Horford finished with 14 rebounds and three assists after the Celtics built an early first quarter lead before holding on to claim a vital win at the TD Garden.
The injury-plagued Celtics now have the opportunity to clinch the NBA Eastern Conference first round series in game six in Milwaukee on Thursday.
Boston had led 23-15 at the end of the first period and held a 48-37 advantage at half-time.
Milwaukee responded strongly after the interval but were unable to overturn their first-half deficit despite rallying to outscore the Celtics 26-20 in the final quarter.
Boston's fans were made to sweat through a nervy final few seconds, when an Eric Bledsoe jumper put the Bucks within three points at 87-84.
But the Celtics hit back with two Terry Rozier free throws to put Boston 89-84 ahead, giving the hosts a five-point cushion they defended through to the buzzer.