German midfielder Kramer came off the bench to cancel out Alexandru Maxim’s 51st-minute opener for the visitors and deny Armin Veh a first win in his second spell as Stuttgart manager.
“Getting a point was hugely important,” M'gladbach manager Lucien Favre said. “It’s a huge relief.
“In the first half, we didn’t use the ball intelligently with patience and pace to make them tired,” “We did that a lot better in the second half.”
Stuttgart’s Antonio Ruediger had a goal rightly disallowed on 32 minutes before Raffael headed off target for Gladbach.
The home team then wasted an even better chance three minutes into the second half when Branimir Hrgota sliced wide with the goal gaping.
Stuttgart soon made Gladbach pay. Christian Gentner helped the ball on to Maxim in the area and the Romanian midfielder volleyed past Yann Sommer from the corner of the six-yard box.
Hrgota had a goal ruled out for offside before Gladbach sub Thorgan Hazard thumped against the post with 16 minutes left.
But with time running out, Alvaro Dominguez found the unmarked Kramer in the box and he kept his cool to fire into the corner.
“We played really well for a long time and then got a bit tired,” said Veh, who led Stuttgart to the German title in 2007. “Overall, Borussia deserved the point.”
Earlier, Paderborn were denied a win on their Bundesliga debut after Koo Ja-Cheol scored a penalty four minutes into injury time to earn Mainz a 2-2 draw.
Shinji Okazaki put Mainz in front after 33 minutes with his 26th goal in the German top flight – equalling the record for a Japanese player.
Paderborn’s Elias Kachunga smashed home four minutes later to bring the home side level and Uwe Huenemeier’s header put them ahead with three minutes left.
But with time almost up the Paderborn captain conceded a penalty, which South Korean midfielder Koo calmly stroked into the corner.
“Towards the end we had chances to keep the ball better,” Paderborn manager Andre Breitenreiter said. “We were bitterly punished for not doing so.”
Mainz, who finished 7th in the top flight last season, took the lead when Johannes Geis saw a dipping free-kick come back off the post. Okazaki, who top scored for Mainz with 15 goals last season, followed up to score from the rebound.
The home side equalised three minutes later. Mainz midfielder Geis passed straight to Paderborn’s Kachunga on the edge of the area and he hammered the ball past visiting keeper Loris Karius.
Paderborn continued to press after the break and they moved in front when Marvin Bakalorz picked out Huenemeier with a brilliant cross field pass. The defender’s header looped into the net despite Karius getting a hand to it.
Okazaki headed against the bar in the last minute but there was another twist when Huenemeier barged into the forward and Koo’s spot kick salvaged a point for Mainz.
On Saturday evening, Bayer Leverkusen's Karim Bellarabi hit the Bundesliga's fastest ever goal as title hopefuls Borussia Dortmund lost 2-0 in their opening game.
The Leverkusen attacker struck after just nine seconds and then set up Stefan Kiessling in injury time.
On Friday, goals from Thomas Mueller and Arjen Robben helped defending champions Bayern Munich open with a 2-1 home win over VfL Wolfsburg.