Gisele Bundchen Instagram Gisele Bündchen and Tom Brady
On Wednesday, a source tells PEOPLE that while the marriage between Brady and supermodel Bündchen, 42, is not "over by any means," the greatest quarterback in NFL history knows he will have to retire permanently from playing after the season to smooth over any issues after he retracted his February retirement announcement just over one month later.
"She doesn't hate that he's playing football, but she sure hates the way he handled the retirement and coming back," a source tells PEOPLE. "He knows that, and he's doing what he can do to smooth things over. He does know that this is his last season if he wants to stay married."
"He can't do this again," the source adds.
"So now all he can do is make this season count, spend time with his family whenever he can, and then retire at the end of the season," the source adds.
On Monday, Brady shrugged off rumors that he already plans to retire after his 23rd NFL season on his Sirius XM show, saying he will simply "take it day by day, and I'll evaluate everything as it comes."
George Pimentel/Getty Tom Brady and Gisele Bündchen
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up to date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
"One of these days, they're going to be right, I will say that," Brady added, referencing those who try to guess his next moves. "One of these days, if you make enough predictions and so forth."
In a wide-ranging recent cover interview for ELLE's October issue, Bündchen shared she would like for Brady, 45, to "be more present" amid his busy football career as she continues to raise son Benjamin, 12, daughter Vivian, 9 and stepson Jack, 15.
"They decided together that he would retire, and then he decided by himself that he'd come back," the source tells PEOPLE about Bündchen's feelings on Brady's return to the NFL. "She said they'd talk about it more, but then he came back anyway, before they had really talked it all through."
"She's a very 'pros vs cons' person, and she sees very few pros to him playing anymore," the source added.