Tom Brady on Wife Gisele Bündchen's Role at Home with Their Kids: She's 'Held Down the House'

Gisele/Instagram Tom Brady, Gisele Bündchen and family

Tom Brady is speaking out about wife Gisele Bündchen and her focus on their family.

In Monday's episode of his SiriusXM podcast, Let's Go! With Tom Brady, Larry Fitzgerald and Jim Gray, the athlete, 44, opened up about how he and his wife balance their careers while being parents to three kids.

Brady — who shares daughter Vivian Lake, 8, and son Benjamin Rein, 11, with wife Gisele Bündchen, plus son John "Jack" Edward, 14, with ex Bridget Moynahan — says his wife, a former supermodel, has "held down the house for a long time now" as he continues to build his NFL career.

"And I think there's things that she wants to accomplish," he continues. "You know, she hasn't worked as much in the last 10, 12 years just raising our family and kind of committing to being in a life in Boston and then moving to Florida."

"But that's an issue, and it's a very difficult issue to reconcile without just saying, 'Hey, it's time to retire.' And I think there's, you know, we're coming to the end here too, so I don't want to miss any of the kids' stuff," Brady adds.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

RELATED: Tom Brady Says He Wishes He Was 'There More' with His Kids amid NFL Career: 'I Try to Do My Best'

Elsewhere in the episode, the Tampa Bay Buccaneers discussed how he makes time for his three kids amid his busy sports schedule.

Brady says that as a football player, he gets a "good healthy off-season" and is able to have "a lot of time off" to spend with his family.

"I try to do my best with the kids," he says. "Although I wish I was there more, but I think if I was there too much, you know, they might be sick and tired of me screwing up everything that's been going on in the house for a long time too."

Earlier this month, Brady shared that while he could continue his football career "until I'm 50 or 55" if I wanted to," he most likely won't play that long because of his family.

"... My physical body won't be the problem," he said while chatting with Amazon Prime Video's Andrea Kremer and Hannah Storm during the Thursday Night Football face-off against the Bucs and the Philadelphia Eagles. "I think it'll just be, I'm just missing too much of life with my family."

At age 44, Brady is currently the oldest player who is still active in the NFL.