So far six Patriots players -- Martellus Bennett, LeGarrette Blount, Devin McCourty, Dont'a Hightower, Chris Long and Alan Branch -- have confirmed they will not attend the traditional reception for NFL championship-winning teams.
Some of the players have said they will snub the visit as a protest to President Donald Trump, who has regularly described Brady as a friend.
Brady told Pro Football Talk Live on Tuesday that he had no issue with team-mates deciding not to attend the White House.
"Everybody has their own choice," said Brady, who missed the Patriots' White House visit in 2015 hosted by President Barack Obama.
"There's certain years, like a couple years ago, I wanted to go and didn't get the opportunity based on the schedule," Brady said.
"We didn't get told until I think like 10 days before we were going, and at that point I had something I'd been planning for months and couldn't get there."
Brady did attend three other White House celebrations following New England Super Bowl wins -- in 2001, 2003 and 2004 -- during George W. Bush's presidency. Brady also attended a reception hosted by Bill Clinton in 1997 when Michigan won the national college title.
"It really is a great experience," Brady said. "Putting politics aside, it never was a political thing. At least, it never was to me. It meant you won a championship and you got to experience something cool with your team, with your teammates."
Meanwhile Patriots owner Robert Kraft -- also a friend of Trump -- said he was not bothered by players who chose not to attend.
"It's interesting, this is our fifth Super Bowl in the last 16 years, and every time we've had the privilege of going to the White House, a dozen of our players don't go. This is the first time it's gotten any media attention," Kraft said Monday.
"Some other players have the privilege of going (to the White House) in college because they're on national championship teams. Others have family commitments. But this is America; we're all free to do whatever's best for us. We're just privileged to be in a position to be going."