During a Thursday appearance on the YouTube series Hot Ones, Lowe, 57 — who starred as Sam Seaborn on the Aaron Sorkin drama from 1999 through 2006 — recalled a set visit from Secretary of State Madeleine Albright, plus a time when he thought he had gotten in trouble with the National Security Advisor.
Hot Ones host Sean Evans asked Lowe, "Is there any truth to the internet rumor that Madeleine Albright once crashed the set of West Wing at 3 o'clock in the morning in a robe to accost you guys for being too loud and for not having a secretary of state character?"
The actor laughed and told Evans that the question was "conflating many different urban legends," but confirmed it was true that Albright did visit the West Wing set.
First We Feast/Youtube
"Madeleine Albright absolutely came to the set, very very late," Lowe recalled. "We were shooting at the [John F. Kennedy Center for the Performing Arts]. She wanted to meet everybody."
At the time, Albright was sitting secretary of state, Lowe explained, and he said that the cast was surprised and agreed to meet her.
He then broke down the rest of the rumor, telling Evans, "I was in the White House with Bill Clinton and was summoned to the national security director's office, who was a really gnarly, tough, bare-knuckle dude named Sandy Berger."
NBCU Photo Bank/Getty
Berger worked as President Clinton's National Security Advisor, a job he held from 1997 to 2001.
Lowe told Evans that Berger snapped his fingers and told him, "Sit down," making the actor instantly nervous. At the time, Lowe said he was thinking, "'Oh, I'm f-----. This has got to be about, like, my taxes.' "
But the actor said the meeting took an unexpected turn.
"And he was like, 'Why isn't there a national security advisor on your show?' And I was like, 'This is what this guy's thinking about in the middle of the day?'" Lowe told Evans.
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
While The West Wing concluded with its seventh and final season in 2006, the cast of the hit political drama reunited last fall in an HBO Max special to encourage viewers to vote in the 2020 presidential election.
Lowe told PEOPLE last year that filming the special was "surreal," explaining, "in getting ready to do it, it felt like it had been a thousand eons ago."
"Then when the cameras rolled, and it was just [West Wing costar] Martin Sheen and I sitting down as Sam and President Bartlet, it felt like it was yesterday, and everybody was as good as I remembered them," he continued. "It was like a great, classic rock band that hadn't lost a step."