John Krasinski's Kids Are Convinced He Works in an Actual Office: 'They Don't Know What I Do'

John Krasinski may be a famous actor, director, and producer, but his kids don't buy it.

The Jack Ryan star appeared on The Late Show with Stephen Colbert Thursday night, where he opened up about how daughters Hazel, 8, and Violet, 6, think he's lying when it comes to his role on The Office, the long-running comedy that launched his career.

In fact, they think he works in an actual office.

"The reason I think that is because I was walking them to school one day, and this guy came up, and he was very nice, he went, 'John! From The Office!,' " he began. "I went 'Yeah,' and he said, 'Oh my God, it's so good to see you,' and I said, 'Thanks' and he kept walking."

After the interaction, Krasinski's daughter said, 'So, you work with him?' I went, 'No.' And she goes, 'Why are you lying to me?' I went, 'I'm not lying to you.' She goes, 'No, he just said he knows you from the office.' "

At the time of the interaction, Krasinski's girls had only seen one episode of the show, which aired from 2005-2013.

"Their actual quote — we were driving — and said, 'So what's the deal with this Office thing?' " he recalled. "I was like 'OK, that's a big question, we'll cut it down in bite-sized morsels.' "

The Late Show with Stephen Colbert

RELATED: John Krasinski Reveals His Daughters' Reaction to Watching One of His Movies for the First Time

Wife Emily Blunt, whom he wed in 2010, suggested showing them the pilot.

"They were watching the first episode and my oldest, it turns out she's a screen talker-atter, so she's talking to the screen," he said. "When Steve fired Jenna, she goes, 'What are you doing?,' then she goes 'You're a horrible person.' "

"'You should meet Steve Carell,' " he yucked.

Krasinski couldn't convince Violet either.

"My 6-year-old kept asking, 'Is that you?' " he remembered. "And she says, 'You look very different.' I was like, 'I'll try not to take that as a hurtful thing,' but sure, I say, 'No, that's me, I was just playing a character, and she went 'Pfff, that's not you.' Then we watched two more episodes, and she goes, 'That's not you. That guy looks a lot like you though!' "

RELATED: All About John Krasinski and Emily Blunt's 2 Kids

ANGELA WEISS/AFP via Getty Images

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Colbert suggested showing her "the Ricky Gervais version," referencing the U.K. version that aired from 2001-2003.

"She'll love that," Krasinski scoffed. "Then she'll be like, 'There's a British version of you?' And I'll be like, 'Yeah.' "

Noting that Blunt had a head start on cool points by playing Mary Poppins, Krasinski admitted, "I think my kids thought she had married me out of charity, like, 'You're so nice to marry an accountant' because they don't know what I do."

The Late Show with Stephen Colbert airs weeknights (11:30 p.m. ET) on CBS.