Hasan Minhaj Defiantly Takes On Celebrity Jeopardy! Backlash: 'Fans Weren't Having It'

Tyler Golden/ABC

Hasan Minhaj had a few things to say about the response he's received after appearing on Celebrity Jeopardy!

The Patriot Act host confronted the intense backlash from his appearance on the long-running game show during a sit-down with Jimmy Fallon on Thursday's episode of The Tonight Show.

"This past week I was on Celebrity Jeopardy! and now fans of Jeopardy! hate my guts," Minhaj, 37, told Fallon, 48. "I have this unique power to piss off dedicated nerds."

Handily, Fallon's crew had crafted a supercut of Minhaj's most animated moments — actions that led some Twitter users to deem "annoying" and "painfully unfunny." One critic went so far as to call Minhaj the worst contestant in Jeopardy! history.

"I was just playing with vigor and passion," Minhaj told Fallon.

"I was into it," Minhaj said after the clip played. "If you get it right, hang on the rim."

But, he acknowledged, the "fans weren't having it."

Minhaj did single out one comment as especially tough to take — the slam that he was "the most annoying contestant" in the show's history.

"That one hurt because, you know, the show's been on for over 58 years — and they've had 15,000 contestants on the show. And Dr. Oz was a contestant," he said.

The comedian — who'd told host Mayim Bialik during his appearance in the tournament that he first connected to Jeopardy! while watching with his father — then quipped, "Maybe my dad was wrong. He said I'd never break records on that show."

RELATED: Comedian Hasan Minhaj Explains Why He Corrected Ellen DeGeneres's Pronunciation of His Name

Santiago Felipe/Getty

Minhaj then set the stage for an apology to Jeopardy! fans — though it didn't go as expected.

"To the fans of Jeopardy!," he began, reading from a notecard.

"I want to take this moment to apologize. Jeopardy! fans, look at me. I'm sorry I desiccated an American institution. I'm sorry I ripped your 7 p.m. linear TV pacifier out of your geriatric mouth. And most importantly, I'm sorry for trying to make Jeopardy! fun."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

The Celebrity Jeopardy! tournament continues — without Minhaj — Sunday at 9 p.m. ET on ABC.