Duff Goldman, Baltimore Ravens Superfan, Says He's 'Good' at 'Trash Talk' — and Has a Perfect Game Day Snack

Michael Tran/FilmMagic

Duff Goldman's love for the Baltimore Ravens truly takes the cake.

The star of Food Network's Ace of Taste tells PEOPLE he became a fan of the team when he was in college, right around the time the Ravens played their first game in Baltimore back in the mid-1990s.

"I went to undergrad in Baltimore right when the Ravens became a team, and I liked their swagger," the 48-year-old chef says.

The Ravens, which had previously been the original Cleveland Browns, famously moved to Maryland after owner Art Modell reached a deal to move the franchise in November 1995. Goldman says he's supported the team from "day one."

"If they win, the sun shines a little brighter," he said. "If they lose, I try not to spiral into depression."

It seems he picked the right team to root for — the club won the Super Bowl twice since moving to Baltimore, in 2001 and 2012. (As of now, 12 out of the NFL's 32 teams have never won the title.)

RELATED: Duff Goldman Gushes Over Wife Johnna and Their Life 'Out in the Woods': 'We Are Insanely in Love'

Goldman — who recently shared his recipe for Rainbow Meringues with PEOPLE — says that even though he now lives thousands of miles away from Baltimore in California, he still catches games on TV or with The West Wing, a group of Ravens fans in Los Angeles.

His favorite Ravens memory is when he went to a game with Hall of Famer Jonathan Ogden, who took him to the players' parking lot, where he was able to meet Ravens stars Steve Smith, Justin Tucker, and Ronnie Stanley.

"That was fun," Goldman recalls. "My wife has a crush on Justin Tucker, so there were mixed emotions. Just kidding!"

Duff Goldman/Twitter Duff Goldman and the Ravens' Haloti Ngata teamed up to make a Ravens cake with Charm City Cakes in 2009

RELATED: Former NFL QB Kurt Warner Says Tom Brady, Aaron Rodgers Look 'Exhausted': 'It's OK to Walk Away'

And if you find yourself seated next to Goldman at a game, don't be surprised if you're subjected to a little "trash talk."

"I'm good at it," he admits.

RELATED VIDEO: Pastry Chef Duff Goldman Shares His Must-Have Kitchen Items For Baking While in Self-Isolation

So what does a celebrity chef serve on game day?

"Crab cakes!" he says. "Baltimore has the best crab cakes, so I'll order some from Jimmy's Famous in Baltimore and have them shipped to L.A."

The Ravens will be fighting to remain in the playoffs when they face the Cincinnati Bengals in the Wild Card Round on Sunday — and Goldman is confident they may be Super Bowl-bound.

"Of course," he says. "We should win every year and go undefeated!"