Glee Christmas Reunion! Lea Michele and Darren Criss Sing Their Way Through NYC for Carpool Karaoke

Glee alums Lea Michele and Darren Criss are marching — and singing! — to the beat of their own little drummer boy.

In a sneak peek at the latest episode of Carpool Karaoke: The Series on Apple TV+, the former drive around New York City singing songs from their respective Christmas albums and sharing some songwriting secrets.

The stars jokingly kick off the video with Criss, 35, saying, "Hey everybody, I'm Lea Michele!" Michele, 36, adds, "No I'm Darren Criss." The duo then go on to wish viewers "a very Happy Holidays" before jumping in to sing Michele's "Christmas in New York."

Michele begins singing the song, and Criss follows suit, hilariously singing off-key midway through the first chorus as he teases Michele, "You're flat. Flat. I sounded better." Criss eventually harmonizes along as the song progresses with the two finishing off on a high note.

The Late Late Show with James Corden/YouTube

Criss then goes on to discuss his own song, "Christmas Dance," explaining: "I'm a fan of a lot of Christmas things so I wrote a little song about all the things that make me wanna dance at Christmas and there's even a little dance in it."

He then asks Michele as the song begins, "Can I have this Christmas dance?" to which she responds, "Yes, you may!"

RELATED: Lea Michele Reveals She Got a Letter from Barbra Streisand — and Called Jonathan Groff Right After!

The two stars dance their way through the song in the car, and there's even a moment where a "frost" filter comes up, turning the car blue and making it appear as if it is snowing.

"Christmas banger!" I Love it!" she says as the song ends. Criss closes out the preview saying, "It's Christmas, it's time to get up and get down."

The Late Late Show with James Corden/YouTube

But those two songs weren't the only ones teased. Two weeks ago, Michele shared a fun TikTok of the duo passionately singing along to "Don't Stop Believing" — one of the most popular covers from their show Glee – while on set filming the Carpool Karaoke episode.

"When you're with your best friend and Don't Stop Believin' randomly comes on the radio," Michele captioned the video, which shows the two banging their heads to the song with the car windows down.

RELATED: Lea Michele's Funny Girl Sets Box Office Record for Broadway's August Wilson Theatre

The two have stayed close over the years following their stint on Glee, with Criss and his wife Mia Swier even showing up to support Michele in Funny Girl on Broadway in October. Michele took over the role of Fanny Brice just a month before in September, stepping into the shoes of the iconic character after Beanie Feldstein's departure in July.

Michele alluded to her Glee character Rachel Berry's obsession with the show (she even performed it on Glee as Berry).

"The amount of times this guy has seen me sing 'Don't Rain on My Parade,'" Michele wrote on Instagram with a backstage photo at the time. "I love you @darrencriss and @miavoncriss."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Carpool Karaoke: The Series is now streaming on Apple TV+.