Rob Lowe Tells Drew Barrymore He Thinks His Dad Once Hooked Up with Her Mom: 'I Wouldn't Doubt It'

Rob Lowe thinks his dad and Drew Barrymore's mom might have linked up years ago.

The actress is Lowe's next guest on his Literally! With Rob Lowe podcast on Thursday, and in a clip shared exclusively with PEOPLE, the stars reflect on memories from the Los Angeles hot spot Helena's back in the 1980s. Barrymore, 47, recalls frequenting the star-studded club with her mom Jaid, now 76.

Lowe tells Barrymore he has "suspicions" that his father Charles and Jaid, as Barrymore puts it, "ended up together" one night at the club. "I wouldn't doubt it," she says, as Lowe agrees, "Neither would I."

"I kind of like the thought of it, I have to say," he continues.

Says Barrymore, "Literally, my mom was fun, is fun. She was a good-time gal." Adds Lowe, "My sense is, as mothers go, she was in her wheelhouse at Helena's."

"She and I used to go to Helena's, I feel, like multiple times a week. You know how people hit the gym a few times a week? We hit Helena's a few times a week."

RELATED: Drew Barrymore Discusses Mending Relationship with Her Formerly Estranged Mom: We're 'Good Now'

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

Barrymore also recalls seeing Sean Penn and Madonna there "all the time," as well as Jack Nicholson, whom she would go on to party with years later outside of the nightclub. Lowe called Helena's "amazing" and noted the "disparate, insane, high level of people that were there."

Team Coco Podcasts

RELATED: Rob Lowe on His Wild Journey from Teen Idol to Sober Family Man: 'I'm Grateful for All of It'

"Jack Nicholson was a fixture [at Helena's]," Barrymore says. "He went to Helena's as many nights a week as me and my mom. And funny enough, I ended up, like, partying with him a lot years later at his house. It was never inappropriate. We would just party and talk and hang out. I love Jack. But I feel like I made friends with him at Helena's, which then matriculated into, a decade later, us, like, partying at his house."

The Literally! With Rob Lowe podcast is on the Team Coco podcast network.