Kevin Bacon on How He and Kyra Sedgwick Recovered After Losing 'Most of Our Money' in Madoff Scheme

Amanda Edwards/WireImage Kyra Sedgwick and Kevin Bacon

Kevin Bacon is opening up about losing "most" of his fortune in Bernie Madoff's Ponzi scheme.

On Monday's episode of the Smartless podcast, hosted by Jason Batemen, Will Arnett, and Sean Hayes, Bacon revealed that he and his wife Kyra Sedgewick, "had most of our money in Madoff."

The 64-year-old actor admitted there's likely something to be learned in hindsight.

"There's obvious life lessons there –– if something is too good to be true, it's too good to be true," he said.

Madoff's scheme is considered to be the largest Ponzi scheme in history, worth about $64.8 billion.

Bacon told the hosting trio that Sedgwick, 57, and he had a "portion" of their losses returned, but not all. In the wake of the scheme blowing up, the Madoff Victim Fund has been working to restore losses to those affected. So far, 40,454 victims have received assistance and payouts now exceed $4.078 billion.

Mario Tama/Getty Bernie Madoff

RELATED: Kevin Bacon and Kyra Sedgwick's Relationship Timeline

"Certainly, you get angry and stuff, but I have to say, there were a lot of people who were much worse off than we were – old people, people whose retirement funds were completely decimated," the A Few Good Men actor said.

"So there's always going to be somebody that's going to have it a lot worse than you."

As they dealt with the fallout, Bacon and Sedgwick focused on what they have as a family — not what they lost. "When something like that happens, you look at each other and you go, 'Well, that sucks, and let's roll up our sleeves and get to work,'" the Tremors star shared. "

RELATED VIDEO: Kevin Bacon Hasn't Done Footloose Dance in Years: 'I Couldn't Believe How Challenging It Was'

We've made it this far, our kids are healthy, we're healthy, you know? Let's look at what we have that's good. We can still both work."The actor then got real with the hosts. "I think people will be not happy to hear me whining about money," Bacon said.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up to date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Madoff was arrested in December 2008 on allegations that his company, Bernard L. Madoff Investment Securities, had swindled billions of dollars from individuals, businesses and charities.

The disgraced financier died in April 2021 at the age of 82 while serving his 150-year sentence.