Craig Conover Says He's Down 20 Lbs. Since Returning to the Gym: 'I Was Eating Everything'

Charles Sykes/Bravo/NBCU Photo Bank via Getty

Craig Conover has been on a fitness journey that he says has left him in the best shape he's been in in years.

The Southern Charm star, 33, opened up to PEOPLE over the weekend about what got him back in the gym, revealing that he's down 20 lbs. since he started focusing on his health again.

"I'm f---ing fit, more fit now than I have been in years," Conover says at Mohegan Sun's final Toast with the Host event at their Ultra-Luxe Lounge. "And I'm finally excited to take my shirt off again instead of hiding it on the show."

"I got so many compliments [when I was filming] Summer House last week," he adds. "Mya [Allen] was like, 'I've never been attracted to you and like, you look fantastic.' And Mya's tough!"

Conover says that his weight fluctuated over the years, especially as he stopped taking Adderall. The reality star, who was a founding cast member on Southern Charm when it premiered in 2014, has been open about his addiction to the ADHD medication during the early seasons of the Bravo show, and says that he began putting on weight after giving it up in 2018.

"Adderall Craig, at my heaviest, was 165 lbs.," he says. "When I wasn't addicted to amphetamines, I hit a healthy weight of like, 185 lbs."

RELATED: Craig Conover Opens Up About the ADHD Meds 'He Should've Stopped Taking After Law School'

Rich Polk/Getty

RELATED: Tom Arnold Reveals How He Lost 75 Lbs. After Suffering Mini Stroke: 'I Made Exercise a Ritual'

But Conover put on what pal and Southern Charm costar Austen Kroll jokes was "too much healthy weight."

"What is it called? Cushion?" teases Conover. "But Austen's right. I was unhealthy. I was eating everything. I was eating a lot of ice cream! And I hit the scale one day and I was 205 lbs. I started seeing my face and my neck. You can see it on Southern Charm this year. I'm big on [season 2 of] Winter House, too, which you'll see when that comes out. I had it on my neck and my waist. So when I started buying bigger pants, I knew I had to do something to change."

Change meant getting to the gym every day despite his disdain for it.

"I made a decision one day that I was going to go back to the gym no matter how much it sucked," he tells PEOPLE. "And it did suck. I was like, 'How the hell did I let myself get out of this habit?'"

With the help of a trainer — Brian Albrecht, of Swolly Mammoth — and workout buddies that include Kroll and their Southern Charm costar Shep Rose, Conover got into a routine that's worked for him.

Not only has he lost weight, but going every days has improved other areas of his life. "I've made more money since I started going back to the gym than I ever have," he says, "because I'm going to the gym in the morning and just calling people all day."

RELATED: Craig Conover Says Girlfriend Paige DeSorbo Is Moving to Charleston with Him for the Summer

Courtesy of Mohegan Sun

He's also learned things about himself he didn't previously know, like the fact that he has what he calls "dormant depression."

"I never saw it before," he said. "I used to take Adderall as a happy pill. Now, if I miss the gym for a few days in a row, I feel my depression creeping up on me. It's really helped me stay on track. I've tried to figure out a way to articulate it to people. I'm like, 'Guys, this isn't right or wrong, but if I wake up and go to the gym, as hard is that is for me — and I know that sounds silly — but I have a good day. If I don't, I just f---ing order food and just lay on my couch and watch movies."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

The only thing Conover struggles with now? Balance.

"Balance, I've never found," he says. "I don't have a balance. No matter how many people want to tell me that there's balance, for me there's not. I'm ether not doing s--- or I'm doing well. There's no middle ground. One day I hope to find balance, but until then, I just have to keep momentum going in a good way."