Austen Kroll Calls Out the 'Most Hurtful' Thing Craig Conover Has Said About Him

Romantic relationships often cause a lot of drama on Bravo's Southern Charm, but it's Craig Conover and Austen Kroll's friendship that has taken the biggest hit this season after Kroll learned about Conover reconnecting with ex-girlfriend Naomie Olindo.

"It wasn't that I was hurt, I was shocked," Kroll, 35, tells PEOPLE. "I thought he would've told me because if he tells me these things, then I can help to mitigate what other people are talking about. I took his word at face [value], and that was clearly wrong. I was curious from Naomie's point of view on what had happened and what she had to say because, obviously, there are three sides."

RELATED: Naomie Olindo on Hooking Up with Ex Craig Conover: 'Neither of Us Thought We'd Get Back Together'

Kroll took particular offense to Conover, 33, previously saying the he didn't think his Pillows and Beer podcast co-host fully backed his Sewing Down South venture.

"I've been supportive of Craig forever," he says, "and for him to try to spin it that I was jealous of him in any sort of way — I mean, are you kidding me?"

Tommaso Boddi/WireImage

Kroll continues, "I'm like, 'Craig, I know that you like this business and pillows, and Shep [Rose] and Whitney [Sudler-Smith] will always laugh at you and say it's girly or whatever, and I've always been like, 'Dude, if you want to do it, then do it.'"

He adds, "And when he was single, I was a champion for it. Because I wasn't the biggest fan of his ex — just like he wasn't the biggest fan of mine. So for him to think that I wasn't supportive is the most hurtful thing that he could have said. For him to call me unsupportive, I think was not factual at all. In fact, that part hurt my feelings."

Kroll points out, "I was at the soft opening. I went and saw the space prior to him opening."

So, he insists, "Don't say for one second that I was not there every step of the way. It's ridiculous. It's such a huge accomplishment. I was always happy for him."

RELATED: Craig Conover Reveals the Most Difficult Chapters to Write in His Book Pillow Talk

The Kings Calling Brewing Co. founder feels Conover only said those things because it endears him to "the female viewership" of the show.

"He knows that when he says it, it gains favor. He knows it," Kroll continues. "It's like, stop it. You know what you're doing. You're pandering so, so hard. You're playing the victim, and I see right through it, Craig."

RELATED VIDEO: Southern Charm's Naomie Olindo Had to 'Beg' to Get Back on the Show After Split from Ex Metul Shah

Kroll also thinks that Conover's priorities shift when he gets into a relationship, such as when and Summer House's Paige DeSorbo became official last year.

"When Craig gets into a relationship, all of a sudden it's him and his girlfriend versus the world. And it's like, It doesn't have to be that way," Kroll says. "Yes, Paige can be your confidante for so many things, but that's why it's important to have someone like me in your life. I don't know what you're doing, but that's why I was very anxious to have that talk with him, which was only four months late."

Charles Sykes/Bravo/NBCU Photo Bank via Getty

Kroll admits, "I really, really was on the fence" about his conversation with Conover (whjich takes place in Thursday's episode).

"I was like, if we show up and we begin to yell at each other and we don't listen to each other, then I can see us both throwing our hands in the air and saying, 'You know what, man, I'm out,'" he says.

Ultimately, "the conversation went very well, which I think surprised us both," Kroll says. "I don't want to have to feel any sort of weirdness between me and my best friend."

RELATED: Naomie Olindo Doubts Cameran Eubanks and Chelsea Meissner Will Ever Return to Southern Charm

As for his own dating life and a possible future with Southern Charm newcomer Olivia Flowers this season, Kroll teases, "There are some ups and downs."

"There are some things that I have to figure out with myself first and foremost, like if I'm ready to date someone," he adds. "And she just moves back to town. I want her to find her footing and not just be an attachment of me, like, 'Oh, that's Austen's girl.' We definitely have that attraction and chemistry together, but you're going to see how we try to navigate this, which is not always the easiest."

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

Southern Charm airs Thursdays at 9 p.m. ET on Bravo.