Southern Charm 's Shep Rose Claims Taylor Ann Green Was 'Always Worried' About Him Cheating

The cast of Southern Charm experienced some rocky waters this week.

Thursday's episode picked up at Madison LeCroy and Venita Aspen's joint birthday garden party, where Venita didn't appreciate Madison inviting Kathryn Dennis and Olivia Flowers.

"I want to make sure at my party I'm good with everybody at the table," Venita said.

Venita and Olivia talked one-on-one, and Venita explained that she didn't like how Olivia walked away from their conversation at Naomie Olindo's party. "Because I heard you throw out the word 'racist' and I'm not about to stick around for that," Olivia told Venita.

Venita wanted clarity. "So you're not about to stick around for it because you're racist or you're not racist?" she asked Olivia.

RELATED: Paige DeSorbo Tells BF Craig Conover It's 'Inappropriate' To Hang Out with His Southern Charm Ex

Bravo

Olivia said she felt "uncomfortable hearing that word so easily tossed out in a conversation that involved me."

"You and I have known each other five minutes, so when I hear that word thrown out, I don't even know what the reason was you said it, I'm checking out because that not okay to throw that disgusting … you're not going to cancel culture me out on this," Olivia continued.

Venita said that if she had that mentality, Kathryn wouldn't be around.

"You will never understand as a Black individual what it's like to have Kathryn say those things and then to be in her presence," Venita said, referencing when Kathryn was accused of racism for using a monkey emoji in an interaction with a Black radio show host. (She apologized for the racially insensitive incident at the time, acknowledging her actions were "offensive.")

Bravo

Olivia pointed out, "I have nothing to do with that." Still, Venita wished, Olivia would "address it instead of walking away."

They ultimately agreed to put a pin in their conversation and rejoin the party. "That didn't get us anywhere," Venita said.

POST-PARTY RECAP

Olivia later recounted what went down at the party to Austen Kroll over drinks, explaining how Madison treated the event like her engagement celebration.

"Surprise, she's like, 'Wait, wait, make it about me, me, me!'" Austen said.

Olivia quickly realized that she should refrain from bringing up Madison to Austen, who is her ex. "I hate that l our s--- has so much to do with Madison," Olivia said to Austen.

Tommy Garcia/Bravo; John Valkos/Bravo

Austen assured "it doesn't" and explained why talk of Madison elicited such a strong response: "I was so vulnerable with my last relationship, it was like emotional whiplash."

Austen told Olivia he wanted to take their relationship slow, but Olivia told the cameras she thought "he's overcomplicating the part that should be simple."

Venita had Leva Bonaparte over for tea and they also debriefed about the garden party. "I wouldn't say I was the best version of myself at the garden party, but I was the most open that I'd ever been," Venita said.

Leva then told Venita she thought the influencer needed to have "a super honest conversation" with Olivia. "I just want it to get to a point where like she says her piece about how whatever I said has made her feel, I say my piece about how what she's done has made me feel and then decide where is this going?" Venita said.

RELATED: Craig Conover Dishes on His Relationship with Paige DeSorbo Formed on Winter House

CHAOS IN CHARLOTTE

Craig Conover and Shep Rose accompanied Austen to Charlotte, where he had a meeting about expanding the distribution of his Trop Hop beer. When they arrived at the hotel, Shep FaceTimed then-girlfriend Taylor Ann Green (PEOPLE confirmed their split on Wednesday) and let her know that he, Craig and Austen planned to go bowling later and then hit up a few bars.

In response, Taylor told Shep to "be good."

"What am I going to do?" Shep asked Taylor. "Are you crazy?"

Shep realized Taylor felt anxious about him straying due to a previous transgression. "She definitely is skeptical of our time apart and that's because of the past," he said in an on-camera interview.

Still, he told Taylor, "I'm not going to stop going places and doing things" just because he was in a relationship.

"I will never be that woman that's left because of a wandering eye," Taylor said.

After hanging up with Taylor, Shep met Craig for a drink and told him about the conversation. "She's always worried," Shep said. "I just sensed her angst, and it happens every time I go somewhere."

Craig understood Taylor's concerns. "That's always going to be a consequence of scars of infidelity in a relationship," he said.

Taylor Ann Green/Instagram

RELATED: Southern Charm's Shep Rose and Taylor Ann Green Split: 'He Refused to Commit,' Source Say

Shep didn't want this to be an ongoing issue in his relationship. "If it's going to be a problem every single time, then there's going to be a problem," he stated.

Craig told Shep he needed to regain Taylor's trust. "If I haven't already, then I never will," Shep said.

Austen arrived with good news from his meeting with the beer distributor. "Drinks are on me because I just sold a s--- ton of beer," Austen announced.

Then they headed out bowling and Shep sent Taylor a selfie with the guys to prove he was having wholesome fun. But Taylor responded with long text messages. "Please, I'm begging, don't be defensive towards me," she wrote. "I have severe anxiety about how I've acted."

Afterwards, Shep headed the bar and flirted with a woman there, even offering to buy her a drink.

Never miss a story — sign up for PEOPLE's free weekly newsletter to get the biggest news of the week delivered to your inbox every Friday.

Back in Charleston, Kathryn and Madison went out with some girlfriends and newly-single Kathryn proposed a toast to "that new new."

"Let's flirt and be weird," she said to start the night.

Thursday's episode aired one day after PEOPLE exclusively reported that Shep and Taylor had broken up after two years of dating. According to insiders, problems in their romance came down to Rose's disinterest in being faithful.

"Shep has a real issue with monogamy and refused to commit to Taylor, telling her that he doesn't want to be with just one person and change his lifestyle," one source said.

"She's incredibly hurt, but she knows that she deserves a guy who will fight for her, not one who is so selfish," added another. "He'll regret this."

Southern Charm airs Thursdays (9 p.m. ET) on Bravo.