In PEOPLE’s exclusive sneak peek at Thursday's episode of Southern Charm, Austen, 33, admits that "it hurt" when a guy Madison, 31, once slept with came to the party.
Though she and Austen were broken up when the one night stand happened, his appearance at the party led Madison to suspect that Shep, 40, had set the whole thing up to create conflict. (The two famously didn't get along last season.) Shep, however, insisted it was just a coincidence.
"I think it kind of sucks that obviously the Peter thing came out," Madison tells Austen the next morning. "But you can't let what people say affect how we are right now, because right now we've been doing so well."
"I mean, yeah, it hurts," Austen replies.
"Listen," Madison continues. "People are going to continue to try to break this relationship up and it makes me sick."
Though Austen acknowledges that he can't hold the one night stand against Madison because they weren't together at the time, he's worried that Shep is trying to sabotage their relationship.
"It just sucks because you and I made a choice to come back to each other and let it all go," he says. "I mean, I can't crucify you for something that happened when we weren't officially together. I just don't want Shep creating a wedge [between us] — that can't happen."
"I just want all this to stop," Madison says before Austen adds of Shep, "I hate that he has knowledge that I don't."
"Don't you understand that when these things come out, they're like, testing to see how strong we are?" Madison tells him. "We've been through enough s--- that you can't break it, and I don't want to be with anybody but you."
RELATED VIDEO: Austen Kroll on Shep's Girlfriend, 3 Cast Members' Exits and Pitching Friends for Southern Charm
Austen Kroll On Shep's Girlfriend, 3 Cast Members Exit & Pitching Friends for Southern Charm
PLUS: Austen Kroll reveals that he would not be doing Instagram Live with Madison if they were "just good friends"
Austen and Madison have had a whirlwind relationship, often breaking up and then coming back to each other. Last November, he opened up about the on-off romance during the Southern Charm cast panel at BravoCon, saying the two were "doing well" at the time.
"This week I'm doing well with Madison," he said last year. "Maybe next week that will change, but I'll let y'all know."
"We've tried to go our separate ways," he added. "Madison has been a person for me for a long time and it's really tough to walk away from something like that. She's a hell of a woman and I find it hard to walk away from."
Southern Charm airs Thursdays (9 p.m. ET) on Bravo.