Wes Bergmann Was Hoping The Challenge: All Stars Season 3 Would Be 'a Vacation ... I Was Wrong'

Laura Barisonzi/MTV ENTERTAINMENT/Paramount+

With two Challenge wins on his résumé, Wes Bergmann heads to All Stars season 3 to try to pick up another victory.

"I signed up to go on a vacation, and then when I showed up it was all the best guys that had ever been on the show in one group," Bergmann, 37, tells PEOPLE. "I was like, 'This feels a little bit like a bait and switch.' You look at the competition and I have an easier chance of just waiting until [The Challenge season] 38 and winning it than I do of winning All Stars."

The new installment of Paramount+'s All Stars requires past contestants to have qualified for or won at least one Challenge final. The cast, which Bergmann calls "very intimidating," includes heavy hitters like Brad Fiorenza, Derrick Kosinski, Mark Long, Jonna Mannion, Veronica Portillo, Darrell Taylor and Jordan Wiseley.

RELATED: The Challenge Heads to CBS with Global Tournament Featuring 'Reality Titans'

The producers delivered especially tough daily challenges for this fierce batch of competitors.

"This was as big of a budget and as creative of a shoot as they've ever done," Bergmann says. "Every day that you show up is another day that you could die. The Challenge never rests."

Laura Barisonzi/MTV ENTERTAINMENT/Paramount+

Bergmann considers season 3 participants Nehemiah Clark, Melinda Collins and Fiorenza, 41, his "squad" and thought he'd be off to a solid start heading into the game with a group of allies.

"But anytime you have a good start, the rest of the eyes are kind of looking at you," Bergmann says. "I knew there was going to be somewhat of a target on my back, but I find out very quickly that there wasn't a target on my back — I was the only one where I was in literally everyone's crosshairs, including friends to a certain extent. If I knew on paper exactly how badly the mass majority of that cast didn't want me there, I would have never come."

With many rookies targeting him on recent seasons of MTV's The Challenge, Bergmann "was looking for a break from that, while still getting to scratch my Challenge itch," he says of choosing to go on All Stars. "I'm so tired of being public enemy No. 1 or 2 on the flagship show. I thought this was going to be different, and I was wrong. It's obvious that I'm going to have a hard time winning either show."

RELATED VIDEO: The Challenge's Leroy Garrett Opens Up About Camila Nakagawa's Racist Outburst on Dirty 30, MTV Apologizes

Bergmann thinks other competitors target him out of "jealousy and insecurity."

"More propaganda is pointed toward me than any other Challenger in the history of the show," the reality star says. "And I'm just sitting there like a redheaded angel, never doing anything wrong, just constantly bullied by everybody. But it makes me the star of the show, so it's a self-fulfilling prophecy on their end. Without guys like me, there would be no show."

The Real World: Austin alum doesn't even think he should qualify as one of the show's biggest threats. "I'm an average-size person that's also aging. And as far as I'm concerned, I should be in the middle of the pack. Like, how old do I have to get before maybe you just consider me one of the people in the crowd?" he wonders. "I'm up there just because I'm wily."

Riccardo Giardina/MTV Wes Bergmann

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Despite finding himself a major target even among a group of vets on All Stars season 3, Bergmann admits he didn't try to get on anyone's good side.

"I don't think it's much of a spoiler to say that I didn't make amends with my rivals, but instead, just made it worse," he teases. "I do that because I love making their lives miserable, but I also know that the fans love to watch me make their lives miserable. So I did it for everybody."

Bergmann's pestering goes beyond the show, too. Even though his top frenemy Johnny "Bananas" Devenanzio doesn't appear on this season of All Stars, Bergmann promises, "I'm going to bully him for the rest of my life" because "we're family."

"I'm going to make his life miserable as much as possible, within the confines of the law, for the rest of my life," continues Bergmann, the producer of the competition series The Blox. "But no one outside of my family is allowed to bully him, because he's my family. He's just the most hated member of my family. There's a lot of respect between me and him, but g-- d---, he is so annoying."

But Bergmann prefers to stay out of the online back and forth that takes place between cast members between filming the series.

"I am such an asynchronous troll — and I'll have to define asynchronous for my colleagues — which is, I pop in, I see something, I think through the lens of a comedian for a couple of minutes, I'll leave a comment and I'll bounce," Bergmann says of his approach to social media engagement. "The reality of it is I've muted all of them. I follow no one on Twitter. I just pop in to troll them on the main accounts every once in a while, to tell some good jokes on Twitter, and then I forget about them."

MTV

RELATED: MTV's The Challenge Is Launching Its First Workout Series Featuring Tori Deal, Mark Long and More

Even though Bergmann likes to distance himself from most of his costars and feels "I have more of an uphill battle on every season that I show up," he says he plans to continue doing The Challenge "when it lines up with my schedule."

"Every single decision that I make in business and in life always has the caveat of, if I choose this, does that make it hard for me to fit a Challenge into that window of time?" Bergmann explains. "I do my best to always make it an option, and I will continue to do so for years. All Stars made me fall in love again with the whole franchise and the whole game, and it relit my fire that was growing dim. It is a hobby that I want to do every time."

The Challenge: All Stars season 3 premieres on Paramount+ this Wednesday.