“VPR”’s Jax Taylor Teases a Return to “House of Villains” Following Elimination: ‘I'll Be Back’ (Exclusive)

"You think the No. 1 guy's going to go that easily? I don't think so,” the 'Vanderpump Rules' star tells PEOPLE

Vanderpump may rule, but not when it comes to being a villain, apparently.

Jax Taylor got voted out by his House of Villains castmates on Thursday’s episode, making him the season’s first eliminee. But the Vanderpump Rules alum, 44, claims viewers haven’t seen the last of him.

“You think the No. 1 guy's going to go that easily? I don't think so,” Taylor tells PEOPLE, referencing one of his most memorable VPR quotes. “I'm not going anywhere. I'll be back. I'm definitely making a comeback. Don't worry.”

Related: House of Villains' Shake Gets Busted Trash-Talking Jax Taylor Behind His Back: 'Are You Dumb?' (Exclusive)

Taylor, 44, thinks his costars — who include The Challenge’s Johnny “Bananas” Devenanzio, Love Is Blind’s Shake Chatterjee, Survivor’s Jonny Fairplay, The Apprentice's Omarosa Manigault Newman and Flavor of Love’s Tiffany “New York” Pollard — targeted him because of his athleticism.

“It was 100 percent, that was the reason,” he says. “I am a physical threat. I dominate them with everything I do. I'm athletic. I'm an alpha male. I dominate them with everything I do. It's between me and Johnny Bananas as far as the dominant male in the house. And that's totally fine.”

Manigault Newman, 49, put Taylor up for elimination alongside Chatterjee, 36, and The Bachelor alum Corinne Olympios.

“Omarosa wasn't just trying to get me out,” Taylor claims. “Omarosa is a very big character; her and New York, they're characters. They're playing off script. They want to look like these ominous figures. It's all for show. She's a great person. they're all great people, except for one.”

Taylor, of course, means his House of Villains nemesis: Chatterjee.

“I don't like the kind of person he is,” the Bravo vet says. “He's a very immature person. And I have to understand, I was that immature, too. He's got a lot of growing up to do. He's very entitled and I think he was very, very lucky to be on that show considering the people that he was on the show with. Everybody that was on the House of Villains are true icons. I don't consider him an icon.”

Taylor thinks Chatterjee lucked his way onto the show. “I consider him doing something stupid on TV and it got him lucky,” Taylor continues. “He was kind of like the lottery winner out of everybody.”

On House of Villains, Chatterjee stated that he didn’t think Taylor needed the $200,000 prize money because of his other businesses and partnerships. Taylor says the presumption came from a conversation he had with Chatterjee in which the Netflix star asked about his career path.

“He was Googling everybody at night trying to figure out what everybody made for a living. And he's like, ‘You do really well for yourself,’” Taylor claims of Chatterjee. “I wish I would've met somebody like me when I started in reality TV,  so I was trying to give him my playbook. I was saying, ‘I've been doing this for a decade, but I've had to hustle my butt off. This stuff didn't just happen to me. You have to work for it. Being on reality TV is a blessing, but they've given me a platform and then you have to do the rest. You have to build a brand.’”

Related: Shake Chatterjee Teases House of Villains Feud with Jax Taylor: 'I Had to Get an Edge on Him' (Exclusive) 

Taylor says he got a sense that Chatterjee “was struggling” and “doesn't really like his job” as a veterinarian and a loan officer.

“He doesn't make a lot of money, living in an apartment and he was venting to me a little bit,” the dad of 2-year-old Cruz says. “I sensed a little vulnerability there, so I thought, I'm going to be a little vulnerable too. I thought, ‘I'll give you some advice, so you don't make so many bad decisions.’ And he took it as, ‘He's cocky, conceited, he doesn't need this.’ And when I heard that, I was like, ‘You son of a bitch. You used that against me.’ I went into the House of Villains being the nicest guy there and I screwed up.”

Taylor showed even more vulnerability when he teared up after Pollard, 41, called him a “demon.”

“I took that to heart,” he says. “That's the first time I've ever gotten emotional on television. That was a real tear. I was like, ‘You don't even know me. You've met me five minutes ago and you're calling me a demon.’”

Taylor and Pollard came to an understanding at House of Villains’ premiere party earlier this month, but the situation made him realize the difference between his background and those of his castmates.

“I come from a reality show that's actually following our lives, not these shows that try to be other people and stuff like that,” Taylor says. “I didn't figure out how her shtick is. I shouldn't have took offense to it, but I did. But I didn't realize until after I left the show who these people are and what they do and their games. Because after the cameras are gone, they're not the same people. I am the same person.”

The When Reality Hits podcast co-host says he “was super intimidated” by some of his female costars. “They're so good at what they do, and I was just in awe,” Taylor says. “Tanisha, Omarosa, New York, those women are the scariest human beings to be around. They are scary, in a good way, but they're scary. You just definitely want to be on their good side.”

Taylor admits he “should have learned a little bit more” about how his castmates’ shows work, because he felt at somewhat of a disadvantage when it came to going against people such as Devenanzio, 41, and Fairplay, 49, who come from competition shows.

“I did not do my research at all,” Taylor says. “I should have watched Survivor, Big Brother, MTV’s The Challenge.”

Taylor has also been out of the reality TV game for a bit. He left VPR in 2020 and calls the time away “refreshing.”

“I've been on reality TV for almost a decade, and I just needed a break,” Taylor says. “Vanderpump Rules is a very intense show. I can pretty much take anything, but even the toughest guys like myself could get worn down a little bit. I needed to recharge my batteries. I need to get in touch back with my wife. I needed to have a baby. And I just needed to get back to my normal self.”

Related: 'House of Villains': Jax Taylor Threatens to Go 'Back to Jail' Over Shake Chatterjee Feud and 5 Celebs Make Surprise Cameos

These days, Taylor doesn’t consider his normal self a bad guy.

“That villain guy, it just is not as there used to be,” he says. “I got a wife now, I have a kid, so I have to work hard to be that ‘asshole.’ I needed that guy for this, and he just wasn't there. It was a blessing to be in that house with those legends, but I was definitely out of my element on that show.”

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

House of Villains airs Thursdays at 10 p.m. ET on E! and can be streamed on the E! app.

For more People news, make sure to sign up for our newsletter!

Read the original article on People.