Simon Cowell Has ‘Rubbed a Lot of People the Wrong Way’ at AGT, But He's Rarely Told 'No': Source

Gabrielle Union isn’t the only America’s Got Talent employee who had grievances with judge and executive producer Simon Cowell.

In the wake of Union’s controversial and unexpected firing from the NBC reality competition series, a source tells PEOPLE that Cowell, 60, “runs the show” and gets what he asks for a majority of the time.

“It’s true that America’s Got Talent is a boy’s club. Simon runs the show and as long as he’s there, he will continue to do so,” says the source.

“Everything goes by him and he’s rubbed a lot of people the wrong way,” the source explains, adding that people who work on the show will rarely “tell him, ‘no.’”

PEOPLE is out to Cowell’s rep for comment.

On Monday, Howard Stern, who previously served as a judge on AGT for four seasons from 2012 through 2015, hinted at Cowell’s allegedly questionable behavior on the show, both slamming and accusing him of having “orchestrated” Union’s firing.

RELATED: Gabrielle Union Speaks Out After NBC Says It’s ‘Working with’ the Actress Following Her AGT Firing

“How is it that Simon Cowell has orchestrated this?” said Stern, 65. “He sets it up that the men stay, no matter how ugly they are, no matter how old they are, no matter how fat they are, no matter how talentless they are.”

“But what he manages to do on all his shows is he constantly replaces the hot chicks with hotter chicks and younger chicks. Which is so obvious,” Stern added.

A rep for Cowell did not immediately respond to PEOPLE’s request for comment regarding Stern’s remarks.

News broke on Nov. 22 that Union, 47, and Julianne Hough — who joined the season 14 AGT judging panel back in February, replacing spots vacated by Mel B and Heidi Klumwould not be returning for the show’s upcoming 15th season. (It has yet to be announced who will replace Hough and Union for the upcoming season.)

On Sunday, Syco Entertainment, which produces AGT, released a joint statement with NBC and production company Fremantle. “We remain committed to ensuring a respectful workplace for all employees and take very seriously any questions about workplace culture,” they said.

Howie Mandel, Gabrielle Union, Julianne Hough and Simon Cowell | Trae Patton/NBC
Howie Mandel, Gabrielle Union, Julianne Hough and Simon Cowell | Trae Patton/NBC

“We are working with Ms. Union through her representatives to hear more about her concerns, following which we will take whatever next steps may be appropriate,” the statement concluded.

Last Tuesday, a report by Variety claimed that while working on the show, Union had expressed concerns over racially insensitive situations during her time as a judge, including a joke guest judge Jay Leno allegedly made that was later edited out of the episode.

Union and Hough, 31, were also both subject to “excessive notes” on their physical appearance, sources alleged in Variety‘s report.

A spokesperson for NBC and Fremantle previously responded to such claims with a statement to PEOPLE on Nov. 26, saying: “America’s Got Talent has a long history of inclusivity and diversity in both our talent and the acts championed by the show. The judging and host line-up has been regularly refreshed over the years and that is one of the reasons for AGT’s enduring popularity. NBC and the producers take any issues on set seriously.”

RELATED: Ellen Pompeo Sends Scathing Message to NBC After Gabrielle Union Gets Fired from AGT

Representatives for Hough, Union and Leno did not respond to PEOPLE’s request for comment.

Blog Love B Scott was first to report about complaints Union had allegedly made prior to her departure, with an NBC source saying that she had spoken out about “problematic” situations on set.

According to Variety, during a taping of AGT over the summer, guest judge Leno allegedly made a joke about a painting on the wall that featured Cowell surrounded by his dogs.

Gabrielle Union and Simon Cowell | Frazer Harrison/Getty Images; Tibrina Hobson/WireImage
Gabrielle Union and Simon Cowell | Frazer Harrison/Getty Images; Tibrina Hobson/WireImage

Sources on the series told the outlet that Leno allegedly joked the dogs looked like something one would find “on the menu at a Korean restaurant,” alluding to the stereotype that all Asian people eat dog meat.

Multiple people who were on set during the joke, Variety reports, including one Asian staffer, were offended and made uncomfortable by it.

Union also soon learned of the joke and allegedly urged producers to report the joke to NBC’s human resources department in an effort to bring awareness to the problematic issue, three sources told Variety. Several insiders told the outlet that it was never addressed by HR. It was eventually cut by producers from the Aug. 6 episode.

In another instance, sources told Variety that Union found herself again speaking up against racial insensitivity when a white male contestant auditioned with an act where he portrayed people of color and impersonated singers through multiple quick costume changes.

During his performance, the man impersonated Beyoncé and, according to Variety‘s sources, emerged with black hands. The audition didn’t sit well with Union and she allegedly addressed the matter with a segment producer and an executive producer, explaining that the audience should not have to watch something so “racially insensitive.”

RELATED: Gabrielle Union Says She’s ‘Grateful’ as She Shares Thanksgiving Family Photo After AGT Firing

After the actress requested that they remove the audition from the show, the producers allegedly told her, “We’ll take it out later.” Like Leno’s dog joke, the contestant’s audition was also reportedly cut from the show.

Variety‘s report went on to describe claims that Union and Hough often received “excessive notes” on their appearance — noting that Union was allegedly told that her ever-changing hairstyles were “too black” for AGT‘s audience, while Hough reportedly received constant criticism on her hair, makeup and wardrobe.

However, in a statement to Variety, Hough denied that she had a negative experience working on the show. “I had a wonderful time on America’s Got Talent, I loved working with the cast, crew, and producers. I am happy to continue my working relationship with NBC,” said Hough, who is set to star in two upcoming NBC Christmas specials. “I’m looking forward to what the future holds.”

When it came to Union’s critique, four sources told Variety that the actress allegedly received the note “over half a dozen times.” But one NBC insider denied the frequency of that feedback and claimed that it was given to both judges due to their hair continuity throughout the episodes.

Gabrielle Union and Jay Leno | Steve Granitz/WireImage; Vivien Killilea/Getty
Gabrielle Union and Jay Leno | Steve Granitz/WireImage; Vivien Killilea/Getty

Additionally, according to multiple reports, Union allegedly complained that Cowell was smoking indoors, which is illegal in California where AGT films. She is allergic to cigarette smoke.

RELATED: Jay Leno Breaks Silence on Gabrielle Union’s AGT Firing After Allegedly Objecting to His Joke

Amid the drama, Union is leaning on her husband Dwyane Wade and family for support. On Friday, the actress shared a selfie of herself and Wade, in which the NBA legend could be seen sweetly kissing her forehead as she leaned back on his chest.

“Lord, you KNOW I’m tryin…” Union captioned on Instagram. “Whew 🥴 and breathe. Support is everything. ❤.”

And although Union’s departure from the series reportedly involved her being offended about a joke Leno made, the former late-night TV host has no hard feelings towards her.

On Monday, Leno spoke out about Union’s exit for the first time, telling TMZ, “I love Gabrielle Union. She’s a great girl. I really enjoyed working with her.”

“She’s really good,” he told the outlet.

When asked if she was treated fairly on the hit competition series, Leno said, “I don’t know … but I think she’s a great girl.”