The View 's Show Culture 'Is Not What Barbara Walters Wanted,' Says Source amid Whoopi Goldberg's Suspension

Things are not what they once were at The View, especially in the wake of the controversy involving Whoopi Goldberg, according to a source familiar with the show.

Following Goldberg's suspension from the daytime talk show for claiming that the Holocaust was "not about race," the source tells PEOPLE that "everyone is despondent" at The View.

The source believes the recent situation would be particularly upsetting to Barbara Walters, who put together the initial panel of women on The View in 1997 and served as a co-host until her retirement in 2014.

"It's like watching a family member spiral downward," says the source. "It was described by people there as 'mayhem.' "

The source continues, "There is no accountability [anymore] ... part of being 'produced' means being told what you don't want to hear, so you get it right. When you make a mistake or look arrogant — it's important to not be surrounded by 'yes men.' "

"Whoopi is the alpha. She's the moderator and a pop culture icon. This is not what Barbara Walters wanted and there was nobody there to push back," the source adds.

Adds a separate insider: "Words matter and when you are on live TV, this is why you have to think before you speak."

Reps for ABC and Goldberg did not immediately respond to PEOPLE's request for comment.

Jacopo Raule/WireImage; Jamie McCarthy/WireImage Whoopi Goldberg and Barbara Walters

RELATED: Anti-Defamation League Accepts Whoopi Goldberg's Apology After Her 'Misinformed' Holocaust Statement

Goldberg inaccurately asserted that the "Holocaust isn't about race" on Monday's episode of The View.

The comment came as she and her co-hosts discussed a Tennessee school board's ban of the Pulitzer Prize-winning book Maus, which tells the story of illustrator and author Art Spiegelman's Polish Jewish parents' experience during the Holocaust. The school board cited concerns about female nudity and profanity and removed the material from the eighth-grade curriculum.

Goldberg, who immediately faced criticism on social media, shared a statement and apologized on the show the next day.

"I said something that I feel a responsibility for not leaving unexamined because my words upset so many people, which was never my intention," she said. "And I understand why now, and for that, I am deeply, deeply grateful because the information I got was really helpful and it helped me understand some different things."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Goldberg also provided a platform to the Anti-Defamation League (ADL) CEO Jonathan Greenblatt during Tuesday's episode, and he suggested the show consider a Jewish co-host to fill Meghan McCain's vacant seat.

Goldberg has since been suspended for two weeks over her "wrong and hurtful comments," according to ABC News President Kim Godwin's statement, which was previously obtained by PEOPLE. Godwin also said that Goldberg has been asked to "take time and reflect and learn about the impact of her comments."

Charles Sykes/Bravo/NBCU Photo Bank/NBCUniversal via Getty Images Whoopi Goldberg

RELATED: Whoopi Goldberg Apologizes on The View After Holocaust Comment Controversy: 'I Stand Corrected'

The source familiar with the show notes that while "[Whoopi] has a long record of saying controversial things," she is beginning to wrap her head around the impact of her words. (Goldberg publicly defended Bill Cosby on The View in 2015 amid his sexual assault allegations and previously said "it wasn't rape-rape" when discussing accusations against Roman Polanski on the show in 2009.)

"She's very upset and is beginning to understand the gravity of what she has done. People love her and maybe she can survive," the source says. "It feels like the dam finally broke and the bad behavior and the things they have gotten away with for so long finally came out on camera. I hope the show can recover."