James Franco Skips Critics' Choice Awards as Source Says He's 'in a Really Bad Place' After Allegations

James Franco is stepping back from the spotlight as allegations of sexually inappropriate behavior emerge against him.

The actor skipped Thursday’s Critics Choice Awards where he won the Best Actor honor for his role in The Disaster Artist.

“He’s in a really bad place, so bad that he changed his phone number,” a source close to the situation told PEOPLE. “His close friends are trying to be there for him but it’s been hard – he’s only talking to a select group of people. For now, he’s just hiding out.”

During an appearance on Late Night with Seth Meyerson Wednesday, the actor addressed the sexual harassment allegations issued by a number of women on Twitter after wearing a Time’s Up pin at Sunday’s Golden Globe Awards.

“There are people that need to be heard,” the 39-year-old actor told Meyers. “I have my own side of this story, but I believe in these people that have been underrepresented getting their stories out enough that I will hold back things that I could say, just because I believe in it that much,” he said. “So if I have to take a knock because I’m not going to try and actively refute things, then I will, because I believe in it that much.”

One accuser, The Breakfast Club star Ally Sheedy, who worked with the actor in 2014 on his Off-Broadway directorial debut, The Long Shrift, wrote in now-deleted tweets, “Why is James Franco allowed in? Said too much.” Adding “James Franco just won. Please never ever ask me why I left the film/tv business.”

When Meyers asked for clarification, Franco was quick to deny his former co-star’s claims.

Shaking his head no, he said, “I had a great relationship with her. She took the tweet down. I don’t know, I really don’t. I don’t know, it was so shocking. I guess I’m just letting it be.”

In an article published in the Los Angeles Times on Thursday, five women accused the actor of abusing his power as an acting teacher and mentor in a sexually exploitative manner.

Two students claimed the actor would often become angry on set when they would refuse to film topless while another former acting student at the film school Franco founded said he once removed safety guards while filming an oral sex scene on the set of the 2015 film The Long Home.

Additionally, actress Violet Paley recounted her previous social media claims that Franco exposed himself and tried to pressure her into oral sex. Though she said they had a consensual relationship, Paley said: “that time wasn’t consensual.” She also alleged that he told her friend to meet him in a hotel when the friend was 17.

Franco’s attorney, Michael Plonsker, denied each of the women’s allegations, and cited Franco’s comments on The Late Show With Stephen Colbert on Tuesday as his formal denial.

“Look, in my life I pride myself on taking responsibility for things that I have done,” Franco told Colbert. “I have to do that to maintain my well being. The things that I heard that were on Twitter are not accurate. But I completely support people coming out and being able to have a voice because they didn’t have a voice for so long. So I don’t want to shut them down in any way.”