Two years ago, Matt Lauer‘s world was turned upside down.
At the time, a source told PEOPLE that Lauer, 61, was let go due to sexual misconduct throughout 2014 with the woman who made the complaint, including at the Winter Olympics in Sochi that year. Another source with knowledge of the situation said at the time that Lauer had viewed this relationship as “consensual” and was “dumbfounded” by the accusation.
Details of the complaint weren’t made public until Oct. 8 this year, when Variety published an excerpt from Ronan Farrow‘s book, Catch and Kill: Lies, Spies, and a Conspiracy to Protect Predators, in which former NBC News employee Brooke Nevils alleged that Lauer anally raped her in his hotel room at the Sochi Olympics. Nevils says she told Farrow she was “too drunk to consent” and also stated multiple times that she did not want to have anal intercourse.
Nevils said in the book that she had more sexual encounters with Lauer back in New York City, telling Farrow: “It was completely transactional. It was not a relationship.”
Lauer, who recently finalized his divorce from longtime wife Annette Roque, penned a lengthy letter in response, claiming the encounter was “extramarital, but consensual.” He said the encounter in Sochi was the beginning of his affair with Nevils and “the first of many sexual encounters between us over the next several months.”
Lauer, who pointed out what he claims are “contradictions” in Nevils’ story, concluded by stating that he has “never assaulted anyone or forced anyone to have sex. Period.”
Lauer has maintained an extremely low profile in the Hamptons since his firing, and a source told PEOPLE this fall that he continues to “lay very low.”
“He’s angry about what he sees as the media’s unfair treatment of him, and hasn’t been taking things well,” said the source. “He had done a lot of work to repair his relationships with his kids and they’ve been sticking by him.”
“He went from being the biggest deal and being able to do whatever he wanted to being an outcast and it’s been a bitter pill to swallow,” the source continued.
According to the source, Lauer’s circle has only gotten smaller since Nevils went public with her rape allegation.
“After the most recent revelations, some of the people who had stayed friendly with him since his firing have iced him out,” said the source. “They’re horrified by the allegations — and the fact that he still doesn’t think he did anything wrong.”
A separate insider says Lauer “still maintains several friendships, including with people at NBC.”
“He’s really close with his kids as well,” says the friend.
PEOPLE is out to Lauer’s attorney for comment.
Nevils, meanwhile, has slammed Lauer’s open letter, calling it a “case study in victim shaming.”
“There’s the Matt Lauer that millions of Americans watched on TV every morning for two decades, and there is the Matt Lauer who this morning attempted to bully a former colleague into silence,” she said in a statement on Oct. 10. “I am not afraid of him now. Regardless of his threats, bullying, and the shaming and predatory tactics I knew he would (and now has) tried to use against me.”
In a statement read on-air on the Today show after Nevils’ rape accusation was first publicly reported earlier this month, NBC News said, “Matt Lauer’s conduct was appalling, horrific and reprehensible, as we said at the time. That’s why he was fired within 24 hours of us first learning of the complaint. Our hearts break again for our colleague.”
If you or someone you know has been sexually assaulted, please contact the National Sexual Assault Hotline at 1-800-656-HOPE (4673) or go to online.rainn.org.