Matt Lauer's Wife Worried He Would Cheat Even Before They Wed, Source Says: 'He Was a Player'

More details about Matt Lauer‘s relationship with his wife of 19 years, Annette Roque, are coming to light.

Since NBC News fired the longtime Today anchor last week after receiving a “detailed complaint from a colleague about inappropriate sexual behavior,” with reason to believe “this may not have been an isolated incident,” several insiders have told PEOPLE that Lauer was allegedly often unfaithful to Roque, whom he wed in 1998 after meeting on a blind date. (Lauer, 59, was previously married to television producer Nancy Alspaugh from 1981-88.)

“By Matt’s own admission there were struggles and challenges with his wife,” a longtime friend tells PEOPLE in this week’s issue. “Like many couples, they struggled with a modern marriage.”

Matt Lauer and Annette Roque
Matt Lauer and Annette Roque

A former longtime colleague of Lauer’s says the marriage “was never good,” adding that Lauer’s reputation as a ladies’ man bothered his wife, a former Dutch model, even before they wed.

“He was a player when she met him and she knew that,” the colleague says. “Even many years ago, he was a flirt, a philanderer. He did his thing.”

  • For more on Matt Lauer, pick up this week’s issue of PEOPLE, on newsstands Friday

Roque briefly filed for divorce in 2006, claiming in court documents obtained by the National Enquirer that Lauer committed “cruel and inhumane” acts against her, demonstrating “extreme anger and hostility” that endangered her mental and physical wellbeing, as well as repeatedly criticized her parenting skills.

Despite the allegations, Roque withdrew the court papers three weeks after filing them. But while they reconciled, “they lived totally separate lives,” says the former colleague, with Roque raising the couple’s three children in the Hamptons and Lauer spending weekdays at his apartment in Manhattan.

Of Lauer’s son Jack, 16, daughter Romy, 14, and son Thijs, 11, the former colleague says Lauer “loves his kids” and “is a solid father” — though, according to one network producer, he “was essentially a single guy.”

On Wednesday, PEOPLE confirmed that Lauer and Roque are no longer wearing their wedding rings — but according to a source, the couple are still under one roof and together.

“They’re taking it day by day right now,” the source said. “His focus is on his family.”

Lauer, who joined NBC in 1992 as a newsreader on Today, had been co-anchor of the morning show since January 1997 and reportedly signed a $20 million dollar contract last year.

On Nov. 29, NBC announced Lauer’s termination, and a source later told PEOPLE that Lauer was let go due to sexual misconduct throughout 2014 with a female staffer, including at the Winter Olympics in Sochi, Russia. Another source with knowledge of the situation said Lauer had viewed this relationship as “consensual” and was “dumbfounded” by the accusation.

Women have also anonymously accused him of sexual harassment and assault in reports published by Varietyand The New York Times.

Lauer broke his silence on the allegations in a statement read Nov. 30 by his former co-hosts on-air.

“There are no words to express my sorrow and regret for the pain I have caused others by words and actions,” he said. “To the people I have hurt, I am truly sorry. As I am writing this, I realize the depth of the damage and disappointment I have left behind at home and at NBC.”

“Some of what is being said about me is untrue or mischaracterized, but there is enough truth in these stories to make me feel embarrassed and ashamed. I regret that my shame is now shared by the people I cherish dearly,” he continued. “Repairing the damage will take a lot of time and soul searching, and I’m committed to beginning that effort. It is now my full time job. The last two days have forced me to take a very hard look at my own troubling flaws. It’s been humbling. I am blessed to be surrounded by people I love. I thank them for their patience and grace.”

One of Lauer’s former Today show colleagues told PEOPLE the ousted anchor is still reeling over the situation.

“He is weak and broken and ashamed by his own admission,” the colleague said.

But, the colleague added, “He is my friend. He is human, and he is flawed. But is he going to use his power to be a predator on women? Until I see solid evidence of that, I just can’t believe it. We don’t know the details.”