Jay Cutler Made Kristin Cavallari 'Cry All the Time' During Filming and 'Belittled' Her: Sources

Kristin Cavallari gave fans a peek at her marital ups and downs with estranged husband Jay Cutler on Very Cavallari, but sources say things were much more contentious behind the scenes.

"He would make her cry all the time during production. He talked down to her and belittled her," a Cavallari friend tells PEOPLE. "She couldn't take it anymore."

Another source close to Cavallari, 33, says the exes have been "struggling" for "at least three years," and that the Uncommon James founder has long been unhappy.

"He belittles Kristin and makes her cry in front of other people," the source says. "He flies off the handle at the littlest thing and would make Kristin absolutely miserable."

A lawyer for Cutler, 37, has not responded to multiple requests for comment from PEOPLE.

Season 2 of her E! reality show, Very Cavallari, featured a memorable argument between Cavallari and Cutler at the dinner table with friends. Cutler, who appeared to be inebriated, told the entire group how Cavallari was not the breadwinner, but "the breadspender," wasting their money, before abruptly leaving the table. During a confessional interview, Cavallari called his remarks "embarrassing." Later on, they reflected on how they'd "switched roles": When he supported the family financially with his football career, she was a stay-at-home mom. But after he retired in 2017, she started to focus on growing her clothing and jewelry business — and didn't feel supported by him at home. And in season 3 — following rumors Cutler had cheated with Cavallari's then-friend Kelly Henderson, which all parties denied — she admitted they still had "issues."

"Jay has been great, yes. From the outside, things are so perfect and things are so great. But actually, they're not. And that sucks," she said on camera. "That's the thing, though, with marriage ... it's ups and downs. But in two months we could be up here again ... that's how we've always been, our whole relationship. It is what it is."

Mark Humphrey/AP/Shutterstock Kristin Cavallari and Jay Cutler

RELATED: Kristin Cavallari Accuses Jay Cutler of 'Inappropriate Marital Conduct' in Divorce Filing

Insiders have told PEOPLE Cavallari and Cutler held off on officially splitting because of their three children: sons Camden, 7, and Jaxon, 5, and daughter Saylor, 4.

But now Cavallari is focused on moving forward. "She just wants to be happy and to be a good mom," the source says. "They tried to make it work for the kids, but Kristin realized Jay is not going to change."

While announcing their decision to divorce on Sunday, Cavallari and Cutler told fans they had "come to a loving conclusion to get a divorce."

RELATED VIDEO: Kristin Cavallari and Jay Cutler's Split Had 'Absolutely Nothing to Do' with Cheating Rumors: Source

Kristin Cavallari and Jay Cutler's Split Had 'Absolutely Nothing to Do' with Cheating Rumors: Source

"On their recent trip, they already knew they were splitting up," a source tells PEOPLE of Kristin Cavallari and Jay Cutler

“We have nothing but love and respect for one another and are deeply grateful for the years shared, memories made, and the children we are so proud of. This is just the situation of two people growing apart," they continued in a statement posted on their respective Instagram accounts.

However, just one day after their announcement, their divorce filings painted a more complicated picture.

In documents obtained by PEOPLE, Cavallari cited both "irreconcilable differences" and "inappropriate marital conduct" as grounds for their divorce. She also claimed in her filing that "further cohabitation" with Cutler would be "unsafe and improper."

The exes also disagree over their official date of separation. While Cavallari listed their date of separation as April 7 — the day they returned back to the U.S. following several weeks spent in the Bahamas amid the coronavirus pandemic — Cutler listed their date of separation as April 21.

RELATED: Kristin Cavallari and Jay Cutler 'Were Not Suited for Each Other' from the Start, Source Says

In his petition, which was filed first, the retired football player claims he "has always been the available at home parent and primary caretaker" of their children.

Jay Cutler and Kristin Cavallari on Very Cavallari

RELATED: Kristin Cavallari Has Been 'Torn' over Divorcing Jay Cutler 'Because of the Kids,' Source Says

Cavallari — who is seeking primary physical custody of their children — hit back in her response, claiming she "has been the primary residential parent."

While they are sharing the same house outside of Nashville, Tennessee amid their split, the source says they are each living there "three days on, three days off, but not at the same time."

In a new legal filing obtained by PEOPLE, Cavallari claims Cutler is refusing to move out from their current home and release funds for her to purchase a new house. The legal filing — which confirms Cavallari and Cutler split time in their home and stay with friends on off days — was submitted on Tuesday in response to a sealed emergency motion filed by Cutler.

According to the documents, Cavallari claims she began looking for a new home last fall during a rough period in their marriage, but she "decided not to purchase the house" after they "attempted to salvage the marriage."

After deciding they were moving ahead with the divorce in March, Cavallari says she told Cutler she wanted to put an offer in on a residence she found last year.

However, Cutler "informed her that his attorney was going to stop her from purchasing the house she wanted" after Cavallari "refused to accept" a settlement that was "unfavorable to her," she claims in the filing.

"Wife does not believe that it is in the best interest of either party or their minor children to reside in the same house ... Wife fears that if the parties are required to remain in the same residence, it would be an unhealthy environment, causing irreparable harm to their minor children," according to the filing. "Husband makes inappropriate statement to and about Wife and attempts to draw her into an argument in front of their minor children. Now that Husband has announced he will not leave so Wife can exercise parenting time with the children, Wife fears the situation will escalate."