Britney Spears and Dad Jamie 'Still Not Speaking' After Incident but He Will Remain Conservator: Source

Though Britney Spears‘ father Jamie stepped down as her primary conservator, the move was only temporary.

A source tells PEOPLE the 67-year-old Spears patriarch “never stepped down fully” from the responsibilities of overseeing his daughter’s personal affairs.

“He will continue to be the conservator,” the source says. “Britney and Jamie always had a complicated relationship. Her dad can be very stern, firm and stubborn. Britney many times rebelled against him.”

Last Thursday, PEOPLE confirmed a judge granted a motion to seal court transcripts after lawyers for Britney and Jamie asked that personal details — including medical information and discussion related to Britney’s sons — remain protected.

RELATED: Britney Spears Celebrates Sons’ Birthdays with Sweet Photo Amid Family Drama: ‘Mamma Loves You’

According to court documents, filed in Los Angeles and obtained by PEOPLE, a judge approved the singer’s longtime “care-manager” Jodi Montgomery to temporarily take over and was granted the “same powers” previously granted to Jamie.

An insider previously told PEOPLE that Jaime “decided to temporarily step down” after Britney’s ex-husband Kevin Federline filed a police report accusing the pop star’s father of physically abusing their son Sean Preston, 14.

However, on Tuesday, the Ventura County District Attorney’s Office announced that criminal child abuse charges will not be filed against Jamie as there was “insufficient proof that a criminal offense was committed.”

The source tells PEOPLE on Monday that Britney and Jamie “are still not speaking after the Sean incident” despite charges being dismissed. “Britney is still upset about it,” the source says.

A rep for Britney did not immediately return PEOPLE’s request for comment.

According to Federline’s attorney Mark Vincent Kaplan’s testimony as recorded in the police report, the pair’s 13-year-old son Jayden James was also present during Jamie and Sean’s fight on Aug. 24. Jamie allegedly “violently shook” Sean after breaking down a bedroom door to reach him, The Blast reported. “Britney did the right thing and removed the children and took them out of there,” said Kaplan, who added that Federline, 41, still wants his ex-wife to have “continuing and meaningful contact with their mother.”

RELATED: Britney Spears ‘Removed’ Her Kids After Son Sean’s Alleged Fight with Dad Jamie, Says K-Fed’s Lawyer

Britney recently went on a vacation to Hawaii amid the conservatorship drama and after the alleged incident. “Mentally, she is definitely doing a lot better than a few months ago,” the source says.

However, Britney is “just very sensitive and fragile. She very easily gets overwhelmed.” Amid the temporary change in conservator, the source says Britney “needs someone around that makes daily decisions for her. Jamie has done a good job so far.”

Earlier this month, an insider told PEOPLE how Britney was “upset” with her father and couldn’t “believe that her dad would jeopardize her relationship with her boys.”

The insider added, “Britney is always terrified that she will lose custody.”

PEOPLE also confirmed that Spears and Federline recently came to a new child custody arrangement which gives Spears 10 percent of custodial rights and Federline 90 percent. Kaplan tells PEOPLE Spears’ visits are required to be supervised.