"It's a bit complicated," Barton told E! in an interview published Tuesday. "It started pretty early on because it had a lot to do with them adding Rachel [Bilson] in last minute as, after the first season, a series regular and evening out everybody's pay — and sort of general bullying from some of the men on set that kind of felt really s**tty."
It didn't help that Barton was simultaneously coping with everything that came along with becoming a teen idol. Just 17 when the show began, O.C. fame was intense for her, even though she had appeared in projects like the Sixth Sense and Guiding Light since she was 9. Barton, who played the starring role of the always cool Marissa Cooper, and her co-stars were everywhere. (Bilson played her best friend, Summer Roberts, while Ben McKenzie played bad boy with a heart of gold Ryan Atwood and Adam Brody played Seth Cohen.)
"You know, I also loved the show and had to build up my own walls and ways of getting around dealing with that and the fame that was thrust specifically at me," Barton said. "Just dealing with like the amount of invasion I was having in my personal life, I just felt very unprotected, I guess is the best way to put it.
Still, the actress stayed on for three seasons, from 2003 to 2006, even though some of that time was rough.
"So halfway through season two, I would say, when we started doubling up on episodes and shooting [became] so much harder, and again a lot of that was too much for me," Barton explained. "I didn't know where the character was going. I look back on it pretty fondly, but there's stuff I think people did wrong and the way they handled it. So, I just didn't feel I could keep going."
Barton said she was presented with the choice of whether she wanted Marissa to die or be written off in a way that would allow her to return if she ever wanted to. Barton opted to go with a bang, and former fans of the show continue to talk to her about that ending.
Barton remembers that, as she signed off, she felt several different emotions.
"I was really sad I was going because that was like my family, but there had also been some things that weren't so cool and I would be lying to say I wasn't a little bit relieved that I was going to extricate myself from that situation," she said. "For whatever reasons, years later, certain people, when they see me from the show they are so excited to see me and they only remember the good times. So it's a bit of a mixed bag how we all feel about it. I was young, but I was excited to try and get to do new stuff, too, and didn't know if I could keep handling the stress of that environment that I was put in."
Warner Bros. Television, which produced the show, declined Yahoo Entertainment's request for comment.
Bilson also has been reflecting on her time in the O.C. in her new podcast, Welcome to the O.C., Bitches!, which she co-hosts with one of their former co-stars, Melinda Clarke. Earlier this month, she said on an episode of Monét X Change Today that she'd "love" to have Barton as a guest.
Read more on Yahoo Entertainment: