Jessica Simpson Says Her Daisy Dukes Set a ‘Gold Standard’ for Her Body That She Couldn’t Escape

In 2005, Jessica Simpson wore the famed Daisy Duke shorts for her scene-stealing role in Dukes of Hazzard. She was praised at the time for her body — and her legs in particular — but the singer says it set her up for years of “scrutiny” any time her weight changed.

Writing in her new memoir Open Book, out today, Simpson, 39, says that the Daisy Duke moment “created a gold standard Jessica, the ‘before’ for every ‘is she fat or is she thin’ story for the rest of my career.”

And those body expectations did follow her for the next 15 years, through three pregnancies and the infamous “mom jeans” look in 2009 that had people calling her “fat.” Simpson tells PEOPLE that she never expected that level of judgement when she signed her record deal at age 17.

RELATED: Jessica Simpson Reveals in Brave New Memoir: ‘I Was Killing Myself with All the Drinking and Pills’

“I thought it was about my voice. I didn’t know that it was going to end up being about how I looked in a dress,” she says. “It’s heartbreaking and I mean, I punished myself for it. I took diet pills. I heard it and I couldn’t not hear it in the back of my mind every time I was on stage, every time I walked out the door.”

Simpson says her own insecurities — and ones she had never thought of — were constantly thrown at her.

“It is not easy, it hurts,” she says. “And still, I feel like we all look in the mirror and are not 100% all the time. I mean, we all see our flaws. Some, the others don’t see. And mine were just out there for the world to rip apart, when they weren’t even flaws. When they were made into flaws that I didn’t know I had.”

Now, as a mom of three and after another tough pregnancy that put her at over 200 lbs., Simpson says that she “just can’t read” the comments about her body anymore, even the ones that are veiled compliments about her weight loss.

“Even now, people [are] commenting on my Instagram, ‘Oh, snap back?’ [about her weight loss] No, it wasn’t a snap back and I don’t even know what that word means,” she says. “It’s like, I work hard and when I work out, a lot of it is to release anxiety. That’s one of my tools for sobriety. Just walking, just going and talking, walking and talking with my husband. Even some of my biggest fans… They’re saying it as a compliment, but it’s like, that’s not what I was trying to get with this picture but okay.”

RELATED: Jessica Simpson’s 100-Lb. Weight Loss Happened ‘Outside of the Gym’

It helps, though, that Simpson is now comfortable in her body and looks at it as a source of strength.

“It’s one of those things that I do accept about myself,” she says. “I do want to look my best of course. And be my best and be active with my kids and run around and not feel too weighed down. It’s not that I am all about strict dieting or anything like that.”

And Simpson is thrilled that curvier figures like hers and those of stars like Lizzo and Ashley Graham are mainstream.

“I just thank God times are changing a little bit and people are standing up for themselves and making it not all about body image,” she says. “I can have hopefully be part of the change that my daughters grow up in a world where she can accept herself at any size.”