Hilary Duff 'Always' Knew She Wanted to Start Her Family Young: 'I Was Meant to Be a Mom'

Dia Dipasupil/Getty Hilary Duff

Hilary Duff knows she started her family at the right time for her.

The How I Met Your Father actress, 35, chatted with Josh Peck and Ben Soffer on Dear Media's Good Guys podcast this week, where she talked about how she became a mom at what Hollywood considered a young age, 24, when she welcomed son Luca Cruz in 2012.

"I was a teen mom," she joked.

"I think that I was just really ready. I told you, the touring was really hard, and I just decided after one of my tours, my last tour, I was like, I need a, a serious break, and I need to lock myself in my house and like figure out how to take care of myself, feed myself, what I like, what I don't like," Duff shared.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

Hilary Duff Instagram

RELATED: Hilary Duff's 3 Kids: Everything to Know

Explaining that she wasn't always comfortable having people "taking care" of her as an "independent person," she was prepared to spend more time focusing on herself.

"Then I met my first husband [Mike Comrie], and like two years later, maybe three years later, I just felt like really ready to open that chapter of my life," she said of starting a family. "I always wanted to be a young parent."

While Duff feels she "always knew" she was "meant to be a mom," it wasn't always an easy road, noting that at one point she "had a two-year-old and was divorced," which kept her from bonding with a lot of her peers.

hilary duff/Instagram Hilary Duff's family holiday card

"It was hard cause I didn't have any friends that were having kids yet. But I just like figured it out and loved it. I love being a mom so much," she said.

Noting that it "was young, for sure," Duff added, "I wouldn't change it at all. And now my son, he's turning 11 in March."

In addition to Luca, Duff shares daughters Banks Violet, 4, and Mae James, who turns 2 this month, with husband Matthew Koma.