Hilary Duff Says Breastfeeding Has Always Been 'Really Hard for Me': It's 'Just Still Painful'

Hilary Duff is continuing to be open about her breastfeeding struggles.

The 33-year-old Younger star speaks with Dr. Elliot Berlin for a post-birth edition of Informed Pregnancy Podcast, sharing how she's handling postpartum life after welcoming daughter Mae James on March 24. Duff also shares daughter Banks Violet, 2, with husband Matthew Koma, plus she's mom to son Luca Cruz, 9.

Duff says breastfeeding has always been "really hard for me."

"I'd say it was the easiest with Luca. All of the babies latch really great, I'm just not a huge milk-producer, so it's emotional for me," says the mom of three. "In fact, this is the first baby that I haven't supplemented with yet, so I've just exclusively been breastfeeding her. I'm going to keep trying that for a couple of weeks."

She continues, "Just still painful and it's hard, and it's even harder having the other two that I know need me so much, and this takes up such a huge portion of the day. It seems like every 20 minutes I'm feeding the baby, and I have to be sitting in one place, and Banks is still not quite old enough to understand, even though she has been amazing with the baby. It's just hard."

Want to get the biggest stories from PEOPLE every weekday? Subscribe to our new podcast, PEOPLE Every Day, to get the essential celebrity, entertainment and human interest news stories Monday through Friday.

RELATED: Hilary Duff Shares Photos of Herself Breastfeeding Newborn Daughter Mae: 'Babe'

Duff says anxiety about not producing enough breastmilk for her baby tends to impact her output.

She explains, "Right now, I don't know that I'm not producing as much as I need, but I think since I haven't in the past, I have tons of anxiety that I'm not, and that she's not getting enough, and then I'm in my head, and then I don't feel like enough, and then the spiral continues from there."

"I'm not even at week three yet, so I need to just sit back and chill and trust that my body is doing the right thing. And she's gaining weight," the actress/singer adds.

The Walt Disney Company/Image Group LA via Getty Images

In a May 2019 Instagram post, Duff opened up about her decision to stop breastfeeding her second child at that time. She said she decided to share her outlook because stopping breastfeeding "was so emotional and hard."

Duff added in her lengthy written message at the time that she felt "fine and happy and relieved and silly that I even stressed on it so hard" after feeling conflicted about switching to formula.

"Banks is thriving and I get even more time with her and daddy gets to do more feeds! And mommy gets a tiny bit more sleep!" she said at the time. "Whether you are pre or postpartum. Or just a busy mom, You are a [superhero] everyday for all that you do. Always putting yourself last and running a mile a minute…while carrying all of the bags lol. Love you all and hope this helps anyone struggling! See you nursing bras✌🏻 until next time!"