Jenna Ushkowitz Hid Her Pregnancy Filming RuPaul's Secret Celebrity Drag Race : 'Nobody Knew'

Emma McIntyre/Getty; VH1/World of Wonder

Jenna Ushkowitz's identity wasn't the only thing being kept under wraps during her time on RuPaul's Secret Celebrity Drag Race.

The actress and Glee alum — who was voted off of the VH1 reality competition series's second season in an episode that aired Sept. 2 — tells PEOPLE that beneath the elaborate costumes, wigs and makeup she donned to transform into drag queen Milli Von Sunshine for the show, she was cooking up another big surprise.

"When we filmed, I was actually 9 weeks pregnant," says Ushkowitz, in an exclusive interview. "I had agreed to do the show, and when I found out I was pregnant, I was like, 'This is crazy. Is this wild to do?' So Drag Race was actually the beginning of my journey into motherhood. And I couldn't tell anybody, nobody knew. It was my own little secret."

"Getting kicked off was a blessing in a way," she adds. "It was just around the time in my pregnancy that I was starting to get really tired and kind of ill. It was definitely one of those things I was glad I was able to do earlier on."

RELATED: Glee's Jenna Ushkowitz Welcomes First Baby with David Stanley: 'Our Hearts Have Burst Wide Open'

Ushkowitz announced in early June that she had given birth to her first child with husband David Stanley, a baby girl.

"Motherhood is just such a wild whirlwind," she tells PEOPLE. "We're exhausted but it's such a beautiful experience. I just feel like everything is so complete now."

She's purposefully kept her daughter's name and likeness off of social media. But she has used the platform to open up about some of the early challenges she's experienced with motherhood, especially when she started heading back to work.

"It's hard, to balance work and motherhood," Ushkowitz tells PEOPLE. "There's so much guilt that comes with it. And yet, it's also fulfilling to me that being a mom is not my only job; it's definitely the most important to me, but it's not my entire identity. So it's this fine balance. And nobody has quite figured out the one formula for it yet. I don't think there is one! It's definitely hard."

As difficult as it can sometimes be, Ushkowitz still notes how grateful she feels to be able to make that choice herself. "Not everybody gets that," she says. "So many people have to go back to a 9-to-5 or go into an office; they don't have the luxury to stay home. We're over 12 weeks out — I can't imagine people who have to go back to work at six weeks or even sooner than that. 12 weeks is hard enough! So I feel really blessed to be able to choose how I want the next year to look and base my work projects around that."

And for the days that she does struggle? Ushkowitz turns to her close confidants, many of them moms themselves.

"I feel very lucky to have such an army of mothers," she says. "It can be really isolating, going through something like this all by yourself. But I had all these new moms around me — like my best friend; we were exactly a week apart in our pregnancies — and it's been so special. Even some of our Glee cohorts are moms who are constantly checking on me. It's been really nice."

Michael Kovac/Getty

RELATED: Glee Star Jenna Ushkowitz Marries David Stanley in Los Angeles: 'The Day of Our Dreams'

Stanley has been a source of support, too. "He is amazing," she tells PEOPLE of her husband, whom she married in July 2021. "In the beginning, the mother plays such a prominent role the father is often overlooked, but he's been so incredible in pitching in every single way that he can be helpful. He's really taking care of me, so I can take care of her. It's just been sweet to watch him evolve into this new role, and I know he feels the same for me."

"It's funny, we still feel like kids sometimes but then you look at her and you're like, 'Where did you come from? Who are you? What do you mean I have to keep you alive?' " Ushkowitz laughs.

Right now, Ushkowitz and Stanley are focused on being new parents — though they're enjoying it so much, it might just happen again.

"I don't know," she says when asked about having more kids. "It's hard to even think about doing it again, and then there's a part of you that can't imagine not doing it again. So I have no idea what we'll do. I have no clue."

VH1/World of Wonder

RELATED: Taylor Dayne 'Didn't Recognize' Herself on RuPaul's Secret Celebrity Drag Race: 'That's You, Girl!'

Until that decision is made, Ushkowitz can look back on the good times she's had with her other baby: Milli Von Sunshine, the drag persona she created for RuPaul's Secret Celebrity Drag Race.

"Doing the show, for me, was such fun," Ushkowitz tells PEOPLE. "It's hard to find opportunities in Hollywood to combine live performance and television, these two things that I love so much. Somehow I was able to do that for six years on Glee, and it was one of these dream experiences. Like, 'Who gets to sing and dance on screen for millions of people for that long?' So to do something like that again was really gratifying. And then to do drag with these icons of the industry, starting with RuPaul? It was just the best way to experience it."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up to date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

And while Ushkowitz didn't take home the crown, she gained something even better.

"It's really easy to go in thinking, 'Oh this will be just silly fun. I'll put on these costumes and make some money for charity.' But the truth is, when you do drag, you really gain such a deep understanding of yourself — which for me was the big takeaway.

"Yes, Milli was bubbly and bright, but creating her allowed me to peel back all the layers of myself — those deeper, scarier, more vulnerable and weirder places of myself — and really embrace all the parts that make you you. Becoming Milli taught me how to really be Jenna. I'll always have the utmost respect for the art form of drag because of it."

RuPaul's Secret Celebrity Drag Race airs Fridays (8 p.m. ET) on VH1.