Meredith Vieira Says She Was Told She 'Set Women Back' When She Quit 60 Minutes to Raise Her Kids

Taylor Hill/Filmmagic

Meredith Vieira is looking back at her controversial decision to step away from 60 Minutes to raise a family.

The journalist, 68, joined Hoda Kotb on Thursday's episode of Today with Hoda & Jenna where she explained how she came to the decision to leave the news show and focus on her kids. At the time, Vieira had a newborn child and another baby on the way. (She's now mom to three kids: Lily, 29, Gabriel 31, and Benjamin, 33, with husband Richard Cohen.)

"When you started out in the business, you played by your own rules," Kotb told Vieira. "You were one of the first women at 60 Minutes and they were grinding at that point, especially the women in that group. And you did what was right for you. You told [creator] Don Hewitt that this might not be working for me."

"And to be a woman in the industry at that moment was big," she continued. "But then to say to the boss and all the women out there, 'I've had a change of heart]…' "

"But my back was up against the wall, literally," Vieira explained. "I was picking between my family and my job at that point, so it was an easy call the minute that it happened. And Don's baby, to be honest, and fair to him, was 60 Minutes. So we both were taking care of our kids."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

RELATED: Hoda Kotb Plays Dress-Up with Daughters Hope and Haley: 'Somebody Raided Mommy's Closet'

Kotb then asked if Vieira was "terrified" to go through with her departure.

"No, it was the next morning that I thought 'Oh my god.' I slept like a baby that night because I knew it was the right decision but the next morning, I had a few second thoughts," Vieira admitted.

The TV personality recalled going to a party a few days later when a woman "backed [her] up into a wall" and said, "You have set us back. You were a poster child for doing it all and now you do this. And it's such a negative message."

"And I said, 'Wait a minute, first of all, I don't want to be a poster child for anybody. Secondly, if I wasn't true to myself, doesn't that send a worse message?' But that really threw me and it came from a woman," she continued. "And I get where she was coming from, but it was sort of like, 'Whoa.' "

Courtesy Meredith Vieira Meredith Vieira and Richard Cohen with family

Speaking about her time leaving The View, Vieira said it "wasn't the best job for me at that time."

"I also knew, timing is everything and it's good to go out on top and to know when you're ready for another chapter and just be confident that something will happen," she added.

As for closing out her time on Today, Vieira said she "had this life that was wonderful between 7 and 9 in the morning" but was "really tired" in other aspects of her life. "I realized that if I signed up [to continue the show], I would be signing up for a paycheck — a really nice paycheck — and at the time, I didn't want to do that."

She added, "I also had certain money in the bank and I had a secure lifestyle. And it's easier to make choices when you're in that situation."