Dax Shepard Says He and Kristen Bell Didn't Want a Second Child, Shares What Changed Their Mind

kristenanniebell/Instagram

Dax Shepard is opening up about how his family of four came to be.

Appearing on The Endless Honeymoon podcast hosted by Natasha Leggero and Moshe Kasher, the Armchair Expert podcast host, 47, shared how he and wife Kristen Bell came to have two kids after having their minds set on just one.

While sharing advice with a caller debating on growing their own family, Shepard said that he and Bell "did not want a second child, as I'm sure you guys are feeling right now."

"You must feel so content and so full, you're not wanting for anything," he noted of the time after having a first child. "So I think for Kristen and I, we had two thoughts."

"One is we travel a lot. It's not fair to bring this little human everywhere we go and deal with only adults. We owe it to her to give her a playmate that travels with us," he shared. "We love her enough to do something we don't really want to do,  which is, have a second 'cause we were so absolutely happy with just the one."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

RELATED: Kristen Bell and Dax Shepard Take Daughters to Mount Rushmore — See the Family Photos!

Secondly, Shepard added, "Our kids are already so privileged beyond belief."

"It rattles both of us being from very modest backgrounds," he continued. "So minimally to make this spoiled bitch, my firstborn, live in the same room with another person and have to share everything, like I needed a force of compromise and sharing and discomfort, 'cause I wasn't gonna give it to her in the other ways. So I, we, just thought it would be really helpful to make her a better person to have to deal with someone else."

Suggesting to the caller that should they go for a second baby to do it "as quick as possible," he went on to celebrate the bond between his daughters.

"For a minute that was difficult, 'cause when you're 5 and the baby's 3, that's no fun. But I will say the corner we've turned is now the party. Not only do they party together, they're united against us, which I love."

Photo by Kevin Mazur/WireImage

Shepard said that when he and Bell are "giving it to one of the girls, the other one comes over like, 'You're not being nice to Lincoln, you didn't listen to what she said.'"

"And I'm like, 'That's right. That's your role. You guys gang up and kill me. It's you two against the  world like that.'"

Reflecting on their decision to have two kids, Shepard noted, "It's ironic, 'cause we've pretty much stopped traveling."

"We had two kids and all of a sudden we were like, 'Oh, we're never leaving town to work again,'" he explained. "So ironically it didn't even — the reason we had — it didn't even come to fruition. But that doesn't matter."