Ryan Phillippe on Trying to Keep Up with His Son, 18, in the Gym: 'It's Really Defeating'

Ryan Phillippe is as chiseled as ever at 47 years old, but the actor admits staying in shape these days takes a bit more work.

"It's like the law of diminishing returns, right?" he told PEOPLE on set of his Sexiest Man Alive shoot last month. "You've got to work harder in the gym for fewer results."

Phillippe's go-to workout buddy is his 18-year-old son Deacon, and the two often hike in L.A. or hit the gym together.

"We'll do the same workout, and the way he looks after compared to the way I look is really defeating," jokes the star.

In all seriousness, Phillippe — who starts most mornings with an "incredibly intense" hike — says staying active has been crucial for his mental health.

For more on the Sexiest Man Alive — and more of Hollywood's hottest men — pick up the latest issue of PEOPLE, on newsstands Friday.

"It's exhausting. You sweat so hard and it changes your body chemistry, literally. It's a run, hike and climb," he says of his early morning workout on a mountain near his home. "When I finish that, I feel clearer. I feel less down. It gets me out of my head a little."

Yu Tsai

RELATED: Reese Witherspoon and Ryan Phillippe Celebrate Son Deacon's 18th Birthday Together: 'Bursting with Pride'

After decades in Hollywood, Phillippe acknowledges it's still a struggle to feel fully comfortable in his skin.

"I don't like aging. When I see the signs of aging, which is every day, it drives me insane. But there's nothing you can do, really," he says. "You can't prolong the inevitable. We're all going to age, if we're lucky enough, to last into our old age."

Getting older, however, has given Phillippe one of life's biggest blessings: cherished relationships with his three kids, Ava, 22, Deacon and Kai, 10.

"They're both better looking than me, thankfully," he jokes of his older children (with ex-wife Reese Witherspoon). "They obviously have a very pretty mom. Deacon and I connect very heavily in regards to music. He's like my best friend, and Ava is very much my personality."

While he still has his moments of insecurity, Phillippe says he and Witherspoon have worked hard to instill self-confidence in their children.

"They're very self-possessed, conscientious people who care about the world, who treat people with respect and take care of themselves," he says. "I think that's also something that you model as a parent."