After 33 years and four children, Baby Boomers Marta and Octavian Dragos say they feel trapped in what was once their dream home in El Cerrito, California.

Both over 70, the Dragos are empty nesters, and like many of their generation, they’re trying to figure out how to downsize from their 3,000-square-foot, five-bedroom home.

“We are here in a huge house with no family nearby, trying to make a wise decision, both financially and for our well-being,” said Dragos, a retired teacher.

But selling and downsizing isn’t easy, appealing or even financially advantageous for many homeowners like the Dragos family.

Many Boomers whose homes have surged in value now face massive capital gains tax bills when they sell. This is a kind of tax on the profit you make when selling an investment or an asset, like a home, that has increased in value.

Plus, smaller homes or apartments in the neighborhoods they’ve come to love are rare. And with current prices and mortgage rates so high, there is often a negligible cost difference between their current home and a smaller one.

“For now we’re staying put,” Dragos said. “Better to hold steady than to do something we will regret.”

Fewer older homeowners selling is part of what is keeping the inventory of homes historically low and pushing prices ever higher in markets across the US. Empty nesters of this age own more larger homes — three bedrooms or more — than Millennials with kids do.

Dragos said she understands that, as homeowners, theirs are enviable problems. They own an asset that has soared in value, after all.

But as she and her husband sit at their dining table discussing the morbid math — what is left after capital gains taxes, what happens if he dies first, what if she goes before him — she says they see no good options for how to get out from under their home while keeping an acceptable amount of profit from its sale, which they’d like to use to fund their retirement.

Here are some of the hurdles many older homeowners say they’re facing.

Taxes on capital gains drain profits from a sale

Federal and possibly state capital gains taxes can be significant for long-time homeowners who have seen their property values soar over several decades.

Most homeowners don’t have to pay capital gains on their home when they sell. Thanks to tax legislation from the ’90s, a gain of up to $250,000 for a single tax filer or $500,000 for a couple filing jointly is exempt from tax. That’s providing the sale is of the homeowner’s primary residence and that they meet other requirements such as living in the property for two of the past five years.

That means if a couple bought a median priced home in 1987 for $100,000 and they’ve lived there as their primary residence and are selling it today for $550,000, the $450,000 gain from that investment is not taxed because it falls under the $500,000 exclusion to capital gains taxes.

However, if those same $100,000 homebuyers lived for 37 years in an area that has seen enormous growth in home values — as is the case for many parts of California — and their home now sells for $2 million dollars, that’s nearly $1.9 million in profit, of which only $500,000 is excluded from taxes.

The taxable gain of $1.4 million at 20% would mean those homeowners are facing a $280,000 tax bill. In a state like California with additional tax, the overall payment would be over $450,000.

“It’s more than a little pill to swallow, it’s a serious issue,” said Peter Poulsen, a retired physicist, who together with his wife sold their home of 35 years in Livermore, California, last year and moved into a retirement community.

They paid the capital gains and moved because their home was isolated and they wanted to be closer to conveniences, health care and other people as they aged. But they were frustrated by the current system because they say a lot of the increase in value of their home was due to inflation.

“The exclusions are based on real estate values from many years ago,” said Poulsen. “None of it is indexed to inflation. A $250,000 per person exclusion? In a state like California that doesn’t quite do the job.”

Last March, California Representative Jimmy Panetta introduced the “More Homes on the Market Act.” The bill increases the tax exclusion of gain from the sale of a primary home and requires an annual inflation adjustment to the increased amount. It is co-sponsored by a bipartisan mix of nearly three dozen lawmakers, but is currently going nowhere in Congress.

Few smaller homes in areas they’ve long lived

Many neighborhoods where older homeowners have long lived are zoned for single-family homes and have few smaller homes or multi-family properties like condos or rental buildings.

This kind of housing is often called the “missing middle” in housing.

While this sector has been underbuilt and, in many areas, forbidden, some communities are slowly changing this by allowing for the construction of smaller scale apartment buildings and accessory dwelling units like granny flats.

Availability of apartment buildings isn’t going to happen overnight and allowing granny flats isn’t enough of a solution, but some Boomers may find that they can move out of their larger home if they can build an accessory dwelling unit on an adult child’s property.

Savings after selling a big home and buying a smaller one are negligible

Older homeowners who want to downsize have been scared into staying put by how expensive a smaller home would be in the current market.

A homeowner who keeps all the profit of a home that sells for $500,000, for example, may find that a condo in their same area, where they can age in place, is $450,000. After calculating realtor fees and closing costs, the profit hardly covers the new purchase, let alone provides any extra income for retirement.

Many homeowners are asking why they should downsize if doing so isn’t that much cheaper.

For more CNN news and newsletters create an account at CNN.com