Since L Brands (LB) said its Victoria's Secret brand would stop selling swimwear and dial back on promotions , the company's shares have plunged as steeply as one of its asset-baring bras.

But one analyst fears the retailer's woes may extend beyond this shift in strategy — meaning a recovery in its stock price could take longer than expected.

Jefferies analyst Randal Konik on Monday downgraded L Brands' shares from hold to underperform, saying industry and consumer trends could cause Victoria's Secret to lose its dominance in intimate apparel. Among those trends are a slowdown in spending on the category and increased competition from specialty stores and online start-ups.

But perhaps more importantly, Americans are moving away from the "bombshell" look Victoria's Secret has built its reputation on, and are instead stocking up on more natural, unstructured bralettes. Because bralettes are easier to construct, they make it simpler for nontraditional lingerie brands to challenge the stalwart. Both American Eagle (AEO)'s aerie lingerie line and Urban Outfitters (URBN) have recently cited the item as one trend fueling their sales.

"We believe the growth in popularity of bralettes is reflective of a broader shift in the consumers' mindset, particularly among younger consumers, which are increasingly rejecting traditional beauty standards in favor of more natural, attainable looks," Konik told investors.

The analyst pointed to a decline in breast augmentations as one example of consumers' changing tastes. Roughly 279,000 American women went under the knife in 2015, down from nearly 331,000 three years earlier. It was the third consecutive year that number had fallen.

And in its December issue, Vogue UK declared that showing cleavage is no longer fashionable.

As shoppers' definition of beauty has evolved, their interest in the Victoria's Secret brand has also waned, Konik said. Whereas 10 million viewers tuned into the brand's annual fashion show in 2011, that number has since fallen nearly 30 percent.

Victoria's Secret will key into the "natural" trend when the event returns to CBS (CBS) next month. This year's show-stopping Fantasy bra is fashioned from its new "Beautiful" line, which boasts push-up without the padding.

Off the runway, Victoria's Secret is playing into the bralette trend by offering a broader assortment and price promotions in its stores. Yet tapping into the item's popularity comes with its own risks. Konik noted the average price for a bralette is roughly 60 percent lower than a regular bra, meaning every 10 percent shift in sales mix toward the item would result in a 7 cent per share hit for L Brands, he estimates.

L Brands' management has rejected the idea that bralettes will weigh on the company's sales, telling investors that the unit volume and frequency of purchases makes up in part for their lower prices. They're also helping Victoria's Secret appeal to a younger customer. L Brands is regarded as one of the best operators in the space, thanks to its vastly profitable store fleet and consistent sales and earnings growth.

Indeed, not everyone agrees that Victoria's Secret is losing its edge with shoppers. Data released in September by The NPD Group's Checkout Tracking found that Victoria's Secret shoppers make more frequent purchases than the average intimates consumer, and also spend more.

"The exclusive and differentiated merchandising mix that Victoria's Secret offers shoppers has led to a unique breed of consumer that is extremely loyal," NPD chief industry analyst Marshal Cohen said at the time.

L Brands last week said it expects third-quarter earnings of 40 cents per share, at the low end of its prior forecast and below Wall Street's expectations. Its October comparable sales also disappointed, rising 1 percent. That included a 2 percent drop at Victoria's Secret.

The company's shares are down 30 percent this year and at midday Monday were trading up 1.3 percent at $65.88.

For more episodes and highlights of The Profit visit Yahoo View, available now on iOS.

More From CNBC

• Top News and Analysis

• Latest News Video

• Personal Finance