History suggests that as Wall Street goes, so do sky-high real estate prices in the Hamptons, the ultra-wealthy enclave that lies east of New York City.

That's because many of the prospective buyers in the ritzy suburb work on Wall Street, where the bonus pool has shrunk, according to a recent report from the Office of the New York State Comptroller. That has at least some drag on local property prices, but experts say that when it comes to the real estate calculus of the Hamptons wealthy, there's far more to consider than financial conditions and monetary policy.

"Typically we deal with a lot of finance-based individuals who focus on the markets in Manhattan, or are involved in Wall Street to some extent, or very high-end entrepreneurs who've built amazing businesses," Zachary Vichinsky told CNBC in a recent interview.

Along with brother Cody, the siblings founded Bespoke Real Estate, which has turned over more than half a billion dollars' worth of real estate in less than two years. According to the firm, the average sale price of a Bespoke-listed house tops $15 million.

The brothers explained that many Hamptons property owners are in fact city dwellers who come to the area seeking a few days of rest and solitude before heading back to the concrete jungle. It's not uncommon to see a property listed by Bespoke that is fully furnished, like one recent listing in Bridgehampton, a testament to the transient nature of many Hamptons residents.

Prospective buyers can rub shoulders with neighbors that include Coach CEO Victor Luis and Goldman Sachs chief Lloyd Blankfein, according to the Vichinsky brothers.

The Hamptons real estate market ended 2015 on a strong note, hitting an 11-year-high, according to a sales report from Douglas Elliman.The report said the average sale price of a home in the area was $2.38 million, a 15.6 percent year-over-year increase.

For the ultra-wealthy customers Bespoke caters to, it is more about time and less about things like interest rates, said Cody, who said many on his clients pay in cash.

"There's a lot of insulated wealth, so it's more about when is the right time to come into the market, and how you think the market is going to swing," he said. "So we have a lot more conversation based upon supply and demand and timing of where we are in the market versus interest rates."

The base of Hamptons buyers is also becoming more diverse. International home buyers are becoming increasingly commonplace, and come from all parts of the globe.

Hamptons' real estate is starting to have "a European purchaser, an Asian purchaser, and tech purchasers as well," Cody said.

"We are seeing a new wave of [the] tech generation come to the Hamptons, and certainly as foreign markets evolve or digress in some cases, the Hamptons has certainly been seen as a great place to put capital and enjoy an asset that you know is appreciating overtime," he added. "We do have a lot of investment conversations with people all over the planet."