Google's 46-camera 'light field videos' let you change perspective and peek around corners

Devin Coldewey

Google is showing off one of the most impressive efforts yet turning traditional photography and video into something more immersive: 3D video that lets the viewer change their perspective and even look around objects in frame. Unfortunately, unless you have 46 spare cameras to sync together, you probably won't be making these "light field videos" any time soon.

The new technique, due to be presented at SIGGRAPH, uses footage from dozens of cameras shooting simultaneously, forming a sort of giant compound eye. These many perspectives are merged into a single one in which the viewer can move their viewpoint and the scene will react correspondingly in real time.

This image from the researchers' paper shows how the cameras capture and segment the view.

The effect of high-definition video and freedom of movement gives these light field videos a real sense of reality. Existing VR-enhanced video generally uses fairly ordinary stereoscopic 3D, which doesn't really allow for a change in viewpoint. And while Facebook's method of understanding depth in photos and adding perspective to them is clever, it's far more limited, creating only a small shift in perspective.

In Google's videos, you can move your head a foot to the side to peek around a corner or see the other side of a given object — the image is photorealistic and full motion but in fact rendered in 3D, so even slight changes to the viewpoint are accurately reflected.

Image Credits: Google

And because the rig is so wide, parts of the scene that are hidden from one perspective are visible from others. When you swing from the far right side to the far left and zoom in, you may find entirely new features — eerily reminiscent of the infamous "enhance" scene from "Blade Runner."

It's probably best experienced in VR, but you can test out a static version of the system at the project's website, or look at a number of demo light field videos as long as you have Chrome and have experimental web platform features enabled (there are instructions at the site).

The experiment is a close cousin to the LED egg used for volumetric capture of human motion we saw late last year. Clearly Google's AI division is interested in enriching media, though how they'll do it in a Pixel smartphone rather than a car-sized camera array is anyone's guess.

This room-sized LED egg captures amazing 3D models of the people inside it


More From

  • Facebook discovers it shared user data with at least 5,000 app developers after a cutoff date

    Facebook says it accidentally allowed around 5,000 developers to access data from their app's inactive users, even though that access should have been cut off. The company explained on Wednesday it recently discovered an issue that had allowed app developers to continue receiving this information beyond the 90 days of inactivity that is meant to cut off data access until the user returns to the app and again re-authenticates. In 2018, Facebook had announced a change to the way app developers would be able access Facebook user data in the wake of the Cambridge Analytica scandal which saw the personal data of 87 million Facebook users compromised.

  • Walmart partners with Tribeca to turn 160 store parking lots into drive-in theaters

    Just ahead of the Fourth of July weekend, Walmart announced a partnership with Tribeca Enterprises (most notably the purveyors of the film festival of the same name) that’s set to convert 160 store locations into makeshift drive-in movie theaters. The move is an extension of the existing Tribeca-led Drive-In program that has already announced events for a handful of cities, including Los Angeles, New York, Arlington, TX, Miami and Seattle, with help from IMAX and AT&T. The Hollywood Reporter has a bit more detail about the new initiative. Walmart Drive-In follows a number of smaller scale initiatives that have helped the largely extinguished category see a resurgence as consumers are understandably wary of returning to an indoor theater experience as COVID-19 continues to spike across the country.

  • QuestDB nabs $2.3M seed to build open source time series database

    Episode1 Ventures led the round with assistance from Seedcamp, 7percent Ventures, YCombinator, Kima Ventures and several unnamed angel investors. The database was originally conceived in 2013 when current CTO Vlad Ilyushchenko was building trading systems for a financial services company and he was frustrated by the performance limitations of the databases available at the time, so he began building a database that could handle large amounts of data and process it extremely fast. For a number of years, QuestDB was a side project, a labor of love for Ilyushchenko until he met his other co-founders Nicolas Hourcard, who became CEO and Tancrede Collard, who became CPO, and the three decided to build a startup on top of the open source project last year.

  • Growth capital investor Kennet raises $250M fund, backed by Edmond de Rothschild

    Venture capital is “not the only fruit” for entrepreneurs, as the often quieter ‘Growth Capital’ can also see great returns for entrepreneurs who prefer to retain a lot of ownership and control but are also willing to bootstrap over a longer period in order to reach revenues and profits. Thus it is that Kennet Partners, a leading European technology growth equity investor, has raised $250m (€223m) for its fifth fund, 'Kennet V', in partnership with Edmond de Rothschild Private Equity, the Private Equity division of the Edmond de Rothschild Group. Kennet is perhaps best know for its involvement in companies such as Receipt Bank, Spatial Networks and its exist from Vlocity, IntelePeer, and MedeAnalytics.