AI Becomes Him: How De-Aging Progressed from ‘Benjamin Button’ to ‘Indiana Jones and the Dial of Destiny’

De-aging has come a long way, from “The Curious Case of Benjamin Button” to “Indiana Jones and the Dial of Destiny” in the last 15 years. While “Benjamin Button” won the VFX Oscar for its trailblazing reverse-aging of Brad Pitt, Harrison Ford’s Indy finale (which premiered this week at the Cannes Film Festival ahead of its June 30 theatrical opening) is already being talked up as an Oscar contender for its breakthrough AI-generated de-aging, making the actor look 35 years younger fighting Nazis in the 25-minute opening flashback.

With David Fincher’s “Benjamin Button,” digital technology caught up with need, and its success spawned a wave of de-aging that fought off the Uncanny Valley with varying degrees of success. This included Patrick Stewart in “X-Men Origins: Wolverine,” Arnold Schwarzenegger in “Terminator Salvation,” Jeff Bridges in “Tron: Legacy,” Michael Douglas in “Ant-Man,” Kurt Russell in “Guardians of the Galaxy Vol. 2,” Samuel L. Jackson in “Captain Marvel,” Peter Cushing and Carrie Fisher in “Rogue One: A Star War’s Story,” Sean Young in “Blade Runner 2049,” Will Smith in “Gemini Man,” and Robert De Niro, Al Pacino, and Joe Pesci in “The Irishman.”

More from IndieWire

• Channing Tatum Fears the Future of Streamers: 'Less Good Storytelling and a Lot More Product'

• Ezra Miller Won't Be Recast in Potential 'Flash' Sequel, Says Director: They 'Excelled' in the Role

Then deepfake started taking off a few years ago, with computer-learning software combining photographs and footage of actors at younger ages to create CG composites of their faces. This AI revolution (an important factor in the current WGA strike) has rapidly evolved to the point of not only achieving high-resolution face replacement through a network of neural skinning but also accurate movement through a more advanced network of neural performance. This means that machine learning has become so sophisticated that after weeks or months of building these neural networks, it delivers rough 3D models in a matter of days.

Which brings us to “Dial of Destiny.” What we know so far is that ILM’s AI system analyzed reels of footage of Ford from the first three “Indiana Jones” films (both used and unused) to get the look and performance right for the opening flashback in 1944, before jumping to 1969, where Indy combats a neo-Nazi revival led by Mads Mikkelsen.

“We had hundreds of hours of footage of him in close-ups, in mediums, in wides, in every kind of lighting, night and day,” director James Mangold told Total Film. “I could shoot Harrison [with mo-cap dots on his face] on a Monday as, you know, a 79-year-old playing a 35-year-old, and I could see dailies by Wednesday with his head already replaced.”

We also know that AI is ushering in a new wave of de-aging. This includes Robert Zemeckis’ upcoming “Here,” adapted from the graphic novel about people who inhabit a single room throughout several decades, which de-ages Tom Hanks and Robin Wright with innovative software from AI studio Metaphysic.

As a prelude to our coverage of ILM’s “Dial of Destiny” de-aging, here’s a recap of how far we’ve come in the last 15 years.

“The Curious Case of Benjamin Button” (2008)

After years of R&D, Digital Domain cracked the code of aging Pitt’s face with different body actors. Utilizing the studio’s volumetric deformation rig, the animators used micro-expressions to manipulate individual movements. This could be driven by either keyframe animation or image analysis because they were built from Pitt’s own faces. Mova’s Contour multi-camera, high-resolution facial capture system was a key component in this part of the process. The trick was letting Pitt be Pitt. Some of the hurdles were making the animation believable, and tracking and creating a lighting and compositing system so that the heads could be dropped into Claudio Miranda’s cinematography.

In addition, for the de-aging sequences between Pitt and Cate Blanchett, Lola FX (which introduced the process in “X-Men: The Last Stand” with an opening flashback featuring a de-aged Patrick Stewart and Ian McKellan as Professor X and Magneto) refined its Photoshop-like 2D/3D technique involving skin smoothing and shape warping with improved animation, tracking, and compositing.

“Tron: Legacy” (2010)

DD raised its game for performance capturing Bridges as the younger Clu 2.0 avatar in Joseph Kosinki’s sequel. This included improving the workflow, management system, pipeline, and animation. But de-aging the 60-year-old actor 25 years was much harder than reverse-aging Pitt. Although the capturing process and facial animation improved, there was no getting around the problem of trying to recapture Bridges in his prime, even if he was driving the performance. His physiology was different and you couldn’t hide the fact that it was older Bridges uncomfortably fitting into his younger self. The result was artificial, but then so was Clu 2.0, who wasn’t human anyway.

“Rogue One: A Star Wars Story” (2016)

ILM’s controversial de-aging of the late Cushing (Grand Moff Tarkin) and Fisher (Princess Leia) in “Rogue One” without the participation of the actors proved problematic. The tech wasn’t there yet to create younger versions of the characters with only old footage and mo-capped doubles. But it was a useful exercise in testing ILM’s evolving facial capture system and animation rig. ILM tried the procedure again in 2019 with Mark Hamill and Fisher for “The Rise of Skywalker,” but the results still weren’t impressive.

There was slight improvement, however, with the cameo of a de-aged Luke at the end of “The Mandalorian” Season 2. This featured the brief participation of Hamill, who was shot on set in alternating scenes with a double before stepping into a special lighting rig called “The Egg” for facial lighting adjustments. Lola handled the de-aging with a deepfake used for reference only. Better still was Luke’s appearance in “The Book of Boba Fett,” where deepfake played a much larger role.

“Blade Runner 2049” (2017)

MPC was part of the Oscar-winning VFX team responsible for the impressive de-aging of Young. Body double Loren Peta played the young Rachael (in costume, makeup, and with dotted face) and performed on set for the two-minute sequence with Ford (reprising Deckard) and Jared Leto (as replicant manufacturer Wallace). She was directed by Denis Villeneuve with Young on set for reference. The facial capture of Peta and Young was done on a Saturday in Budapest, where the film was shot. Production stills and scenes from the original “Blade Runner” were used for reference to create an identical match.

Then, about four weeks before the end of production, they decided to insert footage of Rachael from the original “Blade Runner” into the scene, which set the bar higher. MPC drilled into Young’s mannerisms even further. The director, though, envisioned a three-beat arc for this surprise reunion between Deckard and Rachael: confidence, longing, and rejection when she doesn’t measure up to Deckard’s expectation. But when the animated performance didn’t measure up to Roger Deakins’ beautiful Oscar-winning cinematography, the animators mimicked Young’s original performance until they nailed the emotion.

“Captain Marvel” (2019)

It was a banner year for de-aging in 2019 with three films (“The Irishman,” “Gemini Man,” and “Captain Marvel”) and three distinct techniques on display. Lola, the industry leader as a result of its extensive work for Marvel, reached a milestone with de-aging Samuel L. Jackson as Nick Fury. This marked the first time they didn’t use a digital double. Because the actor performed in two-thirds of the film, it would’ve been too time-consuming and too difficult to match performances for every shot. Fortunately, the actor aged well. Lola had to modify its methods to accommodate the number of shots, and used some makeup to pull back the skin on his neck, but relied on the actor’s performance with no grafting — just slimming and tightening and smoothing over.

“Gemini Man” (2019)

Weta Digital constructed its best digital human yet for Ang Lee’s high frame rate/3D action-adventure, as 50-year-old Will Smith fights 23-year-old Will Smith in a face-off between a hitman and his clone. Smith played the aging Henry as well as his clone, Junior, who appeared in more than half the movie, with the actor channeling his younger self.

Weta created the CG Junior by studying the morphology of aging as it applied to the actor. They then made great strides in skin and eye work. The animators created new procedural software that simulates areas between the pores and along the natural fall lines for a more realistic look. They also modeled a dark retina for the eyes to reveal more depth, and provided an additional film surface that sits across the eye for greater fidelity. The result arguably achieved the first fully convincing digital human.

“The Irishman” (2019)

For Martin Scorsese’s mob epic, ILM created a breakthrough light-based, markerless facial capture system to make Robert De Niro, Al Pacino, and Joe Pesci look decades younger as hitman Frank Sheeran, Teamsters leader Jimmy Hoffa, and Philly crime boss Russell Bufalin. That’s because the director and star demanded an unobtrusive system with a light capture footprint and no reshoots in a controlled environment. In addition, they only showed finished renders to Scorsese, so there were no key-frame animated enhancements.

The camera system and companion software that ILM developed captured the actors’ facial performances on set and then translated those unaltered performances to full CG versions of their younger selves with its proprietary models. ILM’s special three-camera rig (termed “the three-headed monster” by cinematographer Rodrigo Prieto for its bulkiness) consisted of two witness cameras on either side of the director’s camera.

However, despite the impressive Oscar-nominated VFX work, the de-aging couldn’t overcome the fact that it still looked like three old men with young faces. This created a disconnect between appearance and performance, which ILM apparently solved with their AI-generated tech on “Dial of Destiny.” But not before hiring YouTube deepfake expert Shamook after showing what he could do with de-aged Luke in “The Mandalorian” and “The Irishman.”

Best of IndieWire

• Where to Watch This Week's New Movies, from 'The Little Mermaid' to 'You Hurt My Feelings'

• The 55 Best Animated Movies of the 21st Century

• The 80 Best Comedies of the 21st Century, from 'Neighbors' and 'Frances Ha' to 'Jackass Forever' and 'Borat'

Sign up for Indiewire's Newsletter. For the latest news, follow us on Facebook, Twitter, and Instagram.

Click here to read the full article.