‘The Irishman’: How Industrial Light & Magic’s Innovative De-Aging VFX Rescued Martin Scorsese’s Mob Epic

Bill Desowitz

Click here to read the full article.

IndieWireFallTV
IndieWireFallTV

Once again technology caught up with need when Industrial Light & Magic developed an innovative, digital de-aging process for Martin Scorsese’s character-driven, non-linear mob epic, “The Irishman.” Thanks to a new capture system based on light and texture and a new software that altered the shape of faces without animation, Robert De Niro, Al Pacino, and Joe Pesci looked decades younger as hitman Frank Sheeran, Teamsters leader Jimmy Hoffa, and Philly crime boss Russell Bufalino. The impressive results put ILM on the Academy’s longlist this week for the VFX Oscar.

Frankly, it couldn’t have been done any other way to preserve their career-capping performances in this zigzagging meditation on friendship, betrayal, and mortality, spanning four decades.

More from IndieWire

“This was a movie about conversations and we needed a new system because the actors wanted to be themselves,” said ILM VFX supervisor Pablo Helman, who first prepared a test with De Niro from “Goodfellas” in 2015 that got the film greenlit by Netflix. The costly VFX de-aging, therefore, became the tech centerpiece, with 1,750 shots created for two and a half hours of footage, which was the equivalent of making two movies in one — but in nine months instead of 18.

Helman and ILM undertook a two-year, NASA-like science project, developing a special camera rig (with cinematographer Rodrigo Prieto and ARRI) along with a markerless, light-based performance capture software called FLUX. The great advancement of FLUX was in manipulating the younger faces through rendering and compositing rather than traditional keyframe animation. That’s because Scorsese wanted unadulterated performances. One noticeable difference was in the lip movements. Normally facial markers separate the movement of the upper and lower lips and obscure the deformation. Not anymore. Here they had the proper deformation by adding reverb.

In prep, the actors were shot separately, each performing various facial movements, then again on a special lightstage to capture a range of lighting conditions. That data was then used to create 3D models of each contemporary actor (all in their 70s).

All scenes were shot on set using the proprietary camera rig (dubbed “the three-headed monster”), which consisted of two high-res ARRI witness cameras attached to and synced with the primary RED production camera. This system captured the facial performances from multiple angles with all of the necessary on-set lighting, and also threw infrared light onto the actors’ faces while remaining invisible to the production camera.

“The problem is, the system gets confused when there are really big changes between key light and shadows, so we had to calibrate and equalize the light,” Helman said. “That’s why the left and right witness cameras are infrared, which don’t have any shadows. It doesn’t affect the lighting on set but it does affect the way the computer sees the data.

“The system analyzes the lighting and texture information and makes a renderable mesh of geometry for every frame,” Helman continued. “The system then compares it to the digital double of the contemporary actor. Then it gets re-targeted to the younger model.” The modeling, muscle and skin development were all customized for FLUX on a per-frame basis. Additionally, ILM researched and accumulated thousands of historical photographic and video references of the actors’ younger performances, and built an AI-based engine to guide them in real-time to find the closest match when de-aging.

ILM created four separate models for each character at different ages with built in sliders. This allowed them to best match the raw performance, most often by aging up slightly. The shapes of the neck and chin were completely redone on each model. “There’s a basic outline to the face that needed to be addressed,” said Helman. “The eyebrows, eyelids, the wrinkles around the face, around the mouth, and around the nose. When you take that away, then even if you keep the performance, you have to make some concessions.

“To Marty, these were damaged people and he defined the characters based on having a rough life, and de-aging was skewed toward older,” Helman added. “From that point of view, you will see Frank Sheeran as a young version of the older Frank [played by De Niro in his 80s, and relying more heavily on makeup except for the hairline]. We did some 2D work on the hands and on the bodies. But we also aged up De Niro if he didn’t look old enough. For instance, when he goes shopping for a casket.”

Since Scorsese was only interested in seeing final results, Helman would deliver a wide range of complete renders once a week for the director and editor Thelma Schoomaker to evaluate during post-production. “It was really difficult to finish one sequence and then move to another one because it’s kind of a puzzle,” Helman said. “Marty would look at it side by side and sometimes catch something that was important to him. Oddly enough, the three and a half hour cut had not changed for nearly a year. I was surprised. I thought Marty was going to have a problem with different takes and we would have to recompute everything. But he did not do that. Obviously, he had a great vision of what he wanted with Thelma.

By far, De Niro’s Sheeran was the most demanding to de-age because he’s so reserved. And the most difficult scene to get right was when Sheeran was summoned to a restaurant by Harvey Keitel’s Angelo Bruno (the head of the Philly crime family) and Bufalino to explain his surreptitious arson gig. “That was a very difficult scene for all of us because we haven’t observed how ruthless Angelo Bruno can be,” Helman said. “The only reason we think that he’s ruthless is because of De Niro’s performance because he acts like a little kid that did something wrong. If you saw the original dailies, he looks like a turtle trying to hide in its shell. If you take the neck out and unwrinkle it, then you lose the body language.

“That was a conversation that took months to get Marty and Thelma to understand and for me to go back and show them what we could do. And those were shots that we did at the beginning and then we went back after the software was acting a lot better and faster to get the emotion right. And I think it made a difference.”

Helman reflected on an early question by Scorsese about ILM’s de-aging gamechanger: “What is this going to do to my movie?” Helman’s answer: “It shouldn’t do anything to your movie. The only thing it should provide is context for the narrative so that the story makes sense.”

Best of IndieWire

Sign up for Indiewire's Newsletter. For the latest news, follow us on Facebook, Twitter, and Instagram.