Martin Scorsese gave ILM special effects supervisor Pablo Helman the script for The Irishman in 2015 on the set of Silence, and he was hooked. It took the visual effects maestro and a team of 500 people four years to work with Scorsese to de-age Robert De Niro, Al Pacino, and Joe Pesci for the epic mob drama. Helman told me the 1,750 shots they created for the film is the most difficult work he’s done in his 30-year career.
During our conversation, Helman tells us every face, gesture, and movement in the film is intentional, with Scorsese utilizing the new technology Helman pioneered to reflect on the passage of time through Frank Sheeran’s memories–as if the character remembers his life of crime filtered through his current age and physical condition.
I spoke at length with Helman about transitioning from the helmet cams made famous by Star Wars: The Force Awakens to a pain-free system that’s going to be used industry-wide going forward. Actors can thank De Niro for his reluctance to use the bulky system; this innovation is going to change the game. In our wide-ranging, in-depth conversation, we also discussed deepfakes and how, or if, they’ll ever make it into mainstream movies.
The Film Stage: I was looking at your background. You studied music and education. How did you get into visual effects?
Pablo Helman: When I finished my degree in composition, I went to school at UCLA and I wanted to write music for films and so I started writing music for television in Los Angeles and this is the mid-1980s. I started writing music for documentaries and I worked for a PBS station in Los Angeles and they told me my music fit with all their edits and that they didn’t have a job for a musician but they had a position open for an editor and that they could teach me. So right out of college I was an editor for about seven years at PBS in Los Angeles and it was right about the time when things were changing from optical to digital and so I was really into technology. I was getting my master’s in educational technology and I was always into computers. So I was kinda the perfect fit to help them make that change from optical to digital. I got a job on Star Trek: The Next Generation as a digital compositor. So that was kinda the beginning and it fell in my lap. I was always a musician; the funny thing about it is that there is a lot of stuff that is in music that is in visual effects in cinema: color, texture, sequence of scenes, the rhythm of telling a story, having a beginning, middle, and end. Those kinds of things are all the same, those disciplines, so it’s all very natural.
Will you talk about the new technology you created for The Irishman? And how that technology works?
The whole thing started in 2015. I had a conversation with Scorsese. I was working on his film Silence in Taiwan and we were talking about this technology of making actors younger and he said to me that there was a project he was thinking about and he was trying to do for a long time called The Irishman that sounded like something we could work on. He sent me the script overnight and I read it overnight and in the morning I was really hooked. I told him we should get together and do a test. I knew it was Robert De Niro and Marty told me that one of the things that was going to happen was that Bob would probably not want to wear any kind of markers on his face and also not the helmet cams that everybody was using in 2015. And so I was even more intrigued, having a call with ILM and saying we should go after this project and innovate to create a system that was markerless so the actors could be liberated from us, the technology people. [Laughs.]
We did a test with De Niro with no markers in which we recreated a scene from Goodfellas, and that turned out great and basically that greenlit the movie. For two years we worked on this system that could capture the performance on somebody’s face without any kind of markers or helmet cam and on set with theatrical lighting. In two years we went through all kinds of variations of what we wanted to do and we came up with a combination of software and hardware. The software picks up the performance but the hardware is basically three cameras rigged to shoot the actor.
The center camera is the director camera and the left and right of the director camera are two infrared cameras that are film grain, so they’re high-resolution cameras, and they throw infrared light onto the actor and that is to get rid of any shadows that the actor has on the face. Because it’s in a different spectrum that the eye cannot see then we have to influence the lighting from the DP. And then the software takes a look at those three cameras and takes a look at the lighting that is falling on the actor’s face and the textures and comes out with what we call geometry mesh that mimics one frame of whatever the actor is doing by performing that geometry. And then it takes a look at the next frame and the next and so on and so forth and then you end up with the scene built.
What do you call that innovation?
The software is called FLUX and the three-camera rig, the DP called it “three-headed monster.” [Laughs.] It’s basically three thirty-inch rigs so the cameras are right next to each other and they all need to move together. It weighs about sixty-four pounds so that it can actually fit in any of the rigs. Other than that, there was really no interference with the acting, that’s part of the reason we had to come up with something like this. There’s no way this movie could be made with these actors wearing markers on wearing gray suits and this other stuff that would. There was no way of doing this in 2015.
I think one of the reasons why we had to do this, I saw the difference in performance right away. I had been working with that other technology with markers with other actors and sometimes when you’re in the shot and you’re thinking, “There’s something wrong here but I can’t really tell what it is. Maybe the eyes are not connecting, the actors don’t know exactly where to look and there’s a lot of hesitation in some performances.” Research tells us when two actors are on set they check each other’s eyes, but they also check each other’s bodies for clues about whether that other person is reacting to their acting and they check the set and they check back with the actors that are in front of each other. So there’s a lot of reality to that performance, what they call the truth, actors are always looking for the truth and so the only way to get to that truth is to get the actors to be in the moment, in that set, with the lighting, without having to worry about technology.
Did you use the FLUX software in conjunction with ILM’s cineview app so Scorsese could see how the actors would look in real-time?
In this specific project, that wouldn’t help because Marty is such a performance person and has known these actors for fifty years and so he directs the performance that is in front of him, not anything else. And basically it’s our job in post-production to get to a younger performance that says exactly what Marty was seeing. The other thing is that because the actors are living through forty years of their life in the movie, there’s a design aspect to the whole thing and that is something Marty hadn’t thought of and that was very difficult. It would have been very difficult to do live for him to see.
One of the things he did, and I think this is one of the things the audience needs to realize, is that Marty doesn’t want to rewind thirty years and see Jimmy Conway from Goodfellas. He wants to see a younger version of Frank Sheeran (Robert De Niro)–you know the character, not a younger version of De Niro. Same thing with Joe Pesci, in this movie at fifty-three when we start seeing him is very thin and Pesci was never that thin when he was fifty-three so it’s a design thing with Marty. I always talk about Bob in The Untouchables, when you see him playing Al Capone he had gained weight, he lost some of his hair, and it’s not like anybody would look at him and go, “he doesn’t look like De Niro!” You take a look at the characters as how they are designed for that specific narrative that you’re trying to portray.
Even when Frank is in the WWII scene, he looks a lot like current-day Robert De Niro, not what he looked like during Taxi Driver.
Some people have asked why Marty didn’t just cast another actor for the younger scenes. The way the movie’s edited which is back and forth-back and forth, Marty was looking for a way for the audience to connect the actors with the characters and not have to worry about having to figure out who this person is, they’ll make the connection. The brain is desperately trying to make sense out of what we see. Same as if you’re reading and there’s a word you don’t know, your brain is trying to take a look at context, and then makes up the reason why that word is there. It’s the same thing with visuals effects. There might be one shot that doesn’t quite match there because the other shots are around it, in context, your brain can accept it.
De Niro has one scene where young Frank is at a spa and he’s shirtless. How did you de-age his body?
That’s a good story because we had a body double lined-up for that but Bob didn’t want to use him. He thought that he could do it. He was in the robe and I said, “Uhhh, let’s see what you got.” [Laughs.] And I said, “Fine you look great.” So yeah it is him that is bare–we did not use a body double. We reshaped some of his body because he’s seventy-six years old, but it’s mainly his body. I think it’s part of the whole method acting and the whole philosophy that those actors, incredible actors have. Which is that they are looking again for that truth and it doesn’t matter what they do. They’re themselves and they will do that performance that makes them who they are.
Marty and Thelma Schoonmaker have worked with these actors for over fifty years and they know what they are looking for. Thelma knows Bob’s hands, for instance. We couldn’t use a hand model, we had to use Bob’s and then de-age the hands. And that was because Thelma is very aware of the way Bob walks, so we had to use them as they are and then there were hundreds of shots in which we had to change their appearance a little bit. Like the way they kept gaining weight as they got older for instance. At the end of the day, Marty didn’t mind some of the things the actors were doing with their bodies because in his mind this whole movie was basically somebody else’s point of view. And it’s kind of like the memories that we have, like when I was four or five years old, I lived in a house and I remember the room where I slept being very big and then thirty years later I visited the house and I realized the room was very small. So the memories and point of view that Marty is going by is the point of view of Frank Sheeran. I don’t know if it happens to you, but being the age that I am, I always think of myself as seventeen. And I don’t think anything has changed in me. I’m pretty sure other people would beg to differ but to me, I look the same way I did before. It’s part of what’s happening in the movie. The points of view are definitely of the older characters.
Talking about movement coaches, De Niro’s arms are stiff throughout the movie. Is that a reality of his age you couldn’t get around?
No, you can get around anything really. For that matter, we could’ve replaced it completely, but that wasn’t the idea. I think it was an aesthetic decision for the mood of the movie. The movie is about many things but one of them is about the passage of time. Especially from the point of view of somebody who had gone through that time. I think that Marty does not mind at all. Same thing with the younger self section of the movie. We worked on that transition for months. We had softer approaches, harsher approaches, all kinds of different ways to get into the younger section of the movie. That was something that Marty really liked to do. He liked to just plunge himself into it, and adjust yourself to a new way of seeing this character. So I’d say that anything that Marty does is something that he has thought of. It’s part of his vision. It’s not something that’s been overlooked.
Since we’re talking about de-aging faces and bodies, I can’t help but think about Deepfakes. What’s your impression of them?
I think it’s an A.I. way of looking at things, but is very dependent on the resolution of the images you’re working with. I don’t think it’s something that is going to permeate into a 4K movie that you see in the theater. Just because those things don’t hold up at high resolution. I think it’s a good exercise and I think it’s a way of looking at technology or a way of looking into the future. It’ll probably take five years to come up with something that is actually high resolution to make those changes.
An example of Deepfake VFX technology with De Niro and Pacino.
In this movie, we did use some A.I. technology just as a sanity check, which we spent two years building a library of all the actors in the right ages from all these movies that they had made. A big video database with some high-resolution images so we would render Robert De Niro in the movie at 45 years old and then we would run that image through our library so the program goes through the library and finds a life frame from all these different movies. That gives us a chance of how the image that we render compares to the images of the actor at that age and then we would, as a sanity check, take a look and see if we were missing something. So that tells you that kind technology is going to make itself into movie making. The visual effects part of things eventually. So it’s out there, but I wouldn’t replace it with anything we’re doing right now. I don’t think it’s quite ready.
I was reading there are 300 scenes in this movie. Which one was the most challenging to create?
Any scene with a bunch of people sitting down and talking. Anytime something was in close-up, from forehead to chin, that delivers dialogue and delivers extreme poses. Something like Al Pacino where he’s talking to a crowd and he’s exerting and trying to make a point. Those kinds of scenes with a lot of emotions are very tricky and very difficult and it’s something we pay attention to more than others. Having said that, there’s no one character, none of the three characters, that is more difficult than the other. Once you’re in the shot, the subtleness that they’re going for from forehead to chin with all three characters, they’re all difficult.
I think it’s a very difficult thing to get behavioral likeness; how somebody is so iconic, you want to make sure that you hit all the right notes at the right time. You want to make sure that their behavioral likeness is always there or else you’re gonna be in trouble.
What’s the VFX shot count for the film?
The VFX count for the show was 1,750 shots.
Wow.
Yes, it’s because the movie is like three and a half hours, so basically it’s like two movies that we worked on. I was just watching a movie the other day and it was like two and a half hours and we still see all these visual tech shots go by. They’re all back to back to back. I mean Star Wars is probably about 3,300 shots, but these 1,750 are the most difficult stuff that I have done in my 30-year career.
The Irishman’s performances are subtle. Sometimes the difference between a smile and a wince is about two pixels and a smile is one thing and a wince is a completely different thing and Marty picks up all these different things. The concern that he feels on one face has to be there a hundred percent or else he’s not fine with the shot.
What was the size of your team on the project?
The team was 500 people throughout the world. They were in San Francisco, Vancouver, there was a green scene in Ireland, there were some people in China. So it was 500 people which was great and I worked on it for four years and it was a project that required a lot of thinking, a lot of rethinking and a lot of staff. And it’s a project in which the chief mandate was for the actors to be on set without markers and without interference from the technology. It’s a project that was completely geared towards performance and actors. I think that markerless technology is gonna be the next thing. Actors are going to look at this and think, “Wait, I don’t have to wear these 38 markers on my face? And don’t have to go through two hours of make-up and a bunch of technology that is not a part of what I’m hired to do?”
It’s definitely going to change a lot, especially with these comic book movies.
Yeah. I think that’s the idea.
The Irishman is now on Netflix.