If you’ve had any desire to see one of the greatest films ever made in its proper form, you’re out of luck. Unless you’ve been able to catch one of its rare 35mm screenings or got your hands on an even-rarer (and poor-quality) out-of-print DVD, you’ve likely had to resort to a shoddy-looking YouTube rip to see Elaine May’s 1972 black-comedy masterpiece The Heartbreak Kid. Following the journey of immediate regret and romantic conniving on the part of Charles Grodin’s character, who just a few days into the marriage to one woman (Jeannie Berlin) falls for another (Cybill Shepherd), the film has sadly wallowed in distribution woes. Its rights are in the hands of a pharmaceutical company that has little interest in making any version of it widely available, much less restoring it. Thankfully, a tech-fluent cinephile has come to the rescue.
Gus Lanzetta, a Brazilian with extensive experience in videogame preservation, has taken on a number of projects to help rescue moving pictures that have succumbed to such unfortunate fates. Through AI upscaling, he now created a much-improved version of The Heartbreak Kid that looks better than any home release we’ve yet seen––and you can watch it for free on the Internet Archive, where it is also available to download. We spoke to him for an in-depth conversation about his process and why his mission to preserve culture is particularly vital.
While not a software engineer himseflf, Lanzetta uses models made by others to create these AI upscales. “AI, in this case, means machine-learning,” he begins. “So you train a model by making it look at stuff. You show it a high-res picture of something and then a very low-res/compressed version of the same picture or image. You make the model learn what is lost in detail when you do that. So from looking at millions of images like that, the model then can start to create its own technique, in a way, to do the reverse process. It starts to learn if this image loses these kinds of details and information when it is downscaled, then its upscaled ‘original’ should look something like this. So it tries to approximate what the original file would be.”
Following an upscale of Todd Solondz’s Happiness, Lanzetta polled Twitter and said he was open to suggestions of movies stuck in distribution limbo. Of course, there was more than one response for Elaine May’s little-seen masterwork. “I looked it up and remembered knowing of the movie and having a somewhat fresh memory of it because it was mentioned a lot when Grodin passed, so it seemed like a good place to put in the effort. I love Grodin, Berlin, and Shepherd’s work in other things, and the movie’s look of a soft focus combined with a very grainy picture lent itself well to the upscale. So I went for it.”
Before getting into the particulars of this upscale, he offers a bit of history of the source file he used, which was made from a process called Telecine. “Telecine is the process when you get a film and you want to make it digital video, be it Beta Tape [aka Betamax] way back in the ‘70s and ‘80s, or you want to make it digital video nowadays. When you do that transfer you are basically scanning the film, like you would scan your vacation photos or just a document. So we have to realize this movie was probably scanned in the ‘80s or ‘90s or, worst-case scenario, in the ‘70s to tape and the DVD was made from the original tape master––which is a Beta Tape master, still better than VHS but not ideal. Scanners were not as good and maybe the process was not as well done or as carefully done as it would have been today. So you find the best that you can and also, after the scan, you have to realize DVDs were digital video when digital video was in its infancy. So compression technology was bad. DVD-compression technology is MPEG-2-based. It was derived from the JPEG compression stuff that we still use today for still images, but it has largely been obsolete by better codecs and ways to make video into digital information and make it an efficient, deliverable package of video. So you don’t store every single frame as a full image. You basically have keyframes and between keyframes, you have every frame pointing out changes in color and what changes in the frame. You do some relative changes between keyframes which really helps and that’s why we have very high-resolution playback of stuff on our phones and TVs, etc.”
For his AI upscale projects, the first step is figuring out what is the best version he can find, and as a Brazilian with Internet access in the early days of piracy, he knows where to look. “In this case, it was a DVD release of the movie in the ‘90s. So that means at some point—probably not in the ‘90s—it was sourced from the tape, the Telecine. The first thing I need to do is get it back to the original framerate.” From the DVD-sourced file of 29.97 frames, which was an NTSC standard that added unnecessary frames, he returns to 24 frames a second––or, more accurately, 23.976 frames a second. “I get it back to 24 frames because now we live in a beautiful digital world with metadata on files, so your players and your TVs know what the supposed framerate is and they just go with it,” he added. “So you tell it this is the file, this is how many frames it has, and this is how many of those frames you play every second. That gets us back to all frames that are beautiful and unique, and you don’t waste time upscaling frames you aren’t going to need.”
Now with the right framerate, he takes a deeper look at the image quality. “It has whatever the problems the scan of its time had—so not a great scan, maybe a little bit out-of-focus, maybe the film itself was not cleaned. It had some dust and scratches; in this case there are a few very notable scenes, which I could go in and clean by hand.” He continues, “Once I have the correct frame-rate version of the movie in my hard drive, I then take it to Topaz Video Enhance AI, which is a tool I use because for projects like this––TV shows and movies––they are really long and it has a good user interface for me to just do a bunch of previews and test out my parameters on a bunch of different scenes and see what the results will look like before just running a render out. So that’s how I chose this particular tool, which is commercially available, but there are models and tools to do this process that anyone can use at home right now. It’s just how long do you want to leave your computer doing this, is the question.”
Using NVIDIA RTX 3070 hardware, the rendering process takes about three or four times as long as the movie’s runtime, which Lanzetta usually does overnight in an overall process that lasts a few days. “What I do is finetune the parameters as far as I can go and by looking at very different scenes in the movie and seeing what is the best overall solution. What is the best I can get that is not going to make some scenes look very weird but will make most of this look great and not create artifacts or really weird stuff? Because sometimes you have something with very little detail and the model is going to spend a lot of time trying to sharpen that or recover details or try to revert the effects of compression, like those blocky shapes in the image, especially darker parts of the image, which is a problem in this upscale of The Heartbreak Kid. So you find a happy medium. You see enough scenes of good lighting, bad lighting, a lot of noise, not a lot of noise, zoom-ins, close-ups, nice wide shots [that the AI knows what it is looking for].” After the process completes, he syncs back to the original audio and applies whatever the original settings the project had in terms of chapter marks, original audio dubs, or subtitle tracks.
“The biggest benefit of this upscale is that modern TVs, whether they be HD or 4K, are not good at upscaling content,” says Lanzetta, speaking to a larger issue in the world of home entertainment. “One of the biggest things in retro gaming is how the old analog signals in videos are not properly treated by newer TVs. They upscale in a very laggy, fuzzy, blurry way––and it applies to movies as well. So you have a DVD player or just a DVD digital file server, or even if Netflix is giving you a DVD-like version of a file, when it gets to your screen, it’s not treated the best.”
He adds, “So just getting the image above 1080p is the best thing you can do because TVs nowadays just do not handle any digital content below 1080p very well. When they try to upscale, they make it blurry, it lags a bit, and they add post-processing effects that are not really good because the manufacturers are not putting the best hardware in those smart TVs. When it receives small-resolution content it tries to use the very little processing power it has to make it ‘look good.’ But what ‘looking good’ to manufacturers of televisions is not something most of fans of the medium of film would agree with it. So what I’m doing is making a file of the movie that, when you press play on your TV, it’s always going to look better than what we had before the upscale without losing information. I’m not cropping out anything. I’m not blurring out anything. I’m not making anyone look too smooth, smoothing out faces or framerates.”
While his main goal is to have a copy that is “delightfully watchable on modern TVs or modern digital projection equipment than what we have right now,” his hobby also has a more noble and vital mission. “We’re stuck with these companies that will not put in an inch or a second of effort into things that are not profitable,” he added. “The preservation of our culture is not something to be left to motives of profit. We should have learned that with film a long time ago. Decades and decades of film in the early years of the medium were lost. Great, classic films that we have today we have because of pirated copies.”
As for his work, he hopes it brings greater attention to the creation of a proper, official Blu-ray/4K upgrade down the road. “Thankfully [the Internet Archive] has a nice DMCA exemption, so it makes it easier. It won’t be taken down as fast, but certainly rights-holders can––and in a lot of cases will––extra-judicially tell them to take it down and they will comply. But in most cases, I’m focusing on stuff that rights-holders are so many parties and maybe uninterested parties that they won’t do this, or at least will create a reason for them to care,” he says. “So if something is abandoned and no one is taking care of it and we go, ‘Hey, here’s an upscale,’ and they go, ‘Hey, I own that,’ now they’ve gone out of the shadows and we can talk. [We would ask], ‘So why is this sitting here in your warehouse and not available to people?’ So hopefully what I’m doing is getting the people who own the rights to these things to perk up their ears and realize they need to make this available––or that they need to shut up and realize they haven’t been good stewards to these works and let it be. Let people who are interested go to the Internet Archive and download it. Once the genie is out of the bottle, there’s no putting it back in. Once these copies are out there, they are never going to go away.”
The AI upscaled version of The Heartbreak Kid is now available on the Internet Archive, along with more from Gus Lanzetta.