04Film & TV· Vision Team

The Uncanny Valley

ScienceProbability 90/100

// A story from 2051

Day 25 of shooting Lake Kolyma, in the year 2051. The director of photography, Joel, is watching a scene on his VR glasses. Actress South West glides across an eerie landscape. She is playing the part of a two-headed tribal leader who has fallen in love with an intrepid explorer, played by actor Heath Hoffman. A 1000-strong crowd is cheering them on as they fight-dance in an epic showdown of their love. It’s safe to say that romcoms will be somewhat weird in thirty years’ time. The setting – an abandoned mining town in Kolyma, Russia – has lush forests, snow-capped mountains and a frozen radioactive lake. Buildings, transformed into post-apocalyptic Snow-White fortresses, are overgrown with animated thorns.

The director, Margot, yells cut. The acting is locked but she wants to make a change to the background. She motions to Joel to move a mountain on the horizon. He huddles with the technical team to make it happen, which takes about an hour. The delay could irritate a diva actress but it’s no bother to megastar South. Why? Because she’s not even there. The actress you see on the screen is a synthesised version of North, created in a Hollywood backlot a few months ago. Her co-star? A meta-human mash-up of the iconic deceased actors Heath Ledger and Philip Seymour Hoffman – as voted for by the audience of the streaming channel. Fans also voted on the costume the actor is wearing, which doesn’t quite fit with Margot’s vision but is good for marketing.

We may be moving mountains in 2051 but that’s not what’s happening here. When Joel takes off his VR glasses, all he sees is an enormous circular LED wall. On it, we see a super-realistic projection, complete with crowds of people (a mix of extras and meta-humans). Only some placeholder stand-in actors and key props are there for scale in the studio – the rest of the magic happens on a massive interactive holodeck.

When they wrap for the day, Margot sits down with the scripts for the second series – created by an Oscar-winning script bot, Mollywood. Known for her intense family dramas, this will be her first foray into adventure comedy, so there is a lot of hype around it. Soon, she finds herself laughing out loud. The humour is razor-sharp and remarkably on point for an AI-generated script.

Meanwhile, Joel heads into an editing suite to oversee the live rendering of the next locations. The plot sees characters time travelling to a tropical destination, so a team has been dispatched to Tristan da Cunha – a group of volcanic islands in the South Atlantic Ocean. It had an eruption earlier that year, plus it’s so incredibly remote that it wouldn’t be cost-effective to take the crew out there. Luckily, it’s no longer necessary with specialised equipment that captures scenes in 3D down in pixel-perfect detail. The small team is touring the remote island, capturing specific mountain and beach scenes while the shoot is happening. Margot wants a bigger mountain, so a bigger mountain she’ll have by morning – captured and rendered in just 24 hours. If that doesn’t work, there is always a vast library of lidar (light detection and ranging) scanned locations from every corner of the world to choose from. Lidar scanning of Tristan isn’t possible because of the mirror town recently built by an NFT billionaire. 3D volumetric scanning has advanced a lot but still can’t deal with objects like mirrored surfaces.

// The science behind it

Welcome to a world where (literally) nothing is real

Filmmaking in 2051 will be in the hands of a small group of creatives, and powerful virtual production software with unimaginable processing power. Where before millions of dollars were poured into moving hundreds of crew to exotic locations abroad and huge set builds, everything is now in-camera at once.

Locations and props will be hand-picked from content libraries, offering an endless supply of textures, animals, furniture, foliage, vehicles and more to blend into the scene in real time. No more working in foreign locations at the mercy of weather or daylight shooting hours. No more spending months in post-production. No more expensive re-shoots. No more movie flops because the director couldn’t see it all coming together until it’s sadly too late. Assets are created once and used repeatedly. Films and TV shows can now be created on more flexible schedules, with actors working on several projects at once. Talented creatives will design and 3D print imaginative props, while smart costumes will be programmed to display specific textures to boggle the human eye.

Casting will also take an interesting turn. Actors will now film a few key scenes, and their digital alter egos will step in to finish the job. If budgets are tight, they don’t need to be present at all. Deepfakes – machine learning systems that can learn to exactly mimic the data you feed them – will have progressed so far that it will be quite possible to create super-realistic facial overlays. Actors can be recreated posthumously, or filmmakers could decide to use deepfakes to create new actors out of 10 different faces. Audiences will be involved with content creation – just as the NewNew app polls followers to tell influencers what to say, wear or do. Writers will still create initial scripts but bots will finish them off.

It will be a world where almost nothing is real unless it has true artistic meaning. A world of wonderful efficiency and collaboration.

Virtual production is evolving fast

One of the biggest stories of 2020 in film and gaming circles was the revolutionary technology used to make the Star Wars spin-off, The Mandalorian. A collaboration between Industrial Light & Magic and a real-time game engine by Unreal, it was the biggest invention since the green screen. This monster LED screen (now commercially available as Stagecraft) is the largest, most sophisticated virtual filmmaking environment ever created.

The virtual production unit measures 20 feet tall and 75 feet across, offering a curved 270-degree cinematic landscape. Before this, either giant sets were built, or actors behind CG characters would play out their roles in front of a green screen. Now, lighting, sound, location, VFX and acting all come together in the moment. Soon, the entire workflow will go real-time, with less time waiting for computers to process. Even sooner, there will be LED screen set-ups from Cape Town to Mumbai. Unreal is already marketing its gaming engine beyond filmmaking – architecture, broadcast and live events, training, automotive and cultural tours are some of the industries it’s focussing on.

As creatives were getting excited about the opportunities this virtual environment presents, Unreal unveiled another jaw-dropping innovation – an app that can build lifelike virtual humans for games and movies. Called MetaHuman Creator, it makes face customisation as simple as a Sims video game. Skin complexion, wrinkles, stubble and freckles – it’s all there, down to each creepy detail. These digital humans are ready for their close-ups. They are very close to conquering the so-called ‘Uncanny Valley’ – where human replicas that look almost like real human beings elicit feelings of eeriness and revulsion.

Film purists like Quentin Tarantino will argue that synthesised actors will never give you the raw emotion or the Jim Carrey-esque facial acrobatics of a real-life actor. It will be the “death of cinema”, they’ll protest – just as Tarantino did in the digital versus film debate ten years ago. But at $70 a minute, film is expensive. It probably won’t be long until he too succumbs to digital filmmaking. Perhaps even digital actors.

Speaking of production costs, another huge expense is of course shooting on location. This is where lidar technology is bringing exciting advancements, fast. Lidar is becoming increasingly popular for creating realistic computer-generated imagery and visual special effects. Traditionally used for mapping in industries like civil engineering, mining and transportation, lidar is now used to scan buildings or even entire cities in 3D for visual effects. The HBO series Game of Thrones relied heavily on lidar to create and recreate their sets. The old city of Dubrovnik was rendered in 3D as the model for the fictional city of King’s Landing.

Thirty years from now, we may well have most of the earth’s cities mapped out in this way. Perhaps drones could capture scenery inside a volcano, a rain forest, underground cave, or mountain range?

The one thing that won’t change: talent will always have a place in this world. When researching this article, I came across a fun 2004 interview from the future where Julia Roberts “reflects on acting with her synthetic younger self” in the year 2022. While some of the technology sounds plausible (a suit with thousands of location sensors, lifting a ‘skin’ from her Pretty Woman performance), we also know that the legendary actress has nothing to fear from digital actors. And there is no reason to use digital stand-ins with the constant influx of incredible acting talent.

People love people. Real people to whom they can relate. People that age like they do and portray the emotions they feel. They love the versatility of an actor playing a lawyer, then an addict, then a dragon slayer. They love their messy private lives and dodgy accents. Digital actors are boring and will never take centre stage.

But, perhaps actors should be less worried about losing royalties, and rather capitalise on their performances. If digital art can be sold as NFTs, why couldn’t an actor sell iconic facial expressions that can be mashed up with meta-humans? Perhaps everyone should be less wary of digital versions ‘taking over’ and more excited about the endless creative opportunities that lie beyond the uncanny valley.

// Sources & further reading