An Oscar for Medusa

“It was midnight when I got the email from California. I had to call my parents in Canada right away. After that, I barely slept a wink all night,” says Derek Bradley, recounting the moment in early December when he found out that they had won the Sci-Tech Oscar. Bradley was the last to join the group of four who had the honour of accepting the coveted award for their Medusa capture system in Hollywood last Saturday.

Thabo Beeler, who laid the foundations for Medusa with his doctoral thesis at ETH Zurich and who currently heads the Capture and Effects team at Disney Research Studios (DRS), was in a meeting in California at the time. “When I returned to my desk, I was congratulated by everyone – but I didn’t even know why at first,” he says. The news that Medusa had won the Sci-Tech Oscar travelled like wildfire: “Seeing people’s overwhelming emotion made it clear just what a great honour this is – it’s the top award in our field, after all.”

A revolutionary new facial scanner

The Medusa system uses scanners to record facial movements and then renders them into dense animated grids (meshes), breathing new, more realistic looking life into animated figures, for instance. What makes the system so special is that it does not require any anchor points on the face, such as markers or makeup. “[Medusa is] pushing the boundaries of visual fidelity and productivity for character facial performances in motion pictures,” the Academy of Motion Pictures noted. The Sci-Tech Oscar is awarded annually to people and companies whose inventions and innovations have had a significant and lasting impact on film production.

The true father – or grandfather – of Medusa is Markus Gross, a professor of computer science at ETH and Director of Disney Research Studios (DRS). He came up with the basic concept for the system together with his former doctoral student Bernd Bickel, who is currently a professor at the Institute of Science and Technology Austria. The two men gave Thabo Beeler the task of building a high-resolution facial scanner for his Master’s thesis. The idea was to develop something completely novel – a simple and inexpensive scanner. Most of the existing systems used laser scanners and were complex, which made them expensive.

Capturing animated faces

Once the three men were satisfied with the potential demonstrated by one of the prototypes, Beeler’s Master’s degree was finished, and he continued on with his doctoral studies – as one of the first joint doctoral students at ETH and DRS. Derek Bradley joined the team during this period. The Canadian had mainly worked on clothing motion during his own doctoral studies. An ideal combination, he says: “Together, we developed software that uses the revolutionary static scanner the three men had developed in order to capture facial movements and, most importantly, longer sequences.”

The challenge was to separate two movement patterns: facial expressions, such as when we grimace, and the movement of the head itself. Bradley and Beeler worked out a way to achieve this separation, which is crucial for the visual effects industry. Medusa was then finally ready to use in Hollywood.

In recent years, Medusa has been used in over a dozen productions, including Avengers: Infinity War, Pirates of the Caribbean: Dead Men Tell No Tales, Star Wars: The Last Jedi, and Spider-Man: Homecoming, to name just a few of the more popular recent films. The Sci-Tech Oscar has now paid tribute to the four-man team’s impressive achievement.

A long-term vision pays off

For Markus Gross and Disney Research Studios, this second Oscar follows the one they already received in 2013. Does anything feel different? “I’m just as elated, but I’m particularly proud of the second Oscar, because it recognises our commitment and perseverance,” says Gross. His team won its first Oscar for its Wavelet Turbulence software, which is used for low-cost visual renderings of explosions, smoke, and liquids. “The actual scientific work took a few months,” says Gross. “It was a brilliant idea, and we did the right thing by making the software open-source and not patenting it,” he says, explaining the reason for its rapid distribution.

The research for Medusa was much more extensive. It is a complex hardware and software system that was developed based on a series of scientific publications and that contains several patented elements. The development took several years and involved a number of different people. What is most important, however, says Gross, is that: “Medusa is the product of a far-sighted vision that we followed when we founded Disney Research Studios (DRS): to overcome what is known as the ‘uncanny valley’ effect.” The term refers to the phenomenon that occurs when animated faces in films or video games appear not quite realistic, i.e. uncanny.

A successful partnership pays off

Medusa can also be seen as a great example of the partnership between ETH and Disney, says Gross happily: “An ETH student doing research at the DRS lab, collaborative scientific publications, intellectual property developed and patented as a team, and, finally, the overall success of a project that will have a lasting impact on the film industry and has now won an Oscar – this is a fairytale collaboration!”