In conversation with “Digital Einstein”

When ETH Zurich President Joël Mesot lowered himself into an armchair in ETH’s biggest lecture hall, he was faced with an unexpected guest. Sitting opposite him was one of the most famous personalities of the 20th century, instantly recognisable by his moustache and dishevelled hair. The surprise guest at this year’s Rössler Prize ceremony had come in a very unusual guise: as a digital human, who asked questions, gesticulated wildly and bluntly informed Mesot that he couldn’t imagine a worse job than being ETH President. All that from no less than Albert Einstein!

Albert Einstein was awarded the Nobel Prize “for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect” in 1921. To mark the 100-year anniversary of Einstein receiving the Nobel Prize, the idea came about to bring ETH’s Zurich’s most famous alumnus back to life in digital form.

By order of Joël Mesot, researchers led by professor of computer science Markus Gross, together with the ETH spin-off Animatico (external page https://animati.co/), developed an interactive platform in which Einstein – as a digital twin – talks about his years in Zurich. The platform is a new aspect of the university’s strategy to strengthen its dialogue with society and, above all, to give younger generations a greater understanding of science by drawing on Einstein’s fascinating character. The platform can be viewed at various locations around ETH Zurich (see box 1) and is designed to be used at ETH events in Switzerland and abroad.

Combining art and cutting-edge technology

The digital twin is as communicative as the original Einstein. “Digital humans like Einstein who can see, hear and interact with the person in front of them are driven by complex algorithmic systems,” says Christian Schüller, one of the three founders of Animatico. Key technologies include natural-language processing and dynamic rendering of the individual’s expressions and body language. For the artistic 3D modelling, the creators deliberately opted for a younger Einstein, as he was when he lived in Zurich.

The screen and comfy-looking armchair – the visible parts of the platform – are packed with technology. To help Einstein identify interlocutors and ask them questions, a small camera is used to scan people’s movements and reactions. Whenever someone addresses Einstein, microphones filter their voice out of the ambient noise. Their words are then converted into text and analysed by language-processing software. The program uses machine learning to determine what they wish to say and then employs a dialogue algorithm to choose the most appropriate answer along with expressions and gestures from a finite universe of possible reactions.

Intuitive responses as a unique selling point

The answers are part of a defined dialogue tree that encompass various topics, such as Einstein’s famous theories, his time as a student and his personal friendships. Swiss artist and comic Karpi (external page http://karpi.squarespace.com/), wrote these dialogues – and is the voice behind the digital version of Einstein. A special recording technology was used to sync the voice, facial expressions and gestures as effectively as possible. An algorithm is responsible for adding a sense of randomness to the answers, so that the dialogue can keep taking new twists and turns.

If, for example, the “Digital Einstein” notices that his partner in dialogue has turned away, he will quickly ask whether they are distracted or bored. If he doesn’t understand something, he will apologise and ask the person to repeat the question. “These intuitive responses differentiate digital figures like Einstein from conventional question-and-answer assistants like Siri or Alexa,” explains Schüller.

Einstein still has a lot to learn

Being able to take part in an entirely unscripted conversation is still the stuff of pipe dreams. Einstein is the main driving force behind the conversation. He asks the questions and proposes topics that the user can explore further by answering him. Schüller notes that one particular challenge involves the audio and visual data having to be processed in real time to ensure the conversations and interactions are fluent. Protracted waits and delays would immediately disrupt the conversation and destroy the illusion.

At present, all of Einstein’s potential answers and responses have to be defined in advance by the developers. While this allows the content to be controlled based on preset storylines, it does restrict the fluency of the conversation. “In future, it will no doubt be possible to have more open-ended conversations,” states Professor Gross. “However, digital figures like Einstein will need to have access to a pool of knowledge and have self-learning abilities. They also require a working memory and emotions. The machine learning and language processing applications required for this have not yet been sufficiently developed.”

Digital avatars – a research field

Advances in the field of artificial intelligence and computer graphics have led to progress in the development of avatars and chatbots. Digital characters like Einstein offer the chance to make interactions between humans and machines more intuitive as well as more accessible for people who are less tech-savvy, or those with disabilities. There are many potential areas of application. Animated figures, for example, could be used at ticket offices or hotel check-in desks, in product consultation sessions, or as virtual coaches in the health care industry.

At 20 October, “Digital Einstein” will take up residence in ETH’s Campus Info Hönggerberg for one week. Students, employees and visitors to ETH will have the opportunity to speak to Einstein and find out more about his time as a student and professor at ETH.