How top-flight researchers draw global companies to Switzerland

This combination is offered by mixed-reality headsets such as the HoloLens. Equipped with a var­iety of sensors and cameras, this device has numerous features, including the ability to track eye movements, recognise gestures and objects, and understand speech. It is also able to map a physical space in three dimensions and then calculate the position and orientation of objects within this envir­onment. Armed with this information, the HoloLens can then generate digital 3D objects.

Such headsets can be used to display circuit diagrams, for example, or plans of an object or a building. “By superimposing a virtual plan on what the user actually sees of the real building, we en­able them to perform tasks without ever having seen the plan before,” Pollefeys explains. Mixed real­ity offers many other potential uses: helping train medical staff to perform complex surgery is one example; another is the ability to use robots to perform hazardous tasks, remotely controlling them through the HoloLens by means of simple hand gestures.

Real-time instructions

Faced with automation, the Internet of Things (IoT) and a general shortage of skilled workers, industry is having to rethink its approach, says Pollefeys: “What companies need now is people who are flexible enough to perform a whole range of tasks.”

When equipment is mishandled, improperly operated or poorly maintained, that can end up being expensive for an employer and also put em­ployees at risk. Mixed reality provides users of such equipment with digital instructions when and where they need them.

When a machine operator has to carry out a new or complex task, for example, arrows and other symbols can be displayed to guide them through the process step by step. Similarly, data from IoT sensors can be displayed in a headset wearer’s field of view to update them on the live status of a machine. Unlike virtual reality, where you are completely immersed in a synthetic environment, mixed reality overlays the real world – which remains visible – with computer-generated 3D holograms.

Research, education, application

In his role as Director of the Mixed Reality and AI Lab in Zurich and Director of Science at Microsoft, Pollefeys manages a team of 25 engineers and scientists in Switzerland. He also has one team member in Prague and 15 others at Microsoft’s headquarters in Redmond, Washington. “I’m proud to have assembled a team from such a fantastic mix of people,” says Pollefeys. “Their outstanding talent and expertise are crucial to our work on this pioneering technology – especially since we can’t be sure exactly what we need and how we should go about building it.” With technology evolving at such a rapid pace, he says, the team has to be incredibly flexible.

Pollefeys divides his time equally between ETH and Microsoft. He works, researches and teaches at two different locations, with a separate office at each site, following a schedule that is broken down into individual days or even half-days. “That makes it easier for me to organise my various tasks in a more structured way,” he says.

There are times when finding a solution for a specific mixed-reality application is so complex that he works exclusively with his Microsoft team. But, he adds, there are actually very few cases where it makes sense to draw a strict line between Microsoft and the university. “If we’re working on a fundamental kind of problem that requires a general solution, then we put our heads together with both ETH and Microsoft at the same time,” says Pollefeys. Sometimes, Pollefeys encounters a problem at Microsoft that he can channel directly into his teaching and research at ETH. “The students tend to find that very exciting and motivating, because they understand it’s a real-world issue,” he says.

But regardless of whether he’s wearing his Microsoft or ETH Zurich hat, he always does his best to publish research results and share code. Collaboration with other companies is also integral to his work, says Pollefeys: “It’s important for researchers like me to maintain a neutral stance.” One ex­ample of the fruits of his professorship is a key algorithm that provided the basis for the Google Live View navigation tool, which is now in use worldwide.

A win-win situation

Microsoft’s collaboration with ETH gives it a deeper insight into this area of research and helps spur innovation at the company. Finding the time to try out new things can be challenging within the confines of a corporate product team, says Pollefeys. But by collaborating with ETH, Microsoft gains an edge when it comes to exploring new perspectives and finding new solutions.

Marc Holitscher, National Technology Officer at Microsoft Switzerland, has seen the benefits of this approach first hand. “Our collaboration with ETH, and Marc Pollefeys in particular, has already yielded tremendous results, driving innovation across various sectors and contributing to Switzerland's position as a global technology leader,” he says. “Together, we continue to push the boundaries of what is possible, leveraging cutting-edge research, talent and technology to address some of the world's most pressing challenges.”

Pollefeys is quick to point out that working with the private sector is equally beneficial for university teaching: “Students get to hear about the latest ideas and challenges not just from professors like me, but also from Microsoft engineers. That gives them the opportunity to work on exciting topics, put their own ideas to the test and learn about new applied technologies.”

The collaboration between the university and the international technology giant is clearly a win-win situation, he says: “When we work on a problem with our students and ask them to explore how a new technology might be applied to a specific situation, both they and the product developers gain interesting new insights. The students are supervised by an expert who has enough experience and in-depth product knowledge to know when a certain step might be relevant.”

And there are benefits for ETH too, says Pollefeys, when professors gain a better understanding of how industry works: “My job with Microsoft gives me valuable insight into how decisions are made and how processes are run at big companies. That helps me see the whole process of technological development within a wider context.”

An additional benefit of joint projects is that they encourage networking within ETH itself – just as Pollefeys envisaged when he first took up his professorship in Zurich. He is currently collaborating with a colleague, Siyu Tang, to develop a new method of generating data using autonomous avatars. “Kitting out these virtual figures with a HoloLens turns out to be a great way of obtaining the data we need to train and test algorithms – it’s much easier than using the HoloLens ourselves to walk through a physical space and scan its features.”