New Microsoft research laboratory collaborates with ETH Zurich

ETH News: Professor Pollefeys, what is mixed reality?
Marc Pollefeys: It’s all about combining the physical world with virtual reality. For instance, getting smart glasses to add virtual objects to the wearer’s field of vision. What’s great about a mixed reality environment is that it lets users interact with the real world and with the virtual objects at the same time.

Can you give us some examples of applications?
Let’s take the example of a technician who has to service a machine. One possibility would be for the machine’s manufacturer to publish a manual and for the service technician to then painstakingly search through it to find the information they need. Mixed reality can guide them through the maintenance work directly on the machine and much more intuitively by showing them each work step “live”. What’s more, today’s mixed reality glasses use cameras to track the exact position and movements of the wearer’s hands. This lets people use their hands to manipulate virtual objects, for instance as a way to remotely control a robot. Another application is navigation within buildings: the service technician’s mixed reality glasses can guide them to a defective fire alarm, and at the same time give them certainty that they are repairing the right one.

You’re leading the new Microsoft lab in Zurich, which is devoted to mixed reality and artificial intelligence. How do the two interact?
Taking someone through a particular task step by step is a good example of that. The computers in today’s mixed reality glasses are not yet very good at knowing which step a user has reached. Artificial intelligence methods can help to determine that. This turns the device into an assistant that helps users do their job.

Why are Microsoft and ETH Zurich working together?
Because this collaboration is attractive for both parties. Microsoft and ETH have been working together on various individual projects for the past eleven years, and the new lab intensifies this partnership. ETH doctoral students can use it to work on real problems from industry, and it will give them prized access to state-of-the-art hardware. For its part, Microsoft benefits from ETH’s long-term and wide-ranging research, as conducted by our extremely well-qualified doctoral students. For years now, ETH has been successfully collaborating with Disney along similar lines as it will now with Microsoft. Indeed, that’s what inspired us to take this approach.

ETH is a public university, Microsoft is a company driven by profit. Who owns the developments that ETH doctoral students come up with in the Microsoft lab?
As is the case in all ETH’s projects with industry, the details are precisely laid out in a collaboration contract. To the extent that nothing different is agreed as part of a project, ETH retains the rights to the intellectual property relating to such developments, while Microsoft has the right to use the development free of charge. But ETH may also grant licences to third parties. If Microsoft wants exclusive use of the technology, the company can negotiate a paid licence with ETH.

What specific research projects are Microsoft and ETH working on together?
Several projects deal with using cameras to create 3D maps of the environment. One topic this involves is crowdmapping, where information from individual mobile cameras is combined to create a large-scale 3D model. How to guarantee the privacy of passersby is a research topic in its own right. And, finally, there is the hand and object recognition using camera images I mentioned before. While many traditional research and development labs in industry focus on short-term applicability, we have the opportunity to take on longer-term research projects that might lead to interesting applications and products in the future.