Rewind reality and experience it again

“I’ll have to leave now” is a phrase often heard in video conferencing. But imagine a future in which you could not only disconnect from online meetings but also leave the hubbub of the office with a click whenever you wanted to focus on an important task. The events happening around you would become invisible to you. Instead, they would be recorded and you could play them back later on demand. With recordings in 3D and so lifelike as to be almost indistinguishable from reality.

Researchers at ETH Zurich have achieved at least part of this vision. Christian Holz, Assistant Professor of Intelligent Interactive Systems, and Postdoc Andreas Rene Fender equipped an office with depth cameras capable of capturing objects and people in three dimensions. A computer system continuously records everything that happens in the space and detects how events are causally connected. The researchers recently presented their concept and the corresponding system at a conference.

In this “future office”, the user wears a virtual reality (VR) headset and headphones. Per default, the headset renders the physical office space around the user as well as virtual display screens and objects, embedding these in the physical environment. When the user wants to withdraw from what’s going on around them, they can activate “Focus Mode”, which hides everything that isn’t happening in their immediate workspace. Later on, they can play all the “hidden” events back and experience them as if they would happen “here and now” from the user’s point of view.

Much more than an answering machine

It’s tempting to compare this system to an answering machine, but Holz qualifies that idea: “When someone leaves a message on an answerphone, they actively formulate a brief message, putting thought into what they want it to say. But in asynchronous reality, the ‘caller’ doesn’t have to actively communicate at all – they can just do and say things as if the other person were present. Later, when they are ready, the ‘recipient’ can play the events back in the same space and interact directly with objects to experience for themself what has been recorded. That makes the recording feel very real.” So the two people involved experience the same events in the same space – only at different times.