21st-century human-computer interaction

Apart from sounding really techy and innovative, what makes manipulating virtual buttons by gestures more attractive compared to pushing actual buttons?

Flexibility! A virtual button does not require any hardware changes when changing its functionality. In one application it can be a switch, in another a knob and in yet another a scroll bar. Also, the ergonomics can be adjusted to fulfill Fitts’ law. This is an empirical law, which describes how fast we can actually move to a specific target and select it with precision. Take smartphones and smartwatches as an example: Nowadays, we have an information overload on increasingly smaller screens, where it becomes challenging to select the miniature icons or keys with a normal-sized index finger. On average, people spend more than one day per year just unlocking their phones. We can reduce this time significantly by designing better accessible virtual knobs and keys. Moreover, we can make switching between apps much faster.

What is your contribution to project SOLI?

Google worked on the motion sensor and the electronics for the signal processing. Our job was the development of the algorithm for recognizing the gestures that go along with manipulating various virtual buttons.

We worked very closely together with Google, because sensor, electronics and algorithm were developed simultaneously and interfaced tightly. Our work on the algorithm influenced the design of the sensor and the electronics and vice versa. My students, Jie Song and Saiwen Wang, and I travelled multiple times to Sunnyvale, California, where we had direct interaction with the Google ATAP (Advanced Technology And Projects) division. After the project, Saiwen actually stayed in California and started a job with Google. That was a nice side effect of our industry collaboration.

How does gesture recognition work?

The signaling is based on a 60 GHz Doppler radar. Doppler radar is sensitive to motion, but not so much to the position of a given target. This means, we look at gestures under the aspect of what is moving, how fast, in which direction and if we see a combination of movements, e.g. snipping fingers where the thumb goes up and the index finger moves down.

Now here comes the crux, which makes our job tough: The radar signal is messy, because our limbs are not rigid objects. We tend to deform while moving. The signal is low, because skin does not reflect radar as well as for example metal. Last but not least, catching the signal pushes the limits of the fastest electronics that we have. We work with 60 GHz radar. If you want to catch a millimeter movement in space, it translates to a tiny signal shift in time of a few picoseconds (10^-12 seconds). So, the data is produced at an extremely high frame rate, which makes it technically challenging and academically very interesting.

What was the outcome of the project?

We succeeded in distinguishing 11 different gestures with an average recognition rate of 87%. This is a very good result. We used a machine-learning algorithm, which we fed, with our own set of training data. Also, we made sure that the algorithm runs on commodity hardware with low energy consumption, which is ideal for smartphones or other wearable electronics. For those who are interested in learning more about the project, watch our external pagevideocall_made or visit our project webpage.