WorldPoint: Finger Pointing as a Rapid and Natural Trigger for In-the-Wild Mobile Interactions

Pointing with one's finger is a natural and rapid way to denote an area or object of interest. It is routinely used in human-human interaction to increase both the speed and accuracy of communication, but it is rarely utilized in human-computer interactions. In this work, we use the recent inclusion of wide-angle, rear-facing smartphone cameras, along with hardware-accelerated machine learning, to enable real-time, infrastructure-free, finger-pointing interactions on today's mobile phones. We envision users raising their hands to point in front of their phones as a "wake gesture". This can then be coupled with a voice command to trigger advanced functionality. For example, while composing an email, a user can point at a document on a table and say "attach". Our interaction technique requires no navigation away from the current app and is both faster and more privacy-preserving than the current method of taking a photo.



Kim, D. and Harrison, C. 2023. WorldPoint: Finger Pointing as a Rapid and Natural Trigger for In-The-Wild Mobile Interactions. In Proceedings of the ACM on Human-Computer Interaction (PACMHCI), Volume 7, Issue ISS, (December 2023). Presented at the 18th ACM International Conference on Interactive Surfaces and Spaces (November 5 – 8, 2023). ACM, New York, NY. pp 357–375.

© Chris Harrison