Ubi-TOUCH: Ubiquitous Tangible Object Utilization through Consistent Hand-object interaction in Augmented Reality

by | Oct 16, 2023

Authors: Rahul Jain*, Jingyu Shi*, Runlin Duan, Zhengzhe Zhu, Xun Qian, Karthik Ramani
In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST 2023)

Utilizing everyday objects as tangible proxies for Augmented Reality (AR) provides users with haptic feedback while interacting with virtual objects. Yet, existing methods focus on the attributes of the objects, constraining the possible proxies and yielding inconsistency in user experience. Therefore, we propose Ubi-TOUCH, an AR system that assists users in seeking a wider range of tangible proxies for AR applications based on the hand-object interaction (HOI) they desire. Given the target interaction with a virtual object, the system scans the users’ vicinity and recommends object proxies with similar interactions. Upon user selection, the system simultaneously tracks and maps users’ physical HOI to the virtual HOI, adaptively optimizing object 6 DoF and the hand gesture to provide consistency between the interactions. We showcase promising use cases of Ubi-TOUCH, such as remote tutorials, AR gaming, and Smart Home control. Finally, we evaluate the performance and usability of Ubi-TOUCH with a user study.

Rahul Jain

Rahul Jain

Rahul Jain has been a Ph.D. student in the School of Electrical and Computer Engineering at Purdue University since Spring 2022. He is conducting research under Professor Karthik Ramani’s Convergence Design Lab. He received his Master’s in Electrical and Computer Engineering at Purdue University and Bachelor’s in Civil Engineering at Indian Institute of Technology (IIT), Patna. His current research focuses on area of Computer Vision, Machine Learning and human-computer interactions utilizing AR/VR.