Saikat Gupta, Sujin Jang, and Karthik Ramani
PuppetX: A Framework for Gestural Interactions With User Constructed Playthings
Proceedings of the ACM Conference on Advanced Visual Interfaces, May 27-30, Como, Italy. pp. 73-80, 2014

We present PuppetX, a framework for both constructing playthings and playing with them using spatial body and hand gestures. This framework allows users to construct various playthings similar to puppets with modular components representing basic geometric shapes. It is topologically-aware, i.e. depending on its configuration; PuppetX automatically determines its own topological construct. Once the plaything is made the users can interact with them naturally via body and hand gestures as detected by depth-sensing cameras. This gives users the freedom to create playthings using our components and the ability to control them using full body interactions. Our framework creates affordances for a new variety of gestural interactions with physically constructed objects. As its by-product, a virtual 3D model is created, which can be animated as a proxy to the physical construct. Our algorithms can recognize hand and body gestures in various configurations of the playthings. Through our work, we push the boundaries of interaction with user-constructed objects using large gestures involving the whole body or fine gestures involving the fingers. We discuss the results of a study to understand how users interact with the playthings and conclude with a demonstration of the abilities of gestural interactions with PuppetX by exploring a variety of interaction scenarios.


PuppetX Framework A) Components developed B) Puppets created using these components.


A) Scenario 1 – Play with body gestures. B) Scenario 2 – Play with nger gestures. Red arrow shows the depth sensors.

About Sujin Jang

Sujin Jang received his Ph.D. in August 2017 from the School of Mechanical Engineering at Purdue University. His research work in the C-Design Lab broadly involved human-computer interaction, visual analytics, and machine learning. In particular, the goal of his research aimed (i) to create natural human-computer interaction systems using the human body as an interaction input and (ii) to establish the fundamental principles when designing such novel interaction systems. His past research experience also includes computer vision, robotics, and non-linear control/estimation. He has received the Estus H. and Vashti L. Magoon Award for Teaching Excellence in 2015.  Currently, he is a staff research member at the Motorola-Lenovo Research group. [Personal Website][LinkedIn]

Posted in 2014, Embedded Input, HUMAN SHAPE INTERACTION, Karthik Ramani, Mid-air Interaction, Publications, Sujin Jang, TANGIBLE, EMBEDDED & EMBODIED INTERFACES, Tangibles