Towards Design Guidelines for Natural-Language-Based Human-Multi-Robot Collaboration in Domestic Environments

Towards Design Guidelines for Natural-Language-Based Human-Multi-Robot Collaboration in Domestic Environments

Xinyi Wang, Shao-Kang Hsia, Ziyi Liu, Chenfei Zhu, Zhengzhe Zhu, Xiyun Hu, Anastasia Kouvaras Ostrowski, Karthik Ramani
In Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems

As domestic robots become prevalent, households will involve multiple robots with different capabilities, requiring people to coordinate and collaborate with several robots daily. This demands understanding key human–multi-robot interaction that...

AmIWrite: Exploring Scalable One-on-One Handwriting-Based Tutoring for Mathematical Problem-Solving with an LLM-Powered AI Tutor

AmIWrite: Exploring Scalable One-on-One Handwriting-Based Tutoring for Mathematical Problem-Solving with an LLM-Powered AI Tutor

Ziyi Liu, Yuzhao Chen, Haoyu Ji, Runlin Duan, Zhengzhe Zhu, Xiyun Hu, Kylie Peppler, Karthik Ramani
In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems

Real-time handwriting interactions between tutors and students —where tutors observe individual problem-solving processes, provide personalized annotations, and adapt explanations based on students’ work—are fundamental to effective STEM tutoring....

GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

Xiyun Hu, Dizhi Ma, Fengming He, Zhengzhe Zhu, Shao-Kang Hsia, Chenfei Zhu, Ziyi Liu, Karthik Ramani
In Proceedings of the 2025 ACM Designing Interactive Systems Conference

Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal...

avaTTAR: Table Tennis Stroke Training with On-body and Detached Visualization in Augmented Reality

avaTTAR: Table Tennis Stroke Training with On-body and Detached Visualization in Augmented Reality

Dizhi Ma*, Xiyun Hu*, Jingyu Shi, Mayank Patel, Rahul Jain, Ziyi Liu, Zhengzhe Zhu, Karthik Ramani
In The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24)

Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both "on-body" (first-person view) and "detached" (third-person...

ClassMeta: Designing Interactive Virtual Classmate to Promote VR Classroom Participation

ClassMeta: Designing Interactive Virtual Classmate to Promote VR Classroom Participation

Ziyi Liu*, Zhengzhe Zhu*, Lijun Zhu, Enze Jiang, Xiyun Hu, Kylie A Peppler, Karthik Ramani
In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-17. 2024.

Peer influence plays a crucial role in promoting classroom participation, where behaviors from active students can contribute to a collective classroom learning experience. However, the presence of these active students depends on several...

Ubi-TOUCH: Ubiquitous Tangible Object Utilization through Consistent Hand-object interaction in Augmented Reality

Ubi-TOUCH: Ubiquitous Tangible Object Utilization through Consistent Hand-object interaction in Augmented Reality

Rahul Jain*, Jingyu Shi*, Runlin Duan, Zhengzhe Zhu, Xun Qian, Karthik Ramani
In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST 2023)

Utilizing everyday objects as tangible proxies for Augmented Reality (AR) provides users with haptic feedback while interacting with virtual objects. Yet, existing methods focus on the attributes of the objects, constraining the possible proxies...

InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied

InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied

Ziyi Liu, Zhengzhe Zhu, Enze Jiang, Feichi Huang, Ana Villanueva, Tianyi Wang, Xun Qian, Karthik Ramani
In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Augmented Reality tutorials, which provide necessary context by directly superimposing visual guidance on the physical referent, represent an effective way of scaffolding complex instrument operations. However, current AR tutorial authoring...

LearnIoTVR: An End-to-End Virtual Reality Environment Providing Authentic Learning Experiences for Internet of Things

LearnIoTVR: An End-to-End Virtual Reality Environment Providing Authentic Learning Experiences for Internet of Things

Zhengzhe Zhu, Ziyi Liu, Youyou Zhang, Lijun Zhu, Joey Huang, Ana villanueva, Xun Qian, Kylie Peppler, Karthik Ramani
In the Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

The rapid growth of Internet-of-Things (IoT) applications has generated interest from many industries and a need for graduates with relevant knowledge. An IoT system is comprised of spatially distributed interactions between humans and various...

MechARspace: An Authoring System Enabling Bidirectional Binding of Augmented Reality with Toys in Real-time

MechARspace: An Authoring System Enabling Bidirectional Binding of Augmented Reality with Toys in Real-time

Zhengzhe Zhu, Ziyi Liu, Tianyi Wang, Youyou Zhang, Xun Qian, Pashin Farsak Raja, Ana Villanueva, Karthik Ramani
In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (pp. 1-16).

Augmented Reality (AR), which blends physical and virtual worlds, presents the possibility of enhancing traditional toy design. By leveraging bidirectional virtual-physical interactions between humans and the designed artifact, such AR-enhanced...