Xiyun Hu

Xiyun Hu

PhD student in Mechanical Engineering, Robotics area. Focusing on human-computer iteration in AR/VR/MR/XR and mechatronics.
GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

Xiyun Hu, Dizhi Ma, Fengming He, Zhengzhe Zhu, Shao-Kang Hsia, Chenfei Zhu, Ziyi Liu, Karthik Ramani
In Proceedings of the 2025 ACM Designing Interactive Systems Conference

Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal...

avaTTAR: Table Tennis Stroke Training with On-body and Detached Visualization in Augmented Reality

avaTTAR: Table Tennis Stroke Training with On-body and Detached Visualization in Augmented Reality

Dizhi Ma*, Xiyun Hu*, Jingyu Shi, Mayank Patel, Rahul Jain, Ziyi Liu, Zhengzhe Zhu, Karthik Ramani
In The 37th Annual ACM Symposium on User Interface Software and Technology (UIST ’24)

Table tennis stroke training is a critical aspect of player development. We designed a new augmented reality (AR) system, avaTTAR, for table tennis stroke training. The system provides both "on-body" (first-person view) and "detached" (third-person...

ClassMeta: Designing Interactive Virtual Classmate to Promote VR Classroom Participation

ClassMeta: Designing Interactive Virtual Classmate to Promote VR Classroom Participation

Ziyi Liu*, Zhengzhe Zhu*, Lijun Zhu, Enze Jiang, Xiyun Hu, Kylie A Peppler, Karthik Ramani
In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-17. 2024.

Peer influence plays a crucial role in promoting classroom participation, where behaviors from active students can contribute to a collective classroom learning experience. However, the presence of these active students depends on several...

Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality

Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality

Fengming He, Xiyun Hu, Jingyu Shi, Xun Qian, Tianyi Wang, Karthik Ramani
In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Edges are one of the most ubiquitous geometric features of physical objects. They provide accurate haptic feedback and easy-totrack features for camera systems, making them an ideal basis for Tangible User Interfaces (TUI) in Augmented Reality...

ARnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation

ARnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation

Xun Qian, Fengming He, Xiyun Hu, Tianyi Wang, Karthik Ramani
In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (pp. 1-14)

Vision-based 3D pose estimation has substantial potential in hand-object interaction applications and requires user-specified datasets to achieve robust performance. We propose ARnnotate, an Augmented Reality (AR) interface enabling end-users to...

ScalAR: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality

ScalAR: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality

Xun Qian, Fengming He, Xiyun Hu, Tianyi Wang, Ananya Ipsita, and Karthik Ramani
In the Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Augmented Reality (AR) experiences tightly associate virtual contents with environmental entities. However, the dissimilarity of different environments limits the adaptive AR content behaviors under large-scale deployment. We propose ScalAR, an...

GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications

GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications

Tianyi Wang, Xun Qian, Fengming He, Xiyun Hu, Yuanzhi Cao, Karthik Ramani
In The 34th Annual ACM Symposium on User Interface Software and Technology (UIST '21)

Freehand gesture is an essential input modality for modern Augmented Reality (AR) user experiences. However, developing AR applications with customized hand interactions remains a challenge for end-users. Therefore, we propose GesturAR, an...

CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications

CAPturAR: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications

Tianyi Wang*, Xun Qian*, Fengming He, Xiyun Hu, Ke Huo, Yuanzhi Cao, Karthik Ramani
In Proceedings of the 2020 UIST 33rd ACM User Interface Software and Technology Symposium

Recognition of human behavior plays an important role in context-aware applications. However, it is still a challenge for end-users to build personalized applications that accurately recognize their own activities. Therefore, we present CAPturAR,...