agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents

by | Sep 30, 2025

Authors: Chenfei Zhu, Shao-Kang Hsia, Xiyun Hu, Ziyi Liu, Jingyu Shi, Karthik Ramani
In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25).
https://dl.acm.org/doi/full/10.1145/3746059.3747676

Creating Augmented Reality (AR) applications requires expertise in both design and implementation, posing significant barriers to entry for non-expert users. While existing methods reduce some of this burden, they often fall short in flexibility or usability for complex or varied use cases. To address this, we introduce agentAR, an AR authoring system that leverages a tool-augmented large language model (LLM)–based autonomous agent to support end-to-end, in-situ AR application creation from natural language input. Built on an application structure and tool library derived from state-of-the-art AR research, the agent autonomously creates AR applications from natural language dialogue. We demonstrate the effectiveness of agentAR through a case study of six AR applications and a user study with twelve participants, showing that it significantly reduces user effort while supporting the creation of diverse and functional AR experiences.

 

Chenfei Zhu

Chenfei Zhu

I’m Chenfei Zhu (朱辰飞), a Ph.D. student in Mechanical Engineering at Purdue University’s Convergence Design Lab, advised by Prof. Karthik Ramani. My research focuses on agentic AI for AR/MR authoring, with an emphasis on human behavior and human–AI interaction. Before coming to Purdue, I earned my B.Eng. degree in Automotive Engineering at Wuhan University of Technology and my M.S. degree in Mechanical Engineering at Columbia University, where I worked with Prof. Sunil K. Agrawal.