{"id":8585,"date":"2025-09-30T21:04:25","date_gmt":"2025-10-01T02:04:25","guid":{"rendered":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/?p=8585"},"modified":"2025-09-30T21:18:31","modified_gmt":"2025-10-01T02:18:31","slug":"agentar","status":"publish","type":"post","link":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/","title":{"rendered":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents"},"content":{"rendered":"<p>Creating Augmented Reality (AR) applications requires expertise in both design and implementation, posing significant barriers to entry for non-expert users. While existing methods reduce some of this burden, they often fall short in flexibility or usability for complex or varied use cases. To address this, we introduce agentAR, an AR authoring system that leverages a tool-augmented large language model (LLM)\u2013based autonomous agent to support end-to-end, in-situ AR application creation from natural language input. Built on an application structure and tool library derived from state-of-the-art AR research, the agent autonomously creates AR applications from natural language dialogue. We demonstrate the effectiveness of agentAR\u00a0through a case study of six AR applications and a user study with twelve participants, showing that it significantly reduces user effort while supporting the creation of diverse and functional AR experiences.<\/p>\n<div style=\"width: 1080px;\" class=\"wp-video\"><!--[if lt IE 9]><script>document.createElement('video');<\/script><![endif]-->\n<video class=\"wp-video-shortcode\" id=\"video-8585-1\" width=\"1080\" height=\"608\" preload=\"metadata\" controls=\"controls\"><source type=\"video\/mp4\" src=\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentAR_Video_v5.mp4?_=1\" \/><a href=\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentAR_Video_v5.mp4\">https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentAR_Video_v5.mp4<\/a><\/video><\/div>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Creating Augmented Reality (AR) applications requires expertise in both design and implementation, posing significant barriers to entry for non-expert users. While existing methods reduce some of this burden, they often fall short in flexibility or usability for complex or varied use cases. To address this, we introduce agentAR, an AR authoring system that leverages a [&hellip;]<\/p>\n","protected":false},"author":100,"featured_media":8590,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","_links_to":"","_links_to_target":""},"categories":[475,476,203,469,359,268,478,466,465],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v19.0 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University\" \/>\n<meta property=\"og:description\" content=\"Creating Augmented Reality (AR) applications requires expertise in both design and implementation, posing significant barriers to entry for non-expert users. While existing methods reduce some of this burden, they often fall short in flexibility or usability for complex or varied use cases. To address this, we introduce agentAR, an AR authoring system that leverages a [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/\" \/>\n<meta property=\"og:site_name\" content=\"Convergence Design Lab, Purdue University\" \/>\n<meta property=\"article:published_time\" content=\"2025-10-01T02:04:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-01T02:18:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail-1024x490.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1024\" \/>\n\t<meta property=\"og:image:height\" content=\"490\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Chenfei Zhu\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization\",\"name\":\"Convergence Design Lab - Distinguished Professor Karthik Ramani - Purdue University\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/\",\"sameAs\":[],\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2020\/07\/Convergence-Design-Lab-Logo-2020-07-1.png\",\"contentUrl\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2020\/07\/Convergence-Design-Lab-Logo-2020-07-1.png\",\"width\":960,\"height\":100,\"caption\":\"Convergence Design Lab - Distinguished Professor Karthik Ramani - Purdue University\"},\"image\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/logo\/image\/\"}},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#website\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/\",\"name\":\"Convergence Design Lab, Purdue University\",\"description\":\"AR+VR+MR, knowledge and skill transfer, collaborative intelligence, robotics, smart product design, wearables,  geometric deep learning, AI, computer vision.\",\"publisher\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png\",\"contentUrl\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png\",\"width\":3152,\"height\":1508},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/\",\"name\":\"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University\",\"isPartOf\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage\"},\"datePublished\":\"2025-10-01T02:04:25+00:00\",\"dateModified\":\"2025-10-01T02:18:31+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents\"}]},{\"@type\":\"Article\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage\"},\"author\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/5c50cf082cbdf40404b728846a86a1b4\"},\"headline\":\"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents\",\"datePublished\":\"2025-10-01T02:04:25+00:00\",\"dateModified\":\"2025-10-01T02:18:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage\"},\"wordCount\":163,\"publisher\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization\"},\"image\":{\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png\",\"articleSection\":[\"2025\",\"Chenfei Zhu\",\"Featured Publications\",\"Jingyu Shi\",\"Karthik Ramani\",\"Recent Publications\",\"Shao-Kang Hsia\",\"Xiyun Hu\",\"Ziyi Liu\"],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/5c50cf082cbdf40404b728846a86a1b4\",\"name\":\"Chenfei Zhu\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2023\/10\/mmexport1642038338610-1-96x96.jpg\",\"contentUrl\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2023\/10\/mmexport1642038338610-1-96x96.jpg\",\"caption\":\"Chenfei Zhu\"},\"description\":\"I\u2019m Chenfei Zhu (\u6731\u8fb0\u98de), a Ph.D. student in Mechanical Engineering at Purdue University\u2019s\u00a0Convergence Design Lab, advised by\u00a0Prof. Karthik Ramani. My research focuses on agentic AI for AR\/MR authoring, with an emphasis on human behavior and human\u2013AI interaction. Before coming to Purdue, I earned my B.Eng. degree in Automotive Engineering at Wuhan University of Technology and my M.S. degree in Mechanical Engineering at Columbia University, where I worked with\u00a0Prof. Sunil K. Agrawal.\",\"url\":\"https:\/\/engineering.purdue.edu\/cdesign\/wp\/author\/zhu1237\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/","og_locale":"en_US","og_type":"article","og_title":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University","og_description":"Creating Augmented Reality (AR) applications requires expertise in both design and implementation, posing significant barriers to entry for non-expert users. While existing methods reduce some of this burden, they often fall short in flexibility or usability for complex or varied use cases. To address this, we introduce agentAR, an AR authoring system that leverages a [&hellip;]","og_url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/","og_site_name":"Convergence Design Lab, Purdue University","article_published_time":"2025-10-01T02:04:25+00:00","article_modified_time":"2025-10-01T02:18:31+00:00","og_image":[{"width":1024,"height":490,"url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail-1024x490.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"Written by":"Chenfei Zhu","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Organization","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization","name":"Convergence Design Lab - Distinguished Professor Karthik Ramani - Purdue University","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/","sameAs":[],"logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/logo\/image\/","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2020\/07\/Convergence-Design-Lab-Logo-2020-07-1.png","contentUrl":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2020\/07\/Convergence-Design-Lab-Logo-2020-07-1.png","width":960,"height":100,"caption":"Convergence Design Lab - Distinguished Professor Karthik Ramani - Purdue University"},"image":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/logo\/image\/"}},{"@type":"WebSite","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#website","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/","name":"Convergence Design Lab, Purdue University","description":"AR+VR+MR, knowledge and skill transfer, collaborative intelligence, robotics, smart product design, wearables,  geometric deep learning, AI, computer vision.","publisher":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png","contentUrl":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png","width":3152,"height":1508},{"@type":"WebPage","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/","name":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents - Convergence Design Lab, Purdue University","isPartOf":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#website"},"primaryImageOfPage":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage"},"datePublished":"2025-10-01T02:04:25+00:00","dateModified":"2025-10-01T02:18:31+00:00","breadcrumb":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/"},{"@type":"ListItem","position":2,"name":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents"}]},{"@type":"Article","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#article","isPartOf":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage"},"author":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/5c50cf082cbdf40404b728846a86a1b4"},"headline":"agentAR: Creating Augmented Reality Applications with Tool-Augmented LLM-based Autonomous Agents","datePublished":"2025-10-01T02:04:25+00:00","dateModified":"2025-10-01T02:18:31+00:00","mainEntityOfPage":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#webpage"},"wordCount":163,"publisher":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#organization"},"image":{"@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/agentar\/#primaryimage"},"thumbnailUrl":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2025\/09\/agentar_thumbnail.png","articleSection":["2025","Chenfei Zhu","Featured Publications","Jingyu Shi","Karthik Ramani","Recent Publications","Shao-Kang Hsia","Xiyun Hu","Ziyi Liu"],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/5c50cf082cbdf40404b728846a86a1b4","name":"Chenfei Zhu","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/#\/schema\/person\/image\/","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2023\/10\/mmexport1642038338610-1-96x96.jpg","contentUrl":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-content\/uploads\/2023\/10\/mmexport1642038338610-1-96x96.jpg","caption":"Chenfei Zhu"},"description":"I\u2019m Chenfei Zhu (\u6731\u8fb0\u98de), a Ph.D. student in Mechanical Engineering at Purdue University\u2019s\u00a0Convergence Design Lab, advised by\u00a0Prof. Karthik Ramani. My research focuses on agentic AI for AR\/MR authoring, with an emphasis on human behavior and human\u2013AI interaction. Before coming to Purdue, I earned my B.Eng. degree in Automotive Engineering at Wuhan University of Technology and my M.S. degree in Mechanical Engineering at Columbia University, where I worked with\u00a0Prof. Sunil K. Agrawal.","url":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/author\/zhu1237\/"}]}},"_links":{"self":[{"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/posts\/8585"}],"collection":[{"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/users\/100"}],"replies":[{"embeddable":true,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/comments?post=8585"}],"version-history":[{"count":1,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/posts\/8585\/revisions"}],"predecessor-version":[{"id":8588,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/posts\/8585\/revisions\/8588"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/media\/8590"}],"wp:attachment":[{"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/media?parent=8585"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/categories?post=8585"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/engineering.purdue.edu\/cdesign\/wp\/wp-json\/wp\/v2\/tags?post=8585"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}