From Language to Contact: Control-Grounded Multimodal World Models for Robot Manipulation
Project Description
Robots that understand language still struggle to execute long-horizon, contact-rich tasks because semantic reasoning, physical prediction, and feedback control are typically developed separately. This project will create a control-grounded physical-AI framework that unifies vision-language reasoning, imitation and reinforcement learning, multimodal world models, and real-time optimal control. A vision-language model will translate open-vocabulary instructions into structured subgoals, skill constraints, and verifiable effects. Demonstration data will initialize vision-language-action policies, while visual, tactile, and proprioceptive observations will train a predictive world model of contact dynamics. We will investigate Koopman-structured latent representations that preserve task-relevant geometry and interaction dynamics while enabling efficient linear rollout. These models will support model-based reinforcement learning and high-frequency MPC/MPPI for online policy refinement, constraint handling, and recovery from disturbances. The research will study how semantic representations, multimodal predictive states, and control objectives should be co-designed, and will characterize how latent-model error affects robustness and closed-loop performance. Experiments on insertion, screwing, disassembly, and multi-stage manipulation will evaluate generalization, sample efficiency, constraint satisfaction, and real-time performance. The outcome will be a principled bridge from language-level reasoning to physically reliable robot action.
Start Date
Spring or Summer 2027, with flexibility for a mutually agreed start date within calendar year 2027.
Postdoc Qualifications
Candidates should hold a Ph.D. in Robotics, Computer Science, Electrical Engineering, Mechanical Engineering, Civil Engineering, or a closely related field by the start of the appointment. The ideal candidate will have demonstrated expertise in robotics, autonomous systems, motion planning, control, optimization, machine learning, or foundation models, with a strong publication record in leading conferences and journals. Experience with generative AI, world models, reinforcement learning, human-autonomy teaming, ROS/ROS2, or real-world robotic systems is highly desirable. Applicants should be excited to conduct interdisciplinary research at the intersection of Physical AI, robotics, and human-centered autonomous systems, and to collaborate across multiple research groups. This description aligns well with Sixu's background in combining model-based control, optimization, and learning for trustworthy autonomous systems.
Co-advisors
Yu She
Assistant Professor
School of Industrial Engineering
Purdue University
Email: shey@purdue.edu
Website: https://www.purduemars.com/
Shaoshuai Mou
Elmer Bruhn Associate Professor
School of Aeronautics and Astronautics
Purdue University
Email: mous@purdue.edu
Website: https://engineering.purdue.edu/AIMS
Bibliography
Guo, Pengyuan, Zhonghao Mai, Zhengtong Xu, Kaidi Zhang, Quan Khanh Luu, Heng Zhang, Zichen Miao et al. "PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulation." In The 2nd Workshop on Foundation Models Meet Embodied Agents at CVPR 2026.
Zhang, Kaidi, Heng Zhang, Zhengtong Xu, Zhiyuan Zhang, Md Rakibul Islam Prince, Xiang Li, Xiaojing Han, Yuhao Zhou, Arash Ajoudani, and Yu She. "Tacvla: Contact-aware tactile fusion for robust vision-language-action manipulation." arXiv preprint arXiv:2603.12665 (2026).
Zhang, Zhiyuan, Pokuang Zhou, Kaidi Zhang, Adeesh Desai, Temitope Amosa, Davood Soleymanzadeh, Jiuzhou Lei, Minghui Zheng, and Yu She. "ContactWorld: What Matters in Vision-Tactile World Models for Contact-Rich Manipulation." arXiv preprint arXiv:2606.13877 (2026).
Hao, Wenjian, Yuxuan Fang, Zehui Lu, and Shaoshuai Mou. "Accelerating Sampling-Based Control via Learned Linear Koopman Dynamics." arXiv preprint arXiv:2603.05385 (2026).
Hao, Wenjian, Yuxuan Fang, Zehui Lu, and Shaoshuai Mou. "Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems." arXiv preprint arXiv:2604.19980 (2026).