“If you’re sitting in the back row of a classroom and not engaging, it’s no better than just watching a recording,” said Karthik Ramani, Donald W. Feddersen Distinguished Professor of Mechanical Engineering, Professor of Electrical and Computer Engineering, and Professor of Educational Studies by courtesy. “We’ve found a way to use AI in virtual classrooms to break that ice, helping students to participate and ask more questions, while retaining the best characteristics of a physical classroom.”

Ramani’s Convergence Design Lab focuses on the convergence of AI, augmented reality, and virtual reality in real-world industrial and educational settings. His team has created many projects, including authoring tools that help instructors build augmented reality environments in their classrooms, and worker training tools that use virtual reality to pre-skill and up-skill workers in manufacturing environments. Ramani has testified to their application at a recent congressional briefing in Washington, DC.

Their newest project came from the frustration of one of Ramani’s teaching assistants. “I was helping to teach an undergraduate class,” said Ziyi Liu, a Ph.D student in mechanical engineering, and lead author of the paper. “Instructors always want students to engage with the lecture, but sometimes we receive silence or blank stares, which doesn’t make for a great learning atmosphere. I wondered if we could use AI technology to boost the students’ performance in class.”

The result is ClassMeta, a virtual reality environment with a typical classroom, instructor, students, and desks. But among the students are AI avatars, whose behavior is designed to blend in with the human students. Should the conversation lag, one of the ClassMeta “students” will sense the awkward silence, and ask a prompting question in the hopes that human students will continue the conversation.

ClassMeta utilizes ChatGPT to “listen” to the lecture, and then suggest questions based on what has been said (or unsaid).

“This wouldn’t have been possible just a year or two ago,” Liu said. “Our previous AI work required us to train our own custom large language models (LLMs) for each specific context, which takes a long time and lots of data. But ChatGPT’s LLM is already broad and robust enough to respond to almost any query immediately, whatever the subject matter may be. Rather than trying to craft custom responses, we just let ChatGPT figure out the context of the lecture, and then interpret what question should come next.”

The team presented ClassMeta at the 2024 CHI Conference on Human Factors in Computing Systems, which took place in Honolulu, Hawaii in May 2024. The paper received an Honorable Mention, as one of the top 5% of papers of more than 1,000 accepted to the conference.

ClassMeta embeds A.I. students in a virtual classroom environment to help break the ice for human students. They "listen" to the lecture, which is fed to ChatGPT. When it senses an awkward pause, ChatGPT generates a pertinent question, which it then sends to an ElevenLabs text-to-speech tool to ask aloud, as if it were a real student. Their study shows that "breaking the ice" results in better overall educational outcomes.

Incorporating an AI avatar into a human classroom isn’t as easy as just plugging in ChatGPT and hoping for the best. Liu programmed specific guardrails and behaviors into the agent. He designed the AI avatar to appear to take notes, which helps to prompt human students to also take more notes. The AI avatar raises its hand and waits to be acknowledged, like a human student. It intentionally answers only part of the question, leaving human students to fill in the rest. And if the LLM determines that the instructor hasn’t fully explained a concept, the AI student will jump in with a question, prompting the instructor to offer more detail.

Because students are likely to sniff out AI chatbots, a text-to-speech tool from ElevenLabs ensures that the human voice response sounds sufficiently human — hesitating, adding “ums” and “uhs,” and using the same vocabulary level as the other students in the class. “Natural language processing has come very far in the last few years,” Ramani said. “In many ways, it’s a solution looking for a problem. We’ve shown that the classroom is a perfect environment for this kind of AI chatbot, reframed to be a fellow student.”

While the psychology of “breaking the ice” can be observed anecdotally, Liu collaborated with Kylie Peppler, professor of education at UC Irvine, to also confirm it experimentally. They designed an experiment where an instructor taught a class in a virtual environment with just human students, and then in a similar environment where two of the “students” were ClassMeta agents, indistinguishable to the other participants.

The results? Human students in the ClassMeta environment took better quality notes, generated more insights during discussion, and acquired higher learning gains than the students in the baseline group. “When we saw the human participants engage more with the instructor, we were thrilled,” Liu said. “We can see this being used in the future, where remote students are on equal footing with in-person students, and everyone has an opportunity to learn in the best possible environment.”

“During the pandemic, we saw the best and the worst of what virtual classrooms had to offer,” Ramani said. “But virtual classrooms have definitive advantages, as we’ve demonstrated with ClassMeta. This has the potential to level out inequalities that naturally happen in a physical classroom environment. VR helps large classrooms feel small, and small classrooms feel large.”

This work is partially supported by the NSF under the Future of Work at the Human Technology Frontier (FW-HTF) grant 1839971. Patent is pending with the Purdue Innovates Office of Technology Commercialization; for licensing information, please contact the OTC and refer to track code 2024-RAMA-70603.

Writer: Jared Pike, jaredpike@purdue.edu, 765-496-0374

Source: Karthik Ramani, ramani@purdue.edu

ClassMeta: Designing Interactive Virtual Classmate to Promote VR Classroom Participation
Ziyi Liu, Zhengzhe Zhu, Lijun Zhu, Enze Jiang, Xiyun Hu, Kylie A Peppler, Karthik Ramani
https://doi.org/10.1145/3613904.3642947
ABSTRACT: Peer influence plays a crucial role in promoting classroom participation, where behaviors from active students can contribute to a collective classroom learning experience. However, the presence of these active students depends on several conditions and is not consistently available across all circumstances. Recently, Large Language Models (LLMs) such as GPT have demonstrated the ability to simulate diverse human behaviors convincingly due to their capacity to generate contextually coherent responses based on their role settings. Inspired by this advancement in technology, we designed ClassMeta, a GPT-4 powered agent to help promote classroom participation by playing the role of an active student. These agents, which are embodied as 3D avatars in virtual reality, interact with actual instructors and students with both spoken language and body gestures. We conducted a comparative study to investigate the potential of ClassMeta for improving the overall learning experience of the class.