For the high-flex option, register in advance. You will receive a confirmation email containing information about joining the meeting.

Title:

From Natural Language to New Understandings: Developing AI-Powered Methodologies for Engineering Education Research and Practice

Abstract:
The integration of artificial intelligence and natural language processing (NLP) into engineering education research and practice represents a set of opportunities to expand how we understand student learning, assess educational outcomes, and scale qualitative research methods. This talk presents a comprehensive overview of methodological innovations developed in the IDEEAS Lab over the past five years to leverage the power of generative AI, large language models, and NLP techniques for advancing engineering education. I will present three interconnected methodological contributions: (1) AI-assisted qualitative analysis, where I demonstrate how generative text models can perform traditional thematic data analysis while maintaining rigor and validity; (2) AI-assisted assessment and feedback systems, showcasing applications of open-source language models for analyzing student responses, teamwork feedback, and ethics case studies; and (3) mental models research at scale, illustrating how NLP techniques can help uncover engineering faculty and student mental models of complex topics like sustainability, assessment, and now generative AI itself.

Through concrete examples from NSF-funded research projects, I will share methodological lessons learned from analyzing thousands of student responses, faculty interviews, and educational artifacts using AI tools. Key methodological insights include: strategies for prompt engineering in educational contexts, approaches for validating AI-generated qualitative codes against human coding, and frameworks for maintaining ethical standards when using AI in educational research. The talk will conclude with a forward-looking discussion of emerging opportunities and challenges, including strategies for development and dissemination of open-source AI tools for education researchers, the implications of AI for traditional qualitative research paradigms, and how these methodological advances can inform evidence-based improvements to engineering curricula and pedagogy. This work represents a new frontier in engineering education research methodology. The ultimate vision is to leverage AI not to replace human insight, but to amplify our capacity to understand and improve engineering education at unprecedented scale and depth.

Bio:
Andrew Katz is Associate Professor of Engineering Education and Assistant Department Head for Graduate Programs at Virginia Tech. He is a researcher in the application of artificial intelligence and natural language processing to engineering education, with expertise spanning applications of generative AI, mental models research, and qualitative research methodologies. Dr. Katz has secured over $1.5 million in NSF funding as Principal Investigator, including an NSF CAREER Award (2024-2029) focused on faculty mental models of generative AI in education. His methodological innovations in AI-assisted qualitative analysis have been published in venues including the Journal of Engineering Education and International Journal of Qualitative Methods. He has developed novel approaches for using open-source large language models to analyze student feedback, assess ethics reasoning, and uncover mental models at unprecedented scale. His research has analyzed thousands of student responses and faculty interviews using locally-hosted AI tools, contributing to evidence-based improvements in engineering curricula and pedagogy. He earned his PhD in Engineering Education from Purdue University (2019), where he developed foundational expertise in engineering ethics education and qualitative research methods that now inform his AI-enabled methodological innovations.

Citation:
Anakok, I., Katz, A., Chew, K. J., & Matusovich, H. (2025). Leveraging Generative Text Models and Natural Language Processing to Perform Traditional Thematic Data Analysis. International Journal of Qualitative Methods, 24, 16094069251338898.