Opportunities and challenges of large language models in physics education

Recent advances in large language models (LLMs) such as ChatGPT demonstrate compelling performances in complex tasks such as writing coherent essays, reasoning, and even performing well in untrained tasks. While these models also hold a great promise for physics education, there are concerns about biases and output reliability as well as impacts on learning willingness, exam fraud, and misuse. Thus, it is crucial for their integration into physics learning and teaching to provide scientific evidence regarding their effectiveness and safety. In this contribution, we demonstrate how LLMs may support physics teachers and learners and conclude with a perspective of how AI may transform physics education.

Specifically, we report on a comparison of pre-service teachers creating physics tasks using either ChatGPT or a textbook. The correctness of tasks was similar across both groups and both groups adjusted the difficulty level well for their target audience. Moreover, both groups struggled with task specificity and often omitted key solution information. However, those using textbooks created clearer tasks and embedded them more effectively in contexts. While ChatGPT users praised its user-friendliness, they faced issues with output quality.

Additionally, we show how LLMs are sensitive to students’ misconceptions and can aid in developing and validating Concept Inventories. Concept inventories are crucial for checking learners' understanding of physics concepts, but the creation and validation of such inventories are often resource intensive. In this context, we show how ChatGPT enables the creation synthetically generated empirical data, significantly accelerating the creation and validation of concept tests.

Overall, LLMs offer novel perspectives on the intersection of AI and physics education. However, their integration into teaching and learning should be carefully managed, considering their limitations and ensuring they complement traditional teaching methods.