Investigating Problem-Solving in Physics Using NLP-Based Analysis of Chatbot Interactions

Assessing and improving problem-solving skills in physics remains a challenge. This study explores how an AI-driven chatbot supports students' problem-solving development through Natural Language Processing (NLP) analysis of interaction transcripts. Using text classification, topic modeling, and linguistic complexity measures, we examine the types of questions students ask, their specificity, and their progression from novice to expert-like behavior. We categorize queries into conceptual, procedural, and verification-based types while evaluating broad vs. specific questioning patterns. Additionally, a Likert-scale survey assesses students' confidence, engagement, and perceived chatbot effectiveness in structuring problem-solving, fostering metacognition, and improving efficiency. Preliminary findings suggest that the chatbot may help students refine their approach and engage in reflective inquiry. This study highlights the potential of NLP-driven educational tools to enhance problem-solving in STEM education.