Discovering Misconceptions by IRT and Machine Learning from FCI Data

Student misconceptions result in associated patterns of distractors on research-designed multiple choice instruments. We wrote a Hierarchical Bayesian realization of the Multidimensional Nominal Categories Model. Applying non-orthogonal rotations (from the r-package) in the multi-dimensional space generated one "Newtonian Correct" dimension and ~20 "sparse" dimensions each of which loaded heavily on only several distractors. These proved to be highly robust with respect to different rotation methods and to selecting new data sets using bootstrap methods. On most dimensions, the sparse vectors of our bootstrap samples correlated with the best sparse vector at above 0.9. Intellectual similarities among distractors associated with each vector enabled us to identify its misconception or misunderstanding. We found many known misconceptions (i.e., last force governs direction, gravity stronger near ground, etc.), that separate misconceptions for impetus force on linear and circular paths, and new misconceptions like moving masses accelerated sideways travel on straight paths.