Assessing high-level scientific reasoning in a physics exam: Pipe-dream or reality?

What do we want students to be able to do when they have finished their introductory physics course? In addition to learning the physics content, we want students to learn to think like physicists. We want students to develop specific scientific reasoning abilities that are the hall-mark of scientific thinking. These include, analyzing and interpreting experimental data, designing an experiment to test different hypotheses, identifying assumptions in a physical model amongst many others. Physics courses such as the Investigative Science Learning Environment (ISLE) have been developed to focus specifically on developing students' scientific reasoning abilities. Research has shown that ISLE is successful in achieving its goal. We would like our assessments to directly reflect our learning goals for our students. In order to measure higher-level scientific reasoning, we can, for example, require students to participate in a laboratory practical exam in which they have to engage in experimental design and analysis. However, this assessment method could become very difficult to administer and grade in a large-enrollment class. Is it possible to assess scientific thinking abilities of students using traditional formats such as paper and pencil exams? In this talk I will present some of our latest ideas about how to re-design traditional exam questions to measure a range of scientific reasoning abilities.