Cognitive difficulty and format of exams predicts gender and socio-economic gaps in exam performance of students in introductory science courses

National agencies have called for the promotion of deeper conceptual thinking in STEM students and the development of assessments that test this conceptual understanding. However, changing the characteristics of questions can result in bias against historically underrepresented groups. As instructors change their assessments to test higher-order thinking, it will be important to understand if and how these changes impact students. We collected 4800$+$ students' exam data from 87 instructor-generated exams taken across three years of the introductory biology series at a large research university. We determined the Bloom's level and the percentage of constructed-response questions (Percent.CR) for each exam. We explored how changing the characteristics of an exam differentially impact the exam scores of male and female as well as high- and low socio-economic status (SES) students while controlling for prior academic ability. We found that males and high-SES students disproportionately benefitted as the Bloom`s level of exams increased. Male and female students equally benefitted from increasing the Percent.CR of exams; however, high-SES students disproportionately benefitted from increasing the Percent.CR of exams. Given that we controlled for prior academic ability, our findings do not likely reflect differences in academic ability level. We discuss possible explanations for our findings (i.e., stereotype threat) and future research directions.