Evaluating scientific impact: A control group study at the Gordon and Betty Moore Foundation

Measuring scientific impact is at the heart of the question, 'Do people who receive grant funding have more scientific impact than their equal-potential counterparts who do not get funded?' In partnership with the Gordon and Betty Moore Foundation Experimental Physics Investigator (EPI) Initiative, we present a method and preliminary results for measuring the scientific impact of funded vs. unfunded applicants. The method draws on the network normalized citation index, Ĉ. This citation index addresses concerns common to other citation indices, particularly that different fields have different publication practices and that citation counts increase over time. The citation index, Ĉ, compares papers to others in the same subdiscipline not by determining a priori the subdiscipline, but instead by creating a citation network and then using the average yearly citations of nearest neighbors to normalize. Because Ĉ uses average yearly citations, it also accounts for the cumulative nature of citations. We present a preliminary analysis of the 2022 cohort of the EPI Initiative, as we intend to follow a similar analysis pattern for other cohorts in a pre-registered study of scientific impact. In that study, we will not include the 2022 cohort, but use this to demonstrate proof of the viability of the data collection and analysis. The 2022 cohort included 16 funded EPI investigators and 8 members of a comparison group (i.e., equal-potential, unfunded applicants). This analysis is based on the set of papers from the 2022 cohort of EPI investigators (N = 734) and comparison group members (N = 434). These papers were drawn from the OpenAlex database on 22 October 2024.