As Benedict Carey wrote about this week in the NYT (http://nyti.ms/1U8G10F), few scientific studies are replicated. An effort recently undertaken by the Reproducibility Project at the Center for Open Science highlights why replication is important–particularly for studies that may influence policy and practice. This group recently published a report summarizing their replication of 100 psychology studies. They found that many of these peer-reviewed, published papers reported effect sizes that could not be substantiated. In fact, some of the original studies reported effect sizes larger than what was found upon replication within the population to which the study was designed to be generalizable.
A few things to remember about social science studies: (1) most studies recruit participants who are, as a group, meant to be representative of some larger population; (2) analysis of data using statistical models is meant to reveal population trends but the findings are only as good as: (a) the measures used, (b) the degree to which the sample represents the population of interest, and (c) the fit of the questions being asked, the data collected, and the analytic techniques selected to address the questions; (3) effect sizes are meant to translate population estimates generated by the statistical models used to answer the research questions into a more meaningful metric. Generally speaking, the larger the effect size, the more successful the (experimental) manipulation was in generating a difference between one group (who experienced one thing) and another group (who experienced something else).
What is the real-life significance of this report? First, it highlights how different the acceptable time frames for action are for policymakers and practitioners, on the one hand, and researchers, on the other. Policymakers and practitioners are working on a more compressed time frame–responding to the needs of specific people being served by policies and programs. Researchers may be motivated by similar things; yet, researchers are beholden to a scientific process that yields evidence more relevant to the “average” person in any population.
Furthermore, the process of science is lengthy and complex. Social and psychological phenomena, in particular, are often influenced by a complex array of interdependent characteristics of people and the settings in which they are embedded. When a study seems to identify a silver bullet for a persistent and puzzling social problem, everyone ought to call for more evidence. The scientific process, after all, is best deployed to falsify hypotheses and generate testable theories rather than reveal universally-valid truths.
But, what constitutes adequate replication? Are there other safeguards that can be put in place that would increase the replicability of studies? Or, standards that would help researchers understand what steps ought to be taken in reporting the generalizability of the findings and real-world significance of statistically significant findings? The IES’s What Works Clearinghouse addresses these issues and more by tackling research tailored to educational settings.
In future posts, I hope to explore these topics more. Join me!
