False Dichotomies and Health Policy Research Designs: Randomized Trials Are Not Always the Answer

In the Journal of General Internal Medicine, Stephen Soumerai and colleagues, including LDI Senior Fellow Ross Koppel, highlight the benefits of using quasi-experimental designs and observational data when a study neither permits or requires a randomized controlled trial (RCT). Using Campbell and Stanley’s classic research design monograph, which classifies research as randomized experiments, strong quasi-experiments, and weak pre-experiments, the authors provide a simple hierarchy of common strong and weak designs, with RCTs and interrupted time series (ITS) both classified as strong designs that often show trustworthy effects. ITS designs allow researchers to control for baseline secular trends, observe a sudden effect of an intervention (i.e., change in level or slope) and assess the stability of the change over time. These research designs are most trustworthy when control or comparison groups are available. The authors document that ITS designs have become more prevalent in recent years: between 1996 and 2015, the number of ITS studies in PubMed increased from 12 to 239 per year. They recommend expanded training in quasi-experimental designs in medical and graduate schools and in postdoctoral curricula to foster the use of thesde designs and to reduce the prevalence of poorly controlled studies.