States are considered “laboratories of democracy,” allowing experimentation with innovative reforms that could potentially be adopted at the national level. To test new approaches in their Medicaid programs, states have embraced Section 1115 waivers, which permit innovations that do not meet federal program rules, but still promote the program’s objectives. Recent proposals have included controversial, previously untested policies, foremost among them community engagement requirements, which mandate that nondisabled, nonpregnant adults participate in one or more of a wide range of activities – such as work, job search, volunteer activities, or school – to maintain Medicaid coverage. In a New England Journal of Medicine Perspective, the principal investigators of the independent academic team evaluating Kentucky’s 1115 Medicaid waiver, Kristen Underhill of Columbia University and Atheen Venkataramani and Kevin Volpp of the University of Pennsylvania, note that rigorous evaluations of such untested policies are needed to produce meaningful evidence and yield useful lessons for Medicaid program design.

The Centers for Medicare and Medicaid Services (CMS) requires independent evaluation of all Section 1115 demonstrations, and public posting of evaluation design and results. But prior evaluations have proven of little use to policymakers. Methodologic issues, including the lack of a comparison group, insufficient sample sizes, and selective reporting of outcomes has made it difficult for policymakers to discern what has worked. Reports are often delayed for years after waiver cycles, limiting their usefulness to CMS and to other states.

The authors argue that using rigorous study designs, particularly randomized, controlled trials, can strengthen evaluations. They identify and counter three concerns regarding the incorporation of randomization into evaluations of Section 1115 demonstrations.

  1. Administrative capacity: Randomization would require states to implement a different version of Medicaid for each study arm, which could be an administrative strain on states. However, many states already do something similar; for example, recent waivers focus on people covered under Medicaid expansion, but exempt other beneficiaries, such as those who are pregnant, elderly or medically frail. States have the infrastructure in place to provide different sets of benefits to an experimental and a control group.
  2. Equipoise. Research ethics requires that there be true uncertainty (equipoise) about the benefits or harms of the intervention, compared to the existing program. Is that the case for current program beneficiaries? Currently, if CMS approves a new waiver policy, all eligible beneficiaries are exposed to the new terms of the waiver. The authors point out that this exposes a large group of beneficiaries to an untested policy, and defeats the purpose of waivers, which is to test the effects of changes to Medicaid policy. Instead, this can be better achieved through randomization.
  3. Methodologic choices: Randomized, controlled trials require a number of design choices, such as determining whether to test the waiver in individual terms or as a whole, and defining units of randomization. However, these choices are manageable with evaluation expertise, and not as difficult as identifying a control group in nonrandomized evaluations.

The authors also encourage evaluators to embrace a mixed-methods approach, drawing insights from both quantitative analyses and qualitative methods. They conclude:

“Section 1115 waivers rely on experimentalism, and the model of states as laboratories can best fulfill this commitment by producing meaningful evidence of the effects of experimental programs. For untested waiver policies, we believe that CMS and states should take ‘experimentalism’ literally and harness the rigor of randomized, controlled trials.”

Read the full Perspective here