Discussions of drug pricing often link high prices with the need to promote innovation in the pharmaceutical industry. New technology can drive improved health outcomes, social welfare, and economic growth, but assigning value to specific innovations is technically difficult.

For economists, counts of patents and their citations often serve as measures of technological and scientific discovery, but the robustness of these indicators remains an open question. At a recent LDI seminar, Bhaven Sampat of Columbia University presented preliminary data from a forthcoming NBER working paper with LDI Senior Fellow David Abrams, which tests the validity of patent citations (the number of times a patent is cited by subsequent patents) as a measure of innovation in the pharmaceutical industry. Their findings challenge the notion that patent citations fully capture the social and private value of drugs, while indicating a potential mismatch between the social and private value of drugs.

Pharmaceuticals have many advantages for assessing the link between patent citations and the value of new technologies: drug companies rely on patents to protect revenues, nearly all new drugs are patented, and firms have strong incentives to cite all relevant literature in patent applications because the financial risk of having a patent invalidated are enormous. Furthermore, unlike manufactured goods—such as consumer electronics and software—drugs rely on a handful of keystone patents, rather than hundreds or thousands of seemingly unrelated innovations. Thus, studying patents in the context of drugs allows researchers to reasonably assume that the right patents are cited and that patents can be linked to products.

Furthermore, measuring the private and public value of drugs is relatively straightforward compared to other technologies. Despite ongoing controversy over how to measure the value of drugs, pharmacoeconomics provide reasonably well-accepted metrics of  the social value of drugs: aggregate and average quality adjusted life years (QALYs) per intervention. Many drugs have data on QALYs because numerous payers—especially in Europe—require some form of cost effectiveness analysis. Finally, the marginal cost of production of drugs post-development are generally low, so data on drug revenues can accurately estimate profits (the private value of specific drugs). Thus, compared to consumer electronics, automobiles, or clothing, drugs are a relatively straightforward value proposition: patent citations are easy to track, and economists generally agree on measures of social and private value.

Sampat and Abrams used data on 101 new molecular entities from the FDA, the US Patent and Trade Office, cost effectiveness analyses, and pharmacy claims to estimate technological advancement and value (both private and social) as a function of patent citations from 1987 to 2011. The results suggest only weak associations between patent citations, social value, private value, and technological advancement. Additionally, they found significant variation in how well the private and social value of drugs are aligned.

As a measure of technological advancement, drugs that had no benefit over existing treatments had fewer citations on average, but citation counts failed to strongly differentiate between drugs that had some benefit over existing therapies (known as “advanced-in-class”) and breakthroughs (known as “first in class” interventions). This suggests that while citations are associated with some level of technological advancement, they can’t differentiate levels of innovation above a certain threshold.

The findings for social and private value were similarly mixed. There was no observed relationship between total additional QALYs and citations; however, there was a small, positive correlation when drugs with negative or zero additional QALYs—i.e., drugs that offer no benefit beyond existing therapies–were excluded. This suggests that patent citations are only a reasonable measure of marginal social value when the value is already positive. From a revenue perspective, drugs with more citations did, on average, have more revenue—but the effect size was weak.

Most stakeholders agree that drugs with substantial social value ought to be rewarded with high revenue, and Abrams and Sampat investigated the extent to which social and private value are correlated, using a $50,000/QALY value. On the one hand, the mean and median ratio of social to private value were 10 and 2, respectively, suggesting that most drugs provide benefits beyond their costs. However, these positive averages mask tremendous heterogeneity: of 101 drugs, 43 had lower social value than private value.  Many, but not all, innovations are worth the cost.

What is the upshot for policymakers? Prescription drugs are often held as the leading case for a strong patent system: billions of dollars in R&D investment can only be incentivized via ironclad intellectual property protections. With Americans spending over $300 billion annually on prescription drugs, the stakes for incentivizing the right kinds of innovation and paying for value are high. Options for reform of the drug development and purchase process, such as using federal innovation prizes instead of patent protection, rest on the assumption that private and social value of drugs are misaligned, and that patents do not track well with true innovation. This recent work lends credence to those arguments: drugs that rack up patent citations may not be the most innovative, and the most valuable drugs may not be generating the most revenue.