Despite the bias against publishing null results, they are important for policy, helping to kill bad ideas.
I recently had the pleasure of hosting the closing panel at the CSAE Conference. I will write a separate post summarising this discussion, but one point that stuck out was made by Ranil Dissanayake, who highlighted a key role that evidence can play - killing bad ideas. It’s one thing encouraging a policymaker to fund and implement a new programme, it’s much easier to convince them that an existing programme is not having it’s intended impact. Whether it’s providing textbooks or the Millennium Villages Project, economic research has played an important role in stopping ineffective interventions, freeing up resources to (hopefully) be put towards more productive uses.
While these results ended up influencing policy, there are a host of well-documented reasons why there is a bias against publishing null results in economics journals. I am not going to get into these here. Instead, I thought I would highlight research featured on VoxDev that did not find statistically significant impacts. In my experience, these often result in more interesting, thoughtful summary articles. Instead of going through the motions, authors need to more carefully interrogate why there was no impact. This often means they adopt a more policy-friendly lens, thinking through implementation and other factors more important to policymakers than academics more carefully.
Microfinance: Transformative impacts? Nulls across studies
One area which held much promise and hype, but has since come under greater scrutiny, is microcredit. Evidence from the first generation of microcredit RCTs, highlighted in our recently updated VoxDevLit on Microfinance, highlights that access to microcredit does not translate into increases in profit, income, labour supply, average consumption or gains in social indicators. Rachael Meager outlines the appropriate method to understand these average impacts across studies here.
At this point, it is worth highlighting that there are different types of nulls – many of the studies in the microcredit literature are imprecise, meaning that while we can’t rule out zero effects (i.e. the null) we also can’t rule out the possibility there were large effects that studies were not powered to pick up – for examples of precise nulls, read on.
These insights from the first round of RCTs on microcredit have encouraged researchers to dig deeper and interrogate findings more carefully – rather than assuming this intervention would benefit all, asking when it could work and for who.
Improving parenting practices: Cheap alternative? Null at scale
Can we improve early childhood parenting practices at scale with low-cost technology? Improving parenting practices through home visits and group meetings has benefitted child development in low- and middle-income countries, but these policies are expensive and impractical to scale. Given this, Irma Arteaga, Andreas de Barros and Alejandro Ganimian evaluated a programme in India which sent automated phone calls with parenting advice to thousands of caregivers of young children, aiming to replicate the success of intensive home-visit programmes but at a fraction of the cost. However, the evaluation found the intervention was not effective in improving child-rearing practices or child development outcomes. Cheap, easily scalable interventions can fail to achieve impact at scale. Recently, John A. List discussed his work on this “voltage drop”, and how we can identify effective policies at scale.
Free contraception: Popular hypothesis meets precise null
A popular hypothesis in development is that unmet need for contraceptives is a key driver of high fertility rates. Pascaline Dupas, Seema Jayachandran, Adriana Lleras-Muney and Pauline Rossi tested this by conducting a large three-year RCT in rural Burkina Faso, which provided free access to all modern contraceptives to thousands of married women aged 17 to 35 via local health clinics. The outcome was striking: removing all financial constraints to accessing contraceptives had no detectable effect on birth rates, suggesting that in this context, high fertility reflected desired family size rather than lack of contraceptive access.
Laptops: 10 years on, still a null
The One Laptop per Child programme was a widely publicised initiative that gave students in over 40 developing countries a personal laptop. A large scale experiment in Peru found no effects on test scores or enrolment after 15 months. Santiago Cueto, Diether W. Beuermann, Julian Cristia, Ofer Malamud and Francisco Pardo follow up 10 years on, and find that laptops did not improve academic achievement in the long run. This might have seemed obvious in retrospect, but long-term follow ups are important and can uncover impacts that don’t materialise in the short term – see, for example, research on the lifetime benefits of the New Deal’s youth employment programme.
Refugee crisis: Policy-induced null result
During the Syrian refugee crisis, the number of Syrians in Jordan’s public schools increased from 558 to 108,913 between 2012 and 2013. Despite this, Ragui Assaad, Thomas Ginn and Mohamed Saleh found no evidence that Syrian refugees affected Jordanian students’ educational outcomes. Why? This null result was due to policy action. The Jordanian government – with donor support – responded aggressively by opening new schools, hiring teachers, and adding classrooms in areas with many refugee children. As a result, student-teacher ratios and class sizes for Jordanian students remained largely unchanged despite the influx.
Other nulls and null-related writing
- The limited effect of a mobile-linked deposit service on increasing savings: Evidence from Sri Lanka – Suresh De Mel, Craig McIntosh, Ketki Sheth and Christopher Woodruff
- Policies attempting to encourage firm formalisation (Determinants of informality, in Gabriel Ulyssea’s VoxDevLit on Informality).
- Implementation matters: Measure it and account for it – Noam Angrist and Rachael Meager
- Causal claims in economics - Prashant Garg and Thiemo Fetzer
- The null result penalty – Andreas Stegmann, Ingar Haaland, Christopher Roth and Felix Chopra
- Michala Riis-Vestergaard on J-PAL: So, you got a null result. Now what?
- How to Publish Statistically Insignificant Results in Economics - David Evans (thanks Owen Ozier for the suggestion).
Care about policy? Write about your nulls
If you are an academic with a null result, please don’t be dissuaded in submitting your paper to VoxDev. While aiming to encourage the adoption of better policies, lets not forget economists important role in killing bad ones by building up an evidence base on what doesn’t work.