Will It Work Here Too? External Validity and Scaling Up

Why a programme that succeeds in one study may not work elsewhere or at a larger scale, and how economists try to judge when results travel.

A study might show convincingly that a programme worked. But policymakers usually want to know something else: will it work in their country, city or school, and will it still work if rolled out to millions of people? This is the question of external validity.

Two kinds of validity

Internal validity asks whether a study correctly identified the effect in its own setting. A well-run randomised trial has high internal validity. External validity asks whether the result applies beyond that setting. A study can be strong on the first and weak on the second.

Why results may not travel

Several reasons can stop a result from carrying over:

Different contexts: a programme that worked in one country may depend on local institutions, culture or prices.
Who runs it: small pilots are often run by motivated researchers or charities. Government systems at scale may deliver the programme differently.
Scale effects: when a programme becomes large, it can change prices and wages across the economy. Economists call these general equilibrium effects.

Contract teachers in Kenya

A study found that hiring extra contract teachers through a non-profit organisation improved pupils' test scores in Kenya. When the same programme was run through the Kenyan government as part of a larger scale-up, researchers found no significant improvement. Differences in how the programme was implemented, and political pressures around teacher contracts, appear to have mattered. The result that worked in a pilot did not survive scaling up.

Making results travel better

Economists try to improve external validity by testing programmes in several countries, studying why programmes work and not just whether they do, and running trials at larger scale with governments. A 2015 set of studies of a programme giving very poor households assets such as livestock, along with training and support, tested it in six countries and found positive effects in most of them, giving more confidence that it could work elsewhere.

Esther Duflo has described the economist’s role in policy as something like a plumber: paying close attention to the practical details of how programmes are delivered, because those details often decide success.

Assuming a proven programme will work anywhere

A strong result from one place is valuable evidence, but it is not a guarantee. Before copying a programme, it helps to ask whether the conditions that made it work are present, and whether it can be delivered as well at a larger scale.

Key takeaways

Internal validity asks if a study got the right answer in its setting; external validity asks if it applies elsewhere.
Context, delivery and scale can all change a programme's effect.
A Kenyan contract teacher programme worked in a pilot but not when scaled through government.
Multi-country trials and attention to delivery details help results travel.