Professor of Economics at Yale University and researcher at the Abdul Latif Jameel Poverty Action Lab of the Massachusetts Institute of Technology, Dean Karlan is president and founder of Innovations for Poverty Action (IPA), an NGO that seeks to evaluate and implementation of innovative anti-poverty programs that rely on behavioral economics and randomized controlled trials.
Youphil.com: How can behavioral economics inform anti-poverty programs?
Dean Karlan: Many poverty programs are designed a bit abstractly, on a high level. We are learning that we have to take into account the reality of individual decision-making in order to design better programs. The best insights are not about poverty, they are about people. How do we make decisions, rich and poor alike? We then take those insights and apply them in the fight against poverty.
For example, we all respond to incentives and succumb to temptations. There are programs that try to work with the poor to deal with temptation issues, such as finding better ways of managing cash so that it gets saved for longer-term goals.
A study in Kenya, tested a product that helped farmers commit to invest in fertilizers: at a moment when they are flushed with cash, farmers buy vouchers which can only be exchanged for fertilizer at planting season. In the Philippines, a similar project was done for people to quit smoking. They put money in the bank and could have access to it only if they met their commitment to stop smoking. The bank would give them a urine test 6 months after their commitment to check whether they had failed or not.
Behavioral economics as a field was very much focused on laboratory evidence. Now we are seeing a new wave that one may call behavioral economics 2.0, that takes these ideas and tests them in the real world as ways of improving public or business policy.
Do incentives really help the poor get out of poverty?
Certainly not alone and it depends on how you do it. But price does matter - that's all we are saying when we say that incentives matter. IPA researchers did a study in Bangladesh with 8 dollars loans that enabled villagers to cover their moving costs to the city during the lean season, so that they could find a job there. It had a huge impact on making people move, increasing their income and the remittances they were sending back. Plus, it was really just a seasonal move.
Do you feel like development has become more scientific?
Definitely. It's exhilarating to see the enthusiasm that many have for using evidence to guide decisions. It’s actually critical for being more effective and it's been a change in the past ten years.
You are a big advocate for randomized controlled trials (RCTs). What are they?
The goal of an impact study is to answer the following question: how has the live of the people involved in a program changed. This requires thinking through what to ask and to measure, as well as careful attention to data collection. It’s pretty straightforward. But you also need to compare how people’s lives would have been if they hadn’t been involved in the program. This is a hypothetical question: what would they have done if they had not had access to this program or service? An RCT stems from the idea that it’s easier to see why a program is good by explaining why the other alternatives can be weak.
What are these other alternatives that you focus on?
One idea is to focus on other people who haven’t used a service. The problem is that the fact that people chose to participate in a program, or are selected to participate, is usually predictive of some changes in their lives. The success of a program might just be causal: they succeeded because they already wanted a change. Take the example of a job-training program and compare those who joined and those who didn’t. If you look at the employment rates a year later and say that they have increased because of the program, you could be wrong. It could simply be that those who were trying to get a job were more likely to get a job, compared to those who were not trying to get a job. That second interpretation is not very interesting.
So you randomize it?
Yes. An RCT makes sure that the allocation of the program has some random component to it. In many cases projects are already allocated in a random way for ethical and political purposes, as a fair way of choosing who will have access to it when there is limited space. Research takes advantage of this randomization and measures the impact of it. RCTs in social science are often very different from the traditional medical trial: we are trying hard to operate in the real world, and often times we build the randomization into the operations of an organization.
It’s also similar to the way a lot of businesses are run. Take the credit card business: when we receive letters about credit cards in the mail, we are the subjects of a randomized trial. They are testing whether we care about the price and the product. In the past 10 years economists have been taking practices from the corporate world and bringing them to the non-profit and government world. But we do it for public good to help the world allocate their money better and build better public policies.
When do randomized controlled trials come into behavioral economics and how does it help in terms of overall development?
Behavioral economics don’t necessarily go hand in hand with RCTs. RCTs come in because they enable us to sift through the mountains of ideas out there and guide us towards which ones to do. It's about good clean evidence on the effectiveness of ideas. Some of these ideas might be behaviorally motivated or some might be good old ideas like building schools or passing out bed nets.
There is a conditional cash transfer program [that makes welfare programs conditional to the beneficiary's actions] in Colombia that found that delaying the cash transfer increased education by timing the cash transfer closer to the date people have to pay for education. That could sound counter intuitive as we often think more money sooner is strictly better. But the study showed the opposite.
Is one of the issues of RCT scaling and how to transpose something that works in one country to another country?
This is not an issue only with RCTs but with any sort of study. You have to look at the contextual factors that made it work and see whether they are present in the new location. RCTs should not be used strictly for accountability purposes, to look back at a setting and see how a program did. They should say: what should we do?
RCTs are often criticized because they are expensive. Are there other drawbacks to RCTs?
RCTs are not more expensive than any other methods; they are just more expensive than doing nothing or just doing a before/after juxtaposition. If you attempt to make comparisons with people who are not in the program, RCTs are actually cheaper. The real question one should ask is how important it is to have the results, compared to the existing knowledge in the world, and how will that information get used to change future policies. This is a basic question one should ask to justify the expense of any research. As for other drawbacks, they do require a certain level of expertise, and there are settings in which you can't do them. I wouldn't call it a drawback but a limitation.
What are these settings?
Whenever your unit under study is too big. Let’s take it to an extreme: you cannot randomize the monetary policy of a country nor migration policies or any sort of communication campaign where you cannot control the boundaries, such as a campaign that has been blasted on TV or on the radio.
When asked whether organizations should collect data or not, you often refer to the Goldilocks problem. What is it exactly?
The Goldilocks problem is that organizations often either collect too much or too little data, rather than getting it right in the middle. A lot of the push comes from donors that are very data hungry but not analytically rigorous. They might be pushing the funded organization to provide a lot of data with a before/after analysis that doesn't really tell you whether that program caused the change.
Let' s say you are funding nutritional programs and you ask your NGO to weigh people before and to track if the weight of the beneficiaries increases. This would be an over collection, because it is impossible to know if the increase is linked to the work of the NGO. If you already have the information that tells you that these nutritional supplements are going to work, you don't need to track the impact. You need to track that the NGO is doing what it said it was going to do. It’s a huge burden on the resources of the organization.
How does a small NGO know that their idea is good?
There are lots of studies underway. If it's good enough to get funding, it should be good enough to get funding to prove the idea. There are venture philanthropists who say that they want to fund something new and want to document that it works so that others can learn from it.
So should NGOs be proving the idea as well? Their resources are limited.
No - it's a collaboration. IPA is an NGO and we do RCTs. Most of our projects are partnerships with implementing organizations. You need to be at a certain scale so that doing RCTs makes sense. If something is in its early stage, it’s probably too early to do an RCT. Most of the time small NGOs do something that has evidence somewhere else. If 17 NGOs are doing 17 different remedial education projects in 17 different villages, they shouldn’t all do RCTs. But what would be great is if they use evidence out there to guide their work.
Is there good communication between institutions like yours who have the data and NGOs who implement on the ground?
It’s a spectrum. It needs to improve for sure and it’s a huge focus of ours these days. In early September, IPA hosted a three day conference with the Asian Development Bank in Bangkok with 300 people from governments, NGOs and donors, hearing about results and thinking proactively about what can be done and how they can be used to better the work of each of these entities. It was an exhilarating experience.
What are the challenges in making sure this information travels?
Some of it is a matter of time and labor. The problem is that programs can be transferred in some ways but can never be exactly the same. That type of customization is difficult to get out, and is different than just handing in the result on paper. That is why interaction is better.
The other bottleneck is that there are still areas where we are much lighter on the evidence than we want to be. There are many situations where there is only one solid evaluation. It should push people to do more studies to see if we can find consistent results or not. But from an institutional perspective, academics don’t have the incentive to repeat a study. Yet from a public policy perspective this is what we need to do.
Does impact investing have the same challenges in terms of measurement?
Impact investing has a challenge that it is failing to take head on: it wants to take the word impact seriously yet it doesn't do anything to show its impact in a rigorous way. The reason is that it wants to appeal to profit investors who make a positive return on their money, but it costs money to show your impact. They are putting the word impact in front of things and yet they are not showing the evidence. They are lacking assessment, and many say that as long as the intent is good, it's good for us.
But don’t they have to measure the impact to give out the returns?
That’s not what is happening. It goes back to the Goldilocks problem: they will present really bad data – such as the number of poor people they are reaching – and then they argue that they are doing good simply based on the theory of the idea. If the only empirical criterion is that you reach a number people, then the most effective impact investing programs of the world are the alcohol and cigarette industries. A lot of impact investing should just be called investing; but these people should hold their heads high and say that they are putting capital in places that need it and are making the world more efficient. Considering the way people define impact investing today, Walmart should be considered one of the most successful impact investment programs: it has brought prices down for millions of people by lowering their costs of distribution channels and that’s huge for expanding access to goods.
In Europe, we talk more and more about “the new poor clients”. Can RCTs done in the developing world be used for policies in the developed world?
Economic insights are about people. They don't listen to national boundaries and many cut across economic strata as well. For example, the smoking studies done in the Philippines can apply to America and Europe. I started a company, stickK.com, that works with american firms to improve their wellness programs for employees. The research at the root of this company was done in both developed and developing countries. So yes, some of the innovations from developing countries can be applied to developed ones. And some of them, like mobile money for instance, are even further ahead.
Photo credit: UN Photo/Kibae Park
