The Perils of PlayPumps and Other Cautionary Tales – An Interview with Mulago’s Kevin Starr
Mulago Foundation Director Kevin Starr’s no-holds-barred talk at PopTech in 2010 charted some of the more spectacular flame-outs and fiascoes in the world of global social impact at the time. We caught up with Starr recently to talk about the dynamics of failure, his formula for success at Mulago, and whether he thinks anyone’s learned any lessons since he last spoke out.
Impact Design Hub: Can you describe the Mulago Foundation and its mission?
Kevin Starr: We’re a private foundation and our mission is to meet the basic needs of the very poor. We do that through unrestricted funding to organizations that we think have a scalable solution in health, livelihood, education, and agriculture. We follow them over time, we continue funding as long as we’re in our trajectory of impact.
IDH: You’re pretty critical of some of the darlings in the international development field, projects like One Laptop Per Child and PlayPumps. Do you think those projects could have become successful through a pivot or through iteration, or was there something fundamentally flawed about how they were drawn up from the outset?
KS: I guess everybody can pivot towards something. I think that if you talk to computer people about One Laptop Per Child, they’d point out that there were some really useful innovations about low cost computers that made it into industry, and, hence, eventually made (or will make) it into innovations that benefit the poor. It’s a long and winding road and I don’t know enough to comment on it. But PlayPumps, while I suppose they could have pivoted to a better pump technology, they went way out on a limb for this thing that they built all their fundraising around and all their messaging around and they marched themselves right off the edge of a cliff. They just went well beyond the point of no return.
IDH: How does Mulago evaluate which projects are worth funding and which you think will fail?
KS: We have four fairly simple questions: Is it needed? Does it work? Will people use it well? And will it reach those who need it most?
IDH: How did you arrive at that process? Was it based on projects that had failed in the past?
KS: I think it came from the basic question we ask ourselves when we look at an organization, and that’s whether their mission aligns with ours, whether they have the same beliefs about impact that we do. We’ve always looked for some evidence of impact to date that hits the target population, evidence of behavior change, and clear thinking about how to go to scale. These kinds of questions are what morphed into our current four big questions, which are really just a useful simplification of everything we’ve always looked for.
IDH: You put a lot of emphasis on measurements and measuring the right thing. Can you give some examples from your experience of how measuring the wrong thing leads to failure?
KS: Usually it’s because people measure what we would call “delivery,” which are more outputs and activities rather than the eventual outcomes that represent an impact in the world. People will measure things like changes in attitudes and knowledge and assume they’ve accomplished something. At the very least we want to see evidence of behavior change that is in some way rigorously connected to impact, but what’s best is if impact is measured directly.
I think micro-lending is still the best example, or the most visible example of an instance where there was an assumption about measurement that was leading people astray. The thought was that the behavior of taking loans and paying them back made it a slam dunk, that people were actually getting better off, more prosperous, or wealthy as a result. People took repayment as a proxy for impact — for recipients of microcredit increasing their wealth — when in fact there wasn’t enough rigorous data analysis to validate that conclusion. Nobody checked that for far too long. Then we found out that the net impact of all this micro-finance is, in our view, more or less negligible.
IDH: So how can you be confident that you are, in fact, measuring the right thing in the right way?
KS: That’s why we are really obsessive about getting the mission right with a formula of “verb, target population, outcome.” We try to make that crystal clear. Verb, target, outcome — get, African families, out of extreme poverty, for example. Once you determine that, then you need to work with the statisticians to get an idea of what would produce good enough numbers — what methods and sample sizes you need. Then you have to make a case for attribution, because your impact is the difference between what happened with you and what would have happened without you. We don’t always get the most rigorous counter-factual that we’d like, but what we want to see is a persuasive case for attribution that can be validated over time.
IDH: You’ve said that you didn’t really have a clue about this work starting out, so how did you learn to be effective? There must have been a learning curve and some failures of your own along the way.
KS: My international health mentor from medical school died while we were working together in Bolivia, and his family established the Mulago Foundation to carry on his work, so I was definitely thrown suddenly into the world of philanthropy and social impact on a broader scale. Because I had no real background in the social sector, I didn’t realize that it was the norm not to measure, so I didn’t have that bad habit. I was trained as a physician to be oriented toward outcomes, and to use data to make good decisions.
IDH: Did that model pretty much start out fully formed. Is it the same now as it was then?
KS: Oh, god no. The first ten years were wandering around in the wilderness, literally and figuratively. Nowadays I talk to smart, young philanthropists who face the same struggles, just on a wildly accelerated version of my learning curve. They’ve learned from our experiences among many others. Also, more and more people are coming into impact from the business world with an intuitive sense that impact can be viewed similarly to profit. I don’t mean profit-making businesses creating impact, but simply that organizations need to be run in a businesslike fashion toward the goal of maximizing impact.
IDH: How do you guide your grantees through their own wilderness, through the failure that, presumably and inevitably, they make and have to iterate through?
KS: The best analogy for us is venture capital, where you take good people with what seems like at least the germ of a great idea, and you work with them through an iterative process to get to a fully formed idea that is executed out of quality. That’s exactly what we do. We have a design process to try to help that along, but it is intended as a vehicle for iterative design changes and evolution of a model over time. Beyond that, it’s simply going on the journey with them and trying to be useful as they confront the challenges of a growing organization that is refining a specific methodology.
IDH: Have you had any grantees that have had to go through a pretty major pivot? Do any examples come to mind?
KS: Yeah, a lot of them have, but one of our great investees was Off Grid Electric, whose work is installing pay-as-you-go solar systems onto people’s houses so they can get high quality light at an affordable price. When we first met them, they were interested in the idea of micro grids using the excess power from mobile tower generation — diesel and solar generation. That was an interesting idea, but they pretty quickly abandoned that idea for a better one and went on to what has proven to be a very successful model.
IDH: Do you think there’s something that ties all those products you’ve criticized in the past together — a common reason that projects like PlayPumps failed, or something they were missing that you’ve discovered is crucial to success in the projects you support like Off Grid Electric?
KS: Here’s the thing: I think people are getting smarter and we’re not seeing as many of those kinds of initiatives anymore. In fact I cannot think of anything that quite hits that low mark of late. A lot of it has to do with the emergence of more of the user-centered design approach, where if you do it right, you’re just not going to end up with products that don’t have a point to them or that simply don’t work. The funders are also getting smarter and they don’t want to be caught with their pants down investing in something stupid. It means that both the funders and doers are eager to hear the voices of people more experienced than them and who have experienced a few failures themselves, and a few successes, and who have a good overview of the field and are willing to listen. A lot of what happened with PlayPumps was because they simply didn’t listen. There were some very smart people who were saying that this was a bad idea all along.
IDH: So has your viewpoint changed in the last few years, since your PopTech talk? Would that talk be different today?
KS: Yeah, I think things are way better today. The social entrepreneurship world is in full gear and is producing a lot of really smart people with really good ideas. The funding world is getting smarter. The awareness of what it means to do dumb stuff is growing. People are a little bit more careful. And there’s a lot more eyes on development now than there used to be. It’s a really good time to be doing this.
IDH: Have funders and organizations coming in “from the outside” become more humble? Because there’s always the danger of thinking of yourself as the agent of change with superior knowledge, or technology, or whatever, to save people or to come up with a solution. That can make people blind to local capacities, and the various complexities and challenges that are going to exist. Is that an attitude you think is changing — are people becoming more sophisticated in that regard?
KS: The short answer is yes, and the long answer is that we’re still held back in a profound way by the lack of a market for impact. A certain circle of funders is getting smarter, and because of that they’re rewarding the efforts of doers who are smarter, but the whole system doesn’t really function as a whole. If you blundered into the Democratic Republic of Congo with a bad idea for making money, you’d go broke and your investors would go broke, and no one would invest in you next time unless you had a way better idea to execute. If your investors did that too many times, they’d go out of business or get fired, and so there would be this whole system that eliminates dumb doers and dumb funders, but that doesn’t exist in the social sector yet. Foundation heads do not get fired for lack of impact, and charismatic doers can keep raising money for stuff that doesn’t work.
IDH: So in a sense there’s too much tolerance for failure, so to speak, in the system right now?
KS: Yeah, too much tolerance of big failure, and too much fear of little failure. What we want to know is, if something fails, were they treating it as an experiment, and are they going to do something different as a result. Was it a stupid thing to do, or was it a smart thing to try that didn’t work that leads to something else that might work? The whole notion of failure has just been poorly defined in the social sector. You see these people wanting to celebrate failure, and I feel like we’ve been celebrating failure for a long time in this sector, and yet we’re not always as encouraging of the cycle of constant testing and changing course that could be framed as failure.
If you think about PlayPump’s failure — tens of millions of dollars later, organizations dissolving, people let down — and then you think about Off Grid Electric. Off Grid put a certain amount of investment — not that much — into an idea that turned out to not be that great and then quickly pivoted toward a way better idea that they’ve since put a lot of money into. Well, was that first idea a failure, or was it just a first shot at a good idea?
IDH: Do you see any prospects for getting more accountability, like there is in the for-profit sector, into the development sector?
KS: Yeah, I think we’re making progress. I’d like to see it be a lot faster and a lot more pervasive. What’s driving good funders right now is mostly pride — a sense of duty, integrity. There are no fundamental rewards for impact in the funding world and it’s completely unaccountable in terms of validating impact. It’s likely that only creating new behavioral norms and culture can drive good funder behavior. There’s no equivalent to going out of business.We’re social animals and funders do hang together: maybe someday those who don’t have real impact will at least be embarrassed.