Evaluating after the barn-door has been left open: Evaluating Heifer’s Give-a-Goat or Give-a-Cow Programs

The gift of a dairy goat represents a lasting, meaningful way for you to help a little boy or girl on the other side of the world (US$120); or how about a flock of ducks (US$20); or go all out and donate a water buffalo ($250). People need "a cow, not a cup"—cows that could produce milk so families would not have to depend on temporary aid.

International animal donation programs of the type mentioned here are one of the most well-known types of charitable requests, and are used by a number of major charities worldwide. The best known exponent of this program is Heifer International, from which the examples above are taken.  The purported goals of such programs are to improve the nutritional outcomes of participating households and provide a pathway out of poverty.

To my knowledge, none of these programs have been subject to a randomized evaluation, nor to a prospective non-experimental evaluation. So one question is what we can learn from the programs that have already occurred? A new paper by Svetlana Pimkina, Rosemary Rawlins, Chris Barrett, Sarah Pedersen and Bruce Wydick tries to do this, by conducting an ex-post non-experimental evaluation of two of Heifer’s animal donation programs in Rwanda, one that donates dairy cows, and one that donates meat goats. The details are as follows:

Interventions:  In both cases recipients presumably get training in livestock care, although this is not described in the paper. Then there are two interventions, which take place in separate regions.

Cow Treatment: beneficiaries are given an imported, pregnant cow which produces a lot more milk than local cow breeds. Recipients are required to give away the first female offspring, but can sell male offspring. Ongoing insemination services are provided so the dairy cows can continue to produce milk and offspring. Approximate value is US$3000. This is given in northern Rwanda in Ruli.

Goat Treatment – beneficiaries are given two female goats. A few beneficiaries in each area are also given a male goat to breed with the area’s female goats. Heifer expects households to breed the goats for sale and/or consumption. The paper doesn’t give the cost. This is given in eastern Rwanda, in Kirehe.

How are recipients selected? Households apply to receive an animal. To get a cow, households must not already have a high producing dairy cow, must have at least one hectare of land, and must commit to constructing a shed with a cement floor and metal roof (through a Heifer loan) if selected. But beyond this “Heifer International was unable to provide a clear set of secondary selection criteria… beneficiaries are selected based on need as determined by a team of assessors…. in some cases the first round of donations were distributed to highly prioritized families, while in other cases the decision was made based on logistics.”

Evaluation methodology: The researchers conducted a cross-sectional survey in the summer of 2011. They surveyed 406 households, consisting of 224 that had applied for the dairy cow program, and 182 for the meat goat program. Sample sizes for anthropometrics are even smaller: a total of 99 children in the cows region and 129 in the goats region. The paper is silent on what the attrition levels were relative to the target list of all such households. Surveys collected information on household dietary diversity and on child anthropometrics. They surveyed three groups of households: those who had received the program one year earlier, those who had been approved to receive an animal sometime in the near future (“prospective households”), and those that had applied, but were not selected by Heifer.  The paper contains no table comparing means across these three groups, so it is hard to see how similar they appear on time-invariant observables.

Estimation is then by simple linear regression and tobit estimation, controlling for region, acceptance into the program, household size, and a few household asset holdings (e.g. phones, machetes, rabbits), and the age and education of the respondent. Some estimation is also by propensity-score matching, where the propensity score is based by matching on only 5 variables given the small sample size.

Estimated Impacts:
• Households receiving a cow consumed 9.3 more liters of milk per month than members of prospective households (p• Households receiving a goat had no increase in dietary diversity;  had a marginally significant increase in meat consumption (of 0.20 kg/person/month)  in some specifications; a 0.47 reduction in wasting measures for children under 5; and no change in stunting measures.
• The authors use Rosenbaum bounds to see how severe the self-selection on unobservables would have to be to explain the results: for the impact of a dairy cow on milk consumption, there would need to be unobserved characteristics perfectly correlated with dairy consumption that make the odds of self-selection 5.5 times higher for those who were treated, which the authors consider unlikely – whereas even mild self-selection would kill the results on goats.

Thoughts on the evaluation
The paper is a valiant effort to provide some first evidence on these types of programs. But it also highlights the difficulties of having to come in after the fact and do an ex-post evaluation – it is unclear how people were selected, there is no pre-program available on them to enable difference-in-differences or better matching, etc. The authors themselves conclude “The promise evident in animal donation programs is apparent, even if the impacts remain difficult to estimate precisely in small, observational studies such as ours.

A follow-up study using a randomized control trial design with long-term tracking would enrich our understanding…”.  So my hope is that such an evaluation does not serve to enable organizations like Heifer to claim they now have rigorous evidence to support their programs and hence that they see no need for prospective evaluations. Moreover, the paper focuses on short-run outcomes in a very narrow domain. To my mind the three most interesting evaluation questions in an evaluation of these types of animal donation programs would be: 

1. Do these programs provide a lasting pathway out of poverty, or do most of the animals end up getting eaten or dying, and households converge back to their former status?
2. Do the programs provide better long-term impacts than simply giving cash? A horse-race, or perhaps a cow or goat race between an animal grant and pure cash grant would be welcome.
3. My sense is that the main advantage of a program like Heifer’s is probably not that it delivers much better outcomes for the recipients than would cash, but rather that it raises a lot more money from people who might not otherwise donate to the poor. So even if Heifer has only one half the impact of a pure cash grant per dollar granted, it may have a much larger impact on the poor if people are many times more likely to give when they have the gimmick of an animal they think they are buying. Testing this empirically would be very interesting.