How do you evaluate an organization that does all of this?

Elie and I have been wrestling with this problem for the past few days, especially today. We’ve put together what we think is a reasonable final-round application (see below), and sent it to five of our strongest huge-comprehensive-giganto-mega-charities to see how they feel about it. Generally, they haven’t felt very good, and we’re trying to figure out what this means. We’d love your thoughts.

In our Face/Off a few days ago, Elie argued that a good organization ought to be able to give us a picture of what they do and whether it works – no matter how big they are. If they’re too big to give this picture, then – well, they’re too big. I found this pretty convincing, so we crowned Elie as the winner of our inaugural Face/Off (don’t despair, Holden diehards, there will be more to come) and created the following application:

• Application for mega-charities: instructions
• Application for mega-charities: The Matrix

We’ve dubbed this app The Matrix, because its key feature is a gigantic matrix of regions and indicators – we want to know what each charity does and doesn’t have data on, in every region it works in. It’s visually gargantuan, but we’re not asking applicants to fill in statistics in the cells. All we’re asking is that they tell us what they do and don’t do – and what they do and don’t measure – in each of their regions.

All of the people we got to talk to today agreed that the application was off-putting/overwhelming, at least at first glance. We then explained how we want them to go about it: send us what they already have, and use The Matrix just to tell us what they sent (we’re not asking them to do any writing, interpretation, or summarizing – just send what they have and classify it). One of the applicants said this was fine, but the others both hesitated even then. We were told that just compiling what activities are done in each region could take a major project; that pulling up all the relevant reports (and categorizing them) could take weeks.

So my response was: How do you tell what’s going on at a bird’s-eye view level? How does your Executive Director know what’s going on and how you’re doing? How does your Board know? What do you look at to decide which programs to expand and which ones to change?

The basic picture we got is that priorities are often driven project-by-project: the people on the ground (whether local organizations seeking help, or staff in the field) make proposals, and the central office reviews them individually. After some discussion, we came to differing agreements with each organization, and we’ll see what they send us (and keep you posted).

But in my mind, the biggest and most important question (the one in bold above) remains largely unanswered. It seems to me like if you’re running 200 different programs in 20 different countries, you need to be monitoring the heck out of them, and you need to have some kind of summary view that you can show to the people in charge and give them a real picture of what’s going on. Otherwise, how are those people in charge?

I don’t hold small organizations, or simple organizations, to the same standard of measurement and organization. A bicycle doesn’t need a dashboard, because you can tell immediately if something’s wrong; unmetaphorically, if you work in one place, doing one thing, you can be part of the day-to-day activities and understand them intuitively, without ever measuring or documenting a thing. But for the life of me, I can’t understand how it’s possible to have an “intuitive” feel for your work when you’re trying to help thousands of different people, thousands of miles away, living in cultures and regions you didn’t grow up in and will never truly understand. It seems like the only way to have any idea of what’s going on is to collect an enormous quantity of facts and put great care into interpreting and organizing them. Elie and I recognize that we aren’t experienced in these matters … but the idea that an organization would take weeks to put together a summary of what it does and whether it works is just hard for us to swallow.

What do you think? Are we barking up the wrong tree? Is it unreasonable to ask charities for this much organized information? Should we be trying to evaluate charities without getting the full picture of their activities? How can we?

Can you be comfortable donating to a mega-vehicle, without seeing the dashboard?