Non-profit organizations operate in treacherous terrain: not only must they solicit capital externally from often unreliable pools of funding, they must also provide evidence proving that their programs and operations are effective within the communities that they aim to serve. Methods for such efficacy measurement can vary widely and depend heavily upon the organization that they aim to evaluate. Among most, if not all, of such assessment tools, however, is a desire to quantify qualitative statements about an organization’s reach. Examples can include evaluations of an overall mission statement, relevance of the mission statement, or even a specific program’s contribution to an organization’s mission overall. While such assessments vary in scope and function, McKinsey and Company offers a starting point for such strategies in the form of key indicators such as capacity metrics, activity metrics, and mission-impact metrics, to name a few.
Capacity Metrics are a measure of progress at all levels of the organization, thereby enabling it to get things done. These measurements are typically aimed at the organization as a whole, and not necessarily at how (or how effectively) it interacts with its community. The idea, overall, is to assess top-line “success” of an organization operationally. Typical metrics include total membership, public funding, growth in private fund-raising, “Market share”, or executive director salary.
Whereas the aforementioned benchmarks focus on the operational components directly involved with an organization’s sustainability, Activity Metrics are those that are put into place to measure progress toward the goals and programs that drive organizational behavior, though they do not necessarily study the impact of these services on their target communities. These indicators typically focus on an organization’s output, proliferation, or services rendered. Measurable activities may include the number of projects that an organization has launched or the amount of people to whom services were provided.
Up to this point, most of the impact metrics mentioned have probably seemed fairly intuitive and, indeed, many non-profits have such indicators in place. Mission-Impact Metrics, however, are much harder to quantify yet are also arguably the most important in evaluating the success of a non-profit. The aim for these types of metrics is typically to measure progress toward the mission and long-term objectives that drive organizational focus. Such metrics vary greatly by organization and may include figures such as households saved, lives changed, or wildlife threats abated. Such statements are much more difficult to quantify and the process for doing so must be carefully considered.
Evaluating such impact is not only seeking evidence of a programs efficacy, but also entails ensuring both the transparency and objectivity of a report in the form of controls. The equation, then, becomes one of both quantification and controls. This is where a consulting firm, particularly one with a project portfolio as diverse as that of EMH, can add significant value as a long-term advisor. One of the greatest boons of a consultant is the benefit of a high-level, experienced, and data-driven perspective of an outside observer. Where non-profits tend to go wrong is often not in the evaluation methods that they choose, but rather the fashion in which they choose to employ them. As such, we believe that EMH could offer services such as the following to non-profit clients during the course of their programming (among others):
Assurance of Randomization – a common complaint of foundations and other non-profit funders in identifying “high-status” charitable organizations is the evidence of selection bias in proprietary impact evaluations. When evaluating statements of impact, funders cannot be certain that the chosen reports and cases are truly representative of the organization’s services (as opposed to being outliers) or even that the services were not initially offered to those with a propensity to benefit from their use. To better understand this struggle, it is worth noting that, broadly, non-profits resort to one of two types of performance measurement procedures when seeking to quantify mission impact: the Average Treatment Effect (ATE) and the Average Effect on the Treated (ATT). The ATE method aims at evaluating the comparison between “test” and “control” groups that did or did not receive their services (respectively) while the ATT method evaluates the test group contrafactually by assessing their projected outcomes had they not received the services in question. In each case, objectivity is difficult to prove and a significant amount of assumptions have to be both made and proven throughout their documentation. Among other factors, absence of sufficient evidence of randomization in both sample cases and focus groups appears frequently among complaints of funders while evaluating impact. Given that most funders will approach a potential fundee with scrutiny, assuming that a certain amount of selection bias exists in their impact studies (ie. Assuming that they may have offered their services to those that may most aptly receive them, or selecting studies that represent an inordinate scope of improvement), this lack of evidence can prove especially detrimental (let’s call it corporate confirmation bias). It is at this point in the evaluation process that many non-profit organizations will have failed to ensure that accurate objectivity is accounted for in their documentation. At EMH, we can provide counsel on the best practices for implementing randomization processes into impact evaluation and that could prevent this issue from arising during a time of assessment.
Locating, and employing, quantitative instrumental methods during a program or study – So often, impact evaluations will rely on purely qualitative measurements such as open-ended household surveys, before-and-after evaluations, and control groups to assess the impact of their services and programs. To be sure, qualitative metrics are indispensible when assessing the value of a non-profit. In fact, such assessments are often the starting point not only when evaluating an organization’s impact per its mission, but also when crafting a mission initially. Where these metrics can be strengthened is in the introduction of quantifiable metrics that aim to irrefutably portray the success of these statements. In a given study or report, identifying quantifiable, “instrumental” factors throughout the research process can often prove enlightening to the non-profits seeking to prove efficacy. Such factors may include phenomena such as economic impact, frequency of use, or location of services (ie. Was a particular location positively impacted due to nearer proximity to a service than was another?). Searching for these methods can be difficult and involve many efforts to cross-reference variables but, as such, falls squarely into the wheelhouse of firms such as EMH Strategy.
While quantifying impact in non-profits can be difficult, and even stressful, it is not impossible and is typically more successful if the organization operates with a metrics-driven framework in mind early on. With diligence, persistence, and objectivity, impact measurement can become both effective and second nature to any mission-driven organization.
By EMH Consultant Andrew Foley