NVSQ Blog

The Nobel prize in economics awarded to Esther Duflo, Abhijit Banerjee and Michael Kremer in December 2019 has consecrated their game-changing work against poverty. At the heart of their work are experimental approaches using Randomized Controlled Trials (RCTs), which have shed new light on the way the impact of social innovations can be assessed. RCTs compare the impact of a measure between a treatment group and a control group whose participants are selected at random. They are a powerful way to remove biases and isolate a specific action from the great swirl of other factors that may affect the result. However, they are far from being a “one-size-fits-all” approach because they are complex to set up and impose significant technical and financial demands on the organization. They also frequently require long timeframes to set up and run – running into years and decades – making them poor tools to help businesses and investors execute short- to medium-term strategies. RCTs work well to establish causal links between a given intervention and social impact. However, in many instances, the impact evaluation needs for innovators and their supports are quite different – with much more need for tools that can guide performance improvements rather than prove outcomes. The latest research by Anne-Claire Pache and Greg Molecke for the Handbook of Inclusive Innovation suggests that these needs vary based on where social innovators stand in the innovation cycle. We need to focus on what organizations need and what they can actually do if we want impact assessments to truly drive development and increase impact.

The growing concern for social impact assessment

The best impact assessment method depends on their specific evaluation needs, and these needs vary depending on where they are in the innovation life cycle. They often fall into two categories: needs to “improve” operations to enhance impact and needs to “prove” their impact to attract external stakeholders. The relative importance of proving vs improving shifts as enterprises progress through their innovation’s lifecycle. Competitive funding environments and performance-based management schemes have led to a large array of new assessment methods, with varying costs and scope of proof: outcome-based metrics, before/after comparisons, experimental methods, ethnographic thick descriptions… Assessing social impact however remains highly challenging. Assessment must deal with scarce data and difficult-to-measure effects, such as self-respect, freedom, or quality of life. Causal relationships between the impact and the intervention are often difficult to accurately trace and translate through the entire chain of cause and effect. Further, funders seek comparability across assessments; but comparing the effect of different innovations remains difficult, especially when innovations improve lives in very different ways and in different contexts. Social innovators also find that different assessments unevenly support internal (staff, beneficiaries…) and external (funders, regulators…) stakeholders with different priorities.

The experimentation stage: an iterative process

The social innovation life cycle starts with the experimentation stage. During this stage, resources dedicated to the project are scarce. The need to prove is low, while the need to improve is high. Innovators iteratively design their solution and attempt to get feedback. Their main focus is to understand the needs of beneficiaries and improve their design to best address them. This is often done with the help of in-depth qualitative methods (interviews, ethnographic observations), which provide “thick” descriptions of the lived experiences of the beneficiaries These methods are relatively easier and less costly to do. They enable iteration and provide actionable levers. They also allow the creation of detailed “theories of change” and “logic models” that can map how their enterprise’s activities will lead to social impact. An organization attempting to tackle homelessness may, for example, map out the key factors that cause homelessness and then create a chain of steps that would lead to alleviating each factor (called logic models). An innovation that improves only one of the causes of homelessness – a telehealth app, for example – can then track their progress on each of the steps in the specific logic model on how their innovation alleviates the specific underlying cause they are targeting. This way they can demonstrate social performance through progress on the steps in their chains, even if their innovation won’t singlehandedly end homelessness.  On the flip side, they cannot allow comparability across heterogeneous innovations, and don’t give certitude as to whether the expected social changes observed will actually result.

The business model stage: improving and proving the pertinence of the innovation

As an innovation enters the next lifecycle stage – the business model stage – innovators need to mobilize financial and political resources. The need to prove the potential of the innovation to external stakeholders becomes prominent. As more resources come in, a new array of tools start to come in handy: crafting “indicators” and “scorecards” to monitor and optimize their activities.

Outputs and Outcomes measured by performance monitoring can be used as proxies for impact measurement. Take solar lanterns, which can be used to replace polluting, hazardous kerosene lamps in off-grid rural areas in Sub Saharan Africa. The number of solar lamps distributed can be used as a proxy for impact, given the direct link between the use of the lamp and its impact (improved quality of life, decreased indoor air pollution). While “key performance indicators” reduce impact to simplified metrics that cannot fully capture the subtleties of the change, they can be useful tools to monitor and improve operations, as well as to report a sense of impact to the first external funders.

Scaling up impact: taking the innovation to the next level

As the innovation matures and reaches the scaling stage, it is time to provide rigorous proof of impact to funders, regulators and policymakers. As institutional funders, impact investors, and philanthropists get involved, they can bring in expertise and resources which make sophisticated impact assessments more accessible.

Among them, monetary measures translate social value into monetary terms. This sets the ground for comparability and considerations of cost-efficiency. These measures include Trucost Environmental Impact Metrics, Avoided Cost Methods, and Social Return on Investment (SROI). SROI measures compare the social impact generated to the investment costs required to launch the innovation. One Acre Fund, a nonprofit which works with smallholders in rural Africa, uses SROI to compare the additional monetary value of the crops their programs help farmers grow to sell and eat to the net costs of the program. The nonprofit has quickly found that this metric works best when included in scorecards which take other factors into account such as nutrition or soil health as well as scale. Monetary measures often raise a tricky question: how can a monetary value be attributed to social impact? Economists attempt to overcome this challenge by using proxies for the quantitative value of the innovation, such as the willingness to pay for an innovation, even though it is free. How much would people be willing to pay for free malaria bed nets? Once an economic value has been attributed, it can be included in a cost-benefit analysis. Avoided Cost Methods, might calculate repair, replacement, and substitution costs that are avoided when shifting course away from damaging status quo trajectories. For example, the malaria treatment costs avoided by providing bed nets.

Health-based methods, such as disability adjusted life years (DALYs), can also be useful. Through DALYs, health issues are converted into the number of years of life impaired or lost or due to sickness, disability or early death. So, if an enterprise successfully transitions rural areas from cooking over fire to cooking with natural gas, the extra years of healthy life and freedom from disability and burden that are gained by the woman and girls whose lungs no longer fill with smoke every day can be calculated. And the costs per extra DALY generated can be compared to the cost per DALY gained. This process allows funders and investors to direct their resources to places with the best cost/DALY.

These methods, unfortunately, can be difficult to calculate accurately and rely on best estimates of “what would have happened without the innovation” to calculate the actual impact. Only stringent experimental methods, such as RCTs can truly provide conclusive answers to the impact to a specific intervention. Yet, their outreach is limited by their significant cost, technical demands, and timescale. The scaling and mainstream lifecycle stages provide opportunities to implement them, but often only with external help to start.

Few organizations reach the mainstream stage. For those which do, their focus shifts to maintaining the innovation disruptive and making it adopted by others. It is then that organizations are most likely to choose methods that really suit their needs: they are no longer constrained by limited resources and legitimacy concerns.

A new way to look at impact assessment?

What does this lifecycle analysis teach us about assessing the impact of social innovations? Mainly that social innovators need to move beyond the mere question of “how to assess my social impact” to instead ask: “What do we want to do with social impact assessment?” Funders and investors have a key role in this process. First, they can help by aligning their impact assessment demands with other funders’ so that reporting requirements do not balloon with each new stakeholder brought aboard. Second, they can understand which measures are appropriate at different phases of the innovation lifecycle. Third, they can encourage impact assessments that focus on innovators’ needs to learn and improve over funders and investors’ need to prove and compare impact – especially during earlier stages in an innovation’s lifecycle. And finally, they can realize that social impact assessment requires specialized knowledge and resource that most innovators are unlikely to have until the very height of their innovation’s lifecycles, if ever. It may be wiser for funders and investors to provide the social impact assessments as a form of additional support they can deploy across their portfolios rather than having each innovator “reinvent the wheel” of social impact proofs. This keeps each innovation’s resources and attention where both the innovators and their stakeholders want it – on improving and scaling their impact.

Michaela Neumayr & Astrid Pennerstorfer, WU Vienna University of Economics and Business.

What amount of donations can you expect from people when fundraising for your organization? Who are the people who do not donate any money at all, and who are those who donate substantial amounts? Basically, charitable donations are considered to strongly vary across income classes, and fundraisers know that the amounts of individuals’ donations increase with their financial resources. But what proportion of income can you expect to be donated by people of different income groups?  Which income groups are more ‘generous’, those with higher or lower incomes?

Such questions have been discussed in scholarly research since the 1990s and answers to these questions are surprisingly diverse and even contradictory. In the debate concerning the “charitable giving profile” all kinds of shapes have been proposed. Some argue that it follows a U-shaped curve, with individuals at both ends of the income distribution donating the highest proportions of their income. For the US, for instance, those below an annual income of 10,000 USD donate about 4.6% of their income, while those with an income higher than 150,000 USD give 2.2%, and those in the middle 1.4%. Other studies do not find that lower-income groups are more generous, but rather describe the charitable-giving profile as a flat curve with an upward slope for higher-income groups. Yet other studies doubt the upwards swing on the right-hand side and instead find higher relative donation levels for lower income groups. We could go on with an account of yet other assumed shapes of previous studies, but we stop at this point to tell you about our study.

Our study The Relation Between Income and Donations as a Proportion of Income Revisited: Literature Review and Empirical Application consists of two parts. First, we systematically review 26 empirical studies that investigate the relation between income and proportion of income given, trying to make sense of the variety of existing results. In this review, one important explanation for diverging findings is the great number of varying methodological approaches in the previous studies. Second, we use the findings of this systematic review for an empirical application on Austrian income tax data (n = 20,000). In this second part, we are able to demonstrate how the various methods that have been used in previous studies lead to diverging findings. Based on that, we are able to tell which of the previous findings can be considered certain.

Returning to the first part of our study, we find three major reasons why results of previous studies vary so much:

– The first refers to the sample analysed. When describing the charitable giving profile it is crucial whether specific parts of the population are included in the sample (e.g. non-donors) or represented by a group that is large enough to draw unambiguous conclusions (e.g. high-income groups). Additionally, we find that both representative survey data and administrative tax data have their advantages and flaws for use in analysis of the charitable giving profile.  

– The second refers to the specification of variables. Previous studies have used different income and donation measures and this variation can also influence the result.

– The third refers to the method of analysis. The method how the charitable giving shape was determined varied greatly in previous studies ranging from mere visual inspection of donation/income ratios for different income classes, over bivariate to multivariate analyses.

In the second part, we take a sample of Austrian income tax data set to show how results regarding the charitable giving profile change when you vary the sample, define the variables differently or vary the statistical method. We find a visual inspection of the relation between income and the proportion of income donated to be especially unsatisfactory because results vary greatly depending on income class definitions and because visual inspection leaves a lot of room for interpretation. Results of various regression analyses point towards a U-shaped charitable giving profile. However, this is not what we find in the results when applying a more precise estimation technique (semi-parametric estimation), which displayed a downward-sloping relation for most income levels. We find this final method especially helpful for this question, as it allows to exactly describing the relationship between the income and proportion donated without any restrictions.

What can fundraisers take away from this for their practice? While our study has shown that results of previous analysis vary with the method in use, three conclusions regarding the relation between income and the proportion donated are safe to say. First, people in the lowest income group donate the largest proportion of income. While the share itself varies depending on the country context, it is always higher compared to other income groups within the respective country. Second, the profile of the charitable giving curve seems to be fairly flat for middle and high income groups. This means, middle and high income groups can be expected to give the same share of their income. Third, regarding the very high income groups the relation is more difficult to describe Important to note, this very high income group refers to the top 5-3% of the income profile only. Within this group, fundraisers still have to find out for themselves what amounts can be achieved, while for all other groups it is known statistically what proportions and thus amounts can be expected.

Click here to access the full NVSQ research article: Neumayr, M. & Pennerstorfe, A. (2020). The Relation Between Income and Donations as a Proportion of Income Revisited: Literature Review and Empirical Application, Nonprofit & Voluntary Sector Quarterly, https://doi.org/10.1177/0899764020977667.

Bryant Crubaugh, Pepperdine University, Malibu.

Neighborhood equity requires more than the inclusion of nonprofits. How is the relationship between neighborhood development organizations and neighborhood disadvantage dependent on race, resources, and residential mobility? The answer to this question is vital as cities attempt to correct structural inequalities and rely on nonprofit organizations in their plans.

For example, Chicago’s mayor, Lori Lightfoot, is attempting to tackle two persistent, decades-old issues within Chicago: violence and concentrated poverty. Throughout her first two years in office, nonprofit leaders have been meeting with and standing at her side. In February, before the onset of Covid-19 in the US, Chicago’s city leaders and nonprofit organizational representatives met to discuss plans to reinvest in long divested corridors in the West and South sides of the city, hoping to spur job growth and help long-isolated communities escape concentrated poverty. More recently, as Chicago has seen a continued increase in gun violence, Lightfoot has responded to calls with a formalized plan to lean on nonprofit organizations to help reduce violence, though the plan so far has not included significant steps away from traditional policing.

Whenever a major initiative makes its way to a mayor’s desk in the US, community nonprofits are likely to have had a role in getting it there and likely have a role in implementing the solution. Nonprofits are routinely lauded for taking local knowledge into account, for supporting city governments in times of financial strain, and for helping bring people together to shape their efforts. However, community nonprofits do not universally and evenly serve the entire city. When they are tasked with administering social programs instead of the government, they often run at a deficit and lack the funding needed to adequately administer social programs.

It is clear that nonprofit organizations have a large role to play in city administration, but I am left asking: does this role help alleviate inequalities or does it perpetuate them?

In my analyses of Chicago from 1990-2010, I ask how neighborhood development organizations (NDOs) are associated with varying levels of neighborhood disadvantage—a composite measure of impoverished families, rates of unemployment, families receiving public assistance, single-mother households, and vacant units. Traditional social research and popular accounts of this process would lead us to expect that nonprofit organizations are likely to lower disadvantage through processes of developing community trust, increasing neighborly care, and connecting neighbors to external resources. In general, I find that NDOs are associated with decreasing levels of neighborhood disadvantage. But this general pattern does not fit all neighborhoods.

In predominately Black neighborhoods, NDOs do not have this same association. Instead, having more NDOs in predominately Black neighborhoods is associated with increasing disadvantage. This is not a story of predominately Black neighborhoods lacking NDOs. Predominately Black neighborhoods have more NDOs on average than predominately non-Black neighborhoods, by a rate of nearly 2 to 1, as seen in the figure below. Yet when it comes to resources given to NDOs, from any public or private source, we can see that predominately Black neighborhoods are being left out by a rate of over 6 to 1. My results show that when it comes to reducing neighborhood disadvantage, having over 500,000 dollars of NDO income in a census tract in a year is associated with decreasing neighborhood disadvantage—approximately one fifth of a standard deviation of neighborhood disadvantage. Unequal and inequitable funding of Black NDOs is stunting their opportunity to address neighborhood disadvantages.

Residential mobility and potential gentrification also impact the strength of the relationship between NDOs and neighborhood disadvantage. NDOs may encourage gentrification through their focus on beautification, historical preservation, and economic redevelopment, all factors that may disrupt the social fabric of a neighborhood. These processes are not random but are also dependent on the racial makeup of neighborhoods, with predominately Black neighborhoods being more likely to lack external investments and remain in a disinvested state. In the figure below, I show how neighborhoods with significant residential mobility, NDOs are associated with declines in neighborhood disadvantage. This suggests that as neighbors are moving out in neighborhoods with more NDOs, there is a replacement of disadvantaged neighbors with advantaged ones.

So, why does this matter? In a moment when Mayor Lightfoot and mayors across the United States are attempting to tackle large issues such as concentrated poverty by relying on nonprofit organizations, these findings should give us pause.

Given these associations between NDOs and neighborhood disadvantage, more of which I discuss in my article, I see two paths forward. First, when mayors propose the further use of nonprofit organizations like NDOs for neighborhood development or social service administration, funding needs to be equitable. Black communities in Chicago are organized and ready to tackle the large issues that have isolated and disadvantaged many of their neighborhoods, but without equitable funding that gives a disproportionate share of funds from these new programs to those that need it most, resource gaps will still be present and no amount of neighborhood organizing can overcome that disadvantage compared to their neighboring communities.

Second, this should make us question programs and policies that rely on nonprofits for implementation altogether. Given that social service provision often operates from a deficit when administered through nonprofits and the results here, nonprofit administration of neighborhood development and other social programs is likely to reproduce the inequalities already present. In attempting to tackle the large structural inequalities of the day, Mayor Lightfoot and others need to address the roots of the problem that even equitable funding may not address. Without dismantling the structural conditions that produced concentrated disadvantage and violence in the first place, nonprofits are unlikely to be able to make a substantial and structural impact.

Click here for the original NVSQ article – free access to the full text until the end of May 2021!