For a pragmatic approach to social impact assessment

Authors: Anne-Claire Pache1 & Greg Molecke2 Contributor: Eléonore Delanoë1

1ESSEC Business School, 2University of Exeter Business School

The Nobel prize in economics awarded to Esther Duflo, Abhijit Banerjee and Michael Kremer in December 2019 has consecrated their game-changing work against poverty. At the heart of their work are experimental approaches using Randomized Controlled Trials (RCTs), which have shed new light on the way the impact of social innovations can be assessed. RCTs compare the impact of a measure between a treatment group and a control group whose participants are selected at random. They are a powerful way to remove biases and isolate a specific action from the great swirl of other factors that may affect the result. However, they are far from being a “one-size-fits-all” approach because they are complex to set up and impose significant technical and financial demands on the organization. They also frequently require long timeframes to set up and run – running into years and decades – making them poor tools to help businesses and investors execute short- to medium-term strategies. RCTs work well to establish causal links between a given intervention and social impact. However, in many instances, the impact evaluation needs for innovators and their supports are quite different – with much more need for tools that can guide performance improvements rather than prove outcomes. The latest research by Anne-Claire Pache and Greg Molecke for the Handbook of Inclusive Innovation suggests that these needs vary based on where social innovators stand in the innovation cycle. We need to focus on what organizations need and what they can actually do if we want impact assessments to truly drive development and increase impact.

The growing concern for social impact assessment

The best impact assessment method depends on their specific evaluation needs, and these needs vary depending on where they are in the innovation life cycle. They often fall into two categories: needs to “improve” operations to enhance impact and needs to “prove” their impact to attract external stakeholders. The relative importance of proving vs improving shifts as enterprises progress through their innovation’s lifecycle. Competitive funding environments and performance-based management schemes have led to a large array of new assessment methods, with varying costs and scope of proof: outcome-based metrics, before/after comparisons, experimental methods, ethnographic thick descriptions… Assessing social impact however remains highly challenging. Assessment must deal with scarce data and difficult-to-measure effects, such as self-respect, freedom, or quality of life. Causal relationships between the impact and the intervention are often difficult to accurately trace and translate through the entire chain of cause and effect. Further, funders seek comparability across assessments; but comparing the effect of different innovations remains difficult, especially when innovations improve lives in very different ways and in different contexts. Social innovators also find that different assessments unevenly support internal (staff, beneficiaries…) and external (funders, regulators…) stakeholders with different priorities.

The experimentation stage: an iterative process

The social innovation life cycle starts with the experimentation stage. During this stage, resources dedicated to the project are scarce. The need to prove is low, while the need to improve is high. Innovators iteratively design their solution and attempt to get feedback. Their main focus is to understand the needs of beneficiaries and improve their design to best address them. This is often done with the help of in-depth qualitative methods (interviews, ethnographic observations), which provide “thick” descriptions of the lived experiences of the beneficiaries These methods are relatively easier and less costly to do. They enable iteration and provide actionable levers. They also allow the creation of detailed “theories of change” and “logic models” that can map how their enterprise’s activities will lead to social impact. An organization attempting to tackle homelessness may, for example, map out the key factors that cause homelessness and then create a chain of steps that would lead to alleviating each factor (called logic models). An innovation that improves only one of the causes of homelessness – a telehealth app, for example – can then track their progress on each of the steps in the specific logic model on how their innovation alleviates the specific underlying cause they are targeting. This way they can demonstrate social performance through progress on the steps in their chains, even if their innovation won’t singlehandedly end homelessness.  On the flip side, they cannot allow comparability across heterogeneous innovations, and don’t give certitude as to whether the expected social changes observed will actually result.

The business model stage: improving and proving the pertinence of the innovation

As an innovation enters the next lifecycle stage – the business model stage – innovators need to mobilize financial and political resources. The need to prove the potential of the innovation to external stakeholders becomes prominent. As more resources come in, a new array of tools start to come in handy: crafting “indicators” and “scorecards” to monitor and optimize their activities.

Outputs and Outcomes measured by performance monitoring can be used as proxies for impact measurement. Take solar lanterns, which can be used to replace polluting, hazardous kerosene lamps in off-grid rural areas in Sub Saharan Africa. The number of solar lamps distributed can be used as a proxy for impact, given the direct link between the use of the lamp and its impact (improved quality of life, decreased indoor air pollution). While “key performance indicators” reduce impact to simplified metrics that cannot fully capture the subtleties of the change, they can be useful tools to monitor and improve operations, as well as to report a sense of impact to the first external funders.

Scaling up impact: taking the innovation to the next level

As the innovation matures and reaches the scaling stage, it is time to provide rigorous proof of impact to funders, regulators and policymakers. As institutional funders, impact investors, and philanthropists get involved, they can bring in expertise and resources which make sophisticated impact assessments more accessible.

Among them, monetary measures translate social value into monetary terms. This sets the ground for comparability and considerations of cost-efficiency. These measures include Trucost Environmental Impact Metrics, Avoided Cost Methods, and Social Return on Investment (SROI). SROI measures compare the social impact generated to the investment costs required to launch the innovation. One Acre Fund, a nonprofit which works with smallholders in rural Africa, uses SROI to compare the additional monetary value of the crops their programs help farmers grow to sell and eat to the net costs of the program. The nonprofit has quickly found that this metric works best when included in scorecards which take other factors into account such as nutrition or soil health as well as scale. Monetary measures often raise a tricky question: how can a monetary value be attributed to social impact? Economists attempt to overcome this challenge by using proxies for the quantitative value of the innovation, such as the willingness to pay for an innovation, even though it is free. How much would people be willing to pay for free malaria bed nets? Once an economic value has been attributed, it can be included in a cost-benefit analysis. Avoided Cost Methods, might calculate repair, replacement, and substitution costs that are avoided when shifting course away from damaging status quo trajectories. For example, the malaria treatment costs avoided by providing bed nets.

Health-based methods, such as disability adjusted life years (DALYs), can also be useful. Through DALYs, health issues are converted into the number of years of life impaired or lost or due to sickness, disability or early death. So, if an enterprise successfully transitions rural areas from cooking over fire to cooking with natural gas, the extra years of healthy life and freedom from disability and burden that are gained by the woman and girls whose lungs no longer fill with smoke every day can be calculated. And the costs per extra DALY generated can be compared to the cost per DALY gained. This process allows funders and investors to direct their resources to places with the best cost/DALY.

These methods, unfortunately, can be difficult to calculate accurately and rely on best estimates of “what would have happened without the innovation” to calculate the actual impact. Only stringent experimental methods, such as RCTs can truly provide conclusive answers to the impact to a specific intervention. Yet, their outreach is limited by their significant cost, technical demands, and timescale. The scaling and mainstream lifecycle stages provide opportunities to implement them, but often only with external help to start.

Few organizations reach the mainstream stage. For those which do, their focus shifts to maintaining the innovation disruptive and making it adopted by others. It is then that organizations are most likely to choose methods that really suit their needs: they are no longer constrained by limited resources and legitimacy concerns.

A new way to look at impact assessment?

What does this lifecycle analysis teach us about assessing the impact of social innovations? Mainly that social innovators need to move beyond the mere question of “how to assess my social impact” to instead ask: “What do we want to do with social impact assessment?” Funders and investors have a key role in this process. First, they can help by aligning their impact assessment demands with other funders’ so that reporting requirements do not balloon with each new stakeholder brought aboard. Second, they can understand which measures are appropriate at different phases of the innovation lifecycle. Third, they can encourage impact assessments that focus on innovators’ needs to learn and improve over funders and investors’ need to prove and compare impact – especially during earlier stages in an innovation’s lifecycle. And finally, they can realize that social impact assessment requires specialized knowledge and resource that most innovators are unlikely to have until the very height of their innovation’s lifecycles, if ever. It may be wiser for funders and investors to provide the social impact assessments as a form of additional support they can deploy across their portfolios rather than having each innovator “reinvent the wheel” of social impact proofs. This keeps each innovation’s resources and attention where both the innovators and their stakeholders want it – on improving and scaling their impact.

How much of their income do people usually donate to charity?

Photo by Pixabay on Pexels.com

Michaela Neumayr & Astrid Pennerstorfer, WU Vienna University of Economics and Business.

What amount of donations can you expect from people when fundraising for your organization? Who are the people who do not donate any money at all, and who are those who donate substantial amounts? Basically, charitable donations are considered to strongly vary across income classes, and fundraisers know that the amounts of individuals’ donations increase with their financial resources. But what proportion of income can you expect to be donated by people of different income groups?  Which income groups are more ‘generous’, those with higher or lower incomes?

Such questions have been discussed in scholarly research since the 1990s and answers to these questions are surprisingly diverse and even contradictory. In the debate concerning the “charitable giving profile” all kinds of shapes have been proposed. Some argue that it follows a U-shaped curve, with individuals at both ends of the income distribution donating the highest proportions of their income. For the US, for instance, those below an annual income of 10,000 USD donate about 4.6% of their income, while those with an income higher than 150,000 USD give 2.2%, and those in the middle 1.4%. Other studies do not find that lower-income groups are more generous, but rather describe the charitable-giving profile as a flat curve with an upward slope for higher-income groups. Yet other studies doubt the upwards swing on the right-hand side and instead find higher relative donation levels for lower income groups. We could go on with an account of yet other assumed shapes of previous studies, but we stop at this point to tell you about our study.

Our study The Relation Between Income and Donations as a Proportion of Income Revisited: Literature Review and Empirical Application consists of two parts. First, we systematically review 26 empirical studies that investigate the relation between income and proportion of income given, trying to make sense of the variety of existing results. In this review, one important explanation for diverging findings is the great number of varying methodological approaches in the previous studies. Second, we use the findings of this systematic review for an empirical application on Austrian income tax data (n = 20,000). In this second part, we are able to demonstrate how the various methods that have been used in previous studies lead to diverging findings. Based on that, we are able to tell which of the previous findings can be considered certain.

Returning to the first part of our study, we find three major reasons why results of previous studies vary so much:

– The first refers to the sample analysed. When describing the charitable giving profile it is crucial whether specific parts of the population are included in the sample (e.g. non-donors) or represented by a group that is large enough to draw unambiguous conclusions (e.g. high-income groups). Additionally, we find that both representative survey data and administrative tax data have their advantages and flaws for use in analysis of the charitable giving profile.  

– The second refers to the specification of variables. Previous studies have used different income and donation measures and this variation can also influence the result.

– The third refers to the method of analysis. The method how the charitable giving shape was determined varied greatly in previous studies ranging from mere visual inspection of donation/income ratios for different income classes, over bivariate to multivariate analyses.

In the second part, we take a sample of Austrian income tax data set to show how results regarding the charitable giving profile change when you vary the sample, define the variables differently or vary the statistical method. We find a visual inspection of the relation between income and the proportion of income donated to be especially unsatisfactory because results vary greatly depending on income class definitions and because visual inspection leaves a lot of room for interpretation. Results of various regression analyses point towards a U-shaped charitable giving profile. However, this is not what we find in the results when applying a more precise estimation technique (semi-parametric estimation), which displayed a downward-sloping relation for most income levels. We find this final method especially helpful for this question, as it allows to exactly describing the relationship between the income and proportion donated without any restrictions.

What can fundraisers take away from this for their practice? While our study has shown that results of previous analysis vary with the method in use, three conclusions regarding the relation between income and the proportion donated are safe to say. First, people in the lowest income group donate the largest proportion of income. While the share itself varies depending on the country context, it is always higher compared to other income groups within the respective country. Second, the profile of the charitable giving curve seems to be fairly flat for middle and high income groups. This means, middle and high income groups can be expected to give the same share of their income. Third, regarding the very high income groups the relation is more difficult to describe Important to note, this very high income group refers to the top 5-3% of the income profile only. Within this group, fundraisers still have to find out for themselves what amounts can be achieved, while for all other groups it is known statistically what proportions and thus amounts can be expected.

Click here to access the full NVSQ research article: Neumayr, M. & Pennerstorfe, A. (2020). The Relation Between Income and Donations as a Proportion of Income Revisited: Literature Review and Empirical Application, Nonprofit & Voluntary Sector Quarterly, https://doi.org/10.1177/0899764020977667.

Neighborhood Development Organizations – Tackling Neighborhood Disadvantage at the Intersection of Race and Resource Inequity

Photo by Chait Goli on Pexels.com

Bryant Crubaugh, Pepperdine University, Malibu.

Neighborhood equity requires more than the inclusion of nonprofits. How is the relationship between neighborhood development organizations and neighborhood disadvantage dependent on race, resources, and residential mobility? The answer to this question is vital as cities attempt to correct structural inequalities and rely on nonprofit organizations in their plans.

For example, Chicago’s mayor, Lori Lightfoot, is attempting to tackle two persistent, decades-old issues within Chicago: violence and concentrated poverty. Throughout her first two years in office, nonprofit leaders have been meeting with and standing at her side. In February, before the onset of Covid-19 in the US, Chicago’s city leaders and nonprofit organizational representatives met to discuss plans to reinvest in long divested corridors in the West and South sides of the city, hoping to spur job growth and help long-isolated communities escape concentrated poverty. More recently, as Chicago has seen a continued increase in gun violence, Lightfoot has responded to calls with a formalized plan to lean on nonprofit organizations to help reduce violence, though the plan so far has not included significant steps away from traditional policing.

Whenever a major initiative makes its way to a mayor’s desk in the US, community nonprofits are likely to have had a role in getting it there and likely have a role in implementing the solution. Nonprofits are routinely lauded for taking local knowledge into account, for supporting city governments in times of financial strain, and for helping bring people together to shape their efforts. However, community nonprofits do not universally and evenly serve the entire city. When they are tasked with administering social programs instead of the government, they often run at a deficit and lack the funding needed to adequately administer social programs.

It is clear that nonprofit organizations have a large role to play in city administration, but I am left asking: does this role help alleviate inequalities or does it perpetuate them?

In my analyses of Chicago from 1990-2010, I ask how neighborhood development organizations (NDOs) are associated with varying levels of neighborhood disadvantage—a composite measure of impoverished families, rates of unemployment, families receiving public assistance, single-mother households, and vacant units. Traditional social research and popular accounts of this process would lead us to expect that nonprofit organizations are likely to lower disadvantage through processes of developing community trust, increasing neighborly care, and connecting neighbors to external resources. In general, I find that NDOs are associated with decreasing levels of neighborhood disadvantage. But this general pattern does not fit all neighborhoods.

In predominately Black neighborhoods, NDOs do not have this same association. Instead, having more NDOs in predominately Black neighborhoods is associated with increasing disadvantage. This is not a story of predominately Black neighborhoods lacking NDOs. Predominately Black neighborhoods have more NDOs on average than predominately non-Black neighborhoods, by a rate of nearly 2 to 1, as seen in the figure below. Yet when it comes to resources given to NDOs, from any public or private source, we can see that predominately Black neighborhoods are being left out by a rate of over 6 to 1. My results show that when it comes to reducing neighborhood disadvantage, having over 500,000 dollars of NDO income in a census tract in a year is associated with decreasing neighborhood disadvantage—approximately one fifth of a standard deviation of neighborhood disadvantage. Unequal and inequitable funding of Black NDOs is stunting their opportunity to address neighborhood disadvantages.

This image has an empty alt attribute; its file name is image.png

Residential mobility and potential gentrification also impact the strength of the relationship between NDOs and neighborhood disadvantage. NDOs may encourage gentrification through their focus on beautification, historical preservation, and economic redevelopment, all factors that may disrupt the social fabric of a neighborhood. These processes are not random but are also dependent on the racial makeup of neighborhoods, with predominately Black neighborhoods being more likely to lack external investments and remain in a disinvested state. In the figure below, I show how neighborhoods with significant residential mobility, NDOs are associated with declines in neighborhood disadvantage. This suggests that as neighbors are moving out in neighborhoods with more NDOs, there is a replacement of disadvantaged neighbors with advantaged ones.

So, why does this matter? In a moment when Mayor Lightfoot and mayors across the United States are attempting to tackle large issues such as concentrated poverty by relying on nonprofit organizations, these findings should give us pause.

Given these associations between NDOs and neighborhood disadvantage, more of which I discuss in my article, I see two paths forward. First, when mayors propose the further use of nonprofit organizations like NDOs for neighborhood development or social service administration, funding needs to be equitable. Black communities in Chicago are organized and ready to tackle the large issues that have isolated and disadvantaged many of their neighborhoods, but without equitable funding that gives a disproportionate share of funds from these new programs to those that need it most, resource gaps will still be present and no amount of neighborhood organizing can overcome that disadvantage compared to their neighboring communities.

Second, this should make us question programs and policies that rely on nonprofits for implementation altogether. Given that social service provision often operates from a deficit when administered through nonprofits and the results here, nonprofit administration of neighborhood development and other social programs is likely to reproduce the inequalities already present. In attempting to tackle the large structural inequalities of the day, Mayor Lightfoot and others need to address the roots of the problem that even equitable funding may not address. Without dismantling the structural conditions that produced concentrated disadvantage and violence in the first place, nonprofits are unlikely to be able to make a substantial and structural impact.

Click here for the original NVSQ article – free access to the full text until the end of May 2021!

Generating satisfying volunteer experiences: How to design National Days of Service volunteer projects

Photo by Min An on Pexels.com

Stephanie Maas1,2,Lucas Meijs1, Jeffrey Brudney3

1Erasmus University Rotterdam, The Netherlands, 2Vrije Universiteit Amsterdam, The Netherlands, 3University of North Carolina Wilmington, USA.

The way in which people volunteer is changing. An increasing trend is event-based, short-term activities, such as National Day of Service (NDS) volunteering events, rather than long-term, traditional or ongoing volunteer commitments. NDS events are common across the globe, for example, 9/11 Day, Make a Difference Day and Martin Luther King Jr. Day in the United States, Sewa Day in 25 different countries, Mandela Day in South Africa, and NLdoet in the Netherlands – just to name a few. All of these examples are nationwide volunteering events in which individuals and groups support nonprofit organizations by contributing their time to one-day service projects. In NDS events volunteers may cook for the elderly, maintain buildings, gardens and playgrounds, support a fun afternoon for people with disabilities, and so forth.

These events mobilize large numbers of people to engage in volunteer service, creating enormous amounts of donated labor to help communities. But organizers of NDS events also intend to enhance the profile and create an ethic of volunteering. Intentions to continue volunteering typically depend on volunteer satisfaction, and volunteer managers are interested in achieving retention. This goal is not easy, due to the limited contact time between volunteer and volunteer management; moreover, general volunteer management practices used for long-term volunteer commitments might not work. Nevertheless, nonprofits can plan, structure, and organize NDS projects far in advance to enhance satisfaction of volunteer participants. So, how to design a NDS volunteer project to promote volunteer satisfaction?

To answer this question, we conducted semi-structured interviews, participant observations, and focus groups in the context of a NDS event in the Netherlands: NLdoet. Data emanate from nonprofit organizations, volunteer centers, participant observers and volunteers involved in NLdoet.

The answer: Host nonprofit organizations can elicit volunteer job satisfaction by designing NDS projects that create 1) a sense of added value, 2) support productivity, and 3) make volunteers feel comfortable. You now might ask: how to create this sense of added value, support productivity and make volunteers feel comfortable within a NDS?

First, simply put, to create a sense of added value NDS volunteers must feel that they add value and have meaningful experiences: Organize a volunteer activity that is meaningful, has real impact and affects beneficiaries, the nonprofit and/or the community at large. Moreover, the volunteer activity should be a suitable and logical choice as a NDS project. An organization should pick or create a NDS activity that shows importance, urgency and necessity — something that cannot be done regular staff and volunteers. One example is repairing the fence of a Petting Zoo, especially when it is clear that the regular volunteers are already overloaded with keeping the Petting Zoo open; or creating a wall-painting in a residential care environment, based on the understanding that the organization lacks the budget to do so professionally.

Additionally, to create greater impact, NDS volunteers should be provided the opportunity to interact with beneficiaries or clients. In addition, providing feedback to the one-day volunteers – either by nonprofit staff, regular volunteers and/or the beneficiaries themselves — and recognizing their efforts also create a sense of added value. Tell them how the organization will profit from the work done at the NDS! And of course, give them a pat on the back, a thank you or other small momentos.

Second, to support a sense of productivity toward NDS volunteers, the chosen one-day service project should have a clear beginning and end with a visible result. Most importantly, it should be completed at the end of a long day. When the volunteer project cannot be completed, or the result of volunteer efforts is unnoticeable the next day, a lack of fulfillment can surface. Necessary equipment and tools should be available and working properly, as operational problems may frustrate volunteers. Likewise, host nonprofits should make decisions about structuring NDS projects in advance, to ensure that NDS volunteers start immediately and can focus on carrying out the work at hand. Interestingly, a satisfying NDS project should feel a bit heroic; volunteers will welcome some muscle strain in the evening or the next day after completing a project that could only be completed by working hard.

Lastly, how to make volunteers feel comfortable? NDS volunteers typically are not engaged with and are largely uninformed about the host nonprofit organization. Host nonprofits can make volunteers feel comfortable by providing sufficient social support and having sufficient staff members or regular volunteers present at the NDS to assist and answer questions. Furthermore, limiting volunteers’ autonomy and responsibility over the NDS project and interaction with beneficiaries also makes NDS volunteers feel more comfortable. Compared to ongoing volunteers who have background and experience at the host nonprofit, NDS volunteers prefer that the organization exercise greater direction over their work.

When both volunteers as well as the host organization look back on the NDS with satisfaction, the event is definitely a success. The above are only a few suggestions on how to design a NDS volunteer project to elicit satisfying volunteer experiences. If you want to know more about how to design a one-day service project to promote volunteer satisfaction, please check out our recently published full paper here.

Original open access NVSQ article: Maas, S.A., Meijs, L.C.P.M. & Brudney, J.L. (2020). ‘Designing “National Day of Service” Projects to Promote Volunteer Job Satisfaction’. Nonprofit & Voluntary Sector Quarterly. DOI: 10.1177/0899764020982664.