For Whom Do We Learn?
AmbitioUS is an initiative of the Center for Cultural Innovation to act as an experimental arm of the artist-support sector. Throughout the initiative, Angie Kim, President and CEO, will share her observations and reflections related to this cross-sector, experimental, and systems level work.
The distribution of wealth depends, not wholly, indeed, but largely, on a [society’s] institutions; and the character of [a society’s] institutions is determined, not by immutable economic laws, but by the values, preferences, interests and ideals which rule at any moment in a given society. –Richard Henry Tawney, Equality
In the spirit of not going back to the way things were pre-COVID, I am turning my attention to institutional foundations’ evaluation practices. Given the growing number of funders ramping up their equity work, this is a critical moment to think differently about the purposes, practices, and principles of foundations’ use of evaluation: (a) the claim on validity, (b) its alleged relevancy to the causes and people we serve, and (c) its use as an inculcating force discriminating against different cultural worldviews.
Before I joined the Center for Cultural Innovation (CCI), I worked as an independent consultant to foundations intersecting two of my passions–combining program design with evaluation. I synthesized grantmaking aims to fit into theories of change and logic models. I still love applying these skills, and there is something immensely satisfying in being able to see colorful lines connect foundations’ actions with measurable social change. These tools are useful under certain applications, such as when funders are trying to uphold or repair already existing systems, such as expanding healthy food access into communities facing food apartheid. In some cases, theories of change and logic models provide a guidebook for considering all sorts of factors, from the type of companies to introduce, the types of goods to sell, and to their purchase orders and employment practices that create a circuitous economy wherein residents will be able to afford this social intervention.
But many of today’s foundations have become more interested in equity, justice, reparations, and liberation–all of which means they are interested in funding systems change as these terms assume that harms are built into existing systems. This shift marks a new moment in the trajectory of institutional philanthropy, from responding to apparent needs to addressing upstream system failures. Unfortunately, if evaluation processes and tools are applied to systems change without recalibration, these tools actually put blinkers on funders, narrowing their vision to only the factors listed, and prevent them from being able to act nimbly to ever-changing political, economic, legal, environmental, and cultural shifts and opportunities.
The Human Error Component
In 2005, Dr. John P. A. Ioannidis published “Why Most Published Research Findings Are False.” Rather than sparking controversy, his research was so widely accepted to become a cult classic and remains one of the most cited. This paper, and his many subsequent ones, such as this one with a similarly reproving title, “The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses” (2016), allowed medical and social scientists, clinicians, statisticians, translational researchers, among others to openly acknowledge an endemic problem (Freedman, 2010). There’s nothing wrong with science per se. The problem rests with humans: User errors based on personal gain, bias, and conflicts of interest lead to questionable and unreliable outcomes. It’s no wonder we are generally confused about whether or not to take baby aspirin, to apply heat or cold to sore muscles, or to exercise slowly or intensely.
While those in the sciences go through a reckoning process sparked by Dr. Ioannidis’s meta-analyses, the foundation sector has not caught up to questioning its own evaluation practices. Thus, they (a) crudely apply social science tools without regard for situation and context and (b) inhibit progress toward using diverse practices of generating and assessing knowledge (community-based practices, shared learning, shifting ownership of knowledge and data, etc.) that would make it possible to walk the talk of equity work. It is time to re-consider the foundation industry’s inherited practices of evaluation.
Shifting Author-ship/-ity
Besides questioning specious application of scientific methodologies as Dr. Ioannidis’s work points out, relevancy is also an important question: For whom do scientific approaches matter?
Consider the work of Dr. Kim Anderson-Erisman who is a 2020 recipient of the $1 million unrestricted Craig H. Neilsen Foundation’s Visionary Prize. This is clearly someone whose work merits attention. What makes her remarkable is that as a spinal cord injury (SCI) wheelchair user herself she advanced the notion that research on SCI should be relevant for those with SCI. As she succinctly described, the problem is that “the research community doesn’t always know what’s important to the spinal cord injury community” (2016).
Before Dr. Anderson-Erisman spoke up, SCI researchers focused on solving paralysis but largely ignored exploring topics that could actually help those living with SCI, such as bladder function, exercise treatments, pain management, sexual health, and well being. No doubt that it is far easier to attract funding and recognition by trying to solve paralysis than pursuing research areas more likely to yield quieter rewards. Dr. Anderson-Erisman not only advocated for the patients but, through her own translational research, she set a new bar for relevancy. We all benefit from being treated as humans with agency rather than subjected to the medical gaze. It’s also worth noting that her impact on turning researchers from theorists to allies came about by overcoming great odds. We don’t often see wheelchair users in medical research, or medicine at all, because their barriers are physical ones–from the height of lab tables to the accessibility of equipment.
Dr. Anderson-Erisman’s story raises two important considerations for institutional funders using evaluation tools. Her work underscores the importance of giving authority to those with the most at stake. In shifting SCI researchers’ purpose and practices from personal intellectual pursuit to being in service to those whose outcomes matter most, she has helped improve the quality of life for her SCI peers. In addition, she was influential despite great odds. How many structural barriers do foundations put up that make it difficult for the very people their funding is impacting to be influential? Hiring practices, cultural assumptions, institutions to institutions funding, and even headquartering offices outside of the communities meant to be impacted are all ways that foundations deny themselves the opportunity to be swayed by those whom they say they care about the most.
Removing barriers (like using third-party evaluation firms) so as to directly engage with those whose lives we are trying to impact will yield more meaningful knowledge and will help foundations act more like partners humanizing the lives of people instead of faceless institutions.
Conventional Evaluation Practices Uphold the Status Quo
It helps to remember that our conventional grantmaking practices are a construct that reflect a Western European cultural worldview. When foundations use social science techniques, we perpetuate Christian, more specifically Protestant, and, most specifically Calvinistic ideas. To understand this is to recognize the influence of the Russell Sage Foundation in shaping today’s private foundation practice to reflect Christian worldviews.
Margaret Olivia Sage, widow of railroad executive Russell Sage, established the Russell Sage Foundation in 1907 to attend to “the improvement of social and living conditions in the United States.” Sage was responding to a moment of deep poverty, waves of migrants, and an absence of a recognizable elitist culture and class willing to apply “elevated” and educated minds to address social problems. To that end, the Russell Sage Foundation is credited with introducing social scientific methods as the best way to fix society’s problems (Alchon, 1999). According to historian Guy Alchon, the foundation’s influence was greatly shaped by Mary van Kleek, the director of the foundation’s industrial studies program in the early twentieth century. She “insist[ed] upon the essentially spiritual affinities between Christian and social-science idealism” (Alchon, 1999, p.152). She brought to her work a background as a researcher studying female factory workers and child laborers. She was one of many, particularly women, modernist, reformist leaders who wanted to address social issues through academic expertise, facts over other forms of knowledge, and scientific managerial authority (Alchon, 1999, p.153). There were previous generations of women, like Jane Addams, who demonstrated what it was to be an “evangelical Christian as social scientist and social savior.” But it is van Kleek who is credited as the “fountainhead” for institutional philanthropic practices to be based on Christian ideals manifested in social science methods (Alchon, p. 153).
The reliance on scientific application to social problems stemmed from van Kleek’s Calvinistic beliefs, which were originally introduced to North America by British Protestant colonizers. Unlike other Christian sects that scarily predestined people to hell unless they lead sinless lives, Calvinists believed doing good work in the name of God secures eternal rewards. Knowing that a seat in heaven requires having a positive impact on Earth lent to attacking social problems analytically and in ways that demonstrate measurable impacts. Calvinists’ emphasis on goodly, and Godly, action helped fuel the Progressive movement at the time, which depended on the charitable actions of educated elites to create order and harmony in the face of extreme wealth, race, and class divides.
The can-do spirit of Calvinism with the rational ordering of a social scientific mind has had tremendous influence on our American intellectual paradigm, forces that are still perpetuated by Ivy League colleges and charitable foundations. Even capitalism is rooted in Calvinism (Engerman, n.d.). According to good ole Wikipedia, the 18th century aphorism popularly credited to Benjamin Franklin that “time is money” exemplifies the concept that industrious (i.e., not sinful laziness) use of our God-given time can be quantified by how much money is earned and spent (2021).
Consider the impact of the Russell Sage Foundation on today’s philanthropic practices: theory of change tools managerially dictate grantmaking no matter how specious the connections between output and intended impacts; foundations’ “expertise” privileges approximating social science practices over grantees’ and communities’ lived and acquired experiences; arms-length information sources (outside evaluators and impersonal narrative reports) supersede the knowledge gained from direct relationships; and PRI investors’ use of financial risk assessments to “pencil out” overlook the potential of dependable returns by community members who work together to ensure mutually beneficial success.
So, let’s recap. Dr. Ioannidis’s meta-analyses research have demonstrated that researchers are too often biased and have conflicts of interest which prevent yielding valid, useful, and relevant information. Not only does this mean that our theories of changes should be looked at with suspicion; it means that most evaluation practices, particularly for systems change, divert funders’ attention away from actually achieving impact. Furthermore, in a moment of greater awareness of failed and unjust systems, institutions must recalibrate whose voices are considered authoritative. Dr. Anderson-Erisman’s recognized contributions demonstrate that scientific methods should be applied so as to be relevant to the end-beneficiary and not be clouded by personal (or institutional) objectives and gains. As her own work demonstrates, this means that not only must the end-beneficiaries’ viewpoints be included, they must have controlling interest in shaping the questions, direction, application, and assessment.
For foundations and its evaluation personnel, the switch begins by asking, For whom do we serve? For those trying to solve systems-level problems, enacting equity is a critical framework, not just a layer of consideration through inclusionary efforts. Whether trying, for example, to help specific Black communities gain agency over their futures as the desired outcome or because doing so is the means to a mutually beneficial end of land stewardship and social harmony, the locus of impact and measurable change needs to center on the conditions and voices of actual people at community, not organizational, levels. Trickling down foundation resources hasn’t been transformative enough.
This is the moment to fundamentally recalibrate from having evaluations drive grantmaking to having the lived experiences of people drive interventions.
Reference List
Alchon, Guy. (1999). Mary van Kleeck of the Russell Sage Foundation: Religion, Social Science, and the Ironies of Parasitic Mondernity. In Ellen Condliffe Lagemann (Ed.), Philanthropic Foundations: New Scholarship, New Possibilities (pp. 151–166). Indiana University Press.
Anderson-Erisman, Kim. (2016, October 29). Spinal Cord Injury Consumer Engagement in Research [Conference presentation]. Spinal Cord Injury Research and Science, Unite 2 Fight Paralysis. https://spinalcordresearchandadvocacy.wordpress.com/2016/11/23/spinal-cord-injury-consumer-engagement-in-research-dr-kim-anderson-erisman/
Engerman, L. Stanley (n.d.). Capitalism, Protestantism, and Economic Development [Review of the book The Protestant Ethic and The Spirit of Capitalism, by Max Weber]. EH.Net. https://eh.net/book_reviews/the-protestant-ethic-and-the-spirit-of-capitalism
Freedman, David H. (2010). Lies, Damned Lies, and Medical Science. The Atlantic https://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/
Ioannidis, John P. (2016). The Mass Production of Redundant, Misleading,
and Conflicted Systematic Reviews and Meta-analyses. MilbankQuarterly, 94(3), 485–514. https://pubmed.ncbi.nlm.nih.gov/27620683/
Ioannidis, John P. (2005). Why Most Published Research Findings Are False. PLoSMed 2(8). https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.002012
The Protestant Ethic and the Spirit of Capitalism. (2021, September 30). In Wikipedia. https://en.wikipedia.org/wiki/The_Protestant_Ethic_and_the_Spirit_of_Capitalism
