May 31, 2019 — by Elizabeth McGee
In recent decades, “systems thinking” has become more commonplace in both the public and private sectors and has been proven to be a more effective way to solve complex societal problems. Along with the shift to more comprehensive systems-level interventions, there has also been a need to design systems-level monitoring and evaluation efforts that allow us to examine large-scale impact and progress towards short- and long-term goals. As one who is interested in evaluating system interventions, I am lucky to have the opportunity to work with agents of systems change in South Dakota who are changing the landscape of STEM research and education infrastructure with support from the National Science Foundation (NSF).
South Dakota EPSCoR
STEM research, innovation, and education are high priority issues for South Dakota. The South Dakota EPSCoR (Established Program to Stimulate Competitive Research) program aims to develop and increase the state’s STEM research infrastructure and capacity. Their mission is to build science, technology, engineering, and math (STEM) education, increase science literacy, and drive science-based economic development across the state through a partnership of public, private and tribal higher education communities, K-12 education, the state government, and the private sector.
Program Philosophy and Approach
The South Dakota EPSCoR program has several philosophical orientations that underpin how the program approaches change-making in the state. All of SD EPSCoR’s change-making is rooted in a network orientation approach, which translates to efforts that primarily target change through building institutional collaborations and networks, rather than the use of single interventions that target individual-level change (i.e. change among a single organization or entity as opposed to the larger system). As a fundamental principle, the network orientation concept asserts that building infrastructure and fostering collaboration are drivers for creating higher-level and more long-term systemic changes in South Dakota.
The South Dakota EPSCoR program is rightfully complex, thus the evaluation of this initiative needed to be set up in a way that was able to evaluate this complexity. The multi-year evaluation required four evaluative streams, described below as Systems Change Capacities, Early Changes, Systems Changes, and Population Change, all of which provided data to understand SD EPSCoR’s progress towards state-level change.
As South Dakota EPSCoR’s Lead Evaluator (at LEAP Consulting), I had the pleasure of presenting evaluation findings for the final year of the program to The South Dakota Research, Excellence: A Critical Hallmark (REACH) Committee this past week. The REACH Committee provides independent and unbiased coordination of South Dakota’s EPSCoR grant from NSF, as well as other STEM-related initiatives in the state. Also in attendance was the core EPSCoR project team.
Themes
LEAP Consulting’s presentation introduced several themes that emerged over the 5-year evaluation. These themes, outlined below, demonstrate how evaluations that are well-positioned to evaluate systems change can become powerful tools to increase a project’s impact.
- Developing a Theory of Change, and revising that Theory of Change throughout the program, was important to maintain a clear perspective of how the program components work together to elicit impact.
- Distinguishing between early changes, systems changes, and population-level changes was important for explicating what changes we expect to see more concretely. Ensuring each level of change with a distinctive evaluative approach, yet reporting on them collectively was important to paint a clear picture of the system change underway.
- For example, an early change may involve piloting an internship program within a single university, a systems change may involve scaling that same internship program state-wide, and a population-level change may involve the project’s contribution to STEM graduate rates in South Dakota.
- Reflecting critically on program sustainability and the capacities needed to sustain a state-level, outcome-producing infrastructure were important for planning future program activities.
Sustainable Evaluation
Again, using the example of a state-wide internship program. How would this program be funded beyond the EPSCoR grant in the state?
It was evident throughout this presentation and discussion that the systems approach to the evaluation of SD EPSCoR’s program activities helped the REACH Committee and the EPSCoR program team think more concretely about the impact that EPSCoR has had at all levels over the life of the program. These conversations are critical for future planning of program activities, especially related to the sustainability of the change that program components have had in promoting STEM research, innovation, and education across the state. By using a systems approach to evaluation, LEAP Consulting was better able to account for and monitor the complexities of this type of work in achieving large scale goals, such as changing the landscape of STEM state-wide.
LEAP Consulting specializes in systems approaches to evaluation as used in the South Dakota EPSCoR evaluation. LEAP helps nonprofits, foundations, public institutions and funders across the United States achieve greater impact through meaningful evaluations and strategic support. We are social change strategists who offer evaluation, systems, placemaking, and community engagement expertise in a variety of formats. Learn more about our areas of support here. Want to get in touch about your program or next project? Contact us here.
Originally published here.