For agencies in Champaign County that implement programs for individuals with developmental disabilities or mental health concerns, there is no shortage of pressing issues that warrant immediate attention. But how can these agencies best allocate their time and resources to project the greatest benefit? How can agencies assess their reach and effect of services on participant outcomes?
These questions are ones that the Family Resiliency Center (FRC) has been studying over the past year and are the subject of their year-one annual report to the Champaign County Developmental Disability and Mental Health Boards on the project entitled, “Empowerment and Participatory Approaches to Building Agency Evaluation Capacity.” On Wednesday, May 22, FRC Director Jacinda Dariotis offered a presentation of top-line takeaways from the project’s first year.
The FRC report outlines the center’s analysis of the agencies’ needs and how those needs might gesture toward future actions.
An overarching goal of the project is to establish local agencies as cultures of continuous learning. This means that agencies implement a set of sustainable practices by offering staff ongoing opportunities to reflect, cultivate knowledge and add skills through feedback and training. Ideally, a continuous learning culture is the result of the application of an empowerment approach to evaluation capacity building.
The project’s participatory approach, which solicits input from partners at many levels of each organization to maximize the inclusion of individual voices, meant that the FRC evaluation team hosted a group level assessment (GLA) session. GLA allows a large group to anonymously share their opinions and experiences before collectively analyzing the group’s responses by identifying themes and potential action items.
Hosted at the Champaign Public Library, the GLA captured perspectives from 33 staff across 19 agencies and yielded over 2000 insights. The evaluation team conducted 13 focus groups and interviews with agency leaders, board members and past evaluators in addition to pre- and post-study surveys for participants. In total, the report’s readiness and needs assessment included perspectives from 76 stakeholders.
Just as important is the qualitative data from participants included in the report. Because of its participatory approach, the team was able to gather valuable insights during the GLA from both agencies and board members. One board member offered their perspective on feeling drawn to help agencies but also sensing a conflict of interest in doing so. “We end up providing a lot of support from our office,” said the board staff member, “but it creates some cross purposes for us. We cannot help people fill out their reports and then be the ones evaluating the reports. It's really a challenge.” At the same time, agency staff expressed excitement about evaluation capacity building. Said one agency participant: “I am new to the nonprofit world and am excited to have the chance to absorb the different vocabulary/focus points relating to evaluation/reporting vs the more corporate background I come from.” In short, although the GLA identified such challenges as limited time, staff shortages and participants’ reticence to complete surveys, the GLA also yielded a roadmap for ways that evaluators and researchers can support agencies.
One of the needs was for brief educational trainings related to the challenges agencies face. To this end, the FRC team has embarked on creating a suite of microlearnings, two of which are currently available to the public at http://go.illinois.edu/FRC-toolkits. One addresses how to avoid overpromising and underdelivering with regard to program outcomes, and another explains how and why to apply different evaluation tools to a program’s activities and participation versus its short-, intermediate- and long-term outcomes.
The year-one annual report was well-received. Vicki Niswander, the president of the Champaign County Developmental Disability Board, remarked that she could see the utility of extending evaluation capacity building far beyond developmental disability-related agencies. Niswander noted that the needs assessment and evaluation capacity building could be valuable for virtually all non-profit organizations. This is exactly the value-add that the FRC provides: accessible and sustainable capacity building resources that support programs and their evaluation and research activities.