Capturing the real impact of youth development [7] projects that aim to change attitudes and behaviors is a challenge—it’s not found in numbers of people trained or workshops held. Traditional monitoring and evaluation (M&E) tools have their limitations; for example, young people often want to give the “right” answers on surveys or in focus groups– a bias that can be even stronger in cultures that prioritize respect for elders. Personal stories can provide great insight into how a program has changed lives, but they’re generally anecdotal rather than statistically significant M&E data.
However, at a recent panel called Beyond the Numbers: Storytelling as a Youth Development Evaluation Tool [8], hosted by the Society for International Development – Washington’s Youth in Development Workgroup [9] at IREX’s headquarters, I learned about two creative, quantifiable M&E techniques that can strengthen a mixed-method evaluation plan. Soledad Muñiz presented the Participatory Video [10] methodology her organization, InsightShare [11], has used in Kenya and around the world, while Matt Gallagher shared his experience analyzing stories as an evaluator for the Administration for Native Americans [12] (ANA).
Two valuable common threads stood out for me—the use of storytelling, an important feature in both Kenyan and Native American culture, and the shift in focus from beneficiaries as subjects of evaluation to partners in the evaluation process. InsightShare trains communities to make videos about important local issues. In the evaluation of a Mercy Corps [13] sports and tolerance program in Kenya, participants selected the video stories they felt were most representative of the program’s impact.
To evaluate grant projects supported by the ANA, Gallagher made space for story circles, where participants talked freely about the changes they saw in their communities and drew their own conclusions about the project’s impact. Through additional story coding and analysis, he documented connections and themes– for example, “reclaiming identity” as the result of a Native American language school project—and used the data in ANA budget justifications to Congress.
Janet Kerley, Chief of Evaluation, Research and Measurement at Peace Corps [14], served as a respondent from the donor’s perspective. (Kerley previously held senior positions in evaluation at USAID and the State Department.) “An anecdote is not evidence, but the plural of anecdote is data,” she said, quoting UC Berkeley political scientist Ray Wolfinger. “Stories are legitimate sources of qualitative data.”
They can’t stand in isolation, but these innovative approaches show that when proper standards and rigor are applied, stories are more than just promotional material. As a youth development professional focusing on attitudinal and behavioral change, I look forward to exploring new ways to give youth a voice in capturing the impact they value most about IREX programs.