Today I attended an AEA Coffee Break webinar.1 Today’s session focused on evaluation tool kits. The tool kits discussed today were from a Farm to School program in Colorado. This was an interesting webinar in its own right. Without getting into the presentation at this moment (which was well done), a thought occurred to me while listening to some of the early description of the program.
There really are very different targeted beneficiaries whose interests align for this program when its successful. Students (children) who it is hoped will have improved nutrition, food knowledge, that kind of thing. Farmers, who want to see their products integrated into the local economy and are likely looking for some improved business results. The school district looking to keep its costs down and nutritional content up. (There may be others).
The question that popped into my mind right at the beginning of the session was, “Everyones’ interests align when the funding is secure, the context is consistent, and the program is headed toward a degree of success. What happens if things aren’t going well, if funding is cut (e.g. for budgetary or other reasons), or there is some other change in context (crop fail, teachers strike…)?”
Really, who is the more important beneficiary / stakeholder? Is there a favourite? What does the funder / program delivery organization do if the program goes badly? And, most tricky in my own mind, what happens if one stakeholder group’s needs must be preferenced over the needs of the others?
These are very much programming questions, it’s true. From an evaluation perspective though, how does the evaluation team tell that story? How do the evaluators capture the data to demonstrate whether or not an issue is in the offing; whether there is or is not a priority list; the consequences of a change in their context? How do they explain to someone that their needs will be secondary if things go sideways? Or more, interestingly, how does the evaluation team demonstrate the implications on various outcomes (which will necessarily vary by stakeholder group) what is likely to happen if the context or success of the program changes?2
It strikes me that if a program context changed or was likely to change, some evaluations wouldn’t necessarily catch the above questions – at least not if the program hadn’t thought about it in advance. (Am I wrong?) If no-one has worked through the implications of competing interests between different stakeholders at the front end, would it come up without a change in context.
Then, as I was typing this out and continued to see ‘context change’ as integral to explaining the issue that popped in my head, I got to thinking about John Mayne’s and others’ work on contribution analysis. That work is just full of capturing and describing context.
The thing is, I’m not sure my experience with contribution analysis is complete enough to capture the sceario I have in mind.3 When I think of contribution analysis, I think of an evaluation technique focussed on the front end of the initiative, the group of contributing factors needed to build solutions and in some ways on ascribing attribution to those working on and funding the initiative. I think of a scenario in which many groups are working together to support one, or a few similar beneficiaries.
When I’ve use contribution analysis before, the beneficiaries really weren’t that disparate. They were fairly similar really, with a collection of funders/programs working towards a similar goals. So, I haven’t used contribution analysis to look at context at the other end, when there are few funders/programs trying to simultaneously help different beneficiaries achieve some very different outcomes.
And that’s as far as my thoughts got before I finished coffee and needed to get back to other work.
If you have any thoughts, ideas, references, on this please let me know.
Also, it’s been a while since a drafted a long, rambling thought. Feel free to critique the style, tone, spelling, diction of this post.
All the best for now,
1 Today’s session was CFB 197: Evaluation Made Easy: Creating Meaningful Toolkits; a Farm to School example. Thanks to Dr. Lyn Kathlene from @SparkPolicy for taking the time to present today. I really appreciate your presentation style (aka clear) and a number of the implicit assumptions in your discussions (not everyone uses spheres of influence versus spheres of control, which I think are integral to evaluation. Similarly, I appreciate the ‘qualitative – quantitative data continuum’.)
Thanks also to AEA for coordinating and managing the series. If you haven’t taken advantage of these, I recommend them. The sessions are brief, so it is not too difficult to carve out the time from one’s work day to attend. There is a good variety of topics.
2 Understand, once I got thinking down these lines, I was no longer limiting my questions to just this example. I really have no idea how Dr Kathlene or her team would deal with that. The 20 minute presentation is no place to address it. Also, now I can think of all kinds of programs that might try to simultaneously support multiple beneficiaries.
3 I’ve been able to work with Steve Montague and Performance Management Network on a project or two.