Tag Archives: evaluation

Attended @CGDev’s Measuring and evaluating women’s economic empowerment agenda

This morning I had the opportunity to attend the Center for Global Development’s Public Event
Measuring and Evaluating Women’s Economic Empowerment

A great start to the day, "What Works for Whom and here?" are the baseline questions that make up the evaluations in which I have the most confidence .

A great start to the day, “What Works for Whom and here?” are the baseline questions that make up the evaluations in which I have the most confidence .

While there was much to consider, here are a few interesting tidbits that I took note of during the discussions.

NOTE: These are my on-the-fly notes. If I’ve mis-quoted or mis-attributed anything, I am happy to update the post. 

The day started with excellent context setting by Melanne Verneer. As an advocate for and user of evaluation, she speaks of the (refreshingly) knowledgeably about the field and its application to decision making.

— Mayra Buvinic, Senior Fellow, United Nations Foundation https://twitter.com/unfoundation

A thought on gender equity over time “Being ridiculed, to being resisted, to beginning to see change”
— William Savedoff, Senior Fellow, Center for Global Development https://twitter.com/billsavedoff

Spending priorities vary wildly within a household, 94% of the time. Her favourite pet statistic is from FAO study on Cote D’Ivoire that $10 in the hands of a woman well advance the same development goals as $110 in the hands of a man.
—Krisila Benson, Senior Director of Program Services, TechnoServe https://twitter.com/TechnoServe

Q: we have seen when empowering women, there’s been a need for someone on their household to take up the slack and often it’s their daughters. how do you track and address unintended consequences?
— (paraphrased, question from the audience – did not catch name/organization in time)

Q: (following a discussion of rigour in monitoring and evaluation) do we need more rigour in evaluation or do we need something else to influence decision-making, policy, and behaviour change on the ground?
— (paraphrased question from the audience, a colleague at the Center for Global Development, but I did not catch the name)

As funders, we may have crippled innovation be demanding programs that are built on prior data.
— Deborah Birx, Ambassador-at-Large, US Global AIDS Coordinator and US Special Representative for Global http://twitter.com/pepfar

Request for feedback and thoughts on DREAMS partnership, http://www.pepfar.gov/partnerships/ppp/dreams/index.htm
— Deborah Birx, Ambassador-at-Large, US Global AIDS Coordinator and US Special Representative for Global​​ http://twitter.com/pepfar

All in all, an excellent set of discussions.

Leave a comment

Filed under evaluation, training

evalc2015 What I did at conference

I very nearly titled this post ‘What I did at summer camp‘. Last week, I and 450 (+/-) people went to Montreal to join the CES 2015 Annual Conference. It’s fairly often that I find I need to explain to people what evaluation is, where it fits in the world. It’s not difficult, but it’s nearly always at an introductory level. Working remotely or on-site as the external evaluator, there are rarely opportunities to have significant, in-depth, free flowing conversations about what I do or new ideas on how to do it better. Also, I’m a bit of an extrovert. I like being in the same room as people. So conferences are a lot like professional summer camp, heavy on the learning and relationships.

I’ve never understood employers who won’t pay for conferences, at least in part. I’m baffled by those who won’t give time to attend. One needn’t attend everything every year, but there is so much that can be accomplished in such a short period of time when everyone is in the same space. It’s different for the sole proprietor or small non-profit organization that has very limited funds, but really, everyone else should get to something at least every couple of years.

Leave a comment

Filed under Uncategorized

a thought occurred to me today

aea coffee break webinars
well worth the time

Today I attended an AEA Coffee Break webinar.1 Today’s session focused on evaluation tool kits. The tool kits discussed today were from a Farm to School program in Colorado. This was an interesting webinar in its own right. Without getting into the presentation at this moment (which was well done), a thought occurred to me while listening to some of the early description of the program.

There really are very different targeted beneficiaries whose interests align for this program when its successful. Students (children) who it is hoped will have improved nutrition, food knowledge, that kind of thing. Farmers, who want to see their products integrated into the local economy and are likely looking for some improved business results. The school district looking to keep its costs down and nutritional content up. (There may be others).

The question that popped into my mind right at the beginning of the session was, “Everyones’ interests align when the funding is secure, the context is consistent, and the program is headed toward a degree of success. What happens if things aren’t going well, if funding is cut (e.g. for budgetary or other reasons), or there is some other change in context (crop fail, teachers strike…)?”

Really, who is the more important beneficiary / stakeholder? Is there a favourite? What does the funder / program delivery organization do if the program goes badly? And, most tricky in my own mind, what happens if one stakeholder group’s needs must be preferenced over the needs of the others?

These are very much programming questions, it’s true. From an evaluation perspective though, how does the evaluation team tell that story? How do the evaluators capture the data to demonstrate whether or not an issue is in the offing; whether there is or is not a priority list; the consequences of a change in their context? How do they explain to someone that their needs will be secondary if things go sideways? Or more, interestingly, how does the evaluation team demonstrate the implications on various outcomes (which will necessarily vary by stakeholder group) what is likely to happen if the context or success of the program changes?2

It strikes me that if a program context changed or was likely to change, some evaluations wouldn’t necessarily catch the above questions – at least not if the program hadn’t thought about it in advance. (Am I wrong?) If no-one has worked through the implications of competing interests between different stakeholders at the front end, would it come up without a change in context.

Then, as I was typing this out and continued to see ‘context change’ as integral to explaining the issue that popped in my head, I got to thinking about John Mayne’s and others’ work on contribution analysis. That work is just full of capturing and describing context.

The thing is, I’m not sure my experience with contribution analysis is complete enough to capture the sceario I have in mind.3 When I think of contribution analysis, I think of an evaluation technique focussed on the front end of the initiative, the group of contributing factors needed to build solutions and in some ways on ascribing attribution to those working on and funding the initiative. I think of a scenario in which many groups are working together to support one, or a few similar beneficiaries.

When I’ve use contribution analysis before, the beneficiaries really weren’t that disparate. They were fairly similar really, with a collection of funders/programs working towards a similar goals. So, I haven’t used contribution analysis to look at context at the other end, when there are few funders/programs trying to simultaneously help different beneficiaries achieve some very different outcomes.

And that’s as far as my thoughts got before I finished coffee and needed to get back to other work.

If you have any thoughts, ideas, references, on this please let me know.

Also, it’s been a while since a drafted a long, rambling thought. Feel free to critique the style, tone, spelling, diction of this post.

All the best for now,

Today’s session was CFB 197: Evaluation Made Easy: Creating Meaningful Toolkits; a Farm to School example. Thanks to Dr. Lyn Kathlene from for taking the time to present today. I really appreciate your presentation style (aka clear) and a number of the implicit assumptions in your discussions (not everyone uses spheres of influence versus spheres of control, which I think are integral to evaluation. Similarly, I appreciate the ‘qualitative – quantitative data continuum’.)

Thanks also to AEA for coordinating and managing the series. If you haven’t taken advantage of these, I recommend them. The sessions are brief, so it is not too difficult to carve out the time from one’s work day to attend. There is a good variety of topics.

Understand, once I got thinking down these lines, I was no longer limiting my questions to just this example. I really have no idea how Dr Kathlene or her team would deal with that. The 20 minute presentation is no place to address it. Also, now I can think of all kinds of programs that might try to simultaneously support multiple beneficiaries.

3 I’ve been able to work with Steve Montague and Performance Management Network on a project or two.

Leave a comment

Filed under Uncategorized

evaluation work + advice

I had a great professional compliment recently – I was cold called by a new evaluator looking for a job.  “I know your firm and its reputation…” he said. Isn’t that lovely. (Although, he didn’t mention any details of my reputation. ‘Fun at parties‘?)

Nonetheless, given the pending move to the other side of the planet, I thought it best not to take on new staff. Rather than being a jerk and just saying no, or worse, sending out the professional version of it’s-not-you-out’s-me, I thought the least I could do is add to his bag of tricks in looking for work. I put together a few tricks that I have used to get work / make professional contacts. Once I’d wrote them out, I thought that they might be useful to the 4 people likely to read my blog. So, here they are:*

  • MERX.  It is free to register for federal (and some provincial) projects on MERX. Hint: register and download an RFP for a project you are interested in. That then gives you access to a list of other individuals / firms interested in the same project. You can then target specific firms for specific projects. (When I first started, I would do that and then either phone / email and offer to take on a specific area that I knew I could do very well. It doesn’t pay off every time, but will get you one or two projects a year. Good for your reputation and helpful with contacts)
  • geds, as it is called. I’m sure you’re familiar with them. A search by evaluat* or phone number gets you a good list of potential government contacts. Directors are too low on the food chain to do hiring, (that’s at the DG level), but you can ask to ‘do lunch’ with directors to get a sense of the industry and/or that specific department. 
  • Body shops (professional recruitment firms) can get you onto some larger contracts. They take a cut, a fairly sizeable one if you’re junior, but you can still make decent money. Also, if you speak to them and/or one of the other body shops, ask them about the different contracting tools currently available to government. Often times, client department’s program staff really don’t know how to do contracting side, and that can prevent employment. 
  • Charity village  Access to the not-profit list of evaluation jobs (or half program / half evaluation). They don’t pay as well as the feds, but are more forgiving if you’d like to take on a contract arrangement on your own. If you are hired on permanently somewhere, not-profits are loyal employers and good places to grow in the field
  • Twitter don’t underestimate twitter as a place to find out about work (feel free to copy the folks I follow @lisaoreillyca or those from @CESBCY). I’ve seen some fabulous jobs go out on twitter alone.
  • devnet has some interesting things, especially for bilingual and/or willing to travel
  • on a similar tract, at the AEA conference last year an American not-profit was desperate for evaluators in Haiti. They were accepting Anglophones because they have so few Francophone speakers. If you can do some travel, you can look into the American aid organizations. Your language skills will get you a long way in markets you may not expect. 
  • if you are going to a  conference (CES, AEA, anything else): network ruthlessly. Learning is for people with full time jobs. Networking is for everyone else. (and presenting is for the tenured or truly dedicated).
  • if you aren’t going, Conference sponsors and Case Competition sponsors are great lists of potential employers (after all, they have enough money in the budget to sponsor something, they may be able to take on staff)
*oh, and if I suddenly get millions of readers who are applying for evaluation work, I’m going to want some kind of thank you / hug / percentage of the take

Leave a comment

Filed under Uncategorized