The Most Effective Way to Check Your Evaluation Baggage

IMG_1119.JPG

Harnessing the superpower of key evaluation questions

Harnessing the superpower of key evaluation questions

By Morgan Valley

Key evaluation questions have a superpower — the ability to ensure the findings can guide strategic decision-making and improve the impact of the work.

As an evaluator, I’ve worked with many wonderful people at purpose-driven organizations who hear the word “evaluation” and visibly cringe. The first time I witnessed this reaction, I was undone that others didn’t share the love I have for my field. But after hearing their stories about bad experiences with evaluations that gave them findings devoid of useful information, I began to understand why they didn’t view evaluation as a useful learning tool, as I do.

The best way to help people move past their evaluation baggage and unlock the potential of evaluation is by crafting key evaluation questions. Key evaluation questions, which set the priorities of the evaluation up front, have a superpower -- the ability to ensure the evaluation’s findings can guide strategic decision-making and improve the impact of the work.

By articulating the most important things you want to learn from an evaluation at the beginning of the evaluation project, key evaluation questions do two important jobs:

  • First, they guide your evaluation by helping you specify why you want to do an evaluation and determine what you want to learn from the evaluation.

  • Second, they strengthen evaluation reports. When evaluation findings are synthesized and reported by each key evaluation question, you and others who read your reports can easily access the takeaways and supporting evidence to meet your learning priorities.

The best key evaluation questions guide you to the information that you need the most, right now. Here are some strategies for getting there:

Note: Key evaluation questions live in the middle ground between strategy questions and survey or interview questions. They dictate what you are going to ask participants and feed into the strategy questions that you need to answer as a nonprofit.

  • Involve others (your team, other key stakeholders, and program beneficiaries when possible) in determining which questions to prioritize. The questions should feel relevant both to the people doing the legwork of the evaluation; and to those whose experience will be influenced by the evaluation findings.

  • Decide the topics your key evaluation questions should focus on. Three great places to start looking for key evaluation question topics include:

    • Your Brain and Your Team’s Collective Intelligence

      • What do you and your staff want to know about the program?

      • What keeps you and your staff up at night?

      • What information would help you and your team do your jobs better?

    • Your Program Map

      • Which parts of your program are most critical to achieving your goals?

      • Which components of your program would cause the whole program to fall apart if they didn’t work as planned?

      • Which areas of your program act as important entry points for your clients?

    • The Existing Evidence Base

      • Where can you build on what is already known in existing research and evaluation on your content area?

      • What are the areas with the least evidence that need more information?

Once you’ve decided the content of your key evaluation questions, you can use these guidelines from Lori Wingate’s Key Evaluation Checklist to help frame your questions. She writes that key evaluation questions should be evaluative, pertinent, reasonable, specific, and answerable.

Building off the guidelines, here are a few tips to help make the most out of your key evaluation questions:

  • Try to pick no more than 3-5  key evaluation questions that are feasible (with a couple of sub-questions to provide some depth). This, of course, depends on the scope and resources available for your evaluation.

  • Keep your questions open to nuanced answers by avoiding yes/no questions and instead beginning questions with “to what extent…”

  • Use common language and define any terms you use so that your findings will be accessible to stakeholders outside of your immediate staff.

After you’ve articulated your learning priorities in a set of key evaluation questions, then you can decide which evaluation approaches are best suited to answer your key evaluation questions and feel confident that if done well, the evaluation will provide you with answers to questions that you and your team can use to inform your work.


Morgan.Site_480x320.jpg

Morgan Valley, PhD | Evaluator

Morgan is a recovering researcher, skilled in bringing mixed methodologies and advanced data synthesis to clients in way they can understand and use.


Morgan Valley