Evaluation in the Wild: Beer and Unintended Consequences

WILD_binoculars.jpg

How values and nuance shape understanding

How values and nuance shape understanding

By Elena Harman

I use family visits as an excuse to explore our local cultural institutions. So when my husband’s family came to visit, off we went to History Colorado’s “Beer Here!” exhibit

“Beer Here!” explores how Colorado has shaped brewing and how brewing has shaped Colorado. Walking around the exhibit, I came around the corner from Prohibition to post-Prohibition and boom! A question about what works! Now, I’ve gotten used to seeing evaluation of museum exhibits, but I was pleasantly surprised to find evaluation in a museum exhibit.

Prohibition did work.  The display asks whether Prohibition achieved its stated goal: reducing alcohol consumption. By this measure, Prohibition was a success: it dropped alcohol consumption by 70%, and even today, drinking remains lower than pre-Prohibition. So we are done, right? Prohibition was a success, and maybe we should try that model for other social ills. Right?

Prohibition did work. The display asks whether Prohibition achieved its stated goal: reducing alcohol consumption. By this measure, Prohibition was a success: it dropped alcohol consumption by 70%, and even today, drinking remains lower than pre-Prohibition. So we are done, right? Prohibition was a success, and maybe we should try that model for other social ills. Right?

Prohibition didn’t work.  Not so fast. When you twist the plate, a different story emerges. What were the unintended consequences of Prohibition? Here we learn that prohibition fueled the rise of the mafia and crime throughout Colorado. By this measure, Prohibition was a failure: it increased crime – particularly violent crime.

Prohibition didn’t work. Not so fast. When you twist the plate, a different story emerges. What were the unintended consequences of Prohibition? Here we learn that prohibition fueled the rise of the mafia and crime throughout Colorado. By this measure, Prohibition was a failure: it increased crime – particularly violent crime.

beer.jpg

So where does this leave us? Did Prohibition work or not? The display leaves that determination to you. Does the reduction in alcohol consumption make up for the increase in crime? Or does the increase in crime nullify the reduction in alcohol consumption? 

This small display is such a good example of evaluation in action. At its core, evaluation is the process of asking questions about what works. But it gets complicated quickly: how do we define “what works”? Who defines “what works”? What do we conclude when a program generates both positive and negative effects? 

This display is an elegant example of evaluation truths: 

  1. Unintended consequences matter. If we had only looked at the intended impact (reduced alcohol consumption), we would have missed Prohibition’s impact on crime and falsely concluded that it was a raving success. Think about your own programs – how do you integrate unintended consequences into your evaluation? 

  2. Evaluation judgments of success or failure are never 100% black or white. All programs have elements that work and elements that do not. And the opportunities to improve are in that nuance. How might we design a program that reduces consumption and crime? I wonder how, if at all, the experience with Prohibition informed Colorado’s implementation of legalized marijuana.

  3. Values drive judgments of success and failure. When you see a success/failure conclusion, ask whose values are behind that determination. Who decides whether crime or alcohol consumption matters more? Often we find that the people in power are the ones whose values determine our conclusions about program success. What might be different if we prioritized the values of those served by the program?

For me, the power of this museum display was how it encompassed two truths and challenged visitors to rethink their conclusions. For the child next to me, the power of this display was in the fun knobs that turned the plates…but to each her own.


Elena.Site.jpg

Elena Harman, PhD | CEO

Elena is the author of The Great Nonprofit Evaluation Reboot: A New Approach Every Staff Member Can Understand. She pushes everyone around her to think bigger about what evaluation can be, and how it can help improve our communities. Elena brings a deep expertise of systems, nonprofits, and foundations, as well as how to engage diverse audiences in a productive conversation about evaluation.


Elena Harman