Building evaluation capacity leads to positive changes for the kids they serve

DPL_main_AdobeStock_245067021.jpg

How the Denver Public Library’s Read Aloud program overcomes evaluation hurdles, leading to clarity and confidence in evolving their approach

 
 

Leaders at the library wanted more than a “gut feeling” that their program was making a difference. Find out how they overcame their evaluation paralysis, and what they are accomplishing now.

 
 

dpl_logo.jpg

Introducing Denver Public Library’s Read Aloud Program

The Read Aloud Program provides research-based, early learning-enriched storytimes in approximately 350 early childhood classrooms across the city of Denver each week. This program aims to support early literacy skill development, helps kids learn to love to read, and builds social and emotional skills.


 

What they did

JaggedLine3_peach_350px.png

Make positive improvements to their program with confidence

After convincing their primary grant funder to include a one-time investment for evaluating their program, they experienced several false starts before seeing Elena, our CEO, present at a conference. They felt energized by Elena’s vision for evaluation, as well as her passion for helping nonprofits learn how to do evaluation well on their own.

After embracing a new definition of evaluation, learning new methods, and grasping how to analyze the data, the team was able to see clear trends. These findings gave them the confidence to make some significant changes to their volunteer training process and what they expect from their volunteers, all of which has lead to a bigger impact for the kids.

 
The capacity building option was a really good one for us because we had a lot of tools in place, and only a one-time investment. We just didn’t have the knowledge and training we needed to be successful. Now we do!
— Sarah McNeil // Senior Librarian, Early Learning Department • Denver Public Library
 

 

How we did it together

JaggedLine3_peach_350px.png

Overcome two major mindset hurdles that had been standing in their way

The first stumbling block was the internal departmental perception that evaluation clashed with one of their core values: the library is about creating an environment where everyone feels welcome to pursue learning in their own way, in their own time, and is not about testing kids. The second roadblock was the impression that evaluation has to be a complex, double-blind study showing direct impact on reading at grade level by third grade, and the fear that if they couldn’t do that, there was no point.

Together, we overcame these hurdles by:

  • Shifting the interpretation of evaluation from “testing” to “learning in our own way.”

    The team came to evolve their view of evaluation from something that clashes with their core values to one that celebrates them.

  • Learning qualitative methods that felt authentic to the organization.

    Training on focus groups showed them a way to embrace the stories that felt important, but that they had previously believed had no place in evaluation.

 
Vantage helped us see evaluation as a genuine quest for more information so we can do things better, rather than a rigid, quantitative tool to catch us doing something wrong. We no longer fear the evaluation process.
— Sarah McNeil // Senior Librarian, Early Learning Department • Denver Public Library
 

Tweak the tools to make their evaluation efforts more impactful

The team experienced many a-ha moments together, and we modified the evaluation tools they already had in place.

  • Ask more intentional questions to get more valuable insights. Rather than asking teachers about general satisfaction, they now drill deeper to input around early literacy and social-emotional skill development. This has inspired improvements to the program.

  • Tweak the tools to get a higher response. Small, but significant changes to their surveys have led to an increase in quality data.

    For example, shortening surveys, adjusting how and when they send them, and closing the feedback loop afterward.

  • Build skills in analyzing the data. Learning to use pivot tables and make graphs in Excel allowed them to visualize the findings, as well as reach a deeper level of understanding.

 
Vantage took us through an exercise, which made us realize that what we had been measuring did nothing to answer the core questions we had. This ‘ah-a’ had a wonderful ripple effect.
— Sarah McNeil // Senior Librarian, Early Learning Department • Denver Public Library
 
 

The Results

JaggedLine3_peach_350px.png

Deeper understanding leads to considerable changes around the volunteer component of their program

The evaluation had concrete evidence, rather than just “gut feelings.” These findings gave the library team the confidence necessary to make significant changes to their volunteer training process, as well as what they expect from their volunteers. This change led to a bigger impact for the kids.

  • Shift program elements from optional to required. A profound understanding of the impact singing and dancing has on early literacy skill development inspired the team to make this a requirement, rather than an optional bonus activity.

  • See volunteers in a new light. As findings led to increasing program requirements, the team gained courage to expect more from volunteers, understanding they are a critical component of running a successful program.

  • Significant improvements to their volunteer training. From the content to the amount of training offered per year, and expectations for how frequently volunteers must attend these trainings, they have made substantial shifts for more impact.

 
One of the biggest shifts has been our mindset around volunteers, from an audience we serve to a critical component of our program. To best serve the kids, we need to have certain expectations of our volunteers, and that’s okay.
— Sarah McNeil // Senior Librarian, Early Learning Department • Denver Public Library
 

 

New evaluation skills lead to ongoing greatness

The capacity building project has led to new skills and a positive shift within the team, which will continue to pay dividends for years to come.

  • Evaluation is part of ongoing discussions.

    Now, they are thinking about evaluation at the beginning of a project, and have inspired other programs and departments to rethink how they evaluate.

  • More effective grant applications.

    The team can now present specific data regarding the impact of their program, rather than having to rely solely on stories and pictures of cute kids.

  • Proactively applying new evaluation skills.

    They have conducted their own focus groups, and felt empowered to apply the concepts to improve their site observations of volunteer readers on their own.

 
The Vantage team really listened to us with compassion and empathy. Then, they created a project that fit our specific needs, rather than fitting us into a mold.
— Sarah McNeil // Senior Librarian, Early Learning Department • Denver Public Library
 

 

Ready to get started?