This is the first of two blogs that discusses what motivated TAI to commission an external evaluation and insights from the process. Part two of this series will discuss TAI’s main evaluation insights and what the collaborative is doing to act on those learnings. You can read a summary of the main evaluation findings and learnings in TAI’s evaluation brief.
How do you measure donor collaboration? What value does a funder collaborative yield for members? These are the sorts of questions that we always have at the back of our minds at Transparency and Accountability Initiative (TAI). We imagine other donor groups do, too. Having just completed an evaluation we feel we are better positioned to offer some answers.
Why Did We Evaluate?
This was not a funder-required evaluation – so why in the world would TAI decide to evaluate?
Regular evaluation is good practice for a healthy learning organization. As a funder collaborative, evaluative learning is crucial. Our work hinges on providing value for diverse members and inspiring continuous member engagement to drive the collective’s work forward. As we crossed the mid-point line of our previous strategy, TAI sensed questions emerging about our current and future work.
TAI planned this evaluation as an opportunity to systematically assess strategic progress and performance to date beyond our regular data collection. We benefitted greatly from the expertise and external perspectives of Julie Slay, Allison Shean, and Abby Degenhart of Arabella Advisors, who designed and led this evaluation with us.
How Did We Position the Evaluation to Maximize Learning?
Though not a funder-mandated evaluation, the success of this effort required funder engagement. We started with decision-informing questions and drew on timing and participatory processes to maximize learning with TAI’s funder members.
Design driven by purpose. TAI started the evaluation planning process by defining potential purposes and decisions that the evaluation might serve to inform. For example, we wanted to assess our implementation of TAI’s 2016-2019 strategy to inform changes to the theory of change and we wanted to explore progress against our strategic outcomes to inform assumptions about our sphere of influence. This helped us to surface Secretariat and member interests and needs. And it informed the evaluation scope by matching the type of evidence we expected to generate with appropriate questions.
Timing serves decisions. Timing is a key factor in designing a useful evaluation and producing usable findings. TAI positioned the evaluation to precede and inform our strategy planning process. This is truly a balancing act as there are many factors driving when and how evaluative and strategic processes unfold. Shared member commitment to funding and using evidence helped us to walk this tightrope of timing between evaluation and strategy. If a sequential evaluation-to-strategy approach is not right for your context, design a timeline that is workable for the purposes and decision points you prioritize.
Use starts well before the final report. In partnership with Arabella Advisors and strategy consultant Chris Michael, TAI designed and delivered several participatory touchpoints for member engagement. This included virtual and in-person moments with members to inform evaluation questions, raise awareness about the evaluation, and interpret findings. But these sessions also pushed us to roll up our sleeves and sense make the evaluation evidence against our experience and other evidence about funder collaboratives. We dug into uncomfortable areas of ambiguity and some divergent data points. TAI continued to apply evaluation insights and other evidence into our strategy process.
What Did We Learn from This Process?
Working with external evaluation and strategy partners was an incredible advantage as we explored several topics that would have been difficult to assess without an external perspective (i.e., effectiveness of the Secretariat staff, or relevance of current membership).
In practice, aligning evaluation and strategy efforts required continuous attention and realignment, which resulted in a bit more effort than we anticipated. We are appreciative of the can-do spirit not only of our funder members but especially the evaluation and strategy partners that supported us. In the future, starting the design efforts for both processes together may have helped us anticipate these needs.
We were pleased to learn of our challenges and successes through this evaluation, including the different ways TAI offers value to its members. We have also found great value in our investment in this evaluation to inform TAI’s work ahead.
