Photo: Manki Kim on Unsplash
We’ve heard the phrase “this can’t be measured” more times than we can count (*bud-um-ch*). In fact, working across the domains of natural resource governance, public budgets and taxation, corruption, and open government writ large (collectively what we’ll call fiscal governance fields for the purposes of this project), we seem to run up against this sentiment more often than not. And what’s more, we really disagree.
When it comes to the fiscal governance fields[1], there are certainly indices that help us track global or even country-level governance trends (suggesting some agreement on the level of measurability!), but there are few indicators that enable individual organizations to track shorter-term change or their contributions towards macro-level change (as an example check out the huge difference in what’s available for practitioner use on Indikit’s SMART indicator guidance for Good Governance versus Agriculture or Gender Equality). This means a heavy reliance on the dreaded output-level indicators. So if it’s possible to measure, why the gap?
No one can clear the gap alone
For every measurement detractor in the “this can’t be measured” camp, there seems to be another trying to figure it out. Across the board, donors and organizations have been working simultaneously to find effective, and useful proxies in fiscal governance, but are coming up short. We’ve seen many organizations undergoing substantial revisions to their monitoring, evaluation, and learning frameworks and approaches. While they have many (and increasing) methods and tools to draw on that complement the complex nature of their work (e.g., outcome mapping, contribution tracing, outcome harvesting, PDIA)–a suite of tested, relevant indicators for their work is not among them. We recognize that identifying and developing proxies for fiscal governance field is a lengthy and expensive process which cannot be addressed by one CSO/NGO or donor alone. This is a collective action problem.
Confronting complexity to clear the gap
To be certain, fiscal governance work is complex, ever-changing, multilayered and messy. On the best days, improved fiscal governance means navigating contexts of regime change, shortages of resources and skills to implement good policies, tensions between centralized and devolved government, and lengthy time horizons for change. On the worst days, improved fiscal governance means battling entrenched, corrupt political and economic interests, insufficient accountability enforcement mechanisms, and limited incentives for decision-makers and powerholders to change. So we understand that thinking about ‘success’ is complicated when fighting the Hydra (i.e. closing one channel of corruption sometimes means three others spring open in its place). Moreover, like in many other fields, organizations are rarely the only ones tackling a particular issue-not to mention the multilayered nature of most programs and campaigns. Thus, attributing change to a particular organization, campaign, or intervention is rarely possible. In fiscal governance, we are almost always talking about contribution of an individual organization’s work and the collective impact of many. Even still, fiscal governance work is not the only complex work happening to make the world a safer, fairer place.
Learning from the Gap-Closers before us
We can’t help but see the extraordinary complexity covered in other arenas–like child marriage, sexual and reproductive rights, leadership and empowerment of marginalized individuals, and emergency response–and be impressed by the successful development of sophisticated and helpful proxy indicators (some we love: Sisterhood Survey method, Women’s Empowerment in Agriculture Index, and Poverty Probability Index) . We see practitioner-friendly and practical tools for M&E cropping up in many complex arenas. So why these fields and not fiscal governance? Our hunch: Resources have been invested to explicitly enable organizations and researchers to track change and impact in these other fields. Though we recognize that it’s complex, we are convinced that what our indicator gap needs is more resources and creative minds.
A small offering to close the gap
Recognizing this, Open Society Foundations’ Economic Justice Program (formerly known as Fiscal Governance Program) with a steering committee of talented MEL experts from leading fiscal governance organizations (IBP, NRGI, OGP, TAI, TJNA) have teamed up with Results for Development (R4D) to tackle this issue (in a very modest, first-step-of-many kind of way!).
Starting in late 2018, Results for Development (R4D) began a project to (1) explore existing indicators and identify/define gaps from the perspective of a wide range of organizations and (2) based on gaps and demand, begin to design and test a series of new (or improved) indicators in a wide range of contents. If successful, R4D will go on to design practitioner guidance and training on implementing these indicators for organizations. If the design and pilot phase leads to more questions than answers, R4D will work on socializing the extent of their work and propose a series of next steps the field needs to take for this gap to be bridged.
This project is meant to provide a service to the field–one spotted after deep conversations with many, many organizations encountering the same challenge–and not as a sneaky way to introduce new donor requirements or mandated indicators. This is the primary reason it has a dynamic and visionary steering committee to co-own the project! As the project unfolds, we plan to share lessons along the way and invite ample feedback and contributions from a wide range of actors. Check out R4D’s project site here for more information. Interested in offering ideas, cautions, or input on this project? Contact Caroline Poirrier ([email protected]).
[1] We recognize that ‘Fiscal Governance’ is not necessarily a readily or widely used concept. For the purposes of this blog, the scope of ‘Fiscal Governance Fields’ should be considered: natural resource governance, tax, budgets, anticorruption, and trade governance.
Megan Colnar is the Director of Strategy and Impact at the Open Society Foundation’s Economic Justice Program and Soros Economic Development Fund. Suvarna Hulawale is a Monitoring, Evaluation and Learning Officer at International Budget Partnership. Alison Miranda is the Senior Learning Officer at TAI.
