Approaches to Evaluation during the Pandemic

Moderator

• Richard Jones, Chief of Section, IEO, UNDP

Panellists

• Santiago Ramirez Rodriguez, Evaluation Analyst, IEG, World Bank Group
• Baye Elimane Gueye, Coordinator of the Socioeconomic Impact Monitoring Unit, Operational Office for Monitoring the Emerging Senegal Plan, Senegal
• Jenna Smith-Kouassi, Evaluation Policy Analyst, OECD

How did evaluation adapt to the pandemic to ensure continuity and inform decision-making? What lessons and approaches were learnt that should be taken forward?

• Many organizations pivoted quickly in response to the pandemic, producing evaluation guidance and principles focusing on doing no harm, broadening desk reviews, and working remotely using Zoom and WhatsApp.
• There was also a growth in synthesis using pre-existing evaluations and experience, often of responses to past natural and health crises, to provide lessons for different organizations and support decision-making. Partnerships were strengthened. Although we are back in the field, some things remain, such as Zoom, WhatsApp, and the formulation of Reflections.
• Limited travel triggered the use of technology and data, sharing information, lessons and best practices, and better coordination amongst the evaluation community. Local evaluators became essential to the evaluation process and there is a need to continue to build this capacity.
• The pandemic has accelerated the use of triangulation methods at country, portfolio and corporate levels through the response from relief, restructuring to recovery stages. In the context of COVID-19, the use of this method was essential to ensure the precision of the data, validate findings and understand the situation in depth.
• The need for - and value of - collecting data through mobile devices and the use of open data became increasingly important in countries under lockdown, to assess the social and economic impact of the pandemic and movement restrictions on the population. The collection of this data during the pandemic was key to propelling the participatory decision-making process, where stakeholders at various levels, including the government and academia, got involved.
• Governments are now planning to utilize this experience to continue to inform robust decision-making. However, there remain challenges in using mobile data, such as the digital divide, legitimacy, anonymity, and others.
• Evaluation networks, such as EvalNet, are essential in: transforming relationships with programme stakeholders and decision-makers; re-thinking how evaluation evidence is gathered and communicated; increasing communication, coordination, and collaboration across the global evaluation community.
• Without the COVID-19 pandemic, capacity-building, information sharing and coordination, as illustrated through the COVID-19 Global Evaluation Coalition, would not have increased, so in the end, some positive shifts were triggered. Additional benefits of doing remote evaluations were reducing the carbon footprint and conducting remote interviews with a larger number of people in the comfort of their homes.

Conclusion

The pandemic pushed evaluators out of their comfort zone. First, it has accelerated the process of sharing findings that are good enough and not necessarily perfect. This means that timeliness instead of perfection has become a priority. Second, reliance on national evaluators increased their responsibilities on the ground. This has also sped up the transfer of the evaluation process from international to national levels, encouraging more collaborative work in a hybrid format. Third, the pandemic has also accelerated the adoption of innovative methods, and the experience of using those serves as the basis for fast and better decision-making.