Digital Innovations in Evaluation

Moderator

• Claudia Oehl, International Project Manager, evaluator and trainer, Organizational Development and Project Services Programme, ITC, ILO

Panellists

• Daniel Jacobo, Technical Advisor, Division of Evaluation of Development Policies - Ministry of Foreign Affairs, European Union and Cooperation, Spain
• Gonzalo Gomez, Chief of Knowledge and Data Management, IEO, UNDP
• Nodumo Fikile Magadlela, Co-founder and CEO, KPI Lens Limited
• Deo-Gracias Houndolo, Lead, African Institute for Monitoring and Evaluation of the Impact of Development Policies, Programmes and Projects (Le Baromètre)
• Toshiyuki Yokota, Principal Evaluation Specialist, Independent Evaluation Department, Asian Development Bank

Which new pathways do data-heavy, cutting-edge technologies open to evaluation? Data are vital for understanding the progress and impact of development strategies. New technologies, coupled with increased computing power, are creating opportunities for gathering and analysing ever larger amounts of data from a greater range of sources. In addition, remote data collection is playing an increasingly important role, and can make evaluation nimbler when coupled with appropriate technology use.

This panel discussed some illustrative application cases and the new possibilities and limits encountered. What are the trends and where are the limits?

• Several examples were presented of technological innovations in the use of artificial intelligence, data collection and the availability of findings, and the use of geospatial data:
  • The Spanish Ministry of Foreign Affairs presented their evaluation resource dashboard. The system illustrates how to construct a complete common reporting and analysis tool of their cooperation initiatives. Tagging of projects to the SDGs by project managers allows for filtering by SDG contribution. Artificial intelligence is used to further analyse items for their contribution to SDGs;
  • UNDP IEO showcased its AIDA project (Artificial Intelligence for Development Analytics) which uses 5000+ UNDP evaluation reports to extract and classify findings, conclusions and recommendations. The next phase is to develop the platform to generate insights from UNDP evaluations;
  • KPILens Ghana highlighted an award-winning cloud-based all-in-one M&E solution. It can track and report on KPIs linked to several projects;
  • Le Baromètre (African Institute of Monitoring and Impact Evaluation of Public Policy Programmes and Projects, Benin), showed their award-winning smart device application which provides on-demand SDG progress data relevant for evidence‑based government decision-making at municipal, district and national levels;
  • The Asian Development Bank Independent Evaluation Department demonstrated the use of global datasets of geospatial data for portfolio and economic impact analysis.
• Speakers discussed the use of data-heavy, cutting-edge technologies, which open new pathways for evaluation. Data can be accessed more easily, and made more immediately and easily available to where they can make a change. In addition, technology helps to tap into and make sense of existing rich data and analysis.
• Artificial intelligence offers considerable analytical advantages. The Spanish portal INFOODS2030 is a complete homogenous reporting and analysis tool for their cooperation initiatives. Artificial intelligence is used to further analyse items for their contributions to the SDGs.
• AIDA manages any query on any question with artificial intelligence, and the system is able to recognise relevant data and its classification as finding, conclusion or recommendation. In this way, existing evaluations can be tapped into in a much more meaningful way than was previously feasible.
• Future developments underway in AIDA include sentiment analysis and the creation of insights based on the combination of primary evaluation evidence, programme data and external country context data.
• Cloud-based all-in-one solutions make remote data collection instantly usable through mobile real-time input and analysis. A Ghanaian start-up firm illustrated an all-in-one project M&E solution that combines the logical framework and indicator tracking to allow project managers and other stakeholders to track and access key data in real time.
• Le Baromètre from Benin displayed another smart device based app which shows progress on SDG indicators at municipal, district or country level, and makes it available in a geographical display to policy- and decision-makers.
• Geospatial data are available retrospectively and remotely, which is particularly useful for evaluators working in countries with data access constraints, including those caused by the COVID-19 pandemic and in the context of fragility, conflict and violence. Another advantage is that data quality is comparable across countries.
• However, even with the advances being made in automated analytical techniques, such as those seen through artificial intelligence, including very strong document intelligence capacities, it is important to include humans in the loop to ensure that lessons extracted are both useful and appropriately targeted.

Conclusion

The data revolution is present in evaluation just as everywhere else. This session illustrated how old and new data can be used, and showed how new data science applications can be applied in thoughtful and meaningful ways. Technological innovation is not a panacea but is a tool for a purpose. It must be used with a clear purpose and for quality, and always with “the human in the loop”. Technology also comes with a cost and has a lifespan. The challenge will be who develops, and who will use, the new solutions.