Share
The Foresight model from the Danish Refugee Council (DRC) uses open data from 18 sources to predict forced displacement of internally displaced people and refugees, with an initial focus on Afghanistan and Myanmar but more recently applied to the Sahel. In July, DRC worked with the Centre to assess the technical rigor and operational readiness of the model. Below, we share the Model Card and Model Evaluation Matrix, which are the first steps in our Peer Review Framework. We are grateful to the DRC team for helping us to operationalize this process.
The Centre’s Peer Review Framework aims to create standards and processes for the use of models in our sector. It consists of five steps: model submission, technical review, implementation plan submission, ethical review, and final report. There are two types of models that can be reviewed: those that are fully developed and can respond to the requirements of the stakeholders who will implement the related actions, and those that are exploratory. For the latter, it is not possible to evaluate the ethical risks as the actions that the model would generate are not yet defined. This is the case for the DRC model and for this reason, we only performed steps one and two of the Framework.
Step 1: Model Submission through the Model Card
DRC and IBM, their technical partners in developing the model, completed a Model Card, which includes information about the intended use, model development, model evaluation, and operational readiness.
Foresight is a machine learning and bayesian network model for predicting total forced displacement from a given country one to three years ahead. It provides what-if capabilities for end-users to see the potential impact on displacement if selected drivers change and includes an analysis of the different factors driving displacement. Like most machine learning approaches, the main assumption is that past relationships are valid for the future. Read more details about Foresight in the Model Card.
Step 2: Technical Review
For the technical review, we connected the modelers from DRC and IBM with the team at UNHCR Innovation. UNHCR has valuable experience in predicting the movement of displaced populations within and outside of Somalia, based on their Jetson model. The Centre’s Predictive Analytics team facilitated the discussion between the UNHCR Innovation team and colleagues from DRC and IBM on the details of the model. The output of the UNHCR Innovation’s review is summarised in the Model Evaluation Matrix, which indicates the strengths and weaknesses of the model and identifies the scope within which it should be considered for anticipatory action.
The review process highlighted how the model’s scope is well understood and documented. In addition to providing displacement estimates, the model provides insights on the root causes of displacement and can inform humanitarian response at the national level. Due to limitations in the data, the model is not able to capture dynamic or sudden-onset displacement situations.
Next steps
Since we completed the review in August, the Foresight model has been further expanded to include the impact of COVID-19 on displacement in the Sahel region (see coverage in The Guardian). Based on the recommendations that emerged from our evaluation, the DRC and IBM teams are refining the model and targeting specific decision-makers to identify the best use cases. They plan to expand the model to cover more countries with displacement dynamics, and they are exploring how it could inform the annual humanitarian planning cycle. Once the model is ready to be implemented with stakeholders, the Centre will conduct an ethical review and issue a final report for the peer review process.
Based on our experience with DRC, we will continue to apply the steps from our Peer Review Framework with other partners. We welcome feedback on the format of the Model Card, which builds on the work done in other sectors, and the Model Evaluation Matrix. Are these products useful and is the language accessible? How can they be improved?
Over the coming months, the Centre will be setting up a reviewer pool for our peer review process so that we can support partners with the many models being developed for humanitarian use (see our catalogue of models). We are also engaging with epidemiologists and ethical experts to review our own ‘OCHA-Bucky’ predictive model for COVID-19 spread and mitigation in humanitarian crises, created with Johns Hopkins University Applied Physics Laboratory. We are hosting a virtual event on 28 October to mark the public release of the code and documentation for the model (register here).
Special thanks to Alexander Kjærum and Bo Schwartz Madsen from DRC, Rahul Nair from IBM and the UNHCR Innovation team. We look forward to working with DRC and other partners to increase the use of trusted models in humanitarian operations.