Share
As part of the Centre’s Peer Review Framework, we worked with independent technical and ethical experts to assess the application of the 510 typhoon impact model as a trigger for anticipatory action by the UN’s Central Emergency Response Fund (CERF) in the Philippines. This is the first time a model used as a trigger in an OCHA anticipatory action framework has gone through our peer review process. The review found that the model has a clear use case with well-defined assumptions and limitations. It also identified aspects that require more explanation or justification. The review recommended the inclusion of additional documentation and model release tracking to improve reproducibility. It further suggested directly addressing false negatives due to rapid intensification.
Anticipatory Action in the Philippines
The Philippines is the world’s most exposed country to tropical storms, with an average of 20 typhoons per year. Since 2016, 510, an initiative of the Netherlands Red Cross, has been developing an impact model for typhoons in the Philippines. The model uses the typhoon path, windspeed, and rainfall forecasts to predict the percentage of damaged buildings per municipality should a typhoon occur. The model has been used as a funding-release trigger for the IFRC’s Disaster Response Emergency Fund (DREF).
In mid-2021, the Centre’s predictive analytics team began supporting the OCHA office in the Philippines with their anticipatory action framework for typhoons. Anticipatory action enables humanitarian organizations to get ahead of a shock and mitigate its impact on lives and livelihoods. Our work with the Philippines office involved designing the trigger mechanism, which included model selection and evaluation, choosing thresholds for the anticipatory action framework activation, and monitoring support throughout the typhoon season. Once a threshold is met, funding is automatically disbursed by CERF and the pre-agreed plan is set in motion.
The 2021-2022 framework aimed to reach over 250,000 people with multi-purpose cash assistance in the regions of Bicol and Eastern Visayas in the event that a typhoon is predicted to cause severe damage to at least 5,000 homes. In that case, CERF would release up to $7.5 million to fund the plan.
In September 2022, the Centre took part in a simulation exercise organized by OCHA Philippines to test the effectiveness of the anticipatory action framework. The main objective was to practice and strengthen the preparedness and coordinated execution of the anticipatory action. For the trigger, the primary goal was to further test and refine the activation protocol and the information that the typhoon impact model will provide to partners. Through this exercise, we were able to fully simulate how agencies will use and interpret the model output during the 2022 typhoon season.
Typhoon Trigger Development
Part of the trigger mechanism design process included the evaluation of a number of models that predicted the risk of major damage due to typhoons. We recommended the use of the 510 model because it could be operationalized quickly (since it was already being used by DREF), it was open source and free to use, and it was (and still is) continually being improved upon.
The Centre and the OCHA office in the Philippines worked with 510 to tailor the model’s trigger thresholds based on the predicted severity of the event and the uncertainty in the model predictions. The anticipatory action framework was endorsed in September 2021. A few months later in December, category-5 Super Typhoon Rai struck the Philippines, damaging 1.9 million homes, killing hundreds, and displacing millions. Unfortunately, due to rapid intensification of the typhoon, the forecasts did not predict the scale of damage until it was too late, and the trigger threshold was not reached. To address this issue in the future, 510 is working on adding additional typhoon-track forecast sources to the model.
Peer Review of the Model
The Centre’s Peer Review Framework aims to create standards and processes for the use of models in the humanitarian sector. The Model Report for 510’s typhoon impact model includes all of the documentation as well as the findings from the technical and ethical reviews.
- Technical Review
The 510 model uses machine learning applied to ensemble typhoon track forecasts from the European Centre for Medium-Range Weather Forecasts (ECMWF) to provide a probabilistic estimate for the percentage of completely damaged houses per municipality. The model is built from several indicators including historical typhoon track, weather and housing damage data. The technical review was performed by Daniela Gawehns, a PhD candidate in the Data Science Research Programme at Leiden University. The results of the technical review can be found in the Model Evaluation Matrix.
- Ethical Review
The model output is the main trigger for activation of the anticipatory action framework in the Philippines. The ethical review was performed by Anulekha Nandi, a PhD researcher in Information Systems and Innovation at the London School of Economics. The results of ethical review can be found in the Ethical Matrix.
“Thanks to the peer review, we’ve been able to identify key actions that will enhance the model for use as a trigger by both DREF and CERF in 2022.”
-Aklilu Teklesadik, Lead Developer at 510
Main Findings and Next Steps
A key finding of the review is that additional documentation is needed to better justify or explain some aspects of the model. For example, expanded rationale for the damage thresholds, more information about the sources of input data and inclusion of the latest performance scores would be welcome additions. Other recommendations include creating a model release for each version and keeping an updated list of associated input data. This would make reproducibility of earlier results easier and help keep track of the data sources.
The review uncovered a crucial issue regarding false negatives. In the cases of typhoons Goni in 2020 and Rai in 2021, the observed wind speed rapidly increased less than 24 hours before landfall. The anticipatory action framework was therefore not activated as this phenomenon, known as rapid intensification, was not forecasted in time by the models available. For meteorological models, forecasting rapid intensification of tropical cyclones remains an operational challenge, mainly due to the incomplete understanding of its physical mechanisms. The possibility of rapid intensification is not addressed directly in the anticipatory action framework or in the model implementation plan. The recommendation is to plan for it since it may occur routinely.
These findings and recommendations have been an important step towards improving the typhoon model and anticipatory action framework in the Philippines. Applying the peer review process to current and future trigger mechanisms will enable the Centre to continuously improve and monitor the quality of OCHA’s frameworks as we scale up for new shocks and contexts.