In 2024, the Humanitarian Data Exchange (HDX) will reach its 10th anniversary. Launched in July 2014 with just 800 datasets and a dozen organizations, HDX is now an anchor in the humanitarian data ecosystem, bringing together 20,000 datasets from hundreds of partners. To ensure the next ten years are as productive as the last, we have developed a product roadmap that creates new opportunities for faster access to reliable data and insight about humanitarian crises.

Over the years, the organizations on HDX have become increasingly sophisticated in how they contribute data, with more of them using automated data sharing methods rather than manually uploading files. The same can be said for HDX users who increasingly want to access data through frictionless services rather than by going to the site to download files. At the same time, while our team has been able to manage ever greater quantities of data passing through the platform, a lot of our maintenance work still relies on manual processes, which slows things down considerably. 

With the twin goals of sophistication and automation, the HDX roadmap will improve the platform in a number of ways, allowing our team and our users to handle data at scale. In collaboration with argo, a global design firm, and based on extensive user research, we have coalesced on five critical opportunity areas and sequenced the work over the next 18-24 months. The areas include: data as a service, geodata services, onboarding, notifications and workflow management. 

Crawl, walk, run, fly

The roadmap starts by addressing foundational aspects of our infrastructure and process. For example, we are creating the ability to detect changes between dataset versions, which will enable smart notifications to our team (initially) and users (eventually) about what needs to be checked or updated on the site. We will also get the most out of internal tools, such as Jira, to work more efficiently across our teams. These improvements will allow us to set up an operation that is future ready and able to tackle more complex undertakings like data as a service and automated data quality checks.

Below is a summary of the five aspects of our roadmap.

  • Data as a service – Advancing programmatic access to reliable and relevant humanitarian data

HDX will leverage its role as a central platform to provide the most essential data in a common system, simplifying programmatic access to data. This new ‘humanitarian API’ (HAPI) will provide a consistent, standardized and machine-readable interface to retrieve data from a set of high-value, subnational humanitarian indicators drawn from HDX contributors. The data will be available through the API for use in applications, search engines and business intelligence tools. It will also power our own HDX Data Explorers. The initial scope of HAPI will be the data included in the HDX Data Grids, which cover all locations with a humanitarian response plan. 

  • Geographic data services – Expanding HDX’s geo-specific features 

Geographic data is shared on HDX in a variety of formats. It is also the most popular data on the platform, year after year. We will automatically convert this data into industry-standard geodata formats and machine-readable services. These services can be added to mapping applications on the web or on users’ desktops, providing continually updated geographic information for situational awareness in humanitarian operations. We will also leverage existing standards like P-Codes (place codes) to provide geographic views of tabular datasets on HDX. 

  • Onboarding – Providing contextual help and content for an enhanced user experience

As HDX has grown, so has the need to help users get what they need from the platform. We will provide guidance during key moments, such as account setup, organization creation and publishing data, to help users understand the platform’s features. We will also improve the onboarding process to clarify the value proposition for becoming a registered user and to reduce false positives around requests to create an organization.

  • Notifications – Delivering automated smart notifications tailored to user preferences

Tailored notifications will lead to higher user engagement and help our community stay informed about data that’s critical to them. We will provide customizable notifications for datasets, locations, crises, data contributors, or themes, and deliver these as emails or text alerts, depending on user preference. We anticipate that this proactive data push will be particularly valuable for non-technical users or senior humanitarians who want the latest figures about a crisis but may not necessarily visit the HDX site. We will also re-design the user dashboard so it is more intuitive for registered users. 

  • Workflow management – Centralizing and automating operations for better workflow efficiency

Every dataset that is added or updated on HDX goes through a quality assurance (QA) process that entails a number of checks, such as whether the metadata is complete and the files open or if there is personal or sensitive data included. The existing QA workflow has become difficult to manage due to its growing complexity. We will streamline this process, reducing manual tasks so the HDX team can prioritize high-value partnerships with data contributors. We will also deploy workflow tools to provide transparency into the status of tasks and to automate repeatable processes. These efforts will lay the groundwork for further enhancements to measuring the quality of data on the site, something we are researching with our friends at the Data Nutrition Project.

Get in touch 

The HDX roadmap will guide our team’s work over the coming two years. During this time, we will undertake continuous user research to make sure we are meeting demand and releasing new features at optimal moments. If you have ideas on how to improve HDX or would like to be interviewed, fill in this form and we will be in touch. 

We would like to thank our partners at argo who worked with us for many months to help refine our priority areas for improvement and sequence the development work. Their deep knowledge of technology and system design informed our thinking about the direction of HDX and we are grateful for their continued collaboration and advice. You can learn more about our work with argo in this case study. 

For those of you who would like more details about the HDX roadmap, you can look at these slides. Contact us with questions or feedback at hdx@un.org.