Forecasting outbreaks on Azure: HPC lessons from CDC's forecasting center

From CDC outbreak-analytics work · JRSS Engineering

Outbreak science has a brutal compute profile. For weeks, demand is modest. Then a pathogen emerges — mpox, a new COVID variant — and modelers need to run large ensembles of epidemiological simulations, process genomic sequences, and turn around forecasts while the answers still matter. Supporting CDC's outbreak-analytics and forecasting mission, JRSS engineers built for exactly that spike-shaped world on Azure.

Elasticity is the requirement, not a bonus

A fixed cluster sized for the surge wastes money for most of the year; one sized for the average fails during the only weeks that count. Azure Batch resolves the dilemma: simulation workloads fan out across pools of compute nodes that exist only while the job runs. Millions of records per run stop being an infrastructure negotiation and become a job parameter.

Pipelines that keep up with the data

Forecasts are only as fresh as their inputs. Databricks with Delta Live Tables gives the ingestion side declarative, continuously validated pipelines — massively parallel processing that keeps surveillance and laboratory feeds moving without an engineer hand-tending every load. When source data arrives late, malformed, or revised (as outbreak data always does), the pipeline's built-in expectations catch it before a bad number reaches a model.

Reproducibility is part of the science

A forecast that informs public decisions must be re-runnable: same code, same data, same environment, same result. Domino Data Lab gives modelers governed workspaces where every run is captured with its environment and inputs. When results are questioned — and in outbreak response they are always questioned — the answer is a re-execution, not a reconstruction from memory.

Cost discipline makes the mission sustainable

HPC in the cloud can fail financially even while succeeding technically. Consolidating scattered compute onto shared, governed infrastructure — right-sized pools, spot capacity where interruption is tolerable, storage tiered to access patterns — delivered meaningful savings while increasing available capacity. The uncomfortable truth of cloud HPC is that architecture, not negotiation, is where the money is.

What transfers beyond public health

Any agency with burst-shaped analytics — fraud sweeps, climate models, sensor reprocessing, end-of-quarter risk runs — faces this same profile. The pattern that works is consistent: elastic fan-out compute, declarative validated pipelines, reproducible-by-default model execution, and cost treated as a first-class engineering requirement. That is what lets a mission ask bigger questions the day the emergency starts.