Keeping four petabytes analysis-ready: medallion architecture at CDC scale

From the CDC EDAV program · JRSS Engineering

Every modern data platform pitch includes the same diagram: raw data lands in a bronze layer, gets cleaned and conformed in silver, and is served analysis-ready in gold. On CDC's EDAV platform, JRSS engineers live with that diagram at genuine scale — 4+ petabytes of public health data organized under a governed ingestion framework, feeding analytics for thousands of users across hundreds of projects. At that scale, the diagram is the easy part.

Bronze is a contract, not a dumping ground

The bronze layer preserves source data exactly as received — which sounds passive but is actually a promise: any downstream dataset can be rebuilt, any anomaly can be traced to what actually arrived, and no cleanup step silently destroys evidence. In public health, where a surveillance feed may be re-examined years later, that reproducibility is not a nicety. It is the difference between an answer and a shrug.

Silver is where governance is enforced

Standardization, validation, deduplication, and conformance all happen in the silver layer — through governed pipelines, not ad-hoc scripts. EDAV runs on the order of a hundred-plus Azure Data Factory pipelines with Databricks doing the heavy transformation work. The rule that keeps the estate trustworthy is simple to state and hard to hold: data changes only through the pipeline. If an engineer can hand-edit a silver table, the platform has no silver layer — it has folders.

Gold is shaped by questions, not by sources

Gold datasets are modeled around what programs actually ask — case trends, coverage, lab throughput — not around the quirks of whichever system supplied the data. That is also where access control earns its keep: Power BI dashboards with row-level security mean a state health department and a national program office can use the same governed dataset and each see exactly what they are entitled to see.

Performance is a feature of architecture

Two examples of what disciplined layering buys. Storage tiering on ADLS Gen2 — hot, cool, and archive matched to actual access patterns — keeps petabyte-scale storage costs rational without sacrificing availability. And when a critical dashboard's refresh time was cut from over eighty seconds to thirty-to-forty, the fix wasn't a bigger capacity SKU: it was restructuring the gold model the dashboard sat on. At scale, most performance problems are modeling problems wearing a disguise.

The lesson for any agency

Medallion architecture works in government not because the pattern is clever but because it assigns responsibility: bronze answers "what did we receive," silver answers "what do we trust," gold answers "what do we tell the mission." An agency that can answer those three questions on demand has a data platform. One that can't has a very large bill for storage.