What a million conversations taught us about secure GenAI in government

From the CDC EDAV program · JRSS Engineering

Most federal GenAI conversations stall at the same place: the mission wants the capability, and the security office — correctly — refuses to let sensitive data anywhere near a public model endpoint. On CDC's Enterprise Data Analytics and Visualization (EDAV) platform, JRSS engineers helped resolve that tension the only way it actually resolves: by building the assistant inside the boundary instead of asking the boundary to move.

The result is an internal AI assistant that has now handled over a million interactions for public health staff — summarizing documents, accelerating analysis, and answering questions against agency knowledge — while every prompt and every response stays on the agency's own network.

Decision one: the model comes to the data

The assistant runs on Azure OpenAI provisioned inside the agency's cloud tenancy, on dedicated compute. Nothing a user types is sent to a public consumer endpoint, used to train a vendor's model, or commingled with another customer's traffic. That single architectural decision converted GenAI from a policy problem into an engineering project.

Decision two: retrieval, not memorization

A model that answers from its training data will eventually answer confidently and wrongly — unacceptable in a public health context. The assistant instead uses retrieval-augmented generation (RAG): Azure AI Search indexes the agency's own approved content, the relevant passages are retrieved at question time, and the model is instructed to answer from that material. Answers stay grounded in sources the agency controls, and stale knowledge is fixed by updating an index, not retraining a model.

Decision three: log everything, from day one

Every conversation is logged. That was designed in before launch, not retrofitted after an incident — and it changed the character of the rollout. Security reviewers could audit real usage instead of speculating about it; the platform team could see which questions the assistant handled well and where it needed better source material; and leadership could measure adoption with real numbers rather than anecdotes. A million logged interactions later, the audit trail is also the product roadmap.

Decision four: ship to a platform, not a pilot

The assistant succeeded partly because it did not launch into a vacuum. EDAV already gave it an authenticated user base, a governed data estate, and a DevSecOps pipeline with security controls built into every release. GenAI added a capability to a trusted platform — which is a much easier accreditation conversation than standing up a new system and asking an agency to trust it.

Where this goes next: from assistants to agents

An assistant that answers questions is the first chapter, not the whole book. JRSS teams are now building the second chapter: agentic RAG systems that plan multi-step retrieval and tool use instead of answering from a single lookup; enterprise data assistants that let program staff interrogate governed data in plain language; and SQL data assistants that translate a question into a correct, permission-aware query — so the analyst sees the answer and the query that produced it. The engineering foundation is the same on either cloud — we work across Azure AI Foundry and AWS Bedrock Agents — and so are the rules that made the first million conversations safe: grounded answers, logged actions, and a human in charge of anything consequential.

What we tell agencies now

Secure GenAI in government is not blocked on model quality — today's models are more than capable. It is blocked on architecture: where the model runs, what grounds its answers, what gets logged, and what platform it inherits its security posture from. Get those four right and the mission side stops asking whether they may use AI, and starts asking what to point it at next.