Prompt Versioning & Governance: IDs, Tests, and Rollbacks

Prompts are product, not notes. This guide shows how to version, test, approve, and roll back prompts with the same rigor you use for code. Two tracks: Manager Mode for ownership and KPIs, Builder Mode for IDs, schemas, test harness, and release checklist.

Manager Mode — ownership, approvals, and KPIs

Use this section to define who owns prompts, how changes get approved, and how you measure impact.

Ownership & approvals

• Prompt Owner: accountable for performance and updates.
• Editor/Reviewer: signs off on tone, claims, and compliance.
• Change gates: minor (owner + auto tests) • major (owner + reviewer + pilot).
• Rollback policy: any regression over threshold auto-reverts to last good tag.

The table below maps change types to required checks and who approves for quick governance.

Success metrics (blend product + safety)

• Task accuracy: % outputs meeting acceptance criteria on eval set.
• Cycle time: trigger → approved output (P95).
• Edit rate: human edits per 10 outputs.
• Safety flags: blocked terms / PII hits per 1k runs.
• Business KPIs: replies sent, leads qualified, articles shipped.

Builder Mode — IDs, schemas, tests, rollout, rollback

This appendix gives you the artifacts to run prompts like code: consistent IDs, a JSON spec, a test harness, and a release checklist.

Prompt ID + versioning schema

Use stable IDs per task and semantic versions. Keep metadata together for audits and quick rollbacks.

{
  "prompt_id":"EMAIL_REPLY",
  "version":"1.3.0",
  "owner":"alex@yourco.com",
  "description":"Reply to inbound support emails with policy-safe tone.",
  "inputs":{"ticket_text":"string","customer_tier":"gold|std","language":"en|da"},
  "output_schema":"EMAIL_REPLY_SCHEMA_V2",
  "safety":{"blocked_terms":["refund guarantee","medical cure"],"pii_masking":true},
  "dependencies":{"retrieval_profile":"support_kb_v4","tools":["ticket_lookup","kb_search"]},
  "changelog":[
    {"version":"1.3.0","date":"2025-10-24","notes":"Added tone softeners; tightened refund claim guard"},
    {"version":"1.2.0","date":"2025-09-15","notes":"JSON schema V2 with escalation_reason"}
  ],
  "links":{"eval_set":"gs://.../email_reply_eval.csv","dashboard":"https://.../dash"}
}

Output schema (strict JSON)

Lock the output shape to avoid downstream errors and make evals deterministic.

EMAIL_REPLY_SCHEMA_V2 = {
  "reply_text":"string",
  "tone":"friendly|formal|neutral",
  "escalation_required":"boolean",
  "escalation_reason":"string|null",
  "citations":[{"title":"string","url":"string"}],
  "metadata":{"language":"string","model":"string","prompt_id":"string","version":"string"}
}

Deterministic generation prompt

This frame forces JSON-only outputs and ties runs back to the prompt/version for auditability.

System:
You output ONLY valid JSON matching EMAIL_REPLY_SCHEMA_V2. If info is missing, set null. No extra text.
User:
Task: Draft a safe support reply.
Inputs:
- Ticket: {{ticket_text}}
- Customer tier: {{customer_tier}}
- Language: {{language}}
Constraints:
- No guarantees. No medical/financial claims.
- Cite sources if quoting policy lines.
Schema: {{EMAIL_REPLY_SCHEMA_V2}}
Return ONLY JSON.

Eval harness (unit + regression)

Run this small suite before promoting a new version. Track results by prompt_id and version.

Safe rollout (A/B + canary)

• Tag: promote EMAIL_REPLY.v1.3.0 → stable for 10% traffic.
• Monitor: live KPIs (accuracy proxy, edit rate, flags) for 48 hours.
• Promote: if thresholds pass, ramp to 50% then 100%.
• Rollback: one-click revert to last stable tag; auto-notify stakeholders.

Release checklist (copy-paste)

Use this list every time you ship a new prompt version.

• ✅ Changelog updated with version/date/notes.
• ✅ Eval harness green (schema, blocked terms, key metrics).
• ✅ Safety review for claims and PII handling.
• ✅ Canary plan + rollback link in the ticket.
• ✅ Dashboard updated to filter by prompt_id/version.

Storage pattern (folder layout)

Keep prompts and evals co-located so versions don’t drift from tests.

/prompts/
  EMAIL_REPLY/
    v1.3.0/
      prompt.json
      schema.json
      evals.csv
      blocked_terms.txt
      changelog.md
    v1.2.0/
      ...

Observability fields (log per run)

• prompt_id, version, model, ts.
• policy_flags: pii_hit, blocked_term, jailbreak.
• review: edited_yes/no, editor_id, edit_reason.
• latency_ms, tokens_in/out, tenant_id (if multi-tenant).

FAQ — prompt versioning & governance

Do I need semantic versioning?
Yes — it clarifies risk and review depth. Patch = fixes, minor = behavior tweaks, major = structure/logic changes.

Where should prompts live?
In your repo or a prompt registry — not random docs. Treat them like code with reviews.

What if models change underneath?
Pin model names/versions in metadata; re-run evals on provider changes and log deltas.

How big should the eval set be?
Start with 30–100 cases per prompt; grow over time with real-world failures.

Who approves major changes?
Owner + Reviewer + a business stakeholder who owns the outcome.

Further reading

• Prompt Playbooks for Teams — reusable prompts for briefs, emails & QA.
• Evaluations & Guardrails — reduce hallucinations with tests & filters.

Final thoughts

Give every prompt a stable ID, strict schema, and versioned changelog. Ship with evals, monitor live KPIs, and be ready to roll back in one click. That’s how prompts become reliable product components — not fragile one-offs.