From Inbox Chaos to 80% Auto-Triage: Support Bot Playbook (SME, 6 Weeks)

From inbox chaos to 80% auto-triage in 6 weeks. This case playbook shows exactly how a 12-person SME rolled out a support assistant that classifies emails, drafts safe replies, and logs to the CRM with approvals. Two tracks: Manager Mode for rollout and ROI, Builder Mode for data, prompts, and safeguards.

Manager Mode — baseline → pilot → expand

This section covers the rollout plan, staffing impact, and the KPIs that prove value without boiling the ocean.

Baseline (Week 0)

• Volume: 420 tickets/month (email + contact form).
• Mix: 38% “how-to”, 24% billing, 18% bugs, 12% account changes, 8% other.
• Metrics: median first response 1.8 days; resolution 3.7 days; CSAT 4.2/5.
• Pain: routing mistakes, slow after-hours responses, inconsistent tone.

6-Week timeline (decision-first)

1. Week 1: sample 200 tickets, define intents & macros, pick 3 risky claims to block.
2. Week 2: pilot in shadow mode (no sends) on 25% traffic; measure precision/recall.
3. Week 3: go live for how-to + FAQs with drafts-only; human approve required.
4. Week 4: enable auto-send for low-risk intents < 300€ impact; add after-hours coverage.
5. Week 5: integrate CRM write-back & tagging; add billing templates (draft-first).
6. Week 6: expand to 80% auto-triage, 35% auto-approved drafts; tighten guardrails.

KPIs to track (weekly)

Staffing & ROI (quarterly view)

• Hours saved: 12–18 h/week redirected to complex cases and proactive docs.
• Coverage: bot handles after-hours triage + first replies with next-morning review.
• Cost: LLM + automation €180–450/mo; payback < 60 days at SME volumes.

Builder Mode — data, intents, prompts, approvals, CRM

This appendix gives you the concrete artifacts to reproduce the result.

Intent set (v1)

• HOW_TO, BILLING_GENERAL, INVOICE_COPY, ACCOUNT_UPDATE, BUG_REPORT, SPAM/OUT_OF_SCOPE

Router prompt (classification → JSON)

System: Return ONLY JSON. No prose.
User: Classify the email into one intent from:
[HOW_TO, BILLING_GENERAL, INVOICE_COPY, ACCOUNT_UPDATE, BUG_REPORT, SPAM_OUT_OF_SCOPE]
Also extract: priority (low|normal|high), language (iso-2), customer_id (if present), and redaction flags.
Schema:
{
  "intent":"enum",
  "priority":"low|normal|high",
  "language":"string",
  "customer_id":"string|null",
  "pii_detected":true|false
}
Email:
{{RAW_EMAIL_TEXT}}

Draft reply prompt (deterministic, policy-safe)

System: You produce ONLY JSON matching REPLY_SCHEMA_V1. Do not invent facts or prices.
User:
Context:
- Intent={{INTENT}} • Language={{LANG}}
- Customer tier={{TIER}} • Region={{REGION}}
- Policies: no guarantees; no medical/financial claims; link invoices via secure portal only.
- Include exactly 2 short options if info is missing.
REPLY_SCHEMA_V1 = {
  "subject":"string",
  "body_markdown":"string",
  "needs_handoff":true|false,
  "handoff_reason":"string|null",
  "kb_links":["url","url"]
}
Draft the reply. Return ONLY JSON.

Policy checks (validators)

• Blocked terms: refund guarantees, price promises, medical/financial claims.
• Regex: invoice numbers, order IDs, emails, URLs; reject if malformed.
• PII redaction: mask emails/phones in logs (<EMAIL_1>, <PHONE_1>).

Approval step (one-click)

Subject: Approve draft for {{ticket_id}} ({{intent}})
Auto-checks → policy={{policy_ok}}, pii={{pii_ok}}, links={{links_ok}}
Actions:
✅ Send  |  ✏️ Edit  |  ❌ Reject
Diff: {{review_url}} • JSON: {{json_url}} • SLA: 4h (escalates once)

CRM write-back (safe fields)

• Create/update ticket with intent, priority, tags, and reply status.
• Attach redacted transcript + KB links; never store raw secrets.
• Log approver ID, model name, and prompt version for audits.

Automation outline (Zapier/Make/n8n)

1. Trigger: new email in support inbox.
2. Router: classify → JSON; add priority & language.
3. Draft: generate reply → JSON; attach KB links.
4. Validate: blocked terms, regex, link status 200/3xx.
5. Approve: send single message to Slack/Email for one-click send.
6. CRM: write ticket, tags, and transcript; set next step.
7. Metrics: log outcomes, timings, flags for dashboard.

Dashboard metrics (minimal)

Quality guardrails

• Disable auto-send for BILLING_GENERAL until CS approves templates.
• All ACCOUNT_UPDATE replies require human approval.
• BUG_REPORT drafts must include repro steps link and ticket ID.

FAQ — support bot rollout

What about hallucinations?
We never allow free-text sends. JSON-only outputs go through policy checks and approvals before customers see them.

How do you handle languages?
Detect language at the router; use bilingual templates; require editor spot checks for low-resource languages.

Can we use our CRM macros?
Yes — map intents to macros and keep them in source control with version tags.

What if volume spikes?
Autoscale the draft path; keep approval queue SLAs; after-hours, send acknowledgement only.

Further reading

• CRM Copilots — summaries, next steps, pipeline hygiene.
• Automation Workflows — approvals, archives, and audits.
• Evaluations & Guardrails — tests and filters to keep outputs safe.

Final thoughts

Win fast with a narrow intent set, JSON outputs, and one-click approvals. Once the low-risk lanes are stable, add billing templates and CRM write-backs. That’s how you reach 80% auto-triage in weeks, not quarters.