build.allanninal.dev

Part 5 of 7 · Survey analyzer series ~5 min read

How an urgent answer gets flagged

A weekly summary is the right speed for most feedback. It is exactly the wrong speed for the customer who just wrote “third time the order was wrong, I’m done, cancel my account.” That answer can’t wait five days to show up in a tidy theme. So every answer, the moment it lands, runs a quick urgent check — and anything that trips jumps the whole queue and goes straight to a human. This post walks through what the check looks for, what happens when it fires, and why a real person, not the system, decides what to do next.

Key takeaways

  • Every answer runs a quick urgent check at intake — a small, cheap Claude Haiku 4.5 call.
  • It looks for an angry customer, a threat to leave, a safety or legal concern, or a callback request.
  • If it trips, the answer is sent to a human within a minute via SNS — email or Slack.
  • The system only flags and routes; the human decides what to actually do.
  • Every flag is logged, and the week’s flags are appended to the weekly summary too.

Three outcomes of the urgent check

Three outcomes of the urgent check on a new answer A diagram showing one input on the left flowing through a quick urgent check, then branching into three outcome paths. Far left: a "New answer at intake" box showing a just-arrived answer — its text, survey, date, and rating — about to be checked. The intake runs a small Claude Haiku 4.5 call that asks whether this answer needs a human today. The middle column shows the three outcomes. Outcome one, Notify now: the answer looks angry, threatens to leave, or raises a safety or legal concern; the analyzer publishes to the SNS urgent topic right away, which fans out to the on-call person by email and Slack, carrying the full answer text and a short note on why it tripped. The answer is still stored and still grouped later, but the human has it within a minute. Outcome two, Callback queued: the answer asks to be contacted but isn't a crisis; it's added to a short callback queue the on-call person clears once a day, rather than paging anyone. The answer is also stored and grouped as normal. Outcome three, Normal: the answer is ordinary feedback; nothing fires now, and it simply waits in the store for the weekly grouping pass. The right side shows the convergence: every outcome writes a row to the sa-flags DynamoDB table with timestamp, answer id, outcome, reason, and who was notified. A note at the bottom: the system only flags and routes — a human decides what to do, and every flag is logged. New answer at intake text, survey, date, rating [Haiku urgent check] needs a human today? one cheap call Outcome 1 Notify now • Angry, leaving, safety or legal concern • SNS → email + Slack • Full text + reason Outcome 2 Callback queued • Asked to be contacted, not a crisis • Added to callback list cleared once a day Outcome 3 Normal • Ordinary feedback nothing fires now • Waits in the store — grouped weekly Flag log DynamoDB sa-flags timestamp · answer_id outcome · reason notified The system only flags and routes — a human decides what to do. Every flag is logged.
Fig 5. One quick check at intake, three outcomes. Notify now pages a human within a minute. Callback queued holds it for a daily sweep. Normal just waits for the weekly pass. Every outcome writes to the flag log.

The check itself: one small call at intake

When an answer lands — from any of the three lanes in Part 2 — the intake runs a single Claude Haiku 4.5 call before the answer settles in the store. The prompt is short and the rules doc supplies the definition of urgent, so the business can tune it: “Read this survey answer. Does it need a human today? Answer urgent, callback, or normal, and give a one-line reason. Urgent means an angry customer, a threat to cancel or leave, a safety or legal concern, or anything that reads like a complaint that will escalate. Callback means they asked to be contacted but it isn’t a crisis. Otherwise, normal.” It’s Haiku, not Sonnet, because this runs on every answer and the judgment is a simple sort, not an essay — the cheap model is both fast enough and good enough.

The check looks only at the one answer in front of it. It doesn’t need the embeddings, the clusters, or the rest of the week — it’s a fast triage at the door, deliberately kept separate from the slow, thorough grouping that happens later.

Outcome 1: notify now

If the answer trips the urgent bar, the analyzer publishes it to an SNS topic immediately. SNS fans the message out to whoever the rules doc named as on-call — an email address, a Slack channel, or both. The message carries the full answer text (not a summary — the human needs the real words), the one-line reason it tripped, the survey and date it came from, and any rating. The whole path from “answer lands” to “phone buzzes” is well under a minute. The answer is still stored and still grouped into a theme the following week; flagging it urgent doesn’t pull it out of the normal picture, it just adds a fast lane on top.

Crucially, the system stops there. It does not reply to the customer, does not offer a refund, does not promise anything. It hands a real person the words and the reason and gets out of the way. The most expensive mistake a feedback system can make is auto-answering an angry customer with something tone-deaf, so this lane is flag-and-route only — a human owns every word that goes back.

Outcome 2: callback queued

Plenty of answers ask for contact without being a crisis — “could someone call me about my account?”, “happy to chat more if useful.” Paging the on-call person for those would train them to ignore the alerts, which is the worst outcome of all. So a callback answer goes onto a short list that the on-call person clears once a day rather than firing an instant alert. It’s the middle speed: faster than the weekly summary, calmer than a page. The answer is stored and grouped as normal too.

Outcome 3: normal

The common case. Most feedback is ordinary — a compliment, a small gripe, a suggestion — and nothing needs to happen today. The answer simply waits in the store for the weekly grouping pass, where it becomes part of a theme and a count. No alert, no queue, no noise. Keeping this the default is what makes the urgent lane trustworthy: because the bar is set so that most answers are normal, an alert actually means something when it fires.

Why a human always decides, and why every flag is logged

Two design rules hold this lane together. First, the system never acts on the customer’s behalf — it surfaces and routes, and a person decides what to do. That keeps the analyzer firmly in “read-only” territory: it can’t send a bad reply because it can’t send a reply at all. Second, every outcome — urgent, callback, or normal — writes a row to the sa-flags table with the timestamp, the answer id, the outcome, the reason, and who was notified. That log is what lets you tune the rules (“we’re getting paged too often — tighten the definition”), prove a complaint was seen the day it arrived, and append the week’s flags to the bottom of the summary so the slow and fast lanes meet in one place.

Next post: the cost breakdown. The whole pipeline above — embeddings, a weekly grouping pass, a per-answer urgent check, and one summary — runs in coffee-money territory at SMB volume; Part 6 explains exactly where the dollars go.

All posts