build.allanninal.dev

Part 4 of 7 · Content moderator series ~5 min read

How flagged content reaches a moderator

The checker held an item or marked it borderline. Now something has to get it in front of a person — the right person, on the right channel, at a sane time, with the reason already attached. Get any of those wrong and the review is worse than none: a 2am ping, a card that doesn’t say why, fifty separate alerts for one spammer’s fifty comments. Four small guardrails sit between the verdict and the card landing.

Key takeaways

  • Held and borderline items go into a review queue (an SQS queue with a dead-letter backstop).
  • Reviewer resolution: per-area owner beats the configured admin fallback.
  • Quiet hours batch normal items into a digest; only “hold + notify” pages at once.
  • Repeat offenders are grouped into one card, so one spammer doesn’t mean fifty pings.
  • Every card ships with the item, the verdict, the confidence, the rule, and three buttons.

Four guardrails on every review card

Four guardrails between the held verdict and the review card A horizontal flow diagram. On the far left, a "Verdict ready" box: the checker emitted an event with the item, the area, the rule cited, and one of the sending verdicts — hold, send-to-human, or hold-plus-notify. Four guardrail gates sit in a row to the right, each drawn as a vertical bar. Gate 1: Resolve reviewer — looks up the per-area reviewer from the rules doc first; if blank, falls back to the configured admin; returns a Slack member ID if one is set, otherwise an email address. Gate 2: Quiet hours — checks the local time against the rules-doc quiet-hours window; normal holds and borderline items are batched into the next digest rather than pinged; only a hold-plus-notify breaks through to page at once. Gate 3: Group repeats — checks whether the same author already has open items in the queue; if so, the new item is folded into one grouped card so a single spammer doesn't generate one ping per comment. Gate 4: Final compose — builds the review card from the voice-doc template: the original text, the verdict, the confidence, the exact rule cited, a link to where the item lives, and three buttons — publish, remove, edit-and-publish. After all four gates pass, the card ships via Slack Block Kit for the resolved Slack ID, or via SES outbound email with action links for the resolved email. A note at the bottom: every card is logged so the queue knows it was sent and a second pass won't double-send. Verdict ready hold, send-to-human, or hold + notify Gate 1 Resolve reviewer per-area owner? admin fallback? prefer Slack ID, fall back to email Gate 2 Quiet hours local time in quiet window? if so, batch to next digest (notify breaks through) Gate 3 Group repeats same author already queued? if yes, fold into one grouped card Gate 4 Compose + reason voice template for the area + verdict, rule, confidence, link, three buttons Review card — Slack DM (preferred) or SES outbound email (fallback) Block Kit buttons for Slack · action links for email every card logged to DDB cm-queue — a second pass won’t double-send Every gate is a deterministic check — no model calls, no surprise pings at midnight.
Fig 4. Four guardrails between the verdict and the review card. Resolve the reviewer. Honor quiet hours. Group repeat offenders. Compose with the full reason. Then ship via Slack or email and log the card so the queue doesn’t double-send.

The review queue

Held and borderline items don’t go straight to a person; they go into a queue first. The queue is an SQS queue — a simple, durable line of work that holds items until something processes them, so nothing is lost if a delivery hiccups. Behind it sits a dead-letter queue: if an item somehow fails to send a few times in a row, it drops into that backstop instead of vanishing or looping forever, and the weekly digest flags it. A held item is out of public view the moment the verdict lands; the queue just decides when and how the card reaches a moderator.

Gate 1: resolve the reviewer

Two places the dispatch looks for who owns an item, in order. First, the per-area reviewer in the rules doc — “all review-page items go to Priya, all community-post items go to Sam.” Second, the configured admin fallback, the person who set the system up and gets anything unowned. The fallback should rarely fire; when it does, the weekly digest names every item that hit it so the rules doc can be fixed.

Once the reviewer is known, the dispatch looks up how to reach them. The voice doc maps each reviewer to a Slack member ID if one is set, otherwise to an email address. Slack is preferred — a private message with action buttons is faster to act on than an email link — and email is the fallback so nobody is left out.

Gate 2: quiet hours and batching

Items arrive at all hours, but a moderator shouldn’t. Gate 2 reads the rules doc’s quiet-hours setting (default 8pm to 8am, configurable). During quiet hours, a normal hold or borderline item isn’t pinged — it’s batched into the next digest, a single grouped message that goes out when business hours start. Since a held item is already out of public view, nothing is harmed by waiting a few hours for a person to confirm.

The one exception is a hold + notify — the severe call from Part 3. Those break through quiet hours and page the on-call reviewer at once, because a credible threat or a doxxing post is the rare case where waiting until morning is the wrong trade. Everything else respects the window.

Gate 3: group repeat offenders

Spam comes in bursts. One bot can drop fifty near-identical comments in a minute. Fifty separate review cards would bury the moderator and make the real borderline items impossible to find. Gate 3 checks whether the same author already has open items in the queue; if so, the new item is folded into one grouped card — “this author has 50 held comments, all matching the same blocked-link rule” — with a single set of buttons that can publish or remove the whole group at once. One spammer, one card.

Gate 4: compose with the full reason, then ship

The voice doc has one review-card template per area. The dispatch fills it with the original text, the verdict, the confidence, the exact rule the checker cited, and a link to where the item lives. It attaches three buttons — Publish, Remove, Edit & publish — and ships the card via Slack’s Block Kit (the format Slack uses for messages with buttons). For email fallback, the same fields are wrapped in a small HTML email whose buttons are action links that hit a Function URL.

Every card — Slack or email, single or grouped — writes a row to cm-queue in DynamoDB marking that it was sent. A second pass over the queue reads that row and knows not to send the same card twice.

Why the guardrails exist

None of these gates are clever. They’re the small care a thoughtful person would take if they were handing off the flags themselves — send it to whoever actually owns this area, don’t ping at midnight, don’t fire fifty alerts for one bot, and include enough that the reviewer doesn’t have to go dig for the reason. Putting them in code as four small gates makes them part of the design, not something you’re hoping the writer of any one alert remembers.

Next post: what happens when the moderator taps a button — publish, remove, or edit-and-publish — how the item comes down (or goes up), and how every overturn teaches the system the better call.

All posts