build.allanninal.dev

Part 5 of 7 · Website chat assistant series ~4 min read

How gaps become better answers

An assistant that can’t answer a question isn’t a failure — an assistant that can’t answer the same question in two months is. Here’s the small loop that turns every miss into a paragraph in your docs, and turns next month’s identical question into a clean auto-reply.

Key takeaways

  • Four-step clockwise loop: log every miss, group similar questions, write a paragraph, re-index automatically. Ten minutes a week.
  • Every clarify, hand off, and non-hostile decline appends a row to tbl-gaps with the visitor turn, page URL, timestamp, and the closest passages found — no extra PII.
  • A weekly EventBridge cron (0 6 ? * MON *) groups gaps by Titan Embeddings cosine similarity so “ship internationally?” and “order from the UK?” collapse into one row with a count.
  • Writing standard: a sentence the assistant could quote word-for-word and have it sound right. No marketing voice, no hedging.
  • Re-indexing is automatic — the Bedrock managed KB watches the Drive folder via fn-drive-sync and picks up new paragraphs in about ten minutes (5 to sync, 5 to index). No deploy.

The loop, in one diagram

A circular four-step learning loop: log, group, write, re-index A loop with four boxes arranged clockwise. Top: "Log every miss" — when a turn ends in clarify, hand off, or decline, the question, the page URL, and the closest passages found are appended to a Drive doc called the gaps log. Right: "Group similar questions" — once a week, a small batch job groups gaps with similar wording so duplicates collapse into one row. Bottom: "Write a paragraph" — a human spends ten minutes adding short answers to the knowledge folder, one paragraph per group. Left: "Re-index automatically" — the managed knowledge base picks up the new paragraphs within minutes; nothing to deploy. Arrows form a clockwise circle with a small label "weekly" under the gap between top and right. A note in the centre reads: "Each loop turn upgrades the assistant. Month two answers the questions month one missed." Step 1 Log every miss turn + page + closest passages appended to gaps log Step 2 Group similar duplicates collapse into one row, by wording Step 3 Write a paragraph human, ten minutes, straight into the Drive folder Step 4 Re-index automatically managed KB picks up new paragraphs in minutes weekly Each loop turn upgrades the assistant. Month two answers what month one missed.
Fig 5. A small clockwise loop. Misses become groups, groups become paragraphs, paragraphs become better answers.

Step 1: log every miss, automatically

Every time the answerer picks “clarify,” “hand off,” or “decline” for a question that wasn’t hostile, the cloud appends a row to a small Drive doc — the gaps log. Each row is short: the visitor’s message, the page URL, the timestamp, and the closest passage the search did find (which might just be a near miss). Nothing else — no PII the visitor didn’t paste in themselves, no tracking IDs.

The reason this lives in Drive and not a database is simple: the team already lives in Drive. No dashboard to log in to, no permissions to set up. Open the doc, read this week’s misses, write the answers, save — that’s the whole workflow.

Step 2: group similar questions

Most weeks, the gaps log isn’t a long list of unique problems — it’s a short list of repeated ones, each asked by a few different visitors in slightly different words. “Do you ship internationally?”, “Can I order from the UK?”, “Where do you ship to?” are the same question dressed differently.

A small batch job runs once a week and groups gaps by similar meaning. Duplicates collapse into a single row with a count next to it: “international shipping (asked 7 times this week).” This is what makes ten minutes enough — you’re not writing seventy answers, you’re writing five or six, each of which covers ten or twenty visitor turns.

The grouping is rough, not perfect, and that’s fine. The human reading the log still scans the list and might split or merge groups manually. The job’s only job is to make the list short enough that opening it doesn’t feel like a chore.

Step 3: write a paragraph

For each group, you write one short answer — two or three sentences — and add it to the right doc in your knowledge folder. International shipping goes into the shipping policy doc. A pricing question goes into the pricing FAQ. If no doc fits, make a small new one with a clear name (“Returns FAQ” instead of “misc-2.docx”); the search doesn’t care about the file name, but you and the next person to edit will.

The writing standard is “a sentence the assistant could quote word-for-word and have it sound right.” That usually means plain language, no marketing voice, and no hedging (“it really depends” is a bad answer; “we ship to the UK; orders take 7–10 business days” is a good one). If you can’t write a clean answer because the policy itself is undecided, that’s a real find — the gap log just surfaced it. Decide the policy, then write the paragraph.

Step 4: re-index, automatically

You don’t deploy anything. The managed knowledge base watches the Drive folder; it sees the new paragraph within a few minutes and re-indexes the affected file. The next visitor who asks “do you ship to the UK?” gets a real answer with a citation pointing back at the paragraph you just wrote.

This last step is what makes the loop work. If updating the assistant needed a code change, a build, or a deploy — or even logging into a dashboard and re-uploading a file — the loop would die after the first week. Keeping the knowledge in Drive and the index on autopilot is what lets a non-engineer keep the assistant sharp.

What you’ll notice in month two

Three things tend to happen, in roughly this order. First, the handoff volume drops — not because visitors are asking less, but because more of their questions now have grounded answers. Second, the gap log gets shorter and weirder — the bulk topics get covered, leaving only the genuine edge cases (which are usually fine to hand off; that’s what humans are for). Third, the team starts noticing knowledge gaps in their own docs — the assistant is, in effect, auditing the FAQ by trying to use it. That last one is the most underrated benefit and the one that tends to surprise business owners who didn’t expect a chat widget to also be a docs-quality tool.

All posts