build.allanninal.dev

Part 5 of 7 · Website chat assistant series ~4 min read

How gaps become better answers

An assistant that can’t answer a question isn’t a failure — an assistant that can’t answer the same question in two months is. Here’s the small loop that turns every miss into a paragraph in your docs, and turns next month’s identical question into a clean auto-reply.

The loop, in one diagram

A circular four-step learning loop: log, group, write, re-index A loop with four boxes arranged clockwise. Top: "Log every miss" — when a turn ends in clarify, hand off, or decline, the question, the page URL, and the closest passages found are appended to a Drive doc called the gaps log. Right: "Group similar questions" — once a week, a small batch job groups gaps with similar wording so duplicates collapse into one row. Bottom: "Write a paragraph" — a human spends ten minutes adding short answers to the knowledge folder, one paragraph per group. Left: "Re-index automatically" — the managed knowledge base picks up the new paragraphs within minutes; nothing to deploy. Arrows form a clockwise circle with a small label "weekly" under the gap between top and right. A note in the centre reads: "Each loop turn upgrades the assistant. Month two answers the questions month one missed." Step 1 Log every miss turn + page + closest passages appended to gaps log Step 2 Group similar duplicates collapse into one row, by wording Step 3 Write a paragraph human, ten minutes, straight into the Drive folder Step 4 Re-index automatically managed KB picks up new paragraphs in minutes weekly Each loop turn upgrades the assistant. Month two answers what month one missed.
Fig 5. A small clockwise loop. Misses become groups, groups become paragraphs, paragraphs become better answers.

Step 1: log every miss, automatically

Every time the answerer picks “clarify,” “hand off,” or “decline” for a question that wasn’t hostile, the cloud appends a row to a small Drive doc — the gaps log. Each row is short: the visitor’s message, the page URL, the timestamp, and the closest passage the search did find (which might just be a near miss). Nothing else — no PII the visitor didn’t paste in themselves, no tracking IDs.

The point of putting this in Drive, not a database, is that it lives where the team already lives. No dashboard to log into, no permissions to manage. Open the doc, see this week’s misses, write the answers, save the doc — that’s the entire workflow.

Step 2: group similar questions

Most weeks, the gaps log isn’t a long list of unique problems — it’s a short list of repeated ones, each asked by a few different visitors in slightly different words. “Do you ship internationally?”, “Can I order from the UK?”, “Where do you ship to?” are the same question dressed differently.

A small batch job runs once a week and groups gaps by similar meaning. Duplicates collapse into a single row with a count next to it: “international shipping (asked 7 times this week).” This is what makes ten minutes enough — you’re not writing seventy answers, you’re writing five or six, each of which covers ten or twenty visitor turns.

The grouping is rough, not perfect, and that’s fine. The human reading the log still scans the list and might split or merge groups manually. The job’s only job is to make the list short enough that opening it doesn’t feel like a chore.

Step 3: write a paragraph

For each group, you write one short answer — two or three sentences — and add it to the appropriate doc in your knowledge folder. International shipping goes into the shipping policy doc. A pricing question goes into the pricing FAQ. If no doc fits, you create a small new one with a clear name (“Returns FAQ” rather than “misc-2.docx”); the search doesn’t care about the file name, but you and the next person to edit will.

The writing standard is “a sentence the assistant could quote verbatim and have it sound right.” That tends to mean plain language, no marketing voice, no hedging (“it really depends” is a bad answer; “we ship to the UK; orders take 7–10 business days” is a good one). If you can’t write a clean answer because the policy itself is undecided — that’s a real find, and the gap log just told you about it. Decide the policy, then write the paragraph.

Step 4: re-index, automatically

You don’t deploy anything. The managed knowledge base watches the Drive folder; it sees the new paragraph within a few minutes and re-indexes the affected file. The next visitor who asks “do you ship to the UK?” gets a real answer with a citation pointing back at the paragraph you just wrote.

This last step is what makes the loop possible. If updating the assistant required a code change, a build, or a deploy — or even logging into a dashboard and re-uploading a file — the loop would die after the first week. Putting the knowledge in Drive and the index on autopilot is what lets a non-engineer keep the assistant sharp.

What you’ll notice in month two

Three things tend to happen, in roughly this order. First, the handoff volume drops — not because visitors are asking less, but because more of their questions now have grounded answers. Second, the gap log gets shorter and weirder — the bulk topics get covered, leaving only the genuine edge cases (which are usually fine to hand off; that’s what humans are for). Third, the team starts noticing knowledge gaps in their own docs — the assistant is, in effect, auditing the FAQ by trying to use it. That last one is the most underrated benefit and the one that tends to surprise business owners who didn’t expect a chat widget to also be a docs-quality tool.

All posts