build.allanninal.dev

Part 7 of 7 · Feedback collector series ~8 min read

Engineering reference: the feedback collector architecture

Same system, drawn for engineers. Region, service names, resource identifiers, Bedrock model IDs, Lambda inventory, IAM scopes, the SES inbound rule set, EventBridge Scheduler config, the DynamoDB schemas, and the star-tap and webhook flow. Read alongside the previous six posts; this one’s the build sheet.

Region and account shape

Default region: ap-southeast-1 (Singapore). SES inbound, Bedrock Global cross-Region inference, and EventBridge Scheduler are all in good shape there. A second region for multi-region resilience isn’t worth the extra setup work at SMB volume — the failure mode for an SMB is a feedback ask that goes out a few hours late, not a regional outage. One AWS account dedicated to the collector (separate from your other workloads) keeps the IAM blast radius small and lets a single AWS Budgets alarm cover the whole system. Deploy with GitHub Actions using OIDC and AWS SAM, so there are no long-lived AWS keys anywhere — the CI role is assumed per run.

Topology

AWS topology of the feedback collector A topology diagram with three regions stacked vertically inside one AWS account boundary. Top region: ingress. Three boxes show the three intake lanes — a Drive sheet sync via the drive-sync Lambda triggered every 15 minutes by EventBridge Scheduler that mirrors the customer-list CSV to s3://fc-customers-source/, an intake-webhook Lambda with a Function URL that the point-of-sale tool calls when a sale closes and that appends a row to the sheet via the Sheets API, and a calendar-sync Lambda triggered hourly by EventBridge Scheduler that polls Google Calendars for finished bookings and adds rows the same way. Middle region: scheduled processing. The request-builder Lambda is triggered when a new row appears; it reads the rules doc, computes the right send time per visit type, checks quiet hours and the holiday calendar, and creates a one-off EventBridge Scheduler rule that fires the ask at that moment; the ask is sent via SES outbound with a signed star-tap link and a signed reply address. Bottom region: reply and routing. The reply-handler Lambda is triggered by a Function URL on a star-tap link click or by an S3 PUT when SES inbound writes a written reply; it verifies the signed token, dedupes against fc-feedback, and hands the reply to the router Lambda, which scores a star tap by plain rules or calls Bedrock Haiku 4.5 to read the mood of free text and emits a bucket. The dispatch Lambda acts on the bucket: happy gets an SES review nudge, unhappy gets a private owner ping, unclear is held for the daily sweep. CloudWatch Logs collects from every Lambda at 7-day retention. Across the right edge: a small box labelled AWS Budgets alarm at $15 monthly threshold, posting to SNS topic fc-cost-alarm. A note at the bottom: happy goes public, unhappy goes private — and every interaction is logged to fc-audit. Ingress Lambda · drive-sync every 15 min Sheets API → s3://fc-customers-source/ customers.csv Lambda · intake-webhook Function URL POS calls on sale verify shared secret → append row to sheet Lambda · calendar-sync hourly poll Calendar API for finished bookings → append row to sheet Drive customer list canonical store · mirrored to S3 Scheduled processing EventBridge Scheduler one-off per ask in TZ_NAME target: send the ask + daily sweep Lambda · request-builder reads CSV from S3 + rules.txt + voice.txt picks the send time, SES sends signed ask Lambda · router star tap → rules free text → Haiku 4.5 emits a bucket: happy / unhappy / unclear Reply & routing Lambda · reply-handler Function URL (tap) + SES inbound (text); verify token, dedupe, hand to router Lambda · dispatch happy → review nudge unhappy → owner ping unclear → daily sweep SES out / private DM Lambda · sweep-summary daily: list unclear, one follow-up, then stop; monthly: Haiku writes the owner summary Happy goes public, unhappy goes private — and every interaction is logged to fc-audit.
Fig 7. AWS topology, in three regions of the diagram: ingress (three lanes into the customer list), scheduled processing (the builder sending asks, the router bucketing replies), reply and routing (the handler receives, dispatch acts, the sweep cleans up). Every Lambda is event- or schedule-driven; nothing is synchronous-chained.

Lambda functions

All Lambdas use the arm64 (Graviton) architecture, the smallest memory size that meets latency targets (typically 256 MB), Python 3.14 runtime, and CloudWatch Logs at 7-day retention. Each function has its own least-privilege IAM role. None run inside a VPC.

  • drive-sync — EventBridge Scheduler target, fires every 15 minutes. Uses the Google Drive API + Sheets API (service-account credentials in Secrets Manager under fc/drive/sa) to export the customer-list sheet as CSV and write to s3://fc-customers-source/customers.csv only if the sheet has changed since the last sync. Same pattern syncs the rules and voice docs to s3://fc-rules-source/. Memory: 256 MB. Timeout: 30 s.
  • intake-webhook — Lambda Function URL, public with AuthType: NONE; verifies a shared HMAC secret (fc/pos/webhook-secret in Secrets Manager) on each request. Called by the point-of-sale or e-commerce tool when a sale closes. Validates and normalizes the payload, drops refunds and test transactions, and appends a customer-list row via the Sheets API. Memory: 256 MB. Timeout: 15 s.
  • calendar-sync — EventBridge Scheduler target, hourly. Uses the Google Calendar API events.list to scan configured calendars for bookings whose end time has just passed and which are confirmed/done; appends a row per finished booking. For lower-latency setups you can switch to events.watch and have Calendar push notifications to a Function URL instead of polling, at the cost of refreshing the channel before its TTL expires. Memory: 256 MB. Timeout: 30 s.
  • request-builder — triggered when a new row appears (a short EventBridge Scheduler poll on the mirrored CSV, or invoked directly by intake-webhook / calendar-sync). Reads rules.txt for the per-visit-type wait, computes the send time, applies quiet hours and the holiday calendar, and creates a one-off EventBridge Scheduler rule (at(...) with --action-after-completion DELETE) that invokes the send. The ask is rendered from voice.txt and sent via SES outbound; each star link and the reply-to address carry a signed token (HMAC over customer_id|ask_id|exp). Writes a pending row to fc-state. Memory: 256 MB. Timeout: 30 s. No Bedrock calls.
  • reply-handler — two triggers: a Lambda Function URL for star-tap link clicks, and an S3 PUT on s3://fc-raw-mime/ when SES inbound writes a written reply. Verifies the signed token, dedupes against fc-feedback (first reply wins), marks the customer answered, and forwards the payload to router. Renders a thank-you page for taps; for a happy tap it can show the review link inline. Memory: 256 MB. Timeout: 15 s. No Bedrock calls.
  • router — invoked by reply-handler. A star tap is scored by plain rules from rules.txt (4–5 happy, 1–2 unhappy, 3 unclear). Free text gets one Bedrock Haiku 4.5 call (anthropic.claude-haiku-4-5-20251001-v1:0 via global.anthropic.claude-haiku-4-5-20251001-v1:0) returning {mood, confidence}; confidence below the rules-doc threshold is forced to unclear. Writes the bucket and reason to fc-feedback and emits the bucket to dispatch. Memory: 256 MB. Timeout: 30 s.
  • dispatch — invoked by router. Happy: render the review nudge from voice.txt and send via SES outbound (or the SMS provider) with the public-review link. Unhappy: send a private owner ping (Slack DM via chat.postMessage, or SES email) with the customer’s words and one-tap call/message links — never the review link. Unclear: enqueue for the daily sweep. Writes a row to fc-audit after each send. Memory: 256 MB. Timeout: 30 s.
  • sweep-summary — EventBridge Scheduler target. Daily run: lists unclear items for a human, and for asks with no reply past the rules-doc window, sends at most one gentle follow-up via request-builder, then marks the row closed. Monthly run (first Monday 9am): reads the past month’s fc-feedback and fc-audit, calls Bedrock Haiku 4.5 to write a one-paragraph owner summary (asked, happy, unhappy, reviews earned, complaints fixed before they went public), emails it via SES. Memory: 512 MB.

Storage

  • DynamoDB · fc-feedback — one row per reply. PK (customer_id, ask_id); attributes: received_at, channel (tap/email), raw (star count or text), bucket (happy/unhappy/unclear), reason. On-demand. The dedupe and answered-once guarantees both key off this table.
  • DynamoDB · fc-state — one row per pending ask. PK (customer_id, ask_id); attributes: visit_type, send_at, sent_at, follow_up_sent, status (pending/sent/answered/closed), schedule_arn of the one-off rule. On-demand.
  • DynamoDB · fc-audit — one row per move of any kind. PK (customer_id, ts); attributes: bucket, move (review_nudge/owner_ping/held), sent_via, by_user (for human sweep actions). On-demand. No TTL — this is the long-term audit trail the monthly summary reads.
  • S3 · fc-customers-source — mirrored CSV from the Drive customer list. Versioning enabled. Lifecycle to Glacier at 90 days; expiry at 2 years.
  • S3 · fc-rules-source — mirrored rules and voice docs as plain text. Versioning enabled.
  • S3 · fc-raw-mime — raw inbound MIME from written replies. Lifecycle to Glacier at 30 days; expiry at 2 years.

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0 via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Two callsites: router for the free-text mood read, and sweep-summary for the monthly owner narrative. Claude Sonnet 4.6 (global.anthropic.claude-sonnet-4-6-20250930-v1:0) is wired as an optional fallback for replies Haiku flags as genuinely ambiguous, but in practice a one-sentence mood read is squarely Haiku’s job and the fallback rarely fires.
  • Embeddings. Not used. Reading the mood of one short reply is a direct classification, not a retrieval problem. No Knowledge Base, no S3 Vectors, no Titan embeddings.
  • Quotas. Default account quotas are more than enough at SMB volume. The common path (star taps) never calls Bedrock; only the free-text minority and the monthly summary do.

EventBridge Scheduler config

  • fc-drive-sync — rate(15 minutes). Target: drive-sync Lambda.
  • fc-calendar-sync — rate(1 hour). Target: calendar-sync Lambda.
  • fc-daily-sweep — cron(0 17 * * ? *) in TZ. Target: sweep-summary Lambda (daily mode).
  • fc-monthly-summary — cron(0 9 ? * 2#1 *) (first Monday at 9am) in TZ. Target: sweep-summary Lambda (monthly mode).
  • One-off ask rules — created on the fly by request-builder, one per scheduled ask. Use at(YYYY-MM-DDTHH:MM:SS) expressions in TZ_NAME (e.g. Asia/Singapore) with --action-after-completion DELETE so the rule self-cleans after firing.

SES inbound and outbound

  • Set the MX record on a dedicated subdomain (e.g. feedback.your-company.com) to inbound-smtp.ap-southeast-1.amazonaws.com.
  • SES inbound rule set fc-inbound-rules: one rule with recipient feedback@your-company.com → spam scan → S3 PUT to s3://fc-raw-mime/<message-id> → stop. The S3 PUT triggers reply-handler. The signed token rides in the unique reply-to address per ask, so the handler can match a written reply to its ask.
  • SES outbound for the asks, the happy review nudges, and email owner pings: verify a sender identity at hello@your-company.com with DKIM and SPF on the parent domain. Out of sandbox by request. If you send asks as SMS instead, the request-builder calls your SMS provider (or SNS) rather than SES for that leg.

IAM (least privilege per Lambda)

Each Lambda has its own role with policies scoped to exact ARNs. Sketch:

  • request-builder role: s3:GetObject on the customer-list and rules keys; scheduler:CreateSchedule for the one-off ask rules; secretsmanager:GetSecretValue on the token-signing secret; ses:SendRawEmail from the verified sender; dynamodb:PutItem on fc-state. No bedrock:*.
  • reply-handler role: s3:GetObject on fc-raw-mime; secretsmanager:GetSecretValue on the token-signing secret; dynamodb:GetItem + PutItem on fc-feedback; lambda:InvokeFunction on router. No bedrock:*.
  • router role: bedrock:InvokeModel on the Haiku (and optional Sonnet) ARNs; s3:GetObject on the rules key; dynamodb:PutItem on fc-feedback; lambda:InvokeFunction on dispatch.
  • dispatch role: ses:SendRawEmail from the verified sender; secretsmanager:GetSecretValue on the Slack bot token (for owner DMs); dynamodb:PutItem on fc-audit; outbound network access to hooks.slack.com / slack.com and, if used, the SMS provider.
  • intake-webhook role: secretsmanager:GetSecretValue on the POS webhook secret and the Drive service-account secret; outbound network to sheets.googleapis.com.
  • drive-sync and calendar-sync roles: secretsmanager:GetSecretValue on the Google service-account secret; s3:PutObject on the customer-list and rules buckets; outbound network to www.googleapis.com.

Signed-token flow

Every ask carries a token: an HMAC-SHA256 over customer_id|ask_id|expiry, signed with a secret in Secrets Manager (fc/token/signing-secret). For star taps the token is a URL parameter on each star link alongside the star count; for written replies it’s embedded in a unique reply-to address. The reply-handler recomputes the HMAC and rejects anything tampered, expired, or replayed (a token whose (customer_id, ask_id) already has a row in fc-feedback is a no-op). This is what lets the star links be plain, login-free URLs while staying un-forgeable — nobody can spoof a five-star tap for a customer they aren’t.

Observability and cost gates

  • CloudWatch Logs: all Lambdas, 7-day retention, structured JSON. Subscription filter on "error" + "throttle" + "timeout" to a CloudWatch metric for alerting.
  • Alarms: request-builder failures > 0 in a day (an ask that never schedules is a lost review); reply-handler signature-verification failures > 5/hour (might mean the signing secret rotated); dispatch failure rate > 1% in 24h.
  • X-Ray: off by default. Not worth the cost at SMB volume.
  • AWS Budgets: $15/month threshold, alarm at 80% and 100%, posts to SNS topic fc-cost-alarm subscribed to the on-call admin’s email and Slack.

Config and secrets

Service-account credentials for Drive, Sheets, and Calendar APIs all live in Secrets Manager under fc/drive/sa (one service account with scopes for all three APIs). The token-signing secret is fc/token/signing-secret; the POS webhook secret is fc/pos/webhook-secret; the Slack bot token for owner DMs is fc/slack/bot-token. The configured timezone, holiday list reference, quiet-hours window, per-visit-type wait times, the star thresholds, the model confidence threshold, the public-review link, and the owner’s private channel all live in Parameter Store under /fc/config/. Lambdas fetch config on cold start and cache for the lifetime of the execution environment.

Deploy

GitHub Actions with OIDC into a deploy role, AWS SAM for the stack — no long-lived keys. The opinionated bits: deploy the SES rule set as a separate stack (rule-set changes affect mail flow), turn on S3 versioning for both fc-customers-source and fc-rules-source so a bad Drive edit can be rolled back in one click, and version the EventBridge Scheduler timezone setting so you don’t accidentally start sending asks at 3am after a CI rotation. Total deployable surface: around eight Lambdas, three DDB tables, three S3 buckets, the recurring and one-off Scheduler rules, one SES rule set, and one Budgets alarm.

That’s the full system. Six narrative posts and this engineering reference. If you want to talk about adapting it for your business, see Work with me.

All posts