build.allanninal.dev

Part 7 of 7 · Invoice chaser series ~8 min read

Engineering reference: the invoice chaser architecture

Same system, drawn for engineers. Region, service names, resource identifiers, Bedrock model IDs, Lambda inventory, IAM scopes, the SES inbound rule set, EventBridge Scheduler config, the DynamoDB schemas, and the Slack interactive flow. Read alongside the previous six posts; this one’s the build sheet.

Region and account shape

Default region: ap-southeast-1 (Singapore). SES inbound, Bedrock Global cross-Region inference, and EventBridge Scheduler are all in good shape there. A second region for multi-region resilience isn’t worth the extra setup work at SMB volume — the failure mode for an SMB is a reminder that goes out a day late, not a regional outage. One AWS account dedicated to the chaser (separate from your other workloads) keeps the IAM blast radius small and lets a single AWS Budgets alarm cover the whole system.

Topology

AWS topology of the invoice chaser A topology diagram with three regions stacked vertically inside one AWS account boundary. Top region: ingress. Three boxes show the three intake lanes — a Drive sheet sync via the drive-sync Lambda triggered every 15 minutes by EventBridge Scheduler that mirrors the invoice CSV to s3://ic-invoices-source/, an SES inbound rule set with action S3 PUT to s3://ic-raw-mime/ plus the parser Lambda intake-ses-parser that runs Textract on PDFs and Bedrock Haiku 4.5 to propose a row for Slack approval, and a webhook Function URL intake-webhook that the accounting tool calls on issue and that proposes rows the same way. Middle region: scheduled processing. The chaser Lambda is triggered daily at 9am local by EventBridge Scheduler; it reads s3://ic-invoices-source/invoices.csv, iterates rows, computes days_past_due per invoice, looks up the cadence in s3://ic-rules-source/rules.txt, reads send and status state from DynamoDB, and emits one of three events to the EventBridge default bus per invoice that needs an action: ic.first_nudge, ic.follow_up, or ic.escalate. Bottom region: send and acknowledgment. The sender Lambda is triggered by an EventBridge rule on those three event types; it resolves the contact, checks quiet hours, weekends, and the holiday calendar, fetches the tone template from s3://ic-rules-source/voice.txt, sends the reminder via SES outbound with a pay-link, and writes a row to DynamoDB ic-sends. Slack interactive button clicks and customer pay-link clicks land on a Function URL Lambda action-handler that updates ic-state with the action (pause, disputed, write-off, paid) and, on write-off, marks the row closed in the sheet via the Google Sheets API. CloudWatch Logs collects from every Lambda at 7-day retention. Across the right edge: a small box labelled AWS Budgets alarm at $15 monthly threshold, posting to SNS topic ic-cost-alarm. A note at the bottom: every reminder leaves with a pay-link — and every interaction is logged to ic-audit. Ingress Lambda · drive-sync every 15 min Sheets API → s3://ic-invoices-source/ invoices.csv SES inbound rule set ic-inbound-rules action: S3 PUT s3://ic-raw-mime/ trigger: intake-ses-parser Function URL · webhook intake-webhook accounting tool posts new invoice on issue → Slack proposal Drive invoice sheet canonical store · mirrored to S3 Scheduled processing EventBridge Scheduler cron(0 9 * * ? *) in TZ_NAME target: chaser Lambda + deferred one-offs Lambda · chaser reads CSV from S3 + rules.txt + voice.txt computes days, picks one of four moves EventBridge default bus ic.first_nudge ic.follow_up ic.escalate (current → no event) Send & acknowledgment Lambda · sender resolves contact, quiet hours, weekends; SES outbound with a pay-link Slack + pay-link review [Pause] [Disputed] [Write off] clicks → Function URL Lambda · action-handler writes ic-state, ic-audit, and on write-off marks the row via Sheets API Every reminder leaves with a pay-link — and every interaction is logged to ic-audit.
Fig 7. AWS topology, in three regions of the diagram: ingress (three lanes into the invoice sheet), scheduled processing (the daily chaser tick emitting events), send and acknowledgment (the reminder ships and the owner’s or customer’s response is recorded). Every Lambda is event- or schedule-driven; nothing is synchronous-chained.

Lambda functions

All Lambdas use the arm64 architecture, the smallest memory size that meets latency targets (typically 256 MB), Python 3.14 runtime, and CloudWatch Logs at 7-day retention. Each function has its own least-privilege IAM role. None run inside a VPC.

  • drive-sync — EventBridge Scheduler target, fires every 15 minutes. Uses the Google Drive API + Sheets API (service-account credentials in Secrets Manager under ic/drive/sa) to export the invoice sheet as CSV and write to s3://ic-invoices-source/invoices.csv only if the sheet has changed since the last sync. Same pattern syncs the rules and voice docs to s3://ic-rules-source/. Memory: 256 MB. Timeout: 30 s.
  • intake-webhook — Lambda Function URL, AuthType: NONE with a shared-secret check on the request (the secret lives in Secrets Manager under ic/webhook/secret). The accounting tool posts each newly issued invoice; the handler maps the tool’s payload to the row shape and either posts a Slack proposal or, for a trusted source, writes the row directly via the Sheets API. Memory: 256 MB. Timeout: 15 s.
  • intake-ses-parser — S3 PUT trigger on s3://ic-raw-mime/. Parses MIME, extracts the PDF attachment, runs Textract via StartDocumentTextDetection + StartDocumentAnalysis (asynchronously to handle multi-page invoices). On Textract completion (via SNS notification), reads the structured text and calls Bedrock Haiku 4.5 (anthropic.claude-haiku-4-5-20251001-v1:0 via global.anthropic.claude-haiku-4-5-20251001-v1:0) to propose an invoice row. Posts the proposal to Slack via chat.postMessage with Approve/Edit/Discard buttons. For DOCX attachments (Textract doesn’t accept them), falls back to python-docx; XLSX uses openpyxl. Both packages are stable and widely used in 2026, though their maintenance velocity is light — for an invoice-parsing path that only runs a few times a month, that’s acceptable. If extraction precision becomes a concern, the active community fork python-docx-oss is a drop-in alternative. Memory: 512 MB. Timeout: 60 s.
  • chaser — EventBridge Scheduler target, daily at 9am local time (the schedule expression runs in TZ_NAME set to the SMB’s timezone, e.g. Asia/Singapore). Reads s3://ic-invoices-source/invoices.csv and the rules and voice docs. For each row, computes days_past_due, reads chase state from ic-sends and ic-state, decides on a move. Emits one event per row that needs action: ic.first_nudge, ic.follow_up, or ic.escalate, with the invoice context as the event payload. Current invoices emit nothing. Reconciles the paid column against ic-state and archives chases for newly paid invoices. Memory: 512 MB. Timeout: 60 s. No Bedrock calls.
  • sender — EventBridge rule on the three move events. Resolves contact, checks quiet hours, weekends, and the holiday calendar, formats the reminder from the voice tone template, and ships via SES SendRawEmail. The escalate move resolves to the internal account owner instead of the customer. On a quiet-hours, weekend, or holiday defer, creates a one-off EventBridge Scheduler rule that re-invokes sender at the next available business minute. Writes a row to ic-sends after a successful send. Memory: 256 MB. Timeout: 30 s.
  • action-handler — Lambda Function URL, public with AuthType: NONE; verifies a Slack signature on Slack-sourced requests and a signed token on pay-link requests. Triggered by Slack interactive button clicks (Pause/Disputed/Write-off) and by customer pay-link clicks. Writes to ic-state and ic-audit; on write-off, marks the row closed in the Drive sheet via the Sheets API and archives the chase in ic-sends-archive; on a recorded payment, flips state to paid and stops the chase. Memory: 256 MB. Timeout: 15 s.
  • digest — EventBridge Scheduler target, weekly Monday 9am. Reads ic-sends for the past week and the invoice list; sends a digest message to a configured Slack channel summarizing reminders sent, invoices coming due, and the oldest open balances. No Bedrock; the message is a plain summary table. Memory: 256 MB.
  • summary — EventBridge Scheduler target, monthly on the first Monday at 9am. Reads the past month’s ic-sends, ic-state, and ic-audit; calls Bedrock Haiku 4.5 to write a one-paragraph cash-flow narrative (total outstanding, oldest invoices, biggest at-risk amounts, written-off total); emails it via SES to the configured stakeholder list. Memory: 512 MB.

Storage

  • DynamoDB · ic-sends — one row per reminder sent. PK (invoice_id, step_index); attributes: send_date, dispatched_via (email), recipient, move (first_nudge/follow_up/escalate). On-demand. No TTL.
  • DynamoDB · ic-state — one row per invoice. PK invoice_id; attributes: status (open/paid/paused/disputed/written-off), paused_until (if paused), pause_count, paid_date and paid_amount (if paid), by_user. On-demand.
  • DynamoDB · ic-audit — one row per write action of any kind. PK (invoice_id, ts); attributes: action, by_user, before, after. On-demand. No TTL — this is the long-term audit trail.
  • DynamoDB · ic-sends-archive — archived chases after payment or write-off. Same shape as ic-sends; PK (invoice_id, chase_id, step_index). On-demand.
  • S3 · ic-invoices-source — mirrored CSV from the Drive invoice sheet. Versioning enabled. Lifecycle to Glacier at 90 days; expiry at 7 years.
  • S3 · ic-rules-source — mirrored rules and voice docs as plain text. Versioning enabled.
  • S3 · ic-raw-mime — raw inbound MIME from forwarded invoices. Lifecycle to Glacier at 30 days; expiry at 7 years.
  • S3 · ic-source-pdfs — the parsed source invoices after the inbound parser handles them, kept for reference if the row links to one.

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0 via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Two callsites: intake-ses-parser for the inbound invoice parsing, and summary for the monthly cash-flow narrative. Claude Sonnet 4.6 (global.anthropic.claude-sonnet-4-6-20250930-v1:0) is available as a fallback for invoices whose layout the Haiku pass flags as low-confidence, but in practice invoices are structured enough that Haiku handles them.
  • Embeddings. Not used. The invoice list is structured rows; deterministic lookup beats vector retrieval here. No Knowledge Base, no S3 Vectors.
  • Quotas. Default account quotas are more than enough at SMB volume. The chaser itself doesn’t call Bedrock; the parsing lane fires a few times a month at most.

EventBridge Scheduler config

  • ic-daily-tick — cron(0 9 * * ? *) in the SMB’s timezone. Target: chaser Lambda.
  • ic-drive-sync — rate(15 minutes). Target: drive-sync Lambda.
  • ic-weekly-digest — cron(0 9 ? * MON *) in TZ. Target: digest Lambda.
  • ic-monthly-summary — cron(0 9 ? * 2#1 *) (first Monday at 9am) in TZ. Target: summary Lambda.
  • One-off rules — created on the fly by sender when a quiet-hours, weekend, or holiday defer is needed. Use at(YYYY-MM-DDTHH:MM:SS) expressions with --action-after-completion DELETE so the rule self-cleans.

SES inbound and outbound

  • Set the MX record on a dedicated subdomain (e.g. invoices.your-company.com) to inbound-smtp.ap-southeast-1.amazonaws.com.
  • SES inbound rule set ic-inbound-rules: one rule with recipient invoices@your-company.com → spam scan → S3 PUT to s3://ic-raw-mime/<message-id> → stop. The S3 PUT triggers intake-ses-parser.
  • SES outbound for the reminder emails: verify a sender identity at billing@your-company.com with DKIM and SPF on the parent domain, and set the reply-to to a monitored inbox so customer replies reach a human. Out of sandbox by request.

IAM (least privilege per Lambda)

Each Lambda has its own role with policies scoped to exact ARNs. Sketch:

  • chaser role: s3:GetObject on the invoices, rules, and voice keys; dynamodb:Query + GetItem on ic-sends, ic-state; dynamodb:BatchWriteItem for archiving paid chases to ic-sends-archive; events:PutEvents on the default bus. No bedrock:*.
  • sender role: events:ListSchedules + CreateSchedule for the deferred-send one-offs; ses:SendRawEmail from the verified sender identity; dynamodb:PutItem on ic-sends; secretsmanager:GetSecretValue on the pay-link signing secret.
  • action-handler role: dynamodb:PutItem on ic-state and ic-audit; secretsmanager:GetSecretValue on the Slack signing secret and the Sheets-API service-account secret; outbound network access to sheets.googleapis.com; dynamodb:Query for chase state lookup; on write-off or payment, dynamodb:BatchWriteItem for archiving the chase to ic-sends-archive.
  • intake-ses-parser role: s3:GetObject on ic-raw-mime; textract:StartDocumentTextDetection + StartDocumentAnalysis; bedrock:InvokeModel on the Haiku ARN; secretsmanager:GetSecretValue on the Slack bot token.
  • drive-sync and intake-webhook roles: secretsmanager:GetSecretValue on the Google service-account secret and the webhook secret; s3:PutObject on the invoices and rules buckets; outbound network to www.googleapis.com.

Slack interactive flow

The Slack incoming webhook is the simplest delivery surface but doesn’t support interactive button responses. So the daily-review and intake-proposal messages are posted via the chat.postMessage Web API instead, with Block Kit blocks containing the action buttons. Button clicks are sent by Slack to the configured Interactivity request URL, which is the action-handler Function URL. action-handler verifies the Slack signing secret on the inbound request, parses the action_id (pause, disputed, write_off; or approve, edit, discard on intake), opens a modal if needed (Pause opens a modal; Disputed and Write-off are one-tap), and processes the response when the modal is submitted.

The Slack app needs chat:write, im:write, and the Interactivity URL configured. The bot token lives in Secrets Manager under ic/slack/bot-token. The signing secret is ic/slack/signing-secret.

Observability and cost gates

  • CloudWatch Logs: all Lambdas, 7-day retention, structured JSON. Subscription filter on "error" + "throttle" + "timeout" to a CloudWatch metric for alerting.
  • Alarms: chaser Lambda failures > 0 in a day (the daily tick is the one piece that has to run); sender failure rate > 1% in 24h; action-handler signature-verification failures > 5/hour (might mean the Slack secret rotated); SES bounce or complaint rate above the SES reputation thresholds (a reminder to a stale contact that hard-bounces should page).
  • X-Ray: off by default. Not worth the cost at SMB volume.
  • AWS Budgets: $15/month threshold, alarm at 80% and 100%, posts to SNS topic ic-cost-alarm subscribed to the on-call admin’s email and Slack.

Config and secrets

Service-account credentials for Drive and Sheets APIs live in Secrets Manager under ic/drive/sa (one service account with scopes for both APIs). Slack bot token and signing secret under ic/slack/*. The webhook shared secret and the pay-link signing secret under ic/webhook/* and ic/paylink/*. SES sender identity lives in IAM and the verified-domain config. The configured timezone, holiday list reference, quiet-hours window, weekend rule, and account-owner fallback all live in Parameter Store under /ic/config/. Lambdas fetch config on cold start and cache for the lifetime of the execution environment.

Deploy

GitHub Actions with OIDC into a deploy role (no long-lived keys) and AWS SAM for the stack. The opinionated bits: deploy the SES rule set as a separate stack (rule-set changes affect mail flow), turn on S3 versioning for both ic-invoices-source and ic-rules-source so a bad Drive edit can be rolled back in one click, and version the EventBridge Scheduler timezone setting so you don’t accidentally start running the daily tick in UTC after a CI rotation. SAM fits cleanly; a CDK Python stack also works. Total deployable surface: around eight Lambdas, four DDB tables, four S3 buckets, one EventBridge rule on the default bus (plus the Scheduler rules), one SES rule set, and one Budgets alarm.

That’s the full system. Six narrative posts and this engineering reference. If you want to talk about adapting it for your business, see Work with me.

All posts