build.allanninal.dev

Part 7 of 7 · Document expiry watcher series ~8 min read

Engineering reference: the expiry watcher architecture

Same system, drawn for engineers. Region, service names, resource identifiers, Bedrock model IDs, Lambda inventory, IAM scopes, the SES inbound rule set, EventBridge Scheduler config, the DynamoDB schemas, and the Slack interactive flow. Read alongside the previous six posts; this one’s the build sheet.

Region and account shape

Default region: us-east-1. SES inbound, Bedrock cross-Region inference, and EventBridge Scheduler are all in good shape there. A second region for multi-region resilience isn’t worth the extra setup work at SMB volume — the failure mode for an SMB is somebody missing a renewal alert, not a regional outage. One AWS account dedicated to the watcher (separate from your other workloads) keeps the IAM blast radius small and lets a single AWS Budgets alarm cover the whole system.

Topology

AWS topology of the expiry watcher A topology diagram with three regions stacked vertically inside one AWS account boundary. Top region: ingress. Three boxes show the three intake lanes — a Drive sheet sync via the drive-sync Lambda triggered every 15 minutes by EventBridge Scheduler that mirrors the registry CSV to s3://ew-registry-source/, an SES inbound rule set with action S3 PUT to s3://ew-raw-mime/ plus the parser Lambda intake-ses-parser that runs Textract on PDFs and Bedrock Haiku 4.5 to propose a row for Slack approval, and a calendar-sync Lambda triggered hourly by EventBridge Scheduler that polls Google Calendars for events tagged hashtag-expires and proposes rows the same way. Middle region: scheduled processing. The watcher Lambda is triggered daily at 8am local by EventBridge Scheduler; it reads s3://ew-registry-source/registry.csv, iterates rows, computes days_to_expiry per item, looks up the chain in s3://ew-rules-source/rules.txt, reads ack and ping state from DynamoDB, and emits one of three events to the EventBridge default bus per item that needs an action: ew.first_alert, ew.reminder, or ew.escalate. Bottom region: dispatch and acknowledgment. The dispatch Lambda is triggered by an EventBridge rule on those three event types; it resolves the owner, checks quiet hours and the holiday calendar, fetches the alert template from s3://ew-rules-source/voice.txt, posts the message to Slack via incoming webhook with an Acknowledge button or sends an email via SES outbound, and writes a row to DynamoDB ew-pings. Slack interactive button clicks land on a Function URL Lambda ack-handler that updates ew-ack with the action (renew, snooze, ack-only) and, on renew, updates the registry sheet via the Google Sheets API. CloudWatch Logs collects from every Lambda at 7-day retention. Across the right edge: a small box labelled AWS Budgets alarm at $15 monthly threshold, posting to SNS topic ew-cost-alarm. A note at the bottom: every alert leaves with full context — and every interaction is logged to ew-audit. Ingress Lambda · drive-sync every 15 min Sheets API → s3://ew-registry-source/ registry.csv SES inbound rule set ew-inbound-rules action: S3 PUT s3://ew-raw-mime/ trigger: intake-ses-parser Lambda · calendar-sync hourly poll Calendar API for events tagged #expires → Slack proposal Drive registry sheet canonical store · mirrored to S3 Scheduled processing EventBridge Scheduler cron(0 8 * * ? *) in TZ_NAME target: watcher Lambda + deferred one-offs Lambda · watcher reads CSV from S3 + rules.txt + voice.txt computes days, picks one of four moves EventBridge default bus ew.first_alert ew.reminder ew.escalate (healthy → no event) Dispatch & acknowledgment Lambda · dispatch resolves owner, quiet hours, holidays; Slack webhook or SES outbound Slack interactive DM with [Renew] [Snooze] [Ack only] button clicks → Function URL Lambda · ack-handler writes ew-ack, ew-audit, and on renew updates the Sheet via Sheets API Every alert leaves with full context — and every interaction is logged to ew-audit.
Fig 7. AWS topology, in three regions of the diagram: ingress (three lanes into the registry), scheduled processing (the daily watcher tick emitting events), dispatch and acknowledgment (the alert ships and the owner’s response is recorded). Every Lambda is event- or schedule-driven; nothing is synchronous-chained.

Lambda functions

All Lambdas use the arm64 architecture, the smallest memory size that meets latency targets (typically 256 MB), Python 3.14 runtime, and CloudWatch Logs at 7-day retention. Each function has its own least-privilege IAM role. None run inside a VPC.

  • drive-sync — EventBridge Scheduler target, fires every 15 minutes. Uses the Google Drive API + Sheets API (service-account credentials in Secrets Manager under ew/drive/sa) to export the registry sheet as CSV and write to s3://ew-registry-source/registry.csv only if the sheet has changed since the last sync. Same pattern syncs the rules and voice docs to s3://ew-rules-source/. Memory: 256 MB. Timeout: 30 s.
  • calendar-sync — EventBridge Scheduler target, hourly. Uses the Google Calendar API events.list to scan configured calendars for events with #expires in the description; for any new events, creates a Slack interactive proposal message. For lower-latency setups you can switch to events.watch and have Calendar push notifications to a Function URL instead of polling, at the cost of renewing the channel before it expires (Calendar push channels have a finite TTL and need a small refresh job). Memory: 256 MB. Timeout: 30 s.
  • intake-ses-parser — S3 PUT trigger on s3://ew-raw-mime/. Parses MIME, extracts the PDF attachment, runs Textract via StartDocumentTextDetection + StartDocumentAnalysis (asynchronously to handle multi-page contracts). On Textract completion (via SNS notification), reads the structured text and calls Bedrock Haiku 4.5 (anthropic.claude-haiku-4-5-20251001-v1:0 via global.anthropic.claude-haiku-4-5-20251001-v1:0) to propose a registry row. Posts the proposal to Slack via the incoming webhook with Approve/Edit/Discard buttons. For DOCX attachments (Textract doesn’t accept them), falls back to python-docx; XLSX uses openpyxl. Both packages are stable and widely used in 2026, though their maintenance velocity is light — for a contract-parsing path that only runs a few times a month, that’s acceptable. If extraction precision becomes a concern, the active community fork python-docx-oss is a drop-in alternative. Memory: 512 MB. Timeout: 60 s.
  • watcher — EventBridge Scheduler target, daily at 8am local time (the schedule expression runs in TZ_NAME set to the SMB’s timezone, e.g. America/New_York). Reads s3://ew-registry-source/registry.csv and the rules and voice docs. For each row, computes days_to_expiry, reads chain state from ew-pings and ew-ack, decides on a move. Emits one event per row that needs action: ew.first_alert, ew.reminder, or ew.escalate, with the item context as the event payload. Healthy items emit nothing. Memory: 512 MB. Timeout: 60 s. No Bedrock calls.
  • dispatch — EventBridge rule on the three move events. Resolves owner, checks quiet hours and holiday calendar, formats the alert from the voice template, and ships via Slack incoming webhook (ew/slack/webhook in Secrets Manager) or SES SendRawEmail. On quiet-hours or holiday defer, creates a one-off EventBridge Scheduler rule that re-invokes dispatch at the next available business minute. Writes a row to ew-pings after a successful send. Memory: 256 MB. Timeout: 30 s.
  • ack-handler — Lambda Function URL, public with AuthType: NONE; verifies a Slack signature on the request body. Triggered by Slack interactive button clicks (Renew/Snooze/Ack-only) and by email-link clicks. Writes to ew-ack and ew-audit; on renew, updates the Drive sheet via the Sheets API and archives the old chain in ew-pings-archive. Memory: 256 MB. Timeout: 15 s.
  • digest — EventBridge Scheduler target, weekly Sunday 6pm. Reads ew-pings for the past week and the registry; sends a digest message to a configured Slack channel summarizing pings sent and items coming up. No Bedrock; the message is a plain summary table. Memory: 256 MB.
  • summary — EventBridge Scheduler target, monthly on the first Monday at 9am. Reads the past month’s ew-pings, ew-ack, and ew-audit; calls Bedrock Haiku 4.5 to write a one-paragraph board narrative; emails it via SES to the configured stakeholder list. Memory: 512 MB.

Storage

  • DynamoDB · ew-pings — one row per dispatch. PK (item_id, chain_index); attributes: ping_date, dispatched_via (slack/email), recipient, move (first_alert/reminder/escalate). On-demand. No TTL.
  • DynamoDB · ew-ack — one row per acknowledgment. PK item_id; sort key ack_date; attributes: action (renew/snooze/ack-only), by_user, snooze_until (if action = snooze), old_expiry, new_expiry (if action = renew). On-demand.
  • DynamoDB · ew-audit — one row per write action of any kind. PK (item_id, ts); attributes: action, by_user, before, after. On-demand. No TTL — this is the long-term audit trail.
  • DynamoDB · ew-pings-archive — archived chains after a renewal. Same shape as ew-pings; PK (item_id, chain_id, chain_index). On-demand.
  • S3 · ew-registry-source — mirrored CSV from the Drive registry sheet. Versioning enabled. Lifecycle to Glacier at 90 days; expiry at 7 years.
  • S3 · ew-rules-source — mirrored rules and voice docs as plain text. Versioning enabled.
  • S3 · ew-raw-mime — raw inbound MIME from forwarded contracts. Lifecycle to Glacier at 30 days; expiry at 7 years.
  • S3 · ew-source-pdfs — the parsed source contracts after the inbound parser handles them, kept for reference if the registry row links to one.

Bedrock

  • Foundation model. anthropic.claude-haiku-4-5-20251001-v1:0 via the Global cross-Region inference profile global.anthropic.claude-haiku-4-5-20251001-v1:0. Two callsites: intake-ses-parser for the inbound contract parsing, and summary for the monthly board narrative.
  • Embeddings. Not used. The registry is structured rows; deterministic lookup beats vector retrieval here. No Knowledge Base, no S3 Vectors.
  • Quotas. Default account quotas are more than enough at SMB volume. The watcher itself doesn’t call Bedrock; the parsing lane fires a few times a month at most.

EventBridge Scheduler config

  • ew-daily-tick — cron(0 8 * * ? *) in the SMB’s timezone. Target: watcher Lambda.
  • ew-drive-sync — rate(15 minutes). Target: drive-sync Lambda.
  • ew-calendar-sync — rate(1 hour). Target: calendar-sync Lambda.
  • ew-weekly-digest — cron(0 18 ? * SUN *) in TZ. Target: digest Lambda.
  • ew-monthly-summary — cron(0 9 ? * 2#1 *) (first Monday at 9am) in TZ. Target: summary Lambda.
  • One-off rules — created on the fly by dispatch when a quiet-hours or holiday defer is needed. Use at(YYYY-MM-DDTHH:MM:SS) expressions with --action-after-completion DELETE so the rule self-cleans.

SES inbound and outbound

  • Set the MX record on a dedicated subdomain (e.g. expires.your-company.com) to inbound-smtp.us-east-1.amazonaws.com.
  • SES inbound rule set ew-inbound-rules: one rule with recipient expires@your-company.com → spam scan → S3 PUT to s3://ew-raw-mime/<message-id> → stop. The S3 PUT triggers intake-ses-parser.
  • SES outbound for the email-fallback alerts: verify a sender identity at watcher@your-company.com with DKIM and SPF on the parent domain. Out of sandbox by request.

IAM (least privilege per Lambda)

Each Lambda has its own role with policies scoped to exact ARNs. Sketch:

  • watcher role: s3:GetObject on the registry, rules, and voice keys; dynamodb:Query + GetItem on ew-pings, ew-ack; events:PutEvents on the default bus. No bedrock:*.
  • dispatch role: events:ListSchedules + CreateSchedule for the deferred-dispatch one-offs; secretsmanager:GetSecretValue on the Slack webhook secret; ses:SendRawEmail from the verified sender identity; dynamodb:PutItem on ew-pings; outbound network access to hooks.slack.com.
  • ack-handler role: dynamodb:PutItem on ew-ack and ew-audit; secretsmanager:GetSecretValue on the Sheets-API service-account secret; outbound network access to sheets.googleapis.com; dynamodb:Query for chain state lookup; on renew, dynamodb:BatchWriteItem for archiving the old chain to ew-pings-archive.
  • intake-ses-parser role: s3:GetObject on ew-raw-mime; textract:StartDocumentTextDetection + StartDocumentAnalysis; bedrock:InvokeModel on the Haiku ARN; secretsmanager:GetSecretValue on the Slack webhook.
  • drive-sync and calendar-sync roles: secretsmanager:GetSecretValue on the Google service-account secret; s3:PutObject on the registry and rules buckets; outbound network to www.googleapis.com.

Slack interactive flow

The Slack incoming webhook is the simplest delivery surface but doesn’t support interactive button responses. So the alert messages are posted via the chat.postMessage Web API instead, with Block Kit blocks containing the action buttons. Button clicks are sent by Slack to the configured Interactivity request URL, which is the ack-handler Function URL. ack-handler verifies the Slack signing secret on the inbound request, parses the action_id (renew, snooze, ack_only), opens a modal if needed (Renew/Snooze open modals; Ack-only is one-tap), and processes the response when the modal is submitted.

The Slack app needs chat:write, im:write, and the Interactivity URL configured. The bot token lives in Secrets Manager under ew/slack/bot-token. The signing secret is ew/slack/signing-secret.

Observability and cost gates

  • CloudWatch Logs: all Lambdas, 7-day retention, structured JSON. Subscription filter on "error" + "throttle" + "timeout" to a CloudWatch metric for alerting.
  • Alarms: watcher Lambda failures > 0 in a day (the daily tick is the one piece that has to run); dispatch failure rate > 1% in 24h; ack-handler signature-verification failures > 5/hour (might mean the Slack secret rotated).
  • X-Ray: off by default. Not worth the cost at SMB volume.
  • AWS Budgets: $15/month threshold, alarm at 80% and 100%, posts to SNS topic ew-cost-alarm subscribed to the on-call admin’s email and Slack.

Config and secrets

Service-account credentials for Drive, Sheets, and Calendar APIs all live in Secrets Manager under ew/drive/sa (one service account with scopes for all three APIs). Slack bot token, signing secret, and webhook URL all under ew/slack/*. SES sender identity lives in IAM and the verified-domain config. The configured timezone, holiday list reference, quiet-hours window, and admin fallback owner all live in Parameter Store under /ew/config/. Lambdas fetch config on cold start and cache for the lifetime of the execution environment.

Deploy

Whichever IaC you prefer. The opinionated bits: deploy the SES rule set as a separate stack (rule-set changes affect mail flow), turn on S3 versioning for both ew-registry-source and ew-rules-source so a bad Drive edit can be rolled back in one click, and version the EventBridge Scheduler timezone setting so you don’t accidentally start running the daily tick in UTC after a CI rotation. CDK with a Python stack file works well; SAM also fits. Total deployable surface: around eight Lambdas, four DDB tables, four S3 buckets, one EventBridge rule on the default bus (plus the Scheduler rules), one SES rule set, and one Budgets alarm.

That’s the full system. Six narrative posts and this engineering reference. If you want to talk about adapting it for your business, see Work with me.

All posts