How We Built the S3-Backed Content Bootstrap

Backend Team

A deep dive into the transactional S3 scanner that hydrates the news, slides, and legal tables at boot — and the advisory-lock design that makes it safe on multi-replica deployments.

How We Built the S3-Backed Content Bootstrap

When we migrated authored content from the git repo to S3, we needed a reliable way to populate the Postgres index at boot without blocking startup or racing between replicas.

The Design

The bootstrap runs in three phases:

  1. S3 scan — list and fetch every manifest.json under the configured prefix. No database client is held during this phase, so a slow S3 round-trip does not pin a connection.

  2. Advisory lock — a single pg_try_advisory_lock call gates the write phase. If another replica already holds the lock, the current process skips writes entirely.

  3. Transactional upsert — all content rows are upserted inside a single transaction. A stale-row DELETE prunes objects that disappeared from S3 since the last scan.

Why a 60-Second Throttle

The watchdog calls scanAndReindex once per boot. The throttle prevents stampedes: a successful scan blocks re-entry for 60 seconds per process.