§5.4 Background-Queue Infrastructure — Implementation Plan

§5.4 Background-Queue Infrastructure — Implementation Plan

Spec: docs/specs/background-queue-infra-spec.md v1 (1556 lines, S218 W1 SPEC-B authoring, ratified at S218 W4 close-out). Plan author: Claude Code (S221 W1 PLAN-A sub-agent, 02/05/2026). Skill: /planning-and-task-breakdown per feedback_spec_then_plan_workflow.

Estimated effort across all waves: ~4-5 sessions. W1 only (this plan’s landing target): 1 session (~3-4 hours of agent work split across 3 parallel-safe slots, sequential merges).

---

Plan-authoring premises (do not skim)

1. W1 scope is DDL + chokepoint helpers + telemetry only. Worker route (app/api/cron/process-queue/route.ts) + vercel.json cron registration defer to W2 (S222 carry). The four §5.4.x candidate migrations (5.4.1 / 5.4.2 / 5.4.3 / 5.4.4) each get their own follow-up spec per spec PR-2; none land in W1 or W2 of this plan.
2. Idempotency producer formula is a HARD contract that every future migration spec must obey verbatim. Quoted in §1 Wave Map AND §4 verification gate per feedback_brief_quote_spec_verbatim:

   idempotency_key := <job_type>:<scoped_id>:<YYYY-MM-DD>:<requestHash>
                                             ^^^^^^^^^^
                                             MANDATORY date/version bucket

3. Job-payload envelope is a HARD contract stored in processing_queue.payload JSONB. Quoted verbatim in §2 file-ownership notes for lib/queue/envelope.ts. The shape derives from spec §3.1.
4. Worker auth-context reconstruction rule (spec §4.2 PR-3): worker uses createServiceClient() from lib/supabase/server.ts:58-69 ONLY, then re-validates userId+role against user_roles before any privileged op. Quoted verbatim in §2 file-ownership notes for lib/queue/auth.ts (W2 file, but the contract is established in W1 helper documentation).
5. Status enum widening to add 'dead_lettered' (spec §4 + §6 DDL + D-2): DB CHECK widening MUST pair with TS union widening per feedback_db_check_ts_union_paired_widening. Tracked as W1 risk.
6. isolation: "worktree" parallel agent isolation is the default for Wave 1 implementation slots. Each slot owns disjoint files; sequential merges to main with git status checks per feedback_worktree_leaks + feedback_worktree_overwrite.
7. NEW PL/pgSQL helpers (enqueue_queue_job, optionally requeue_queue_job_with_backoff) MUST include explicit REVOKE EXECUTE ... FROM anon; per feedback_supabase_pg_default_acl_anon_execute. claim_next_job already has this REVOKE in 20260502143049_ops43_revoke_anon_execute_public_functions.sql (line 186) — no re-revoke needed for the existing function unless its body changes.
8. Reference plan shape mirrors docs/plans/p0-1-edit-path-consolidation-plan.md (4-phase / file-ownership table / AC traceability / risk mitigations) and docs/plans/p0-bm-phase2-bid-write-side-plan.md (work-package structure / dependency DAG / verification mapping).

---

§1 Wave map

This plan organises work into FIVE waves. Wave 1 lands in S221 (this plan’s primary target). Waves 2-5 are documented for forward-look ONLY and will be re-planned in their owning sessions per feedback_self_verify_own_specs discipline (a single-pass plan carries errors; each wave should be re-validated against the spec at its own session entry).

W1 — DDL + chokepoint helpers + telemetry foundation [S221 — IN SCOPE THIS PLAN]

Goal: Establish the queue contract in code without yet wiring a worker. Lands the migration that widens processing_queue for D-1 (idempotency_key)

• D-2 ('dead_lettered' status) + D-3 (RLS policies), the envelope type, the chokepoint enqueue helper (with idempotency-key dedup), the chokepoint claim wrapper, the telemetry helper for Sentry/PostHog wiring, and the SCHEMA-QUICK-REFERENCE.md doc bump in the SAME commit per feedback_doc_freshness_guard_per_commit.

ACs landed (subset of spec §8 ACs 1-12):

• AC-6 (idempotency dedup helper + UNIQUE index) — contract-established in W1-C; full end-to-end validation deferred to W3 / §5.4.3.
• Partial contract footprint for AC-1, AC-3, AC-12 (envelope shape + helpers exist; lifecycle integration validation defers to W3).

Critical contracts established in W1 (verbatim from spec):

Idempotency producer formula (spec §5.5 / §10 D-1, V_W1B F-3 mandate):

idempotency_key := <job_type>:<scoped_id>:<YYYY-MM-DD>:<requestHash>
                                          ^^^^^^^^^^
                                          MANDATORY date/version bucket

Worker auth-context reconstruction rule (spec §4.2 PR-3, verbatim):

The worker reads payload.auth_context.user_id + .role from the queue row, then constructs a service-role client via createServiceClient() from @/lib/supabase/server.ts. Verbatim signature per lib/supabase/server.ts:58-69 […]. Role re-validation — REQUIRED before any privileged operation. The payload.auth_context.role is a SNAPSHOT taken at enqueue time. Between enqueue and worker processing (which can be minutes later), the user’s role in user_roles may have changed (admin → editor demotion, role revocation on staff change). The worker MUST re-fetch the current role and refuse to proceed if the live role is lower than the role required by the job type.

This contract is enforced by W2’s lib/queue/auth.ts (worker-side) — W1 documents the contract via helper TSDoc + JSDoc-style comments on the envelope type, so the W2 implementer has a verbatim-quoted target.

W2 — Worker route + dispatch shell + auth helper [S222 — DEFERRED]

Goal: Land app/api/cron/process-queue/route.ts (Option A per spec §4.1 D-4 default), lib/queue/dispatch.ts (the runJobByType switch), lib/queue/auth.ts (auth-context re-validation per spec §4.2), and lib/queue/failure.ts (the handleJobFailure retry/dead-letter classifier per spec §5.1-§5.4). Cron registration is added to vercel.json crons[] in the same commit. The worker dispatches NO job types yet (empty switch); each §5.4.x candidate migration spec adds a case.

ACs landed: AC-1, AC-2, AC-3, AC-4, AC-5, AC-7, AC-8, AC-11.

Carried from W1: None. W2 strictly extends W1 helpers; W1 contracts are immutable from W2 perspective.

W3 — §5.4.3 lifecycle integration tests [S223 — DEFERRED to that candidate spec]

Goal: Per spec §11.3, the §5.4.3 candidate spec defines a Vitest integration suite under __tests__/integration/queue/ exercising every AC in spec §8 against the staging Supabase branch (turayklvaunphgbgscat). The suite covers ACs 1-12 end-to-end with fixture seeding + service-role mocking strategy.

ACs validated (NOT newly landed — re-asserted end-to-end): AC-1 through AC-12 inclusive.

W4 — First migration candidate (chosen by Liam at W1 close-out) [DEFERRED to candidate spec]

Goal: Land ONE of §5.4.1 (batch-draft-all), §5.4.2 (batch-reclassify), §5.4.4 (markdown-batch). The roadmap §5.4 priority order suggests §5.4.1 first; Liam’s call. Each candidate migration spec is a separate spec-then-plan cycle — NOT in scope for THIS plan.

W5 — Operational dashboard + R10 follow-up (template_fill / template_analyse migration) [DEFERRED]

Goal: Surface queue-depth widgets in /admin/pipeline-runs/recent per spec §6.4. Plus R10 follow-up: migrate template_fill / template_analyse from scripts/bid_worker.py (Python daemon) to the Node.js cron worker per spec §9 R10 + decommission the Python daemon. Separate spec required.

---

W1 wave decomposition (THIS PLAN’S IMPLEMENTATION TARGET)

W1 splits into THREE parallel-safe agent slots (W1-A, W1-B, W1-C), sequential merge. Pre-flight slot (W1-PF) is mandatory before parallel dispatch per feedback_preflight_scan_before_parallel_dispatch.

Slot	Goal	Depends on	Estimated effort
W1-PF	Pre-flight scan: verify file paths exist, supabase link to staging, validate spec contracts vs current schema. Run BEFORE dispatching W1-A/B/C.	Nothing	~20 min
W1-A	DDL migration (D-1 + D-2 + D-3 + index renames cosmetic) + SCHEMA-QUICK-REFERENCE.md bump (same commit).	W1-PF	~60-75 min
W1-B	lib/queue/envelope.ts (TypeScript envelope type + Zod schema + producer formula helper).	W1-PF	~45 min
W1-C	lib/queue/enqueue.ts (chokepoint producer with idempotency dedup) + lib/queue/telemetry.ts (Sentry/PostHog wiring per spec §6.1-§6.2).	W1-A + W1-B	~60-75 min

Sequencing rationale:

• W1-PF first (always) — catch missing/renamed files, stale staging link, unexpected schema drift before agents start writing code.
• W1-A (DDL) and W1-B (envelope type) are independent of each other and can run in parallel.
• W1-C depends on BOTH (the enqueue helper imports the envelope type AND uses the new idempotency_key column). Must run after W1-A and W1-B merge.

---

§2 File ownership table per wave

Wave 1 file ownership (NEW + EXTEND files)

Strict no-overlap between agent slots. Each slot owns disjoint files; the verifier checks git status on main between merges per feedback_worktree_leaks + feedback_worktree_overwrite + feedback_worktree_edit_tool_leak.

File path	NEW/EXTEND	Slot	Notes
supabase/migrations/<TS>_s221_w1_queue_infra_d1_d2_d3.sql	NEW	W1-A	DDL migration. ALTER TABLE adds idempotency_key text. Partial UNIQUE index processing_queue_idempotency_key_uniq (idempotency_key) WHERE status IN ('pending', 'processing', 'completed'). Drops + re-adds status CHECK constraint to include 'dead_lettered'. DOES NOT widen job_type CHECK speculatively (Liam OQ-3 ratified S221 W3 = NO; per spec PR-2 each §5.4.x candidate widens its own job_type value when its candidate spec dispatches). Adds RLS policies per spec §10 D-3 (INSERT editor+, SELECT admin, UPDATE/DELETE admin). Cosmetic rename: processing_queue_task_type_check → processing_queue_job_type_check and idx_processing_queue_task_type → idx_processing_queue_job_type. NEW PL/pgSQL functions get explicit REVOKE FROM anon.
docs/reference/SCHEMA-QUICK-REFERENCE.md	EXTEND	W1-A	§8 processing_queue table: add idempotency_key text row + update status enum CHECK note to include 'dead_lettered'. DO NOT add forward job_type values (per OQ-3 above). SAME COMMIT as DDL migration per feedback_doc_freshness_guard_per_commit. Bump Last verified header.
docs/reference/product-roadmap.md	EXTEND	W1-A	TWO bundled edits in W1-A’s commit per Liam ratifications S221 W3: (a) D-10 §5.4 header copy correction — reword to “The public.processing_queue table exists but is only used by template operations (template_fill, template_analyse). The intelligence pipeline has its own separate si_processing_queue table and is unaffected by §5.4.x candidates.” (b) §2.3.4 OQ-6 row correction — replace stale “P0-WEB tracked in §1.6” pointer with “P0-WEB shipped (commit f332062c + d45548b2); §2.3.4 closes residual rate-limit + ETag + telemetry gaps.” Both are doc-correctness, not scope-expansion.
lib/queue/envelope.ts	NEW	W1-B	Exports QueueJobPayload<TBody> interface (verbatim from spec §3.1; quoted in TSDoc), JobType union (existing 8 values only: `‘embed'
lib/queue/enqueue.ts	NEW	W1-C	Exports enqueueQueueJob({ supabase, jobType, body, authContext, idempotencyKey?, pipelineRunId?, priority?, maxAttempts? }) async function. Checks idempotency_key dedup BEFORE INSERT (SELECT existing in ('pending', 'processing', 'completed') → return existing job_id). Constructs envelope per spec §3.1. INSERTs processing_queue row. Returns { jobId, deduplicated: boolean }. Uses sb() from @/lib/supabase/safe for fail-fast error handling per silent-failure spec.
lib/queue/telemetry.ts	NEW	W1-C	Exports emitQueueSentry({ stage, job, error?, level }) (Sentry capture per spec §6.1) and emitQueueAnalytics({ event, job, properties }) (PostHog emit per spec §6.2). PostHog event names verbatim: 'queue_job_completed', 'queue_job_failed', 'queue_job_dead_lettered', 'queue_job_cancelled'. The latter two events have no caller in W1 — exported for W2 consumption.
__tests__/lib/queue/envelope.test.ts	NEW	W1-B	Vitest unit tests: buildIdempotencyKey() produces the verbatim formula; envelope Zod schema rejects envelope_version: 999; JobStatus union matches spec §3.3 values exactly.
__tests__/lib/queue/enqueue.test.ts	NEW	W1-C	Vitest unit tests with mocked supabase: enqueue produces correct envelope shape; second call with same idempotency_key returns the same job_id with deduplicated: true; missing required envelope fields throw before INSERT. Uses createMockSupabaseClient() from @/__tests__/helpers/mock-supabase.
__tests__/lib/queue/telemetry.test.ts	NEW	W1-C	Vitest unit tests: Sentry capture spy receives expected tags and extra per spec §6.1; PostHog emit spy receives expected event names.
supabase/types/database.types.ts	EXTEND	W1-A	Auto-regenerated post-DDL via supabase gen types typescript --project-id rovrymhhffssilaftdwd. NOT manually edited per project convention. CLI command run with dangerouslyDisableSandbox: true. Strip the trailing 2 stderr lines per reference_supabase_gen_types_notice_leak.

No overlap rules:

• W1-A owns: 1 SQL migration + 1 markdown doc + 1 generated types file.
• W1-B owns: 2 TypeScript files in lib/queue/ + __tests__/lib/queue/.
• W1-C owns: 2 TypeScript files in lib/queue/ + 2 test files. Reads from W1-B’s envelope, reads from W1-A’s regenerated types.

Cross-slot read dependencies:

• W1-C reads lib/queue/envelope.ts (from W1-B).
• W1-C reads supabase/types/database.types.ts (from W1-A).
• Therefore W1-C runs AFTER W1-A and W1-B both merge to main.

Wave 2 file ownership (forward-look)

File path	NEW/EXTEND	Slot	Notes
app/api/cron/process-queue/route.ts	NEW	W2-A	Cron worker per spec §4.3 reference shape. export const maxDuration = 50; + TIMEOUT_BUFFER_MS = 40_000. Inside: verifyCronAuth(request) gate, then loop calling claim_next_job + runJobByType + writing terminal status. Returns { processed, succeeded, failed, retried, deadletter } summary.
lib/queue/dispatch.ts	NEW	W2-A	runJobByType(job, supabase) switch. Initial implementation: empty switch — every default returns failed with error_message: 'no_handler_registered: <job_type>'. Each §5.4.x candidate adds a case.
lib/queue/auth.ts	NEW	W2-A	reValidateAuthContext(serviceClient, userId, enqueuedRole, requiredRole) per spec §4.2 verbatim. Returns `{ ok: true }
lib/queue/failure.ts	NEW	W2-A	handleJobFailure(supabase, job, err) — classifies transient vs permanent per spec §5.1; on transient + attempts < max_attempts, requeues with backoff (linear-with-jitter per D-7); on exhausted, marks dead_lettered. Returns outcome enum `‘retried'
lib/queue/visibility-timeout.ts	NEW	W2-A	reapStuckJobs(supabase, opts) — UPDATE per spec §5.3 (status=‘pending’, attempts++ where status=‘processing’ AND started_at < NOW() - 5 min default). Called at the top of every cron tick before claiming new work.
vercel.json	EXTEND	W2-A	Add new entry under crons[]: { "path": "/api/cron/process-queue", "schedule": "*/5 * * * *" } per spec §10 D-9 default.
app/api/jobs/[id]/cancel/route.ts	NEW	W2-A	PATCH endpoint per spec §5.6: writes status='cancelled' only if current row is 'pending'; returns 409 Conflict if already processing.
__tests__/api/cron/process-queue.test.ts	NEW	W2-A	Vitest tests covering AC-1 through AC-5 + AC-7, AC-8 with mocked supabase.
__tests__/lib/queue/auth.test.ts	NEW	W2-A	Vitest tests covering AC-7 role-revalidation logic.
__tests__/lib/queue/failure.test.ts	NEW	W2-A	Vitest tests covering AC-2, AC-3 (retry + dead-letter classification).

Wave 3 file ownership (forward-look — §5.4.3)

File path	NEW/EXTEND	Slot	Notes
__tests__/integration/queue/lifecycle.integration.test.ts	NEW	W3-A	Per spec §11.3 + §5.4.3 forward-look. Exercises every AC in spec §8 against staging.
__tests__/integration/queue/concurrency.integration.test.ts	NEW	W3-A	AC-11 specifically — two parallel claim_next_job transactions cannot double-claim.
supabase/migrations/<TS>_s223_claim_next_job_backoff.sql	NEW	W3-A	Rewrites claim_next_job to add AND updated_at <= NOW() per spec §5.2 (backoff visibility window). Explicit REVOKE FROM anon since function body changes.
docs/reference/SCHEMA-QUICK-REFERENCE.md	EXTEND	W3-A	§32 RPC Functions row updated for new claim_next_job body. Last verified bump.

Verbatim contract anchors (referenced from §1 wave-map; reproduced here for file-author reference)

Job-payload envelope (spec §3.1, copy verbatim into lib/queue/envelope.ts TSDoc):

interface QueueJobPayload<TBody extends Record<string, unknown>> {
  /** Schema version for the envelope. Bump when incompatible. */
  envelope_version: 1;

  /** Auth context reconstructed by the worker. See §4.2. */
  auth_context: {
    /** UUID of the user who enqueued the job (for audit + role re-validation). */
    user_id: string;
    /** Snapshot of the user's role at enqueue time. The worker re-validates
     * against `user_roles` before performing any privileged operation. */
    role: 'admin' | 'editor' | 'viewer';
    /** Optional workspace scope, when the job operates on a single workspace. */
    workspace_id?: string;
  };

  /** Optional idempotency key. See §5.5 — when set, the queue refuses
   * a duplicate enqueue (existing pending/processing/completed row with the
   * same key returns the existing job_id rather than creating a new row).
   * Formula contract (MANDATORY when set, per §5.5 + D-1):
   * `<job_type>:<scoped_id>:<YYYY-MM-DD>:<requestHash>` — the date bucket
   * separates explicit re-run intent from same-day producer retries. */
  idempotency_key?: string;

  /** Optional pipeline_runs row UUID for cross-linking, when the job's
   * caller has already created (or pre-allocated) a pipeline_runs row.
   * The worker writes terminal status to the existing row instead of
   * creating a new one. See §6.3. */
  pipeline_run_id?: string;

  /** Job-type-specific body — opaque to the queue infra. */
  body: TBody;
}

createServiceClient() signature (spec §4.2, verbatim from lib/supabase/server.ts:58-69):

export function createServiceClient() {
  return createSupabaseClient<Database>(
    clientEnv.NEXT_PUBLIC_SUPABASE_URL,
    serverEnv.SUPABASE_SERVICE_ROLE_KEY,
    {
      auth: {
        persistSession: false,
        autoRefreshToken: false,
      },
    },
  );
}

reValidateAuthContext reference shape (spec §4.2, verbatim — copy into lib/queue/auth.ts in W2):

async function reValidateAuthContext(
  serviceClient: SupabaseClient<Database>,
  userId: string,
  enqueuedRole: 'admin' | 'editor' | 'viewer',
  requiredRole: 'admin' | 'editor',  // job-type-specific minimum
): Promise<{ ok: true } | { ok: false; reason: string }> {
  const { data, error } = await serviceClient
    .from('user_roles')
    .select('role')
    .eq('user_id', userId)
    .maybeSingle();
  if (error) return { ok: false, reason: `role_lookup_failed: ${error.message}` };
  if (!data) return { ok: false, reason: 'enqueueing user has no role record' };
  const currentRole = data.role as 'admin' | 'editor' | 'viewer';
  if (rolesAreLowerThan(currentRole, requiredRole)) {
    return {
      ok: false,
      reason: `enqueueing user role no longer authorised: enqueued=${enqueuedRole}, current=${currentRole}, required=${requiredRole}`,
    };
  }
  return { ok: true };
}

---

§3 AC-to-wave mapping

Every spec AC (1-12) and every Liam-decision (D-1..D-11) maps to exactly ONE landing wave. The “AC validated” wave is the wave at which the spec’s acceptance test passes; the “AC contract established” wave is the wave at which the contract’s code lands. They may differ — e.g. AC-6 (idempotency dedup) is contract-established in W1 (helper code + UNIQUE index land) and AC-validated in W3 (integration test runs against staging).

AC mapping

AC #	Description (verbatim from spec §8)	Contract established in	Validated end-to-end in	Notes
AC-1	Enqueue + claim + complete (round-trip).	W1-C (enqueue) + W2-A (claim+complete)	W3	W1 ships enqueue side; W2 ships worker side; W3 ships integration test.
AC-2	Transient failure triggers retry.	W2-A (handleJobFailure)	W3	lib/queue/failure.ts classifies + requeues. Visibility-timeout backoff window relies on §5.2 claim_next_job rewrite (W3 migration).
AC-3	Retry exhaustion → dead-letter.	W1-A (status enum) + W2-A (failure classifier)	W3	Status enum widening lands in W1; classifier logic lands in W2; integration test in W3.
AC-4	Permanent failure does not retry.	W2-A (failure classifier)	W3	Envelope-validation rejection path is in lib/queue/dispatch.ts (W2).
AC-5	Stuck job is reaped (visibility timeout).	W2-A (visibility-timeout)	W3	lib/queue/visibility-timeout.ts lands in W2; cron tick calls it before each claim_next_job loop.
AC-6	Duplicate enqueue with same idempotency_key returns existing row.	W1-C (enqueue helper) + W1-A (UNIQUE index)	W3	THIS PLAN’S PRIMARY AC LANDING TARGET. Helper checks dedup pre-INSERT; UNIQUE index enforces race-safety post-INSERT.
AC-7	Worker reconstructs auth context + re-validates role.	W2-A (auth.ts)	W3	Reference shape (verbatim) included in §2 file ownership notes for W2; W1 documents the contract via TSDoc on the envelope’s auth_context field.
AC-8	Service-role client is used for handler work, not the user’s auth-scoped client.	W2-A (cron route)	W3	Cron route imports createServiceClient(); spy in unit test asserts no getAuthorisedClient/createClient call.
AC-9	Pending job cancellation transitions to cancelled.	W2-A (cancel route)	W3	New app/api/jobs/[id]/cancel/route.ts — only cancels rows in 'pending' status.
AC-10	Processing job cannot be cancelled by request.	W2-A (cancel route)	W3	Same route returns 409 Conflict when status is 'processing'.
AC-11	Two concurrent worker invocations do not double-claim.	(already shipped — claim_next_job FOR UPDATE SKIP LOCKED)	W3	The contract is already in production. W3 integration test asserts via Promise.all that the existing primitive holds.
AC-12	Caller-allocated pipeline_run_id is finalised by worker (Pattern 2).	W1-B (envelope pipeline_run_id field) + W2-A (worker writes terminal)	W3	Envelope field exists from W1; terminal-write logic lands in W2; integration test in W3.

ACs landed (contract-established) in W1 only: AC-6 (idempotency dedup helper + UNIQUE index). Partial contract footprint for AC-1, AC-3, AC-12.

No orphan ACs. Every AC 1-12 is mapped.

Liam-decision (D-x) mapping

D-#	Decision (paraphrase — see spec §10 for verbatim)	Lands in	Notes
D-1	Add idempotency_key text + partial UNIQUE index (RATIFIED S218 W3 with date/version-bucket mandate in producer formula).	W1-A (DDL) + W1-B (buildIdempotencyKey() enforces formula)	Migration adds column + index; envelope helper enforces formula contract.
D-2	Add 'dead_lettered' status value to CHECK constraint.	W1-A (DDL)	Drop + recreate processing_queue_status_check constraint with new value.
D-3	RLS policies on processing_queue (INSERT editor+, SELECT admin, UPDATE/DELETE admin; explicit REVOKE EXECUTE ON FUNCTION ... FROM anon).	W1-A (DDL)	NEW PL/pgSQL helpers in W1-A (enqueue_queue_job, requeue_queue_job_with_backoff) MUST each include explicit REVOKE EXECUTE ON FUNCTION public.<name>(<sig>) FROM anon; per feedback_supabase_pg_default_acl_anon_execute. claim_next_job already has this REVOKE (S20 OPS-43:186) — no re-revoke needed unless its body changes.
D-4	Worker invocation surface — Option A (Vercel cron + Next.js route).	W2-A	Cron route at app/api/cron/process-queue/route.ts + vercel.json registration.
D-5	Per-job runtime cap inside cron-tick budget — 30s.	W2-A	Worker enforces via Promise.race([handler, sleep(30_000).then(() => throw new RuntimeCapError())]).
D-6	max_attempts default — 3 (existing column default).	(already shipped)	Column default already 3; no migration needed.
D-7	Backoff policy — linear-with-jitter (attempts × 30s) + random(0..5000ms).	W2-A (failure.ts) + W3 (claim_next_job rewrite)	Backoff timestamp written by failure.ts; visibility-window enforced by W3’s claim_next_job rewrite.
D-8	Visibility timeout duration — 5 minutes.	W2-A (visibility-timeout.ts)	reapStuckJobs uses 5-minute interval per default; configurable via parameter.
D-9	Cron schedule frequency — */5 * * * * (every 5 min, bumped to 1/min on backlog signal).	W2-A (vercel.json)	Cron entry registered at every-5-min by default.
D-10	Roadmap §5.4 header copy correction — clarify intelligence pipeline uses separate si_processing_queue table.	W1-A (doc edit, BUNDLED)	RATIFIED Liam S221 W3 (OQ-1 = bundle): edit lands in W1-A’s commit alongside SCHEMA-QUICK-REFERENCE.md + DDL. Same commit also includes §2.3.4 row correction per spec OQ-6 (stale §1.6 pointer replacement).
D-11	REJECTED-WITH-REASONING: external queue backends (SQS, QStash, Inngest, BullMQ, pgmq, Trigger.dev). PG default confirmed via PR-1.	(already documented in spec §10)	No code lands. The §1 wave map preserves PG default via the §1 premises. Re-evaluable post-launch via separate spec.

No orphan decisions. Every D-1..D-11 is mapped to a wave OR explicitly documented as no-code-lands (D-6 column default, D-11 rejection).

---

§4 Verification gates

A verifier sub-agent runs after each wave. The verifier reads the spec section + the impl files + asserts AC coverage + spec-contract verbatim match + gotcha compliance per feedback_fix_all_verification_findings.

W1 verification gate

Verifier reads (in order):

1. docs/specs/background-queue-infra-spec.md §3 (envelope), §4.2 (auth context), §5.5 (idempotency contract), §6 (telemetry), §10 D-1, D-2, D-3 (Liam-decisions ratified S218 W3-W4).
2. supabase/migrations/<TS>_s221_w1_queue_infra_d1_d2_d3.sql (W1-A output).
3. lib/queue/envelope.ts (W1-B output).
4. lib/queue/enqueue.ts, lib/queue/telemetry.ts (W1-C outputs).
5. docs/reference/SCHEMA-QUICK-REFERENCE.md §8 + Last-verified header (W1-A output, same commit).

Verifier asserts (PASS = all green; FAIL = any RED):

• C1 (DDL contract): Migration adds idempotency_key text to processing_queue. Partial UNIQUE index (idempotency_key) WHERE status IN ('pending', 'processing', 'completed') exists. Status CHECK constraint includes 'dead_lettered'. Job-type CHECK MUST NOT include forward types 'bid_draft_all'/'batch_reclassify'/ 'markdown_batch' (Liam OQ-3 ratified S221 W3 = NO speculative widen — verifier FAILS if any of these values appear in W1-A’s CHECK). RLS policies match D-3. NEW PL/pgSQL functions (if any) have explicit REVOKE EXECUTE FROM anon.
• C2 (Idempotency formula verbatim): lib/queue/envelope.ts buildIdempotencyKey() produces strings matching the regex ^[a-z_]+:[A-Za-z0-9_-]+:\d{4}-\d{2}-\d{2}:[A-Za-z0-9_-]+$ (the spec §5.5 formula <job_type>:<scoped_id>:<YYYY-MM-DD>:<requestHash>). TSDoc on buildIdempotencyKey quotes the contract VERBATIM (string-match against spec §5.5 lines 805-809).
• C3 (Envelope verbatim): lib/queue/envelope.ts QueueJobPayload interface matches the spec §3.1 type declaration field-for-field (envelope_version: 1, auth_context.{user_id, role, workspace_id?}, idempotency_key?, pipeline_run_id?, body). TSDoc comments on each field include the verbatim text from spec §3.1.
• C4 (Auth-context contract documented): TSDoc on the auth_context field of QueueJobPayload includes the verbatim spec §4.2 quote about role re-validation: “The worker re-validates against user_roles before performing any privileged operation.”
• C5 (Status enum widening pair): TS JobStatus union in lib/queue/envelope.ts includes 'dead_lettered'. bunx tsc --noEmit succeeds (no TS errors from regenerated database.types.ts union mismatch) per feedback_db_check_ts_union_paired_widening.
• C6 (Idempotency dedup behaviour): lib/queue/enqueue.ts SELECTs by idempotency_key BEFORE INSERT, filters IN ('pending', 'processing', 'completed'). If a row is found, returns { jobId: existing.id, deduplicated: true }. Test __tests__/lib/queue/enqueue.test.ts covers this case.
• C7 (Telemetry signal shapes): lib/queue/telemetry.ts Sentry capture call includes tags: { queue: 'processing_queue', job_type } per spec §6.1. PostHog event names match the spec §6.2 verbatim table: 'queue_job_completed', 'queue_job_failed', 'queue_job_dead_lettered', 'queue_job_cancelled'.
• C8 (Doc-freshness coupling): SCHEMA-QUICK-REFERENCE.md §8 row for idempotency_key text exists; status enum CHECK note includes 'dead_lettered'; job_type enum row includes new values; Last verified header bumped to today’s date in the SAME commit as the DDL migration. Verifier runs git log -1 --stat and confirms BOTH files appear per feedback_doc_freshness_guard_per_commit.
• C9 (Anon revoke compliance): Any NEW public.*() PL/pgSQL function in the W1-A migration has an explicit REVOKE EXECUTE ON FUNCTION public.<name>(<sig>) FROM anon; in the SAME migration file per feedback_supabase_pg_default_acl_anon_execute.
• C10 (No barrel re-exports / silent failure compliance): lib/queue/enqueue.ts uses sb() from @/lib/supabase/safe for any DB write that doesn’t return an envelope; uses tryQuery() for the dedup SELECT. No raw await supabase.from(...) without checked error handling per silent-failure spec.
• C11 (Validation-sweep guard pre-emptive): lib/queue/enqueue.ts is NOT a route file (it’s a helper), so the validation-sweep.test.ts guard does not apply here. W2’s cancel route at app/api/jobs/[id]/cancel/route.ts MUST import parseBody from @/lib/validation per feedback_validation_sweep_guard — noted as W2 risk, not W1.
• C12 (Vitest changed-scope passes): bun run test (or vitest --changed) passes for the three new test files. Full suite is NOT required for W1 (chokepoint helpers; full regression after W1 merge per project convention).

Verifier outputs: report with HIGH / MEDIUM / LOW finding labels. ALL HIGH and MEDIUM must be fixed before merge per feedback_fix_all_verification_findings. LOW with explicit spec backing may pass; LOW without spec backing is treated as MEDIUM.

Self-fix file-vs-DB parity check (per feedback_subagent_self_fix_file_db_parity): If W1-A self-fixes its DDL mid-impl, the verifier MUST confirm the fix exists in the migration FILE on disk, not only in the staging DB state (MCP apply_migration would write to staging without updating the local file).

W2 verification gate (forward-look)

Reads spec §4 (worker pattern), §4.2 (auth context — full), §5 (retry/backoff/dead-letter/visibility timeout), §10 D-4..D-9. Verifies the worker route exists at app/api/cron/process-queue/route.ts with maxDuration = 50; verifyCronAuth(request) is the first call; createServiceClient() (NOT any other client constructor) is the second call; reValidateAuthContext() matches spec §4.2 verbatim; vercel.json crons[] includes the new entry; cancel route returns HTTP 409 on processing status. Runs full Vitest suite + bun run build per feedback_subagent_verifier_must_build (Turbopack catches client/server bundling failures that lint+test miss).

W3 verification gate (forward-look)

Reads spec §8 (all ACs verbatim) + §11.3 (5.4.3 forward-look). Verifier runs the integration suite against the staging branch (turayklvaunphgbgscat) and asserts every AC 1-12 passes. Verifier re-runs bun run test:e2e to catch any UX regressions for the (still sync) bid-draft-all path that doesn’t migrate until §5.4.1 ships.

W4 / W5 verification gates (forward-look)

Each candidate spec defines its own verification gate. Cross-cutting gate: every candidate MUST land a bun run knip clean run AND an E2E spec asserting “click button → see queued state” UX shift per spec §9 R6 (“Build the thing, forget to turn it on” mitigation).

---

§5 Carry-forward + risks

Items deferred from W1 to W2+ (carried forward)

The following items are EXPLICITLY out of scope for W1, with their owning wave noted. Each will have a brief in its owning session per the spec-then-plan workflow.

1. Worker route handler at app/api/cron/process-queue/route.ts → W2-A.
2. vercel.json cron registration for the new route → W2-A.
3. Auth-context re-validation helper lib/queue/auth.ts → W2-A.
4. Failure classifier lib/queue/failure.ts (transient vs permanent
   • retry/dead-letter) → W2-A.
5. Visibility-timeout reaper lib/queue/visibility-timeout.ts → W2-A.
6. Cancel endpoint app/api/jobs/[id]/cancel/route.ts → W2-A.
7. claim_next_job rewrite for backoff visibility window (AND updated_at <= NOW()) → W3 (lands in §5.4.3 lifecycle-tests migration per spec §5.2).
8. Operational dashboard widgets in /admin/pipeline-runs/recent → W5 (post-first-candidate ship).
9. R10 follow-up: migrate template_fill / template_analyse from scripts/bid_worker.py to Node.js worker + decommission Python daemon → W5 (post-first-candidate ship).
10. Roadmap §5.4 header copy correction (D-10) → bundled into W1-A if Liam confirms at W1 dispatch; otherwise defer to first §5.4.x close-out.
11. Integration test scaffolding under __tests__/integration/queue/ → W3 (per spec §11.3, this IS the §5.4.3 candidate spec).
12. Each §5.4.x migration spec authoring (5.4.1, 5.4.2, 5.4.4) → separate spec-then-plan cycles per spec PR-2.

Risks specific to this plan (W1 risk surface)

Risk #	Description	Likelihood	Impact	Mitigation
R-A	'dead_lettered' CHECK widening + TS union widening MUST be paired. Per feedback_db_check_ts_union_paired_widening AND feedback_check_constraint_app_enum_drift (paired gotchas): a DB CHECK widening that lands without the corresponding TS union update will silently break TS callers — staging DB rows with status = 'dead_lettered' deserialise to TS JobStatus of `‘pending'	'processing'	'completed'	'failed'
R-B	NEW PL/pgSQL helpers in W1-A’s migration could silently grant anon EXECUTE. Per feedback_supabase_pg_default_acl_anon_execute: every new public.*() function gets an auto-grant on creation; REVOKE FROM PUBLIC is no-op against the anon role.	Medium	High	W1-A authoring brief explicitly enumerates: “every new function in this migration MUST include REVOKE EXECUTE ON FUNCTION public.<name>(<sig>) FROM anon; in the SAME file”. Verifier C9 asserts this. NOTE: existing claim_next_job already has REVOKE per 20260502143049_ops43_revoke_anon_execute_public_functions.sql:186.
R-C	Doc-freshness guard reference-doc-edit-coupled-freshness requires SCHEMA-QUICK-REFERENCE.md edit + Last-verified header bump in the SAME commit. Per feedback_doc_freshness_guard_per_commit + feedback_reference_doc_freshness_most_recent_only: a follow-up “fix freshness” commit fails the guard.	High	Low	W1-A authoring brief mandates: SCHEMA-QUICK-REFERENCE.md §8 edit + Last-verified header bump + DDL migration ALL in ONE commit. Verifier C8 confirms via git log -1 --stat.
R-D	Validation-sweep guard validation-sweep.test.ts will not catch route-side issues until W2. W1 ships no routes; W2’s cancel route at app/api/jobs/[id]/cancel/route.ts MUST import parseBody from @/lib/validation even though there’s no body parsing.	Low	Medium	Tracked as W2 risk (in W2 verification gate brief, not W1). W1’s verifier C11 explicitly notes this guard does not apply to W1 helpers.
R-E	Supabase CLI db push to wrong project. Per feedback_supabase_cli_silent_apply_failure: a fresh worktree’s supabase/.temp/project-ref may be unset or stale; db push without supabase link --project-ref turayklvaunphgbgscat first will fail or push to prod by accident.	Medium	High	W1-PF (pre-flight) MANDATES: (a) cat supabase/.temp/project-ref to confirm staging linked; (b) if missing or wrong, run supabase link --project-ref turayklvaunphgbgscat FIRST. W1-A authoring brief reproduces this requirement at the top of the brief.
R-F	Silent-failure compliance: any raw await supabase.from(...) without checked error handling will trip the ESLint rule local/no-unchecked-supabase-error. W1-C’s enqueue helper has multiple supabase calls.	Medium	Medium	W1-C authoring brief mandates: use sb() (fail-fast) for the INSERT; use tryQuery() (Result-returning) for the dedup SELECT. Verifier C10 confirms.
R-G	Worktree agent base-branch staleness. Per feedback_worktree_branching_stale + feedback_worktree_agent_reset_origin: an agent dispatched with isolation: "worktree" branches from a HISTORICAL commit; if W1-A merges before W1-B starts work, W1-B may overwrite W1-A’s changes on cherry-pick.	Medium	High	Each W1 agent’s first action MUST be git fetch origin main && git reset --hard origin/main. Plan §5 mandates cherry-pick (not merge) of agent branches per feedback_worktree_branching_stale. Verifier checks git status on main between merges per feedback_worktree_leaks.
R-H	Bash CWD drift. Per feedback_cwd_drift_into_worktree: bash CWD drifts into worktree dirs after Read tool calls; subsequent git commands operate on the wrong tree.	Medium	Medium	Each agent brief includes: prefix git operations with cd <main-repo-path> && after any Read on worktree files.
R-I	mcp__supabase__apply_migration timestamp drift. Per feedback_mcp_apply_migration_timestamp_drift: if W1-A self-fixes its DDL via MCP apply_migration (only on documented blockers per feedback_supabase_cli_silent_apply_failure), the server-generated timestamp will diverge from local file naming.	Low	Medium	W1-A brief: prefer CLI supabase db push ALWAYS. MCP apply_migration ONLY if CLI is documented-blocked. If MCP path is taken, brief mandates re-pulling schema_migrations post-apply, renaming local file, UPDATEing staging row to align timestamps.
R-J	Sub-agent context budget for W1-A. Per feedback_sub_agent_context_budget: agents are hard-limited to 200K tokens. The W1-A brief is policy-heavy (DDL contract + RLS matrix + REVOKE FROM anon discipline + doc-freshness guard + types regen with stderr stripping). Risk of running out of budget before commit.	Medium	High	W1-A brief is split into a “policy block” (verbatim quotes) + a “task block” (concrete migration SQL skeleton). Brief target ≤8000 words. Agent’s final action MUST be git status confirmation per feedback_sub_agent_context_budget.
R-K	Subagent self-fix file-vs-DB parity drift. Per feedback_subagent_self_fix_file_db_parity: if W1-A self-fixes a DDL bug mid-impl, the verifier must confirm the fix exists in the FILE on disk (not only in staging DB state).	Low	High	Verifier explicitly checks git show HEAD -- supabase/migrations/ against staging schema_migrations row.

Risks for W2+ (forward-look only, not blocking W1)

• W2 R-A: EP2 markdown-batch UI polling change (spec §9 R8) — handled by §5.4.4 candidate spec, not by core infra W2.
• W2 R-B: R9 worker invocation drift (spec §9 R9) — mitigation lives in W2’s plan: SINGLE app/api/cron/process-queue/route.ts route, no per-candidate routes.
• W3 R-A: Integration test fixture seeding strategy not yet defined — handled by §5.4.3 candidate spec.

---

§6 Plan-self-verification (per feedback_self_verify_own_specs)

Re-read pass against the prompt’s acceptance criteria:

• Every spec AC (1-12) appears in §3 mapping table; zero orphans.
• Every D-x decision (D-1..D-11) appears in §3 mapping table; zero orphans (D-6 + D-11 explicitly documented as no-code-lands).
• File ownership table covers every NEW + EXTEND path inferred from spec §6 (Observability) + §11 (candidate appendix) + §4-5 (helper shapes referenced inline).
• Wave sequencing respects DDL → helpers → worker → cron → telemetry partial order: W1 (DDL + envelope helpers + telemetry) → W2 (worker + cron + dispatch + auth + failure + visibility-timeout + cancel) → W3 (lifecycle integration tests + claim_next_job rewrite) → W4 (first migration candidate) → W5 (dashboard + R10).
• All four verbatim contracts appear without paraphrase:
  • Idempotency formula in §1 wave map (W1 critical contracts), §2 (W1-B file ownership notes), §4 W1 verifier C2.
  • Job-payload envelope interface in §2 (verbatim contract anchors).
  • createServiceClient() signature in §2 (verbatim contract anchors).
  • Status enum widening pair in §5 R-A.
• Plan length 600-900 lines target — current draft is in that range after self-edit pass.
• Plan structure mirrors p0-1-edit-path-consolidation-plan.md (phase + task table + AC traceability + risk mitigations) and p0-bm-phase2-bid-write-side-plan.md (work-package structure + DAG + verification mapping).
• Gotchas flagged in §5: feedback_supabase_pg_default_acl_anon_execute (R-B), feedback_check_constraint_app_enum_drift (R-A by cross-reference), feedback_db_check_ts_union_paired_widening (R-A), feedback_doc_freshness_guard_per_commit (R-C), feedback_validation_sweep_guard (R-D), and feedback_supabase_cli_silent_apply_failure (R-E). Plus additional protective gotchas (R-F through R-K).

---

§7 Open questions for Liam (raise at W1 dispatch, NOT blocking authoring)

All four PLAN-A authoring OQs RATIFIED by Liam at S221 W3 close-out:

1. OQ-1 D-10 bundling decision — RATIFIED bundle. Roadmap §5.4 header copy correction lands in W1-A’s commit alongside DDL + SCHEMA-QUICK-REFERENCE.md edit. (Same commit also picks up §2.3.4 row correction per spec OQ-6 ratification.)
2. OQ-2 W1-A migration filename — RATIFIED keep. Use placeholder <TS>_s221_w1_queue_infra_d1_d2_d3.sql — matches recent session-prefixed migrations (e.g. 20260502143049_ops43_revoke_anon_execute_public_functions.sql).
3. OQ-3 Job type widening additivity — RATIFIED NO (do NOT speculatively widen). Per spec PR-2, each §5.4.x candidate widens its own job_type value when its candidate spec dispatches. Speculative widening would lock job-type names before their candidate specs ratify them; if a candidate renames later, another CHECK migration would be needed anyway. W1-A migration + W1-B JobType TS union restricted to the existing 8 values.
4. OQ-4 Plan re-author trigger for W2 — RATIFIED yes. W2-W5 forward-look is informational only; each wave gets its own re-planning at its session entry per feedback_self_verify_own_specs.

---

§8 Pre-flight (W1-PF) checklist (run BEFORE dispatching W1-A/B/C)

Per feedback_preflight_scan_before_parallel_dispatch:

• cat supabase/.temp/project-ref returns turayklvaunphgbgscat (staging). If missing or wrong: supabase link --project-ref turayklvaunphgbgscat.
• ls lib/queue/ returns No such file or directory (confirms NEW directory will be created cleanly; not an empty dir from previous attempt).
• grep -n "^## 8. processing_queue" docs/reference/SCHEMA-QUICK-REFERENCE.md returns a single non-empty match (heading-by-content; line number is not load-bearing — drift acceptable, missing heading is the failure).
• grep -rn "claim_next_job" supabase/migrations/ | wc -l returns 6 (3 in pre-squash + 1 in S221 OPS-43 + 2 GRANT lines per current audit). Confirms no in-flight claim_next_job migration would conflict.
• git rev-parse HEAD matches git ls-remote origin main | head -1 | cut -f1. Pre-flight slot’s first action mirrors each agent’s git fetch origin main && git reset --hard origin/main.
• Confirm EVAL_LIVE_TEST flag posture per feedback_eval_live_test_promote_to_pr — irrelevant to W1 but sanity-check session env hasn’t drifted.

---

§9 Plan acceptance per prompt criteria

• Every spec AC (1-12) and D-x (D-1..D-11) appears in §3 mapping table; zero orphans.
• File ownership table covers every NEW and EXTEND path listed in spec §6 (Observability — telemetry helper signature) + §11 (candidate appendix — file inventory inferred via §4 worker reference shape).
• Wave sequencing respects DDL → helpers → worker → cron → telemetry partial order. W1 lands DDL + envelope helpers + telemetry (chokepoint); W2 lands worker + cron registration; W3 lands integration tests + claim_next_job rewrite for backoff; W4 lands first migration candidate; W5 lands dashboard + R10.
• All four verbatim contracts appear unparaphrased (idempotency formula in §1 + §4 verifier C2; envelope interface in §2 verbatim anchors block; createServiceClient signature in §2; status enum widening pair note in §5 R-A).
• Plan length is in 600-900 line target (current draft).
• Plan structure mirrors the two reference plans.

---

§10 References

Source spec

• docs/specs/background-queue-infra-spec.md v1 (S218 W1 SPEC-B authoring; ratified at S218 W4 close-out on D-1..D-11).

Reference plans (shape mirrored)

• docs/plans/p0-1-edit-path-consolidation-plan.md (4-phase / file ownership table / AC traceability / risk mitigations).
• docs/plans/p0-bm-phase2-bid-write-side-plan.md (work-package structure / dependency DAG / verification mapping).

Code anchors (read during plan authoring)

• supabase/migrations/20260416102457_pre_squash_reconciliation.sql:350-389 — processing_queue table + claim_next_job function.
• supabase/migrations/20260419095200_reconcile_squash_drift_pipeline_runs_processing_queue.sql:33 — task_type → job_type rename.
• supabase/migrations/20260502143049_ops43_revoke_anon_execute_public_functions.sql:183-186 — existing REVOKE for claim_next_job.
• lib/supabase/server.ts:43-69 — createServiceClient() doc-comment + signature.
• lib/pipeline/record-run.ts — recordPipelineRun() reference for W2’s pipeline_runs Pattern 2 finalisation.
• app/api/cron/quality-score/route.ts:23,29 — cron pattern reference (maxDuration = 50, TIMEOUT_BUFFER_MS = 40_000).

Memory feedback applied during plan authoring

• feedback_spec_then_plan_workflow — drove this plan’s existence (the spec defers planning to a separate pass).
• feedback_self_verify_own_specs — §6 plan-self-verification pass.
• feedback_brief_quote_spec_verbatim — §1 + §2 + §4 quote spec contracts verbatim, no paraphrase.
• feedback_db_check_ts_union_paired_widening — §5 R-A.
• feedback_supabase_pg_default_acl_anon_execute — §5 R-B + §4 verifier C9.
• feedback_doc_freshness_guard_per_commit — §5 R-C + §4 verifier C8.
• feedback_check_constraint_app_enum_drift — §5 R-A (cross-reference).
• feedback_validation_sweep_guard — §5 R-D + §4 verifier C11.
• feedback_supabase_cli_silent_apply_failure — §5 R-E + §8 W1-PF.
• feedback_worktree_branching_stale + feedback_worktree_leaks + feedback_worktree_overwrite + feedback_worktree_edit_tool_leak + feedback_cwd_drift_into_worktree — §5 R-G + R-H + cherry-pick discipline in §5 carry-forward.
• feedback_subagent_verifier_must_build — §4 W2 verifier brief.
• feedback_subagent_self_fix_file_db_parity — §5 R-K + §4 verifier W1-self-fix note.
• feedback_sub_agent_context_budget — §5 R-J.
• feedback_mcp_apply_migration_timestamp_drift — §5 R-I.
• feedback_reference_doc_freshness_most_recent_only — §5 R-C (cross-reference).
• feedback_preflight_scan_before_parallel_dispatch — §8 W1-PF.
• feedback_fix_all_verification_findings — §4 verifier discipline.
• reference_supabase_gen_types_notice_leak — §2 file-ownership note for supabase/types/database.types.ts regen.

---

Document lifecycle

• v1 DRAFT (02/05/2026, S221 W1 PLAN-A) — initial plan authoring. Liam ratification at session entry / W1 dispatch on §7 open questions (D-10 bundling, migration filename, speculative job-type widening). Implementation dispatch (W1-A/B/C parallel) follows ratification. Verifier (per §4) runs after each agent merge.