Queue Operational Dashboard — Specification
Queue Operational Dashboard — Specification
Section titled “Queue Operational Dashboard — Specification”Status: v1 DRAFT — S225 W3-A authoring (05/05/2026). Decisions
require Liam ratification at S225 close-out (or subsequent session).
Single-pass spec — V_W3 verifier wave to follow before ratification per
feedback_self_verify_own_specs.
Severity: forward-looking surface spec. SPEC ONLY — IMPL is deferred
to a follow-up session. This candidate ratifies the surface choice +
metric set + data source + alert thresholds so a future IMPL session
can dispatch with no further design questions. No migration files, no
UI page, no RPCs ship in this candidate.
Author: Claude Code (S225 W3-A SPEC sub-agent).
Roadmap placement: docs/reference/product-roadmap.md §5.4 W5+
operational tweaks — “Queue operational dashboard for processing_queue
visibility” — 1-2 sessions, Could (post §5.4 W4 + W5 ship).
Origin: docs/specs/§5.4.1-batch-draft-all-spec.md §1.3 (out-of-scope
forward-look — “A queue-depth dashboard for bid_draft_all is deferred
to operational dashboard spec post-§5.4 W5”) + docs/specs/§5.4.2-batch-reclassify-spec.md
§1.3 (same forward-look) + docs/specs/background-queue-infra-spec.md
§9 R3-R4 (“table bloat” + “operator visibility”) forward-look. Per
infra-spec PR-2 the dashboard surfaces the queue but does NOT introduce a
new job-type so it has its OWN spec rather than being a §5.4.x candidate.
Dispatch prerequisites (for IMPL session):
- §5.4.1 batch-draft-all spec IMPL SHIPPED (the §5.4.1 IMPL is the first
consumer of the queue’s terminal-state semantics; without it the
dashboard has no real job-type traffic to surface beyond
template_fill/template_analyse). - §5.4.2 batch-reclassify spec IMPL SHIPPED (second consumer; admin-only workspace-wide writes; the dashboard’s per-job_type drill-down depends on having ≥2 job-types live to validate the IA actually works).
- §5.4 W5 operational cron-frequency tweak (
*/5 * * * *→* * * * *) SHIPPED — without minute-granularity ticks the age-of-oldest-pending metric is bounded below by tick interval, which trivialises the metric. - W3 lifecycle integration tests SHIPPED (§5.4.3 spec) — provides the test harness this dashboard’s AC-3 alert assertions reuse.
Pre-resolved decisions (PR-x)
Section titled “Pre-resolved decisions (PR-x)”The following are PR (pre-resolved) at spec authoring time — they do NOT appear in the §11 decision matrix because the parent infra spec or prior ratification already locked them in.
- PR-1:
processing_queueis the canonical source of queue state. Per infra spec PR-1 + §5.4.1 PR-1 + §5.4.2 PR-1. The dashboard readsprocessing_queue(andpipeline_runsfor pipeline-level joins) — NOT a separate metrics store, NOT a sidecar Postgres view that mirrors the data, NOT a Sentry/PostHog snapshot. Sentry + PostHog are observability sinks downstream ofprocessing_queue; the dashboard treats them as alternative renderings, not authoritative state. - PR-2: Worker invocation surface remains the existing cron route.
Per infra spec D-4. The dashboard does NOT introduce a new cron route,
a new background poller, or a new sidecar process. It reads from the
same
processing_queuetable the cron worker drains. - PR-3: Admin-only access. The dashboard is admin-surface only.
processing_queueRLS already restricts SELECT to admin perprocessing_queue_select_admin(verbatim fromdocs/reference/SCHEMA-QUICK-REFERENCE.md§8 RLS row). The dashboard page + the metric-source RPCs/endpoints inheritrequiredRole: 'admin'viagetAuthorisedClient(['admin'])per the existing admin-route pattern (e.g.app/api/admin/pipeline-runs/recent/route.ts:50). - PR-4: Sentry + PostHog event names verbatim. The four PostHog
terminal-state event names (
queue_job_completed,queue_job_failed,queue_job_dead_lettered,queue_job_cancelled) and the three Sentry signal stages (invocation,handler,visibility_timeout) are the contract fromlib/queue/telemetry.ts:28-30, 169-172+ the comment at L75-83. The dashboard MUST NOT rename, re-case, or coalesce these — PostHog dashboards + Sentry inbox filters depend on the exact strings. - PR-5: No new auth flows. No new role types, no new role bindings. Admin-or-deny per PR-3.
- PR-6: pipeline_runs Pattern 2 finalisation is preserved. The
dashboard’s “is the pipeline_run row live + writing forward progress?”
signal reads
pipeline_runs.completed_at IS NULLagainst the producer’s pre-allocated row (Pattern 2 per infra §6.3 + §5.4.1 §6.3). The dashboard does NOT re-implement, override, or replace this contract.
Table of Contents
Section titled “Table of Contents”- Summary + Scope
- Background + Current State
- Surface — what the dashboard IS
- Metric set — what it surfaces
- Data sources
- UX flows
- Acceptance Criteria
- Risks + Watch-outs
- Decisions Required (Liam ratification)
- References
1. Summary + Scope
Section titled “1. Summary + Scope”1.1 What this spec ships
Section titled “1.1 What this spec ships”A formal contract for the operator-facing queue operational
dashboard surfacing background-queue health metrics: queue depth per
job-type, dead-letter count per job-type, age-of-oldest-pending per
job-type, mean runtime per job-type, retry-rate per job-type. The
dashboard is admin-only, lives at /admin/queue (per D-1 authored
default), and reads from processing_queue aggregations via new
PL/pgSQL RPCs (per D-3 authored default).
The deliverable is a specification + AC matrix for Liam ratification. IMPL is deferred to a follow-up session. No migration files, no RPCs, no UI page, no test cases ship in this candidate. This candidate exists to ratify the surface + metric set + data source + alert thresholds so a future IMPL session can dispatch with no further design questions.
1.2 What is in scope (of THIS spec — not the IMPL it gates)
Section titled “1.2 What is in scope (of THIS spec — not the IMPL it gates)”- A surface choice: dedicated
/admin/queuepage vs admin-tile-only vs Sentry/PostHog-only (D-1 ratification surfaces the choice). - A metric set: minimum vs comprehensive (D-2 ratification surfaces the choice). Each metric has a definition + data-source pointer.
- A data source: direct
processing_queueaggregations vs new RPCs vs Sentry/PostHog (D-3 ratification surfaces the choice). - Alert thresholds for Sentry warn vs page operations (D-4 ratification surfaces the choice). Two tiers: WARN (Sentry warning level, no paging) and PAGE (Sentry error level + PagerDuty if integrated).
- A refresh cadence: manual vs auto-refresh (D-5 surfaces the choice).
- A cost-surface decision: show $ per-job-type estimates or hide (D-6 surfaces the choice).
- Acceptance criteria for the IMPL session to gate against (§7).
- Risks + mitigations the IMPL session must address (§8).
- A list of dispatch prerequisites the IMPL session inherits (front matter “Dispatch prerequisites”).
1.3 What is out of scope
Section titled “1.3 What is out of scope”- The IMPL itself — see §1.4 + §10. No code, no migration files, no UI files, no tests in this candidate.
- A separate “queue analytics” data warehouse (Snowflake / BigQuery feed) — out of scope for an operator dashboard. PostHog covers longitudinal trends if/when needed.
- Per-user telemetry attribution (which user enqueued the slow jobs?)
—
processing_queue.created_byexists but the dashboard does NOT surface it as a primary axis. Users showing up as drilldown context is acceptable; per-user aggregation is out of scope. - A “click to retry just this dead-lettered job” admin UX — surfacing
the dead-letter row count is in scope; first-class re-enqueue from
the dashboard UI is deferred (operators currently re-enqueue via
manual
INSERTagainstprocessing_queueor by re-running the producer endpoint withforce=truestyle options; an admin re-enqueue surface is its own follow-up spec). - A queue-depth-by-priority breakdown —
processing_queue.priorityis declared in the schema but not yet differentiated by any producer. Surfacing the column when no producer varies it would be misleading. - Cross-job-type orchestration views (e.g. “show me all jobs for workspace X across job_types”) — out of scope; the dashboard is job-type-axis primary, not workspace-axis primary.
- Real-time push (WebSocket / Server-Sent Events) — D-5 ratifies polling cadence; push is out of scope.
1.4 Why a forward-look spec now (and not an IMPL)
Section titled “1.4 Why a forward-look spec now (and not an IMPL)”Three forces converge to make this the right time for spec-only:
- §5.4.1 + §5.4.2 IMPL produce real traffic. Until either §5.4.x
IMPL ships, the dashboard would have only
template_fill/template_analysejob-type traffic to surface, which means the per-job-type drill-down (D-2 minimum metric set) cannot validate. Spec-now-IMPL-later lets the IMPL session dispatch the moment the §5.4.x IMPLs are live. - Liam ratification on D-1 (surface choice) is load-bearing. D-1 forks the IMPL into one of three different worktrees-worth of work: a dedicated page (~1.5 sessions of UI + endpoint + RPC + tests), an admin-tile-only (~0.5 sessions of one tile + one endpoint), or Sentry/PostHog-only (~0.1 sessions of dashboard wiring on existing infra). Without ratification the IMPL session would burn time second-guessing scope.
- D-4 alert thresholds are operator-experience-driven. Liam (as product owner + sole on-call operator) must ratify the WARN vs PAGE thresholds against his expected-recovery-time tolerance. A spec-author-chosen threshold is a guess; a Liam-ratified threshold is a contract.
This spec exists to surface those three forks (D-1 + D-2 + D-4) and their alternatives so ratification is a single Liam-pass rather than mid-IMPL re-design.
2. Background + Current State
Section titled “2. Background + Current State”2.1 Current state — no operational dashboard exists
Section titled “2.1 Current state — no operational dashboard exists”Today, queue-state visibility for an operator is split across four disjoint surfaces:
- Raw
processing_queueSELECT queries via Supabase Studio ormcp__supabase__execute_sql. Operator opens a query interface, typesSELECT job_type, status, count(*) FROM processing_queue GROUP BY 1, 2;— manual, ad-hoc, error-prone, no alerting layer. - Sentry dashboard inbox. Sentry receives
queue.handler,queue.invocation,queue.visibility_timeoutcapture events perlib/queue/telemetry.ts:132-157. Useful for “this just broke” but does NOT surface queue-depth, age-of-oldest-pending, or retry-rate trends — only the discrete failure events. - PostHog event stream (when wired).
lib/queue/telemetry.tsnotes (L32-40) that no PostHog client exists yet — theemitQueueAnalyticshelper is a Sentry-breadcrumb stub. Once wired (post-W1, tracked separately) PostHog will receive the four terminal-state events. Even then PostHog dashboards surface event-frequency trends, not live “right now” state. /api/admin/pipeline-runs/recentadmin endpoint. This exists (app/api/admin/pipeline-runs/recent/route.ts:48-132) and powerscomponents/intelligence/pipeline-runs-panel.tsx— but it surfacespipeline_runs(Pattern 2 caller-allocated rows for the §5.4.x candidates), NOTprocessing_queue. A dead-lettered queue row whosepipeline_run_idwas never updated pastrunningwill show as “running” inpipeline-runs-panel, NOT as “dead_lettered”. This is correct for the existing pipeline_runs panel but means the panel does NOT cover queue-state visibility.
2.2 Pain points (operator perspective)
Section titled “2.2 Pain points (operator perspective)”- No “is the queue draining?” glance. An operator landing on
/admincannot see in <5 seconds whether the queue is clearing pending work or backing up. Current path: open Supabase Studio, run aSELECT count(*) FROM processing_queue WHERE status IN ('pending', 'processing') GROUP BY job_type;query, compare against intuition. - No age-of-oldest-pending alarm. A
bid_draft_alljob stuck inpendingfor an hour withattempts < max_attemptsand no visibility-timeout firing (because it never claimed) is invisible. ThereapStuckJobsreaper handlesprocessingrows past timeout but does NOT alert onpendingrows that never claim. Per spec §5.3 D-8 the visibility-timeout default is 5 minutes — but for apending-with-no-claim row, no alarm fires until an operator notices. - No mean-runtime baseline. When a
bid_draft_alljob runs in 90s on a 10-question bid (normal) vs 45min on a 50-question bid (also normal), the operator has no baseline to call out the outlier. PostHogduration_msevent property exists perlib/queue/telemetry.ts:235but only as breadcrumbs in Sentry until PostHog client wires. - Dead-lettered jobs accumulate silently. Per
processing_queueschema §8,dead_letteredis a terminal status. No alerting fires on count growth — the row count grows until someone notices. Sentry receives the per-jobReaped stuck queue jobwarning but not a “you have 17 dead-letteredbid_draft_allrows now” rollup. - Retry-rate signal is invisible. A
batch_reclassifyjob that retries 2 of 3 attempts, then completes, looks identical inprocessing_queue(status=‘completed’, attempts=3) to one that completed first try (status=‘completed’, attempts=1). Theattemptscolumn is observable via raw SELECT but no aggregation surfaces “what % of completedbatch_reclassifyjobs needed at least one retry today” — the canary signal for upstream Anthropic/Supabase health.
2.3 Why a dashboard now (post-§5.4 W4 + W5 ship)
Section titled “2.3 Why a dashboard now (post-§5.4 W4 + W5 ship)”Two §5.4.x candidates landing in production (§5.4.1 + §5.4.2) means:
- More job-types in flight — beyond
template_fill/template_analysethe queue starts carryingbid_draft_all,batch_reclassify. A per-job-type axis becomes informative. - More throughput — admin-triggered batch operations (workspace reclassify, draft-all-50-questions) generate larger queue depths than today’s template flows.
- More retries possible — the §5.4.1 / §5.4.2 handlers call Anthropic; rate-limit / 503 surface frequently enough that retry- rate becomes a real signal.
Per docs/specs/background-queue-infra-spec.md §9 R3-R4, “table
bloat” + “operator visibility” are forward-look risks the infra spec
explicitly defers to a follow-up. This is that follow-up.
3. Surface — what the dashboard IS
Section titled “3. Surface — what the dashboard IS”3.1 Information architecture
Section titled “3.1 Information architecture”The dashboard surfaces ONE primary axis (per-job-type) and SIX metric columns (M1..M6 per the D-2 minimum metric set authored default — §4.1 below). It is a TABLE, not a chart-heavy view.
Authored-default IA (per D-1 ratification):
/admin/queue (NEW page — admin-only) ├── Header: "Queue overview — last refreshed: <timestamp>" + manual refresh button ├── Summary tile row (4 tiles): │ - Pending count (sum across all job_types) │ - Dead-letter count (sum across all job_types) │ - Mean runtime (last 24h, weighted across job_types) │ - Oldest pending age (max across all job_types) ├── Per-job-type table (one row per job_type): │ | job_type | pending | processing | dead_lettered | oldest_pending_age | mean_runtime_24h | retry_rate_24h | │ | template_fill | 12 | 1 | 0 | 00:00:32 | 4.2s | 8% | │ | bid_draft_all | 3 | 0 | 1 | 00:01:58 | 18.4s | 21% | │ | ... | ... | ... | ... | ... | ... | ... | └── Click on a job_type row → /admin/queue/[jobType] drill-down (per §6.2 UX flow)3.2 Page layout proposal
Section titled “3.2 Page layout proposal”- Mobile-first not a priority — this is an admin-only operator surface; ops will be on desktop in 95%+ of cases. But the page MUST be at-least-readable on mobile (Pixel 5 viewport ≥ 393px width) so an on-call operator can glance at their phone.
- Warm Meridian semantic tokens — per
docs/design/warm-meridian-implementation-spec.md. No raw Tailwind colours. Status indicators use semantic foreground tokens (text-success,text-warning,text-destructive— exact names per spec, sourced fromapp/globals.css). - Severity-coloured rows — per WCAG 2.1 AA “never colour
alone for meaning”. A row with dead_letter > 0 gets a
text- destructivecount + a small “alert” icon (e.g. lucideAlertCircle) — colour + icon + text-suffix (“(N dead)”). A row with mean_runtime_24h > 1.5× weekly baseline gets atext-warningcolour + warning icon + ”(↑ vs baseline)” suffix. - No charts in v1 — minimum metric set is point-in-time +
24h mean. Trend lines are out of scope per D-2 authored
default. (D-2 alternative
comprehensivewould add P50/P95/ P99 + hourly enqueue-rate sparklines; defer to that ratification.)
3.3 Mobile / responsive considerations
Section titled “3.3 Mobile / responsive considerations”- Summary tile row collapses to 2-column grid on viewports <640px.
- Per-job-type table becomes a stacked card list (one card per job_type, fields stacked vertically) on viewports <768px.
- Header refresh-timestamp wraps below the title on mobile.
- All interactive elements ≥44px tap target per WCAG 2.5.5.
4. Metric set — what it surfaces
Section titled “4. Metric set — what it surfaces”4.1 List of metrics with definition + data source
Section titled “4.1 List of metrics with definition + data source”Per D-2 authored default (minimum), the dashboard surfaces SIX metrics per job_type (M1..M6) plus FOUR top-of-page summary tiles (S1..S4). Each metric has a precise definition + a data-source pointer.
| # | Metric | Definition | Data source | Notes |
|---|---|---|---|---|
| M1 | Pending count | count(*) FROM processing_queue WHERE status='pending' GROUP BY job_type | processing_queue — direct or via queue_metrics_summary() RPC per D-3 | Lower bound 0; no upper cap. Includes pending rows backed off via updated_at > NOW() (per background-queue-infra-spec.md §5.2 backoff policy + §10 D-7 — claim_next_job gates on updated_at <= NOW()). |
| M2 | Processing count | count(*) FROM processing_queue WHERE status='processing' GROUP BY job_type | processing_queue — same source | Should be ≤ 1 per job_type at any moment given current single-worker claim_next_job semantics. Higher = visibility-timeout reaper not firing OR multiple workers accidentally. |
| M3 | Dead-letter count | count(*) FROM processing_queue WHERE status='dead_lettered' GROUP BY job_type | processing_queue — same source | Persistent count (not 24h-windowed). Operators care about totals — a stale dead_lettered row from 3 weeks ago is still actionable. Archive policy per §5.4.1 D-5 (30 days) bounds growth. |
| M4 | Age of oldest pending | EXTRACT(EPOCH FROM (NOW() - MIN(created_at))) FROM processing_queue WHERE status='pending' GROUP BY job_type | processing_queue — same source | Returns seconds. Render as HH:MM:SS. Returns NULL when no pending row for that job_type. |
| M5 | Mean runtime (last 24h) | AVG(EXTRACT(EPOCH FROM (completed_at - started_at))) FROM processing_queue WHERE status='completed' AND completed_at > NOW() - INTERVAL '24 hours' GROUP BY job_type | processing_queue — same source | Returns seconds. NULL when no completed jobs in the 24h window. Weights all completions equally — does NOT exclude outliers (a single 30-min bid_draft_all skews the mean for low-traffic types). Operators see raw mean; outlier-trimmed alternative is D-2 comprehensive. |
| M6 | Retry rate (last 24h) | (SUM(attempts > 1) / count(*)) over status IN ('completed', 'completed_with_errors', 'failed', 'dead_lettered') rows in last 24h, GROUP BY job_type | processing_queue — same source | Returns float 0..1; rendered as %. NULL when no terminal-state jobs in the 24h window. A high retry rate is a leading indicator of upstream Anthropic/Supabase health. |
| S1 | Total pending (summary tile) | Sum of M1 across all job_types | Aggregate of M1 — same RPC call returns | Top-of-page glance. |
| S2 | Total dead-lettered (summary tile) | Sum of M3 across all job_types | Aggregate of M3 — same RPC call returns | Top-of-page glance. |
| S3 | Weighted mean runtime (summary tile) | Sum of (M5 × completion-count) / total-completion-count across job_types | Aggregate of M5 — same RPC call returns | NULL when no completions in the 24h window across any job_type. |
| S4 | Max oldest pending age (summary tile) | Max of M4 across all job_types | Aggregate of M4 — same RPC call returns | NULL when no pending rows across any job_type. |
4.2 Alert thresholds — when does Sentry/PostHog warn vs page operations
Section titled “4.2 Alert thresholds — when does Sentry/PostHog warn vs page operations”Per D-4 authored default, two tiers:
WARN (Sentry warning level, no paging — Sentry inbox visibility only):
- Pending > 100 for any job_type for >5 min — queue depth building. Likely cause: cron worker is up but Anthropic 429s are forcing transient retries → pending count grows during burst enqueue.
- Dead-letter > 0 for any job_type — fired ONCE per dead_letter transition (NOT every dashboard render). Drives investigation; even a single dead-lettered row is unusual if upstream is healthy.
- Mean-runtime per job_type increased >50% vs 24h-ago baseline
— leading indicator of upstream slowdown (Anthropic latency,
Supabase blip). Compute: `current_24h_mean / previous_24h_mean
1.5`.
- Retry-rate per job_type > 25% for >10 min — upstream flakiness. Below the PAGE threshold but above reasonable-steady-state.
PAGE (Sentry error level + PagerDuty if integrated — operator notification with paging tier):
- Pending > 1000 for any job_type for >15 min — queue depth unmanageable. Indicates either worker is wedged OR enqueue rate exceeds drain rate by a factor that won’t self-correct.
- Age-of-oldest-pending > 30 min for any job_type — a single
row stuck pending for 30+ minutes means either visibility-
timeout reaper isn’t firing, or the cron worker is wedged, or
a
pendingrow created 30 min ago has never been claimed (worker outage). - Retry-rate per job_type > 50% for >10 min — upstream failing fast enough that majority of jobs are retrying. Anthropic outage / Supabase outage.
- Dead-letter > 10 in any 1h window for any job_type — systematic dead-lettering, NOT one-off. Indicates the failure classifier is incorrectly marking persistent permanent failures as transient (and they exhaust retries).
PagerDuty integration is OUT OF SCOPE for this spec — D-4 ratifies the THRESHOLDS; PagerDuty wiring is a separate ops-track follow-up. WARN events go to Sentry inbox today; PAGE events go to Sentry inbox
- trigger PagerDuty IF wired (gracefully degrade to Sentry-only when not wired).
4.3 Per-job-type alert tuning (out of scope for this spec)
Section titled “4.3 Per-job-type alert tuning (out of scope for this spec)”Different job_types have different reasonable steady-state values.
A bid_draft_all job legitimately running for 18 minutes (50
questions × 20s/question) is normal; an embed job running 18
minutes is a wedge. D-4 ratifies UNIFORM thresholds across job_types
for v1; per-job-type tuning is a future operational tweak (recorded
as D-4 alternative “per-job_type tuning”).
5. Data sources
Section titled “5. Data sources”5.1 processing_queue aggregations — direct queries OR new RPCs
Section titled “5.1 processing_queue aggregations — direct queries OR new RPCs”Per D-3 authored default, the dashboard reads via NEW PL/pgSQL RPCs:
queue_metrics_summary()— returns JSONB with all M1..M6 + S1..S4 metrics in a single round-trip. Per-job-type breakdown + cross-job-type aggregate. Powers the main dashboard render.queue_jobs_recent(p_job_type text, p_limit int)— returns the N most-recent rows for a single job_type with their full status + attempts + error_message. Powers the drill-down (§6.2 UX flow).
Both RPCs run as SECURITY INVOKER (per kh-prod-readiness 45
batch convention; admin-only RLS already gates processing_queue
SELECT — the RPC inherits the caller’s RLS context). Both RPCs MUST
include SET search_path = public, extensions per the CLAUDE.md
“Function search_path” gotcha. Both RPCs MUST be paired with
explicit REVOKE EXECUTE ON FUNCTION public.queue_metrics_summary() FROM anon; per feedback_supabase_pg_default_acl_anon_execute.
D-3 authored-default RATIONALE:
- Direct SELECT queries via supabase-js would work (PR-3 gates SELECT to admin via RLS) but force the dashboard route to compute aggregations in JS — slower, less optimisable, and means the query plan is opaque to ops without enabling supabase query logs.
- Sentry/PostHog only — no DB load, but freshness lag (PostHog events are 1-5min behind), and during a Sentry/PostHog OUTAGE the dashboard goes blind. The dashboard’s purpose is to be the in-product fallback when external observability is degraded.
- RPCs — centralise the metric computation, can be unit-tested
via SQL fixtures, can have purpose-specific indexes added (e.g.
processing_queuealready hasidx_processing_queue_job_typeperdocs/reference/SCHEMA-QUICK-REFERENCE.md§8 cross-link; composite(status, job_type, created_at)index addition is trivially additive in IMPL).
5.2 Sentry — queue_* tags from lib/queue/telemetry.ts
Section titled “5.2 Sentry — queue_* tags from lib/queue/telemetry.ts”The dashboard does NOT itself read Sentry — Sentry’s JS-SDK
hosted-API is not designed for in-product telemetry-fetching.
Instead, the dashboard surfaces Sentry as the SINK for its own
WARN/PAGE alerts (per §4.2 above) — it WRITES to Sentry via
emitQueueSentry() from lib/queue/telemetry.ts when crossing
threshold transitions. Operators correlate against the Sentry
dashboard manually for trend / inbox-search.
Verbatim contract from lib/queue/telemetry.ts:75-83 — three Sentry
signal stages the dashboard MUST NOT add new stages to (else PR-4
violation):
invocation— worker-harness crashhandler— job-type handler throwvisibility_timeout— reaper rescue
The dashboard’s WARN/PAGE alert emissions reuse stage: 'handler'
with a new tags.dashboard_alert: 'true' discriminator + the
threshold tier (tags.alert_tier: 'warn' | 'page') — per D-4 IMPL
detail. NO new stages.
5.3 PostHog — queue_job_* events from lib/queue/telemetry.ts
Section titled “5.3 PostHog — queue_job_* events from lib/queue/telemetry.ts”The dashboard does NOT itself read PostHog — PostHog dashboards are
the longitudinal-trend surface; the operational dashboard is
point-in-time + 24h windowed. PostHog event names from
lib/queue/telemetry.ts:169-172 — verbatim contract per PR-4:
queue_job_completed(success)queue_job_failed(transient or permanent failure)queue_job_dead_lettered(exhausted retries)queue_job_cancelled(user cancellation)
These events are EMITTED by the worker (via emitQueueAnalytics())
on every terminal-state transition. PostHog dashboards COULD be
queried later to surface 7-day / 30-day trend lines but D-2
authored-default (minimum metric set) does NOT include trend lines
— deferred to D-2 alternative comprehensive.
5.4 Cross-source reconciliation — when sources disagree which wins
Section titled “5.4 Cross-source reconciliation — when sources disagree which wins”Three classes of disagreement are possible:
processing_queuesaysdead_lettered, PostHog saysqueue_job_dead_letteredcount is 1 lower. Cause: the worker emitted the event AFTER the row update (the spec §6.2 contract — event emission is best-effort, post-write). PostHog can lag indefinitely. Resolution:processing_queuewins. PostHog is for trends, not for live state.processing_queuesays row hasattempts=2, Sentry shows 3queue.handlercapture events for the same job_id. Cause: Sentry de-dupes by stack-trace fingerprint; multiple distinct throws from the same job re-render as one Sentry issue with event-count 3. Or: the worker captured a transient aserrorand a permanent on retry exhaustion as anothererror. Resolution:processing_queue.attemptswins for the count. Sentry is for error-investigation, not retry-counting.- PostHog
duration_msfor a job_id differs fromcompleted_at - started_atby >100ms. Cause: telemetry measurement is wall-clock-from-worker-claim, whileprocessing_queue.started_atis set byclaim_next_jobRPC andcompleted_atis set by the worker writing back. Inter-segment gaps (RPC return latency, JS event-loop) cause small differences. Resolution:processing_queueinterval wins for the dashboard. PostHogduration_msis for user-centric percentile aggregation.
The dashboard’s UI MUST NOT show “PostHog says X, DB says Y” —
operators see ONE value per cell, sourced from processing_queue.
6. UX flows
Section titled “6. UX flows”6.1 Flow — admin opens the dashboard
Section titled “6.1 Flow — admin opens the dashboard”- Admin navigates to
/admin/queue(link from main/adminlanding page — the IMPL must add the link). - Page renders the four summary tiles + per-job-type table from a
single GET
/api/admin/queue/metricscall (which proxies thequeue_metrics_summary()RPC per §5.1). - “Last refreshed:
” + manual refresh button at top. - Auto-refresh (per D-5 authored default — 30s interval).
- Total render time target: <1.5s p95 on a populated queue (per
§7 AC-1). RPC plan must hit
idx_processing_queue_job_typefor theGROUP BY job_typeclause.
6.2 Flow — admin drills into a single job_type
Section titled “6.2 Flow — admin drills into a single job_type”- Admin clicks a row in the per-job-type table (e.g. clicks the
bid_draft_allrow). - Routes to
/admin/queue/[jobType](e.g./admin/queue/bid_draft_all). - Drill-down page calls GET
/api/admin/queue/[jobType]/recent?limit=50(which proxies thequeue_jobs_recent()RPC per §5.1). - Renders a table of the 50 most recent jobs of that type with
columns:
id,status,attempts,created_at,started_at,completed_at,error_message(truncated), andpipeline_run_id(link topipeline-runs-panelfiltered by that pipeline_run_id). - NO re-enqueue / cancel actions on the drill-down (per §1.3 out-of-scope — surfacing only, no actions). Future spec adds re-enqueue UX.
6.3 Flow — admin sees an alert + investigates
Section titled “6.3 Flow — admin sees an alert + investigates”- Sentry inbox surfaces a
queue.handlercapture withtags.dashboard_alert: 'true'andtags.alert_tier: 'page'forpending > 1000 bid_draft_all for 15 min. - Admin opens
/admin/queuefrom the link in the Sentry alert (or directly). - Summary tile S1 (Total pending) shows
1247intext-destructive. - Per-job-type table row for
bid_draft_allshowspending: 1184intext-destructive. - Admin clicks the row → drill-down → sees that the recent 50 are
all
status='pending', oldestcreated_atis ~17 min ago, none havestarted_atset. - Admin investigates worker health (Vercel logs, cron-job-monitor) — the dashboard’s role ends at “show the queue is wedged”; root- cause is via Vercel + Sentry in conjunction.
6.4 Flow — admin re-enqueues a dead-lettered job
Section titled “6.4 Flow — admin re-enqueues a dead-lettered job”OUT OF SCOPE for v1 IMPL per §1.3. Documented here as the deferred-flow shape so the IMPL session knows what NOT to build:
- Admin navigates to
/admin/queue/bid_draft_all(drill-down). - Sees a row with
status='dead_lettered',attempts=3,error_message='Anthropic timeout'. - (future) Admin clicks “Re-enqueue” button → POST
/api/admin/queue/[jobId]/re-enqueue→ server inserts a freshprocessing_queuerow with the SAMEpayloadand a freshidempotency_key(or NULL key — tradeoff documented in the future spec). - Admin sees the new row appear at the top of the drill-down table on next refresh.
This UX shape is not built in v1. Operators currently dead-letter
via raw INSERT against processing_queue or by re-running the
producer endpoint (e.g. POST /api/bids/.../draft-all with
fresh request body — different requestHash → different
idempotency_key → fresh enqueue per §5.4.1 §3.2).
7. Acceptance Criteria
Section titled “7. Acceptance Criteria”The IMPL session MUST gate against ALL of the following.
AC-1 — Admin lands on /admin/queue and sees the metric set within 1.5s p95
Section titled “AC-1 — Admin lands on /admin/queue and sees the metric set within 1.5s p95”- Admin authenticates as a user with
adminrole. - Navigates to
/admin/queue. - Page renders the four summary tiles + per-job-type table for ALL
currently-known job_types (per
JobTypeunion inlib/queue/envelope.ts— 10 values post-S225 W1-IMPL ('batch_reclassify'widening shipped on commit1c9e88fe)). - P95 render time (TTFB → first contentful paint) < 1.5s when the
processing_queuetable has ≤ 100,000 rows. - Empty job_type rows (no rows of that type in queue) are NOT
omitted — they render with all SIX metrics surfaced (
pending: 0,processing: 0,dead_lettered: 0,oldest_pending_age: --,mean_runtime_24h: --,retry_rate_24h: --) so the operator sees the full universe of job_types.
AC-2 — Admin drills into one job_type and sees the 50 most recent jobs
Section titled “AC-2 — Admin drills into one job_type and sees the 50 most recent jobs”- Admin clicks a job_type row in the per-job-type table.
- Routes to
/admin/queue/[jobType]. - Page renders a table of the 50 most-recent jobs of that type
ordered by
created_at DESC, includingid,status,attempts,created_at,started_at,completed_at,error_message(truncated to 200 chars), andpipeline_run_id(clickable link). - P95 render time < 1.0s for the drill-down (smaller dataset, no GROUP BY).
- Clicking the
pipeline_run_idlink routes to a filtered view ofpipeline-runs-panelfor that single row.
AC-3 — Sentry alerts fire at the ratified D-4 thresholds
Section titled “AC-3 — Sentry alerts fire at the ratified D-4 thresholds”- With a populated test queue meeting each D-4 threshold (mocked
timestamps + status values), the IMPL emits the corresponding
Sentry capture via
emitQueueSentry({ stage: 'handler', tags: { dashboard_alert: 'true', alert_tier: 'warn' | 'page' } }). - The capture includes
extra.threshold_breacheddescribing which D-4 rule matched (e.g.'pending_gt_1000_for_15min'). - WARN alerts fire at level
warning; PAGE alerts fire at levelerror. - Each threshold-rule fires AT MOST ONCE per 5-minute window for
the same (job_type, alert_tier, threshold_rule) tuple — to
prevent Sentry inbox flood. Implementation detail: a small
processing_queue_alertstable OR a Sentry-side rate-limit rule. Choice deferred to IMPL. - Test cases exercise each threshold via fixture-driven
processing_queuestate.
AC-4 — Empty-state handling
Section titled “AC-4 — Empty-state handling”- With ZERO rows in
processing_queue,/admin/queuerenders without error. - The summary tiles show
0,0,--,--(pending, dead- lettered, weighted-mean-runtime, max-oldest-pending-age). - The per-job-type table renders with one row per
JobTypeunion value, all metrics showing0or--. - A small “Queue empty — no jobs in the last 24h” caption renders
below the table when
S1 + S2 + total-completions-in-24h === 0. - NO Sentry alert fires for “empty queue”.
AC-5 — Auto-refresh fires at the D-5-ratified cadence without state-flicker
Section titled “AC-5 — Auto-refresh fires at the D-5-ratified cadence without state-flicker”- With D-5 ratified (authored default 30s), the dashboard polls
GET
/api/admin/queue/metricsevery 30s via TanStack QueryrefetchInterval(perfeedback_searchbar_query_provider). NOT rawsetInterval. - Stale data continues to display during refetch — NO loading spinner replaces the populated table.
- On successful refetch, the “Last refreshed:
” header updates atomically with the table data. - On failed refetch (network blip), the header shows “Last
refreshed:
— retrying” but the stale data continues to display. - Manual refresh button cancels any in-flight refetch and starts a fresh one.
AC-6 — Admin-only access enforced (PR-3 gate)
Section titled “AC-6 — Admin-only access enforced (PR-3 gate)”- A
vieweruser navigating to/admin/queueis redirected to/loginper the existing admin-route pattern (see e.g.app/admin/content-dedup/page.tsx’s admin-gate behaviour). - An
editoruser navigating to/admin/queueis redirected to/(or sees a 403) — editors are NOT permitted on the operator dashboard per PR-3. - The metrics endpoint GET
/api/admin/queue/metricsreturns 403 foreditorcallers and 401 for unauthenticated callers, perauthFailureResponse(auth)semantics inapp/api/admin/pipeline-runs/recent/route.ts:50-52.
8. Risks + Watch-outs
Section titled “8. Risks + Watch-outs”| Risk | Description | Likelihood | Impact | Mitigation |
|---|---|---|---|---|
| R1 | Dashboard query performance on a large processing_queue. Post-archive cleanup the table should stay <1M rows per infra §9 R3 (30-day retention per §5.4.1 D-5). Pre-archive — or if the archive job lapses — the table can grow unbounded. | Medium | Medium | Indexes already exist on (job_type) per idx_processing_queue_job_type. RPCs queue_metrics_summary() GROUP BY job_type use this index. For per-status filtering, add a composite (status, job_type) index in the IMPL migration. Re-test query EXPLAIN at 5M-row staging fixture before merging IMPL. |
| R2 | Metric staleness during high-throughput periods. With cron tick at * * * * * (post §5.4 W5) + dashboard polling at 30s, the dashboard may show 30s-stale state mid-burst. Operators may misread “still 800 pending” when in fact it has drained to 200. | Low | Low | D-5 ratifies refresh cadence; manual refresh button always available. The auto-refresh interval is a knob — tune per operator feedback. |
| R3 | Surface-choice paralysis during ratification. D-1 has 4 alternatives (dedicated page / tile-only / Sentry-only / hybrid). Without ratification the IMPL session cannot dispatch. | High | High | This spec exists to surface D-1 explicitly. Liam ratification gates IMPL dispatch. The spec author has a strong-but-overridable preference (dedicated page) per §9 D-1 authored default; alternatives are honestly enumerated. |
| R4 | ”Build the thing, forget to turn it on”. Per feedback_build_not_wired. The dashboard could ship as a page + endpoint + RPC but never get linked from /admin landing — operators wouldn’t find it. | Medium | High | IMPL acceptance MUST include a hard step “add /admin/queue link to /admin landing page (or wherever the admin-tile-row lives)”. bun run knip post-IMPL to catch orphaned files. AC-1 trace runs via Playwright from the /admin landing → click link → lands on /admin/queue → renders. |
| R5 | Alert flood on first deploy. Threshold rules in D-4 are static; on first deploy, an existing 200-row pending backlog (from before the dashboard existed) instantly fires the WARN threshold for “pending > 100 for >5 min”. Sentry inbox gets a one-off flood. | Medium | Low | IMPL adds a 5-minute rate-limit per (job_type, alert_tier, threshold_rule) tuple per AC-3 step 4. First-deploy noise self-bounds within one window. Post-deploy ops review: clear the backlog OR raise the WARN threshold to 200 if the 100-threshold proves too sensitive in practice. |
| R6 | Cross-track migration drift on RPC creation. Per feedback_cross_track_migration_blocks_cli_sync. The queue_metrics_summary() + queue_jobs_recent() RPCs are NEW PL/pgSQL functions; if the production-readiness track concurrently ships an 45 batch widening RPC ACLs, the migration order matters. | Low | Medium | IMPL session does the cross-track sync check pre-dispatch (per feedback_cross_track_migration_blocks_cli_sync recipe). The RPC migrations include explicit REVOKE EXECUTE ... FROM anon per feedback_supabase_pg_default_acl_anon_execute, AND SECURITY INVOKER per 45 conventions. |
| R7 | Comprehensive metric set scope creep. D-2 alternative comprehensive adds P50/P95/P99 + sparklines + cost-per-job. Each is independently valuable; together they 3x the IMPL scope. Without explicit Liam ratification at “minimum”, the IMPL session may scope-creep. | Medium | Medium | D-2 authored default is minimum with explicit “comprehensive” alternative. Ratification at minimum unblocks IMPL; comprehensive becomes its own follow-up spec. The dashboard’s first cut is “enough to call ops to action”, not “enough to do retrospectives”. |
9. Decisions Required (Liam ratification)
Section titled “9. Decisions Required (Liam ratification)”Each D-x has an authored default + alternatives + Liam-decision-pending
marker. Ratification at spec-review close-out unless flipped per
feedback_spec_ratification_senior_pass_flip (1-2 flips per
ratification expected). Anchor IDs use the d-N slug.
D-1 — Surface choice — admin tile vs dedicated /admin/queue page vs Sentry-only vs PostHog-only?
Section titled “D-1 — Surface choice — admin tile vs dedicated /admin/queue page vs Sentry-only vs PostHog-only?”Why: The dashboard could live in any of FOUR shapes:
- A tile on the existing
/adminlanding page (lightweight; no drill-down; surface is one rectangle that summarises the full queue). - A dedicated
/admin/queuepage with per-job-type drill-down to/admin/queue/[jobType](full IA per §3.1 above). - Sentry-only — no in-product dashboard; the WARN/PAGE alerts go to Sentry inbox + admin uses Sentry’s own dashboard for visualisation.
- PostHog-only — same as Sentry-only but PostHog as the surface.
- Hybrid: tile on landing for glance + dedicated page for detail.
Authored default: Dedicated /admin/queue page (option 2).
Reasons: matches existing admin info-architecture (cf.
app/admin/content-dedup/); gives drill-down per-job-type without
cramming a tile; Sentry/PostHog continue to host alerts but the page
is the operator-facing summary; a dedicated page is also the
substrate for future “re-enqueue dead-lettered” actions per §6.4
deferred-flow.
Alternatives:
- Admin-tile-only on
/adminlanding. Lighter; no drill-down; admin sees aggregate counts on landing. Cost: drilling into per-job-type detail requires rawprocessing_queueSELECT — back to today’s pain point. REJECTED in authored default for insufficient drill-down. - Sentry-only. No new surface; relies on Sentry’s own dashboard. Cost: during a Sentry outage the dashboard goes dark; no in-product fallback. Sentry’s queue-state filtering is via tag search, not aggregation — operators must mentally aggregate. REJECTED in authored default — the dashboard’s purpose is operator-in-product visibility.
- PostHog-only. Similar to Sentry-only but PostHog is event- driven (not state-driven) — PostHog cannot show “right now 47 pending” because it only has events of “job claimed” + “job completed”. REJECTED — fundamentally wrong surface for live queue depth.
- Hybrid: tile-on-landing + page-for-detail. Adds the tile in addition to the page. Cost: maintenance burden of two surfaces rendering similar data. Acceptable if Liam wants the landing glance; defers to future iteration.
Liam decision: PENDING (authored default = dedicated
/admin/queue page). Liam to ratify at S225 close-out (or
subsequent session).
D-2 — Metric set — minimum vs comprehensive?
Section titled “D-2 — Metric set — minimum vs comprehensive?”Why: §4.1 surfaces SIX per-job-type metrics (M1-M6) + FOUR summary tiles (S1-S4). A “comprehensive” alternative would add P50/P95/P99 runtimes, throughput-per-hour graph, hourly enqueue rate, cost-per-job_type estimates. Comprehensive is more visually impressive but 3x the IMPL scope (per R7).
Authored default: Minimum (M1..M6 per §4.1 only). Reasons: the dashboard’s first cut is “enough to call ops to action”, not “enough to do retrospectives”. Trends are PostHog’s job once PostHog is wired. Per-job cost is a future operational concern (cf. D-6).
Alternatives:
- Comprehensive: add P50/P95/P99 runtimes per job_type. Useful
for tail-latency tracking. Cost: requires
percentile_cont()computation which is more expensive on a largeprocessing_queue— must run async-after-render OR via materialised view. Adds ~1 session of IMPL effort. - Comprehensive: add throughput-per-hour graph (sparkline). Adds Recharts (or similar) dependency to the dashboard route. Adds ~0.5 session of IMPL effort.
- Comprehensive: add hourly enqueue-rate sparkline. Same as above; same incremental cost.
- Comprehensive: add cost-per-job_type estimates. Per
bid_draft_all: 50 questions × 3 model passes × token estimates. Requires per-pass cost data the worker doesn’t currently emit. Adds ~1 session of IMPL effort + a worker-side cost-emission migration. REJECTED in authored default — too much new infrastructure. - Trace-style: per-job timeline view (visual) for debugging individual failed jobs. Powerful for debugging; massive scope expansion. REJECTED in authored default — first-cut dashboard is aggregate-axis, not single-job-axis.
Liam decision: PENDING (authored default = minimum).
D-3 — Data source — direct processing_queue aggregations OR new RPCs OR Sentry/PostHog?
Section titled “D-3 — Data source — direct processing_queue aggregations OR new RPCs OR Sentry/PostHog?”Why: The metrics could read DIRECTLY via supabase-js (filtered SELECT + JS-side aggregation), or via NEW PL/pgSQL RPCs (centralised SQL aggregation), or could derive from Sentry/PostHog event streams.
Authored default: processing_queue aggregations via NEW
RPCs (option B) — queue_metrics_summary() returning JSONB with
all metrics in one round-trip; queue_jobs_recent() for drill-down.
Reasons: Sentry/PostHog have data freshness lag (PostHog 1-5min,
Sentry sampling); raw aggregation queries via supabase-js move
GROUP BY computation into JS which is slower + opaque to ops; RPCs
centralise the metric computation, can be unit-tested via SQL
fixtures, and can have purpose-specific indexes added.
Alternatives:
- Direct supabase-js queries (no RPC; route reads SELECT directly
- GROUP BY in JS). Simpler, no new SQL surface, but no caching at the RPC layer; query plan is opaque to ops; aggregation crosses the JS↔Postgres boundary which costs network round-trips when N job_types proliferate.
- Sentry/PostHog only. No DB load; freshness lag; dashboard goes blind during external observability outages. REJECTED — the dashboard’s purpose is in-product fallback during external outage.
- Hybrid: RPCs for live state, Sentry for trends. Acceptable evolution — D-2 minimum metric set has no trend lines, so trend-source is moot for v1; if D-2 flipped to comprehensive, trend-source would re-open.
Liam decision: PENDING (authored default = processing_queue
aggregations via NEW RPCs).
D-4 — Alert thresholds — what triggers Sentry/PostHog warn vs page operations?
Section titled “D-4 — Alert thresholds — what triggers Sentry/PostHog warn vs page operations?”Why: Without thresholds, the dashboard is passive — operators must remember to check it. Without operator-tuned thresholds, a too- noisy alert ruleset trains operators to ignore the page-tier notifications. Without too-quiet thresholds, real outages go unnoticed.
Authored default: Two-tier — WARN (Sentry warning, no paging) and PAGE (Sentry error + PagerDuty if integrated). Verbatim thresholds per §4.2:
WARN tier:
- pending > 100 for any job_type for >5 min
- dead-letter > 0 for any job_type (one-shot per transition)
- mean-runtime per job_type increased >50% vs 24h-ago baseline
- retry-rate per job_type > 25% for >10 min
PAGE tier:
- pending > 1000 for any job_type for >15 min
- age-of-oldest-pending > 30 min for any job_type
- retry-rate per job_type > 50% for >10 min
- dead-letter > 10 in any 1h window for any job_type
Alternatives:
- PagerDuty-free (Sentry-warn-only — operator polls dashboard). Simpler; no PagerDuty dependency. Cost: ops loses paging-tier notification — for a 3am outage the queue stays wedged till business hours. REJECTED in authored default IF PagerDuty is available; ACCEPTED in authored default for now since PagerDuty is OUT OF SCOPE for this spec — alerts go to Sentry, paging is follow-up.
- Aggressive (paging at lower thresholds). WARN at 50, PAGE at 500. Higher signal sensitivity; higher false-positive rate. REJECTED in authored default — operator (Liam) is solo; false-positives consume his evening time.
- Per-job-type tuning.
bid_draft_alllegitimately runs for 18 min;embedrunning 18 min is a wedge. Static thresholds across job_types miss this. REJECTED in v1 authored default for IMPL simplicity; first-cut uniform thresholds — tune per ops feedback in v2.
Liam decision: PENDING (authored default = the two-tier threshold table above).
D-5 — Refresh cadence — manual vs auto-refresh?
Section titled “D-5 — Refresh cadence — manual vs auto-refresh?”Why: The dashboard is point-in-time data; cadence drives operator experience.
Authored default: Auto-refresh every 30s via TanStack Query
refetchInterval. Manual refresh button always available.
Alternatives:
- Manual-only. Operator clicks “refresh” to see new state. Cost: operator-glance during outage stays stale; less effective during active investigation.
- Faster auto-refresh (10s or 5s). More responsive, but more
query load on
processing_queue. With cron tick at 1m post W5, 10s polling is 6x more aggregate-query traffic than tick-rate. Likely overkill. - Slower auto-refresh (60s or 5min). Less query load but lag during burst.
- Visibility-aware refresh (refetch on tab focus + when
document.visibilityState === 'visible'). Saves polling when the operator’s tab is backgrounded. NICE-TO-HAVE; defers to IMPL discretion.
Liam decision: PENDING (authored default = 30s auto-refresh + manual refresh button).
D-6 — Cost surface — show $ per-job-type estimates or hide?
Section titled “D-6 — Cost surface — show $ per-job-type estimates or hide?”Why: Each bid_draft_all job costs Anthropic API spend (3
passes × ~50 questions × token cost). A per-job-type cost estimate
informs ops of “is the queue churning expensively?”.
Authored default: Hide cost estimates in v1. Reasons: the
worker doesn’t currently emit per-pass cost data; surfacing
estimated cost from token-count heuristics is fragile and may
mislead ops; the §5.4.1 §6.3 pipeline_runs.cost field DOES
capture per-job cost-on-completion for bid_draft_all but not
historic baseline for comparison.
Alternatives:
- Surface estimated cost-per-job-type as a column in the per-job-
type table. Computed as
AVG(pipeline_runs.cost)over recent completed jobs of that type. Cost: requires the join topipeline_runsand assumes Pattern 2 finalisation has populatedcost. May be NULL for older rows. ACCEPTABLE in a future iteration oncepipeline_runs.costcoverage is high. - Surface MTD (month-to-date) total cost per job-type. Larger aggregation surface; useful for finance not ops. REJECTED — the dashboard is for ops, not finance.
Liam decision: PENDING (authored default = hide cost
estimates in v1; revisit when pipeline_runs.cost coverage is
high across all job_types).
10. References
Section titled “10. References”10.1 Source files (read during spec authoring)
Section titled “10.1 Source files (read during spec authoring)”docs/specs/§5.4.1-batch-draft-all-spec.mdv1 RATIFIED (1269 lines). Section-structure model + verbatim PR-1..PR-6 + verbatim Pattern 2 contract from §6.3.docs/specs/§5.4.2-batch-reclassify-spec.mdv1 DRAFT (1762 lines). Section-structure model — symmetry with §5.4.x family.docs/specs/background-queue-infra-spec.mdv1 — primary parent spec. Contracts referenced:- PR-1 + §3.3 lifecycle states (drives PR-1 of this spec).
- §5.5 idempotency formula contract (drives PR-1 implication
that the dashboard treats
processing_queue.idempotency_keyas a queryable column, NOT as a source-of-truth re-derivation). - §6.1 Sentry signal stages (drives PR-4 + §5.2 of this spec).
- §6.2 PostHog event names (drives PR-4 + §5.3 of this spec).
- §9 R3-R4 forward-look (drives §1.4 + §2.3 of this spec).
lib/queue/dispatch.ts:31-199—runJobByTypeswitch + Pattern 2 finalisation (drives §5.4 cross-source reconciliation rule 2).lib/queue/failure.ts:113-173—handleJobFailure(drives §4.1 M6 retry-rate metric + §4.2 dead-letter alert tier).lib/queue/visibility-timeout.ts:59-72—reapStuckJobsRPC (drives §2.2 “no age-of-oldest-pending alarm” pain point + §4.1 M4 metric).lib/queue/envelope.ts:34-43—JobTypeunion (10 values post-S225 W1-IMPL —'batch_reclassify'widening shipped on commit1c9e88fe) + lifecycle states (drives §4.1 M1..M6 GROUP BY job_type axis).lib/queue/telemetry.ts:75-83, 132-157, 169-172, 223-251—emitQueueSentry+emitQueueAnalytics+ Sentry stage constants + PostHog event names (drives PR-4 + §5.2 + §5.3 verbatim contract).lib/queue/auth.ts—reValidateAuthContext(cross-reference for PR-3 admin-only).app/api/admin/pipeline-runs/recent/route.ts:48-132— existing admin endpoint pattern (drives §6.1 endpoint shape + AC-6 auth semantics).lib/pipeline/record-run.ts:127-211—recordPipelineRun()signature + Pattern 2 INSERT-only (drives PR-6 cross-link to §5.4.1 §6.3 Pattern 2 finalisation note).docs/reference/SCHEMA-QUICK-REFERENCE.md§8 (lines 280-311) —processing_queueschema verbatim. Drives §4.1 metric definitions (every column is a metric source) + AC-6 RLS gate reference.app/admin/content-dedup/page.tsx(existing admin page) — layout reference for the v1 IMPL session. Not load-bearing on this spec.
10.2 Specification cross-references
Section titled “10.2 Specification cross-references”docs/specs/background-queue-infra-spec.mdv1 — primary parent spec (R3-R4 forward-look this spec executes on).docs/specs/§5.4.1-batch-draft-all-spec.mdv1 RATIFIED — the out-of-scope “queue-depth dashboard forbid_draft_allis deferred to operational dashboard spec post-§5.4 W5” call-out in §1.3 originated this spec.docs/specs/§5.4.2-batch-reclassify-spec.mdv1 DRAFT — same out-of-scope call-out.docs/specs/silent-failure-prevention-spec.md—sb()/tryQuery()conventions used by the metric-source RPC route (and any future IMPL).docs/reference/product-roadmap.md§5.4 W5+ — operational tweaks track.docs/design/warm-meridian-implementation-spec.md— semantic tokens for the dashboard’s status colours per §3.2.
10.3 Memory-feedback references applied during authoring
Section titled “10.3 Memory-feedback references applied during authoring”feedback_brief_quote_spec_verbatim— drove §5.2 + §5.3 verbatim PostHog event names + Sentry stage constants fromlib/queue/telemetry.ts. Per the memory: “Copy interface shapes/regex literals from spec; never paraphrase.”feedback_self_verify_own_specs— single-pass spec carries errors; verifier wave to follow this spec authoring before Liam ratification. Brief specifically called this out.feedback_supabase_pg_default_acl_anon_execute— drove §5.1 RPC migration requirement to include explicitREVOKE EXECUTE ON FUNCTION public.queue_metrics_summary() FROM anon;.feedback_cross_track_migration_blocks_cli_sync— drove R6 cross-track migration drift mitigation.feedback_searchbar_query_provider— drove AC-5 TanStack QueryrefetchInterval(NOT rawsetInterval) for auto-refresh.feedback_build_not_wired— drove R4 mitigation + AC-1 Playwright trace from the/adminlanding to the dashboard page.feedback_audit_all_pipeline_entry_points— drove §5.4 cross-source reconciliation + the survey of FOUR existing surfaces in §2.1.feedback_no_silent_promise_catch(viasilent-failure-prevention-spec.md) — drove the implicit contract that any IMPL of the metrics endpoint usessb()/tryQuery()not raw.then().catch()swallow.
Document Lifecycle
Section titled “Document Lifecycle”- v1 DRAFT (05/05/2026, S225 W3-A SPEC). Initial spec authoring
per §5.4.1 + §5.4.2 §1.3 forward-look + parent infra-spec §9
R3-R4 forward-look. PR-1..PR-6 lift contract from the parent
spec + §5.4.x ratifications. IMPL is deferred — this
candidate is a forward-looking surface spec ratifying D-1..D-6
so a future IMPL session can dispatch with no further design
questions. Verifier wave V_W3 to run before ratification per
feedback_self_verify_own_specs. - v1 RATIFIED (PENDING). Liam ratification on D-1..D-6 at
S225 close-out (or subsequent session). Once ratified, the
IMPL session can dispatch with the ratified surface +
metric set + data source + alert thresholds + refresh cadence
- cost-surface decision in hand.