Spike S12 — Graphify Confidence-Label Taxonomy Adopt (Q4.14)

Spike S12 — Graphify Confidence-Label Taxonomy Adopt (Q4.14)

Branch: content-items-investigation Session: S229 Status: RESOLVED-ADOPT (Q4.14 ratified S228 by Liam) — this spike confirms integration path with Q4.7 KG-provenance enum. Output spec: 0.9-spike-plan.md §S12 (lines 564-584). Decision gate G12: see §6.

---

1. Context

1.1 What Q4.14 is

Q4.14 in 0.9-decision-graph.md (line 119) is RESOLVED-ADOPT, ratified verbatim by Liam:

“regardless I agree that we should be adopting Graphify’s confidence label taxonomy. This has been an architectural oversight on our part so far, and as you noted, brings several interesting benefits and is very low risk for minimal effort. We also won’t need to backfill, due to the upcoming re-ingestion.”

(Source: docs/plans/phase-0-investigation/graphify-evaluation-feedback.md line 26, §5.2.C response.)

1.2 What Q4.7 is

Q4.7 in 0.9-decision-graph.md (line 112) is RESOLVED:

“KG provenance enum (EXTRACTED / INFERRED / AMBIGUOUS) on entity_relationships? — Yes per wave-08 (mempalace Open Q response 6).”

Confidence: 90%. Depends on: Q4.1.

1.3 The relationship between Q4.7 and Q4.14

Q4.7 originated from the mempalace wave-08 feedback (KG-provenance enum, with the three values EXTRACTED / INFERRED / AMBIGUOUS). Q4.14 originated from the graphify §5.2.C feedback (confidence-label taxonomy).

Both feedback streams independently converged on the same three-value enum and the same three label names. This is the key finding of this spike: there is no semantic collision because Q4.7’s enum values and graphify’s confidence labels are literally the same set: EXTRACTED, INFERRED, AMBIGUOUS.

Therefore Q4.14 does not introduce a parallel taxonomy — it ratifies the values Q4.7 already named, plus adds graphify’s per-label semantics + rubrics.

---

2. Graphify’s confidence-label taxonomy — verbatim

Source: ~/.claude/skills/graphify/SKILL.md (v6 per .graphify_version), lines 293-340.

2.1 Label semantics

Label	Definition (graphify SKILL.md L293-295)
EXTRACTED	”relationship explicit in source (import, call, citation, ‘see §3.2’)“
INFERRED	”reasonable inference (shared data structure, implied dependency)“
AMBIGUOUS	”uncertain — flag for review, do not omit”

2.2 Companion numeric confidence_score (REQUIRED on every edge)

Source: ~/.claude/skills/graphify/SKILL.md L327-339.

Label	Allowed confidence_score values	Notes
EXTRACTED	1.0 always	Single value, not a range.
INFERRED	One of {0.95, 0.85, 0.75, 0.65, 0.55} — discrete rubric	”never 0.5 as a default”. Rubric: 0.95 direct structural evidence; 0.85 strong inference; 0.75 reasonable inference; 0.65 weak inference; 0.55 speculative but plausible.
AMBIGUOUS	0.1 – 0.3 (range)	“if no value above [0.55] fits, mark the edge AMBIGUOUS rather than picking 0.4 or below”

Why the discrete rubric matters: graphify observed bimodal behaviour in production (>50% of values at 0.5, >40% at 0.85+) when models were given a continuous 0.0–1.0 range — models collapse continuous ranges into a near-binary distribution. The discrete five-point rubric for INFERRED + the empty 0.4–0.5 band forcing escalation to AMBIGUOUS is a deliberate prompt-engineering pattern.

2.3 Real-world distribution

From .planning/codebase/graphify-baselines/client-docs-with-llm/GRAPH_REPORT.md, KH’s own client-docs corpus baseline:

“Extraction: 98% EXTRACTED · 2% INFERRED · 0% AMBIGUOUS · INFERRED: 22 edges (avg confidence: 0.82)”

This validates the rubric in our actual data: a corpus of 1,100+ edges had ~22 INFERRED (clustered around 0.85) and zero AMBIGUOUS. Heavy tail on EXTRACTED is the expected shape — most relationships are explicit citations / structural references.

---

3. KH’s current confidence model

3.1 Schema today

Source: docs/reference/SCHEMA-QUICK-REFERENCE.md §19 (entity_relationships, lines 545-561):

Column	Type	Nullable	Default	Notes
id	uuid	NO	gen_random_uuid()	PK
source_entity	text	NO		Canonical name of source entity
relationship_type	text	NO		CHECK constraint (holds, complies_with, delivers_to, uses, demonstrated_by, requires, part_of, supersedes, references, evidences)
target_entity	text	NO		Canonical name of target entity
source_item_id	uuid	YES		FK → content_items.id (SET NULL)
confidence	numeric(3,2)	YES	1.0	Extraction confidence 0–1
created_at	timestamptz	YES	now()

No provenance / confidence_label / equivalent column exists. Q4.7 has not yet been implemented — confirmed by absence in supabase/migrations/.

3.2 Writer behaviour today

Source: lib/ai/classify.ts L1772-1793 (Pass 2 relationship upsert):

const relRows = result.relationships.map((r) => ({
  source_entity: resolveAlias(canonicalise(r.source)).toLowerCase(),
  relationship_type: r.relationship,
  target_entity: resolveAlias(canonicalise(r.target)).toLowerCase(),
  source_item_id: itemId,
  confidence: 1.0,   // ← hardcoded
}));

The Pass 2 classifier hardcodes confidence: 1.0 on every emitted relationship — KH does not currently differentiate confidence at write time. The numeric confidence column has been a placeholder.

This is consistent with §5.2.C feedback (“This has been an architectural oversight on our part so far”) — KH had the column but never used it.

3.3 Entity_mentions parallel

Source: docs/reference/SCHEMA-QUICK-REFERENCE.md §18 (entity_mentions, line 527): confidence numeric(3,2) default 1.0 — same shape, same hardcoded write.

Same gap applies. Q4.14 adopt should be considered for entity_mentions too (out of strict scope for S12 which is entity_relationships, but flagging for the schema decision below).

---

4. Mapping table — graphify labels ↔ Q4.7 enum

Graphify label	Q4.7 enum value	Identity?	Gap?	Redundancy?
EXTRACTED	EXTRACTED	Yes — same name, same definition	None	None
INFERRED	INFERRED	Yes — same name, same definition	None	None
AMBIGUOUS	AMBIGUOUS	Yes — same name, same definition	None	None

Gaps (graphify labels with no Q4.7 equivalent): zero. Redundancies (Q4.7 values graphify wouldn’t emit): zero.

The two enums are identical at the level of labels. The only difference is that graphify pairs each label with a companion confidence_score rubric, which Q4.7 did not specify because Q4.7’s source (mempalace wave-08) was concerned only with provenance labelling, not numeric scoring.

4.1 Where the two streams differ — the rubric

Graphify adds three pieces of guidance Q4.7 did not specify:

1. Numeric confidence_score is mandatory on every edge (not nullable, no default placeholder).
2. Discrete bucket rubric for INFERRED (five values: 0.55 / 0.65 / 0.75 / 0.85 / 0.95).
3. Empty 0.4–0.5 band forcing escalation to AMBIGUOUS.

These don’t introduce a new column type — graphify’s confidence_score field maps cleanly to KH’s existing entity_relationships.confidence numeric(3,2). They’re prompt-engineering rules the Pass 2 classifier should follow when emitting, not schema additions.

4.2 Semantic-similar edge — additional graphify pattern

Graphify also defines a relation semantically_similar_to (SKILL.md L312-316), always emitted as INFERRED with confidence_score 0.6-0.95. KH’s relationship_type CHECK constraint does not include this value (current list: holds, complies_with, delivers_to, uses, demonstrated_by, requires, part_of, supersedes, references, evidences).

Out-of-strict-scope for Q4.14 (which is the confidence-label taxonomy, not the relationship-type taxonomy), but flagging for the post-Phase-0.9 backlog: if KH adopts graphify analysis primitives later (per Option A in graphify-evaluation.md §5.2.A — backlogged as OPS-G-1), semantically_similar_to will need a CHECK widening migration.

---

5. Schema decision — extend vs replace

The spike-plan §S12 decision gate offers two shapes:

Extend: keep Q4.7 enum + add graphify_confidence text column — pluralism Replace: Q4.7 enum becomes graphify-aligned — uniformity

5.1 The decision gate as framed is moot

Because graphify’s three labels ARE Q4.7’s three values (§4), there is no “graphify-aligned” version of Q4.7 that differs from Q4.7. There is no graphify_confidence column to add separately from the Q4.7 column — they would carry identical data.

Adding a graphify_confidence text column alongside a Q4.7 provenance enum column would be pure duplication. Replacing Q4.7 with a graphify-aligned enum would mean…ratifying Q4.7 as-is.

5.2 The real decision

The real choice is: what shape does the single confidence-label column take?

Shape	Description	Pros	Cons
A. Postgres ENUM type	CREATE TYPE confidence_label AS ENUM ('EXTRACTED', 'INFERRED', 'AMBIGUOUS'); then ALTER TABLE entity_relationships ADD COLUMN provenance confidence_label NOT NULL DEFAULT 'EXTRACTED';	Type-safe at DB level; index-friendly; clean Supabase type generation.	Adding new values later requires ALTER TYPE … ADD VALUE (DDL, not data migration). Locked at three values.
B. CHECK-constrained text column	ALTER TABLE entity_relationships ADD COLUMN provenance text NOT NULL DEFAULT 'EXTRACTED' CHECK (provenance IN ('EXTRACTED','INFERRED','AMBIGUOUS'));	Same correctness as ENUM; constraint editable without type alter; familiar pattern (matches relationship_type CHECK already in this table).	Slightly looser type-gen (string union not enum); equivalent index cost.
C. JSONB on metadata	Stash provenance + confidence_score inside an existing or new metadata jsonb column.	Schema-free; can carry future fields.	Violates the CLAUDE.md principle “Prefer proper schema — tables and columns over JSONB for key data”. Hidden from type generation; index requires GIN expression.

Recommendation: Shape B (CHECK-constrained text column).

Rationale:

• Matches the existing relationship_type shape in the same table (precedent from 20260421171520_entity_relationships_unique_tuple_constraint.sql series — CHECK is the house pattern).
• CLAUDE.md §Supabase & Schema explicitly says “Prefer proper schema — tables and columns over JSONB for key data.”
• Future expansion (e.g. a new label like MANUAL for human-corrected edges) needs only ALTER TABLE … DROP CONSTRAINT … ADD CONSTRAINT …, no ENUM type alter.
• Type generation produces a clean string union via bun run supabase gen types.

5.3 Companion numeric column

The existing entity_relationships.confidence numeric(3,2) column stays and is repurposed from “placeholder hardcoded 1.0” to “graphify’s confidence_score value following the discrete rubric in §2.2.”

This matches the recommendation in docs/plans/phase-0-investigation/graphify-evaluation.md §7 Open Question 4:

“If we add provenance (EXTRACTED/INFERRED/AMBIGUOUS) to entity_relationships, do we also keep classification_confidence numeric, or migrate to categorical entirely? Recommend: keep both — numeric for ranking, categorical for filtering.”

Confirmed by this spike: keep numeric confidence + add categorical provenance. Different jobs.

5.4 Pass 2 classifier write-path change

lib/ai/classify.ts Pass 2 prompt + write path needs updating so the classifier emits a provenance label AND a graphify-rubric-compliant confidence_score:

// before
const relRows = result.relationships.map((r) => ({
  ...,
  confidence: 1.0,   // hardcoded
}));

// after
const relRows = result.relationships.map((r) => ({
  ...,
  provenance: r.provenance ?? 'EXTRACTED',         // new column
  confidence: r.confidence_score ?? 1.0,            // graphify-rubric value
}));

The Pass 2 prompt template (lib/ai/classify-prompt.ts or wherever the Anthropic prompt for relationship extraction lives) needs the EXTRACTED/INFERRED/AMBIGUOUS rubric inlined verbatim from graphify’s SKILL.md §confidence_score block. This work lives in a follow-on implementation task, not in this spike.

5.5 Backfill — none required

Per Liam (graphify-evaluation-feedback.md L26): “We also won’t need to backfill, due to the upcoming re-ingestion.”

The pre-launch re-ingest plan (CX.22 RESOLVED, decision-graph §4) wipes and re-populates entity_mentions + entity_relationships. The Q4.14 migration adds the column with a safe default ('EXTRACTED') — pre-re-ingest rows get the default, get overwritten on re-ingest with classifier-emitted values. No bespoke backfill code path needed.

5.6 Migration shape (illustrative — not in spike scope)

-- migration: 20260512XXXXXX_add_provenance_to_entity_relationships.sql
ALTER TABLE entity_relationships
  ADD COLUMN provenance text NOT NULL DEFAULT 'EXTRACTED'
  CHECK (provenance IN ('EXTRACTED', 'INFERRED', 'AMBIGUOUS'));

-- Optional companion index for filter-by-provenance queries
CREATE INDEX idx_entity_relationships_provenance
  ON entity_relationships (provenance, source_item_id);

COMMENT ON COLUMN entity_relationships.provenance IS
  'Graphify confidence-label taxonomy: EXTRACTED (explicit in source), INFERRED (reasonable inference), AMBIGUOUS (uncertain — flag for review). Paired with the existing numeric confidence column carrying graphify''s discrete confidence_score rubric (1.0 / {0.55-0.95} / 0.1-0.3 respectively).';

Same migration would optionally apply to entity_mentions (which also has placeholder confidence numeric default 1.0). Mention deferred to a follow-on decision because S12’s spike-plan scope names entity_relationships only.

---

6. G12 decision-gate verdict

G12 status: EXTEND-EXISTING-PROPER (revised framing).

The spike-plan §S12 binary (extend Q4.7 + add graphify_confidence column OR replace Q4.7 with graphify-aligned enum) is moot in its original framing because Q4.7 and Q4.14 name the same three values. The genuinely-open question was the column shape (ENUM type vs CHECK text vs JSONB).

Adopted:

• Add one new column provenance text NOT NULL DEFAULT 'EXTRACTED' CHECK (...) to entity_relationships (Shape B in §5.2).
• Keep the existing numeric confidence column; repurpose its semantics from “placeholder 1.0” to “graphify’s discrete confidence_score rubric value.”
• Update Pass 2 classifier prompt + write path to emit both fields, following the graphify rubric.
• No backfill (per CX.22 re-ingest, ratified Liam S228).

Rejected:

• A separate graphify_confidence column distinct from a provenance column — would be pure duplication.
• ENUM type — CHECK text matches relationship_type precedent in the same table.
• JSONB metadata — violates CLAUDE.md “Prefer proper schema” principle.

Mapping outcome: 1:1 identity between graphify labels and Q4.7 enum values. Zero gaps, zero redundancies.

Effort estimate: ~1 day (migration + Pass 2 prompt update + write-path update + Pass 2 unit-test update). Implementation goes onto the post-Phase-0.9 backlog under “CX.32” or “Q4.14-IMPL” tag (TBD when architecture-implementation phase opens).

---

7. Follow-on flags (out of S12 scope, for backlog)

1. entity_mentions parallel adopt. The same column shape applies to entity_mentions.confidence (also a placeholder hardcoded 1.0). Decision deferred but flagged.
2. relationship_type CHECK widening for semantically_similar_to. If Option A (graphify analysis primitives) lands post-launch (OPS-G-1), the CHECK constraint on entity_relationships.relationship_type will need semantically_similar_to added.
3. Pass 2 prompt rubric inline. The discrete confidence_score rubric needs to be inlined into the Pass 2 prompt template verbatim (graphify SKILL.md L327-339). Prompt-engineering work, separate from migration.
4. Pass 2 test fixtures. __tests__/lib/ai/classify.test.ts will need fixture updates to cover the new provenance + non-1.0 confidence write path. Vitest, scoped to lib/ai/.
5. Cross-track read-side awareness. lib/mcp/tools/entities.ts (governance / entities tool surface) currently does not filter or expose provenance. Post-migration, MCP tool outputs should optionally surface provenance so AI consumers can apply the §5.2.C filtering pattern (“AI-generated content can use only EXTRACTED relationships, never INFERRED unless cited” — graphify-evaluation.md L176).

---

8. Confidence

Spike confidence: 92%.

High confidence because:

• Mapping is 1:1 identical labels — no semantic interpretation required.
• Graphify rubric is fully documented in SKILL.md and validated by real-world distribution in the client-docs-with-llm baseline (98/2/0 split, INFERRED avg 0.82).
• Schema shape (CHECK text) has direct precedent in the same table (relationship_type).
• No backfill risk because re-ingest wipes the table.

Drag from:

• The “extend vs replace” decision-gate framing in spike-plan §S12 turned out to be moot, which suggests a minor planning-doc inconsistency. Flagging — when spike-plan is next edited, replace the §S12 decision-gate text with the §6 verdict above.
• Implementation in Pass 2 prompt is not in S12 scope; the rubric needs careful prompt-template work and a fresh Anthropic eval to confirm the model emits the discrete bucket values cleanly (not the 0.5 / 0.85 bimodal failure mode graphify saw). That’s an implementation-phase eval, not a spike output.

---

End of spike output. No code or schema changes made in this session — documentation only.