Normalize descriptor strings into insert-only reference tables

## What does this PR do? Adopts **D-0031**: normalizes the low-cardinality descriptor strings repeated on the growing fact tables into four **insert-only reference tables** (`sources`, `classifiers`, `policies`, `action_providers`). Fact tables (`raw_observations`, `events`, `audit_records`, `classifications`, `decisions`, `actions`) now carry int foreign keys; the domain model and ports keep speaking strings, so this is purely a persistence-layer change. Key pieces: - **ReferenceResolver**: translates both directions with process-lifetime caches (insert-only rows can never go stale). Concurrent first-inserts race benignly on the unique index (loser re-reads the winner). A null `sources.source_type` is filled exactly once by the first typed writer; rows are never renamed or deleted, so audit history is never rewritten retroactively. - **RawObservation gains `SourceType`**: observation provenance now records the source kind (previously not captured); all three adapters set it from their existing type constants. - **3NF cleanups**: `events.source_type` and `decisions.policy_version` move into their reference tables; both depended on the descriptor, not the row. - **AddReferenceTables migration**: backfills the reference tables from existing distinct values, then swaps each text column for the int FK via explicit add/backfill/drop/rename SQL (PostgreSQL has no implicit text-to-integer cast, so the scaffolded AlterColumn operations were replaced). Schema-agnostic per the search_path design; reversible Down included. Existing VM data migrates in place, no reset needed. - **Deliberately not normalized** (recorded in D-0031): queue tables (transient rows), `corrected_by` (user identity; future users table under admin auth), `category`/`recommended_action`/`operation_id` (natural follow-ons once the D-0029 config store defines those vocabularies; tracked in TODO.md). ## How was this tested? - [x] `dotnet build` - 0 errors, 0 warnings - [x] `dotnet test` - 267 passed, 9 skipped (live-PostgreSQL integration tests; local WSL Docker unavailable) - [x] New integration tests: store round-trip through reference resolution, cross-process dedup (fresh resolver finds the existing row), and fill-once source-type upgrade - [ ] Live verification on the Debian VM post-merge: AutoMigrate applies the backfill migration to the existing data ## Checklist - [x] I have read the [Contributing Guide](../CONTRIBUTING.md) - [x] Changes are focused - one logical change per PR - [x] Documentation updated (README.md / DECISIONS.md / TODO.md / AGENT-README.md, if applicable) - [x] New dependencies are MIT/Apache-2.0/BSD, security-vetted, and recorded in THIRD-PARTY-NOTICES.md (none added) - [x] No secrets, credentials, or real infrastructure identifiers in the diff - [x] No commented-out code or debug leftovers

hannah-vernon commented

2026-08-25 18:16:53 -05:00

Owner

What does this PR do?

Adopts D-0031: normalizes the low-cardinality descriptor strings repeated on the growing fact tables into four insert-only reference tables (sources, classifiers, policies, action_providers). Fact tables (raw_observations, events, audit_records, classifications, decisions, actions) now carry int foreign keys; the domain model and ports keep speaking strings, so this is purely a persistence-layer change.

Key pieces:

ReferenceResolver: translates both directions with process-lifetime caches (insert-only rows can never go stale). Concurrent first-inserts race benignly on the unique index (loser re-reads the winner). A null sources.source_type is filled exactly once by the first typed writer; rows are never renamed or deleted, so audit history is never rewritten retroactively.
RawObservation gains SourceType: observation provenance now records the source kind (previously not captured); all three adapters set it from their existing type constants.
3NF cleanups: events.source_type and decisions.policy_version move into their reference tables; both depended on the descriptor, not the row.
AddReferenceTables migration: backfills the reference tables from existing distinct values, then swaps each text column for the int FK via explicit add/backfill/drop/rename SQL (PostgreSQL has no implicit text-to-integer cast, so the scaffolded AlterColumn operations were replaced). Schema-agnostic per the search_path design; reversible Down included. Existing VM data migrates in place, no reset needed.
Deliberately not normalized (recorded in D-0031): queue tables (transient rows), corrected_by (user identity; future users table under admin auth), category/recommended_action/operation_id (natural follow-ons once the D-0029 config store defines those vocabularies; tracked in TODO.md).

How was this tested?

dotnet build - 0 errors, 0 warnings
dotnet test - 267 passed, 9 skipped (live-PostgreSQL integration tests; local WSL Docker unavailable)
New integration tests: store round-trip through reference resolution, cross-process dedup (fresh resolver finds the existing row), and fill-once source-type upgrade
Live verification on the Debian VM post-merge: AutoMigrate applies the backfill migration to the existing data

Checklist

I have read the Contributing Guide
Changes are focused - one logical change per PR
Documentation updated (README.md / DECISIONS.md / TODO.md / AGENT-README.md, if applicable)
New dependencies are MIT/Apache-2.0/BSD, security-vetted, and recorded in THIRD-PARTY-NOTICES.md (none added)
No secrets, credentials, or real infrastructure identifiers in the diff
No commented-out code or debug leftovers

hannah-vernon added 1 commit

Normalize descriptor strings into insert-only reference tables 09e7d2efe6

Adopts D-0031: the low-cardinality descriptor strings repeated on the
growing fact tables are normalized into four insert-only reference
tables (sources, classifiers, policies, action_providers).  Fact rows
carry int foreign keys; the domain model and ports keep speaking
strings.

PostgreSQL heap tables have no page/dictionary compression, so a
repeated ~30-byte descriptor costs real bytes on every row and index
entry.  Reference tables give compact fact rows, referential
integrity, an enumerable catalog for the admin GUI, and an attachment
point for D-0029 per-classifier runtime config.

- ReferenceResolver: process-lifetime two-way caches (insert-only rows
  never go stale); concurrent first-inserts race benignly on the
  unique index; a null sources.source_type is filled exactly once by
  the first typed writer (audit path does not know the type)
- RawObservation gains SourceType: observation provenance now records
  the source kind; all three adapters set it
- events.source_type and decisions.policy_version move into their
  reference tables (3NF: both depended on the descriptor, not the row)
- AddReferenceTables migration backfills reference tables from
  existing distinct values and swaps columns in place via explicit
  add/backfill/drop/rename SQL (PostgreSQL has no implicit
  text-to-integer cast); schema-agnostic, reversible Down
- Deliberately not normalized: queue tables (transient),
  corrected_by (user identity; future users table), category /
  recommended_action / operation_id (await D-0029 vocabularies)
- Integration tests: round-trip with reference resolution, cross-
  process dedup, fill-once type upgrade
- D-0031 recorded; AGENT-README and TODO.md updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannah-vernon merged commit c98214584c into dev

2026-08-25 18:40:23 -05:00

hannah-vernon referenced this pull request from a commit

Merge pull request 'Normalize descriptor strings into insert-only reference tables' (#15) from feat/reference-tables into dev

Rows
Columns

Normalize descriptor strings into insert-only reference tables #15

What does this PR do?

How was this tested?

Checklist