hannah-vernon/viegard-sentinel
hannah-vernon/viegard-sentinel
Watch
1
0
Fork
You've already forked viegard-sentinel
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Release: phases 1-5 complete (skeleton, sources, persistence, detection, policy, classification) #10

Merged
hannah-vernon merged 37 commits from dev into main 2026-08-21 13:23:28 -05:00
Conversation 0 Commits 37 Files changed 181 +15573 -121
hannah-vernon commented 2026-08-21 13:22:30 -05:00
Owner
Copy link

What does this PR do?

Release PR bringing main up to date with everything merged to dev since repository bootstrap: PRs #1 through #9, decisions D-0001 through D-0028.

Delivered, in merge order:

  • #1-#2 Phase 3 skeleton: solution structure (Viegard.slnx, NuGetAudit enforced), domain model, application ports, broker-semantics channel queue, secret providers, prompt-injection-resistant PromptAssembler, fail-closed ClassificationOutputValidator, queue telemetry with traffic-light evaluation, Dockerfiles + compose example.
  • #3 IMAP source (D-0019..D-0022): per-account workers, app-password auth with OAuth2 seam, IDLE with polling fallback, read-only folders, attachment metadata only.
  • #4 Syslog UDP source (D-0023): fail-closed source allowlist, rate/size caps, nginx access-log parsing, SWAG setup guide.
  • #5 PostgreSQL persistence (D-0024): EF Core + Npgsql, jsonb polymorphic payloads, durable SKIP LOCKED + LISTEN/NOTIFY queues with crash-safe dead-lettering, nightly pg_dump rigging; defects fixed via live integration testing (18/18 against postgres:17).
  • #6 Phase 5 detection + correlation: deterministic HTTP/mail rules (linear scans on untrusted input, no regex), TimeWindowCorrelator, CorrelationWorker.
  • #7 MDaemon source (D-0013, D-0025): both real log formats parsed (session transcripts + DynScrn), satellite-ready, sanitized fixtures.
  • #8 Judgment policy engine (D-0026, D-0027): full guardrail-chain evaluation, fail-closed protected-address list, provisional thresholds, shipped posture dry-run ON / providers OFF / approval ON.
  • #9 Deterministic classification spine (D-0028): detect -> correlate -> classify -> decide end to end with zero LLM dependency.

Docs kept current throughout: ARCHITECTURE.md (approved D-0017), DECISIONS.md, TODO.md, AGENT-README.md (agent-readme.md draft v0.1 spec-conformant).

How was this tested?

  • dotnet build Viegard.slnx - 0 errors, 0 warnings (re-verified on dev tip ea9192b)
  • dotnet test Viegard.slnx - 240/240 unit tests; 18/18 PostgreSQL integration tests passed live 2026-08-20
  • End-to-end verifications along the way: syslog UDP smoke test, pipeline host boots per role, flagship dry-run spine test

Checklist

  • I have read the Contributing Guide
  • Changes are focused - release sync of reviewed, already-merged work
  • Documentation updated (continuously, per-PR)
  • New dependencies security-vetted and recorded in THIRD-PARTY-NOTICES.md (MailKit/MimeKit, BouncyCastle, Npgsql/EF Core, MS.Extensions, test tooling)
  • No secrets, credentials, or real infrastructure identifiers
  • No commented-out code or debug leftovers
## What does this PR do? Release PR bringing `main` up to date with everything merged to `dev` since repository bootstrap: PRs #1 through #9, decisions D-0001 through D-0028. Delivered, in merge order: - **#1-#2 Phase 3 skeleton:** solution structure (`Viegard.slnx`, NuGetAudit enforced), domain model, application ports, broker-semantics channel queue, secret providers, prompt-injection-resistant `PromptAssembler`, fail-closed `ClassificationOutputValidator`, queue telemetry with traffic-light evaluation, Dockerfiles + compose example. - **#3 IMAP source** (D-0019..D-0022): per-account workers, app-password auth with OAuth2 seam, IDLE with polling fallback, read-only folders, attachment metadata only. - **#4 Syslog UDP source** (D-0023): fail-closed source allowlist, rate/size caps, nginx access-log parsing, SWAG setup guide. - **#5 PostgreSQL persistence** (D-0024): EF Core + Npgsql, jsonb polymorphic payloads, durable `SKIP LOCKED` + `LISTEN/NOTIFY` queues with crash-safe dead-lettering, nightly pg_dump rigging; defects fixed via live integration testing (18/18 against postgres:17). - **#6 Phase 5 detection + correlation:** deterministic HTTP/mail rules (linear scans on untrusted input, no regex), `TimeWindowCorrelator`, `CorrelationWorker`. - **#7 MDaemon source** (D-0013, D-0025): both real log formats parsed (session transcripts + DynScrn), satellite-ready, sanitized fixtures. - **#8 Judgment policy engine** (D-0026, D-0027): full guardrail-chain evaluation, fail-closed protected-address list, provisional thresholds, shipped posture dry-run ON / providers OFF / approval ON. - **#9 Deterministic classification spine** (D-0028): detect -> correlate -> classify -> decide end to end with zero LLM dependency. Docs kept current throughout: ARCHITECTURE.md (approved D-0017), DECISIONS.md, TODO.md, AGENT-README.md (agent-readme.md draft v0.1 spec-conformant). ## How was this tested? - [x] `dotnet build Viegard.slnx` - 0 errors, 0 warnings (re-verified on dev tip `ea9192b`) - [x] `dotnet test Viegard.slnx` - 240/240 unit tests; 18/18 PostgreSQL integration tests passed live 2026-08-20 - [x] End-to-end verifications along the way: syslog UDP smoke test, pipeline host boots per role, flagship dry-run spine test ## Checklist - [x] I have read the [Contributing Guide](../CONTRIBUTING.md) - [x] Changes are focused - release sync of reviewed, already-merged work - [x] Documentation updated (continuously, per-PR) - [x] New dependencies security-vetted and recorded in THIRD-PARTY-NOTICES.md (MailKit/MimeKit, BouncyCastle, Npgsql/EF Core, MS.Extensions, test tooling) - [x] No secrets, credentials, or real infrastructure identifiers - [x] No commented-out code or debug leftovers 
- ARCHITECTURE.md: components, solution layout, core interfaces, event
  and decision model, deployment topology, security boundaries, failure
  handling, and proposed dependencies; awaiting Hannah's approval
- DECISIONS.md: D-0010 pipeline host and admin API as separate services
- TODO.md / AGENT-README.md: reflect Phase 2 in progress

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ARCHITECTURE.md: assumption 3 now requires ack/abandon semantics,
  serializable ID-carrying messages, idempotent consumers, and a
  poison-message policy so an external broker can replace the in-process
  channels without a rewrite; add IWorkQueue to the interface table
- TODO.md: track queue/broker technology evaluation (Service Broker,
  Kafka KIP-932 share groups, RabbitMQ, NATS JetStream, Redis Streams,
  Postgres SKIP LOCKED)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ARCHITECTURE.md: role-configurable pipeline host deployable N times;
  singleton roles (correlator, policy/action engine) enforced by config
  validation; multi-instance data sources (one worker per configured
  mail account); durable DB-backed IWorkQueue as first cross-process
  transport
- DECISIONS.md: record D-0011 per Hannah's requirement (many mail
  accounts across many providers, including self-hosted MDaemon)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ARCHITECTURE.md: queue health monitor section (signals, green/amber/
  red derivation, transport-independent telemetry via shared
  persistence with stale-heartbeat-as-red)
- DECISIONS.md: record D-0012 per Hannah's requirement
- TODO.md: track configurable traffic-light thresholds needing approval

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ARCHITECTURE.md: Viegard.Sources.MDaemonLogs adapter + tests project
- DECISIONS.md: D-0013, MDaemon log ingestion in initial scope
- TODO.md: open questions (log selection, formats, transport, paths);
  Phase 4 ordering updated
- AGENT-README.md: planned integrations updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ARCHITECTURE.md: viegard-admin is a mobile-compatible PWA GUI + API;
  Viegard.Notifications.Email and Viegard.Notifications.WebPush projects;
  INotificationProvider initial implementations defined
- DECISIONS.md: D-0014 per Hannah's requirements
- TODO.md: frontend tech, Web Push privacy sign-off, SMTP details, and
  phone-to-GUI network access as open questions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Web Push relays (FCM/Apple/Mozilla) carry E2E-encrypted payloads but
expose delivery metadata; Hannah wants further privacy consideration
before choosing.  Notification port stays pluggable; operator email
notifications and the mobile GUI proceed unaffected.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Static SSR by default, Interactive Server components only where needed;
approve/deny degradable to plain form posts; mandatory staleness UX
hint with automatic refresh per Hannah's requirement.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Directory.Build.props: net10.0, nullable, warnings-as-errors, NuGetAudit
  (mode=all, level=low, NU1901-NU1904 as errors)
- Viegard.Domain: RawObservation, NormalizedEvent, EntityRef, payloads,
  Incident/Evidence, Classification (validated invariants), Decision with
  guardrail evaluations, ActionRecord, AuditRecord, Correction,
  ComponentHealth, AdminCommand
- Viegard.Application: ports (IDataSource, IEventNormalizer, IWorkQueue,
  ICommandQueue, stores, ICorrelator, IClassifier, IInferenceProvider with
  trust-tagged prompt variables, IPolicyEngine, IActionProvider,
  INotificationProvider, IAuditLedger, ISecretProvider, IHealthContributor);
  ChannelWorkQueue with ack/abandon, dead-lettering, and age-preserving
  redelivery; FileSecretProvider (path-traversal-safe) and
  ConfigurationSecretProvider; Secret type that never reveals via ToString
- Viegard.Persistence: development-only in-memory stores and audit ledger
- Viegard.PipelineHost: role-configurable host (D-0011) with startup
  validation that refuses unknown/empty/duplicate roles; secret provider
  selection via configuration; heartbeat worker
- Viegard.AdminApi: Blazor Web App template shell (D-0016) plus /healthz
- Tests: 33 passing (classification invariants, queue semantics including
  FIFO, redelivery counts, dead-lettering, stale-age telemetry, double-settle
  protection; secret masking, path traversal rejection, config secrets)
- Docs: AGENT-README verified commands, THIRD-PARTY-NOTICES entries,
  TODO Phase 3 progress + HashiCorp Vault discussion item, .gitattributes
  *.slnx rule

Verified: dotnet build (0 warnings), dotnet test (33/33), host boots with
valid roles and fails fast on invalid topology.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #1
- PromptAssembler: SYSTEM/APPLICATION/UNTRUSTED sections; untrusted values
  only inside per-assembly random-boundary data blocks; placeholders can
  never resolve to untrusted variables; oversized values truncated with
  marker; malicious variable names sanitized (D-0018)
- ClassificationOutputValidator: strict fail-closed validation of model
  output on System.Text.Json; required fields/ranges enforced, unknown
  properties rejected, oversized output rejected, no partial data on
  failure (D-0018)
- Queue telemetry (D-0012): QueueTelemetrySnapshot domain record,
  IQueueTelemetryStore port + in-memory impl, IQueueStatsSource on
  IWorkQueue, QueueTelemetryPublisher worker, QueueHealthEvaluator with
  configurable green/amber/red thresholds (defaults pending approval)
- deploy/: Dockerfile.pipeline, Dockerfile.admin, sanitized compose
  example, .dockerignore.  NOT verified: no container tooling on this
  workstation; verification deferred to the Debian VM per Hannah
- Tests: 70 passing (37 new: prompt-injection containment, boundary
  forgery resistance, validator strictness, traffic-light derivation)
- Docs: AGENT-README enforcement points + state, TODO Phase 3 status,
  DECISIONS D-0018

Verified: dotnet build (0 warnings), dotnet test (70/70), pipeline host
boots with telemetry publisher running.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #2
- Viegard.Sources.Imap (MailKit 4.17.0, supply-chain reviewed):
  - ImapMailSource per account: implicit TLS (993), app-password auth
    via ISecretProvider, folders opened read-only (retrieval never sets
    Seen), IDLE with polling fallback and bounded IDLE cycle as safety
    poll, UIDVALIDITY-aware offset resume, reconnect with delay,
    IHealthContributor per account
  - ImapEventNormalizer: MailFetchDto JSON -> MailMessageEvent; fails
    closed on malformed payloads; body truncation; sender entity refs
  - LinkExtractor: http/https from text + hrefs, deduped, capped
  - Options validation refuses duplicate accounts, missing fields, and
    the unimplemented OAuth2 seam
- Domain: MailMessageEvent/MailAddressInfo/AttachmentInfo payloads
- Application: ObservedItem (observation + raw payload), SourceType on
  IDataSource, ISourceOffsetStore port
- PipelineHost: IngestionWorker (persist raw -> normalize -> store ->
  enqueue -> audit, per stage); events ChannelWorkQueue registered for
  telemetry; per-account source registration; ingestion gated on the
  sources role
- Tests: 99 passing (29 new: normalizer incl. prompt-injection-as-data
  and truncation, link extraction incl. hostile-mail cap, options
  validation)
- Docs: THIRD-PARTY-NOTICES (MailKit, MimeKit, BouncyCastle, MS.Ext),
  AGENT-README integrations/structure, TODO (first-run baseline needs
  confirmation; live-account verification outstanding)

Verified: dotnet build (0 warnings), dotnet test (99/99), host boots
with ingestion worker idle when no accounts are configured.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Viegard.Sources.Syslog:
  - SyslogUdpSource: thin UDP loop; per-source health incl. drop counters
  - SyslogDatagramHandler (testable core): fail-closed source-IP
    allowlist, datagram size cap, per-source token-bucket rate cap,
    replacement-character decoding so hostile bytes never throw
  - SyslogEnvelopeParser: RFC 3164 (incl. pid tags, year inference with
    Dec/Jan wrap) and RFC 5424; garbage degrades to message-only
  - NginxAccessLogParser: combined format + Viegard host=/rt= extension;
    malformed/hostile request lines yield partial fields, never throw
  - SyslogEventNormalizer: nginx_access tags -> HttpRequestEvent with
    IP/host/URI/UA entity refs; everything else -> generic SyslogEvent
    keyed to peer IP (claimed hostname untrusted)
  - Options validation: enabled listener requires non-empty allowlist
- Domain: HttpRequestEvent + SyslogEvent payloads
- PipelineHost: syslog registration (off by default); debug log on
  successful ingestion
- docs/swag-syslog-setup.md: sanitized SWAG nginx configuration guide
- Tests: 130 passing (31 new incl. spoofed-hostname handling, injection
  text as data, rate-limit recovery with FakeTimeProvider)

Verified end-to-end: live UDP datagrams to the running host produced a
normalized HttpRequestEvent (nginx line) and generic SyslogEvent
(hostile message) queued for correlation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #4
- DECISIONS.md: D-0024 resolves the deferred database choice (D-0004):
  PostgreSQL 17 dedicated container, EF Core + Npgsql, in-database
  durable queues (SKIP LOCKED + LISTEN/NOTIFY), nightly pg_dump sidecar
  riding the weekly Hyper-V export; SQL Server stays a documented
  possible future provider behind the store/queue ports
- deploy/docker-compose.example.yml: viegard-db (postgres:17, secret-file
  password, healthcheck, unpublished port), viegard-db-backup (nightly
  custom-format pg_dump to the VM disk, 14-day retention), DB settings
  wired into pipeline and admin services
- .gitignore: never commit deploy data/backups/secrets directories
- TODO.md: database decided; queue/broker item narrowed; MDaemon
  transport recommendation recorded pending confirmation
- ARCHITECTURE.md: deployables table updated

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Viegard.Persistence.Postgres:
  - ViegardDbContext + InitialSchema migration: raw observations, events
    (jsonb entities/payload), incidents, classifications, decisions,
    actions, audit records, corrections, queue telemetry, source
    offsets, queue messages + counters
  - Postgres store implementations for every persistence port, mapping
    immutable domain records to EF row types
  - PostgresWorkQueue: SKIP LOCKED visibility-timeout leases, delivery
    counting with dead-lettering enforced at lease time (crash-safe
    poison protection), LISTEN/NOTIFY wakeups with fallback poll,
    per-queue counters; PostgresCommandQueue for admin commands
  - DatabaseOptions (+ validator): password via ISecretProvider secret
    name, never configuration; AutoMigrate at host startup
  - Service registration extension + design-time factory for dotnet-ef
- Domain: JSON polymorphism discriminators on EventPayload
- Application: IQueueStatsSource.GetStatsAsync (durable queues compute
  stats in the database); ChannelWorkQueue keeps a sync core
- PipelineHost: persistence provider switch (inmemory default,
  postgres via Viegard:Persistence:Provider) with startup migration
- Tests: 140 passing + 7 PostgreSQL integration tests (queue
  round-trip, redelivery, dead-letter, lease expiry, concurrent
  consumers, LISTEN/NOTIFY wakeup, polymorphic store round-trip) gated
  on VIEGARD_TEST_POSTGRES pending local Docker (WSL reboot)
- Deps (supply-chain reviewed): Npgsql.EntityFrameworkCore.PostgreSQL
  10.0.3, EF Core 10.0.11, recorded in THIRD-PARTY-NOTICES.md

Verified: dotnet build (0 warnings), dotnet test (140/140 + 7 skips),
host boots with default inmemory provider.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #5
All 17 Viegard.Persistence.Postgres tests now pass against a live
postgres:17 container (Docker CE in WSL2 Debian); full suite 147/147.

Defects the integration tests caught:
- EF generated PascalCase quoted columns while the queue's raw SQL uses
  snake_case: added a snake_case naming convention to ViegardDbContext
  and regenerated the InitialSchema migration (never shipped)
- Enqueue SQL omitted dead_lettered (NOT NULL, no default): insert false
- Stats query used a FULL JOIN with a non-equijoin condition PostgreSQL
  rejects (0A000): rewritten with scalar subqueries
- jsonb does not preserve key order, so the \ discriminator
  could arrive mid-object and strict STJ polymorphism refused it:
  enabled AllowOutOfOrderMetadataProperties in the mapping serializer
- Shared LISTEN connection could be entered while in Waiting state
  (NpgsqlOperationInProgressException): each wait now uses a dedicated
  pooled connection; PostgresWorkQueue no longer needs IAsyncDisposable

Docs: AGENT-README (WSL/Docker test-container workflow incl. IPv4-only
forwarding and VM idle-termination), TODO (verification complete).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add configurable deterministic HTTP and mail detection rules for sensitive paths, traversal, injection indicators, suspicious User-Agents, unusual methods, error statuses, mail link heuristics, reply-to mismatches, and attachment metadata.

Add the time-window correlator, incident caps, evidence truncation, latest-open in-memory incident lookup, and the role-gated correlation worker that consumes queued event IDs and writes correlation audit records.

Add xUnit coverage for detection safety and correlation behavior, and update Phase 5 documentation.

Tests: 156 passed, 7 skipped.
Verification:
- dotnet build Viegard.slnx: 0 warnings, 0 errors
- dotnet test Viegard.slnx: 156 passed, 7 skipped
- dotnet run --project src\Viegard.PipelineHost --no-build: host booted and logged Correlation worker started

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #6
Add the MDaemon domain payload, flat-file tailing source, session-transcript and Dynamic Screening parsers, normalizer, disabled default configuration, and PipelineHost wiring. Add deterministic MDaemon evidence scoring and IP correlation support with sanitized parser fixtures and tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #7
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adopt the spec's section set and order (Purpose, Setup & commands,
Guardrails, Conventions, Current state, Surprises, Architecture,
Contacts, Changes), the colon-delimited metadata header, checkable
imperative guardrails, and the facts/preferences split.  Condensed to
the spec's one-to-two-page target by linking out to ARCHITECTURE.md,
DECISIONS.md, and TODO.md instead of inlining depth.  No guidance was
weakened; all prior security rules are preserved in Guardrails.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add D-0027 policy options, protected CIDR guardrail, in-memory guardrail state, and default policy evaluation with explainable guardrail decisions.

Wire the policy role in PipelineHost with a placeholder worker for Phase 6 classification intake.

Cover protected CIDRs, policy gates, rate caps, circuit breaker, repeat-offender duration, and guardrail state behavior. Update AGENT-README and TODO with remaining follow-up work.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the deterministic evidence classifier, typed incident and classification queue messages, and durable queue registration for both in-memory and PostgreSQL providers.

Wire correlation to hand off incidents, add the classification worker, replace the policy placeholder with classification intake, and persist policy decisions while marking incident subjects decided.

Document the deterministic-only state and add classifier plus end-to-end spine coverage.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reviewed-on: #9
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
hannah-vernon/viegard-sentinel!10
No description provided.