Fix non-UTC timestamp crash; harden ingestion against per-item failures #12

Merged

hannah-vernon merged 1 commit from fix/timestamp-offsets into dev

2026-08-25 17:31:45 -05:00

hannah-vernon commented

2026-08-25 17:26:18 -05:00

Owner

What does this PR do?

Fixes the crash Hannah hit the moment real SWAG traffic reached the deployed stack, plus the resilience gap it exposed.

The crash: nginx log timestamps carry the server's local offset ([25/Aug/2026:17:15:00 -0500]); the parser faithfully preserves it; Npgsql rejects any non-zero-offset DateTimeOffset for timestamptz; the events INSERT threw on the first real datagram.

The deeper defect: that exception escaped IngestionWorker, and the host's default BackgroundServiceExceptionBehavior.StopHost took down the entire pipeline: a single poisoned datagram was a crash-loop primitive, violating the "malformed input must never crash ingestion" requirement.

Changes:

Mapping layer normalizes every DateTimeOffset to UTC in the ToRow direction (12 fields across all row types). The instant is preserved exactly; only the representation changes. Read-side values come back from Postgres as UTC, which round-trips equal.
IngestionWorker catches per-item failures: logs, best-effort audits the failure, and keeps consuming the source. Correlation/classification/policy workers already had per-item catch+abandon; ingestion was the only unprotected stage.
Regression tests construct events, raw observations, and incident windows with a -05:00 offset and assert UTC-offset rows preserving the instant.

Fixes #

How was this tested?

dotnet build Viegard.slnx - 0 errors, 0 warnings
dotnet test Viegard.slnx - 263/263 pass (3 new regression tests)
Live verification on the Debian VM after merge: git pull && docker compose up -d --build, then re-drive SWAG traffic and confirm events/incidents accumulate without pipeline restarts (Hannah)

Checklist

I have read the Contributing Guide
Changes are focused - one logical change per PR
Documentation updated (n/a: behavior fix; no doc surface changed)
New dependencies: none
No secrets, credentials, or real infrastructure identifiers in the diff
No commented-out code or debug leftovers

## What does this PR do? Fixes the crash Hannah hit the moment real SWAG traffic reached the deployed stack, plus the resilience gap it exposed. **The crash:** nginx log timestamps carry the server's local offset (`[25/Aug/2026:17:15:00 -0500]`); the parser faithfully preserves it; Npgsql rejects any non-zero-offset `DateTimeOffset` for `timestamptz`; the `events` INSERT threw on the first real datagram. **The deeper defect:** that exception escaped `IngestionWorker`, and the host's default `BackgroundServiceExceptionBehavior.StopHost` took down the entire pipeline: a single poisoned datagram was a crash-loop primitive, violating the "malformed input must never crash ingestion" requirement. Changes: - **Mapping layer normalizes every `DateTimeOffset` to UTC** in the ToRow direction (12 fields across all row types). The instant is preserved exactly; only the representation changes. Read-side values come back from Postgres as UTC, which round-trips equal. - **`IngestionWorker` catches per-item failures**: logs, best-effort audits the failure, and keeps consuming the source. Correlation/classification/policy workers already had per-item catch+abandon; ingestion was the only unprotected stage. - Regression tests construct events, raw observations, and incident windows with a `-05:00` offset and assert UTC-offset rows preserving the instant. Fixes # ## How was this tested? - [x] `dotnet build Viegard.slnx` - 0 errors, 0 warnings - [x] `dotnet test Viegard.slnx` - 263/263 pass (3 new regression tests) - [ ] Live verification on the Debian VM after merge: `git pull && docker compose up -d --build`, then re-drive SWAG traffic and confirm events/incidents accumulate without pipeline restarts (Hannah) ## Checklist - [x] I have read the [Contributing Guide](../CONTRIBUTING.md) - [x] Changes are focused - one logical change per PR - [x] Documentation updated (n/a: behavior fix; no doc surface changed) - [x] New dependencies: none - [x] No secrets, credentials, or real infrastructure identifiers in the diff - [x] No commented-out code or debug leftovers

hannah-vernon added 1 commit

2026-08-25 17:26:18 -05:00

Fix non-UTC timestamp crash and harden ingestion against item failures ef968f58ea

Found live: the first real nginx datagram carried a -0500 log
timestamp; Npgsql rejects non-zero-offset DateTimeOffset for
timestamptz, the events INSERT threw, and the unhandled exception in
IngestionWorker stopped the whole pipeline host (crash loop on a
single poisoned datagram).

- Mapping layer now normalizes every DateTimeOffset to UTC in the
  ToRow direction (instant preserved); regression tests cover events,
  raw observations, and incident windows with a -05:00 offset
- IngestionWorker catches per-item failures: logs, audits the failure
  (best-effort), and keeps consuming the source; downstream workers
  already had per-item catch+abandon, ingestion was the only
  unprotected stage
- InternalsVisibleTo added so mapping tests can exercise the internal
  converter

Verified: dotnet build 0 warnings; dotnet test 263 passing (Postgres
integration tests skip locally; live verification on the Debian VM
after merge).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>