Phase 4: IMAP data source (D-0019..D-0022)

## What does this PR do? Implements the IMAP data source (Phase 4, first source) per decisions D-0019 through D-0022, and wires the front of the ingestion pipeline. - **`Viegard.Sources.Imap`** (MailKit 4.17.0, supply-chain reviewed 2026-08-18, low risk): - `ImapMailSource`, one instance per configured account: implicit TLS (993), app-password authentication via `ISecretProvider` (D-0019), folders opened **read-only** so retrieval can never mark messages seen (D-0022), IMAP IDLE with automatic polling fallback after repeated failures plus a bounded IDLE cycle acting as the safety poll (D-0020), UIDVALIDITY-aware offset resume, reconnect-with-delay on any failure, per-account health contributor. - `ImapEventNormalizer`: `MailFetchDto` JSON -> `MailMessageEvent`; fails closed on malformed payloads; truncates oversized bodies; emits sender `EntityRef`s. - `LinkExtractor`: http/https links from text bodies and HTML hrefs, deduplicated, capped at 100 against hostile mail. - Startup options validation: duplicate account IDs, missing fields, invalid ports, and the unimplemented OAuth2 seam are all refused. - **Domain:** `MailMessageEvent` / `MailAddressInfo` / `AttachmentInfo` payload records (metadata only for attachments, D-0022). - **Application:** `ObservedItem` (observation + raw payload, so sources never touch persistence), `SourceType` on `IDataSource`, `ISourceOffsetStore` port (+ in-memory implementation). - **PipelineHost:** `IngestionWorker` pumps every registered source through persist-raw -> normalize -> store-event -> enqueue-for-correlation, auditing each stage separately; the `events` queue is registered for D-0012 telemetry; ingestion is gated on the `sources` role (D-0011). Notes for reviewer: - Account configuration (hosts, usernames) is environment-specific: committed `appsettings.json` ships an empty account list; real config belongs in user-secrets (dev) or mounted config (prod). - First run baselines to **new-mail-only** (`IngestExistingOnFirstRun` = false); flagged in TODO.md for your confirmation. - The adapter has **not yet run against a live server**; that needs an app password from you (TODO.md). Fixes # ## How was this tested? - [x] `dotnet build Viegard.slnx` - 0 errors, 0 warnings - [x] `dotnet test Viegard.slnx` - 99/99 pass (29 new: normalizer incl. prompt-injection-content-stays-data and truncation, malformed-payload fail-closed, link extraction incl. hostile-mail cap and scheme filtering, options validation) - [x] Manually tested: host boots; ingestion worker idles cleanly with zero accounts; telemetry publisher reports the events queue - [ ] Live IMAP connection NOT tested (requires app password; tracked in TODO.md) ## Checklist - [x] I have read the [Contributing Guide](../CONTRIBUTING.md) - [x] Changes are focused - one logical change per PR - [x] Documentation updated (AGENT-README.md, TODO.md, THIRD-PARTY-NOTICES.md) - [x] New dependencies are MIT/Apache-2.0 and security-vetted (MailKit/MimeKit 4.17.0 MIT, BouncyCastle.Cryptography 2.6.2 MIT, MS.Extensions abstractions MIT; all recorded) - [x] No secrets, credentials, or real infrastructure identifiers in the diff - [x] No commented-out code or debug leftovers

hannah-vernon commented

2026-08-19 11:42:51 -05:00

Owner

What does this PR do?

Implements the IMAP data source (Phase 4, first source) per decisions D-0019 through D-0022, and wires the front of the ingestion pipeline.

Viegard.Sources.Imap (MailKit 4.17.0, supply-chain reviewed 2026-08-18, low risk):
- ImapMailSource, one instance per configured account: implicit TLS (993), app-password authentication via ISecretProvider (D-0019), folders opened read-only so retrieval can never mark messages seen (D-0022), IMAP IDLE with automatic polling fallback after repeated failures plus a bounded IDLE cycle acting as the safety poll (D-0020), UIDVALIDITY-aware offset resume, reconnect-with-delay on any failure, per-account health contributor.
- ImapEventNormalizer: MailFetchDto JSON -> MailMessageEvent; fails closed on malformed payloads; truncates oversized bodies; emits sender EntityRefs.
- LinkExtractor: http/https links from text bodies and HTML hrefs, deduplicated, capped at 100 against hostile mail.
- Startup options validation: duplicate account IDs, missing fields, invalid ports, and the unimplemented OAuth2 seam are all refused.
Domain: MailMessageEvent / MailAddressInfo / AttachmentInfo payload records (metadata only for attachments, D-0022).
Application: ObservedItem (observation + raw payload, so sources never touch persistence), SourceType on IDataSource, ISourceOffsetStore port (+ in-memory implementation).
PipelineHost: IngestionWorker pumps every registered source through persist-raw -> normalize -> store-event -> enqueue-for-correlation, auditing each stage separately; the events queue is registered for D-0012 telemetry; ingestion is gated on the sources role (D-0011).

Notes for reviewer:

Account configuration (hosts, usernames) is environment-specific: committed appsettings.json ships an empty account list; real config belongs in user-secrets (dev) or mounted config (prod).
First run baselines to new-mail-only (IngestExistingOnFirstRun = false); flagged in TODO.md for your confirmation.
The adapter has not yet run against a live server; that needs an app password from you (TODO.md).

Fixes #

How was this tested?

dotnet build Viegard.slnx - 0 errors, 0 warnings
dotnet test Viegard.slnx - 99/99 pass (29 new: normalizer incl. prompt-injection-content-stays-data and truncation, malformed-payload fail-closed, link extraction incl. hostile-mail cap and scheme filtering, options validation)
Manually tested: host boots; ingestion worker idles cleanly with zero accounts; telemetry publisher reports the events queue
Live IMAP connection NOT tested (requires app password; tracked in TODO.md)

Checklist

I have read the Contributing Guide
Changes are focused - one logical change per PR
Documentation updated (AGENT-README.md, TODO.md, THIRD-PARTY-NOTICES.md)
New dependencies are MIT/Apache-2.0 and security-vetted (MailKit/MimeKit 4.17.0 MIT, BouncyCastle.Cryptography 2.6.2 MIT, MS.Extensions abstractions MIT; all recorded)
No secrets, credentials, or real infrastructure identifiers in the diff
No commented-out code or debug leftovers

hannah-vernon added 1 commit

Implement IMAP data source (Phase 4, D-0019..D-0022) 89e3d95ad4

- Viegard.Sources.Imap (MailKit 4.17.0, supply-chain reviewed):
  - ImapMailSource per account: implicit TLS (993), app-password auth
    via ISecretProvider, folders opened read-only (retrieval never sets
    Seen), IDLE with polling fallback and bounded IDLE cycle as safety
    poll, UIDVALIDITY-aware offset resume, reconnect with delay,
    IHealthContributor per account
  - ImapEventNormalizer: MailFetchDto JSON -> MailMessageEvent; fails
    closed on malformed payloads; body truncation; sender entity refs
  - LinkExtractor: http/https from text + hrefs, deduped, capped
  - Options validation refuses duplicate accounts, missing fields, and
    the unimplemented OAuth2 seam
- Domain: MailMessageEvent/MailAddressInfo/AttachmentInfo payloads
- Application: ObservedItem (observation + raw payload), SourceType on
  IDataSource, ISourceOffsetStore port
- PipelineHost: IngestionWorker (persist raw -> normalize -> store ->
  enqueue -> audit, per stage); events ChannelWorkQueue registered for
  telemetry; per-account source registration; ingestion gated on the
  sources role
- Tests: 99 passing (29 new: normalizer incl. prompt-injection-as-data
  and truncation, link extraction incl. hostile-mail cap, options
  validation)
- Docs: THIRD-PARTY-NOTICES (MailKit, MimeKit, BouncyCastle, MS.Ext),
  AGENT-README integrations/structure, TODO (first-run baseline needs
  confirmation; live-account verification outstanding)

Verified: dotnet build (0 warnings), dotnet test (99/99), host boots
with ingestion worker idle when no accounts are configured.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

hannah-vernon merged commit 577ae9b16d into dev

2026-08-19 11:51:44 -05:00

hannah-vernon referenced this pull request from a commit

2026-08-19 11:51:46 -05:00

Merge pull request 'Phase 4: IMAP data source (D-0019..D-0022)' (#3) from feature/imap-source into dev

hannah-vernon referenced this pull request

2026-08-21 13:22:30 -05:00

Release: phases 1-5 complete (skeleton, sources, persistence, detection, policy, classification) #10

Rows
Columns

Phase 4: IMAP data source (D-0019..D-0022) #3

What does this PR do?

How was this tested?

Checklist