philipp.baumann/dse-security-reports
Watch
0
0
Fork
You've already forked dse-security-reports
 0
Code Issues 1 Pull requests Releases Activity

feat(pipeline): network reachability probe, machine layout diagram and Defender switch-status table #18

Merged
Philipp Baumann merged 4 commits from fm/dse-security-reports-network-probe-layout into main 2026-10-27 21:36:09 +00:00
Conversation 0 Commits 4 Files changed 11 +1665 -29
Philipp Baumann commented 2026-10-27 17:39:50 +00:00
Owner
Copy link

Summary

The reachability probe of 2026-10-27, the machine layout and the Defender switch states existed only in conversation and chat history. This change captures all three as reproducible pipeline stages: a committed, deterministic network probe recipe with its unreachable-vs-closed rule and control-host gate, a deterministically generated Mermaid layout diagram (source committed, rendered asset embedded like the logo), and an authenticated Defender-for-Cloud configuration fetch cached once so the report renders switch and plan tables offline and byte-identically.

What changed

  • scripts/network_probe.sh + Justfile (just network-probe): bounded, read-only reachability probe per unit from the workstation (one ICMP echo, 20 TCP connects with 3 s timeout, name resolution) into .local/network/<machine>.json plus _context.json (method record, port set, control/auxiliary hosts). A silent host is unreachable, never closed; a closed-port finding requires a demonstrable answer; the control-host gate (two public estate endpoints from the committed mapping plus the corporate DNS server) aborts without writing a cache when nothing answers. Units without a committed endpoint (ASP-DSE-PROD-MISC) get an explicit not-attempted file.
  • scripts/derive_findings.py + src/dse_security_reports/derive.py (mirrored, just py-crosscheck green): load_network/network_aggregate joining the probe cache with the same fixture-only gate as the evidence cache, covering the same unit union as the locations aggregate; load_defender_config/defender_config_aggregate for the configuration cache; the covered-machines union is factored into _covered_machines (locations output preserved byte-for-byte).
  • scripts/fetch_defender_config.sh + Justfile (just defender-config): integration switches, auto-provisioning, pricing plans (via az rest — this az CLI build rejects az security pricing list --scope, recorded in the script header) and the WindowsDefenderATP service-principal facts (role assignments, Graph grants), cached once to .local/defender-config.json; failed queries leave explicit unknowns with their reason.
  • scripts/gen_layout_diagram.py + reports/machine-layout.mmd/.svg + Justfile (just layout): diagram generated deterministically from config/systems.json and the derived locations/network aggregates, rendered with pinned mmdr 0.3.1 into the committed, cache-like SVG; the committed facts layer (PTR sweep, second subscription, ASP VNet) carries its provenance; the environment DSE cannot read appears as an explicitly unread external boundary.
  • reports/defender-triage.typ: three new sections — Netzwerk-Reichweite der Einheiten (per-unit states, never a blank cell), Maschinen- und Ressourcen-Layout (embedded SVG with source attribution) and Defender-Konfigurationsstatus (switch and plan tables with the read date; unknowns with reason, never off/none; absent cache stated as a gap) — plus Grundlagen/Reproduzierbarkeit updates.
  • tests/test_derive.py: network aggregate states, cache gating next to the fixture, and real-shaped joins proving both lanes stay byte-identical with the live caches.
  • AGENTS.md: durable notes (probe rule, az CLI quirk, mmdr layout limits) and the extended published-round restore list.

Behaviour change

  • The first committed probe run reproduced the 2026-10-27 baseline: the three non-DSE Windows units and both domain controllers answered nothing (unreachable, names NXDOMAIN on the reachable corporate DNS), the eight Linux units answered on their open ports, controls answered, ASP-DSE-PROD-MISC is explicitly not measured.
  • The first cached Defender fetch reproduced the baseline: WDATP/WDATP_UNIFIED_SOLUTION/MCAS on, Sentinel and WDATP_EXCLUDE_LINUX_PUBLIC_PREVIEW off, auto-provisioning off, VirtualMachines Standard/P1, KeyVaults Standard/PerKeyVault, Containers/CloudPosture/Discovery/FoundationalCspm Standard without sub-plan; the WindowsDefenderATP principal carries zero role assignments and zero Graph grants.
  • No derived number moved: the new real-data payload equals the published 2026-10-27 round apart from the two new aggregate keys; just repro proves byte-identical PDFs on the fixture and on the real round.
  • The published 2026-10-27 round was re-rendered from the restored caches plus the fresh network/defender caches and published into the firstmate artifacts dir (previous same-round copies moved to artifacts/superseded/, the 2026-10-26 set untouched), with network-2026-10-27/ and defender-config-2026-10-27.json added so the round stays restorable.

Validation: just all, just repro (byte-identical, both reports, fixture and real data), just test (33), just lint, just py-crosscheck, just py-repro green.

Instructions-PR: none

## Summary The reachability probe of 2026-10-27, the machine layout and the Defender switch states existed only in conversation and chat history. This change captures all three as reproducible pipeline stages: a committed, deterministic network probe recipe with its unreachable-vs-closed rule and control-host gate, a deterministically generated Mermaid layout diagram (source committed, rendered asset embedded like the logo), and an authenticated Defender-for-Cloud configuration fetch cached once so the report renders switch and plan tables offline and byte-identically. ## What changed - `scripts/network_probe.sh` + `Justfile` (`just network-probe`): bounded, read-only reachability probe per unit from the workstation (one ICMP echo, 20 TCP connects with 3 s timeout, name resolution) into `.local/network/<machine>.json` plus `_context.json` (method record, port set, control/auxiliary hosts). A silent host is `unreachable`, never `closed`; a closed-port finding requires a demonstrable answer; the control-host gate (two public estate endpoints from the committed mapping plus the corporate DNS server) aborts without writing a cache when nothing answers. Units without a committed endpoint (ASP-DSE-PROD-MISC) get an explicit `not-attempted` file. - `scripts/derive_findings.py` + `src/dse_security_reports/derive.py` (mirrored, `just py-crosscheck` green): `load_network`/`network_aggregate` joining the probe cache with the same fixture-only gate as the evidence cache, covering the same unit union as the locations aggregate; `load_defender_config`/`defender_config_aggregate` for the configuration cache; the covered-machines union is factored into `_covered_machines` (locations output preserved byte-for-byte). - `scripts/fetch_defender_config.sh` + `Justfile` (`just defender-config`): integration switches, auto-provisioning, pricing plans (via `az rest` — this az CLI build rejects `az security pricing list --scope`, recorded in the script header) and the WindowsDefenderATP service-principal facts (role assignments, Graph grants), cached once to `.local/defender-config.json`; failed queries leave explicit unknowns with their reason. - `scripts/gen_layout_diagram.py` + `reports/machine-layout.mmd`/`.svg` + `Justfile` (`just layout`): diagram generated deterministically from `config/systems.json` and the derived locations/network aggregates, rendered with pinned mmdr 0.3.1 into the committed, cache-like SVG; the committed facts layer (PTR sweep, second subscription, ASP VNet) carries its provenance; the environment DSE cannot read appears as an explicitly unread external boundary. - `reports/defender-triage.typ`: three new sections — Netzwerk-Reichweite der Einheiten (per-unit states, never a blank cell), Maschinen- und Ressourcen-Layout (embedded SVG with source attribution) and Defender-Konfigurationsstatus (switch and plan tables with the read date; unknowns with reason, never off/none; absent cache stated as a gap) — plus Grundlagen/Reproduzierbarkeit updates. - `tests/test_derive.py`: network aggregate states, cache gating next to the fixture, and real-shaped joins proving both lanes stay byte-identical with the live caches. - `AGENTS.md`: durable notes (probe rule, az CLI quirk, mmdr layout limits) and the extended published-round restore list. ## Behaviour change - The first committed probe run reproduced the 2026-10-27 baseline: the three non-DSE Windows units and both domain controllers answered nothing (`unreachable`, names NXDOMAIN on the reachable corporate DNS), the eight Linux units answered on their open ports, controls answered, ASP-DSE-PROD-MISC is explicitly not measured. - The first cached Defender fetch reproduced the baseline: WDATP/WDATP_UNIFIED_SOLUTION/MCAS on, Sentinel and WDATP_EXCLUDE_LINUX_PUBLIC_PREVIEW off, auto-provisioning off, VirtualMachines Standard/P1, KeyVaults Standard/PerKeyVault, Containers/CloudPosture/Discovery/FoundationalCspm Standard without sub-plan; the WindowsDefenderATP principal carries zero role assignments and zero Graph grants. - No derived number moved: the new real-data payload equals the published 2026-10-27 round apart from the two new aggregate keys; `just repro` proves byte-identical PDFs on the fixture and on the real round. - The published 2026-10-27 round was re-rendered from the restored caches plus the fresh network/defender caches and published into the firstmate artifacts dir (previous same-round copies moved to `artifacts/superseded/`, the 2026-10-26 set untouched), with `network-2026-10-27/` and `defender-config-2026-10-27.json` added so the round stays restorable. Validation: `just all`, `just repro` (byte-identical, both reports, fixture and real data), `just test` (33), `just lint`, `just py-crosscheck`, `just py-repro` green. Instructions-PR: none 
## Summary

The manual reachability probe of 2026-10-27 existed only in conversation; this change captures it as a committed, deterministic recipe and renders its result per unit in the triage report. The probe's core rule carries through the whole chain: a host that does not respond at all is recorded unreachable, never closed -- a closed-port finding is only valid when the host demonstrably answers something, so an unrouted subnet can never read as a hardened host.

## What changed

- `scripts/network_probe.sh`: bounded, read-only probe per unit from the workstation -- one ICMP echo, 20 TCP connects (3 s timeout each), name resolution (ssh.cfg alias for the Linux units, corporate DNS over TCP for the 10.0.1.0/24 environment). Own control-host gate first (two public endpoints on the DSE-controlled Azure estate from the committed mapping plus the corporate DNS server); without an answering estate control the script aborts without writing any cache. Units without a committed endpoint (ASP-DSE-PROD-MISC) get an explicit not-attempted file. Output: `.local/network/<machine>.json` per unit plus `_context.json` (method record, port set, control and auxiliary hosts, PTR-swept domain controllers). Collection date only, no moving timestamps.
- `Justfile`: recipe `network-probe`.
- `scripts/derive_findings.py` + `src/dse_security_reports/derive.py` (mirrored, `just py-crosscheck` green): `load_network`/`network_aggregate` reading the probe cache with the same fixture-only gate as the evidence cache, a `network` payload aggregate covering the same unit union as the locations aggregate (export, evidence, cloud side); a unit without a cache renders as the explicit not-attempted, never a clean result. The covered-machines union is factored into `_covered_machines`, shared with the locations aggregate (byte-identical output preserved).
- `reports/defender-triage.typ`: new section "Netzwerk-Reichweite der Einheiten (Workstation-Sicht)" -- per-unit table (target, name resolution, ICMP, TCP states, status) plus the control/auxiliary hosts table, all rendered from the aggregate; never a blank cell. Prose states the unreachable-vs-closed rule and the control gate; the no-route observation of the manual round stays attributed to that round.
- `tests/test_derive.py`: network aggregate function test (unreachable stays unreachable, no cache stays the explicit not-attempted, controls/auxiliary pass through), fixture-gate test extended with a network cache next to the synthetic lane, and a real-shaped join test proving both lanes stay byte-identical with the live cache in place.

## Behaviour change

- The first committed run reproduced the 2026-10-27 baseline: the three non-DSE Windows units and both domain controllers answered nothing (unreachable, names NXDOMAIN on the reachable corporate DNS), the eight Linux units answered on their open ports, the estate control hosts and the corporate DNS answered, ASP-DSE-PROD-MISC is explicitly not measured (no committed endpoint).
- Renders read the probe cache only; without a run every unit renders as explicitly not measured.

Validation: `just all`, `just repro` (byte-identical), `just test` (33), `just lint`, `just py-crosscheck`, `just py-repro` green on the fixture; the live first run wrote the baseline cache and re-derivation joined it byte-identically.
## Summary

The machine layout is now known well enough to draw: eleven units across two platforms, an Azure estate DSE controls and one environment it does not. This change generates the diagram deterministically from the committed mapping and the derived data, commits the Mermaid source for review, renders it with the pinned mmdr and embeds the rendered asset in the triage report like the logo -- the standard-library-only, no-`@preview` rule stays intact.

## What changed

- `scripts/gen_layout_diagram.py`: generates `reports/machine-layout.mmd` from `config/systems.json` (resource groups, hosting, roles) and the derived locations/network aggregates (IMDS/az regions, probe states). A committed facts layer records the topology facts of the 2026-10-27 round with their provenance: the PTR sweep over 10.0.1.0/24, the second visible subscription (Vov Azure, RG-DWHBI-DEV/-PROD, zero VMs), the ASP VNet read (VN-AZ-VPN in RG-VPN, default subnet with no VM/NIC). Refuses to overwrite the committed diagram from a synthetic-fixture derivation.
- `reports/machine-layout.mmd` + `reports/machine-layout.svg` (committed): the generated source is reviewable and hand-editable; the rendered SVG is committed as a cache-like asset, refreshed only when the generated source changes -- mmdr output is byte-stable across runs (verified), but committing the asset keeps `just repro` independent of a renderer on the render path.
- `Justfile`: recipe `layout` (generate + render, mmdr pinned to 0.3.1).
- `reports/defender-triage.typ`: new section "Maschinen- und Ressourcen-Layout" embedding the SVG as a local file (like `allianceswisspass-logo.svg`), with source attribution and regeneration pointers; Grundlagen and Reproduzierbarkeit updated.

## Behaviour change

- The report carries a data-derived layout: the DSE-controlled subscription with its six resource groups and units (regions from the measurement precedence), the second visible subscription, and the non-DSE environment as an explicitly unread external boundary -- its nodes are PTR-swept names, the dotted edge to the ASP-VNet states the measured prefix reuse without a route, and no structure is guessed.

Validation: `just repro` byte-identical with the committed asset embedded; the diagram source renders deterministically (three consecutive mmdr runs byte-identical).
## Summary

The Defender for Cloud switch states were read live on 2026-10-27 and lived only in chat. This change captures them as an authenticated refresh stage in the same shape as the existing Azure refresh -- query once, cache to a gitignored file, render only from the cache -- and renders the deterministic switch and plan tables in the triage report.

## What changed

- `scripts/fetch_defender_config.sh`: fetches the integration switches (`az security setting list`), agent auto-provisioning (`az security auto-provisioning-setting list`), the pricing plans via the REST form (`az rest` against the Microsoft.Security/pricings endpoint -- recorded in the script header because `az security pricing list --scope` is rejected by this CLI build with "unrecognized arguments: --scope" and a bare invocation fails with a missing scope_id) and the WindowsDefenderATP service-principal facts (`az ad sp list`, `az role assignment list --assignee <id> --all`, Graph oauth2PermissionGrants via `az rest`). Writes the sorted, date-only cache `.local/defender-config.json`; a failed query leaves its surface as an explicit unknown with its reason and the script exits non-zero, but the cache is always written so the honest unknown renders.
- `scripts/derive_findings.py` + `src/dse_security_reports/derive.py` (mirrored, `just py-crosscheck` green): `load_defender_config`/`defender_config_aggregate` gated like every live cache; the payload carries `defenderConfig` (present plus the cache contents, or the explicit absent state).
- `Justfile`: recipe `defender-config`.
- `reports/defender-triage.typ`: new section "Defender-Konfigurationsstatus (Cloud)" with a switch table (values an/aus plus the read date) and a plan table (tier, sub-plan) -- an unreadable or missing value renders as an explicit unknown with its reason, never as off or none, and without a cache the section says so instead of rendering an empty table. The service-principal context (no Azure role assignments, no Graph permissions -- hence machines visible outside any held subscription, data reachable only through Defender roles) renders only from the recorded queries.
- `tests/test_derive.py`: fixture-gate assertions (cache next to the synthetic lane stays absent) and the real-shaped join test asserting the full cache pass-through in both lanes byte-identically.

## Behaviour change

- The first cached run reproduced the 2026-10-27 baseline: WDATP/WDATP_UNIFIED_SOLUTION/MCAS on, Sentinel and WDATP_EXCLUDE_LINUX_PUBLIC_PREVIEW off, auto-provisioning off, VirtualMachines Standard/P1, KeyVaults Standard/PerKeyVault, Containers/CloudPosture/Discovery/FoundationalCspm Standard without sub-plan; the WindowsDefenderATP principal carries zero role assignments and zero Graph grants.

Validation: `just all`, `just repro` (byte-identical), `just test` (33), `just lint`, `just py-crosscheck`, `just py-repro` green on the fixture; the live fetch reproduced the baseline and re-derivation joined it byte-identically.
## Summary

The three new live stages carry project knowledge worth keeping for every future session: the unreachable-vs-closed rule of the reachability probe, the az CLI pricing quirk of the Defender config fetch, and mmdr's layout limitations for the diagram. This change records those in the project memory and extends the published-round restore list with the new caches.

## What changed

- `AGENTS.md`: entries for `just network-probe` (rule, control gate, committed target tables), `just defender-config` (surfaces, REST quirk, cache semantics) and `just layout` (committed mmd/svg assets, pinned mmdr, layout-engine limitations, fixture refusal); the restore recipe now names `network-*/` and `defender-config-<date>.json`; the triage-report table list names the three new sections.

## Behaviour change

- None (documentation only).
Philipp Baumann deleted branch fm/dse-security-reports-network-probe-layout 2026-10-27 21:36:09 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
philipp.baumann/dse-security-reports!18
No description provided.