feat(report): non-DSE Windows entity section and per-unit physical location #17

Merged

Philipp Baumann merged 1 commit from fm/dse-security-reports-non-dse-entity-section into main

2026-10-27 16:36:13 +00:00

Philipp Baumann commented

2026-10-27 16:31:53 +00:00

Owner

Summary

The three Windows units the report labeled "On-Prem" are Azure VMs in a VoeV environment the DSE side does not control, and the coarse Azure/On-Prem location label carried no physical meaning (nine Linux VMs rendered identically as "Azure" although eight run in westeurope and one in switzerlandnorth). This change records the hosting and ownership facts in the committed baseline, renders a dedicated deterministic section for the non-DSE Windows entity with each machine's role, adds a bounded instance-metadata probe (with a VM-vs-physical fallback) to the evidence stage, derives and renders each unit's real physical location, and makes the published 2026-10-27 round restorable by adding the missing baseline entry for ASP-DSE-PROD-MISC plus a deliberate loud failure for unknown override match keys.

What changed

config/systems.json: new hosting/environment/organisation/role/region fields on every entry. The three Windows units are corrected to hosting: Azure, category Compute, subcategory "Azure-VMs in fremder Umgebung", environment: not-DSE, organisation: unknown (stated unknown, never a guess) and a per-machine role from the Defender software inventory (dor01 = SQL Server database server, ent01 = Tomcat/Java web server, cop01 = ASP.NET Core application server). Added the exact baseline entry for ASP-DSE-PROD-MISC (RG-DSE-PROD, empty sshAlias, zero Defender rows naturally).
scripts/ssh_evidence.sh: bounded, read-only physical-location probe in the fixed sequence -- one curl -s -m 5 -H Metadata:true http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01 per unit (region, availability zone, platform fault domain, VM id, resource group), and where that call fails systemd-detect-virt plus the readable /sys/class/dmi/id files (VM vs physical hardware, vendor/product/board); a host answering neither is recorded unknown, never guessed. Caches predating the probe get a bounded metadata-only refresh merged in (a failed probe leaves the file untouched).
scripts/az_refresh_systems.sh: writes the real Azure region per VM (region field from az vm list) as the cloud-side fallback for units whose in-machine probe is missing or failed.
scripts/derive_findings.py + src/dse_security_reports/derive.py (mirrored byte-for-byte, just py-crosscheck green): extended mapping schema; physical_location precedence IMDS -> az region -> host facts -> explicit unknown (regions normalized to the az lowercase spelling); a payload-level locations aggregate over export machines, evidence units and (real data only, the fixture lane stays fixture-only) cloud-side units; the merged mapping now fails loudly on an unknown override match key.
reports/defender-triage.typ: new section "Die Windows-Entitaet: Azure-VMs in einer nicht-DSE-Umgebung" (evidence prose + deterministic table: role, hosting, environment, organisation, service and URL as explicit stated gaps) and new section "Physischer Standort der Einheiten" (per-unit region/zone/source/detail, explicit unknown for the three Windows units); On-Prem wording corrected throughout (map headings, method bullets, P1/P5 prose).
reports/defender-findings.typ: the RG-cell fallback renders the foreign Azure hosting instead of "On-Prem"; prose corrected.
tests/test_derive.py: schema/order assertions updated; new tests for the location precedence, the cloud-side unit join, the fixture gate and the loud unknown-override-key failure in both lanes (31 tests green).

Behaviour change

The 2026-10-27 report renders the real fleet: 8 Linux units measured in westeurope (IMDS), ASP-ANALYTICS-PROD-MISC in switzerlandnorth (IMDS), ASP-DSE-PROD-MISC in westeurope via the az fallback (its only possible path: no SSH alias, zero Defender rows), and the three Windows units with an explicit unknown location instead of inheriting the coarse label. Service and URL of the three Windows machines render as an explicit stated gap.
The hosting classification is a committed fact per machine and is never inferred from a resource group being absent from the Defender export -- that inference is what produced the wrong On-Prem label.
Decision stated explicitly (Part C): the derive keeps failing when an az-override match key is absent from the committed baseline, but the failure is now deliberate and named ("systems-az override names X, which the committed baseline lacks; add the baseline entry first") instead of an incidental KeyError deep in resolution. Rationale: a machine live in Azure but absent from the baseline would flow into the report without its committed facts (alias, hosting, ownership), so mapping drift fails fast rather than silently passing. With the ASP-DSE-PROD-MISC baseline entry in place, the archived published override restores cleanly (verified: derived aggregates match the published round and the derive no longer crashes with the override in place).
The published 2026-10-27 round was re-rendered from the restored caches plus the fresh metadata round and published into the firstmate artifacts dir (previous copies moved to artifacts/superseded/; the 2026-10-26 PDFs untouched); the dated evidence-2026-10-27/ and derived-2026-10-27/ caches were refreshed so the round stays restorable.

Validation: just all, just repro, just test (31), just lint, just py-crosscheck, just py-repro all green on the fixture; the real-data leg re-derived and rendered both reports with just repro byte-identical.

Instructions-PR: none

## Summary The three Windows units the report labeled "On-Prem" are Azure VMs in a VoeV environment the DSE side does not control, and the coarse Azure/On-Prem location label carried no physical meaning (nine Linux VMs rendered identically as "Azure" although eight run in westeurope and one in switzerlandnorth). This change records the hosting and ownership facts in the committed baseline, renders a dedicated deterministic section for the non-DSE Windows entity with each machine's role, adds a bounded instance-metadata probe (with a VM-vs-physical fallback) to the evidence stage, derives and renders each unit's real physical location, and makes the published 2026-10-27 round restorable by adding the missing baseline entry for ASP-DSE-PROD-MISC plus a deliberate loud failure for unknown override match keys. ## What changed - `config/systems.json`: new `hosting`/`environment`/`organisation`/`role`/`region` fields on every entry. The three Windows units are corrected to `hosting: Azure`, category Compute, subcategory "Azure-VMs in fremder Umgebung", `environment: not-DSE`, `organisation: unknown` (stated unknown, never a guess) and a per-machine role from the Defender software inventory (dor01 = SQL Server database server, ent01 = Tomcat/Java web server, cop01 = ASP.NET Core application server). Added the exact baseline entry for ASP-DSE-PROD-MISC (RG-DSE-PROD, empty `sshAlias`, zero Defender rows naturally). - `scripts/ssh_evidence.sh`: bounded, read-only physical-location probe in the fixed sequence -- one `curl -s -m 5 -H Metadata:true http://169.254.169.254/metadata/instance/compute?api-version=2021-02-01` per unit (region, availability zone, platform fault domain, VM id, resource group), and where that call fails `systemd-detect-virt` plus the readable `/sys/class/dmi/id` files (VM vs physical hardware, vendor/product/board); a host answering neither is recorded unknown, never guessed. Caches predating the probe get a bounded metadata-only refresh merged in (a failed probe leaves the file untouched). - `scripts/az_refresh_systems.sh`: writes the real Azure region per VM (`region` field from `az vm list`) as the cloud-side fallback for units whose in-machine probe is missing or failed. - `scripts/derive_findings.py` + `src/dse_security_reports/derive.py` (mirrored byte-for-byte, `just py-crosscheck` green): extended mapping schema; `physical_location` precedence IMDS -> az region -> host facts -> explicit unknown (regions normalized to the az lowercase spelling); a payload-level `locations` aggregate over export machines, evidence units and (real data only, the fixture lane stays fixture-only) cloud-side units; the merged mapping now fails loudly on an unknown override match key. - `reports/defender-triage.typ`: new section "Die Windows-Entitaet: Azure-VMs in einer nicht-DSE-Umgebung" (evidence prose + deterministic table: role, hosting, environment, organisation, service and URL as explicit stated gaps) and new section "Physischer Standort der Einheiten" (per-unit region/zone/source/detail, explicit unknown for the three Windows units); On-Prem wording corrected throughout (map headings, method bullets, P1/P5 prose). - `reports/defender-findings.typ`: the RG-cell fallback renders the foreign Azure hosting instead of "On-Prem"; prose corrected. - `tests/test_derive.py`: schema/order assertions updated; new tests for the location precedence, the cloud-side unit join, the fixture gate and the loud unknown-override-key failure in both lanes (31 tests green). ## Behaviour change - The 2026-10-27 report renders the real fleet: 8 Linux units measured in westeurope (IMDS), ASP-ANALYTICS-PROD-MISC in switzerlandnorth (IMDS), ASP-DSE-PROD-MISC in westeurope via the az fallback (its only possible path: no SSH alias, zero Defender rows), and the three Windows units with an explicit unknown location instead of inheriting the coarse label. Service and URL of the three Windows machines render as an explicit stated gap. - The hosting classification is a committed fact per machine and is never inferred from a resource group being absent from the Defender export -- that inference is what produced the wrong On-Prem label. - **Decision stated explicitly (Part C):** the derive keeps failing when an az-override match key is absent from the committed baseline, but the failure is now deliberate and named ("systems-az override names X, which the committed baseline lacks; add the baseline entry first") instead of an incidental `KeyError` deep in resolution. Rationale: a machine live in Azure but absent from the baseline would flow into the report without its committed facts (alias, hosting, ownership), so mapping drift fails fast rather than silently passing. With the ASP-DSE-PROD-MISC baseline entry in place, the archived published override restores cleanly (verified: derived aggregates match the published round and the derive no longer crashes with the override in place). - The published 2026-10-27 round was re-rendered from the restored caches plus the fresh metadata round and published into the firstmate artifacts dir (previous copies moved to `artifacts/superseded/`; the 2026-10-26 PDFs untouched); the dated `evidence-2026-10-27/` and `derived-2026-10-27/` caches were refreshed so the round stays restorable. Validation: `just all`, `just repro`, `just test` (31), `just lint`, `just py-crosscheck`, `just py-repro` all green on the fixture; the real-data leg re-derived and rendered both reports with `just repro` byte-identical. Instructions-PR: none

Philipp Baumann added 1 commit

2026-10-27 16:31:53 +00:00

feat(report): non-DSE Windows entity section and per-unit physical location 93f1812912

## Summary
The three Windows units the report labeled On-Prem are Azure VMs in a
VoeV environment DSE does not control, and the coarse Azure/On-Prem
location label carried no physical meaning. This change records the
hosting/ownership facts in the committed baseline, renders a dedicated
non-DSE entity section with each machine's role (service/URL as an
explicit stated gap), adds a bounded instance-metadata probe with a
VM-vs-physical fallback to the evidence stage, derives a per-unit
physical location (IMDS -> az refresh -> host facts -> explicit
unknown), renders it in its own section, and makes the published round
restorable by adding the missing baseline entry for ASP-DSE-PROD-MISC
plus a deliberate loud failure for unknown override match keys.

## What changed
- config/systems.json: hosting/environment/organisation/role/region
  fields on every entry (the three Windows units corrected to Azure,
  Compute, environment not-DSE, organisation unknown, per-machine role
  from the Defender software inventory) and the exact baseline entry for
  ASP-DSE-PROD-MISC.
- scripts/ssh_evidence.sh: bounded read-only instance-metadata probe
  (169.254.169.254, 5 s, no credentials) plus the systemd-detect-virt
  and /sys/class/dmi/id fallback in the fixed sequence, with a bounded
  metadata-only refresh merged into caches that predate the probe.
- scripts/az_refresh_systems.sh: writes the real Azure region per VM
  (region field) as the cloud-side fallback.
- scripts/derive_findings.py and src/dse_security_reports/derive.py:
  extended system-field schema, physical_location precedence, a
  payload-level locations aggregate over export units, evidence units
  and cloud-side units (real data only), and the explicit unknown-override-key
  failure; both lanes stay byte-identical (just py-crosscheck).
- reports/defender-triage.typ: entity section (roles, hosting,
  environment, organisation unknown, service/URL gap) and physical
  location section; On-Prem wording corrected throughout.
- reports/defender-findings.typ: RG fallback renders the foreign Azure
  hosting instead of On-Prem; prose corrected.
- tests/test_derive.py: schema, precedence, fixture-gate, cloud-side
  and loud-failure coverage (31 tests green).

## Behaviour change
The 2026-10-27 report shows nine measured Linux locations (8x
westeurope IMDS, switzerlandnorth for ANALYTICS, westeurope via az for
ASP-DSE-PROD-MISC) and three explicit unknown Windows locations; the
hosting classification never falls back to inferring On-Prem from a
missing resource group. An az override naming a machine the baseline
lacks now fails both lanes with a message naming the key (decision
kept from the published-round restore crash) instead of an incidental
KeyError; the archived published override merges cleanly now that
ASP-DSE-PROD-MISC has its baseline entry.

Philipp Baumann merged commit 81ccffd23f into main

2026-10-27 16:36:13 +00:00

Philipp Baumann deleted branch fm/dse-security-reports-non-dse-entity-section

2026-10-27 16:36:14 +00:00

Philipp Baumann referenced this pull request from a commit

2026-10-27 16:36:15 +00:00

feat(report): non-DSE Windows entity section and per-unit physical location (#17)