feat(report): resource categories, DSE/ASP split, CVE links, SSH evidence, trivy agreement and patch posture #4

Merged

Philipp Baumann merged 5 commits from fm/dse-security-reports-cve-overview into main

2026-10-26 17:59:32 +00:00

Philipp Baumann commented

2026-10-26 17:43:35 +00:00

Owner

Summary

PR 2 (the real-data leg) was merged mid-flight, so this follow-up carries the five additions the captain requested on top of it, rebased onto current main (including the polars data layer from #3).
Each Defender finding now carries a cli-cloud-bi-style resource categorization, a DSE-vs-ASP domain split, a cached Red Hat/AlmaLinux CVE link, an SSH-evidence installed-version ground truth, a trivy scanner-agreement dimension, and each scanned unit gains a patch-management posture summary.

What changed

config/systems.csv -- category (Compute / On-Prem), subcategory (Persistente VMs, CI-Runner-VMs for crowci*, On-Prem Systeme), domain (DSE vs ASP) and SSH alias columns; exact rows for the three DSE Alma machines map to their dse-ansible aliases (dse_dev_misc, dse_dev_workbench, dse_prod_workbench), all other units are SSH-unreachable.
scripts/derive_findings.py -- resolves category/subcategory/domain/sshAlias per finding and system; derives per-category (with subcategories and severity split) and per-domain aggregations; resolves CVE links from the raw cache (Red Hat CVE page preferred, best-matching AlmaLinux erratum as fallback); compares claimed versions against the SSH evidence cache; joins trivy detections into the scanner-agreement dimension (trivy-confirms / trivy-silent with matching installed version / defender-only) with scan metadata and counts; computes patch posture incl. lifecycle position against the EOL table; merges optional az-refresh rows field-wise over the committed baseline.
scripts/fetch_cve_links.sh + just cve-links -- fetches the raw link cache once, live (Alma errata indexes 8-10, HTTP status per Red Hat CVE page, snapshot date) into gitignored .local/cve-cache/.
scripts/ssh_evidence.sh + just evidence -- connects once, live, per SSH-reachable unit (dse-ansible ssh.cfg alias, admin account philipp_baumann, publickey) and runs a fixed read-only command sequence: sorted rpm -qa, openssl/glibc versions, rpm matches for the Defender software tokens, and the patch posture (dnf-automatic timer states, last dnf transaction, Alma release, pending security updates with an explicit check status so a failed repo-metadata fetch is not reported as zero) -- one stable-sorted JSON per unit under gitignored .local/evidence/.
scripts/scan_trivy.sh + just scan -- runs trivy once, live, over the evidence inventories: the vulnerability DB is cached once under .local/trivy/ (then --skip-db-update), each inventory becomes a minimal CycloneDX SBOM (distro family alma plus pkg:rpm/alma purls, which trivy needs for the advisory lookup) and the raw report is normalized into one stable-sorted JSON per unit under gitignored .local/scans/ with trivy version and DB date.
scripts/az_refresh_systems.sh -- az runs under the dse-ansible prod environment convention (direnv exec over the prod envrc pulling pipeline credentials from Azure Key Vault via the authenticated az session).
config/alma-eol.csv -- committed static EOL date table (8: 2029-05-31, 9: 2032-05-31, 10: 2035-05-31) so lifecycle positions stay offline-deterministic.
reports/defender-findings.typ -- CVE-Uebersicht gains the Link column (RH/ALMA badge, hyperlinked CVE id) and the Trivy column (compact T+ / T- / D markers next to the false-positive column, with legend and trivy/DB versions); new sections Befunde nach Ressourcen-Kategorie, Befunde nach Domaene and Patch-Management pro System; per-system table shows category and domain; Evidence section compares claimed versions against installed RPMs.
AGENTS.md -- records the live caches, mapping columns, patch/EOL rules, the Azure direnv convention and the vulnix availability note.

Behaviour change

The committed fixture pipeline stays byte-identical (just all, just repro verified). With the real exports under .local/real/ the report renders: 203 Critical / 157 CVSS>9.5 findings with machine and resource group per row, 27 Red Hat CVE links (Alma fallback verified), On-Prem 163 vs Compute 40 and ASP 183 vs DSE 20 splits, SSH evidence from all three DSE Alma machines, trivy agreement for every DSE finding (all 14 currently defender-only -- the Defender claims are pip packages with no RPM counterpart), and patch posture: Alma 9.8 (supported until 2032-05-31), dnf-automatic enabled everywhere, last automatic runs on 2026-10-25/26, and the security-update check currently failing on repo metadata (Microsoft repo GPG signature) -- explicit context for the queued dse-ansible/opentofu challenge.
All live inputs and caches stay under gitignored .local/ and renders remain offline and deterministic.

Notes:

vulnix 1.12.5 is installed but stays unwired (Nix-derivation-focused, no Nix-managed unit); noted in AGENTS.md instead of the handwritten root README, which is deliberately untouched.
For the polars lane: just py-crosscheck now fails by design -- src/dse_security_reports still mirrors the pre-categorization payload shape while the stdlib pipeline (this lane, per the ownership split in the Justfile) extended it. The polars layer needs a resync; this PR deliberately does not touch src/.

Validation: just repro byte-identical on fixture and real data; live CVE fetch (85 CVEs), live SSH evidence (3 units), live trivy scans (1595/1850/1850 detections, DB 2026-10-26) and patch posture collected; az-override field-merge and baseline restoration tested; ingest error path tested.

Instructions-PR: none

## Summary PR 2 (the real-data leg) was merged mid-flight, so this follow-up carries the five additions the captain requested on top of it, rebased onto current main (including the polars data layer from #3). Each Defender finding now carries a cli-cloud-bi-style resource categorization, a DSE-vs-ASP domain split, a cached Red Hat/AlmaLinux CVE link, an SSH-evidence installed-version ground truth, a trivy scanner-agreement dimension, and each scanned unit gains a patch-management posture summary. ## What changed - `config/systems.csv` -- category (Compute / On-Prem), subcategory (Persistente VMs, CI-Runner-VMs for crowci*, On-Prem Systeme), domain (DSE vs ASP) and SSH alias columns; exact rows for the three DSE Alma machines map to their dse-ansible aliases (dse_dev_misc, dse_dev_workbench, dse_prod_workbench), all other units are SSH-unreachable. - `scripts/derive_findings.py` -- resolves category/subcategory/domain/sshAlias per finding and system; derives per-category (with subcategories and severity split) and per-domain aggregations; resolves CVE links from the raw cache (Red Hat CVE page preferred, best-matching AlmaLinux erratum as fallback); compares claimed versions against the SSH evidence cache; joins trivy detections into the scanner-agreement dimension (trivy-confirms / trivy-silent with matching installed version / defender-only) with scan metadata and counts; computes patch posture incl. lifecycle position against the EOL table; merges optional az-refresh rows field-wise over the committed baseline. - `scripts/fetch_cve_links.sh` + `just cve-links` -- fetches the raw link cache once, live (Alma errata indexes 8-10, HTTP status per Red Hat CVE page, snapshot date) into gitignored `.local/cve-cache/`. - `scripts/ssh_evidence.sh` + `just evidence` -- connects once, live, per SSH-reachable unit (dse-ansible ssh.cfg alias, admin account philipp_baumann, publickey) and runs a fixed read-only command sequence: sorted rpm -qa, openssl/glibc versions, rpm matches for the Defender software tokens, and the patch posture (dnf-automatic timer states, last dnf transaction, Alma release, pending security updates with an explicit check status so a failed repo-metadata fetch is not reported as zero) -- one stable-sorted JSON per unit under gitignored `.local/evidence/`. - `scripts/scan_trivy.sh` + `just scan` -- runs trivy once, live, over the evidence inventories: the vulnerability DB is cached once under `.local/trivy/` (then --skip-db-update), each inventory becomes a minimal CycloneDX SBOM (distro family alma plus pkg:rpm/alma purls, which trivy needs for the advisory lookup) and the raw report is normalized into one stable-sorted JSON per unit under gitignored `.local/scans/` with trivy version and DB date. - `scripts/az_refresh_systems.sh` -- az runs under the dse-ansible prod environment convention (direnv exec over the prod envrc pulling pipeline credentials from Azure Key Vault via the authenticated az session). - `config/alma-eol.csv` -- committed static EOL date table (8: 2029-05-31, 9: 2032-05-31, 10: 2035-05-31) so lifecycle positions stay offline-deterministic. - `reports/defender-findings.typ` -- CVE-Uebersicht gains the Link column (RH/ALMA badge, hyperlinked CVE id) and the Trivy column (compact T+ / T- / D markers next to the false-positive column, with legend and trivy/DB versions); new sections Befunde nach Ressourcen-Kategorie, Befunde nach Domaene and Patch-Management pro System; per-system table shows category and domain; Evidence section compares claimed versions against installed RPMs. - `AGENTS.md` -- records the live caches, mapping columns, patch/EOL rules, the Azure direnv convention and the vulnix availability note. ## Behaviour change The committed fixture pipeline stays byte-identical (`just all`, `just repro` verified). With the real exports under `.local/real/` the report renders: 203 Critical / 157 CVSS>9.5 findings with machine and resource group per row, 27 Red Hat CVE links (Alma fallback verified), On-Prem 163 vs Compute 40 and ASP 183 vs DSE 20 splits, SSH evidence from all three DSE Alma machines, trivy agreement for every DSE finding (all 14 currently defender-only -- the Defender claims are pip packages with no RPM counterpart), and patch posture: Alma 9.8 (supported until 2032-05-31), dnf-automatic enabled everywhere, last automatic runs on 2026-10-25/26, and the security-update check currently failing on repo metadata (Microsoft repo GPG signature) -- explicit context for the queued dse-ansible/opentofu challenge. All live inputs and caches stay under gitignored `.local/` and renders remain offline and deterministic. Notes: - vulnix 1.12.5 is installed but stays unwired (Nix-derivation-focused, no Nix-managed unit); noted in AGENTS.md instead of the handwritten root README, which is deliberately untouched. - For the polars lane: `just py-crosscheck` now fails by design -- `src/dse_security_reports` still mirrors the pre-categorization payload shape while the stdlib pipeline (this lane, per the ownership split in the Justfile) extended it. The polars layer needs a resync; this PR deliberately does not touch `src/`. Validation: `just repro` byte-identical on fixture and real data; live CVE fetch (85 CVEs), live SSH evidence (3 units), live trivy scans (1595/1850/1850 detections, DB 2026-10-26) and patch posture collected; az-override field-merge and baseline restoration tested; ingest error path tested. Instructions-PR: none

Philipp Baumann added 3 commits

2026-10-26 17:43:35 +00:00

feat(report): categorize resources cli-cloud-bi style and split DSE vs ASP domain 783f6810e5

This change will:
- extend the committed systems mapping with category, subcategory and
  domain columns: Azure VMs are Compute (Persistente VMs, or CI-Runner-VMs
  (Crow CI) for crowci-prefixed units), on-prem machines are their own
  On-Prem category; DSE covers the RG-DSE units and Analytics-Prod, ASP
  covers myRIDE, QMS, VS and the on-prem machines
- resolve category, subcategory and domain per finding and per system
  through the same deterministic exact-then-longest-prefix matching as
  the resource groups
- aggregate per category and subcategory (counts, systems, exploit
  availability and severity split) and per domain, sorted by findings
  descending like the cost report's per-category table
- render two new report sections (Befunde nach Ressourcen-Kategorie,
  Befunde nach Domäne) with dynamically sized severity columns and add
  category and domain columns to the per-system table
- make az vm list refresh rows carry the full mapping shape and merge
  them over the committed baseline per Match string so baseline-only
  rows such as the on-prem machines survive a live refresh

feat(report): add CVE link column resolved from a cached Red Hat / AlmaLinux snapshot 05c1d8c8e7

This change will:
- add just cve-links and scripts/fetch_cve_links.sh, fetching the raw link
  cache once, live: the static AlmaLinux errata indexes (releases 8-10),
  the HTTP status of each Red Hat CVE page, and the snapshot date, all
  under gitignored .local/cve-cache/
- resolve each CVE link deterministically from that cache during derive,
  preferring the Red Hat security page and falling back to the best
  matching AlmaLinux erratum (security advisories first, then lowest
  release, then advisory id); without a cache every link stays empty and
  renders remain offline
- add the Link column to the CVE-Uebersicht and triage table (RH / ALMA
  badge, CVE id itself hyperlinked) and note the cache snapshot date in
  the section intro
- keep the raw/derived split of the cli-cloud-bi pipeline: fetched live
  once, then reproducible byte-identical renders from what is stored

feat(report): add SSH evidence stage as false-positive ground truth ed39c6c1b4

This change will:
- add just evidence and scripts/ssh_evidence.sh, connecting once, live,
  per SSH-reachable unit from the committed mapping (dse-ansible ssh.cfg
  alias, admin account philipp_baumann, publickey) and running a fixed
  read-only command sequence (sorted rpm -qa, openssl/glibc versions, rpm
  matches for every Defender software token and its suffix-stripped base
  name) into one stable-sorted JSON per unit under gitignored
  .local/evidence/
- extend config/systems.csv with an SshAlias column, carried on exact
  rows for the three DSE Alma machines (dse_dev_misc,
  dse_dev_workbench, dse_prod_workbench); units without an alias are
  SSH-unreachable
- merge optional az-refresh rows field-wise over the committed baseline
  so live resource groups refresh while SSH aliases, categories and
  domains keep their committed values
- compare, when evidence exists for a unit, every finding record against
  the cache (installed packages per software token, a mechanical
  claimed-version-installed hint, the collection date) and aggregate a
  per-unit evidence view (claimed versions vs installed packages, plus
  openssl/glibc)
- render the evidence comparison as a new report section noting the
  snapshot date, with an explanatory fallback when no cache exists

Philipp Baumann added 2 commits

2026-10-26 17:57:43 +00:00

feat(report): join trivy scans into a scanner-agreement dimension 24c03ef991

This change will:
- add just scan and scripts/scan_trivy.sh, running trivy once, live,
  over the SSH evidence package inventories: the vulnerability DB is
  fetched once into .local/trivy (then --skip-db-update against the
  cached DB), each inventory becomes a minimal CycloneDX SBOM (distro
  family alma plus pkg:rpm/alma purls, which trivy needs for the Red
  Hat / AlmaLinux advisory lookup), and the raw report is normalized
  into one stable-sorted JSON per unit under gitignored .local/scans/
  with the trivy version and DB date recorded
- join Defender findings, SSH installed-version evidence and trivy
  detections into a scanner-agreement value per finding
  (trivy-confirms / trivy-silent with matching installed version /
  defender-only), recorded per finding plus aggregate counts and scan
  metadata in the payload
- add the Trivy column to the CVE-Uebersicht and triage table (compact
  T+ / T- / D markers next to the existing false-positive column,
  table re-densified to fit nine columns) with a legend and the trivy
  version / DB date noted in the section intro
- note in AGENTS.md that vulnix 1.12.5 is installed but stays unwired
  (Nix-derivation-focused, no Nix-managed unit); the handwritten root
  README is deliberately not touched

feat(report): add patch-management posture and Azure direnv convention 6b9feab4a1

This change will:
- extend the SSH evidence stage with a fixed patch-posture sequence
  per unit: dnf-automatic timer states, the last dnf transaction
  (dnf history info last, date parsed from the C-locale format), the
  Alma release and the pending security updates from
  dnf --security check-update, recording the check status so a failed
  repo-metadata fetch is reported as error instead of a fake zero count
- add the committed static EOL date table config/alma-eol.csv and
  compute each unit's lifecycle position (supported/eol with days
  remaining) deterministically from the cache's collection date
- render a per-system patch-management table in the report (automation,
  last dnf run, Alma version, lifecycle, pending security updates with
  an explicit Fehler state when the repo metadata is not fetchable)
- run az-refresh's az calls under the dse-ansible prod environment
  convention (direnv exec over the prod envrc, which pulls pipeline
  credentials from Azure Key Vault via the authenticated az session at
  runtime) instead of a bare az

Philipp Baumann changed title from ~~feat(report): resource categories, DSE/ASP domain split, CVE links and SSH evidence~~ to feat(report): resource categories, DSE/ASP split, CVE links, SSH evidence, trivy agreement and patch posture

2026-10-26 17:58:01 +00:00

Philipp Baumann merged commit 0ba72e6cbe into main

2026-10-26 17:59:32 +00:00

Philipp Baumann deleted branch fm/dse-security-reports-cve-overview

2026-10-26 17:59:32 +00:00

Philipp Baumann referenced this pull request from a commit

2026-10-26 17:59:32 +00:00

feat(report): resource categories, DSE/ASP split, CVE links, SSH evidence, trivy agreement and patch posture (#4)