dse-security-reports

Deterministic, data-driven Typst reports over Microsoft Defender vulnerability exports (Alliance Swiss Pass DSE). The pipeline follows the deterministic approach of asp/cli-cloud-bi: a Justfile drives the recipes, data-driven .typ templates read derived JSON, and typst compile --root . --creation-timestamp 0 renders byte-identical PDFs.

Sensitive data handling

All real Defender exports, triage annotations (triage.csv sidecar), SSH evidence, trivy scans, SBOMs and everything derived from them live under .local/ (gitignored; *.pdf is ignored as well). Nothing real is ever committed; the committed pipeline runs entirely on a synthetic, deterministic fixture (just fixture). Real exports carry the capture date in their filename (ASP_Defender_*_YYYY-MM-DD.csv) and are placed in .local/real/ for just ingest; artifacts copied out for the captain's records keep that date in their name (see "Regenerating the captain's record artifacts").

Pipeline

The Justfile is the reference; this is the map.

• just fixture — writes small synthetic CSVs of both export shapes plus a triage sidecar to .local/samples/ (fake SYNTH-* devices, impossible CVE-0000-* ids, clearly marked synthetic).
• just ingest — installs exactly one ASP_Defender_Critical_Findings_*.csv and one ASP_Defender_CVSS_*.csv from .local/real/ as the active samples and derives from them.
• just derive — deterministic stdlib derivation (scripts/derive_findings.py) from CSV to canonical JSON in .local/derived/; joins the evidence, scan, CVE-link and systems caches when present, and keeps evidence-covered units that have no Defender rows as explicit coverage gaps.
• just cve-links — manual, once: fetches the Red Hat/AlmaLinux CVE link cache into .local/cve-cache/, so later derives stay offline.
• just evidence — manual, once: collects SSH evidence per reachable Linux unit into .local/evidence/<machine>.json (running kernel, installed kernel packages, pip inventories over application venvs, dnf timers, pending security updates with failure reason, the mdatp sensor state incl. its health, and the root-only Ricochet deployment stores); endpoint and account come from config/systems.json.
• just backfill-verified — manual, once per restored round: merges the verified 2026-10-28 probe results (Ricochet-store facts, mdatp health, security-check failure reasons) into a restored 2026-10-27 evidence cache; idempotent, and scripts/ssh_evidence.sh writes the same shapes in future rounds so the bridge retires itself.
• just ssh-<short> — interactive SSH shorthand per Linux unit (short = lowercased machine name without the ASP- prefix, e.g. just ssh-dse-prod-wireguard); the endpoint resolves at run time via the same shared resolution as just evidence (scripts/ssh_common.sh), so nothing connection-related lives in the Justfile.
• just ssh-list — lists every SSH shorthand with its resolved endpoint and the command that would run, without connecting.
• just scan — manual, once: runs trivy over the evidence inventories into .local/scans/<machine>.json (vulnerability DB cached once under .local/trivy/), joined by derive into the scanner-agreement dimension.
• just sbom — manual, once: emits a deterministic CycloneDX 1.7 SBOM per unit into .local/sbom/<machine>.json from the same cached DB.
• just report — renders reports/defender-findings.typ to .local/out/defender-findings.pdf.
• just triage — renders reports/defender-triage.typ to .local/out/defender-triage.pdf; all its tables are generated from the derived JSONs.
• just az-refresh — manual, once: refreshes the systems mapping live via az vm list into .local/systems-az.json, merged field-wise over the committed baseline during derive.
• just all — fixture + derive + both reports on the synthetic fixture.
• just repro — derives and renders both reports twice and compares the PDF bytes (cmp).
• just local — lists the local (gitignored) pipeline artifacts.

Requirements: just, python3, typst (standard on the DSE workstation profile); uv for the Python data lane; trivy only for the optional scan/SBOM stages.

Python data lane (polars)

src/dse_security_reports mirrors the stdlib derivation byte-for-byte on the fixture, under uv with a committed uv.lock. just install syncs the environment, just test runs the pytest suite, just lint / just fmt run ruff on src/tests only. just py-derive derives into .local/derived-polars/, just py-repro checks its determinism, and just py-crosscheck compares both lanes for identical JSON bytes.

Unit registry (config/systems.json)

One entry per match key carries resource group, location, category, subcategory (cli-cloud-bi style), domain (DSE vs ASP) and an SSH endpoint (sshAlias: an ssh.cfg alias or a raw address). Derive resolves exact match first, then longest prefix on the short machine name. just az-refresh merges live values over that baseline; a match key the baseline lacks crashes the merge, so add the baseline entry first when a genuinely new Azure VM appears. The Windows on-prem units are SSH-unreachable and simply lack evidence.

Application lock pins (config/app-locks.json)

The committed pins of the deployed application projects (tracking-quality on QMS, myride_simulation on MYRIDE): per lock file (root uv.lock and the deployed src/app/uv.lock) which package is pinned at which version, read when, last committed when, and which member lock the deployment resolves from. The derive joins these pins against the Ricochet store sweeps into the payload-level ricochet aggregate; the triage report renders it as the deployed-vs-pinned table. A refused or un-swept store stays a stated gap, never an absence.

Reports and theme

Two German Typst reports render from the derived JSON: the findings report and the triage report, whose curated problem rows stay in the template while every cell value is derived. Both use the standard library only, no @preview imports, and share the theme of the captain's cost report in asp/cli-cloud-bi/cost-overview/asp-cost-report.typ. The reports state that the two exports use different filters and are never a temporal comparison, and that Defender findings may be false positives, so they are an analysis and triage aid, never a verdict.

Regenerating the captain's record artifacts

Real-data runs happen in the primary checkout; the disposable worktrees only ever see the synthetic fixture. After a real-data run, the sensitive deliverables are copied into the private firstmate artifacts dir (~/firstmate/data/dse-security-reports-triage/artifacts/, outside the repo and never committed), with date-stamped PDF names:

# render both reports from the active real derivation
just report
just triage
# copy the PDFs (and, on a fresh evidence/scan round, the caches) into the private artifacts dir
cp .local/out/defender-findings.pdf ~/firstmate/data/dse-security-reports-triage/artifacts/
cp .local/out/defender-triage.pdf ~/firstmate/data/dse-security-reports-triage/artifacts/defender-triage-YYYY-MM-DD.pdf

The committed reports/defender-triage.typ is the only source of truth for the triage report; do not keep hand-edited template copies in the artifacts dir.

Planned follow-ups (out of scope of the pipeline)

• Steer parallel analysis agents on the derived dispatch structure (per system and per CVE), each producing a verifiable ~10-simple-sentence analysis hint per problem.
• Challenge the existing dse-ansible and dse-opentofu setups against the findings; az CLI and the Azure pi extensions are approved for these analysis stages (the pipeline itself stays offline and deterministic).
• Optional Azure API integration for fetching the exports.