feat(py): uv-managed polars data layer with pytest setup #3

Merged

Philipp Baumann merged 2 commits from fm/dse-security-reports-pyproject into main 2026-10-26 17:39:11 +00:00

Conversation 0 Commits 2 Files changed 11 +1037 -1

Philipp Baumann commented 2026-10-26 17:32:52 +00:00

Owner

Copy link

Summary

The report pipeline needs a proper Python project for its data frame queries.
This change adds a uv-managed package (pyproject.toml + committed uv.lock) with the newest polars release (2.1.0) as the data frame engine and a pytest test setup driven by just recipes.
A first polars derivation module reads both Defender export shapes (plus the triage sidecar) and produces exactly the same canonical per-finding JSON as the stdlib scripts/derive_findings.py, so the two paths cross-check byte-for-byte on the synthetic fixture.

What changed

• pyproject.toml - new uv-compatible package (src layout, hatchling) pinning polars==2.1.* and pytest in the test dependency group.
• uv.lock - committed lock for reproducible installs (uv sync --locked).
• src/dse_security_reports/derive.py - polars derivation: reads both export shapes as strings, merges the triage sidecar, resolves resource groups from config/systems.csv (with the optional systems-az.csv override), and emits the same canonical payload (importance sort, per-system/per-CVE dispatch structure, aggregates) as the stdlib path.
• src/dse_security_reports/__init__.py - public API of the data layer.
• src/dse_security_reports/__main__.py - CLI mirroring scripts/derive_findings.py [samples-dir] [out-dir], writing to .local/derived-polars/.
• src/dse_security_reports/py.typed - typed package marker.
• tests/conftest.py - session fixtures: synthetic fixture generation and the stdlib reference derivation.
• tests/test_derive.py - 8 tests on synthetic data only: byte-identical output vs the stdlib derivation, deterministic across runs, payload/finding key order, importance sort, triage merge, system resolution, synthetic-marker semantics.
• Justfile - appended recipes: install (uv sync), test (pytest), py-derive, py-repro (byte-identical across two runs), py-crosscheck (stdlib vs polars).
• .gitignore - ignore .venv/ and Python bytecode.
• AGENTS.md - one-line pointer to the data layer and its recipes.

Behaviour change

The jq/stdlib pipeline is unchanged; all additions are new files plus appended just recipes.
A new parallel derivation path exists (just py-derive), proven byte-identical to the stdlib path on the fixture (just py-crosscheck) and byte-stable across runs (just py-repro); just install / just test give the Python side a reproducible environment and test suite.

Instructions-PR: none

## Summary The report pipeline needs a proper Python project for its data frame queries. This change adds a uv-managed package (`pyproject.toml` + committed `uv.lock`) with the newest polars release (2.1.0) as the data frame engine and a pytest test setup driven by just recipes. A first polars derivation module reads both Defender export shapes (plus the triage sidecar) and produces exactly the same canonical per-finding JSON as the stdlib `scripts/derive_findings.py`, so the two paths cross-check byte-for-byte on the synthetic fixture. ## What changed - `pyproject.toml` - new uv-compatible package (src layout, hatchling) pinning `polars==2.1.*` and pytest in the `test` dependency group. - `uv.lock` - committed lock for reproducible installs (`uv sync --locked`). - `src/dse_security_reports/derive.py` - polars derivation: reads both export shapes as strings, merges the triage sidecar, resolves resource groups from `config/systems.csv` (with the optional `systems-az.csv` override), and emits the same canonical payload (importance sort, per-system/per-CVE dispatch structure, aggregates) as the stdlib path. - `src/dse_security_reports/__init__.py` - public API of the data layer. - `src/dse_security_reports/__main__.py` - CLI mirroring `scripts/derive_findings.py [samples-dir] [out-dir]`, writing to `.local/derived-polars/`. - `src/dse_security_reports/py.typed` - typed package marker. - `tests/conftest.py` - session fixtures: synthetic fixture generation and the stdlib reference derivation. - `tests/test_derive.py` - 8 tests on synthetic data only: byte-identical output vs the stdlib derivation, deterministic across runs, payload/finding key order, importance sort, triage merge, system resolution, synthetic-marker semantics. - `Justfile` - appended recipes: `install` (uv sync), `test` (pytest), `py-derive`, `py-repro` (byte-identical across two runs), `py-crosscheck` (stdlib vs polars). - `.gitignore` - ignore `.venv/` and Python bytecode. - `AGENTS.md` - one-line pointer to the data layer and its recipes. ## Behaviour change The jq/stdlib pipeline is unchanged; all additions are new files plus appended just recipes. A new parallel derivation path exists (`just py-derive`), proven byte-identical to the stdlib path on the fixture (`just py-crosscheck`) and byte-stable across runs (`just py-repro`); `just install` / `just test` give the Python side a reproducible environment and test suite. Instructions-PR: none

Philipp Baumann added 1 commit 2026-10-26 17:32:52 +00:00

feat(py): add uv-managed polars data layer with pytest setup ... 95bb3b24fd

- Add pyproject.toml (packaged src layout, hatchling) with the newest
  polars release (2.1.*) as the data frame engine and pytest in the
  test dependency group; commit uv.lock for reproducible builds.
- Implement src/dse_security_reports: a polars derivation that reads
  both Defender export shapes (plus the triage sidecar) and emits the
  same canonical per-finding JSON as scripts/derive_findings.py, so
  both paths cross-check byte-for-byte on the synthetic fixture.
- Add pytest tests (synthetic data only): byte-identical output vs
  the stdlib derivation, deterministic across runs, payload shape,
  importance sort, triage merge and system resolution.
- Append just recipes: install (uv sync --locked), test (pytest),
  py-derive, py-repro (determinism smoke test) and py-crosscheck
  (stdlib vs polars, byte-identical).
- Ignore .venv/ and Python bytecode; note the data layer in AGENTS.md.

Philipp Baumann added 1 commit 2026-10-26 17:34:30 +00:00

chore(py): add ruff lint and fmt recipes from py-containerio ... f1c37e82f9

- Add ruff (newest release) in a lint dependency group and a
  [tool.ruff] config (line-length 88 chosen from the tree's measured
  distribution, google docstring convention, D1xx missing-docstring
  ignores).
- Port the good Justfile parts from asp/py-containerio: lint
  (ruff check --fix) and fmt (ruff format), scoped to src/ and tests/
  only so the stdlib pipeline lane under scripts/ stays untouched;
  its sync/test recipes add nothing over the existing locked
  install/test recipes, prek and rumdl need config this repo does not
  carry.
- Apply the formatter to the data layer sources.

Philipp Baumann merged commit 3c5f486460 into main 2026-10-26 17:39:11 +00:00

Philipp Baumann deleted branch fm/dse-security-reports-pyproject 2026-10-26 17:39:12 +00:00

Philipp Baumann referenced this pull request from a commit 2026-10-26 17:39:13 +00:00

feat(py): uv-managed polars data layer with pytest setup (#3)

Philipp Baumann referenced this pull request 2026-10-26 17:43:35 +00:00

feat(report): resource categories, DSE/ASP split, CVE links, SSH evidence, trivy agreement and patch posture #4

Philipp Baumann referenced this pull request from a commit 2026-10-26 17:59:32 +00:00

feat(report): resource categories, DSE/ASP split, CVE links, SSH evidence, trivy agreement and patch posture (#4)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

philipp.baumann/dse-security-reports!3

No description provided.