Skip to content

feat(engine): add training diagnostics and XAI widget - #409

Draft
devanj08 wants to merge 1 commit into
dm-batch-prediction-xaifrom
feat/training-xai-widget
Draft

devanj08 wants to merge 1 commit into
dm-batch-prediction-xaifrom
feat/training-xai-widget

Conversation

@devanj08

@devanj08 devanj08 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Display retained training diagnostics and explanations with training_result.visualize().
The widget includes evaluation charts, train/validation epoch history, column importance and sampled entity subgraphs.
It supports PDF printing of every chart view on the active tab, offline interactive HTML export and a static saved-notebook snapshot.

What Happens and Why

The existing Python API exposes retained artifacts and individual plots, but has no combined interactive diagnostic view.
This view reads the saved model and explanation outputs without starting training or inference.
Missing artifacts produce bounded views with explicit coverage rather than fabricated results.

The Fix

Reuse the shared notebook widget transport and typed subgraph readers, and add task-specific diagnostic adapters with a self-contained SVG frontend.
Keep binary cutoff controls separate from class-labeled tasks and bound graph height and initial exploration depth.
Preserve the existing UI data rules, with documented safe heatmap alignment/count handling and explicit missing/equal sensitivity scores.

Blast Radius

This is a read-only display API; training and prediction execution are unchanged.
This stack is based on the batch-XAI API in #289; retarget to main after its prerequisites merge.

Verification

The validation report records passing artifact tests and Chromium, Firefox and WebKit checks of plotted PDF content and offline HTML state using retained data.
Native notebook print dialogs and the full host/OS matrix were not validated by these local checks.

Change map.

@copy-pr-bot

copy-pr-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@devanj08
devanj08 deployed to gha-kumo-workflow-tests November 19, 2026 06:50 — with GitHub Actions Active
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Pre-merge checks: Passed 1 · Inconclusive 2

Failed Failed checks: 2 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive ## Summary I could not read the PR conversation comment that holds the Change map, so I could not assess its design. ## Evidence gathered - The check requires the PR conversation comment that starts w… Provide the full body of the PR conversation comment that begins with <!-- kumo-pr-change-map --> (the comment linked as #issuecomment-7002011813). Then re-run this check. The review will test the Change map against the repository code …
Pr Size Inconclusive The check fails only if the PR changes more than 40 files. The provider file count for PR #409 is not available in the supplied context, and the base branch is not in the local clone. Evidence gathere… Confirm the PR file count in the provider (GitHub "Files changed" tab for #409, compared with base dm-batch-prediction-xai). Fetch the base branch tip and run git diff --name-only <base-tip> <head> to count files. If the…
Passed Passed checks: 1 passed

Check name Status Explanation
Public Resource Access Passed PR #409 (author devanj08, base dm-batch-prediction-xai, head feat/training-xai-widget) was checked at commit 0cde40b against its parent 7640f27. The added lines in the repository d…
Full details: Public Resource Access

Explanation

PR #409 (author devanj08, base dm-batch-prediction-xai, head feat/training-xai-widget) was checked at commit 0cde40b against its parent 7640f27. The added lines in the repository diff contain no added URL, git, package-index, CDN, or repository source location that a user must access to build, test, use, or contribute to the changed code: - The only http strings are SVG namespace identifiers (xmlns="http://www.w3.org/2000/svg" in the Python SVG generators, and const NS = 'http://www.w3.org/2000/svg' in the frontend JavaScript). These are XML namespace identifiers, not fetched resources, so they are excluded. - packages/kumo-engine/test/widget_exports.cjs adds require('playwright') and documents NODE_PATH=&lt;directory-containing-playwright&gt;. This names a package and uses a placeholder path. It adds no package source URL. - packages/kumo-engine/test/widget_validation.md and explainability.md add no links. The pyproject.toml change adds only a package-data entry. - No pip install, npm install, CDN, unpkg, jsdelivr, or esm.sh reference was added. The PR description links to a GitHub blob for widget_validation.md at commit 0cde40b and to a PR comment. Those links are outside the repository diff. The same validation content is in the repository file packages/kumo-engine/test/widget_validation.md, which is added by this PR. Anonymous access to those GitHub links was not tested, so they are not counted as changed-code references for this check. No added external resource reference was found that fails or cannot be tested reliably.

Full details: Change Map Design Review

Explanation

Summary I could not read the PR conversation comment that holds the Change map, so I could not assess its design. ## Evidence gathered - The check requires the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt;. No tool in this run fetches PR comments. The description links the comment (#issuecomment-7002011813), but that link cannot be opened here. - A search of the checked-out repository for kumo-pr-change-map found only docs/development/review.md, which describes the convention. No copy of the comment exists in the repository. - The checkout is at commit 0cde40b5, the PR head, with widget_validation.md present. This file holds validation results only. It does not contain the Change map. - The primary PR record matches the metadata: NVIDIA/kumo-engine #409, base dm-batch-prediction-xai, head feat/training-xai-widget, author devanj08. - No CodeRabbit review threads were supplied, so no review finding bears on the Change map. ## Why the verdict is INCONCLUSIVE The Change map is the only input the check evaluates. Without its text, I cannot test whether each change is necessary, whether it fits the right layer, or whether it has the observed failure, inspected behavior, smallest sufficient change, focused verification, and stopping condition. Missing evidence cannot establish PASS. The check does not define a FAIL condition for an unavailable comment, so FAIL is also not supported.

Resolution

Provide the full body of the PR conversation comment that begins with &lt;!-- kumo-pr-change-map --&gt; (the comment linked as #issuecomment-7002011813). Then re-run this check. The review will test the Change map against the repository code at commit 0cde40b. It will check the observed failure or release requirement, the inspected existing behavior, the proposed change set, the verification, and the stopping condition. It will also check the behavior the table omits.

Full details: Pr Size

Explanation

The check fails only if the PR changes more than 40 files. The provider file count for PR #409 is not available in the supplied context, and the base branch is not in the local clone. Evidence gathered: - The local HEAD is 0cde40b, "Add training diagnostics and explainability widget". It matches the PR title and touches the _explainability widget files. - Parent-to-head diff of 0cde40b touches 32 files (git diff --name-only HEAD^ HEAD). - The base branch dm-batch-prediction-xai is not present locally. Its tip is unknown. - Seven earlier commits on the local branch (68892c8 through 7640f27) are prediction-explanation work. The PR description says the stack is based on the batch-XAI API in #289, so these commits may belong to the base or to the PR. - The PR file count depends on the base tip. Counted against candidate base tips: - 7640f27: 32 files (PASS) - 5b5e466: 34 files (PASS) - 8444357: 34 files (PASS) - 1c5711a: 39 files (PASS) - 68892c8: 41 files (FAIL) - 23df659: 41 files (FAIL) - ea49aa2: 43 files (FAIL) - 0ef29e7: 48 files (FAIL) - The head commit alone is 32 files. Whether the PR also includes the earlier prediction-explanation commits decides the result. Required evidence (the exact changed-file count against the real base) remains unavailable. Missing evidence cannot establish PASS. No code defect is asserted.

Resolution

Confirm the PR file count in the provider (GitHub "Files changed" tab for #409, compared with base dm-batch-prediction-xai). Fetch the base branch tip and run git diff --name-only &lt;base-tip&gt; &lt;head&gt; to count files. If the count is 40 or fewer, the check passes. If the count exceeds 40, the check fails. In that case, split the PR into reviewable units, for example by separating the batch-XAI prerequisites from the training diagnostics widget.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@devanj08

Copy link
Copy Markdown
Contributor Author

Change map

Requested feature: retained training diagnostics with offline exports.
Existing artifact readers and shared widget shell are reused; no training or prediction generation is added.
Focused tests cover artifact and rendering contracts; local browser checks cover exports.

File Function / scope What changed Why
packages/kumo-engine/explainability.md widget display and saved-snapshot guide Document supported display/export behavior, retained-artifact requirements and inactive-notebook output. Explain the display APIs, exports and retained-artifact limitations.
packages/kumo-engine/kumo/engine/_explainability/__init__.py ExplainabilityView; for_result; open_run Expose a lazy training view through HTML, SVG and anywidget display using the existing shared widget transport. Keep notebook rendering lazy and share existing transport.
packages/kumo-engine/kumo/engine/_explainability/_cards.py _cards Compose only diagnostic and explanation cards supported by the retained task and artifacts. Avoid controls for unsupported task/artifact combinations.
packages/kumo-engine/kumo/engine/_explainability/_classification.py _classification Compute binary threshold confusion matrices and ROC/PR/gain views from retained predictions; keep class-labeled tasks binary-control-free. Keep threshold charts consistent with the same retained rows.
packages/kumo-engine/kumo/engine/_explainability/_columns.py cohorts; importance; groups Adapt retained gradients and cohort vectors to importance and grouped predictions with explicit coverage. Make saved cohort distributions navigable with or without importance scores.
packages/kumo-engine/kumo/engine/_explainability/_coverage.py _coverage Distinguish full and retained-prefix coverage so sampled metrics and charts are not represented as complete. Expose incomplete data instead of implying full-population results.
packages/kumo-engine/kumo/engine/_explainability/_files.py RunFiles; locate; bundle_file; holdout Locate published, selected and refit artifacts through fsutil and read only the retained columns needed for display. Resolve artifacts consistently across local and supported storage paths.
packages/kumo-engine/kumo/engine/_explainability/_forecasting.py _forecasting Render retained forecast totals and horizon summaries with reference baselines. Make horizon performance and forecast magnitude inspectable.
packages/kumo-engine/kumo/engine/_explainability/_format.py _format Produce consistent finite-number, percentage and missing-value display text across charts and static snapshots. Use consistent missing and finite-value formatting.
packages/kumo-engine/kumo/engine/_explainability/_header.py _header Identify the model query, target entity, dates and retained holdout coverage. Keep model and evaluation provenance visible.
packages/kumo-engine/kumo/engine/_explainability/_holdout.py _holdout Align task-specific finite prediction and target rows before computing diagnostics. Give each diagnostic the same valid prediction/target pairs.
packages/kumo-engine/kumo/engine/_explainability/_metrics.py fits; baselines; table; _history; card Compose baseline comparisons, all metrics, original last-epoch training scores and train/validation epoch histories. Compare scores and training progress with correct fit attribution.
packages/kumo-engine/kumo/engine/_explainability/_regression.py _regression Compute retained prediction/residual charts and UI-aligned heatmaps while safely preserving bin counts. Expose residual patterns while preserving heatmap totals and alignment.
packages/kumo-engine/kumo/engine/_explainability/_render.py _render Package the same self-contained frontend resources for both notebook transports and offline HTML export. Share the renderer between live notebooks and offline exports.
packages/kumo-engine/kumo/engine/_explainability/_run.py _run Read one supported training run once and share its task, plan, predictions and coverage across cards. Avoid repeated artifact parsing and tolerate optional malformed inputs.
packages/kumo-engine/kumo/engine/_explainability/_snapshot.py Snapshot: metrics, charts and static MIME Preserve basic diagnostics as script-free SVG/HTML/plain MIME for inactive notebook sessions. Retain useful diagnostics when no kernel is active.
packages/kumo-engine/kumo/engine/_explainability/_subgraphs.py card; collector_card Normalize existing typed subgraph records without regenerating explanations, preserving missing scores and large IDs. Display saved graph samples without fabricating scores or reading unlisted batch artifacts.
packages/kumo-engine/kumo/engine/_explainability/static/charts.js charts Draw responsive SVG axes, rows and legends, and share accessible selectors across cards without a CDN dependency. Share chart primitives and selectors across task views.
packages/kumo-engine/kumo/engine/_explainability/static/classification.js classification Render binary cutoff controls and their linked confusion, ROC, PR and gain charts. Make cutoff changes update every binary chart consistently.
packages/kumo-engine/kumo/engine/_explainability/static/epoch-history.js epoch-history Display selectable train/validation metrics and experiments with missing values represented as gaps. Inspect training progress without adding a separate page.
packages/kumo-engine/kumo/engine/_explainability/static/exports.js exports Print every chart view on the active page and export the full interactive widget with saved selections. Make all active-page views printable and saved interaction state portable.
packages/kumo-engine/kumo/engine/_explainability/static/forecasting.js forecasting Render saved forecast horizon and total comparisons. Expose totals and horizon comparisons from the saved payload.
packages/kumo-engine/kumo/engine/_explainability/static/regression.js regression Render saved residual, heatmap and prediction distribution views. Display residual and prediction distributions responsively.
packages/kumo-engine/kumo/engine/_explainability/static/shell.js KX.cards.metrics; KX.cards.importance; KX.cards.groups; mount Coordinate tabs, metrics and explanation cards and their linked column state within one widget instance. Coordinate page navigation and linked explanation selections.
packages/kumo-engine/kumo/engine/_explainability/static/subgraphs.js training sample graph exploration and outcome details Explore sampled entity graphs with sensitivity sizing, hover paths, details, filters and bounded initial depth. Keep large saved graphs usable within the notebook viewport.
packages/kumo-engine/kumo/engine/_explainability/static/widget.css scoped training headers, controls, charts and graph styles Style scoped responsive light/dark diagnostics and bounded graphs using the approved widget palette. Keep charts readable across themes and notebook widths.
packages/kumo-engine/kumo/engine/_widget.py _widget Preserve optional static MIME output and fallback on failed anywidget mount, retaining existing graph defaults. Preserve script-free diagnostics when interactive mounting is unavailable.
packages/kumo-engine/kumo/engine/trainer/trainer.py TrainingResult.visualize Expose the requested visualize entrypoint on TrainingResult. Expose widget display on returned results without rerunning jobs.
packages/kumo-engine/pyproject.toml pyproject Include widget CSS and JavaScript in installed Engine distributions. Ensure installed wheels contain the required frontend resources.
packages/kumo-engine/test/test_explainability.py test_explainability Cover training task families, numeric diagnostics, missing/partial artifacts, typed subgraphs, static MIME and self-contained scripts. Verify numerical, partial-data and saved-notebook behavior across training tasks.
packages/kumo-engine/test/widget_exports.cjs offline export browser contracts Verify actual SVG plots in all-view PDF output, saved importance expansion, offline exports and task-specific layout in three browsers. Catch print-time rendering and saved-state regressions that Python and syntax checks cannot detect.
packages/kumo-engine/test/widget_validation.md training results Record reproducible validation commands, actual outcomes and platform limitations for the widget candidate. Provide reviewable evidence for the checks reported in the PR.

This branch was successfully deployed

1 active deployment
gha-kumo-workflow-tests — 0cde40b5 Deployed Nov 19, 2026 by devanj08 via import #2678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant