Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
a8ee9f2
Support macOS engine wheels and Metal retrieval
haoweiyu-kumo Oct 20, 2026
4d8a759
Expose per-trainer accelerator selection
haoweiyu-kumo Oct 20, 2026
848b8f1
Import Faiss without a type-checking guard
haoweiyu-kumo Oct 20, 2026
e06547d
Fix CI typing, imports, and connector test isolation
haoweiyu-kumo Oct 20, 2026
ad6d76d
Revert unrelated BigQuery test changes
haoweiyu-kumo Oct 20, 2026
98ab785
Make accelerator explicit through Lightning construction
haoweiyu-kumo Oct 20, 2026
4a9b861
Add built-in SVG graph visualization
devanj08 Oct 21, 2026
02147a8
Keep uppercase and Unicode graph labels within cards
devanj08 Oct 21, 2026
7e10946
Match graph connections to the UI curved flow
devanj08 Oct 21, 2026
5ea3789
Merge macOS wheel and accelerator support into graph visualization
devanj08 Oct 21, 2026
f4c8dab
Add runnable graph visualization notebook
devanj08 Oct 21, 2026
cbee55d
Preserve spaces in graph visualization labels
devanj08 Oct 21, 2026
d8b8029
Bound graph notebook output with scrolling and fullscreen
devanj08 Oct 21, 2026
72335e0
Show graph visualization in existing notebooks
devanj08 Oct 21, 2026
448526a
Add stable graph layout controls and notebook previews
devanj08 Oct 26, 2026
498bdbc
Polish graph navigation, tooltip placement, and time column display
devanj08 Oct 28, 2026
ab5d378
Consolidate graph column visibility documentation
devanj08 Oct 28, 2026
9cc8614
Extract shared notebook widget presentation and lifecycle
devanj08 Oct 28, 2026
a595b5a
Use widget abort signal for graph resize lifetime
devanj08 Oct 28, 2026
5d8c43b
Trim redundant graph visualization tests
devanj08 Oct 28, 2026
d8c8145
Label graph time columns as TS
devanj08 Oct 28, 2026
5992e42
Simplify graph visualization and refresh against main
devanj08 Oct 29, 2026
b72f880
Show six graph samples in a two-column tooltip
devanj08 Oct 29, 2026
d92abbc
Format graph timestamp samples as plain date and time
devanj08 Oct 29, 2026
3ea920e
Add graph PDF export and deliberate hover navigation
devanj08 Oct 29, 2026
93d63bf
Restore previous graph hover behavior and zoom order
devanj08 Oct 29, 2026
ec88713
Keep graph hover camera adjustments horizontal
devanj08 Oct 29, 2026
baa55a6
Smooth deliberate horizontal graph hover adjustments
devanj08 Oct 29, 2026
c3c93d1
Preserve right-side foreign-key hover placement
devanj08 Oct 29, 2026
5eb5b5e
Place graph download after zoom controls
devanj08 Oct 29, 2026
89f25a1
Balance fullscreen graph toolbar padding
devanj08 Oct 29, 2026
067c965
Use consistent graph toolbar spacing in both views
devanj08 Oct 29, 2026
16e4f98
Install graph PDF dependencies in the CI test environment
devanj08 Oct 29, 2026
e76c786
Export graphs to Markdown and PDF using existing dependencies
devanj08 Oct 29, 2026
01cd9c6
Redesign dark graph theme and keep hover placement stationary
devanj08 Oct 29, 2026
f008457
Increase dark theme contrast and use neutral row highlights
devanj08 Oct 29, 2026
2ac54b2
Harden graph exports and shared widget composition
devanj08 Oct 30, 2026
022bec2
Preserve widget themes in Databricks HTML outputs
devanj08 Oct 30, 2026
53e89eb
Support widget display and persistent notebook graph outputs
devanj08 Oct 30, 2026
8f59710
Select widget display automatically in Snowflake Workspace
devanj08 Oct 30, 2026
1d63aed
Preserve declared end-time columns in graph visualization
devanj08 Oct 30, 2026
3c1c1bd
Merge branch 'main' into feat/graph-visualize
devanj08 Oct 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions example_notebooks/colab_example_notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,7 @@
" ],\n",
")\n",
"graph.validate()\n",
"display(graph.visualize())\n",
"print('Graph ID:', graph.id)\n",
"print('Tables:', list(graph._named_tables))\n",
"print('Edges:', len(graph.edges))\n",
Expand Down
1 change: 1 addition & 0 deletions example_notebooks/databricks_example_notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -617,6 +617,7 @@
")\n",
"\n",
"graph.validate()\n",
"display(graph.visualize())\n",
"print(\"graph id:\", graph.id)"
]
},
Expand Down
2 changes: 1 addition & 1 deletion example_notebooks/snowflake_example_notebook.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@
"language": "python"
},
"outputs": [],
"source": "graph = kumo.Graph(\n tables=[customer, part, orders, lineitem],\n edges=[\n Edge(src_table=\"ORDERS\", fkey=\"O_CUSTKEY\", dst_table=\"CUSTOMER\"),\n Edge(src_table=\"LINEITEM\", fkey=\"L_ORDERKEY\", dst_table=\"ORDERS\"),\n Edge(src_table=\"LINEITEM\", fkey=\"L_PARTKEY\", dst_table=\"PART\"),\n ],\n)\n\ngraph.validate()\n\nsnapshot = graph.snapshot()\n\nprint(\"Snapshot:\", snapshot.id)"
"source": "from IPython.display import display\n\ngraph = kumo.Graph(\n tables=[customer, part, orders, lineitem],\n edges=[\n Edge(src_table=\"ORDERS\", fkey=\"O_CUSTKEY\", dst_table=\"CUSTOMER\"),\n Edge(src_table=\"LINEITEM\", fkey=\"L_ORDERKEY\", dst_table=\"ORDERS\"),\n Edge(src_table=\"LINEITEM\", fkey=\"L_PARTKEY\", dst_table=\"PART\"),\n ],\n)\n\ngraph.validate()\ndisplay(graph.visualize())\n\nsnapshot = graph.snapshot()\n\nprint(\"Snapshot:\", snapshot.id)"
},
{
"cell_type": "markdown",
Expand Down
120 changes: 120 additions & 0 deletions packages/kumo-engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,126 @@ To use your own data, point `FileConnector` at a directory of CSV or Parquet fil

`graph.snapshot()` ingests the tables into an on-disk graph index under the scratch directory, which also holds training runs, model bundles, and predictions. It defaults to `~/.cache/kumo-engine`; to put it on a large local disk, call `kumo.engine.configure(scratch_dir=...)` first.

## Visualize table relationships

Display declared tables and relationships in a notebook, or export SVG, PDF, or Markdown:

```python
graph.visualize()
graph.visualize("relationships.svg", show_cols=True)
graph.visualize("relationships.pdf", show_cols=True, layout="compact")
graph.visualize("relationships.md")
graph.visualize(layout="compact")
```

Visualization reads no source data by default and uses Kumo Engine's existing dependencies.
The return value is SVG text with notebook display support.
The default view includes primary keys, relationship foreign keys, and declared time (`TS`) and end-time (`TE`) columns.
`show_cols=True` adds other columns up to 10 rows per table, or 5 above 10 tables; key and time columns always remain visible.
Graphs are limited to 30 tables and require valid declared schemas without self-relations.

`layout="auto"` chooses Standard or Compact from graph structure; either is also selectable in the notebook.
Drag to pan, use the controls or Ctrl/Cmd+wheel to zoom, and choose Fit to reset the view.
Hover or focus a header or column for details without moving the graph.
The default notebook view follows the browser’s light or dark appearance; SVG and PDF exports use the light palette.
Notebook display automatically uses the widget protocol in detected Snowflake Workspace notebooks; Databricks and other hosts use HTML.
Installing the Snowflake connector extra does not select the display protocol.
HTML interactivity requires trusted output; fullscreen also needs permission from the notebook host.
Saved HTML output includes the complete, sample-free SVG image in the requested initial layout and “Rerun this cell to activate animation.”
Rerun the visualization cell once after upgrading and save the notebook to store this fallback; VS Code Restricted Mode can refuse HTML/SVG output entirely.
The Snowflake Workspace view uses the environment's `anywidget>=0.9,<1` and `ipywidgets` packages, imported only when displaying the result.
Snowflake Workspace Runtime 2.6 and later supplies these packages, including the documented GPU Runtime 2.9 target; Runtime 2.5 and earlier requires installation according to your environment's package policy.
See [Snowflake's widget installation guidance](https://docs.snowflake.com/en/user-guide/ui-snowsight/notebooks-in-workspaces/notebooks-in-workspaces-interactive-widgets#installation).
Kumo Engine does not install these packages as base dependencies or connector extras.
If the widget packages are unavailable or incompatible, the Workspace view shows a static graph and a diagnostic; SVG, PDF, and Markdown exports remain available.
The optional `graph.visualize().as_widget()` helper explicitly selects the same widget view on compatible hosts.
The widget view uses the same controls, and widget restoration still depends on the notebook host.
It inherits a host color scheme declared when mounted, with browser appearance as the fallback.
Each displayed view has its own camera; redisplay after restarting the kernel.
SVG exports contain the initial arrangement and native tooltips.
The Export commands button reveals a selectable PDF save command followed by a Markdown save command.
Run the desired command with your graph variable to save the file in the notebook filesystem.
Opening the panel does not generate a PDF, start a download, or write a file.
The PDF includes a paginated index of all declared tables, columns, data types, key/time roles, and connections, including columns omitted from the diagram.
Passing a `.pdf` path to `visualize` saves the same document on the filesystem where Kumo Engine runs, using the requested initial layout; it overwrites an existing file.
PDF export uses Matplotlib and its embedded DejaVu fonts; unsupported characters are shown as explicit Unicode escapes.
For example, `_ = graph.visualize("graph.pdf", layout="standard")` saves a PDF in the notebook's working directory; use `layout="compact"` for Compact.
Use `_ = graph.visualize("graph.md")` to save Markdown.
The assignment suppresses automatic display of the returned SVG.
In Snowflake Workspaces, retrieve the resulting file through the workspace file browser; saving requires write access to the directory.
This writes a notebook-side file and does not start a browser download.

Passing a `.md` or `.markdown` path writes a Mermaid relationship diagram and Markdown tables containing every declared column, data type, key/time role, and outgoing connection.
The file is saved on the machine running Kumo Engine and overwrites an existing file.
Mermaid diagrams render on GitHub and in compatible Markdown viewers, using the viewer's layout rather than the notebook arrangement.
The schema tables are readable without Mermaid support.

`show_samples=True` shows up to six non-null examples per visible column in two columns and up to three rows, sampled independently from up to 30 preview rows per table.
The default `show_samples=False` hides samples and reads no source data.
Previews can run warehouse queries and incur costs; the row limit does not bound bytes scanned.
Unavailable or empty previews are labeled.
Sample values are saved in notebook HTML or widget state, but never included in SVG, PDF, or Markdown exports.

### Developing another notebook widget

Use the private `_widget` helpers to define one feature specification for both notebook transports.
The shared shell supplies theme tokens, controls, view identity, and disposal; keep feature behavior inside its initializer.
This chart uses the same specification with trusted HTML and with real widget MIME:

```python
from IPython.display import HTML, display
from kumo.engine._widget import (
CSS_THEME, button, render_anywidget, render_widget,
)

chart = dict(
class_name="kumo-contribution",
body=(
'<div class="kumo-widget-toolbar">'
+ button("chart-toggle", "Toggle contribution", text="↔")
+ '</div><svg width="320" height="48" role="img" '
'aria-label="Contribution"><rect y="8" height="24"/></svg>'
),
style=f'.kumo-contribution rect{{fill:{CSS_THEME["accent"]}}}',
data={"fraction": 0.75},
script="""
const svg=host.querySelector('svg'),bar=svg.querySelector('rect');
const toggle=host.querySelector('.chart-toggle');
let fraction=data.fraction;
const update=()=>{bar.setAttribute('width',300*fraction);
svg.setAttribute('aria-label','Contribution: '+Math.round(100*fraction)+'%')};
toggle.addEventListener('click',()=>{fraction=1-fraction;update()},{signal});
update();toggle.disabled=false;
""",
)
display(HTML(render_widget(**chart)))
display(render_anywidget(**chart)) # uses environment-provided widget packages
```

`render_anywidget` lazily imports the optional dependency and provides an AFM `render({model, el})` entry point.
It transports markup and structured `data` in model state; executable feature code stays in the module.
Each view initializes its own closure, so repeated displays of one model have independent controls and camera state.
The specification is a snapshot; updating Python dictionaries does not update existing views.

- Escape user text in markup and use `data` for values; never interpret model state as code or include executable scripts in `body`.
Keep frontend code self-contained, without CDN imports, `eval`, or `new Function`.
- Initializers receive `host`, `widgetId`, `data`, `signal`, and `onDispose(callback)`.
Scope queries to `host`, attach listeners with `signal`, and enable controls after initialization.
Register timer/animation cancellation, observer disconnection, pointer release, and object-URL revocation with `onDispose`; the shell invokes cleanup on HTML removal or AFM disposal.
- Scope CSS under the feature's root class and use `CSS_THEME` for interactive colors.
Both shells retain `nodarkmode`; the widget transport inherits a declared host color scheme available at mount and otherwise follows browser appearance.
Redisplay to inherit a scheme that becomes available only after mounting.
- Use `widgetId` for IDs created by the initializer; for IDs and references already in SVG markup, pass their shared unique prefix as `namespace` so each view rewrites it independently.
- Keep static export serialization in the feature, using the resolved `THEME` palette for light exports.
Notebook-only sample values may enter saved HTML or widget state, but must stay out of static exports.
- HTML needs a trusted execution surface; anywidget needs a compatible active notebook host.
Request fullscreen directly from a user action and handle rejection; use `host.matches(":fullscreen")` to recognize the actual fullscreen element inside Shadow DOM.
Fullscreen remains subject to host permissions, and widget support alone does not prove that capability.
Show explicit notebook-file export commands with accurate destinations; keep file generation outside display initialization.

AFM initializes once attached; temporary host detachment preserves the view until the host calls its disposer.
Disposed views must be rendered again.

## Documentation

- [ADLS connector guide](https://github.com/NVIDIA/kumo-engine/blob/main/packages/kumo-adls/README.md): setup, authentication, supported reads and exports.
Expand Down
Loading
Loading