Skip to content

Add a generic compute client driven by task discovery - #395

Open
amirabbas-jalali-bentley wants to merge 2 commits into
SeequentEvo:mainfrom
amirabbas-jalali-bentley:add-generic-compute-engine
Open

amirabbas-jalali-bentley wants to merge 2 commits into
SeequentEvo:mainfrom
amirabbas-jalali-bentley:add-generic-compute-engine

Conversation

@amirabbas-jalali-bentley

@amirabbas-jalali-bentley amirabbas-jalali-bentley commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Description

DiscoveryClient (#387) lists the tasks an organization can run, with the JSON Schema of each task's parameters. This adds ComputeClient, which uses that catalogue to run any listed task without a hand-written client for it.

Series overview & trying the full preview build

Today evo-compute only covers tasks that have a hand-written client, so every new task or parameter change requires dedicated SDK code. Building on DiscoveryClient (#387), this PR starts a 6-part series that brings a discovery-driven generic compute engine to evo-compute:

  1. Add a generic compute client driven by task discovery #395 (this PR — ready for review): Core ComputeClient (client.<topic>.<task>.run(...) and arun(...)) with lazy discovery, schema-shaped runtime signatures, and explicit preview opt-in.
  2. Validate compute task parameters against their schema #407 (draft): Schema-driven parameter validation (ParameterValidationError, required/nullability checks by default, opt-in JSON Schema Draft 2020-12 deep validation, and discriminated union diagnostics).
  3. Resolve compute references and hydrate task results generically #408 (draft): Generic reference resolution for typed geoscience objects, attributes, and files on input, plus TaskResult / ResultNode hydration (.load() and .to_dataframe()) on output.
  4. Generate static type stubs for the generic compute engine #409 (draft): Offline .pyi stub generator and bundled catalogue snapshot (engine.pyi + py.typed) for static autocomplete and mypy / pyright checking while keeping execution live.
  5. Use custom runners for compute tasks and honor schema extra-field rules #410 (draft): Custom task runner seam (evo.compute.overrides, used by Kriging for cross-field checks, folded filter/diagnostic inputs, and KrigingResult) and schema-driven additionalProperties handling.
  6. Add compute job handles and unify task execution on ComputeClient #411 (draft): Non-blocking TaskJob handles (submit(...), asubmit(...), TaskJob.from_url(...)), context-shared discovery caching, and routing evo.compute.tasks.run through ComputeClient so existing notebooks work unchanged on the engine.

End outcome & test build: With the full stack in place, callers can run any catalogued compute task with typed SDK objects, static editor hints, hydrated results, and optional job handles, while existing evo.compute.tasks code and example notebooks continue to work as drop-in calls. To try the complete end-state in an existing evo-sdk environment today, install the preview wheel from the 0.0.7a1 preview release:

pip install https://github.com/aminabedi-bentley/evo-python-sdk/releases/download/evo-compute-preview-0.0.7a1/evo_compute-0.0.7a1-py3-none-any.whl
# Switch back anytime with: pip install evo-compute==0.0.6

The follow-up PRs (#407–#411) are open in draft so the full design is visible end to end. Please take a look at this first PR (#395) — once it lands, we will rebase and open the next PRs in sequence.

Changes

  • ComputeClient(context) (exported from evo.compute) exposes the catalogue as client.<topic>.<task>.run(...). Topic, task and parameter names are the platform's names with hyphens replaced by underscores, so normal-score is client.geostatistics.normal_score. arun(topic, task, parameters) does the same for names held as strings.
  • run fetches the catalogue on first use, binds the arguments to a keyword-only signature built from the task's parameter schema, submits the job through JobClient and returns its results as a dict. Parameters are sent as given, in the form the schema declares. Only the parameters passed are sent, so the platform applies its own defaults, while an explicit None or 0 is still sent. A missing or unknown parameter raises TypeError before anything is submitted.
  • preview=True opts in to preview tasks. It is off by default, including for tasks behind a feature flag.
  • Once the catalogue is cached, run shows the schema-shaped signature and the task description, and dir() lists the cached topics and tasks for tab-completion. Attribute access alone never makes a request.
  • If a task's parameter names can't all be keyword arguments (a reserved word, or two names that match once hyphens become underscores), run takes **parameters with the schema's names instead, while still checking required and unknown schema keys before submission.
  • DiscoveryClient.peek_tasks() returns the cached catalogue without fetching it, or an empty list while the cache is unset or expired. The signature and tab-completion lookups use it. Like the other accessors, it returns copies.
  • ComputeClient is added to the API client list. Its generated docs page is included, because the docs workflow only updates pages that already exist.
  • Fixes the filter example in kriging_multiple.ipynb. It imported the removed RegionFilter, passed target_region_filter, which KrigingParameters doesn't have, and called to_dict(), which doesn't exist. It now uses target_filter=Filter(where=FilterCondition(...)).
from evo.compute import ComputeClient

client = ComputeClient(manager)

# Arguments take the form the task's parameter schema declares.
results = await client.geostatistics.declustering.run(
    source=..., grid=..., target=..., neighborhood=..., preview=True
)

# The same task by name
results = await client.arun(
    "geostatistics",
    "declustering",
    {"source": ..., "grid": ..., "target": ..., "neighborhood": ..., "preview": True},
)

No breaking changes. This adds a class and a method, and the existing task clients are unchanged.

Testing

  • New ComputeClient tests run against a mocked discovery endpoint with job submission patched. They cover when discovery happens, name normalisation for topics, tasks and parameters, which parameters are sent, preview handling, errors, the **parameters fallback (including required and unknown parameter rejection), signatures and descriptions after caching, and dir(). New peek_tasks tests cover the cache, its expiry and the returned copies.
  • evo-compute tests: 541 passed (24 new). ruff check, ruff format --check and the pre-commit hooks pass.

Checklist

  • I have read the contributing guide and the code of conduct

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The **parameters fallback bypasses required and unknown parameter validation, allowing invalid calls to reach job submission; schema-key validation and rejection tests are needed.

1 open issue
What changed in this PR

Adds a discovery-driven ComputeClient so callers can run catalogued compute tasks without a dedicated task client.

Changes:

  • Adds dynamic task access, schema-shaped signatures, parameter forwarding, and preview opt-in.
  • Adds cache-only discovery access and tests for execution, signatures, and cache behavior.
  • Publishes API documentation and updates the kriging notebook’s filter example.
File Description
packages/​evo-compute/​tests/​test_engine.py Tests the generic compute client.
packages/​evo-compute/​tests/​test_discovery.py Tests cache-only reads, expiry, and copy isolation.
packages/​evo-compute/​tests/​data/​discovery-tasks.json Extends schema fixtures for signature tests.
packages/​evo-compute/​src/​evo/​compute/​engine.py Implements discovery-driven task execution.
packages/​evo-compute/​src/​evo/​compute/​discovery.py Adds peek_tasks() for non-fetching cache access.
packages/​evo-compute/​src/​evo/​compute/​__init__.py Exports ComputeClient.
packages/​evo-compute/​docs/​examples/​kriging_multiple.ipynb Updates filtering and removes an obsolete method call.
mkdocs/​site/​packages/​evo-compute/​index.html Links navigation to the new client page.
mkdocs/​site/​packages/​evo-compute/​DiscoveryClient.html Documents peek_tasks() and updates navigation.
mkdocs/​site/​packages/​evo-compute/​ComputeClient.html Adds the rendered client reference.
mkdocs/​docs/​packages/​evo-compute/​ComputeClient.md Adds the client documentation stub.
mkdocs/​api_clients.txt Registers the client for documentation generation.

Review effort: Balanced

ComputeClient exposes every task in the discovery catalogue as
client.<topic>.<task>.run(...), with arun() for names held as strings.
A run reads the task's schema to build a keyword-only signature, submits
only the parameters the caller passed, and waits for the results.
Preview stays opt-in.

DiscoveryClient gains peek_tasks(), so tab-completion and signature help
can read the cached catalogue without making a request.

Also fixes the filter example in the kriging_multiple notebook, which
still used the removed RegionFilter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants