Skip to content

Wire cache lookup, result saving, REST endpoints, and cleanup into execution #5885

Open

Description

@Xiao-zhen-Liu

Task Summary

Part of #5881 (design: #5880).

Connect the pieces so the cache is used: look up matched ports at submission, save a result when an output port finishes, expose the cache endpoints, and remove entries when executions are deleted. After this task the feature works on the backend.

Scope

  • Submission-time matching: WorkflowExecutionService asks the cache service for the matched ports and stores them on WorkflowSettings.cachedOutputs, which the scheduler task (Add cache-reuse planning and skipped-region execution to the scheduler #5884) reads.
  • Saving on completion: when an output port finishes, PortCompletedHandler sends a new PortMaterialized event; ExecutionCacheService receives it and writes a cache entry. The engine sends an event and the service layer handles it, matching the existing pattern for statistics and worker updates, so the engine stays unaware of the web layer.
  • Two small websocket events so the UI can show cache entries and which ones the current run can use, plus the state stores that hold them; both default to empty.
  • Cache endpoints on WorkflowExecutionsResource: list entries, clear all, clear for a selected operator, and remove entries whose cache key no longer matches the current workflow. The result location is left out of the listing.
  • Cleanup: when executions are deleted, or a computing unit is torn down, the cache entries those executions produced are removed.

Behavior change to call out. To make reuse possible, this stops eagerly clearing the previous run's results when a new run starts. This affects re-runs whether or not the cache is used, so it must be described clearly.

Why the rest is safe with no matched ports. The lookup returns nothing when the table is empty, so cachedOutputs stays empty and the scheduler behaves as before. The saving path is wrapped so a write failure logs and does not fail the run. The endpoints are new routes, and the cleanup is a no-op with no rows.

Depends on. #5882 (cache table and service) and #5884 (the scheduler that reads cachedOutputs). Activates the feature.

Out of scope. The cache panel and canvas display (#5886). Any cost-based or eviction logic.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Activity

  1. changed the title [-][Cache PR 4] Turn the feature on: lookup, saving, endpoints, and cleanup[/-] [+]Wire cache lookup, result saving, REST endpoints, and cleanup into execution[/+] on Jun 24, 2026
  2. changed the issue type fromtoon Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions