Skip to content

Add cache-reuse planning and skipped-region execution to the scheduler #5884

Open

Description

@Xiao-zhen-Liu

Task Summary

Part of #5881 (design: #5880).

Plan cache reuse directly on the full physical plan inside CostBasedScheduleGenerator without changing Amber's four-layer pipeline (LogicalPlan -> PhysicalPlan -> RegionPlan -> Schedule). Package scheduling adds a 4-edge-state SearchState (Materialized, Pipelined, CacheRead, Unused), resolves input requirements and skipped upstream branches in one reverse-topological pass (resolveInputRequirements), prices retained sub-plans with saved-result reader URIs attached in Pass 2, and places skipped operators into disconnected skip regions (Region.skipped = true) so ScheduleGenerator puts them at schedule level 0. Follow-up execution PRs run level 0 skip regions without workers and persist saved row counts.

Issue #5884 cache-reuse system and execution architecture

Pull Requests

PR Scope Stack
#8731 Carry WorkflowContext.matchedResults; add Region.skipped, OutputPortConfig.cachedTupleCount, level 0 schedule placement, and region.getLinks scoping in Region.getSourceOperators. Base
#8732 Add EdgeType, SearchState (private[scheduling]), requireOutputPortsOfPlan, resolveInputRequirements, and scheduler-input helpers. Sits on #8731
#8733 Wire SearchState, Pass 2 saved-result URIs, isolated DefaultResourceAllocator(fullPlan), shared DefaultCostEstimator.withResourceAllocator, and buildRegionPlan into CostBasedScheduleGenerator. Sits on #8731, #8732
#8734 Scope FriesReconfigurationAlgorithm to region.getLinks so omitted CacheRead links stay outside reconfiguration scope. Independent
Still to open Fix worker-free run completion and execute Region.skipped = true at level 0 in RegionExecutionManager (closes #5884). After #8731 to #8733

Scope & Invariants

Planning (#8731 to #8733):

Mechanism Design rule
WorkflowContext.matchedResults Maps matched output ports to CachedResult(storageUri, tupleCount) on the computing unit (#5885); omitted from editor WorkflowSettings.
SearchState.resolveInputRequirements Enforces plan-port membership (requireOutputPortsOfPlan), keeps loop/warehouse checks on cache lookup, and resolves both skipped (Unused) and required (CacheRead or active) inputs in one reverse-topological pass.
Pass 2 & allocator isolation Attaches saved URIs in IntermediateInputPortConfig and isolates DefaultResourceAllocator(fullPlan) plus memoization per (retainedPlan, readerUrisByInputPort) while sharing DB runtime stats.
Extended RegionPlan & level 0 Appends Region.skipped = true components with IDs above active IDs; ScheduleGenerator rejects RegionLinks on skip regions and schedules them at level 0.

Running a skip region (still to open):

Aspect Planned behavior
Level 0 execution Starts zero workers, emits saved results as this run's results, marks operators COMPLETED, and starts level 1 in the same coordinator turn.
Cache flag & stats Sets reused_from_cache (#6729) only when all physical operators of a logical operator skip; output ports report cachedTupleCount and input ports report no counts.
Completion & cleanup Zero-worker runs emit COMPLETED once and run final result collection; execution deletion drops only result documents written by this run.

Safe default and eager-reuse scope. When matchedResults is empty, CostBasedScheduleGenerator uses the fast path with zero skip regions and produces the exact same schedule as main. Every matched result is reused (CacheRead stays fixed in the search queue); cost-based reuse choices will be added later inside package scheduling (SearchState and CostBasedScheduleGenerator) and CostEstimator, keeping Region, RegionPlan, ScheduleGenerator, and WorkflowScheduler intact.

Depends on: #6729 (#5883, merged). Nothing populates matchedResults until #5885.

Out of scope: Cost-based reuse choices; reading/writing cache metadata at lookup/save (#5885); UI reuse display (#5886).

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Activity

  1. changed the title [-][Cache PR 3] Scheduler: reuse planner, skeleton generator, and schedule assembly[/-] [+]Add cache-reuse planning to the scheduler (skeleton generation and schedule assembly)[/+] on Jun 24, 2026
  2. changed the issue type fromtoon Jun 24, 2026
  3. changed the title [-]Add cache-reuse planning to the scheduler (skeleton generation and schedule assembly)[/-] [+]Add cache-reuse planning and skipped-region execution to the scheduler[/+] on Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions