Repository navigation
Description
Task Summary
Part of #5881 (design: #5880).
Plan cache reuse directly on the full physical plan inside CostBasedScheduleGenerator without changing Amber's four-layer pipeline (LogicalPlan -> PhysicalPlan -> RegionPlan -> Schedule). Package scheduling adds a 4-edge-state SearchState (Materialized, Pipelined, CacheRead, Unused), resolves input requirements and skipped upstream branches in one reverse-topological pass (resolveInputRequirements), prices retained sub-plans with saved-result reader URIs attached in Pass 2, and places skipped operators into disconnected skip regions (Region.skipped = true) so ScheduleGenerator puts them at schedule level 0. Follow-up execution PRs run level 0 skip regions without workers and persist saved row counts.
Pull Requests
| PR | Scope | Stack |
|---|---|---|
| #8731 | Carry WorkflowContext.matchedResults; add Region.skipped, OutputPortConfig.cachedTupleCount, level 0 schedule placement, and region.getLinks scoping in Region.getSourceOperators. |
Base |
| #8732 | Add EdgeType, SearchState (private[scheduling]), requireOutputPortsOfPlan, resolveInputRequirements, and scheduler-input helpers. |
Sits on #8731 |
| #8733 | Wire SearchState, Pass 2 saved-result URIs, isolated DefaultResourceAllocator(fullPlan), shared DefaultCostEstimator.withResourceAllocator, and buildRegionPlan into CostBasedScheduleGenerator. |
Sits on #8731, #8732 |
| #8734 | Scope FriesReconfigurationAlgorithm to region.getLinks so omitted CacheRead links stay outside reconfiguration scope. |
Independent |
| Still to open | Fix worker-free run completion and execute Region.skipped = true at level 0 in RegionExecutionManager (closes #5884). |
After #8731 to #8733 |
Scope & Invariants
Planning (#8731 to #8733):
| Mechanism | Design rule |
|---|---|
WorkflowContext.matchedResults |
Maps matched output ports to CachedResult(storageUri, tupleCount) on the computing unit (#5885); omitted from editor WorkflowSettings. |
SearchState.resolveInputRequirements |
Enforces plan-port membership (requireOutputPortsOfPlan), keeps loop/warehouse checks on cache lookup, and resolves both skipped (Unused) and required (CacheRead or active) inputs in one reverse-topological pass. |
| Pass 2 & allocator isolation | Attaches saved URIs in IntermediateInputPortConfig and isolates DefaultResourceAllocator(fullPlan) plus memoization per (retainedPlan, readerUrisByInputPort) while sharing DB runtime stats. |
Extended RegionPlan & level 0 |
Appends Region.skipped = true components with IDs above active IDs; ScheduleGenerator rejects RegionLinks on skip regions and schedules them at level 0. |
Running a skip region (still to open):
| Aspect | Planned behavior |
|---|---|
Level 0 execution |
Starts zero workers, emits saved results as this run's results, marks operators COMPLETED, and starts level 1 in the same coordinator turn. |
| Cache flag & stats | Sets reused_from_cache (#6729) only when all physical operators of a logical operator skip; output ports report cachedTupleCount and input ports report no counts. |
| Completion & cleanup | Zero-worker runs emit COMPLETED once and run final result collection; execution deletion drops only result documents written by this run. |
Safe default and eager-reuse scope. When matchedResults is empty, CostBasedScheduleGenerator uses the fast path with zero skip regions and produces the exact same schedule as main. Every matched result is reused (CacheRead stays fixed in the search queue); cost-based reuse choices will be added later inside package scheduling (SearchState and CostBasedScheduleGenerator) and CostEstimator, keeping Region, RegionPlan, ScheduleGenerator, and WorkflowScheduler intact.
Depends on: #6729 (#5883, merged). Nothing populates matchedResults until #5885.
Out of scope: Cost-based reuse choices; reading/writing cache metadata at lookup/save (#5885); UI reuse display (#5886).
Task Type
- Refactor / Cleanup
- DevOps / Deployment / CI
- Testing / QA
- Documentation
- Performance
- Other

Activity
[-][Cache PR 3] Scheduler: reuse planner, skeleton generator, and schedule assembly[/-][+]Add cache-reuse planning to the scheduler (skeleton generation and schedule assembly)[/+]feat(amber): carry cache-reuse status as a metrics flag
[-]Add cache-reuse planning to the scheduler (skeleton generation and schedule assembly)[/-][+]Add cache-reuse planning and skipped-region execution to the scheduler[/+]feat(amber): carry cache-reuse status as a metrics flag
feat(amber): carry cache-reuse status as a metrics flag (apache#6729)