From 445f08de9a831902153f465758821a9704975ab8 Mon Sep 17 00:00:00 2001 From: Preetam Dwivedi Date: Wed, 8 Jul 2026 10:14:03 -0700 Subject: [PATCH] feat(speculation): add path scorer extension MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the scorer seam from the speculation RFC, as a vendor-agnostic extension interface under submitqueue/extension/speculation/pathscorer/. The scorer computes each speculation path's predicted-success score from the current state: the per-batch scores of the path's base batches (entity.Batch.Score) and which of those dependencies have resolved (landed or build-passed), plus optionally other signals. It is a prediction over live state, so the controller re-runs it on every respeculate right after reconciling status, and persists the result; the scorer owns only the formula. The controller hands it the batch's speculation tree directly — the subject it scores. Any richer signal an implementation needs (dependency batch scores, historical pass rates) is injected at its Factory, not put in the signature. It never writes: its only output is per-path scores ([]entity.PathScore — path ID plus fresh score, an entity-level seam-output type alongside the path-decision type), which the controller merges into the tree and persists, staying the single writer of tree state; structure and status never pass through the scorer. Scores are probabilities in [0, 1] — the contract every implementation must satisfy, enforced by the controller on consume. This is the per-path scorer, distinct from the existing per-batch score stage (`extension/scorer`) that sets entity.Batch.Score — the path scorer consumes those to score whole paths. Follows the repo extension contract: Factory.For(Config) (Scorer, error) with Config carrying only QueueName. Includes README, gomock package, and a programmable fake. The speculation RFC's seam descriptions are updated to match the identity-keyed minimal-output contracts (and gain a design-decision entry for assigned path identity). Interface only; concrete impls and controller wiring are deferred. --- Makefile | 2 +- doc/rfc/submitqueue/speculation.md | 10 +- submitqueue/entity/speculation_tree.go | 13 +++ .../speculation/enumerator/enumerator.go | 2 +- .../speculation/pathscorer/BUILD.bazel | 9 ++ .../speculation/pathscorer/README.md | 19 ++++ .../speculation/pathscorer/fake/BUILD.bazel | 23 +++++ .../speculation/pathscorer/fake/fake.go | 72 ++++++++++++++ .../speculation/pathscorer/fake/fake_test.go | 51 ++++++++++ .../speculation/pathscorer/mock/BUILD.bazel | 13 +++ .../pathscorer/mock/pathscorer_mock.go | 97 +++++++++++++++++++ .../speculation/pathscorer/pathscorer.go | 76 +++++++++++++++ 12 files changed, 381 insertions(+), 6 deletions(-) create mode 100644 submitqueue/extension/speculation/pathscorer/BUILD.bazel create mode 100644 submitqueue/extension/speculation/pathscorer/README.md create mode 100644 submitqueue/extension/speculation/pathscorer/fake/BUILD.bazel create mode 100644 submitqueue/extension/speculation/pathscorer/fake/fake.go create mode 100644 submitqueue/extension/speculation/pathscorer/fake/fake_test.go create mode 100644 submitqueue/extension/speculation/pathscorer/mock/BUILD.bazel create mode 100644 submitqueue/extension/speculation/pathscorer/mock/pathscorer_mock.go create mode 100644 submitqueue/extension/speculation/pathscorer/pathscorer.go diff --git a/Makefile b/Makefile index cec663fa..2ee51d77 100644 --- a/Makefile +++ b/Makefile @@ -364,7 +364,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service mocks: ## Generate mock files using mockgen @echo "Generating mocks..." - @$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/... + @$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/enumerator/... ./submitqueue/extension/speculation/dependencylimit/... ./submitqueue/extension/speculation/pathscorer/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/... @echo "Mocks generated successfully!" proto: ## Generate protobuf files from .proto definitions diff --git a/doc/rfc/submitqueue/speculation.md b/doc/rfc/submitqueue/speculation.md index 78dd069e..97fad456 100644 --- a/doc/rfc/submitqueue/speculation.md +++ b/doc/rfc/submitqueue/speculation.md @@ -187,13 +187,13 @@ Re-speculation needs no special undo path: the controller refreshes statuses, an ## Interfaces -The seams are vendor-agnostic extensions, each in its own package; the exact Go signatures live in the source. All are per-queue: the system hands a `Factory` the queue identity, and the factory builds the seam for that queue — so the queue is bound at construction and never re-passed to a method. +The seams are vendor-agnostic extensions, each in its own package; the exact Go signatures live in the source. All are per-queue: the system hands a `Factory` the queue identity, and the factory builds the seam for that queue — so the queue is bound at construction and never re-passed to a method. Persisted paths carry a controller-assigned identity, and every seam output names a path by that ID rather than restating its Base/Head split — each seam returns only its verdict, and everything else about a path stays controller-owned. **Decision seams:** -- **Enumerator** (`extension/speculation/enumerator`) — given a batch ID and its ordered active dependencies, returns the batch's speculation tree *structure*: the candidate paths, each a Base/Head split. Pure and deterministic; sets no score and no status. -- **Scorer** (`extension/speculation/scorer`) — given the speculation tree and the current dependency batches, returns each path's predicted-success score. Called by the controller on every respeculate (during reconciliation) so scores track the live state — dependencies landing, dependency builds passing, siblings failing. Owns the score formula; combines the base batches' `Batch.Score` and their resolved/unresolved state (and optionally other signals). -- **Selector** (`extension/speculation/selector`) — given a speculation tree (with each path's controller-stamped status and freshly recomputed score), returns a per-path action (`Build` or `Cancel`) for the paths it chooses to act on. It reads status and score and emits actions only; paths it leaves alone are omitted. It is constructed with its **selection limit** and calls it to cap how many paths it builds in parallel. +- **Enumerator** (`extension/speculation/enumerator`) — given a batch and its ordered active dependencies, returns the batch's candidate paths, each a Base/Head split — structure only. Pure and deterministic; the controller assembles the persisted tree, assigning each path's identity and stamping its status. +- **Scorer** (`extension/speculation/pathscorer`) — given the speculation tree and the current dependency batches, returns each path's predicted-success score, keyed by path identity. Scores are probabilities in [0, 1]; the controller enforces the range when it merges them into the tree. Called by the controller on every respeculate (during reconciliation) so scores track the live state — dependencies landing, dependency builds passing, siblings failing. Owns the score formula; combines the base batches' `Batch.Score` and their resolved/unresolved state (and optionally other signals). +- **Selector** (`extension/speculation/selector`) — given a speculation tree (with each path's controller-stamped status and freshly recomputed score), returns a per-path action (`Build` or `Cancel`) for the paths it chooses to act on, each decision naming its path by identity, at most one decision per path. It reads status and score and emits actions only; paths it leaves alone are omitted. It is constructed with its **selection limit** and calls it to cap how many paths it builds in parallel. - **Prioritizer** (`extension/speculation/prioritizer`) — given the queue's pending build candidates, returns the subset admitted to run, ranked by score plus any fairness policy. It is constructed with its **prioritization limit** and applies it itself. Operates queue-wide, across all of the queue's in-flight batches. **Limit policies** — each a signal-driven "how much" seam returning a bound from build-resource and other signals: @@ -206,6 +206,8 @@ The scorer, prioritizer, and the three limit policies are design-level here — ## Design decisions +**Persisted paths are referenced by assigned identity.** Each path entry in a persisted tree gets a controller-assigned, immutable, opaque ID when it is first written; scores, decisions, and durable links (such as the path→build mapping) all name paths by it. *Why:* seam outputs stay minimal — an ID plus a verdict — instead of restating structure, and cross-entity links stay valid however the tree is re-derived. *Rejected:* structural reference (restating Base/Head in every output) — couples every consumer to path structure and forces ordered-slice comparison everywhere; an ID derived from the structure — an ID that encodes structure invites parsing, and identity should be free to survive re-enumeration on the controller's terms. + **Two layers: decisions and limits.** Decision seams (enumerator, scorer, selector, prioritizer) — enumeration and scoring *describe* the tree, selection and prioritization *act* on it; limit policies (dependency, selection, prioritization) decide *how much*. *Why:* the "which" is qualitative policy that is stable, while the "how much" must scale with volatile build resources; separating them lets the resource-aware knobs move independently of the decision logic, and lets each be tested in isolation. *Rejected:* baking counts into each decision seam as constants — it hard-codes a policy that needs to breathe with CI capacity. **Limits are signal-driven, and resources are the primary but not the only signal.** A limit is whatever its policy computes — from available capacity, and optionally historical pass rates, cost, time, or experiment flags. *Why:* speculation aggression should rise and fall with the build system, and the design should not foreclose other inputs. *Rejected:* a single fixed constant, or a static per-queue config value — neither can react to load. diff --git a/submitqueue/entity/speculation_tree.go b/submitqueue/entity/speculation_tree.go index 94bdfdec..8d35e9bb 100644 --- a/submitqueue/entity/speculation_tree.go +++ b/submitqueue/entity/speculation_tree.go @@ -130,6 +130,19 @@ type SpeculationPathInfo struct { BuildID string } +// PathScore is the path scorer's verdict for a single path: the +// path's identity and its freshly computed predicted-success score. It is the +// scorer seam's only output — the controller merges scores into the tree by +// path ID and persists them; tree structure and status never pass through the +// scorer. Like SpeculationPathDecision, it is ephemeral and never persisted. +type PathScore struct { + // PathID identifies the scored path (SpeculationPathInfo.ID) within the + // tree the scorer was handed. + PathID string + // Score is the path's predicted-success probability, in [0, 1]. + Score float32 +} + // SpeculationPathDecision is a seam's decision for a single path: the action the // controller should take for it. It is the output of both the selector (per // batch) and the prioritizer (queue-wide), and is not persisted. A seam returns diff --git a/submitqueue/extension/speculation/enumerator/enumerator.go b/submitqueue/extension/speculation/enumerator/enumerator.go index 64727145..50919a13 100644 --- a/submitqueue/extension/speculation/enumerator/enumerator.go +++ b/submitqueue/extension/speculation/enumerator/enumerator.go @@ -29,7 +29,7 @@ import ( // deliberately dumb and purely structural: it mechanically lists candidate // Base/Head paths from the dependency batches it is handed and nothing else. It // does not score paths — that is the scorer's job (see -// extension/speculation/scorer), which the controller re-runs on every +// extension/speculation/pathscorer), which the controller re-runs on every // respeculate — it does not decide which paths to build — that is the selector's // job (see extension/speculation/selector) — and it does not decide how far // back to speculate: the controller gates on the dependency limit and hands diff --git a/submitqueue/extension/speculation/pathscorer/BUILD.bazel b/submitqueue/extension/speculation/pathscorer/BUILD.bazel new file mode 100644 index 00000000..e3be76ed --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/BUILD.bazel @@ -0,0 +1,9 @@ +load("@rules_go//go:def.bzl", "go_library") + +go_library( + name = "go_default_library", + srcs = ["pathscorer.go"], + importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer", + visibility = ["//visibility:public"], + deps = ["//submitqueue/entity:go_default_library"], +) diff --git a/submitqueue/extension/speculation/pathscorer/README.md b/submitqueue/extension/speculation/pathscorer/README.md new file mode 100644 index 00000000..6258deea --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/README.md @@ -0,0 +1,19 @@ +# Speculation Path Scorer + +Vendor-agnostic interface for scoring the paths in a batch's **speculation tree** — the predicted-success probability of each candidate bet, recomputed as the batch's world changes. + +See the [Speculation RFC](/doc/rfc/submitqueue/speculation.md) for the end-to-end design and how scoring fits into the orchestrator pipeline. + +## Scorer + +A path's score is a **prediction**: *how likely is this bet to pay off, right now?* The scorer answers it from the current state — the per-batch success probabilities of a path's base batches (`entity.Batch.Score`, set by the score stage), which of those dependencies have already landed or had their build pass (resolved assumptions raise confidence), and optionally other signals such as how long the batch has waited or historical pass rates. The score is the common currency the [selector](../selector) and prioritizer both rank on, so keeping it current is what makes both act on the latest reality. + +Because it is a prediction over live state, the scorer is **re-run on every respeculate**, right after the controller reconciles path status — so when a dependency lands, its build passes, or a sibling path fails, the surviving paths' scores are recomputed before anything is selected or prioritized. The controller drives *when* to rescore (it is part of reconciliation) and persists the result; the scorer owns the *formula*. + +This is the per-**path** scorer, distinct from the per-**batch** [score stage](../../scorer), which sets `entity.Batch.Score`. The path scorer consumes those batch scores to score whole paths. The controller hands it the batch's **speculation tree** directly — the subject it scores — and any richer signal an implementation needs (the dependency batches' scores, historical pass rates) is injected at its factory, not passed in. It never writes: its only output is per-path scores, each naming a path by its ID, and the controller merges them into the tree and persists — the controller stays the single writer of tree state, and everything else about a path (structure, status) never passes through the scorer at all. Paths omitted from the result keep their last persisted score. + +Scores are **probabilities in [0, 1]** — 0 is a bet certain to lose, 1 a bet certain to pay off. That is the contract every implementation must satisfy, and the controller enforces the range when it consumes the result. The selector and prioritizer rank on these values, so implementations sharing a queue must agree on this scale. + +## Factory + +A per-queue factory returns the scorer for a queue, following the repo's extension contract. It is handed only the queue identity; scoring knobs and read access to any extra signals are injected at construction by the integrator in the wiring layer, which resolves per-queue settings through `queueconfig`. Scoring itself stays config-free. diff --git a/submitqueue/extension/speculation/pathscorer/fake/BUILD.bazel b/submitqueue/extension/speculation/pathscorer/fake/BUILD.bazel new file mode 100644 index 00000000..a3f1b933 --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/fake/BUILD.bazel @@ -0,0 +1,23 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "go_default_library", + srcs = ["fake.go"], + importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer/fake", + visibility = ["//visibility:public"], + deps = [ + "//submitqueue/entity:go_default_library", + "//submitqueue/extension/speculation/pathscorer:go_default_library", + ], +) + +go_test( + name = "go_default_test", + srcs = ["fake_test.go"], + embed = [":go_default_library"], + deps = [ + "//submitqueue/entity:go_default_library", + "@com_github_stretchr_testify//assert:go_default_library", + "@com_github_stretchr_testify//require:go_default_library", + ], +) diff --git a/submitqueue/extension/speculation/pathscorer/fake/fake.go b/submitqueue/extension/speculation/pathscorer/fake/fake.go new file mode 100644 index 00000000..aa3d5519 --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/fake/fake.go @@ -0,0 +1,72 @@ +// Copyright (c) 2025 Uber Technologies, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package fake provides a programmable pathscorer.Scorer for tests and +// examples. By default Score echoes each input path's current score back, +// keyed by its ID; Returns overrides that with canned path scores, and +// FailWith injects an error on every call. It is intended for examples and +// tests only, never production. +package fake + +import ( + "context" + + "github.com/uber/submitqueue/submitqueue/entity" + "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer" +) + +// Scorer is a programmable pathscorer.Scorer. +type Scorer struct { + scores []entity.PathScore + hasScores bool + err error +} + +// New returns a fake Scorer that echoes each input path's current score back. +// Override the returned scores with Returns. +func New() *Scorer { + return &Scorer{} +} + +// Returns makes every Score call return scores instead of echoing its input. +func (s *Scorer) Returns(scores []entity.PathScore) *Scorer { + s.scores = scores + s.hasScores = true + return s +} + +// FailWith makes every Score call return err. +func (s *Scorer) FailWith(err error) *Scorer { + s.err = err + return s +} + +// Score returns the canned scores if set with Returns, otherwise one PathScore +// per input path echoing the path's current score, unchanged. +func (s *Scorer) Score(_ context.Context, tree entity.SpeculationTree) ([]entity.PathScore, error) { + if s.err != nil { + return nil, s.err + } + if s.hasScores { + return s.scores, nil + } + scores := make([]entity.PathScore, 0, len(tree.Paths)) + for _, p := range tree.Paths { + scores = append(scores, entity.PathScore{PathID: p.ID, Score: p.Score}) + } + return scores, nil +} + +// ensure the fake satisfies the interface. +var _ pathscorer.Scorer = (*Scorer)(nil) diff --git a/submitqueue/extension/speculation/pathscorer/fake/fake_test.go b/submitqueue/extension/speculation/pathscorer/fake/fake_test.go new file mode 100644 index 00000000..09815b4c --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/fake/fake_test.go @@ -0,0 +1,51 @@ +// Copyright (c) 2025 Uber Technologies, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package fake + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "github.com/uber/submitqueue/submitqueue/entity" +) + +func TestScore_EchoesInputScoresByDefault(t *testing.T) { + tree := entity.SpeculationTree{ + BatchID: "q/batch/2", + Paths: []entity.SpeculationPathInfo{ + {ID: "q/batch/2/path/0", Path: entity.SpeculationPath{Head: "q/batch/2"}, Score: 0.9}, + }, + } + got, err := New().Score(context.Background(), tree) + require.NoError(t, err) + assert.Equal(t, []entity.PathScore{{PathID: "q/batch/2/path/0", Score: 0.9}}, got) +} + +func TestScore_ReturnsCanned(t *testing.T) { + in := entity.SpeculationTree{BatchID: "q/batch/2"} + scores := []entity.PathScore{{PathID: "q/batch/2/path/0", Score: 0.75}} + got, err := New().Returns(scores).Score(context.Background(), in) + require.NoError(t, err) + assert.Equal(t, scores, got) +} + +func TestScore_FailWith(t *testing.T) { + sentinel := errors.New("boom") + _, err := New().FailWith(sentinel).Score(context.Background(), entity.SpeculationTree{BatchID: "q/batch/1"}) + require.ErrorIs(t, err, sentinel) +} diff --git a/submitqueue/extension/speculation/pathscorer/mock/BUILD.bazel b/submitqueue/extension/speculation/pathscorer/mock/BUILD.bazel new file mode 100644 index 00000000..16082c77 --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/mock/BUILD.bazel @@ -0,0 +1,13 @@ +load("@rules_go//go:def.bzl", "go_library") + +go_library( + name = "go_default_library", + srcs = ["pathscorer_mock.go"], + importpath = "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer/mock", + visibility = ["//visibility:public"], + deps = [ + "//submitqueue/entity:go_default_library", + "//submitqueue/extension/speculation/pathscorer:go_default_library", + "@org_uber_go_mock//gomock:go_default_library", + ], +) diff --git a/submitqueue/extension/speculation/pathscorer/mock/pathscorer_mock.go b/submitqueue/extension/speculation/pathscorer/mock/pathscorer_mock.go new file mode 100644 index 00000000..a6997c7b --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/mock/pathscorer_mock.go @@ -0,0 +1,97 @@ +// Code generated by MockGen. DO NOT EDIT. +// Source: pathscorer.go +// +// Generated by this command: +// +// mockgen -source=pathscorer.go -destination=mock/pathscorer_mock.go -package=mock +// + +// Package mock is a generated GoMock package. +package mock + +import ( + context "context" + reflect "reflect" + + entity "github.com/uber/submitqueue/submitqueue/entity" + pathscorer "github.com/uber/submitqueue/submitqueue/extension/speculation/pathscorer" + gomock "go.uber.org/mock/gomock" +) + +// MockScorer is a mock of Scorer interface. +type MockScorer struct { + ctrl *gomock.Controller + recorder *MockScorerMockRecorder + isgomock struct{} +} + +// MockScorerMockRecorder is the mock recorder for MockScorer. +type MockScorerMockRecorder struct { + mock *MockScorer +} + +// NewMockScorer creates a new mock instance. +func NewMockScorer(ctrl *gomock.Controller) *MockScorer { + mock := &MockScorer{ctrl: ctrl} + mock.recorder = &MockScorerMockRecorder{mock} + return mock +} + +// EXPECT returns an object that allows the caller to indicate expected use. +func (m *MockScorer) EXPECT() *MockScorerMockRecorder { + return m.recorder +} + +// Score mocks base method. +func (m *MockScorer) Score(ctx context.Context, tree entity.SpeculationTree) ([]entity.PathScore, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "Score", ctx, tree) + ret0, _ := ret[0].([]entity.PathScore) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// Score indicates an expected call of Score. +func (mr *MockScorerMockRecorder) Score(ctx, tree any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "Score", reflect.TypeOf((*MockScorer)(nil).Score), ctx, tree) +} + +// MockFactory is a mock of Factory interface. +type MockFactory struct { + ctrl *gomock.Controller + recorder *MockFactoryMockRecorder + isgomock struct{} +} + +// MockFactoryMockRecorder is the mock recorder for MockFactory. +type MockFactoryMockRecorder struct { + mock *MockFactory +} + +// NewMockFactory creates a new mock instance. +func NewMockFactory(ctrl *gomock.Controller) *MockFactory { + mock := &MockFactory{ctrl: ctrl} + mock.recorder = &MockFactoryMockRecorder{mock} + return mock +} + +// EXPECT returns an object that allows the caller to indicate expected use. +func (m *MockFactory) EXPECT() *MockFactoryMockRecorder { + return m.recorder +} + +// For mocks base method. +func (m *MockFactory) For(cfg pathscorer.Config) (pathscorer.Scorer, error) { + m.ctrl.T.Helper() + ret := m.ctrl.Call(m, "For", cfg) + ret0, _ := ret[0].(pathscorer.Scorer) + ret1, _ := ret[1].(error) + return ret0, ret1 +} + +// For indicates an expected call of For. +func (mr *MockFactoryMockRecorder) For(cfg any) *gomock.Call { + mr.mock.ctrl.T.Helper() + return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "For", reflect.TypeOf((*MockFactory)(nil).For), cfg) +} diff --git a/submitqueue/extension/speculation/pathscorer/pathscorer.go b/submitqueue/extension/speculation/pathscorer/pathscorer.go new file mode 100644 index 00000000..641a2e3a --- /dev/null +++ b/submitqueue/extension/speculation/pathscorer/pathscorer.go @@ -0,0 +1,76 @@ +// Copyright (c) 2025 Uber Technologies, Inc. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package pathscorer + +//go:generate mockgen -source=pathscorer.go -destination=mock/pathscorer_mock.go -package=mock + +import ( + "context" + + "github.com/uber/submitqueue/submitqueue/entity" +) + +// Scorer computes the predicted-success score of every path in a batch's +// speculation tree. +// +// A path's score is a prediction — "how likely is this bet to pay off?" — and +// predictions must move as evidence arrives. The scorer answers "how good is +// each path right now" from the current state: the per-batch success +// probabilities of a path's base batches (entity.Batch.Score, set by the score +// stage), which of those dependencies have already landed or had their build +// pass (resolved assumptions raise confidence), and optionally other signals +// (how long the batch has waited, historical pass rates). +// +// The controller re-runs the scorer on every respeculate, right after it +// reconciles path status — so when a dependency lands, its build passes, or a +// sibling path fails, the surviving paths' scores are recomputed against the new +// reality before anything is selected or prioritized. The controller drives +// *when* to rescore and persists the result; the scorer owns the *formula*. +// +// This is the per-*path* scorer, distinct from the per-*batch* score stage +// (extension/scorer), which sets entity.Batch.Score. The path scorer consumes +// those batch scores to score whole paths. +// +// The controller hands the scorer the batch's speculation tree directly — the +// subject it scores. Any richer signal an implementation needs (the dependency +// batches' scores, historical pass rates) is injected at its Factory, not passed +// here. It never writes: it returns per-path scores and the controller merges +// them into the tree and persists, keeping the controller the single writer of +// tree state. +type Scorer interface { + // Score returns the freshly computed predicted-success score of each path in + // the tree as entity.PathScore values, keyed by path ID + // (entity.SpeculationPathInfo.ID). Paths omitted from the result keep their + // last persisted score. The combination formula is the implementation's + // concern; the returned values must be probabilities in [0, 1] — the + // controller enforces the range when it consumes them. + Score(ctx context.Context, tree entity.SpeculationTree) ([]entity.PathScore, error) +} + +// Config carries the per-queue identity handed to a Factory. The system knows +// only the queue name; everything an implementation needs (scoring knobs, and +// read access to any extra signals such as the dependency batches' scores) is +// injected at construction by the integrator. +type Config struct { + // QueueName identifies the queue this Scorer serves. + QueueName string +} + +// Factory builds the Scorer for a queue. Implementations are provided by +// integrators (and tests) and inject whatever they need at construction. +type Factory interface { + // For returns the Scorer for the given queue. + For(cfg Config) (Scorer, error) +}