From 6d5cdcbed2a7c7ace3f31d54fd23b20284c3bc55 Mon Sep 17 00:00:00 2001 From: Dante Date: Wed, 14 Oct 2026 14:43:47 +0800 Subject: [PATCH 01/51] MUL-7488: fix(execenv): exclude committed local HEAD changes from resume replay (#8549) * fix(execenv): exclude clean HEAD advances from resume replay * fix(execenv): normalize carried WIP before replaying local edits Co-authored-by: multica-agent * fix(execenv): resume after conflicting local draft commits Co-authored-by: multica-agent * fix(execenv): limit snapshot conflict handling to affected paths Co-authored-by: multica-agent --------- Co-authored-by: Sol-Boy Co-authored-by: multica-agent --- apps/docs/content/docs/project-resources.mdx | 1 + .../internal/daemon/execenv/local_worktree.go | 171 +++++++- .../daemon/execenv/local_worktree_test.go | 406 ++++++++++++++++++ 3 files changed, 575 insertions(+), 3 deletions(-) diff --git a/apps/docs/content/docs/project-resources.mdx b/apps/docs/content/docs/project-resources.mdx index 16b43d695f9..06540af3d86 100644 --- a/apps/docs/content/docs/project-resources.mdx +++ b/apps/docs/content/docs/project-resources.mdx @@ -81,6 +81,7 @@ What the agent sees, and what you get back: - **The agent starts from what you see, not from `HEAD`.** Your uncommitted edits and your untracked files are replayed into the worktree, so the agent is not reviewing code you no longer have. Your own working copy, index, and stash list are never touched. If that state cannot be reproduced faithfully — including when there is more untracked content than the replay covers (2000 files / 200 MiB) — the run fails instead of starting from a tree you would not recognise. The usual fix is to gitignore or clean up build output that is not already ignored. - **The result is a branch in your repo**, one per issue rather than one per run: `agent//` (a chat gets `agent//chat-`). The run's **Run details** panel shows the branch name (and lets you copy it); you can also find it with `git branch`. Review it with `git log` / `git diff`, and merge or cherry-pick it yourself. Multica never merges it for you. A run that failed partway still reports its branch, because it still committed whatever the agent had produced. - **A follow-up continues where the last run stopped.** The next comment on the same issue checks that branch out again, so the agent is standing on the work it just delivered instead of starting from `HEAD` with its own changes stranded on another branch. Only what you changed in your own directory since the last run is replayed on top — the branch already carries the rest. Merge the branch and the next run starts from your `HEAD` again. If two runs on one issue overlap, the second one forks from the first's work onto `agent//-`, because git allows one worktree per branch. +- **Committed updates to your local `HEAD` are kept separate from uncommitted edits.** A pull or local commit between turns is not replayed onto the issue branch; merge those commits into it explicitly when needed. If the previous run captured uncommitted work, comparing it on the new `HEAD` requires Git 2.40 or later. If those older edits conflict with the new committed baseline, the runtime logs a warning and uses the current `HEAD` only for the affected paths; other drafts keep their incremental comparison. Finishing and committing one draft therefore does not block future runs or re-propose your unchanged drafts against the agent's work. Any conflict between your current edits and the agent's work is still offered to the agent to resolve. - **If your edits collide with the agent's, the agent resolves them.** When you rewrite the same lines the agent rewrote, git cannot decide which version wins, so the run starts on a conflicted worktree and is told to finish the merge before anything else — `git status` in that worktree lists the unmerged paths. A run that leaves any file unmerged delivers nothing: it fails, keeps the worktree, and the next run offers the same edits again, so your change is never quietly dropped. - **Only branches Multica owns are continued.** Continuing a branch is decided by an ownership record, not by its name: the record names the conversation, and it names the commit the branch was left at. A branch you created yourself that happens to be called `agent//` is never checked out, appended to, or read as a previous run — and neither is one that no longer contains the recorded commit, so deleting Multica's branch and making your own under that name, or force-moving it onto other history, is safe. In those cases the run takes `agent//-` instead. Committing on top of a delivered branch is the normal case and keeps it continuing — your commit is simply there in the next run. Every branch also starts with a baseline commit of its own, marking the tree the first run began from — that commit is what identifies the branch later, and `git diff ..` is exactly the agent's work. A run that could not record its branch — it failed before finishing, the record could not be written, or it reset its worktree past the commit that carried your edits into the branch — reports failure and keeps its worktree, and later runs start a new branch rather than continuing one they cannot prove is theirs. - **Nothing is silently dropped.** Anything the agent leaves uncommitted is committed to that branch before the worktree is removed, including when the run fails partway. In the rare case that commit itself cannot be made — for example a repo with `commit.gpgSign` and no signing key available to the runtime — the worktree is deliberately kept instead of removed, the run is reported as failed, and `git worktree list` in your repo points at the directory holding the work. diff --git a/server/internal/daemon/execenv/local_worktree.go b/server/internal/daemon/execenv/local_worktree.go index 440bee6c4ef..773a0265d6e 100644 --- a/server/internal/daemon/execenv/local_worktree.go +++ b/server/internal/daemon/execenv/local_worktree.go @@ -1329,6 +1329,11 @@ type replayResult struct { // itself, so the whole snapshot applies and cannot conflict. For a continued // branch it is the snapshot that branch recorded, which is what makes this a // replay of the user's LAST-TURN-TO-NOW edits rather than of their whole tree. +// If the local HEAD changed, first transplant the previous uncommitted edits +// onto the new HEAD. Comparing snapshots on that shared baseline excludes +// committed updates without re-proposing edits the branch already carries. +// If that transplant conflicts, use current HEAD for the affected paths while +// retaining the successfully normalized edits in all other paths. // // Replaying the whole tree onto a continued branch is the tempting version and // it is wrong: that merge takes the user's HEAD as its base, so it re-proposes @@ -1346,7 +1351,11 @@ type replayResult struct { func replayUserState(worktreePath string, plan taskBranchPlan, snapshot string, logger *slog.Logger) (replayResult, error) { carried := plan.base if plan.continues { - carried = plan.priorState + var err error + carried, err = normalizeReplayBaseline(worktreePath, plan.priorState, snapshot, logger) + if err != nil { + return replayResult{}, err + } } if carried == "" { return replayResult{}, fmt.Errorf("execenv: no baseline to replay the local directory against for branch %s", plan.name) @@ -1404,6 +1413,161 @@ func replayUserState(worktreePath string, plan taskBranchPlan, snapshot string, return replayResult{conflicts: conflicts}, nil } +// normalizeReplayBaseline puts the previously carried WIP on the current local +// HEAD before calculating its incremental change. Writes are limited to Git +// objects and disposable scratch files; neither live checkout/index nor the task +// branch or its record is moved. +func normalizeReplayBaseline(worktreePath, priorState, snapshot string, logger *slog.Logger) (string, error) { + // A record has two parents; its first is a snapshot or, after a conflict + // resolved without a commit, another record. Follow only that parent until + // the first one-parent commit: the actual snapshot, parented at local HEAD. + // Limit the result to that snapshot, not the repository's entire history. + previous, err := runGitTrimmed(worktreePath, "rev-list", "--first-parent", "--parents", + "--min-parents=1", "--max-parents=1", "--max-count=1", priorState) + if err != nil { + return "", fmt.Errorf("execenv: could not read the previous local snapshot: %w", err) + } + fields := strings.Fields(previous) + if len(fields) != 2 { + return "", errors.New("execenv: the previous local snapshot has no captured HEAD") + } + previousSnapshot, previousHead := fields[0], fields[1] + currentHead, err := runGitTrimmed(worktreePath, "rev-parse", "--verify", snapshot+"^1") + if err != nil { + return "", fmt.Errorf("execenv: could not read the current local snapshot's HEAD: %w", err) + } + if previousHead == currentHead { + return priorState, nil + } + // With no previous WIP, transplanting it is the identity operation. This + // also avoids requiring merge-tree when a clean local directory advances. + if _, err := runGit(worktreePath, "diff", "--quiet", previousHead, previousSnapshot); err == nil { + return currentHead, nil + } + + // Use the captured HEAD as an explicit base. A record's second parent is + // the agent checkpoint, which may already contain newer mainline commits; + // using Git's inferred merge base could then subtract the wrong changes. + merged, err := runGitStdout(worktreePath, "merge-tree", "--write-tree", "--name-only", "--messages", "-z", + "--merge-base="+previousHead, currentHead, previousSnapshot) + tree, _, hasTree := strings.Cut(merged, "\x00") + if err != nil { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) && exitErr.ExitCode() == 1 { + paths, pathsErr := mergeTreeConflictPaths(merged) + if pathsErr != nil { + return "", fmt.Errorf("execenv: could not read the conflicted local snapshot: %w", pathsErr) + } + // A single conflicting draft must not re-propose every untouched + // draft against the agent's edits. Preserve the merged tree and + // replace only the conflicting paths with their current HEAD state. + if len(paths) == 0 { + // Git can report a tree-level conflict without affected paths. + // Retain the resumable current-WIP behavior in that case. + if logger != nil { + logger.Warn("execenv: local snapshot merge reported a conflict without paths; replaying current uncommitted edits", + "path", worktreePath, "previous_head", previousHead, "current_head", currentHead) + } + return currentHead, nil + } + tree, err = replaceConflictedSnapshotPaths(worktreePath, tree, currentHead, paths) + if err != nil { + return "", err + } + if logger != nil { + logger.Warn("execenv: previous local edits conflict with the committed HEAD update; replaying current edits only for the affected paths", + "path", worktreePath, "previous_head", previousHead, "current_head", currentHead, "files", quotedPaths(paths)) + } + } else { + return "", fmt.Errorf("execenv: could not normalize previous uncommitted edits onto the current local HEAD "+ + "(this operation requires Git 2.40 or later); the task branch and its previous snapshot are preserved: %w", err) + } + } + if !hasTree || tree == "" { + return "", errors.New("execenv: the normalized local snapshot has no tree") + } + // The replay commit needs a commit parent, not just a tree. This object is + // only a comparison baseline: its committed HEAD changes never enter the + // task branch, which cherry-picks only baseline-to-current-snapshot edits. + args := append(commitIdentityArgs(worktreePath), "commit-tree", tree, "-p", currentHead, + "-m", "multica: previous local edits on the current HEAD") + baseline, err := runGitTrimmed(worktreePath, args...) + if err != nil { + return "", fmt.Errorf("execenv: could not describe the normalized local snapshot: %w", err) + } + return baseline, nil +} + +// mergeTreeConflictPaths reads merge-tree's NUL-delimited file and message +// sections. Conflict messages also name paths for directory-rename conflicts +// that have no unmerged file entries. Never parse the human-readable message. +func mergeTreeConflictPaths(output string) ([]string, error) { + fields := strings.Split(output, "\x00") + if len(fields) < 3 || fields[0] == "" || fields[len(fields)-1] != "" { + return nil, errors.New("invalid merge-tree output") + } + var paths []string + seen := make(map[string]bool) + add := func(name string) { + if name != "" && !seen[name] { + paths = append(paths, name) + seen[name] = true + } + } + i := 1 + for i < len(fields) && fields[i] != "" { + add(fields[i]) + i++ + } + i++ // End of the unmerged-file section. + for i < len(fields)-1 { + count, err := strconv.Atoi(fields[i]) + i++ + if err != nil || count < 0 || count > len(fields)-i-3 { + return nil, errors.New("invalid merge-tree conflict message") + } + if strings.HasPrefix(fields[i+count], "CONFLICT") { + for _, name := range fields[i : i+count] { + add(name) + } + } + i += count + 2 // Affected paths, conflict type, and message. + } + return paths, nil +} + +// replaceConflictedSnapshotPaths edits only a private index beside the task +// worktree. Resetting explicit literal paths handles additions, deletions, +// renames and file/directory conflicts without touching either live index. +func replaceConflictedSnapshotPaths(worktreePath, tree, currentHead string, paths []string) (string, error) { + if len(paths) == 0 { + return "", errors.New("execenv: no conflicted snapshot paths to replace") + } + dir, err := os.MkdirTemp(filepath.Dir(worktreePath), ".multica-normalize-") + if err != nil { + return "", fmt.Errorf("execenv: create private normalization index: %w", err) + } + defer os.RemoveAll(dir) + env := []string{"GIT_INDEX_FILE=" + filepath.Join(dir, "index"), "GIT_LITERAL_PATHSPECS=1"} + if out, err := runGitEnv(worktreePath, env, "read-tree", tree); err != nil { + return "", fmt.Errorf("execenv: seed private normalization index: %s: %w", strings.TrimSpace(out), err) + } + pathFile := filepath.Join(dir, "paths") + if err := os.WriteFile(pathFile, []byte(strings.Join(paths, "\x00")+"\x00"), 0o600); err != nil { + return "", fmt.Errorf("execenv: write conflicted snapshot paths: %w", err) + } + // The nonempty pathspec file selects reset's index-only form. Using a file + // avoids command-length limits and preserves whitespace in repository names. + if out, err := runGitEnv(worktreePath, env, "reset", currentHead, "--pathspec-from-file="+pathFile, "--pathspec-file-nul"); err != nil { + return "", fmt.Errorf("execenv: replace conflicted snapshot paths: %s: %w", strings.TrimSpace(out), err) + } + normalized, err := runGitTrimmedEnv(worktreePath, env, "write-tree") + if err != nil { + return "", fmt.Errorf("execenv: write normalized local snapshot: %w", err) + } + return normalized, nil +} + // quotedPaths renders repository paths for a human-facing message. Quoted // because a git path may contain newlines, quotes or control characters, and // this string is read in logs and task errors where a raw one would look like @@ -1704,7 +1868,8 @@ func runGitTrimmedEnv(dir string, extraEnv []string, args ...string) (string, er // runGitStdout is runGitTrimmed without the trimming, for output where // whitespace is significant — NUL-separated file listings, where a leading or -// trailing space is part of a filename. +// trailing space is part of a filename. Retain partial stdout on error for +// plumbing like merge-tree, which emits a useful tree on a conflict. func runGitStdout(dir string, args ...string) (string, error) { ctx, cancel := context.WithTimeout(context.Background(), gitTimeout) defer cancel() @@ -1714,7 +1879,7 @@ func runGitStdout(dir string, args ...string) (string, error) { cmd.WaitDelay = 5 * time.Second out, err := cmd.Output() if err != nil { - return "", withGitStderr(err) + return string(out), withGitStderr(err) } return string(out), nil } diff --git a/server/internal/daemon/execenv/local_worktree_test.go b/server/internal/daemon/execenv/local_worktree_test.go index 6543ddf1d38..8bf81b38c92 100644 --- a/server/internal/daemon/execenv/local_worktree_test.go +++ b/server/internal/daemon/execenv/local_worktree_test.go @@ -888,6 +888,412 @@ func TestPrepareLocalWorktreeContinuesTheConversationBranch(t *testing.T) { } } +// A clean local HEAD advancing during a turn is committed work, not a new +// user edit to replay onto the conversation branch (#8544). +func TestPrepareLocalWorktreeDoesNotReplayCleanHEADAdvance(t *testing.T) { + for _, dirty := range []bool{false, true} { + t.Run(fmt.Sprintf("new_local_edit_%t", dirty), func(t *testing.T) { + repo := newTestRepo(t) + first := prepareTurn(t, repo, "MUL-6881", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "turn-one.txt"), "agent work\n") + writeFile(t, filepath.Join(repo, "upstream.txt"), "committed upstream work\n") + gitRun(t, repo, "add", "upstream.txt") + gitRun(t, repo, "commit", "-m", "advance local HEAD") + finalizeOK(t, first) + firstTip := gitRun(t, repo, "rev-parse", first.Branch) + if dirty { + writeFile(t, filepath.Join(repo, "local-edit.txt"), "new uncommitted work\n") + } + + second := prepareTurn(t, repo, "MUL-6881", turnTwoTask) + if !second.Continued || second.Branch != first.Branch { + t.Fatalf("expected to continue %s, got %s (continued=%v)", first.Branch, second.Branch, second.Continued) + } + if _, err := os.Stat(filepath.Join(second.WorkDir, "upstream.txt")); !os.IsNotExist(err) { + t.Fatalf("committed local HEAD advance was replayed: %v", err) + } + if got := readFile(t, filepath.Join(second.WorkDir, "turn-one.txt")); got != "agent work\n" { + t.Fatalf("agent work changed: %q", got) + } + if dirty { + if got := readFile(t, filepath.Join(second.WorkDir, "local-edit.txt")); got != "new uncommitted work\n" { + t.Fatalf("new local edit missing: %q", got) + } + } else if second.BaseCommit != firstTip { + t.Fatalf("clean resume authored a commit: %s != %s", second.BaseCommit, firstTip) + } + finalizeOK(t, second) + secondTip := gitRun(t, repo, "rev-parse", second.Branch) + third := prepareTurn(t, repo, "MUL-6881", turnThreeTask) + if third.BaseCommit != secondTip { + t.Fatalf("unchanged third turn replayed local state again: %s != %s", third.BaseCommit, secondTip) + } + finalizeOK(t, third) + }) + } +} + +func TestPrepareLocalWorktreeExcludesHEADAdvanceWithExistingUserEdits(t *testing.T) { + for _, kind := range []string{"untracked", "tracked", "deleted"} { + for _, newEdits := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/new_edits_%t", kind, newEdits), func(t *testing.T) { + repo := newTestRepo(t) + switch kind { + case "untracked": + writeFile(t, filepath.Join(repo, "draft.txt"), "user draft\n") + case "tracked": + writeFile(t, filepath.Join(repo, "tracked.txt"), "user WIP\n") + case "deleted": + if err := os.Remove(filepath.Join(repo, "tracked.txt")); err != nil { + t.Fatal(err) + } + } + writeFile(t, filepath.Join(repo, "notes.txt"), "old user notes\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "task.txt"), "agent work\n") + if kind == "untracked" { + writeFile(t, filepath.Join(first.WorkDir, "draft.txt"), "draft improved by the agent\n") + } else if kind == "tracked" { + writeFile(t, filepath.Join(first.WorkDir, "tracked.txt"), "WIP improved by the agent\n") + } + finalizeOK(t, first) + firstTip := gitRun(t, repo, "rev-parse", first.Branch) + + // Advancing HEAD commits only upstream work, leaving all WIP alone. + writeFile(t, filepath.Join(repo, "upstream.txt"), "committed upstream work\n") + gitRun(t, repo, "add", "upstream.txt") + gitRun(t, repo, "commit", "-m", "advance local HEAD") + if newEdits { + writeFile(t, filepath.Join(repo, "keep.txt"), "new tracked user edit\n") + writeFile(t, filepath.Join(repo, "scratch.txt"), "new untracked user edit\n") + if err := os.Remove(filepath.Join(repo, "notes.txt")); err != nil { + t.Fatal(err) + } + } + sourceStatus := gitRun(t, repo, "status", "--porcelain") + sourceIndex := gitRun(t, repo, "ls-files", "--stage") + second := prepareTurn(t, repo, "MUL-7488", turnTwoTask) + if !second.Continued || second.Branch != first.Branch || len(second.ReplayConflicts) != 0 { + t.Fatalf("unexpected continuation: %s, continued=%v, conflicts=%v", second.Branch, second.Continued, second.ReplayConflicts) + } + if _, err := os.Stat(filepath.Join(second.WorkDir, "upstream.txt")); !os.IsNotExist(err) { + t.Fatalf("committed HEAD advance was replayed: %v", err) + } + if kind == "untracked" { + if got := readFile(t, filepath.Join(second.WorkDir, "draft.txt")); got != "draft improved by the agent\n" { + t.Fatalf("agent's draft was overwritten: %q", got) + } + } else if kind == "tracked" { + if got := readFile(t, filepath.Join(second.WorkDir, "tracked.txt")); got != "WIP improved by the agent\n" { + t.Fatalf("agent's tracked edit was overwritten: %q", got) + } + } else if _, err := os.Stat(filepath.Join(second.WorkDir, "tracked.txt")); !os.IsNotExist(err) { + t.Fatalf("user's deletion was lost: %v", err) + } + if newEdits { + for name, want := range map[string]string{"keep.txt": "new tracked user edit\n", "scratch.txt": "new untracked user edit\n"} { + if got := readFile(t, filepath.Join(second.WorkDir, name)); got != want { + t.Fatalf("new user edit missing in %s: %q", name, got) + } + } + if _, err := os.Stat(filepath.Join(second.WorkDir, "notes.txt")); !os.IsNotExist(err) { + t.Fatalf("new untracked deletion was lost: %v", err) + } + } else if second.BaseCommit != firstTip { + t.Fatalf("unchanged WIP authored a replay commit: %s != %s", second.BaseCommit, firstTip) + } + if got := gitRun(t, repo, "status", "--porcelain"); got != sourceStatus { + t.Fatalf("source status changed: %q != %q", got, sourceStatus) + } + if got := gitRun(t, repo, "ls-files", "--stage"); got != sourceIndex { + t.Fatal("source index changed during prepare") + } + finalizeOK(t, second) + secondTip := gitRun(t, repo, "rev-parse", second.Branch) + third := prepareTurn(t, repo, "MUL-7488", turnThreeTask) + if third.BaseCommit != secondTip || len(third.ReplayConflicts) != 0 { + t.Fatalf("unchanged third turn replayed state again: %s, conflicts=%v", third.BaseCommit, third.ReplayConflicts) + } + finalizeOK(t, third) + }) + } + } +} + +func TestPrepareLocalWorktreeNormalizesEditsAcrossChangesInTheSameFile(t *testing.T) { + repo := newTestRepo(t) + base := "one\ntwo\nthree\nfour\nfive\nsix\nseven\neight\n" + writeFile(t, filepath.Join(repo, "tracked.txt"), base) + gitRun(t, repo, "add", "tracked.txt") + gitRun(t, repo, "commit", "-m", "multi-line base") + writeFile(t, filepath.Join(repo, "tracked.txt"), strings.Replace(base, "one", "user one", 1)) + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "tracked.txt"), strings.Replace(base, "one", "agent one", 1)) + finalizeOK(t, first) + firstTip := gitRun(t, repo, "rev-parse", first.Branch) + + // Pulling a change to another line and restoring the same user WIP is not + // a new uncommitted edit; neither line should be replayed into the task. + upstream := strings.Replace(base, "eight", "upstream eight", 1) + writeFile(t, filepath.Join(repo, "tracked.txt"), upstream) + gitRun(t, repo, "add", "tracked.txt") + gitRun(t, repo, "commit", "-m", "upstream edit to another line") + writeFile(t, filepath.Join(repo, "tracked.txt"), strings.Replace(upstream, "one", "user one", 1)) + second := prepareTurn(t, repo, "MUL-7488", turnTwoTask) + if second.BaseCommit != firstTip || len(second.ReplayConflicts) != 0 { + t.Fatalf("unchanged WIP caused replay: %s, conflicts=%v", second.BaseCommit, second.ReplayConflicts) + } + if got := readFile(t, filepath.Join(second.WorkDir, "tracked.txt")); got != strings.Replace(base, "one", "agent one", 1) { + t.Fatalf("committed source edit entered the task or agent edit was lost: %q", got) + } + finalizeOK(t, second) +} + +func TestPrepareLocalWorktreeExcludesHEADAdvanceAfterUncommittedConflictResolution(t *testing.T) { + repo := newTestRepo(t) + writeFile(t, filepath.Join(repo, "tracked.txt"), "user A\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "tracked.txt"), "agent B\n") + finalizeOK(t, first) + writeFile(t, filepath.Join(repo, "tracked.txt"), "user C\n") + for _, taskID := range []string{turnTwoTask, turnThreeTask} { + pending := prepareTurn(t, repo, "MUL-7488", taskID) + if len(pending.ReplayConflicts) != 1 { + t.Fatalf("expected a genuine user conflict, got %v", pending.ReplayConflicts) + } + gitRun(t, pending.Path, "reset", "--hard", pending.BaseCommit) + finalizeOK(t, pending) + } + record := gitRun(t, repo, "rev-parse", userStateRef(first.Branch)) + if parents := strings.Fields(gitRun(t, repo, "rev-list", "--parents", "-n", "1", record+"^1")); len(parents) != 3 { + t.Fatalf("expected a nested two-parent record, got %v", parents) + } + writeFile(t, filepath.Join(repo, "upstream.txt"), "committed upstream work\n") + gitRun(t, repo, "add", "upstream.txt") + gitRun(t, repo, "commit", "-m", "advance local HEAD") + resumed := prepareTurn(t, repo, "MUL-7488", "11112222-3333-4444-5555-dddddddddddd") + if len(resumed.ReplayConflicts) != 1 || resumed.ReplayConflicts[0] != "tracked.txt" { + t.Fatalf("pending user conflict was lost: %v", resumed.ReplayConflicts) + } + if _, err := os.Stat(filepath.Join(resumed.WorkDir, "upstream.txt")); !os.IsNotExist(err) { + t.Fatalf("HEAD advance entered the task through a nested record: %v", err) + } + writeFile(t, filepath.Join(resumed.WorkDir, "tracked.txt"), "agent B and user C\n") + gitRun(t, resumed.Path, "add", "tracked.txt") + finalizeOK(t, resumed) + resolvedTip := gitRun(t, repo, "rev-parse", resumed.Branch) + next := prepareTurn(t, repo, "MUL-7488", "11112222-3333-4444-5555-eeeeeeeeeeee") + if next.BaseCommit != resolvedTip || len(next.ReplayConflicts) != 0 { + t.Fatalf("resolved user edit was replayed: %s, conflicts=%v", next.BaseCommit, next.ReplayConflicts) + } + finalizeOK(t, next) +} + +func TestPrepareLocalWorktreeContinuesAfterUserCommitsConflictingDraft(t *testing.T) { + for _, kind := range []string{"untracked", "tracked"} { + for _, newEdits := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/new_edits_%t", kind, newEdits), func(t *testing.T) { + repo := newTestRepo(t) + name := "draft.txt" + if kind == "tracked" { + name = "tracked.txt" + } + writeFile(t, filepath.Join(repo, name), "previous user draft\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, name), "draft improved by the agent\n") + writeFile(t, filepath.Join(first.WorkDir, "task.txt"), "agent work\n") + finalizeOK(t, first) + record := gitRun(t, repo, "rev-parse", userStateRef(first.Branch)) + tip := gitRun(t, repo, "rev-parse", first.Branch) + + // The finished, committed draft conflicts with the old snapshot. + // Its committed changes must not enter the task or block resumes. + writeFile(t, filepath.Join(repo, name), "finished draft committed by the user\n") + gitRun(t, repo, "add", name) + gitRun(t, repo, "commit", "-m", "finish and commit the draft") + if newEdits { + writeFile(t, filepath.Join(repo, "scratch.txt"), "current uncommitted user edit\n") + } + sourceStatus := gitRun(t, repo, "status", "--porcelain") + sourceIndex := gitRun(t, repo, "ls-files", "--stage") + for _, taskID := range []string{turnTwoTask, turnThreeTask} { + resumed, err := prepareTurnAs(t, repo, "MUL-7488", taskID, testBranchOwner) + if err != nil { + t.Errorf("resume %s failed after the draft was committed: %v", taskID, err) + continue + } + if !resumed.Continued || resumed.Branch != first.Branch || len(resumed.ReplayConflicts) != 0 { + t.Fatalf("unexpected continuation: %s, continued=%v, conflicts=%v", resumed.Branch, resumed.Continued, resumed.ReplayConflicts) + } + if (!newEdits || taskID == turnThreeTask) && resumed.BaseCommit != tip { + t.Fatalf("unchanged user WIP authored a commit on resume %s", taskID) + } + if got := readFile(t, filepath.Join(resumed.WorkDir, name)); got != "draft improved by the agent\n" { + t.Fatalf("committed draft overwrote the agent's work: %q", got) + } + if got := readFile(t, filepath.Join(resumed.WorkDir, "task.txt")); got != "agent work\n" { + t.Fatalf("earlier agent work was lost: %q", got) + } + if newEdits { + if got := readFile(t, filepath.Join(resumed.WorkDir, "scratch.txt")); got != "current uncommitted user edit\n" { + t.Fatalf("current user edit was lost: %q", got) + } + } + finalizeOK(t, resumed) + tip = gitRun(t, repo, "rev-parse", first.Branch) + if got := gitRun(t, repo, "rev-parse", userStateRef(first.Branch)); got == record { + t.Fatal("successful resume did not advance the captured user state") + } + if got := gitRun(t, repo, "status", "--porcelain"); got != sourceStatus { + t.Fatalf("resume changed the user's checkout status: %q", got) + } + if got := gitRun(t, repo, "ls-files", "--stage"); got != sourceIndex { + t.Fatal("resume changed the user's index") + } + if got := readFile(t, filepath.Join(repo, name)); got != "finished draft committed by the user\n" { + t.Fatalf("resume changed the user's committed draft: %q", got) + } + } + }) + } + } +} + +func TestPrepareLocalWorktreeHandsCurrentEditsToAgentAfterHEADNormalizationConflict(t *testing.T) { + repo := newTestRepo(t) + writeFile(t, filepath.Join(repo, "tracked.txt"), "previous user draft\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "tracked.txt"), "draft improved by the agent\n") + finalizeOK(t, first) + writeFile(t, filepath.Join(repo, "tracked.txt"), "finished draft committed by the user\n") + gitRun(t, repo, "add", "tracked.txt") + gitRun(t, repo, "commit", "-m", "finish and commit the draft") + writeFile(t, filepath.Join(repo, "tracked.txt"), "current uncommitted user edit\n") + + second := prepareTurn(t, repo, "MUL-7488", turnTwoTask) + if len(second.ReplayConflicts) != 1 || second.ReplayConflicts[0] != "tracked.txt" { + t.Fatalf("current user edit was not offered as a conflict: %v", second.ReplayConflicts) + } + conflict := readFile(t, filepath.Join(second.WorkDir, "tracked.txt")) + for _, want := range []string{"draft improved by the agent", "current uncommitted user edit", "<<<<<<<"} { + if !strings.Contains(conflict, want) { + t.Fatalf("conflicted file is missing %q: %s", want, conflict) + } + } + writeFile(t, filepath.Join(second.WorkDir, "tracked.txt"), "agent draft and current user edit\n") + gitRun(t, second.Path, "add", "tracked.txt") + finalizeOK(t, second) + resolvedTip := gitRun(t, repo, "rev-parse", second.Branch) + third := prepareTurn(t, repo, "MUL-7488", turnThreeTask) + if third.BaseCommit != resolvedTip || len(third.ReplayConflicts) != 0 { + t.Fatalf("resolved edit was replayed again: %s, conflicts=%v", third.BaseCommit, third.ReplayConflicts) + } + if got := readFile(t, filepath.Join(third.WorkDir, "tracked.txt")); got != "agent draft and current user edit\n" { + t.Fatalf("resolved draft was lost: %q", got) + } + finalizeOK(t, third) +} + +func TestPrepareLocalWorktreePreservesOtherDraftsWhenNormalizationConflicts(t *testing.T) { + for _, committedChange := range []string{"finish", "delete", "directory"} { + for _, newEdit := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/new_edit_%t", committedChange, newEdit), func(t *testing.T) { + repo := newTestRepo(t) + // A tracked draft allows both content and modify/delete conflicts. + writeFile(t, filepath.Join(repo, "notes.txt"), "original notes\n") + gitRun(t, repo, "add", "notes.txt") + gitRun(t, repo, "commit", "-m", "initial notes") + writeFile(t, filepath.Join(repo, "notes.txt"), "old user notes\n") + writeFile(t, filepath.Join(repo, "scratch.txt"), "old user scratch\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, "scratch.txt"), "scratch improved by the agent\n") + finalizeOK(t, first) + tip := gitRun(t, repo, "rev-parse", first.Branch) + + switch committedChange { + case "finish": + writeFile(t, filepath.Join(repo, "notes.txt"), "finished committed notes\n") + case "delete", "directory": + if err := os.Remove(filepath.Join(repo, "notes.txt")); err != nil { + t.Fatal(err) + } + if committedChange == "directory" { + writeFile(t, filepath.Join(repo, "notes.txt", "finished.txt"), "finished committed notes\n") + } + } + gitRun(t, repo, "add", "-A", "--", "notes.txt") + gitRun(t, repo, "commit", "-m", "commit the notes change") + if newEdit { + writeFile(t, filepath.Join(repo, "keep.txt"), "new tracked user edit\n") + } + sourceHead := gitRun(t, repo, "rev-parse", "HEAD") + sourceStatus := gitRun(t, repo, "status", "--porcelain") + sourceIndex := gitRun(t, repo, "ls-files", "--stage") + for _, taskID := range []string{turnTwoTask, turnThreeTask} { + resumed := prepareTurn(t, repo, "MUL-7488", taskID) + if len(resumed.ReplayConflicts) != 0 { + t.Fatalf("an unchanged draft was replayed as a conflict: %v", resumed.ReplayConflicts) + } + if (!newEdit || taskID == turnThreeTask) && resumed.BaseCommit != tip { + t.Fatal("unchanged drafts caused an extra replay commit") + } + if got := readFile(t, filepath.Join(resumed.WorkDir, "scratch.txt")); got != "scratch improved by the agent\n" { + t.Fatalf("unchanged user scratch overwrote the agent's work: %q", got) + } + if got := readFile(t, filepath.Join(resumed.WorkDir, "notes.txt")); got != "old user notes\n" { + t.Fatalf("committed notes change entered the task branch: %q", got) + } + if newEdit { + if got := readFile(t, filepath.Join(resumed.WorkDir, "keep.txt")); got != "new tracked user edit\n" { + t.Fatalf("new user edit was lost: %q", got) + } + } + finalizeOK(t, resumed) + tip = gitRun(t, repo, "rev-parse", first.Branch) + if got := gitRun(t, repo, "rev-parse", "HEAD"); got != sourceHead { + t.Fatal("normalization moved the user's HEAD") + } + if got := gitRun(t, repo, "status", "--porcelain"); got != sourceStatus { + t.Fatalf("normalization changed source status: %q", got) + } + if got := gitRun(t, repo, "ls-files", "--stage"); got != sourceIndex { + t.Fatal("normalization changed the user's index") + } + } + }) + } + } +} + +func TestPrepareLocalWorktreeNormalizesConflictPathsLiterally(t *testing.T) { + repo := newTestRepo(t) + // Brackets are valid filenames on every platform but are also Git glob + // syntax. Restoring the first path must not reset the second draft's WIP. + conflicting := "notes [1].txt" + unchanged := "notes 1.txt" + writeFile(t, filepath.Join(repo, conflicting), "old user notes\n") + writeFile(t, filepath.Join(repo, unchanged), "old user scratch\n") + first := prepareTurn(t, repo, "MUL-7488", turnOneTask) + writeFile(t, filepath.Join(first.WorkDir, unchanged), "scratch improved by the agent\n") + finalizeOK(t, first) + tip := gitRun(t, repo, "rev-parse", first.Branch) + writeFile(t, filepath.Join(repo, conflicting), "finished committed notes\n") + gitRun(t, repo, "add", "--", ":(literal)"+conflicting) + gitRun(t, repo, "commit", "-m", "finish notes with a literal name") + + second := prepareTurn(t, repo, "MUL-7488", turnTwoTask) + if len(second.ReplayConflicts) != 0 || second.BaseCommit != tip { + t.Fatalf("unrelated matching filename was replayed: %v, base=%s", second.ReplayConflicts, second.BaseCommit) + } + if got := readFile(t, filepath.Join(second.WorkDir, unchanged)); got != "scratch improved by the agent\n" { + t.Fatalf("unrelated matching draft was overwritten: %q", got) + } + if got := readFile(t, filepath.Join(second.WorkDir, conflicting)); got != "old user notes\n" { + t.Fatalf("committed notes were replayed: %q", got) + } + finalizeOK(t, second) +} + // The user's uncommitted work reaches the branch once, as turn one's baseline, // and every later turn replays only what they changed since. Replaying their // whole tree again would ask git to merge their copy of a file against the From 52b26900c51f759b87707b4ea35fa9f141b83b52 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=B0=8F=E5=A6=8D=E5=84=BF=20=E2=9C=A8?= <258399167+xiaoyaner0201@users.noreply.github.com> Date: Wed, 14 Oct 2026 14:55:37 +0800 Subject: [PATCH 02/51] MUL-8012: fix(inbox): restrict notifications to current workspace recipients (#9242) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(inbox): restrict notifications to current workspace recipients Removing a member or leaving a workspace deletes their subscriptions, but several notification sources keep addressing the user by a retained ID: reactions to their issues and comments, retained assignments, explicit mentions, squad mentions through surviving squad-member rows, quick-create outcomes and children_done for a retained parent assignee. CreateInboxItem inserted those rows unconditionally and the inbox:new event was then sent to the user's own connections, so a former member could keep receiving new issue titles and comment content. Guard the shared insert instead of each source: a member recipient must still be a member of the item's workspace and an agent recipient must belong to it. An ineligible recipient yields no row (pgx.ErrNoRows), which every caller now treats as a quiet skip without publishing inbox:new or failing the underlying operation. The check runs in the statement's snapshot and takes no row locks; historical inbox rows are left untouched. Existing delivery tests created recipients without membership rows; their fixtures now add the membership the production path requires. * test(inbox): assert children_done settles receipts when the recipient left * refactor(inbox): centralize eligible notification writes Co-authored-by: multica-agent --------- Co-authored-by: 千乘妍 (Xiaoyaner) Co-authored-by: Sol-Boy Co-authored-by: multica-agent --- .../cmd/server/autopilot_failure_monitor.go | 7 +- server/cmd/server/inbox_recipient_test.go | 74 +++++ server/cmd/server/notification_listeners.go | 16 +- .../cmd/server/notification_listeners_test.go | 49 +-- .../server/notification_membership_test.go | 292 ++++++++++++++++++ server/internal/inbox/create.go | 23 ++ server/internal/service/autopilot.go | 6 +- .../service/autopilot_quota_notifications.go | 6 +- .../internal/service/issue_wakeup_system.go | 9 +- server/internal/service/task.go | 14 +- server/pkg/db/generated/inbox.sql.go | 10 +- server/pkg/db/queries/inbox.sql | 10 +- 12 files changed, 482 insertions(+), 34 deletions(-) create mode 100644 server/cmd/server/inbox_recipient_test.go create mode 100644 server/cmd/server/notification_membership_test.go create mode 100644 server/internal/inbox/create.go diff --git a/server/cmd/server/autopilot_failure_monitor.go b/server/cmd/server/autopilot_failure_monitor.go index 8e4883aded9..158b3d1ad8b 100644 --- a/server/cmd/server/autopilot_failure_monitor.go +++ b/server/cmd/server/autopilot_failure_monitor.go @@ -14,6 +14,7 @@ import ( "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgtype" "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/service" "github.com/multica-ai/multica/server/internal/util" db "github.com/multica-ai/multica/server/pkg/db/generated" @@ -245,7 +246,7 @@ func emitAutopilotPausedNotifications( } emitted[key] = true - item, err := queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: autopilot.WorkspaceID, RecipientType: r.Type, @@ -269,6 +270,10 @@ func emitAutopilotPausedNotifications( continue } + if !inserted { + continue + } + bus.Publish(events.Event{ Type: protocol.EventInboxNew, WorkspaceID: workspaceID, diff --git a/server/cmd/server/inbox_recipient_test.go b/server/cmd/server/inbox_recipient_test.go new file mode 100644 index 00000000000..53d7dcce29d --- /dev/null +++ b/server/cmd/server/inbox_recipient_test.go @@ -0,0 +1,74 @@ +package main + +import ( + "context" + "errors" + "testing" + + "github.com/jackc/pgx/v5" + "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/testutil" + "github.com/multica-ai/multica/server/internal/util" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/dbid" + "github.com/multica-ai/multica/server/pkg/protocol" +) + +func TestCreateInboxItem_RecipientWorkspace(t *testing.T) { + fx := workspaceFixture(t) + otherWorkspace := fx.Workspace(t, "Other workspace", "inbox-recipient-other") + member := fx.User(t, "Member", "inbox-query-member@example.test") + fx.Member(t, testWorkspaceID, member, "member") + otherMember := fx.User(t, "Other member", "inbox-query-other@example.test") + fx.Member(t, otherWorkspace, otherMember, "member") + agent := fx.Agent(t, "Current agent", "") + otherAgent := fx.Agent(t, "Other agent", "", testutil.Cols{"workspace_id": otherWorkspace}) + deletedAgent := fx.Agent(t, "Deleted agent", "") + fx.Exec(t, `DELETE FROM agent WHERE id = $1`, deletedAgent) + issue := fx.Issue(t, "Recipient guard") + fx.Cleanup(t, `DELETE FROM inbox_item WHERE issue_id = $1`, issue) + queries := db.New(testPool) + for _, tc := range []struct { + name, recipientType, recipientID string + allowed bool + }{ + {"current member", "member", member, true}, + {"other workspace member", "member", otherMember, false}, + {"current agent", "agent", agent, true}, + {"other workspace agent", "agent", otherAgent, false}, + {"deleted agent", "agent", deletedAgent, false}, + {"member ID as agent", "agent", member, false}, + {"agent ID as member", "member", agent, false}, + } { + t.Run(tc.name, func(t *testing.T) { + params := db.CreateInboxItemParams{ID: dbid.NewV7(), WorkspaceID: util.MustParseUUID(testWorkspaceID), RecipientType: tc.recipientType, RecipientID: util.MustParseUUID(tc.recipientID), IssueID: util.MustParseUUID(issue), Type: "reaction_added", Severity: "info", Title: "Recipient guard", Details: emptyDetails} + item, err := queries.CreateInboxItem(context.Background(), params) + if tc.allowed { + if err != nil { + t.Fatal(err) + } + if item.ID != params.ID || item.RecipientType != tc.recipientType || item.RecipientID != params.RecipientID { + t.Fatalf("wrong inserted item: %+v", item) + } + fx.Exec(t, `DELETE FROM inbox_item WHERE id = $1`, item.ID) + } else if !errors.Is(err, pgx.ErrNoRows) { + t.Fatalf("CreateInboxItem error = %v, want pgx.ErrNoRows", err) + } + bus := events.New() + count := 0 + bus.Subscribe(protocol.EventInboxNew, func(events.Event) { count++ }) + notifyDirect(context.Background(), queries, bus, tc.recipientType, tc.recipientID, testWorkspaceID, + events.Event{ActorType: "member", ActorID: testUserID}, issue, "todo", "reaction_added", "info", "Recipient guard", "", emptyDetails) + want := 0 + if tc.allowed { + want = 1 + } + if count != want { + t.Errorf("inbox:new events = %d, want %d", count, want) + } + if n := fx.Count(t, `SELECT count(*) FROM inbox_item WHERE issue_id = $1 AND recipient_type = $2 AND recipient_id = $3`, issue, tc.recipientType, tc.recipientID); n != want { + t.Errorf("inbox rows = %d, want %d", n, want) + } + }) + } +} diff --git a/server/cmd/server/notification_listeners.go b/server/cmd/server/notification_listeners.go index 0946b4be6ad..e5fcccee68f 100644 --- a/server/cmd/server/notification_listeners.go +++ b/server/cmd/server/notification_listeners.go @@ -8,6 +8,7 @@ import ( "github.com/jackc/pgx/v5/pgtype" "github.com/multica-ai/multica/server/internal/events" "github.com/multica-ai/multica/server/internal/handler" + "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/issuestatus" "github.com/multica-ai/multica/server/internal/util" db "github.com/multica-ai/multica/server/pkg/db/generated" @@ -447,7 +448,7 @@ func notifyIssueSubscribers( continue } - item, err := queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: parseUUID(workspaceID), RecipientType: "member", @@ -466,6 +467,9 @@ func notifyIssueSubscribers( "subscriber_id", subID, "type", notifType, "error", err) continue } + if !inserted { + continue + } notified[subID] = true resp := inboxItemToResponse(item) @@ -513,7 +517,7 @@ func notifyDirect( } } - item, err := queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: parseUUID(workspaceID), RecipientType: recipientType, @@ -532,6 +536,9 @@ func notifyDirect( "issue_id", issueID, "recipient_id", recipientID, "type", notifType, "error", err) return } + if !inserted { + return + } resp := inboxItemToResponse(item) resp["issue_status"] = issueStatus @@ -631,7 +638,7 @@ func notifyMentionedMembers( if p, ok := mentionPrefs[id]; ok && isNotifMuted(p, "mentioned") { continue } - item, err := queries.CreateInboxItem(context.Background(), db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(context.Background(), queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: parseUUID(e.WorkspaceID), RecipientType: "member", @@ -648,6 +655,9 @@ func notifyMentionedMembers( slog.Error("mention inbox creation failed", "mentioned_id", id, "error", err) continue } + if !inserted { + continue + } resp := inboxItemToResponse(item) resp["issue_status"] = issueStatus bus.Publish(events.Event{ diff --git a/server/cmd/server/notification_listeners_test.go b/server/cmd/server/notification_listeners_test.go index 6de35127cc5..465048c12b3 100644 --- a/server/cmd/server/notification_listeners_test.go +++ b/server/cmd/server/notification_listeners_test.go @@ -14,6 +14,15 @@ import ( // notificationTest helpers — reuse the integration test fixtures from TestMain // (testPool, testUserID, testWorkspaceID are set in integration_test.go). +// createNotificationMember gives delivery controls real workspace membership. +// A user account alone is not an eligible notification recipient. +func createNotificationMember(t *testing.T, email string) string { + t.Helper() + id := createTestUser(t, email) + workspaceFixture(t).Member(t, testWorkspaceID, id, "member") + return id +} + // inboxItemsForRecipient returns all non-archived inbox items for a given recipient. func inboxItemsForRecipient(t *testing.T, queries *db.Queries, recipientID string) []db.ListInboxItemsRow { t.Helper() @@ -84,7 +93,7 @@ func TestNotification_IssueCreated_AssigneeNotified(t *testing.T) { bus := newNotificationBus(t, queries) assigneeEmail := "notif-assignee-created@multica.ai" - assigneeID := createTestUser(t, assigneeEmail) + assigneeID := createNotificationMember(t, assigneeEmail) t.Cleanup(func() { cleanupTestUser(t, assigneeEmail) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -244,11 +253,11 @@ func TestNotification_StatusChanged(t *testing.T) { // Create two extra users as subscribers sub1Email := "notif-sub1-status@multica.ai" - sub1ID := createTestUser(t, sub1Email) + sub1ID := createNotificationMember(t, sub1Email) t.Cleanup(func() { cleanupTestUser(t, sub1Email) }) sub2Email := "notif-sub2-status@multica.ai" - sub2ID := createTestUser(t, sub2Email) + sub2ID := createNotificationMember(t, sub2Email) t.Cleanup(func() { cleanupTestUser(t, sub2Email) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -323,11 +332,11 @@ func TestNotification_StatusChanged_Muted(t *testing.T) { bus := newNotificationBus(t, queries) mutedEmail := "notif-muted-status@multica.ai" - mutedID := createTestUser(t, mutedEmail) + mutedID := createNotificationMember(t, mutedEmail) t.Cleanup(func() { cleanupTestUser(t, mutedEmail) }) normalEmail := "notif-normal-status@multica.ai" - normalID := createTestUser(t, normalEmail) + normalID := createNotificationMember(t, normalEmail) t.Cleanup(func() { cleanupTestUser(t, normalEmail) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -382,11 +391,11 @@ func TestNotification_CommentCreated(t *testing.T) { bus := newNotificationBus(t, queries) commenterEmail := "notif-commenter@multica.ai" - commenterID := createTestUser(t, commenterEmail) + commenterID := createNotificationMember(t, commenterEmail) t.Cleanup(func() { cleanupTestUser(t, commenterEmail) }) sub1Email := "notif-sub1-comment@multica.ai" - sub1ID := createTestUser(t, sub1Email) + sub1ID := createNotificationMember(t, sub1Email) t.Cleanup(func() { cleanupTestUser(t, sub1Email) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -462,13 +471,13 @@ func TestNotification_SystemCommentSkipsInboxAndMentions(t *testing.T) { // Subscriber on the issue who would normally receive new_comment. subEmail := "notif-system-comment-sub@multica.ai" - subID := createTestUser(t, subEmail) + subID := createNotificationMember(t, subEmail) t.Cleanup(func() { cleanupTestUser(t, subEmail) }) // A second member whose UUID we will smuggle into the system-comment // body as a fake mention to prove the listener does not parse it. targetEmail := "notif-system-comment-target@multica.ai" - targetID := createTestUser(t, targetEmail) + targetID := createNotificationMember(t, targetEmail) t.Cleanup(func() { cleanupTestUser(t, targetEmail) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -556,15 +565,15 @@ func TestNotification_AssigneeChanged(t *testing.T) { bus := newNotificationBus(t, queries) oldAssigneeEmail := "notif-old-assignee@multica.ai" - oldAssigneeID := createTestUser(t, oldAssigneeEmail) + oldAssigneeID := createNotificationMember(t, oldAssigneeEmail) t.Cleanup(func() { cleanupTestUser(t, oldAssigneeEmail) }) newAssigneeEmail := "notif-new-assignee@multica.ai" - newAssigneeID := createTestUser(t, newAssigneeEmail) + newAssigneeID := createNotificationMember(t, newAssigneeEmail) t.Cleanup(func() { cleanupTestUser(t, newAssigneeEmail) }) bystanderEmail := "notif-bystander@multica.ai" - bystanderID := createTestUser(t, bystanderEmail) + bystanderID := createNotificationMember(t, bystanderEmail) t.Cleanup(func() { cleanupTestUser(t, bystanderEmail) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -735,7 +744,7 @@ func TestNotification_PriorityChanged(t *testing.T) { bus := newNotificationBus(t, queries) sub1Email := "notif-sub1-priority@multica.ai" - sub1ID := createTestUser(t, sub1Email) + sub1ID := createNotificationMember(t, sub1Email) t.Cleanup(func() { cleanupTestUser(t, sub1Email) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -800,7 +809,7 @@ func TestNotification_DueDateChanged(t *testing.T) { bus := newNotificationBus(t, queries) sub1Email := "notif-sub1-duedate@multica.ai" - sub1ID := createTestUser(t, sub1Email) + sub1ID := createNotificationMember(t, sub1Email) t.Cleanup(func() { cleanupTestUser(t, sub1Email) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -861,7 +870,7 @@ func TestNotification_StartDateChanged(t *testing.T) { bus := newNotificationBus(t, queries) sub1Email := "notif-sub1-startdate@multica.ai" - sub1ID := createTestUser(t, sub1Email) + sub1ID := createNotificationMember(t, sub1Email) t.Cleanup(func() { cleanupTestUser(t, sub1Email) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) @@ -921,7 +930,7 @@ func TestNotification_ParentBubble_StatusChanged(t *testing.T) { bus := newNotificationBus(t, queries) parentSubEmail := "notif-parent-sub-status@multica.ai" - parentSubID := createTestUser(t, parentSubEmail) + parentSubID := createNotificationMember(t, parentSubEmail) t.Cleanup(func() { cleanupTestUser(t, parentSubEmail) }) parentID := createTestIssue(t, testWorkspaceID, testUserID) @@ -983,11 +992,11 @@ func TestNotification_ParentBubble_NewCommentSuppressed(t *testing.T) { bus := newNotificationBus(t, queries) commenterEmail := "notif-parent-bubble-commenter@multica.ai" - commenterID := createTestUser(t, commenterEmail) + commenterID := createNotificationMember(t, commenterEmail) t.Cleanup(func() { cleanupTestUser(t, commenterEmail) }) parentSubEmail := "notif-parent-sub-comment@multica.ai" - parentSubID := createTestUser(t, parentSubEmail) + parentSubID := createNotificationMember(t, parentSubEmail) t.Cleanup(func() { cleanupTestUser(t, parentSubEmail) }) parentID := createTestIssue(t, testWorkspaceID, testUserID) @@ -1035,7 +1044,7 @@ func TestNotification_ParentBubble_PriorityChangeSuppressed(t *testing.T) { bus := newNotificationBus(t, queries) parentSubEmail := "notif-parent-sub-priority@multica.ai" - parentSubID := createTestUser(t, parentSubEmail) + parentSubID := createNotificationMember(t, parentSubEmail) t.Cleanup(func() { cleanupTestUser(t, parentSubEmail) }) parentID := createTestIssue(t, testWorkspaceID, testUserID) @@ -1141,7 +1150,7 @@ func TestNotification_StatusChange_ArchivesStaleTaskFailed(t *testing.T) { bus := newNotificationBus(t, queries) subEmail := "notif-archive-task-failed-sub@multica.ai" - subID := createTestUser(t, subEmail) + subID := createNotificationMember(t, subEmail) t.Cleanup(func() { cleanupTestUser(t, subEmail) }) issueID := createTestIssue(t, testWorkspaceID, testUserID) diff --git a/server/cmd/server/notification_membership_test.go b/server/cmd/server/notification_membership_test.go new file mode 100644 index 00000000000..9c602b5da2d --- /dev/null +++ b/server/cmd/server/notification_membership_test.go @@ -0,0 +1,292 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + + "github.com/go-chi/chi/v5" + "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/handler" + "github.com/multica-ai/multica/server/internal/service" + "github.com/multica-ai/multica/server/internal/testutil" + "github.com/multica-ai/multica/server/internal/util" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/protocol" +) + +// Use the real removal handlers, including their transactional subscription +// cleanup, rather than deleting only the member row in the regression fixture. +func removeNotificationMember(t *testing.T, userID, memberID, method string) { + t.Helper() + h := &handler.Handler{Queries: db.New(testPool), TxStarter: testPool, Bus: events.New()} + r := httptest.NewRequest(http.MethodDelete, "/", nil) + actor := testUserID + if method == "leave" { + actor = userID + } + r.Header.Set("X-User-ID", actor) + ctx := r.Context() + route := chi.NewRouteContext() + route.URLParams.Add("id", testWorkspaceID) + route.URLParams.Add("memberId", memberID) + r = r.WithContext(context.WithValue(ctx, chi.RouteCtxKey, route)) + if method == "leave" { + testutil.Call(t, h.LeaveWorkspace, r).Want(http.StatusNoContent) + } else { + testutil.Call(t, h.DeleteMember, r).Want(http.StatusNoContent) + } + fx := workspaceFixture(t) + if n := fx.Count(t, `SELECT count(*) FROM member WHERE workspace_id = $1 AND user_id = $2`, testWorkspaceID, userID); n != 0 { + t.Fatalf("removal left %d member rows", n) + } +} + +func TestNotification_CurrentMembership(t *testing.T) { + for _, source := range []string{ + "issue reaction", "comment reaction", "created assignee", "updated assignee", "previous assignee", + "description mention created", "description mention updated", "comment mention", + "description squad created", "description squad updated", "comment squad", + "all", "subscriber comment", "subscriber status", "subscriber task failure", "parent subscriber", + } { + for _, state := range []string{"current", "delete", "leave"} { + removed := state != "current" + t.Run(source+"/"+state, func(t *testing.T) { + fx := workspaceFixture(t) + recipient := fx.User(t, "Recipient", "membership-recipient@example.test") + memberID := fx.Member(t, testWorkspaceID, recipient, "member") + issueID := fx.Issue(t, "Membership regression", testutil.Cols{"creator_id": recipient}) + commentID := fx.Comment(t, issueID, "Original comment", testutil.Cols{"author_id": recipient}) + fx.Cleanup(t, `DELETE FROM inbox_item WHERE issue_id = $1`, issueID) + fx.Cleanup(t, `DELETE FROM issue_subscriber WHERE issue_id = $1`, issueID) + memberType := "member" + issue := handler.IssueResponse{ID: issueID, WorkspaceID: testWorkspaceID, Title: "Membership regression", Status: "todo", CreatorType: "member", CreatorID: recipient} + e := events.Event{WorkspaceID: testWorkspaceID, ActorType: "member", ActorID: testUserID} + payload := map[string]any{"issue_id": issueID, "issue_title": issue.Title, "issue_status": issue.Status} + expectedType := "mentioned" + content := "[@Recipient](mention://member/" + recipient + ")" + if source == "description squad created" || source == "description squad updated" || source == "comment squad" { + agent := fx.Agent(t, "Leader", "") + squad := fx.Squad(t, "Squad", agent) + fx.SquadMember(t, squad, "member", recipient) + content = "[@Squad](mention://squad/" + squad + ")" + } + switch source { + case "issue reaction": + e.Type = protocol.EventIssueReactionAdded + payload["reaction"] = handler.IssueReactionResponse{Emoji: "👍"} + payload["creator_type"], payload["creator_id"] = "member", recipient + expectedType = "reaction_added" + case "comment reaction": + e.Type = protocol.EventReactionAdded + payload["reaction"] = handler.ReactionResponse{Emoji: "👍"} + payload["comment_author_type"], payload["comment_author_id"], payload["comment_id"] = "member", recipient, commentID + expectedType = "reaction_added" + case "created assignee", "updated assignee", "previous assignee": + e.Type = protocol.EventIssueCreated + issue.AssigneeType, issue.AssigneeID = &memberType, &recipient + expectedType = "issue_assigned" + if source != "created assignee" { + e.Type = protocol.EventIssueUpdated + payload["assignee_changed"] = true + } + if source == "previous assignee" { + issue.AssigneeType, issue.AssigneeID = nil, nil + payload["prev_assignee_type"], payload["prev_assignee_id"] = &memberType, &recipient + expectedType = "unassigned" + } + payload["issue"] = issue + case "description mention created", "description squad created", "description mention updated", "description squad updated": + e.Type = protocol.EventIssueCreated + if source == "description mention updated" || source == "description squad updated" { + e.Type = protocol.EventIssueUpdated + payload["description_changed"] = true + } + issue.Description = &content + payload["issue"] = issue + case "comment mention", "comment squad", "all", "subscriber comment": + e.Type = protocol.EventCommentCreated + if source == "all" { + content = "[@All](mention://all/all)" + } + if source == "subscriber comment" { + content = "No mentions" + addTestSubscriber(t, issueID, "member", recipient, "manual") + expectedType = "new_comment" + } + payload["comment"] = handler.CommentResponse{ID: commentID, IssueID: issueID, AuthorType: "member", AuthorID: testUserID, Content: content} + case "subscriber status", "parent subscriber": + subscriberIssue := issueID + if source == "parent subscriber" { + subscriberIssue = fx.Issue(t, "Parent") + fx.Exec(t, `UPDATE issue SET parent_issue_id = $1 WHERE id = $2`, subscriberIssue, issueID) + fx.Cleanup(t, `DELETE FROM issue_subscriber WHERE issue_id = $1`, subscriberIssue) + } + addTestSubscriber(t, subscriberIssue, "member", recipient, "manual") + e.Type = protocol.EventIssueUpdated + issue.Status = "done" + payload["issue"], payload["status_changed"], payload["prev_status"] = issue, true, "todo" + expectedType = "status_changed" + case "subscriber task failure": + addTestSubscriber(t, issueID, "member", recipient, "manual") + e.Type = protocol.EventTaskFailed + expectedType = "task_failed" + } + if removed { + removeNotificationMember(t, recipient, memberID, state) + if n := fx.Count(t, `SELECT count(*) FROM issue_subscriber WHERE user_id = $1`, recipient); n != 0 { + t.Fatalf("removal left %d subscriptions", n) + } + } + // Register only notification listeners: source events are delivered + // synchronously, with no sleeps or asynchronous observation window. + bus := events.New() + registerNotificationListeners(bus, db.New(testPool)) + eventCount := 0 + bus.Subscribe(protocol.EventInboxNew, func(e events.Event) { + item := e.Payload.(map[string]any)["item"].(map[string]any) + if item["recipient_id"] == recipient { + eventCount++ + if item["type"] != expectedType { + t.Errorf("event type = %v, want %s", item["type"], expectedType) + } + } + }) + e.Payload = payload + bus.Publish(e) + want := 1 + if removed { + want = 0 + } + if n := fx.Count(t, `SELECT count(*) FROM inbox_item WHERE workspace_id = $1 AND recipient_id = $2`, testWorkspaceID, recipient); n != want { + t.Errorf("inbox rows = %d, want %d", n, want) + } + if eventCount != want { + t.Errorf("inbox:new events = %d, want %d", eventCount, want) + } + }) + } + } +} + +// Only the origin lookup is fault-injected to select the unconfirmed branch; +// task transitions and inbox writes still run against real PostgreSQL. +type originLookupFailure struct{ *pgxpool.Pool } +type failedOriginRow struct{} + +func (failedOriginRow) Scan(...any) error { return errors.New("injected origin lookup failure") } +func (q originLookupFailure) QueryRow(ctx context.Context, sql string, args ...any) pgx.Row { + if strings.HasPrefix(sql, "-- name: GetIssueByOrigin ") { + return failedOriginRow{} + } + return q.Pool.QueryRow(ctx, sql, args...) +} + +func TestNotification_ServiceMembership(t *testing.T) { + for _, source := range []string{"quick_create_done", "quick_create_failed", "quick_create_unconfirmed", "children_done"} { + for _, state := range []string{"current", "delete", "leave"} { + t.Run(source+"/"+state, func(t *testing.T) { + ctx := context.Background() + fx := workspaceFixture(t) + recipient := fx.User(t, "Requester", "membership-service@example.test") + member := fx.Member(t, testWorkspaceID, recipient, "member") + bus := events.New() + queries := db.New(testPool) + if source == "quick_create_unconfirmed" { + queries = db.New(originLookupFailure{testPool}) + } + tasks := service.NewTaskService(queries, testPool, nil, bus) + eventCount := 0 + bus.Subscribe(protocol.EventInboxNew, func(e events.Event) { + item := e.Payload.(map[string]any)["item"].(map[string]any) + if item["recipient_id"] == recipient { + eventCount++ + if item["type"] != source { + t.Errorf("event type = %v, want %s", item["type"], source) + } + } + }) + fx.Cleanup(t, `DELETE FROM inbox_item WHERE recipient_id = $1`, recipient) + var trigger func() error + if source == "children_done" { + parent := fx.Issue(t, "Parent", testutil.Cols{"assignee_type": "member", "assignee_id": recipient, "status": "in_progress"}) + child := fx.Issue(t, "Child", testutil.Cols{"parent_issue_id": parent, "status": "in_progress"}) + fx.Cleanup(t, `DELETE FROM activity_log WHERE issue_id = $1`, parent) + fx.Cleanup(t, `DELETE FROM issue_wakeup_receipt WHERE wakeup_id IN (SELECT id FROM issue_wakeup WHERE issue_id = $1)`, parent) + fx.Cleanup(t, `DELETE FROM issue_wakeup WHERE issue_id = $1`, parent) + fx.Cleanup(t, `DELETE FROM issue_child_event WHERE parent_id = $1`, parent) + wakeups := &service.IssueWakeupService{Tasks: tasks} + if err := wakeups.ProcessChildEvents(ctx, util.MustParseUUID(parent)); err != nil { + t.Fatal(err) + } + trigger = func() error { + fx.Exec(t, `UPDATE issue SET status = 'done' WHERE id = $1`, child) + want, wantOutcome := 1, "notified" + if state != "current" { + want, wantOutcome = 0, "none" + } + for _, phase := range []string{"trigger", "repeat"} { + if err := wakeups.ProcessChildEvents(ctx, util.MustParseUUID(parent)); err != nil { + return err + } + if n := fx.Count(t, `SELECT count(*) FROM issue_wakeup_receipt r JOIN issue_wakeup w ON w.id = r.wakeup_id WHERE w.issue_id = $1 AND r.processed_at IS NULL`, parent); n != 0 { + t.Errorf("%s: pending wakeup receipts = %d, want 0", phase, n) + } + var count int + var outcome string + fx.QueryRow(t, `SELECT count(*), COALESCE(min(details->>'outcome'), '') FROM activity_log WHERE issue_id = $1 AND action = 'wakeup_triggered'`, parent).Scan(&count, &outcome) + if count != 1 || outcome != wantOutcome { + t.Errorf("%s: triggered activities = %d, outcome = %q; want 1, %q", phase, count, outcome, wantOutcome) + } + if n := fx.Count(t, `SELECT count(*) FROM inbox_item WHERE recipient_id = $1`, recipient); n != want { + t.Errorf("%s: inbox rows = %d, want %d", phase, n, want) + } + if eventCount != want { + t.Errorf("%s: inbox:new events = %d, want %d", phase, eventCount, want) + } + } + return nil + } + } else { + runtime := fx.Runtime(t, "Shared runtime") + agent := fx.Agent(t, "Shared agent", runtime) + qc, err := json.Marshal(service.QuickCreateContext{Type: service.QuickCreateContextType, RequesterID: recipient, WorkspaceID: testWorkspaceID, Prompt: "Create issue"}) + if err != nil { + t.Fatal(err) + } + taskID := fx.Task(t, agent, testutil.Cols{"runtime_id": runtime, "status": "running", "context": qc}) + if source == "quick_create_done" { + fx.Issue(t, "Created issue", testutil.Cols{"creator_type": "agent", "creator_id": agent, "origin_type": "quick_create", "origin_id": taskID}) + } + trigger = func() error { + _, err := tasks.CompleteTask(ctx, util.MustParseUUID(taskID), []byte(`{"output":"Finished"}`), "", "", "", false, "", "") + return err + } + } + if state != "current" { + removeNotificationMember(t, recipient, member, state) + } + if err := trigger(); err != nil { + t.Fatalf("trigger: %v", err) + } + want := 1 + if state != "current" { + want = 0 + } + if n := fx.Count(t, `SELECT count(*) FROM inbox_item WHERE recipient_id = $1`, recipient); n != want { + t.Errorf("inbox rows = %d, want %d", n, want) + } + if eventCount != want { + t.Errorf("inbox:new events = %d, want %d", eventCount, want) + } + }) + } + } +} diff --git a/server/internal/inbox/create.go b/server/internal/inbox/create.go new file mode 100644 index 00000000000..1fb9d233693 --- /dev/null +++ b/server/internal/inbox/create.go @@ -0,0 +1,23 @@ +package inbox + +import ( + "context" + "errors" + + "github.com/jackc/pgx/v5" + db "github.com/multica-ai/multica/server/pkg/db/generated" +) + +// CreateIfEligible returns inserted=false without an error when the recipient +// does not belong to the workspace. Other database errors remain failures. +// Eligibility is checked by the insert's snapshot, without additional locks. +func CreateIfEligible(ctx context.Context, queries *db.Queries, params db.CreateInboxItemParams) (item db.InboxItem, inserted bool, err error) { + item, err = queries.CreateInboxItem(ctx, params) + if errors.Is(err, pgx.ErrNoRows) { + return db.InboxItem{}, false, nil + } + if err != nil { + return db.InboxItem{}, false, err + } + return item, true, nil +} diff --git a/server/internal/service/autopilot.go b/server/internal/service/autopilot.go index 4d20084398f..88f891f87d2 100644 --- a/server/internal/service/autopilot.go +++ b/server/internal/service/autopilot.go @@ -18,6 +18,7 @@ import ( "github.com/multica-ai/multica/server/internal/dispatch" "github.com/multica-ai/multica/server/internal/entitlement" "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/issueguard" "github.com/multica-ai/multica/server/internal/issueposition" "github.com/multica-ai/multica/server/internal/issuestatus" @@ -898,7 +899,7 @@ func (s *AutopilotService) notifyAutopilotSubscribersOnCreate( if sub.UserType != "member" { continue } - item, err := s.Queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, s.Queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: ap.WorkspaceID, RecipientType: "member", @@ -921,6 +922,9 @@ func (s *AutopilotService) notifyAutopilotSubscribersOnCreate( ) continue } + if !inserted { + continue + } s.Bus.Publish(events.Event{ Type: protocol.EventInboxNew, WorkspaceID: util.UUIDToString(ap.WorkspaceID), diff --git a/server/internal/service/autopilot_quota_notifications.go b/server/internal/service/autopilot_quota_notifications.go index 203741f2edf..ea69f8d254f 100644 --- a/server/internal/service/autopilot_quota_notifications.go +++ b/server/internal/service/autopilot_quota_notifications.go @@ -13,6 +13,7 @@ import ( "github.com/jackc/pgx/v5/pgtype" "github.com/multica-ai/multica/server/internal/entitlement" "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/util" db "github.com/multica-ai/multica/server/pkg/db/generated" "github.com/multica-ai/multica/server/pkg/dbid" @@ -158,7 +159,7 @@ func (s *AutopilotService) createAutopilotQuotaInboxItems( items := make([]db.InboxItem, 0, len(recipients)) for _, recipient := range recipients { - item, err := q.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, q, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: facts.WorkspaceID, RecipientType: recipient.Type, RecipientID: recipient.ID, Type: "autopilot_quota_exceeded", Severity: "attention", IssueID: pgtype.UUID{}, @@ -169,6 +170,9 @@ func (s *AutopilotService) createAutopilotQuotaInboxItems( if err != nil { return nil, fmt.Errorf("create autopilot quota inbox item: %w", err) } + if !inserted { + continue + } items = append(items, item) } return items, nil diff --git a/server/internal/service/issue_wakeup_system.go b/server/internal/service/issue_wakeup_system.go index 8afbaf2d11c..b6dcc8585d2 100644 --- a/server/internal/service/issue_wakeup_system.go +++ b/server/internal/service/issue_wakeup_system.go @@ -12,6 +12,7 @@ import ( "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgtype" "github.com/multica-ai/multica/server/internal/events" + inboxwriter "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/issuestatus" "github.com/multica-ai/multica/server/internal/util" db "github.com/multica-ai/multica/server/pkg/db/generated" @@ -554,7 +555,7 @@ func (s *IssueWakeupService) dispatchSystem(ctx context.Context, prev db.IssueWa if current.Type == "member" { facts["outcome"] = "notified" details, _ := json.Marshal(facts) - item, err := q.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inboxwriter.CreateIfEligible(ctx, q, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: issue.WorkspaceID, RecipientType: "member", RecipientID: current.ID, Type: "children_done", Severity: "info", IssueID: issue.ID, Title: issue.Title, ActorType: pgtype.Text{String: "system", Valid: true}, Details: details, @@ -562,7 +563,11 @@ func (s *IssueWakeupService) dispatchSystem(ctx context.Context, prev db.IssueWa if err != nil { return err } - inbox = append(inbox, item) + if inserted { + inbox = append(inbox, item) + } else { + facts["outcome"] = "none" + } } if err = q.ConsumeWakeupReceipts(ctx, db.ConsumeWakeupReceiptsParams{Ids: ids}); err != nil { return err diff --git a/server/internal/service/task.go b/server/internal/service/task.go index 7ddf3f90e14..5893204d906 100644 --- a/server/internal/service/task.go +++ b/server/internal/service/task.go @@ -22,6 +22,7 @@ import ( "github.com/multica-ai/multica/server/internal/entitlement" "github.com/multica-ai/multica/server/internal/events" "github.com/multica-ai/multica/server/internal/featureflags" + "github.com/multica-ai/multica/server/internal/inbox" "github.com/multica-ai/multica/server/internal/issuestatus" obsmetrics "github.com/multica-ai/multica/server/internal/metrics" "github.com/multica-ai/multica/server/internal/realtime" @@ -6507,7 +6508,7 @@ func (s *TaskService) exhaustDelegatedFailureRecovery(ctx context.Context, targe if err != nil { return fmt.Errorf("encode delegated failure exhaustion details: %w", err) } - exhaustedInbox, err = qtx.CreateInboxItem(ctx, db.CreateInboxItemParams{ + exhaustedInbox, inboxCreated, err = inbox.CreateIfEligible(ctx, qtx, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: target.issue.WorkspaceID, RecipientType: "member", @@ -6524,7 +6525,6 @@ func (s *TaskService) exhaustDelegatedFailureRecovery(ctx context.Context, targe if err != nil { return fmt.Errorf("create delegated failure exhaustion inbox item: %w", err) } - inboxCreated = true } return nil }); err != nil { @@ -7929,7 +7929,7 @@ func (s *TaskService) notifyQuickCreateCompleted(ctx context.Context, task db.Ag "identifier": identifier, "original_prompt": qc.Prompt, }) - item, err := s.Queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, s.Queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: workspaceID, RecipientType: "member", @@ -7947,6 +7947,9 @@ func (s *TaskService) notifyQuickCreateCompleted(ctx context.Context, task db.Ag slog.Error("quick-create completion: inbox write failed", "task_id", util.UUIDToString(task.ID), "error", err) return } + if !inserted { + return + } s.publishQuickCreateInbox(item, qc.WorkspaceID, util.UUIDToString(task.AgentID), issue.Status) } @@ -8030,7 +8033,7 @@ func (s *TaskService) writeQuickCreateOutcomeInbox(ctx context.Context, task db. "error": redact.Text(errMsg), "source_context_id": qc.SourceContextID, }) - item, err := s.Queries.CreateInboxItem(ctx, db.CreateInboxItemParams{ + item, inserted, err := inbox.CreateIfEligible(ctx, s.Queries, db.CreateInboxItemParams{ ID: dbid.NewV7(), WorkspaceID: workspaceID, RecipientType: "member", @@ -8048,6 +8051,9 @@ func (s *TaskService) writeQuickCreateOutcomeInbox(ctx context.Context, task db. slog.Error("quick-create failure: inbox write failed", "task_id", util.UUIDToString(task.ID), "error", err) return } + if !inserted { + return + } s.publishQuickCreateInbox(item, qc.WorkspaceID, util.UUIDToString(task.AgentID), "") } diff --git a/server/pkg/db/generated/inbox.sql.go b/server/pkg/db/generated/inbox.sql.go index 789b048dcca..9634931feae 100644 --- a/server/pkg/db/generated/inbox.sql.go +++ b/server/pkg/db/generated/inbox.sql.go @@ -261,7 +261,13 @@ INSERT INTO inbox_item ( workspace_id, recipient_type, recipient_id, type, severity, issue_id, title, body, actor_type, actor_id, details, id -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, COALESCE($12::uuid, gen_random_uuid())) +) +SELECT $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, COALESCE($12::uuid, gen_random_uuid()) +WHERE ($2::text = 'member' AND EXISTS ( + SELECT 1 FROM member WHERE workspace_id = $1 AND user_id = $3 +)) OR ($2::text = 'agent' AND EXISTS ( + SELECT 1 FROM agent WHERE workspace_id = $1 AND id = $3 +)) RETURNING id, workspace_id, recipient_type, recipient_id, type, severity, issue_id, title, body, read, archived, created_at, actor_type, actor_id, details ` @@ -280,6 +286,8 @@ type CreateInboxItemParams struct { ID pgtype.UUID `json:"id"` } +// A missing recipient is a quiet skip (pgx.ErrNoRows), not a delivery failure. +// Check the insertion snapshot without adding recipient row locks. func (q *Queries) CreateInboxItem(ctx context.Context, arg CreateInboxItemParams) (InboxItem, error) { row := q.db.QueryRow(ctx, createInboxItem, arg.WorkspaceID, diff --git a/server/pkg/db/queries/inbox.sql b/server/pkg/db/queries/inbox.sql index aad0404fb80..b5996db6369 100644 --- a/server/pkg/db/queries/inbox.sql +++ b/server/pkg/db/queries/inbox.sql @@ -90,11 +90,19 @@ SELECT * FROM inbox_item WHERE id = $1 AND workspace_id = $2; -- name: CreateInboxItem :one +-- A missing recipient is a quiet skip (pgx.ErrNoRows), not a delivery failure. +-- Check the insertion snapshot without adding recipient row locks. INSERT INTO inbox_item ( workspace_id, recipient_type, recipient_id, type, severity, issue_id, title, body, actor_type, actor_id, details, id -) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, COALESCE(sqlc.narg('id')::uuid, gen_random_uuid())) +) +SELECT $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, COALESCE(sqlc.narg('id')::uuid, gen_random_uuid()) +WHERE ($2::text = 'member' AND EXISTS ( + SELECT 1 FROM member WHERE workspace_id = $1 AND user_id = $3 +)) OR ($2::text = 'agent' AND EXISTS ( + SELECT 1 FROM agent WHERE workspace_id = $1 AND id = $3 +)) RETURNING *; -- name: MarkInboxRead :one From d3c1b44d757f941ab2376d6a775d1ebe54d68284 Mon Sep 17 00:00:00 2001 From: Multica Eve Date: Thu, 15 Oct 2026 12:36:00 +0800 Subject: [PATCH 03/51] MUL-8014: fix(realtime): revoke workspace WebSocket access (#9249) * fix(realtime): revoke workspace access on membership removal Close removed members' sockets, revalidate registration, and gate captured write batches with fresh coalesced membership queries. Cover relay failure, replay, queue races, and transactional removal/rollback (MUL-8014). Co-authored-by: multica-agent * fix(realtime): keep registration free of database waits Co-authored-by: multica-agent * perf(realtime): narrow removal event prefilter Co-authored-by: multica-agent * test(wakeup): avoid global backfill in workspace default test Co-authored-by: multica-agent * fix(realtime): bound concurrent authorization and defer relay revocation Co-authored-by: multica-agent * test(realtime): check caller budget without waiting for expiry Co-authored-by: multica-agent * fix(realtime): decouple member-list invalidation from revocation Co-authored-by: multica-agent * perf(realtime): skip removal decoding for member actor metadata Co-authored-by: multica-agent --------- Co-authored-by: Sol-Boy Co-authored-by: multica-agent --- server/cmd/server/realtime_membership_test.go | 201 ++++++ server/cmd/server/router.go | 44 +- .../handler/issue_system_wakeup_test.go | 21 +- server/internal/handler/workspace.go | 2 + server/internal/realtime/hub.go | 111 ++- server/internal/realtime/hub_test.go | 13 +- server/internal/realtime/membership.go | 190 ++++++ server/internal/realtime/membership_batch.go | 150 ++++ .../realtime/membership_filter_test.go | 48 ++ .../realtime/membership_performance_test.go | 218 ++++++ .../internal/realtime/membership_removal.go | 72 ++ .../realtime/membership_removal_test.go | 283 ++++++++ server/internal/realtime/membership_test.go | 646 ++++++++++++++++++ server/internal/realtime/metrics.go | 30 +- server/pkg/db/generated/member.sql.go | 40 ++ server/pkg/db/queries/member.sql | 9 + 16 files changed, 2025 insertions(+), 53 deletions(-) create mode 100644 server/cmd/server/realtime_membership_test.go create mode 100644 server/internal/realtime/membership.go create mode 100644 server/internal/realtime/membership_batch.go create mode 100644 server/internal/realtime/membership_filter_test.go create mode 100644 server/internal/realtime/membership_performance_test.go create mode 100644 server/internal/realtime/membership_removal.go create mode 100644 server/internal/realtime/membership_removal_test.go create mode 100644 server/internal/realtime/membership_test.go diff --git a/server/cmd/server/realtime_membership_test.go b/server/cmd/server/realtime_membership_test.go new file mode 100644 index 00000000000..977ec245934 --- /dev/null +++ b/server/cmd/server/realtime_membership_test.go @@ -0,0 +1,201 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "net" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/go-chi/chi/v5" + "github.com/gorilla/websocket" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" + "github.com/multica-ai/multica/server/internal/analytics" + "github.com/multica-ai/multica/server/internal/auth" + "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/realtime" + "github.com/multica-ai/multica/server/internal/testutil" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/protocol" +) + +type membershipFailCommitPool struct{ pool *pgxpool.Pool } +type membershipFailCommitTx struct{ pgx.Tx } + +func (p membershipFailCommitPool) Begin(ctx context.Context) (pgx.Tx, error) { + tx, err := p.pool.Begin(ctx) + if err != nil { + return nil, err + } + return membershipFailCommitTx{tx}, nil +} + +func (membershipFailCommitTx) Commit(context.Context) error { + return errors.New("injected membership commit failure") +} + +func realtimeMembershipConn(t *testing.T, server *httptest.Server, userID, workspaceID string) (*websocket.Conn, string) { + t.Helper() + token, err := generateTestJWT(userID, "realtime@example.test", "Realtime") + if err != nil { + t.Fatal(err) + } + headers := http.Header{"Cookie": []string{auth.AuthCookieName + "=" + token}} + conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http")+"/ws?workspace_id="+workspaceID, headers) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { conn.Close() }) + if err := conn.WriteJSON(map[string]string{"type": "ping"}); err != nil { + t.Fatal(err) + } + wantRealtimeMembershipFrame(t, conn, "pong") + return conn, token +} + +func wantRealtimeMembershipFrame(t *testing.T, conn *websocket.Conn, typ string) { + t.Helper() + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + _, raw, err := conn.ReadMessage() + var frame struct{ Type string } + if err != nil || json.Unmarshal(raw, &frame) != nil || frame.Type != typ { + t.Fatalf("want %s, got %s, err=%v", typ, raw, err) + } +} + +func wantRealtimeMembershipClosed(t *testing.T, conn *websocket.Conn) { + t.Helper() + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + for { + _, raw, err := conn.ReadMessage() + if err != nil { + var netErr net.Error + if errors.As(err, &netErr) && netErr.Timeout() { + t.Fatalf("revoked socket stayed open: %v", err) + } + return + } + var frame struct{ Type string } + if json.Unmarshal(raw, &frame) != nil || frame.Type != protocol.EventMemberRemoved { + t.Fatalf("revoked socket received workspace content: %s", raw) + } + } +} + +func TestRealtimeMemberRevocation(t *testing.T) { + for _, method := range []string{"delete", "leave"} { + for _, failCommit := range []bool{false, true} { + name := method + if failCommit { + name += "/rollback" + } + t.Run(name, func(t *testing.T) { + fx := workspaceFixture(t) + wsID := fx.Workspace(t, "Realtime revocation", "realtime-revocation") + otherWS := fx.Workspace(t, "Other realtime workspace", "realtime-other") + userID := fx.User(t, "Leaver", "realtime-leaver@example.test") + fx.Member(t, wsID, testUserID, "owner") + memberID := fx.Member(t, wsID, userID, "member") + fx.Member(t, otherWS, userID, "owner") + hub := realtime.NewHub() + go hub.Run() + bus := events.New() + registerListeners(bus, hub) + router, h := NewRouterWithOptions(testPool, hub, bus, analytics.NoopClient{}, nil, RouterOptions{}) + server := httptest.NewServer(router) + t.Cleanup(server.Close) + old, token := realtimeMembershipConn(t, server, userID, wsID) + second, _ := realtimeMembershipConn(t, server, userID, wsID) + other, _ := realtimeMembershipConn(t, server, userID, otherWS) + current, _ := realtimeMembershipConn(t, server, testUserID, wsID) + if failCommit { + h.TxStarter = membershipFailCommitPool{testPool} + } + request := httptest.NewRequest(http.MethodDelete, "/", nil) + actor := testUserID + if method == "leave" { + actor = userID + } + request.Header.Set("X-User-ID", actor) + route := chi.NewRouteContext() + route.URLParams.Add("id", wsID) + route.URLParams.Add("memberId", memberID) + request = request.WithContext(context.WithValue(request.Context(), chi.RouteCtxKey, route)) + endpoint := h.DeleteMember + if method == "leave" { + endpoint = h.LeaveWorkspace + } + status := http.StatusNoContent + if failCommit { + status = http.StatusInternalServerError + } + testutil.Call(t, endpoint, request).Want(status) + if failCommit { + if fx.Count(t, `SELECT count(*) FROM member WHERE id=$1`, memberID) != 1 { + t.Fatal("failed revocation did not restore membership") + } + } else { + wantRealtimeMembershipClosed(t, old) + wantRealtimeMembershipClosed(t, second) + wantRealtimeMembershipFrame(t, current, protocol.EventMemberRemoved) + fresh, resp, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http")+"/ws?workspace_id="+wsID, + http.Header{"Cookie": []string{auth.AuthCookieName + "=" + token}}) + if fresh != nil { + fresh.Close() + } + if resp != nil && resp.Body != nil { + defer resp.Body.Close() + } + if err == nil || resp == nil || resp.StatusCode != http.StatusForbidden { + t.Fatalf("revoked reconnect accepted: response=%v, err=%v", resp, err) + } + } + // Use the real event listeners after the removal API returned. + for _, typ := range []string{protocol.EventCommentCreated, protocol.EventIssueUpdated} { + bus.Publish(events.Event{Type: typ, WorkspaceID: wsID, Payload: map[string]string{"body": "new-workspace-content"}}) + wantRealtimeMembershipFrame(t, current, typ) + if failCommit { + wantRealtimeMembershipFrame(t, old, typ) + wantRealtimeMembershipFrame(t, second, typ) + } + } + bus.Publish(events.Event{Type: protocol.EventIssueUpdated, WorkspaceID: otherWS, Payload: map[string]string{"title": "other-workspace"}}) + wantRealtimeMembershipFrame(t, other, protocol.EventIssueUpdated) + }) + } + } +} + +func TestRealtimeBatchMembershipPairs(t *testing.T) { + fx := workspaceFixture(t) + firstWS := fx.Workspace(t, "Batch one", "realtime-batch-one") + secondWS := fx.Workspace(t, "Batch two", "realtime-batch-two") + firstUser := fx.User(t, "Batch A", "realtime-batch-a@example.test") + secondUser := fx.User(t, "Batch B", "realtime-batch-b@example.test") + fx.Member(t, firstWS, firstUser, "owner") + fx.Member(t, secondWS, secondUser, "owner") + checker := &membershipChecker{queries: db.New(testPool)} + // Both users and both workspaces exist, but cross-pair identities must + // never inherit another pair's membership through a Cartesian product. + requested := []realtime.MembershipIdentity{ + {WorkspaceID: firstWS, UserID: secondUser}, + {WorkspaceID: secondWS, UserID: firstUser}, + {WorkspaceID: firstWS, UserID: firstUser}, + {WorkspaceID: secondWS, UserID: secondUser}, + {WorkspaceID: strings.ToUpper(firstWS), UserID: strings.ToUpper(firstUser)}, + } + members, err := checker.Members(context.Background(), requested) + if err != nil { + t.Fatal(err) + } + for i, identity := range requested { + if members[identity] != (i >= 2) { + t.Fatalf("incorrect membership for pair %+v", identity) + } + } +} diff --git a/server/cmd/server/router.go b/server/cmd/server/router.go index 54485467beb..ffb6234d38c 100644 --- a/server/cmd/server/router.go +++ b/server/cmd/server/router.go @@ -3,6 +3,7 @@ package main import ( "context" "crypto/sha256" + "errors" "fmt" "log/slog" "net/http" @@ -14,6 +15,7 @@ import ( "github.com/go-chi/chi/v5" chimw "github.com/go-chi/chi/v5/middleware" "github.com/go-chi/cors" + "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgtype" "github.com/jackc/pgx/v5/pgxpool" "github.com/redis/go-redis/v9" @@ -1449,6 +1451,7 @@ func NewRouterWithOptions(pool *pgxpool.Pool, hub *realtime.Hub, bus *events.Bus // WebSocket mc := &membershipChecker{queries: queries} + hub.SetMembershipChecker(mc) pr := &patResolver{queries: queries, cache: patCache} slugResolver := realtime.SlugResolver(func(ctx context.Context, slug string) (string, error) { ws, err := queries.GetWorkspaceBySlug(ctx, slug) @@ -2550,12 +2553,49 @@ type membershipChecker struct { queries *db.Queries } -func (mc *membershipChecker) IsMember(ctx context.Context, userID, workspaceID string) bool { +func (mc *membershipChecker) IsMember(ctx context.Context, userID, workspaceID string) (bool, error) { _, err := mc.queries.GetMemberByUserAndWorkspace(ctx, db.GetMemberByUserAndWorkspaceParams{ UserID: parseUUID(userID), WorkspaceID: parseUUID(workspaceID), }) - return err == nil + if errors.Is(err, pgx.ErrNoRows) { + return false, nil + } + return err == nil, err +} + +func (mc *membershipChecker) Members(ctx context.Context, identities []realtime.MembershipIdentity) (map[realtime.MembershipIdentity]bool, error) { + params := db.GetRealtimeMembershipsParams{} + for _, identity := range identities { + userID, err := util.ParseUUID(identity.UserID) + if err != nil { + return nil, err + } + workspaceID, err := util.ParseUUID(identity.WorkspaceID) + if err != nil { + return nil, err + } + params.UserIds = append(params.UserIds, userID) + params.WorkspaceIds = append(params.WorkspaceIds, workspaceID) + } + rows, err := mc.queries.GetRealtimeMemberships(ctx, params) + if err != nil { + return nil, err + } + type pair struct{ workspaceID, userID pgtype.UUID } + current := make(map[pair]bool, len(rows)) + for _, row := range rows { + current[pair{row.WorkspaceID, row.UserID}] = true + } + // Return the exact request keys, including valid UUID spellings that are + // different from the database's canonical lowercase rendering. + members := make(map[realtime.MembershipIdentity]bool, len(rows)) + for i, identity := range identities { + if current[pair{params.WorkspaceIds[i], params.UserIds[i]}] { + members[identity] = true + } + } + return members, nil } // patResolver implements realtime.PATResolver using database queries. diff --git a/server/internal/handler/issue_system_wakeup_test.go b/server/internal/handler/issue_system_wakeup_test.go index cf4dfe05ce6..5abcf95b977 100644 --- a/server/internal/handler/issue_system_wakeup_test.go +++ b/server/internal/handler/issue_system_wakeup_test.go @@ -180,9 +180,28 @@ func TestWorkspaceSystemWakeupDefault(t *testing.T) { if w := putChildDoneRule(t, custom.parent.ID, map[string]any{"enabled": true}); w.Code != http.StatusOK { t.Fatalf("customize: %d", w.Code) } - if _, err := (&service.IssueWakeupService{Tasks: testHandler.TaskService}).BackfillChildDoneRules(context.Background(), 1000); err != nil { + // Initialize only this fixture's uncustomized rule. A global backfill can + // race another package's workspace/rule cleanup in CI. + ctx := context.Background() + tx, err := testPool.Begin(ctx) + if err != nil { t.Fatal(err) } + defer tx.Rollback(ctx) + q := testHandler.Queries.WithTx(tx) + parent, err := q.GetIssue(ctx, parseUUID(follows.parent.ID)) + if err != nil { + t.Fatal(err) + } + if _, err := service.EnsureChildDoneRule(ctx, tx, q, parent); err != nil { + t.Fatal(err) + } + if err := tx.Commit(ctx); err != nil { + t.Fatal(err) + } + if rules := listSystemWakeupsFor(t, follows.parent.ID); len(rules) != 1 || rules[0].ID == "" || !rules[0].Enabled || rules[0].Customized { + t.Fatalf("fixture did not initialize its default-following rule: %+v", rules) + } put := func(body map[string]any, user string) *httptest.ResponseRecorder { req := withURLParam(newRequest("PUT", "/api/system-wakeups/child_done", body), "rule", "child_done") req.Header.Set("X-User-ID", user) diff --git a/server/internal/handler/workspace.go b/server/internal/handler/workspace.go index 0772ad3212e..35f652fdd1c 100644 --- a/server/internal/handler/workspace.go +++ b/server/internal/handler/workspace.go @@ -694,6 +694,7 @@ func (h *Handler) DeleteMember(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusInternalServerError, "failed to delete member") return } + h.Hub.DisconnectMember(uuidToString(target.WorkspaceID), uuidToString(target.UserID), uuidToString(target.ID)) h.settleMemberCapacityRelease(r.Context(), uuid.UUID(target.WorkspaceID.Bytes), uuid.UUID(target.ID.Bytes)) h.MembershipCache.Invalidate(r.Context(), uuidToString(target.UserID), workspaceID) @@ -738,6 +739,7 @@ func (h *Handler) LeaveWorkspace(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusInternalServerError, "failed to leave workspace") return } + h.Hub.DisconnectMember(uuidToString(member.WorkspaceID), uuidToString(member.UserID), uuidToString(member.ID)) h.settleMemberCapacityRelease(r.Context(), uuid.UUID(member.WorkspaceID.Bytes), uuid.UUID(member.ID.Bytes)) h.MembershipCache.Invalidate(r.Context(), uuidToString(member.UserID), workspaceID) diff --git a/server/internal/realtime/hub.go b/server/internal/realtime/hub.go index 807433bde0c..7012bfbaa9b 100644 --- a/server/internal/realtime/hub.go +++ b/server/internal/realtime/hub.go @@ -22,7 +22,13 @@ import ( // MembershipChecker verifies a user belongs to a workspace. type MembershipChecker interface { - IsMember(ctx context.Context, userID, workspaceID string) bool + IsMember(ctx context.Context, userID, workspaceID string) (bool, error) + Members(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) +} + +type MembershipIdentity struct { + WorkspaceID string + UserID string } // SlugResolver translates a workspace slug to its UUID. @@ -226,6 +232,8 @@ type Client struct { send chan []byte userID string workspaceID string + writeMu sync.Mutex // serializes application writes and the final removal notice + stopped atomic.Bool // sendMu guards sendClosed so a late inbound-response or fanout send // can never write to the closed send channel. closeSend flips @@ -281,12 +289,19 @@ type SubscriptionCallback func(scopeType, scopeID string) // Hub manages WebSocket connections organized into scope-based rooms. type Hub struct { - rooms map[scopeKey]map[*Client]bool - clients map[*Client]bool // every connected client (used by global Broadcast and snapshots) - broadcast chan []byte - register chan *Client - unregister chan *Client - mu sync.RWMutex + rooms map[scopeKey]map[*Client]bool + clients map[*Client]bool // every connected client (used by global Broadcast and snapshots) + broadcast chan []byte + register chan *Client + unregister chan *Client + mu sync.RWMutex + membership MembershipChecker // guarded by mu + writeMembership chan membershipWriteRequest + writeMembershipOnce sync.Once + memberRemovals chan memberRemovalJob + memberRemovalOnce sync.Once + memberRemovalPending atomic.Int64 + memberRemovalDropped atomic.Int64 authorizer ScopeAuthorizer @@ -298,11 +313,13 @@ type Hub struct { // NewHub creates a new Hub instance. func NewHub() *Hub { return &Hub{ - rooms: make(map[scopeKey]map[*Client]bool), - clients: make(map[*Client]bool), - broadcast: make(chan []byte), - register: make(chan *Client), - unregister: make(chan *Client), + rooms: make(map[scopeKey]map[*Client]bool), + clients: make(map[*Client]bool), + broadcast: make(chan []byte), + register: make(chan *Client), + unregister: make(chan *Client), + writeMembership: make(chan membershipWriteRequest, maxMembershipBatch), + memberRemovals: make(chan memberRemovalJob, memberRemovalQueueSize), } } @@ -328,18 +345,7 @@ func (h *Hub) Run() { for { select { case client := <-h.register: - h.mu.Lock() - h.clients[client] = true - total := len(h.clients) - h.mu.Unlock() - M.ConnectsTotal.Add(1) - M.ActiveConnections.Add(1) - // Auto-subscribe to the workspace and user scopes. - h.subscribe(client, ScopeWorkspace, client.workspaceID) - if client.userID != "" { - h.subscribe(client, ScopeUser, client.userID) - } - slog.Info("ws client connected", "workspace_id", client.workspaceID, "user_id", client.userID, "total_clients", total) + h.registerClient(client) case client := <-h.unregister: h.removeClient(client) @@ -505,6 +511,15 @@ func (h *Hub) BroadcastToScopeDedup(scopeType, scopeID string, message []byte, e if scopeType == "" || scopeID == "" { return } + // Both the local fast path and every relay backend converge here. Removal + // invalidations reach peers immediately; target connections are revalidated. + if scopeType == ScopeWorkspace && h.queueMemberRemoval(scopeID, message, eventID) { + return + } + h.broadcastToScope(scopeType, scopeID, message, eventID, "") +} + +func (h *Hub) broadcastToScope(scopeType, scopeID string, message []byte, eventID, excludeUserID string) { key := sk(scopeType, scopeID) h.mu.RLock() @@ -512,6 +527,9 @@ func (h *Hub) BroadcastToScopeDedup(scopeType, scopeID string, message []byte, e var slow []*Client var sent int64 for client := range clients { + if excludeUserID != "" && client.userID == excludeUserID { + continue + } if !client.markSeen(eventID) { continue } @@ -601,10 +619,9 @@ func (h *Hub) fanoutUser(userID string, message []byte, excludeWorkspace, eventI if !client.markSeen(eventID) { continue } - select { - case client.send <- message: + if client.trySend(message) { sent++ - default: + } else { slow = append(slow, client) } } @@ -674,8 +691,10 @@ func (h *Hub) Snapshot() map[string]any { rooms[key.Type]++ } return map[string]any{ - "connections": len(h.clients), - "rooms": rooms, + "connections": len(h.clients), + "rooms": rooms, + "membership_removals_pending": h.memberRemovalPending.Load(), + "membership_removals_dropped_total": h.memberRemovalDropped.Load(), } } @@ -804,7 +823,12 @@ func HandleWebSocket(hub *Hub, mc MembershipChecker, pr PATResolver, resolveSlug http.Error(w, errMsg, status) return } - if !mc.IsMember(r.Context(), uid, workspaceID) { + member, lookupErr := checkMembership(mc, uid, workspaceID) + if lookupErr != nil { + http.Error(w, `{"error":"membership lookup failed"}`, http.StatusServiceUnavailable) + return + } + if !member { http.Error(w, `{"error":"not a member of this workspace"}`, http.StatusForbidden) return } @@ -836,7 +860,12 @@ func HandleWebSocket(hub *Hub, mc MembershipChecker, pr PATResolver, resolveSlug writeWSAuthErrorAndClose(conn, []byte(errMsg), "workspace_id", workspaceID) return } - if !mc.IsMember(r.Context(), uid, workspaceID) { + member, lookupErr := checkMembership(mc, uid, workspaceID) + if lookupErr != nil { + writeWSAuthErrorAndClose(conn, []byte(`{"error":"membership lookup failed"}`), "workspace_id", workspaceID) + return + } + if !member { writeWSAuthErrorAndClose( conn, []byte(`{"error":"not a member of this workspace"}`), @@ -1082,18 +1111,28 @@ func (c *Client) writePump() { for { select { case message, ok := <-c.send: - c.conn.SetWriteDeadline(time.Now().Add(writeWait)) if !ok { - c.conn.WriteMessage(websocket.CloseMessage, []byte{}) + c.writeMu.Lock() + if !c.stopped.Load() { + c.conn.SetWriteDeadline(time.Now().Add(writeWait)) + c.conn.WriteMessage(websocket.CloseMessage, []byte{}) + } + c.writeMu.Unlock() return } - if err := c.conn.WriteMessage(websocket.TextMessage, message); err != nil { - slog.Warn("websocket write error", "error", err, "user_id", c.userID, "workspace_id", c.workspaceID) + if !c.writeBatch(message) { return } case <-ticker.C: + c.writeMu.Lock() + if c.stopped.Load() { + c.writeMu.Unlock() + return + } c.conn.SetWriteDeadline(time.Now().Add(writeWait)) - if err := c.conn.WriteMessage(websocket.PingMessage, nil); err != nil { + err := c.conn.WriteMessage(websocket.PingMessage, nil) + c.writeMu.Unlock() + if err != nil { return } } diff --git a/server/internal/realtime/hub_test.go b/server/internal/realtime/hub_test.go index a1723d790ea..ea3dba2918e 100644 --- a/server/internal/realtime/hub_test.go +++ b/server/internal/realtime/hub_test.go @@ -25,8 +25,16 @@ const testUserID = "test-user" // mockMembershipChecker always returns true. type mockMembershipChecker struct{} -func (m *mockMembershipChecker) IsMember(_ context.Context, _, _ string) bool { - return true +func (m *mockMembershipChecker) IsMember(_ context.Context, _, _ string) (bool, error) { + return true, nil +} + +func (*mockMembershipChecker) Members(_ context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + members := make(map[MembershipIdentity]bool, len(identities)) + for _, identity := range identities { + members[identity] = true + } + return members, nil } func makeTestToken(t *testing.T) string { @@ -89,6 +97,7 @@ func newTestHub(t *testing.T) (*Hub, *httptest.Server) { go hub.Run() mc := &mockMembershipChecker{} + hub.SetMembershipChecker(mc) mux := http.NewServeMux() mux.HandleFunc("/ws", func(w http.ResponseWriter, r *http.Request) { HandleWebSocket(hub, mc, nil, nil, w, r) diff --git a/server/internal/realtime/membership.go b/server/internal/realtime/membership.go new file mode 100644 index 00000000000..2c32a1e9d7a --- /dev/null +++ b/server/internal/realtime/membership.go @@ -0,0 +1,190 @@ +package realtime + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "log/slog" + "time" + + "github.com/gorilla/websocket" +) + +const ( + membershipLookupTimeout = 2 * time.Second + maxWriteBatch = 32 + removalNoticeWait = 100 * time.Millisecond +) + +// SetMembershipChecker wires the primary database lookup before serving sockets. +// The handshake and content writer must use this same authority. +func (h *Hub) SetMembershipChecker(checker MembershipChecker) { + h.mu.Lock() + h.membership = checker + h.mu.Unlock() +} + +func checkMembership(checker MembershipChecker, userID, workspaceID string) (bool, error) { + if checker == nil { + return false, errors.New("membership checker unavailable") + } + ctx, cancel := context.WithTimeout(context.Background(), membershipLookupTimeout) + defer cancel() + member, err := checker.IsMember(ctx, userID, workspaceID) + if err != nil { + slog.Warn("ws membership lookup failed", "user_id", userID, "workspace_id", workspaceID, "error", err) + } + return member, err +} + +func (h *Hub) registerClient(client *Client) { + // Keep the Hub event loop free of database waits. A connection whose + // handshake raced revocation is denied by its first content write. + h.mu.Lock() + // A transport/write failure can abort this client before registration. + // Check under the same lock used by removeClient. + if client.stopped.Load() { + h.mu.Unlock() + return + } + h.clients[client] = true + total := len(h.clients) + h.mu.Unlock() + M.ConnectsTotal.Add(1) + M.ActiveConnections.Add(1) + h.subscribe(client, ScopeWorkspace, client.workspaceID) + h.subscribe(client, ScopeUser, client.userID) + slog.Info("ws client connected", "workspace_id", client.workspaceID, "user_id", client.userID, "total_clients", total) +} + +// DisconnectMember is a post-commit signal, not an authorization cache. Check +// current membership so delayed/replayed signals cannot disconnect a rejoined +// member. Lookup errors fail closed without claiming membership was removed. +func (h *Hub) DisconnectMember(workspaceID, userID, memberID string) { + if h == nil || workspaceID == "" || userID == "" { + return + } + notice, _ := json.Marshal(map[string]any{ + "type": "member:removed", + "payload": map[string]string{"workspace_id": workspaceID, "user_id": userID, "member_id": memberID}, + }) + job := h.snapshotMemberRemoval(workspaceID, userID) + if len(job.clients) == 0 { + return + } + job.notice = notice + h.disconnectMember(job) +} + +// Only confirmed removals send the target a notice; stale signals are ignored +// and lookup failures close captured connections without a navigation notice. +func (h *Hub) disconnectMember(job memberRemovalJob) { + member, err := checkMembership(job.checker, job.userID, job.workspaceID) + if err == nil && member { + return + } + if err != nil { + job.notice = nil + } + // One total deadline bounds notification attempts across all sockets. An + // idle writer can send the existing client navigation signal; busy/slow + // writers are closed immediately instead of draining content first. + deadline := time.Now().Add(removalNoticeWait) + for _, client := range job.clients { + h.stopClient(client, job.notice, deadline) + } +} + +// Returns the target and whether this is a removal frame. An empty target on +// a removal frame means its payload is invalid and the frame must be dropped. +func parseMemberRemoval(workspaceID string, raw []byte) (string, bool) { + if !memberRemovalCandidate(raw) { + return "", false + } + var frame struct { + Type string `json:"type"` + Payload struct { + UserID string `json:"user_id"` + WorkspaceID string `json:"workspace_id"` + } `json:"payload"` + } + if json.Unmarshal(raw, &frame) != nil || frame.Type != "member:removed" { + return "", false + } + if frame.Payload.UserID == "" || frame.Payload.WorkspaceID != workspaceID { + return "", true + } + return frame.Payload.UserID, true +} + +func memberRemovalCandidate(raw []byte) bool { + // Require an event prefix, not the actor_type value "member" attached to + // ordinary comments and issue updates. A separator escape starts with '\'. + // An escaped "member" prefix must contain \u006* (m/e/b) or \u0072 + // (r). HTML and line-separator escapes in ordinary content do not qualify. + return bytes.Contains(raw, []byte("member:")) || bytes.Contains(raw, []byte(`member\`)) || + bytes.Contains(raw, []byte(`\u006`)) || bytes.Contains(raw, []byte(`\u0072`)) +} + +func (h *Hub) stopClient(client *Client, notice []byte, deadline time.Time) { + client.stopped.Store(true) + h.removeClient(client) + client.closeSend() // a writer may stop before registration completes + for range client.send { + // Discard queued content, including frames whose writer is doing a + // membership lookup concurrently. The writer also checks stopped. + } + if client.conn == nil { + return + } + if len(notice) != 0 && time.Now().Before(deadline) && client.writeMu.TryLock() { + client.conn.SetWriteDeadline(deadline) + _ = client.conn.WriteMessage(websocket.TextMessage, notice) + client.writeMu.Unlock() + } + client.conn.Close() +} + +// Capture the bounded batch BEFORE querying membership. A successful lookup +// authorizes only these already queued frames, never content arriving during +// or after the lookup. In particular there is no positive TTL or singleflight +// result that a post-revocation event on another node can reuse. +func (c *Client) writeBatch(first []byte) bool { + if c.stopped.Load() { + return false + } + batch := make([][]byte, 0, maxWriteBatch) + batch = append(batch, first) +drain: + for len(batch) < maxWriteBatch { + select { + case frame, open := <-c.send: + if !open { + break drain + } + batch = append(batch, frame) + default: + break drain + } + } + if !c.hub.authorizeWrites(MembershipIdentity{WorkspaceID: c.workspaceID, UserID: c.userID}) { + c.hub.stopClient(c, nil, time.Time{}) + return false + } + c.writeMu.Lock() + defer c.writeMu.Unlock() + // One deadline bounds the whole batch instead of allowing each frame a + // fresh timeout. A revocation can still interrupt it via conn.Close. + c.conn.SetWriteDeadline(time.Now().Add(writeWait)) + for _, frame := range batch { + if c.stopped.Load() { + return false + } + if err := c.conn.WriteMessage(websocket.TextMessage, frame); err != nil { + slog.Warn("websocket write error", "error", err, "user_id", c.userID, "workspace_id", c.workspaceID) + return false + } + } + return true +} diff --git a/server/internal/realtime/membership_batch.go b/server/internal/realtime/membership_batch.go new file mode 100644 index 00000000000..3660884c349 --- /dev/null +++ b/server/internal/realtime/membership_batch.go @@ -0,0 +1,150 @@ +package realtime + +import ( + "context" + "errors" + "log/slog" + "sync" + "time" +) + +const ( + maxMembershipBatch = 256 + membershipWorkers = 4 +) + +type membershipWriteRequest struct { + identity MembershipIdentity + result chan bool + ctx context.Context +} + +func (r membershipWriteRequest) pending() bool { + return r.ctx == nil || r.ctx.Err() == nil +} + +func (r membershipWriteRequest) finish(member bool) { + r.result <- member && r.pending() +} + +func (h *Hub) authorizeWrites(identity MembershipIdentity) bool { + h.writeMembershipOnce.Do(func() { go h.runMembershipChecks() }) + ctx, cancel := context.WithTimeout(context.Background(), membershipLookupTimeout+time.Second) + defer cancel() + request := membershipWriteRequest{identity: identity, result: make(chan bool, 1), ctx: ctx} + select { + case h.writeMembership <- request: + case <-ctx.Done(): + return false + } + select { + case member := <-request.result: + return member && ctx.Err() == nil + case <-ctx.Done(): + return false + } +} + +// Coalesce queued write batches across sockets into one indexed database +// query. Close the request cohort BEFORE starting the query: joining a lookup +// already in flight could authorize newly created content with an old snapshot. +func (h *Hub) runMembershipChecks() { + // One collector fixes each cohort before passing it to a bounded pool. + // The unbuffered handoff avoids a second unbounded backlog of SQL jobs. + jobs := make(chan []membershipWriteRequest) + var workers sync.WaitGroup + for i := 0; i < membershipWorkers; i++ { + workers.Add(1) + go func() { + defer workers.Done() + for requests := range jobs { + h.checkWriteMembership(requests) + } + }() + } + for first := range h.writeMembership { + if !first.pending() { + first.finish(false) + continue + } + requests := make([]membershipWriteRequest, 0, maxMembershipBatch) + requests = append(requests, first) + timer := time.NewTimer(time.Millisecond) + collect: + for len(requests) < maxMembershipBatch { + select { + case request, open := <-h.writeMembership: + if !open { + break collect + } + if !request.pending() { + request.finish(false) + continue + } + requests = append(requests, request) + case <-timer.C: + break collect + } + } + timer.Stop() + jobs <- requests + } + close(jobs) + workers.Wait() +} + +func (h *Hub) checkWriteMembership(requests []membershipWriteRequest) { + live := requests[:0] + identities := make([]MembershipIdentity, 0, len(requests)) + seen := make(map[MembershipIdentity]bool, len(requests)) + var latestDeadline time.Time + allHaveDeadlines := true + for _, request := range requests { + if !request.pending() { + request.finish(false) + continue + } + live = append(live, request) + if !seen[request.identity] { + identities = append(identities, request.identity) + seen[request.identity] = true + } + if request.ctx == nil { + allHaveDeadlines = false + } else if deadline, ok := request.ctx.Deadline(); !ok { + allHaveDeadlines = false + } else if deadline.After(latestDeadline) { + latestDeadline = deadline + } + } + if len(live) == 0 { + return + } + // A cohort that waited for a worker gets no database time beyond the + // last live caller's budget. Expired callers never authorize a write. + deadline := time.Now().Add(membershipLookupTimeout) + if allHaveDeadlines && latestDeadline.Before(deadline) { + deadline = latestDeadline + } + h.mu.RLock() + checker := h.membership + h.mu.RUnlock() + ctx, cancel := context.WithDeadline(context.Background(), deadline) + var members map[MembershipIdentity]bool + var err error + if checker == nil { + err = errors.New("membership checker unavailable") + } else { + members, err = checker.Members(ctx, identities) + } + if err == nil { + err = ctx.Err() + } + cancel() + if err != nil { + slog.Warn("ws batch membership lookup failed", "identities", len(identities), "error", err) + } + for _, request := range live { + request.finish(err == nil && members[request.identity]) + } +} diff --git a/server/internal/realtime/membership_filter_test.go b/server/internal/realtime/membership_filter_test.go new file mode 100644 index 00000000000..aa738d982ca --- /dev/null +++ b/server/internal/realtime/membership_filter_test.go @@ -0,0 +1,48 @@ +package realtime + +import ( + "encoding/json" + "fmt" + "strings" + "testing" +) + +func membershipCommentFrame(actor string, bodyBytes int) []byte { + // Include the metadata attached by registerListeners' SubscribeAll path. + raw, err := json.Marshal(map[string]any{ + "type": "comment:created", "actor_type": actor, "actor_id": "actor-id", + "payload": map[string]string{"body": strings.Repeat("x", bodyBytes)}, + }) + if err != nil { + panic(err) + } + return raw +} + +func TestMembershipRemovalPrefilterSkipsOrdinaryActorMetadata(t *testing.T) { + for _, actor := range []string{"member", "agent"} { + raw := membershipCommentFrame(actor, 8192) + if memberRemovalCandidate(raw) { + t.Fatalf("actor_type=%q made an ordinary comment a removal candidate", actor) + } + if _, removal := parseMemberRemoval(testWorkspaceID, raw); removal { + t.Fatal("ordinary comment was misclassified as a removal") + } + } +} + +func BenchmarkMembershipOrdinaryEventRemovalFilter(b *testing.B) { + for _, n := range []int{8192, 65536} { + for _, actor := range []string{"agent", "member"} { + b.Run(fmt.Sprintf("body=%d/actor=%s", n, actor), func(b *testing.B) { + raw := membershipCommentFrame(actor, n) + b.ReportAllocs() + b.SetBytes(int64(len(raw))) + b.ResetTimer() + for i := 0; i < b.N; i++ { + parseMemberRemoval(testWorkspaceID, raw) + } + }) + } + } +} diff --git a/server/internal/realtime/membership_performance_test.go b/server/internal/realtime/membership_performance_test.go new file mode 100644 index 00000000000..8f8bd7ed360 --- /dev/null +++ b/server/internal/realtime/membership_performance_test.go @@ -0,0 +1,218 @@ +package realtime + +import ( + "context" + "fmt" + "sort" + "sync" + "sync/atomic" + "testing" + "time" + + redismock "github.com/go-redis/redismock/v9" + "github.com/redis/go-redis/v9" +) + +// Regression adapted from the independent review probe: every identity is valid and +// every database query succeeds before the configured two-second deadline. +func TestMembershipBurstDoesNotRejectValidWriters(t *testing.T) { + for _, tc := range []struct { + writers int + latency time.Duration + }{ + {1024, 5 * time.Millisecond}, + {1024, 800 * time.Millisecond}, + {4096, 200 * time.Millisecond}, + } { + t.Run(fmt.Sprintf("writers=%d/query=%s", tc.writers, tc.latency), func(t *testing.T) { + hub := NewHub() + var calls, active, peak, queryErrors atomic.Int32 + hub.SetMembershipChecker(membershipBatchOverride{ + batch: func(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + calls.Add(1) + n := active.Add(1) + defer active.Add(-1) + for p := peak.Load(); n > p; p = peak.Load() { + if peak.CompareAndSwap(p, n) { + break + } + } + timer := time.NewTimer(tc.latency) + defer timer.Stop() + select { + case <-timer.C: + case <-ctx.Done(): + queryErrors.Add(1) + return nil, ctx.Err() + } + members := make(map[MembershipIdentity]bool, len(identities)) + for _, identity := range identities { + members[identity] = true + } + return members, nil + }, + }) + workerDone := make(chan struct{}) + hub.writeMembershipOnce.Do(func() { go func() { hub.runMembershipChecks(); close(workerDone) }() }) + start := make(chan struct{}) + latencies := make([]time.Duration, tc.writers) + var denied atomic.Int32 + var wg sync.WaitGroup + wg.Add(tc.writers) + for i := 0; i < tc.writers; i++ { + go func(i int) { + defer wg.Done() + <-start + begin := time.Now() + if !hub.authorizeWrites(MembershipIdentity{WorkspaceID: fmt.Sprintf("workspace-%d", i%32), UserID: fmt.Sprintf("user-%d", i)}) { + denied.Add(1) + } + latencies[i] = time.Since(begin) + }(i) + } + close(start) + wg.Wait() + close(hub.writeMembership) + <-workerDone + sort.Slice(latencies, func(i, j int) bool { return latencies[i] < latencies[j] }) + t.Logf("valid=%d denied=%d queries=%d query_errors=%d peak_query_concurrency=%d p50=%s p95=%s max=%s", tc.writers, denied.Load(), calls.Load(), queryErrors.Load(), peak.Load(), latencies[len(latencies)/2].Round(time.Millisecond), latencies[len(latencies)*95/100].Round(time.Millisecond), latencies[len(latencies)-1].Round(time.Millisecond)) + if queryErrors.Load() != 0 { + t.Fatal("unexpected query error") + } + if denied.Load() != 0 { + t.Fatal("queue rejected valid writers") + } + if peak.Load() > membershipWorkers || (tc.latency != 5*time.Millisecond && peak.Load() < 2) { + t.Fatal("membership queries did not use bounded concurrency") + } + }) + } +} + +// A caller that already exhausted its queue budget must do no database work. +func TestMembershipExpiredQueuedRequestDoesNotQuery(t *testing.T) { + hub := NewHub() + var calls atomic.Int32 + hub.SetMembershipChecker(membershipBatchOverride{ + batch: func(_ context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + calls.Add(1) + return map[MembershipIdentity]bool{identities[0]: true}, nil + }, + }) + // Hold the worker while a request exhausts its queue budget. + hub.writeMembershipOnce.Do(func() {}) + if hub.authorizeWrites(MembershipIdentity{WorkspaceID: "workspace", UserID: "valid-user"}) { + t.Fatal("expected queue timeout") + } + close(hub.writeMembership) + hub.runMembershipChecks() + t.Logf("database queries for already expired writer: %d", calls.Load()) + if calls.Load() != 0 { + t.Fatal("expired caller triggered a database query") + } +} + +func TestMembershipCancelledRequestDoesNotEnterLiveCohort(t *testing.T) { + hub := NewHub() + cancelled, cancel := context.WithCancel(context.Background()) + cancel() + dead := membershipWriteRequest{ctx: cancelled, identity: MembershipIdentity{UserID: "cancelled"}, result: make(chan bool, 1)} + live := membershipWriteRequest{ctx: context.Background(), identity: MembershipIdentity{UserID: "current"}, result: make(chan bool, 1)} + var calls atomic.Int32 + hub.SetMembershipChecker(membershipBatchOverride{batch: func(_ context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + calls.Add(1) + if len(identities) != 1 || identities[0] != live.identity { + return nil, fmt.Errorf("cancelled request entered the SQL cohort: %v", identities) + } + return map[MembershipIdentity]bool{live.identity: true}, nil + }}) + hub.writeMembership <- dead + hub.writeMembership <- live + close(hub.writeMembership) + hub.runMembershipChecks() + if <-dead.result || !<-live.result || calls.Load() != 1 { + t.Fatal("cancellation affected another writer or caused extra SQL") + } +} + +func TestMembershipQueryUsesRemainingCallerBudget(t *testing.T) { + hub := NewHub() + caller, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + deadline, _ := caller.Deadline() + var observed time.Time + request := membershipWriteRequest{ctx: caller, identity: MembershipIdentity{UserID: "current"}, result: make(chan bool, 1)} + hub.SetMembershipChecker(membershipBatchOverride{batch: func(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + observed, _ = ctx.Deadline() + cancel() + return map[MembershipIdentity]bool{identities[0]: true}, nil + }}) + hub.writeMembership <- request + close(hub.writeMembership) + hub.runMembershipChecks() + if <-request.result || observed.IsZero() || observed.After(deadline) { + t.Fatal("query outlived its callers or granted an expired authorization") + } +} + +// Use the actual shard reader with a mocked XREAD. A workspace with no local +// subscribers should not make another workspace wait for its removal lookup. +func TestMembershipRemovalWithoutSubscribersDoesNotQuery(t *testing.T) { + hub := NewHub() + const latency = 300 * time.Millisecond + var calls atomic.Int32 + hub.SetMembershipChecker(membershipFunc(func(ctx context.Context, _, _ string) (bool, error) { + calls.Add(1) + timer := time.NewTimer(latency) + defer timer.Stop() + select { + case <-timer.C: + return false, nil + case <-ctx.Done(): + return false, ctx.Err() + } + })) + rdb, mock := redismock.NewClientMock() + t.Cleanup(func() { _ = rdb.Close() }) + relay := NewShardedStreamRelay(hub, rdb, rdb, ShardedStreamRelayConfig{}) + removedWorkspace := "workspace-with-no-local-clients" + shard := relay.shardFor(ScopeWorkspace, removedWorkspace) + var otherWorkspace string + for i := 0; ; i++ { + otherWorkspace = fmt.Sprintf("other-workspace-%d", i) + if relay.shardFor(ScopeWorkspace, otherWorkspace) == shard { + break + } + } + client := attachRealtimeTestClient(hub, ScopeWorkspace, otherWorkspace) + client.workspaceID = otherWorkspace + stream, lastID := ShardedStreamKey(shard), "0-0" + mock.ExpectXRead(&redis.XReadArgs{ + Streams: []string{stream, lastID}, Count: relay.config.ReadCount, Block: relay.config.ReadBlock, + }).SetVal([]redis.XStream{{Stream: stream, Messages: []redis.XMessage{ + {ID: "1-0", Values: envelopeRedisValues(envelope{ + EventID: "removal-event", Scope: ScopeWorkspace, ScopeID: removedWorkspace, + PayloadJSON: fmt.Sprintf(`{"type":"member:removed","payload":{"workspace_id":%q,"user_id":"removed-user"}}`, removedWorkspace), + })}, + {ID: "2-0", Values: envelopeRedisValues(envelope{ + EventID: "unrelated-update", Scope: ScopeWorkspace, ScopeID: otherWorkspace, + PayloadJSON: `{"type":"issue:updated"}`, + })}, + }}}) + finished := make(chan struct{}) + begin := time.Now() + go func() { relay.readShardOnce(context.Background(), shard, stream, &lastID); close(finished) }() + select { + case <-client.send: + t.Logf("unrelated workspace enqueue delay=%s; removal queries with no local subscriber=%d", time.Since(begin).Round(time.Millisecond), calls.Load()) + if calls.Load() != 0 { + t.Error("removal queried membership without local recipients") + } + case <-time.After(3 * time.Second): + t.Error("unrelated workspace update was not delivered") + } + <-finished + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} diff --git a/server/internal/realtime/membership_removal.go b/server/internal/realtime/membership_removal.go new file mode 100644 index 00000000000..c62e91143e0 --- /dev/null +++ b/server/internal/realtime/membership_removal.go @@ -0,0 +1,72 @@ +package realtime + +const ( + memberRemovalWorkers = 2 + memberRemovalQueueSize = 64 +) + +type memberRemovalJob struct { + workspaceID string + userID string + checker MembershipChecker + clients []*Client + notice []byte +} + +func (h *Hub) snapshotMemberRemoval(workspaceID, userID string) memberRemovalJob { + h.mu.RLock() + defer h.mu.RUnlock() + job := memberRemovalJob{workspaceID: workspaceID, userID: userID, checker: h.membership} + for client := range h.clients { + if client.workspaceID == workspaceID && client.userID == userID { + job.clients = append(job.clients, client) + } + } + return job +} + +// Removal signals are advisory invalidations. Their database revalidation +// must never block a Redis shard reader, even when this queue is saturated. +// Content writers independently enforce fresh primary-database membership. +func (h *Hub) queueMemberRemoval(workspaceID string, raw []byte, eventID string) bool { + userID, removal := parseMemberRemoval(workspaceID, raw) + if !removal { + return false + } + if userID == "" { + return true + } + // Freeze targets before fanout and queueing, not when a worker starts: + // a later registration must not inherit this signal or lookup result. + job := h.snapshotMemberRemoval(workspaceID, userID) + // Peers use member:* events to refetch the current member list. Deliver + // that invalidation in reader order, independently of lookup latency or + // queue overflow. Exclude the target: its client would navigate away even + // if the removal is stale and it has already rejoined. + h.broadcastToScope(ScopeWorkspace, workspaceID, raw, eventID, userID) + if len(job.clients) == 0 { + return true + } + job.notice = raw + h.memberRemovalOnce.Do(func() { + for i := 0; i < memberRemovalWorkers; i++ { + go h.runMemberRemovals() + } + }) + h.memberRemovalPending.Add(1) + select { + case h.memberRemovals <- job: + default: + h.memberRemovalPending.Add(-1) + h.memberRemovalDropped.Add(1) + M.MembershipRemovalDroppedTotal.Add(1) + } + return true +} + +func (h *Hub) runMemberRemovals() { + for job := range h.memberRemovals { + h.disconnectMember(job) + h.memberRemovalPending.Add(-1) + } +} diff --git a/server/internal/realtime/membership_removal_test.go b/server/internal/realtime/membership_removal_test.go new file mode 100644 index 00000000000..d01239f82a0 --- /dev/null +++ b/server/internal/realtime/membership_removal_test.go @@ -0,0 +1,283 @@ +package realtime + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "sync" + "sync/atomic" + "testing" + "time" + + redismock "github.com/go-redis/redismock/v9" + "github.com/redis/go-redis/v9" +) + +func removalFrame(workspaceID, userID string) []byte { + raw, _ := json.Marshal(map[string]any{ + "type": "member:removed", "payload": map[string]string{"workspace_id": workspaceID, "user_id": userID}, + }) + return raw +} + +func TestMembershipRemovalPeerNotificationsKeepArrivalOrder(t *testing.T) { + for _, outcome := range []string{"removed", "rejoined", "lookup error"} { + t.Run(outcome, func(t *testing.T) { + hub := NewHub() + target := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + peer := &Client{hub: hub, userID: "current-member", workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(target) + hub.registerClient(peer) + entered, release := make(chan struct{}), make(chan struct{}) + var releaseOnce sync.Once + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { + close(entered) + <-release + if outcome == "lookup error" { + return false, errors.New("database unavailable") + } + return outcome == "rejoined", nil + })) + t.Cleanup(func() { + releaseOnce.Do(func() { close(release) }) + waitFor(t, "removal completion", func() bool { return hub.memberRemovalPending.Load() == 0 }) + close(hub.memberRemovals) + hub.removeClient(target) + hub.removeClient(peer) + }) + removed := removalFrame(testWorkspaceID, testUserID) + hub.BroadcastToScopeDedup(ScopeWorkspace, testWorkspaceID, removed, "removal") + <-entered + if target.stopped.Load() || len(target.send) != 0 { + t.Fatal("target received an unverified removal") + } + // A rejoin can arrive while revocation still holds the old DB + // snapshot. Peer invalidations must retain the reader's order. + added := []byte(`{"type":"member:added"}`) + hub.BroadcastToWorkspace(testWorkspaceID, added) + for _, want := range [][]byte{removed, added} { + select { + case got := <-peer.send: + if string(got) != string(want) { + t.Fatalf("peer notification reordered: got %s, want %s", got, want) + } + default: + t.Fatal("peer notification waited for the removal query") + } + } + // A later subscriber must not inherit a deferred notification. + late := &Client{hub: hub, userID: "later-member", workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(late) + t.Cleanup(func() { hub.removeClient(late) }) + releaseOnce.Do(func() { close(release) }) + waitFor(t, "removal completion", func() bool { return hub.memberRemovalPending.Load() == 0 }) + if len(peer.send) != 0 || len(late.send) != 0 || target.stopped.Load() != (outcome != "rejoined") { + t.Fatal("removal completion replayed a notification or closed a rejoined member") + } + }) + } +} + +func TestMembershipRemovalWithOnlyPeersSkipsRevalidationAndDeduplicates(t *testing.T) { + hub := NewHub() + peer := &Client{hub: hub, userID: "current-member", workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(peer) + t.Cleanup(func() { hub.removeClient(peer) }) + var queries atomic.Int32 + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { + queries.Add(1) + return false, nil + })) + raw := removalFrame(testWorkspaceID, testUserID) + for i := 0; i < 2; i++ { + hub.BroadcastToScopeDedup(ScopeWorkspace, testWorkspaceID, raw, "removal") + } + waitFor(t, "removal completion", func() bool { return hub.memberRemovalPending.Load() == 0 }) + close(hub.memberRemovals) + if queries.Load() != 0 || len(peer.send) != 1 { + t.Fatalf("peer-only removal issued %d queries and sent %d notifications", queries.Load(), len(peer.send)) + } +} + +func TestMembershipRemovalLookupDoesNotBlockShardWithSubscribers(t *testing.T) { + hub := NewHub() + target := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(target) + other := attachRealtimeTestClient(hub, ScopeWorkspace, "other-workspace") + other.workspaceID = "other-workspace" + entered, release := make(chan struct{}, 1), make(chan struct{}) + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { + entered <- struct{}{} + <-release + return false, nil + })) + rdb, mock := redismock.NewClientMock() + t.Cleanup(func() { _ = rdb.Close() }) + relay := NewShardedStreamRelay(hub, rdb, rdb, ShardedStreamRelayConfig{Shards: 1, ReadCount: 2, ReadBlock: time.Millisecond}) + stream, lastID := ShardedStreamKey(0), "0-0" + mock.ExpectXRead(&redis.XReadArgs{Streams: []string{stream, lastID}, Count: relay.config.ReadCount, Block: relay.config.ReadBlock}).SetVal([]redis.XStream{{ + Stream: stream, Messages: []redis.XMessage{ + {ID: "1-0", Values: envelopeRedisValues(envelope{Scope: ScopeWorkspace, ScopeID: testWorkspaceID, PayloadJSON: string(removalFrame(testWorkspaceID, testUserID))})}, + {ID: "2-0", Values: envelopeRedisValues(envelope{Scope: ScopeWorkspace, ScopeID: other.workspaceID, PayloadJSON: `{"type":"issue:updated"}`})}, + }, + }}) + finished := make(chan struct{}) + go func() { relay.readShardOnce(context.Background(), 0, stream, &lastID); close(finished) }() + t.Cleanup(func() { + close(release) + <-finished + waitFor(t, "removal completion", func() bool { return hub.memberRemovalPending.Load() == 0 }) + close(hub.memberRemovals) + hub.removeClient(other) + }) + select { + case <-entered: + case <-time.After(2 * time.Second): + t.Fatal("removal lookup did not start") + } + select { + case raw := <-other.send: + if string(raw) != `{"type":"issue:updated"}` { + t.Fatalf("unexpected unrelated event: %s", raw) + } + case <-time.After(2 * time.Second): + t.Fatal("subscribed workspace removal blocked unrelated shard traffic") + } + <-finished + if lastID != "2-0" || target.stopped.Load() { + t.Fatal("reader did not advance independently of the pending lookup") + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +func TestMembershipRemovalQueueOverflowDoesNotAllowContent(t *testing.T) { + conn, peer := membershipSocketPair(t) + hub := NewHub() + target := &Client{hub: hub, conn: conn, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(target) + other := attachRealtimeTestClient(hub, ScopeWorkspace, "other-workspace") + const signals = memberRemovalWorkers + memberRemovalQueueSize + 1 + member := &Client{hub: hub, userID: "current-member", workspaceID: testWorkspaceID, send: make(chan []byte, signals)} + hub.registerClient(member) + entered, release := make(chan struct{}, memberRemovalWorkers), make(chan struct{}) + hub.SetMembershipChecker(membershipBatchOverride{ + membershipFunc: membershipFunc(func(context.Context, string, string) (bool, error) { + select { + case entered <- struct{}{}: + default: + } + <-release + return true, nil // stale removal signals must not close rejoined connections + }), + batch: func(context.Context, []MembershipIdentity) (map[MembershipIdentity]bool, error) { + return nil, nil // current content authorization sees the committed removal + }, + }) + t.Cleanup(func() { + close(release) + waitFor(t, "saturated removal queue to drain", func() bool { return hub.memberRemovalPending.Load() == 0 }) + close(hub.memberRemovals) + hub.removeClient(target) + hub.removeClient(other) + hub.removeClient(member) + }) + raw := removalFrame(testWorkspaceID, testUserID) + for i := 0; i < memberRemovalWorkers; i++ { + hub.BroadcastToWorkspace(testWorkspaceID, raw) + select { + case <-entered: + case <-time.After(2 * time.Second): + t.Fatal("removal worker did not start") + } + } + for i := 0; i < memberRemovalQueueSize+1; i++ { + hub.BroadcastToWorkspace(testWorkspaceID, raw) + } + if hub.memberRemovalDropped.Load() != 1 || hub.memberRemovalPending.Load() != memberRemovalQueueSize+memberRemovalWorkers { + t.Fatal("removal queue did not enforce its bound") + } + if len(member.send) != signals { + t.Fatalf("queue overflow lost peer invalidations: got %d, want %d", len(member.send), signals) + } + hub.BroadcastToWorkspace("other-workspace", []byte(`{"type":"issue:updated"}`)) + if raw := <-other.send; string(raw) != `{"type":"issue:updated"}` { + t.Fatalf("overflow stalled unrelated traffic: %s", raw) + } + if target.writeBatch([]byte(`{"type":"private-content"}`)) { + t.Fatal("dropped removal signal permitted revoked content") + } + wantMembershipClosed(t, peer) +} + +func TestMembershipQueuedRemovalKeepsRegistrationSnapshot(t *testing.T) { + hub := NewHub() + old := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(old) + entered, release := make(chan struct{}, memberRemovalWorkers), make(chan struct{}) + var releaseOnce sync.Once + hub.SetMembershipChecker(membershipFunc(func(_ context.Context, userID, _ string) (bool, error) { + if userID != testUserID { + entered <- struct{}{} + <-release + } + return false, nil + })) + t.Cleanup(func() { + releaseOnce.Do(func() { close(release) }) + waitFor(t, "queued removals to finish", func() bool { return hub.memberRemovalPending.Load() == 0 }) + close(hub.memberRemovals) + hub.removeClient(old) + }) + for i := 0; i < memberRemovalWorkers; i++ { + busy := &Client{hub: hub, userID: fmt.Sprintf("busy-user-%d", i), workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(busy) + t.Cleanup(func() { hub.removeClient(busy) }) + hub.BroadcastToWorkspace(testWorkspaceID, removalFrame(testWorkspaceID, busy.userID)) + <-entered + } + hub.BroadcastToWorkspace(testWorkspaceID, removalFrame(testWorkspaceID, testUserID)) + if len(hub.memberRemovals) != 1 { + t.Fatal("target removal was not queued behind the blocked workers") + } + conn, peer := membershipSocketPair(t) + current := &Client{hub: hub, conn: conn, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 4)} + hub.registerClient(current) + t.Cleanup(func() { hub.removeClient(current) }) + releaseOnce.Do(func() { close(release) }) + waitFor(t, "queued removal to be applied", func() bool { return hub.memberRemovalPending.Load() == 0 }) + if !old.stopped.Load() || current.stopped.Load() || len(current.send) != 0 { + t.Fatal("queued signal inherited a later registration") + } + // This late registration has a stale handshake. It was not in the signal's + // snapshot, but its own first content write must still deny and close it. + if current.writeBatch([]byte(`{"type":"private-content"}`)) { + t.Fatal("late registration bypassed current write authorization") + } + wantMembershipClosed(t, peer) +} + +func TestMembershipRemovalClosesResourceOnlySubscriber(t *testing.T) { + hub := NewHub() + mc := &changingMembership{} + conn := membershipConn(t, membershipServer(t, hub, mc), testUserID, testWorkspaceID) + if err := conn.WriteJSON(map[string]any{"type": "subscribe", "payload": map[string]string{"scope": ScopeTask, "id": "resource"}}); err != nil { + t.Fatal(err) + } + wantMembershipFrame(t, conn, "subscribe_ack") + if err := conn.WriteJSON(map[string]any{"type": "unsubscribe", "payload": map[string]string{"scope": ScopeWorkspace, "id": testWorkspaceID}}); err != nil { + t.Fatal(err) + } + wantMembershipFrame(t, conn, "unsubscribe_ack") + if hub.HasLocalSubscribers(ScopeWorkspace, testWorkspaceID) { + t.Fatal("workspace subscription was not removed") + } + mc.revoked.Store(true) + hub.BroadcastToWorkspace(testWorkspaceID, removalFrame(testWorkspaceID, testUserID)) + wantMembershipClosed(t, conn) + if hub.HasLocalSubscribers(ScopeTask, "resource") { + t.Fatal("revocation skipped the resource-only connection") + } +} diff --git a/server/internal/realtime/membership_test.go b/server/internal/realtime/membership_test.go new file mode 100644 index 00000000000..2acd77f217a --- /dev/null +++ b/server/internal/realtime/membership_test.go @@ -0,0 +1,646 @@ +package realtime + +import ( + "context" + "encoding/json" + "errors" + "net" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/gorilla/websocket" +) + +type changingMembership struct{ revoked atomic.Bool } + +func (m *changingMembership) IsMember(context.Context, string, string) (bool, error) { + return !m.revoked.Load(), nil +} + +func (m *changingMembership) Members(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + return membershipFunc(m.IsMember).Members(ctx, identities) +} + +// A missed revocation signal must not let a live socket receive new content. +func TestMembershipRevocationWithoutRelay(t *testing.T) { + hub := NewHub() + go hub.Run() + mc := &changingMembership{} + hub.SetMembershipChecker(mc) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + HandleWebSocket(hub, mc, nil, nil, w, r) + })) + defer server.Close() + conn := connectWS(t, server) + defer conn.Close() + waitFor(t, "workspace subscription", func() bool { return hub.HasLocalSubscribers(ScopeWorkspace, testWorkspaceID) }) + mc.revoked.Store(true) + hub.BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"comment:created","payload":{"body":"after-revocation"}}`)) + wantMembershipClosed(t, conn) +} + +type membershipFunc func(context.Context, string, string) (bool, error) + +func (f membershipFunc) IsMember(ctx context.Context, userID, workspaceID string) (bool, error) { + return f(ctx, userID, workspaceID) +} + +func (f membershipFunc) Members(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + members := make(map[MembershipIdentity]bool, len(identities)) + for _, identity := range identities { + member, err := f(ctx, identity.UserID, identity.WorkspaceID) + if err != nil { + return nil, err + } + members[identity] = member + } + return members, nil +} + +func membershipServer(t *testing.T, hub *Hub, mc MembershipChecker) *httptest.Server { + t.Helper() + hub.SetMembershipChecker(mc) + go hub.Run() + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + HandleWebSocket(hub, mc, nil, nil, w, r) + })) + t.Cleanup(server.Close) + return server +} + +func membershipConn(t *testing.T, server *httptest.Server, userID, workspaceID string) *websocket.Conn { + t.Helper() + conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http")+"/?workspace_id="+workspaceID, nil) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { conn.Close() }) + if err := conn.WriteJSON(map[string]any{"type": "auth", "payload": map[string]string{"token": makeTestTokenForUser(t, userID, "")}}); err != nil { + t.Fatal(err) + } + wantMembershipFrame(t, conn, "auth_ack") + // A ping response goes through the registered client's writer. + if err := conn.WriteJSON(map[string]string{"type": "ping"}); err != nil { + t.Fatal(err) + } + wantMembershipFrame(t, conn, "pong") + return conn +} + +func wantMembershipFrame(t *testing.T, conn *websocket.Conn, typ string) { + t.Helper() + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + _, raw, err := conn.ReadMessage() + var frame struct{ Type string } + if err != nil || json.Unmarshal(raw, &frame) != nil || frame.Type != typ { + t.Fatalf("want %s, got %s, err=%v", typ, raw, err) + } +} + +func wantMembershipClosed(t *testing.T, conn *websocket.Conn) { + t.Helper() + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + for { + _, raw, err := conn.ReadMessage() + if err != nil { + var netErr net.Error + if errors.As(err, &netErr) && netErr.Timeout() { + t.Fatalf("revoked socket remained open: %v", err) + } + return + } + var frame struct{ Type string } + if json.Unmarshal(raw, &frame) != nil || frame.Type != "member:removed" { + t.Fatalf("revoked socket received content: %s", raw) + } + } +} + +func TestMembershipRemovalAcrossHubsAndScopes(t *testing.T) { + var revoked atomic.Bool + mc := membershipFunc(func(_ context.Context, userID, workspaceID string) (bool, error) { + return !(revoked.Load() && userID == testUserID && workspaceID == testWorkspaceID), nil + }) + local, remote := NewHub(), NewHub() + localServer := membershipServer(t, local, mc) + remoteServer := membershipServer(t, remote, mc) + oldLocal := membershipConn(t, localServer, testUserID, testWorkspaceID) + oldRemote := membershipConn(t, remoteServer, testUserID, testWorkspaceID) + secondRemote := membershipConn(t, remoteServer, testUserID, testWorkspaceID) + current := membershipConn(t, remoteServer, "current-user", testWorkspaceID) + otherWS := membershipConn(t, remoteServer, testUserID, "other-workspace") + for _, scope := range []string{ScopeTask, ScopeChat} { + if err := oldRemote.WriteJSON(map[string]any{"type": "subscribe", "payload": map[string]string{"scope": scope, "id": "resource-id"}}); err != nil { + t.Fatal(err) + } + wantMembershipFrame(t, oldRemote, "subscribe_ack") + } + revoked.Store(true) + local.DisconnectMember(testWorkspaceID, testUserID, "member-id") + removed := `{"type":"member:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}` + // Exercise the exact common receiver used by legacy, sharded and mirrored + // relays, including duplicate processing on another node. + for i := 0; i < 2; i++ { + deliverEnvelope(remote, nil, nil, envelope{Scope: ScopeWorkspace, ScopeID: testWorkspaceID, PayloadJSON: removed, EventID: "removal"}) + } + for _, conn := range []*websocket.Conn{oldLocal, oldRemote, secondRemote} { + wantMembershipClosed(t, conn) + } + for _, scope := range []string{ScopeTask, ScopeChat} { + if remote.HasLocalSubscribers(scope, "resource-id") { + t.Fatalf("revocation left %s subscription", scope) + } + } + wantMembershipFrame(t, current, "member:removed") + remote.BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"comment:created"}`)) + remote.BroadcastToWorkspace("other-workspace", []byte(`{"type":"issue:updated"}`)) + wantMembershipFrame(t, current, "comment:created") + wantMembershipFrame(t, otherWS, "issue:updated") +} + +type failedMembershipPublisher struct{} + +func (failedMembershipPublisher) PublishWithID(string, string, string, []byte, string) error { + return errors.New("relay unavailable") +} + +func TestMembershipRelayFailureAndLookupError(t *testing.T) { + for _, lookupError := range []bool{false, true} { + t.Run(map[bool]string{false: "revoked", true: "database unavailable"}[lookupError], func(t *testing.T) { + var failed atomic.Bool + mc := membershipFunc(func(context.Context, string, string) (bool, error) { + if !failed.Load() { + return true, nil + } + if lookupError { + return true, errors.New("database unavailable") + } + return false, nil + }) + hub := NewHub() + conn := membershipConn(t, membershipServer(t, hub, mc), testUserID, testWorkspaceID) + failed.Store(true) + // The remote node processes a new local event while the removal + // signal is missing and Redis publication also fails. + NewDualWriteBroadcaster(hub, failedMembershipPublisher{}).BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"issue:updated"}`)) + wantMembershipClosed(t, conn) + }) + } +} + +func TestMembershipRemovalReplayAfterRejoin(t *testing.T) { + hub := NewHub() + mc := &mockMembershipChecker{} + conn := membershipConn(t, membershipServer(t, hub, mc), testUserID, testWorkspaceID) + removed := `{"type":"member:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}` + deliverEnvelope(hub, nil, nil, envelope{Scope: ScopeWorkspace, ScopeID: testWorkspaceID, PayloadJSON: removed}) + waitFor(t, "stale removal to be checked", func() bool { return hub.memberRemovalPending.Load() == 0 }) + hub.BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"issue:updated"}`)) + // Neither a hard close nor a stale frontend removal notification is valid + // when the primary database says the user has rejoined. + wantMembershipFrame(t, conn, "issue:updated") +} + +func TestMembershipReplayCannotRestoreRevokedAccess(t *testing.T) { + hub := NewHub() + mc := &changingMembership{} + conn := membershipConn(t, membershipServer(t, hub, mc), testUserID, testWorkspaceID) + mc.revoked.Store(true) + deliverEnvelope(hub, nil, nil, envelope{ + Scope: ScopeWorkspace, ScopeID: testWorkspaceID, + PayloadJSON: `{"type":"comment:created","payload":{"body":"retained-content"}}`, + }) + wantMembershipClosed(t, conn) +} + +func TestMembershipRegistrationDoesNotQueryDatabase(t *testing.T) { + hub := NewHub() + var queries atomic.Int32 + checker := membershipFunc(func(context.Context, string, string) (bool, error) { + queries.Add(1) + return false, errors.New("database unavailable") + }) + hub.SetMembershipChecker(checker) + go hub.Run() + clients := make([]*Client, 20) + for i := range clients { + clients[i] = &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.register <- clients[i] + t.Cleanup(func() { hub.removeClient(clients[i]) }) + } + // A later event is a barrier for all registrations in the single loop. + hub.unregister <- &Client{} + if queries.Load() != 0 { + t.Fatalf("registration issued %d database queries in the Hub event loop", queries.Load()) + } + if totalClients(hub) != len(clients) { + t.Fatal("database availability affected in-memory registration") + } + hub.unregister <- clients[0] + hub.Broadcast([]byte(`{"type":"hub-progress"}`)) + if _, open := <-clients[0].send; open { + t.Fatal("event loop did not clean up the disconnected client") + } + for _, client := range clients[1:] { + select { + case frame := <-client.send: + if string(frame) != `{"type":"hub-progress"}` { + t.Fatalf("unexpected broadcast: %s", frame) + } + case <-time.After(2 * time.Second): + t.Fatal("event loop did not deliver the global broadcast") + } + } +} + +func TestMembershipStaleHandshakeCannotReceiveContent(t *testing.T) { + hub := NewHub() + var checks atomic.Int32 + mc := membershipFunc(func(context.Context, string, string) (bool, error) { + // The handshake succeeded before the simulated revocation; all later + // queries see the committed absence. + return checks.Add(1) == 1, nil + }) + server := membershipServer(t, hub, mc) + conn := connectWS(t, server) + defer conn.Close() + waitFor(t, "stale handshake registration", func() bool { return totalClients(hub) == 1 }) + if checks.Load() != 1 { + t.Fatal("registration rechecked membership instead of leaving it to the writer") + } + hub.BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"comment:created","payload":{"body":"after-revocation"}}`)) + wantMembershipClosed(t, conn) + waitFor(t, "revoked connection cleanup", func() bool { return totalClients(hub) == 0 }) + if totalClients(hub) != 0 || hub.HasLocalSubscribers(ScopeWorkspace, testWorkspaceID) { + t.Fatal("stale handshake left a registered connection") + } +} + +func TestMembershipRegistrationAfterRemovalCannotReceiveContent(t *testing.T) { + conn, peer := membershipSocketPair(t) + hub := NewHub() + checker := membershipFunc(func(context.Context, string, string) (bool, error) { + return false, nil + }) + hub.SetMembershipChecker(checker) + // The handshake passed earlier, but registration reached this node after + // revocation already took its connection snapshot and completed. + hub.DisconnectMember(testWorkspaceID, testUserID, "member-id") + client := &Client{hub: hub, conn: conn, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.registerClient(client) + if totalClients(hub) != 1 { + t.Fatal("connection was not registered in memory") + } + if client.writeBatch([]byte(`{"type":"after-revocation"}`)) { + t.Fatal("late registration received content after revocation") + } + wantMembershipClosed(t, peer) + if totalClients(hub) != 0 || hub.HasLocalSubscribers(ScopeWorkspace, testWorkspaceID) || !client.stopped.Load() { + t.Fatal("write-time denial left a registered connection") + } +} + +func TestMembershipRegistrationProgressDuringRemovalLookup(t *testing.T) { + hub := NewHub() + entered, release := make(chan struct{}), make(chan struct{}) + var queries atomic.Int32 + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { + if queries.Add(1) == 1 { + close(entered) + } + <-release + return false, nil + })) + old := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.registerClient(old) + go hub.Run() + removed := make(chan struct{}) + go func() { hub.DisconnectMember(testWorkspaceID, testUserID, "member-id"); close(removed) }() + <-entered + clients := make([]*Client, 20) + progress := make(chan struct{}) + go func() { + for i := range clients { + clients[i] = &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.register <- clients[i] + } + hub.unregister <- clients[0] + hub.Broadcast([]byte(`{"type":"hub-progress"}`)) + close(progress) + }() + t.Cleanup(func() { + close(release) + <-removed + <-progress + for _, client := range clients { + hub.removeClient(client) + } + }) + // The lookup remains blocked until cleanup; completion cannot depend on + // its latency or on database availability. + select { + case <-progress: + case <-time.After(2 * time.Second): + t.Fatal("removal lookup blocked registration, cleanup or global broadcast") + } + if queries.Load() != 1 { + t.Fatal("registrations issued extra membership lookups") + } + if _, open := <-clients[0].send; open { + t.Fatal("disconnected client was not cleaned up while the lookup was blocked") + } + for _, client := range clients[1:] { + select { + case frame := <-client.send: + if string(frame) != `{"type":"hub-progress"}` { + t.Fatalf("unexpected broadcast: %s", frame) + } + case <-time.After(2 * time.Second): + t.Fatal("broadcast did not progress while the lookup was blocked") + } + } +} + +func TestMembershipRemovalLookupDoesNotAffectLaterRejoin(t *testing.T) { + hub := NewHub() + old := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + other := &Client{hub: hub, userID: "current-user", workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.registerClient(old) + hub.registerClient(other) + t.Cleanup(func() { hub.removeClient(other) }) + entered, release := make(chan struct{}), make(chan struct{}) + var rejoined atomic.Bool + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { + member := rejoined.Load() + close(entered) + <-release + return member, nil + })) + removed := []byte(`{"type":"member:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`) + done := make(chan struct{}) + go func() { hub.BroadcastToWorkspace(testWorkspaceID, removed); close(done) }() + <-entered + // The database now has a fresh membership, while the removal query still + // holds the old absence snapshot. Its result must apply only to old. + rejoined.Store(true) + current := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.registerClient(current) + t.Cleanup(func() { hub.removeClient(current) }) + close(release) + <-done + waitFor(t, "removal lookup and fanout", func() bool { return hub.memberRemovalPending.Load() == 0 }) + if !old.stopped.Load() || current.stopped.Load() || totalClients(hub) != 2 { + t.Fatal("removal result affected a connection registered after its snapshot") + } + select { + case frame := <-other.send: + if string(frame) != string(removed) { + t.Fatalf("remaining member missed removal notification: %s", frame) + } + case <-time.After(2 * time.Second): + t.Fatal("remaining member missed removal notification") + } + select { + case frame := <-current.send: + t.Fatalf("rejoined member received an old frontend removal notification: %s", frame) + default: + } + hub.BroadcastToWorkspace(testWorkspaceID, []byte(`{"type":"issue:updated"}`)) + select { + case frame := <-current.send: + if string(frame) != `{"type":"issue:updated"}` { + t.Fatalf("rejoined connection could not receive new updates: %s", frame) + } + case <-time.After(2 * time.Second): + t.Fatal("rejoined connection could not receive new updates") + } +} + +func TestMembershipRemovalCandidateFilter(t *testing.T) { + for _, tt := range []struct { + name string + raw string + excluded string + removal bool + }{ + {"ordinary event", `{"type":"issue:updated"}`, "", false}, + {"marker in content", `{"type":"comment:created","payload":{"body":"member:removed"}}`, "", false}, + {"removal", `{"type":"member:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`, testUserID, true}, + {"escaped type", `{"type":"member:\u0072emoved","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`, testUserID, true}, + {"escaped separator", `{"type":"member\u003aremoved","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`, testUserID, true}, + {"escaped prefix", `{"type":"m\u0065mber:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`, testUserID, true}, + {"fully escaped prefix", `{"type":"\u006d\u0065\u006d\u0062\u0065\u0072:removed","payload":{"workspace_id":"test-workspace","user_id":"test-user"}}`, testUserID, true}, + {"wrong workspace", `{"type":"member:removed","payload":{"workspace_id":"other-workspace","user_id":"test-user"}}`, "", true}, + {"missing user", `{"type":"member:removed","payload":{"workspace_id":"test-workspace"}}`, "", true}, + } { + t.Run(tt.name, func(t *testing.T) { + excluded, removal := parseMemberRemoval(testWorkspaceID, []byte(tt.raw)) + if excluded != tt.excluded || removal != tt.removal { + t.Fatalf("got target=%q removal=%v, want %q %v", excluded, removal, tt.excluded, tt.removal) + } + }) + } +} + +func TestMembershipRemovalPrefilterSkipsEscapedCommentContent(t *testing.T) { + raw, err := json.Marshal(map[string]any{ + "type": "comment:created", + "payload": map[string]string{"body": "Markdown: `
x > y && y < z
`\u2028\u2029"}, + }) + if err != nil { + t.Fatal(err) + } + if memberRemovalCandidate(raw) { + t.Fatalf("HTML and line-separator escapes made an ordinary comment a removal candidate: %s", raw) + } +} + +func TestMembershipRegistrationDoesNotRestoreStoppedClient(t *testing.T) { + hub := NewHub() + client := &Client{hub: hub, userID: testUserID, workspaceID: testWorkspaceID, send: make(chan []byte, 2)} + hub.stopClient(client, nil, time.Time{}) + hub.registerClient(client) + if totalClients(hub) != 0 || hub.HasLocalSubscribers(ScopeWorkspace, testWorkspaceID) { + t.Fatal("registration restored an already stopped connection") + } +} + +// A real socket pair lets tests control the queue and lookup without timing +// assumptions about a concurrently running writePump. +func membershipSocketPair(t *testing.T) (*websocket.Conn, *websocket.Conn) { + t.Helper() + accepted := make(chan *websocket.Conn, 1) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := upgrader.Upgrade(w, r, nil) + if err == nil { + accepted <- conn + } + })) + t.Cleanup(server.Close) + peer, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http"), nil) + if err != nil { + t.Fatal(err) + } + conn := <-accepted + t.Cleanup(func() { conn.Close(); peer.Close() }) + return conn, peer +} + +func TestMembershipWriteBatchDoesNotCacheAuthorization(t *testing.T) { + conn, peer := membershipSocketPair(t) + hub := NewHub() + var revoked atomic.Bool + var lookups atomic.Int32 + entered, release := make(chan struct{}), make(chan struct{}) + mc := membershipFunc(func(context.Context, string, string) (bool, error) { + member := !revoked.Load() + if lookups.Add(1) == 1 { + close(entered) + <-release + } + return member, nil + }) + client := &Client{hub: hub, conn: conn, send: make(chan []byte, 8), userID: testUserID, workspaceID: testWorkspaceID} + hub.SetMembershipChecker(mc) + for i := 0; i < 3; i++ { + client.send <- []byte(`{"type":"before-revocation"}`) + } + done := make(chan bool, 1) + go func() { done <- client.writeBatch(<-client.send) }() + <-entered + revoked.Store(true) + client.send <- []byte(`{"type":"after-revocation"}`) + close(release) + if !<-done { + t.Fatal("previously authorized batch failed") + } + for i := 0; i < 3; i++ { + wantMembershipFrame(t, peer, "before-revocation") + } + if lookups.Load() != 1 { + t.Fatal("queued frames were checked individually instead of as a batch") + } + if client.writeBatch(<-client.send) { + t.Fatal("new content reused the previous batch's authorization") + } + wantMembershipClosed(t, peer) +} + +func TestMembershipHardCloseDropsCapturedAndQueuedFrames(t *testing.T) { + conn, peer := membershipSocketPair(t) + hub := NewHub() + var revoked atomic.Bool + hub.SetMembershipChecker(membershipFunc(func(context.Context, string, string) (bool, error) { return !revoked.Load(), nil })) + entered, release := make(chan struct{}), make(chan struct{}) + client := &Client{hub: hub, conn: conn, send: make(chan []byte, 4), userID: testUserID, workspaceID: testWorkspaceID} + batchChecker := membershipFunc(func(context.Context, string, string) (bool, error) { + close(entered) + <-release + return true, nil + }) + hub.SetMembershipChecker(membershipBatchOverride{ + membershipFunc: membershipFunc(func(context.Context, string, string) (bool, error) { return !revoked.Load(), nil }), + batch: batchChecker.Members, + }) + hub.clients[client] = true + hub.subscribe(client, ScopeWorkspace, testWorkspaceID) + done := make(chan bool, 1) + go func() { done <- client.writeBatch([]byte(`{"type":"captured-secret"}`)) }() + <-entered + client.send <- []byte(`{"type":"queued-secret"}`) + revoked.Store(true) + hub.DisconnectMember(testWorkspaceID, testUserID, "member-id") + close(release) + if <-done { + t.Fatal("stopped writer continued after its membership lookup") + } + wantMembershipClosed(t, peer) + if _, open := <-client.send; open { + t.Fatal("revoked send queue remained open") + } +} + +type membershipBatchOverride struct { + membershipFunc + batch func(context.Context, []MembershipIdentity) (map[MembershipIdentity]bool, error) +} + +func (m membershipBatchOverride) Members(ctx context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + return m.batch(ctx, identities) +} + +func TestMembershipWriteCohortDoesNotJoinInflightLookup(t *testing.T) { + hub := NewHub() + var revoked atomic.Bool + var calls atomic.Int32 + entered, release := make(chan struct{}), make(chan struct{}) + mc := membershipBatchOverride{ + membershipFunc: membershipFunc(func(context.Context, string, string) (bool, error) { return !revoked.Load(), nil }), + batch: func(_ context.Context, identities []MembershipIdentity) (map[MembershipIdentity]bool, error) { + member := !revoked.Load() + if calls.Add(1) == 1 { + close(entered) + <-release + } + return map[MembershipIdentity]bool{identities[0]: member}, nil + }, + } + hub.SetMembershipChecker(mc) + identity := MembershipIdentity{WorkspaceID: testWorkspaceID, UserID: testUserID} + first, second := make(chan bool, 1), make(chan bool, 1) + go func() { first <- hub.authorizeWrites(identity) }() + <-entered + revoked.Store(true) + go func() { second <- hub.authorizeWrites(identity) }() + if <-second { + close(release) + t.Fatal("post-revocation content joined an in-flight membership snapshot") + } + close(release) + if !<-first || calls.Load() != 2 { + t.Fatal("post-revocation content reused an in-flight membership snapshot") + } +} + +func TestMembershipBatchCoalescesAndDeduplicatesIdentities(t *testing.T) { + hub := NewHub() + var calls atomic.Int32 + identities := []MembershipIdentity{ + {WorkspaceID: "one", UserID: "a"}, + {WorkspaceID: "one", UserID: "b"}, + {WorkspaceID: "two", UserID: "a"}, + } + hub.SetMembershipChecker(membershipBatchOverride{ + batch: func(_ context.Context, requested []MembershipIdentity) (map[MembershipIdentity]bool, error) { + calls.Add(1) + if len(requested) != len(identities) { + return nil, errors.New("identities were not deduplicated") + } + return map[MembershipIdentity]bool{identities[0]: true, identities[2]: true}, nil + }, + }) + requests := make([]membershipWriteRequest, 12) + for i := range requests { + requests[i] = membershipWriteRequest{identity: identities[i%3], result: make(chan bool, 1)} + hub.writeMembership <- requests[i] + } + go hub.runMembershipChecks() + for _, request := range requests { + select { + case allowed := <-request.result: + if allowed != (request.identity.UserID == "a") { + t.Fatal("batched membership was applied to the wrong user/workspace") + } + case <-time.After(2 * time.Second): + t.Fatal("batch did not complete") + } + } + close(hub.writeMembership) + if calls.Load() != 1 { + t.Fatal("one queued wave was checked with multiple queries") + } +} diff --git a/server/internal/realtime/metrics.go b/server/internal/realtime/metrics.go index 3983cc40b18..80fa4b44f5d 100644 --- a/server/internal/realtime/metrics.go +++ b/server/internal/realtime/metrics.go @@ -18,6 +18,10 @@ type Metrics struct { SlowEvictionsTotal atomic.Int64 MessagesSentTotal atomic.Int64 MessagesDroppedTotal atomic.Int64 + // MembershipRemovalDroppedTotal counts target revalidations discarded when + // the bounded queue is full. Peer invalidation and content authorization + // still run. + MembershipRemovalDroppedTotal atomic.Int64 // InboundTooLargeTotal counts connections closed because a peer sent a // message over inboundReadLimit, on either the pre-auth or the @@ -199,18 +203,19 @@ func (m *Metrics) Snapshot() map[string]any { nodeID, _ = v.(string) } return map[string]any{ - "connects_total": m.ConnectsTotal.Load(), - "disconnects_total": m.DisconnectsTotal.Load(), - "active_connections": m.ActiveConnections.Load(), - "slow_evictions_total": m.SlowEvictionsTotal.Load(), - "messages_sent_total": m.MessagesSentTotal.Load(), - "messages_dropped_total": m.MessagesDroppedTotal.Load(), - "inbound_too_large_total": m.InboundTooLargeTotal.Load(), - "events_sent_by_type": snapshotCounters(&m.eventSent), - "subscribes_total": snapshotCounters(&m.subscribeTotal), - "unsubscribes_total": snapshotCounters(&m.unsubscribeTotal), - "subscribe_denied_total": snapshotCounters(&m.subscribeDeniedTotal), - "active_scope_rooms": snapshotCounters(&m.scopeRooms), + "connects_total": m.ConnectsTotal.Load(), + "disconnects_total": m.DisconnectsTotal.Load(), + "active_connections": m.ActiveConnections.Load(), + "slow_evictions_total": m.SlowEvictionsTotal.Load(), + "messages_sent_total": m.MessagesSentTotal.Load(), + "messages_dropped_total": m.MessagesDroppedTotal.Load(), + "membership_removals_dropped_total": m.MembershipRemovalDroppedTotal.Load(), + "inbound_too_large_total": m.InboundTooLargeTotal.Load(), + "events_sent_by_type": snapshotCounters(&m.eventSent), + "subscribes_total": snapshotCounters(&m.subscribeTotal), + "unsubscribes_total": snapshotCounters(&m.unsubscribeTotal), + "subscribe_denied_total": snapshotCounters(&m.subscribeDeniedTotal), + "active_scope_rooms": snapshotCounters(&m.scopeRooms), "redis": map[string]any{ "connected": m.RedisConnected.Load(), "node_id": nodeID, @@ -238,6 +243,7 @@ func (m *Metrics) Snapshot() map[string]any { // Reset zeroes all counters. Tests only. func (m *Metrics) Reset() { + m.MembershipRemovalDroppedTotal.Store(0) m.ConnectsTotal.Store(0) m.DisconnectsTotal.Store(0) m.ActiveConnections.Store(0) diff --git a/server/pkg/db/generated/member.sql.go b/server/pkg/db/generated/member.sql.go index 8f13d782cde..ec381f1147a 100644 --- a/server/pkg/db/generated/member.sql.go +++ b/server/pkg/db/generated/member.sql.go @@ -86,6 +86,46 @@ func (q *Queries) GetMemberByUserAndWorkspace(ctx context.Context, arg GetMember return i, err } +const getRealtimeMemberships = `-- name: GetRealtimeMemberships :many +SELECT m.user_id, m.workspace_id +FROM member m +JOIN ( + SELECT unnest($1::uuid[]) AS user_id, + unnest($2::uuid[]) AS workspace_id +) AS requested + ON m.user_id = requested.user_id AND m.workspace_id = requested.workspace_id +` + +type GetRealtimeMembershipsParams struct { + UserIds []pgtype.UUID `json:"user_ids"` + WorkspaceIds []pgtype.UUID `json:"workspace_ids"` +} + +type GetRealtimeMembershipsRow struct { + UserID pgtype.UUID `json:"user_id"` + WorkspaceID pgtype.UUID `json:"workspace_id"` +} + +func (q *Queries) GetRealtimeMemberships(ctx context.Context, arg GetRealtimeMembershipsParams) ([]GetRealtimeMembershipsRow, error) { + rows, err := q.db.Query(ctx, getRealtimeMemberships, arg.UserIds, arg.WorkspaceIds) + if err != nil { + return nil, err + } + defer rows.Close() + items := []GetRealtimeMembershipsRow{} + for rows.Next() { + var i GetRealtimeMembershipsRow + if err := rows.Scan(&i.UserID, &i.WorkspaceID); err != nil { + return nil, err + } + items = append(items, i) + } + if err := rows.Err(); err != nil { + return nil, err + } + return items, nil +} + const listMembers = `-- name: ListMembers :many SELECT id, workspace_id, user_id, role, created_at FROM member WHERE workspace_id = $1 diff --git a/server/pkg/db/queries/member.sql b/server/pkg/db/queries/member.sql index 74e1dc70ab8..98c1f945c7a 100644 --- a/server/pkg/db/queries/member.sql +++ b/server/pkg/db/queries/member.sql @@ -16,6 +16,15 @@ WHERE id = $1; SELECT * FROM member WHERE user_id = $1 AND workspace_id = $2; +-- name: GetRealtimeMemberships :many +SELECT m.user_id, m.workspace_id +FROM member m +JOIN ( + SELECT unnest(sqlc.arg(user_ids)::uuid[]) AS user_id, + unnest(sqlc.arg(workspace_ids)::uuid[]) AS workspace_id +) AS requested + ON m.user_id = requested.user_id AND m.workspace_id = requested.workspace_id; + -- name: CreateMember :one INSERT INTO member (workspace_id, user_id, role) VALUES ($1, $2, $3) From 83422dc262e38edc0b1363494f22aa753dad66a8 Mon Sep 17 00:00:00 2001 From: YYClaw <197375+yyclaw@users.noreply.github.com> Date: Thu, 15 Oct 2026 13:50:03 +0800 Subject: [PATCH 04/51] MUL-7464: fix(agents): recover completed builder drafts and align recovery feedback (#8514) * fix(agents): recover completed builder drafts and align recovery feedback * fix(agents): reject array builder draft payloads --- packages/core/agents/builder-protocol.test.ts | 64 +++++ packages/core/agents/builder-protocol.ts | 32 ++- .../agents/create/builder-conversation.tsx | 18 +- .../agents/create/builder-workspace.test.tsx | 224 ++++++++++++++++++ .../views/agents/create/builder-workspace.tsx | 27 ++- packages/views/locales/en/agents.json | 1 + packages/views/locales/fr/agents.json | 1 + packages/views/locales/ja/agents.json | 1 + packages/views/locales/ko/agents.json | 1 + packages/views/locales/zh-Hans/agents.json | 1 + 10 files changed, 352 insertions(+), 18 deletions(-) create mode 100644 packages/views/agents/create/builder-workspace.test.tsx diff --git a/packages/core/agents/builder-protocol.test.ts b/packages/core/agents/builder-protocol.test.ts index 0293c508241..a95c5c5005f 100644 --- a/packages/core/agents/builder-protocol.test.ts +++ b/packages/core/agents/builder-protocol.test.ts @@ -1,3 +1,5 @@ +// @vitest-environment node + import { describe, expect, it } from "vitest"; import { decodeBuilderInput, @@ -26,6 +28,19 @@ const draft = (): AgentDraft => ({ }); describe("agent builder protocol", () => { + it.each([ + "[]", + '[{"name":"Reviewer"}]', + '[{"instructions":"First line\nSecond line"}]', + ])("rejects array payloads: %j", (payload) => { + expect( + parseBuilderDraft(`${payload}`), + ).toBeNull(); + expect( + parseBuilderDraft(`${payload}`, { completed: true }), + ).toBeNull(); + }); + it("parses and hides the structured draft block", () => { const content = 'Here is a first draft.\n{"name":"Researcher","permission_scope":"workspace"}'; @@ -293,3 +308,52 @@ Return findings."}`; }); }); }); + +describe("completed builder reply recovery", () => { + const reply = 'Here is your draft.\n{"name":"Poet","instructions":"# Role\\nWrite poetry."}'; + + it("recovers a complete JSON object only after the reply completes", () => { + expect(parseBuilderDraft(reply)).toBeNull(); + expect(parseBuilderDraft(reply, { completed: false })).toBeNull(); + expect(parseBuilderDraft(reply, { completed: true })).toEqual({ + name: "Poet", + instructions: "# Role\nWrite poetry.", + }); + }); + + it("recovers an unclosed completed draft with literal string control characters", () => { + const content = '{"name":"诗仙李白","instructions":"第一行\n\n## 交流方式\r\n- 以第一人称对话。\t保持诗意。","permission_scope":"private"}'; + expect(parseBuilderDraft(content)).toBeNull(); + expect(parseBuilderDraft(content, { completed: false })).toBeNull(); + expect(parseBuilderDraft(content, { completed: true })).toEqual({ + name: "诗仙李白", + instructions: "第一行\n\n## 交流方式\r\n- 以第一人称对话。\t保持诗意。", + permission_scope: "private", + }); + }); + + it.each([ + '{"name":"Poet"', + '{"instructions":"line one\nline two"', + '{"instructions":"line one\nline two"} trailing prose', + '{"name":"Poet",}', + '{"name":"Poet"} trailing prose', + '{"name":"Poet"}{"name":"Other"}', + '[]', + '', + '', + '{"name":"Poet"}null', + '"Poet"', + '{"name":"Poet"}', + ])("rejects malformed or ambiguous completed drafts: %s", (content) => { + expect(parseBuilderDraft(content, { completed: true })).toBeNull(); + }); + + it("preserves closed-block parsing and literal-newline repair", () => { + expect(parseBuilderDraft('{"instructions":"line one\nline two"}')) + .toEqual({ instructions: "line one\nline two" }); + expect(parseBuilderDraft(`${reply}`, { completed: true })) + .toEqual(parseBuilderDraft(`${reply}`)); + }); +}); diff --git a/packages/core/agents/builder-protocol.ts b/packages/core/agents/builder-protocol.ts index d072db9da1d..abfe887495d 100644 --- a/packages/core/agents/builder-protocol.ts +++ b/packages/core/agents/builder-protocol.ts @@ -34,14 +34,32 @@ export interface BuilderDraftPayload { member_ids?: unknown; } -export function parseBuilderDraft(content: string): BuilderDraftPayload | null { +function isBuilderDraftPayload(value: unknown): value is BuilderDraftPayload { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +export function parseBuilderDraft( + content: string, + { completed = false }: { completed?: boolean } = {}, +): BuilderDraftPayload | null { const match = content.match(/([\s\S]*?)<\/agent_draft>/); - if (!match?.[1]) return null; + if (!match?.[1]) { + // Completed CLI replies can omit the closing tag and include literal + // string control characters. Apply the same repair as closed blocks, + // but never recover a live stream or truncate trailing text. + if (!completed || content.includes("")) return null; + const parts = content.split(""); + if (parts.length !== 2 || !parts[1]) return null; + try { + const value = JSON.parse(escapeJsonStringControlCharacters(parts[1])); + return isBuilderDraftPayload(value) ? value : null; + } catch { + return null; + } + } try { const value = JSON.parse(match[1]); - return value && typeof value === "object" - ? (value as BuilderDraftPayload) - : null; + return isBuilderDraftPayload(value) ? value : null; } catch { // Some CLI-backed models emit literal newlines in the Markdown // instructions string even when asked for compact JSON. Repair only JSON @@ -49,9 +67,7 @@ export function parseBuilderDraft(content: string): BuilderDraftPayload | null { // other syntax still have to pass JSON.parse. try { const value = JSON.parse(escapeJsonStringControlCharacters(match[1])); - return value && typeof value === "object" - ? (value as BuilderDraftPayload) - : null; + return isBuilderDraftPayload(value) ? value : null; } catch { return null; } diff --git a/packages/views/agents/create/builder-conversation.tsx b/packages/views/agents/create/builder-conversation.tsx index 2ac44698f43..711e1d78a22 100644 --- a/packages/views/agents/create/builder-conversation.tsx +++ b/packages/views/agents/create/builder-conversation.tsx @@ -18,6 +18,7 @@ import type { import { Button } from "@multica/ui/components/ui/button"; import { cn } from "@multica/ui/lib/utils"; import { ChatInput } from "../../chat/components/chat-input"; +import { CHAT_COLUMN, CHAT_GUTTER } from "../../chat/components/chat-column"; import { ChatMessageList, ChatMessageSkeleton, @@ -228,12 +229,17 @@ export function BuilderConversation({ )} {error ? ( -
- {error} +
+
+ {error} +
) : null} diff --git a/packages/views/agents/create/builder-workspace.test.tsx b/packages/views/agents/create/builder-workspace.test.tsx new file mode 100644 index 00000000000..3320b4335c7 --- /dev/null +++ b/packages/views/agents/create/builder-workspace.test.tsx @@ -0,0 +1,224 @@ +// @vitest-environment jsdom + +import { useState, type ReactNode } from "react"; +import { cleanup, fireEvent, screen, waitFor } from "@testing-library/react"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { EMPTY_AGENT_DRAFT, toStoredAgentDraft, type AgentDraft } from "@multica/core/agents"; +import type { AgentBuilderSessionSummary, ChatMessage } from "@multica/core/types"; +import type { SupportedLocale } from "@multica/core/i18n"; +import { renderWithI18n } from "../../test/i18n"; +import { BuilderWorkspace } from "./builder-workspace"; + +const h = vi.hoisted(() => ({ + messages: [] as ChatMessage[], + pending: false, + loading: false, + settled: true, + error: null as string | null, + session: undefined as AgentBuilderSessionSummary | undefined, + save: vi.fn().mockResolvedValue(undefined), +})); +vi.mock("@multica/core/api", () => ({ + api: { saveAgentBuilderDraft: h.save }, +})); +vi.mock("./use-builder-session", () => ({ + useBuilderSession: () => ({ + messages: h.messages, + pending: h.pending, + messagesLoading: h.loading, + error: h.error, + }), +})); +vi.mock("./use-create-agent-form", () => ({ + useCreateAgentForm: () => { + const [draft, setDraft] = useState({ ...EMPTY_AGENT_DRAFT, runtimeId: "local" }); + return { + draft, setDraft, selectedRuntime: null, + workspaceSkills: [], members: [], runtimes: [], draftReady: true, + }; + }, +})); +vi.mock("./use-create-agent-submit", () => ({ + useCreateAgentSubmit: () => ({ creating: false }), +})); +vi.mock("./agent-configuration-panel", () => ({ + AgentConfigurationPanel: ({ draft, onChange }: { + draft: AgentDraft; onChange: (draft: AgentDraft) => void; + }) => ( + onChange({ ...draft, name: event.target.value })} /> + ), +})); +vi.mock("./builder-conversation", () => ({ + BuilderConversation: ({ error }: { error: string | null }) => + error ?
{error}
: null, +})); +vi.mock("./create-agent-footer", () => ({ CreateAgentFooter: () => null })); +vi.mock("react-resizable-panels", () => ({ useDefaultLayout: () => ({}) })); +vi.mock("@multica/ui/components/ui/resizable", () => ({ + ResizablePanelGroup: ({ children }: { children: ReactNode }) => <>{children}, + ResizablePanel: ({ children }: { children: ReactNode }) => <>{children}, + ResizableHandle: () => null, +})); + +const guidance = "This reply could not fill the form. Ask the assistant to generate the draft again, or fill it in manually."; +const unclosed = '{"name":"Poet","instructions":"# Role\nWrite poetry."}'; +function reply(content: string, id = "reply-1"): ChatMessage { + return { + id, content, role: "assistant", chat_session_id: "session-1", + task_id: "task-1", created_at: "2026-09-17T05:00:00Z", + }; +} +function workspace() { + return ; +} +function mount(locale: SupportedLocale = "en") { + const qc = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + const view = renderWithI18n( + {workspace()}, + { locale }, + ); + return { + ...view, + refresh: () => view.rerender( + {workspace()}, + ), + }; +} + +describe("builder draft recovery wiring", () => { + beforeEach(() => { + h.messages = []; + h.pending = false; + h.loading = false; + h.settled = true; + h.error = null; + h.session = undefined; + h.save.mockClear(); + }); + afterEach(cleanup); + + it.each([unclosed, `${unclosed}`])("waits for completion, fills once, and persists subsequent manual edits: %s", async (content) => { + h.messages = [ + reply('{"name":"Older draft"}', "older"), + reply(content), + ]; + h.pending = true; + const view = mount(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + expect(screen.queryByRole("alert")).toBeNull(); + + h.pending = false; + view.refresh(); + await waitFor(() => expect(screen.getByLabelText("Agent name")).toHaveValue("Poet")); + fireEvent.change(screen.getByLabelText("Agent name"), { target: { value: "Edited poet" } }); + h.messages = [...h.messages]; + view.refresh(); + expect(screen.getByLabelText("Agent name")).toHaveValue("Edited poet"); + expect(screen.queryByRole("alert")).toBeNull(); + h.pending = true; + view.refresh(); + expect(screen.getByLabelText("Agent name")).toHaveValue("Edited poet"); + h.pending = false; + view.refresh(); + expect(screen.getByLabelText("Agent name")).toHaveValue("Edited poet"); + view.unmount(); + expect(h.save).toHaveBeenLastCalledWith("session-1", expect.objectContaining({ + name: "Edited poet", applied_message_id: "reply-1", + })); + }); + + it("shows recovery guidance for an invalid final reply and clears it on a valid retry", async () => { + h.messages = [reply('{"name":"Unfinished')]; + h.pending = true; + const view = mount(); + expect(screen.queryByRole("alert")).toBeNull(); + h.pending = false; + view.refresh(); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + + h.messages = [...h.messages, reply(unclosed, "reply-2")]; + view.refresh(); + await waitFor(() => expect(screen.getByLabelText("Agent name")).toHaveValue("Poet")); + expect(screen.queryByRole("alert")).toBeNull(); + }); + + it.each([unclosed, `${unclosed}`])("does not recover a failed turn or replace its existing failure presentation: %s", (content) => { + h.messages = [{ ...reply(content), failure_reason: "agent_error" }]; + mount(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + expect(screen.queryByRole("alert")).toBeNull(); + }); + + it("preserves saved manual edits when reopening a recovered reply", async () => { + h.messages = [reply(unclosed)]; + h.session = { + session_id: "session-1", title: "Poet", runtime_id: "local", + created_at: "", updated_at: "", last_message_at: "", + last_message_content: unclosed, last_message_role: "assistant", + draft: toStoredAgentDraft({ ...EMPTY_AGENT_DRAFT, name: "Saved edit" }, "reply-1"), + }; + mount(); + await waitFor(() => expect(screen.getByLabelText("Agent name")).toHaveValue("Saved edit")); + expect(screen.queryByRole("alert")).toBeNull(); + expect(h.save).not.toHaveBeenCalled(); + }); + + it("waits for history and stored configuration before recovering a draft", async () => { + h.messages = [reply(unclosed)]; + h.loading = true; + h.settled = false; + const view = mount(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + h.loading = false; + view.refresh(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + h.settled = true; + view.refresh(); + await waitFor(() => expect(screen.getByLabelText("Agent name")).toHaveValue("Poet")); + }); + + it("keeps transport errors visible ahead of draft guidance", () => { + h.messages = [reply('{"name":"Unfinished')]; + h.error = "Connection lost"; + mount(); + expect(screen.getByRole("alert")).toHaveTextContent("Connection lost"); + expect(screen.getByRole("alert")).not.toHaveTextContent(guidance); + }); + + it.each(["Stopped.", "No draft"])("does not warn for a reply without a draft block, including after reopening: %s", (content) => { + h.messages = [reply(content)]; + const view = mount(); + expect(screen.queryByRole("alert")).toBeNull(); + view.unmount(); + mount(); + expect(screen.queryByRole("alert")).toBeNull(); + }); + + it("does not warn for empty history, user messages, or no-response replies", () => { + const view = mount(); + expect(screen.queryByRole("alert")).toBeNull(); + h.messages = [{ ...reply("Create a poet"), role: "user" }]; + view.refresh(); + expect(screen.queryByRole("alert")).toBeNull(); + h.messages = [{ ...reply(""), message_kind: "no_response" }]; + view.refresh(); + expect(screen.queryByRole("alert")).toBeNull(); + }); + + it.each([ + ["en", "This reply could not fill the form."], + ["zh-Hans", "本次回复未能填入表单。"], + ["ja", "この返信からフォームを入力できませんでした。"], + ["ko", "이 응답으로 양식을 채울 수 없습니다."], + ["fr", "Cette réponse n’a pas permis de remplir le formulaire."], + ] as const)("renders draft recovery guidance in %s", (locale, message) => { + h.messages = [reply('{"name":"Unfinished')]; + mount(locale); + expect(screen.getByRole("alert")).toHaveTextContent(message); + }); +}); diff --git a/packages/views/agents/create/builder-workspace.tsx b/packages/views/agents/create/builder-workspace.tsx index 122d82e204a..5d67d691e03 100644 --- a/packages/views/agents/create/builder-workspace.tsx +++ b/packages/views/agents/create/builder-workspace.tsx @@ -181,11 +181,24 @@ export function BuilderWorkspace({ // Realtime chat updates can mutate the cached messages array in place. Use // the latest structured message's scalar identity/content as effect inputs // so a draft is still applied when the array reference itself is unchanged. + // These are durable replies from chat:done/history, not streaming chunks. + // Wait until the active turn settles before recovering an unclosed block. + const completed = !builder.pending && !builder.messagesLoading; + const latestMessage = builder.messages.at(-1); + const draftReplyInvalid = + completed && + latestMessage?.role === "assistant" && + !latestMessage.failure_reason && + latestMessage.message_kind !== "no_response" && + latestMessage.content.includes("") && + !parseBuilderDraft(latestMessage.content, { completed }); const latestDraftMessage = [...builder.messages] .reverse() .find( (message) => - message.role === "assistant" && parseBuilderDraft(message.content), + message.role === "assistant" && + !message.failure_reason && + parseBuilderDraft(message.content, { completed }), ); const latestDraftMessageId = latestDraftMessage?.id; const latestDraftMessageContent = latestDraftMessage?.content; @@ -198,7 +211,7 @@ export function BuilderWorkspace({ // Gated on the restore: merging a reply into the form before the stored // configuration lands would be overwritten a tick later, and the merge // would have been computed against an empty draft. - if (!restored) return; + if (!restored || !completed) return; if ( !latestDraftMessageId || !latestDraftMessageContent || @@ -206,7 +219,7 @@ export function BuilderWorkspace({ ) { return; } - const payload = parseBuilderDraft(latestDraftMessageContent); + const payload = parseBuilderDraft(latestDraftMessageContent, { completed }); if (!payload) return; markApplied(latestDraftMessageId); setDraft((current) => @@ -221,6 +234,7 @@ export function BuilderWorkspace({ }, [ latestDraftMessageContent, latestDraftMessageId, + completed, markApplied, memberIdSet, restored, @@ -312,7 +326,12 @@ export function BuilderWorkspace({ onStop={() => void builder.stop()} restoreDraftRequest={builder.restoreDraftRequest} onRestoreDraftApplied={builder.handleRestoreDraftApplied} - error={builder.error} + error={ + builder.error || + (draftReplyInvalid + ? t(($) => $.creation_studio.builder.draft_parse_failed) + : null) + } /> diff --git a/packages/views/locales/en/agents.json b/packages/views/locales/en/agents.json index b70531df0fe..8f32330bb4f 100644 --- a/packages/views/locales/en/agents.json +++ b/packages/views/locales/en/agents.json @@ -339,6 +339,7 @@ "setup_description": "This runtime powers setup and is the default for the new agent.", "start": "Start conversation", "connect_runtime": "Connect a runtime", + "draft_parse_failed": "This reply could not fill the form. Ask the assistant to generate the draft again, or fill it in manually.", "start_failed": "Could not start Agent Builder.", "send_failed": "Could not send your message.", "stop_failed": "Could not stop the current generation.", diff --git a/packages/views/locales/fr/agents.json b/packages/views/locales/fr/agents.json index 56a4acf89c2..77339e6ece1 100644 --- a/packages/views/locales/fr/agents.json +++ b/packages/views/locales/fr/agents.json @@ -339,6 +339,7 @@ "setup_description": "Ce runtime sert à la configuration et sera celui du nouvel agent par défaut.", "start": "Démarrer la conversation", "connect_runtime": "Connecter un runtime", + "draft_parse_failed": "Cette réponse n’a pas permis de remplir le formulaire. Demandez à l’agent de générer à nouveau le brouillon ou remplissez le formulaire manuellement.", "start_failed": "Impossible de démarrer l'Agent Builder.", "send_failed": "Impossible d'envoyer votre message.", "stop_failed": "Impossible d'arrêter la génération en cours.", diff --git a/packages/views/locales/ja/agents.json b/packages/views/locales/ja/agents.json index 94536f9ee8e..3a651417093 100644 --- a/packages/views/locales/ja/agents.json +++ b/packages/views/locales/ja/agents.json @@ -920,6 +920,7 @@ "setup_description": "このランタイムは作成時の対話と、新しいエージェントの既定値に使われます。", "start": "会話を開始", "connect_runtime": "ランタイムを接続", + "draft_parse_failed": "この返信からフォームを入力できませんでした。エージェントに下書きの再生成を依頼するか、手動で入力してください。", "start_failed": "Agent Builder を開始できませんでした。", "send_failed": "メッセージを送信できませんでした。", "stop_failed": "現在の生成を停止できませんでした。", diff --git a/packages/views/locales/ko/agents.json b/packages/views/locales/ko/agents.json index e80ea7760f4..e772236a039 100644 --- a/packages/views/locales/ko/agents.json +++ b/packages/views/locales/ko/agents.json @@ -920,6 +920,7 @@ "setup_description": "이 런타임은 생성 대화에 사용되며 새 에이전트의 기본값이 됩니다.", "start": "대화 시작", "connect_runtime": "런타임 연결", + "draft_parse_failed": "이 응답으로 양식을 채울 수 없습니다. 에이전트에게 초안을 다시 생성하도록 요청하거나 직접 입력하세요.", "start_failed": "Agent Builder를 시작하지 못했습니다.", "send_failed": "메시지를 보내지 못했습니다.", "stop_failed": "현재 생성을 중지하지 못했습니다.", diff --git a/packages/views/locales/zh-Hans/agents.json b/packages/views/locales/zh-Hans/agents.json index 54de8b7f270..3864b4cad06 100644 --- a/packages/views/locales/zh-Hans/agents.json +++ b/packages/views/locales/zh-Hans/agents.json @@ -329,6 +329,7 @@ "setup_description": "该运行时用于创建对话，并默认用于新智能体。", "start": "开始对话", "connect_runtime": "连接运行时", + "draft_parse_failed": "本次回复未能填入表单。请让智能体重新生成草稿，或手动填写。", "start_failed": "无法启动智能体构建器。", "send_failed": "无法发送消息。", "stop_failed": "无法停止当前生成任务。", From 640788c059537cd60daf11d152bd76a2e3fc2a9f Mon Sep 17 00:00:00 2001 From: "[S]" Date: Thu, 15 Oct 2026 01:07:09 -0500 Subject: [PATCH 05/51] MUL-6945 & MUL-6944: fix(daemon): preserve Git content during artifact GC (#7893) * fix(daemon): preserve tracked files during artifact GC * fix(daemon): close artifact GC Git preservation gaps Co-authored-by: multica-agent --------- Co-authored-by: SSK Co-authored-by: Sol-Boy Co-authored-by: multica-agent --- server/internal/daemon/gc.go | 160 +++++++++++++++++- .../daemon/gc_artifact_tracking_test.go | 157 +++++++++++++++++ .../daemon/gc_junction_windows_test.go | 3 +- .../daemon/gc_managed_artifact_test.go | 10 +- server/internal/daemon/gc_test.go | 63 +++++-- 5 files changed, 367 insertions(+), 26 deletions(-) create mode 100644 server/internal/daemon/gc_artifact_tracking_test.go diff --git a/server/internal/daemon/gc.go b/server/internal/daemon/gc.go index 135bfa3b7aa..19efc2fc9c3 100644 --- a/server/internal/daemon/gc.go +++ b/server/internal/daemon/gc.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "log/slog" "net/http" "os" "os/exec" @@ -226,7 +227,7 @@ func (d *Daemon) gcWorkspace(ctx context.Context, wsDir string, stats *gcStats) } } action := d.shouldCleanTaskDir(ctx, taskDir) - cleanedHere += d.applyGCAction(taskDir, action, stats) + cleanedHere += d.applyGCAction(ctx, taskDir, action, stats) } workspaceIDs := make([]string, 0, len(issueCandidatesByWorkspace)) for workspaceID := range issueCandidatesByWorkspace { @@ -306,13 +307,13 @@ func (d *Daemon) gcWorkspaceIssues(ctx context.Context, workspaceID string, cand // data stays. The regenerable Codex cache is still fair game — // see applyManagedArtifactFallback. action := d.applyManagedArtifactFallback(candidate.taskDir, candidate.meta, gcActionSkip) - cleaned += d.applyGCAction(candidate.taskDir, action, stats) + cleaned += d.applyGCAction(ctx, candidate.taskDir, action, stats) continue } action := d.gcDecisionIssueResult(candidate.taskDir, candidate.meta, result) action = d.applyLocalDirectoryGCOverride(candidate.meta, action) action = d.applyManagedArtifactFallback(candidate.taskDir, candidate.meta, action) - cleaned += d.applyGCAction(candidate.taskDir, action, stats) + cleaned += d.applyGCAction(ctx, candidate.taskDir, action, stats) } return cleaned } @@ -320,7 +321,7 @@ func (d *Daemon) gcWorkspaceIssues(ctx context.Context, workspaceID string, cand // applyGCAction performs one decision and updates cycle stats. Each mutation // atomically reserves the env root because a task can start while the server // reconciliation request is in flight. -func (d *Daemon) applyGCAction(taskDir string, action gcAction, stats *gcStats) int { +func (d *Daemon) applyGCAction(ctx context.Context, taskDir string, action gcAction, stats *gcStats) int { if action != gcActionSkip { if _, err := d.gcTaskDirOwner(taskDir); err != nil { d.logger.Warn("gc: refusing to mutate unowned task directory", "dir", taskDir, "error", err) @@ -361,7 +362,7 @@ func (d *Daemon) applyGCAction(taskDir string, action gcAction, stats *gcStats) stats.bytesReclaimed += bytes return 1 case gcActionCleanArtifacts: - removed, bytes, perPattern := d.cleanTaskArtifacts(taskDir, d.cfg.GCArtifactPatterns) + removed, bytes, perPattern := d.cleanTaskArtifacts(ctx, taskDir, d.cfg.GCArtifactPatterns) recordArtifactCleanup(stats, removed, bytes, perPattern) stats.skipped++ // task dir itself preserved case gcActionCleanManagedArtifacts: @@ -939,13 +940,20 @@ const linkedDirModes = os.ModeSymlink | os.ModeIrregular // - patterns are basename-only; entries with a path separator are dropped. // - .git subtrees are never descended into, so the agent's git history stays // intact even if a pattern would otherwise match. +// - a basename-matched directory containing Git-tracked files is retained. Artifact +// names are conventions, not proof that the repository treats the content +// as regenerable; deleting tracked output mutates the agent's checkout. +// - a basename-matched directory that is or contains a Git checkout is retained, even +// when its containing repository does not track the nested checkout. +// Exact daemon-managed cache paths keep their explicit regenerable contract, +// consistent with cleanManagedTaskArtifacts. // - linked directories are skipped entirely — neither the link nor its // target is touched, so a malicious or stale link can't redirect the GC // outside the workdir. See linkedDirModes for what counts as a link. // - every removal target is verified to live inside taskDir, so a tampered // .gc_meta.json can't trick the daemon into deleting outside its sandbox. -func (d *Daemon) cleanTaskArtifacts(taskDir string, patterns []string) (removed int, bytes int64, perPattern map[string]int) { - return d.cleanTaskArtifactsMatching(taskDir, newArtifactMatcher(patterns, execenv.ManagedReclaimableArtifactSubpaths())) +func (d *Daemon) cleanTaskArtifacts(ctx context.Context, taskDir string, patterns []string) (removed int, bytes int64, perPattern map[string]int) { + return d.cleanTaskArtifactsMatching(ctx, taskDir, newArtifactMatcher(patterns, execenv.ManagedReclaimableArtifactSubpaths())) } // cleanManagedTaskArtifacts removes the exact daemon-managed artifact subpaths @@ -1041,7 +1049,7 @@ func hasManagedArtifact(taskDir string) bool { return false } -func (d *Daemon) cleanTaskArtifactsMatching(taskDir string, matcher artifactMatcher) (removed int, bytes int64, perPattern map[string]int) { +func (d *Daemon) cleanTaskArtifactsMatching(ctx context.Context, taskDir string, matcher artifactMatcher) (removed int, bytes int64, perPattern map[string]int) { perPattern = map[string]int{} if taskDir == "" || (len(matcher.basenames) == 0 && len(matcher.exactPaths) == 0) { return @@ -1053,6 +1061,9 @@ func (d *Daemon) cleanTaskArtifactsMatching(taskDir string, matcher artifactMatc } walkErr := filepath.WalkDir(absRoot, func(path string, entry os.DirEntry, err error) error { + if ctx.Err() != nil { + return context.Cause(ctx) + } if err != nil { return nil // best-effort — keep walking } @@ -1080,7 +1091,33 @@ func (d *Daemon) cleanTaskArtifactsMatching(taskDir string, matcher artifactMatc if !ok { return nil } - size := dirSize(path) + rel, relErr := filepath.Rel(absRoot, path) + if relErr != nil { + return relErr + } + if _, managed := matcher.exactPaths[rel]; !managed { + preserve, preserveErr := artifactDirectoryNeedsPreservation(ctx, absRoot, path) + if preserveErr != nil { + // Fail closed. A malformed or temporarily unreadable checkout is not + // evidence that a matched directory is safe to remove. + d.logger.Warn("gc: artifact retained because git tracking could not be determined", "path", path, "error", preserveErr) + return filepath.SkipDir + } + if preserve { + if d.logger.Enabled(ctx, slog.LevelDebug) { + retainedBytes, _ := dirSizeContext(ctx, path) + d.logger.DebugContext(ctx, "gc: artifact directory retained to preserve git content", "path", path, "pattern", pattern, "retained_bytes", retainedBytes) + } + return filepath.SkipDir + } + } + size, sizeErr := dirSizeContext(ctx, path) + if sizeErr != nil { + return sizeErr + } + if ctx.Err() != nil { + return context.Cause(ctx) + } if rmErr := os.RemoveAll(path); rmErr != nil { d.logger.Warn("gc: artifact remove failed", "path", path, "error", rmErr) return filepath.SkipDir @@ -1098,6 +1135,111 @@ func (d *Daemon) cleanTaskArtifactsMatching(taskDir string, matcher artifactMatc return } +// artifactDirectoryNeedsPreservation protects both tracked content and Git +// checkout boundaries. The index, not git status, decides whether content is +// tracked: a clean checkout containing a committed .next directory must stay. +// A nested checkout's files need not appear in the outer repository's index, +// so its metadata must also be checked before deleting the entire subtree. +func artifactDirectoryNeedsPreservation(ctx context.Context, taskRoot, artifactPath string) (bool, error) { + if ctx.Err() != nil { + return false, context.Cause(ctx) + } + absRoot, err := filepath.Abs(taskRoot) + if err != nil { + return false, err + } + absArtifact, err := filepath.Abs(artifactPath) + if err != nil { + return false, err + } + if !isPathWithin(absRoot, absArtifact) { + return false, fmt.Errorf("artifact path escapes task root") + } + + repoRoot, err := containingGitCheckout(absRoot, absArtifact) + if err != nil { + return false, err + } + if repoRoot == absArtifact { + return true, nil // the removal target is itself an entire checkout + } + if repoRoot != "" { + rel, err := filepath.Rel(repoRoot, absArtifact) + if err != nil { + return false, fmt.Errorf("resolve artifact path relative to git checkout: %w", err) + } + if !isPathWithin(repoRoot, absArtifact) { + return false, fmt.Errorf("artifact path is not below git checkout") + } + // -- ends option parsing, not pathspec magic. These are filesystem + // paths, so names such as :(top)fixture must be interpreted literally. + // NUL output also makes whitespace-only tracked names unambiguous. + out, err := runGitGCCommandContext(ctx, repoRoot, "--literal-pathspecs", "ls-files", "-z", "--", filepath.ToSlash(rel)) + if err != nil { + return false, fmt.Errorf("list tracked files below %s: %w", rel, err) + } + if out != "" { + return true, nil + } + } + + found := false + err = filepath.WalkDir(absArtifact, func(path string, entry os.DirEntry, err error) error { + if ctx.Err() != nil { + return context.Cause(ctx) + } + if err != nil { + return err // an unreadable subtree cannot be proven safe to delete + } + if entry.Name() == ".git" { + found = true // includes linked worktrees and damaged metadata + return filepath.SkipAll + } + if entry.IsDir() { + info, err := os.Lstat(path) + if err != nil { + return err + } + if info.Mode()&linkedDirModes != 0 { + return filepath.SkipDir + } + } + return nil + }) + if err != nil { + return false, fmt.Errorf("inspect artifact subtree for git checkouts: %w", err) + } + return found, nil +} + +// containingGitCheckout returns the nearest ancestor carrying Git worktree +// metadata without walking above taskRoot. Linked worktrees use a .git file; +// ordinary clones use a .git directory, so either shape is authoritative. +func containingGitCheckout(taskRoot, start string) (string, error) { + current := filepath.Clean(start) + for isPathWithin(taskRoot, current) { + if _, err := os.Lstat(filepath.Join(current, ".git")); err == nil { + return current, nil + } else if !os.IsNotExist(err) { + return "", fmt.Errorf("inspect git metadata: %w", err) + } + if current == taskRoot { + break + } + parent := filepath.Dir(current) + if parent == current { + break + } + current = parent + } + return "", nil +} + +func isPathWithin(root, path string) bool { + rel, err := filepath.Rel(root, path) + return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) && !filepath.IsAbs(rel) +} + // dirSize returns the total size of all regular files under root, in bytes. // Linked content is not counted: os.RemoveAll would drop the link and leave // the target, so counting it would overstate what a removal reclaims. diff --git a/server/internal/daemon/gc_artifact_tracking_test.go b/server/internal/daemon/gc_artifact_tracking_test.go new file mode 100644 index 00000000000..02e62e90a62 --- /dev/null +++ b/server/internal/daemon/gc_artifact_tracking_test.go @@ -0,0 +1,157 @@ +package daemon + +import ( + "bytes" + "context" + "log/slog" + "net/http" + "os" + "path/filepath" + "runtime" + "strings" + "testing" +) + +func TestCleanTaskArtifacts_PreservesCheckoutBoundaries(t *testing.T) { + t.Parallel() + + for _, layout := range []string{"clone", "linked worktree"} { + for _, location := range []struct { + name string + repo string + outer bool + }{ + {name: "matched root", repo: "workdir/target"}, + {name: "nested checkout", repo: "workdir/node_modules/dependency"}, + {name: "nested checkout inside another repo", repo: "workdir/outer/node_modules/dependency", outer: true}, + } { + t.Run(layout+"/"+location.name, func(t *testing.T) { + d := newGCTestDaemon(t, http.NewServeMux()) + root := t.TempDir() + repo := filepath.Join(root, filepath.FromSlash(location.repo)) + if location.outer { + outer := filepath.Join(root, "workdir", "outer") + if err := os.MkdirAll(outer, 0o755); err != nil { + t.Fatal(err) + } + runGitForGC(t, outer, "init") + } + source := repo + if layout == "linked worktree" { + source = t.TempDir() + } + file := filepath.Join(source, "src", "main.go") + if err := os.MkdirAll(filepath.Dir(file), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(file, []byte("tracked source"), 0o644); err != nil { + t.Fatal(err) + } + runGitForGC(t, source, "init") + runGitForGC(t, source, "add", "src/main.go") + runGitForGC(t, source, "-c", "commit.gpgsign=false", "commit", "-m", "tracked source") + if layout == "linked worktree" { + runGitForGC(t, source, "worktree", "add", "--detach", repo) + } + writeFile(t, filepath.Join(root, "workdir", ".turbo", "cache"), 8) + + removed, bytes, perPattern := d.cleanTaskArtifacts(context.Background(), root, []string{"target", "node_modules", ".turbo"}) + if removed != 1 || bytes != 8 || perPattern[".turbo"] != 1 { + t.Fatalf("removed=%d bytes=%d patterns=%v, want only the unrelated cache removed", removed, bytes, perPattern) + } + if got, err := os.ReadFile(filepath.Join(repo, "src", "main.go")); err != nil || string(got) != "tracked source" { + t.Fatalf("checkout source was removed: data=%q err=%v", got, err) + } + if _, err := os.Lstat(filepath.Join(repo, ".git")); err != nil { + t.Fatalf("checkout metadata was removed: %v", err) + } + }) + } + } +} + +func TestCleanTaskArtifacts_UsesLiteralTrackedPaths(t *testing.T) { + t.Parallel() + + for _, prefix := range []string{":(top)fixture", ":(exclude)fixture", "dist[1]", "star*fixture"} { + t.Run(prefix, func(t *testing.T) { + if runtime.GOOS == "windows" && strings.ContainsAny(prefix, ":*") { + t.Skip("the fixture name is not valid on Windows") + } + d := newGCTestDaemon(t, http.NewServeMux()) + root := t.TempDir() + repo := filepath.Join(root, "workdir", "repo") + rel := filepath.Join(prefix, ".next", "server", "page.js") + tracked := filepath.Join(repo, rel) + if err := os.MkdirAll(filepath.Dir(tracked), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(tracked, []byte("tracked output"), 0o644); err != nil { + t.Fatal(err) + } + runGitForGC(t, repo, "init") + runGitForGC(t, repo, "--literal-pathspecs", "add", "--", filepath.ToSlash(rel)) + writeFile(t, filepath.Join(repo, "node_modules", "cache"), 8) + + removed, bytes, perPattern := d.cleanTaskArtifacts(context.Background(), root, []string{".next", "node_modules"}) + if removed != 1 || bytes != 8 || perPattern["node_modules"] != 1 { + t.Fatalf("removed=%d bytes=%d patterns=%v, want only untracked node_modules removed", removed, bytes, perPattern) + } + if got, err := os.ReadFile(tracked); err != nil || string(got) != "tracked output" { + t.Fatalf("literal tracked path was removed: data=%q err=%v", got, err) + } + }) + } +} + +func TestCleanTaskArtifacts_PreservesInvalidGitMetadata(t *testing.T) { + t.Parallel() + + d := newGCTestDaemon(t, http.NewServeMux()) + root := t.TempDir() + repo := filepath.Join(root, "workdir", "repo") + writeFile(t, filepath.Join(repo, ".next", "cache"), 8) + if err := os.WriteFile(filepath.Join(repo, ".git"), []byte("gitdir: missing"), 0o644); err != nil { + t.Fatal(err) + } + if removed, _, _ := d.cleanTaskArtifacts(context.Background(), root, []string{".next"}); removed != 0 { + t.Fatalf("removed=%d, want fail-closed retention", removed) + } +} + +func TestCleanTaskArtifacts_CancellationPreservesArtifacts(t *testing.T) { + t.Parallel() + + d := newGCTestDaemon(t, http.NewServeMux()) + root := t.TempDir() + artifact := filepath.Join(root, "workdir", "node_modules", "cache") + writeFile(t, artifact, 8) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if removed, _, _ := d.cleanTaskArtifacts(ctx, root, []string{"node_modules"}); removed != 0 { + t.Fatalf("removed=%d after cancellation, want 0", removed) + } + if _, err := os.Stat(artifact); err != nil { + t.Fatalf("cancelled cleanup removed artifact: %v", err) + } +} + +func TestCleanTaskArtifacts_DebugLogsRetainedBytes(t *testing.T) { + t.Parallel() + + d := newGCTestDaemon(t, http.NewServeMux()) + var logs bytes.Buffer + d.logger = slog.New(slog.NewJSONHandler(&logs, &slog.HandlerOptions{Level: slog.LevelDebug})) + root := t.TempDir() + repo := filepath.Join(root, "workdir", "repo") + writeFile(t, filepath.Join(repo, ".next", "tracked"), 8) + writeFile(t, filepath.Join(repo, ".next", "cache"), 16) + runGitForGC(t, repo, "init") + runGitForGC(t, repo, "add", ".next/tracked") + if removed, _, _ := d.cleanTaskArtifacts(context.Background(), root, []string{".next"}); removed != 0 { + t.Fatalf("removed=%d, want tracked directory retained", removed) + } + if log := logs.String(); !strings.Contains(log, `"level":"DEBUG"`) || !strings.Contains(log, `"retained_bytes":24`) { + t.Fatalf("missing debug retention byte count: %s", log) + } +} diff --git a/server/internal/daemon/gc_junction_windows_test.go b/server/internal/daemon/gc_junction_windows_test.go index 6b70d4ec155..866f1f0e904 100644 --- a/server/internal/daemon/gc_junction_windows_test.go +++ b/server/internal/daemon/gc_junction_windows_test.go @@ -3,6 +3,7 @@ package daemon import ( + "context" "net/http" "os" "os/exec" @@ -58,7 +59,7 @@ func TestCleanTaskArtifacts_DoesNotFollowDirectoryJunction(t *testing.T) { // user's ~/.codex/skills/. createJunction(t, userSkill, filepath.Join(taskDir, "codex-home", "skills", "alpha")) - removed, _, _ := d.cleanTaskArtifacts(taskDir, []string{"node_modules"}) + removed, _, _ := d.cleanTaskArtifacts(context.Background(), taskDir, []string{"node_modules"}) if removed != 0 { t.Errorf("removed %d artifacts through a junction, want 0", removed) } diff --git a/server/internal/daemon/gc_managed_artifact_test.go b/server/internal/daemon/gc_managed_artifact_test.go index b35cf1021a8..a878063cd26 100644 --- a/server/internal/daemon/gc_managed_artifact_test.go +++ b/server/internal/daemon/gc_managed_artifact_test.go @@ -80,7 +80,7 @@ func TestManagedArtifact_IdleActiveChatReclaimsSandboxBin(t *testing.T) { if action != gcActionCleanManagedArtifacts { t.Fatalf("want gcActionCleanManagedArtifacts, got %d", action) } - d.applyGCAction(taskDir, action, stats) + d.applyGCAction(context.Background(), taskDir, action, stats) assertGone(t, taskDir, sandboxBinRel) assertKept(t, taskDir, @@ -212,7 +212,7 @@ func TestManagedArtifact_NonTerminalAutopilotAndQuickCreate(t *testing.T) { if action != gcActionCleanManagedArtifacts { t.Fatalf("%s: want gcActionCleanManagedArtifacts, got %d", tc.name, action) } - d.applyGCAction(taskDir, action, &gcStats{byPattern: map[string]int{}}) + d.applyGCAction(context.Background(), taskDir, action, &gcStats{byPattern: map[string]int{}}) assertGone(t, taskDir, sandboxBinRel) assertKept(t, taskDir, ".gc_meta.json") } @@ -261,7 +261,7 @@ func TestManagedArtifact_LocalDirectoryChatReclaimsSandboxBin(t *testing.T) { if action != gcActionCleanManagedArtifacts { t.Fatalf("want gcActionCleanManagedArtifacts, got %d", action) } - d.applyGCAction(taskDir, action, &gcStats{byPattern: map[string]int{}}) + d.applyGCAction(context.Background(), taskDir, action, &gcStats{byPattern: map[string]int{}}) assertGone(t, taskDir, sandboxBinRel) assertKept(t, taskDir, "logs/run.log", ".gc_meta.json") } @@ -287,7 +287,7 @@ func TestManagedArtifact_SecondCycleIsANoOp(t *testing.T) { if first != gcActionCleanManagedArtifacts { t.Fatalf("first cycle: want gcActionCleanManagedArtifacts, got %d", first) } - d.applyGCAction(taskDir, first, &gcStats{byPattern: map[string]int{}}) + d.applyGCAction(context.Background(), taskDir, first, &gcStats{byPattern: map[string]int{}}) assertGone(t, taskDir, sandboxBinRel) for cycle := 2; cycle <= 3; cycle++ { @@ -460,7 +460,7 @@ func TestManagedArtifact_DirectRemovalMatchesTreeWalk(t *testing.T) { setup(t, walked) setup(t, direct) - wRemoved, wBytes, wPattern := d.cleanTaskArtifactsMatching(walked, walkMatcher) + wRemoved, wBytes, wPattern := d.cleanTaskArtifactsMatching(context.Background(), walked, walkMatcher) dRemoved, dBytes, dPattern := d.cleanManagedTaskArtifacts(direct) if wRemoved != dRemoved || wBytes != dBytes || fmt.Sprint(wPattern) != fmt.Sprint(dPattern) { diff --git a/server/internal/daemon/gc_test.go b/server/internal/daemon/gc_test.go index 8e90b38c0a2..ab2b1cec39b 100644 --- a/server/internal/daemon/gc_test.go +++ b/server/internal/daemon/gc_test.go @@ -597,7 +597,7 @@ func TestApplyGCAction_RefusesEveryMutationWithoutOwner(t *testing.T) { } stats := &gcStats{byPattern: map[string]int{}} - if removed := d.applyGCAction(taskDir, tc.action, stats); removed != 0 { + if removed := d.applyGCAction(context.Background(), taskDir, tc.action, stats); removed != 0 { t.Fatalf("applyGCAction reported %d removals on an unowned directory", removed) } if stats.cleaned != 0 || stats.orphaned != 0 || stats.artifactDirs != 0 || stats.bytesReclaimed != 0 { @@ -1489,8 +1489,10 @@ func TestCleanTaskArtifacts_RemovesOnlyMatchedDirs(t *testing.T) { mustMkdir("workdir/repo/src") mustWrite("workdir/repo/src/index.ts", "console.log('hi')") - mustMkdir("workdir/repo/.git/objects") - mustWrite("workdir/repo/.git/objects/pack", "binary") + if out, err := exec.Command("git", "-C", filepath.Join(taskDir, "workdir/repo"), "init").CombinedOutput(); err != nil { + t.Fatalf("git init: %v: %s", err, out) + } + mustWrite("workdir/repo/.git/objects/pack/test.pack", "binary") mustMkdir("workdir/repo/node_modules/lodash") mustWrite("workdir/repo/node_modules/lodash/index.js", "module.exports = {}") mustMkdir("workdir/repo/.next/cache") @@ -1503,7 +1505,7 @@ func TestCleanTaskArtifacts_RemovesOnlyMatchedDirs(t *testing.T) { mustMkdir("output") mustWrite("output/result.txt", "done") - removed, bytes, perPattern := d.cleanTaskArtifacts(taskDir, []string{"node_modules", ".next", ".turbo"}) + removed, bytes, perPattern := d.cleanTaskArtifacts(context.Background(), taskDir, []string{"node_modules", ".next", ".turbo"}) if removed != 3 { t.Fatalf("expected 3 artifact dirs removed, got %d", removed) @@ -1518,7 +1520,7 @@ func TestCleanTaskArtifacts_RemovesOnlyMatchedDirs(t *testing.T) { // Verify protected paths are intact. for _, rel := range []string{ "workdir/repo/src/index.ts", - "workdir/repo/.git/objects/pack", + "workdir/repo/.git/objects/pack/test.pack", "workdir/repo/dist/main.js", "output/result.txt", ".gc_meta.json", @@ -1540,6 +1542,45 @@ func TestCleanTaskArtifacts_RemovesOnlyMatchedDirs(t *testing.T) { } } +func TestCleanTaskArtifacts_PreservesTrackedArtifactDirectory(t *testing.T) { + t.Parallel() + + d := newGCTestDaemon(t, http.NewServeMux()) + taskDir := t.TempDir() + repoDir := filepath.Join(taskDir, "workdir", "repo") + tracked := filepath.Join(repoDir, ".next", "server", "page.js") + untracked := filepath.Join(repoDir, "node_modules", "dependency", "index.js") + for path, content := range map[string]string{ + tracked: "committed build output", + untracked: "regenerable dependency", + } { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + } + if out, err := exec.Command("git", "-C", repoDir, "init").CombinedOutput(); err != nil { + t.Fatalf("git init: %v: %s", err, out) + } + if out, err := exec.Command("git", "-C", repoDir, "add", ".next/server/page.js").CombinedOutput(); err != nil { + t.Fatalf("git add tracked artifact: %v: %s", err, out) + } + + removed, _, perPattern := d.cleanTaskArtifacts(context.Background(), taskDir, []string{"node_modules", ".next"}) + + if removed != 1 || perPattern["node_modules"] != 1 || perPattern[".next"] != 0 { + t.Fatalf("removed=%d perPattern=%v, want only untracked node_modules removed", removed, perPattern) + } + if got, err := os.ReadFile(tracked); err != nil || string(got) != "committed build output" { + t.Fatalf("tracked artifact was mutated: data=%q err=%v", got, err) + } + if _, err := os.Stat(filepath.Dir(filepath.Dir(untracked))); !os.IsNotExist(err) { + t.Fatalf("untracked artifact directory survived: %v", err) + } +} + func TestCleanTaskArtifacts_RejectsPatternsWithSeparators(t *testing.T) { t.Parallel() @@ -1549,7 +1590,7 @@ func TestCleanTaskArtifacts_RejectsPatternsWithSeparators(t *testing.T) { t.Fatal(err) } - removed, _, _ := d.cleanTaskArtifacts(taskDir, []string{"workdir/node_modules", "../etc"}) + removed, _, _ := d.cleanTaskArtifacts(context.Background(), taskDir, []string{"workdir/node_modules", "../etc"}) if removed != 0 { t.Fatalf("expected 0 removals from separator-bearing patterns, got %d", removed) } @@ -1577,7 +1618,7 @@ func TestCleanTaskArtifacts_DoesNotFollowSymlinks(t *testing.T) { t.Skipf("symlink not supported: %v", err) } - removed, _, _ := d.cleanTaskArtifacts(taskDir, []string{"node_modules"}) + removed, _, _ := d.cleanTaskArtifacts(context.Background(), taskDir, []string{"node_modules"}) if removed != 0 { t.Fatalf("expected 0 removals (symlinked node_modules), got %d", removed) } @@ -1597,7 +1638,7 @@ func TestCleanTaskArtifacts_ManagedPathIsExactAndDeduplicated(t *testing.T) { writeFile(t, filepath.Join(taskDir, "workdir/repo/.sandbox-bin/keep"), 400) writeFile(t, filepath.Join(taskDir, "codex-home/auth.json"), 10) - removed, bytes, perPattern := d.cleanTaskArtifacts(taskDir, nil) + removed, bytes, perPattern := d.cleanTaskArtifacts(context.Background(), taskDir, nil) if removed != 1 || bytes != 300 { t.Fatalf("removed=%d bytes=%d, want 1/300", removed, bytes) } @@ -1616,7 +1657,7 @@ func TestCleanTaskArtifacts_ManagedPathIsExactAndDeduplicated(t *testing.T) { writeFile(t, filepath.Join(taskDir, "codex-home/.sandbox-bin/codex.exe"), 300) writeFile(t, filepath.Join(taskDir, "workdir/repo/.sandbox-bin/cache"), 400) - removed, bytes, perPattern := d.cleanTaskArtifacts(taskDir, []string{".sandbox-bin"}) + removed, bytes, perPattern := d.cleanTaskArtifacts(context.Background(), taskDir, []string{".sandbox-bin"}) if removed != 2 || bytes != 700 { t.Fatalf("removed=%d bytes=%d, want 2/700 without double counting", removed, bytes) } @@ -1651,7 +1692,7 @@ func TestCleanTaskArtifacts_ManagedPathDoesNotFollowSymlinks(t *testing.T) { t.Skipf("symlink not supported: %v", err) } - removed, _, _ := d.cleanTaskArtifacts(taskDir, nil) + removed, _, _ := d.cleanTaskArtifacts(context.Background(), taskDir, nil) if removed != 0 { t.Fatalf("removed=%d, want 0 for symlinked managed path", removed) } @@ -2523,7 +2564,7 @@ func TestShouldCleanTaskDir_ChatActiveResistsOldMtime(t *testing.T) { if action == gcActionClean || action == gcActionOrphan { t.Fatalf("active chat session's directory must never be removed, got action %d", action) } - d.applyGCAction(taskDir, action, &gcStats{byPattern: map[string]int{}}) + d.applyGCAction(context.Background(), taskDir, action, &gcStats{byPattern: map[string]int{}}) for _, rel := range []string{".", "logs/run.log", "output/result.md", ".gc_meta.json"} { if _, err := os.Stat(filepath.Join(taskDir, rel)); err != nil { From e326a561ae835907310dd48f98c09ab91dd2eec6 Mon Sep 17 00:00:00 2001 From: Multica Eve Date: Thu, 15 Oct 2026 14:46:17 +0800 Subject: [PATCH 06/51] MUL-8000: fix(agents): persist builder draft warning dismissal (#9265) * fix(agents): persist builder draft warning dismissal Co-authored-by: multica-agent * fix(agents): preserve builder warnings during automatic model cleanup Co-authored-by: multica-agent --------- Co-authored-by: Sol-Boy Co-authored-by: multica-agent --- packages/core/agents/stored-draft.test.ts | 24 ++++ packages/core/agents/stored-draft.ts | 2 + packages/core/api/schemas.ts | 1 + packages/core/types/agent.ts | 2 + .../agents/components/model-dropdown.test.tsx | 36 ++++- .../agents/components/model-dropdown.tsx | 7 +- .../create/agent-configuration-panel.tsx | 6 + .../agents/create/builder-conversation.tsx | 21 ++- .../agents/create/builder-workspace.test.tsx | 128 ++++++++++++++++-- .../views/agents/create/builder-workspace.tsx | 33 ++++- .../agents/create/use-builder-draft-sync.ts | 22 +-- 11 files changed, 253 insertions(+), 29 deletions(-) diff --git a/packages/core/agents/stored-draft.test.ts b/packages/core/agents/stored-draft.test.ts index 2b253c34f28..a560e54d7c5 100644 --- a/packages/core/agents/stored-draft.test.ts +++ b/packages/core/agents/stored-draft.test.ts @@ -1,4 +1,7 @@ +// @vitest-environment node + import { describe, expect, it } from "vitest"; +import { StoredAgentDraftSchema } from "../api/schemas"; import type { AgentDraft } from "./draft"; import { fromStoredAgentDraft, @@ -54,6 +57,27 @@ describe("stored agent draft", () => { expect(toStoredAgentDraft(draft(), null).applied_message_id).toBeNull(); }); + it("stores dismissed notices separately from the applied draft", () => { + const stored = toStoredAgentDraft(draft(), "applied", "dismissed"); + expect(StoredAgentDraftSchema.parse(stored)).toMatchObject({ + applied_message_id: "applied", + dismissed_message_id: "dismissed", + }); + expect(fromStoredAgentDraft(stored, "runtime-1")).toEqual(draft()); + expect(storedAgentDraftsEqual(stored, toStoredAgentDraft(draft(), "applied"))).toBe(false); + }); + + it("accepts older drafts and ignores malformed dismissal markers", () => { + const stored = toStoredAgentDraft(draft(), "applied"); + expect(stored).not.toHaveProperty("dismissed_message_id"); + expect(StoredAgentDraftSchema.parse(stored).dismissed_message_id).toBeUndefined(); + expect(StoredAgentDraftSchema.parse({ ...stored, dismissed_message_id: 42 })).toMatchObject({ + name: stored.name, + applied_message_id: "applied", + dismissed_message_id: null, + }); + }); + // Autosave compares on the stored form: a Set rebuilt with the same members // is a new reference every render and would otherwise write on every keystroke // elsewhere in the form. diff --git a/packages/core/agents/stored-draft.ts b/packages/core/agents/stored-draft.ts index e0ee16fc264..827b8808553 100644 --- a/packages/core/agents/stored-draft.ts +++ b/packages/core/agents/stored-draft.ts @@ -6,6 +6,7 @@ export type { StoredAgentDraft }; export function toStoredAgentDraft( draft: AgentDraft, appliedMessageId: string | null, + dismissedMessageId: string | null = null, ): StoredAgentDraft { return { name: draft.name, @@ -21,6 +22,7 @@ export function toStoredAgentDraft( member_ids: [...draft.memberIds], team_ids: [...draft.teamIds], applied_message_id: appliedMessageId, + ...(dismissedMessageId ? { dismissed_message_id: dismissedMessageId } : {}), }; } diff --git a/packages/core/api/schemas.ts b/packages/core/api/schemas.ts index 9e6785707f9..70703c647f0 100644 --- a/packages/core/api/schemas.ts +++ b/packages/core/api/schemas.ts @@ -2204,6 +2204,7 @@ export const StoredAgentDraftSchema = z.object({ member_ids: z.array(z.string()).catch([]), team_ids: z.array(z.string()).catch([]), applied_message_id: z.string().nullable().catch(null), + dismissed_message_id: z.string().nullish().catch(null), }).loose(); /** diff --git a/packages/core/types/agent.ts b/packages/core/types/agent.ts index ff3a3b06854..1653cdf74c5 100644 --- a/packages/core/types/agent.ts +++ b/packages/core/types/agent.ts @@ -730,6 +730,8 @@ export interface StoredAgentDraft { member_ids: string[]; team_ids: string[]; applied_message_id: string | null; + /** Recovery notice acknowledged by manual editing or dismissal. */ + dismissed_message_id?: string | null; } /** One unfinished agent-creation conversation, as listed by the studio. */ diff --git a/packages/views/agents/components/model-dropdown.test.tsx b/packages/views/agents/components/model-dropdown.test.tsx index 9644b06c241..722a70797d1 100644 --- a/packages/views/agents/components/model-dropdown.test.tsx +++ b/packages/views/agents/components/model-dropdown.test.tsx @@ -1,7 +1,8 @@ // @vitest-environment jsdom import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; -import { render, screen, fireEvent, cleanup } from "@testing-library/react"; +import type { ComponentProps } from "react"; +import { render, screen, fireEvent, cleanup, waitFor } from "@testing-library/react"; import { I18nProvider } from "@multica/core/i18n/react"; import type { RuntimeModelsResult } from "@multica/core/types"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -46,7 +47,7 @@ vi.mock("@multica/core/runtimes", () => ({ // Bumped per test so React Query cannot serve a previous case's cached result. let discoveryKey = 0; -function renderDropdown() { +function renderDropdown(props: Partial> = {}) { const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } }, }); @@ -59,6 +60,7 @@ function renderDropdown() { runtimeOnline value="" onChange={onChange} + {...props} /> , @@ -87,7 +89,8 @@ describe("ModelDropdown", () => { }); it("offers the gpt-5.6 Codex models and submits their canonical IDs", async () => { - const { container, onChange } = renderDropdown(); + const onAutomaticChange = vi.fn(); + const { container, onChange } = renderDropdown({ onAutomaticChange }); openDropdown(container); expect(await screen.findByText("GPT-5.6 Sol")).toBeTruthy(); @@ -99,6 +102,33 @@ describe("ModelDropdown", () => { fireEvent.click(screen.getByText("GPT-5.6 Terra")); expect(onChange).toHaveBeenCalledWith("gpt-5.6-terra"); + expect(onAutomaticChange).not.toHaveBeenCalled(); + }); + + it("routes unsupported model cleanup separately from user selections", async () => { + discovery = async () => ({ models: [], supported: false }); + const onAutomaticChange = vi.fn(); + const { onChange } = renderDropdown({ value: "saved-model", onAutomaticChange }); + + await waitFor(() => expect(onAutomaticChange).toHaveBeenCalledWith("")); + expect(onChange).not.toHaveBeenCalled(); + }); + + it("still clears unsupported models for callers with one change handler", async () => { + discovery = async () => ({ models: [], supported: false }); + const { onChange } = renderDropdown({ value: "saved-model" }); + + await waitFor(() => expect(onChange).toHaveBeenCalledWith("")); + }); + + it("treats explicit clearing as a user selection", async () => { + const onAutomaticChange = vi.fn(); + const { container, onChange } = renderDropdown({ value: "saved-model", onAutomaticChange }); + openDropdown(container); + fireEvent.click(await screen.findByText(enAgents.model_dropdown.clear_full)); + + expect(onChange).toHaveBeenCalledWith(""); + expect(onAutomaticChange).not.toHaveBeenCalled(); }); it("offers an explicit refresh that requests the runtime's live catalog", async () => { diff --git a/packages/views/agents/components/model-dropdown.tsx b/packages/views/agents/components/model-dropdown.tsx index 3380cf79e0d..f64bcaeb82d 100644 --- a/packages/views/agents/components/model-dropdown.tsx +++ b/packages/views/agents/components/model-dropdown.tsx @@ -37,12 +37,15 @@ export function ModelDropdown({ runtimeOnline, value, onChange, + onAutomaticChange = onChange, disabled, }: { runtimeId: string | null; runtimeOnline: boolean; value: string; onChange: (value: string) => void; + /** Automatic value cleanup, separate from explicit user selections. */ + onAutomaticChange?: (value: string) => void; disabled?: boolean; }) { const { t } = useT("agents"); @@ -80,9 +83,9 @@ export function ModelDropdown({ // persist a ghost configuration that never takes effect. useEffect(() => { if (!supported && value !== "") { - onChange(""); + onAutomaticChange(""); } - }, [supported, value, onChange]); + }, [supported, value, onAutomaticChange]); const filtered = useMemo(() => { if (!search.trim()) return grouped; diff --git a/packages/views/agents/create/agent-configuration-panel.tsx b/packages/views/agents/create/agent-configuration-panel.tsx index 2f06da98f2b..ba840fe812c 100644 --- a/packages/views/agents/create/agent-configuration-panel.tsx +++ b/packages/views/agents/create/agent-configuration-panel.tsx @@ -38,6 +38,7 @@ const PERMISSION_SCOPES: AgentPermissionScope[] = [ export function AgentConfigurationPanel({ draft, onChange, + onAutomaticChange = onChange, runtimes, runtimesLoading, members, @@ -51,6 +52,8 @@ export function AgentConfigurationPanel({ }: { draft: AgentDraft; onChange: (draft: AgentDraft) => void; + /** Value cleanup without user interaction, such as an unsupported model. */ + onAutomaticChange?: (draft: AgentDraft) => void; runtimes: RuntimeDevice[]; runtimesLoading: boolean; members: MemberWithUser[]; @@ -222,6 +225,9 @@ export function AgentConfigurationPanel({ runtimeOnline={selectedRuntime?.status === "online"} value={draft.model} onChange={(value) => onChange(applyDraftModelChange(draft, value))} + onAutomaticChange={(value) => + onAutomaticChange(applyDraftModelChange(draft, value)) + } // A successful switch clears the model, so an edit made while the // rebind is in flight would be silently discarded. disabled={!selectedRuntime || runtimeSwitchInFlight} diff --git a/packages/views/agents/create/builder-conversation.tsx b/packages/views/agents/create/builder-conversation.tsx index 711e1d78a22..8a042b10d2f 100644 --- a/packages/views/agents/create/builder-conversation.tsx +++ b/packages/views/agents/create/builder-conversation.tsx @@ -1,7 +1,7 @@ "use client"; import type { ReactNode } from "react"; -import { Loader2, MessageSquare } from "lucide-react"; +import { Loader2, MessageSquare, X } from "lucide-react"; import { applyDraftModelChange, applyDraftRuntimeChange, @@ -143,6 +143,7 @@ export function BuilderConversation({ restoreDraftRequest, onRestoreDraftApplied, error, + onDismissError, }: { sessionId: string; messages: ChatMessage[]; @@ -158,8 +159,10 @@ export function BuilderConversation({ restoreDraftRequest: BuilderRestore | null; onRestoreDraftApplied: () => void; error: string | null; + onDismissError?: () => void; }) { const { t } = useT("agents"); + const { t: tModals } = useT("modals"); const pending = !!pendingTask?.task_id; const draftKey = `agent-builder:${sessionId}`; const prompts = [ @@ -235,10 +238,22 @@ export function BuilderConversation({ aria-live="polite" className={cn( CHAT_COLUMN, - "rounded-md bg-destructive/5 px-3 py-2 text-body text-destructive", + "flex items-start gap-2 rounded-md bg-destructive/5 px-3 py-2 text-body text-destructive", )} > - {error} + {error} + {onDismissError && ( + + )}
) : null} diff --git a/packages/views/agents/create/builder-workspace.test.tsx b/packages/views/agents/create/builder-workspace.test.tsx index 3320b4335c7..dd04b924090 100644 --- a/packages/views/agents/create/builder-workspace.test.tsx +++ b/packages/views/agents/create/builder-workspace.test.tsx @@ -1,6 +1,6 @@ // @vitest-environment jsdom -import { useState, type ReactNode } from "react"; +import { useEffect, useState, type ReactNode } from "react"; import { cleanup, fireEvent, screen, waitFor } from "@testing-library/react"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -17,6 +17,7 @@ const h = vi.hoisted(() => ({ settled: true, error: null as string | null, session: undefined as AgentBuilderSessionSummary | undefined, + clearUnsupportedModel: false, save: vi.fn().mockResolvedValue(undefined), })); vi.mock("@multica/core/api", () => ({ @@ -40,19 +41,37 @@ vi.mock("./use-create-agent-form", () => ({ }, })); vi.mock("./use-create-agent-submit", () => ({ - useCreateAgentSubmit: () => ({ creating: false }), + useCreateAgentSubmit: () => ({ creating: false, clearNameError: vi.fn() }), })); vi.mock("./agent-configuration-panel", () => ({ - AgentConfigurationPanel: ({ draft, onChange }: { + AgentConfigurationPanel: ({ draft, onChange, onNameChange, onAutomaticChange }: { draft: AgentDraft; onChange: (draft: AgentDraft) => void; - }) => ( + onNameChange: (name: string) => void; + onAutomaticChange?: (draft: AgentDraft) => void; + }) => { + const clearUnsupportedModel = h.clearUnsupportedModel; + useEffect(() => { + if (clearUnsupportedModel && draft.model) { + (onAutomaticChange ?? onChange)({ ...draft, model: "" }); + } + }, [clearUnsupportedModel, draft, onAutomaticChange, onChange]); + return ( + <> onChange({ ...draft, name: event.target.value })} /> - ), + onChange={(event) => onNameChange(event.target.value)} /> + onChange({ ...draft, description: event.target.value })} /> + + + ); + }, })); vi.mock("./builder-conversation", () => ({ - BuilderConversation: ({ error }: { error: string | null }) => - error ?
{error}
: null, + BuilderConversation: ({ error, onDismissError }: { + error: string | null; onDismissError?: () => void; + }) => error ?
{error} + {onDismissError && } +
: null, })); vi.mock("./create-agent-footer", () => ({ CreateAgentFooter: () => null })); vi.mock("react-resizable-panels", () => ({ useDefaultLayout: () => ({}) })); @@ -70,6 +89,13 @@ function reply(content: string, id = "reply-1"): ChatMessage { task_id: "task-1", created_at: "2026-09-17T05:00:00Z", }; } +function savedSession(draft: AgentBuilderSessionSummary["draft"]): AgentBuilderSessionSummary { + return { + session_id: "session-1", title: "Poet", runtime_id: "local", + created_at: "", updated_at: "", last_message_at: "", + last_message_content: "", last_message_role: "assistant", draft, + }; +} function workspace() { return { h.settled = true; h.error = null; h.session = undefined; + h.clearUnsupportedModel = false; h.save.mockClear(); }); afterEach(cleanup); @@ -147,6 +174,88 @@ describe("builder draft recovery wiring", () => { expect(screen.queryByRole("alert")).toBeNull(); }); + it.each(["name", "description"])("acknowledges a failed draft after editing %s, including after reopening", (field) => { + h.messages = [ + reply('{"name":"Older draft"}', "older"), + reply('{"name":"Unfinished'), + ]; + h.session = savedSession(toStoredAgentDraft({ ...EMPTY_AGENT_DRAFT, name: "Older draft" }, "older")); + const view = mount(); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + + const label = field === "name" ? "Agent name" : "Agent description"; + fireEvent.change(screen.getByLabelText(label), { target: { value: "Manual edit" } }); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText(label)).toHaveValue("Manual edit"); + view.unmount(); + const stored = h.save.mock.lastCall?.[1]; + expect(stored).toMatchObject({ [field]: "Manual edit", applied_message_id: "older", dismissed_message_id: "reply-1" }); + + h.session = savedSession(stored); + const reopened = mount(); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText(label)).toHaveValue("Manual edit"); + h.messages = [...h.messages, reply('{"name":"Still unfinished', "reply-2")]; + reopened.refresh(); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + expect(screen.getByLabelText(label)).toHaveValue("Manual edit"); + h.messages = [...h.messages, reply(unclosed, "reply-3")]; + reopened.refresh(); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText("Agent name")).toHaveValue("Poet"); + }); + + it("persists dismissal even when the configuration is unchanged", () => { + h.messages = [reply('{"name":"Unfinished')]; + const view = mount(); + fireEvent.click(screen.getByRole("button", { name: "Close" })); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + view.unmount(); + const stored = h.save.mock.lastCall?.[1]; + expect(stored).toMatchObject({ name: "", applied_message_id: null, dismissed_message_id: "reply-1" }); + + h.session = savedSession(stored); + mount(); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText("Agent name")).toHaveValue(""); + }); + + it("keeps draft guidance after automatic model cleanup and reopening", () => { + h.messages = [reply('{"name":"Unfinished')]; + h.session = savedSession(toStoredAgentDraft({ ...EMPTY_AGENT_DRAFT, model: "saved-model" }, null)); + const view = mount(); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + + h.clearUnsupportedModel = true; + view.refresh(); + expect(screen.getByLabelText("Agent model")).toHaveValue(""); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + view.unmount(); + const stored = h.save.mock.lastCall?.[1]; + expect(stored).toMatchObject({ model: "" }); + expect(stored).not.toHaveProperty("dismissed_message_id"); + + h.session = savedSession(stored); + mount(); + expect(screen.getByRole("alert")).toHaveTextContent(guidance); + fireEvent.change(screen.getByLabelText("Agent name"), { target: { value: "Manual edit" } }); + expect(screen.queryByRole("alert")).toBeNull(); + }); + + it("waits for stored configuration before showing draft guidance", () => { + h.messages = [reply('{"name":"Unfinished')]; + h.settled = false; + const view = mount(); + expect(screen.queryByRole("alert")).toBeNull(); + + h.session = savedSession(toStoredAgentDraft({ ...EMPTY_AGENT_DRAFT, name: "Saved edit" }, null, "reply-1")); + h.settled = true; + view.refresh(); + expect(screen.queryByRole("alert")).toBeNull(); + expect(screen.getByLabelText("Agent name")).toHaveValue("Saved edit"); + }); + it.each([unclosed, `${unclosed}`])("does not recover a failed turn or replace its existing failure presentation: %s", (content) => { h.messages = [{ ...reply(content), failure_reason: "agent_error" }]; mount(); @@ -188,6 +297,9 @@ describe("builder draft recovery wiring", () => { mount(); expect(screen.getByRole("alert")).toHaveTextContent("Connection lost"); expect(screen.getByRole("alert")).not.toHaveTextContent(guidance); + expect(screen.queryByRole("button", { name: "Close" })).toBeNull(); + fireEvent.change(screen.getByLabelText("Agent name"), { target: { value: "Manual edit" } }); + expect(screen.getByRole("alert")).toHaveTextContent("Connection lost"); }); it.each(["Stopped.", "No draft"])("does not warn for a reply without a draft block, including after reopening: %s", (content) => { diff --git a/packages/views/agents/create/builder-workspace.tsx b/packages/views/agents/create/builder-workspace.tsx index 5d67d691e03..06b33d938bb 100644 --- a/packages/views/agents/create/builder-workspace.tsx +++ b/packages/views/agents/create/builder-workspace.tsx @@ -25,6 +25,7 @@ import { mergeBuilderDraft, parseBuilderDraft, stripBuilderDraft, + type AgentDraft, } from "@multica/core/agents"; import { runtimeDisplayLabel, @@ -185,9 +186,18 @@ export function BuilderWorkspace({ // Wait until the active turn settles before recovering an unclosed block. const completed = !builder.pending && !builder.messagesLoading; const latestMessage = builder.messages.at(-1); + const { + restored, + appliedMessageId, + markApplied, + dismissedMessageId, + markDismissed, + } = draftSync; const draftReplyInvalid = + restored && completed && latestMessage?.role === "assistant" && + latestMessage.id !== dismissedMessageId && !latestMessage.failure_reason && latestMessage.message_kind !== "no_response" && latestMessage.content.includes("") && @@ -203,9 +213,13 @@ export function BuilderWorkspace({ const latestDraftMessageId = latestDraftMessage?.id; const latestDraftMessageContent = latestDraftMessage?.content; - const { restored, markApplied } = draftSync; - const appliedRef = useRef(null); - appliedRef.current = draftSync.appliedMessageId; + const acknowledgeInvalidDraft = () => { + if (draftReplyInvalid && latestMessage) markDismissed(latestMessage.id); + }; + const handleDraftChange = (next: AgentDraft) => { + acknowledgeInvalidDraft(); + setDraft(next); + }; useEffect(() => { // Gated on the restore: merging a reply into the form before the stored @@ -215,7 +229,7 @@ export function BuilderWorkspace({ if ( !latestDraftMessageId || !latestDraftMessageContent || - latestDraftMessageId === appliedRef.current + latestDraftMessageId === appliedMessageId ) { return; } @@ -234,6 +248,7 @@ export function BuilderWorkspace({ }, [ latestDraftMessageContent, latestDraftMessageId, + appliedMessageId, completed, markApplied, memberIdSet, @@ -272,6 +287,7 @@ export function BuilderWorkspace({ if (!runtimeId || runtimeId === draft.runtimeId) return; const bound = await builder.switchRuntime(runtimeId); if (!bound) return; + acknowledgeInvalidDraft(); // Model ids are per-runtime; clear it — with the per-model thinking / // speed overrides — so the new runtime resolves its own defaults instead // of keeping values it may not serve. @@ -332,6 +348,11 @@ export function BuilderWorkspace({ ? t(($) => $.creation_studio.builder.draft_parse_failed) : null) } + onDismissError={ + !builder.error && draftReplyInvalid + ? acknowledgeInvalidDraft + : undefined + } /> @@ -357,7 +378,8 @@ export function BuilderWorkspace({ { submit.clearNameError(); + acknowledgeInvalidDraft(); setDraft((current) => ({ ...current, name })); }} onRuntimeSelect={(runtimeId) => { diff --git a/packages/views/agents/create/use-builder-draft-sync.ts b/packages/views/agents/create/use-builder-draft-sync.ts index 50b73dfbb78..ece03ac3187 100644 --- a/packages/views/agents/create/use-builder-draft-sync.ts +++ b/packages/views/agents/create/use-builder-draft-sync.ts @@ -30,6 +30,8 @@ interface BuilderDraftSync { /** The assistant message whose draft is already reflected in the form. */ appliedMessageId: string | null; markApplied: (messageId: string) => void; + dismissedMessageId: string | null; + markDismissed: (messageId: string) => void; } /** @@ -46,6 +48,8 @@ interface BuilderDraftSync { * what stops a restore from re-applying the last reply over edits made after * it. Held only in memory it died with the page, which is precisely when the * next restore needs it. + * Dismissed recovery notices have a separate marker: older clients must still + * recognize the last applied draft rather than replay it over manual edits. */ export function useBuilderDraftSync(options: { sessionId: string; @@ -62,7 +66,8 @@ export function useBuilderDraftSync(options: { options; const [restored, setRestored] = useState(false); - const appliedMessageIdRef = useRef(null); + const [appliedMessageId, markApplied] = useState(null); + const [dismissedMessageId, markDismissed] = useState(null); // The last shape known to be on the server, so an unchanged draft never // produces a request. Also seeded by the restore, which must not immediately // save back what it just read. @@ -71,7 +76,8 @@ export function useBuilderDraftSync(options: { useEffect(() => { if (!sessionId || restored || !storedSettled) return; if (stored) { - appliedMessageIdRef.current = stored.applied_message_id; + markApplied(stored.applied_message_id); + markDismissed(stored.dismissed_message_id ?? null); savedRef.current = stored; setDraft(fromStoredAgentDraft(stored, runtimeId)); } @@ -102,14 +108,14 @@ export function useBuilderDraftSync(options: { useEffect(() => { if (!sessionId || !restored) return; - const next = toStoredAgentDraft(draft, appliedMessageIdRef.current); + const next = toStoredAgentDraft(draft, appliedMessageId, dismissedMessageId); if (savedRef.current && storedAgentDraftsEqual(savedRef.current, next)) { return; } pendingRef.current = next; const timer = setTimeout(() => save(next), AUTOSAVE_DELAY_MS); return () => clearTimeout(timer); - }, [draft, restored, save, sessionId]); + }, [appliedMessageId, dismissedMessageId, draft, restored, save, sessionId]); // Unmount is the deadline: a tab switch, a route change or a session switch // all tear this down, and the debounce timer dies with it. @@ -123,9 +129,9 @@ export function useBuilderDraftSync(options: { return { restored, - appliedMessageId: appliedMessageIdRef.current, - markApplied: (messageId: string) => { - appliedMessageIdRef.current = messageId; - }, + appliedMessageId, + markApplied, + dismissedMessageId, + markDismissed, }; } From d3730f29f966405282c36caad544aecaa78fa965 Mon Sep 17 00:00:00 2001 From: AbdulrhmanKassemSalla <62542159+AbdulrhmanKassemSalla@users.noreply.github.com> Date: Thu, 15 Oct 2026 11:05:24 +0300 Subject: [PATCH 07/51] fix(editor): render mixed RTL/LTR text with per-block direction (#9261) Arabic/Hebrew/Persian comments and descriptions that mix in English words, paths or numbers render left-aligned with scrambled word order, because every block inherits the page's LTR direction. Use `unicode-bidi: plaintext` so each paragraph, list item, heading and blockquote resolves its base direction from its own first strong character, and `text-align: start` so alignment follows it. Applied to .rich-text-editor (prose.css) and .markdown-content (markdown.css). LTR-only content resolves to LTR and is unchanged. Co-authored-by: Claude Opus 5.5 --- packages/ui/markdown/markdown.css | 7 +++++++ packages/views/editor/styles/prose.css | 10 ++++++++++ 2 files changed, 17 insertions(+) diff --git a/packages/ui/markdown/markdown.css b/packages/ui/markdown/markdown.css index 5b258bd17b1..ed961e867b8 100644 --- a/packages/ui/markdown/markdown.css +++ b/packages/ui/markdown/markdown.css @@ -7,3 +7,10 @@ .markdown-content .katex-display > .katex { white-space: nowrap; } + +/* Mixed RTL/LTR text: resolve each block's direction from its own content + (same rule as .rich-text-editor in prose.css). */ +.markdown-content :is(p, li) { + unicode-bidi: plaintext; + text-align: start; +} diff --git a/packages/views/editor/styles/prose.css b/packages/views/editor/styles/prose.css index 13aa26bf7b6..30582e19b89 100644 --- a/packages/views/editor/styles/prose.css +++ b/packages/views/editor/styles/prose.css @@ -38,6 +38,16 @@ overflow-wrap: break-word; } +/* Bidirectional text (Arabic, Hebrew, Persian mixed with English). Each block + takes its base direction from its own first strong character instead of the + page, so an Arabic paragraph is right-aligned and keeps English words, paths + and numbers in the right order, while English blocks stay left-aligned. + `start` follows the resolved direction, so LTR-only content is unchanged. */ +.rich-text-editor :is(p, li, h1, h2, h3, h4, h5, h6, blockquote) { + unicode-bidi: plaintext; + text-align: start; +} + /* Headings — compact but with clear visual hierarchy */ .rich-text-editor h1 { font-size: var(--text-display-sm); From 83527872634824012dd26bb183aa216fab631c05 Mon Sep 17 00:00:00 2001 From: ZIce <39822906+vicksiyi@users.noreply.github.com> Date: Thu, 15 Oct 2026 16:43:24 +0800 Subject: [PATCH 08/51] MUL-7658: fix(daemon): recover undelivered batch claims after timeout (ZIC-260) (#8774) * fix(daemon): recover undelivered batch claims after request timeout Co-authored-by: multica-agent * test(daemon): avoid copying task service locks in recovery fixture Co-authored-by: multica-agent * test(tasks): isolate delegated recovery assertions (MUL-7658) Co-authored-by: multica-agent * test(daemon): use one clock for deferred claim hints (MUL-7658) Co-authored-by: multica-agent * fix(daemon): keep claim recovery within client deadlines (MUL-7658) Co-authored-by: multica-agent * fix(execenv): preserve racy Git index timestamps in snapshots (MUL-7658) Co-authored-by: multica-agent * fix(daemon): deliver claim responses before recovery side effects (MUL-7658) Co-authored-by: multica-agent * fix(daemon): isolate HTTP claim retries from recovery (MUL-7658) Co-authored-by: multica-agent --------- Co-authored-by: multica-agent Co-authored-by: Sol-Boy --- server/internal/daemon/client.go | 2 +- .../internal/daemon/execenv/local_worktree.go | 25 +- .../daemon/execenv/local_worktree_test.go | 45 +++ server/internal/daemonws/hub.go | 3 + server/internal/daemonws/rpc_completion.go | 28 ++ server/internal/daemonws/rpc_dispatch_test.go | 40 ++ server/internal/handler/daemon.go | 120 ++++-- .../daemon_batch_claim_deadline_test.go | 269 ++++++++++++++ .../daemon_batch_claim_notifications_test.go | 346 ++++++++++++++++++ .../daemon_batch_claim_recovery_test.go | 200 ++++++++++ .../handler/daemon_batch_claim_test.go | 3 +- .../delegated_failure_recovery_test.go | 41 ++- server/pkg/protocol/claim.go | 8 + 13 files changed, 1094 insertions(+), 36 deletions(-) create mode 100644 server/internal/daemonws/rpc_completion.go create mode 100644 server/internal/handler/daemon_batch_claim_deadline_test.go create mode 100644 server/internal/handler/daemon_batch_claim_notifications_test.go create mode 100644 server/internal/handler/daemon_batch_claim_recovery_test.go create mode 100644 server/pkg/protocol/claim.go diff --git a/server/internal/daemon/client.go b/server/internal/daemon/client.go index 66c83217021..46959da1ac4 100644 --- a/server/internal/daemon/client.go +++ b/server/internal/daemon/client.go @@ -271,7 +271,7 @@ func (c *Client) ResolveRemoteMCPCredential(ctx context.Context, daemonToken, ta // starvation; a claim that commits server-side after the client gives up is // recovered by ReclaimStaleDispatchedTasksForRuntimes on the next poll. Kept // comfortably above p99 claim latency so recovery stays the exception. -const batchClaimRequestTimeout = 5 * time.Second +const batchClaimRequestTimeout = protocol.DaemonBatchClaimRequestTimeout // claimTasksResult carries optional scheduling metadata understood only by // daemons advertising claim-poll-hints-v1. A zero-value result is deliberately diff --git a/server/internal/daemon/execenv/local_worktree.go b/server/internal/daemon/execenv/local_worktree.go index 773a0265d6e..aace124836f 100644 --- a/server/internal/daemon/execenv/local_worktree.go +++ b/server/internal/daemon/execenv/local_worktree.go @@ -7,6 +7,7 @@ import ( "encoding/json" "errors" "fmt" + "io" "log/slog" "os" "os/exec" @@ -983,7 +984,29 @@ func seedSnapshotIndex(gitRoot, path string) bool { if !filepath.IsAbs(src) { src = filepath.Join(gitRoot, src) } - return copyFile(src, path) == nil + in, err := os.Open(src) + if err != nil { + return false + } + defer in.Close() + info, err := in.Stat() + if err != nil { + return false + } + out, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644) + if err != nil { + return false + } + _, copyErr := io.Copy(out, in) + closeErr := out.Close() + // Git compares file timestamps with the index timestamp to identify racy + // same-size edits. Giving the copy a fresh mtime would turn those edits + // into apparently clean stat-cache hits, silently omitting user drafts. + if copyErr != nil || closeErr != nil || os.Chtimes(path, info.ModTime(), info.ModTime()) != nil { + _ = os.Remove(path) + return false + } + return true } // snapshotExcludes keeps the daemon's own sidecars out of the user's snapshot. diff --git a/server/internal/daemon/execenv/local_worktree_test.go b/server/internal/daemon/execenv/local_worktree_test.go index 8bf81b38c92..5187c8d94ba 100644 --- a/server/internal/daemon/execenv/local_worktree_test.go +++ b/server/internal/daemon/execenv/local_worktree_test.go @@ -1741,6 +1741,51 @@ func TestCaptureUserSnapshotExcludesMulticaSidecars(t *testing.T) { } } +// Git must rehash a same-size edit when its timestamp equals the index's. +// A private index with a newer timestamp would incorrectly make it look clean. +func TestCaptureUserSnapshotPreservesRacyIndexTimestamp(t *testing.T) { + repo := newTestRepo(t) + gitRun(t, repo, "config", "core.trustctime", "false") + gitRun(t, repo, "config", "core.checkStat", "minimal") + path := filepath.Join(repo, "tracked.txt") + stamp := time.Now().Add(-time.Minute).Truncate(time.Second) + if err := os.Chtimes(path, stamp, stamp); err != nil { + t.Fatal(err) + } + gitRun(t, repo, "update-index", "--refresh") + indexPath := filepath.Join(repo, ".git", "index") + if err := os.Chtimes(indexPath, stamp, stamp); err != nil { + t.Fatal(err) + } + indexBefore, err := os.ReadFile(indexPath) + if err != nil { + t.Fatal(err) + } + writeFile(t, path, "modified\n") // Same byte length as "original\n". + if err := os.Chtimes(path, stamp, stamp); err != nil { + t.Fatal(err) + } + head := gitRun(t, repo, "rev-parse", "HEAD") + snapshot, err := captureUserSnapshot(repo, t.TempDir(), head, worktreeTestLogger()) + if err != nil { + t.Fatal(err) + } + if got := gitRun(t, repo, "show", snapshot+":tracked.txt"); got != "modified" { + t.Fatalf("same-size uncommitted edit was omitted from snapshot: %q", got) + } + indexAfter, err := os.ReadFile(indexPath) + if err != nil { + t.Fatal(err) + } + info, err := os.Stat(indexPath) + if err != nil { + t.Fatal(err) + } + if string(indexBefore) != string(indexAfter) || !info.ModTime().Equal(stamp) { + t.Fatal("snapshot changed the user's live index") + } +} + // The record is what proves a branch is this conversation's: an owner AND the // tip it was written at. func TestBranchRecordRoundTrips(t *testing.T) { diff --git a/server/internal/daemonws/hub.go b/server/internal/daemonws/hub.go index ce0f332f214..2ce80ccad0c 100644 --- a/server/internal/daemonws/hub.go +++ b/server/internal/daemonws/hub.go @@ -1036,6 +1036,9 @@ func (c *client) handleRPCFrame(raw json.RawMessage) { hctx, cancel = context.WithTimeout(c.ctx, time.Duration(req.TimeoutMs)*time.Millisecond) defer cancel() } + completion := &rpcCompletion{} + hctx = context.WithValue(hctx, rpcCompletionKey{}, completion) + defer completion.run() status, body, err := handler(hctx, c.identity, req.Method, req.Body) if err != nil { if status < 400 { diff --git a/server/internal/daemonws/rpc_completion.go b/server/internal/daemonws/rpc_completion.go new file mode 100644 index 00000000000..7506afb5062 --- /dev/null +++ b/server/internal/daemonws/rpc_completion.go @@ -0,0 +1,28 @@ +package daemonws + +import "context" + +type rpcCompletionKey struct{} + +type rpcCompletion struct { + callbacks []func() +} + +// AfterRPCResponse registers synchronous request cleanup to run after the RPC +// response is queued. The existing handler goroutine and concurrency slot stay +// owned until cleanup finishes; no detached worker is started. Call only from +// the RPC handler goroutine. False means this is not a hub-dispatched request. +func AfterRPCResponse(ctx context.Context, cleanup func()) bool { + completion, _ := ctx.Value(rpcCompletionKey{}).(*rpcCompletion) + if completion == nil { + return false + } + completion.callbacks = append(completion.callbacks, cleanup) + return true +} + +func (c *rpcCompletion) run() { + for _, cleanup := range c.callbacks { + cleanup() + } +} diff --git a/server/internal/daemonws/rpc_dispatch_test.go b/server/internal/daemonws/rpc_dispatch_test.go index c077d77d018..a285d23dca8 100644 --- a/server/internal/daemonws/rpc_dispatch_test.go +++ b/server/internal/daemonws/rpc_dispatch_test.go @@ -90,6 +90,46 @@ func TestRPCDispatch_RoundTrip(t *testing.T) { } } +func TestRPCDispatch_ResponsePrecedesCleanup(t *testing.T) { + hub := NewHub() + entered := make(chan struct{}) + release := make(chan struct{}) + finished := make(chan struct{}) + hub.SetRPCHandler(func(ctx context.Context, _ ClientIdentity, _ string, _ json.RawMessage) (int, json.RawMessage, error) { + if !AfterRPCResponse(ctx, func() { + close(entered) + <-release + close(finished) + }) { + t.Error("hub request has no completion phase") + } + return http.StatusOK, json.RawMessage(`{"ok":true}`), nil + }) + defer func() { + close(release) + select { + case <-finished: + case <-time.After(2 * time.Second): + t.Error("RPC cleanup did not finish") + } + }() + conn := dialRPCTestConn(t, hub, ClientIdentity{DaemonID: "daemon-cleanup", RuntimeIDs: []string{"rt-cleanup"}}) + resp := sendRPCRequest(t, conn, protocol.RPCRequestPayload{RequestID: "cleanup", Method: "tasks.claim"}) + if resp.Status != http.StatusOK || string(resp.Body) != `{"ok":true}` { + t.Fatalf("response = %+v", resp) + } + select { + case <-entered: + case <-time.After(2 * time.Second): + t.Fatal("cleanup was not called") + } + select { + case <-finished: + t.Fatal("cleanup finished before being released") + default: + } +} + // TestRPCDispatch_HandlerError maps a handler error to a non-2xx response so // the daemon can fall back to HTTP. func TestRPCDispatch_HandlerError(t *testing.T) { diff --git a/server/internal/handler/daemon.go b/server/internal/handler/daemon.go index 2daba13eeb7..23dade08163 100644 --- a/server/internal/handler/daemon.go +++ b/server/internal/handler/daemon.go @@ -1708,6 +1708,10 @@ const claimBatchMaxTasksCap = 32 // it self-heals via the heartbeat path). func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { start := time.Now() + buildDeadline, responseDeadline := batchClaimDeadlines(r.Context(), start) + buildCtx, cancelBuild := context.WithDeadline(r.Context(), buildDeadline) + defer cancelBuild() + r = r.WithContext(buildCtx) var req struct { DaemonID string `json:"daemon_id"` RuntimeIDs []string `json:"runtime_ids"` @@ -1835,6 +1839,9 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { out := make([]AgentTaskResponse, 0, len(claimed)) for i := range claimed { + if r.Context().Err() != nil { + break + } task := claimed[i] rt, ok := runtimeByID[uuidToString(task.RuntimeID)] if !ok { @@ -1856,8 +1863,7 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { if failure != nil { // Builder rejected this task (workspace isolation / chat-input); // it has already settled the task where the failure requires it. - // Skip it — non-settling failures leave the task dispatched for - // the reclaim path. + // Undelivered claims are recovered below with a separate budget. continue } if !rt.OwnerID.Valid { @@ -1873,20 +1879,12 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { if terr != nil { slog.Error("batch claim: generate task token failed; requeueing claim", "task_id", uuidToString(task.ID), "error", terr) - if _, rerr := h.TaskService.RequeueTaskAfterClaimFailure(r.Context(), task); rerr != nil { - slog.Error("batch claim: requeue after token-gen failure failed", - "task_id", uuidToString(task.ID), "error", rerr) - } continue } remoteMCPToken, daemonTokens, derr := remoteMCPDaemonTokenForClaim(resp, rt) if derr != nil { slog.Error("batch claim: generate Remote MCP daemon token failed; requeueing claim", "task_id", uuidToString(task.ID), "error", derr) - if _, rerr := h.TaskService.RequeueTaskAfterClaimFailure(r.Context(), task); rerr != nil { - slog.Error("batch claim: requeue after Remote MCP token failure failed", - "task_id", uuidToString(task.ID), "error", rerr) - } continue } // Route through the SAME finalization as the per-runtime endpoint so the @@ -1909,10 +1907,6 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { if ferr != nil { slog.Error("batch claim: finalize task claim failed; requeueing claim", "task_id", uuidToString(task.ID), "error", ferr) - if _, rerr := h.TaskService.RequeueTaskAfterClaimFailure(r.Context(), task); rerr != nil { - slog.Error("batch claim: requeue after finalize failure failed", - "task_id", uuidToString(task.ID), "error", rerr) - } continue } if deliveryFailure != nil { @@ -1940,8 +1934,9 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { // poll; a task that crossed fire_at during this request yields a bounded // follow-up hint and is promoted on the next claim. If the lookup fails, // omit the support bit so the daemon conservatively retains its ordinary - // PollInterval. - if len(out) < maxTasks && requestHasClientCapability(r, protocol.DaemonCapabilityClaimPollHintsV1) { + // PollInterval. Omitted claims also retain that short poll in case recovery + // failed or the task_available wakeup raced with this response. + if len(out) == len(claimed) && len(out) < maxTasks && requestHasClientCapability(r, protocol.DaemonCapabilityClaimPollHintsV1) { nextDeferred, nextErr := h.Queries.NextDeferredTaskFireAtForRuntimes(r.Context(), db.NextDeferredTaskFireAtForRuntimesParams{ RuntimeIds: authorized, RuntimeStaleSecs: service.RuntimeClaimFreshnessSeconds, @@ -1956,7 +1951,81 @@ func (h *Handler) ClaimTasksByRuntime(w http.ResponseWriter, r *http.Request) { } } } + if len(out) != len(claimed) { + // HTTP/1.1 cannot process another request until this handler returns. + // Keep the next claim off this connection while recovery is running. + w.Header().Set("Connection", "close") + } writeMeasuredJSON(w, http.StatusOK, response) + if len(out) != len(claimed) { + // Deliver successful payloads before batch recovery side effects. Cache + // invalidation, event listeners and cross-node wakeups can use their own + // timeouts, so the recovery context alone cannot bound the response wait. + // HTTP already advertises Content-Length; flushing sends the whole body. + // WS keeps cleanup in the existing RPC worker after queuing its response. + cleanup := func() { + h.requeueUndeliveredBatchClaims(r.Context(), claimed, out, responseDeadline) + } + if !daemonws.AfterRPCResponse(r.Context(), cleanup) { + _ = http.NewResponseController(w).Flush() + cleanup() + } + } +} + +const ( + batchClaimRecoveryTimeout = time.Second + batchClaimResponseMargin = 250 * time.Millisecond +) + +// Reserve recovery and response time INSIDE the installed HTTP client's budget. +// The WS response grace is for transport, not additional database work. Shorter +// caller deadlines get proportionally smaller reserves; longer ones cannot +// extend the HTTP-compatible batch budget. +func batchClaimDeadlines(ctx context.Context, start time.Time) (build, response time.Time) { + deadline := start.Add(protocol.DaemonBatchClaimRequestTimeout) + if caller, ok := ctx.Deadline(); ok && caller.Before(deadline) { + deadline = caller + } + budget := max(deadline.Sub(start), 0) + response = deadline.Add(-min(batchClaimResponseMargin, budget/10)) + build = response.Add(-min(batchClaimRecoveryTimeout, budget/4)) + return build, response +} + +// Share one deadline for recovery DB work across the batch. Post-commit cache, +// event and wakeup side effects may outlive it, so callers deliver the response +// first while retaining cleanup in the current HTTP/RPC request worker. +func (h *Handler) requeueUndeliveredBatchClaims(ctx context.Context, claimed []db.AgentTaskQueue, delivered []AgentTaskResponse, responseDeadline time.Time) { + if len(claimed) == len(delivered) { + return + } + deliveredIDs := make(map[string]bool, len(delivered)) + for _, task := range delivered { + deliveredIDs[task.ID] = true + } + deadline := time.Now().Add(batchClaimRecoveryTimeout) + if responseDeadline.Before(deadline) { + deadline = responseDeadline + } + ctx, cancel := context.WithDeadline(context.WithoutCancel(ctx), deadline) + defer cancel() + for _, task := range claimed { + if deliveredIDs[uuidToString(task.ID)] { + continue + } + if ctx.Err() != nil { + slog.Warn("batch claim: recovery budget exhausted; stale reclaim will recover remaining claims", "error", ctx.Err()) + break + } + // The existing CAS matches runtime + dispatched_at and requires an + // unstarted dispatched row. Settled, already requeued, started, and + // newer claims must remain untouched (ErrNoRows). + if _, err := h.TaskService.RequeueTaskAfterClaimFailure(ctx, task); err != nil && !errors.Is(err, pgx.ErrNoRows) { + slog.Error("batch claim: undelivered claim recovery failed; stale reclaim will recover it", + "task_id", uuidToString(task.ID), "error", err) + } + } } func claimPollHintDelay(now, fireAt time.Time) time.Duration { @@ -2118,9 +2187,9 @@ type claimBuildFailure struct { // The two failures settle differently: // // - The read FAILED (transient: DB blip, timeout, reset). Preserve the -// just-dispatched task so the stale-dispatched reclaim redelivers it, -// exactly as the chat-input load does (MUL-4351). This is the reachable -// case and the one worth fixing. +// just-dispatched task for redelivery. The batch endpoint requeues +// omitted claims within its response budget; the singular endpoint +// retains stale-dispatch recovery, as for chat-input reads (MUL-4351). // - The row is GONE (ErrNoRows). Cancel: the task can never run, and // preserving it would spin in the reclaim loop forever. Legacy FKs // currently keep every one of these states from arising — @@ -2153,10 +2222,9 @@ func (h *Handler) rejectClaimSourceLoad(ctx context.Context, task *db.AgentTaskQ } } -// rejectClaimSkillLoad settles a claim whose agent skills could not be read. -// It mirrors rejectClaimSourceLoad's transient branch — preserve the -// just-dispatched task so the stale-dispatched reclaim redelivers it — because -// this is the same kind of failure: a transient read on the claim-build path. +// rejectClaimSkillLoad preserves a claim whose agent skills could not be read. +// Like other transient source failures, the batch endpoint requeues it within +// its response budget; the singular endpoint retains stale-dispatch recovery. // // Dispatching with whatever loaded is not the safer half-measure it looks // like. LoadAgentSkills reads the whole skill set in two queries, so a failure @@ -3154,9 +3222,9 @@ func (h *Handler) buildClaimedTaskResponse(r *http.Request, task *db.AgentTaskQu inputLoadErr = err } // A read failure must NOT masquerade as "zero input". Preserve the - // just-dispatched task (the stale-dispatched reclaim redelivers it) - // and reject the claim with 5xx, rather than cancelling a valid direct - // task on a transient DB error (MUL-4351 review). + // just-dispatched task for batch recovery (or singular stale reclaim) + // and reject its payload, rather than cancelling a valid direct task + // on a transient DB error (MUL-4351 review). if inputLoadErr != nil { slog.Error("chat claim: load chat input messages failed; preserving task for redelivery", "task_id", uuidToString(task.ID), diff --git a/server/internal/handler/daemon_batch_claim_deadline_test.go b/server/internal/handler/daemon_batch_claim_deadline_test.go new file mode 100644 index 00000000000..8fed75ade2f --- /dev/null +++ b/server/internal/handler/daemon_batch_claim_deadline_test.go @@ -0,0 +1,269 @@ +package handler + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gorilla/websocket" + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgtype" + "github.com/multica-ai/multica/server/internal/daemonws" + "github.com/multica-ai/multica/server/internal/middleware" + "github.com/multica-ai/multica/server/internal/service" + "github.com/multica-ai/multica/server/internal/testutil" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/protocol" +) + +type slowBatchClaimDB struct { + db.DBTX + failingAgent pgtype.UUID + expired bool + recoveryDelay time.Duration +} + +func (d *slowBatchClaimDB) QueryRow(ctx context.Context, query string, args ...any) pgx.Row { + if !d.expired && strings.Contains(query, "-- name: GetAgent :one") && args[0] == d.failingAgent { + d.expired = true + <-ctx.Done() + return errRow{err: ctx.Err()} + } + if strings.Contains(query, "-- name: RequeueAgentTaskAfterClaimFailure :one") && ctx.Err() == nil { + timer := time.NewTimer(d.recoveryDelay) + defer timer.Stop() + select { + case <-timer.C: + case <-ctx.Done(): + return errRow{err: ctx.Err()} + } + } + return d.DBTX.QueryRow(ctx, query, args...) +} + +// A builds successfully, B exhausts the build deadline, and recovery remains +// slow. A must still reach installed WS (7s) and HTTP (5s) clients. This failed +// at about 8s with the previous independent five-second recovery budget. +func TestClaimTasksByRuntime_PartialSuccessFitsClientDeadlines(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + for _, transport := range []string{"ws", "http"} { + for _, recoveryDelay := range []time.Duration{0, 3 * time.Second} { + t.Run(fmt.Sprintf("%s/recovery_%s", transport, recoveryDelay), func(t *testing.T) { + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "Partial claim runtime") + agentA, issueA := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Successful claim") + taskA := seedQueuedIssueTask(t, ctx, agentA, runtimeID, issueA) + agentB, issueB := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Timed out claim") + taskB := seedQueuedIssueTask(t, ctx, agentB, runtimeID, issueB) + dbfx.Exec(t, `UPDATE agent_task_queue SET priority = 100 WHERE id = $1`, taskA) + + h := *testHandler + h.Queries = db.New(&slowBatchClaimDB{DBTX: testPool, failingAgent: parseUUID(agentB)}) + hub := daemonws.NewHub() + recoveryDB := &slowBatchClaimDB{DBTX: testPool, expired: true, recoveryDelay: recoveryDelay} + h.TaskService = service.NewTaskService(db.New(recoveryDB), testPool, testHandler.TaskService.Hub, testHandler.TaskService.Bus, hub) + completed := make(chan struct{}) + hub.SetRPCHandler(func(ctx context.Context, identity daemonws.ClientIdentity, method string, body json.RawMessage) (int, json.RawMessage, error) { + status, response, err := h.DaemonRPCHandler(ctx, identity, method, body) + daemonws.AfterRPCResponse(ctx, func() { close(completed) }) + return status, response, err + }) + identity := daemonws.ClientIdentity{ + DaemonID: batchClaimTestDaemonID, UserID: testUserID, + WorkspaceID: testWorkspaceID, WorkspaceIDs: []string{testWorkspaceID}, + RuntimeIDs: []string{runtimeID}, Capabilities: protocol.DaemonCapabilityClaimPollHintsV1, + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if transport == "ws" { + hub.HandleWebSocket(w, r, identity) + return + } + defer close(completed) + r.Header.Set("X-Client-Capabilities", identity.Capabilities) + h.ClaimTasksByRuntime(w, r.WithContext(middleware.WithDaemonContext(r.Context(), testWorkspaceID, batchClaimTestDaemonID))) + })) + defer server.Close() + body, err := json.Marshal(map[string]any{ + "daemon_id": batchClaimTestDaemonID, "runtime_ids": []string{runtimeID}, "max_tasks": 2, + }) + if err != nil { + t.Fatal(err) + } + var response batchClaimResponse + var readErr error + start := time.Now() + if transport == "ws" { + conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http"), nil) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + payload, err := json.Marshal(protocol.RPCRequestPayload{ + RequestID: "partial", Method: "tasks.claim", TimeoutMs: 5000, Body: body, + }) + if err != nil { + t.Fatal(err) + } + if err := conn.WriteJSON(protocol.Message{Type: protocol.EventDaemonRPCRequest, Payload: payload}); err != nil { + t.Fatal(err) + } + if err := conn.SetReadDeadline(start.Add(7 * time.Second)); err != nil { + t.Fatal(err) + } + for { + var msg protocol.Message + readErr = conn.ReadJSON(&msg) + if readErr != nil { + break + } + if msg.Type != protocol.EventDaemonRPCResponse { + continue + } + var rpc protocol.RPCResponsePayload + if readErr = json.Unmarshal(msg.Payload, &rpc); readErr == nil { + if rpc.Status != http.StatusOK { + readErr = fmt.Errorf("RPC status %d: %s", rpc.Status, rpc.Error) + } else { + readErr = json.Unmarshal(rpc.Body, &response) + } + } + break + } + } else { + client := &http.Client{Timeout: 5 * time.Second} + resp, err := client.Post(server.URL, "application/json", bytes.NewReader(body)) + readErr = err + if err == nil { + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + readErr = fmt.Errorf("HTTP status %d", resp.StatusCode) + } else { + readErr = json.NewDecoder(resp.Body).Decode(&response) + } + } + } + elapsed := time.Since(start) + // Collect the handler even when exercising the broken implementation. + select { + case <-completed: + case <-time.After(10 * time.Second): + t.Fatal("claim handler did not finish") + } + if readErr != nil { + t.Fatalf("successful task missed %s client deadline after %s: %v", transport, elapsed, readErr) + } + if elapsed >= 5*time.Second { + t.Fatalf("claim took %s; must leave response time inside the existing 5s budget", elapsed) + } + if len(response.Tasks) != 1 || response.Tasks[0].ID != taskA || response.Tasks[0].AuthToken == "" || response.ClaimPollHintSupported { + t.Fatalf("partial response lost A or enabled long polling: %+v", response) + } + rowA, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(taskA)) + if err != nil || rowA.Status != "dispatched" || rowA.StartedAt.Valid { + t.Fatalf("successful claim changed: task=%+v err=%v", rowA, err) + } + wantB := "queued" + if recoveryDelay > 0 { + wantB = "dispatched" // Slow recovery is cancelled; stale reclaim remains safe. + } + rowB, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(taskB)) + if err != nil || rowB.Status != wantB { + t.Fatalf("omitted claim status=%s err=%v, want %s", rowB.Status, err, wantB) + } + t.Logf("%s response in %s; successful task delivered; omitted task %s", transport, elapsed.Round(time.Millisecond), rowB.Status) + }) + } + } +} + +func TestBatchClaimDeadlines(t *testing.T) { + start := time.Now() + for _, tt := range []struct { + name string + caller time.Duration + build, response time.Duration + }{ + {name: "HTTP without deadline", build: 3750 * time.Millisecond, response: 4750 * time.Millisecond}, + {name: "WS budget", caller: 5 * time.Second, build: 3750 * time.Millisecond, response: 4750 * time.Millisecond}, + {name: "shorter caller", caller: 2 * time.Second, build: 1300 * time.Millisecond, response: 1800 * time.Millisecond}, + {name: "longer caller", caller: 30 * time.Second, build: 3750 * time.Millisecond, response: 4750 * time.Millisecond}, + {name: "expired caller", caller: -time.Second, build: -time.Second, response: -time.Second}, + } { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + if tt.caller != 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithDeadline(ctx, start.Add(tt.caller)) + defer cancel() + } + build, response := batchClaimDeadlines(ctx, start) + if build.Sub(start) != tt.build || response.Sub(start) != tt.response { + t.Fatalf("build=%s response=%s, want %s/%s", build.Sub(start), response.Sub(start), tt.build, tt.response) + } + }) + } +} + +type retryChatInputDB struct { + db.DBTX + failures int +} + +func (d *retryChatInputDB) Query(ctx context.Context, query string, args ...any) (pgx.Rows, error) { + if d.failures > 0 && strings.Contains(query, "-- name: ListChatInputMessages :many") { + d.failures-- + return nil, context.DeadlineExceeded + } + return d.DBTX.Query(ctx, query, args...) +} + +func TestClaimTasksByRuntime_ChatRecoversAfterRepeatedInputFailures(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "Chat recovery runtime") + agentID := dbfx.Agent(t, "Chat recovery agent", runtimeID) + chatID := dbfx.ChatSession(t, agentID, testutil.Cols{"runtime_id": runtimeID}) + taskID := dbfx.Task(t, agentID, testutil.Cols{"runtime_id": runtimeID, "chat_session_id": chatID}) + dbfx.Exec(t, `UPDATE agent_task_queue SET chat_input_task_id = id WHERE id = $1`, taskID) + const input = "preserve this chat input through three failed claims" + dbfx.Insert(t, "chat_message", testutil.Cols{ + "chat_session_id": chatID, "task_id": taskID, "role": "user", "content": input, + }) + h := *testHandler + h.Queries = db.New(&retryChatInputDB{DBTX: testPool, failures: 3}) + h.TaskService = service.NewTaskService(testHandler.Queries, testPool, testHandler.TaskService.Hub, testHandler.TaskService.Bus) + claim := func() []AgentTaskResponse { + t.Helper() + var response struct { + Tasks []AgentTaskResponse `json:"tasks"` + } + testutil.Call(t, h.ClaimTasksByRuntime, batchClaimRequest(testWorkspaceID, []string{runtimeID}, 1, "")).Want(http.StatusOK).JSON(&response) + return response.Tasks + } + for attempt := 0; attempt < 3; attempt++ { + if tasks := claim(); len(tasks) != 0 { + t.Fatalf("failed input read returned a task: %+v", tasks) + } + task, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(taskID)) + if err != nil || task.Status != "queued" || task.DispatchedAt.Valid || task.ChatInputTaskID != task.ID { + t.Fatalf("claim %d did not preserve retryable chat input: task=%+v err=%v", attempt+1, task, err) + } + } + tasks := claim() // No stale-recovery wait between attempts. + if len(tasks) != 1 || tasks[0].ID != taskID || !strings.Contains(tasks[0].ChatMessage, input) || tasks[0].AuthToken == "" { + t.Fatalf("retry did not deliver the original chat input: %+v", tasks) + } + if duplicate := claim(); len(duplicate) != 0 { + t.Fatalf("successful chat delivery was duplicated: %+v", duplicate) + } +} diff --git a/server/internal/handler/daemon_batch_claim_notifications_test.go b/server/internal/handler/daemon_batch_claim_notifications_test.go new file mode 100644 index 00000000000..29efad8b620 --- /dev/null +++ b/server/internal/handler/daemon_batch_claim_notifications_test.go @@ -0,0 +1,346 @@ +package handler + +import ( + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/http/httptrace" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + chimw "github.com/go-chi/chi/v5/middleware" + "github.com/gorilla/websocket" + "github.com/jackc/pgx/v5" + "github.com/multica-ai/multica/server/internal/daemonws" + "github.com/multica-ai/multica/server/internal/events" + "github.com/multica-ai/multica/server/internal/middleware" + "github.com/multica-ai/multica/server/internal/realtime" + "github.com/multica-ai/multica/server/internal/service" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/protocol" + "github.com/redis/go-redis/v9" +) + +// Exercise the production notifier/relay. XADD honors the actual context the +// relay supplies, including its independent two-second timeout. +type stalledWakeupRedis struct { + redis.UniversalClient + calls atomic.Int64 +} + +func (r *stalledWakeupRedis) XAdd(ctx context.Context, _ *redis.XAddArgs) *redis.StringCmd { + r.calls.Add(1) + <-ctx.Done() + cmd := redis.NewStringCmd(ctx) + cmd.SetErr(ctx.Err()) + return cmd +} + +type stalledClaimCache struct { + redis.UniversalClient + calls atomic.Int64 +} + +func (r *stalledClaimCache) ZRem(ctx context.Context, _ string, _ ...any) *redis.IntCmd { + r.calls.Add(1) + <-ctx.Done() + cmd := redis.NewIntCmd(ctx) + cmd.SetErr(ctx.Err()) + return cmd +} + +func (r *stalledClaimCache) Pipeline() redis.Pipeliner { return &stalledClaimPipeline{cache: r} } + +type stalledClaimPipeline struct { + redis.Pipeliner + cache *stalledClaimCache +} + +func (p *stalledClaimPipeline) Incr(ctx context.Context, _ string) *redis.IntCmd { + return redis.NewIntCmd(ctx) +} + +func (p *stalledClaimPipeline) Expire(ctx context.Context, _ string, _ time.Duration) *redis.BoolCmd { + return redis.NewBoolCmd(ctx) +} + +func (p *stalledClaimPipeline) Exec(ctx context.Context) ([]redis.Cmder, error) { + p.cache.calls.Add(1) + <-ctx.Done() + return nil, ctx.Err() +} + +type claimRecoveryCacheDB struct { + db.DBTX + tasks *service.TaskService + cache *stalledClaimCache +} + +func (d *claimRecoveryCacheDB) QueryRow(ctx context.Context, query string, args ...any) pgx.Row { + // Only stall recovery caches, not the preceding ordinary claim operations. + if ctx.Err() == nil && strings.Contains(query, "-- name: RequeueAgentTaskAfterClaimFailure :one") { + d.tasks.EmptyClaim = service.NewEmptyClaimCache(d.cache) + d.tasks.ReclaimCheck = service.NewReclaimCheckCache(d.cache) + } + return d.DBTX.QueryRow(ctx, query, args...) +} + +func TestClaimTasksByRuntime_HTTPFollowupDoesNotWaitForRecovery(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "HTTP follow-up runtime") + agentA, issueA := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Delivered before recovery") + taskA := seedQueuedIssueTask(t, ctx, agentA, runtimeID, issueA) + agentB, issueB := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Retried during recovery") + taskB := seedQueuedIssueTask(t, ctx, agentB, runtimeID, issueB) + dbfx.Exec(t, `UPDATE agent_task_queue SET priority = 100 WHERE id = $1`, taskA) + + // Hold the first handler inside its post-response recovery, after the + // omitted task is queued. A follow-up claim must run before this is released. + bus := events.New() + eventEntered, eventRelease := make(chan struct{}), make(chan struct{}) + releaseEvent := sync.OnceFunc(func() { close(eventRelease) }) + bus.Subscribe(protocol.EventTaskQueued, func(events.Event) { + close(eventEntered) + <-eventRelease + }) + h := *testHandler + h.TaskService = service.NewTaskService(testHandler.Queries, testPool, testHandler.TaskService.Hub, bus) + firstHandler := h + firstHandler.Queries = db.New(&slowBatchClaimDB{DBTX: testPool, failingAgent: parseUUID(agentB)}) + var requests atomic.Int64 + firstCompleted := make(chan struct{}) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + wrapped := chimw.NewWrapResponseWriter(chimw.NewWrapResponseWriter(w, r.ProtoMajor), r.ProtoMajor) + r = r.WithContext(middleware.WithDaemonContext(r.Context(), testWorkspaceID, batchClaimTestDaemonID)) + if requests.Add(1) == 1 { + defer close(firstCompleted) + firstHandler.ClaimTasksByRuntime(wrapped, r) + return + } + h.ClaimTasksByRuntime(wrapped, r) + })) + defer server.Close() + defer releaseEvent() // Unblock the handler before server shutdown on failure. + client := &http.Client{ + Timeout: protocol.DaemonBatchClaimRequestTimeout, + Transport: http.DefaultTransport.(*http.Transport).Clone(), + } + defer client.CloseIdleConnections() + body, err := json.Marshal(map[string]any{ + "daemon_id": batchClaimTestDaemonID, "runtime_ids": []string{runtimeID}, "max_tasks": 2, + }) + if err != nil { + t.Fatal(err) + } + claim := func() (batchClaimResponse, bool) { + t.Helper() + var reused atomic.Bool + traceCtx := httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{GotConn: func(info httptrace.GotConnInfo) { + reused.Store(info.Reused) + }}) + req, err := http.NewRequestWithContext(traceCtx, http.MethodPost, server.URL, bytes.NewReader(body)) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Content-Type", "application/json") + resp, err := client.Do(req) + if err != nil { + t.Fatalf("claim blocked by the preceding handler's recovery (connection reused=%t): %v", reused.Load(), err) + } + defer resp.Body.Close() + data, err := io.ReadAll(resp.Body) // Consume the full body so keep-alive is eligible. + if err != nil { + t.Fatal(err) + } + if resp.StatusCode != http.StatusOK { + t.Fatalf("claim status=%d: %s", resp.StatusCode, data) + } + var response batchClaimResponse + if err := json.Unmarshal(data, &response); err != nil { + t.Fatal(err) + } + return response, reused.Load() + } + first, _ := claim() + if len(first.Tasks) != 1 || first.Tasks[0].ID != taskA || first.Tasks[0].AuthToken == "" { + t.Fatalf("first claim lost the successful payload: %+v", first) + } + select { + case <-eventEntered: + case <-time.After(3 * time.Second): + t.Fatal("first claim did not reach recovery notification") + } + start := time.Now() + second, reused := claim() + elapsed := time.Since(start) + if len(second.Tasks) != 1 || second.Tasks[0].ID != taskB || second.Tasks[0].AuthToken == "" { + t.Fatalf("follow-up claim lost the recovered payload: %+v", second) + } + if reused { + t.Fatal("follow-up claim reused the connection held by recovery") + } + select { + case <-firstCompleted: + t.Fatal("recovery finished before the follow-up claim; regression was not exercised") + default: + } + // Only responses with omissions need a new connection. Complete batches + // must retain ordinary keep-alive and must not deliver either task twice. + third, reused := claim() + if len(third.Tasks) != 0 || !reused { + t.Fatalf("complete batch did not retain keep-alive or duplicated delivery: reused=%t tasks=%+v", reused, third.Tasks) + } + releaseEvent() + select { + case <-firstCompleted: + case <-time.After(3 * time.Second): + t.Fatal("first claim did not finish recovery") + } + t.Logf("follow-up delivered the recovered task in %s while the first handler remained blocked", elapsed.Round(time.Millisecond)) +} + +func TestClaimTasksByRuntime_ResponsePrecedesRecoveryNotifications(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + for _, transport := range []string{"http", "ws"} { + t.Run(transport, func(t *testing.T) { + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "Notification deadline runtime") + agentA, issueA := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Successful payload") + taskA := seedQueuedIssueTask(t, ctx, agentA, runtimeID, issueA) + agentB, issueB := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Failed payload") + taskB := seedQueuedIssueTask(t, ctx, agentB, runtimeID, issueB) + dbfx.Exec(t, `UPDATE agent_task_queue SET priority = 100 WHERE id = $1`, taskA) + h := *testHandler + h.Queries = db.New(&slowBatchClaimDB{DBTX: testPool, failingAgent: parseUUID(agentB)}) + hub := daemonws.NewHub() + rdb := &stalledWakeupRedis{} + relay := realtime.NewShardedStreamRelay(testHandler.TaskService.Hub, rdb, rdb, realtime.DefaultShardedStreamRelayConfig()) + bus := events.New() + eventEntered, eventRelease := make(chan struct{}), make(chan struct{}) + releaseEvent := sync.OnceFunc(func() { close(eventRelease) }) + bus.Subscribe(protocol.EventTaskQueued, func(events.Event) { + close(eventEntered) + <-eventRelease + }) + h.TaskService = service.NewTaskService(testHandler.Queries, testPool, testHandler.TaskService.Hub, bus, daemonws.NewRelayNotifier(hub, relay)) + cache := &stalledClaimCache{} + h.TaskService.Queries = db.New(&claimRecoveryCacheDB{DBTX: testPool, tasks: h.TaskService, cache: cache}) + completed := make(chan struct{}) + hub.SetRPCHandler(func(ctx context.Context, identity daemonws.ClientIdentity, method string, body json.RawMessage) (int, json.RawMessage, error) { + status, response, err := h.DaemonRPCHandler(ctx, identity, method, body) + daemonws.AfterRPCResponse(ctx, func() { close(completed) }) + return status, response, err + }) + identity := daemonws.ClientIdentity{ + DaemonID: batchClaimTestDaemonID, UserID: testUserID, + WorkspaceID: testWorkspaceID, WorkspaceIDs: []string{testWorkspaceID}, + RuntimeIDs: []string{runtimeID}, Capabilities: protocol.DaemonCapabilityClaimPollHintsV1, + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if transport == "ws" { + hub.HandleWebSocket(w, r, identity) + return + } + defer close(completed) + // The production logger and metrics both wrap the HTTP writer. + wrapped := chimw.NewWrapResponseWriter(chimw.NewWrapResponseWriter(w, r.ProtoMajor), r.ProtoMajor) + h.ClaimTasksByRuntime(wrapped, r.WithContext(middleware.WithDaemonContext(r.Context(), testWorkspaceID, batchClaimTestDaemonID))) + })) + defer server.Close() + defer releaseEvent() // Release a blocked event before server shutdown on failure. + body, err := json.Marshal(map[string]any{ + "daemon_id": batchClaimTestDaemonID, "runtime_ids": []string{runtimeID}, "max_tasks": 2, + }) + if err != nil { + t.Fatal(err) + } + start := time.Now() + var response batchClaimResponse + var clientErr error + var conn *websocket.Conn + if transport == "http" { + client := &http.Client{Timeout: 5 * time.Second} + resp, err := client.Post(server.URL, "application/json", bytes.NewReader(body)) + clientErr = err + if resp != nil { + clientErr = json.NewDecoder(resp.Body).Decode(&response) + resp.Body.Close() + } + } else { + conn, _, err = websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http"), nil) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + payload, err := json.Marshal(protocol.RPCRequestPayload{RequestID: "notifications", Method: "tasks.claim", TimeoutMs: 5000, Body: body}) + if err != nil { + t.Fatal(err) + } + if err := conn.WriteJSON(protocol.Message{Type: protocol.EventDaemonRPCRequest, Payload: payload}); err != nil { + t.Fatal(err) + } + if err := conn.SetReadDeadline(start.Add(7 * time.Second)); err != nil { + t.Fatal(err) + } + var msg protocol.Message + clientErr = conn.ReadJSON(&msg) + if clientErr == nil { + if msg.Type != protocol.EventDaemonRPCResponse { + t.Fatalf("message before recovery notification = %q, want RPC response", msg.Type) + } + var rpc protocol.RPCResponsePayload + if err := json.Unmarshal(msg.Payload, &rpc); err != nil { + t.Fatal(err) + } + clientErr = json.Unmarshal(rpc.Body, &response) + } + } + elapsed := time.Since(start) + select { + case <-eventEntered: + case <-time.After(3 * time.Second): + releaseEvent() + t.Fatal("recovery did not publish its queued event") + } + releaseEvent() + select { + case <-completed: + case <-time.After(10 * time.Second): + t.Fatal("recovery did not finish in the request worker") + } + if clientErr != nil || elapsed >= 5*time.Second { + t.Fatalf("%s payload waited for recovery notifications: elapsed=%s error=%v", transport, elapsed, clientErr) + } + if len(response.Tasks) != 1 || response.Tasks[0].ID != taskA || response.Tasks[0].AuthToken == "" { + t.Fatalf("successful payload missing: %+v", response) + } + if cache.calls.Load() != 2 || rdb.calls.Load() != 1 { + t.Fatalf("recovery skipped side effects: cache=%d wakeup=%d", cache.calls.Load(), rdb.calls.Load()) + } + for _, task := range []struct{ id, status string }{{taskA, "dispatched"}, {taskB, "queued"}} { + row, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(task.id)) + if err != nil || row.Status != task.status { + t.Fatalf("task %s status=%s error=%v, want %s", task.id, row.Status, err, task.status) + } + } + if conn != nil { + var wakeup protocol.Message + if err := conn.ReadJSON(&wakeup); err != nil || wakeup.Type != protocol.EventDaemonTaskAvailable { + t.Fatalf("local wakeup was lost: type=%s err=%v", wakeup.Type, err) + } + } + t.Logf("%s received complete payload in %s; slow caches, event and production relay completed afterward", transport, elapsed.Round(time.Millisecond)) + }) + } +} diff --git a/server/internal/handler/daemon_batch_claim_recovery_test.go b/server/internal/handler/daemon_batch_claim_recovery_test.go new file mode 100644 index 00000000000..50af86b1d13 --- /dev/null +++ b/server/internal/handler/daemon_batch_claim_recovery_test.go @@ -0,0 +1,200 @@ +package handler + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gorilla/websocket" + "github.com/jackc/pgx/v5" + "github.com/multica-ai/multica/server/internal/daemonws" + "github.com/multica-ai/multica/server/internal/service" + "github.com/multica-ai/multica/server/internal/testutil" + db "github.com/multica-ai/multica/server/pkg/db/generated" + "github.com/multica-ai/multica/server/pkg/protocol" +) + +// Expire the request only after the service has committed the claim. The next +// request uses the same live WebSocket and database, without the injected stall. +type expireClaimBuildDB struct { + db.DBTX + expired bool +} + +func (d *expireClaimBuildDB) QueryRow(ctx context.Context, query string, args ...any) pgx.Row { + if !d.expired && strings.Contains(query, "-- name: GetWorkspace :one") { + d.expired = true + <-ctx.Done() + return errRow{err: ctx.Err()} + } + return d.DBTX.QueryRow(ctx, query, args...) +} + +func TestClaimTasksByRuntime_DeadlineRecoveryOverWebSocket(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "Claim deadline runtime") + agentID, issueID := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Claim deadline agent") + taskID := seedQueuedIssueTask(t, ctx, agentID, runtimeID, issueID) + + h := *testHandler + h.Queries = db.New(&expireClaimBuildDB{DBTX: testPool}) + hub := daemonws.NewHub() + h.TaskService = service.NewTaskService(testHandler.Queries, testPool, testHandler.TaskService.Hub, testHandler.TaskService.Bus, hub) + hub.SetRPCHandler(h.DaemonRPCHandler) + identity := daemonws.ClientIdentity{ + DaemonID: batchClaimTestDaemonID, UserID: testUserID, + WorkspaceID: testWorkspaceID, WorkspaceIDs: []string{testWorkspaceID}, + RuntimeIDs: []string{runtimeID}, Capabilities: protocol.DaemonCapabilityClaimPollHintsV1, + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hub.HandleWebSocket(w, r, identity) + })) + defer server.Close() + conn, _, err := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(server.URL, "http"), nil) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + + woke := false + claim := func(requestID string) batchClaimResponse { + t.Helper() + body, err := json.Marshal(map[string]any{ + "daemon_id": batchClaimTestDaemonID, "runtime_ids": []string{runtimeID}, "max_tasks": 2, + }) + if err != nil { + t.Fatal(err) + } + payload, err := json.Marshal(protocol.RPCRequestPayload{ + RequestID: requestID, Method: "tasks.claim", TimeoutMs: 5000, Body: body, + }) + if err != nil { + t.Fatal(err) + } + if err := conn.WriteJSON(protocol.Message{Type: protocol.EventDaemonRPCRequest, Payload: payload}); err != nil { + t.Fatal(err) + } + // Match the installed daemon's five-second budget plus two-second grace. + if err := conn.SetReadDeadline(time.Now().Add(protocol.DaemonBatchClaimRequestTimeout + 2*time.Second)); err != nil { + t.Fatal(err) + } + for { + var msg protocol.Message + if err := conn.ReadJSON(&msg); err != nil { + t.Fatal(err) + } + if msg.Type == protocol.EventDaemonTaskAvailable { + woke = true + continue + } + if msg.Type != protocol.EventDaemonRPCResponse { + continue + } + var rpc protocol.RPCResponsePayload + if err := json.Unmarshal(msg.Payload, &rpc); err != nil { + t.Fatal(err) + } + if rpc.RequestID != requestID || rpc.Status != http.StatusOK { + t.Fatalf("unexpected RPC response: %+v", rpc) + } + var response batchClaimResponse + if err := json.Unmarshal(rpc.Body, &response); err != nil { + t.Fatal(err) + } + return response + } + } + + failed := claim("expired-build") + if len(failed.Tasks) != 0 || failed.ClaimPollHintSupported { + t.Fatalf("failed build returned tasks or enabled the long poll: %+v", failed) + } + // Cleanup can finish after the response; its wakeup still has to arrive on + // this same healthy connection before another claim is needed. + for !woke { + var msg protocol.Message + if err := conn.ReadJSON(&msg); err != nil { + t.Fatal(err) + } + woke = msg.Type == protocol.EventDaemonTaskAvailable + } + task, err := h.Queries.GetAgentTask(ctx, parseUUID(taskID)) + if err != nil || task.Status != "queued" || task.DispatchedAt.Valid { + t.Fatalf("undelivered claim was not released: task=%+v err=%v", task, err) + } + retried := claim("retry") + if len(retried.Tasks) != 1 || retried.Tasks[0].ID != taskID || retried.Tasks[0].AuthToken == "" { + t.Fatalf("immediate retry did not deliver the task: %+v", retried) + } + if duplicate := claim("no-duplicate"); len(duplicate.Tasks) != 0 { + t.Fatalf("successful delivery was requeued: %+v", duplicate) + } +} + +func TestRequeueUndeliveredBatchClaims_PreservesOtherGenerationsAndStates(t *testing.T) { + if testHandler == nil || testPool == nil { + t.Skip("database not available") + } + for _, state := range []string{"undelivered", "delivered", "started", "cancelled", "newer claim"} { + t.Run(state, func(t *testing.T) { + ctx := context.Background() + runtimeID := createClaimReclaimRuntime(t, ctx, "Recovery fence runtime") + agentID, issueID := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Recovery fence agent") + taskID := dbfx.Task(t, agentID, testutil.Cols{ + "runtime_id": runtimeID, "issue_id": issueID, + "status": "dispatched", "dispatched_at": testutil.Raw("now()"), + }) + task, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(taskID)) + if err != nil { + t.Fatal(err) + } + var delivered []AgentTaskResponse + want := "dispatched" + switch state { + case "undelivered": + want = "queued" + case "delivered": + delivered = []AgentTaskResponse{{ID: taskID}} + case "started": + _, err = testPool.Exec(ctx, `UPDATE agent_task_queue SET started_at = now() WHERE id = $1`, taskID) + case "cancelled": + want = "cancelled" + _, err = testPool.Exec(ctx, `UPDATE agent_task_queue SET status = 'cancelled' WHERE id = $1`, taskID) + case "newer claim": + _, err = testPool.Exec(ctx, `UPDATE agent_task_queue SET dispatched_at = dispatched_at + interval '1 second' WHERE id = $1`, taskID) + } + if err != nil { + t.Fatal(err) + } + expired, cancel := context.WithDeadline(ctx, time.Now().Add(-time.Second)) + defer cancel() + claimed := []db.AgentTaskQueue{task} + if state == "delivered" { + // Exercise a partial batch, so the delivered task must be + // excluded even while another task is recovered. + otherAgent, otherIssue := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Omitted batch agent") + otherID := dbfx.Task(t, otherAgent, testutil.Cols{ + "runtime_id": runtimeID, "issue_id": otherIssue, + "status": "dispatched", "dispatched_at": testutil.Raw("now()"), + }) + other, err := testHandler.Queries.GetAgentTask(ctx, parseUUID(otherID)) + if err != nil { + t.Fatal(err) + } + claimed = append(claimed, other) + } + testHandler.requeueUndeliveredBatchClaims(expired, claimed, delivered, time.Now().Add(batchClaimRecoveryTimeout)) + got, err := testHandler.Queries.GetAgentTask(ctx, task.ID) + if err != nil || got.Status != want { + t.Fatalf("recovery status=%s err=%v, want %s", got.Status, err, want) + } + }) + } +} diff --git a/server/internal/handler/daemon_batch_claim_test.go b/server/internal/handler/daemon_batch_claim_test.go index 852ce3552b2..78de75c7ace 100644 --- a/server/internal/handler/daemon_batch_claim_test.go +++ b/server/internal/handler/daemon_batch_claim_test.go @@ -76,11 +76,12 @@ func TestClaimTasksByRuntime_ClaimPollHintSchedulesNextDeferredTask(t *testing.T ctx := context.Background() runtimeID := createClaimReclaimRuntime(t, ctx, "Batch claim deferred hint") agentID, issueID := createClaimReclaimAgentAndIssue(t, ctx, runtimeID, "Batch claim deferred hint agent") + // Match the handler's clock; a Docker database clock can be slightly ahead. dbfx.Task(t, agentID, testutil.Cols{ "runtime_id": runtimeID, "issue_id": issueID, "status": "deferred", - "fire_at": testutil.Raw("now() + interval '5 seconds'"), + "fire_at": time.Now().Add(5 * time.Second), }) hinted := testutil.Decode[batchClaimResponse](t, testHandler.ClaimTasksByRuntime, diff --git a/server/internal/service/delegated_failure_recovery_test.go b/server/internal/service/delegated_failure_recovery_test.go index ca081eba856..15bf76325e2 100644 --- a/server/internal/service/delegated_failure_recovery_test.go +++ b/server/internal/service/delegated_failure_recovery_test.go @@ -249,17 +249,44 @@ func TestDelegatedFailureRecoveryPreservesSignalWhenStatusUnavailable(t *testing t.Fatalf("unresolved recovery: tasks=%d; want no dispatch or permanent receipt", count) } if !readError { - if result, err := svc.RecoverPendingDelegatedFailures(ctx, 100); err != nil || result != (DelegatedFailureRecoverySweepResult{}) { - t.Fatalf("unresolved signal occupies the batch: %+v, %v", result, err) + pending, err := queries.ListPendingDelegatedFailureRecoveries(ctx, 100) + if err != nil { + t.Fatal(err) + } + for _, candidate := range pending { + if candidate.ID == comment.ID { + t.Fatal("unresolved signal occupies the batch") + } + } + if _, err := svc.RecoverPendingDelegatedFailures(ctx, 100); err != nil { + t.Fatalf("sweep with unresolved signal: %v", err) } createStatus() } - svc.Queries = queries - if result, err := svc.RecoverPendingDelegatedFailures(ctx, 100); err != nil || result != (DelegatedFailureRecoverySweepResult{Scanned: 1, Replayed: 1}) { - t.Fatalf("catalog repaired: %+v, %v; want one replay without repeating the failure hook", result, err) + // Packages share DATABASE_URL in CI. Keep another workspace's + // signal pending to exercise the global sweep without assuming + // this fixture is its only candidate. + other, otherSvc := seedDelegatedFailureFixture(t) + otherFailedID := other.insertWorkerTask(t, "failed", "comment", 1, 2) + if target, created, err := otherSvc.ensureDelegatedFailureRecoveryComment(ctx, otherFailedID); err != nil || target == nil || !created { + t.Fatalf("seed unrelated recovery signal: created=%v err=%v", created, err) } - if result, err := svc.RecoverPendingDelegatedFailures(ctx, 100); err != nil || result != (DelegatedFailureRecoverySweepResult{}) { - t.Fatalf("repeat sweep: %+v, %v; want no duplicate", result, err) + svc.Queries = queries + for _, phase := range []string{"catalog repaired", "repeat sweep"} { + result, err := svc.RecoverPendingDelegatedFailures(ctx, 100) + if err != nil { + t.Fatalf("%s: %+v, %v", phase, result, err) + } + // Sweep totals include other workspaces. Each signal must have + // exactly one recovery task, including after the second sweep. + for _, sourceID := range []pgtype.UUID{failedID, otherFailedID} { + fx.QueryRow(t, `SELECT count(*) FROM agent_task_queue + WHERE trigger_evidence_kind = 'delegated_failure' AND trigger_evidence_ref_id = $1`, sourceID).Scan(&count) + if count != 1 { + t.Fatalf("%s: source %s has %d recovery tasks; want one replay without repeating the failure hook", + phase, util.UUIDToString(sourceID), count) + } + } } }) } diff --git a/server/pkg/protocol/claim.go b/server/pkg/protocol/claim.go new file mode 100644 index 00000000000..75aa5f0ff86 --- /dev/null +++ b/server/pkg/protocol/claim.go @@ -0,0 +1,8 @@ +package protocol + +import "time" + +// DaemonBatchClaimRequestTimeout is the existing HTTP client timeout and WS +// execution budget. Servers must leave room inside it for claim recovery and +// response delivery so installed daemons can still receive partial batches. +const DaemonBatchClaimRequestTimeout = 5 * time.Second From 231dedb55287ab6e40fd103634a024e4d9d1baeb Mon Sep 17 00:00:00 2001 From: Jiayuan Zhang Date: Thu, 15 Oct 2026 21:54:33 +0800 Subject: [PATCH 09/51] MUL-8034: chore: clean historical comments across the codebase (#9274) * chore: clean historical comments (MUL-8034) Co-authored-by: multica-agent * docs: preserve pending Composio expiry contract Co-authored-by: multica-agent --------- Co-authored-by: multica-agent --- AGENTS.md | 1 + apps/desktop/src/main/keyboard-shortcuts.ts | 26 +- apps/desktop/src/main/navigation-guard.ts | 24 +- .../src/renderer/src/components/tab-bar.tsx | 32 +- .../renderer/src/platform/tab-coordinator.ts | 22 +- .../src/renderer/src/stores/tab-store.ts | 28 +- apps/mobile/app/(app)/[workspace]/search.tsx | 6 - .../components/chat/chat-message-list.tsx | 45 +-- .../components/issue/comment-context-menu.tsx | 20 +- .../issue/composer-attachment-row.tsx | 4 - .../mobile/components/issue/timeline-list.tsx | 81 +---- apps/mobile/data/queries/my-issues.ts | 18 +- apps/mobile/data/realtime/chat-ws-updaters.ts | 8 - .../mobile/data/realtime/issue-ws-updaters.ts | 44 +-- apps/mobile/data/schemas.ts | 8 - apps/mobile/lib/markdown/markdown-style.ts | 44 +-- apps/mobile/lib/markdown/split-markdown.ts | 47 +-- apps/web/app/[workspaceSlug]/layout.tsx | 26 +- apps/web/app/manifest.ts | 28 +- .../components/download/all-platforms.tsx | 2 - .../landing/components/download/hero.tsx | 4 - apps/web/platform/in-app-history.ts | 26 +- packages/core/agents/failure-reason.ts | 23 +- packages/core/agents/types.ts | 27 +- packages/core/api/client.ts | 8 - packages/core/api/schemas.ts | 24 -- packages/core/attachments/image-sequence.ts | 6 - packages/core/github/pull-request-status.ts | 10 - packages/core/hooks/use-file-upload.ts | 65 +--- packages/core/inbox/mutations.ts | 34 +-- packages/core/issue-statuses/mutations.ts | 32 +- packages/core/issues/cache-coordinator.ts | 47 +-- packages/core/issues/mutations.ts | 12 - packages/core/onboarding/step-order.ts | 30 +- packages/core/onboarding/types.ts | 32 +- packages/core/types/agent.ts | 4 - packages/core/types/attachment-url.ts | 23 +- packages/core/types/billing.ts | 1 - packages/core/types/composio.ts | 4 +- .../views/agents/components/agents-page.tsx | 8 - .../runtime-machine-filter-dropdown.tsx | 2 - .../use-html-preview-scroll-restore.ts | 38 +-- packages/views/auth/login-page.tsx | 12 - .../components/autopilot-dialog.tsx | 14 - .../components/autopilot-list-actions.tsx | 8 - .../autopilots/components/autopilots-page.tsx | 8 - .../chat/components/use-chat-draft-restore.ts | 75 +---- packages/views/common/picker-keys.ts | 2 - packages/views/dashboard/utils.ts | 6 - .../views/editor/attachment-preview-modal.tsx | 73 +---- packages/views/editor/attachment.tsx | 69 +---- packages/views/editor/bubble-menu.tsx | 16 - packages/views/editor/content-editor.tsx | 6 - .../views/editor/extensions/file-card.tsx | 4 - .../editor/extensions/mention-suggestion.tsx | 10 - .../extensions/slash-command-suggestion.tsx | 2 - packages/views/editor/link-hover-card.tsx | 4 - packages/views/editor/title-editor.tsx | 8 - packages/views/issues/blocked-trigger-copy.ts | 39 +-- .../views/issues/components/comment-card.tsx | 12 - .../deliverables/deliverable-details.tsx | 4 - .../views/issues/components/gantt-view.tsx | 14 - .../views/issues/components/issue-detail.tsx | 16 - .../views/issues/components/issues-header.tsx | 18 -- .../components/pickers/property-picker.tsx | 8 - .../views/issues/components/reply-input.tsx | 4 - .../views/issues/components/status-icon.tsx | 12 - .../issues/components/thread-minimap.tsx | 5 - .../issues/components/use-board-drag-pan.ts | 52 +--- .../views/issues/hooks/use-in-page-find.ts | 2 - packages/views/modals/create-issue.tsx | 2 - packages/views/modals/create-squad.tsx | 4 - .../views/navigation/use-back-or-replace.ts | 27 +- packages/views/navigation/use-row-link.ts | 43 +-- .../onboarding/source-backfill-modal.tsx | 38 +-- .../onboarding/steps/step-platform-fork.tsx | 4 - .../onboarding/steps/step-runtime-connect.tsx | 6 - .../projects/components/project-detail.tsx | 4 - .../projects/components/projects-page.tsx | 8 - packages/views/rich-content/rich-content.tsx | 8 - .../components/connect-remote-dialog.tsx | 8 - .../components/delete-runtime-dialog.tsx | 8 - .../runtimes/components/runtime-list.tsx | 4 - .../components/runtime-profiles-dialog.tsx | 6 - .../runtimes/components/usage-section.tsx | 20 -- packages/views/runtimes/utils.ts | 12 - .../settings/components/composio-tab.tsx | 10 +- .../skills/components/create-skill-dialog.tsx | 10 - .../views/skills/components/file-tree.tsx | 6 - .../runtime-local-skill-import-panel.tsx | 10 - .../skills/components/skill-detail-page.tsx | 10 - .../skills/components/skill-list-actions.tsx | 10 - .../views/skills/components/skills-page.tsx | 10 - .../squads/components/squad-detail-page.tsx | 4 - .../views/squads/components/squads-page.tsx | 8 - server/cmd/multica/cmd_agent.go | 8 - server/cmd/multica/cmd_autopilot.go | 4 - server/cmd/multica/cmd_daemon_autostart.go | 6 - server/cmd/multica/cmd_issue.go | 14 - server/cmd/multica/cmd_label.go | 2 - server/cmd/multica/cmd_project.go | 6 - server/cmd/multica/cmd_property.go | 4 - server/cmd/multica/cmd_runtime.go | 2 - server/cmd/multica/cmd_runtime_profile.go | 2 - server/cmd/multica/cmd_skill.go | 4 - server/internal/analytics/client.go | 16 +- server/internal/attribution/attribution.go | 60 +--- server/internal/daemon/agents_refresh.go | 62 +--- server/internal/daemon/daemon.go | 163 ++-------- .../internal/daemon/execenv/codex_memory.go | 60 +--- server/internal/daemon/execenv/hermes_home.go | 57 +--- .../internal/daemon/execenv/hermes_memory.go | 51 +--- .../daemon/execenv/hermes_sessions.go | 50 +--- .../daemon/execenv/openclaw_config.go | 275 +++-------------- .../daemon/execenv/reply_instructions.go | 107 +------ .../daemon/execenv/runtime_config_sections.go | 226 ++------------ .../daemon/execenv/sidecar_manifest.go | 47 +-- server/internal/handler/actor_guards.go | 97 +----- .../internal/handler/attachment_capability.go | 2 - server/internal/handler/avatar.go | 70 +---- server/internal/handler/chat.go | 10 - server/internal/handler/comment.go | 90 +----- server/internal/handler/daemon.go | 56 +--- server/internal/handler/dashboard.go | 4 - server/internal/handler/file.go | 16 - .../internal/handler/integrations_composio.go | 5 +- server/internal/handler/invitation.go | 14 - server/internal/handler/issue.go | 15 +- server/internal/handler/label.go | 8 - server/internal/handler/property.go | 12 - server/internal/handler/quick_action.go | 10 - server/internal/handler/runtime.go | 2 - .../internal/handler/runtime_model_catalog.go | 2 - server/internal/handler/runtime_models.go | 4 - server/internal/handler/runtime_profile.go | 2 - server/internal/handler/runtime_update.go | 4 - server/internal/integrations/channel/doc.go | 52 +--- .../integrations/composio/dispatch.go | 13 +- server/internal/integrations/lark/doc.go | 71 +---- server/internal/integrations/lark/outbound.go | 47 +-- .../integrations/lark/ws_connector.go | 53 +--- .../internal/integrations/wecom/markdown.go | 94 +----- .../integrations/wecom/media_download.go | 57 +--- .../internal/integrations/wecom/outbound.go | 50 +--- .../integrations/wecom/relay_outbound.go | 57 +--- .../integrations/wecom/stream_store.go | 105 +------ server/internal/integrations/wecom/types.go | 65 +--- .../integrations/wecom/typing_indicator.go | 48 +-- server/internal/metrics/wecom.go | 56 +--- server/internal/service/issue.go | 10 +- server/internal/util/path.go | 54 +--- server/pkg/agent/claude_models.go | 45 +-- server/pkg/agent/codex.go | 94 +----- server/pkg/agent/run_collect_quiet.go | 94 ++---- server/pkg/composio/doc.go | 60 +--- server/pkg/db/generated/agent.sql.go | 272 ++--------------- server/pkg/db/queries/agent.sql | 283 +++--------------- 157 files changed, 618 insertions(+), 4468 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c87d399ee1e..09feabb01da 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -122,6 +122,7 @@ Workspace-scoped queries filter by `workspace_id`; membership gates access and ` ## Change and Delivery Rules - Keep changes scoped; reuse existing patterns. Code comments are English. +- Use names, types, and tests to express behavior. Keep comments for non-obvious rationale and constraints; remove superseded plans and incident narratives when updating code. Preserve tool directives and mixed-version contracts. - Do not add internal compatibility shims, dual writes, fallback paths, or legacy adapters unless requested. This does not relax API response compatibility above. - New global pre-workspace routes use a single word or `/{noun}/{verb}`, not hyphenated root names. Update `server/internal/handler/reserved_slugs.json`, run `pnpm generate:reserved-slugs`, and commit `packages/core/paths/reserved-slugs.ts` when changing reserved slugs. - Use atomic conventional commits and the repository PR template. For releases, follow [.github/RELEASING.md](.github/RELEASING.md); default to a patch bump unless specified otherwise. diff --git a/apps/desktop/src/main/keyboard-shortcuts.ts b/apps/desktop/src/main/keyboard-shortcuts.ts index 3145b3bf762..26b525805ca 100644 --- a/apps/desktop/src/main/keyboard-shortcuts.ts +++ b/apps/desktop/src/main/keyboard-shortcuts.ts @@ -26,28 +26,10 @@ const ZOOM_MIN = -3; const ZOOM_MAX = 4.5; /** - * Inspect a `before-input-event` key and apply (or block) the matching - * window-level shortcut. Returns `true` when the caller should call - * `event.preventDefault()` — that both swallows the renderer keydown and - * prevents the application menu accelerator from firing, so we don't - * double-trigger zoom on macOS where the default menu also binds these - * keys. - * - * Why we don't rely on the menu's `zoomIn` / `zoomOut` roles: on macOS the - * default `Cmd+-` accelerator does not fire reliably across keyboard - * layouts (issue MUL-2354 — Cmd+= zooms in but Cmd+- doesn't undo it). - * Handling the shortcuts here gives identical behavior on every platform - * and every layout. - */ -/** - * Result of handleAppShortcut: - * - `false`: not handled, let Electron continue - * - `true`: handled (preventDefault), no further action - * - `"close-tab"`: Cmd/Ctrl+W intercepted — caller should send IPC to renderer - * - `"open-settings"`: Cmd/Ctrl+, intercepted — caller should route the - * request to the tabbed main window - * - `{ action: "select-tab", key }`: Cmd/Ctrl+1..9 intercepted — caller - * should route the requested browser-style tab position to the main window + * Window shortcut outcomes tell the caller whether to continue, preventDefault, + * or dispatch a renderer action. Prevent handled input from also firing the menu + * accelerator. Handle zoom here for consistent keyboard-layout behavior. + * close-tab, open-settings, and select-tab actions route to the main window. */ export type ShortcutResult = | boolean diff --git a/apps/desktop/src/main/navigation-guard.ts b/apps/desktop/src/main/navigation-guard.ts index 59651b2fdb8..498ba050aa5 100644 --- a/apps/desktop/src/main/navigation-guard.ts +++ b/apps/desktop/src/main/navigation-guard.ts @@ -1,24 +1,8 @@ /** - * Top-level navigation hardening for renderer windows. - * - * Scope note (MUL-4899) — this is origin hardening ONLY, and deliberately NOT - * the mechanism that handles in-app links. - * - * Client-side routing never fires `will-navigate`: React Router pushes history - * entries without a document load. So an in-app path — a good one like - * `/acme/issues/1` or a bad one like `/Users/me/shot.png` pasted by an agent — - * never reaches this guard at all. Bad in-app paths are a routing concern, - * answered by the renderer's 404 page. Trying to police application paths here - * would be both ineffective (the hook does not see them) and wrong (it would - * break legitimate reloads). - * - * What it does catch is a real document-level navigation away from the trusted - * renderer: `window.location = 'https://evil.example'`, an anchor Chromium - * resolves to a remote URL in the same window, a dropped file URL. These windows - * run with `webSecurity: false` (the process sandbox is enabled), so a - * foreign document landing in one would inherit privileged context. New - * windows and external links already route through `setWindowOpenHandler` → - * `openExternalSafely`; this closes the same-window path. + * Guard document-level renderer navigation to trusted origins or the packaged + * entry file. Same-window foreign documents must not inherit renderer privileges. + * Client-side route changes do not fire will-navigate and remain the router's + * responsibility. New windows and external links use openExternalSafely. */ /** The subset of BrowserWindow this module needs — keeps it unit-testable. */ diff --git a/apps/desktop/src/renderer/src/components/tab-bar.tsx b/apps/desktop/src/renderer/src/components/tab-bar.tsx index 8cd04478d2c..7ab3964cab8 100644 --- a/apps/desktop/src/renderer/src/components/tab-bar.tsx +++ b/apps/desktop/src/renderer/src/components/tab-bar.tsx @@ -47,33 +47,11 @@ import { parseIssueWindowPath } from "../../../shared/issue-window"; const TAB_SCROLL_FADE_SIZE = 24; const TAB_ENTRY_EASE = [0.22, 1, 0.36, 1] as const; -// Chrome-style merged tab: the active tab shares the content surface's fill and -// flares into it through concave bottom corners. Each flare is a small square -// whose radial gradient carves a quarter-circle notch (transparent, so the -// strip behind the flare shows through), strokes a 1px arc that continues the -// tab's side border into the content card's top ring, and fills the rest with -// the surface colour. The 0.4px spread on either side of the --surface-border -// pair anti-aliases that arc. -// -// That background-color is load-bearing, not decoration: the flare's bottom row -// overlaps the content card's top ring, and in dark mode --surface-border is -// translucent (oklch(1 0 0 / 10%)), so without an opaque layer beneath it the -// arc would composite over the ring instead of replacing it and the two keylines -// would stack into a markedly lighter line right where the straight ring meets -// the curve. It therefore has to be the strip's real backdrop — the sidebar -// wrapper's fill, which is conditional and not this file's to re-derive. -// SIDEBAR_WRAPPER_FILL_CLASS reads it off the wrapper itself. -// -// The backing may only reach as far as it is needed, though. A flare hangs 9px -// past the tab's edge, over the neighbouring tab, so an opaque square there -// prints over whatever that neighbour draws: its hover pill lost the whole -// corner it shares with the flare and read as a dark bite (MUL-6160). Masking -// the notch away removes the backing exactly where the gradient is transparent -// anyway, so the notch is a hole rather than a painted-on copy of the strip and -// shows whatever is actually behind it — bare strip usually, the neighbour's -// pill where it reaches in. The mask is opaque again by the radius where the -// arc's anti-aliasing starts, so every pixel the arc and the canvas fill cover -// keeps its backing. +// Concave tab flares need an opaque backing beneath translucent border arcs +// to avoid compositing a second keyline over the content ring. Read that backdrop +// from SIDEBAR_WRAPPER_FILL_CLASS rather than duplicating the wrapper's fill rule. +// Mask the transparent notch so the backing cannot paint over a neighboring hover +// pill, while retaining backing beneath the arc's antialiasing. const TAB_FLARE_RADIUS = 10; const tabFlareGradient = (side: "left" | "right") => { const r = TAB_FLARE_RADIUS; diff --git a/apps/desktop/src/renderer/src/platform/tab-coordinator.ts b/apps/desktop/src/renderer/src/platform/tab-coordinator.ts index 7df5b666f94..afe07e16b86 100644 --- a/apps/desktop/src/renderer/src/platform/tab-coordinator.ts +++ b/apps/desktop/src/renderer/src/platform/tab-coordinator.ts @@ -14,24 +14,10 @@ import { } from "@/stores/tab-store"; /** - * Tab Coordinator (MUL-4741 Phase 2) — the ONLY writer of the single app - * router. - * - * Architecture: the tab store is the source of truth; the router is a - * projection of the active session's URL. The Coordinator subscribes to the - * store and *reconciles* the router to `activeSession.url`, stamping every - * navigation with a token. All entry points (tab bar clicks, adapter - * push/replace, shell back/forward, error recovery) mutate the store — none - * of them touch the router. This makes invariant 1 hold by construction: - * - * - Router location change WITH a pending token → expected, consume it. - * - Router location change WITHOUT a pending token → protocol error → - * bounded recovery (re-reconcile toward the session URL, capped). - * - * The router's own history is never used: the Coordinator always navigates - * with `replace`, and per-tab back/forward is a session operation over the - * session's virtual history stack. A POP therefore can only come from - * unforeseen code and is handled by the same protocol-error path. + * Project the active tab session onto the single app router. Navigation entry + * points mutate the store; only this coordinator writes the router, using replace + * and a pending token. Untokened location changes trigger bounded reconciliation. + * Per-tab virtual history owns back/forward; router POP is a protocol error. */ let router: DataRouter | null = null; diff --git a/apps/desktop/src/renderer/src/stores/tab-store.ts b/apps/desktop/src/renderer/src/stores/tab-store.ts index 0df8289f526..f8c8be09b3c 100644 --- a/apps/desktop/src/renderer/src/stores/tab-store.ts +++ b/apps/desktop/src/renderer/src/stores/tab-store.ts @@ -5,17 +5,9 @@ import { createPersistStorage, defaultStorage } from "@multica/core/platform"; import { createSafeId } from "@multica/core/utils"; import { isReservedSlug } from "@multica/core/paths"; -// --------------------------------------------------------------------------- -// Types -// --------------------------------------------------------------------------- -// -// MUL-4741 Phase 2: tabs are *sessions*, not routers. A TabSession is pure -// serializable state — URL, identity, virtual history, and a memento of -// restorable view state. The single app router is a projection of the active -// session, driven exclusively by the tab Coordinator (src/platform/ -// tab-coordinator.ts) which subscribes to this store and reconciles the -// router to `activeSession.url` with a navigation token. Nothing in this -// file touches react-router. +// Tab sessions hold serializable URLs, identity, virtual history, and view +// mementos. The coordinator projects the active session onto the single router; +// this store does not operate the router. /** * One captured scroll position. @@ -293,9 +285,7 @@ interface TabStore { reset: () => void; } -// --------------------------------------------------------------------------- // Session identity helpers -// --------------------------------------------------------------------------- // // A tab's icon is NOT part of this model. It is derived from `tab.url` at // render time via `routeIconForPath` (@multica/views/layout), which shares the @@ -312,9 +302,7 @@ function extractWorkspaceSlug(path: string): string | null { return first; } -// --------------------------------------------------------------------------- // URL helpers -// --------------------------------------------------------------------------- /** Split a tab URL into pathname and the search+hash suffix. */ export function splitTabUrl(url: string): { pathname: string; suffix: string } { @@ -405,9 +393,7 @@ export function sanitizeTabPath(path: string): string | null { return path; } -// --------------------------------------------------------------------------- // Session factory -// --------------------------------------------------------------------------- function createId(): string { return createSafeId(); @@ -618,9 +604,7 @@ function normalizeBrowsingHistoryTitles( return result; } -// --------------------------------------------------------------------------- // Group helpers -// --------------------------------------------------------------------------- /** * Build a group with its MRU order reconciled against the new (tabs, @@ -711,9 +695,7 @@ function buildCloseOtherTabsResult( }; } -// --------------------------------------------------------------------------- // Store -// --------------------------------------------------------------------------- export const useTabStore = create()( persist( @@ -1497,9 +1479,7 @@ function setHistoryIndex( }); } -// --------------------------------------------------------------------------- // Persisted shapes (for migration) -// --------------------------------------------------------------------------- interface V1Tab { id: string; @@ -1711,9 +1691,7 @@ export function migrateV1ToV2(v1: Partial): V2Persisted { return { activeWorkspaceSlug, byWorkspace }; } -// --------------------------------------------------------------------------- // Selectors (convenience hooks) -// --------------------------------------------------------------------------- /** * Pure non-hook helper — useful from event handlers / effects that already diff --git a/apps/mobile/app/(app)/[workspace]/search.tsx b/apps/mobile/app/(app)/[workspace]/search.tsx index 4b976102dca..d276372550c 100644 --- a/apps/mobile/app/(app)/[workspace]/search.tsx +++ b/apps/mobile/app/(app)/[workspace]/search.tsx @@ -65,9 +65,7 @@ const SEARCH_HEADER_LABELS: Record = { "h-cancelled": "issues:status.cancelled", }; -// ===================================================== // HighlightText — mobile port of web's HighlightText -// ===================================================== // Web uses an HTML which doesn't exist in RN, so we segment the // string ourselves and wrap matched parts in a styled . Same regex // escape + case-insensitive substring match as @@ -125,9 +123,7 @@ function HighlightText({ ); } -// ===================================================== // Row item types — drives the single FlatList render -// ===================================================== // RowItem + buildSearchRows live in lib/search-rows.ts so the ordering rules // (including the cancelled partition) are testable without mounting the screen. @@ -304,9 +300,7 @@ function RecentRow({ item, slug }: RecentRowProps) { ); } -// ===================================================== // Screen -// ===================================================== interface SearchResultsState { issues: SearchIssueResult[]; diff --git a/apps/mobile/components/chat/chat-message-list.tsx b/apps/mobile/components/chat/chat-message-list.tsx index b78f459a968..05cbd457f1f 100644 --- a/apps/mobile/components/chat/chat-message-list.tsx +++ b/apps/mobile/components/chat/chat-message-list.tsx @@ -1,44 +1,11 @@ /** - * Chat message list — user / assistant bubbles, oldest at top, newest at - * bottom. Initial render lands at the bottom; new arrivals auto-scroll - * when the user is anchored near the bottom; reading history is never - * yanked down. + * Chronological FlashList chat with initial bottom positioning and MVCP. + * Follow new messages only while the user is near the bottom; preserve reading + * position through asynchronous Markdown and image resizing. * - * Behavioral parity (apps/mobile/CLAUDE.md): - * - Render ALL message roles. Unknown role values are downgraded to - * "assistant" by ChatMessageSchema's `.catch()`, so this list never - * needs to silently drop a row. - * - Render `failure_reason` messages with destructive styling — same - * boolean as web's destructive bubble + failureReasonLabel(). - * - * v1 simplifications: - * - No "Replied in Ns" badge under assistant bubbles (elapsed_ms is - * parsed but not displayed). Easy v2 add — show below the bubble. - * - Attachments bound to a message but NOT referenced inline in `content` - * render as standalone cards below the bubble via `CommentAttachmentList` - * (same component the comment thread uses; mirrors web reusing - * `AttachmentList` in chat). Inline `![](url)` / `[name](url)` still flow - * through the markdown renderer and are de-duped out of the card list. - * - * Interaction: long-press inside a bubble fires a native iOS - * `ActionSheetIOS` (Copy / Select Text / Cancel). While the sheet is on - * screen the targeted bubble's border highlights. The assistant branch - * has no border baseline because its bubble has no shell — adding a 2px - * baseline would shift layout per message. See `useChatMessageLongPress` - * in `./message-long-press.tsx`. - * - * List engine: FlashList v2 (Shopify). FlatList was the original choice - * (per the now-outdated "no FlashList" baseline in apps/mobile/CLAUDE.md - * — written before FlashList v2 stabilised). FlatList's `scrollToEnd` is - * janky on variable-height lists by RN's own docs admission, and our - * markdown bubbles render in multiple async passes (Shiki highlight, - * image natural-size, lightbox provider injection) — each pass used to - * fire onContentSizeChange and trigger another forced scroll, causing - * the "open chat → feels stuck" jank. FlashList v2 replaces the manual - * scroll dance with `maintainVisibleContentPosition` - * (default-on; locks visible item across content changes) + - * `startRenderingFromBottom` (initial paint at bottom, no setTimeout - * hacks). Cell recycling also keeps scroll-up smooth. + * Render all normalized message roles and failure styling. Attachments not used + * inline appear as standalone cards. Long-press actions highlight the target + * without adding a border that would shift the layout. */ import { useMemo, useState } from "react"; import { ActivityIndicator, Pressable, View } from "react-native"; diff --git a/apps/mobile/components/issue/comment-context-menu.tsx b/apps/mobile/components/issue/comment-context-menu.tsx index 8b5caf57a55..50a6675279c 100644 --- a/apps/mobile/components/issue/comment-context-menu.tsx +++ b/apps/mobile/components/issue/comment-context-menu.tsx @@ -1,21 +1,9 @@ /** - * Long-press handler for a comment bubble. Exposes `onLongPress` (drives a - * native iOS ActionSheetIOS) and `isPressed` (drives the caller's highlight - * ring while the sheet is on screen). + * Native long-press actions for a comment bubble, with a pressed highlight + * while the sheet is visible. Reply targets the inline composer. * - * iOS-native first per apps/mobile/CLAUDE.md §UI components → waterfall step - * 1: `ActionSheetIOS.showActionSheetWithOptions`. Zero custom layout, zero - * animation, zero overflow math, zero new deps. - * - * Item set (conditional, mirrors web's comment context menu): - * Reply (stub) · React… (opens nested sheet) · Copy · Select Text · - * Copy Link · Resolve/Unresolve Thread (root only) · Delete (own only) · - * Cancel - * - * The nested React… sheet (5 quick emojis + More reactions… + Cancel) is - * fired from INSIDE the outer sheet's completion callback rather than - * inline, because iOS will refuse to present a second ActionSheet while the - * first is still dismissing — the callback runs after dismissal completes. + * Present nested reaction sheets from the outer sheet's completion callback; + * iOS cannot present the next ActionSheet while the first is dismissing. */ import { useCallback, useState } from "react"; import { ActionSheetIOS, Alert } from "react-native"; diff --git a/apps/mobile/components/issue/composer-attachment-row.tsx b/apps/mobile/components/issue/composer-attachment-row.tsx index 00f588a8bc7..4308572c31d 100644 --- a/apps/mobile/components/issue/composer-attachment-row.tsx +++ b/apps/mobile/components/issue/composer-attachment-row.tsx @@ -121,9 +121,7 @@ export function ComposerAttachmentRow({ ); } -// --------------------------------------------------------------------------- // Mention chip — small capsule, no tap (only × removes). -// --------------------------------------------------------------------------- function MentionChipView({ mention, @@ -166,9 +164,7 @@ function MentionChipView({ ); } -// --------------------------------------------------------------------------- // Attachment chip — image / file capsule with status overlay. -// --------------------------------------------------------------------------- interface AttachmentChipProps { item: ComposerAttachmentItem; diff --git a/apps/mobile/components/issue/timeline-list.tsx b/apps/mobile/components/issue/timeline-list.tsx index 20b2e513f93..a9dd10cf3f3 100644 --- a/apps/mobile/components/issue/timeline-list.tsx +++ b/apps/mobile/components/issue/timeline-list.tsx @@ -1,79 +1,12 @@ /** - * The scrolling timeline. ASC chronological — oldest at top, newest near the - * bottom (above the composer). Pull-to-refresh refetches issue + timeline. + * Chronological issue timeline backed by FlashList. + * Normal opens show the header; deep links start at the bottom and highlight a + * target when its row is rendered. MVCP keeps the viewport stable as asynchronous + * Markdown, images, and WebSocket patches change row heights. * - * Backend returns the full timeline in one shot (server-side pagination - * was dropped in #2322 — p99 ~30 entries per issue, cursor walking only - * created bugs at reply-thread boundaries). The previous "Pull to load - * older" UX and top-edge `fetchOlder` trigger are gone. - * - * Inbox deep-link — FlashList v2 `startRenderingFromBottom` (mirrors - * chat-message-list.tsx): - * When `highlightCommentId` is set, we pass - * `maintainVisibleContentPosition.startRenderingFromBottom: true` to - * FlashList and remount the list via `key={`hl-${highlightNonce}`}` once - * timeline data has arrived. FlashList's `getInitialScrollIndex()` then - * lands on the last data item; after the initial paint MVCP keeps the - * visible content stable across async resizes (Shiki highlight, image - * natural-size, WS `comment:updated`). No JS-side scroll dance. - * - * The previous implementation hand-rolled a `landed` state gate that - * re-fired `scrollToEnd` on every `onContentSizeChange`. That deadlocked - * the user at the bottom: `onScrollBeginDrag` queued `setLanded(true)` - * but a concurrent `onContentSizeChange` (from MVCP's own ScrollAnchor - * reposition, or from a markdown pass finishing) read the stale React - * closure, saw `landed === false`, and slammed the user back to bottom - * before the commit landed. State-vs-event races don't survive iOS - * markdown rendering hogging the JS thread. - * - * The matching 's `RootHighlightOverlay` fires when the - * target row enters the render window — so for a deep-link pointing - * at an old comment, the user scrolls up and the flash plays as the - * row mounts. `HIGHLIGHT_HOLD_MS` (5s) is the window for that. - * - * Why not `scrollToIndex`: it requires accurate height estimates that - * variable-height markdown bubbles can't provide, even with - * `onScrollToIndexFailed` retry. Same lesson web learned (see - * `packages/views/issues/components/issue-detail.tsx:1822-1850` — they - * split the path into virtualized-for-browse / flat-for-deep-link - * precisely because "virtualization and 'land precisely on a target' - * have fundamentally opposed contracts"). Mobile sidesteps the split - * by not pretending to land precisely. - * - * `maintainVisibleContentPosition` is enabled by default on FlashList v2 - * and is implemented inside the C++ shadow tree — it compensates the - * scroll offset both when a row is INSERTED above the viewport AND when - * an upper row RESIZES (a WS `comment:updated` / `reaction:added` / - * `resolved` event on an older comment used to push the user's read - * position down by the delta). On FlatList the same prop was iOS-only - * + JS-side; the FlashList path is steadier and animates the offset - * compensation rather than snapping it. - * - * List engine: FlashList v2 (Shopify). Migrated from FlatList because: - * 1. Cell recycling — markdown bubbles (Shiki highlight, image natural- - * size, lightbox provider injection) are expensive to mount; FlashList - * keeps them in recycled cells when scrolling through history rather - * than re-running the multi-pass render each time a row re-enters the - * window. - * 2. Native MVCP — see paragraph above. Smoother behavior than FlatList's - * JS-side implementation when WS events resize an upper row. - * 3. Async-render stability — async markdown size changes inside the - * viewport no longer cause the visual "twitch" we saw with FlatList, - * because FlashList re-layouts inside the shadow tree without - * surfacing a JS-side onContentSizeChange storm. - * - * What FlashList v2 does NOT change about this file: - * - The deep-link "land at bottom + flash on row mount" pattern (Linear - * iOS). FlashList's `scrollToIndex` is still estimate-based, so we - * intentionally don't try to land precisely on a specific comment — - * we land at the bottom and let `RootHighlightOverlay` claim the - * target when the user scrolls past it. `startRenderingFromBottom` - * is NOT enabled: a normal issue-open should show the header (title, - * description, status) first, not jump straight to the latest reply. - * - Spacing between rows. FlashList ignores `gap-*` on - * `contentContainer` the same way it does in chat-message-list.tsx — - * we use `ItemSeparatorComponent` for the 12 px breathing room and - * `ListHeaderComponentStyle` to add the same gap below the header. + * Avoid repeated scrollToEnd calls during content resizing: they can override + * user scrolling. New replies use the explicit jump-to-new chip instead of + * automatically following appends. Item separators provide spacing between rows. */ import { useCallback, useEffect, useMemo, useRef, useState } from "react"; import { diff --git a/apps/mobile/data/queries/my-issues.ts b/apps/mobile/data/queries/my-issues.ts index f7f1571f88d..f765c6a8937 100644 --- a/apps/mobile/data/queries/my-issues.ts +++ b/apps/mobile/data/queries/my-issues.ts @@ -1,18 +1,10 @@ /** - * "My Issues" list, server-filtered by scope. Mirrors the three scopes web - * exposes in `packages/views/my-issues/components/my-issues-page.tsx:48-65`: - * - assigned: issues where assignee_id = me - * - created: issues where creator_id = me - * - agents: issues where the assignee is an *indirect* extension of me — - * an owned agent, OR a squad I'm a human member of, lead, or - * have an owned agent inside. Driven server-side by the - * `involves_user_id` predicate (see MUL-2397, 2026-05-19). - * Direct member assignment is intentionally EXCLUDED — that's - * the `assigned` scope's meaning. + * Server-filtered My Issues scopes: direct assignments, creations, or involvement + * through owned agents and squad membership. The agents scope excludes direct + * member assignment, which belongs to assigned. * - * Cache key shape is `issueKeys.myList(wsId, scope, filter)` — same prefix - * as web's `packages/core/issues/queries.ts` so a future WS handler can - * invalidate `issueKeys.myAll(wsId)` and reach both clients. + * Mobile owns these query keys and its QueryClient. useMyIssuesRealtime patches + * or invalidates local lists; each client receives its own server events. */ import { queryOptions } from "@tanstack/react-query"; import { api } from "@/data/api"; diff --git a/apps/mobile/data/realtime/chat-ws-updaters.ts b/apps/mobile/data/realtime/chat-ws-updaters.ts index 4986501766a..4038cffe83e 100644 --- a/apps/mobile/data/realtime/chat-ws-updaters.ts +++ b/apps/mobile/data/realtime/chat-ws-updaters.ts @@ -37,9 +37,7 @@ import type { } from "@multica/core/types"; import { chatKeys } from "@/data/queries/chat"; -// ===================================================== // Sessions list (ChatSession[] keyed by wsId) -// ===================================================== export function patchSessionListAfterRename( qc: QueryClient, @@ -90,9 +88,7 @@ export function flipSessionUnread( ); } -// ===================================================== // Messages cache (ChatMessage[] keyed by sessionId) -// ===================================================== /** * Apply `chat:done` to the messages cache. @@ -205,9 +201,7 @@ export async function applyChatQuickActionsToCache( ); } -// ===================================================== // Pending task (ChatPendingTask keyed by sessionId) -// ===================================================== export function seedPendingTaskFromQueued( qc: QueryClient, @@ -296,9 +290,7 @@ export function seedAcceptedPendingTask( invalidatePendingTask(qc, payload.chat_session_id); } -// ===================================================== // Task messages (live timeline, keyed by taskId) -// ===================================================== /** * Append a `task:message` payload into the per-task timeline cache. diff --git a/apps/mobile/data/realtime/issue-ws-updaters.ts b/apps/mobile/data/realtime/issue-ws-updaters.ts index 60ae28f4ef2..984a5d9b27f 100644 --- a/apps/mobile/data/realtime/issue-ws-updaters.ts +++ b/apps/mobile/data/realtime/issue-ws-updaters.ts @@ -1,29 +1,9 @@ /** - * Mobile-owned WS cache patchers for the issue domain. These are pure - * functions over `QueryClient` — no React, no WS plumbing. The hooks in - * `use-issue-realtime.ts` and `use-my-issues-realtime.ts` translate WS - * events into calls into this module. - * - * Why mobile-owned (and not importing from packages/core/issues/ws-updaters): - * - Web's updaters reference `issueKeys` from `packages/core/issues/queries`, - * a different runtime instance than mobile's `data/queries/issue-keys.ts`. - * TanStack Query keys are compared structurally so this would *appear* - * to work, but binding cache mutation to a foreign key factory invites - * drift the moment either side adjusts its key shape. - * - Mobile cache shapes are simpler: no status-bucketed list, no children - * subtree, no childProgress, no label-byIssue cache. A direct mirror - * would be ~120 lines of conditional dead-code paths. - * - * Cache shapes (the design contract here): - * - Issue detail: Issue (keyed by detail(wsId, id)) - * - Issue timeline: TimelineEntry[] (keyed by timeline(wsId, id)) - * ASC oldest-first; new entries inserted at sorted position. - * - My Issues list: Issue[] (keyed by myList(wsId, scope, filter)) - * Multiple list caches per wsId (one per scope/filter combo). - * Patch ALL of them via setQueriesData on myAll(wsId). - * - Workspace list: Issue[] (keyed by list(wsId)) - * Single cache per wsId (no scope/filter in the key — - * filtering happens client-side off the same list). + * Mobile-owned issue cache patchers, called by record and listing subscriptions. + * Use mobile's key factories and cache shapes rather than web's bucketed caches. + * Detail stores an Issue; timelines store chronological TimelineEntry arrays; + * My Issues stores one Issue array per scope/filter and patches all matching keys; + * the workspace list stores a single client-filtered Issue array. */ import type { QueryClient } from "@tanstack/react-query"; import type { @@ -122,9 +102,7 @@ function acceptsRevision(current: number | undefined, incoming: number | undefin return incoming !== undefined && incoming >= current; } -// ===================================================== // Issue detail cache (single Issue per id) -// ===================================================== export function patchIssueDetail( qc: QueryClient, @@ -233,9 +211,7 @@ export function invalidateIssueAfterReconnect( qc.invalidateQueries({ queryKey: issueKeys.tasks(wsId, issueId) }); } -// ===================================================== // Issue timeline (flat TimelineEntry[], ASC oldest-first) -// ===================================================== export function appendTimelineEntry( qc: QueryClient, @@ -385,9 +361,7 @@ export function removeCommentCascade( ); } -// ===================================================== // My Issues list (flat Issue[] across many keys) -// ===================================================== export function patchMyIssuesList( qc: QueryClient, @@ -420,9 +394,7 @@ export function removeFromMyIssuesList( ); } -// ===================================================== // Workspace Issues list (flat Issue[] under list(wsId)) -// ===================================================== export function patchIssuesList( qc: QueryClient, @@ -464,9 +436,7 @@ export function removeFromIssuesList( ); } -// ===================================================== // Reactions -// ===================================================== export function addCommentReaction( qc: QueryClient, @@ -599,9 +569,7 @@ export function removeIssueReaction( ); } -// ===================================================== // Labels -// ===================================================== export function patchIssueLabels( qc: QueryClient, @@ -637,9 +605,7 @@ export function patchIssueLabels( onIssueAuxiliaryRevision(qc, wsId, issueId, issueRevision, "labels"); } -// ===================================================== // Helpers — payload normalization -// ===================================================== /** * Convert a Comment WS payload into a TimelineEntry. The two types share diff --git a/apps/mobile/data/schemas.ts b/apps/mobile/data/schemas.ts index 18aaddb1f9c..436155225dd 100644 --- a/apps/mobile/data/schemas.ts +++ b/apps/mobile/data/schemas.ts @@ -238,9 +238,7 @@ export const EMPTY_LIST_PROJECT_RESOURCES_RESPONSE: ListProjectResourcesResponse total: 0, }; -// ===================================================== // Chat (sessions / messages / pending task) -// ===================================================== // Lenient on every field that's purely informational (status enum, timestamps, // agent/creator ids). `.loose()` so server-added fields pass through. The two // fields mobile keys behaviour on — `id` and `chat_session_id` — are required. @@ -363,9 +361,7 @@ export const TaskMessageListSchema = z.array(TaskMessagePayloadSchema).default([ export const EMPTY_TASK_MESSAGE_LIST: TaskMessagePayload[] = []; -// ===================================================== // Search (issues + projects) -// ===================================================== // Mirrors SearchIssueResult / SearchProjectResult in packages/core/types/api.ts. // Web does not currently route search responses through parseWithFallback, so // the schemas live mobile-side. Promote to core when web adopts the same @@ -401,9 +397,7 @@ export const EMPTY_SEARCH_PROJECTS_RESPONSE: SearchProjectsResponse = { projects: [], }; -// ===================================================== // Agent tasks (per-issue runs, active + history) -// ===================================================== // Mirrors AgentTask in packages/core/types/agent.ts. Backend handlers: // GET /api/issues/{id}/active-task → { tasks: AgentTask[] } (may be empty) // GET /api/issues/{id}/task-runs → AgentTask[] @@ -473,9 +467,7 @@ export interface ActiveTasksResponse { export const EMPTY_AGENT_TASK_LIST: AgentTask[] = []; export const EMPTY_ACTIVE_TASKS_RESPONSE: ActiveTasksResponse = { tasks: [] }; -// ===================================================== // User / Workspace / Inbox / Member / Agent -// ===================================================== // Mobile reads these on every cold start (auth → workspaces → inbox → members // → agents form the boot sequence). A schema drift in any of them used to // cascade — getMe failure flushed the user, listWorkspaces failure landed the diff --git a/apps/mobile/lib/markdown/markdown-style.ts b/apps/mobile/lib/markdown/markdown-style.ts index 2f3342e506e..7f2a84a7b05 100644 --- a/apps/mobile/lib/markdown/markdown-style.ts +++ b/apps/mobile/lib/markdown/markdown-style.ts @@ -179,46 +179,10 @@ export function useMarkdownStyle() { color: t.brand, underline: true, }, - // Inline code — monospace + muted-foreground tint, NO background chip. - // - // Why no background: on iOS, NSAttributedString draws - // NSBackgroundColorAttributeName as a rectangle spanning the full - // line-box height (24pt for our CJK-tuned body leading). PingFang - // SC places the baseline at ~75-80% of the line-box, so monospaced - // glyphs sit low inside that rectangle, leaving ~11pt empty at the - // top vs ~6pt at the bottom — a visible 2:1 asymmetry in CJK - // paragraphs. This is a platform-level constraint, not a library - // bug: the library's STYLES.md exposes only fontFamily / fontSize / - // color / backgroundColor / borderColor for inline code — no - // padding, lineHeight, or baselineOffset knob. Confirmed 2026-05-21 - // (researcher brief): zero open PRs / forks / prereleases touching - // inline-code vertical geometry; upstream issue #255 has no - // maintainer response since 2026-04-20. The only real fixes are at - // the native layer (custom NSLayoutManager.drawBackground override - // or TextKit 2 NSTextLayoutManager.enumerateTextSegments(.highlight)) - // which would require forking the library — not worth the perpetual - // patch-rebase cost for a polish issue. - // - // Visual identification stays strong without the chip: monospace - // font + muted-foreground tint matches Apple Notes / iA Writer. - // - // Color choice — mutedForeground (NOT brand): - // - mutedForeground is a neutral gray, reads as "subdued prose". - // - brand is reserved for links; tinting code blue confused users - // into tapping it as if it were a link (tried 2026-05-19, then - // reverted same day for this reason). - // - // fontFamily intentionally NOT set: enriched-markdown's default is - // the platform system monospace (SF Mono on iOS, monospace on - // Android), which has a larger visual x-height than the previously- - // explicit "Menlo" override. - // - // Revisit when: - // - upstream #255 ships a paddingVertical / lineHeight knob, OR - // - we contribute the TextKit 2 fix upstream (the right hook is - // NSTextLayoutManager.enumerateTextSegments(in:type:.highlight:)) - // Either way, switch back to a tinted-chip style for cross-platform - // visual parity with web/desktop. + // Keep inline code backgrounds transparent: native attributed-string chips + // span the line box and cannot be vertically padded by this renderer. Monospace + // and a neutral tint identify code without making it look like a link. + // Leave fontFamily unset to use the platform's system monospace. code: { color: t.mutedForeground, backgroundColor: "transparent", diff --git a/apps/mobile/lib/markdown/split-markdown.ts b/apps/mobile/lib/markdown/split-markdown.ts index 307ea994b0e..b723be8115e 100644 --- a/apps/mobile/lib/markdown/split-markdown.ts +++ b/apps/mobile/lib/markdown/split-markdown.ts @@ -1,47 +1,10 @@ /** - * Hybrid-render segment splitter. + * Split Markdown into native prose, React code blocks, and interactive images. + * Use lexer token boundaries so fenced/indented code and paragraph images are + * handled without fragile regex reconstruction. Preserve prose through token.raw. * - * react-native-enriched-markdown gives us native-quality rendering of - * prose (paragraphs / headings / lists / quotes / tables / inline code / - * links / mentions) but does NOT let us inject React for any leaf node - * (see issues #54, #232, #246 — maintainer's stance is "no custom - * renderers, ever, by design"). That blocks three product needs: - * - * 1. Shiki syntax-highlighted code blocks (web parity). - * 2. Tap-to-lightbox on images. - * 3. Horizontal scroll + copy button on code. - * - * Maintainer-endorsed workaround (issue #246): split the markdown at - * those boundaries, render each fragment as its own enriched instance - * for prose and as React for the rest. This file is that splitter. - * - * Strategy: - * - * - Run `marked.lexer` (we already depend on it for preprocess). - * - Walk the top-level token list. - * - `code` token → emit a `code` segment. - * - `paragraph` token containing image tokens → split the paragraph - * at each image boundary. Each image becomes its own `image` - * segment; the surrounding text rejoins the prose stream. (RN's - * can't be inline in a , and tap-to-lightbox needs a - * Pressable wrapper that's not viable inside an attributed string - * anyway. GitHub mobile and Linear iOS both promote inline images - * to block-level — we follow.) - * - Every other block stays in the prose buffer and is handed to - * enriched verbatim via `token.raw`. - * - * Why not regex: code inside lists, indented code, paragraph-embedded - * images, and consecutive code blocks without a blank line all have - * boundaries regex can't reliably detect. marked.lexer already knows; - * reuse it. - * - * Known trade-off: code blocks nested inside a list item stay with the - * list and render via enriched (no Shiki). That's because the splitter - * only inspects top-level tokens — descending into list/quote children - * would force us to also re-serialise their surrounding markdown, - * complicating prose reconstruction. Top-level code is the >95% case - * for issue descriptions / comments; revisit if real content shows - * otherwise. + * Only top-level code is extracted. Nested list/quote code stays in the native + * renderer to preserve its surrounding Markdown structure. */ import { marked, type Tokens } from "marked"; diff --git a/apps/web/app/[workspaceSlug]/layout.tsx b/apps/web/app/[workspaceSlug]/layout.tsx index f150e61d5fa..ce2744e2eae 100644 --- a/apps/web/app/[workspaceSlug]/layout.tsx +++ b/apps/web/app/[workspaceSlug]/layout.tsx @@ -54,28 +54,10 @@ export default function WorkspaceLayout({ enabled: !!user, }); - // Render-phase sync: feed the URL slug into the platform singleton so - // the first child query's X-Workspace-Slug header is already correct. - // setCurrentWorkspace self-dedupes + runs rehydrate as a side effect; - // safe to call on every render. - // - // Gated on the live pathname because this is a module-global write from - // render, and the App Router can keep a previous workspace's layout mounted - // beside the incoming one. Both instances re-render on their own query - // activity, so an unguarded write let them alternate — every render flipping - // the singleton back to its own slug, indefinitely. Each flip tore down and - // rebuilt the realtime socket (packages/core/realtime/provider.tsx binds it - // to this slug) and re-pointed the @mention lookup, which reads the same - // singleton synchronously and falls back to an empty list when the workspace - // it names has no warm cache (packages/views/editor/extensions/ - // mention-suggestion.tsx). Realtime went dead and the mention list came back - // empty for whichever workspace the user was actually looking at. - // - // Comparing against the pathname resolves it without an ownership handshake: - // the URL is the source of truth for workspace identity, and usePathname() - // is one value shared by every mounted instance, so exactly one layout — the - // routed one — can define it. Desktop needs its own protocol instead - // (workspace-route-layout.tsx) because it has no URL bar to appeal to. + // Sync routed workspace identity during render so the first child request has + // the correct workspace header. Gate this global write on the live pathname: + // App Router can retain old layouts whose rerenders would otherwise switch the + // singleton and realtime connection back to a stale workspace. if (workspace && workspaceSlug === workspaceSlugFromPathname(pathname)) { setCurrentWorkspace(workspaceSlug, workspace.id); } diff --git a/apps/web/app/manifest.ts b/apps/web/app/manifest.ts index 873108e06b3..a4c2fb49bd2 100644 --- a/apps/web/app/manifest.ts +++ b/apps/web/app/manifest.ts @@ -1,30 +1,12 @@ import type { MetadataRoute } from "next"; /** - * Web app manifest — what makes the mobile web app installable and lets it run - * in its own window instead of a browser tab. + * Installable web manifest. Start at /inbox so session-aware proxy routing + * opens the workspace or login; the marketing host's root would open the landing + * page even for a signed-in user. * - * `start_url` is deliberately NOT "/". The official marketing hosts keep the - * root path on the public site even for a signed-in session (see - * `isOfficialMarketingHost` in proxy.ts), so an installed app pointed at "/" - * would open the landing page. "/inbox" is one of `LEGACY_ROUTE_SEGMENTS`, - * which proxy.ts resolves per session: signed in with a known workspace it - * lands on that workspace's inbox, signed in without one it lands on /login - * (which resolves against the workspace list), and signed out it lands on - * /login too. All three are pinned in manifest.test.ts, because a launcher - * icon has no URL bar to recover from a wrong destination. - * - * The icons under /icons are generated, not hand-drawn. To regenerate after a - * brand change, edit public/icons/icon.svg and run from public/icons: - * - * sips -s format png --resampleHeightWidth 512 512 icon.svg --out icon-maskable-512.png - * sips -s format png --resampleHeightWidth 180 180 icon.svg --out apple-touch-icon.png - * sips -s format png --resampleHeightWidth 512 512 ../../../desktop/build/icon.png --out icon-512.png - * sips -s format png --resampleHeightWidth 192 192 ../../../desktop/build/icon.png --out icon-192.png - * - * The two `any` icons come from the desktop app icon so an installed web app - * and an installed desktop app show the same artwork; the maskable one is - * full-bleed because Android crops it to the launcher's shape. + * Standard icons share desktop artwork. The maskable SVG supplies full-bleed + * artwork for launcher cropping; regenerate raster sizes from those source assets. */ /** Launch path. Exported so manifest.test.ts can run it through the proxy. */ diff --git a/apps/web/features/landing/components/download/all-platforms.tsx b/apps/web/features/landing/components/download/all-platforms.tsx index 1c3cd0680e9..8c8f7f6ca2f 100644 --- a/apps/web/features/landing/components/download/all-platforms.tsx +++ b/apps/web/features/landing/components/download/all-platforms.tsx @@ -148,9 +148,7 @@ export function AllPlatforms({ ); } -// ------------------------------------------------------------ // Row -// ------------------------------------------------------------ interface RowProps { icon: React.ReactNode; diff --git a/apps/web/features/landing/components/download/hero.tsx b/apps/web/features/landing/components/download/hero.tsx index 0f35eb69662..13028684b41 100644 --- a/apps/web/features/landing/components/download/hero.tsx +++ b/apps/web/features/landing/components/download/hero.tsx @@ -78,9 +78,7 @@ export function DownloadHero({ ); } -// ------------------------------------------------------------ // Content resolver — maps (detect, assets) → CTA props -// ------------------------------------------------------------ interface HeroContent { title: string; @@ -207,9 +205,7 @@ export function resolveContent( }; } -// ------------------------------------------------------------ // Pieces -// ------------------------------------------------------------ function PrimaryCta({ href, diff --git a/apps/web/platform/in-app-history.ts b/apps/web/platform/in-app-history.ts index 57fbf178867..a796c04eb81 100644 --- a/apps/web/platform/in-app-history.ts +++ b/apps/web/platform/in-app-history.ts @@ -1,25 +1,9 @@ /** - * "Is there a Multica page behind this one?" for the web navigation adapter. - * - * Callers that want to step back (deleting an issue returns the user to the - * list they opened it from) must not step back off the app when the current - * page was opened cold — a shared link, a pasted URL, a new tab. - * - * Only the browser can answer this, and exactly one API does: the Navigation - * API's `entries()` spans just the contiguous same-origin run around the - * current entry, so arriving from an external site reports `canGoBack` false - * even though the browser technically has somewhere to go. `history.length` - * is no substitute — it counts other origins' entries too. - * - * Nothing else here is a guess. Counting the adapter's own `router.push` - * calls looks like a workable fallback and is not: a push is a request, not a - * committed history entry. Next drops it to `replaceState` when the canonical - * URL is unchanged (`app-router.js`, the `pendingPush && href !== canonicalUrl` - * branch), and pushes can also be superseded or abandoned mid-transition. Any - * count derived from calls can therefore claim history that does not exist, - * and a wrong `true` walks the user out of Multica — the precise failure this - * exists to prevent. So where the browser cannot answer we report `false`, - * and callers take their fallback path. + * Determine whether web navigation has a safe in-app back destination. + * Use Navigation API entries for committed same-origin history. history.length + * includes external entries, and counting router.push requests includes navigations + * that may be replaced or abandoned. Return false when the browser cannot prove + * a destination so callers use their fallback instead of leaving the app. */ /** Minimal shape of the Navigation API — not in TypeScript's DOM lib yet. */ diff --git a/packages/core/agents/failure-reason.ts b/packages/core/agents/failure-reason.ts index 1f01cdecc62..0875dab9b73 100644 --- a/packages/core/agents/failure-reason.ts +++ b/packages/core/agents/failure-reason.ts @@ -1,25 +1,10 @@ import type { TaskFailureReason } from "../types"; /** - * Picks the copy key a UI surface should render for a server-supplied - * `failure_reason`, degrading instead of falling off a cliff. - * - * Resolution order: - * 1. Exact match — the surface has copy for this precise reason. - * 2. Coarse family — `agent_error.provider_network` degrades to - * `agent_error` when the surface only has family-level copy. - * 3. `undefined` — nothing usable; the caller renders its own fallback. - * - * Why this exists: the backend's taxonomy is refined (`agent_error.*`) and - * grows independently of shipped clients. Before MUL-5370 the surfaces did an - * exact `Record` lookup against the six-value coarse enum, so every refined - * reason missed and rendered a generic "something went wrong" — the useful - * classification the backend had already computed was thrown away at the last - * step. Step 2 makes an unseen `agent_error.` land on the family - * line rather than the fallback, which is the repo's "enum drift downgrades, - * not crashes" rule applied to copy. - * - * Pure and dependency-free so mobile can import it directly. + * Resolve failure copy by exact reason, then coarse family, then caller fallback. + * The server taxonomy can grow independently of installed clients; unfamiliar + * agent_error subtypes should retain family-level guidance. Keep this helper pure + * so mobile can share it. */ export function resolveFailureReasonKey( reason: TaskFailureReason | string | null | undefined, diff --git a/packages/core/agents/types.ts b/packages/core/agents/types.ts index 3bb23ef5628..a306c928ee0 100644 --- a/packages/core/agents/types.ts +++ b/packages/core/agents/types.ts @@ -1,27 +1,6 @@ -// Derived presence types for agents — the user-facing state we display -// across the UI (list dots, hover cards, status lines). Computed in the -// front-end from raw server data (agent + runtime + recent tasks); the -// back-end never knows about these enums. -// -// Two orthogonal dimensions, derived independently and answering only -// "what's true right now?" — historical / error context lives on the -// agent detail page (Recent Work, failure_reason) and Inbox, not in the -// list-level summary state: -// -// 1. AgentAvailability — "Can this agent take work right now?" -// Depends only on runtime reachability. The dot colour everywhere in -// the app reflects this single dimension; never sticky-red because of -// a past task outcome. -// -// 2. Workload — "What is on this agent's plate right now?" -// Depends only on the workspace task snapshot. Three states, each -// pointing at a clear user action: -// working → tasks running, normal -// queued → tasks queued but nothing running (= stuck if availability -// is offline/unstable; momentary if online) -// idle → nothing to do -// No `failed` / `completed` / `cancelled` states — those are historical, -// surfaced via Recent Work + Inbox. +// Agent presence has two independent current-state dimensions: runtime +// availability and active task workload. Past failures and completed tasks belong +// in Recent Work and Inbox rather than persisting as a list-level error state. // Runtime-reachability dimension. `unstable` is the transient amber state // during the runtime sweeper's grace window (offline < 5 min); it decays diff --git a/packages/core/api/client.ts b/packages/core/api/client.ts index 265d2845718..7719f247ad7 100644 --- a/packages/core/api/client.ts +++ b/packages/core/api/client.ts @@ -2054,12 +2054,10 @@ export class ApiClient { }); } - // --------------------------------------------------------------------- // Cloud Billing — proxies to multica-cloud /api/v1/billing/*. The // multica-api server stamps X-User-ID and forwards bytes; everything // here is upstream-shaped. See packages/core/types/billing.ts for the // response field documentation. - // --------------------------------------------------------------------- async getCloudBillingBalance(): Promise { const raw = await this.fetch("/api/cloud-billing/balance"); @@ -2183,7 +2181,6 @@ export class ApiClient { ); } - // --------------------------------------------------------------------- // Workspace subscriptions — the server resolves the workspace from the // authenticated request context, so no caller names one. // @@ -2193,7 +2190,6 @@ export class ApiClient { // - non-2xx (older cloud without the route, 403, 503) throws ApiError from // `fetch`, so a React Query caller sees `isError`; // - a 2xx body that does not match the contract returns null here. - // --------------------------------------------------------------------- async getWorkspaceSubscriptionSummary(): Promise { const raw = await this.fetch("/api/cloud-subscriptions/summary"); @@ -2375,10 +2371,8 @@ export class ApiClient { }); } - // --------------------------------------------------------------------- // Custom runtime profiles (MUL-3284). All workspace-scoped: the caller // passes the workspace id the same way the runtimes list resolves it. - // --------------------------------------------------------------------- async listRuntimeProfiles(workspaceId: string): Promise { const res = await this.fetch<{ runtime_profiles?: RuntimeProfile[] }>( @@ -2520,12 +2514,10 @@ export class ApiClient { ); } - // --------------------------------------------------------------------------- // Workspace dashboard — three independent rollups for `/{slug}/dashboard`. // Each accepts an optional `project_id` to narrow the scope to one project. // Cost is computed client-side from the model pricing table (same contract // as the per-runtime endpoints above). - // --------------------------------------------------------------------------- async getDashboardUsageDaily( params: { days?: number; project_id?: string | null; tz?: string }, diff --git a/packages/core/api/schemas.ts b/packages/core/api/schemas.ts index 70703c647f0..a4999b3011b 100644 --- a/packages/core/api/schemas.ts +++ b/packages/core/api/schemas.ts @@ -780,7 +780,6 @@ export interface AppConfigResponse { server_version?: string; } -// --------------------------------------------------------------------------- // Schemas for the highest-risk API endpoints — those whose responses drive // the issue detail page (timeline, comments, subscribers) and the issues // list. These are the surfaces that white-screened in #2143 / #2147 / #2192. @@ -806,7 +805,6 @@ export interface AppConfigResponse { // literal unions, which would defeat the leniency above. `parseWithFallback` // returns the parsed value cast to the caller-supplied `T`, so the strict // type still flows out at the call site; the schema only guards shape. -// --------------------------------------------------------------------------- const ReactionSchema = z.object({ id: z.string(), @@ -1679,7 +1677,6 @@ export const EMPTY_CLOUD_RUNTIME_NODE: CloudRuntimeNode = { updated_at: "", }; -// --------------------------------------------------------------------------- // Workspace dashboard schemas // // The dashboard hits three independent rollup endpoints. Each returns a flat @@ -1688,7 +1685,6 @@ export const EMPTY_CLOUD_RUNTIME_NODE: CloudRuntimeNode = { // String fields default to "" (no enum narrowing) to survive future model / // agent ID drift, and so a single null from tz-aware SQL bucketing fails // only that row instead of dropping the whole array to the `[]` fallback. -// --------------------------------------------------------------------------- // Cost split carried by every usage row. `cost_usd_ticks` is what the provider // itself charged for the rows behind this aggregate (1e-10 USD); the @@ -1788,12 +1784,10 @@ export const DashboardFailureByAgentListSchema = z.array( DashboardFailureByAgentSchema, ); -// --------------------------------------------------------------------------- // Runtime usage schemas — the runtime-detail page's four usage endpoints // (`/api/runtimes/:id/usage*`). Same leniency rules as the dashboard // schemas above: numbers default to 0, strings to "", `.loose()` passes // unknown fields. -// --------------------------------------------------------------------------- const RuntimeUsageSchema = z.object({ runtime_id: z.string().default(""), @@ -1843,10 +1837,8 @@ const RuntimeUsageByHourSchema = z.object({ export const RuntimeUsageByHourListSchema = z.array(RuntimeUsageByHourSchema); -// --------------------------------------------------------------------------- // Agent task responses. The base object stays loose so daemon/runtime fields // can drift while task-list consumers still validate the fields they render. -// --------------------------------------------------------------------------- // Human attribution (MUL-4302 §9): who an agent run is accountable to, and how // that human was resolved. Every field is defensive so a departed member, an @@ -2320,7 +2312,6 @@ export const SquadMemberStatusListResponseSchema = z.object({ export const EMPTY_SQUAD_MEMBER_STATUS_LIST = { members: [] }; -// --------------------------------------------------------------------------- // Structured error body — POST /api/workspaces/:wsId/issues 409 conflict. // // When the server detects an active issue with the same title in the same @@ -2340,7 +2331,6 @@ export const EMPTY_SQUAD_MEMBER_STATUS_LIST = { members: [] }; // so a future server-side rename of `status` must not knock this branch // out. `.loose()` lets the field pass through unchanged for any other // consumer. -// --------------------------------------------------------------------------- export const DuplicateIssueErrorBodySchema = z.object({ code: z.literal("active_duplicate_issue"), @@ -2362,14 +2352,12 @@ export interface DuplicateIssueErrorBody { }; } -// --------------------------------------------------------------------------- // Webhook delivery schemas — backing the Autopilot Deliveries section. Enums // (`status`, `signature_status`, `provider`) are kept as `z.string()` so a // future server-side value (e.g. a Stripe provider, a new dedupe state) // degrades to a generic UI fallback rather than collapsing the list into // the empty array. `.loose()` lets unknown fields pass through, matching // the rule used by every other endpoint here. -// --------------------------------------------------------------------------- const WebhookDeliverySchema = z.object({ id: z.string(), @@ -2416,13 +2404,11 @@ export const EMPTY_LIST_WEBHOOK_DELIVERIES_RESPONSE: ListWebhookDeliveriesRespon total: 0, }; -// --------------------------------------------------------------------------- // Autopilot list schema. Enums (`status`, `execution_mode`, `trigger_kinds`, // `last_run_status`) stay `z.string()` so future server-side values degrade // to a generic UI fallback. The three derived fields (trigger_kinds / // next_run_at / last_run_status) are list-endpoint-only and absent on older // servers — optional by contract, the list renders "—" without them. -// --------------------------------------------------------------------------- const AutopilotListItemSchema = z.object({ id: z.string(), @@ -2551,7 +2537,6 @@ export const EMPTY_WEBHOOK_DELIVERY: WebhookDelivery = { created_at: "", }; -// --------------------------------------------------------------------------- // User (`/api/me` GET + PATCH). The auth store and Settings → Account both // trust this shape — a drift here would knock both surfaces out. Kept // lenient by the same rules as IssueSchema: enums stay `z.string()`, @@ -2560,7 +2545,6 @@ export const EMPTY_WEBHOOK_DELIVERY: WebhookDelivery = { // MUL-2406; the server emits `""` when unset (NOT NULL DEFAULT ''), so // the schema defaults to `""` too — keeps the type tight without // breaking older backends that don't return the column yet. -// --------------------------------------------------------------------------- export const UserSchema = z.object({ id: z.string(), @@ -2592,13 +2576,11 @@ export const EMPTY_USER: User = { updated_at: "", }; -// --------------------------------------------------------------------------- // Cross-workspace unread inbox summary (`/api/inbox/unread-summary` GET). // One entry per workspace the user belongs to that has unread items; the // sidebar derives the workspace-switcher dot from it. Lenient per the usual // rules so a future field addition can't blank the dot — on malformed JSON // parseWithFallback returns the empty list, which simply hides the dot. -// --------------------------------------------------------------------------- export const InboxUnreadSummarySchema = z.array( z @@ -2611,7 +2593,6 @@ export const InboxUnreadSummarySchema = z.array( export const EMPTY_INBOX_UNREAD_SUMMARY: InboxWorkspaceUnread[] = []; -// --------------------------------------------------------------------------- // Inbox items (`/api/inbox` and `/api/inbox/archived` GET). // Lenient per the usual rules: `severity` / `type` / `recipient_type` stay // `z.string()` so a notification kind this client doesn't know yet still @@ -2621,7 +2602,6 @@ export const EMPTY_INBOX_UNREAD_SUMMARY: InboxWorkspaceUnread[] = []; // empty list — the affected view then reads as empty rather than white- // screening the inbox. Both endpoints share this boundary because they return // the same row shape and both feed the status/priority filter UI. -// --------------------------------------------------------------------------- export const InboxItemListSchema = z.array( z @@ -2664,7 +2644,6 @@ export const ArchivedInboxFacetsSchema = z.object({ export const EMPTY_INBOX_ITEMS: InboxItem[] = []; -// --------------------------------------------------------------------------- // Billing schemas (cloud-billing proxy surface) // // All billing JSON we receive comes from multica-cloud verbatim — we proxy @@ -2834,7 +2813,6 @@ export const EMPTY_CREATE_BILLING_PORTAL_SESSION_RESPONSE: CreateBillingPortalSe url: "", }; -// --------------------------------------------------------------------------- // Workspace subscriptions (`/api/cloud-subscriptions/*`) // // These schemas are the compatibility boundary with multica-cloud. Three rules @@ -3130,7 +3108,6 @@ export const CreateWorkspaceSubscriptionPortalResponseSchema = z }), ); -// --------------------------------------------------------------------------- // Runtime model discovery (`POST /api/runtimes/:id/models`, // `GET /api/runtimes/:id/models/:requestId`). Both endpoints return the same // request record, and the UI drives a state machine off `status`, so the two @@ -3146,7 +3123,6 @@ export const CreateWorkspaceSubscriptionPortalResponseSchema = z // // `cached` / `cached_at` are additive markers for a snapshot served from the // server-side catalog cache (MUL-5444); an older backend omits them. -// --------------------------------------------------------------------------- const RuntimeModelThinkingLevelSchema = z.object({ value: z.string(), diff --git a/packages/core/attachments/image-sequence.ts b/packages/core/attachments/image-sequence.ts index c6ccf620c7d..c16c21f016b 100644 --- a/packages/core/attachments/image-sequence.ts +++ b/packages/core/attachments/image-sequence.ts @@ -72,9 +72,7 @@ export function isImageAttachment( return ext !== "" && IMAGE_EXTENSIONS.has(ext); } -// --------------------------------------------------------------------------- // URL → attachment matching -// --------------------------------------------------------------------------- function stripQueryAndFragment(url: string): string { return url.split(/[?#]/, 1)[0] ?? ""; @@ -177,9 +175,7 @@ export function orderStandaloneAttachments a.index - b.index); } -// --------------------------------------------------------------------------- // Sequence -// --------------------------------------------------------------------------- /** One attachment the viewer can page to. */ export interface ImageSequenceItem { diff --git a/packages/core/github/pull-request-status.ts b/packages/core/github/pull-request-status.ts index 68cb2088785..474bb8742f8 100644 --- a/packages/core/github/pull-request-status.ts +++ b/packages/core/github/pull-request-status.ts @@ -21,9 +21,7 @@ import type { // fields; each rule defaults defensively (`?? 0`, `?? []`, explicit `=== "..."` // checks) so an absent field never fabricates a positive verdict. -// --------------------------------------------------------------------------- // CI status -// --------------------------------------------------------------------------- // Discriminated union for the CI element. `none` is a current snapshot with no // checks; `unavailable` means there is no current snapshot region to render. @@ -94,9 +92,7 @@ export function deriveChecksStatus(input: PullRequestChecksInput): PullRequestCh return input.snapshot_available === true ? { kind: "none" } : { kind: "unavailable" }; } -// --------------------------------------------------------------------------- // Mergeability -// --------------------------------------------------------------------------- // Discriminated union for the mergeability element. `none` renders nothing: // when GitHub has not decided (mergeable unknown/null and no decisive @@ -141,9 +137,7 @@ export function deriveMergeStatus(input: PullRequestMergeInput): PullRequestMerg return { kind: "none" }; } -// --------------------------------------------------------------------------- // Diff stats -// --------------------------------------------------------------------------- export interface PullRequestStatsInput { additions?: number; @@ -163,9 +157,7 @@ export function shouldShowPullRequestStats(input: PullRequestStatsInput): boolea return a + d + f > 0; } -// --------------------------------------------------------------------------- // Row verdict -// --------------------------------------------------------------------------- // The one thing the sidebar row says about a PR: what stands between it and a // merge, or how it ended. `failed` carries the failing check names and whether @@ -224,9 +216,7 @@ export function derivePullRequestVerdict(input: PullRequestVerdictInput): PullRe return { kind: "unknown" }; } -// --------------------------------------------------------------------------- // Row copy -// --------------------------------------------------------------------------- function escapeRegExp(value: string): string { return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); diff --git a/packages/core/hooks/use-file-upload.ts b/packages/core/hooks/use-file-upload.ts index 68e92916df8..603503056ae 100644 --- a/packages/core/hooks/use-file-upload.ts +++ b/packages/core/hooks/use-file-upload.ts @@ -6,49 +6,12 @@ import type { Attachment } from "../types"; import { attachmentDownloadPath } from "../types/attachment-url"; import { MAX_FILE_SIZE } from "../constants/upload"; -// Carries the full Attachment so editors that need preview metadata -// (`content_type`, `download_url`) get it directly. Two URL fields are -// surfaced because they have different lifetimes: -// -// `link` — the same value as `att.url`. Short-lived for the -// LocalStorage backend (HMAC-signed `/uploads/`) -// and a long-lived CDN URL on S3 / CloudFront. This -// is what avatar / logo callers persist into -// `avatar_url` style fields, and what URL-only -// consumers (Markdown renderers without a record -// in hand) get to load directly. Keeping it -// semantically equal to `att.url` preserves the -// pre-MUL-3130 contract for non-markdown callers -// so avatar uploads do not get rerouted through -// the workspace-membership-gated download endpoint. -// -// `markdownLink` — the URL the editor writes into markdown bodies. -// Source: `att.markdown_url` from the server, which -// `buildMarkdownURL` picks per deployment policy -// (public CDN durable URL, or -// `/api/attachments//download`). -// The contract is "absolute, no TTL, loads natively -// on every client" — that's what fixes the Desktop / -// mobile-webview regression where a site-relative -// `/api/attachments//download` couldn't resolve -// against `file://` (MUL-3192). -// -// Falls back through two layers when the server -// didn't populate `markdown_url`: -// 1. `attachmentDownloadPath(att.id)` — the legacy -// site-relative shape. Works on web (Next -// rewrite proxies /api/* to the API host) and -// is what older comments persist; render -// surfaces handle the absolutize for non-web -// clients via attachment.tsx's legacy compat. -// 2. `att.url` — the no-workspace avatar branch -// where there's no attachment-row id at all. -// -// MUL-3130 introduced the persisted-image regression by collapsing -// these two semantics into a single `link` field; MUL-3192 followed up -// by moving the durable-URL choice from the client to the server so -// the persisted shape is correct for the deployment without the client -// having to know whether it's running on web / desktop / mobile. +// UploadResult keeps raw storage and durable Markdown URLs separate. +// link preserves att.url for avatars and other storage-reference consumers. +// markdownLink uses the server's absolute, non-expiring markdown_url so native +// clients can load persisted media across origins. Older responses fall back to +// an attachment-ID path, then the raw URL when no attachment row exists. +// The full Attachment carries preview metadata alongside these conveniences. export type UploadResult = Attachment & { link: string; markdownLink: string; @@ -60,20 +23,8 @@ export interface UploadContext { chatSessionId?: string; } -// pickMarkdownLink chooses the URL the editor will write into markdown. -// -// Order: -// 1. `att.markdown_url` — server-provided durable URL. This is the -// modern contract introduced in MUL-3192; the server (`buildMarkdownURL`) -// decides whether to emit a public CDN URL or an absolute API -// endpoint pinned to `MULTICA_PUBLIC_URL` based on the deployment. -// 2. `attachmentDownloadPath(att.id)` — site-relative legacy shape, -// retained for compatibility with backends old enough to predate -// MUL-3192. Web's Next rewrite makes this load; desktop / mobile -// surfaces hit the attachment.tsx legacy-absolutize fallback. -// 3. `att.url` — no attachment-row id (the no-workspace avatar branch -// of UploadFile). Markdown callers fall back to whatever storage -// backend produced for the upload; persistence is on the caller. +// pickMarkdownLink prefers the server's durable URL. Older responses fall back +// to an attachment-ID path, then raw storage for uploads without a row ID. function pickMarkdownLink(att: Attachment): string { if (att.markdown_url) return att.markdown_url; if (att.id) return attachmentDownloadPath(att.id); diff --git a/packages/core/inbox/mutations.ts b/packages/core/inbox/mutations.ts index 0c6b2880e40..b4becc22f76 100644 --- a/packages/core/inbox/mutations.ts +++ b/packages/core/inbox/mutations.ts @@ -7,35 +7,13 @@ import { useWorkspaceId } from "../hooks"; import type { InboxItem } from "../types"; /** - * Re-read every inbox cache a write can change: both workspace lists and the - * cross-workspace unread summary. + * Refresh workspace inbox lists and the account-level unread summary after a + * confirmed write. Use realtime's refresh helpers so in-flight reads are cancelled + * before refetching; a workspace query prefix does not reach the summary. * - * The unread badge reads that summary (`useInboxUnreadCount`), and it lives - * under its own account-level key which `inboxKeys.all(wsId)` does not reach. - * Every mutation here can change the number it holds, so each one refreshes it - * once the server has confirmed, rather than waiting for the WebSocket echo of - * its own action. - * - * Deliberately the same entry points realtime uses, not local copies: both - * refreshes have to cancel any in-flight request first, and a mutation racing - * a first load hits exactly the same hole a WS event does. A plain - * `invalidateQueries` on the list here would let the list fall behind the - * badge — the two are rendered side by side (see `refreshInboxQuery`). - * - * Not awaited by `onSettled`: the mutation is finished once the server has - * answered, and a background refresh should not hold its lifecycle open. - * - * The rows are patched optimistically but the badge is NOT: it follows the - * server's confirmation, which buys a single writer at the cost of the badge - * trailing the row. Deriving it locally instead — recomputing the count from - * the list cache and writing that back — reads as instant but is unsound: a - * list cache proves only that the list was loaded ONCE, never that it is - * complete or concurrent with the summary, and the account-level summary - * request is not cancelled by the workspace-scoped `cancelQueries` below, so a - * response already in flight lands on top of the local value anyway. Under - * pagination it would be wrong by construction — one loaded page cannot - * produce a global count. If instant feedback is wanted later, it has to be a - * per-group delta that handles the race, not a recomputed total. + * Rows may patch optimistically, but the unread badge follows server confirmation. + * A partial or independently fetched list cannot produce a reliable global total. + * Do not hold the completed mutation open while the refresh runs. */ function refreshInboxAfterWrite(qc: QueryClient, wsId: string) { void onInboxInvalidate(qc, wsId); diff --git a/packages/core/issue-statuses/mutations.ts b/packages/core/issue-statuses/mutations.ts index 55ebdcd3cc0..9905806d855 100644 --- a/packages/core/issue-statuses/mutations.ts +++ b/packages/core/issue-statuses/mutations.ts @@ -11,33 +11,11 @@ import type { } from "../types"; /** - * Catalog mutations (MUL-6243). - * - * No mutation invalidates the catalog on success. The refresh is the - * `issue_status:changed` realtime event, which reaches the writing tab as well - * as every other one, so a catalog edit costs each client exactly ONE catalog - * read. Invalidating here too would make the admin who did the writing the - * only client that reads it twice. (MUL-6458) - * - * That refetch is what CONVERGES the cache, and it is ordered correctly by - * construction: the event is published after the write commits, so the refetch - * it triggers can only read post-commit state. - * - * Which is why no mutation writes its RESPONSE BODY into the cache. A response - * is authoritative for the row as of that write, not as of now — and the two - * channels are independent, so a slow response can land after a refetch that - * already picked up someone else's later write and quietly roll the catalog - * back to a state nothing will correct. Every local write below is instead - * something that stays true no matter what else has landed in between: - * - * - an optimistic field patch (rename, recolor, reorder), - * - an insert of a row nobody else can have seen yet (create), - * - a monotonic flag (archive — there is no unarchive endpoint, and the server - * refuses to edit an archived row, so setting it can never be stale). - * - * On FAILURE the invalidate stays, and it is not symmetry for its own sake: a - * 409 usually means this client's catalog is the stale thing — someone else - * took the name, or archived the row that was being dragged. + * Catalog edits converge through the post-commit issue_status:changed event. + * Avoid a duplicate success invalidation or writing a response snapshot that can + * arrive after a newer refetch. Local patches must remain valid across that race: + * optimistic fields, newly created rows, or monotonic archive flags. + * Invalidate failures because a conflict may reveal stale local catalog state. */ function useCatalogCache() { diff --git a/packages/core/issues/cache-coordinator.ts b/packages/core/issues/cache-coordinator.ts index 439fae58669..a4d6d44ccf1 100644 --- a/packages/core/issues/cache-coordinator.ts +++ b/packages/core/issues/cache-coordinator.ts @@ -39,46 +39,15 @@ export type IssueFlatCache = InfiniteData; export type IssueTableRowCache = IssueTableRowsResponse; /** - * IssueCacheCoordinator — the one rules table for how a single issue change - * propagates through the query cache. + * IssueCacheCoordinator applies shared mutation and WebSocket cache rules. + * Patch determinate loaded rows, remove rows that leave a filter, and mark keys + * stale when membership or an unloaded row's server-sorted position is uncertain. + * Update bucket totals for known transitions without inserting speculative rows. * - * Every write path converges here: `useUpdateIssue` (onMutate optimistic + - * onSuccess server reconcile), `useBatchUpdateIssues`, and the WS - * `issue:updated` handler all call {@link applyIssueChange}, so "I changed - * it" and "someone else changed it" follow the same rules by construction. - * - * The rules, per loaded bucketed list (workspace board + every myList - * scope — My Issues, Project, actor panels, workspace members/agents tabs): - * - * card present, filter untouched by the change → surgical patch (rebucket - * on status, position-slot insert; never a refetch — refetching the - * visible list is what made drags flicker) - * card present, no longer matches the list's filter → surgical REMOVE - * (bucket total decremented) — the "issue left this surface" case that a - * filter-blind patch used to leave behind (MUL-3669) - * card present, membership undecidable client-side (involves / my:all) → - * patch + mark the key stale - * card absent, change can't affect this list → skip - * card absent, stayed a member + status changed → move one unit of the - * server total between the two buckets immediately, then mark the key - * stale: the row's slot in the destination bucket's loaded window is - * server knowledge, and under staleTime: Infinity an explicit - * invalidation is the only channel that ever fills it in - * card absent, left the list (reassigned / re-projected) → old status - * bucket total -1 - * card absent, may have ENTERED, or anything undecidable (no base entity, - * unknown membership) → mark the key stale (never hard-insert: the - * right page/slot under the list's sort+filter is server knowledge) - * - * Stale keys are NOT invalidated here — timing is the caller's contract: - * mutations defer them to onSettled (invalidating mid-flight would refetch - * uncommitted state and stomp the optimistic patch), the WS path invalidates - * immediately (the server already committed). - * - * The detail cache and the Inbox issue projections are patched in the - * same pass. Aggregate projections that cannot be recomputed from one entity - * (assignee-grouped boards, Gantt, project metrics) go through - * {@link invalidateIssueDerivatives}. + * This helper returns stale keys without invalidating them: mutations wait until + * onSettled so refetches cannot overwrite optimistic work, while committed WS + * changes invalidate immediately. Detail and inbox projections are patched here; + * aggregate projections use invalidateIssueDerivatives. */ export interface IssueCacheChangeResult { diff --git a/packages/core/issues/mutations.ts b/packages/core/issues/mutations.ts index 5dfc3cc81d2..8f28beb0731 100644 --- a/packages/core/issues/mutations.ts +++ b/packages/core/issues/mutations.ts @@ -56,10 +56,8 @@ import { reconcileIssueFullSnapshotRevision, } from "./ws-updaters"; -// --------------------------------------------------------------------------- // Shared mutation variable types — used by both mutation hooks and // useMutationState consumers to keep the type assertion in sync. -// --------------------------------------------------------------------------- export type ToggleCommentReactionVars = { commentId: string; @@ -81,13 +79,9 @@ export type UpdateIssueMutationInput = { move_intent?: Pick; } & UpdateIssueRequest; -// --------------------------------------------------------------------------- // Per-status pagination -// --------------------------------------------------------------------------- -// --------------------------------------------------------------------------- // Issue CRUD -// --------------------------------------------------------------------------- // Shared by the single and batch update hooks, so the inbox re-read can wait // for the last of them (see `owedInboxRereads`). @@ -816,9 +810,7 @@ export function useBatchDeleteIssues() { }); } -// --------------------------------------------------------------------------- // Comments / Timeline -// --------------------------------------------------------------------------- type TimelineCache = TimelineEntry[]; @@ -1091,9 +1083,7 @@ export function useToggleCommentReaction(issueId: string) { }); } -// --------------------------------------------------------------------------- // Issue-level Reactions -// --------------------------------------------------------------------------- export function useToggleIssueReaction(issueId: string) { const qc = useQueryClient(); @@ -1119,9 +1109,7 @@ export function useToggleIssueReaction(issueId: string) { }); } -// --------------------------------------------------------------------------- // Issue Subscribers -// --------------------------------------------------------------------------- export function useToggleIssueSubscriber(issueId: string) { const qc = useQueryClient(); diff --git a/packages/core/onboarding/step-order.ts b/packages/core/onboarding/step-order.ts index ad4423d5d77..ece7828d9c3 100644 --- a/packages/core/onboarding/step-order.ts +++ b/packages/core/onboarding/step-order.ts @@ -1,32 +1,10 @@ import type { OnboardingStep } from "./types"; /** - * Canonical order of the persisted onboarding steps. - * - * Single source of truth for "what step comes after what" — consumed - * by the UI progress indicator to compute `index of current_step` and - * `total step count`. Inserting, reordering, or removing a step only - * requires changing this array; every call site that reads it updates - * automatically. - * - * Intentionally excludes "welcome": welcome is a first-entry product - * intro, not a persisted step. It doesn't show a progress indicator - * for the same reason — users shouldn't think of reading the intro - * as progress toward completing setup. - * - * Two questions are intentionally NOT steps anymore: - * - * - "source" (How did you hear about Multica?) is pure attribution - * data with zero user-facing payoff, so it no longer taxes the - * critical path. It is collected post-onboarding by the workspace - * source-backfill prompt, and only after agents have completed - * work for the user — see `needs-backfill.ts`. - * - "role" / "use_case" merged into the single "about_you" step so - * related personalization inputs stay together. - * - * Runtime is the final form step. A connected path provisions Mika and opens - * the interactive onboarding chat as part of the runtime step's submit action; - * that chat is the product experience itself, not another progress-screen step. + * Canonical persisted onboarding order for progress and navigation. + * Welcome is an unpersisted introduction; source attribution is collected later. + * About-you combines personalization questions. Runtime is the final form step; + * its connected path provisions Mika and enters chat without another progress step. */ export const ONBOARDING_STEP_ORDER: readonly OnboardingStep[] = [ "about_you", diff --git a/packages/core/onboarding/types.ts b/packages/core/onboarding/types.ts index 46b66b7d911..91b70604a77 100644 --- a/packages/core/onboarding/types.ts +++ b/packages/core/onboarding/types.ts @@ -55,33 +55,13 @@ export type UseCase = | "other"; /** - * Questionnaire shape. `use_case` allows multiple values (users hire - * Multica for several jobs at once); `source` and `role` are single- - * select — for `source` we capture the primary acquisition channel - * for clean self-reported-attribution math (the array shape is - * preserved for back-compat with v2 multi-select rows; the client - * now always commits a one-element array), and `role` stays single - * because downstream personalization wants a primary identity. + * Questionnaire persistence shared by onboarding and source backfill. + * use_case is multi-select; source and role are single-select. Source retains an + * array shape for stored compatibility and is collected after agent work completes. + * Explicit *_skipped flags distinguish declines from unanswered fields. * - * `role` / `use_case` are collected in-flow on the About-you step; - * `source` is no longer asked during onboarding — it is collected - * after the user has seen agents complete work, via the workspace - * source-backfill prompt (see `needs-backfill.ts`). The slots stay in - * this one shape because they share the same JSONB column and PATCH - * endpoint. - * - * `*_skipped: true` distinguishes an explicit Skip / decline from a - * slot the user never reached. Both states are "unknown" for - * personalization purposes; the skip marker exists for analytics and - * so future re-prompts can avoid nagging users who already declined. - * - * Backward compat: prior versions of this app wrote `source` and - * `use_case` as a single string. `mergeQuestionnaire` in - * `onboarding-flow.tsx` upgrades those rows to single-element arrays - * on read; the server's `questionnaireAnswers.UnmarshalJSON` does the - * same. `version` stays at 2 — the JSONB column is schema-less so a - * mechanical bump would only show up in analytics, not in storage, - * and we keep one funnel cohort. + * Readers normalize older single-string source/use_case values to arrays. + * Version 2 remains the shared analytics cohort; the JSONB storage is schema-less. */ export interface QuestionnaireAnswers { source: Source[]; diff --git a/packages/core/types/agent.ts b/packages/core/types/agent.ts index 1653cdf74c5..8d1759d8016 100644 --- a/packages/core/types/agent.ts +++ b/packages/core/types/agent.ts @@ -7,7 +7,6 @@ export type AgentRuntimeMode = "local" | "cloud"; export type AgentVisibility = "workspace" | "private"; -// --------------------------------------------------------------------------- // Agent invocation permissions (MUL-3963) // // `permission_mode` + `invocation_targets` are the AUTHORITATIVE gate for who @@ -24,7 +23,6 @@ export type AgentVisibility = "workspace" | "private"; // - permission_mode "public_to" + workspace target: any workspace member // - permission_mode "public_to" + member target: only the matching user // - team target: reserved, INERT in v1 (never grants) -// --------------------------------------------------------------------------- export type AgentPermissionMode = "private" | "public_to"; @@ -93,7 +91,6 @@ export interface RuntimeDevice { export type AgentRuntime = RuntimeDevice; -// --------------------------------------------------------------------------- // Custom runtime profiles (MUL-3284) // // A RuntimeProfile is a workspace-level *definition* of a custom runtime @@ -101,7 +98,6 @@ export type AgentRuntime = RuntimeDevice; // *instance*. An admin authors a profile (display name + base protocol // family + the CLI command to launch), and daemons can then register // runtimes against it; those instances carry `profile_id` pointing back here. -// --------------------------------------------------------------------------- // The fixed allow-list of base protocol families a custom runtime can wrap. // These are the only backends the create flow may select; the server rejects diff --git a/packages/core/types/attachment-url.ts b/packages/core/types/attachment-url.ts index 9f73af88546..9b9a980594d 100644 --- a/packages/core/types/attachment-url.ts +++ b/packages/core/types/attachment-url.ts @@ -1,24 +1,7 @@ /** - * Stable URL helpers for attachment references that get persisted into - * markdown bodies (issue descriptions, comment bodies, chat messages). - * - * Background — MUL-3130: - * - * The original upload flow returned `att.url` as the link to embed in - * markdown. After PR #3903 (MUL-3132), `att.url` for the LocalStorage - * backend became a 30-minute HMAC-signed `/uploads/?exp&sig` URL. - * That URL is short-lived by design — it's the in-memory token that - * keeps native browser /