Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions changes/2026-10-23-issue-21-workflow-and-artifact-tokens/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Design

> **Issue 21 workflow and artifact tokens** · `2026-10-23-issue-21-workflow-and-artifact-tokens`

## Approach

Count files that are already on disk. `countTokens` in `packages/core/src/search/tokenize.ts` is the cl100k counter. `sdd workflows` keeps its one-line list, then prints the token inventory under it.

## Seams

| Piece | Change |
|---|---|
| `formatTokenInventory` | Plain text for every workflow, then the active change when one exists |
| Workflow file | Token count of `.sdd/workflows/<name>.yaml` |
| Artifact template | The same template lookup `sdd next` uses, counted before `{{title}}` is filled in |
| Active change | Token count of each artifact path under the change directory |
| CLI | `sdd workflows` prints that text. No new command. |

A directory artifact is labeled `directory` and is not added to the total. A missing template is `missing`. An unwritten change file is `not written`. The artifact total adds only counted files.

## Risks

The same template used by two stages is counted twice, once per artifact file that would be written. The active-change section counts the filled files, so those numbers differ from the template numbers.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Feature

> **Issue 21 workflow and artifact tokens** · `2026-10-23-issue-21-workflow-and-artifact-tokens` · 2026-10-23

## Reading

`sdd workflows` shows each workflow and the token count of every artifact, using the same cl100k count the search pack already uses. A workflow line counts the workflow file. Each artifact line counts the template that stage writes. When a change is active, a second section counts the artifact files on disk. Dollar cost stays out of this view.

## Source

Issue #21, and the request to see those counts without a session import or a price file. This is the product manager's reading. The person can correct it.

## Product owner

The request is one view: tokens for each workflow and its artifacts. Dollar amounts stay off this command. Proposed checks:

- One `sdd workflows` run prints a cl100k count for every copied workflow and for each of its artifact templates.
- A missing template prints `missing` and is left out of the artifact total.
- The active change, when one exists, lists each artifact file and its token count. A file that is not written yet prints `not written`.
- The view prints no dollar amount.

## Non-goals

- No change to `sdd report` or `sdd cost`.
- No price file and no session-log import.
- External search providers stay in issue #25. This slice does not close epic #21.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Local test plan

> **Issue 21 workflow and artifact tokens** · `2026-10-23-issue-21-workflow-and-artifact-tokens`

Node on this machine is v20.19.3. The `EBADENGINE` warning for `engines.node >= 24` is expected.

| # | Check | Expected |
|---|--------|----------|
| 1 | `vitest run src/__tests__/token-inventory.test.ts` | Pass. `hello world` is 2 tokens. A missing template prints `missing`. No dollar amount. |
| 2 | `pnpm --filter @structured-vibe-coding/core test` | Pass. |
| 3 | `tsc --noEmit` for core and cli, and biome check | Pass. |
| 4 | `sdd workflows` | Prints `Tokens (cl100k):` and a line for the active change. |

These checks are proposed evidence. They are not the person's recorded QA result.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Local test results

> **Issue 21 workflow and artifact tokens** · `2026-10-23-issue-21-workflow-and-artifact-tokens`

Node v20.19.3. `EBADENGINE` warnings for `engines.node >= 24` are expected.

| # | Result |
|---|--------|
| 1 | PASS. `token-inventory.test.ts` — 1 test, exit 0. `hello world` counted as 2. |
| 2 | PASS. Full core vitest, 282 tests, 37 files, exit 0. |
| 3 | PASS. `tsc --noEmit` for core and cli. `biome check` on `packages/core/src` and `packages/cli/src`, 141 files, exit 0. |
| 4 | PASS. `sdd workflows` printed `Tokens (cl100k):` for enterprise-feature, feature, greenfield, hotfix, patch, and spike, then the active change's artifact files. No dollar amount. |

QA proposes this result. The person has not recorded it. `sdd verify` ran `pnpm test` and recorded exit 0. The stamp is HEAD `1c62be3`, the commit before this change was committed. The command ran on the working tree that contains this change.

Dollar cost is left on `sdd report` and `sdd cost`. This slice does not close epic #21. Issue #25 remains open.

---

## Latest run (2026-10-23T10:35:52.601Z)

> Overall: PASS
> Stage: local_verify
<!-- sdd:verify stage="local_verify" head_sha="1c62be3e941f260d197a41a614c92b38cb4cdda8" result_count="1" -->

- **Workspace tests**: exit 0 (`pnpm test`)
41 changes: 41 additions & 0 deletions changes/2026-10-23-issue-21-workflow-and-artifact-tokens/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
id: 2026-10-23-issue-21-workflow-and-artifact-tokens
title: Issue 21 workflow and artifact tokens
workflow: feature
created: 2026-10-23T10:32:16.610Z
updated: 2026-10-23T10:35:52.601Z
status: in_progress
stage: local_verify
epic: issue-21
depends: []
flags: {}
overrides:
skip_stages:
- clarify_intent
- brainstorm
- clarify_design
- clarify_tasks
gates: {}
extra_stages: []
skipped:
- stage: clarify_intent
reason: the request names tokens and leaves dollar cost out
at: 2026-10-23T10:32:32.972Z
- stage: brainstorm
reason: count the files with the existing tokenizer
at: 2026-10-23T10:32:33.756Z
- stage: clarify_design
reason: the seams are in the design
at: 2026-10-23T10:32:34.472Z
- stage: clarify_tasks
reason: the task list is the work
at: 2026-10-23T10:32:35.201Z
gates: {}
verify_results:
local_verify:
ok: true
status: pass
head_sha: 1c62be3e941f260d197a41a614c92b38cb4cdda8
at: 2026-10-23T10:35:52.601Z
results:
- name: Workspace tests
exitCode: 0
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
{
"version": 1,
"phases": [
{
"phase": "intent",
"startedAt": "2026-10-23T10:32:16.610Z",
"endedAt": "2026-10-23T10:32:35.927Z"
},
{
"phase": "design",
"startedAt": "2026-10-23T10:32:35.927Z",
"endedAt": "2026-10-23T10:32:36.741Z"
},
{
"phase": "tasks",
"startedAt": "2026-10-23T10:32:36.741Z",
"endedAt": "2026-10-23T10:32:37.547Z"
},
{
"phase": "implement",
"startedAt": "2026-10-23T10:32:37.547Z",
"endedAt": "2026-10-23T10:34:34.030Z"
},
{
"phase": "local_verify",
"startedAt": "2026-10-23T10:34:34.030Z"
}
],
"requests": []
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Tasks

> **Issue 21 workflow and artifact tokens** · `2026-10-23-issue-21-workflow-and-artifact-tokens`

1. Add `formatTokenInventory` and cover a known `hello world` file, a missing template, and the absence of a dollar amount.
2. Print that inventory from `sdd workflows`.
3. Document the view next to the `sdd workflows` command.
4 changes: 3 additions & 1 deletion docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Commit `.sdd/config.yaml`, workflows, templates, `protocol.md`, `memory/`, and c
| `sdd branch <name>` | Create that story branch from `main` | Start one story |
| `sdd pr` | Push the change branch and open a pull request | Story is ready |
| `sdd checkout <id>` | Switch active change | Multi-PR |
| `sdd workflows` | List project workflows, and name bundled workflows that are not copied yet | Discover |
| `sdd workflows` | List project workflows, name bundled workflows that are not copied yet, and print cl100k token counts for each workflow file and artifact | Discover |
| `sdd help` | Short overview | First run |
| `sdd doctor` | Health check | Stuck / setup |

Expand Down Expand Up @@ -141,6 +141,8 @@ If a change has no measured requests, the report adds one `estimated (handoff co

Search, context packing, and the handoff retrieved section count tokens with `cl100k_base`. That count is separate from the report estimate.

`sdd workflows` prints those same cl100k counts for every copied workflow. The workflow line counts the workflow file. Each artifact line counts the template that stage copies. A missing template prints `missing`. When a change is active, a `Change` section counts the artifact files in that change. A file that has not been written prints `not written`. This view has no dollar amount.

Embeddings are off by default. Set `embeddings: true` in `.sdd/search.yaml` to download `BAAI/bge-small-en-v1.5` once into `.sdd/models/` and rank with hybrid fusion. A missing model falls back to keyword search.

Related change specs and memory pages can be packed as a short extractive summary. The active change and code stay full. The keyword index is unchanged. Repeated searches are cached in `.sdd/index.db`, and search text does not gain a cache banner.
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ Shipped under `.sdd/workflows/` after `sdd init` (from package defaults). List a
sdd workflows
```

The same command prints a cl100k token count for each workflow file and for each artifact template. When a change is active, it also counts that change's artifact files. It does not print a dollar cost.

A project initialized by an older `sdd` may be missing packs this CLI ships. `sdd workflows` names those under `Not copied yet`. Copy them with `sdd init --here --force`. That command leaves a file alone when its bytes differ from the bundled file. A file that still matches is recorded, so a later upgrade can refresh it.

Pick a pack at change start:
Expand Down
5 changes: 4 additions & 1 deletion packages/cli/src/commands.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
describeKept,
formatJsonSummary,
formatStatus,
formatTokenInventory,
generateCodeContext,
getActiveChangeId,
getStage,
Expand Down Expand Up @@ -801,7 +802,7 @@ const feature = defineCommand({
const workflows = defineCommand({
meta: { name: "workflows", description: "List available workflow packs (no agent launch)" },
async run() {
await withProject(async ({ root }) => {
await withProject(async ({ root, config }) => {
const names = await listWorkflowNames(root);
for (const name of names) {
const wf = await loadWorkflow(root, name);
Expand All @@ -814,6 +815,8 @@ const workflows = defineCommand({
const copy = "Run `sdd init --here --force` to copy new defaults.";
consola.log(`${pc.yellow("Not copied yet:")} ${missing.join(", ")}. ${copy}`);
}
console.log("");
console.log(await formatTokenInventory(root, config));
});
},
});
Expand Down
64 changes: 64 additions & 0 deletions packages/core/src/__tests__/token-inventory.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "pathe";
import { afterEach, describe, expect, it } from "vitest";
import {
createChange,
formatTokenInventory,
initProject,
loadConfig,
setActiveChange,
} from "../index.js";

const temps: string[] = [];

afterEach(async () => {
while (temps.length) {
const dir = temps.pop();
if (dir) await rm(dir, { recursive: true, force: true });
}
});

describe("workflow and artifact tokens", () => {
it("counts each workflow file and artifact, and the active change", async () => {
const root = await mkdtemp(join(tmpdir(), "sdd-tokens-"));
temps.push(root);
await initProject({ projectRoot: root });
await writeFile(
join(root, ".sdd/workflows/token-gap.yaml"),
[
"name: token-gap",
"stages:",
" - id: gap",
" artifacts:",
" - id: gap",
" path: gap.md",
" template: does-not-exist.md",
"",
].join("\n"),
"utf8",
);
const config = await loadConfig(root);
const ctx = await createChange({
projectRoot: root,
config,
title: "Token sample",
workflowName: "hotfix",
});
await writeFile(join(ctx.path, "intent.md"), "hello world", "utf8");
await setActiveChange(root, config, ctx.id);

const text = await formatTokenInventory(root, config);
expect(text.startsWith("Tokens (cl100k):")).toBe(true);
expect(text).toContain("hotfix workflow ");
expect(text).toContain("gap gap.md missing");
expect(text).toContain(" artifacts 0");
expect(text).not.toContain("$");
expect(text.toLowerCase()).not.toContain("usd");

const change = text.slice(text.indexOf("\nChange "));
expect(change).toContain(`Change ${ctx.id} workflow hotfix`);
expect(change).toContain("intent intent.md 2");
expect(change).toContain("local-test-results.md not written");
});
});
1 change: 1 addition & 0 deletions packages/core/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ export * from "./report.js";
export * from "./search/index.js";
export * from "./sync.js";
export * from "./workflow.js";
export * from "./token-inventory.js";
export * from "./change.js";
export * from "./init.js";
export * from "./verify.js";
Expand Down
Loading
Loading