Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions biome.json
Original file line number Diff line number Diff line change
Expand Up @@ -771,7 +771,7 @@
"noExcessiveLinesPerFile": {
"level": "error",
"options": {
"maxLines": 1872
"maxLines": 1880
}
}
}
Expand Down Expand Up @@ -816,7 +816,7 @@
"noExcessiveLinesPerFile": {
"level": "error",
"options": {
"maxLines": 2947
"maxLines": 2980
}
}
}
Expand All @@ -831,7 +831,7 @@
"noExcessiveLinesPerFile": {
"level": "error",
"options": {
"maxLines": 486
"maxLines": 493
}
}
}
Expand Down
8 changes: 8 additions & 0 deletions changes/feat-tracker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
### Added

- **Issue trackers get their own category: `@wirecat/cli-core/tracker`.** A tracker CLI writes one `TrackerSource`
and gets the copy in the shared store (store version 70: `tracker_items` and the word index `tracker_item_words`),
`syncTracker` (changes by cursor, deletions by a daily id pass, a run budget that resumes where it stopped),
`addTrackerCommands` (`sync`, `issues list|show|search|task`, and `projects`, `teams`, `users`, `states`, `labels`
lists), and `/tracker/testing` with `trackerSourceContract` and `fakeTrackerSource`. `issues task` makes an issue
the owner's task by a link, copying only its key. See [Trackers](../docs/dev/TRACKER.md).
1 change: 1 addition & 0 deletions docs/dev/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,7 @@ alias, keyword and person lanes and returns cited evidence with a resume checkpo
| `./constants` | `src/constants/` | the packaged constants (`constants`, `constantsSchema`), the app a sign-in uses (`googleApp`, `microsoftApp`, `oauthApp`) and `doctor`'s report of it (`oauthClientReport`) — see [Constants](CONSTANTS.md) |
| `./mail`, `/mail/testing` | `src/mail/` | the mail source port, the sync engine over `store.mail` (resume points, rules, deletions, attachment text), sender scoring and the contract kit — see [Mail](MAIL.md) |
| `./crm`, `/crm/cli`, `/crm/testing` | `src/crm/` | the CRM port, the sync engine, the guarded writes, the live `people`, `organizations` and `users` commands, the `files` commands, the adapter helpers and the contract kit — see [CRM](CRM.md) |
| `./tracker`, `/tracker/testing` | `src/tracker/` | issue trackers: the `TrackerSource` port, the sync engine over `store.tracker` (changes by cursor, deletions by an id pass), the issue → owner task link, the shared commands and the contract kit — see [Trackers](TRACKER.md) |
| `./google` | `src/google/` | what every Google API client shares: `googleError` turns a refusal into a `CliError` by Google's reasons, and `googlePages` reads every `pageToken` page, refusing a repeated token |
| `./sftp` | `src/sftp/` | a read-only SFTP client and an SSH tunnel that only forwards: `session.ts` holds the host-key pin, deadlines, retries and run events; `ssh2.ts` is the only file that loads `ssh2` (an optional peer, imported on first use) — see [SFTP](../sftp.md) |
| `./sql` | `src/sql/` | a read-only SQL reader: `reader.ts` opens mysql2 or pg (optional peers, imported on first use) read-only, pages a host's table and maps driver errors — see [SQL](../sql.md) |
Expand Down
74 changes: 74 additions & 0 deletions docs/dev/TRACKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Trackers

`src/tracker/`, exported as `@wirecat/cli-core/tracker` and `/tracker/testing`. A tracker CLI (Linear first) writes its
sign-in and one adapter, a `TrackerSource`; the copy in the shared store, the sync, the commands and the contract kit
are shared, as for [CRMs](CRM.md) and [mail](MAIL.md). The tables are in the [schema](../storage/schema.md); this page
covers what runs over them.

## The domain

Seven kinds, in the order a sync reads them: `team`, `user`, `state`, `label`, `project`, `issue`, `comment`
(`TRACKER_KINDS`, `src/domain/tracker.ts`). Each record carries the tracker's `externalId`, its `createdAt`,
`updatedAt` and `archivedAt` in epoch milliseconds, and refers to other rows by their tracker ids: an issue's `state`,
`team`, `project`, `assignee`, `labels` and `parent`, a comment's `issue`. A state has one of six types
(`STATE_TYPES`): `triage`, `backlog`, `unstarted`, `started`, `completed`, `canceled`, so "open work" reads the same
whatever each team named its states.

## The port

| Member | What the adapter does |
|---|---|
| `traits.kinds` | the kinds this tracker has, in `TRACKER_KINDS` order |
| `changes(kind, cursor, page?, context?)` | what changed since `cursor`, a page at a time; `cursor: null` lists everything. A tracker without a change feed reads by its modified time less an overlap and gives that time back as the last page's `cursor`. A dead cursor answers `{ reset: true }` |
| `ids(kind, page?, context?)` | every id the tracker holds of the kind, archived ones too, a page at a time |
| `self?(context?)` | the sign-in's own user id |

Archived rows are listed, with `archivedAt`, and the copy keeps them marked. A row deleted outright is found only by
`ids()`. A source that pages inside one call asks `context.spend()` before each further request, and names what the
sign-in could not read with `context.gap()`.

## `syncTracker(source, store, account, options)`

For each kind: the changes since the kind's cursor, page by page, each page saved with the cursor after it
(`tracker:<kind>` in the account's sync state). Then, once `passEvery` (a day) has passed since the last complete one,
an id pass that marks deleted each stored row the list lacks and that was not saved since the pass began. A pass that
stops keeps its ids and the page it reached.

- **Budget.** The engine takes one request from `options.budget` before each call. A run that stops answers
`state: "partial"` and `stopped`; the kinds it finished are skipped by the next run until every kind has had its
round, so a budget smaller than the number of kinds still moves on.
- **Held passes.** A pass is held (`pass: "held"`, with the reason in `held`) and marks nothing when the list is empty
while rows are stored, when it would mark more than half of ten or more stored rows deleted, or when the sign-in
reported a gap for the kind: a tracker that lost rights lists none of what it can no longer see.
- **Failures.** A kind that fails is named in its result's `error` and the others still sync. A page that says there is
more but gives no next page fails the kind.
- **`rescan`.** Lists everything again and every id, dropping a round in progress, and lets a held pass through. A
rescan that stops goes on as one in the next run.

A save merges: a field the read left out keeps its stored value, so a partial selection never erases.

## `store.tracker`

`save`, `removeUnlisted`, `get`, `issueByKey` (`ENG-12`, any case), `list` with issue filters (team, project,
assignee or none, state, state types through the state rows, label, archived), `find` over the word index
`tracker_item_words` (titles, descriptions, comment bodies), `count`, and `people`: the tracker users whose e-mail an
identity linked to a person has, joined when read. Nothing is written to the people graph.

## An issue as the owner's task

`trackerIssueTask(store, account, issue)` and `issues task <issue>` make an issue the owner's task: `source` is
`tracker:issue/<externalId>`, `sourceKind` is `tracker_issue`, and the title is the issue's key only. The task links
to the issue and copies none of its text. The same issue again answers the task it already has.

## Commands

`addTrackerCommands(program, { app, open })` mounts `sync`, `issues list|show|search|task`, and `list` for
`projects`, `teams`, `users` (each with its person), `states` and `labels`. Every text field from the tracker is named
in the row's `untrusted`.

## The contract kit

`trackerSourceContract(make)` checks an adapter: full lists in the domain's shape, every held row listed, a cursor
read again, `ids()` covering the full list, a dead cursor answering reset, a clock off by ten minutes either way, an
archived row still listed, a deleted row leaving `ids()`, a list cut after its first page, and a spent budget ending a
call that pages inside. `fakeTrackerSource` and `exampleTrackerRows` are an in-memory tracker for tests.
39 changes: 38 additions & 1 deletion docs/storage/schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Every table of the shared store (tg, max, memo, zm) as its migrations create it.
`drizzle/` must agree with it. `pnpm schema:check` fails when they do not, and `src/store/sqlite/schema.test.ts`
checks every table and column of a new store against this page.

137 tables, 26 full-text indexes.
138 tables, 27 full-text indexes.

## Conventions

Expand Down Expand Up @@ -3006,6 +3006,43 @@ How up to date one built chat's conversations and one model's vectors are, as `s

*Keys and indexes:* `PRIMARY KEY (chat_id, model)`, `INDEX (model)`

## Issue trackers

What an issue tracker account holds, as last synced: teams, users, workflow states, labels, projects, issues and comments. A user joins a person only by exact e-mail, when read; nothing here writes to the people graph.

### `tracker_items`

One team, user, workflow state, label, project, issue or comment of an issue tracker account (Linear), as last synced. The columns are what lists, filters and search read; the rest of the record is `data`.

| Column | Type | Constraints | References | Meaning |
|---|---|---|---|---|
| `id` | integer | PK | | |
| `account_id` | integer | not null | → `accounts.id` | the tracker account it was copied from |
| `kind` | text | not null | | team, user, state, label, project, issue or comment |
| `external_id` | text | not null | | its id in the tracker |
| `parent_id` | text | | | the tracker's id of what it hangs under: a comment's issue, an issue's, a state's or a label's team |
| `title` | text | not null | | an issue's title, a comment's first line, anything else's name |
| `title_folded` | text | not null | | the title in one letter case |
| `body` | text | | | an issue's or a project's description, a comment's text |
| `normalized_text` | text | not null | | title and body normalized for the word index |
| `data` | text | not null | | JSON: the record in the tracker domain's field names; a field the tracker did not send keeps its stored value |
| `source_created_at` | integer | | | when the tracker says it was made |
| `source_updated_at` | integer | | | when the tracker says it last changed |
| `archived_at` | integer | | | the tracker archived it; it still exists there |
| `deleted_at` | integer | | | the tracker no longer lists it; sync never hard-deletes |
| `synced_at` | integer | not null | | |

*Keys and indexes:* `UNIQUE (account_id, kind, external_id)`, `INDEX (account_id, kind, parent_id)`

### `tracker_item_words`

Word index over a tracker item's title and text, rowid = `tracker_items.id`.

```sql
CREATE VIRTUAL TABLE tracker_item_words USING fts5(normalized_text, content = '', contentless_delete = 1, tokenize = 'unicode61 remove_diacritics 2', prefix = '3');
CREATE VIRTUAL TABLE tracker_item_words_vocab USING fts5vocab(tracker_item_words, 'col')
```

## Store

Bookkeeping for the file itself.
Expand Down
21 changes: 21 additions & 0 deletions drizzle/20261117211410_version-70-tracker-items/migration.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
CREATE TABLE `tracker_items` (
`id` integer PRIMARY KEY,
`account_id` integer NOT NULL,
`kind` text NOT NULL,
`external_id` text NOT NULL,
`parent_id` text,
`title` text NOT NULL,
`title_folded` text NOT NULL,
`body` text,
`normalized_text` text NOT NULL,
`data` text NOT NULL,
`source_created_at` integer,
`source_updated_at` integer,
`archived_at` integer,
`deleted_at` integer,
`synced_at` integer NOT NULL,
CONSTRAINT `fk_tracker_items_account_id_accounts_id_fk` FOREIGN KEY (`account_id`) REFERENCES `accounts`(`id`),
CONSTRAINT `tracker_items_account_id_kind_external_id_unique` UNIQUE(`account_id`,`kind`,`external_id`)
);
--> statement-breakpoint
CREATE INDEX `tracker_items_by_account_id_kind_parent_id` ON `tracker_items` (`account_id`,`kind`,`parent_id`);
Loading
Loading