From 5c2509bb38cb24f63a59ff85945ea2784676bf37 Mon Sep 17 00:00:00 2001 From: Dibbayajyoti Date: Tue, 27 Oct 2026 12:42:18 +0530 Subject: [PATCH 1/2] feat: live views, logs --watch and batch watch logs --watch is a live, filterable request log; batch watch is a dashboard for a batch job. Both are full-screen (Bubble Tea) only when stdin and stdout are terminals and none of --json, --quiet, CI or TERM=dumb is set. Otherwise logs --watch streams new rows as plain lines (JSONL with --json) and batch watch is batch wait, with the same exit codes (0, 4, 11). The terminal is restored on q, Ctrl-C (exit 130), errors and panics. Failed polls are retried with backoff and shown on the status bar. NO_COLOR is honoured. Rows show tiers, never engine names. New dependencies, all Charm: bubbletea, lipgloss and bubbles v2. --- CHANGELOG.md | 1 + DESIGN.md | 3 + README.md | 15 +- go.mod | 23 +- go.sum | 47 +- internal/cmd/batch_watch.go | 378 ++++++++ internal/cmd/batch_watch_test.go | 432 +++++++++ internal/cmd/logs_watch.go | 336 +++++++ internal/cmd/logs_watch_test.go | 429 +++++++++ internal/cmd/watch.go | 71 ++ internal/cmd/watch_test.go | 70 ++ internal/tui/header.go | 84 ++ internal/tui/header_test.go | 130 +++ internal/tui/run.go | 70 ++ internal/tui/run_test.go | 253 +++++ .../testdata/TestSnapshotBatch80x24.golden | 24 + .../TestSnapshotBatchFailedOnly.golden | 24 + .../tui/testdata/TestSnapshotLogs80x24.golden | 24 + .../testdata/TestSnapshotLogsDetail.golden | 24 + .../tui/testdata/TestSnapshotLogsError.golden | 24 + .../TestSnapshotLogsFilteredSearch.golden | 24 + .../tui/testdata/TestSnapshotLogsWide.golden | 14 + .../tui/testdata/TestSnapshotNoRows.golden | 24 + internal/tui/watch.go | 871 ++++++++++++++++++ internal/tui/watch_test.go | 631 +++++++++++++ 25 files changed, 4022 insertions(+), 4 deletions(-) create mode 100644 internal/cmd/batch_watch.go create mode 100644 internal/cmd/batch_watch_test.go create mode 100644 internal/cmd/logs_watch.go create mode 100644 internal/cmd/logs_watch_test.go create mode 100644 internal/cmd/watch.go create mode 100644 internal/cmd/watch_test.go create mode 100644 internal/tui/header.go create mode 100644 internal/tui/header_test.go create mode 100644 internal/tui/run.go create mode 100644 internal/tui/run_test.go create mode 100644 internal/tui/testdata/TestSnapshotBatch80x24.golden create mode 100644 internal/tui/testdata/TestSnapshotBatchFailedOnly.golden create mode 100644 internal/tui/testdata/TestSnapshotLogs80x24.golden create mode 100644 internal/tui/testdata/TestSnapshotLogsDetail.golden create mode 100644 internal/tui/testdata/TestSnapshotLogsError.golden create mode 100644 internal/tui/testdata/TestSnapshotLogsFilteredSearch.golden create mode 100644 internal/tui/testdata/TestSnapshotLogsWide.golden create mode 100644 internal/tui/testdata/TestSnapshotNoRows.golden create mode 100644 internal/tui/watch.go create mode 100644 internal/tui/watch_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 6039d16..e741079 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman - **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API. - **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`. - **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`). +- **Live views.** `spicrawl logs --watch` is a live, scrollable request log (new rows on top, filter by status, tier and errors, search, Enter for the `logs get` page) and `spicrawl batch watch ` is a dashboard for a batch job (progress bar, done/failed/total, ETA, credits, a per-item table, failed-only filter, item detail). Both are full-screen only when stdin and stdout are terminals and none of `--json`, `-q`, `CI` or `TERM=dumb` is set; otherwise `logs --watch` streams new rows as plain lines (JSONL with `--json`) and `batch watch` is `batch wait`. `batch watch` ends with the final job and `batch wait`'s exit codes (0, 4, 11); `NO_COLOR` is honoured; the terminal is restored on q, Ctrl-C (exit 130), errors and panics. New dependencies, all Charm: `charm.land/bubbletea/v2`, `charm.land/lipgloss/v2`, `charm.land/bubbles/v2`. The release binary grows by about 2.1 MB (10.0 to 12.2 MB, stripped). - **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names. - **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary. diff --git a/DESIGN.md b/DESIGN.md index 5a1d77d..3c2392d 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -170,6 +170,7 @@ spicrawl batch get spicrawl batch results [--status S] [--all] [-o FILE] JSONL spicrawl batch content [-o FILE] one item's document spicrawl batch wait [--max-wait 30m] exit 11 when it gives up, 4 if failed/cancelled +spicrawl batch watch [--poll 2s] [--max-wait 30m] live dashboard on a TTY; else exactly `batch wait` spicrawl batch cancel [--yes] --yes required without a TTY spicrawl batch retry|close spicrawl batch append @@ -182,6 +183,8 @@ spicrawl browser url [--engine browser|js] [--country CC] [--ttl S] spicrawl logs [--errors] [--status S] [--limit N] [--all] [--all-projects] GET /v1/requests (--all-projects: all_projects=true, read scope) +spicrawl logs --watch [--poll 2s] [--errors] [--status S] [--limit N] [--all-projects] + live view: full-screen on a TTY, else plain lines (JSONL with --json) spicrawl logs get spicrawl usage [--from D] [--to D] [--group-by day] GET /v1/usage spicrawl usage summary | reconciliation diff --git a/README.md b/README.md index 023d378..b93ddb4 100644 --- a/README.md +++ b/README.md @@ -180,7 +180,7 @@ The key is passed as an argument, so other users on the machine can see it in th | Command | What it does | |---|---| | `spicrawl scrape ` | Fetch one page, or many from stdin, as Markdown, HTML, text, JSON, PDF or extracted data | -| `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` | +| `spicrawl batch` | Run up to 10,000 URLs as one server-side job: `submit`, `wait`, `watch`, `get`, `list`, `results`, `content`, `retry`, `cancel`, `append`, `close` | | `spicrawl sessions` | Keep one rendering tier and cookie jar across scrapes: `create`, `list`, `get`, `context`, `release`, `delete` | | `spicrawl browser` | Cloud browser over a DevTools Protocol (CDP) WebSocket (`browser url`). Coming soon | | `spicrawl init` | Set up AI agent clients: MCP server config and the agent skill | @@ -190,7 +190,7 @@ The key is passed as an argument, so other users on the machine can see it in th | `spicrawl logout` | Remove the saved API key | | `spicrawl auth` | `auth status`: which key and base URL are in use, where they come from, and whether the key works | | `spicrawl config` | Read and write the config file: `get`, `set`, `unset`, `path` | -| `spicrawl logs` | Recent API requests, newest first; `logs get ` for one | +| `spicrawl logs` | Recent API requests, newest first; `logs get ` for one; `logs --watch` for a live view | | `spicrawl usage` | Billed usage over a time window; `usage summary`, `usage reconciliation` | | `spicrawl status` | Check the API is reachable, ready and accepts your key, and that detected agent clients have the MCP server; prints a repair command for each failed check | | `spicrawl docs` | Print a Spicrawl docs page as Markdown (`--list` for the index); no API key needed | @@ -209,6 +209,17 @@ On a terminal `spicrawl --help` lists the commands grouped by purpose, with exam All of it is on stderr and stdout is byte-for-byte unchanged. It switches itself off, and the CLI prints exactly what it printed before, when stderr is not a terminal (a pipe, a file, an agent or CI log), with `--json` or `-q/--quiet`, with `NO_COLOR` set (or `CLICOLOR=0`), with `TERM=dumb`, or when `CI` is set. +### Live views + +`spicrawl logs --watch` and `spicrawl batch watch ` open a full-screen view that updates as the API answers. They need a terminal for stdin and stdout; the colours follow your terminal's theme and `NO_COLOR` turns them off (the view stays usable: the selected row is marked with `>`). + +- `logs --watch` polls the request log (`--poll`, default `2s`). New requests appear at the top and the selection stays on its row. Arrows, `PgUp`/`PgDn`, `g`/`G` move; `Enter` opens a request (what `logs get` prints); `/` searches; `s`, `t` and `e` filter by status, tier and errors only; `c` clears the filters; `p` pauses; `r` refreshes; `q` or `Esc` quits. `--errors`, `--status`, `--limit` and `--all-projects` work as in `logs`. The filters apply to the requests loaded so far (the newest `--limit`, 200 by default, and everything since), so start with `--errors` or `--status` to narrow the query itself. +- `batch watch ` shows a progress bar with done, failed and total, the ETA and credits, and a table of the finished items (status, HTTP status, tier, duration, URL), newest first. `Enter` opens an item, `f` shows only failed ones, and `s`, `t`, `/` filter as above. When the job finishes the view closes, the final job is printed and the exit code is `batch wait`'s: `0` completed, `4` failed or cancelled, `11` if `--max-wait` elapsed first. `q` leaves the job running (exit `0`); `Ctrl-C` exits `130`. + +A failed poll never closes a view: the error shows on the status bar and the poll is retried with backoff (a `429` waits for its `Retry-After`). A wrong key or an unknown job id fails at once with its usual exit code, before the screen opens. Resizing is followed, and the views fit 80x24; below 40x10 they ask for a bigger terminal. + +When stdin or stdout is not a terminal, or with `--json`, `-q/--quiet`, `CI` or `TERM=dumb`, there is no full-screen view: `logs --watch` streams new requests as plain lines, oldest first (JSONL with `--json`, which is the default when stdout is piped), and `batch watch` is exactly `batch wait`. + ### How to convert a web page to Markdown from the command line ```sh diff --git a/go.mod b/go.mod index b0ec7a9..df7e1d0 100644 --- a/go.mod +++ b/go.mod @@ -3,6 +3,13 @@ module github.com/OfficialSpicrawl/cli go 1.27.1 require ( + charm.land/bubbles/v2 v2.2.1 + charm.land/bubbletea/v2 v2.1.0 + charm.land/lipgloss/v2 v2.0.6 + github.com/charmbracelet/colorprofile v0.4.3 + github.com/charmbracelet/x/ansi v0.11.9 + github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f + github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169 github.com/mattn/go-isatty v0.0.20 github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.9 @@ -10,6 +17,20 @@ require ( ) require ( + github.com/aymanbagabas/go-udiff v0.4.1 // indirect + github.com/charmbracelet/harmonica v0.2.0 // indirect + github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4 // indirect + github.com/charmbracelet/x/term v0.2.2 // indirect + github.com/charmbracelet/x/termios v0.1.1 // indirect + github.com/charmbracelet/x/windows v0.2.2 // indirect + github.com/clipperhouse/displaywidth v0.11.0 // indirect + github.com/clipperhouse/uax29/v2 v2.7.0 // indirect github.com/inconshreveable/mousetrap v1.1.0 // indirect - golang.org/x/sys v0.6.0 // indirect + github.com/lucasb-eyer/go-colorful v1.4.1 // indirect + github.com/mattn/go-runewidth v0.0.30 // indirect + github.com/muesli/cancelreader v0.2.2 // indirect + github.com/rivo/uniseg v0.4.7 // indirect + github.com/xo/terminfo v1.0.0 // indirect + golang.org/x/sync v0.22.0 // indirect + golang.org/x/sys v0.47.0 // indirect ) diff --git a/go.sum b/go.sum index 8edb9ab..d47fc6e 100644 --- a/go.sum +++ b/go.sum @@ -1,16 +1,61 @@ +charm.land/bubbles/v2 v2.2.1 h1:Fq1+qm5hV6GkvzLQDhCBpXXE5tLgvh1PRriCLwSvIQU= +charm.land/bubbles/v2 v2.2.1/go.mod h1:wdMgn+sje1KNXdwFizIWjbf328fIUBxqEmJ/vYPo8yc= +charm.land/bubbletea/v2 v2.1.0 h1:A9fj9h5LFrvQhQHFNggp8K8rvY6CUfgn9dMbb6p2HIw= +charm.land/bubbletea/v2 v2.1.0/go.mod h1:xWHeyxIX9/wiFU2kgPW04ojOZyTzrWf+8lGjUCaftiU= +charm.land/lipgloss/v2 v2.0.6 h1:EaGKeuA8FvF+v2BT5VmZd2LoYLaMZJXA5n34th8nCIQ= +charm.land/lipgloss/v2 v2.0.6/go.mod h1:ipDDJNSGa1hlwDtSfW1s2/xR8Vdhbut4PXh2zEKZd0Q= +github.com/aymanbagabas/go-udiff v0.4.1 h1:OEIrQ8maEeDBXQDoGCbbTTXYJMYRCRO1fnodZ12Gv5o= +github.com/aymanbagabas/go-udiff v0.4.1/go.mod h1:0L9PGwj20lrtmEMeyw4WKJ/TMyDtvAoK9bf2u/mNo3w= +github.com/charmbracelet/colorprofile v0.4.3 h1:QPa1IWkYI+AOB+fE+mg/5/4HRMZcaXex9t5KX76i20Q= +github.com/charmbracelet/colorprofile v0.4.3/go.mod h1:/zT4BhpD5aGFpqQQqw7a+VtHCzu+zrQtt1zhMt9mR4Q= +github.com/charmbracelet/harmonica v0.2.0 h1:8NxJWRWg/bzKqqEaaeFNipOu77YR5t8aSwG4pgaUBiQ= +github.com/charmbracelet/harmonica v0.2.0/go.mod h1:KSri/1RMQOZLbw7AHqgcBycp8pgJnQMYYT8QZRqZ1Ao= +github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4 h1:hISLzUbwcvJbGoNR6A3L/TpguRRnACBmZOee8LcHLJw= +github.com/charmbracelet/ultraviolet v0.0.0-20261008173134-6b8d4baf91b4/go.mod h1:aJDVk+WOVM1K04rdc/71PMPoUGfjxjhvXfP3Pjz1Zio= +github.com/charmbracelet/x/ansi v0.11.9 h1:kIrqLBxtTVbLyyHO2Evoi93EjQo1mylWmpNEsfPxXAM= +github.com/charmbracelet/x/ansi v0.11.9/go.mod h1:DG/hpOnaI2QJ/KMMqLL2h0KVhluuJLazf8973P+Vy5w= +github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f h1:8CnFOYzrMArVN42jYaGvnBo3mxdONgt09fly+9B96GY= +github.com/charmbracelet/x/exp/golden v0.0.0-20251109135125-8916d276318f/go.mod h1:V8n/g3qVKNxr2FR37Y+otCsMySvZr601T0C7coEP0bw= +github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169 h1:jP7g4cGLDeRdqlUbQ9U3pQLm4MdOPiMknRqPId46XyY= +github.com/charmbracelet/x/exp/teatest/v2 v2.0.0-20261025003702-835ad0048169/go.mod h1:aRoQwQWmN9LBG2xi3sVByMFt2fdkPCagd0GAJ1qwOfw= +github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk= +github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI= +github.com/charmbracelet/x/termios v0.1.1 h1:o3Q2bT8eqzGnGPOYheoYS8eEleT5ZVNYNy8JawjaNZY= +github.com/charmbracelet/x/termios v0.1.1/go.mod h1:rB7fnv1TgOPOyyKRJ9o+AsTU/vK5WHJ2ivHeut/Pcwo= +github.com/charmbracelet/x/windows v0.2.2 h1:IofanmuvaxnKHuV04sC0eBy/smG6kIKrWG2/jYn2GuM= +github.com/charmbracelet/x/windows v0.2.2/go.mod h1:/8XtdKZzedat74NQFn0NGlGL4soHB0YQZrETF96h75k= +github.com/clipperhouse/displaywidth v0.11.0 h1:lBc6kY44VFw+TDx4I8opi/EtL9m20WSEFgwIwO+UVM8= +github.com/clipperhouse/displaywidth v0.11.0/go.mod h1:bkrFNkf81G8HyVqmKGxsPufD3JhNl3dSqnGhOoSD/o0= +github.com/clipperhouse/uax29/v2 v2.7.0 h1:+gs4oBZ2gPfVrKPthwbMzWZDaAFPGYK72F0NJv2v7Vk= +github.com/clipperhouse/uax29/v2 v2.7.0/go.mod h1:EFJ2TJMRUaplDxHKj1qAEhCtQPW2tJSwu5BF98AuoVM= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= +github.com/lucasb-eyer/go-colorful v1.4.1 h1:1EO+WB73+EH8EVbzlrG3KLAfEypQWVHIBqlTf+2hNss= +github.com/lucasb-eyer/go-colorful v1.4.1/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0= github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/mattn/go-runewidth v0.0.30 h1:+KUuiDA4fF0R1p5FeueHefjDm+GIM+kWfFnDjybOPgk= +github.com/mattn/go-runewidth v0.0.30/go.mod h1:3qAiGCV4Koz/yuveO58qUefmUTRm8r0IGEXZ9jeHp/8= +github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA= +github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo= +github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ= +github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY= github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/xo/terminfo v1.0.0 h1:2ZpYzqWzyyytjk3TP6aJVDhkMAkc99/1xKQdA3TDTBY= +github.com/xo/terminfo v1.0.0/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= -golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ= +golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI= +golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo= +golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek= +golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw= golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/internal/cmd/batch_watch.go b/internal/cmd/batch_watch.go new file mode 100644 index 0000000..8e18e90 --- /dev/null +++ b/internal/cmd/batch_watch.go @@ -0,0 +1,378 @@ +package cmd + +// "spicrawl batch watch ": a live dashboard for a batch job. On a terminal +// it is the full-screen view of internal/tui; anywhere else it is "batch wait". +// Either way it ends with the job printed and batch wait's exit code: 0 for +// completed, 4 for failed or cancelled, 11 when --max-wait ran out. + +import ( + "bytes" + "cmp" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/url" + "slices" + "strconv" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" + + "github.com/OfficialSpicrawl/cli/internal/api" + "github.com/OfficialSpicrawl/cli/internal/exitcode" + "github.com/OfficialSpicrawl/cli/internal/output" + "github.com/OfficialSpicrawl/cli/internal/tui" + "github.com/OfficialSpicrawl/cli/internal/ui" +) + +var ( + batchWatchPoll time.Duration + batchWatchMax time.Duration +) + +var batchWatchCmd = &cobra.Command{ + Use: "watch ", + Short: "Live dashboard for a batch job (exit codes as batch wait)", + Long: `Follow a batch job on a full-screen dashboard: a progress bar with done, failed +and total, the ETA and credits, and a table of the finished items (status, +HTTP status, tier, duration, URL), newest first. Enter opens an item (its +record and the request log entry behind it), f shows only failed items, s and +t filter by status and tier, / searches, p pauses, q or Esc leaves. + +The table lists finished items, newest first, up to the first 1000 in +submission order; the job's own counters are always complete. A tier shows once +the request log has the item's request. + +The dashboard closes when the job finishes and prints the final job, then exits +as "batch wait" does: 0 if completed, 4 if failed or cancelled, 11 if --max-wait +elapsed first. Quitting with q or Esc leaves the job running (exit 0), Ctrl-C +exits 130. A failed poll is retried with backoff and shown on the status bar. + +When stdin or stdout is not a terminal, or with --json, --quiet, CI or +TERM=dumb, there is no dashboard: this is exactly "batch wait". NO_COLOR is +honoured.`, + Example: ` spicrawl batch watch 01J9Z6V0Q8M4K2T7R3N5B1C9XA + spicrawl batch submit urls.txt --name nightly | jq -r .id | xargs spicrawl batch watch`, + Args: cobra.ExactArgs(1), + RunE: batchRunWatch, +} + +func init() { + f := batchWatchCmd.Flags() + f.DurationVar(&batchWatchPoll, "poll", 2*time.Second, "poll interval") + f.DurationVar(&batchWatchMax, "max-wait", 30*time.Minute, "give up after this long (exit 11); 0 = never") + batchCmd.AddCommand(batchWatchCmd) +} + +func batchRunWatch(cmd *cobra.Command, args []string) error { + id := args[0] + c, err := Client() + if err != nil { + return err + } + p, ctx := Printer(), cmd.Context() + if !watchInteractive() { + return batchWaitAndPrint(cmd, c, p, id, batchWatchPoll, batchWatchMax) + } + + src := &batchWatchSource{c: watchClient(c), id: id, tiers: map[string]string{}} + sp := spin(p, "Fetching batch job %s", id) + first, err := src.Poll(ctx, nil) + sp.Stop() + if err == nil && first.Done { + return batchWaitAndPrint(cmd, c, p, id, batchWatchPoll, batchWatchMax) // already over: print it + } + cfg := tui.Config{ + Title: "spicrawl batch watch", Noun: "items", Empty: "no item has finished yet", + Columns: []tui.Column{ + {Title: "SEQ", Width: 6, Right: true}, + {Title: "STATUS", Width: 9, Status: true}, + {Title: "HTTP", Width: 4, Right: true}, + {Title: "TIER", Width: 7, MinTerm: 60}, + {Title: "TOOK", Width: 6, Right: true}, + {Title: "URL"}, + }, + Statuses: []string{"succeeded", "failed", "cancelled", "skipped"}, + BadKey: "f", BadLabel: "failed", + BadStatus: func(s string) bool { return s == "failed" }, + Source: src, Initial: first, Poll: batchWatchPoll, + } + if batchWatchMax > 0 { + cfg.Deadline = time.Now().Add(batchWatchMax) + } + if err != nil { + if !watchRetryable(err) { + return err + } + cfg.InitialErr = err + } + + reason, err := tui.Run(ctx, cfg, tuiOptions...) + if err != nil { + return err + } + switch reason { + case tui.Finished: + return batchWaitAndPrint(cmd, c, p, id, batchWatchPoll, batchWatchMax) + case tui.Timeout: + raw, job := src.latest() + if err := batchPrintJob(p, raw, job); err != nil { + return err + } + return &ExitError{Code: exitcode.Pending, Err: fmt.Errorf( + "batch %s is still %s after %s; resume with: spicrawl batch watch %s", id, job.Status, batchWatchMax, id)} + case tui.Interrupt: + return &ExitError{Code: exitcode.Interrupted, Err: errors.New("interrupted")} + } + p.Info("stopped watching; the job keeps running: spicrawl batch watch %s", id) + return nil +} + +// batchWatchItem is one result line with the fields the dashboard adds to +// batchResultLine. +type batchWatchItem struct { + batchResultLine + Attempts int `json:"attempts"` + RequestID string `json:"request_id"` + CreditsMicro int64 `json:"credits_micro"` + DurationMS int64 `json:"duration_ms"` + Warnings []string `json:"warnings"` + FinishedAt string `json:"finished_at"` +} + +const ( + batchWatchPage = 500 // result lines per request + batchWatchPages = 2 // so the table holds at most 1000 items + batchWatchTiers = 200 // newest items whose tier is looked up +) + +// batchWatchSource reads a job and its finished items. It is the tui.Source of +// the dashboard. +type batchWatchSource struct { + c *api.Client + id string + + mu sync.Mutex + raw []byte // the last job, as sent + job batchJob + itemsAt batchProgressCounts // progress the table was built for + haveRows bool + items map[string]batchWatchItem // by seq, for Detail + tiers map[string]string // request id -> tier +} + +func (s *batchWatchSource) latest() ([]byte, batchJob) { + s.mu.Lock() + defer s.mu.Unlock() + return s.raw, s.job +} + +func (s *batchWatchSource) Poll(ctx context.Context, _ map[string]struct{}) (tui.Update, error) { + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/batch/" + url.PathEscape(s.id)}) + if err != nil { + return tui.Update{}, watchErr(err) + } + var job batchJob + if err := resp.Decode(&job); err != nil { + return tui.Update{}, err + } + s.mu.Lock() + s.raw, s.job = resp.Body, job + stale := !s.haveRows || s.itemsAt != job.Progress + s.mu.Unlock() + + u := tui.Update{Header: tui.BatchHeader(batchSnapshot(job)), Done: batchTerminal[job.Status]} + if u.Done || !stale { + return u, nil // a finished job ends the view; an unchanged one has the same table + } + rows, truncated, err := s.finished(ctx) + if err != nil { + u.Note = "items: " + err.Error() // keep the progress current, try the table again next poll + return u, nil + } + s.mu.Lock() + s.itemsAt, s.haveRows = job.Progress, true + s.mu.Unlock() + u.Rows, u.Replace = rows, true + if truncated { + u.Note = fmt.Sprintf("table: first %d finished items", len(rows)) + } + return u, nil +} + +func batchSnapshot(j batchJob) tui.BatchSnapshot { + pr := j.Progress + total := pr.Total + if total == 0 { + total = j.TotalItems + } + return tui.BatchSnapshot{ + ID: j.ID, Name: j.Name, Status: j.Status, + Total: total, Completed: pr.Completed, Succeeded: pr.Succeeded, Failed: pr.Failed, + Cancelled: pr.Cancelled, Skipped: pr.Skipped, Credits: pr.CreditsCharged, Reserved: j.EstimatedCredits, + Started: watchTime(j.StartedAt), Finished: watchTime(j.FinishedAt), + Error: strings.TrimSpace(j.ErrorCode + " " + j.ErrorMessage), + } +} + +func watchTime(ts string) time.Time { + t, _ := time.Parse(time.RFC3339Nano, ts) + return t +} + +// finished reads the job's finished items and returns them as rows, newest +// first. truncated says there were more than the table holds. +func (s *batchWatchSource) finished(ctx context.Context) (rows []tui.Row, truncated bool, err error) { + q := url.Values{"limit": {strconv.Itoa(batchWatchPage)}} + var items []batchWatchItem + for page := 1; ; page++ { + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/batch/" + url.PathEscape(s.id) + "/results", Query: q, Accept: "application/x-ndjson"}) + if err != nil { + return nil, false, watchErr(err) + } + dec := json.NewDecoder(bytes.NewReader(resp.Body)) + for { + var it batchWatchItem + if err := dec.Decode(&it); errors.Is(err, io.EOF) { + break + } else if err != nil { + return nil, false, fmt.Errorf("decode results: %w", err) + } + items = append(items, it) + } + next := batchNextCursor(resp) + if next == "" { + break + } + if page == batchWatchPages { + truncated = true + break + } + q.Set("cursor", next) + } + slices.SortStableFunc(items, func(a, b batchWatchItem) int { + if c := watchTime(b.FinishedAt).Compare(watchTime(a.FinishedAt)); c != 0 { + return c + } + return cmp.Compare(b.Seq, a.Seq) + }) + s.learnTiers(ctx, items[:min(len(items), batchWatchTiers)]) + + s.mu.Lock() + defer s.mu.Unlock() + s.items = make(map[string]batchWatchItem, len(items)) + for _, it := range items { + key := strconv.FormatInt(it.Seq, 10) + s.items[key] = it + dur := "—" + if it.DurationMS > 0 { + dur = ui.Elapsed(time.Duration(it.DurationMS) * time.Millisecond) + } + tier := s.tiers[it.RequestID] + code := "" + if it.Error != nil { + code = it.Error.Code + } + rows = append(rows, tui.Row{ + Key: key, Status: it.Status, Tier: tier, + Cells: []string{key, it.Status, logsOptInt(it.HTTPStatus), logsDash(tier), dur, it.URL}, + Search: strings.Join([]string{it.URL, it.ExternalID, key, it.Status, code}, " "), + }) + } + return rows, truncated, nil +} + +// learnTiers fills in the tier of items whose request the log has. The batch +// result line carries no tier, so it comes from the request log, one page for +// all of them; a miss leaves the cell as — and is not an error. +func (s *batchWatchSource) learnTiers(ctx context.Context, items []batchWatchItem) { + s.mu.Lock() + missing := false + for _, it := range items { + if _, ok := s.tiers[it.RequestID]; !ok && it.RequestID != "" { + missing = true + } + } + s.mu.Unlock() + if !missing { + return + } + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/requests", Query: url.Values{"limit": {"200"}}}) + if err != nil { + return + } + var page logsPage + if resp.Decode(&page) != nil { + return + } + s.mu.Lock() + defer s.mu.Unlock() + for _, raw := range page.Data { + var e logsEntry + if json.Unmarshal(raw, &e) == nil { + s.tiers[e.ID] = output.Tier(e.Engine) + } + } +} + +// Detail is the item's record, then the request log entry behind it ("logs get"). +func (s *batchWatchSource) Detail(ctx context.Context, key string) (string, error) { + s.mu.Lock() + it, ok := s.items[key] + tier := s.tiers[it.RequestID] + s.mu.Unlock() + if !ok { + return "", errors.New("this item is no longer in the table") + } + var b strings.Builder + var u *ui.UI + if tui.Colour() { + u = ui.Force(&b) + } + kv := func(k, v string) { + if v != "" { + fmt.Fprintf(&b, "%s %s\n", viewKey(u, 14, k+":"), v) + } + } + kv("item", key) + kv("status", logsStatusCell(u, it.Status)) + kv("url", it.URL) + kv("external id", it.ExternalID) + kv("http status", logsOptInt(it.HTTPStatus)) + kv("tier", tier) + if it.Attempts > 0 { + kv("attempts", strconv.Itoa(it.Attempts)) + } + if it.DurationMS > 0 { + kv("duration", ui.Elapsed(time.Duration(it.DurationMS)*time.Millisecond)) + } + kv("credits", logsCredits(it.CreditsMicro)) + if it.Bytes > 0 { + kv("bytes", strconv.FormatInt(it.Bytes, 10)) + } + if it.Error != nil { + kv("error", it.Error.Code) + } + kv("warnings", strings.Join(it.Warnings, ", ")) + kv("finished", it.FinishedAt) + kv("request id", it.RequestID) + + if it.RequestID != "" { + fmt.Fprintln(&b, "\n"+u.Bold("request log entry:")+"\n") + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/requests/" + url.PathEscape(it.RequestID)}) + var e logsEntry + if err == nil { + err = resp.Decode(&e) + } + if err != nil { + fmt.Fprintf(&b, "unavailable: %v\n", err) + } else { + logsWriteEntry(&b, u, &e) + } + } + return b.String(), nil +} diff --git a/internal/cmd/batch_watch_test.go b/internal/cmd/batch_watch_test.go new file mode 100644 index 0000000..235ee5c --- /dev/null +++ b/internal/cmd/batch_watch_test.go @@ -0,0 +1,432 @@ +package cmd + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/OfficialSpicrawl/cli/internal/api" + "github.com/OfficialSpicrawl/cli/internal/exitcode" +) + +// witem renders one finished item as the results endpoint does. +func witem(seq int, status string) string { + code, hs := "", 200 + if status == "failed" { + code, hs = `,"error":{"code":"ERR::UPSTREAM::BLOCKED"}`, 403 + } + return fmt.Sprintf(`{"seq":%d,"url":"https://docs.example.org/page/%d","status":%q,"attempts":1,"http_status":%d,"request_id":"RQ%d",`+ + `"credits_micro":1000000,"bytes":2048,"duration_ms":%d%s,"finished_at":"2026-09-22T06:%02d:00Z"}`+"\n", + seq, seq, status, hs, seq, 400+seq*100, code, seq) +} + +// batchFake serves one job whose state advances with every poll of it. +type batchFake struct { + srv *httptest.Server + mu sync.Mutex + jobs []string // served in order; the last repeats + items []string // finished items, in submission order + engines map[string]string + polls atomic.Int32 + results atomic.Int32 + logs atomic.Int32 + failRes atomic.Bool // results answers 503 + nextCur string // X-Next-Cursor on the first results page, when set + pages atomic.Int32 +} + +func newBatchFake(t *testing.T, jobs []string, items ...string) *batchFake { + t.Helper() + b := &batchFake{jobs: jobs, items: items, engines: map[string]string{}} + for i := range items { + b.engines[fmt.Sprintf("RQ%d", i)] = []string{"fetch", "moli", "plain-chrome", "obscura"}[i%4] + } + b.srv = httptest.NewServer(http.HandlerFunc(b.serve)) + t.Cleanup(b.srv.Close) + return b +} + +func (b *batchFake) serve(w http.ResponseWriter, r *http.Request) { + b.mu.Lock() + defer b.mu.Unlock() + switch { + case strings.HasSuffix(r.URL.Path, "/results"): + b.results.Add(1) + if b.failRes.Load() { + writeProblem(w, 503, "ERR::INTERNAL::UNAVAILABLE", true) + return + } + w.Header().Set("Content-Type", "application/x-ndjson") + if b.nextCur != "" { + w.Header().Set("X-Next-Cursor", b.nextCur) + } + b.pages.Add(1) + _, _ = fmt.Fprint(w, strings.Join(b.items, "")) + case r.URL.Path == "/v1/requests": + b.logs.Add(1) + var data []string + for id, eng := range b.engines { + data = append(data, fmt.Sprintf(`{"id":%q,"created_at":"2026-09-22T06:00:00Z","project_id":"P","status":"success","engine":%q,"url":"u","credits_micro":0,"proxy":{},"spans":{}}`, id, eng)) + } + _, _ = fmt.Fprintf(w, `{"data":[%s],"page":{"has_more":false,"retention_hours":24,"retained_since":"x"}}`, strings.Join(data, ",")) + case strings.HasPrefix(r.URL.Path, "/v1/requests/"): + id := strings.TrimPrefix(r.URL.Path, "/v1/requests/") + eng, ok := b.engines[id] + if !ok { + writeProblem(w, 404, "ERR::REQUEST::NOT_FOUND", false) + return + } + _, _ = fmt.Fprintf(w, `{"id":%q,"created_at":"2026-09-22T06:00:00Z","project_id":"P","status":"success","engine":%q,"url":"https://docs.example.org/","credits_micro":1000000,"proxy":{},"spans":{"total_ms":900}}`, id, eng) + default: // GET /v1/batch/{id} + i := min(int(b.polls.Add(1))-1, len(b.jobs)-1) + if b.jobs[i] == "" { + writeProblem(w, 503, "ERR::INTERNAL::UNAVAILABLE", true) + return + } + w.Header().Set("Content-Type", "application/json") + _, _ = fmt.Fprint(w, b.jobs[i]) + } +} + +func running(done, failed int) string { + return gJob{Status: "running", Done: done, Failed: failed, Total: 40, Name: "nightly"}.JSON() +} + +func finishedJob(status string, failed int, errCode string) string { + return gJob{Status: status, Done: 40, Failed: failed, Total: 40, Name: "nightly", Finished: true, Err: errCode}.JSON() +} + +func threeItems() []string { + return []string{witem(0, "succeeded"), witem(1, "failed"), witem(2, "succeeded")} +} + +// ---- the plain fallback is "batch wait" -------------------------------------------------------- + +func TestBatchWatchOffATerminalIsBatchWait(t *testing.T) { + for _, tc := range []struct { + name string + jobs []string + code int + }{ + {"completed", []string{running(10, 0), running(30, 1), finishedJob("completed", 1, "")}, 0}, + {"failed", []string{running(10, 0), finishedJob("failed", 5, "ERR::LIMIT::FAILURE_THRESHOLD")}, 4}, + {"cancelled", []string{finishedJob("cancelled", 0, "")}, 4}, + } { + w := newBatchFake(t, tc.jobs) + wo, we, wc := runHuman(t, w.srv.URL, false, "batch", "wait", gBatchID, "--poll", "5ms") + g := newBatchFake(t, tc.jobs) + go_, ge, gc := runHuman(t, g.srv.URL, false, "batch", "watch", gBatchID, "--poll", "5ms") + if wc != tc.code || gc != wc || go_ != wo || ge != we { + t.Errorf("%s: watch differs from wait\nwait %d %q %q\nwatch %d %q %q", tc.name, wc, wo, we, gc, go_, ge) + } + if got := g.results.Load() + g.logs.Load(); got != 0 { + t.Errorf("%s: the plain fallback read the item table (%d calls)", tc.name, got) + } + } +} + +func TestBatchWatchJSONIsBatchWaitToo(t *testing.T) { + a := newBatchFake(t, []string{running(5, 0), finishedJob("completed", 0, "")}) + wo, _, wc := runCLI(t, a.srv.URL, "batch", "wait", gBatchID, "--poll", "5ms") + b := newBatchFake(t, []string{running(5, 0), finishedJob("completed", 0, "")}) + go_, _, gc := runCLI(t, b.srv.URL, "batch", "watch", gBatchID, "--poll", "5ms") + if wc != 0 || gc != 0 || wo != go_ || !strings.Contains(go_, `"status": "completed"`) { + t.Errorf("--json watch differs from wait:\n%s\n%s", wo, go_) + } +} + +// ---- the dashboard ------------------------------------------------------------------------------ + +func TestBatchWatchFullScreenCompletedExitsZeroAndPrintsTheJob(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1), running(3, 1), running(3, 1), finishedJob("completed", 1, "")}, threeItems()...) + ft := newFakeTerminal(t) + out, errs, code := runWatch(t, ft, b.srv.URL, nil, "batch", "watch", gBatchID, "--poll", "60ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } + for _, want := range []string{"status completed", "progress 40/40"} { + if !strings.Contains(out, want) { + t.Errorf("the final job is missing %q:\n%s", want, out) + } + } + if in, restored := ft.Left(); !in || !restored { + t.Errorf("alt screen entered %v, restored %v", in, restored) + } + screen := ansi.ReplaceAllString(ft.Screen.String(), "") + for _, want := range []string{"spicrawl batch watch", "nightly", "3/40 done", "1 failed", "https://docs.example.org/page/2", "ETA", "credits"} { + if !strings.Contains(screen, want) { + t.Errorf("the dashboard never showed %q:\n%s", want, screen) + } + } + if internalNames.MatchString(screen) || internalNames.MatchString(out) { + t.Errorf("an engine name reached the screen or stdout:\n%s", screen) + } +} + +func TestBatchWatchFullScreenExitCodesMatchBatchWait(t *testing.T) { + for _, tc := range []struct { + name, status, code string + want int + msg string + }{ + {"failed", "failed", "ERR::LIMIT::FAILURE_THRESHOLD", exitcode.Request, "failed (40 of 40 items finished): ERR::LIMIT::FAILURE_THRESHOLD"}, + {"cancelled", "cancelled", "", exitcode.Request, "cancelled (40 of 40 items finished)"}, + } { + b := newBatchFake(t, []string{running(5, 0), running(5, 0), finishedJob(tc.status, 2, tc.code)}, threeItems()...) + ft := newFakeTerminal(t) + out, errs, code := runWatch(t, ft, b.srv.URL, nil, "batch", "watch", gBatchID, "--poll", "40ms") + if code != tc.want || !strings.Contains(errs, tc.msg) || !strings.Contains(out, "status "+tc.status) { + t.Errorf("%s: exit %d (want %d)\nstderr %q\nstdout %q", tc.name, code, tc.want, errs, out) + } + if _, restored := ft.Left(); !restored { + t.Errorf("%s: screen not restored", tc.name) + } + } +} + +func TestBatchWatchFullScreenMaxWaitExits11(t *testing.T) { + b := newBatchFake(t, []string{running(5, 0)}, threeItems()...) + ft := newFakeTerminal(t) + out, errs, code := runWatch(t, ft, b.srv.URL, nil, "batch", "watch", gBatchID, "--poll", "20ms", "--max-wait", "150ms") + if code != exitcode.Pending || !strings.Contains(errs, "still running after 150ms; resume with: spicrawl batch watch "+gBatchID) { + t.Errorf("exit %d\n%s", code, errs) + } + if !strings.Contains(out, "status running") { + t.Errorf("the last job was not printed:\n%s", out) + } + if _, restored := ft.Left(); !restored { + t.Error("screen not restored") + } +} + +func TestBatchWatchFullScreenQuitLeavesTheJobRunning(t *testing.T) { + b := newBatchFake(t, []string{running(5, 1)}, threeItems()...) + ft := newFakeTerminal(t) + out, errs, code := runWatch(t, ft, b.srv.URL, func() { + ft.Wait(t, "https://docs.example.org/page/2") + ft.Press("q") + }, "batch", "watch", gBatchID, "--poll", "20ms") + if code != 0 || out != "" || !strings.Contains(errs, "the job keeps running: spicrawl batch watch "+gBatchID) { + t.Errorf("exit %d\nstdout %q\nstderr %q", code, out, errs) + } +} + +func TestBatchWatchFullScreenCtrlCExits130(t *testing.T) { + b := newBatchFake(t, []string{running(5, 1)}, threeItems()...) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, b.srv.URL, func() { + ft.Wait(t, "https://docs.example.org/page/2") + ft.Press("\x03") + }, "batch", "watch", gBatchID, "--poll", "20ms") + if code != exitcode.Interrupted || !strings.Contains(errs, "interrupted") { + t.Errorf("exit %d, stderr %q", code, errs) + } + if _, restored := ft.Left(); !restored { + t.Error("screen not restored after Ctrl-C") + } +} + +func TestBatchWatchFullScreenFilterFailedAndOpenItem(t *testing.T) { + b := newBatchFake(t, []string{running(5, 1)}, threeItems()...) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, b.srv.URL, func() { + ft.Wait(t, "https://docs.example.org/page/2") + ft.Press("f") + ft.Wait(t, "1 shown") + ft.Press("\r") // the only failed item is seq 1 + ft.Wait(t, "request log entry:") + ft.Press("\x1b[6~") // PgDn: the page is longer than 24 rows + ft.Wait(t, "spans (ms):") + ft.Press("q") + time.Sleep(60 * time.Millisecond) + ft.Press("q") + }, "batch", "watch", gBatchID, "--poll", "20ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } + page := ansi.ReplaceAllString(ft.Screen.String(), "") + for _, want := range []string{"item:", "ERR::UPSTREAM::BLOCKED", "request id:", "RQ1", "tier:", "js"} { + if !strings.Contains(page, want) { + t.Errorf("item detail lacks %q", want) + } + } + if internalNames.MatchString(page) { + t.Errorf("engine name in the item detail:\n%s", page) + } +} + +func TestBatchWatchFullScreenAlreadyFinishedNeverOpensTheView(t *testing.T) { + b := newBatchFake(t, []string{finishedJob("failed", 3, "ERR::LIMIT::FAILURE_THRESHOLD")}, threeItems()...) + ft := newFakeTerminal(t) + out, _, code := runWatch(t, ft, b.srv.URL, nil, "batch", "watch", gBatchID) + if code != exitcode.Request || !strings.Contains(out, "status failed") { + t.Errorf("exit %d\n%s", code, out) + } + if in, _ := ft.Left(); in { + t.Error("opened the dashboard for a job that was already over") + } + if b.results.Load() != 0 { + t.Error("read the item table of a finished job") + } +} + +func TestBatchWatchFullScreenUnknownJobFailsAtOnce(t *testing.T) { + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + writeProblem(w, 404, "ERR::REQUEST::NOT_FOUND", false) + }) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, srv.URL, nil, "batch", "watch", "nope") + if code != exitcode.Request || !strings.Contains(errs, "ERR::REQUEST::NOT_FOUND") { + t.Errorf("exit %d: %s", code, errs) + } + if in, _ := ft.Left(); in || len(*reqs) != 1 { + t.Errorf("entered the view %v, %d requests", in, len(*reqs)) + } +} + +func TestBatchWatchFullScreenKeepsRetryingAndShowsTheError(t *testing.T) { + // 503 on the first poll, then a good one, a failing one, and the end. + b := newBatchFake(t, []string{"", running(3, 1), "", finishedJob("completed", 1, "")}, threeItems()...) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, b.srv.URL, nil, "batch", "watch", gBatchID, "--poll", "15ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } + screen := ansi.ReplaceAllString(ft.Screen.String(), "") + if !strings.Contains(screen, "ERR::INTERNAL::UNAVAILABLE") || !strings.Contains(screen, "retrying in") { + t.Errorf("the error never reached the status bar:\n%s", screen) + } +} + +// ---- the source --------------------------------------------------------------------------------------- + +func newSource(b *batchFake) *batchWatchSource { + return &batchWatchSource{c: api.New(b.srv.URL, "k", 2*time.Second), id: gBatchID, tiers: map[string]string{}} +} + +func TestBatchWatchSourceRowsNewestFirstWithTiersFromTheRequestLog(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1)}, threeItems()...) + src := newSource(b) + u, err := src.Poll(context.Background(), nil) + if err != nil || !u.Replace || len(u.Rows) != 3 || u.Header == nil || u.Done { + t.Fatalf("%v %+v", err, u) + } + // finished_at is 06:02, 06:01, 06:00 for seq 2, 1, 0 + if u.Rows[0].Key != "2" || u.Rows[2].Key != "0" { + t.Errorf("not newest first: %s %s %s", u.Rows[0].Key, u.Rows[1].Key, u.Rows[2].Key) + } + r := u.Rows[1] // seq 1: failed, engine "moli" in the log + if got := strings.Join(r.Cells, "|"); got != "1|failed|403|js|0.5s|https://docs.example.org/page/1" || r.Tier != "js" || r.Status != "failed" { + t.Errorf("row %q tier %q", got, r.Tier) + } + if got := u.Rows[0].Tier; got != "browser" { // seq 2 -> plain-chrome + t.Errorf("tier of seq 2 = %q", got) + } + for _, row := range u.Rows { + for _, c := range row.Cells { + if internalNames.MatchString(c) { + t.Errorf("cell %q", c) + } + } + } + if b.logs.Load() != 1 { + t.Errorf("%d request-log reads for one table", b.logs.Load()) + } +} + +func TestBatchWatchSourceOnlyRereadsTheTableWhenProgressMoves(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1), running(3, 1), running(4, 1), finishedJob("completed", 1, "")}, threeItems()...) + src := newSource(b) + ctx := context.Background() + u1, _ := src.Poll(ctx, nil) + u2, _ := src.Poll(ctx, nil) // same counters: header only + if len(u1.Rows) != 3 || u2.Replace || len(u2.Rows) != 0 || u2.Header == nil { + t.Errorf("unchanged progress: %+v", u2) + } + if b.results.Load() != 1 { + t.Errorf("%d result reads after an unchanged poll", b.results.Load()) + } + u3, _ := src.Poll(ctx, nil) + if !u3.Replace || b.results.Load() != 2 { + t.Errorf("moved progress did not refresh the table: %+v", u3) + } + u4, _ := src.Poll(ctx, nil) // terminal: Done, and no download of the inline bodies + if !u4.Done || b.results.Load() != 2 { + t.Errorf("done=%v results=%d", u4.Done, b.results.Load()) + } +} + +func TestBatchWatchSourceKeepsProgressWhenTheTableFails(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1), running(4, 1)}, threeItems()...) + b.failRes.Store(true) + src := newSource(b) + u, err := src.Poll(context.Background(), nil) + if err != nil || u.Header == nil || u.Replace || !strings.HasPrefix(u.Note, "items: ") { + t.Fatalf("err %v update %+v", err, u) + } + b.failRes.Store(false) + u, _ = src.Poll(context.Background(), nil) + if !u.Replace || len(u.Rows) != 3 { + t.Errorf("the table did not come back: %+v", u) + } +} + +func TestBatchWatchSourceSaysWhenTheTableIsCut(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1)}, threeItems()...) + b.nextCur = "more" // every page claims another follows: the table stops at two pages + src := newSource(b) + u, err := src.Poll(context.Background(), nil) + if err != nil || !strings.Contains(u.Note, "first") { + t.Errorf("err %v note %q", err, u.Note) + } + if b.pages.Load() != batchWatchPages { + t.Errorf("%d pages read, want %d", b.pages.Load(), batchWatchPages) + } +} + +func TestBatchWatchSourceDetail(t *testing.T) { + b := newBatchFake(t, []string{running(3, 1)}, threeItems()...) + src := newSource(b) + if _, err := src.Poll(context.Background(), nil); err != nil { + t.Fatal(err) + } + text, err := src.Detail(context.Background(), "1") + if err != nil { + t.Fatal(err) + } + plain := ansi.ReplaceAllString(text, "") + for _, want := range []string{"item:", "failed", "https://docs.example.org/page/1", "403", "tier: js", "attempts: 1", + "ERR::UPSTREAM::BLOCKED", "request id: RQ1", "request log entry:", "spans (ms):", "total 900"} { + if !strings.Contains(plain, want) { + t.Errorf("detail lacks %q:\n%s", want, plain) + } + } + if internalNames.MatchString(plain) { + t.Errorf("engine name in detail:\n%s", plain) + } + if _, err := src.Detail(context.Background(), "999"); err == nil { + t.Error("an unknown item must be an error") + } + delete(b.engines, "RQ1") // the request log no longer has it: the item page still renders + text, err = src.Detail(context.Background(), "1") + if err != nil || !strings.Contains(text, "unavailable:") { + t.Errorf("err %v\n%s", err, text) + } +} + +func TestBatchSnapshotCounts(t *testing.T) { + var j batchJob + if err := (&api.Response{Body: []byte(running(7, 2))}).Decode(&j); err != nil { + t.Fatal(err) + } + s := batchSnapshot(j) + if s.Total != 40 || s.Completed != 7 || s.Failed != 2 || s.Succeeded != 5 || s.Started.IsZero() || s.ID != gBatchID { + t.Errorf("%+v", s) + } +} diff --git a/internal/cmd/logs_watch.go b/internal/cmd/logs_watch.go new file mode 100644 index 0000000..ac88939 --- /dev/null +++ b/internal/cmd/logs_watch.go @@ -0,0 +1,336 @@ +package cmd + +// "spicrawl logs --watch": a live request log. On a terminal it is the +// full-screen view of internal/tui; anywhere else it streams new rows as +// plain lines (JSONL in JSON mode), oldest first, like tail -f. +// +// The flags and the branch are added from here (init wraps logsCmd.RunE), so +// logs.go is untouched. + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "strconv" + "strings" + "time" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" + + "github.com/OfficialSpicrawl/cli/internal/api" + "github.com/OfficialSpicrawl/cli/internal/exitcode" + "github.com/OfficialSpicrawl/cli/internal/output" + "github.com/OfficialSpicrawl/cli/internal/tui" + "github.com/OfficialSpicrawl/cli/internal/ui" +) + +var ( + logsWatch bool + logsPoll time.Duration +) + +func init() { + f := logsCmd.Flags() + f.BoolVar(&logsWatch, "watch", false, "live view: new requests appear at the top (full-screen on a terminal; plain lines, or JSONL with --json, otherwise)") + f.DurationVar(&logsPoll, "poll", 2*time.Second, "with --watch: poll interval") + logsCmd.Example += ` + spicrawl logs --watch + spicrawl logs --watch --errors --all-projects + spicrawl logs --watch --json | jq -r 'select(.status != "success") | .url'` + logsCmd.Long += ` + +--watch polls the log and shows new requests as they arrive. On a terminal it +is full-screen: arrows move, Enter opens a request (what "logs get" prints), +/ searches, s, t and e filter by status, tier and errors, p pauses, q or Esc +quits. Filters apply to the rows loaded so far (the newest --limit, up to 200, +and everything since); --errors and --status narrow the query itself. When +stdin or stdout is not a terminal, or with --json, --quiet, CI or TERM=dumb, it +prints new requests one per line instead (JSONL with --json). NO_COLOR is +honoured. A failed poll is retried with backoff.` + + run := logsCmd.RunE + logsCmd.RunE = func(cmd *cobra.Command, args []string) error { + switch { + case logsWatch: + return logsRunWatch(cmd) + case cmd.Flags().Changed("poll"): + return Usagef("--poll needs --watch") + } + return run(cmd, args) + } +} + +func logsRunWatch(cmd *cobra.Command) error { + if logsAll || logsJSONL { + return Usagef("--watch cannot be combined with --all or --jsonl") + } + if logsPoll <= 0 { + return Usagef("--poll must be positive") + } + q, err := logsWatchQuery(cmd.Flags()) + if err != nil { + return err + } + c, err := Client() + if err != nil { + return err + } + p, ctx := Printer(), cmd.Context() + src := &logsWatchSource{c: watchClient(c), q: q, project: logsAllProjects} + if !watchInteractive() { + return logsWatchPlain(ctx, p, src, logsPoll) + } + + sp := spin(p, "Fetching the request log") + first, err := src.Poll(ctx, nil) + sp.Stop() + cfg := tui.Config{ + Title: "spicrawl logs", Noun: "requests", Empty: "no requests yet: new ones appear here as they arrive", + Columns: src.columns(), Statuses: logsStatuses, BadKey: "e", BadLabel: "errors", + BadStatus: func(s string) bool { return s != "success" }, + Source: src, Initial: first, Poll: logsPoll, + } + if err != nil { + if !watchRetryable(err) { + return err + } + cfg.InitialErr = err + } + reason, err := tui.Run(ctx, cfg, tuiOptions...) + if err != nil { + return err + } + if reason == tui.Interrupt { + return &ExitError{Code: exitcode.Interrupted, Err: errors.New("interrupted")} + } + return nil +} + +// logsWatchQuery is the query of "logs" for the flags --watch shares with it +// (the validation is the same as logs.go's). A watch fetches the largest page +// unless --limit says otherwise, so the filters have rows to work on. +func logsWatchQuery(f *pflag.FlagSet) (url.Values, error) { + q := url.Values{} + limit := 200 + if f.Changed("limit") { + if logsLimit < 1 || logsLimit > 200 { + return nil, Usagef("--limit must be between 1 and 200") + } + limit = logsLimit + } + q.Set("limit", strconv.Itoa(limit)) + if logsStatus != "" { + s := strings.ToLower(logsStatus) + found := false + for _, v := range logsStatuses { + found = found || v == s + } + if !found { + return nil, Usagef("--status must be one of %s, not %q", strings.Join(logsStatuses, ", "), logsStatus) + } + if logsErrors { + return nil, Usagef("--errors and --status cannot be combined") + } + q.Set("status", s) + } + if logsErrors { + q.Set("only_errors", "true") + } + if logsAllProjects { + q.Set("all_projects", "true") + } + return q, nil +} + +// logsWatchSource reads the request log. It is the tui.Source of the view and +// the poller of the plain stream. +type logsWatchSource struct { + c *api.Client + q url.Values + project bool +} + +type logsRec struct { + raw json.RawMessage + e logsEntry +} + +// maxGapPages bounds how far back one poll pages to reach rows it already has. +const maxGapPages = 5 + +// fetch returns the requests newer than known, newest first. The first poll +// (nothing known) reads one page; later ones page back until they reach a +// request they have, so a burst longer than one page leaves no hole. +func (s *logsWatchSource) fetch(ctx context.Context, known map[string]struct{}) ([]logsRec, error) { + q := url.Values{} + for k, v := range s.q { + q[k] = v + } + pages := 1 + if len(known) > 0 { + pages = maxGapPages + } + var out []logsRec + for i := 0; i < pages; i++ { + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/requests", Query: q}) + if err != nil { + return nil, watchErr(err) + } + var page logsPage + if err := resp.Decode(&page); err != nil { + return nil, err + } + reached := false + for _, raw := range page.Data { + var r logsRec + r.raw = raw + _ = json.Unmarshal(raw, &r.e) + if _, ok := known[r.e.ID]; ok { + reached = true + continue + } + out = append(out, r) + } + if reached || !page.Page.HasMore || page.Page.NextBefore == "" || page.Page.NextBeforeID == "" { + break + } + q.Set("before", page.Page.NextBefore) + q.Set("before_id", page.Page.NextBeforeID) + } + return out, nil +} + +func (s *logsWatchSource) columns() []tui.Column { + cols := []tui.Column{{Title: "TIME", Width: 8}} + if s.project { + cols = append(cols, tui.Column{Title: "PROJECT", Width: 8}) + } + return append(cols, + tui.Column{Title: "STATUS", Width: 9, Status: true}, + tui.Column{Title: "HTTP", Width: 4, Right: true}, + tui.Column{Title: "TIER", Width: 7, MinTerm: 60}, + tui.Column{Title: "CREDITS", Width: 8, Right: true, MinTerm: 100}, + tui.Column{Title: "MS", Width: 6, Right: true, MinTerm: 72}, + tui.Column{Title: "URL"}, + ) +} + +func (s *logsWatchSource) row(e logsEntry) tui.Row { + tier := output.Tier(e.Engine) + cells := []string{watchClock(e.CreatedAt)} + if s.project { + cells = append(cells, strings.TrimSpace(tail(e.ProjectID, 8))) + } + cells = append(cells, e.Status, logsOptInt(e.HTTPStatus), logsDash(tier), logsCredits(e.CreditsMicro), logsSpan(e.Spans.Total), e.URL) + return tui.Row{ + Key: e.ID, Cells: cells, Status: e.Status, Tier: tier, + Search: strings.Join([]string{e.URL, e.ID, e.Status, e.ErrorCode, e.ProjectID}, " "), + } +} + +func (s *logsWatchSource) Poll(ctx context.Context, known map[string]struct{}) (tui.Update, error) { + recs, err := s.fetch(ctx, known) + if err != nil { + return tui.Update{}, err + } + rows := make([]tui.Row, len(recs)) + for i, r := range recs { + rows[i] = s.row(r.e) + } + return tui.Update{Rows: rows}, nil +} + +// Detail is the page of "logs get", in colour unless NO_COLOR is set. +func (s *logsWatchSource) Detail(ctx context.Context, id string) (string, error) { + resp, err := s.c.Do(ctx, api.Request{Path: "/v1/requests/" + url.PathEscape(id)}) + if err != nil { + return "", err + } + var e logsEntry + if err := resp.Decode(&e); err != nil { + return "", err + } + var b strings.Builder + var u *ui.UI + if tui.Colour() { + u = ui.Force(&b) + } + logsWriteEntry(&b, u, &e) + return b.String(), nil +} + +// logsWatchPlain streams requests as they arrive, oldest first within a poll: +// the newest page at the start, then only new ones. Failed polls are retried +// with backoff and reported on stderr; Ctrl-C ends it. +func logsWatchPlain(ctx context.Context, p *output.Printer, src *logsWatchSource, every time.Duration) error { + var newest []string // the newest request ids seen, newest first + known := map[string]struct{}{} + failures := 0 + for first := true; ; first = false { + recs, err := src.fetch(ctx, known) + delay := every + if err != nil { + if ctx.Err() != nil { + return ctx.Err() + } + if first && !watchRetryable(err) { + return err + } + failures++ + delay = tui.Backoff(every, failures, err) + p.Warn("request log: %v; retrying in %s", err, delay.Round(time.Second)) + } else { + failures = 0 + ids := make([]string, 0, len(recs)) + for i := len(recs) - 1; i >= 0; i-- { + if err := logsStreamLine(p, recs[i]); err != nil { + return err + } + } + for _, r := range recs { + ids = append(ids, r.e.ID) + } + newest = append(ids, newest...) + newest = newest[:min(len(newest), 400)] + known = make(map[string]struct{}, len(newest)) + for _, id := range newest { + known[id] = struct{}{} + } + } + if err := sleepCtx(ctx, delay); err != nil { + return err + } + } +} + +func logsStreamLine(p *output.Printer, r logsRec) error { + if p.JSON { + return p.Line(r.raw) + } + e := r.e + _, err := fmt.Fprintf(p.Out, "%s %-9s %4s %-7s %8s %6s %s\n", + e.CreatedAt, e.Status, logsOptInt(e.HTTPStatus), logsDash(output.Tier(e.Engine)), + logsCredits(e.CreditsMicro), logsSpan(e.Spans.Total), e.URL) + return err +} + +// watchClock is a timestamp as HH:MM:SS in the local zone; anything that does +// not parse is shown as sent. +func watchClock(ts string) string { + t, err := time.Parse(time.RFC3339Nano, ts) + if err != nil { + return ts + } + return t.Local().Format("15:04:05") +} + +// tail is the last n bytes of s. +func tail(s string, n int) string { + if len(s) <= n { + return s + } + return s[len(s)-n:] +} diff --git a/internal/cmd/logs_watch_test.go b/internal/cmd/logs_watch_test.go new file mode 100644 index 0000000..d3cb430 --- /dev/null +++ b/internal/cmd/logs_watch_test.go @@ -0,0 +1,429 @@ +package cmd + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "strings" + "sync/atomic" + "testing" + "time" + + "github.com/OfficialSpicrawl/cli/internal/api" + "github.com/OfficialSpicrawl/cli/internal/output" + "github.com/OfficialSpicrawl/cli/internal/tui" +) + +func init() { time.Local = time.UTC } // watchClock shows local time; the tests read it as UTC + +func logsEntryJSON(id, created, status, engine, url string) string { + return fmt.Sprintf(`{"id":%q,"created_at":%q,"project_id":"P-ONE","status":%q,"http_status":200,"engine":%q,"credits_micro":1000000,"url":%q,`+ + `"error_code":"","error_detail":"","attempts":1,"proxy":{},"spans":{"total_ms":120}}`, id, created, status, engine, url) +} + +func logsList(more bool, entries ...string) string { + page := `{"has_more":false,"retention_hours":24,"retained_since":"x"}` + if more { + page = `{"has_more":true,"next_before":"2026-09-22T10:00:00Z","next_before_id":"cursor","retention_hours":24,"retained_since":"x"}` + } + return `{"data":[` + strings.Join(entries, ",") + `],"page":` + page + `}` +} + +// logsScripted answers /v1/requests with the next body each call (the last repeats). +func logsScripted(t *testing.T, bodies ...string) (string, func() []recordedRequest) { + t.Helper() + var n atomic.Int32 + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + i := min(int(n.Add(1))-1, len(bodies)-1) + _, _ = w.Write([]byte(bodies[i])) + }) + return srv.URL, func() []recordedRequest { return *reqs } +} + +var ( + e1 = logsEntryJSON("E1", "2026-09-22T10:00:01Z", "success", "fetch", "https://x.test/1") + e2 = logsEntryJSON("E2", "2026-09-22T10:00:02Z", "failed", "moli", "https://x.test/2") + e3 = logsEntryJSON("E3", "2026-09-22T10:00:03Z", "blocked", "plain-chrome", "https://x.test/3") + e4 = logsEntryJSON("E4", "2026-09-22T10:00:04Z", "success", "fetch", "https://x.test/4") + e5 = logsEntryJSON("E5", "2026-09-22T10:00:05Z", "success", "obscura", "https://x.test/5") + e6 = logsEntryJSON("E6", "2026-09-22T10:00:06Z", "success", "fetch", "https://x.test/6") +) + +// streamUntil runs the plain stream until out has n lines, lets it poll a few +// more times (nothing more may appear), then stops it. +func streamUntil(t *testing.T, base string, json bool, n int) (out, errs string, err error) { + t.Helper() + var ob, eb lockedBuf + p := &output.Printer{JSON: json, Out: &ob, Err: &eb, Plain: true} + src := &logsWatchSource{c: api.New(base, "k", 2*time.Second), q: map[string][]string{"limit": {"200"}}} + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + done := make(chan error, 1) + go func() { done <- logsWatchPlain(ctx, p, src, 5*time.Millisecond) }() + count := func() int { + n := 0 + for _, l := range strings.Split(ob.String(), "\n") { + if strings.TrimSpace(l) != "" { + n++ + } + } + return n + } + for count() < n { + select { + case err := <-done: + return ob.String(), eb.String(), err + case <-time.After(5 * time.Millisecond): + } + } + time.Sleep(60 * time.Millisecond) + cancel() + err = <-done + return ob.String(), eb.String(), err +} + +func lineIDs(out string) []string { + var ids []string + for _, l := range strings.Split(strings.TrimSpace(out), "\n") { + if l == "" { + continue + } + if i := strings.Index(l, "https://x.test/"); i >= 0 { + ids = append(ids, "E"+l[i+len("https://x.test/"):]) + } + } + return ids +} + +// ---- flags ------------------------------------------------------------------------------- + +func TestLogsWatchUsageErrors(t *testing.T) { + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) {}) + for _, args := range [][]string{ + {"logs", "--watch", "--all"}, + {"logs", "--watch", "--jsonl"}, + {"logs", "--poll", "5s"}, + {"logs", "--watch", "--poll", "0s"}, + {"logs", "--watch", "--limit", "500"}, + {"logs", "--watch", "--status", "error"}, + {"logs", "--watch", "--errors", "--status", "failed"}, + {"logs", "get", "R1", "--watch"}, + } { + if _, errb, code := runCLI(t, srv.URL, args...); code != 2 { + t.Errorf("%v: exit %d (stderr %q)", args, code, errb) + } + } + if len(*reqs) != 0 { + t.Errorf("a usage error sent %d requests", len(*reqs)) + } +} + +func TestLogsWatchQueryAndFatalFirstPoll(t *testing.T) { + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + writeProblem(w, 401, "ERR::AUTH::INVALID_KEY", false) + }) + _, _, code := runCLI(t, srv.URL, "logs", "--watch", "--errors", "--all-projects") + if code != 3 { + t.Errorf("a wrong key must fail at once with exit 3, got %d", code) + } + if len(*reqs) != 1 || (*reqs)[0].Query != "all_projects=true&limit=200&only_errors=true" { + t.Errorf("requests: %+v", *reqs) + } + + _, _, code = runCLI(t, srv.URL, "logs", "--watch", "--limit", "20", "--status", "Blocked") + if q := (*reqs)[1].Query; code != 3 || q != "limit=20&status=blocked" { + t.Errorf("exit %d, query %q", code, q) + } +} + +func TestLogsWithoutWatchIsUnchanged(t *testing.T) { + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + _, _ = w.Write([]byte(logsList(false, e1))) + }) + out, _, code := runCLI(t, srv.URL, "logs") + if code != 0 || (*reqs)[0].Query != "" || !strings.Contains(out, `"id": "E1"`) { + t.Errorf("exit %d, query %q\n%s", code, (*reqs)[0].Query, out) + } +} + +// ---- the plain stream (stdout is not a terminal, or --json) --------------------------------- + +func TestLogsWatchPlainStreamsNewRowsOldestFirst(t *testing.T) { + srv, reqs := logsScripted(t, + logsList(false, e3, e2), // the newest page at the start + logsList(false, e4, e3, e2), // one new row + logsList(false, e4, e3, e2), // nothing new + ) + out, _, err := streamUntil(t, srv, false, 3) + if !errors.Is(err, context.Canceled) && !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("err %v", err) + } + if got := strings.Join(lineIDs(out), " "); got != "E2 E3 E4" { + t.Errorf("rows: %q\n%s", got, out) + } + if !strings.Contains(out, "failed") || !strings.Contains(out, "2026-09-22T10:00:02Z") { + t.Errorf("plain line lacks status or time:\n%s", out) + } + if len(reqs()) < 3 { + t.Errorf("polled %d times", len(reqs())) + } +} + +func TestLogsWatchPlainJSONLIsTheAPIsRecords(t *testing.T) { + srv, _ := logsScripted(t, logsList(false, e2, e1), logsList(false, e3, e2, e1)) + out, _, _ := streamUntil(t, srv, true, 3) + var ids []string + for _, l := range strings.Split(strings.TrimSpace(out), "\n") { + var v map[string]any + if err := json.Unmarshal([]byte(l), &v); err != nil { + t.Fatalf("line %q is not JSON: %v", l, err) + } + ids = append(ids, v["id"].(string)) + if v["engine"] == nil { + t.Errorf("record was reshaped: %q", l) + } + } + if got := strings.Join(ids, " "); got != "E1 E2 E3" { + t.Errorf("ids %q", got) + } +} + +func TestLogsWatchPlainPagesBackToCloseAGap(t *testing.T) { + srv, reqs := logsScripted(t, + logsList(false, e2, e1), + logsList(true, e6, e5), // a burst: two pages of new rows... + logsList(true, e4, e3), // ...a third page... + logsList(false, e2, e1), // ...and then the rows we already have + logsList(false, e6, e5, e4, e3, e2, e1), + ) + out, _, _ := streamUntil(t, srv, false, 6) + if got := strings.Join(lineIDs(out), " "); got != "E1 E2 E3 E4 E5 E6" { + t.Errorf("rows %q (a burst longer than one page left a hole or a duplicate)\n%s", got, out) + } + r := reqs() + if r[1].Query != "limit=200" || !strings.Contains(r[2].Query, "before=2026-09-22T10%3A00%3A00Z") || !strings.Contains(r[2].Query, "before_id=cursor") { + t.Errorf("second poll queries: %q then %q", r[1].Query, r[2].Query) + } +} + +func TestLogsWatchPlainRetriesAndReports(t *testing.T) { + var n atomic.Int32 + srv, _ := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + switch n.Add(1) { + case 1: + _, _ = w.Write([]byte(logsList(false, e1))) + case 2: + writeProblem(w, 503, "ERR::INTERNAL::UNAVAILABLE", true) + default: + _, _ = w.Write([]byte(logsList(false, e2, e1))) + } + }) + out, errs, _ := streamUntil(t, srv.URL, false, 2) + if got := strings.Join(lineIDs(out), " "); got != "E1 E2" { + t.Errorf("rows %q", got) + } + if !strings.Contains(errs, "warning: request log:") || !strings.Contains(errs, "retrying in") { + t.Errorf("the failed poll was not reported on stderr: %q", errs) + } +} + +func TestLogsWatchPlainFirstPollFatalAndRetryable(t *testing.T) { + srv, _ := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + writeProblem(w, 403, "ERR::AUTH::INSUFFICIENT_SCOPE", false) + }) + _, _, err := streamUntil(t, srv.URL, false, 1) + if ExitCode(err) != 3 { + t.Errorf("exit code %d for %v", ExitCode(err), err) + } +} + +func TestLogsWatchPlainStopsWhenStdoutCloses(t *testing.T) { + srv, _ := logsScripted(t, logsList(false, e1)) + p := &output.Printer{Out: failWriter{}, Err: &lockedBuf{}, Plain: true} + src := &logsWatchSource{c: api.New(srv, "k", time.Second), q: map[string][]string{}} + err := logsWatchPlain(context.Background(), p, src, time.Millisecond) + if err == nil || !strings.Contains(err.Error(), "closed pipe") { + t.Errorf("a failed write must end the stream, got %v", err) + } +} + +type failWriter struct{} + +func (failWriter) Write([]byte) (int, error) { return 0, errors.New("write: closed pipe") } + +func TestLogsWatchShowsTiersNeverEngines(t *testing.T) { + srv, _ := logsScripted(t, logsList(false, e5, e3, e2, e1)) + out, _, _ := streamUntil(t, srv, false, 4) + if internalNames.MatchString(out) { + t.Errorf("an engine name reached the stream:\n%s", out) + } + for _, tier := range []string{"fetch", "js", "browser"} { + if !strings.Contains(out, tier) { + t.Errorf("tier %q missing:\n%s", tier, out) + } + } +} + +func TestLogsWatchSourceRowsAreTiersAndLocalTimes(t *testing.T) { + srv, _ := logsScripted(t, logsList(false, e3, e2)) + src := &logsWatchSource{c: api.New(srv, "k", time.Second), q: map[string][]string{}, project: true} + u, err := src.Poll(context.Background(), nil) + if err != nil || len(u.Rows) != 2 { + t.Fatalf("%v %+v", err, u) + } + r := u.Rows[0] + if r.Key != "E3" || r.Tier != "browser" || r.Status != "blocked" { + t.Errorf("row %+v", r) + } + if got := strings.Join(r.Cells, "|"); got != "10:00:03|P-ONE|blocked|200|browser|1|120|https://x.test/3" { + t.Errorf("cells %q", got) + } + if u.Rows[1].Tier != "js" { + t.Errorf("moli must show as js, got %q", u.Rows[1].Tier) + } + if cols := src.columns(); len(cols) != len(r.Cells) || cols[1].Title != "PROJECT" { + t.Errorf("columns do not line up with cells: %+v", cols) + } + for _, c := range r.Cells { + if internalNames.MatchString(c) { + t.Errorf("cell %q", c) + } + } +} + +func TestWatchErrMarks429WithRetryAfter(t *testing.T) { + err := watchErr(&api.Problem{Status: 429, Code: "ERR::LIMIT::RATE", RetryAfterSeconds: 7}) + if d := tui.Backoff(time.Second, 3, err); d != 7*time.Second { + t.Errorf("backoff ignored Retry-After: %s", d) + } + var plain = &api.Problem{Status: 500, Code: "ERR::INTERNAL::ERROR"} + if watchErr(plain) != error(plain) { + t.Error("only a 429 with Retry-After is wrapped") + } + if !watchRetryable(err) || watchRetryable(&api.Problem{Status: 401}) || !watchRetryable(&api.NetworkError{Err: errors.New("refused")}) { + t.Error("watchRetryable") + } +} + +// ---- the full-screen path --------------------------------------------------------------------- + +func logsTUIServer(t *testing.T) (string, func() []recordedRequest) { + e := logsEntryJSON("R1", "2026-09-22T10:00:00.5Z", "blocked", "moli", "https://example.com/pricing") + e = strings.Replace(e, `"error_detail":""`, `"error_detail":"the site served a plain-chrome challenge"`, 1) + srv, reqs := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + if strings.HasPrefix(r.URL.Path, "/v1/requests/") { + _, _ = w.Write([]byte(e)) + return + } + _, _ = w.Write([]byte(logsList(false, e))) + }) + return srv.URL, func() []recordedRequest { return *reqs } +} + +func TestLogsWatchFullScreenQuitsOnQ(t *testing.T) { + base, _ := logsTUIServer(t) + ft := newFakeTerminal(t) + out, errs, code := runWatch(t, ft, base, func() { + ft.Wait(t, "https://example.com/pricing") + ft.Press("q") + }, "logs", "--watch", "--poll", "20ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } + if out != "" { + t.Errorf("stdout must stay empty under the full-screen view: %q", out) + } + if in, restored := ft.Left(); !in || !restored { + t.Errorf("alt screen entered %v, restored %v", in, restored) + } + if s := ft.Screen.String(); internalNames.MatchString(s) || !strings.Contains(s, "spicrawl logs") { + t.Errorf("screen:\n%s", ansi.ReplaceAllString(s, "")) + } + if errs != "" { + t.Errorf("stderr must stay quiet under the view (a spinner would be drawn over it): %q", errs) + } +} + +func TestLogsWatchFullScreenCtrlCExits130(t *testing.T) { + base, _ := logsTUIServer(t) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, base, func() { + ft.Wait(t, "https://example.com/pricing") + ft.Press("\x03") + }, "logs", "--watch") + if code != 130 || !strings.Contains(errs, "interrupted") { + t.Errorf("exit %d, stderr %q", code, errs) + } + if _, restored := ft.Left(); !restored { + t.Error("the screen was not restored after Ctrl-C") + } +} + +func TestLogsWatchFullScreenEnterShowsLogsGet(t *testing.T) { + base, reqs := logsTUIServer(t) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, base, func() { + ft.Wait(t, "https://example.com/pricing") + ft.Press("\r") + ft.Wait(t, "spans (ms):") + ft.Press("q") // back to the list, not out + time.Sleep(60 * time.Millisecond) + ft.Press("q") + }, "logs", "--watch", "--poll", "50ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } + var detail bool + for _, r := range reqs() { + detail = detail || r.Path == "/v1/requests/R1" + } + if !detail { + t.Error("the detail did not read GET /v1/requests/R1") + } + s := ft.Screen.String() + plain := ansi.ReplaceAllString(s, "") + for _, want := range []string{"error detail:", "blocked", "upstream"} { + if !strings.Contains(plain, want) { + t.Errorf("detail lacks %q", want) + } + } + if internalNames.MatchString(plain) { + t.Errorf("an engine name reached the detail page:\n%s", plain) + } +} + +func TestLogsWatchFullScreenFirstPollFatalNeverOpensTheView(t *testing.T) { + srv, _ := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + writeProblem(w, 401, "ERR::AUTH::INVALID_KEY", false) + }) + ft := newFakeTerminal(t) + _, _, code := runWatch(t, ft, srv.URL, nil, "logs", "--watch") + if code != 3 { + t.Errorf("exit %d", code) + } + if in, _ := ft.Left(); in { + t.Error("the alternate screen was entered for a fatal error") + } +} + +func TestLogsWatchFullScreenRetriesATransientFirstPoll(t *testing.T) { + var n atomic.Int32 + srv, _ := newRecordingServer(t, func(w http.ResponseWriter, r *http.Request) { + if n.Add(1) <= 2 { + writeProblem(w, 503, "ERR::INTERNAL::UNAVAILABLE", true) + return + } + _, _ = w.Write([]byte(logsList(false, e1))) + }) + ft := newFakeTerminal(t) + _, errs, code := runWatch(t, ft, srv.URL, func() { + ft.Wait(t, "retrying in") + ft.Wait(t, "https://x.test/1") + ft.Press("q") + }, "logs", "--watch", "--poll", "20ms") + if code != 0 { + t.Fatalf("exit %d: %s", code, errs) + } +} diff --git a/internal/cmd/watch.go b/internal/cmd/watch.go new file mode 100644 index 0000000..a0ec0e2 --- /dev/null +++ b/internal/cmd/watch.go @@ -0,0 +1,71 @@ +package cmd + +// Shared pieces of the live views: "spicrawl logs --watch" and +// "spicrawl batch watch" (logs_watch.go, batch_watch.go). The full-screen part +// is internal/tui; this file is what both commands need to start it. + +import ( + "context" + "time" + + tea "charm.land/bubbletea/v2" + + "github.com/OfficialSpicrawl/cli/internal/api" + "github.com/OfficialSpicrawl/cli/internal/tui" +) + +// watchInteractive reports whether a full-screen view may run: stdin and +// stdout are terminals, and none of --json, --quiet, CI or TERM=dumb is set. +// Tests replace it. +var watchInteractive = func() bool { return tui.Interactive(flagJSON || flagQuiet) } + +// tuiOptions are extra Bubble Tea options for the view. Tests use them to give +// it a fake terminal. +var tuiOptions []tea.ProgramOption + +// watchClient is c without the stderr spinner and retry notes, which would be +// drawn across the alternate screen. +func watchClient(c *api.Client) *api.Client { + q := *c + q.Wait, q.Notify = nil, nil + return &q +} + +// rateLimited marks a 429 so the view waits for Retry-After, not for its own +// backoff. +type rateLimited struct { + error + after time.Duration +} + +func (e *rateLimited) Unwrap() error { return e.error } +func (e *rateLimited) RetryAfter() time.Duration { return e.after } + +func watchErr(err error) error { + if p, ok := api.AsProblem(err); ok && p.Status == 429 && p.RetryAfterSeconds > 0 { + return &rateLimited{err, time.Duration(p.RetryAfterSeconds) * time.Second} + } + return err +} + +// watchRetryable reports whether a failed first poll is worth opening the view +// for. A wrong key or id (a 4xx) is not: it fails at once with its exit code, +// like any other command. After the first poll every failure is retried. +func watchRetryable(err error) bool { + if p, ok := api.AsProblem(err); ok && p.Status == 429 { + return true + } + return batchPollTransient(err) +} + +// sleepCtx waits d, or returns ctx's error as soon as ctx ends. +func sleepCtx(ctx context.Context, d time.Duration) error { + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + return nil + } +} diff --git a/internal/cmd/watch_test.go b/internal/cmd/watch_test.go new file mode 100644 index 0000000..4d6be65 --- /dev/null +++ b/internal/cmd/watch_test.go @@ -0,0 +1,70 @@ +package cmd + +// Helpers for the live-view tests: a fake terminal for the full-screen path. + +import ( + "io" + "strings" + "testing" + "time" + + tea "charm.land/bubbletea/v2" +) + +// fakeTerminal forces the full-screen path on and gives the view a screen +// (everything it writes) and a keyboard. keys writes to the keyboard once the +// run has started; closing happens when the run ends. +type fakeTerminal struct { + Screen *lockedBuf + kb *io.PipeWriter +} + +func newFakeTerminal(t *testing.T) *fakeTerminal { + t.Helper() + pr, pw := io.Pipe() + ft := &fakeTerminal{Screen: &lockedBuf{}, kb: pw} + prevI, prevO := watchInteractive, tuiOptions + watchInteractive = func() bool { return true } + tuiOptions = []tea.ProgramOption{tea.WithInput(pr), tea.WithOutput(ft.Screen), tea.WithWindowSize(80, 24)} + t.Cleanup(func() { + watchInteractive, tuiOptions = prevI, prevO + pw.Close() + pr.Close() + }) + return ft +} + +// Press types s on the keyboard. +func (f *fakeTerminal) Press(s string) { _, _ = io.WriteString(f.kb, s) } + +// Wait blocks until the screen has shown want, or the test fails. +func (f *fakeTerminal) Wait(t *testing.T, want string) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for !strings.Contains(f.Screen.String(), want) { + if time.Now().After(deadline) { + // Not Fatalf: this runs on the script's goroutine. Ctrl-C ends the run. + t.Errorf("the screen never showed %q:\n%s", want, ansi.ReplaceAllString(f.Screen.String(), "")) + f.Press("\x03") + return + } + time.Sleep(5 * time.Millisecond) + } +} + +// Left reports whether the view entered the alternate screen and left it again +// with the cursor shown, which is what "restored" means. +func (f *fakeTerminal) Left() (entered, restored bool) { + s := f.Screen.String() + in, out := strings.LastIndex(s, "\x1b[?1049h"), strings.LastIndex(s, "\x1b[?1049l") + return in >= 0, in >= 0 && out > in && strings.LastIndex(s, "\x1b[?25h") > in +} + +// runWatch runs the CLI in human mode on the fake terminal while script drives it. +func runWatch(t *testing.T, ft *fakeTerminal, baseURL string, script func(), args ...string) (stdoutS, stderrS string, code int) { + t.Helper() + if script != nil { + go script() + } + return runHuman(t, baseURL, false, args...) +} diff --git a/internal/tui/header.go b/internal/tui/header.go new file mode 100644 index 0000000..af4d7e9 --- /dev/null +++ b/internal/tui/header.go @@ -0,0 +1,84 @@ +package tui + +import ( + "fmt" + "strings" + "time" + + "charm.land/bubbles/v2/progress" + "charm.land/lipgloss/v2" + "github.com/charmbracelet/x/ansi" + + "github.com/OfficialSpicrawl/cli/internal/output" + "github.com/OfficialSpicrawl/cli/internal/ui" +) + +// BatchSnapshot is the job facts the dashboard header shows. +type BatchSnapshot struct { + ID, Name, Status string + Total, Completed, Succeeded, Failed int64 + Cancelled, Skipped int64 + Credits, Reserved int64 + Started, Finished time.Time // zero when unknown + Error string +} + +// BatchHeader is the three-line block above the item table: +// +// batch 01J9Z… · my-job running +// ████████████░░░░░░░░░░░░░░░░░░░ 38% 380/1000 done · 8 failed +// ETA 2m10s · 4.3 items/s · elapsed 1m28s · credits 1140 of ~3000 reserved +func BatchHeader(s BatchSnapshot) func(width int, now time.Time) []string { + return func(w int, now time.Time) []string { + pct := 0.0 + if s.Total > 0 { + pct = float64(s.Completed) / float64(s.Total) + } + title := "batch " + clean(s.ID) + if s.Name != "" { + title += " · " + clean(s.Name) + } + status := clean(s.Status) + pad := max(1, w-ansi.StringWidth(title)-ansi.StringWidth(status)-2) + l1 := sBold.Render(fit(" "+title+strings.Repeat(" ", pad)+status, w, false)) + + counts := fmt.Sprintf(" %3.0f%% %d/%d done", pct*100, s.Completed, s.Total) + failed := "" + for _, c := range []struct { + n int64 + name string + }{{s.Failed, "failed"}, {s.Cancelled, "cancelled"}, {s.Skipped, "skipped"}} { + if c.n > 0 { + failed += fmt.Sprintf(" · %d %s", c.n, c.name) + } + } + bar := progress.New(progress.WithColors(lipgloss.Color("2")), progress.WithoutPercentage(), progress.WithFillCharacters('█', '░'), + progress.WithWidth(max(10, w-ansi.StringWidth(counts+failed)-2))) + l2 := " " + bar.ViewAs(pct) + counts + if failed != "" { + l2 += lipgloss.NewStyle().Foreground(lipgloss.Color("1")).Render(failed) + } + + eta, rate, elapsed := "—", "—", "—" + if !s.Started.IsZero() { + end := now + if !s.Finished.IsZero() { + end = s.Finished + } + d := max(0, end.Sub(s.Started)) + elapsed = ui.Elapsed(d) + if s.Completed > 0 && d > 0 { + per := float64(s.Completed) / d.Seconds() + rate = fmt.Sprintf("%.1f items/s", per) + if left := s.Total - s.Completed; left > 0 && s.Finished.IsZero() { + eta = ui.Elapsed(time.Duration(float64(left) / per * float64(time.Second))) + } + } + } + l3 := fmt.Sprintf(" ETA %s · %s · elapsed %s · credits %d of ~%d reserved", eta, rate, elapsed, s.Credits, s.Reserved) + if s.Error != "" { + l3 = " " + sRed.Render(output.PublicText(clean(s.Error))) + } + return []string{l1, ansi.Truncate(l2, w, "…"), sDim.Render(fit(l3, w, false))} + } +} diff --git a/internal/tui/header_test.go b/internal/tui/header_test.go new file mode 100644 index 0000000..9629c53 --- /dev/null +++ b/internal/tui/header_test.go @@ -0,0 +1,130 @@ +package tui + +import ( + "context" + "strings" + "testing" + "time" + + tea "charm.land/bubbletea/v2" + "github.com/charmbracelet/x/ansi" + "github.com/charmbracelet/x/exp/golden" +) + +var batchNow = time.Date(2026, 10, 27, 12, 10, 0, 0, time.UTC) + +func batchSnap() BatchSnapshot { + return BatchSnapshot{ + ID: "01J9Z6V0Q8M4K2T7R3N5B1C9XA", Name: "nightly-crawl", Status: "running", + Total: 1000, Completed: 380, Succeeded: 372, Failed: 8, Credits: 1140, Reserved: 3000, + Started: batchNow.Add(-90 * time.Second), + } +} + +func batchCfg(src Source, rows ...Row) Config { + return Config{ + Title: "spicrawl batch watch", Noun: "items", Empty: "no item has finished yet", + Columns: []Column{ + {Title: "SEQ", Width: 6, Right: true}, {Title: "STATUS", Width: 9, Status: true}, + {Title: "HTTP", Width: 4, Right: true}, {Title: "TIER", Width: 7, MinTerm: 60}, + {Title: "TOOK", Width: 6, Right: true}, {Title: "URL"}, + }, + Statuses: []string{"succeeded", "failed", "cancelled", "skipped"}, + BadKey: "f", BadLabel: "failed", BadStatus: func(s string) bool { return s == "failed" }, + Source: src, Poll: time.Second, Now: func() time.Time { return batchNow }, + Initial: Update{Rows: rows, Header: BatchHeader(batchSnap())}, + } +} + +func item(seq, status, tier, took, url string) Row { + return Row{Key: seq, Status: status, Tier: tier, Search: url + " " + seq, Cells: []string{seq, status, "200", tier, took, url}} +} + +func batchRows() []Row { + return []Row{ + item("379", "succeeded", "js", "1.8s", "https://docs.example.org/page/379"), + item("378", "failed", "browser", "12.4s", "https://docs.example.org/page/378"), + item("377", "succeeded", "fetch", "0.4s", "https://docs.example.org/page/377"), + item("376", "cancelled", "—", "—", "https://docs.example.org/page/376"), + } +} + +func newBatch() *Model { + m := New(context.Background(), batchCfg(&fakeSource{}, batchRows()...)) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + return m +} + +func TestSnapshotBatch80x24(t *testing.T) { + golden.RequireEqual(t, text(newBatch())) +} + +func TestSnapshotBatchFailedOnly(t *testing.T) { + m := newBatch() + press(m, "f") + if keys(m) != "378" { + t.Errorf("failed only: %s", keys(m)) + } + golden.RequireEqual(t, text(m)) +} + +func TestBatchHeaderNumbers(t *testing.T) { + out := strings.Join(BatchHeader(batchSnap())(80, batchNow), "\n") + out = ansi.Strip(out) + for _, want := range []string{ + "batch 01J9Z6V0Q8M4K2T7R3N5B1C9XA · nightly-crawl", "running", + " 38% 380/1000 done · 8 failed", + "ETA 2m27s", "4.2 items/s", "elapsed 1m30s", "credits 1140 of ~3000 reserved", + } { + if !strings.Contains(out, want) { + t.Errorf("header lacks %q:\n%s", want, out) + } + } + if !strings.Contains(out, "█") || !strings.Contains(out, "░") { + t.Errorf("no progress bar:\n%s", out) + } +} + +func TestBatchHeaderBeforeAndAfterTheRun(t *testing.T) { + queued := batchSnap() + queued.Status, queued.Started, queued.Completed, queued.Failed = "queued", time.Time{}, 0, 0 + out := ansi.Strip(strings.Join(BatchHeader(queued)(80, batchNow), "\n")) + if !strings.Contains(out, "ETA — · — · elapsed —") || !strings.Contains(out, "0% 0/1000 done") { + t.Errorf("queued job:\n%s", out) + } + + done := batchSnap() + done.Status, done.Completed, done.Finished = "completed", 1000, batchNow.Add(-30*time.Second) + out = ansi.Strip(strings.Join(BatchHeader(done)(80, batchNow), "\n")) + if !strings.Contains(out, "ETA — ") || !strings.Contains(out, "elapsed 1m00s") || !strings.Contains(out, "100%") { + t.Errorf("finished job keeps counting:\n%s", out) + } + + failed := batchSnap() + failed.Error = "ERR::LIMIT::FAILURE_THRESHOLD reached in moli" + out = ansi.Strip(strings.Join(BatchHeader(failed)(80, batchNow), "\n")) + if strings.Contains(strings.ToLower(out), "moli") || !strings.Contains(out, "ERR::LIMIT::FAILURE_THRESHOLD") { + t.Errorf("job error:\n%s", out) + } +} + +func TestBatchHeaderFitsNarrowTerminals(t *testing.T) { + for _, w := range []int{40, 50, 80, 140} { + for i, l := range BatchHeader(batchSnap())(w, batchNow) { + if got := ansi.StringWidth(l); got > w { + t.Errorf("width %d: header line %d is %d wide: %q", w, i, got, ansi.Strip(l)) + } + } + } +} + +func TestBatchTableAndHeaderTogether24Rows(t *testing.T) { + // The header is three lines; the table gets what is left of 24. + m := newBatch() + if got := m.listH(); got != 24-5-3 { + t.Errorf("listH = %d", got) + } + if lines := strings.Split(text(m), "\n"); len(lines) != 24 { + t.Errorf("%d lines", len(lines)) + } +} diff --git a/internal/tui/run.go b/internal/tui/run.go new file mode 100644 index 0000000..a40fb21 --- /dev/null +++ b/internal/tui/run.go @@ -0,0 +1,70 @@ +package tui + +import ( + "context" + "errors" + "fmt" + "os" + "strings" + + tea "charm.land/bubbletea/v2" + "github.com/charmbracelet/colorprofile" + "github.com/mattn/go-isatty" +) + +// Interactive reports whether a full-screen view may run: stdin and stdout +// are terminals, noTUI (--json or --quiet) is off, and neither TERM=dumb nor +// CI says the session is not a person's. NO_COLOR is not here: it takes the +// colour away, not the view. +func Interactive(noTUI bool) bool { + return interactive(isTerminal(os.Stdin), isTerminal(os.Stdout), os.Getenv, noTUI) +} + +func isTerminal(f *os.File) bool { + return isatty.IsTerminal(f.Fd()) || isatty.IsCygwinTerminal(f.Fd()) +} + +func interactive(inTTY, outTTY bool, getenv func(string) string, noTUI bool) bool { + if noTUI || !inTTY || !outTTY || getenv("TERM") == "dumb" { + return false + } + switch strings.ToLower(getenv("CI")) { + case "", "0", "false": + return true + } + return false +} + +// Colour reports whether the view may use colour (NO_COLOR is unset). +func Colour() bool { return os.Getenv("NO_COLOR") == "" } + +// Run shows the view until it ends and returns why. The terminal is restored +// however it ends: a key, Ctrl-C (the context), an error or a panic. A +// cancelled ctx returns ctx.Err(). opts are extra tea.ProgramOptions, for tests. +func Run(ctx context.Context, cfg Config, opts ...tea.ProgramOption) (Reason, error) { + if !Colour() { + opts = append([]tea.ProgramOption{tea.WithColorProfile(colorprofile.ASCII)}, opts...) // text styles stay, colours go + } + ctx, cancel := context.WithCancel(ctx) // stops a poll still in flight when the view ends + defer cancel() + m := New(ctx, cfg) + if _, err := runProgram(ctx, m, opts...); err != nil { + return 0, err + } + return m.Reason(), nil +} + +func runProgram(ctx context.Context, m tea.Model, opts ...tea.ProgramOption) (tea.Model, error) { + final, err := tea.NewProgram(m, append([]tea.ProgramOption{tea.WithContext(ctx)}, opts...)...).Run() + switch { + case err == nil: + return final, nil + case ctx.Err() != nil: + return final, ctx.Err() + case errors.Is(err, tea.ErrInterrupted): + return final, context.Canceled + case errors.Is(err, tea.ErrProgramPanic): + return final, fmt.Errorf("the live view crashed (the terminal was restored): %w", err) + } + return final, err +} diff --git a/internal/tui/run_test.go b/internal/tui/run_test.go new file mode 100644 index 0000000..0505811 --- /dev/null +++ b/internal/tui/run_test.go @@ -0,0 +1,253 @@ +package tui + +import ( + "bytes" + "context" + "errors" + "fmt" + "strings" + "sync" + "testing" + "time" + + tea "charm.land/bubbletea/v2" + "github.com/charmbracelet/x/exp/teatest/v2" +) + +func TestInteractiveMatrix(t *testing.T) { + env := func(kv ...string) func(string) string { + return func(k string) string { + for i := 0; i < len(kv); i += 2 { + if kv[i] == k { + return kv[i+1] + } + } + return "" + } + } + for _, tc := range []struct { + name string + in, out, off bool // stdin tty, stdout tty, --json or --quiet + env func(string) string + want bool + }{ + {"terminal", true, true, false, env("TERM", "xterm-256color"), true}, + {"no TERM at all (Windows)", true, true, false, env(), true}, + {"NO_COLOR keeps the view", true, true, false, env("TERM", "xterm", "NO_COLOR", "1"), true}, + {"CI=false", true, true, false, env("CI", "false"), true}, + {"CI=0", true, true, false, env("CI", "0"), true}, + {"--json or --quiet", true, true, true, env(), false}, + {"stdin piped", false, true, false, env(), false}, + {"stdout piped", true, false, false, env(), false}, + {"TERM=dumb", true, true, false, env("TERM", "dumb"), false}, + {"CI=true", true, true, false, env("CI", "true"), false}, + {"CI=1", true, true, false, env("CI", "1"), false}, + } { + if got := interactive(tc.in, tc.out, tc.env, tc.off); got != tc.want { + t.Errorf("%s: got %v, want %v", tc.name, got, tc.want) + } + } +} + +func TestColourFollowsNoColor(t *testing.T) { + t.Setenv("NO_COLOR", "") + if !Colour() { + t.Error("colour off with NO_COLOR empty") + } + t.Setenv("NO_COLOR", "1") + if Colour() { + t.Error("colour on with NO_COLOR=1") + } +} + +// ---- teatest: the model in a real (virtual) program ---------------------------- + +func waitFor(t *testing.T, tm *teatest.TestModel, s string) { + t.Helper() + teatest.WaitFor(t, tm.Output(), func(b []byte) bool { return bytes.Contains(b, []byte(s)) }, + teatest.WithDuration(3*time.Second), teatest.WithCheckInterval(10*time.Millisecond)) +} + +func TestProgramShowsNewRowsAtTheTopAndQuitsOnQ(t *testing.T) { + var n int + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + n++ + return Update{Rows: []Row{row(fmt.Sprintf("%02d", 20+n), "failed", "js", fmt.Sprintf("https://live.example/%d", n))}}, nil + }} + m := New(context.Background(), logsCfg(src, sample()...)) + tm := teatest.NewTestModel(t, m, teatest.WithInitialTermSize(80, 24)) + + waitFor(t, tm, "https://a.example/09") + waitFor(t, tm, "https://live.example/2") // arrives by polling, with no key pressed + tm.Send(key("q")) + tm.WaitFinished(t, teatest.WithFinalTimeout(3*time.Second)) + + final := tm.FinalModel(t).(*Model) + if final.Reason() != Quit { + t.Errorf("reason %v", final.Reason()) + } + if final.rows[0].Key <= "21" { + t.Errorf("newest row is not on top: %s", final.rows[0].Key) + } +} + +func TestProgramFinishesWhenTheSourceSaysDone(t *testing.T) { + var calls int + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + calls++ + return Update{Done: calls >= 3, Header: func(w int, _ time.Time) []string { return []string{"job 3/3"} }}, nil + }} + tm := teatest.NewTestModel(t, New(context.Background(), logsCfg(src, sample()...)), teatest.WithInitialTermSize(80, 24)) + tm.WaitFinished(t, teatest.WithFinalTimeout(3*time.Second)) + if r := tm.FinalModel(t).(*Model).Reason(); r != Finished { + t.Errorf("reason %v", r) + } +} + +func TestProgramKeepsRetryingAfterErrors(t *testing.T) { + var mu sync.Mutex + calls := 0 + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + mu.Lock() + defer mu.Unlock() + calls++ + if calls <= 2 { + return Update{}, errors.New("503 unavailable") + } + return Update{Rows: []Row{row("30", "success", "js", "https://back.example/")}}, nil + }} + cfg := logsCfg(src, sample()...) + cfg.Poll = time.Millisecond + tm := teatest.NewTestModel(t, New(context.Background(), cfg), teatest.WithInitialTermSize(80, 24)) + waitFor(t, tm, "https://back.example/") + tm.Send(key("q")) + tm.WaitFinished(t, teatest.WithFinalTimeout(3*time.Second)) + mu.Lock() + defer mu.Unlock() + if calls < 3 { + t.Errorf("polled %d times", calls) + } +} + +func TestProgramDetailThenQuit(t *testing.T) { + m := New(context.Background(), logsCfg(&fakeSource{}, sample()...)) + tm := teatest.NewTestModel(t, m, teatest.WithInitialTermSize(80, 24)) + waitFor(t, tm, "https://a.example/09") + tm.Send(key("enter")) + waitFor(t, tm, "detail of 09") + tm.Send(key("esc")) + tm.Send(key("ctrl+c")) + tm.WaitFinished(t, teatest.WithFinalTimeout(3*time.Second)) + if r := tm.FinalModel(t).(*Model).Reason(); r != Interrupt { + t.Errorf("reason %v", r) + } +} + +// ---- Run: the terminal is restored however the view ends ----------------------------------- + +// leaveAlt is the sequence that returns to the normal screen; showCursor is the one that brings the cursor back. +const ( + enterAlt = "\x1b[?1049h" + leaveAlt = "\x1b[?1049l" + showCursor = "\x1b[?25h" +) + +func restored(t *testing.T, out string) { + t.Helper() + if !strings.Contains(out, enterAlt) { + t.Errorf("never entered the alternate screen: %q", out) + } + if !strings.Contains(out, leaveAlt) || strings.LastIndex(out, leaveAlt) < strings.LastIndex(out, enterAlt) { + t.Errorf("did not leave the alternate screen: %q", out) + } + if !strings.Contains(out, showCursor) { + t.Errorf("cursor left hidden: %q", out) + } +} + +type syncBuf struct { + mu sync.Mutex + b bytes.Buffer +} + +func (s *syncBuf) Write(p []byte) (int, error) { s.mu.Lock(); defer s.mu.Unlock(); return s.b.Write(p) } +func (s *syncBuf) String() string { s.mu.Lock(); defer s.mu.Unlock(); return s.b.String() } + +func TestRunRestoresTheScreenWhenTheViewEnds(t *testing.T) { + // Done ends it by itself; the others are driven in the tests below. + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + time.Sleep(150 * time.Millisecond) // let the first frame out + return Update{Done: true}, nil + }} + var out syncBuf + r, err := Run(context.Background(), logsCfg(src, sample()...), tea.WithInput(nil), tea.WithOutput(&out), tea.WithWindowSize(80, 24)) + if err != nil || r != Finished { + t.Fatalf("reason %v, err %v", r, err) + } + restored(t, out.String()) +} + +func TestRunRestoresTheScreenOnContextCancel(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + time.Sleep(150 * time.Millisecond) // let the first frame out + cancel() // Ctrl-C as the process sees it: SIGINT cancels the command's context + time.Sleep(50 * time.Millisecond) + return Update{}, nil + }} + var out syncBuf + _, err := Run(ctx, logsCfg(src, sample()...), tea.WithInput(nil), tea.WithOutput(&out), tea.WithWindowSize(80, 24)) + if !errors.Is(err, context.Canceled) { + t.Fatalf("err %v, want context.Canceled", err) + } + restored(t, out.String()) +} + +// panics blows up on a message, in the program's own loop. +type panics struct{ Model } + +func (p *panics) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + if _, ok := msg.(pollMsg); ok { + panic("boom in Update") + } + _, cmd := p.Model.Update(msg) + return p, cmd // keep this model: returning the embedded one would drop the panic +} + +func (p *panics) Init() tea.Cmd { + return func() tea.Msg { time.Sleep(150 * time.Millisecond); return pollMsg{} } // after the first frame +} + +func TestRunProgramRestoresTheScreenAfterAPanic(t *testing.T) { + m := &panics{Model: *New(context.Background(), logsCfg(&fakeSource{}, sample()...))} + var out syncBuf + _, err := runProgram(context.Background(), m, tea.WithInput(nil), tea.WithOutput(&out), tea.WithWindowSize(80, 24)) + if err == nil || !strings.Contains(err.Error(), "crashed") || !errors.Is(err, tea.ErrProgramPanic) { + t.Fatalf("err %v", err) + } + restored(t, out.String()) +} + +func TestRunWithNoColorKeepsAttributesButNoColours(t *testing.T) { + t.Setenv("NO_COLOR", "1") + src := &fakeSource{poll: func(map[string]struct{}) (Update, error) { + time.Sleep(150 * time.Millisecond) + return Update{Done: true}, nil + }} + var out syncBuf + if _, err := Run(context.Background(), logsCfg(src, sample()...), tea.WithInput(nil), tea.WithOutput(&out), tea.WithWindowSize(80, 24)); err != nil { + t.Fatal(err) + } + s := out.String() + if !strings.Contains(s, "\x1b[1;7m") && !strings.Contains(s, "\x1b[7m") { + t.Errorf("no reverse video: the selected row would be invisible without colour: %q", s) + } + for _, colour := range []string{"\x1b[31m", "\x1b[32m", "\x1b[36m", "38;5;", "38;2;", "\x1b[91m"} { + if strings.Contains(s, colour) { + t.Errorf("NO_COLOR but the output has %q", colour) + } + } + if !strings.Contains(s, "> ") { + t.Error("the selected row has no > marker") + } +} diff --git a/internal/tui/testdata/TestSnapshotBatch80x24.golden b/internal/tui/testdata/TestSnapshotBatch80x24.golden new file mode 100644 index 0000000..b6d2b32 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotBatch80x24.golden @@ -0,0 +1,24 @@ + spicrawl batch watch ● live + batch 01J9Z6V0Q8M4K2T7R3N5B1C9XA · nightly-crawl running + █████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 38% 380/1000 done · 8 failed + ETA 2m27s · 4.2 items/s · elapsed 1m30s · credits 1140 of ~3000 reserved + status: all tier: all failed: off search: - + SEQ STATUS HTTP TIER TOOK URL +> 379 succeeded 200 js 1.8s https://docs.example.org/page/379 + 378 failed 200 browser 12.4s https://docs.example.org/page/378 + 377 succeeded 200 fetch 0.4s https://docs.example.org/page/377 + 376 cancelled 200 — — https://docs.example.org/page/376 + + + + + + + + + + + + + 4 items · updated 12:10:00 + ↑↓ move ⏎ detail / search s status t tier f failed c clear q quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotBatchFailedOnly.golden b/internal/tui/testdata/TestSnapshotBatchFailedOnly.golden new file mode 100644 index 0000000..51f9a81 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotBatchFailedOnly.golden @@ -0,0 +1,24 @@ + spicrawl batch watch ● live + batch 01J9Z6V0Q8M4K2T7R3N5B1C9XA · nightly-crawl running + █████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 38% 380/1000 done · 8 failed + ETA 2m27s · 4.2 items/s · elapsed 1m30s · credits 1140 of ~3000 reserved + status: all tier: all failed: on search: - + SEQ STATUS HTTP TIER TOOK URL +> 378 failed 200 browser 12.4s https://docs.example.org/page/378 + + + + + + + + + + + + + + + + 4 items · 1 shown · updated 12:10:00 + ↑↓ move ⏎ detail / search s status t tier f failed c clear q quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotLogs80x24.golden b/internal/tui/testdata/TestSnapshotLogs80x24.golden new file mode 100644 index 0000000..fa6f9d1 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotLogs80x24.golden @@ -0,0 +1,24 @@ + spicrawl logs ● live + status: all tier: all errors: off search: - + TIME STATUS HTTP TIER MS URL + 10:00:09 success 200 fetch 120 https://a.example/09 +> 10:00:08 failed 200 js 120 https://b.example/08 + 10:00:07 blocked 200 browser 120 https://c.example/07 + 10:00:06 success 200 js 120 https://d.example/06 + 10:00:05 failed 200 fetch 120 https://e.example/05 + + + + + + + + + + + + + + + 5 requests · updated 12:00:00 + ↑↓ move ⏎ detail / search s status t tier e errors c clear q quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotLogsDetail.golden b/internal/tui/testdata/TestSnapshotLogsDetail.golden new file mode 100644 index 0000000..0e74457 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotLogsDetail.golden @@ -0,0 +1,24 @@ + spicrawl logs · 09 ● live +✗ blocked https://a.example/09 + +id: 09 +status: blocked +engine: js +error detail: the site served a bot challenge, a long line that has to wrap +because it is wider than the screen is + + + + + + + + + + + + + + + + 100% · ↑↓ PgUp PgDn scroll · esc back · ctrl+c quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotLogsError.golden b/internal/tui/testdata/TestSnapshotLogsError.golden new file mode 100644 index 0000000..cbb89b5 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotLogsError.golden @@ -0,0 +1,24 @@ + spicrawl logs ! offline + status: all tier: all errors: off search: - + TIME STATUS HTTP TIER MS URL +> 10:00:09 success 200 fetch 120 https://a.example/09 + 10:00:08 failed 200 js 120 https://b.example/08 + 10:00:07 blocked 200 browser 120 https://c.example/07 + 10:00:06 success 200 js 120 https://d.example/06 + 10:00:05 failed 200 fetch 120 https://e.example/05 + + + + + + + + + + + + + + + ! 503 ERR::INTERNAL::UNAVAILABLE · retrying in 1s + ↑↓ move ⏎ detail / search s status t tier e errors c clear q quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotLogsFilteredSearch.golden b/internal/tui/testdata/TestSnapshotLogsFilteredSearch.golden new file mode 100644 index 0000000..c1aa1f6 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotLogsFilteredSearch.golden @@ -0,0 +1,24 @@ + spicrawl logs ● live + search: example/08▏ + TIME STATUS HTTP TIER MS URL +> 10:00:08 failed 200 js 120 https://b.example/08 + + + + + + + + + + + + + + + + + + + 5 requests · 1 shown · updated 12:00:00 + type to filter · ⏎ keep · esc cancel \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotLogsWide.golden b/internal/tui/testdata/TestSnapshotLogsWide.golden new file mode 100644 index 0000000..1dff0bb --- /dev/null +++ b/internal/tui/testdata/TestSnapshotLogsWide.golden @@ -0,0 +1,14 @@ + spicrawl logs ● live + status: all tier: all errors: off search: - + TIME STATUS HTTP TIER CREDITS MS URL +> 10:00:09 success 200 fetch 1 120 https://a.example/09 + 10:00:08 failed 200 js 1 120 https://b.example/08 + 10:00:07 blocked 200 browser 1 120 https://c.example/07 + 10:00:06 success 200 js 1 120 https://d.example/06 + 10:00:05 failed 200 fetch 1 120 https://e.example/05 + + + + + 5 requests · updated 12:00:00 + ↑↓ move ⏎ detail / search s status t tier e errors c clear p pause r refresh q quit \ No newline at end of file diff --git a/internal/tui/testdata/TestSnapshotNoRows.golden b/internal/tui/testdata/TestSnapshotNoRows.golden new file mode 100644 index 0000000..710e6f3 --- /dev/null +++ b/internal/tui/testdata/TestSnapshotNoRows.golden @@ -0,0 +1,24 @@ + spicrawl logs ● live + status: all tier: all errors: off search: - + TIME STATUS HTTP TIER MS URL + no requests yet + + + + + + + + + + + + + + + + + + + 0 requests · updated 12:00:00 + ↑↓ move ⏎ detail / search s status t tier e errors c clear q quit \ No newline at end of file diff --git a/internal/tui/watch.go b/internal/tui/watch.go new file mode 100644 index 0000000..fead000 --- /dev/null +++ b/internal/tui/watch.go @@ -0,0 +1,871 @@ +// Package tui is the CLI's full-screen live views (Bubble Tea + Lip Gloss): +// the request log of "spicrawl logs --watch" and the dashboard of +// "spicrawl batch watch". Both are one Model, a filterable table that polls a +// Source; they differ only in their columns, their Source and an optional +// header block. The package knows nothing about the API: the cmd package +// supplies rows already formatted for a person, so it also owns the rule +// that a customer sees tiers (fetch, js, browser) and never an engine name. +package tui + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + "unicode" + "unicode/utf8" + + "charm.land/bubbles/v2/viewport" + tea "charm.land/bubbletea/v2" + "charm.land/lipgloss/v2" + "github.com/charmbracelet/x/ansi" + + "github.com/OfficialSpicrawl/cli/internal/output" +) + +// Row is one table line. Cells are already formatted and line up with the +// Config's Columns. Key identifies the row across polls (request id, item seq). +type Row struct { + Key string + Cells []string + Status string // colours the status cell and drives the status filter + Tier string // fetch, js or browser; "" when unknown + Search string // what the "/" search matches +} + +// Column is one table column. A Width of 0 takes the space left over. +type Column struct { + Title string + Width int + Right bool + Status bool // coloured by the row's Status + MinTerm int // hidden when the terminal is narrower than this +} + +// Update is what one poll brought. +type Update struct { + Rows []Row // new rows, newest first; with Replace, the whole table + Replace bool + // Header, when set, replaces the block drawn above the table (batch + // progress). It is called on every redraw with the width and the clock. + Header func(width int, now time.Time) []string + Note string // tail of the status bar + Done bool // the thing watched has finished: the view exits +} + +// Source is what a view watches. Poll and Detail are never called at the same +// time as themselves; they may run concurrently with each other. +type Source interface { + // Poll returns what changed. known holds the keys of the newest rows the + // view has, so a source can stop paging once it reaches them. + Poll(ctx context.Context, known map[string]struct{}) (Update, error) + // Detail renders one row's page (the content of "logs get"). It may + // contain SGR colour codes; anything else non-printable is stripped. + Detail(ctx context.Context, key string) (string, error) +} + +// RetryAfter is implemented by an error that says when to try again (a 429). +type RetryAfter interface{ RetryAfter() time.Duration } + +// Config describes one view. +type Config struct { + Title string // "spicrawl logs" + Noun string // "requests", "items" + Empty string // shown while there are no rows + Columns []Column + Statuses []string // what the "s" key cycles through + BadKey string // the key that toggles the problems-only filter ("e", "f") + BadLabel string // "errors", "failed" + BadStatus func(status string) bool + Source Source + Initial Update // the poll the caller already did + InitialErr error // or its failure, which the view retries + Poll time.Duration + MaxRows int + Deadline time.Time // the view exits with Timeout after this; zero = never + Now func() time.Time // the clock; time.Now by default +} + +// Reason says why a view returned. +type Reason int + +const ( + Quit Reason = iota // q or Esc + Interrupt // Ctrl-C + Finished // Update.Done + Timeout // Config.Deadline passed +) + +type mode int + +const ( + modeList mode = iota + modeSearch + modeDetail +) + +const ( + minW, minH = 40, 10 + pollTimeout = 20 * time.Second + maxBackoff = 30 * time.Second + knownKeys = 200 +) + +// The messages the model sends itself. +type ( + tickMsg struct{ gen int } + clockMsg struct{} + pollMsg struct { + u Update + err error + } + detailMsg struct { + key string + text string + err error + } +) + +// Model is the Bubble Tea model of a view. +type Model struct { + cfg Config + ctx context.Context + now func() time.Time + + rows []Row + seen map[string]struct{} + shown []Row + header func(int, time.Time) []string + note string + added int + + f filters + cur cursor + + mode mode + searchWas string + vp viewport.Model + detailKey string + detailText string + detailErr error + loading bool + + w, h int + paused bool + polling bool + gen int + failures int + err error + nextAt time.Time + updated time.Time + + done bool + result Reason +} + +// New builds the model of a view. ctx bounds every poll it starts. +func New(ctx context.Context, cfg Config) *Model { + if cfg.Now == nil { + cfg.Now = time.Now + } + if cfg.Poll <= 0 { + cfg.Poll = 2 * time.Second + } + if cfg.MaxRows <= 0 { + cfg.MaxRows = 5000 + } + if cfg.BadStatus == nil { + cfg.BadStatus = func(string) bool { return false } + } + m := &Model{cfg: cfg, ctx: ctx, now: cfg.Now, seen: map[string]struct{}{}, w: 80, h: 24} + m.vp = viewport.New(viewport.WithWidth(m.w), viewport.WithHeight(m.h-3)) + if cfg.InitialErr != nil { + m.fail(cfg.InitialErr) + } else { + m.apply(cfg.Initial) + m.added, m.updated = 0, m.now() + } + return m +} + +// Reason is why the view returned; valid once the program has ended. +func (m *Model) Reason() Reason { return m.result } + +func (m *Model) Init() tea.Cmd { + return tea.Batch(m.tickAfter(m.delay()), tea.Tick(time.Second, func(time.Time) tea.Msg { return clockMsg{} })) +} + +// ---- polling --------------------------------------------------------------- + +func (m *Model) delay() time.Duration { return Backoff(m.cfg.Poll, m.failures, m.err) } + +// Backoff is how long to wait before the next poll: poll while all is well, +// doubling per failure up to 30 s, or what a 429 asked for (at most a minute). +func Backoff(poll time.Duration, failures int, err error) time.Duration { + if err == nil || failures <= 0 { + return poll + } + d := poll + for i := 0; i < failures && d < maxBackoff; i++ { + d *= 2 + } + d = min(d, maxBackoff) + var ra RetryAfter + if errors.As(err, &ra) && ra.RetryAfter() > 0 { + d = min(ra.RetryAfter(), time.Minute) + } + return d +} + +func (m *Model) tickAfter(d time.Duration) tea.Cmd { + m.gen++ + gen := m.gen + m.nextAt = m.now().Add(d) + return tea.Tick(d, func(time.Time) tea.Msg { return tickMsg{gen} }) +} + +// fetch starts one poll. The caller has checked that none is running. +func (m *Model) fetch() tea.Cmd { + m.polling = true + known := make(map[string]struct{}, knownKeys) + for _, r := range m.rows[:min(len(m.rows), knownKeys)] { + known[r.Key] = struct{}{} + } + src, ctx := m.cfg.Source, m.ctx + return func() tea.Msg { + ctx, cancel := context.WithTimeout(ctx, pollTimeout) + defer cancel() + u, err := src.Poll(ctx, known) + return pollMsg{u, err} + } +} + +func (m *Model) fail(err error) { + m.failures++ + m.err = err +} + +// apply folds one poll into the table. +func (m *Model) apply(u Update) { + if u.Header != nil { + m.header = u.Header + } + m.note = u.Note + m.done = m.done || u.Done + m.added = 0 + switch { + case u.Replace: + seen := make(map[string]struct{}, len(u.Rows)) + m.rows = make([]Row, len(u.Rows)) + for i, r := range u.Rows { + r = cleanRow(r) + if _, ok := m.seen[r.Key]; !ok { + m.added++ + } + seen[r.Key] = struct{}{} + m.rows[i] = r + } + m.seen = seen + default: + fresh := make([]Row, 0, len(u.Rows)) + for _, r := range u.Rows { + if _, ok := m.seen[r.Key]; ok { + continue + } + m.seen[r.Key] = struct{}{} + fresh = append(fresh, cleanRow(r)) + } + m.added = len(fresh) + m.rows = append(fresh, m.rows...) + if len(m.rows) > m.cfg.MaxRows { + for _, r := range m.rows[m.cfg.MaxRows:] { + delete(m.seen, r.Key) + } + m.rows = m.rows[:m.cfg.MaxRows] + } + } + m.refilter() +} + +func (m *Model) refilter() { + m.shown = m.shown[:0] + for _, r := range m.rows { + if m.f.match(r, m.cfg.BadStatus) { + m.shown = append(m.shown, r) + } + } + m.cur.resolve(m.shown, m.listH()) +} + +// finish returns tea.Quit when the view is over and nothing is being read or +// typed, so a finished job never pulls a detail page out from under a reader. +func (m *Model) finish() tea.Cmd { + if m.mode != modeList { + return nil + } + switch { + case m.done: + m.result = Finished + case !m.cfg.Deadline.IsZero() && m.now().After(m.cfg.Deadline): + m.result = Timeout + default: + return nil + } + return tea.Quit +} + +// ---- update ---------------------------------------------------------------- + +func (m *Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { + switch msg := msg.(type) { + case tea.WindowSizeMsg: + m.w, m.h = msg.Width, msg.Height + m.layout() + return m, nil + case clockMsg: + return m, tea.Batch(m.finish(), tea.Tick(time.Second, func(time.Time) tea.Msg { return clockMsg{} })) + case tickMsg: + if msg.gen != m.gen { + return m, nil + } + if m.paused || m.polling { + return m, m.tickAfter(m.cfg.Poll) + } + return m, m.fetch() + case pollMsg: + m.polling = false + if msg.err != nil { + m.fail(msg.err) + } else { + m.failures, m.err = 0, nil + m.updated = m.now() + m.apply(msg.u) + } + return m, tea.Batch(m.finish(), m.tickAfter(m.delay())) + case detailMsg: + if m.mode == modeDetail && msg.key == m.detailKey { + m.loading, m.detailText, m.detailErr = false, msg.text, msg.err + m.layout() + } + return m, nil + case tea.KeyPressMsg: + return m.key(msg) + } + return m, nil +} + +func (m *Model) key(msg tea.KeyPressMsg) (tea.Model, tea.Cmd) { + k := msg.String() + if k == "ctrl+c" { + m.result = Interrupt + return m, tea.Quit + } + switch m.mode { + case modeSearch: + switch k { + case "enter": + m.mode = modeList + return m, m.finish() + case "esc": + m.f.text = m.searchWas + m.mode = modeList + m.refilter() + return m, m.finish() + case "backspace": + if r := []rune(m.f.text); len(r) > 0 { + m.f.text = string(r[:len(r)-1]) + } + default: + m.f.text += msg.Text + } + m.cur = cursor{} + m.refilter() + return m, nil + case modeDetail: + switch k { + case "esc", "q", "enter", "backspace", "left", "h": + m.mode = modeList + return m, m.finish() + } + var cmd tea.Cmd + m.vp, cmd = m.vp.Update(msg) + return m, cmd + } + + switch k { + case "q", "esc": + m.result = Quit + return m, tea.Quit + case "up", "k": + m.cur.move(-1, m.shown, m.listH()) + case "down", "j": + m.cur.move(1, m.shown, m.listH()) + case "pgup", "ctrl+b": + m.cur.move(-m.listH(), m.shown, m.listH()) + case "pgdown", "ctrl+f": + m.cur.move(m.listH(), m.shown, m.listH()) + case "home", "g": + m.cur.move(-len(m.shown), m.shown, m.listH()) + case "end", "G": + m.cur.move(len(m.shown), m.shown, m.listH()) + case "enter": + if len(m.shown) == 0 { + return m, nil + } + m.mode, m.detailKey, m.detailText, m.detailErr, m.loading = modeDetail, m.shown[m.cur.idx].Key, "", nil, true + m.layout() + return m, m.fetchDetail(m.detailKey) + case "/": + m.mode, m.searchWas = modeSearch, m.f.text + case "s": + m.f.status = next(m.f.status, m.cfg.Statuses) + m.cur = cursor{} + m.refilter() + case "t": + m.f.tier = next(m.f.tier, []string{"fetch", "js", "browser"}) + m.cur = cursor{} + m.refilter() + case "c": + m.f = filters{} + m.cur = cursor{} + m.refilter() + case "p", " ": + m.paused = !m.paused + case "r": + if !m.polling { + return m, m.fetch() + } + default: + if k == m.cfg.BadKey && k != "" { + m.f.bad = !m.f.bad + m.cur = cursor{} + m.refilter() + } + } + return m, nil +} + +func (m *Model) fetchDetail(key string) tea.Cmd { + src, ctx := m.cfg.Source, m.ctx + return func() tea.Msg { + ctx, cancel := context.WithTimeout(ctx, pollTimeout) + defer cancel() + text, err := src.Detail(ctx, key) + return detailMsg{key, text, err} + } +} + +// ---- layout ---------------------------------------------------------------- + +func (m *Model) headerLines() []string { + if m.header == nil { + return nil + } + return m.header(m.w, m.now()) +} + +// listH is the number of table rows that fit: the screen less the title, the +// header block, the filter line, the column titles, the status bar and help. +func (m *Model) listH() int { + return max(1, m.h-5-len(m.headerLines())) +} + +func (m *Model) layout() { + m.vp.SetWidth(m.w) + m.vp.SetHeight(max(1, m.h-3)) + m.vp.SetContent(m.detailBody()) + m.cur.resolve(m.shown, m.listH()) +} + +func (m *Model) detailBody() string { + switch { + case m.loading: + return "loading…" + case m.detailErr != nil: + return lipgloss.NewStyle().Width(m.w).Render("! " + clean(output.PublicText(m.detailErr.Error()))) + } + return lipgloss.NewStyle().Width(m.w).Render(Sanitize(m.detailText)) +} + +type colLayout struct { + cols []Column + widths []int + at []int // each shown column's index in Config.Columns, and so in Row.Cells +} + +// columns picks the columns that fit and gives the flexible one the rest. +func (m *Model) columns() colLayout { + var l colLayout + used := 2 // the cursor mark + flex := -1 + for i, c := range m.cfg.Columns { + if m.w < c.MinTerm { + continue + } + if c.Width == 0 { + flex = len(l.cols) + } + l.cols = append(l.cols, c) + l.at = append(l.at, i) + l.widths = append(l.widths, c.Width) + used += c.Width + 1 + } + if flex >= 0 { + l.widths[flex] = max(8, m.w-used+1) + } + return l +} + +func (l colLayout) line(cells []string, style func(i int, cell string) string) string { + var b strings.Builder + for i, c := range l.cols { + cell := "" + if at := l.at[i]; at < len(cells) { + cell = cells[at] + } + b.WriteString(style(i, fit(cell, l.widths[i], c.Right))) + if i < len(l.cols)-1 { + b.WriteByte(' ') + } + } + return b.String() +} + +// ---- view ------------------------------------------------------------------ + +var ( + sBar = lipgloss.NewStyle().Bold(true).Reverse(true) + sSel = lipgloss.NewStyle().Reverse(true) + sDim = lipgloss.NewStyle().Faint(true) + sBold = lipgloss.NewStyle().Bold(true) + sGreen = lipgloss.NewStyle().Foreground(lipgloss.Color("2")) + sRed = lipgloss.NewStyle().Foreground(lipgloss.Color("1")) + sCyan = lipgloss.NewStyle().Foreground(lipgloss.Color("6")) + goodSet = map[string]bool{"success": true, "succeeded": true} + badSet = map[string]bool{"failed": true, "timeout": true, "blocked": true, "rejected": true} +) + +func statusStyle(s string) lipgloss.Style { + switch { + case goodSet[s]: + return sGreen + case badSet[s]: + return sRed + } + return sDim +} + +func (m *Model) View() tea.View { + v := tea.NewView(m.screen()) + v.AltScreen = true + return v +} + +// screen is the whole screen as a styled string, exactly m.h lines. +func (m *Model) screen() string { + if m.w < minW || m.h < minH { + return fmt.Sprintf("terminal too small\n%dx%d, need %dx%d\nq quits", m.w, m.h, minW, minH) + } + if m.mode == modeDetail { + return m.detailView() + } + lines := []string{m.titleBar(m.cfg.Title)} + lines = append(lines, m.headerLines()...) + lines = append(lines, m.filterLine()) + + l := m.columns() + titles := make([]string, len(m.cfg.Columns)) + for i, c := range m.cfg.Columns { + titles[i] = c.Title + } + lines = append(lines, sBold.Render(" "+l.line(titles, func(_ int, c string) string { return c }))) + + h := m.listH() + for i := 0; i < h; i++ { + idx := m.cur.top + i + switch { + case idx < len(m.shown): + lines = append(lines, m.rowLine(l, m.shown[idx], idx == m.cur.idx)) + case i == 0 && len(m.shown) == 0: + lines = append(lines, " "+sDim.Render(m.emptyText())) + default: + lines = append(lines, "") + } + } + lines = append(lines, m.statusBar(), m.helpLine()) + return strings.Join(lines, "\n") +} + +func (m *Model) rowLine(l colLayout, r Row, selected bool) string { + if selected { + return sSel.Render(fit("> "+l.line(r.Cells, func(_ int, c string) string { return c }), m.w, false)) + } + return " " + l.line(r.Cells, func(i int, c string) string { + if l.cols[i].Status { + return statusStyle(r.Status).Render(c) + } + return c + }) +} + +func (m *Model) emptyText() string { + switch { + case m.f.active() && len(m.rows) > 0: + return "no " + m.cfg.Noun + " match the filters (c clears them)" + case m.err != nil: + return "waiting for the API…" + } + return m.cfg.Empty +} + +func (m *Model) titleBar(left string) string { + right := "● live" + if m.paused { + right = "‖ paused" + } + if m.err != nil { + right = "! offline" + } + pad := max(1, m.w-ansi.StringWidth(left)-ansi.StringWidth(right)-2) + return sBar.Render(fit(" "+left+strings.Repeat(" ", pad)+right+" ", m.w, false)) +} + +func (m *Model) filterLine() string { + item := func(name, val string, on bool) string { + if on { + return sDim.Render(name+": ") + sCyan.Render(sBold.Render(val)) + } + return sDim.Render(name + ": " + val) + } + if m.mode == modeSearch { + return ansi.Truncate(" "+sDim.Render("search: ")+sCyan.Render(sBold.Render(clean(m.f.text)))+"▏", m.w, "…") + } + return ansi.Truncate(" "+strings.Join([]string{ + item("status", or(m.f.status, "all"), m.f.status != ""), + item("tier", or(m.f.tier, "all"), m.f.tier != ""), + item(m.cfg.BadLabel, onOff(m.f.bad), m.f.bad), + item("search", or(clean(m.f.text), "-"), m.f.text != ""), + }, " "), m.w, "…") +} + +func (m *Model) statusBar() string { + var s string + if m.err != nil { + retry := max(1, int((m.nextAt.Sub(m.now())+time.Second-1)/time.Second)) // rounded up: "0s" would look like a hang + s = sRed.Render(fit(fmt.Sprintf(" ! %s · retrying in %ds", clean(output.PublicText(m.err.Error())), retry), m.w, false)) + return s + } + parts := []string{fmt.Sprintf("%d %s", len(m.rows), m.cfg.Noun)} + if m.f.active() { + parts = append(parts, fmt.Sprintf("%d shown", len(m.shown))) + } + if m.added > 0 { + parts = append(parts, fmt.Sprintf("+%d new", m.added)) + } + if !m.updated.IsZero() { + parts = append(parts, "updated "+m.updated.Format("15:04:05")) + } + if m.note != "" { + parts = append(parts, clean(m.note)) + } + return sDim.Render(fit(" "+strings.Join(parts, " · "), m.w, false)) +} + +func (m *Model) helpLine() string { + keys := [][2]string{{"↑↓", "move"}, {"⏎", "detail"}, {"/", "search"}, {"s", "status"}, {"t", "tier"}} + if m.cfg.BadKey != "" { + keys = append(keys, [2]string{m.cfg.BadKey, m.cfg.BadLabel}) + } + keys = append(keys, [2]string{"c", "clear"}, [2]string{"p", "pause"}, [2]string{"r", "refresh"}) + if m.mode == modeSearch { + return sDim.Render(" type to filter · ⏎ keep · esc cancel") + } + const quit = "q quit" + s := " " + for _, k := range keys { + item := k[0] + " " + k[1] + " " + if ansi.StringWidth(s+item+quit) > m.w { + break + } + s += item + } + return sDim.Render(fit(s+quit, m.w, false)) +} + +func (m *Model) detailView() string { + title := m.titleBar(m.cfg.Title + " · " + clean(m.detailKey)) + foot := fmt.Sprintf(" %3.0f%% · ↑↓ PgUp PgDn scroll · esc back · ctrl+c quit", m.vp.ScrollPercent()*100) + return title + "\n" + padLines(m.vp.View(), m.vp.Height()) + "\n\n" + sDim.Render(fit(foot, m.w, false)) +} + +// padLines pads a block to exactly h lines so the footer stays on the last line. +func padLines(s string, h int) string { + lines := strings.Split(s, "\n") + for len(lines) < h { + lines = append(lines, "") + } + return strings.Join(lines[:h], "\n") +} + +// ---- helpers --------------------------------------------------------------- + +type filters struct { + status, tier string + bad bool + text string +} + +func (f filters) active() bool { return f.status != "" || f.tier != "" || f.bad || f.text != "" } + +func (f filters) match(r Row, bad func(string) bool) bool { + switch { + case f.status != "" && r.Status != f.status, + f.tier != "" && r.Tier != f.tier, + f.bad && !bad(r.Status), + f.text != "" && !strings.Contains(strings.ToLower(r.Search), strings.ToLower(f.text)): + return false + } + return true +} + +// next cycles "" -> opts[0] -> ... -> "" -> ... +func next(cur string, opts []string) string { + if cur == "" { + return opts[0] + } + for i, o := range opts { + if o == cur && i+1 < len(opts) { + return opts[i+1] + } + } + return "" +} + +// cursor is the selected row. It follows the newest row while it is on the +// first one, and otherwise stays on its row as new ones arrive above it. +type cursor struct { + key string + idx, top int +} + +func (c *cursor) resolve(rows []Row, h int) { + if len(rows) == 0 { + *c = cursor{} + return + } + if c.idx > 0 && c.key != "" { + c.idx = min(c.idx, len(rows)-1) + for i, r := range rows { + if r.Key == c.key { + c.idx = i + break + } + } + } else { + c.idx = 0 + } + c.key = rows[c.idx].Key + if c.idx < c.top { + c.top = c.idx + } + if c.idx >= c.top+h { + c.top = c.idx - h + 1 + } + c.top = max(0, min(c.top, len(rows)-h)) +} + +func (c *cursor) move(d int, rows []Row, h int) { + if len(rows) == 0 { + return + } + c.idx = max(0, min(c.idx+d, len(rows)-1)) + c.key = rows[c.idx].Key + c.resolve(rows, h) +} + +// fit truncates s to w cells (with …) and pads it to exactly w. +func fit(s string, w int, right bool) string { + s = ansi.Truncate(s, w, "…") + if pad := w - ansi.StringWidth(s); pad > 0 { + if right { + return strings.Repeat(" ", pad) + s + } + return s + strings.Repeat(" ", pad) + } + return s +} + +func or(s, def string) string { + if s == "" { + return def + } + return s +} + +func onOff(b bool) string { + if b { + return "on" + } + return "off" +} + +// clean makes a server string safe for one terminal line: control characters, +// which could move the cursor or retitle the window, become spaces. +func clean(s string) string { + return strings.Map(func(r rune) rune { + if unicode.IsControl(r) { + return ' ' + } + return r + }, s) +} + +func cleanRow(r Row) Row { + cells := make([]string, len(r.Cells)) + for i, c := range r.Cells { + cells[i] = clean(c) + } + r.Cells, r.Search, r.Key = cells, clean(r.Search), clean(r.Key) + return r +} + +// Sanitize keeps a multi-line page safe to print: colour codes (SGR) and +// newlines pass, every other control character or escape sequence is dropped. +func Sanitize(s string) string { + var b strings.Builder + for i := 0; i < len(s); { + if s[i] == 0x1b { + if j := sgrEnd(s, i); j > 0 { + b.WriteString(s[i:j]) + i = j + continue + } + i++ + continue + } + r, n := utf8.DecodeRuneInString(s[i:]) + if r == '\n' || r == '\t' || !unicode.IsControl(r) { + b.WriteString(s[i : i+n]) + } + i += n + } + return b.String() +} + +// sgrEnd is the end of the colour sequence ESC [ digits/; m at i, or 0. +func sgrEnd(s string, i int) int { + if i+1 >= len(s) || s[i+1] != '[' { + return 0 + } + for j := i + 2; j < len(s); j++ { + switch c := s[j]; { + case c >= '0' && c <= '9', c == ';': + case c == 'm': + return j + 1 + default: + return 0 + } + } + return 0 +} diff --git a/internal/tui/watch_test.go b/internal/tui/watch_test.go new file mode 100644 index 0000000..b7b9787 --- /dev/null +++ b/internal/tui/watch_test.go @@ -0,0 +1,631 @@ +package tui + +import ( + "context" + "errors" + "fmt" + "strings" + "sync" + "testing" + "time" + + tea "charm.land/bubbletea/v2" + "github.com/charmbracelet/x/ansi" + "github.com/charmbracelet/x/exp/golden" +) + +// ---- fixtures ---------------------------------------------------------------- + +var testCols = []Column{ + {Title: "TIME", Width: 8}, + {Title: "STATUS", Width: 9, Status: true}, + {Title: "HTTP", Width: 4, Right: true}, + {Title: "TIER", Width: 7, MinTerm: 60}, + {Title: "CREDITS", Width: 8, Right: true, MinTerm: 100}, + {Title: "MS", Width: 6, Right: true, MinTerm: 72}, + {Title: "URL"}, +} + +func row(id, status, tier, url string) Row { + return Row{ + Key: id, Status: status, Tier: tier, Search: url + " " + id + " " + status, + Cells: []string{"10:00:" + id, status, "200", tier, "1", "120", url}, + } +} + +// fakeSource is a Source scripted by the test. +type fakeSource struct { + mu sync.Mutex + poll func(known map[string]struct{}) (Update, error) + detail func(key string) (string, error) + polls int +} + +func (f *fakeSource) Poll(_ context.Context, known map[string]struct{}) (Update, error) { + f.mu.Lock() + f.polls++ + f.mu.Unlock() + if f.poll == nil { + return Update{}, nil + } + return f.poll(known) +} + +func (f *fakeSource) Detail(_ context.Context, key string) (string, error) { + if f.detail == nil { + return "detail of " + key, nil + } + return f.detail(key) +} + +func logsCfg(src Source, rows ...Row) Config { + return Config{ + Title: "spicrawl logs", Noun: "requests", Empty: "no requests yet", + Columns: testCols, Statuses: []string{"success", "failed", "blocked"}, + BadKey: "e", BadLabel: "errors", BadStatus: func(s string) bool { return s != "success" }, + Source: src, Initial: Update{Rows: rows}, Poll: 10 * time.Millisecond, + Now: func() time.Time { return time.Date(2026, 10, 27, 12, 0, 0, 0, time.UTC) }, + } +} + +func sample() []Row { + return []Row{ + row("09", "success", "fetch", "https://a.example/09"), + row("08", "failed", "js", "https://b.example/08"), + row("07", "blocked", "browser", "https://c.example/07"), + row("06", "success", "js", "https://d.example/06"), + row("05", "failed", "fetch", "https://e.example/05"), + } +} + +func newLogs(rows ...Row) *Model { + m := New(context.Background(), logsCfg(&fakeSource{}, rows...)) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + return m +} + +func key(s string) tea.KeyPressMsg { + switch s { + case "enter": + return tea.KeyPressMsg{Code: tea.KeyEnter} + case "esc": + return tea.KeyPressMsg{Code: tea.KeyEscape} + case "up": + return tea.KeyPressMsg{Code: tea.KeyUp} + case "down": + return tea.KeyPressMsg{Code: tea.KeyDown} + case "backspace": + return tea.KeyPressMsg{Code: tea.KeyBackspace} + case "ctrl+c": + return tea.KeyPressMsg{Code: 'c', Mod: tea.ModCtrl} + } + return tea.KeyPressMsg{Code: []rune(s)[0], Text: s} +} + +// press sends keys and returns the last command. +func press(m *Model, keys ...string) (cmd tea.Cmd) { + for _, k := range keys { + _, cmd = m.Update(key(k)) + } + return cmd +} + +func keys(m *Model) string { + var ids []string + for _, r := range m.shown { + ids = append(ids, r.Key) + } + return strings.Join(ids, ",") +} + +func text(m *Model) string { return ansi.Strip(m.View().Content) } + +// runCmd runs cmd and returns its message, or nil if it blocks (a timer). +func runCmd(cmd tea.Cmd) tea.Msg { + if cmd == nil { + return nil + } + ch := make(chan tea.Msg, 1) + go func() { ch <- cmd() }() + select { + case m := <-ch: + return m + case <-time.After(30 * time.Millisecond): + return nil + } +} + +// quits reports whether cmd (or a command batched in it) is tea.Quit. +func quits(cmd tea.Cmd) bool { + switch msg := runCmd(cmd).(type) { + case tea.QuitMsg: + return true + case tea.BatchMsg: + for _, c := range msg { + if quits(c) { + return true + } + } + } + return false +} + +// ---- filtering ----------------------------------------------------------------- + +func TestFiltersStatusTierErrorsAndSearch(t *testing.T) { + m := newLogs(sample()...) + if got := keys(m); got != "09,08,07,06,05" { + t.Fatalf("all rows: %s", got) + } + + press(m, "s") // success + if got := keys(m); got != "09,06" { + t.Errorf("status=success: %s", got) + } + press(m, "s") // failed + if got := keys(m); got != "08,05" { + t.Errorf("status=failed: %s", got) + } + press(m, "s", "s") // blocked, then back to all + if got := keys(m); got != "09,08,07,06,05" || m.f.status != "" { + t.Errorf("status cycle did not wrap to all: %q %s", m.f.status, got) + } + + press(m, "t") // fetch + if got := keys(m); got != "09,05" { + t.Errorf("tier=fetch: %s", got) + } + press(m, "t") // js + if got := keys(m); got != "08,06" { + t.Errorf("tier=js: %s", got) + } + press(m, "c") + press(m, "e") + if got := keys(m); got != "08,07,05" { + t.Errorf("errors only: %s", got) + } + press(m, "t", "t", "t") // fetch, js, browser: errors + browser + if got := keys(m); got != "07" { + t.Errorf("errors + tier=browser: %s", got) + } + press(m, "c") + if got := keys(m); got != "09,08,07,06,05" || m.f.active() { + t.Errorf("c did not clear: %s", got) + } + + press(m, "/", "d", ".", "e") + if got := keys(m); got != "06" { + t.Errorf("search d.e (live while typing): %s", got) + } + press(m, "enter") + if m.mode != modeList || m.f.text != "d.e" || keys(m) != "06" { + t.Errorf("enter keeps the search: mode %v text %q rows %s", m.mode, m.f.text, keys(m)) + } + press(m, "/", "backspace", "backspace", "backspace", "esc") // esc restores what was there + if m.f.text != "d.e" || keys(m) != "06" { + t.Errorf("esc did not restore the search: %q %s", m.f.text, keys(m)) + } + press(m, "/", "backspace", "backspace", "backspace", "enter") + if m.f.text != "" || len(m.shown) != 5 { + t.Errorf("emptied search still filters: %q %s", m.f.text, keys(m)) + } + press(m, "/", "Z", "Z", "enter") + if len(m.shown) != 0 || !strings.Contains(text(m), "no requests match the filters (c clears them)") { + t.Errorf("empty filtered list says:\n%s", text(m)) + } +} + +func TestSearchIsCaseInsensitive(t *testing.T) { + m := newLogs(sample()...) + press(m, "/", "B", ".", "E", "X", "enter") + if keys(m) != "08" { + t.Errorf("got %s", keys(m)) + } +} + +// ---- polling and the cursor ------------------------------------------------------ + +func TestNewRowsGoOnTopAndTheCursorStaysOnItsRow(t *testing.T) { + m := newLogs(sample()...) + press(m, "down", "down") // on 07 + m.Update(pollMsg{u: Update{Rows: []Row{row("11", "success", "js", "https://n/11"), row("10", "failed", "js", "https://n/10")}}}) + if got := keys(m); got != "11,10,09,08,07,06,05" { + t.Fatalf("order after the poll: %s", got) + } + if m.shown[m.cur.idx].Key != "07" { + t.Errorf("the cursor moved to %s", m.shown[m.cur.idx].Key) + } + if !strings.Contains(text(m), "+2 new") { + t.Errorf("status bar does not count the new rows:\n%s", text(m)) + } + + // On the first row the view follows the newest. + press(m, "g") + m.Update(pollMsg{u: Update{Rows: []Row{row("12", "success", "js", "https://n/12")}}}) + if m.shown[m.cur.idx].Key != "12" { + t.Errorf("the cursor did not follow the newest row: %s", m.shown[m.cur.idx].Key) + } +} + +func TestPollDoesNotDuplicateKnownRows(t *testing.T) { + m := newLogs(sample()...) + m.Update(pollMsg{u: Update{Rows: []Row{row("10", "success", "js", "u"), row("09", "success", "fetch", "u")}}}) + if got := keys(m); got != "10,09,08,07,06,05" { + t.Errorf("got %s", got) + } +} + +func TestReplaceSwapsTheTable(t *testing.T) { + m := newLogs(sample()...) + m.Update(pollMsg{u: Update{Replace: true, Rows: []Row{row("09", "success", "js", "u"), row("20", "failed", "js", "u")}}}) + if got := keys(m); got != "09,20" || m.added != 1 { + t.Errorf("got %s, %d new", got, m.added) + } +} + +func TestRowCapDropsTheOldest(t *testing.T) { + cfg := logsCfg(&fakeSource{}, sample()...) + cfg.MaxRows = 6 + m := New(context.Background(), cfg) + m.Update(pollMsg{u: Update{Rows: []Row{row("11", "success", "js", "u"), row("10", "success", "js", "u"), row("12", "success", "js", "u")}}}) + if len(m.rows) != 6 || m.rows[5].Key != "07" { + t.Errorf("rows %d, last %s", len(m.rows), m.rows[len(m.rows)-1].Key) + } + if _, ok := m.seen["05"]; ok { + t.Error("a dropped row is still remembered as seen, so it could never come back") + } +} + +func TestPollKnownSnapshotIsTheNewestRows(t *testing.T) { + var got map[string]struct{} + src := &fakeSource{poll: func(known map[string]struct{}) (Update, error) { got = known; return Update{}, nil }} + m := New(context.Background(), logsCfg(src, sample()...)) + runCmd(m.fetch()) + if _, ok := got["09"]; !ok || len(got) != 5 { + t.Errorf("known = %v", got) + } +} + +func TestPausedViewSkipsPolling(t *testing.T) { + src := &fakeSource{} + m := New(context.Background(), logsCfg(src, sample()...)) + press(m, "p") + if !strings.Contains(text(m), "paused") { + t.Errorf("no paused marker:\n%s", text(m)) + } + _, cmd := m.Update(tickMsg{gen: m.gen}) + runCmd(cmd) + if src.polls != 0 { + t.Error("a paused view polled") + } + press(m, "p") + _, cmd = m.Update(tickMsg{gen: m.gen}) + runCmd(cmd) + if src.polls != 1 { + t.Errorf("resumed view polled %d times", src.polls) + } + // A stale tick (an old generation) does nothing. + _, cmd = m.Update(tickMsg{gen: m.gen - 1}) + if cmd != nil { + t.Error("a stale tick scheduled work") + } +} + +// ---- errors ------------------------------------------------------------------------ + +type slow struct{ after time.Duration } + +func (slow) Error() string { return "429 slow down" } +func (s slow) RetryAfter() time.Duration { return s.after } + +func TestPollErrorShowsOnTheStatusBarAndBacksOff(t *testing.T) { + m := newLogs(sample()...) + m.Update(pollMsg{err: errors.New("cannot reach the Spicrawl API: dial tcp: refused")}) + bar := text(m) + if !strings.Contains(bar, "! cannot reach the Spicrawl API: dial tcp: refused · retrying in ") { + t.Errorf("no error on the status bar:\n%s", bar) + } + if !strings.Contains(bar, "! offline") { + t.Errorf("title bar does not say offline:\n%s", bar) + } + if got := len(m.shown); got != 5 { + t.Errorf("an error cleared the table (%d rows)", got) + } + + poll := 2 * time.Second + want := []time.Duration{2, 4, 8, 16, 30, 30} + err := errors.New("x") + for i, w := range want { + if got := Backoff(poll, i, err); i > 0 && got != w*time.Second { + t.Errorf("Backoff(failures=%d) = %s, want %ss", i, got, w*time.Second) + } + } + if got := Backoff(poll, 0, nil); got != poll { + t.Errorf("healthy delay = %s", got) + } + if got := Backoff(poll, 3, slow{7 * time.Second}); got != 7*time.Second { + t.Errorf("Retry-After ignored: %s", got) + } + if got := Backoff(poll, 3, slow{time.Hour}); got != time.Minute { + t.Errorf("Retry-After not capped: %s", got) + } + + // The next good poll clears the error and the offline marker. + m.Update(pollMsg{u: Update{Rows: []Row{row("10", "success", "js", "u")}}}) + if m.err != nil || m.failures != 0 || strings.Contains(text(m), "offline") { + t.Errorf("did not recover: err=%v failures=%d", m.err, m.failures) + } +} + +func TestErrorTextNeverShowsEngineNamesOrControlCodes(t *testing.T) { + m := newLogs(sample()...) + m.Update(pollMsg{err: errors.New("moli died\x1b]0;pwn\x07 in plain-chrome")}) + out := m.View().Content + if strings.Contains(strings.ToLower(out), "moli") || strings.Contains(strings.ToLower(out), "chrome") { + t.Errorf("engine name on screen:\n%s", out) + } + if strings.Contains(out, "\x1b]") || strings.Contains(out, "\x07") { + t.Errorf("control code from an error reached the screen: %q", out) + } +} + +func TestFirstPollFailureIsRetried(t *testing.T) { + cfg := logsCfg(&fakeSource{}) + cfg.InitialErr = errors.New("503") + m := New(context.Background(), cfg) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + if m.err == nil || m.failures != 1 || !strings.Contains(text(m), "waiting for the API") { + t.Errorf("start with an error: err=%v\n%s", m.err, text(m)) + } +} + +// ---- detail --------------------------------------------------------------------------- + +func TestEnterOpensDetailAndEscReturns(t *testing.T) { + src := &fakeSource{detail: func(k string) (string, error) { + return "id: " + k + "\nstatus: ok\x1b]0;pwned\x07\x1b[31m red \x1b[0m\x1b[2J", nil + }} + m := New(context.Background(), logsCfg(src, sample()...)) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + press(m, "down") + cmd := press(m, "enter") + if m.mode != modeDetail || !strings.Contains(text(m), "loading") { + t.Fatalf("no loading state:\n%s", text(m)) + } + msg := runCmd(cmd) + m.Update(msg) + out := m.View().Content + if !strings.Contains(ansi.Strip(out), "id: 08") { + t.Errorf("detail of the selected row missing:\n%s", ansi.Strip(out)) + } + if strings.Contains(out, "\x1b]0") || strings.Contains(out, "\x07") || strings.Contains(out, "\x1b[2J") { + t.Errorf("control codes from the page reached the screen: %q", out) + } + if !strings.Contains(out, "\x1b[31m") { + t.Errorf("colour (SGR) was stripped from the page: %q", out) + } + + // q and Enter go back; neither quits from the detail. + if quits(press(m, "q")) || m.mode != modeList { + t.Errorf("q in detail: mode %v", m.mode) + } + press(m, "enter") + press(m, "esc") + if m.mode != modeList { + t.Error("esc did not go back") + } +} + +func TestDetailErrorIsShownInPlace(t *testing.T) { + src := &fakeSource{detail: func(string) (string, error) { return "", errors.New("404 not found") }} + m := New(context.Background(), logsCfg(src, sample()...)) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + m.Update(runCmd(press(m, "enter"))) + if !strings.Contains(text(m), "! 404 not found") { + t.Errorf("detail error missing:\n%s", text(m)) + } +} + +func TestStaleDetailIsIgnored(t *testing.T) { + m := New(context.Background(), logsCfg(&fakeSource{}, sample()...)) + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) + press(m, "enter") + press(m, "esc") + m.Update(detailMsg{key: "09", text: "late"}) + if m.detailText != "" { + t.Error("a detail that arrived after the user left was kept") + } +} + +// ---- quitting and finishing ------------------------------------------------------------- + +func TestQuitKeys(t *testing.T) { + for _, k := range []string{"q", "esc"} { + m := newLogs(sample()...) + if !quits(press(m, k)) || m.Reason() != Quit { + t.Errorf("%s: reason %v", k, m.Reason()) + } + } + m := newLogs(sample()...) + if !quits(press(m, "ctrl+c")) || m.Reason() != Interrupt { + t.Errorf("ctrl+c: reason %v", m.Reason()) + } + // Ctrl-C quits from the search box and the detail too. + for _, setup := range [][]string{{"/"}, {"enter"}} { + m := newLogs(sample()...) + press(m, setup...) + if !quits(press(m, "ctrl+c")) || m.Reason() != Interrupt { + t.Errorf("ctrl+c after %v: reason %v", setup, m.Reason()) + } + } + // q is a letter in the search box. + m = newLogs(sample()...) + if quits(press(m, "/", "q")) || m.f.text != "q" { + t.Error("q in the search box quit or was lost") + } +} + +func TestDoneQuitsButNeverFromUnderADetailPage(t *testing.T) { + m := newLogs(sample()...) + press(m, "enter") + _, cmd := m.Update(pollMsg{u: Update{Done: true}}) + if quits(cmd) { + t.Fatal("the view closed under a reader") + } + if !quits(press(m, "esc")) || m.Reason() != Finished { + t.Errorf("leaving the page did not finish the view: reason %v", m.Reason()) + } + + m = newLogs(sample()...) + _, cmd = m.Update(pollMsg{u: Update{Done: true}}) + if !quits(cmd) || m.Reason() != Finished { + t.Errorf("done in the list: reason %v", m.Reason()) + } + + // Typing a search holds the exit as well. + m = newLogs(sample()...) + press(m, "/") + _, cmd = m.Update(pollMsg{u: Update{Done: true}}) + if quits(cmd) { + t.Fatal("the view closed while typing") + } + if !quits(press(m, "enter")) { + t.Error("did not finish after the search was kept") + } +} + +func TestDeadlineEndsTheViewWithTimeout(t *testing.T) { + now := time.Date(2026, 10, 27, 12, 0, 0, 0, time.UTC) + cfg := logsCfg(&fakeSource{}, sample()...) + cfg.Now = func() time.Time { return now } + cfg.Deadline = now.Add(time.Minute) + m := New(context.Background(), cfg) + if quits(m.finish()) { + t.Fatal("ended before the deadline") + } + now = now.Add(2 * time.Minute) + if !quits(m.finish()) || m.Reason() != Timeout { + t.Errorf("reason %v", m.Reason()) + } +} + +// ---- layout ---------------------------------------------------------------------------------- + +func TestViewIsExactlyTheTerminalSize(t *testing.T) { + for _, sz := range [][2]int{{80, 24}, {120, 40}, {40, 10}, {64, 12}, {200, 50}} { + m := New(context.Background(), logsCfg(&fakeSource{}, sample()...)) + m.Update(tea.WindowSizeMsg{Width: sz[0], Height: sz[1]}) + lines := strings.Split(text(m), "\n") + if len(lines) != sz[1] { + t.Errorf("%dx%d: %d lines", sz[0], sz[1], len(lines)) + } + for i, l := range lines { + if w := ansi.StringWidth(l); w > sz[0] { + t.Errorf("%dx%d: line %d is %d wide: %q", sz[0], sz[1], i, w, l) + } + } + } +} + +func TestTooSmallTerminalSaysSo(t *testing.T) { + m := newLogs(sample()...) + m.Update(tea.WindowSizeMsg{Width: 30, Height: 6}) + if out := text(m); !strings.Contains(out, "terminal too small\n30x6, need 40x10\nq quits") { + t.Errorf("got %q", out) + } + if !quits(press(m, "q")) { + t.Error("q must still quit a too-small terminal") + } + m.Update(tea.WindowSizeMsg{Width: 80, Height: 24}) // growing it again brings the table back + if !strings.Contains(text(m), "https://a.example/09") { + t.Errorf("did not recover after resize:\n%s", text(m)) + } +} + +func TestResizeKeepsTheCursorVisible(t *testing.T) { + var rows []Row + for i := 40; i > 0; i-- { + rows = append(rows, row(fmt.Sprintf("%02d", i), "success", "js", fmt.Sprintf("https://x/%02d", i))) + } + m := newLogs(rows...) + press(m, "G") // the last row + m.Update(tea.WindowSizeMsg{Width: 100, Height: 12}) + if !strings.Contains(text(m), "https://x/01") { + t.Errorf("selected row scrolled out of sight after a resize:\n%s", text(m)) + } + m.Update(tea.WindowSizeMsg{Width: 50, Height: 30}) + if !strings.Contains(text(m), "> ") || !strings.Contains(text(m), "x/01") { + t.Errorf("selected row lost after a second resize:\n%s", text(m)) + } +} + +func TestNarrowTerminalDropsColumnsNotTheURL(t *testing.T) { + m := newLogs(sample()...) + m.Update(tea.WindowSizeMsg{Width: 50, Height: 24}) + out := text(m) + if strings.Contains(out, "TIER") || strings.Contains(out, "CREDITS") { + t.Errorf("columns that do not fit are still shown:\n%s", out) + } + if !strings.Contains(out, "https://a.example/09") { + t.Errorf("the URL does not fit at 50 columns:\n%s", out) + } + m.Update(tea.WindowSizeMsg{Width: 120, Height: 24}) + if !strings.Contains(text(m), "CREDITS") { + t.Errorf("wide terminal lacks the credits column:\n%s", text(m)) + } +} + +func TestCellsFollowTheirColumnsWhenOneIsHidden(t *testing.T) { + m := newLogs(sample()...) // 80 wide: CREDITS (MinTerm 100) is hidden + line := strings.Split(text(m), "\n")[3] + if !strings.Contains(line, "fetch") || !strings.Contains(line, "120") || !strings.Contains(line, "https://a.example/09") || !strings.HasPrefix(line, "> ") { + t.Errorf("cells shifted into the wrong columns: %q", line) + } +} + +func TestControlCharactersInCellsAreNeutralised(t *testing.T) { + r := row("01", "success", "js", "https://evil\x1b[2J\x1b]0;x\x07.example/\n") + m := newLogs(r) + out := m.View().Content + if strings.Contains(out, "\x1b[2J") || strings.Contains(out, "\x1b]0") || strings.Contains(out, "\x07") { + t.Errorf("a control sequence in a row reached the screen: %q", out) + } +} + +// ---- snapshots (golden text of the screen) --------------------------------------------------- + +func TestSnapshotLogs80x24(t *testing.T) { + m := newLogs(sample()...) + press(m, "down") + golden.RequireEqual(t, text(m)) +} + +func TestSnapshotLogsFilteredSearch(t *testing.T) { + m := newLogs(sample()...) + press(m, "e", "/", "e", "x", "a", "m", "p", "l", "e", "/", "0", "8") + golden.RequireEqual(t, text(m)) +} + +func TestSnapshotLogsError(t *testing.T) { + m := newLogs(sample()...) + m.Update(pollMsg{err: errors.New("503 ERR::INTERNAL::UNAVAILABLE")}) + golden.RequireEqual(t, text(m)) +} + +func TestSnapshotLogsWide(t *testing.T) { + m := newLogs(sample()...) + m.Update(tea.WindowSizeMsg{Width: 120, Height: 14}) + golden.RequireEqual(t, text(m)) +} + +func TestSnapshotLogsDetail(t *testing.T) { + m := newLogs(sample()...) + press(m, "enter") + m.Update(detailMsg{key: "09", text: "✗ blocked https://a.example/09\n\nid: 09\nstatus: blocked\nengine: js\nerror detail: the site served a bot challenge, a long line that has to wrap because it is wider than the screen is"}) + golden.RequireEqual(t, text(m)) +} + +func TestSnapshotNoRows(t *testing.T) { + m := newLogs() + golden.RequireEqual(t, text(m)) +} From 832995f6deb58a9007fb8d0da6744959df26c992 Mon Sep 17 00:00:00 2001 From: Dibbayajyoti Date: Tue, 27 Oct 2026 12:45:10 +0530 Subject: [PATCH 2/2] changelog: binary size of the live views --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e741079..2c62e4e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,7 @@ Bug fixes from a review of 0.1.3, and live progress on a terminal. No new comman - **usage summary** shows the enforced monthly allowance. **status** fails when the key check cannot reach the API. - **mcp install / init** back up the client config to `.bak`, write through symlinks, refuse to replace a customised `spicrawl` entry without `--force`, and accept VS Code JSONC/BOM `mcp.json`. - **Docs:** the embedded skill, `openapi.yaml` and schemas are synced with the API (cache hits are billed, `impersonate` defaults to true, batch honours `--format`). -- **Live views.** `spicrawl logs --watch` is a live, scrollable request log (new rows on top, filter by status, tier and errors, search, Enter for the `logs get` page) and `spicrawl batch watch ` is a dashboard for a batch job (progress bar, done/failed/total, ETA, credits, a per-item table, failed-only filter, item detail). Both are full-screen only when stdin and stdout are terminals and none of `--json`, `-q`, `CI` or `TERM=dumb` is set; otherwise `logs --watch` streams new rows as plain lines (JSONL with `--json`) and `batch watch` is `batch wait`. `batch watch` ends with the final job and `batch wait`'s exit codes (0, 4, 11); `NO_COLOR` is honoured; the terminal is restored on q, Ctrl-C (exit 130), errors and panics. New dependencies, all Charm: `charm.land/bubbletea/v2`, `charm.land/lipgloss/v2`, `charm.land/bubbles/v2`. The release binary grows by about 2.1 MB (10.0 to 12.2 MB, stripped). +- **Live views.** `spicrawl logs --watch` is a live, scrollable request log (new rows on top, filter by status, tier and errors, search, Enter for the `logs get` page) and `spicrawl batch watch ` is a dashboard for a batch job (progress bar, done/failed/total, ETA, credits, a per-item table, failed-only filter, item detail). Both are full-screen only when stdin and stdout are terminals and none of `--json`, `-q`, `CI` or `TERM=dumb` is set; otherwise `logs --watch` streams new rows as plain lines (JSONL with `--json`) and `batch watch` is `batch wait`. `batch watch` ends with the final job and `batch wait`'s exit codes (0, 4, 11); `NO_COLOR` is honoured; the terminal is restored on q, Ctrl-C (exit 130), errors and panics. New dependencies, all Charm: `charm.land/bubbletea/v2`, `charm.land/lipgloss/v2`, `charm.land/bubbles/v2`. The release binary grows by about 2 MB (10.2 to 12.3 MB for linux/amd64, stripped; about +20%). - **No engine or browser names.** Customers see three rendering tiers, `fetch`, `js` and `browser`, never an engine. `sessions create --engine` and `browser url --engine` take a tier; the old engine names still work (sent as typed) with a deprecation notice on stderr that does not repeat them. On `scrape` and `batch submit`, `--engine` and the display flag next to it, which the API already ignores, are hidden from `--help` but still accepted. Human output (tables, `--meta`, warnings, errors, `logs get`) shows a tier for any engine name the API sends; `--json` and `--jsonl` stay exactly as the API sent them. The help, `openapi.yaml`, the schemas, the embedded skill and the README use tier names. - **install.sh** accepts `SPICRAWL_VERSION=0.1.3` as well as `v0.1.3` and explains a failed download. The npm launcher forwards Ctrl-C to the binary.