Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.11", "3.12", "3.13", "3.14"]
runs-on: ${{ matrix.os }}
permissions:
contents: read
Expand Down Expand Up @@ -70,7 +70,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.11", "3.12", "3.13", "3.14"]
runs-on: windows-latest
permissions:
contents: read
Expand Down Expand Up @@ -118,7 +118,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python: ["3.10", "3.14"]
python: ["3.11", "3.14"]
runs-on: ${{ matrix.os }}
permissions:
contents: read
Expand Down
2 changes: 1 addition & 1 deletion .serena/memories/core.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# qsv-client — core

Python library that runs the qsv CSV toolkit binary as a subprocess (qsv not bundled). Zero runtime deps. Sync `Qsv` + asyncio `AsyncQsv` with identical API.
Python library that runs the qsv CSV toolkit binary as a subprocess (qsv not bundled). Zero runtime deps (daemon.toml is read with stdlib `tomllib`). Sync `Qsv` + asyncio `AsyncQsv` with identical API.

Related: stack/versions `mem:tech_stack`; dev/test/run commands `mem:suggested_commands`; code style & sync/async mirroring rules `mem:conventions`; done-checklist (ruff/mypy/pytest as CI runs them) `mem:task_completion`.

Expand Down
4 changes: 2 additions & 2 deletions .serena/memories/tech_stack.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Tech stack

- Python >=3.10 (ruff `target-version = py310`; CI matrix 3.10–3.14 on ubuntu + macos). Use `from __future__ import annotations`; no 3.11+ only stdlib features.
- No runtime dependencies — keep it that way (stdlib `subprocess`/`asyncio` only).
- Python >=3.11 (ruff `target-version = py311`; CI matrix 3.11–3.14 on ubuntu + macos). Use `from __future__ import annotations`; no 3.12+ only stdlib features.
- No runtime dependencies — keep it that way (stdlib `subprocess`/`asyncio` only). daemon.toml is read with stdlib `tomllib`.
- Package/env manager: uv (`uv.lock` committed); dev deps in `[dependency-groups].dev`: pytest, pytest-asyncio, mypy, ruff.
- Build: hatchling, src layout (`src/qsv_client`). License AGPL-3.0-or-later.
- pytest: `asyncio_mode = "auto"` (async tests need no marker).
Expand Down
33 changes: 30 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,23 @@ All notable changes to this project are documented here. The format follows

## [Unreleased]

## [0.6.0] - 2026-10-26

This release accompanies qsv 27.0.0, the first qsv whose Data Schematics connect to
`qsv-clientd`. `qsv-clientd viz <URL>` turns a download link or a CKAN page into a live
schematic in one step, and `qsv-clientd` remembers where the data came from, so Fetch latest
gets the current version.

### Added

- **`qsv-clientd viz <URL>`** ([#34](https://github.com/dathere/qsv-client/pull/34)) downloads the data first, then renders, registers and opens
the schematic in one step. The URL can be an http(s) download link, `ckan://<id>`, or a
CKAN resource page copied from the browser. It is also where Fetch latest gets new data.
The data is saved beside the schematic (`--save-as` to choose), readable only by you
since it may be private, never replacing an existing file. `-o` now defaults to
the data's name with `.html`. `--no-open` skips the browser.
since it may be private, never replacing an existing file. A qsv with `get` also keeps a
copy in its own cache (`~/.qsv-cache` by default, or `QSV_CACHE_DIR`), which follows
your umask. `-o` now defaults to the data's name with `.html`. `--no-open` skips the
browser.
- **CKAN pages as sources.** ([#34](https://github.com/dathere/qsv-client/pull/34)) A CKAN resource page (`<site>/dataset/<name>/resource/<id>`) is
fetched as `ckan://<id>` from that site. A CKAN dataset page with exactly one CSV or TSV
resource, found with `package_show`, is fetched as that resource; with several, the
Expand All @@ -23,6 +32,23 @@ All notable changes to this project are documented here. The format follows
URL) for `ckan://` sources.
- `register --schematic` takes the source URL (and its CKAN site) from a schematic that
records one, as qsv records it for data downloaded with `qsv get`. ([#34](https://github.com/dathere/qsv-client/pull/34))
- With a qsv older than 27.0.0, whose schematics can't connect, `qsv-clientd serve` logs a
warning and the tray shows a notification once, instead of leaving schematics silently
offline. Everything else still works with qsv 25.0.0 or later.

### Changed

- **Python 3.11 or later is required**; 3.10 is no longer supported. The `[tray]` extra now
needs Pillow 13.0.0, which fixes its known vulnerabilities and requires Python 3.11.
(The tray opens only its own bundled icon, so they weren't reachable through it.)
- **The daemon's config is now `daemon.toml`**, not `daemon.json`, so it can carry comments.
The tray's Open config… creates it with every setting commented out, each with its
default and the values it takes; nothing changes until you uncomment a line. A
`daemon.json` from an earlier release is no longer read: move its settings across
(`serve --config` with an old JSON file says so). It is read with the standard
library's `tomllib`, so qsv-client still has no runtime dependencies.
- The docs now name the release that brings live Data Schematics: qsv 27.0.0 or later
(before, "a qsv newer than 26.0.0"). Other uses still need only qsv 25.0.0.

### Fixed

Expand Down Expand Up @@ -278,7 +304,8 @@ First release. ([#1](https://github.com/dathere/qsv-client/pull/1))
environment variables, so keys never appear on the command line.
- Type hints (`py.typed`), no runtime dependencies, Python 3.10+.

[Unreleased]: https://github.com/dathere/qsv-client/compare/0.5.0...HEAD
[Unreleased]: https://github.com/dathere/qsv-client/compare/0.6.0...HEAD
[0.6.0]: https://github.com/dathere/qsv-client/compare/0.5.0...0.6.0
[0.5.0]: https://github.com/dathere/qsv-client/compare/0.4.0...0.5.0
[0.4.0]: https://github.com/dathere/qsv-client/compare/0.3.0...0.4.0
[0.3.0]: https://github.com/dathere/qsv-client/compare/0.2.0...0.3.0
Expand Down
9 changes: 5 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Python library that runs the [qsv](https://github.com/dathere/qsv) CSV toolkit as a subprocess
(qsv itself is not bundled). Sync `Qsv` and asyncio `AsyncQsv` clients with the same API.
No runtime dependencies. Python 3.10+.
No runtime dependencies (the daemon's config is read with the stdlib's `tomllib`). Python 3.11+.

## Commands

Expand All @@ -14,9 +14,9 @@ uv run ruff format . && uv run ruff check . && uv run mypy # must be
uv run pytest --cov # with branch coverage (config in pyproject.toml)
```

CI runs lint, format check, and mypy, plus tests on ubuntu/macos x Python 3.10–3.14 against
CI runs lint, format check, and mypy, plus tests on ubuntu/macos x Python 3.11–3.14 against
qsv, qsvlite, and qsvdp (Linux only).
On Windows (Python 3.10–3.14) it runs the full suite against qsv, plus the integration tests
On Windows (Python 3.11–3.14) it runs the full suite against qsv, plus the integration tests
against qsvlite. Windows teardown uses Job Objects via ctypes (`_ProcessTree` in `_process.py`);
type-check it with `uv run mypy --platform win32` (the lint job runs both).
Coverage from every test job is merged on Codecov (`codecov.yml`): the total may not drop
Expand Down Expand Up @@ -72,7 +72,8 @@ below the base commit's, and new or changed lines need 100% coverage.

- `Qsv` and `AsyncQsv` are mirrored by hand. Apply every change to both. In `AsyncQsv`,
`capabilities` and `version` are coroutines, not properties.
- Keep it dependency-free and 3.10-compatible (`from __future__ import annotations`).
- Keep it dependency-free and 3.11-compatible
(`from __future__ import annotations`).
- mypy is strict and covers `tests/` too. Ruff line length is 100.
- New error kind: add a subclass in `errors.py`, map it in `_KIND_TO_CLASS` (and
`_EXIT_CODE_TO_CLASS` if it has its own exit code), export it from `__init__.py`, and update
Expand Down
18 changes: 13 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,15 @@ subprocess pitfalls:
`qsv --capabilities`.
- **Secrets stay off the command line.** LLM settings for `describegpt` are passed as
environment variables, so an API key never shows up in `ps` output.
- **Sync and asyncio** clients with the same API. No runtime dependencies. Python 3.10+ on
- **Sync and asyncio** clients with the same API. No runtime dependencies. Python 3.11+ on
Linux, macOS and Windows.
- **A local daemon, `qsv-clientd`,** that runs qsv for your own processes and lets a Data
Schematic from `qsv viz smart` work with its live data, with an optional system tray to
run and control it. See [The qsv-clientd daemon](#the-qsv-clientd-daemon).

qsv itself is not bundled, and **qsv 25.0.0 or later** is required. Install it from the
[qsv releases](https://github.com/dathere/qsv/releases) or a
qsv itself is not bundled, and **qsv 25.0.0 or later** is required; live Data Schematics
need **qsv 27.0.0 or later** (with an older qsv, `qsv-clientd serve` and the tray say so).
Install it from the [qsv releases](https://github.com/dathere/qsv/releases) or a
[package manager](https://github.com/dathere/qsv#installation-options).

## Install
Expand Down Expand Up @@ -128,17 +129,24 @@ client per call costs nothing extra after the first.

`qsv-clientd` (installed with the package) is a per-user daemon that runs qsv for local
processes, through `DaemonClient` / `AsyncDaemonClient` (the same `run()` as `Qsv`), and for
open **Data Schematics** from `qsv viz smart` (a qsv newer than 26.0.0). Once you approve a
open **Data Schematics** from `qsv viz smart` (qsv 27.0.0 or later). Once you approve a
schematic with a pairing code, it can drill into the rows behind a chart, refresh its
statistics, re-render itself, run read-only queries and fetch the latest data. Large results
are paged from the daemon, so every row is reachable.

```bash
qsv-clientd install # or: qsv-clientd serve
qsv-clientd register data.csv --schematic data.html
qsv-clientd viz data.csv # render, register and open data.html
qsv-clientd viz https://data.example.gov/dataset/housing/resource/<resource id>
qsv-clientd approve K7Q-4MZ # the code the schematic shows
```

`viz` with a URL (an http(s) download link, a CKAN resource page, or `ckan://<id>`)
downloads the data first and remembers the URL, so Fetch latest gets the current data. An
existing schematic can be registered with `qsv-clientd register data.csv --schematic
data.html`. Settings go in `daemon.toml` (where it lives is in the doc); the tray's Open
config… creates it with every setting documented and commented out.

```python
from qsv_client.daemon import DaemonClient

Expand Down
19 changes: 11 additions & 8 deletions docs/DAEMON.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ callers:
the latest data from its source URL or PID. Without the daemon the schematic works exactly
as before.

It needs nothing beyond the Python standard library, and runs on Python 3.10+ like the rest
It needs nothing beyond the Python standard library, and runs on Python 3.11+ like the rest
of the package.

## Running it
Expand All @@ -39,17 +39,19 @@ closes it, and `client()` returns a `DaemonClient` for it. A host starts once: t
stop it and build a new `Daemon` and `DaemonHost`.

Services don't read your shell profile, so `install` finds qsv the way the daemon would
(`binary` in `daemon.json`, then `$QSV_BIN`, then your PATH) and writes that absolute path
(`binary` in `daemon.toml`, then `$QSV_BIN`, then your PATH) and writes that absolute path
into the service as `QSV_BIN`. It fails if no qsv is found. Run `install` again after
moving qsv. Other `QSV_*` settings in your shell are not copied, because the service file
is world-readable and they can hold secrets such as `QSV_LLM_APIKEY`. Set those in the
service's environment yourself if runs through the daemon need them. A Windows scheduled
task gets your user environment variables, as set with `setx`, not those of the current
shell.

Configuration is optional: `daemon.json` in the config directory
Configuration is optional: `daemon.toml` in the config directory
(`~/.config/qsv-client/` on Linux, `~/Library/Application Support/qsv-client/` on macOS,
`%APPDATA%\qsv-client\` on Windows). Keys: `binary`, `port` (default `14470`; `0` turns the
`%APPDATA%\qsv-client\` on Windows), one `key = value` per line, with `#` comments. The
tray's **Open config…** creates it with every setting commented out, each with its default
and the values it takes. Keys: `binary`, `port` (default `14470`; `0` turns the
browser listener off), `socket`, `max_jobs`, `max_timeout`, `inherit_env`, `kill_grace`,
`browser`, `browser_origins` (default `["null"]`), `browser_timeout`, `browser_max_output`,
`browser_max_result`, `browser_max_render`, `token_ttl`, `allow_fetch`, `fetch_max_bytes`
Expand All @@ -71,7 +73,7 @@ when a schematic is waiting to be paired, red on an error. The menu shows the st
version, running jobs and datasets, and lets you start, stop and restart the daemon.

- **Hosted:** when no daemon is running, the tray runs one itself, on a background thread,
with your `daemon.json`. Quitting the tray stops it. Restart reads `daemon.json` again, and
with your `daemon.toml`. Quitting the tray stops it. Restart reads `daemon.toml` again, and
unpairs every open schematic, since tokens are kept in memory.
- **Attached:** when a daemon is already running (the service `install` set up, or
`qsv-clientd serve`), the tray only shows it. Quitting leaves it running. If that daemon
Expand Down Expand Up @@ -112,8 +114,9 @@ it went. A fetch refused because the new data lacks some of the dataset's column
**Fetch latest data, accepting missing columns** next. **Revoke all tokens ▸ Yes** unpairs
every open schematic.

**Files.** **Open config…** opens the default `daemon.json` in the config directory (not a
`--config` file a foreground `serve` may use), creating it as `{}` if it's missing, and says
**Files.** **Open config…** opens the default `daemon.toml` in the config directory (not a
`--config` file a foreground `serve` may use), creating it if it's missing with every setting
commented out, so nothing changes until you uncomment one, and says
how changes apply: **Restart** when the tray hosts the daemon, otherwise a restart of the
service or of `serve`. **Open log…** opens the tray's log, which a daemon it hosts writes to
as well, or the launchd service's `~/Library/Logs/qsv-clientd.log`. For a systemd service the
Expand Down Expand Up @@ -309,7 +312,7 @@ size or modification time, as with an edit or a fetch with `--allow-replace`. A
a registered dataset by it:

```html
<script type="application/json" id="qsv-schematic-source">{"schema":1,"qsv_version":"26.0.0",
<script type="application/json" id="qsv-schematic-source">{"schema":1,"qsv_version":"27.0.0",
"schematic_uuid":"…","fingerprint":{"algo":"blake3","digest":"…","bytes":123},
"rows":4,"viz_args":["--dict-info"],"redacted":[],"pid":"https://doi.org/…"}</script>
```
Expand Down
10 changes: 5 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ build-backend = "hatchling.build"

[project]
name = "qsv-client"
version = "0.5.0"
description = "Run the qsv CSV toolkit (25.0.0+) from Python: timeouts that kill the whole process tree, typed errors, capability detection, and a local daemon (with an optional system tray) for live Data Schematics."
version = "0.6.0"
description = "Run the qsv CSV toolkit (25.0.0+) from Python: timeouts that kill the whole process tree, typed errors, capability detection, and a local daemon (with an optional system tray) for live Data Schematics (qsv 27.0.0+)."
readme = "README.md"
license = "AGPL-3.0-or-later"
license-files = ["LICENSE"]
requires-python = ">=3.10"
requires-python = ">=3.11"
authors = [{ name = "datHere" }]
keywords = ["qsv", "csv", "data-wrangling", "subprocess"]
classifiers = [
Expand All @@ -27,7 +27,7 @@ dependencies = []

[project.optional-dependencies]
# the system tray for qsv-clientd: `pip install 'qsv-client[tray]'`
tray = ["pystray>=0.19.5,<0.20", "Pillow>=10"]
tray = ["pystray>=0.19.5,<0.20", "Pillow>=13"]

[project.scripts]
qsv-clientd = "qsv_client.daemon.__main__:main"
Expand Down Expand Up @@ -87,7 +87,7 @@ ignore_missing_imports = true

[tool.ruff]
line-length = 100
target-version = "py310"
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]
2 changes: 1 addition & 1 deletion src/qsv_client/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
QsvWriteError,
)

__version__ = "0.5.0"
__version__ = "0.6.0"

__all__ = [
"DEFAULT_BINARIES",
Expand Down
2 changes: 1 addition & 1 deletion src/qsv_client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -632,7 +632,7 @@ async def _probe(self, timeout: float | None = None) -> Capabilities:
lock = _async_probe_lock(key)
try:
await asyncio.wait_for(lock.acquire(), limit)
except asyncio.TimeoutError:
except TimeoutError:
raise _probe_wait_timeout(self.binary, limit) from None
try:
cached = _caps_cache.get(key)
Expand Down
4 changes: 2 additions & 2 deletions src/qsv_client/daemon/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ def _parser() -> argparse.ArgumentParser:
sub = p.add_subparsers(dest="cmd", required=True)

s = sub.add_parser("serve", help="run the daemon in the foreground")
s.add_argument("--config", type=Path, help="config file (default: daemon.json)")
s.add_argument("--config", type=Path, help="config file (default: daemon.toml)")
s.add_argument("--port", type=int, help="loopback port for browsers (0 disables)")
s.add_argument("--binary", help="qsv binary to use (overrides daemon.json)")
s.add_argument("--binary", help="qsv binary to use (overrides daemon.toml)")
s.add_argument("--log-file", type=Path, help="also log to this file (rotated at 5 MB, 3 kept)")
s.add_argument("-v", "--verbose", action="store_true")

Expand Down
4 changes: 2 additions & 2 deletions src/qsv_client/daemon/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ def headers(self, body: bytes) -> dict[str, str]:


def _configured_socket() -> Path:
"""The socket the daemon was configured with (daemon.json), else the default."""
"""The socket the daemon was configured with (daemon.toml), else the default."""
try:
return DaemonConfig.load().socket_path()
except (OSError, ValueError, TypeError):
Expand Down Expand Up @@ -304,7 +304,7 @@ async def request(
status, data = await asyncio.wait_for(self._exchange(method, path, payload), wait)
except (FileNotFoundError, ConnectionRefusedError) as exc:
raise QsvDaemonUnavailable(f"qsv-clientd is not running at {t.describe()}") from exc
except asyncio.TimeoutError:
except TimeoutError:
raise QsvDaemonError("qsv-clientd did not answer in time") from None
except OSError as exc:
raise QsvDaemonError(_os_error_message(t, exc)) from exc
Expand Down
Loading