diff --git a/CLAUDE.md b/CLAUDE.md index df430c5..ede3044 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -319,6 +319,32 @@ client-side filter masked it). Because the skill teaches the new flag values and probe semantics, the floor was raised to 0.3.20 (same merge-after-CDN gate — the dist PR merges only after CDN `channels.stable` = 0.3.20). +0.3.21 picks up SDK 0.7.25 (spec v0.7.17) and adds **bulk connector metadata +imports** (features 45→46: `library.collections.imports.v1`; verbs stay 17 — +`imports` is a `library` subcommand family, not a verb): `library imports +create --name --connector ` saves an immutable definition that +lists a data connector server-side (google-drive, dropbox, zoom, gong, +recall, grain, iconik — not S3/GCS) and imports each matching file's +source_metadata into a METADATA collection as collection files — thousands +to hundreds of thousands of records per run, no media processing, runs +consume no credits. Filters are listing passes reusing the `connectors +files` flags (or `--filters ''` for several passes — Dropbox +listing is non-recursive, one pass per folder; unsupported keys are a 400 +at create, never dropped; Zoom/Gong pin their 6-month lookback into the +stored filters at create; iconik caps 10k rows per set). Modes: `append` +(default) imports new files + retries failures; `refresh` re-imports +everything matched and alone honors `--delete-missing` (sweeps ONLY files +this import brought in; a `--max-files`-capped run never sweeps). +`--rate-limit` (clamped per connector) and `--thumbnails` (Grain/iconik +posters) tune runs; `--no-start` saves without running. `imports +list`/`show` page run history; `create`/`run`/`show` report envelope status +`pending` while a run executes (with an `imports show` next-hint, mirroring +`query --background`) and `ready` once settled; one run may be active per +collection (busy trigger = rewritten retryable 409 error); `cancel` +(defaults to the latest run) and `delete` never remove imported files. +Because the skill teaches the subcommand family, the floor was raised to +0.3.21 (same merge-after-CDN gate — the dist PR merges only after CDN +`channels.stable` = 0.3.21). The host-level `profile` verb and the leading global flags `--home`/`--profile` (also `$TINYCLOUD_HOME`; 0.3.3+) relocate state and are intentionally absent from `commands --json` — like the launcher's install/update, they're CLI/host @@ -371,9 +397,9 @@ of printing JSON. Any script invoking the binary must redirect ` --to col_meta --metadata '{"deal":"acme"}' --json tinycloud library connectors refresh cloudglue://files/ --json # re-fetch stale source_metadata, re-index (free) +# Bulk metadata imports (0.3.21+) — load a WHOLE connector corpus in one free run (don't add files one at a time): +tinycloud library imports create col_meta --name "drive backfill" --connector --json # → pending; first run starts now +tinycloud library imports show col_meta --json # poll: pending while running, ready when settled → probe/ask # Publish an HTML artifact to Cloudglue Sites (manage with list / unpublish) tinycloud publish ./tinycloud-output/html/report.html --name report --visibility private --json @@ -231,6 +234,13 @@ Authoring your own recipes: [reference/workflow-authoring.md](reference/workflow - `probe --filter` works only with a collection scope (`--in collection:col_…`). `source_metadata.*` filters are file-level facts, so pair them with `--scope file`. +- `library imports` (0.3.21+): to load a WHOLE connector corpus into a + metadata collection, create a bulk import instead of `add`/`sync`ing files + one at a time — free, no media processing. `imports create`/`imports run` + return `pending` while the run executes; poll `imports show + ` until the envelope is `ready` before querying. One run may be + active per collection at a time; definitions are immutable (delete + + recreate to change filters); cancel/delete never remove imported files. - `query` (0.3.17+) is for analytics, not search: when the task is to COUNT, GROUP, rank, or join across a collection ("how many…", "which … most", "total hours per host"), reach for `query`, not `probe`/`ask`. Run `query diff --git a/skills/tinycloud/reference/verbs.md b/skills/tinycloud/reference/verbs.md index 92d2c69..976ab00 100644 --- a/skills/tinycloud/reference/verbs.md +++ b/skills/tinycloud/reference/verbs.md @@ -336,7 +336,7 @@ finds the query face across one or more collections (`--min-score`, `--group-by file`). `total` reports the server-available count across all modes (never rewritten by client `--min-*`/`--limit` filters). -### library — collections and connectors +### library — collections, connectors, and bulk metadata imports ```bash tinycloud library collections list --json @@ -353,6 +353,16 @@ tinycloud library connectors files [--limit 25] [--page-token tinycloud library connectors inspect [] --json # metadata peek, no file created (0.3.11+) tinycloud library connectors refresh --json # re-fetch stored source_metadata (0.3.15+) tinycloud library connectors sync [] --json +# Bulk metadata imports (0.3.21+) — load a whole connector corpus into a METADATA collection, free: +tinycloud library imports create --name --connector \ + [--from --to --title-search --folder-id --path

--team --meeting-type \ + | --filters ''] \ + [--mode append|refresh] [--delete-missing] [--rate-limit ] [--max-files ] [--thumbnails] [--no-start] --json +tinycloud library imports list [--limit ] [--offset ] --json +tinycloud library imports show [--limit ] [--offset ] --json # run history; PENDING while a run executes +tinycloud library imports run [--mode append|refresh] [--delete-missing|--no-delete-missing] [--max-files ] [--thumbnails|--no-thumbnails] --json +tinycloud library imports cancel [] --json # no run-id = the latest (only possibly-active) run +tinycloud library imports delete --json # imported files always stay ``` `collections create|add|remove|delete` are the only writes in an otherwise @@ -433,6 +443,42 @@ or `ask`, using `--filter` on connector library (titles, participants, dates, tags) before paying for full processing in a `media-descriptions` collection. +**Bulk metadata imports** (0.3.21+, feature `library.collections.imports.v1`) +are the bulk path INTO a metadata collection — never `add`/`sync` files one +at a time when a whole corpus is wanted. `library imports create --name + --connector ` saves a definition that lists the connector server-side +(google-drive, dropbox, zoom, gong, recall, grain, iconik — not S3/GCS) and +imports every matching file's `source_metadata` as collection files: +thousands to hundreds of thousands of records per run, **free — runs consume +no credits**. The first run starts immediately (`--no-start` saves the +definition only), and `create`/`imports run` return **`pending`** while a run +executes — poll `imports show ` until the envelope goes +`ready` (progress counters inline: listed, created, updated, skipped, +indexed, failed, removed), then query with `probe`/`ask`. + +Filters are listing passes reusing the `connectors files` flags (`--from`/ +`--to` YYYY-MM-DD UTC, `--title-search`, `--folder-id` Drive, `--path` +Dropbox — non-recursive, so cover a tree with one pass per folder via +`--filters '[{"path":"/a"},{"path":"/b"}]'` — `--team`/`--meeting-type` +Grain); a key the connector can't honor is a clear 400 at create, never +silently dropped, and no filters means one unfiltered pass over everything. +Zoom/Gong pin their 6-month default lookback into the stored filters at +create (pass `--from` to control the window); iconik caps 10k results per +filter set (split by date windows). Modes: `append` (default) imports new +files and retries previously-failed ones; `refresh` re-imports everything +matched, and only refresh runs honor `--delete-missing` (removes ONLY files +this import brought in that the source stopped returning; a +`--max-files`-capped run never sweeps). `--rate-limit` tunes upstream +listing requests/sec (per-connector safe defaults, clamped); +`--thumbnails` copies Grain/iconik poster images (off by default — slows +large iconik imports). Definitions are **immutable** (delete + recreate to +change one); **one run may be active per collection** — triggering while +busy errors with a clear retryable message; `imports cancel` stops the +active run (already-imported files stay) and `imports delete` removes the +definition + run history but never the imported files. On pre-0.3.21 +binaries every `imports` subcommand fails with an unknown-command error — +fall back to per-file `collections add` / `connectors sync`. + `connectors sync` materializes its argument into a Cloudglue file without starting analysis (idempotent). The connector id is optional — with just a URI or link, sync routes through the matching connector type. Connector URIs diff --git a/skills/tinycloud/scripts/preflight.sh b/skills/tinycloud/scripts/preflight.sh index 1f402d8..521e3a0 100755 --- a/skills/tinycloud/scripts/preflight.sh +++ b/skills/tinycloud/scripts/preflight.sh @@ -9,11 +9,11 @@ set -u # Mirror tinycloud-skill.json: min_version / supported_range upper bound # (CI diffs these against the manifest). -MIN_VERSION="0.3.20" +MIN_VERSION="0.3.21" MAX_VERSION_EXCLUSIVE="0.4.0" INSTALL_CMD='curl -fsSL https://app.cloudglue.dev/tinycloud.sh | bash' # Kept in sync with ../tinycloud-skill.json required_features (CI diffs them). -REQUIRED_FEATURES="envelope.v1 watch.v1 watch.speech.v1 see.v1 extract.v1 extract.images.v1 caption.v1 search.v1 probe.v1 probe.filters.v1 probe.entities.v1 probe.scope.auto.v1 ask.v1 query.v1 query.export.v1 clip.v1 grab.v1 face.v1 jobs.v1 library.collections.v1 library.collections.create.v1 library.collections.mutate.v1 library.collections.entities.v1 library.collections.metadata.v1 library.collections.describe.v1 library.collections.pagination.v1 library.sync.url.v1 workflow.v1 publish.v1 publish.manage.v1 publish.video.v1 publish.link.preview.v1 publish.link.preview.player.v1 setup.v1" +REQUIRED_FEATURES="envelope.v1 watch.v1 watch.speech.v1 see.v1 extract.v1 extract.images.v1 caption.v1 search.v1 probe.v1 probe.filters.v1 probe.entities.v1 probe.scope.auto.v1 ask.v1 query.v1 query.export.v1 clip.v1 grab.v1 face.v1 jobs.v1 library.collections.v1 library.collections.create.v1 library.collections.mutate.v1 library.collections.entities.v1 library.collections.metadata.v1 library.collections.describe.v1 library.collections.pagination.v1 library.collections.imports.v1 library.sync.url.v1 workflow.v1 publish.v1 publish.manage.v1 publish.video.v1 publish.link.preview.v1 publish.link.preview.player.v1 setup.v1" # 1) Binary present and responsive? if ! command -v tinycloud >/dev/null 2>&1; then diff --git a/skills/tinycloud/tinycloud-skill.json b/skills/tinycloud/tinycloud-skill.json index 8047bc9..4b962e4 100644 --- a/skills/tinycloud/tinycloud-skill.json +++ b/skills/tinycloud/tinycloud-skill.json @@ -1,8 +1,8 @@ { - "skill_version": "0.3.20", + "skill_version": "0.3.21", "tinycloud": { - "min_version": "0.3.20", - "supported_range": ">=0.3.20 <0.4.0", + "min_version": "0.3.21", + "supported_range": ">=0.3.21 <0.4.0", "required_features": [ "envelope.v1", "watch.v1", @@ -30,6 +30,7 @@ "library.collections.metadata.v1", "library.collections.describe.v1", "library.collections.pagination.v1", + "library.collections.imports.v1", "library.sync.url.v1", "workflow.v1", "publish.v1",