From bd0457cb6034dac09a98194c089cf28e2fe082f5 Mon Sep 17 00:00:00 2001 From: KDR Date: Tue, 18 Aug 2026 16:08:43 -0700 Subject: [PATCH 1/2] =?UTF-8?q?Prepare=20for=20tinycloud=200.3.22:=20bulk?= =?UTF-8?q?=20MEDIA=20imports,=20--enrich-metadata,=20Dropbox=20--recursiv?= =?UTF-8?q?e=20(skill=20floor=20=E2=86=92=200.3.22)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tinycloud 0.3.22 (SDK 0.7.28, spec v0.7.21) turns bulk imports from a metadata-collection loader into a general one (features 46→49; verbs stay 17): - An import's `import_type` follows the TARGET COLLECTION's type, inferred at create and fixed for its life. A `metadata` collection still imports source_metadata only (free); any other type (media-descriptions, entities, rich-transcripts, face-analysis) now imports the MEDIA itself, processed like `collections add` — BILLED PER FILE and counted against the account's file usage limits, capped at 10000 files/run whatever --max-files says, resumable with an `append` rerun after a credit/limit stop (feature library.collections.imports.media.v1). Summaries name the kind and a media create/run carries an explicit billing warning, so the skill teaches agents to ask the user first and cap with --max-files. - --enrich-metadata / --no-enrich-metadata (feature library.collections.imports.enrich.v1) backfills fields the connector listing omits — Gong parties + Call Spotlight (re-embedded, so the content becomes searchable) and Dropbox media_info duration/dimensions. Metadata-import only; it and --thumbnails are a 400 on a media import. - --recursive (feature library.connectors.recursive.v1) on `connectors files`, and a `recursive` filter-set key on `imports create`, walk a whole Dropbox subtree under --path instead of its direct children. On the wire it is the STRING "true"/"false". Because the skill teaches the media-import cost model and the new flags, the floor rises to 0.3.22: tinycloud-skill.json min_version/supported_range/skill_version → 0.3.22 and required_features +library.collections.imports.media.v1 +library.collections.imports.enrich.v1 (connector-capability ids stay out of the list, as with iconik/refresh), with preflight.sh kept byte-identical. SKILL.md and reference/verbs.md gain the import_type cost table, the metadata-only-flag rule, and --recursive. Plugin/marketplace metadata moves to 0.3.22 (it had lagged at 0.3.19 since the 0.3.20 and 0.3.21 releases skipped the bump). Merge gate unchanged: the live CDN still serves 0.3.21, so this PR merges only once `channels.stable` = 0.3.22 — preflight against the current CDN binary correctly exits 11 (below floor) until then. Verified: npm test 42/42, and 42/42 again against a real 0.3.22 tarball via TINYCLOUD_TEST_TARBALL; shellcheck clean; all three CI plugin/skill metadata checks pass locally; smoke-test.sh green against the 0.3.22 binary (EXPECTED_VERSION=0.3.22); preflight exits 13 on 0.3.22 (compatible, no creds) and 11 on 0.3.21. --- .claude-plugin/marketplace.json | 4 +- .claude-plugin/plugin.json | 2 +- CLAUDE.md | 40 ++++++++++ package.json | 2 +- skills/tinycloud/SKILL.md | 25 ++++-- skills/tinycloud/reference/verbs.md | 109 +++++++++++++++++--------- skills/tinycloud/scripts/preflight.sh | 4 +- skills/tinycloud/tinycloud-skill.json | 8 +- 8 files changed, 145 insertions(+), 49 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index a50ee6c..1f6e9d0 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -7,14 +7,14 @@ }, "metadata": { "description": "Deep video and image work for coding agents: analyze, describe, search, caption, clip, detect faces, and publish via the tinycloud CLI.", - "version": "0.3.19" + "version": "0.3.22" }, "plugins": [ { "name": "tinycloud", "source": "./", "description": "Video and image understanding and editing via the tinycloud CLI (Cloudglue): watch/see/extract/search/ask/face over videos and images, captions, clips, collections, plus one-command workflows for sales coaching, blog posts, ad analysis, meeting breakdowns, and YouTube publishing.", - "version": "0.3.19", + "version": "0.3.22", "author": { "name": "Cloudglue" }, "homepage": "https://tinycloud.sh", "repository": "https://github.com/cloudglue/tinycloud", diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json index e692a47..336aec4 100644 --- a/.claude-plugin/plugin.json +++ b/.claude-plugin/plugin.json @@ -2,7 +2,7 @@ "name": "tinycloud", "displayName": "Tinycloud Video", "description": "Video and image understanding and editing via the tinycloud CLI (Cloudglue): watch/see/extract/search/ask/face over videos and images, captions, clips, collections, and one-command video workflows.", - "version": "0.3.19", + "version": "0.3.22", "author": { "name": "Cloudglue", "url": "https://cloudglue.dev" }, "homepage": "https://tinycloud.sh", "repository": "https://github.com/cloudglue/tinycloud", diff --git a/CLAUDE.md b/CLAUDE.md index ede3044..2aab8e1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -345,6 +345,44 @@ collection (busy trigger = rewritten retryable 409 error); `cancel` Because the skill teaches the subcommand family, the floor was raised to 0.3.21 (same merge-after-CDN gate — the dist PR merges only after CDN `channels.stable` = 0.3.21). +0.3.22 picks up SDK 0.7.28 (spec v0.7.21) and turns bulk imports into a +**general collection loader** (features 46→49, verbs stay 17): the SDK's +`metadataImports` namespace is renamed `bulkImports` (the old name stays as +a back-compat alias, and tinycloud's adapter prefers the new one and falls +back), and an import's `import_type` — inferred from the TARGET +COLLECTION's type at create and fixed for the import's life — decides what +a run ingests. A **metadata** collection still imports source_metadata only +(free, no media processing); **any other** collection type +(media-descriptions, entities, rich-transcripts, face-analysis) now imports +the **media itself**, each file ingested and processed exactly like +`collections add`, so it is **billed per file** and counts against the +account's file usage limits (feature +`library.collections.imports.media.v1`). Media runs are additionally capped +at 10000 files per run whatever `--max-files` says, a run that exhausts +credits or a usage limit stops and keeps everything already imported (rerun +`append` to resume), and `refresh` on a media import re-syncs source +metadata only — media bytes are never re-downloaded. Because the cost story +now depends on the target collection, every summary names the kind +("Created media import …") and a media create/run carries an explicit +per-file billing warning, `imports list`/`show` render `type=media`, and +run progress gains the `files_imported` counter. `--thumbnails` and the new +`--enrich-metadata` are **metadata-import only** (either on a media import +is a clean upstream 400): `--enrich-metadata` (feature +`library.collections.imports.enrich.v1`, off by default) backfills +source-metadata fields the connector's listing omits after each index batch +settles — Gong parties + Call Spotlight content (batched and re-embedded, +so the content becomes searchable) and Dropbox `media_info` +duration/dimensions — reported by the new `files_enriched` counter, a no-op +for other connectors, and costing upstream API budget plus (for Gong) +embedding work. Dropbox listings gain `--recursive` on `connectors files` +and as a `recursive` filter-set key on `imports create` (feature +`library.connectors.recursive.v1`), walking the whole subtree under +`--path` instead of its direct children — so a Dropbox tree is one +recursive pass rather than one pass per folder; on the wire it is the +STRING `"true"`/`"false"`, and tinycloud rejects any other spelling +client-side. Because the skill teaches the media-import cost model and the +new flags, the floor was raised to 0.3.22 (same merge-after-CDN gate — the +dist PR merges only after CDN `channels.stable` = 0.3.22). The host-level `profile` verb and the leading global flags `--home`/`--profile` (also `$TINYCLOUD_HOME`; 0.3.3+) relocate state and are intentionally absent from `commands --json` — like the launcher's install/update, they're CLI/host @@ -400,6 +438,8 @@ of printing JSON. Any script invoking the binary must redirect ` --to col_meta --metadata '{"deal":"acme"}' --json tinycloud library connectors refresh cloudglue://files/ --json # re-fetch stale source_metadata, re-index (free) -# Bulk metadata imports (0.3.21+) — load a WHOLE connector corpus in one free run (don't add files one at a time): +# Bulk imports (0.3.21+) — load a WHOLE connector corpus in one run (don't add files one at a time). +# Into a METADATA collection: source_metadata only, free. Into ANY OTHER type: the media itself, BILLED PER FILE. tinycloud library imports create col_meta --name "drive backfill" --connector --json # → pending; first run starts now tinycloud library imports show col_meta --json # poll: pending while running, ready when settled → probe/ask +tinycloud library imports create col_meta --name "dbx tree" --connector --path /recordings --recursive --enrich-metadata --json # 0.3.22+ +tinycloud library imports create col_desc --name "corpus" --connector --max-files 50 --json # 0.3.22+ MEDIA import — ask the user first, it bills per file # Publish an HTML artifact to Cloudglue Sites (manage with list / unpublish) tinycloud publish ./tinycloud-output/html/report.html --name report --visibility private --json @@ -234,13 +237,25 @@ Authoring your own recipes: [reference/workflow-authoring.md](reference/workflow - `probe --filter` works only with a collection scope (`--in collection:col_…`). `source_metadata.*` filters are file-level facts, so pair them with `--scope file`. -- `library imports` (0.3.21+): to load a WHOLE connector corpus into a - metadata collection, create a bulk import instead of `add`/`sync`ing files - one at a time — free, no media processing. `imports create`/`imports run` - return `pending` while the run executes; poll `imports show +- `library imports` (0.3.21+): to load a WHOLE connector corpus, create a + bulk import instead of `add`/`sync`ing files one at a time. **What a run + ingests follows the target collection's type**, reported as `import_type`: + into a `metadata` collection it imports source_metadata only — free, no + media processing; into ANY OTHER type (media-descriptions, entities, + rich-transcripts, face-analysis) it imports the MEDIA itself, processed + like a manual add and **billed per file** (0.3.22+). Ask the user before + starting a media import, and cap it with `--max-files` (media runs also + stop at 10000 files/run). A media run that runs out of credits keeps what + it imported — rerun in `append` mode to resume. `imports create`/`imports + run` return `pending` while the run executes; poll `imports show ` until the envelope is `ready` before querying. One run may be active per collection at a time; definitions are immutable (delete + recreate to change filters); cancel/delete never remove imported files. + `--thumbnails` and `--enrich-metadata` are metadata-import only (either on + a media import is a clean upstream error); `--enrich-metadata` (0.3.22+) + backfills fields the connector listing omits — Gong parties + Call + Spotlight (re-embedded, so they become searchable) and Dropbox + `media_info` duration/dimensions. - `query` (0.3.17+) is for analytics, not search: when the task is to COUNT, GROUP, rank, or join across a collection ("how many…", "which … most", "total hours per host"), reach for `query`, not `probe`/`ask`. Run `query diff --git a/skills/tinycloud/reference/verbs.md b/skills/tinycloud/reference/verbs.md index 976ab00..18b4e13 100644 --- a/skills/tinycloud/reference/verbs.md +++ b/skills/tinycloud/reference/verbs.md @@ -336,7 +336,7 @@ finds the query face across one or more collections (`--min-score`, `--group-by file`). `total` reports the server-available count across all modes (never rewritten by client `--min-*`/`--limit` filters). -### library — collections, connectors, and bulk metadata imports +### library — collections, connectors, and bulk imports ```bash tinycloud library collections list --json @@ -349,18 +349,19 @@ tinycloud library collections remove --from --json tinycloud library collections delete --json tinycloud library collections entities [--limit ] [--offset ] --json # read a video's entities tinycloud library connectors list --json -tinycloud library connectors files [--limit 25] [--page-token ] --json +tinycloud library connectors files [--limit 25] [--page-token ] [--path [--recursive]] --json # --recursive: Dropbox subtree (0.3.22+) tinycloud library connectors inspect [] --json # metadata peek, no file created (0.3.11+) tinycloud library connectors refresh --json # re-fetch stored source_metadata (0.3.15+) tinycloud library connectors sync [] --json -# Bulk metadata imports (0.3.21+) — load a whole connector corpus into a METADATA collection, free: +# Bulk imports (0.3.21+) — load a whole connector corpus in one run. Into a METADATA +# collection: source_metadata only, free. Into ANY OTHER type: the media itself, BILLED PER FILE (0.3.22+). tinycloud library imports create --name --connector \ - [--from --to --title-search --folder-id --path

--team --meeting-type \ + [--from --to --title-search --folder-id --path

--recursive --team --meeting-type \ | --filters ''] \ - [--mode append|refresh] [--delete-missing] [--rate-limit ] [--max-files ] [--thumbnails] [--no-start] --json + [--mode append|refresh] [--delete-missing] [--rate-limit ] [--max-files ] [--thumbnails] [--enrich-metadata] [--no-start] --json tinycloud library imports list [--limit ] [--offset ] --json tinycloud library imports show [--limit ] [--offset ] --json # run history; PENDING while a run executes -tinycloud library imports run [--mode append|refresh] [--delete-missing|--no-delete-missing] [--max-files ] [--thumbnails|--no-thumbnails] --json +tinycloud library imports run [--mode append|refresh] [--delete-missing|--no-delete-missing] [--max-files ] [--thumbnails|--no-thumbnails] [--enrich-metadata|--no-enrich-metadata] --json tinycloud library imports cancel [] --json # no run-id = the latest (only possibly-active) run tinycloud library imports delete --json # imported files always stay ``` @@ -443,41 +444,77 @@ or `ask`, using `--filter` on connector library (titles, participants, dates, tags) before paying for full processing in a `media-descriptions` collection. -**Bulk metadata imports** (0.3.21+, feature `library.collections.imports.v1`) -are the bulk path INTO a metadata collection — never `add`/`sync` files one -at a time when a whole corpus is wanted. `library imports create --name - --connector ` saves a definition that lists the connector server-side -(google-drive, dropbox, zoom, gong, recall, grain, iconik — not S3/GCS) and -imports every matching file's `source_metadata` as collection files: -thousands to hundreds of thousands of records per run, **free — runs consume -no credits**. The first run starts immediately (`--no-start` saves the -definition only), and `create`/`imports run` return **`pending`** while a run -executes — poll `imports show ` until the envelope goes -`ready` (progress counters inline: listed, created, updated, skipped, -indexed, failed, removed), then query with `probe`/`ask`. +**Bulk imports** (0.3.21+, feature `library.collections.imports.v1`) are the +bulk path INTO a collection — never `add`/`sync` files one at a time when a +whole corpus is wanted. `library imports create --name --connector +` saves a definition that lists the connector server-side (google-drive, +dropbox, zoom, gong, recall, grain, iconik — not S3/GCS) and brings every +matching file into the collection. + +**What a run ingests follows the TARGET COLLECTION's type**, inferred at +create, fixed for the import's life, and reported as `import_type` on the +definition and on every run: + +| Target collection | `import_type` | What a run does | Cost | +|---|---|---|---| +| `metadata` | `metadata` | imports each file's `source_metadata` as a collection file — no media download or processing | **free** (no credits) | +| anything else (`media-descriptions`, `entities`, `rich-transcripts`, `face-analysis`) | `media` | ingests and processes the media itself, exactly like `collections add` | **billed per file** (0.3.22+, feature `library.collections.imports.media.v1`) | + +A **media** import counts against the account's file usage limits, so **ask +the user before starting one** and cap it with `--max-files` (media runs also +stop at 10000 files/run whatever `--max-files` says). A media run that +exhausts credits or a usage limit stops with a clear error and keeps +everything it already imported — rerun in `append` mode to resume. On a media +import, `refresh` re-syncs already-imported files' source metadata only; +media bytes are never re-downloaded. The create/run summary names the kind +("Created media import ...") and carries the per-file billing warning, so +read `data.import.import_type` (or the summary) before assuming a run is +free. + +The first run starts immediately (`--no-start` saves the definition only), +and `create`/`imports run` return **`pending`** while a run executes — poll +`imports show ` until the envelope goes `ready` (progress +counters inline: listed, created, updated, skipped, imported, indexed, +enriched, failed, removed), then query with `probe`/`ask`. Filters are listing passes reusing the `connectors files` flags (`--from`/ `--to` YYYY-MM-DD UTC, `--title-search`, `--folder-id` Drive, `--path` -Dropbox — non-recursive, so cover a tree with one pass per folder via +Dropbox — direct children unless `--recursive` walks the whole subtree +(0.3.22+, feature `library.connectors.recursive.v1`), so a tree is either one +recursive pass or one pass per folder via `--filters '[{"path":"/a"},{"path":"/b"}]'` — `--team`/`--meeting-type` Grain); a key the connector can't honor is a clear 400 at create, never silently dropped, and no filters means one unfiltered pass over everything. -Zoom/Gong pin their 6-month default lookback into the stored filters at -create (pass `--from` to control the window); iconik caps 10k results per -filter set (split by date windows). Modes: `append` (default) imports new -files and retries previously-failed ones; `refresh` re-imports everything -matched, and only refresh runs honor `--delete-missing` (removes ONLY files -this import brought in that the source stopped returning; a +In a `--filters` set, `recursive` is the STRING `"true"`/`"false"`, not a +JSON boolean. Zoom/Gong pin their 6-month default lookback into the stored +filters at create (pass `--from` to control the window); iconik caps 10k +results per filter set (split by date windows). Modes: `append` (default) +imports new files and retries previously-failed ones; `refresh` re-imports +everything matched, and only refresh runs honor `--delete-missing` (removes +ONLY files this import brought in that the source stopped returning; a `--max-files`-capped run never sweeps). `--rate-limit` tunes upstream -listing requests/sec (per-connector safe defaults, clamped); -`--thumbnails` copies Grain/iconik poster images (off by default — slows -large iconik imports). Definitions are **immutable** (delete + recreate to -change one); **one run may be active per collection** — triggering while -busy errors with a clear retryable message; `imports cancel` stops the -active run (already-imported files stay) and `imports delete` removes the -definition + run history but never the imported files. On pre-0.3.21 -binaries every `imports` subcommand fails with an unknown-command error — -fall back to per-file `collections add` / `connectors sync`. +listing requests/sec (per-connector safe defaults, clamped). + +`--thumbnails` and `--enrich-metadata` are **metadata-import only** — either +one on a media import is a clear upstream error (imported media gets real +thumbnails from the processing pipeline, and a media run has no +metadata-only record to enrich). `--thumbnails` copies Grain/iconik poster +images (off by default — slows large iconik imports). `--enrich-metadata` +(0.3.22+, feature `library.collections.imports.enrich.v1`; off by default) +backfills source-metadata fields the connector's listing omits, after each +index batch settles: Gong parties + Call Spotlight content (batched, and the +enriched documents are re-embedded so that content becomes searchable) and +Dropbox `media_info` duration/dimensions (per-file). It is a no-op for other +connectors and costs upstream API budget plus, for Gong, embedding work; the +run's `files_enriched` counter reports how many records it backfilled. + +Definitions are **immutable** (delete + recreate to change one); **one run +may be active per collection** — triggering while busy errors with a clear +retryable message; `imports cancel` stops the active run (already-imported +files stay) and `imports delete` removes the definition + run history but +never the imported files. On pre-0.3.21 binaries every `imports` subcommand +fails with an unknown-command error — fall back to per-file `collections +add` / `connectors sync`. `connectors sync` materializes its argument into a Cloudglue file without starting analysis (idempotent). The connector id is optional — with just a @@ -501,7 +538,9 @@ YouTube URLs cannot sync; use `tinycloud grab` instead. `connectors files` also takes provider-specific filters: `--from`/`--to` (dates — every provider except S3/GCS; Zoom and Gong default to the last 6 months; Iconik filters on asset date_created), `--folder-id` (Google -Drive), `--path` (Dropbox), +Drive), `--path` (Dropbox — direct children only; add `--recursive` on +0.3.22+ to walk the whole subtree under it, Dropbox only and built for bulk +imports rather than interactive browsing), `--bucket`/`--prefix` (S3/GCS — bucket required), `--title-search` (Grain, Zoom, Google Drive, Dropbox, Gong, Iconik), `--team`/`--meeting-type` (Grain). diff --git a/skills/tinycloud/scripts/preflight.sh b/skills/tinycloud/scripts/preflight.sh index 521e3a0..43be71f 100755 --- a/skills/tinycloud/scripts/preflight.sh +++ b/skills/tinycloud/scripts/preflight.sh @@ -9,11 +9,11 @@ set -u # Mirror tinycloud-skill.json: min_version / supported_range upper bound # (CI diffs these against the manifest). -MIN_VERSION="0.3.21" +MIN_VERSION="0.3.22" MAX_VERSION_EXCLUSIVE="0.4.0" INSTALL_CMD='curl -fsSL https://app.cloudglue.dev/tinycloud.sh | bash' # Kept in sync with ../tinycloud-skill.json required_features (CI diffs them). -REQUIRED_FEATURES="envelope.v1 watch.v1 watch.speech.v1 see.v1 extract.v1 extract.images.v1 caption.v1 search.v1 probe.v1 probe.filters.v1 probe.entities.v1 probe.scope.auto.v1 ask.v1 query.v1 query.export.v1 clip.v1 grab.v1 face.v1 jobs.v1 library.collections.v1 library.collections.create.v1 library.collections.mutate.v1 library.collections.entities.v1 library.collections.metadata.v1 library.collections.describe.v1 library.collections.pagination.v1 library.collections.imports.v1 library.sync.url.v1 workflow.v1 publish.v1 publish.manage.v1 publish.video.v1 publish.link.preview.v1 publish.link.preview.player.v1 setup.v1" +REQUIRED_FEATURES="envelope.v1 watch.v1 watch.speech.v1 see.v1 extract.v1 extract.images.v1 caption.v1 search.v1 probe.v1 probe.filters.v1 probe.entities.v1 probe.scope.auto.v1 ask.v1 query.v1 query.export.v1 clip.v1 grab.v1 face.v1 jobs.v1 library.collections.v1 library.collections.create.v1 library.collections.mutate.v1 library.collections.entities.v1 library.collections.metadata.v1 library.collections.describe.v1 library.collections.pagination.v1 library.collections.imports.v1 library.collections.imports.media.v1 library.collections.imports.enrich.v1 library.sync.url.v1 workflow.v1 publish.v1 publish.manage.v1 publish.video.v1 publish.link.preview.v1 publish.link.preview.player.v1 setup.v1" # 1) Binary present and responsive? if ! command -v tinycloud >/dev/null 2>&1; then diff --git a/skills/tinycloud/tinycloud-skill.json b/skills/tinycloud/tinycloud-skill.json index 4b962e4..fb52d83 100644 --- a/skills/tinycloud/tinycloud-skill.json +++ b/skills/tinycloud/tinycloud-skill.json @@ -1,8 +1,8 @@ { - "skill_version": "0.3.21", + "skill_version": "0.3.22", "tinycloud": { - "min_version": "0.3.21", - "supported_range": ">=0.3.21 <0.4.0", + "min_version": "0.3.22", + "supported_range": ">=0.3.22 <0.4.0", "required_features": [ "envelope.v1", "watch.v1", @@ -31,6 +31,8 @@ "library.collections.describe.v1", "library.collections.pagination.v1", "library.collections.imports.v1", + "library.collections.imports.media.v1", + "library.collections.imports.enrich.v1", "library.sync.url.v1", "workflow.v1", "publish.v1", From adca09f6dd99151815bc9d542dff018724002439 Mon Sep 17 00:00:00 2001 From: KDR Date: Thu, 20 Aug 2026 17:01:48 -0700 Subject: [PATCH 2/2] docs: live CDN now serves 0.3.22 --- CLAUDE.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 2aab8e1..8789248 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -435,11 +435,9 @@ of printing JSON. Any script invoking the binary must redirect `