From 787be27bef939a0c5c9234adfb81f48204999d6b Mon Sep 17 00:00:00 2001 From: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:59:02 -0400 Subject: [PATCH 1/6] feat(integrations): add integration documentation framework and data-movement guides Signed-off-by: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> --- docs/integrations/connection-settings.mdx | 20 ++++++ .../data-movement/aws-cli-sdk.mdx | 38 +++++++++++ .../data-movement/cyberduck-mountain-duck.mdx | 46 +++++++++++++ docs/integrations/data-movement/rclone.mdx | 61 +++++++++++++++++ docs/integrations/index.mdx | 26 +++++++ docs/integrations/validate-an-integration.mdx | 18 +++++ package.json | 3 +- scripts/check-integrations.mjs | 68 +++++++++++++++++++ sidebars.js | 17 +++++ static/llms-full.txt | 29 ++++++++ static/llms.txt | 8 +++ 11 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 docs/integrations/connection-settings.mdx create mode 100644 docs/integrations/data-movement/aws-cli-sdk.mdx create mode 100644 docs/integrations/data-movement/cyberduck-mountain-duck.mdx create mode 100644 docs/integrations/data-movement/rclone.mdx create mode 100644 docs/integrations/index.mdx create mode 100644 docs/integrations/validate-an-integration.mdx create mode 100644 scripts/check-integrations.mjs diff --git a/docs/integrations/connection-settings.mdx b/docs/integrations/connection-settings.mdx new file mode 100644 index 0000000..3f68241 --- /dev/null +++ b/docs/integrations/connection-settings.mdx @@ -0,0 +1,20 @@ +--- +title: Integration connection settings +description: Shared endpoint, region, authentication, and addressing settings for Fil One integrations. +--- + +# Integration connection settings + +Use the values shown in your bucket's **Access endpoints** panel. Keep endpoint values in configuration rather than embedding them in scripts or application code. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (also shown in the bucket's **Access endpoints** panel) | +| Region | The bucket's region: `eu-west-1` or `us-east-1` | +| Authentication | AWS Signature Version 4 (SigV4) | +| Addressing | Path style required | +| Access key and secret | A key created for the same region, scoped to the target bucket | + +Create the bucket before configuring an integration. Endpoint, region, key, and bucket must belong to the same region. Keep TLS certificate validation enabled. Never place a secret key in documentation, screenshots, source control, or command-line history. + +Third-party tools may attempt optional S3 operations after a transfer. Review [S3 Compatibility](/reference/s3-compatibility) for current regional behavior and [Encryption](/security/encryption) for security settings. Feature availability differs between the S3 API and the app.fil.one dashboard; see S3 Compatibility for API behavior. diff --git a/docs/integrations/data-movement/aws-cli-sdk.mdx b/docs/integrations/data-movement/aws-cli-sdk.mdx new file mode 100644 index 0000000..bbcd6d8 --- /dev/null +++ b/docs/integrations/data-movement/aws-cli-sdk.mdx @@ -0,0 +1,38 @@ +--- +title: AWS CLI and SDKs +description: Use existing Fil One AWS CLI and SDK guides for data movement workflows. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# AWS CLI and SDKs + +The AWS CLI and SDKs are Fil One's reference clients; see the [SDK guides](/sdk/overview) for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (`forcePathStyle: true` in JavaScript, `UsePathStyle: true` in Go, `addressing_style: path` in boto3 and the CLI) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named AWS CLI or SDK version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Follow the guide for your client: [AWS CLI](/sdk/cli), [Python with boto3](/sdk/python), [JavaScript with AWS SDK v3](/sdk/javascript), or [Go with aws-sdk-go-v2](/sdk/go). Apply the [connection settings](/integrations/connection-settings) to each client. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration). Include a file above the client's multipart threshold so automatic multipart upload is exercised. For supported operations and regional differences, see [S3 Compatibility](/reference/s3-compatibility); for object and part sizes, see [Limits & Quotas](/limits). + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Requests go to a bucket subdomain | Set the client's path-style option. | +| Signature mismatch or `403` | Endpoint, region, and key must belong to the same region and the key must be scoped to the bucket. | +| Optional operation returns `403` or `NotImplemented` | Compare against [S3 Compatibility](/reference/s3-compatibility). | diff --git a/docs/integrations/data-movement/cyberduck-mountain-duck.mdx b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx new file mode 100644 index 0000000..07416c3 --- /dev/null +++ b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx @@ -0,0 +1,46 @@ +--- +title: Cyberduck and Mountain Duck +description: Connect Cyberduck or Mountain Duck to a Fil One bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Cyberduck and Mountain Duck + +Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (enter the hostname without a bucket name or path) | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (Cyberduck's path-style S3 connection profile) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Cyberduck or Mountain Duck version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +1. Install Cyberduck's path-style S3 connection profile from the profiles list. +2. Open **New Bookmark**, choose that profile, and enter the hostname from the bucket's regional endpoint. +3. Enter the access key and secret for the same region. +4. Keep HTTPS and certificate validation enabled. Save the bookmark. + +Mountain Duck uses the same bookmark settings, then mounts the bucket through Finder or File Explorer. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Rename and delete a test object and confirm the remote listing reflects each operation. +2. Test keys containing spaces before a larger migration. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Client requests a bucket subdomain | Use the path-style S3 connection profile rather than the default Amazon S3 profile. | +| `403` | Key, endpoint, and region must belong to the same region and the key must be scoped to the bucket. | +| Unexpected filename encoding | Test keys with spaces and review regional behavior in [S3 Compatibility](/reference/s3-compatibility). | diff --git a/docs/integrations/data-movement/rclone.mdx b/docs/integrations/data-movement/rclone.mdx new file mode 100644 index 0000000..3d152b3 --- /dev/null +++ b/docs/integrations/data-movement/rclone.mdx @@ -0,0 +1,61 @@ +--- +title: rclone +description: Configure rclone for upload, restore, and integrity checks with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# rclone + +rclone connects to Fil One as an S3 remote with `provider = Other`, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style (`force_path_style = true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region | +| Provider | `Other` | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named rclone version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Run `rclone config`, create a new remote, choose **Amazon S3 Compliant Storage Providers**, and select **Other** as the provider. Enter the bucket's region and endpoint when prompted, enable path-style access, and enter credentials at rclone's prompts. The resulting section in `rclone.conf` looks like this: + +```ini +[filone] +type = s3 +provider = Other +access_key_id = ACCESS_KEY +secret_access_key = SECRET_KEY +region = eu-west-1 +endpoint = https://eu-west-1.s3.fil.one +force_path_style = true +``` + +Test discovery without changing remote data: + +```bash +rclone lsd filone: +``` + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Copy with `rclone copy ./fixture filone:BUCKET/VALIDATION_PREFIX` and list with `rclone lsjson filone:BUCKET/VALIDATION_PREFIX`. +2. Restore with `rclone copy filone:BUCKET/VALIDATION_PREFIX ./restored`. +3. Run `rclone check --download ./restored filone:BUCKET/VALIDATION_PREFIX`. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Signature mismatch | Endpoint, region, and key must refer to the same region; confirm `force_path_style = true`. | +| Bucket not found | Confirm the bucket already exists and the key is scoped to it. | +| TLS error | Keep certificate validation enabled; inspect local proxy or trust-store configuration. | +| Cleanup leaves data | List the exact validation prefix and remove only that prefix. | diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx new file mode 100644 index 0000000..ba27927 --- /dev/null +++ b/docs/integrations/index.mdx @@ -0,0 +1,26 @@ +--- +title: Integrations +description: Configure and validate third-party tools with Fil One object storage. +--- + +# Integrations + +These guides explain how to connect common transfer, backup, NAS, DevOps, analytics, and AI tools to Fil One. Each guide gives the connection settings, a validation procedure, and troubleshooting for one tool. + +Every guide displays one of these validation states: + +| Status | Meaning | +| --- | --- | +| **Verified** | A named tool version completed the documented end-to-end workflow, restore, and integrity check in every listed region. The guide includes the evidence and verification date. | +| **Compatible** | An end-to-end run exists for a named version, but not for every region or not within the last 12 months. | +| **Preview** | A runnable validation path exists, but coverage is incomplete. | +| **Not currently tested** | No end-to-end run has been recorded for any version or region. Follow the validation steps before production use. | +| **Not currently supported** | A known product or addressing requirement blocks the integration. | + +Start with [Connection settings](/integrations/connection-settings) and the [validation round trip](/integrations/validate-an-integration), then open the guide for your tool. For API behavior, use the [S3 Compatibility reference](/reference/s3-compatibility). + +| Category | Tool | Status | +| --- | --- | --- | +| Data movement | [rclone](/integrations/data-movement/rclone) | Not currently tested | +| Data movement | [Cyberduck and Mountain Duck](/integrations/data-movement/cyberduck-mountain-duck) | Not currently tested | +| Data movement | [AWS CLI and SDKs](/integrations/data-movement/aws-cli-sdk) | Not currently tested | diff --git a/docs/integrations/validate-an-integration.mdx b/docs/integrations/validate-an-integration.mdx new file mode 100644 index 0000000..86cb96f --- /dev/null +++ b/docs/integrations/validate-an-integration.mdx @@ -0,0 +1,18 @@ +--- +title: Validate an integration +description: The round trip every Fil One integration guide uses to confirm upload, restore, and integrity. +--- + +# Validate an integration + +Every integration guide links to this round trip. Run it with disposable data before pointing production workloads at the bucket. + +1. Create a local fixture directory containing a small text file, a nested file, and a file large enough to trigger multipart upload in the tool (above the tool's part or chunk threshold; see [Limits & Quotas](/limits) for part sizes). +2. Copy the fixture to a unique, disposable prefix in the bucket, for example `validation-YYYYMMDD-HHMM/`. +3. List the prefix and confirm every file is present. +4. Remove the local copy of the fixture. +5. Restore the prefix to a new local directory. +6. Compare byte size and SHA-256 for every restored file against the values recorded before upload. +7. Delete only that prefix, then list it again to confirm it is empty. + +Record the tool version, the region, and the date with the result. The tool-specific guide adds any steps that go beyond this round trip. diff --git a/package.json b/package.json index ebfeca8..775443f 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,8 @@ "swizzle": "docusaurus swizzle", "deploy": "docusaurus deploy", "clear": "docusaurus clear", - "serve": "docusaurus serve" + "serve": "docusaurus serve", + "check:integrations": "node scripts/check-integrations.mjs" }, "dependencies": { "@docusaurus/core": "^3.9.2", diff --git a/scripts/check-integrations.mjs b/scripts/check-integrations.mjs new file mode 100644 index 0000000..4883c59 --- /dev/null +++ b/scripts/check-integrations.mjs @@ -0,0 +1,68 @@ +import fs from "node:fs"; +import path from "node:path"; + +const root = path.resolve("docs/integrations"); +const guides = []; + +function walk(directory) { + for (const entry of fs.readdirSync(directory, { withFileTypes: true })) { + const absolute = path.join(directory, entry.name); + if (entry.isDirectory()) walk(absolute); + else if (entry.name.endsWith(".mdx") && path.dirname(absolute) !== root) guides.push(absolute); + } +} + +walk(root); +guides.sort(); +const failures = []; +const allowedStatuses = new Set(["Verified", "Compatible", "Preview", "Not currently tested", "Not currently supported"]); +const sidebar = fs.readFileSync("sidebars.js", "utf8"); +const mirrors = Object.fromEntries(["static/llms.txt", "static/llms-full.txt"].map((m) => [m, fs.readFileSync(m, "utf8")])); + +for (const file of guides) { + const source = fs.readFileSync(file, "utf8"); + const route = path.relative("docs", file).replace(/\.mdx$/, ""); + const match = source.match(/^---\n([\s\S]*?)\n---/); + if (!match) { + failures.push(`${file}: missing frontmatter`); + continue; + } + const value = (key) => match[1].match(new RegExp(`^${key}:\\s*(.*)$`, "m"))?.[1]?.trim(); + const status = value("integration_status"); + const body = source.slice(match[0].length); + if (!status) failures.push(`${file}: missing integration_status`); + if (status && !allowedStatuses.has(status)) failures.push(`${file}: invalid integration_status`); + const blocked = status === "Not currently supported"; + if (blocked && value("draft") !== "true") failures.push(`${file}: blocked guide must be draft: true`); + if (status === "Verified") { + for (const key of ["tested_versions", "verified_regions", "verification_date", "evidence"]) { + const field = value(key); + if (!field || field === "[]" || field === "null") failures.push(`${file}: Verified requires ${key}`); + } + if (!/^\[[^\]]+\]$/.test(value("tested_versions") ?? "")) failures.push(`${file}: Verified versions must be a non-empty inline list`); + if (!/^\[(eu-west-1|us-east-1)(,\s*(eu-west-1|us-east-1))*\]$/.test(value("verified_regions") ?? "")) failures.push(`${file}: Verified regions must use public region IDs`); + if (!/^\d{4}-\d{2}-\d{2}$/.test(value("verification_date") ?? "")) failures.push(`${file}: Verified requires an ISO verification date`); + if (!/^https:\/\//.test(value("evidence") ?? "")) failures.push(`${file}: Verified evidence must be an HTTPS link`); + } + if (status !== "Verified" && /(^|\W)Verified(\W|$)/.test(body)) { + failures.push(`${file}: body uses Verified without Verified metadata`); + } + if (status && !blocked) { + const statusLines = body.split("\n").filter((line) => line.startsWith("Validation status: ")); + if (statusLines.length !== 1 || !statusLines[0].startsWith(`Validation status: ${status}.`)) { + failures.push(`${file}: body must contain exactly one line starting with "Validation status: ${status}."`); + } + } + if (blocked && sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: blocked guide must not be listed: ${route}`); + if (!blocked && !sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: missing ${route}`); + for (const [mirror, text] of Object.entries(mirrors)) { + if (blocked && text.includes(`/${route}`)) failures.push(`${mirror}: blocked guide must not be mirrored: /${route}`); + if (!blocked && !text.includes(`/${route}`)) failures.push(`${mirror}: missing /${route}`); + } +} + +if (failures.length) { + console.error(failures.join("\n")); + process.exit(1); +} +console.log(`Checked ${guides.length} integration guides.`); diff --git a/sidebars.js b/sidebars.js index 9f232b3..c0b14ac 100644 --- a/sidebars.js +++ b/sidebars.js @@ -77,6 +77,23 @@ const sidebars = { "sdk/cli", ], }, + { + type: "category", + label: "Integrations", + items: [ + "integrations/index", + "integrations/connection-settings", + { + type: "category", + label: "Data movement", + items: [ + "integrations/data-movement/rclone", + "integrations/data-movement/cyberduck-mountain-duck", + "integrations/data-movement/aws-cli-sdk", + ], + }, + ], + }, "faq", "limits", ], diff --git a/static/llms-full.txt b/static/llms-full.txt index d6edb6e..1c405ca 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2913,3 +2913,32 @@ Embeddings and extracted chunk text are held in storage Fil One manages, separat from the bucket. Objects themselves are not moved or modified. Extracted chunk text is retained in the index so answers can quote it — enable Bucket Intelligence deliberately per bucket rather than across everything. +# Integrations + +Source: https://docs.fil.one/integrations + +Integration guides are compatibility guidance, not vendor certification. Verified requires a named tool version, end-to-end workflow, restore, integrity check, evidence link, verification date, and coverage in every listed region. Other states are Compatible, Preview, Not currently tested, and Not currently supported. + +# Integration connection settings + +Source: https://docs.fil.one/integrations/connection-settings + +Use the endpoint in the bucket's Access endpoints panel and keep it in configuration. Endpoint, region, API key, and bucket must belong to the same region. Use SigV4, path-style addressing, HTTPS, and certificate validation. Never put secret keys in docs, screenshots, source control, or command-line history. Current API behavior lives in the S3 Compatibility reference. + +# rclone + +Source: https://docs.fil.one/integrations/data-movement/rclone + +Validation status: Not currently tested. Configure an Other S3 provider with the bucket's endpoint and region, path-style access, and prompted or secret-store credentials. Validate a small, nested, and multipart-sized fixture under a unique prefix; list it, delete the local copy, restore, compare size and SHA-256, run rclone check --download, and delete only that prefix. + +# Cyberduck and Mountain Duck + +Source: https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck + +Validation status: Not currently tested. Use an Amazon S3 bookmark with the endpoint hostname, matching regional key, bucket path, HTTPS, and certificate validation. Validate nested upload, download, SHA-256, rename, delete, and recovery with disposable data. Mountain Duck uses the same bookmark settings for a desktop mount. + +# AWS CLI and SDKs + +Source: https://docs.fil.one/integrations/data-movement/aws-cli-sdk + +Validation status: Not currently tested. Use the maintained AWS CLI, boto3, JavaScript v3, and Go SDK pages rather than duplicated examples. Validate small and automatic-multipart uploads, listing, restore, SHA-256 and byte size, and exact-prefix cleanup. boto3 is the reference API workflow, not backup-product certification. diff --git a/static/llms.txt b/static/llms.txt index e768105..7597343 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -174,6 +174,14 @@ Critical integration notes for developers: download, paginated listing, presigned URLs, and error handling; requires `UsePathStyle: true` +## Integrations + +- [Integrations](https://docs.fil.one/integrations): Validation status definitions and integration guide index +- [Integration connection settings](https://docs.fil.one/integrations/connection-settings): Central endpoint, region, SigV4, path-style, and credential guidance +- [rclone](https://docs.fil.one/integrations/data-movement/rclone): Not currently tested; configuration plus upload, restore, checksum, and cleanup validation +- [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): Not currently tested; bookmark, mount, restore, and troubleshooting guidance +- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): Not currently tested; links to the maintained CLI and SDK guides without duplicating them + ## Optional The browser console at app.fil.one, one page per console screen. No code From c55181d2ee1993c3c97cdcd31e32b80782014fe2 Mon Sep 17 00:00:00 2001 From: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:59:02 -0400 Subject: [PATCH 2/6] docs(integrations): add backup and NAS integration guides Signed-off-by: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> --- .../backup/acronis-cyber-protect.mdx | 42 ++++++++++++++++++ docs/integrations/backup/comet-backup.mdx | 43 +++++++++++++++++++ docs/integrations/backup/msp360.mdx | 42 ++++++++++++++++++ docs/integrations/backup/nakivo.mdx | 41 ++++++++++++++++++ docs/integrations/index.mdx | 7 +++ docs/integrations/nas/qnap-hbs.mdx | 42 ++++++++++++++++++ docs/integrations/nas/synology-cloud-sync.mdx | 42 ++++++++++++++++++ docs/integrations/nas/truenas-cloud-sync.mdx | 42 ++++++++++++++++++ sidebars.js | 19 ++++++++ static/llms-full.txt | 42 ++++++++++++++++++ static/llms.txt | 7 +++ 11 files changed, 369 insertions(+) create mode 100644 docs/integrations/backup/acronis-cyber-protect.mdx create mode 100644 docs/integrations/backup/comet-backup.mdx create mode 100644 docs/integrations/backup/msp360.mdx create mode 100644 docs/integrations/backup/nakivo.mdx create mode 100644 docs/integrations/nas/qnap-hbs.mdx create mode 100644 docs/integrations/nas/synology-cloud-sync.mdx create mode 100644 docs/integrations/nas/truenas-cloud-sync.mdx diff --git a/docs/integrations/backup/acronis-cyber-protect.mdx b/docs/integrations/backup/acronis-cyber-protect.mdx new file mode 100644 index 0000000..b21ad76 --- /dev/null +++ b/docs/integrations/backup/acronis-cyber-protect.mdx @@ -0,0 +1,42 @@ +--- +title: Acronis Cyber Protect +description: Configure a Fil One storage location for Acronis backup and restore. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Acronis Cyber Protect + +Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 (AuthV4) access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Acronis Cyber Protect version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +In the Acronis administration console, add an S3-compatible storage location. Enter the regional endpoint, the matching access key, and AuthV4 signing from [Connection settings](/integrations/connection-settings). Select an existing Fil One bucket when prompted, then assign the location to a protection plan. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Protect the fixture directory with the plan, run it, and confirm the activity completes. +2. Restore to a new location and compare size and SHA-256 for every restored file. +3. Confirm that retention and cleanup operations affect only the test backup set. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection failure | Recheck endpoint, regional key, AuthV4, and TLS validation. | +| Missing bucket | Create it first and ensure the key is scoped to it. | +| Immutability option | This depends on Object Lock semantics. Review [S3 Compatibility](/reference/s3-compatibility) and validate it with the exact Acronis release before enabling it. | diff --git a/docs/integrations/backup/comet-backup.mdx b/docs/integrations/backup/comet-backup.mdx new file mode 100644 index 0000000..4b2c54a --- /dev/null +++ b/docs/integrations/backup/comet-backup.mdx @@ -0,0 +1,43 @@ +--- +title: Comet Backup +description: Configure and validate a Comet S3-compatible Storage Vault with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Comet Backup + +Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (some Comet fields expect the hostname without `https://`; follow the field label) | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Comet Backup version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Create either a Comet Storage Template or a user's Custom Storage Vault, select the S3-compatible type, and enter the regional hostname, region, existing bucket, and matching credentials. Enable encrypted transmission. Use the same values described in [Connection settings](/integrations/connection-settings). + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Run Comet's connection test, protect the fixture directory, and complete an on-demand backup. +2. Restore to a separate directory, run Comet's vault integrity check, and inspect the job history for partial failures. +3. Remove only the test snapshot and prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection test fails | Confirm whether the Comet field expects a hostname or a full URL. | +| Access errors | Check that the key can read, write, list, and delete within the selected bucket. | +| Object Lock vault type | Use the standard S3-compatible storage type unless the Object Lock workflow has been validated with your Comet release; see [S3 Compatibility](/reference/s3-compatibility). | +| Slow or failing large backups | Test concurrency and recovery time from the same network where the Comet server runs. | diff --git a/docs/integrations/backup/msp360.mdx b/docs/integrations/backup/msp360.mdx new file mode 100644 index 0000000..4ffffb4 --- /dev/null +++ b/docs/integrations/backup/msp360.mdx @@ -0,0 +1,42 @@ +--- +title: MSP360 +description: Add Fil One as S3-compatible storage for MSP360 backup plans. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# MSP360 + +MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 (signature version 4) access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named MSP360 version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Add an S3-compatible storage account. Put the regional URL in the HTTPS endpoint field, choose signature version 4, keep certificate validation enabled, and select an existing bucket. Enable native multipart upload after validating it with your MSP360 release. The desktop agent exposes the same core settings. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Create a test plan for the fixture directory and run it. +2. Confirm the remote objects are listed, restore to a new directory, and compare size and SHA-256. +3. Delete only the isolated test backup set. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection or signature errors | Use the S3-compatible provider rather than an AWS preset, confirm path-style addressing, and keep HTTPS enabled. | +| Bucket discovery fails | Check the bucket's region and the key's scope. | +| Backup succeeds but restore fails | Preserve the job logs and rerun the round trip after correcting the settings. | diff --git a/docs/integrations/backup/nakivo.mdx b/docs/integrations/backup/nakivo.mdx new file mode 100644 index 0000000..a7110fb --- /dev/null +++ b/docs/integrations/backup/nakivo.mdx @@ -0,0 +1,41 @@ +--- +title: NAKIVO Backup and Replication +description: Configure Fil One as an S3-compatible NAKIVO repository. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# NAKIVO Backup and Replication + +NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named NAKIVO Backup and Replication version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Add an S3-compatible repository. Enter the regional endpoint, region, matching credentials, and an existing bucket. Keep HTTPS certificate validation enabled. Ensure the Transporter that performs data movement can reach the endpoint. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Back up a disposable workload with known file hashes, then perform file-level recovery to a different location and compare size and SHA-256. +2. Run NAKIVO's repository verification if available. +3. Exercise retention against only the test recovery point and confirm unrelated prefixes remain untouched. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Connection errors | Endpoint, key region, and path-style settings must match. | +| Retention or immutability options | Validate against the named NAKIVO release and current [S3 Compatibility](/reference/s3-compatibility) behavior. | diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index ba27927..ef4f7e5 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -24,3 +24,10 @@ Start with [Connection settings](/integrations/connection-settings) and the [val | Data movement | [rclone](/integrations/data-movement/rclone) | Not currently tested | | Data movement | [Cyberduck and Mountain Duck](/integrations/data-movement/cyberduck-mountain-duck) | Not currently tested | | Data movement | [AWS CLI and SDKs](/integrations/data-movement/aws-cli-sdk) | Not currently tested | +| Backup and disaster recovery | [Acronis Cyber Protect](/integrations/backup/acronis-cyber-protect) | Not currently tested | +| Backup and disaster recovery | [Comet Backup](/integrations/backup/comet-backup) | Not currently tested | +| Backup and disaster recovery | [MSP360](/integrations/backup/msp360) | Not currently tested | +| Backup and disaster recovery | [NAKIVO Backup and Replication](/integrations/backup/nakivo) | Not currently tested | +| NAS and file services | [Synology Cloud Sync](/integrations/nas/synology-cloud-sync) | Not currently tested | +| NAS and file services | [QNAP HBS](/integrations/nas/qnap-hbs) | Not currently tested | +| NAS and file services | [TrueNAS Cloud Sync](/integrations/nas/truenas-cloud-sync) | Not currently tested | diff --git a/docs/integrations/nas/qnap-hbs.mdx b/docs/integrations/nas/qnap-hbs.mdx new file mode 100644 index 0000000..9df6631 --- /dev/null +++ b/docs/integrations/nas/qnap-hbs.mdx @@ -0,0 +1,42 @@ +--- +title: QNAP HBS +description: Configure QNAP Hybrid Backup Sync with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# QNAP HBS + +QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named QNAP HBS version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +In HBS 3, create a storage space of the S3-compatible type. Enter the regional endpoint, region, SigV4 credentials, and existing bucket. Enable SSL and certificate validation. Create a backup or sync job targeting a dedicated remote prefix. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Run the backup job for the fixture, delete the local fixture, and restore it to a new folder; compare size and SHA-256. +2. Review the HBS job log for skipped files. +3. Test retention or sync deletion only inside the validation prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Timestamp or signature errors | Enable NTP on the NAS; choose SigV4 and the correct region. | +| Connection fails | Confirm the service address includes HTTPS and does not construct a bucket subdomain. | +| Large-file failures | Capture a repeatable failure before reducing concurrency or chunk size. | diff --git a/docs/integrations/nas/synology-cloud-sync.mdx b/docs/integrations/nas/synology-cloud-sync.mdx new file mode 100644 index 0000000..260fc21 --- /dev/null +++ b/docs/integrations/nas/synology-cloud-sync.mdx @@ -0,0 +1,42 @@ +--- +title: Synology Cloud Sync +description: Configure Synology Cloud Sync with a Fil One bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Synology Cloud Sync + +Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Synology Cloud Sync version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Install Cloud Sync, add a connection, and choose S3 storage with a custom server URL. Enter the regional endpoint, SigV4 credentials, and existing bucket. Choose a local path, remote path, and one of bidirectional, upload-only, or download-only synchronization. Keep TLS validation enabled. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Sync the fixture with an upload-only task to a dedicated remote prefix and wait for the task to report up to date. +2. Remove the local fixture and create a download-only recovery task to a separate directory; compare size and SHA-256. +3. Test rename and delete semantics before choosing bidirectional sync, because that mode can propagate deletions. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Empty bucket dropdown | Endpoint, region, credentials, or bucket scope do not match. | +| Signature failures | Correct NAS time with NTP. | +| Large files stall | Test a smaller chunk size within the multipart guidance in [Limits & Quotas](/limits). | diff --git a/docs/integrations/nas/truenas-cloud-sync.mdx b/docs/integrations/nas/truenas-cloud-sync.mdx new file mode 100644 index 0000000..d8d52af --- /dev/null +++ b/docs/integrations/nas/truenas-cloud-sync.mdx @@ -0,0 +1,42 @@ +--- +title: TrueNAS Cloud Sync +description: Configure a TrueNAS Cloud Sync push, pull, or sync task with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# TrueNAS Cloud Sync + +TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region (leave signature version 2 disabled) | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named TrueNAS version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +In TrueNAS SCALE 22 and later, create the credential under Backup Credentials; in TrueNAS CORE, use the System cloud credentials page. Choose Amazon S3 with a custom endpoint, enter the regional key and region, keep signature version 2 disabled, and verify the credential. Create a Cloud Sync task using an existing bucket and a dedicated folder. + +Choose the COPY transfer mode for the first test. SYNC can delete destination files that are absent at the source, and MOVE deletes source files after transfer. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Run a PUSH of the fixture, then a PULL into a separate dataset; compare size and SHA-256. +2. Inspect the task logs and confirm cleanup stays within the test folder. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Credential verification fails | Recheck endpoint, region, key scope, SigV4, and system time. | +| Task succeeds with missing files | Compare the source and restored manifests before increasing concurrency. | diff --git a/sidebars.js b/sidebars.js index c0b14ac..53f9769 100644 --- a/sidebars.js +++ b/sidebars.js @@ -92,6 +92,25 @@ const sidebars = { "integrations/data-movement/aws-cli-sdk", ], }, + { + type: "category", + label: "Backup and disaster recovery", + items: [ + "integrations/backup/acronis-cyber-protect", + "integrations/backup/comet-backup", + "integrations/backup/msp360", + "integrations/backup/nakivo", + ], + }, + { + type: "category", + label: "NAS and file services", + items: [ + "integrations/nas/synology-cloud-sync", + "integrations/nas/qnap-hbs", + "integrations/nas/truenas-cloud-sync", + ], + }, ], }, "faq", diff --git a/static/llms-full.txt b/static/llms-full.txt index 1c405ca..7bf528c 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2942,3 +2942,45 @@ Validation status: Not currently tested. Use an Amazon S3 bookmark with the endp Source: https://docs.fil.one/integrations/data-movement/aws-cli-sdk Validation status: Not currently tested. Use the maintained AWS CLI, boto3, JavaScript v3, and Go SDK pages rather than duplicated examples. Validate small and automatic-multipart uploads, listing, restore, SHA-256 and byte size, and exact-prefix cleanup. boto3 is the reference API workflow, not backup-product certification. + +# Acronis Cyber Protect + +Source: https://docs.fil.one/integrations/backup/acronis-cyber-protect + +Validation status: Not currently tested. Add Public cloud storage using S3 compatible, the regional endpoint and key, AuthV4, and an existing bucket. Validate a disposable backup, local deletion, restore, SHA-256 and size, and scoped cleanup. This is not Acronis certification; validate any immutability option separately. + +# Comet Backup + +Source: https://docs.fil.one/integrations/backup/comet-backup + +Validation status: Not currently tested. Configure an S3-compatible Storage Template or Custom Storage Vault with encrypted transmission. Complete Test Connection, backup, restore to a new directory, SHA-256 and size comparison, native vault checking, and isolated cleanup. + +# MSP360 + +Source: https://docs.fil.one/integrations/backup/msp360 + +Validation status: Not currently tested. Add an S3 Compatible storage account with the regional HTTPS endpoint, SigV4, certificate validation, and existing bucket. Validate small and multipart-sized backup, restore, hashes, and exact test-set cleanup. + +# NAKIVO Backup and Replication + +Source: https://docs.fil.one/integrations/backup/nakivo + +Validation status: Not currently tested. Configure an S3-compatible repository with matching endpoint, region, credentials, bucket, and TLS validation. Validate file-level recovery and repository verification using disposable data before relying on multipart, retention, or immutability behavior. + +# Synology Cloud Sync + +Source: https://docs.fil.one/integrations/nas/synology-cloud-sync + +Validation status: Not currently tested. Configure S3 Storage with a custom server URL, SigV4, and an existing bucket. Validate a dedicated-prefix sync and download-only recovery with SHA-256 and size. Test deletion semantics before enabling bidirectional sync. + +# QNAP HBS + +Source: https://docs.fil.one/integrations/nas/qnap-hbs + +Validation status: Not currently tested. Create an S3 Compatible HBS 3 Storage Space with SSL, certificate validation, SigV4, and an existing bucket. Back up disposable data, restore to a new folder, compare hashes, inspect logs, and keep retention or sync deletion inside the validation prefix. + +# TrueNAS Cloud Sync + +Source: https://docs.fil.one/integrations/nas/truenas-cloud-sync + +Validation status: Not currently tested. Configure Amazon S3 credentials with a custom endpoint and SigV4, then start with COPY to a dedicated folder. Validate PUSH and PULL to a separate dataset with size and SHA-256. SYNC can delete destination files and MOVE deletes source files. diff --git a/static/llms.txt b/static/llms.txt index 7597343..c7780a7 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -181,6 +181,13 @@ Critical integration notes for developers: - [rclone](https://docs.fil.one/integrations/data-movement/rclone): Not currently tested; configuration plus upload, restore, checksum, and cleanup validation - [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): Not currently tested; bookmark, mount, restore, and troubleshooting guidance - [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): Not currently tested; links to the maintained CLI and SDK guides without duplicating them +- [Acronis Cyber Protect](https://docs.fil.one/integrations/backup/acronis-cyber-protect): Not currently tested; S3-compatible storage, backup, restore, and integrity validation +- [Comet Backup](https://docs.fil.one/integrations/backup/comet-backup): Not currently tested; Storage Vault setup and restore validation +- [MSP360](https://docs.fil.one/integrations/backup/msp360): Not currently tested; S3-compatible storage account and recovery workflow +- [NAKIVO](https://docs.fil.one/integrations/backup/nakivo): Not currently tested; repository, recovery, and verification guidance +- [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): Not currently tested; custom S3 task and download recovery +- [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): Not currently tested; HBS storage space, backup, and restore +- [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): Not currently tested; credential, push, pull, and integrity workflow ## Optional From 89c866dbca1f107c05d5c170d23a2879461976ab Mon Sep 17 00:00:00 2001 From: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:59:02 -0400 Subject: [PATCH 3/6] docs(integrations): add Kubernetes, DevOps, SIEM, and logging guides Signed-off-by: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> --- docs/integrations/devops/harbor.mdx | 43 +++++++++++++++ docs/integrations/devops/terraform.mdx | 54 +++++++++++++++++++ docs/integrations/devops/velero.mdx | 42 +++++++++++++++ docs/integrations/index.mdx | 5 ++ .../logging/elasticsearch-snapshots.mdx | 43 +++++++++++++++ .../logging/splunk-smartstore.mdx | 41 ++++++++++++++ sidebars.js | 17 ++++++ static/llms-full.txt | 30 +++++++++++ static/llms.txt | 5 ++ 9 files changed, 280 insertions(+) create mode 100644 docs/integrations/devops/harbor.mdx create mode 100644 docs/integrations/devops/terraform.mdx create mode 100644 docs/integrations/devops/velero.mdx create mode 100644 docs/integrations/logging/elasticsearch-snapshots.mdx create mode 100644 docs/integrations/logging/splunk-smartstore.mdx diff --git a/docs/integrations/devops/harbor.mdx b/docs/integrations/devops/harbor.mdx new file mode 100644 index 0000000..3c79311 --- /dev/null +++ b/docs/integrations/devops/harbor.mdx @@ -0,0 +1,43 @@ +--- +title: Harbor +description: Configure Harbor registry storage on a Fil One S3-compatible bucket. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Harbor + +Harbor stores image layers in Fil One through the registry's `s3` storage driver, with `regionendpoint` set to the regional endpoint, `forcepathstyle: true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`regionendpoint`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`forcepathstyle: true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`v4auth: true`) | +| TLS | `secure: true` | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Harbor version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Set Harbor's registry storage driver to `s3`. Provide the existing bucket, `regionendpoint`, `region`, `secure: true`, and `forcepathstyle: true`. Store credentials in the deployment's secret mechanism. Use a dedicated bucket or prefix so registry garbage collection cannot affect unrelated objects. Back up Harbor's database separately; the registry bucket holds only image layers. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Push a disposable image with several layers, remove the local image, and pull it again. Compare the image digest and start a container. +2. Inspect Harbor and registry logs. +3. Delete the test artifact, run garbage collection in dry-run mode first, then confirm only unreferenced test layers are removed. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Redirects or bucket-subdomain requests | Set `forcepathstyle: true`. | +| `403` | Key region does not match the endpoint, or the key is not scoped to the bucket. | +| Multipart or garbage-collection failures | Rerun the validation with the exact Harbor release before production rollout. | diff --git a/docs/integrations/devops/terraform.mdx b/docs/integrations/devops/terraform.mdx new file mode 100644 index 0000000..436e18f --- /dev/null +++ b/docs/integrations/devops/terraform.mdx @@ -0,0 +1,54 @@ +--- +title: Terraform S3 backend +description: Evaluate a Fil One bucket for Terraform remote state. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Terraform S3 backend + +Terraform's `s3` backend stores state in Fil One with `endpoints { s3 = ... }` set to the regional endpoint and `use_path_style = true` (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`endpoints { s3 = ... }`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`use_path_style = true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied outside the configuration | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Terraform version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Use the `s3` backend with an existing bucket, a dedicated state key, the regional endpoint, matching region, and path-style addressing. Keep credentials outside the Terraform configuration. Evaluate the backend's non-AWS options `skip_credentials_validation`, `skip_requesting_account_id`, `skip_metadata_api_check`, and `skip_s3_checksum` for your Terraform release, and enable only the ones that release needs. + +```hcl +terraform { + backend "s3" { + bucket = "BUCKET_NAME" + key = "terraform/ENVIRONMENT.tfstate" + region = "REGION" + endpoints = { s3 = "ENDPOINT_URL" } + use_path_style = true + } +} +``` + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Initialize against a disposable configuration, apply it, and confirm a subsequent `terraform plan` is empty. +2. Copy state to an offline recovery location, remove the local working directory, reinitialize, and confirm state can be read. +3. Simulate two concurrent writers before relying on locking. Test any S3 lock-file option with the exact Terraform version and region before enabling it. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Initialization contacts an AWS hostname | Check the custom endpoint and path-style options supported by your Terraform release. | +| Account or credential lookups fail | Evaluate the `skip_*` options listed above. | +| Backend migration | Preserve an offline state backup before migrating. | diff --git a/docs/integrations/devops/velero.mdx b/docs/integrations/devops/velero.mdx new file mode 100644 index 0000000..0a11cad --- /dev/null +++ b/docs/integrations/devops/velero.mdx @@ -0,0 +1,42 @@ +--- +title: Velero +description: Configure Velero object storage for Kubernetes backup and restore with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Velero + +Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets `s3Url` to the regional endpoint, `s3ForcePathStyle: "true"`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3Url`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (`s3ForcePathStyle: "true"`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through a Kubernetes Secret | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Velero version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Install Velero with its AWS object-store plugin. Configure a backup storage location with the existing bucket, `region`, `s3Url`, and `s3ForcePathStyle: "true"`. Supply the key through a Kubernetes Secret rather than a manifest committed to source control. Volume-snapshot support is separate from object storage and requires its own provider. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Create a disposable namespace containing representative objects and persistent data. +2. Run a backup, inspect `velero backup describe --details` and the logs, delete the namespace, then restore it. +3. Verify resource counts, application data, and file hashes. Confirm that expiration or cleanup affects only the test backup. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Storage location unavailable | Endpoint, region, secret, bucket scope, or `s3ForcePathStyle` is wrong. | +| Version mismatch | Check Velero, plugin, and Kubernetes compatibility as a version set. | +| Metadata restores but volumes do not | Volume recovery needs a snapshot or file-system backup provider; test it separately. | diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index ef4f7e5..5a2a64c 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -31,3 +31,8 @@ Start with [Connection settings](/integrations/connection-settings) and the [val | NAS and file services | [Synology Cloud Sync](/integrations/nas/synology-cloud-sync) | Not currently tested | | NAS and file services | [QNAP HBS](/integrations/nas/qnap-hbs) | Not currently tested | | NAS and file services | [TrueNAS Cloud Sync](/integrations/nas/truenas-cloud-sync) | Not currently tested | +| Kubernetes and DevOps | [Velero](/integrations/devops/velero) | Not currently tested | +| Kubernetes and DevOps | [Harbor](/integrations/devops/harbor) | Not currently tested | +| Kubernetes and DevOps | [Terraform S3 backend](/integrations/devops/terraform) | Not currently tested | +| SIEM and logging | [Elasticsearch searchable snapshots](/integrations/logging/elasticsearch-snapshots) | Not currently tested | +| SIEM and logging | [Splunk SmartStore](/integrations/logging/splunk-smartstore) | Not currently tested | diff --git a/docs/integrations/logging/elasticsearch-snapshots.mdx b/docs/integrations/logging/elasticsearch-snapshots.mdx new file mode 100644 index 0000000..e870463 --- /dev/null +++ b/docs/integrations/logging/elasticsearch-snapshots.mdx @@ -0,0 +1,43 @@ +--- +title: Elasticsearch searchable snapshots +description: Configure Elasticsearch S3 snapshots with Fil One for recovery testing. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Elasticsearch searchable snapshots + +Elasticsearch stores snapshots in Fil One through an `s3` repository whose named client sets `endpoint`, `region`, and `path_style_access: true`, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes `repository-s3`; on 7.x install the plugin on every node. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3.client.CLIENT.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`s3.client.CLIENT.region`) | +| Addressing | Path style (`s3.client.CLIENT.path_style_access: true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, stored in the keystore as `s3.client.CLIENT.access_key` and `s3.client.CLIENT.secret_key` | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Elasticsearch version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Store the access and secret keys in the Elasticsearch keystore under a named S3 client. Configure that client's endpoint, region, and path-style access, then register an `s3` snapshot repository using an existing bucket and a dedicated `base_path`. Run the repository verification API from every master and data node before taking a snapshot. Searchable snapshots require an Enterprise subscription; ordinary snapshot storage does not. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Create a disposable index with known documents, take a snapshot, delete the index, and restore it under a new name. +2. Compare document counts and selected content hashes. +3. If searchable snapshots are in scope, mount the snapshot and execute representative queries. +4. Test cleanup only against the dedicated repository path. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Repository verification fails on some nodes | Confirm every node can reach the endpoint and has the same client settings and keystore entries. | +| Endpoint syntax errors | Review the endpoint format for your Elasticsearch release. | +| Verification passes but restore fails | Repository verification checks connectivity only; rerun the snapshot and restore steps. | diff --git a/docs/integrations/logging/splunk-smartstore.mdx b/docs/integrations/logging/splunk-smartstore.mdx new file mode 100644 index 0000000..0e2a555 --- /dev/null +++ b/docs/integrations/logging/splunk-smartstore.mdx @@ -0,0 +1,41 @@ +--- +title: Splunk SmartStore +description: Evaluate Fil One as remote storage for Splunk SmartStore. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Splunk SmartStore + +Splunk SmartStore uses Fil One as a `remote` volume in `indexes.conf`, with `remote.s3.endpoint` set to the regional endpoint, `remote.s3.auth_region` set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`remote.s3.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`remote.s3.auth_region`) | +| Addressing | Path style (`remote.s3.url_version = v1`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`remote.s3.signature_version = v4`) | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Splunk Enterprise version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Define a `remote` volume in `indexes.conf` with the existing bucket, regional endpoint, auth region, path-style URL mode, and versioning behavior appropriate to the bucket. Reference the volume from each intended index with `remotePath`. Store credentials through the deployment's protected configuration workflow and apply the settings consistently across an indexer cluster. SmartStore licensing and topology requirements apply. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Index disposable events, roll data to remote storage, and confirm searches can fetch evicted buckets. +2. Restart or replace a test indexer and verify remote recovery. Compare event counts and representative hashes or fields. +3. Exercise cache eviction under a bounded test load before choosing concurrency and cache-size settings. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Remote errors | Use `splunk btool` to inspect effective configuration; check endpoint, auth region, URL style, clock skew, and credential scope. | +| Uploads succeed but searches fail after eviction | Review indexer and cache-manager logs and rerun the search-after-eviction step. | diff --git a/sidebars.js b/sidebars.js index 53f9769..98e973b 100644 --- a/sidebars.js +++ b/sidebars.js @@ -111,6 +111,23 @@ const sidebars = { "integrations/nas/truenas-cloud-sync", ], }, + { + type: "category", + label: "Kubernetes and DevOps", + items: [ + "integrations/devops/velero", + "integrations/devops/harbor", + "integrations/devops/terraform", + ], + }, + { + type: "category", + label: "SIEM and logging", + items: [ + "integrations/logging/elasticsearch-snapshots", + "integrations/logging/splunk-smartstore", + ], + }, ], }, "faq", diff --git a/static/llms-full.txt b/static/llms-full.txt index 7bf528c..dfc1e52 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2984,3 +2984,33 @@ Validation status: Not currently tested. Create an S3 Compatible HBS 3 Storage S Source: https://docs.fil.one/integrations/nas/truenas-cloud-sync Validation status: Not currently tested. Configure Amazon S3 credentials with a custom endpoint and SigV4, then start with COPY to a dedicated folder. Validate PUSH and PULL to a separate dataset with size and SHA-256. SYNC can delete destination files and MOVE deletes source files. + +# Velero + +Source: https://docs.fil.one/integrations/devops/velero + +Validation status: Not currently tested. Configure the AWS object-store plugin with an existing bucket, regional endpoint and region, path-style access, and credentials in a Kubernetes Secret. Validate a disposable namespace backup, deletion, restore, resources, data hashes, logs, and scoped cleanup. Volume snapshots are a separate provider concern. This is not a supported-provider or certification claim. + +# Harbor + +Source: https://docs.fil.one/integrations/devops/harbor + +Validation status: Not currently tested. Configure the s3 registry driver with a dedicated bucket or prefix, regional endpoint, region, TLS, path style, and protected credentials. Validate image push, local deletion, pull, digest, container start, and garbage collection. Harbor database backup is separate. + +# Terraform S3 backend + +Source: https://docs.fil.one/integrations/devops/terraform + +Validation status: Not currently tested. Configure an existing bucket, dedicated state key, custom endpoint, matching region, and path style without embedding credentials. Validate init, apply, empty plan, offline state backup, reinitialization, recovery, and concurrent writers. Do not enable S3 lock-file behavior until conditional writes are proven for the exact release and region. + +# Elasticsearch searchable snapshots + +Source: https://docs.fil.one/integrations/logging/elasticsearch-snapshots + +Validation status: Not currently tested. Install repository-s3 where required, keep credentials in the keystore, configure endpoint, region, and path style, and register a dedicated base_path. Validate repository verification, snapshot, index deletion, restore, counts and hashes. Searchable snapshots have separate license and mount validation. + +# Splunk SmartStore + +Source: https://docs.fil.one/integrations/logging/splunk-smartstore + +Validation status: Not currently tested. Define remote storage with the bucket, endpoint, auth region, path-style URL mode, and protected credentials, then reference it from intended indexes. Validate upload, eviction, remote-backed search, test-indexer replacement, recovery, event counts, and bounded cache behavior. Licensing and topology requirements apply. diff --git a/static/llms.txt b/static/llms.txt index c7780a7..5e8929f 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -188,6 +188,11 @@ Critical integration notes for developers: - [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): Not currently tested; custom S3 task and download recovery - [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): Not currently tested; HBS storage space, backup, and restore - [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): Not currently tested; credential, push, pull, and integrity workflow +- [Velero](https://docs.fil.one/integrations/devops/velero): Not currently tested; Kubernetes backup, restore, and volume-scope validation +- [Harbor](https://docs.fil.one/integrations/devops/harbor): Not currently tested; registry push, pull, digest, and garbage-collection guidance +- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): Not currently tested; state migration, recovery, and concurrency cautions +- [Elasticsearch snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): Not currently tested; repository verification, restore, and searchable-snapshot validation +- [Splunk SmartStore](https://docs.fil.one/integrations/logging/splunk-smartstore): Not currently tested; remote storage, eviction, search, and recovery validation ## Optional From 4ac781aa52c628497d488978e1ff9bd1abb39f90 Mon Sep 17 00:00:00 2001 From: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> Date: Tue, 18 Aug 2026 20:59:02 -0400 Subject: [PATCH 4/6] docs(integrations): add data and AI integration guides Signed-off-by: James Kurz <255140170+jameskurz-filecoin@users.noreply.github.com> --- docs/integrations/data-ai/apache-iceberg.mdx | 42 ++++++++++ docs/integrations/data-ai/databricks.mdx | 41 ++++++++++ docs/integrations/data-ai/delta-lake.mdx | 41 ++++++++++ docs/integrations/data-ai/duckdb.mdx | 55 ++++++++++++++ .../data-ai/hugging-face-datasets.mdx | 41 ++++++++++ docs/integrations/data-ai/ray-data.mdx | 41 ++++++++++ docs/integrations/index.mdx | 6 ++ sidebars.js | 13 ++++ static/llms-full.txt | 76 ++++++++++++++----- static/llms.txt | 21 +++-- 10 files changed, 353 insertions(+), 24 deletions(-) create mode 100644 docs/integrations/data-ai/apache-iceberg.mdx create mode 100644 docs/integrations/data-ai/databricks.mdx create mode 100644 docs/integrations/data-ai/delta-lake.mdx create mode 100644 docs/integrations/data-ai/duckdb.mdx create mode 100644 docs/integrations/data-ai/hugging-face-datasets.mdx create mode 100644 docs/integrations/data-ai/ray-data.mdx diff --git a/docs/integrations/data-ai/apache-iceberg.mdx b/docs/integrations/data-ai/apache-iceberg.mdx new file mode 100644 index 0000000..7d40072 --- /dev/null +++ b/docs/integrations/data-ai/apache-iceberg.mdx @@ -0,0 +1,42 @@ +--- +title: Apache Iceberg +description: Evaluate Apache Iceberg object storage workflows with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Apache Iceberg + +Apache Iceberg reads and writes table data in Fil One through `S3FileIO`, with `s3.endpoint` set to the regional endpoint, `s3.path-style-access=true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`s3.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`client.region`) | +| Addressing | Path style (`s3.path-style-access=true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the engine's secret mechanism | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Apache Iceberg version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Use Iceberg's `S3FileIO` when your engine supports it. Configure the warehouse on an existing bucket, the regional endpoint and region, path-style access, and credentials through the engine's secret mechanism. Catalog configuration is separate: REST, JDBC, Hive, Glue-compatible, and filesystem catalogs have different correctness and concurrency properties. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Create a disposable table, append and update data, and read it from a fresh session. +2. Validate partition pruning and inspect metadata and manifest objects under the dedicated warehouse prefix. +3. Run concurrent commits and confirm one writer does not silently overwrite another. +4. Preserve the metadata tree, recover the table from its catalog, and compare row counts and content hashes. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Some components reach an AWS hostname | An engine may use more than one S3 client; apply endpoint and path-style settings to every execution component. | +| Reads work but commits fail | Check the catalog configuration and the engine and catalog versions. | diff --git a/docs/integrations/data-ai/databricks.mdx b/docs/integrations/data-ai/databricks.mdx new file mode 100644 index 0000000..8c9197e --- /dev/null +++ b/docs/integrations/data-ai/databricks.mdx @@ -0,0 +1,41 @@ +--- +title: Databricks +description: Evaluate Databricks access to data stored in Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Databricks + +Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region | +| Region | `eu-west-1` or `us-east-1` | +| Addressing | Path style | +| Authentication | SigV4 access key and secret scoped to the bucket's region, stored in a Databricks secret scope | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Databricks Runtime version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Use a cluster or SQL environment that permits a custom S3-compatible endpoint. Store credentials in a Databricks secret scope and inject them into the supported filesystem or connector configuration. Set the regional endpoint, region, and path-style access from [Connection settings](/integrations/connection-settings). Keep secret values out of notebooks and cluster policies. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Read a disposable Parquet dataset, write results to a dedicated prefix, terminate the compute, and read the output from a fresh cluster. +2. Compare row counts, schema, and deterministic aggregates. +3. For managed table formats, validate their transaction and concurrency behavior separately. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Serverless or governed mode blocks custom endpoints | Use a compute mode that permits them, or treat the integration as unavailable for that workspace. | +| Driver works but jobs fail | Confirm that executors as well as the driver receive endpoint and credential settings. | diff --git a/docs/integrations/data-ai/delta-lake.mdx b/docs/integrations/data-ai/delta-lake.mdx new file mode 100644 index 0000000..c2cc8e1 --- /dev/null +++ b/docs/integrations/data-ai/delta-lake.mdx @@ -0,0 +1,41 @@ +--- +title: Delta Lake +description: Evaluate Delta Lake transaction-log workflows on Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Delta Lake + +Delta Lake on Spark reads and writes Fil One through the `s3a` connector, with `fs.s3a.endpoint` set to the regional endpoint, `fs.s3a.path.style.access=true`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`fs.s3a.endpoint`) | +| Region | `eu-west-1` or `us-east-1` (`fs.s3a.endpoint.region`) | +| Addressing | Path style (`fs.s3a.path.style.access=true`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the platform's secret mechanism | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Delta Lake version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Configure Spark's S3 connector with the existing bucket, regional endpoint and region, path-style access, and credentials supplied through the platform's secret mechanism. Keep a dedicated table prefix. Ensure the driver and every executor receive the same settings. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Create a disposable Delta table, append and update rows, produce a checkpoint, and read the table from a fresh session. +2. Run concurrent writers and verify conflicts do not cause silent data loss. +3. Restore a prior table version after confirming the required log and data objects exist, then compare row counts and deterministic hashes. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Parquet reads work but Delta commits fail | Connector retry, rename, conditional-write, and consistency assumptions vary by release; check the Delta, Spark, and connector version set. | +| Executors reach an AWS hostname | Apply the endpoint and path-style settings to executors as well as the driver. | diff --git a/docs/integrations/data-ai/duckdb.mdx b/docs/integrations/data-ai/duckdb.mdx new file mode 100644 index 0000000..e6c6428 --- /dev/null +++ b/docs/integrations/data-ai/duckdb.mdx @@ -0,0 +1,55 @@ +--- +title: DuckDB +description: Read and write Fil One objects with DuckDB httpfs. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# DuckDB + +DuckDB reads and writes Fil One through the `httpfs` extension with an S3 secret that sets `ENDPOINT` to the regional hostname, `URL_STYLE 'path'`, and a key scoped to the bucket's region. + +| Setting | Value | +| --- | --- | +| Endpoint | `eu-west-1.s3.fil.one` or `us-east-1.s3.fil.one`, matching the bucket's region (`ENDPOINT`, hostname without `https://`, with `USE_SSL true`) | +| Region | `eu-west-1` or `us-east-1` (`REGION`) | +| Addressing | Path style (`URL_STYLE 'path'`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`KEY_ID` and `SECRET`) | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named DuckDB version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Install and load DuckDB's `httpfs` extension. Create an S3 secret with the regional key, region, endpoint hostname, `USE_SSL true`, and `URL_STYLE 'path'`. Use a secret provider appropriate to your environment and keep credentials out of SQL files. + +```sql +INSTALL httpfs; +LOAD httpfs; +CREATE SECRET filone ( + TYPE S3, + KEY_ID 'ACCESS_KEY', + SECRET 'SECRET_KEY', + REGION 'eu-west-1', + ENDPOINT 'eu-west-1.s3.fil.one', + USE_SSL true, + URL_STYLE 'path' +); +``` + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Read a disposable Parquet file from the bucket and compare row count and deterministic aggregates. +2. Use `COPY` to write a result under a unique prefix, start a fresh DuckDB process, and read the output back. Compare schema, row count, byte size, and content-level checks before removing the prefix. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Requests target a bucket subdomain | Set `URL_STYLE 'path'`. | +| Endpoint rejected | The `ENDPOINT` field expects a hostname rather than a full URL. | +| Large datasets | Test range reads and multipart writes separately. | diff --git a/docs/integrations/data-ai/hugging-face-datasets.mdx b/docs/integrations/data-ai/hugging-face-datasets.mdx new file mode 100644 index 0000000..9caee1d --- /dev/null +++ b/docs/integrations/data-ai/hugging-face-datasets.mdx @@ -0,0 +1,41 @@ +--- +title: Hugging Face Datasets +description: Evaluate Hugging Face Datasets access to files stored in Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Hugging Face Datasets + +Hugging Face `datasets` reads files in Fil One through `fsspec` and `s3fs`, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`client_kwargs={"endpoint_url": ...}`) | +| Region | `eu-west-1` or `us-east-1` (`client_kwargs={"region_name": ...}`) | +| Addressing | Path style (`config_kwargs={"s3": {"addressing_style": "path"}}`) | +| Authentication | SigV4 access key and secret scoped to the bucket's region (`key` and `secret`) | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Hugging Face Datasets version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Pass the regional endpoint, region, matching credentials, and path-style behavior through the storage options supported by the installed `s3fs` release. Load credentials from a secret store or environment and keep them out of notebooks. Use explicit object paths under an existing bucket. Pin and record the complete Python package set, because endpoint and addressing options can change across `datasets`, `fsspec`, and `s3fs`. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Load a disposable dataset normally and in streaming mode. Compare feature schema, row count, and deterministic sample hashes. +2. Clear the local cache and load it again to confirm the remote path is sufficient. +3. If preprocessing writes cache or transformed data remotely, isolate that output under a unique prefix and validate its cleanup. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Load works once, then fails | A local cache can mask remote errors; clear the cache and repeat. | +| Requests target a bucket subdomain | Set the path-style addressing option for the installed `s3fs` release. | diff --git a/docs/integrations/data-ai/ray-data.mdx b/docs/integrations/data-ai/ray-data.mdx new file mode 100644 index 0000000..95ef322 --- /dev/null +++ b/docs/integrations/data-ai/ray-data.mdx @@ -0,0 +1,41 @@ +--- +title: Ray Data +description: Evaluate distributed Ray Data reads and writes with Fil One. +integration_status: Not currently tested +tested_versions: [] +verified_regions: [] +verification_date: null +evidence: null +--- + +# Ray Data + +Ray Data reads and writes Fil One through a PyArrow `S3FileSystem` with `endpoint_override` set to the regional endpoint, `region` set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. + +| Setting | Value | +| --- | --- | +| Endpoint | `https://eu-west-1.s3.fil.one` or `https://us-east-1.s3.fil.one`, matching the bucket's region (`endpoint_override`) | +| Region | `eu-west-1` or `us-east-1` (`region`) | +| Addressing | Path style (PyArrow uses path style with `endpoint_override` unless `force_virtual_addressing` is set) | +| Authentication | SigV4 access key and secret scoped to the bucket's region, distributed through the cluster's secret mechanism | + +Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Ray version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. + +## Configure + +Create the S3 filesystem with the regional endpoint, region, path-style access, and matching credentials. Distribute credentials through the cluster's secret mechanism and confirm every worker receives the endpoint configuration. Use a dedicated input and output prefix. + +## Validate + +Follow the [validation round trip](/integrations/validate-an-integration), then: + +1. Read a disposable multi-file Parquet dataset across several workers, apply a deterministic transformation, and write the result to a unique prefix. +2. Start a fresh Ray job, read the output, and compare schema, row count, partition count where relevant, and deterministic aggregates. +3. Exercise worker loss and retry under a bounded test before relying on recovery. + +## Troubleshooting + +| Symptom | Check | +| --- | --- | +| Driver succeeds, workers fail | Inspect worker logs for bucket-subdomain requests, credential errors, and retry loops. | +| Results differ between runs | Validate the exact Ray and PyArrow versions at production concurrency. | diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index 5a2a64c..22330e1 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -36,3 +36,9 @@ Start with [Connection settings](/integrations/connection-settings) and the [val | Kubernetes and DevOps | [Terraform S3 backend](/integrations/devops/terraform) | Not currently tested | | SIEM and logging | [Elasticsearch searchable snapshots](/integrations/logging/elasticsearch-snapshots) | Not currently tested | | SIEM and logging | [Splunk SmartStore](/integrations/logging/splunk-smartstore) | Not currently tested | +| Data and AI | [Apache Iceberg](/integrations/data-ai/apache-iceberg) | Not currently tested | +| Data and AI | [Databricks](/integrations/data-ai/databricks) | Not currently tested | +| Data and AI | [Delta Lake](/integrations/data-ai/delta-lake) | Not currently tested | +| Data and AI | [DuckDB](/integrations/data-ai/duckdb) | Not currently tested | +| Data and AI | [Hugging Face Datasets](/integrations/data-ai/hugging-face-datasets) | Not currently tested | +| Data and AI | [Ray Data](/integrations/data-ai/ray-data) | Not currently tested | diff --git a/sidebars.js b/sidebars.js index 98e973b..22740c5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -83,6 +83,7 @@ const sidebars = { items: [ "integrations/index", "integrations/connection-settings", + "integrations/validate-an-integration", { type: "category", label: "Data movement", @@ -128,6 +129,18 @@ const sidebars = { "integrations/logging/splunk-smartstore", ], }, + { + type: "category", + label: "Data and AI", + items: [ + "integrations/data-ai/apache-iceberg", + "integrations/data-ai/databricks", + "integrations/data-ai/delta-lake", + "integrations/data-ai/duckdb", + "integrations/data-ai/hugging-face-datasets", + "integrations/data-ai/ray-data", + ], + }, ], }, "faq", diff --git a/static/llms-full.txt b/static/llms-full.txt index dfc1e52..3c4a8dc 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2917,100 +2917,142 @@ Intelligence deliberately per bucket rather than across everything. Source: https://docs.fil.one/integrations -Integration guides are compatibility guidance, not vendor certification. Verified requires a named tool version, end-to-end workflow, restore, integrity check, evidence link, verification date, and coverage in every listed region. Other states are Compatible, Preview, Not currently tested, and Not currently supported. +Each integration guide gives the connection settings, a validation procedure, and troubleshooting for one tool. Statuses: Verified (named tool version completed the end-to-end workflow, restore, and integrity check in every listed region, with evidence and date), Compatible (an end-to-end run exists for a named version, but not for every region or not within the last 12 months), Preview (runnable validation path, incomplete coverage), Not currently tested (no end-to-end run recorded for any version or region), Not currently supported (a known product or addressing requirement blocks the integration). All 21 published guides are Not currently tested. The index page lists every guide with its status. # Integration connection settings Source: https://docs.fil.one/integrations/connection-settings -Use the endpoint in the bucket's Access endpoints panel and keep it in configuration. Endpoint, region, API key, and bucket must belong to the same region. Use SigV4, path-style addressing, HTTPS, and certificate validation. Never put secret keys in docs, screenshots, source control, or command-line history. Current API behavior lives in the S3 Compatibility reference. +Endpoint: https://eu-west-1.s3.fil.one or https://us-east-1.s3.fil.one, matching the bucket's region (also shown in the bucket's Access endpoints panel). Region: eu-west-1 or us-east-1. Authentication: SigV4 with a key created for the same region and scoped to the bucket. Addressing: path style required. Endpoint, region, key, and bucket must belong to the same region. Keep TLS certificate validation enabled and secret keys out of docs, screenshots, source control, and command-line history. Feature availability differs between the S3 API and the app.fil.one dashboard; the S3 Compatibility reference covers API behavior. + +# Validate an integration + +Source: https://docs.fil.one/integrations/validate-an-integration + +The round trip every guide links to: create a fixture with a small file, a nested file, and a file large enough to trigger multipart upload; copy it to a unique disposable prefix; list the prefix; remove the local copy; restore to a new directory; compare byte size and SHA-256 for every file; delete only that prefix and confirm it is empty. Run it with disposable data before pointing production workloads at the bucket. Record tool version, region, and date with the result. # rclone Source: https://docs.fil.one/integrations/data-movement/rclone -Validation status: Not currently tested. Configure an Other S3 provider with the bucket's endpoint and region, path-style access, and prompted or secret-store credentials. Validate a small, nested, and multipart-sized fixture under a unique prefix; list it, delete the local copy, restore, compare size and SHA-256, run rclone check --download, and delete only that prefix. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named rclone version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. rclone connects to Fil One as an S3 remote with provider = Other, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Cyberduck and Mountain Duck Source: https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck -Validation status: Not currently tested. Use an Amazon S3 bookmark with the endpoint hostname, matching regional key, bucket path, HTTPS, and certificate validation. Validate nested upload, download, SHA-256, rename, delete, and recovery with disposable data. Mountain Duck uses the same bookmark settings for a desktop mount. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Cyberduck or Mountain Duck version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. Configuration, tool-specific validation steps, and troubleshooting are on the page. # AWS CLI and SDKs Source: https://docs.fil.one/integrations/data-movement/aws-cli-sdk -Validation status: Not currently tested. Use the maintained AWS CLI, boto3, JavaScript v3, and Go SDK pages rather than duplicated examples. Validate small and automatic-multipart uploads, listing, restore, SHA-256 and byte size, and exact-prefix cleanup. boto3 is the reference API workflow, not backup-product certification. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named AWS CLI or SDK version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. The AWS CLI and SDKs are Fil One's reference clients; see the SDK guides for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Acronis Cyber Protect Source: https://docs.fil.one/integrations/backup/acronis-cyber-protect -Validation status: Not currently tested. Add Public cloud storage using S3 compatible, the regional endpoint and key, AuthV4, and an existing bucket. Validate a disposable backup, local deletion, restore, SHA-256 and size, and scoped cleanup. This is not Acronis certification; validate any immutability option separately. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Acronis Cyber Protect version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Comet Backup Source: https://docs.fil.one/integrations/backup/comet-backup -Validation status: Not currently tested. Configure an S3-compatible Storage Template or Custom Storage Vault with encrypted transmission. Complete Test Connection, backup, restore to a new directory, SHA-256 and size comparison, native vault checking, and isolated cleanup. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Comet Backup version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # MSP360 Source: https://docs.fil.one/integrations/backup/msp360 -Validation status: Not currently tested. Add an S3 Compatible storage account with the regional HTTPS endpoint, SigV4, certificate validation, and existing bucket. Validate small and multipart-sized backup, restore, hashes, and exact test-set cleanup. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named MSP360 version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # NAKIVO Backup and Replication Source: https://docs.fil.one/integrations/backup/nakivo -Validation status: Not currently tested. Configure an S3-compatible repository with matching endpoint, region, credentials, bucket, and TLS validation. Validate file-level recovery and repository verification using disposable data before relying on multipart, retention, or immutability behavior. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named NAKIVO Backup and Replication version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Synology Cloud Sync Source: https://docs.fil.one/integrations/nas/synology-cloud-sync -Validation status: Not currently tested. Configure S3 Storage with a custom server URL, SigV4, and an existing bucket. Validate a dedicated-prefix sync and download-only recovery with SHA-256 and size. Test deletion semantics before enabling bidirectional sync. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Synology Cloud Sync version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # QNAP HBS Source: https://docs.fil.one/integrations/nas/qnap-hbs -Validation status: Not currently tested. Create an S3 Compatible HBS 3 Storage Space with SSL, certificate validation, SigV4, and an existing bucket. Back up disposable data, restore to a new folder, compare hashes, inspect logs, and keep retention or sync deletion inside the validation prefix. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named QNAP HBS version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # TrueNAS Cloud Sync Source: https://docs.fil.one/integrations/nas/truenas-cloud-sync -Validation status: Not currently tested. Configure Amazon S3 credentials with a custom endpoint and SigV4, then start with COPY to a dedicated folder. Validate PUSH and PULL to a separate dataset with size and SHA-256. SYNC can delete destination files and MOVE deletes source files. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named TrueNAS version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Velero Source: https://docs.fil.one/integrations/devops/velero -Validation status: Not currently tested. Configure the AWS object-store plugin with an existing bucket, regional endpoint and region, path-style access, and credentials in a Kubernetes Secret. Validate a disposable namespace backup, deletion, restore, resources, data hashes, logs, and scoped cleanup. Volume snapshots are a separate provider concern. This is not a supported-provider or certification claim. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Velero version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets s3Url to the regional endpoint, s3ForcePathStyle: "true", and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Harbor Source: https://docs.fil.one/integrations/devops/harbor -Validation status: Not currently tested. Configure the s3 registry driver with a dedicated bucket or prefix, regional endpoint, region, TLS, path style, and protected credentials. Validate image push, local deletion, pull, digest, container start, and garbage collection. Harbor database backup is separate. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Harbor version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Harbor stores image layers in Fil One through the registry's s3 storage driver, with regionendpoint set to the regional endpoint, forcepathstyle: true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Terraform S3 backend Source: https://docs.fil.one/integrations/devops/terraform -Validation status: Not currently tested. Configure an existing bucket, dedicated state key, custom endpoint, matching region, and path style without embedding credentials. Validate init, apply, empty plan, offline state backup, reinitialization, recovery, and concurrent writers. Do not enable S3 lock-file behavior until conditional writes are proven for the exact release and region. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Terraform version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Terraform's s3 backend stores state in Fil One with endpoints { s3 = ... } set to the regional endpoint and use_path_style = true (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Elasticsearch searchable snapshots Source: https://docs.fil.one/integrations/logging/elasticsearch-snapshots -Validation status: Not currently tested. Install repository-s3 where required, keep credentials in the keystore, configure endpoint, region, and path style, and register a dedicated base_path. Validate repository verification, snapshot, index deletion, restore, counts and hashes. Searchable snapshots have separate license and mount validation. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Elasticsearch version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Elasticsearch stores snapshots in Fil One through an s3 repository whose named client sets endpoint, region, and path_style_access: true, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes repository-s3; on 7.x install the plugin on every node. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Splunk SmartStore Source: https://docs.fil.one/integrations/logging/splunk-smartstore -Validation status: Not currently tested. Define remote storage with the bucket, endpoint, auth region, path-style URL mode, and protected credentials, then reference it from intended indexes. Validate upload, eviction, remote-backed search, test-indexer replacement, recovery, event counts, and bounded cache behavior. Licensing and topology requirements apply. +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Splunk Enterprise version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Splunk SmartStore uses Fil One as a remote volume in indexes.conf, with remote.s3.endpoint set to the regional endpoint, remote.s3.auth_region set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Apache Iceberg + +Source: https://docs.fil.one/integrations/data-ai/apache-iceberg + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Apache Iceberg version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Apache Iceberg reads and writes table data in Fil One through S3FileIO, with s3.endpoint set to the regional endpoint, s3.path-style-access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Databricks + +Source: https://docs.fil.one/integrations/data-ai/databricks + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Databricks Runtime version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Delta Lake + +Source: https://docs.fil.one/integrations/data-ai/delta-lake + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Delta Lake version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Delta Lake on Spark reads and writes Fil One through the s3a connector, with fs.s3a.endpoint set to the regional endpoint, fs.s3a.path.style.access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# DuckDB + +Source: https://docs.fil.one/integrations/data-ai/duckdb + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named DuckDB version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. DuckDB reads and writes Fil One through the httpfs extension with an S3 secret that sets ENDPOINT to the regional hostname, URL_STYLE 'path', and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Hugging Face Datasets + +Source: https://docs.fil.one/integrations/data-ai/hugging-face-datasets + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Hugging Face Datasets version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Hugging Face datasets reads files in Fil One through fsspec and s3fs, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. Configuration, tool-specific validation steps, and troubleshooting are on the page. + +# Ray Data + +Source: https://docs.fil.one/integrations/data-ai/ray-data + +Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Ray version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Ray Data reads and writes Fil One through a PyArrow S3FileSystem with endpoint_override set to the regional endpoint, region set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. Configuration, tool-specific validation steps, and troubleshooting are on the page. diff --git a/static/llms.txt b/static/llms.txt index 5e8929f..412b8fc 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -176,23 +176,30 @@ Critical integration notes for developers: ## Integrations -- [Integrations](https://docs.fil.one/integrations): Validation status definitions and integration guide index -- [Integration connection settings](https://docs.fil.one/integrations/connection-settings): Central endpoint, region, SigV4, path-style, and credential guidance -- [rclone](https://docs.fil.one/integrations/data-movement/rclone): Not currently tested; configuration plus upload, restore, checksum, and cleanup validation +- [Integrations](https://docs.fil.one/integrations): Validation status definitions and a table of every integration guide with its status +- [Integration connection settings](https://docs.fil.one/integrations/connection-settings): Endpoints (https://eu-west-1.s3.fil.one, https://us-east-1.s3.fil.one), region, SigV4, path-style, and credential settings +- [Validate an integration](https://docs.fil.one/integrations/validate-an-integration): The shared upload, list, restore, SHA-256, and cleanup round trip every guide links to +- [rclone](https://docs.fil.one/integrations/data-movement/rclone): Not currently tested; provider Other, force_path_style, config example, rclone check validation - [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): Not currently tested; bookmark, mount, restore, and troubleshooting guidance -- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): Not currently tested; links to the maintained CLI and SDK guides without duplicating them +- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): Not currently tested; the reference clients, with setup in the SDK guides - [Acronis Cyber Protect](https://docs.fil.one/integrations/backup/acronis-cyber-protect): Not currently tested; S3-compatible storage, backup, restore, and integrity validation - [Comet Backup](https://docs.fil.one/integrations/backup/comet-backup): Not currently tested; Storage Vault setup and restore validation - [MSP360](https://docs.fil.one/integrations/backup/msp360): Not currently tested; S3-compatible storage account and recovery workflow -- [NAKIVO](https://docs.fil.one/integrations/backup/nakivo): Not currently tested; repository, recovery, and verification guidance +- [NAKIVO Backup and Replication](https://docs.fil.one/integrations/backup/nakivo): Not currently tested; repository, recovery, and verification guidance - [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): Not currently tested; custom S3 task and download recovery - [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): Not currently tested; HBS storage space, backup, and restore - [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): Not currently tested; credential, push, pull, and integrity workflow - [Velero](https://docs.fil.one/integrations/devops/velero): Not currently tested; Kubernetes backup, restore, and volume-scope validation - [Harbor](https://docs.fil.one/integrations/devops/harbor): Not currently tested; registry push, pull, digest, and garbage-collection guidance -- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): Not currently tested; state migration, recovery, and concurrency cautions -- [Elasticsearch snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): Not currently tested; repository verification, restore, and searchable-snapshot validation +- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): Not currently tested; endpoints and use_path_style backend settings, state migration, recovery, and concurrency validation +- [Elasticsearch searchable snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): Not currently tested; s3 repository client settings, repository verification, restore, and searchable-snapshot validation - [Splunk SmartStore](https://docs.fil.one/integrations/logging/splunk-smartstore): Not currently tested; remote storage, eviction, search, and recovery validation +- [Apache Iceberg](https://docs.fil.one/integrations/data-ai/apache-iceberg): Not currently tested; S3FileIO, catalog, commits, and recovery validation +- [Databricks](https://docs.fil.one/integrations/data-ai/databricks): Not currently tested; custom-endpoint compute, read, write, and fresh-cluster recovery +- [Delta Lake](https://docs.fil.one/integrations/data-ai/delta-lake): Not currently tested; transaction log, checkpoints, concurrency, and recovery +- [DuckDB](https://docs.fil.one/integrations/data-ai/duckdb): Not currently tested; httpfs read, write, and cache-independent validation +- [Hugging Face Datasets](https://docs.fil.one/integrations/data-ai/hugging-face-datasets): Not currently tested; load, streaming, cache clearing, and package-version guidance +- [Ray Data](https://docs.fil.one/integrations/data-ai/ray-data): Not currently tested; distributed reads, writes, worker configuration, and retry validation ## Optional From 17575e7511c8e518ccb70cc6370247283cf3118f Mon Sep 17 00:00:00 2001 From: jameskurz-filecoin Date: Wed, 19 Aug 2026 12:01:33 -0400 Subject: [PATCH 5/6] fix(integrations): tighten checker nits from review Reject whitespace-only tested_versions lists, and require a Validation status line on blocked guides as well as live ones. --- scripts/check-integrations.mjs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/check-integrations.mjs b/scripts/check-integrations.mjs index 4883c59..f49df19 100644 --- a/scripts/check-integrations.mjs +++ b/scripts/check-integrations.mjs @@ -39,7 +39,7 @@ for (const file of guides) { const field = value(key); if (!field || field === "[]" || field === "null") failures.push(`${file}: Verified requires ${key}`); } - if (!/^\[[^\]]+\]$/.test(value("tested_versions") ?? "")) failures.push(`${file}: Verified versions must be a non-empty inline list`); + if (!/^\[[^\]]*\S[^\]]*\]$/.test(value("tested_versions") ?? "")) failures.push(`${file}: Verified versions must be a non-empty inline list`); if (!/^\[(eu-west-1|us-east-1)(,\s*(eu-west-1|us-east-1))*\]$/.test(value("verified_regions") ?? "")) failures.push(`${file}: Verified regions must use public region IDs`); if (!/^\d{4}-\d{2}-\d{2}$/.test(value("verification_date") ?? "")) failures.push(`${file}: Verified requires an ISO verification date`); if (!/^https:\/\//.test(value("evidence") ?? "")) failures.push(`${file}: Verified evidence must be an HTTPS link`); @@ -47,7 +47,7 @@ for (const file of guides) { if (status !== "Verified" && /(^|\W)Verified(\W|$)/.test(body)) { failures.push(`${file}: body uses Verified without Verified metadata`); } - if (status && !blocked) { + if (status) { const statusLines = body.split("\n").filter((line) => line.startsWith("Validation status: ")); if (statusLines.length !== 1 || !statusLines[0].startsWith(`Validation status: ${status}.`)) { failures.push(`${file}: body must contain exactly one line starting with "Validation status: ${status}."`); From addbfeb0504b47e84293b9918fad5fe9f20fa9c7 Mon Sep 17 00:00:00 2001 From: jameskurz-filecoin Date: Fri, 21 Aug 2026 15:54:10 -0400 Subject: [PATCH 6/6] docs(integrations): drop untested banners from public pages Setup guides should read like setup guides. Status stays in frontmatter and the checker still refuses Verified without evidence. --- .../backup/acronis-cyber-protect.mdx | 6 +- docs/integrations/backup/comet-backup.mdx | 6 +- docs/integrations/backup/msp360.mdx | 6 +- docs/integrations/backup/nakivo.mdx | 6 +- docs/integrations/data-ai/apache-iceberg.mdx | 6 +- docs/integrations/data-ai/databricks.mdx | 6 +- docs/integrations/data-ai/delta-lake.mdx | 6 +- docs/integrations/data-ai/duckdb.mdx | 6 +- .../data-ai/hugging-face-datasets.mdx | 6 +- docs/integrations/data-ai/ray-data.mdx | 6 +- .../data-movement/aws-cli-sdk.mdx | 6 +- .../data-movement/cyberduck-mountain-duck.mdx | 6 +- docs/integrations/data-movement/rclone.mdx | 14 ++--- docs/integrations/devops/harbor.mdx | 6 +- docs/integrations/devops/terraform.mdx | 6 +- docs/integrations/devops/velero.mdx | 6 +- docs/integrations/index.mdx | 60 ++++++++----------- .../logging/elasticsearch-snapshots.mdx | 6 +- .../logging/splunk-smartstore.mdx | 6 +- docs/integrations/nas/qnap-hbs.mdx | 6 +- docs/integrations/nas/synology-cloud-sync.mdx | 6 +- docs/integrations/nas/truenas-cloud-sync.mdx | 6 +- docs/integrations/validate-an-integration.mdx | 18 +++--- scripts/check-integrations.mjs | 6 +- static/llms-full.txt | 48 +++++++-------- static/llms.txt | 46 +++++++------- 26 files changed, 109 insertions(+), 203 deletions(-) diff --git a/docs/integrations/backup/acronis-cyber-protect.mdx b/docs/integrations/backup/acronis-cyber-protect.mdx index b21ad76..94db506 100644 --- a/docs/integrations/backup/acronis-cyber-protect.mdx +++ b/docs/integrations/backup/acronis-cyber-protect.mdx @@ -19,15 +19,11 @@ Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, usi | Addressing | Path style | | Authentication | SigV4 (AuthV4) access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Acronis Cyber Protect version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure In the Acronis administration console, add an S3-compatible storage location. Enter the regional endpoint, the matching access key, and AuthV4 signing from [Connection settings](/integrations/connection-settings). Select an existing Fil One bucket when prompted, then assign the location to a protection plan. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Protect the fixture directory with the plan, run it, and confirm the activity completes. 2. Restore to a new location and compare size and SHA-256 for every restored file. diff --git a/docs/integrations/backup/comet-backup.mdx b/docs/integrations/backup/comet-backup.mdx index 4b2c54a..b05dbf6 100644 --- a/docs/integrations/backup/comet-backup.mdx +++ b/docs/integrations/backup/comet-backup.mdx @@ -19,15 +19,11 @@ Comet Backup writes to Fil One through an S3-compatible Storage Vault, using Sig | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Comet Backup version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Create either a Comet Storage Template or a user's Custom Storage Vault, select the S3-compatible type, and enter the regional hostname, region, existing bucket, and matching credentials. Enable encrypted transmission. Use the same values described in [Connection settings](/integrations/connection-settings). -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Run Comet's connection test, protect the fixture directory, and complete an on-demand backup. 2. Restore to a separate directory, run Comet's vault integrity check, and inspect the job history for partial failures. diff --git a/docs/integrations/backup/msp360.mdx b/docs/integrations/backup/msp360.mdx index 4ffffb4..cfcbff3 100644 --- a/docs/integrations/backup/msp360.mdx +++ b/docs/integrations/backup/msp360.mdx @@ -19,15 +19,11 @@ MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path- | Addressing | Path style | | Authentication | SigV4 (signature version 4) access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named MSP360 version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Add an S3-compatible storage account. Put the regional URL in the HTTPS endpoint field, choose signature version 4, keep certificate validation enabled, and select an existing bucket. Enable native multipart upload after validating it with your MSP360 release. The desktop agent exposes the same core settings. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Create a test plan for the fixture directory and run it. 2. Confirm the remote objects are listed, restore to a new directory, and compare size and SHA-256. diff --git a/docs/integrations/backup/nakivo.mdx b/docs/integrations/backup/nakivo.mdx index a7110fb..b29dfbe 100644 --- a/docs/integrations/backup/nakivo.mdx +++ b/docs/integrations/backup/nakivo.mdx @@ -19,15 +19,11 @@ NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named NAKIVO Backup and Replication version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Add an S3-compatible repository. Enter the regional endpoint, region, matching credentials, and an existing bucket. Keep HTTPS certificate validation enabled. Ensure the Transporter that performs data movement can reach the endpoint. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Back up a disposable workload with known file hashes, then perform file-level recovery to a different location and compare size and SHA-256. 2. Run NAKIVO's repository verification if available. diff --git a/docs/integrations/data-ai/apache-iceberg.mdx b/docs/integrations/data-ai/apache-iceberg.mdx index 7d40072..1f5cace 100644 --- a/docs/integrations/data-ai/apache-iceberg.mdx +++ b/docs/integrations/data-ai/apache-iceberg.mdx @@ -19,15 +19,11 @@ Apache Iceberg reads and writes table data in Fil One through `S3FileIO`, with ` | Addressing | Path style (`s3.path-style-access=true`) | | Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the engine's secret mechanism | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Apache Iceberg version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Use Iceberg's `S3FileIO` when your engine supports it. Configure the warehouse on an existing bucket, the regional endpoint and region, path-style access, and credentials through the engine's secret mechanism. Catalog configuration is separate: REST, JDBC, Hive, Glue-compatible, and filesystem catalogs have different correctness and concurrency properties. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Create a disposable table, append and update data, and read it from a fresh session. 2. Validate partition pruning and inspect metadata and manifest objects under the dedicated warehouse prefix. diff --git a/docs/integrations/data-ai/databricks.mdx b/docs/integrations/data-ai/databricks.mdx index 8c9197e..54f2eb1 100644 --- a/docs/integrations/data-ai/databricks.mdx +++ b/docs/integrations/data-ai/databricks.mdx @@ -19,15 +19,11 @@ Availability of custom S3-compatible endpoints varies by Databricks compute mode | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region, stored in a Databricks secret scope | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Databricks Runtime version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Use a cluster or SQL environment that permits a custom S3-compatible endpoint. Store credentials in a Databricks secret scope and inject them into the supported filesystem or connector configuration. Set the regional endpoint, region, and path-style access from [Connection settings](/integrations/connection-settings). Keep secret values out of notebooks and cluster policies. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Read a disposable Parquet dataset, write results to a dedicated prefix, terminate the compute, and read the output from a fresh cluster. 2. Compare row counts, schema, and deterministic aggregates. diff --git a/docs/integrations/data-ai/delta-lake.mdx b/docs/integrations/data-ai/delta-lake.mdx index c2cc8e1..c8bb482 100644 --- a/docs/integrations/data-ai/delta-lake.mdx +++ b/docs/integrations/data-ai/delta-lake.mdx @@ -19,15 +19,11 @@ Delta Lake on Spark reads and writes Fil One through the `s3a` connector, with ` | Addressing | Path style (`fs.s3a.path.style.access=true`) | | Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through the platform's secret mechanism | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Delta Lake version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Configure Spark's S3 connector with the existing bucket, regional endpoint and region, path-style access, and credentials supplied through the platform's secret mechanism. Keep a dedicated table prefix. Ensure the driver and every executor receive the same settings. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Create a disposable Delta table, append and update rows, produce a checkpoint, and read the table from a fresh session. 2. Run concurrent writers and verify conflicts do not cause silent data loss. diff --git a/docs/integrations/data-ai/duckdb.mdx b/docs/integrations/data-ai/duckdb.mdx index e6c6428..da6496d 100644 --- a/docs/integrations/data-ai/duckdb.mdx +++ b/docs/integrations/data-ai/duckdb.mdx @@ -19,8 +19,6 @@ DuckDB reads and writes Fil One through the `httpfs` extension with an S3 secret | Addressing | Path style (`URL_STYLE 'path'`) | | Authentication | SigV4 access key and secret scoped to the bucket's region (`KEY_ID` and `SECRET`) | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named DuckDB version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Install and load DuckDB's `httpfs` extension. Create an S3 secret with the regional key, region, endpoint hostname, `USE_SSL true`, and `URL_STYLE 'path'`. Use a secret provider appropriate to your environment and keep credentials out of SQL files. @@ -39,9 +37,7 @@ CREATE SECRET filone ( ); ``` -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Read a disposable Parquet file from the bucket and compare row count and deterministic aggregates. 2. Use `COPY` to write a result under a unique prefix, start a fresh DuckDB process, and read the output back. Compare schema, row count, byte size, and content-level checks before removing the prefix. diff --git a/docs/integrations/data-ai/hugging-face-datasets.mdx b/docs/integrations/data-ai/hugging-face-datasets.mdx index 9caee1d..4c46189 100644 --- a/docs/integrations/data-ai/hugging-face-datasets.mdx +++ b/docs/integrations/data-ai/hugging-face-datasets.mdx @@ -19,15 +19,11 @@ Hugging Face `datasets` reads files in Fil One through `fsspec` and `s3fs`, pass | Addressing | Path style (`config_kwargs={"s3": {"addressing_style": "path"}}`) | | Authentication | SigV4 access key and secret scoped to the bucket's region (`key` and `secret`) | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Hugging Face Datasets version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Pass the regional endpoint, region, matching credentials, and path-style behavior through the storage options supported by the installed `s3fs` release. Load credentials from a secret store or environment and keep them out of notebooks. Use explicit object paths under an existing bucket. Pin and record the complete Python package set, because endpoint and addressing options can change across `datasets`, `fsspec`, and `s3fs`. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Load a disposable dataset normally and in streaming mode. Compare feature schema, row count, and deterministic sample hashes. 2. Clear the local cache and load it again to confirm the remote path is sufficient. diff --git a/docs/integrations/data-ai/ray-data.mdx b/docs/integrations/data-ai/ray-data.mdx index 95ef322..b2f8e8d 100644 --- a/docs/integrations/data-ai/ray-data.mdx +++ b/docs/integrations/data-ai/ray-data.mdx @@ -19,15 +19,11 @@ Ray Data reads and writes Fil One through a PyArrow `S3FileSystem` with `endpoin | Addressing | Path style (PyArrow uses path style with `endpoint_override` unless `force_virtual_addressing` is set) | | Authentication | SigV4 access key and secret scoped to the bucket's region, distributed through the cluster's secret mechanism | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Ray version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Create the S3 filesystem with the regional endpoint, region, path-style access, and matching credentials. Distribute credentials through the cluster's secret mechanism and confirm every worker receives the endpoint configuration. Use a dedicated input and output prefix. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Read a disposable multi-file Parquet dataset across several workers, apply a deterministic transformation, and write the result to a unique prefix. 2. Start a fresh Ray job, read the output, and compare schema, row count, partition count where relevant, and deterministic aggregates. diff --git a/docs/integrations/data-movement/aws-cli-sdk.mdx b/docs/integrations/data-movement/aws-cli-sdk.mdx index bbcd6d8..0819a9d 100644 --- a/docs/integrations/data-movement/aws-cli-sdk.mdx +++ b/docs/integrations/data-movement/aws-cli-sdk.mdx @@ -19,15 +19,13 @@ The AWS CLI and SDKs are Fil One's reference clients; see the [SDK guides](/sdk/ | Addressing | Path style (`forcePathStyle: true` in JavaScript, `UsePathStyle: true` in Go, `addressing_style: path` in boto3 and the CLI) | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named AWS CLI or SDK version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Follow the guide for your client: [AWS CLI](/sdk/cli), [Python with boto3](/sdk/python), [JavaScript with AWS SDK v3](/sdk/javascript), or [Go with aws-sdk-go-v2](/sdk/go). Apply the [connection settings](/integrations/connection-settings) to each client. -## Validate +## Examples -Follow the [validation round trip](/integrations/validate-an-integration). Include a file above the client's multipart threshold so automatic multipart upload is exercised. For supported operations and regional differences, see [S3 Compatibility](/reference/s3-compatibility); for object and part sizes, see [Limits & Quotas](/limits). +Include a file above the client's multipart threshold so automatic multipart upload is exercised. For supported operations and regional differences, see [S3 Compatibility](/reference/s3-compatibility); for object and part sizes, see [Limits & Quotas](/limits). ## Troubleshooting diff --git a/docs/integrations/data-movement/cyberduck-mountain-duck.mdx b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx index 07416c3..14621ad 100644 --- a/docs/integrations/data-movement/cyberduck-mountain-duck.mdx +++ b/docs/integrations/data-movement/cyberduck-mountain-duck.mdx @@ -19,8 +19,6 @@ Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 | Addressing | Path style (Cyberduck's path-style S3 connection profile) | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Cyberduck or Mountain Duck version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure 1. Install Cyberduck's path-style S3 connection profile from the profiles list. @@ -30,9 +28,7 @@ Validation status: Not currently tested. Fil One has not yet run the round trip Mountain Duck uses the same bookmark settings, then mounts the bucket through Finder or File Explorer. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Rename and delete a test object and confirm the remote listing reflects each operation. 2. Test keys containing spaces before a larger migration. diff --git a/docs/integrations/data-movement/rclone.mdx b/docs/integrations/data-movement/rclone.mdx index 3d152b3..e2ef0ef 100644 --- a/docs/integrations/data-movement/rclone.mdx +++ b/docs/integrations/data-movement/rclone.mdx @@ -20,8 +20,6 @@ rclone connects to Fil One as an S3 remote with `provider = Other`, SigV4 signin | Authentication | SigV4 access key and secret scoped to the bucket's region | | Provider | `Other` | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named rclone version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Run `rclone config`, create a new remote, choose **Amazon S3 Compliant Storage Providers**, and select **Other** as the provider. Enter the bucket's region and endpoint when prompted, enable path-style access, and enter credentials at rclone's prompts. The resulting section in `rclone.conf` looks like this: @@ -43,13 +41,11 @@ Test discovery without changing remote data: rclone lsd filone: ``` -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples -1. Copy with `rclone copy ./fixture filone:BUCKET/VALIDATION_PREFIX` and list with `rclone lsjson filone:BUCKET/VALIDATION_PREFIX`. -2. Restore with `rclone copy filone:BUCKET/VALIDATION_PREFIX ./restored`. -3. Run `rclone check --download ./restored filone:BUCKET/VALIDATION_PREFIX`. +1. Copy with `rclone copy ./fixture filone:BUCKET/example/` and list with `rclone lsjson filone:BUCKET/example/`. +2. Restore with `rclone copy filone:BUCKET/example/ ./restored`. +3. Run `rclone check --download ./restored filone:BUCKET/example/`. ## Troubleshooting @@ -58,4 +54,4 @@ Follow the [validation round trip](/integrations/validate-an-integration), then: | Signature mismatch | Endpoint, region, and key must refer to the same region; confirm `force_path_style = true`. | | Bucket not found | Confirm the bucket already exists and the key is scoped to it. | | TLS error | Keep certificate validation enabled; inspect local proxy or trust-store configuration. | -| Cleanup leaves data | List the exact validation prefix and remove only that prefix. | +| Cleanup leaves data | List the exact prefix and remove only that prefix. | diff --git a/docs/integrations/devops/harbor.mdx b/docs/integrations/devops/harbor.mdx index 3c79311..bc46879 100644 --- a/docs/integrations/devops/harbor.mdx +++ b/docs/integrations/devops/harbor.mdx @@ -20,15 +20,11 @@ Harbor stores image layers in Fil One through the registry's `s3` storage driver | Authentication | SigV4 access key and secret scoped to the bucket's region (`v4auth: true`) | | TLS | `secure: true` | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Harbor version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Set Harbor's registry storage driver to `s3`. Provide the existing bucket, `regionendpoint`, `region`, `secure: true`, and `forcepathstyle: true`. Store credentials in the deployment's secret mechanism. Use a dedicated bucket or prefix so registry garbage collection cannot affect unrelated objects. Back up Harbor's database separately; the registry bucket holds only image layers. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Push a disposable image with several layers, remove the local image, and pull it again. Compare the image digest and start a container. 2. Inspect Harbor and registry logs. diff --git a/docs/integrations/devops/terraform.mdx b/docs/integrations/devops/terraform.mdx index 436e18f..b8ab064 100644 --- a/docs/integrations/devops/terraform.mdx +++ b/docs/integrations/devops/terraform.mdx @@ -19,8 +19,6 @@ Terraform's `s3` backend stores state in Fil One with `endpoints { s3 = ... }` s | Addressing | Path style (`use_path_style = true`) | | Authentication | SigV4 access key and secret scoped to the bucket's region, supplied outside the configuration | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Terraform version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Use the `s3` backend with an existing bucket, a dedicated state key, the regional endpoint, matching region, and path-style addressing. Keep credentials outside the Terraform configuration. Evaluate the backend's non-AWS options `skip_credentials_validation`, `skip_requesting_account_id`, `skip_metadata_api_check`, and `skip_s3_checksum` for your Terraform release, and enable only the ones that release needs. @@ -37,9 +35,7 @@ terraform { } ``` -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Initialize against a disposable configuration, apply it, and confirm a subsequent `terraform plan` is empty. 2. Copy state to an offline recovery location, remove the local working directory, reinitialize, and confirm state can be read. diff --git a/docs/integrations/devops/velero.mdx b/docs/integrations/devops/velero.mdx index 0a11cad..83d85c5 100644 --- a/docs/integrations/devops/velero.mdx +++ b/docs/integrations/devops/velero.mdx @@ -19,15 +19,11 @@ Velero writes backups to Fil One through its AWS object-store plugin, with a bac | Addressing | Path style (`s3ForcePathStyle: "true"`) | | Authentication | SigV4 access key and secret scoped to the bucket's region, supplied through a Kubernetes Secret | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Velero version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Install Velero with its AWS object-store plugin. Configure a backup storage location with the existing bucket, `region`, `s3Url`, and `s3ForcePathStyle: "true"`. Supply the key through a Kubernetes Secret rather than a manifest committed to source control. Volume-snapshot support is separate from object storage and requires its own provider. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Create a disposable namespace containing representative objects and persistent data. 2. Run a backup, inspect `velero backup describe --details` and the logs, delete the namespace, then restore it. diff --git a/docs/integrations/index.mdx b/docs/integrations/index.mdx index 22330e1..2bff1a3 100644 --- a/docs/integrations/index.mdx +++ b/docs/integrations/index.mdx @@ -1,44 +1,32 @@ --- title: Integrations -description: Configure and validate third-party tools with Fil One object storage. +description: Configure third-party tools with Fil One object storage. --- # Integrations -These guides explain how to connect common transfer, backup, NAS, DevOps, analytics, and AI tools to Fil One. Each guide gives the connection settings, a validation procedure, and troubleshooting for one tool. +These guides show how to point common transfer, backup, NAS, DevOps, analytics, and AI tools at Fil One. Start with [Connection settings](/integrations/connection-settings), then open the guide for your tool. For API behavior, see the [S3 Compatibility reference](/reference/s3-compatibility). -Every guide displays one of these validation states: - -| Status | Meaning | +| Category | Tool | | --- | --- | -| **Verified** | A named tool version completed the documented end-to-end workflow, restore, and integrity check in every listed region. The guide includes the evidence and verification date. | -| **Compatible** | An end-to-end run exists for a named version, but not for every region or not within the last 12 months. | -| **Preview** | A runnable validation path exists, but coverage is incomplete. | -| **Not currently tested** | No end-to-end run has been recorded for any version or region. Follow the validation steps before production use. | -| **Not currently supported** | A known product or addressing requirement blocks the integration. | - -Start with [Connection settings](/integrations/connection-settings) and the [validation round trip](/integrations/validate-an-integration), then open the guide for your tool. For API behavior, use the [S3 Compatibility reference](/reference/s3-compatibility). - -| Category | Tool | Status | -| --- | --- | --- | -| Data movement | [rclone](/integrations/data-movement/rclone) | Not currently tested | -| Data movement | [Cyberduck and Mountain Duck](/integrations/data-movement/cyberduck-mountain-duck) | Not currently tested | -| Data movement | [AWS CLI and SDKs](/integrations/data-movement/aws-cli-sdk) | Not currently tested | -| Backup and disaster recovery | [Acronis Cyber Protect](/integrations/backup/acronis-cyber-protect) | Not currently tested | -| Backup and disaster recovery | [Comet Backup](/integrations/backup/comet-backup) | Not currently tested | -| Backup and disaster recovery | [MSP360](/integrations/backup/msp360) | Not currently tested | -| Backup and disaster recovery | [NAKIVO Backup and Replication](/integrations/backup/nakivo) | Not currently tested | -| NAS and file services | [Synology Cloud Sync](/integrations/nas/synology-cloud-sync) | Not currently tested | -| NAS and file services | [QNAP HBS](/integrations/nas/qnap-hbs) | Not currently tested | -| NAS and file services | [TrueNAS Cloud Sync](/integrations/nas/truenas-cloud-sync) | Not currently tested | -| Kubernetes and DevOps | [Velero](/integrations/devops/velero) | Not currently tested | -| Kubernetes and DevOps | [Harbor](/integrations/devops/harbor) | Not currently tested | -| Kubernetes and DevOps | [Terraform S3 backend](/integrations/devops/terraform) | Not currently tested | -| SIEM and logging | [Elasticsearch searchable snapshots](/integrations/logging/elasticsearch-snapshots) | Not currently tested | -| SIEM and logging | [Splunk SmartStore](/integrations/logging/splunk-smartstore) | Not currently tested | -| Data and AI | [Apache Iceberg](/integrations/data-ai/apache-iceberg) | Not currently tested | -| Data and AI | [Databricks](/integrations/data-ai/databricks) | Not currently tested | -| Data and AI | [Delta Lake](/integrations/data-ai/delta-lake) | Not currently tested | -| Data and AI | [DuckDB](/integrations/data-ai/duckdb) | Not currently tested | -| Data and AI | [Hugging Face Datasets](/integrations/data-ai/hugging-face-datasets) | Not currently tested | -| Data and AI | [Ray Data](/integrations/data-ai/ray-data) | Not currently tested | +| Data movement | [rclone](/integrations/data-movement/rclone) | +| Data movement | [Cyberduck and Mountain Duck](/integrations/data-movement/cyberduck-mountain-duck) | +| Data movement | [AWS CLI and SDKs](/integrations/data-movement/aws-cli-sdk) | +| Backup and disaster recovery | [Acronis Cyber Protect](/integrations/backup/acronis-cyber-protect) | +| Backup and disaster recovery | [Comet Backup](/integrations/backup/comet-backup) | +| Backup and disaster recovery | [MSP360](/integrations/backup/msp360) | +| Backup and disaster recovery | [NAKIVO Backup and Replication](/integrations/backup/nakivo) | +| NAS and file services | [Synology Cloud Sync](/integrations/nas/synology-cloud-sync) | +| NAS and file services | [QNAP HBS](/integrations/nas/qnap-hbs) | +| NAS and file services | [TrueNAS Cloud Sync](/integrations/nas/truenas-cloud-sync) | +| Kubernetes and DevOps | [Velero](/integrations/devops/velero) | +| Kubernetes and DevOps | [Harbor](/integrations/devops/harbor) | +| Kubernetes and DevOps | [Terraform S3 backend](/integrations/devops/terraform) | +| SIEM and logging | [Elasticsearch searchable snapshots](/integrations/logging/elasticsearch-snapshots) | +| SIEM and logging | [Splunk SmartStore](/integrations/logging/splunk-smartstore) | +| Data and AI | [Apache Iceberg](/integrations/data-ai/apache-iceberg) | +| Data and AI | [Databricks](/integrations/data-ai/databricks) | +| Data and AI | [Delta Lake](/integrations/data-ai/delta-lake) | +| Data and AI | [DuckDB](/integrations/data-ai/duckdb) | +| Data and AI | [Hugging Face Datasets](/integrations/data-ai/hugging-face-datasets) | +| Data and AI | [Ray Data](/integrations/data-ai/ray-data) | diff --git a/docs/integrations/logging/elasticsearch-snapshots.mdx b/docs/integrations/logging/elasticsearch-snapshots.mdx index e870463..7ff036e 100644 --- a/docs/integrations/logging/elasticsearch-snapshots.mdx +++ b/docs/integrations/logging/elasticsearch-snapshots.mdx @@ -19,15 +19,11 @@ Elasticsearch stores snapshots in Fil One through an `s3` repository whose named | Addressing | Path style (`s3.client.CLIENT.path_style_access: true`) | | Authentication | SigV4 access key and secret scoped to the bucket's region, stored in the keystore as `s3.client.CLIENT.access_key` and `s3.client.CLIENT.secret_key` | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Elasticsearch version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Store the access and secret keys in the Elasticsearch keystore under a named S3 client. Configure that client's endpoint, region, and path-style access, then register an `s3` snapshot repository using an existing bucket and a dedicated `base_path`. Run the repository verification API from every master and data node before taking a snapshot. Searchable snapshots require an Enterprise subscription; ordinary snapshot storage does not. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Create a disposable index with known documents, take a snapshot, delete the index, and restore it under a new name. 2. Compare document counts and selected content hashes. diff --git a/docs/integrations/logging/splunk-smartstore.mdx b/docs/integrations/logging/splunk-smartstore.mdx index 0e2a555..045b63a 100644 --- a/docs/integrations/logging/splunk-smartstore.mdx +++ b/docs/integrations/logging/splunk-smartstore.mdx @@ -19,15 +19,11 @@ Splunk SmartStore uses Fil One as a `remote` volume in `indexes.conf`, with `rem | Addressing | Path style (`remote.s3.url_version = v1`) | | Authentication | SigV4 access key and secret scoped to the bucket's region (`remote.s3.signature_version = v4`) | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Splunk Enterprise version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Define a `remote` volume in `indexes.conf` with the existing bucket, regional endpoint, auth region, path-style URL mode, and versioning behavior appropriate to the bucket. Reference the volume from each intended index with `remotePath`. Store credentials through the deployment's protected configuration workflow and apply the settings consistently across an indexer cluster. SmartStore licensing and topology requirements apply. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Index disposable events, roll data to remote storage, and confirm searches can fetch evicted buckets. 2. Restart or replace a test indexer and verify remote recovery. Compare event counts and representative hashes or fields. diff --git a/docs/integrations/nas/qnap-hbs.mdx b/docs/integrations/nas/qnap-hbs.mdx index 9df6631..473ed2c 100644 --- a/docs/integrations/nas/qnap-hbs.mdx +++ b/docs/integrations/nas/qnap-hbs.mdx @@ -19,15 +19,11 @@ QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named QNAP HBS version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure In HBS 3, create a storage space of the S3-compatible type. Enter the regional endpoint, region, SigV4 credentials, and existing bucket. Enable SSL and certificate validation. Create a backup or sync job targeting a dedicated remote prefix. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Run the backup job for the fixture, delete the local fixture, and restore it to a new folder; compare size and SHA-256. 2. Review the HBS job log for skipped files. diff --git a/docs/integrations/nas/synology-cloud-sync.mdx b/docs/integrations/nas/synology-cloud-sync.mdx index 260fc21..73d2388 100644 --- a/docs/integrations/nas/synology-cloud-sync.mdx +++ b/docs/integrations/nas/synology-cloud-sync.mdx @@ -19,15 +19,11 @@ Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named Synology Cloud Sync version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure Install Cloud Sync, add a connection, and choose S3 storage with a custom server URL. Enter the regional endpoint, SigV4 credentials, and existing bucket. Choose a local path, remote path, and one of bidirectional, upload-only, or download-only synchronization. Keep TLS validation enabled. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Sync the fixture with an upload-only task to a dedicated remote prefix and wait for the task to report up to date. 2. Remove the local fixture and create a download-only recovery task to a separate directory; compare size and SHA-256. diff --git a/docs/integrations/nas/truenas-cloud-sync.mdx b/docs/integrations/nas/truenas-cloud-sync.mdx index d8d52af..9db7b91 100644 --- a/docs/integrations/nas/truenas-cloud-sync.mdx +++ b/docs/integrations/nas/truenas-cloud-sync.mdx @@ -19,17 +19,13 @@ TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a | Addressing | Path style | | Authentication | SigV4 access key and secret scoped to the bucket's region (leave signature version 2 disabled) | -Validation status: Not currently tested. Fil One has not yet run the round trip below with a named TrueNAS version in eu-west-1 and us-east-1; results will be added here with version, date, and evidence. - ## Configure In TrueNAS SCALE 22 and later, create the credential under Backup Credentials; in TrueNAS CORE, use the System cloud credentials page. Choose Amazon S3 with a custom endpoint, enter the regional key and region, keep signature version 2 disabled, and verify the credential. Create a Cloud Sync task using an existing bucket and a dedicated folder. Choose the COPY transfer mode for the first test. SYNC can delete destination files that are absent at the source, and MOVE deletes source files after transfer. -## Validate - -Follow the [validation round trip](/integrations/validate-an-integration), then: +## Examples 1. Run a PUSH of the fixture, then a PULL into a separate dataset; compare size and SHA-256. 2. Inspect the task logs and confirm cleanup stays within the test folder. diff --git a/docs/integrations/validate-an-integration.mdx b/docs/integrations/validate-an-integration.mdx index 86cb96f..b169a35 100644 --- a/docs/integrations/validate-an-integration.mdx +++ b/docs/integrations/validate-an-integration.mdx @@ -1,18 +1,16 @@ --- -title: Validate an integration -description: The round trip every Fil One integration guide uses to confirm upload, restore, and integrity. +title: Check a setup +description: Optional steps to copy a small fixture, restore it, and compare checksums. --- -# Validate an integration +# Check a setup -Every integration guide links to this round trip. Run it with disposable data before pointing production workloads at the bucket. +If you want to confirm a tool can write to a bucket and read the same bytes back, use disposable data and a prefix you can delete. -1. Create a local fixture directory containing a small text file, a nested file, and a file large enough to trigger multipart upload in the tool (above the tool's part or chunk threshold; see [Limits & Quotas](/limits) for part sizes). -2. Copy the fixture to a unique, disposable prefix in the bucket, for example `validation-YYYYMMDD-HHMM/`. +1. Create a local folder with a small text file, a nested file, and a file large enough to trigger multipart upload in the tool (above the tool's part or chunk size; see [Limits & Quotas](/limits)). +2. Copy it to a unique prefix in the bucket, for example `check-YYYYMMDD-HHMM/`. 3. List the prefix and confirm every file is present. -4. Remove the local copy of the fixture. +4. Remove the local copy. 5. Restore the prefix to a new local directory. -6. Compare byte size and SHA-256 for every restored file against the values recorded before upload. +6. Compare size and SHA-256 for every restored file against the values recorded before upload. 7. Delete only that prefix, then list it again to confirm it is empty. - -Record the tool version, the region, and the date with the result. The tool-specific guide adds any steps that go beyond this round trip. diff --git a/scripts/check-integrations.mjs b/scripts/check-integrations.mjs index f49df19..a81135d 100644 --- a/scripts/check-integrations.mjs +++ b/scripts/check-integrations.mjs @@ -47,11 +47,13 @@ for (const file of guides) { if (status !== "Verified" && /(^|\W)Verified(\W|$)/.test(body)) { failures.push(`${file}: body uses Verified without Verified metadata`); } - if (status) { - const statusLines = body.split("\n").filter((line) => line.startsWith("Validation status: ")); + const statusLines = body.split("\n").filter((line) => line.startsWith("Validation status: ")); + if (status === "Verified" || status === "Not currently supported") { if (statusLines.length !== 1 || !statusLines[0].startsWith(`Validation status: ${status}.`)) { failures.push(`${file}: body must contain exactly one line starting with "Validation status: ${status}."`); } + } else if (statusLines.length) { + failures.push(`${file}: untested guides must not include a Validation status line`); } if (blocked && sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: blocked guide must not be listed: ${route}`); if (!blocked && !sidebar.includes(`"${route}"`)) failures.push(`sidebars.js: missing ${route}`); diff --git a/static/llms-full.txt b/static/llms-full.txt index 3c4a8dc..97d10ea 100644 --- a/static/llms-full.txt +++ b/static/llms-full.txt @@ -2917,7 +2917,7 @@ Intelligence deliberately per bucket rather than across everything. Source: https://docs.fil.one/integrations -Each integration guide gives the connection settings, a validation procedure, and troubleshooting for one tool. Statuses: Verified (named tool version completed the end-to-end workflow, restore, and integrity check in every listed region, with evidence and date), Compatible (an end-to-end run exists for a named version, but not for every region or not within the last 12 months), Preview (runnable validation path, incomplete coverage), Not currently tested (no end-to-end run recorded for any version or region), Not currently supported (a known product or addressing requirement blocks the integration). All 21 published guides are Not currently tested. The index page lists every guide with its status. +Each integration guide gives the connection settings, setup steps, and troubleshooting for one tool. For API behavior, see the S3 Compatibility reference. # Integration connection settings @@ -2925,134 +2925,134 @@ Source: https://docs.fil.one/integrations/connection-settings Endpoint: https://eu-west-1.s3.fil.one or https://us-east-1.s3.fil.one, matching the bucket's region (also shown in the bucket's Access endpoints panel). Region: eu-west-1 or us-east-1. Authentication: SigV4 with a key created for the same region and scoped to the bucket. Addressing: path style required. Endpoint, region, key, and bucket must belong to the same region. Keep TLS certificate validation enabled and secret keys out of docs, screenshots, source control, and command-line history. Feature availability differs between the S3 API and the app.fil.one dashboard; the S3 Compatibility reference covers API behavior. -# Validate an integration +# Check a setup Source: https://docs.fil.one/integrations/validate-an-integration -The round trip every guide links to: create a fixture with a small file, a nested file, and a file large enough to trigger multipart upload; copy it to a unique disposable prefix; list the prefix; remove the local copy; restore to a new directory; compare byte size and SHA-256 for every file; delete only that prefix and confirm it is empty. Run it with disposable data before pointing production workloads at the bucket. Record tool version, region, and date with the result. +Optional steps to copy a small fixture (including a file large enough for multipart upload), restore it, and compare size and SHA-256. Use a unique prefix you can delete. # rclone Source: https://docs.fil.one/integrations/data-movement/rclone -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named rclone version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. rclone connects to Fil One as an S3 remote with provider = Other, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +rclone connects to Fil One as an S3 remote with provider = Other, SigV4 signing, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Cyberduck and Mountain Duck Source: https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Cyberduck or Mountain Duck version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Cyberduck and Mountain Duck connect to Fil One through Cyberduck's path-style S3 connection profile, using SigV4 and a key scoped to the bucket's region. Mountain Duck uses the same bookmark and mounts the bucket in Finder or File Explorer. Configuration, tool-specific validation steps, and troubleshooting are on the page. # AWS CLI and SDKs Source: https://docs.fil.one/integrations/data-movement/aws-cli-sdk -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named AWS CLI or SDK version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. The AWS CLI and SDKs are Fil One's reference clients; see the SDK guides for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. Configuration, tool-specific validation steps, and troubleshooting are on the page. +The AWS CLI and SDKs are Fil One's reference clients; see the SDK guides for setup. Each client takes the regional endpoint, SigV4 credentials, and path-style addressing. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Acronis Cyber Protect Source: https://docs.fil.one/integrations/backup/acronis-cyber-protect -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Acronis Cyber Protect version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Acronis Cyber Protect writes to Fil One as an S3-compatible backup location, using SigV4 (AuthV4), path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Comet Backup Source: https://docs.fil.one/integrations/backup/comet-backup -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Comet Backup version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Comet Backup writes to Fil One through an S3-compatible Storage Vault, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # MSP360 Source: https://docs.fil.one/integrations/backup/msp360 -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named MSP360 version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +MSP360 writes to Fil One as an S3-compatible storage account, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # NAKIVO Backup and Replication Source: https://docs.fil.one/integrations/backup/nakivo -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named NAKIVO Backup and Replication version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +NAKIVO Backup and Replication writes to Fil One as an S3-compatible repository, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Synology Cloud Sync Source: https://docs.fil.one/integrations/nas/synology-cloud-sync -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Synology Cloud Sync version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Synology Cloud Sync connects to Fil One as S3 storage with a custom server URL, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # QNAP HBS Source: https://docs.fil.one/integrations/nas/qnap-hbs -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named QNAP HBS version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +QNAP Hybrid Backup Sync (HBS 3) connects to Fil One as an S3-compatible storage space, using SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # TrueNAS Cloud Sync Source: https://docs.fil.one/integrations/nas/truenas-cloud-sync -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named TrueNAS version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +TrueNAS Cloud Sync connects to Fil One with an Amazon S3 credential that uses a custom endpoint, SigV4, path-style addressing, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Velero Source: https://docs.fil.one/integrations/devops/velero -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Velero version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets s3Url to the regional endpoint, s3ForcePathStyle: "true", and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Velero writes backups to Fil One through its AWS object-store plugin, with a backup storage location that sets s3Url to the regional endpoint, s3ForcePathStyle: "true", and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Harbor Source: https://docs.fil.one/integrations/devops/harbor -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Harbor version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Harbor stores image layers in Fil One through the registry's s3 storage driver, with regionendpoint set to the regional endpoint, forcepathstyle: true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Harbor stores image layers in Fil One through the registry's s3 storage driver, with regionendpoint set to the regional endpoint, forcepathstyle: true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Terraform S3 backend Source: https://docs.fil.one/integrations/devops/terraform -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Terraform version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Terraform's s3 backend stores state in Fil One with endpoints { s3 = ... } set to the regional endpoint and use_path_style = true (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Terraform's s3 backend stores state in Fil One with endpoints { s3 = ... } set to the regional endpoint and use_path_style = true (Terraform 1.6 and later backend syntax), signed with SigV4 by a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Elasticsearch searchable snapshots Source: https://docs.fil.one/integrations/logging/elasticsearch-snapshots -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Elasticsearch version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Elasticsearch stores snapshots in Fil One through an s3 repository whose named client sets endpoint, region, and path_style_access: true, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes repository-s3; on 7.x install the plugin on every node. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Elasticsearch stores snapshots in Fil One through an s3 repository whose named client sets endpoint, region, and path_style_access: true, with a key scoped to the bucket's region held in the keystore. Elasticsearch 8.x includes repository-s3; on 7.x install the plugin on every node. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Splunk SmartStore Source: https://docs.fil.one/integrations/logging/splunk-smartstore -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Splunk Enterprise version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Splunk SmartStore uses Fil One as a remote volume in indexes.conf, with remote.s3.endpoint set to the regional endpoint, remote.s3.auth_region set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Splunk SmartStore uses Fil One as a remote volume in indexes.conf, with remote.s3.endpoint set to the regional endpoint, remote.s3.auth_region set to the bucket's region, path-style URLs, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Apache Iceberg Source: https://docs.fil.one/integrations/data-ai/apache-iceberg -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Apache Iceberg version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Apache Iceberg reads and writes table data in Fil One through S3FileIO, with s3.endpoint set to the regional endpoint, s3.path-style-access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Apache Iceberg reads and writes table data in Fil One through S3FileIO, with s3.endpoint set to the regional endpoint, s3.path-style-access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Databricks Source: https://docs.fil.one/integrations/data-ai/databricks -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Databricks Runtime version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Availability of custom S3-compatible endpoints varies by Databricks compute mode. On compute that permits a custom endpoint, Databricks reads and writes Fil One through the S3 connector with the regional endpoint, path-style access, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Delta Lake Source: https://docs.fil.one/integrations/data-ai/delta-lake -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Delta Lake version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Delta Lake on Spark reads and writes Fil One through the s3a connector, with fs.s3a.endpoint set to the regional endpoint, fs.s3a.path.style.access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Delta Lake on Spark reads and writes Fil One through the s3a connector, with fs.s3a.endpoint set to the regional endpoint, fs.s3a.path.style.access=true, and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # DuckDB Source: https://docs.fil.one/integrations/data-ai/duckdb -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named DuckDB version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. DuckDB reads and writes Fil One through the httpfs extension with an S3 secret that sets ENDPOINT to the regional hostname, URL_STYLE 'path', and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. +DuckDB reads and writes Fil One through the httpfs extension with an S3 secret that sets ENDPOINT to the regional hostname, URL_STYLE 'path', and a key scoped to the bucket's region. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Hugging Face Datasets Source: https://docs.fil.one/integrations/data-ai/hugging-face-datasets -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Hugging Face Datasets version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Hugging Face datasets reads files in Fil One through fsspec and s3fs, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Hugging Face datasets reads files in Fil One through fsspec and s3fs, passing the regional endpoint, region, path-style addressing, and a key scoped to the bucket's region as storage options. Configuration, tool-specific validation steps, and troubleshooting are on the page. # Ray Data Source: https://docs.fil.one/integrations/data-ai/ray-data -Validation status: Not currently tested. Fil One has not yet run the round trip in the guide with a named Ray version in eu-west-1 and us-east-1; results will be added to the guide with version, date, and evidence. Ray Data reads and writes Fil One through a PyArrow S3FileSystem with endpoint_override set to the regional endpoint, region set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. Configuration, tool-specific validation steps, and troubleshooting are on the page. +Ray Data reads and writes Fil One through a PyArrow S3FileSystem with endpoint_override set to the regional endpoint, region set to the bucket's region, path-style addressing, and a key scoped to the bucket's region on every worker. Configuration, tool-specific validation steps, and troubleshooting are on the page. diff --git a/static/llms.txt b/static/llms.txt index 412b8fc..f241b48 100644 --- a/static/llms.txt +++ b/static/llms.txt @@ -176,30 +176,30 @@ Critical integration notes for developers: ## Integrations -- [Integrations](https://docs.fil.one/integrations): Validation status definitions and a table of every integration guide with its status +- [Integrations](https://docs.fil.one/integrations): Setup guides for common transfer, backup, NAS, DevOps, analytics, and AI tools - [Integration connection settings](https://docs.fil.one/integrations/connection-settings): Endpoints (https://eu-west-1.s3.fil.one, https://us-east-1.s3.fil.one), region, SigV4, path-style, and credential settings -- [Validate an integration](https://docs.fil.one/integrations/validate-an-integration): The shared upload, list, restore, SHA-256, and cleanup round trip every guide links to -- [rclone](https://docs.fil.one/integrations/data-movement/rclone): Not currently tested; provider Other, force_path_style, config example, rclone check validation -- [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): Not currently tested; bookmark, mount, restore, and troubleshooting guidance -- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): Not currently tested; the reference clients, with setup in the SDK guides -- [Acronis Cyber Protect](https://docs.fil.one/integrations/backup/acronis-cyber-protect): Not currently tested; S3-compatible storage, backup, restore, and integrity validation -- [Comet Backup](https://docs.fil.one/integrations/backup/comet-backup): Not currently tested; Storage Vault setup and restore validation -- [MSP360](https://docs.fil.one/integrations/backup/msp360): Not currently tested; S3-compatible storage account and recovery workflow -- [NAKIVO Backup and Replication](https://docs.fil.one/integrations/backup/nakivo): Not currently tested; repository, recovery, and verification guidance -- [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): Not currently tested; custom S3 task and download recovery -- [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): Not currently tested; HBS storage space, backup, and restore -- [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): Not currently tested; credential, push, pull, and integrity workflow -- [Velero](https://docs.fil.one/integrations/devops/velero): Not currently tested; Kubernetes backup, restore, and volume-scope validation -- [Harbor](https://docs.fil.one/integrations/devops/harbor): Not currently tested; registry push, pull, digest, and garbage-collection guidance -- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): Not currently tested; endpoints and use_path_style backend settings, state migration, recovery, and concurrency validation -- [Elasticsearch searchable snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): Not currently tested; s3 repository client settings, repository verification, restore, and searchable-snapshot validation -- [Splunk SmartStore](https://docs.fil.one/integrations/logging/splunk-smartstore): Not currently tested; remote storage, eviction, search, and recovery validation -- [Apache Iceberg](https://docs.fil.one/integrations/data-ai/apache-iceberg): Not currently tested; S3FileIO, catalog, commits, and recovery validation -- [Databricks](https://docs.fil.one/integrations/data-ai/databricks): Not currently tested; custom-endpoint compute, read, write, and fresh-cluster recovery -- [Delta Lake](https://docs.fil.one/integrations/data-ai/delta-lake): Not currently tested; transaction log, checkpoints, concurrency, and recovery -- [DuckDB](https://docs.fil.one/integrations/data-ai/duckdb): Not currently tested; httpfs read, write, and cache-independent validation -- [Hugging Face Datasets](https://docs.fil.one/integrations/data-ai/hugging-face-datasets): Not currently tested; load, streaming, cache clearing, and package-version guidance -- [Ray Data](https://docs.fil.one/integrations/data-ai/ray-data): Not currently tested; distributed reads, writes, worker configuration, and retry validation +- [Check a setup](https://docs.fil.one/integrations/validate-an-integration): Optional steps to copy a small fixture, restore it, and compare checksums +- [rclone](https://docs.fil.one/integrations/data-movement/rclone): provider Other, force_path_style, config example, rclone check validation +- [Cyberduck and Mountain Duck](https://docs.fil.one/integrations/data-movement/cyberduck-mountain-duck): bookmark, mount, restore, and troubleshooting guidance +- [AWS CLI and SDKs](https://docs.fil.one/integrations/data-movement/aws-cli-sdk): the reference clients, with setup in the SDK guides +- [Acronis Cyber Protect](https://docs.fil.one/integrations/backup/acronis-cyber-protect): S3-compatible storage, backup, restore, and integrity validation +- [Comet Backup](https://docs.fil.one/integrations/backup/comet-backup): Storage Vault setup and restore validation +- [MSP360](https://docs.fil.one/integrations/backup/msp360): S3-compatible storage account and recovery workflow +- [NAKIVO Backup and Replication](https://docs.fil.one/integrations/backup/nakivo): repository, recovery, and verification guidance +- [Synology Cloud Sync](https://docs.fil.one/integrations/nas/synology-cloud-sync): custom S3 task and download recovery +- [QNAP HBS](https://docs.fil.one/integrations/nas/qnap-hbs): HBS storage space, backup, and restore +- [TrueNAS Cloud Sync](https://docs.fil.one/integrations/nas/truenas-cloud-sync): credential, push, pull, and integrity workflow +- [Velero](https://docs.fil.one/integrations/devops/velero): Kubernetes backup, restore, and volume-scope validation +- [Harbor](https://docs.fil.one/integrations/devops/harbor): registry push, pull, digest, and garbage-collection guidance +- [Terraform S3 backend](https://docs.fil.one/integrations/devops/terraform): endpoints and use_path_style backend settings, state migration, recovery, and concurrency validation +- [Elasticsearch searchable snapshots](https://docs.fil.one/integrations/logging/elasticsearch-snapshots): s3 repository client settings, repository verification, restore, and searchable-snapshot validation +- [Splunk SmartStore](https://docs.fil.one/integrations/logging/splunk-smartstore): remote storage, eviction, search, and recovery validation +- [Apache Iceberg](https://docs.fil.one/integrations/data-ai/apache-iceberg): S3FileIO, catalog, commits, and recovery validation +- [Databricks](https://docs.fil.one/integrations/data-ai/databricks): custom-endpoint compute, read, write, and fresh-cluster recovery +- [Delta Lake](https://docs.fil.one/integrations/data-ai/delta-lake): transaction log, checkpoints, concurrency, and recovery +- [DuckDB](https://docs.fil.one/integrations/data-ai/duckdb): httpfs read, write, and cache-independent validation +- [Hugging Face Datasets](https://docs.fil.one/integrations/data-ai/hugging-face-datasets): load, streaming, cache clearing, and package-version guidance +- [Ray Data](https://docs.fil.one/integrations/data-ai/ray-data): distributed reads, writes, worker configuration, and retry validation ## Optional