Skip to content

Add rclone to rsync-transfer image - #75

Merged
istein1 merged 4 commits into
migtools:masterfrom
stillalearner:rclone_add
Aug 4, 2026
Merged

istein1 merged 4 commits into
migtools:masterfrom
stillalearner:rclone_add

Conversation

@stillalearner

@stillalearner stillalearner commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add rclone as a selectively-compiled Go binary to the rsync-transfer container image, enabling indirect PVC data migration via S3-compatible cloud storage.

Changes

New file: cmd/rclone/main.go

Minimal rclone binary with only the backends needed for indirect migration:

  • s3 — S3-compatible storage (AWS, MinIO, Ceph, Noobaa)
  • local — local filesystem (PVC mount)
  • crypt — client-side encryption overlay

Only the sync subcommand is included.

Modified: Dockerfile

Added one go build line and one COPY line following the existing pattern for blockrsync and proxy.

Modified: go.mod / go.sum

Added github.com/rclone/rclone dependency.

Binary sizes (Linux amd64)

Binary Size Status
blockrsync 11MB Existing, unchanged
proxy 10MB Existing, unchanged
rclone 37MB New

Selective backend imports reduce the binary from ~80MB (full rclone) to 37MB.

What this enables

The rclone binary in the image allows future crane transfer-pvc --cloud-storage to create pods that run rclone sync instead of rsync. No crane code changes in this PR — this only adds the tool to the image.

Backward compatibility

Existing rsync and stunnel functionality is completely unaffected. The rclone binary is an addition, not a replacement.

Summary by CodeRabbit

  • New Features

    • Added a bundled rclone command-line utility to the Docker image.
    • Enabled synchronization through the included sync command.
    • Added support for local storage, encrypted remotes, and S3-compatible storage.
  • Improvements

    • The rclone executable is available at /usr/local/bin/rclone without additional installation.
    • Updated the container build environment and produced more portable command-line binaries.
    • Added startup validation to confirm the included synchronization tools are functional.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 034cdff7-d634-460e-8221-11c5b4533168

📥 Commits

Reviewing files that changed from the base of the PR and between 34c1f68 and d7aec25.

📒 Files selected for processing (1)
  • Dockerfile
🚧 Files skipped from review as they are similar to previous changes (1)
  • Dockerfile

📝 Walkthrough

Walkthrough

The module adds an rclone executable with crypt, local, S3, and sync support. The Docker build uses Go 1.25, disables CGO for existing builds, packages rclone, and validates the binaries with smoke tests.

Changes

Rclone integration

Layer / File(s) Summary
Rclone entrypoint and dependencies
go.mod, cmd/rclone/main.go
The module adds rclone v1.75.0, refreshes dependencies, and defines the executable entry point with crypt, local, S3, and sync registrations.
Docker binary packaging
Dockerfile
The Docker build uses Go 1.25, sets CGO_ENABLED=0 for blockrsync and proxy, builds rclone, copies it to /usr/local/bin/rclone, and runs smoke tests for blockrsync, rclone, and rsync.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

  • migtools/crane#755 — Adds rclone with selective S3, local, and crypt backend imports.
  • migtools/crane#751 — Adds the rclone executable, dependency, backend imports, and Docker integration.

Sequence Diagram(s)

sequenceDiagram
  participant DockerBuilder
  participant GoToolchain
  participant FinalImage
  DockerBuilder->>GoToolchain: Build blockrsync, proxy, and rclone
  GoToolchain->>FinalImage: Copy rclone to /usr/local/bin/rclone
  DockerBuilder->>FinalImage: Run blockrsync, rclone, and rsync smoke tests
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the primary change: adding rclone to the rsync-transfer image.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
Dockerfile (1)

30-30: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Use a builder that satisfies the module’s Go version.

go.mod requires Go 1.25.0, but the builder at Line 3 remains golang:1.21. The new build at Line 30 relies on automatic toolchain switching. With GOTOOLCHAIN=local, or without access to download the required toolchain, the build can stop before compiling rclone. Go treats the go directive as a minimum requirement; automatic switching depends on GOTOOLCHAIN. (go.dev)

Pin the builder to Go 1.25 or explicitly provision the required toolchain.

Proposed fix
-FROM docker.io/golang:1.21 as golang-builder
+FROM docker.io/golang:1.25 as golang-builder
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Dockerfile` at line 30, Update the Dockerfile builder image used by the
rclone build so it provides Go 1.25, matching the minimum version declared in
go.mod. Change the existing builder configuration rather than relying on
automatic toolchain downloads, and keep the RUN go build command unchanged.
cmd/rclone/main.go (1)

1-1: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Preserve rclone’s TLS compatibility setting in this executable.

This module is built on Go 1.25+ and does not inherit github.com/rclone/rclone v1.75.0’s dependency godebug directives. If compatibility with S3-compatible servers using negative certificate serial numbers is still required, add the directive to the rclone main package.

Proposed fix
+//go:debug x509negativeserial=1
 package main
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/rclone/main.go` at line 1, Update the rclone main package to declare the
required godebug TLS compatibility setting for negative certificate serial
numbers, matching the setting expected by rclone v1.75.0. Add it at package
scope in cmd/rclone/main.go so the Go 1.25+ executable preserves this behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cmd/rclone/main.go`:
- Line 1: Update the rclone main package to declare the required godebug TLS
compatibility setting for negative certificate serial numbers, matching the
setting expected by rclone v1.75.0. Add it at package scope in
cmd/rclone/main.go so the Go 1.25+ executable preserves this behavior.

In `@Dockerfile`:
- Line 30: Update the Dockerfile builder image used by the rclone build so it
provides Go 1.25, matching the minimum version declared in go.mod. Change the
existing builder configuration rather than relying on automatic toolchain
downloads, and keep the RUN go build command unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 09d1392c-2b28-49ce-ba0a-e83a2c80c105

📥 Commits

Reviewing files that changed from the base of the PR and between 0a019b1 and e8634b6.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (3)
  • Dockerfile
  • cmd/rclone/main.go
  • go.mod

@aufi aufi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
Dockerfile (1)

28-30: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Keep the rclone artifact statically linked.

Lines 28-29 set CGO_ENABLED=0, but line 30 does not. Go uses CGO_ENABLED to control cgo-dependent build selection. (github.com)

If the selected rclone dependencies produce a cgo-linked binary, the final minimal image can fail to execute. Set CGO_ENABLED=0 for the rclone build, or verify that the artifact has no dynamic dependencies for every supported architecture.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Dockerfile` around lines 28 - 30, Update the rclone build command in the
Dockerfile to set CGO_ENABLED=0, matching the blockrsync and proxy builds, so
the rclone artifact remains statically linked across supported architectures.

Source: MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@Dockerfile`:
- Around line 28-30: Update the rclone build command in the Dockerfile to set
CGO_ENABLED=0, matching the blockrsync and proxy builds, so the rclone artifact
remains statically linked across supported architectures.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: dc2a7db5-dc58-48dd-a595-8c88513cc1f4

📥 Commits

Reviewing files that changed from the base of the PR and between e8634b6 and 34c1f68.

📒 Files selected for processing (1)
  • Dockerfile

@istein1

istein1 commented Aug 4, 2026

Copy link
Copy Markdown
Member

Approved.

Few minor claude suggestions:

  1. Confirm existing tests pass with the dependency bumps — especially the snappy major version change.
  2. Consider adding a smoke test — even a simple RUN rclone version in a Dockerfile test stage, or a CI step that runs rclone sync --help in the built image.
  3. Consider the //go:debug x509negativeserial=1 directive CodeRabbit mentioned if this will connect to S3-compatible servers with unusual TLS certificates (Ceph, MinIO in dev setups). Worth evaluating but not critical.

istein1
istein1 previously approved these changes Aug 4, 2026
@istein1

istein1 commented Aug 4, 2026

Copy link
Copy Markdown
Member

Looks good, thanks.

nitpick:
The smoke test suppresses stderr/stdout (> /dev/null 2>&1), which means if a binary fails during the build, the error message won't be visible in the build log. Consider dropping the redirects so failures are diagnosable.

@istein1
istein1 merged commit 13fd699 into migtools:master Aug 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants