rfc: FilOne Appliance Deployment Strategy

[bajtos]

📖 Preview

Summary

FilOne Appliance is a set of services operating FilOne node on infrastructure provided by a regional provider. This RFC proposes how to deploy and operate the appliance - first an MPV to get us off the ground quickly, then a direction towards a more robust & scalable solution.

[Copilot]

Copilot wasn't able to review any files in this pull request.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pyropy]

Have you tried running the appliance with yugabyte? I know curio is using yugabyte and maybe we could reach out to them to find out their experiences with the database.

[bajtos]

I know that Curio uses Yugabyte, but I haven't tried it our myself. That's why this RFC says Postgres-compatible 😄

[hannahhoward]

Yugabyte is a later problem. It's main advantage of stock PostGres is multi-node high availability deployments, which you pay for with additional complexity. This is for a future with super large SPs who want 100% uptime

[pyropy]

I think this is the better solution as they might be using some other container orchestrator service.

[pyropy]

Given the new routing policy RFC, would ingot nodes be able to re-route the requests during the 'draining' process?

[bajtos]

I don't know, I guess it's something we can investigate later.

[pyropy]

Should we get notified for every patch version or only for minor and major version releases?

[bajtos]

What we really need is to get a notification if we haven't applied a security patch in time. I think that means notifying us for both patch & minor versions.

[pyropy]

Same question as for the OpenBao version alerts; are we going to get alerts for every patch release or just for minor releases?

[bajtos]

What we really need is to get a notification if we haven't applied a security patch in time. I think that means notifying us for both patch & minor versions.

[pyropy]

Are we going to aggregate logs in somewhere?

[bajtos]

For appliances operated by us, we should probably forward all logs to Grafana.

For appliances operated by region providers, it's up to them to decide.

[bajtos]

Based on our discussion in Eng Weekly today, we want continuous delivery to the dev environment.

[hannahhoward]

Broadly, this is getting close to GTG. (probably drop the raw notes before merge)

I do want to talk about zero downtime. Agree that it's an unrealistic goal for v1.

However, I think we should be clear about our goals and evolution:

1. I think it's important to get to zero downtime as quickly as possible for Ingot. My believe is this simply needs an audit for where Ingot is storing state that isn't fundamentally multiprocess compatible. I believe that is actually very few places. The goal is to clean up any surfaces and then move quickly to blue-green deployments via Caddy as a load balancer.

2. Next comes Piri -- my first thought is we can seperate the web interface of Piri which is near stateless from the task scheduling parts. This may not be as hard as we think as I think the Harmony task scheduler is pretty multiprocess friendly.

3. Postgres/OpenBao are really future work for super large providers.

I just want to capture this knowledge -- if we can keep Ingot up during upgrades, I think we everything else becomes a much later/downstream problem, especially since Ingot has a cache of most of the requested data and can continue serving many requests without Piri.

[hannahhoward]

My only blocker with the current design is we should clear the raw notes before merge. I'd like @frrist to sign off first though.

[frrist]

Thanks for putting this together. The research went deep and much of it I suspect will be valuable to reflect on as we build. But as it stands I think this RFC is doing too much: architecture doc, MVP plan, operating model, post-mvp plan, runbook, and research archive, for several environments at once. For me, that makes it hard to extract the actual decisions being made and why they are being made.

Given that, I'd propose a rescope to just a CD dev environment in the rough shape we want. The appliance material (Podman, immutable OS, partner trust boundary, pdp-gated windows, detailed runbooks, etc.) can then land as follow-up RFCs where each can be properly motivated, discussed, agreed upon, and land as a smaller scoped change.

A few thoughts on this as it stands in addition to the comments I've left in review:

• Podman + Quadlet needs to be motivated or deferred.: The proposal asserts it but never sells it. The alternatives compare against Portainer/Komodo/Kamal but never against the current baseline: plain docker compose. This deserves its own RFC and is a good thing to consider, but I don't believe we need to make a decision on this point yet.
• Provisioning is a gap: For dev and staging we own the whole stack, so the doc should say how the infra itself gets deployed: Terraform/Tofu for the machine, volumes, DNS, firewall, hardware requirements spec (disk size, CPU, memory, location, cloud provider), and splitting the services across different VMs (e.g. SP instance and Forge instance).
• Dev must be CD: The MVP section is currently waffling on this, but I think we are all in agreement now that CD is needed for Dev per your last comment there. Fortunetly the machinery in this RFC already supports this: the git-pull reconciler on a short timer is a CD engine and CI already auto-publishes new images on every merge to main. Policy (imho) being the differentiating factor across environments (How we promote release from dev -> staging -> prod, and how we validate them also deserves its own RFC).
• Secrets: defer to RFC rfc: regional security principles and key management deployment proposal #21 as the authority: Dev then takes the simplest path consistent with that design, which for now roughly looks like a Vault container running in Dev.

[frrist]

Suggested change

	5. Upgrades must honours timing constraints, e.g. we cannot upgrade in the window where the Piri
	node is required to submit a PDP proof.
	5. Upgrades must honour operator obligations on chain.

[frrist]

nit: I'd drop this into a non-requirements section - and possibly include more non-requirements there, aside from no downtime upgrades.

[frrist]

I'd argue its fairly clear that providers operate hardware given:

Our selling point to regional providers: you provide hard drives and a VM, we bring our software and
customers.

[frrist]

nit: drop virtual - that belongs in the OS layer.

[frrist]

Drop Forest, that implementation is untest/unverified with our stack.

[frrist]

far as I am aware, Piri and Ingot don't store sensitive information in config files. All sensitive information is already kept external files.

[bajtos]

You are right that Piri config file does not contain any sensitive information 👍🏻

Based on fil-forge/smelt#11, Ingot config file contains the following secrets:

• Root S3 account key & secret
• postgres_dsn

See environments/staging/piri/config/ingot/config.yaml.tpl

[bajtos]

Actually, I think Piri will need Postgres connection string in its config file once we switch from SQLite to Postgres for storage.

https://github.com/fil-forge/piri/blob/93b85a6d5bf44ce299d3668674bb7d958cab2617/pkg/config/app/storage.go#L38-L41

type PostgresConfig struct {
	// URL is the PostgreSQL connection string (only used when Type is "postgres").
	// Format: postgres://user:password@host:port/dbname?sslmode=disable
	URL url.URL

[frrist]

For a dev environment I'd expect this to be automated. i.e.: When a service merges to main it gets deployed to dev.

[frrist]

While not here yet, I'll make the case that this needs a more thorough validation. At a minimum, in addition to a smoke test, I'd include a soak test: https://en.wikipedia.org/wiki/Soak_testing

[frrist]

I don't want to block this, so approved!

Your call if you want to incorperate any of the comments I left, they're mainly nits. Happy for a doc on a dev environment to land later, if at all.

Only blocking comment here is Hannah's about the notes - which I'd also like cut

[bajtos]

Responding to @frrist's comment:

Podman + Quadlet needs to be motivated or deferred.: The proposal asserts it but never sells it. The alternatives compare against Portainer/Komodo/Kamal but never against the current baseline: plain docker compose. This deserves its own RFC and is a good thing to consider, but I don't believe we need to make a decision on this point yet.

TL;DR: I need to reconsider this decision.

The long answer written by Claude and lightly edited by me:

Arguments for Podman + Quadlet

• Per-service unit granularity fits the mixed-cadence policy. Each service is its own systemd unit, so the reconciler restarts exactly one thing and leaves the rest untouched — directly serving "Postgres and OpenBao must never be auto-touched." Compose reconciles at the project level, and there are documented cases (Container needlessly recreated and then won't start on docker-compose up docker/compose#9357, [BUG] up Incorrectly recreates containers with no changes docker/compose#10259) of it needlessly recreating containers, including Postgres.
• Native on the immutable-OS path. bootc/FCOS ships Podman and does not ship Docker Engine; getting Docker there means rpm-ostree layering, a reboot, and ongoing maintenance against read-only /usr.
• Notify=healthy (Podman ≥5.0) makes a restart genuinely fail if the healthcheck never passes; OnFailure= gives alerting; ordering handles OpenBao-before-Ingot. Compose's equivalent (up -d --wait) is one project-wide gate, not per-unit.
• Rootless, daemonless on semi-trusted partner hardware. No root daemon socket to steal — a well-known privilege-escalation path in Docker's default posture. Fewer long-running moving parts overall.
• Better secrets story. systemd-creds / LoadCredentialEncrypted= gives host-bound, optionally TPM-sealed blobs, versus SOPS+age decrypting into the daemon's environment on the Compose side.

Downsides

• Distro version sensitivity. Ubuntu LTS ships Podman too old for Notify=healthy (24.04 has 4.9.3); Debian 13's 5.4.2 works but predates ReloadCmd= and podman quadlet (5.6.0), needing workarounds. Compose installs trivially anywhere via Docker's apt repo.
• The reconciler needs custom diff/restart logic. Compose's up -d --wait is itself the reconciler and only recreates changed services; with Quadlet you copy changed units, daemon-reload, and selectively restart yourself — daemon-reload alone won't restart anything.
• Weaker newcomer ergonomics and smaller ecosystem. "One YAML, docker compose up -d" is a friendlier mental model, and most GitOps-lite tooling (Portainer, Komodo) is Compose-only, so those options are foreclosed.
• No native zero-downtime story either way, but Quadlet's is more manual. Compose at least has the docker rollout plugin pattern; Quadlet requires hand-rolled blue/green with templated units.
• Live sharp edges. The Notify=healthy + TimeoutStartSec footgun (podman #27290) must be explicitly configured given Piri's up-to-60-minute drain, and the BoltDB→SQLite transition (#28216) is a known concurrency hazard for Quadlet-at-boot topologies.

Note: I rejected podman auto-update as the CD mechanism because AutoUpdate=registry never fires on digest-pinned images — it would make the registry, not git, the source of truth.

[alanshaw]

Minor but FilOne should be written as "Fil One".

[alanshaw]

I think it's redundant to say accessible via S3 (HTTPS) when we have in the same sentence specified "S3-compatible object storage" which is a HTTP API.

Suggested change

	4. Storage - data plane: an S3-compatible object storage, in the same datacenter, accessible via S3 (HTTPS)
	4. Storage - data plane: an S3-compatible object storage, in the same datacenter

[alanshaw]

Seems as though we're replicating a systems package manager with auto updates enabled. Why don't we just ship an apt package (host our own apt source) and use unattended-upgrades (https://wiki.debian.org/PeriodicUpdates) (or alternative e.g. cron-apt) to get auto upgrades?

The package gets to administer service starts/restarts on install/upgrade via maintenance scripts and we get a really nice, easy, way to distribute the appliance.

Installing the appliance is a case of adding the aptitude source:

sudo add-apt-repository "deb http://deb.fil-forge.com/forge <codename> main"

...and install:

sudo apt install fil-one-appliance

...and adding auto updates:

sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-custom << 'EOF'
Unattended-Upgrade::Allowed-Origins {
    "FilForge:stable";
};
EOF

[bajtos]

Interesting idea!

How do you propose to deal with the dependencies (Postgres, OpenBao, etc.)?

[alanshaw]

They are either dependencies of the package (so get installed automatically by apt when installing the appliance) or they just need to be installed out of band or like sudo apt install postgres openbao fil-one-appliance.

[alanshaw]

That said, I'm not sure these have to be local-local, as in on the same machine. As I understand it they need to be available on the local network...