rfc: forge network service identities

[alanshaw]

📖 Preview

Summary

Proposal for service identities/URLs in the Forge network.

[bajtos]

We are currently using <service>.staging.<DOMAIN> format, so that all staging services share the same higher-level domain name. What are the benefits of the proposed format staging.<service>.<DOMAIN>?

[alanshaw]

I think I'd have done this in the past (<service>.<env>.<DOMAIN>), but I've had push back on doing it like this and in Storacha we switched to the <env>.<service>.<DOMAIN> as above.

I guess the benefit is that when you read it, it's immediately obvious that you are talking to the staging service and not production, whereas if it's buried later in the domain it's easier to miss.

It's also easier to sort and clean up ephemeral environments like pr-456.<service>.<DOMAIN> since they are prefixed with the PR number.

I think in the past it was easier/cheaper to buy a wildcard *.<env>.<DOMAIN> SSL cert but nowadays it does not matter.

so that all staging services share the same higher-level domain name

I am interested in what the benefits of doing it this way are?

[bajtos]

To me, the main benefit of <service>.<env>.<TLD> is matching the way how infrastructure is defined and managed.

Each environment consists of multiple services deployed to that stack. Each domain name (<env>.<TLD>) owns its children (<service>.<env>.<TLD>). It's easy to delegate at environment-level granularity.

IaaC (Terraform/Pulumi/SST) files are organized by environments (one directory per environment containing nested services), not services (one directory per service containing all environments where this service is deployed to).

Adding a new enviroment (staging, pr-123, etc.) means copying & deploying (& late removing) one per-environment group of resources, not adding a new resource to every per-service group.

I guess the benefit is that when you read it, it's immediately obvious that you are talking to the staging service and not production, whereas if it's buried later in the domain it's easier to miss.

Agreed 👍

I think in the past it was easier/cheaper to buy a wildcard *.. SSL cert but nowadays it does not matter.

I agree that wildcard certs are no longer needed.

It's also easier to sort and clean up ephemeral environments like pr-456.<service>.<DOMAIN> since they are prefixed with the PR number.

I have a different opinion here. In my proposal, you look at everything below pr-466.<DOMAIN> to see what's in the environment.

Here are more arguments from Claude in favour of <service>.<env>.<TLD>:

• Single delegable zone per environment. staging.fil.one is one node covering the whole environment. One NS record in the parent delegates the entire staging subtree to a separate Route 53 hosted zone in a separate AWS account — the mechanism AWS documents under "Routing traffic for subdomains.

  • Miroslav: The relevance of this argument depends on how we structure prod vs non-prod hostnames. We use two different AWS accounts for prod and non-prod, and use a short static list of zone delegations from CloudFlare to AWS - staging.fil.one and dev.fil.one is delegated to AWS Route53 in the sandbox account. Having the same auth.fil-forge.com for both prod and non-prod would require one delegation per stage, which would be tricky (if possible at all) to implement. We were also discussing three AWS accounts (prod, staging, dev/PR-previews), which would further complicate this matter.

    I guess an easy way out is to have a different <DOMAIN> for prod & non-prod, basically one domain per AWS account - e.g. fil-forge.com for production, fil-forge.dev for non-prod. But that could make the distinction (prod vs dev) even more difficult to spot because it's at the end of the URL?

• Cookie, HSTS, and PSL containment follows the environment boundary. Cookies scope to eTLD+1 and flow down the subtree (RFC 6265) so a cookie set at staging.fil.one can never reach a production host, and an HSTS policy with includeSubDomains on the staging parent stays inside staging.

  • Miroslav: I think this is relevant to our web console and not much relevant for non-production Forge deployments, because Forge don't use cookies, don't need HSTS & PSL in non-prod.

• Ephemeral environments become whole disposable zones. pr-456.fil.one holds s3.pr-456.fil.one, api.pr-456.fil.one; teardown is "delete one hosted zone," cleaner than deleting N records scattered across N per-service zones.

  • Miroslav:I guess this is easy to automate via Terraform, so it's a weak argument.

• Composes with per-environment Pulumi stacks and per-environment AWS accounts — the DNS boundary, account boundary, cert boundary, and stack boundary all land on the same line.

What are arguments for <env>.<service>.<TLD>:

• The environment is the first thing you read.
• Prefix sorting and tab-completion by environment.
• Ephemeral environments read as a PR-number-prefixed family
• Per-service delegation - s3.fil.one can be handed to the team owning S3, across all environments.
  • Miroslav: this does not apply to us yet, since we have one team owning all services
• Grafana/Loki filtering on a leftmost label ({host=~"staging\\..*"}) is marginally simpler than an embedded label — though =~".*\\.staging\\..*" is hardly onerous.
  • Miroslav: We use different filtering in Fil One. Our service name labe is filone-<stage>-<service>. I think we can achieve the same for forge with service name labels like forge-<stage>-<service>. Having said that, I think these log group names should mirror the domain names. If we have <env>.<service>.<TLD>, then the log group name should be forge-<service>-<env>.

Also: With a bare production apex, staging becomes a child of the production hostname. If prod is s3.fil.one and staging is staging.s3.fil.one, then any cookie production sets on s3.fil.one is transmitted to staging, and production's HSTS includeSubDomains policy governs staging. This is the sharpest argument against the layout, and it is structural — you escape it only by giving production its own explicit label (prod.s3.fil.one) and never serving the bare name.

• Miroslav: As I mentioned above, this is a weak argument, becase Forge consists of API services and no websites, therefore HSTS & Cookie isolation does not apply. (I think?)

[alanshaw]

It's not a hill I will die on. You are currently leading this work so I think you get to choose.

[bajtos]

FWIW, we don't use fil-one domain names for the S3 endpoints in staging, we are using the orchestrator's domain name (https://s3.dev.aur.lu or https://us-east-1.fortilyx.com).

Having wrote that, I think it would be nice to use the FilOne/Forge domain for the staging S3 endpoint.

[alanshaw]

So, yeah, in this case the orchestrator is us so I used the fil-forge.com domain.

[bajtos]

I am curious: why fil-forge.com and not fil-one.com? (I don't have a strong opinion.)

[alanshaw]

I guess for the same reasons we chose to have a fil-forge github org and not put the repos in the fil-one org.

Forge is != Fil One. Fil One is an S3 interface that backs on to multiple storage networks, one of them is the Forge network.

[bajtos]

Do you have any concerns about PSL not being able to distinguish between prod and non-prod stages?

I am wondering if we should adopt filonecontent.dev for non-prod stages, or perhaps keep non-prod S3 endpoints on the fil-forge.com domain.

Thoughts?

[alanshaw]

No, that's the whole point of PSL isn't it? The suffix is not flagged as bad.

[bajtos]

@alanshaw can you please create a follow-up Linear ticket to apply the changes proposed in this RFC? Feel free to assign the ticket to me.