Serve IPFS retrievals

[bajtos]

URL schema:

• https://1-{base32(dataSetId)}-{base32(pieceId)}.ipfs.filbeam.io/{favicon.ico}
• https://link.ipfs.calibration.filbeam.io/{WalletAddres}/{IpfsRootCid}/{favicon.ico} (redirects to the above)

This is a proof of concept that I don't expect to land. Instead, I'll extract smaller pull requests and land them independently.

I will be updating filbeam-ipfs-retriever-calibration worker on CF with the code from this pull request.

I have a special-case path sending the retrieval requests to our Frisbii instance as a workaround until Curio implements IPFS retrievals for PDP deals:

Raw image:

• https://link.ipfs.calibration.filbeam.io/0xc83dbfdf61616778537211a7e5ca2e87ec6cf0ed/bafybeig7sb5kcd3iswkv4tbrgt5hixmto3pdetxiuf723zx2i3pjlv4vae/anthony-delanoix-aDxmYZtYj7g-unsplash.jpg
• https://1-je-0.ipfs.calibration.filbeam.io/anthony-delanoix-aDxmYZtYj7g-unsplash.jpg

CAR format:

• https://link.ipfs.calibration.filbeam.io/0xc83dbfdf61616778537211a7e5ca2e87ec6cf0ed/bafybeig7sb5kcd3iswkv4tbrgt5hixmto3pdetxiuf723zx2i3pjlv4vae/anthony-delanoix-aDxmYZtYj7g-unsplash.jpg?format=car
  • DOES NOT WORK - query string is not preserved after redirect
• https://1-je-0.ipfs.calibration.filbeam.io/anthony-delanoix-aDxmYZtYj7g-unsplash.jpg?format=car

Does not work yet - we need to figure out Cloudflare config:
https://ipfs.calibration.filbeam.io/0x000000000000000000000000000000000000dead/bafybeiagrjpf2rwth5oylc64czsrz2jm7a4fgo67b2luygqjrivjbswuku/rusty-lassie.png

TODO:

• Add CI & CD steps
• Re-enable the skipped tests (see FIXME comments).
  This is blocked until there is at least one calibnet SP running the Curio version that scans PDP pieces for IPFS CAR blocks.
• Create real SP test for subpath support (currently not supported by FOC SPs)
• Support all ?format=raw (later)
• Support for other IPFS Trustless GW options, like dag-scope, etc. (later)
• Calculate egress consumption, account for different size of the cache-miss CAR response vs client-side RAW response. The first iteration could report CAR size instead of the real response size. We will charge clients a bit more (CAR is larger than the raw data), but I think that's okay for a few days until we fix the problem in a follow-up pull request.
• Bot to peridically check IPFS retrievals (later)
• Preserve ?format=car and other query-string params when redirecting from /wallet/cid to 1-{dataset}-{piece}. (later)
• Convert the wallet address to lowercase before querying the DB
• Implement bad-bits lookup
• Don't deploy to mainnet
• Re-enable any skipped tests added in this PR

Links:

• Serve IPFS retrievals #297
• PoC of FilBeam for IPFS roadmap#89

TODO after merge

• Update FilBeam docs
• Update Filecoin Pin docs
• Announce the feature
• Monitor etc., then eventually also deploy to mainnet

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​web3-storage/​car-block-validator@​1.2.2
	@​ipld/​car@​5.4.6
	ipfs-unixfs-exporter@​13.7.3

View full report

[juliangruber]

Manual deployment of IPFS retrievals on FilBeam still works:

• https://link.ipfs.calibration.filbeam.io/0xaEF30c4730CF03b5dE26b57007A0e4a9752df82b/bafkreiheygfzn22dfeos3xoay5cxnfb464znd2rszieyzcinlsgu2z7kau/
• https://1-hdyq-0.ipfs.calibration.filbeam.io/

[rvagg]

but we also skip validation of the CAR? so we pass it on to the client and it becomes their problem to validate? might be fine, but if this is part of the service we're advertising then we should be sure that they follow standard trustless-http protocol for validation of the contents of the CAR.

[BravoNatalie]

Yes, we don't validate it.
Isn't that the expected behavior according to the Trustless Gateway protocol?

I've made that explicit in the documentation.
https://github.com/filbeam/worker/pull/312/changes#diff-cfd04945276d0a793adc966c6641b4a4c93606dc159a46e3ad3195ec757a096aR58

[rvagg]

you have directory entries below, but what about the two cases for non-directories, where you hit file contents but may not know what to do with it; you kind of alude to it in this section but let's be more clear:

1. The root CID itself resolves to some raw content, you don't actually know what it is, you don't even have a filename (we did away with directory wrapping for filecoin-pin in feat(unixfs)!: adopt IPIP-499 unixfs-v1-2025 profile filecoin-project/filecoin-pin#448 so it'll be common even for our content). I think this section says that we just shove it out to the browser and make it their problem to figure out the content type by magic bytes sniffing, I guess that works with modern browsers now.
2. We have a filename, so at least have an extension for the file and could give a reasonable content-type for it like a standard web server. Do we not even do that here and just send it out and make it the browser's problem? Do you know if this is fine and good for modern browsers to render a whole website, JS, CSS, HTML, images, videos, etc.? Is there any reason this might be a risky strategy?

Either way, let's just note that there are two paths here -- we retrieve content that we just know are raw bytes and know nothing else about it, and the case where we at least have the terminal path that tells us the filename which may have an extension.

[rvagg]

see my note there about dag-scope=entity, I think it'd be a better choice in general for everything we're doing here than all.

[rvagg]

Well ... not quite this clean. we made a choice in the frisbii version of trustless http that we'd prioritise streaming, so while traversing, it's possible to find that we don't have a block that we need, perhaps the client didn't upload their full DAG, and we have no useful way of signalling an error in that case, trailers are not a useful error signalling mechanism and we decided not to build in some kind of error signal into the CAR we returned, so we must rely on the fact that the receiver is doing the identical traversal to also discover that the next block isn't present and the CAR terminated prematurely. So it is possible start getting data, be happily traversing down the DAG with unixfs-exporter and then just halt. In which case we need to return some kind of meaningful error. It could be an SP problem, maybe their system is busted in some way, maybe network got disconnected and we don't get a clean disconnect error, but it could also be that the user didn't actually store the entire DAG with that SP.

The nice thing about IPFS retrievals and how they work on the SP side is that the entire DAG doesn't need to be in the same piece. You could have one tiny piece that just has the root CID and then a hundred more pieces that contain CARs with all the rest of the blocks for your content, and frisbii retrieval will just fetch the blocks in the right order out of all of those pieces to serve back to you (so the 1Gb piece limit we currently have in place doesn't stop you from storing and serving a 100Gb movie file using IPFS retrievals). BUT unlike Rainbow and other IPFS gateway software that taps into a broader network, using the DHT or IPNI to discover providers for blocks, the DAG must be all on the same SP for any of this to work. You can't shard your DAG across multiple SPs to have the filbeam retrieval work; but you can shard it and have generic IPFS gateway retrievals work because it does block-by-block lookups as it traverses and can skip from SP to SP to find what it needs. Important distinction that we should probably note somewhere.

[rvagg]

On the other side of the equation here is that there's caching potential for block-by-block, that should be noted, maybe it's in the slack thread linked; /path/to/some/deep/nested/content.html where that html includes 100 jpgs has to re-fetch the DAG for /path/to/some/deep/nested/ for each jpg. Typically these are quite small and it's not really a big deal. There are some cases where it might matter but on balance I'd say that for our narrow case where the client is choosing to opt in to this service by storing the entire DAG on one of the SPs in this service rahter than scattering it around the internet and crossing their fingers, CAR retrieval makes more sense. But it's not a clear-cut case I think.

[rvagg]

also reinforced by my point above that perhaps the SP is having a bad day and their database had problems resolving all the blocks in the CAR, or one of their disks holding a piece that contained part of the DAG was down for a moment and you got an incomplete CAR; you don't want to associated strictly a forever-cache with a potential blip in serving.

It's true that CIDs point to immutable content, but you're dealing with a whole string of CIDs that must be validated together. In theory CID+subpath is unique, but there a ways for it to go bad and not know until you run the traversal.

[rvagg]

You do have the same problem as I outlined above I think if we're streaming the content to the user unless you hard-abort the connection or something (IIRC there's no good, reliable signal for terminating an HTTP stream that would invalidate a cache? I think I remember we wrestled with this for a while in the trustless HTTP design). Might be worth a bit of research to make sure that this isn't going to cause problems -- tell the client that their 100Mb movie file is correct for a year, only able to fetch and serve 50Mb of it, what then? is the movie now 50Mb for that client for a year? Can we terminate in a way that will reliably tell the client that the max-age shouldn't be respected?

[rvagg]

if in doubt, we should try and pull in @lidel to get some answers since this kind of nuance is right up his alley

[BigLep]

@BravoNatalie : I'm going to put this back to TODO since I know this work is currently on pause. If there are still "TODOs after merge" that are listed in the issue description, lets capture those in the project issue #297 which will stay open after the merge so we don't lose track of them.