requests to IPNS URLs may fail

[d70-t]

As noticed in #39, some gateways don't support requesting format=raw for URLs including a path.

Aparrently, trustless-gateway.link also doesn't support format=car for IPNS links (and according to the spec, only format=ipns-record has to be supported). For this reason, ls and info may fail for IPNS links in similar ways as outlined for #39.

E.g.:

This fails:

IPFS_GATEWAY='https://ipfs.io' python3 -c "import fsspec; fs, _ = fsspec.url_to_fs('ipns://ipfs.io'); print(fs.ls('ipfs.io'))"

...
aiohttp.client_exceptions.ClientResponseError: 406, message='Not Acceptable', url='https://trustless-gateway.link/ipns/ipfs.io?format=car&dag-scope=block'

while the following succeeds (assuming a local kubo instance running):

IPFS_GATEWAY='http://127.0.0.1:8080' python3 -c "import fsspec; fs, _ = fsspec.url_to_fs('ipns://ipfs.io'); print(fs.ls('ipfs.io'))"

[d70-t]

It should be possible to circumvent this issue by resolving IPNS names to a CID before doing subsequent requests. However, a question remains, how to do this. IPNS names can be

• true IPNS names (which can be resolved using a trustless gateway)
• DNSLINK names (which can't be resolved in a trustless fashion, the resolution depends on DNS)

One option might be to issue a HEAD request to the /ipns/ API endpoint and inspect the X-Ipfs-Roots response header, which should include the CID for a given name. It's however unclear how to handle name changes and timeouts (e.g. the TTL value from the DNS entry).

[2color]

• true IPNS names (which can be resolved using a trustless gateway)
• DNSLINK names (which can't be resolved in a trustless fashion, the resolution depends on DNS)

In Helia, the TypeScript implementation, we do the following for IPNS URIs:

1. Parse the input string into the following type:

export interface ParsedURL {
  url: URL,
  protocol: 'ipfs' | 'ipns'
  cidOrPeerIdOrDnsLink: string
  path: string[]
  query: Record<string, any>
  fragment: string
}

2. If the protocol is IPNS, try to parse the Peer ID in the string (which has multiple possible representations 🙄)
3. If that fails, try to resolve the DNSLink.

Things to be aware of

• As you correctly noted DNSLink is not self-certifying, so you always have to trust the DNS server, though you can ask more than one for confidence.
• There's this library for DNSLink https://pypi.org/project/dnslink/
• I couldn't find any native python implementation of IPNS. It shouldn't be too hard to write one, though it's a bit of a headache due to the use of both protobufs and CBOR, and the possible record versions for legacy reasons

It's however unclear how to handle name change

What do you mean by name change?

[d70-t]

It's however unclear how to handle name change

What do you mean by name change?

It's not on the top of my head anymore... But I guess what I meant was the name being the same but it's pointing to a different CID. I guess essentially this boils down to handling TTL values from either IPNS or DNS and thinking about implications for caching. As of now, ipfsspec doesn't do caching but fsspec might do down the line. So we might want to figure out if / how we could communicate TTL values down to fsspec, and (probably later) want to figure out if we should do some form of name resolution caching in ipfsspec.

... but maybe for a first implementation it might even be enough to not cache and not care about TTL.

---

The Helia approach seems reasonable. My feeling would be not to go for the Python dnslink library, as

• it didn't see any updates in the last 6 years
• it doesn't do a lot more than calling dnspython
• it doesn't support async resolution, while dnspython does

Thus, I guess I'd propose to directly use async dnspython and (re-) implement the little wrapping layer here.

I guess it will take a while to go through implementing IPNS, but at least we already have some helpful dependencies (i.e. multiformats has CID, dag-cbor and pure-protobuf are already available. I won't have time to dig much deeper into this right now, but I'd be very happy to see more PRs :-)

[2color]

But I guess what I meant was the name being the same but it's pointing to a different CID. I guess essentially this boils down to handling TTL values from either IPNS or DNS and thinking about implications for caching.

Yeah that's correct. DNS libraries handle this automatically for DNS records, for IPNS records, you just rely on the TTL in the record for caching, while also checking that the validity (also referred to as lifetime) timestamp is still valid (in the future).

---

As of now, ipfsspec doesn't do caching but fsspec might do down the line.

I'm pretty new to fsspec and python, so I'm curious how you envision caching working (probably out of scope for this specific issue, so feel free to open a separate issue and link). Happy to share feedback and support.

Thus, I guess I'd propose to directly use async dnspython and (re-) implement the little wrapping layer here.

Yeah that seems sensible.

---

I guess it will take a while to go through implementing IPNS, but at least we already have some helpful dependencies (i.e. multiformats has CID, dag-cbor and pure-protobuf are already available. I won't have time to dig much deeper into this right now, but I'd be very happy to see more PRs :-)

Noted :)