Add ε-serde support (no change to the code)

[vigna]

This PR adds gated ε-serde support to fst. There is no change in the fst code—just a few gated derives and implementations.

ε-serde tries to solve in general the problem fst solves specifically for memory-mapping large amount of data, but it does it using derive macros for generic data structures; it replaces recursively vectors or boxed slices with reference to a byte buffer (e.g., memory-mapped) and strings with references to str.

The additional feature gained by fst is the possibility of serializing and map into memory directly any hierarchical combination of types, not just leaves (there are a few examples and the end of test.rs.

fst was in fact the trigger for expanding the scope of ε-serde: originally, we were just replacing type parameters which where the type of a field; now, we replace them recursively everywhere (which turned out to be fairly complicated).

This PR does not modify any fst code, but because of that it needs to provide hand-made serialization/deserialization implementations for the leaves (Str, Subsequence) because we can only support memory mapping, and not full deserialization.

We are also proposing another PR that just uses gated derives, but in this case we need to add a parameter to Str and Subsequence, so that it can be substituted. We gain full deserialization, which is of interest as ε-serde can fully deserialize into transparent huge pages, which might provide some performance boost. But this requires a small incompatible change to the exposed API.

[BurntSushi]

Thanks for sending this PR! e-serde sounds really cool, but in terms of adding support into fst there are a couple problems:

• Firstly is the LGPL. I don't add dependencies with copyleft licenses. No, I do not want to debate this. Yes, I see that e-serde is dual licensed, but I don't want even the appearance of dependencies of my libraries injecting copyleft licensing. The LGPL is very non-standard in the Rust crate ecosystem.
• Secondly is versioning. e-serde has had 13 breaking change releases. Adding e-serde support means linking fst's semver versioning to be at least as aggressive as e-serde's. Or otherwise maintaining support for multiple versions of e-serde simultaneously.

The rkyv folks, have, I think, figured out how to deal with this. I think they tried adding support to crates like you're doing here, but that didn't work out.

ε-serde tries to solve in general the problem fst solves specifically for memory-mapping large amount of data, but it does it using derive macros for generic data structures

To clarify here, fst is more than this. It's also compressing the data. The format is also bespoke, so it can't be read into any Rust data structure other than a &[u8]. From there, you need the fst library to actually read it.

[vigna]

Thanks for sending this PR! e-serde sounds really cool, but in terms of adding support into fst there are a couple problems:

• Firstly is the LGPL. I don't add dependencies with copyleft licenses. No, I do not want to debate this. Yes, I see that e-serde is dual licensed, but I don't want even the appearance of dependencies of my libraries injecting copyleft licensing. The LGPL is very non-standard in the Rust crate ecosystem.

More than thirty years ago I've been exactly in your position (on the other side), so I understand it perfectly. That's not an issue—we can just remove (and maybe add MIT—Rust projects seems to be mostly MIT + Apache).

• Secondly is versioning. e-serde has had 13 breaking change releases. Adding e-serde support means linking fst's semver versioning to be at least as aggressive as e-serde's. Or otherwise maintaining support for multiple versions of e-serde simultaneously.

Well... let's say it has been complicated 😂. I don't think that's true tho, unless you want to use it for archival purposes, which was never one of our goals. There has been just one change to the major version of the file format, which I think is more relevant for a serialization framework.

The rkyv folks, have, I think, figured out how to deal with this. I think they tried adding support to crates like you're doing here, but that didn't work out.

I think this is kind of apple-and-oranges, as rkyv gives you back a different type, and that type is likely bound to the version of rkyv. That's exactly what we wanted to avoid: when you deserialize with zero-copy aliasing, or you do memory mapping, we give you back your own type, not another one. The version just depends on your crate, not on ours.

To clarify here, fst is more than this. It's also compressing the data. The format is also bespoke, so it can't be read into any Rust data structure other than a &[u8]. From there, you need the fst library to actually read it.

I know what fst does: we stumbled into it exactly when looking for ways to compress strings algorithmically. What I was trying to say is that fst does memory mapping in specific cases, but with ε-serde you serialize and map into memory any hierarchical combination of types in the crate, which is currently not possible. This might also be something for which there is zero interest, of course (e.g., because these hierarchies of types are built on an as-needed basis from the basic memory-mapped types).

[BurntSushi]

Well... let's say it has been complicated 😂. I don't think that's true tho, unless you want to use it for archival purposes, which was never one of our goals. There has been just one change to the major version of the file format, which I think is more relevant for a serialization framework.

0.12.0 -> 0.13.0 is a semver breaking change release in the Cargo crate ecosystem. Regardless of changes to the file format, fst with a public dependency on epserde 0.13.0 requires a semver breaking change release itself (to 0.5.0, for example) if epserde 0.14.0 is released and users expect to use epserde 0.14.0 with fst. The only alternative to this is fst maintaining support for multiple semver incompatible releases of epserde simultaneously.

[vigna]

Oh well, yes, you're right—I wasn't thinking of that, even if it actually happened to me with other crates.