Suggestion: Indexed bzip2 writer that calculates offsets while writing

[thomasj02]

I'd like to use indexed_bzip2 to read files that I've previously written.

Presumably, it's low-cost to calculate the offsets on the fly while writing the file, instead of having to re-open the bz2 file and calculate the offsets after the write is complete. If that were possible, then I could save the offsets as described at https://github.com/mxmlnkn/indexed_bzip2/tree/master/python/indexed_bzip2#storing-and-loading-the-block-offset-map so that they would never have to be calculated separately.

This would make the overhead of using indexed_bzip2 significantly lower.

[mxmlnkn]

Sorry for the very late answer :(. I was not able to reply in January and then lost the notification. This should work to at least reduce disc accesses:

lbzip2 -c file | tee file.bz2 | rapidgzip --export-index file.bz2.index

This compresses the file, while also streaming the compressed data to rapidgzip / ibzip2, which creates the index and nothing more.

Adding compression to this tool feels out of scope, although the sibling repo has such an issue.