crates.io top 20k crates verification process

[paolobarbolini]

I've scraped (I'm lazy, I should have used the database dumps) the top 20k crates by recent downloads count. I've published the list and the script at https://gist.github.com/paolobarbolini/b5101b3ad378bcb6bc5c282349edfd4c.

I'll soon be getting a server from Hetzner with 320 GB of disk and see if I can go through the entire list without running out of disk space. I'll also use the list as a way of fixing some of the shortcomings which have been reported in other issues.

⚠️⚠️⚠️ WARNING ⚠️⚠️⚠️

Before you open issues in the projects you think are affected, investigate the reports thoroughly. This software is still v0.0.1 for a very good reason.

[paolobarbolini]

While analyzing crates I caught an error and got stuck trying to fetch submodules from https://github.com/quickwit-oss/tantivy/tree/dff022b30aff6bcd4df7e908f6fa2f86e551204b because it used git over SSH. I guess GIT_TERMINAL_PROMPT=0 isn't enough.

[paolobarbolini]

I've opened #12 to stop missing repositories from continuously blocking the clone process. This patch is already present on the machine I'm doing the scanning from.

[paolobarbolini]

The clone process was interrupted by ntex-rs/ntex#333 😅. I've applied a patch locally for now and I'll see how to fix it permanently. Turns out with a large enough pool of crates almost every ensure! will probably get hit at some point

[paolobarbolini]

While analyzing crates I caught an error and got stuck trying to fetch submodules from https://github.com/quickwit-oss/tantivy/tree/dff022b30aff6bcd4df7e908f6fa2f86e551204b because it used git over SSH. I guess GIT_TERMINAL_PROMPT=0 isn't enough.

It just happened again with the investments crate

[paolobarbolini]

I'm starting to analyze the logs, which I'll publish once we finish analyzing all crates. I've already encountered something, which I've reported at rust-db/refinery#323

[paolobarbolini]

I've also opened TimelyDataflow/timely-dataflow#559

[link2xt]

Not sure if it is in top 20k crates, but here is a pgp crate issue: rpgp/rpgp#327
EDIT: pgp is in top 20k

[paolobarbolini]

Processing got stuck at ~18.5k crates. Heres the log: output.log.gz WARNING: I've already verified that there are a lot of false positives.

I'll merge #18, #19 and #20 locally and have it re-run on all crates

[paolobarbolini]

pgp is in top 20k

Don't worry about the 20k limit, it's just a number I've picked for doing the "official" scrape after having done a very rough 5k one in the previous days 😃

[link2xt]

Issue for brotli, have not tested if it is reproducible: dropbox/rust-brotli#178
async-rusqlite PR to add repository: jsdw/async-rusqlite#2