feat: Adding settings, utils, write to S3, and writer close

[malteos]

This PR integrates a couple of general changes from the EOT PR (#54):

• Settings variables are loaded from environment variables in settings.py
• Common CLI methods are moved to utils.py
• Read and write to S3 (via fsspec).
• Missing writer.close() statement is added to CLI and example.

[codecov]

Codecov Report

❌ Patch coverage is 97.63780% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.12%. Comparing base (b6f5f02) to head (f5c159b).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cdx_toolkit/cli.py	92.85%	1 Missing ⚠️
cdx_toolkit/commoncrawl.py	83.33%	1 Missing ⚠️
cdx_toolkit/utils.py	97.29%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #67      +/-   ##
==========================================
+ Coverage   95.78%   97.12%   +1.34%     
==========================================
  Files           7        9       +2     
  Lines         877      939      +62     
==========================================
+ Hits          840      912      +72     
+ Misses         37       27      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[malteos]

Given that the test coverage check fails on this one, what about adjusting the coverage check to a fixed value instead of the last coverage value?

[wumpus]

While the coverage check failed, I don't think it will cause a problem in the next CI run? It's just a slightly easier hurdle next time.

[malteos]

While the coverage check failed, I don't think it will cause a problem in the next CI run? It's just a slightly easier hurdle next time.

I added additional unit tests which should increase the coverage above the threshold.

However, codecov seemed to stopped working after moving the repo to the CC org. I already requested access to the org. I guess you just need to approved codecov again @wumpus.

[malteos]

The Codecov checks pass now.

[damian0815]

Looks generally good to me

[damian0815]

'request changes' I think they only blocker issue is the boto3 require

[malteos]

Thanks for the feedback. I changed the PR accordingly.

See #68 for the context manager issue.

[wumpus]

I would put the s3 support in the install target cdx_toolkit[s3] and also make an [all], similar to what warcio is now.

As an external tool, it's worth having a small install. And there may be situations where the full install on an older python version will fail because of s3 dependencies.

[malteos]

@wumpus I agree. The dependencies have been changed accordingly. The CI also run tests with both setups, "all" and the minimal installation.

[damian0815]

looks good, thank you!