Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hydra_base.lib.cacheopens its diskcacheCacheontempfile.gettempdir()with the defaultDisk, so the templates, project trees and network attributes incache.dbcome back throughpickle.load(cache.py#L15-L19). diskcache has no release that fixes CVE-2025-69872 (GHSA-w8v5-vhqr-4h9v); its newest, 5.6.3, dates from 2023. This PR switches hydra-base to diskcache-community, a maintained, faster fork whose license (Apache-2.0), API and cache files are diskcache's own. The cache is then opened with that package'sSafeDisk: a planted row is removed and read as a miss, and nothing in it runs.Closes #320
Changes
diskcachebecomesdiskcache-community>=5.8.4inpyproject.tomland inrequirements.txt.Cacheand the memcached fallback incache.pyboth getdisk=dc.SafeDisk.with_types(JSONObject, Dataset).with_options(on_error="miss"). Templates and network attributes areJSONObjects, holdingDatasets for default values, and the project trees are dicts, lists and sets of ints; nothing else is stored, so a folder filled by 0.1.20.post0 keeps its hits. Every read already goes throughget().get_all_network_resourceattributes(return_orm=True)no longer writes the cache. It stored SQLAlchemy instances, whichSafeDiskrefuses, under the same key as theJSONObjectlist.test_planted_pickle_is_not_loadedintests/templates/test_template_cache.pyplants a pickle under a template's key through a writer with the defaultDisk, then checks thatget_templateanswers from the database and_plantedis never called.Before and after
The script moves
tempfile.gettempdir()to a new folder, keeps a SQLite file there, and counts calls toTemplate.get_typesas database builds. On load, its planted row calls the script'splanted(), which prints one line and hands back an invented template; a row from any other writer could callos.systeminstead.hydra-base 0.1.20.post0 with diskcache 5.6.3:
With this change (diskcache-community 5.8.4):
Script
Testing
On macOS with Python 3.12:
pytest --no-externaldb tests/templates tests/project tests/attributes: same results as onmaster(156 passed), plus the new test passing.Diskback makes the new test fail, as_plantedthen runs.Update
Every day new libraries getting updated, as example several libraries that were updated yesterday lithops and octotools.
improved performance
Measured over 20 rounds per build, each build in its own venv in random order, medians vs PyPI diskcache 5.6.3 — classic mix (100,000 ops/round, 90% gets / 9% sets / 1% deletes) plus size tiers (100 B to 1 MB) and Django backends, millions of ops total.
Cache.gethitCache.getmissDjangoCache.gethitCache.setCache.setwith expire / Django setP.S I created this package myself after running into this issue, and I'm the one maintaining it.