Skip to content

Read the disk cache through diskcache-community's SafeDisk - #321

Open
avgvi wants to merge 1 commit into
hydraplatform:masterfrom
avgvi:fix/diskcache-safedisk
Open

avgvi wants to merge 1 commit into
hydraplatform:masterfrom
avgvi:fix/diskcache-safedisk

Conversation

@avgvi

@avgvi avgvi commented Nov 9, 2026 •

Copy link
Copy Markdown

Summary

hydra_base.lib.cache opens its diskcache Cache on tempfile.gettempdir() with the default Disk, so the templates, project trees and network attributes in cache.db come back through pickle.load (cache.py#L15-L19). diskcache has no release that fixes CVE-2025-69872 (GHSA-w8v5-vhqr-4h9v); its newest, 5.6.3, dates from 2023. This PR switches hydra-base to diskcache-community, a maintained, faster fork whose license (Apache-2.0), API and cache files are diskcache's own. The cache is then opened with that package's SafeDisk: a planted row is removed and read as a miss, and nothing in it runs.

Closes #320

Changes

  • Dependency: diskcache becomes diskcache-community>=5.8.4 in pyproject.toml and in requirements.txt.
  • Disk: the default Cache and the memcached fallback in cache.py both get disk=dc.SafeDisk.with_types(JSONObject, Dataset).with_options(on_error="miss"). Templates and network attributes are JSONObjects, holding Datasets for default values, and the project trees are dicts, lists and sets of ints; nothing else is stored, so a folder filled by 0.1.20.post0 keeps its hits. Every read already goes through get().
  • ORM rows: get_all_network_resourceattributes(return_orm=True) no longer writes the cache. It stored SQLAlchemy instances, which SafeDisk refuses, under the same key as the JSONObject list.
  • Test: test_planted_pickle_is_not_loaded in tests/templates/test_template_cache.py plants a pickle under a template's key through a writer with the default Disk, then checks that get_template answers from the database and _planted is never called.

Before and after

The script moves tempfile.gettempdir() to a new folder, keeps a SQLite file there, and counts calls to Template.get_types as database builds. On load, its planted row calls the script's planted(), which prints one line and hands back an invented template; a row from any other writer could call os.system instead.

hydra-base 0.1.20.post0 with diskcache 5.6.3:

1. template read from the database: 'River basin' ['Reservoir'] | database builds: 1
2. same template, served from the cache: 'River basin' ['Reservoir'] | database builds: 1
3. tampered entry
  code from the cache directory ran in this process
  result: 'Not from the database' [] | database builds: 1

With this change (diskcache-community 5.8.4):

1. template read from the database: 'River basin' ['Reservoir'] | database builds: 1
2. same template, served from the cache: 'River basin' ['Reservoir'] | database builds: 1
3. tampered entry
treating an unreadable entry in '…' as missing: UnpicklingError: SafeDisk refuses to unpickle __main__.planted
  result: 'River basin' ['Reservoir'] | database builds: 2
Script
import logging
import shutil
import tempfile
import warnings
from unittest.mock import patch

try:
    import diskcache_community as diskcache
except ModuleNotFoundError:
    import diskcache  # hydra-base 0.1.20.post0 from PyPI

# hydra-base opens its cache in tempfile.gettempdir() on import; use a fresh folder.
work = tempfile.mkdtemp()
tempfile.tempdir = work
logging.disable(logging.INFO)  # hydra-base logs at INFO on import
warnings.simplefilter("ignore", SyntaxWarning)

import hydra_base as hb  # noqa: E402
from hydra_base.db.model import Template  # noqa: E402
from hydra_base.lib import template as template_lib  # noqa: E402
from hydra_base.lib.cache import cache, clear_cache  # noqa: E402
from hydra_base.lib.objects import JSONObject  # noqa: E402
from hydra_base.util.hdb import create_default_users_and_perms, make_root_user  # noqa: E402

logging.basicConfig(format="%(message)s", force=True)

hb.db.connect(f"sqlite:///{work}/hydra.db")
create_default_users_and_perms()
make_root_user()

reservoir = JSONObject({"name": "Reservoir", "resource_type": "NODE", "typeattrs": []})
new = JSONObject({"name": "River basin", "templatetypes": [reservoir]})
template_id = template_lib.add_template(new, user_id=1).id
clear_cache()

db_builds = []
real_get_types = Template.get_types


def counting_get_types(self, *args, **kwargs):
    db_builds.append(self.id)
    return real_get_types(self, *args, **kwargs)


def planted():
    print("  code from the cache directory ran in this process")
    return JSONObject({"name": "Not from the database", "templatetypes": []})


class Planted:
    def __reduce__(self):
        return (planted, ())


def template_summary():
    tmpl = template_lib.get_template(template_id, user_id=1)
    return repr(tmpl.name), [t.name for t in tmpl.templatetypes]


with patch.object(Template, "get_types", counting_get_types):
    print("1. template read from the database:", *template_summary(), "| database builds:", len(db_builds))
    print("2. same template, served from the cache:", *template_summary(), "| database builds:", len(db_builds))

    print("3. tampered entry")
    with diskcache.Cache(cache.directory) as other_writer:
        other_writer.set(f"template_{template_id}", Planted())
    print("  result:", *template_summary(), "| database builds:", len(db_builds))

hb.db.close_session()
cache.close()
shutil.rmtree(work)

Testing

On macOS with Python 3.12:

  • pytest --no-externaldb tests/templates tests/project tests/attributes: same results as on master (156 passed), plus the new test passing.
  • Putting the default Disk back makes the new test fail, as _planted then runs.
  • A cache written by hydra-base 0.1.20.post0 still gives its hits on this branch.

Update

Every day new libraries getting updated, as example several libraries that were updated yesterday lithops and octotools.

improved performance

Measured over 20 rounds per build, each build in its own venv in random order, medians vs PyPI diskcache 5.6.3 — classic mix (100,000 ops/round, 90% gets / 9% sets / 1% deletes) plus size tiers (100 B to 1 MB) and Django backends, millions of ops total.

Operation vs diskcache 5.6.3
Cache.get hit ~17% faster
Cache.get miss ~12% faster
DjangoCache.get hit ~16% faster
Cache.set ~12–23% faster, best case
Cache.set with expire / Django set ~7–11% faster best case

P.S I created this package myself after running into this issue, and I'm the one maintaining it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache rows in tempfile.gettempdir()/cache.db are loaded with pickle (diskcache, CVE-2025-69872)

1 participant