[Bug]: pillarenv containing "/" leaks temp files since 3008.0

[dequis]

What happened?

Since upgrading to 3008.2, i've been getting spam of this in my master logs:

2026-07-08 22:21:53,318 [salt.master      :3013][ERROR   ][2901879] Error in function _pillar:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cache/__init__.py", line 157, in store
    return self.modules[fun](bank, key, data, expires=expires, **self.kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 177, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1480, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1495, in _run_as
    ret = _func_or_method(*args, **kwargs)
TypeError: store() got an unexpected keyword argument 'expires'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cache/localfs.py", line 65, in store
    salt.utils.atomicfile.atomic_rename(tmpfname, outfile)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/salt/master/pillar/tmpit4hyi64' -> '/var/cache/salt/master/pillar/someserver:someenv/beta.p'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 3007, in run_func
    ret = getattr(self, func)(load)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/master.py", line 2678, in _pillar
    data = pillar.compile_pillar()
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/pillar/__init__.py", line 1477, in compile_pillar
    self.cache.store("pillar", self.pillar_key, pillar_data)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cache/__init__.py", line 169, in store
    return self.modules[fun](bank, key, data, **self.kwargs)
           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 177, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1480, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/loader/lazy.py", line 1495, in _run_as
    ret = _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.14/site-packages/salt/cache/localfs.py", line 67, in store
    raise SaltCacheError(
        f"There was an error writing the cache file, {base}: {exc}"
    )
salt.exceptions.SaltCacheError: There was an error writing the cache file, /var/cache/salt/master/pillar: [Errno 2] No such file or directory: '/var/cache/salt/master/pillar/tmpit4hyi64' -> '/var/cache/salt/master/pillar/someserver:someenv/beta.p'

I have this in my salt master config:

pillar_roots:
  someenv/beta:
    - /srv/pillar/someenv/beta
  someenv/rc:
    - /srv/pillar/someenv/rc

And literal millions of files in /var/cache/salt/master/pillar/tmp* (about 10 per minute, adds up over the weeks)

I don't have pillar_cache in my config so it should be default false, so it's unexpected to see it trying to save a cache entry. But maybe it was always writing files there and because it wasn't failing I never noticed? Seems to be happening through minion_data_cache which defaults to true, and as far as I can tell that didn't change.

This seems to trace back to PR #68030, which seems to define a new cache key that includes minion_id:pillarenv.

+    @property
+    def pillar_key(self):
+        if not self.opts["pillarenv"]:
+            return self.minion_id
+        else:
+            return f"{self.minion_id}:{self.opts['pillarenv']}"

But when passed to the localfs store, the slash becomes a problem.

Ideas to approach this:

• Sanitize slashes in pillarenv? Maybe too specific to localfs, but that's the default anyway.
• Sanitize slashes in localfs?
• Makedirs in localfs?
• Go back to minion_id only as cache key?
• Disallow slashes in pillarenvs?

As a workaround, for the time being, I'll be using 3007.x. I could change the names of the pillarenvs I guess, but would rather not.

(also the bug report template wouldn't let me pick a major version newer than 3007.x)

Type of salt install

Official deb

Major version

3007.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output

Salt Version:
          Salt: 3008.2

Python Version:
        Python: 3.14.6 (main, Jun 11 2026, 02:19:05) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 48.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.50
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.14
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.7
           ZMQ: 4.3.5

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: ubuntu 24.04.4 noble
        locale: utf-8
       machine: x86_64
       release: 6.17.0-1017-aws
        system: Linux
       version: Ubuntu 24.04.4 noble

[donkopotamus]

We have also observed this when attemping to move from 3006.x to 3008. In our case we use:

• git_pillar for pillar data; and
• commonly have pillarenv (and saltenv) containing a /, as that is a very common branch naming scheme (eg feature/<something>, release/<something> etc

This causes the above kind of SaltCacheError on the master when pillar data is being generated, resulting in that data not being available at all downstream, with minions immediately failing during a state.apply with a message such as:

salt.exceptions.SaltClientError: Got a bad pillar from master, type str, expecting dict:

Tip

It appears to be a problem with the default localfs cache only, as / in a key is meaningful to the filesystem. If we instead move to using eg mmap_cache we appear to avoid the problem.