Users not found when password database is in a non-standard path

[ocaisa]

Saw this on the compute nodes of Lumi:

{EESSI/2025.06} alanocai@nid005265:~> id -un
id: cannot find name for user ID 327004947
327004947
{EESSI/2025.06} alanocai@nid005265:~> export LD_PRELOAD=/lib64/libnss_db.so.2
{EESSI/2025.06} alanocai@nid005265:~> id -un
alanocai

So looks like we need another symlink?

[boegel]

@bedroge Thoughts on this?

[ocaisa]

Do we just add another symlink under https://github.com/EESSI/compatibility-layer/blob/main/scripts/prefix-symlink-host-paths.sh#L31 ?

[ocaisa]

Adding another symlink is not really an option, while the LD_PRELOAD worked for 2025.06, it gives a glibc error for 2023.06. I wonder if we can build our own libnss_db.so.2

[boegel]

This library is available in EESSI compat layers:

/cvmfs/software.eessi.io/versions/2025.06/compat/linux/x86_64/lib64/libnss_db.so.2

and

/cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64/lib64/libnss_db.so.2

[bedroge]

@ocaisa So this happens with both 2023.06 and 2025.06, right?

Could you do an strace id -un and grep for var/lib/sss? I'm suspecting that we may have to change https://github.com/EESSI/gentoo-overlay/blob/main/sys-auth/sssd/sssd-2.8.2.ebuild#L152 and point it to the host location if the daemon is not built, as we do below for e.g. --with-pipe-path.

[ocaisa]

Almost nothing for /var at all:

{EESSI/2025.06} alanocai@nid005779:~> strace id -un |& grep '/var'
openat(AT_FDCWD, "/var/db/passwd.db", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

but this is a bit more interesting:

{EESSI/2025.06} alanocai@nid005779:~> strace id -un |& grep passwd
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/var/db/passwd.db", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
{EESSI/2025.06} alanocai@nid005779:~> LD_PRELOAD=/lib64/libnss_db.so.2 strace id -un |& grep passwd
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/var/lib/misc/passwd.db", O_RDONLY|O_CLOEXEC) = 3

[bedroge]

This may be relevant here: NixOS/nixpkgs#229486

[ocaisa]

So it seems like the database is just not where it is expected. Is it possible to add additional options for that?

[ocaisa]

Nope, seems not. Only option would be a symlink at their level, or a variant symlink at ours (and also patching glibc to look there?)

[bedroge]

Couldn't find such an option either, looks like it's baked into glibc. It's defined in $EPREFIX/usr/include/paths.h:

#define _PATH_VARDB     "/var/db/"

and then used here: https://github.com/bminor/glibc/blob/04e750e75b73957cf1c791535a3f4319534a52fc/nss/nss_db/db-init.c#L27.

I suppose we could indeed change that variable and point it to a variant symlink on CVMFS.

[bedroge]

That variable was initially set to $EPREFIX/var/db by the way, but we deliberately deprefixify these paths with a patch provided by @bartoldeman (https://github.com/EESSI/gentoo-overlay/blob/main/etc/portage/env/sys-libs/glibc), with the idea to use the host files for such services. But if distros decide to use non-default locations, we may have to rethink this.

[bartoldeman]

yes, although we're using sssd (which uses pipe or socket) so I haven't seen libnss_db in action yet.
https://linux.die.net/man/5/nsscache.conf says:

nssdb CACHE OPTIONS

These options configure the behaviour of the nssdb cache.

nssdb_dir
    Directory to store the Berkeley DB databases. Defaults to the current directory. Note that nss_db hardcodes the path to /var/lib/misc on Debian systems, and /var/db on Red Hat systems. 

one could go via variant symlinks, or patching libnss_db to check both locations.

[ocaisa]

So one way to do it would be to restore $EPREFIX/var/db as the search location and then make $EPREFIX/var/db/passwd.db a variant symlink which by default points to /var/db/passwd.db...and if we are doing that we should probably evaluate our other choices to see if we should prefer a similar approach for those too

[bedroge]

I reinstalled glibc in a local 2026.06 test build, and these are the files that Gentoo Prefix automatically prefixifies (and we used to undo some of those changes afterwards):

 * Prefixifying hardcoded path
 *   Updating libio/iopopen.c ...                                                                                                                                                [ ok ]
 *   Updating shadow/lckpwdf.c ...
sed: can't read shadow/lckpwdf.c: No such file or directory                                                                                                                      [ !! ]
 *   Updating resolv/netdb.h ...                                                                                                                                                 [ ok ]
 *   Updating resolv/resolv.h ...                                                                                                                                                [ ok ]
 *   Updating elf/rtld.c ...                                                                                                                                                     [ ok ]
 *   Updating nss/nss_compat/compat-grp.c ...                                                                                                                                    [ ok ]
 *   Updating nss/nss_compat/compat-initgroups.c ...                                                                                                                             [ ok ]
 *   Updating nss/nss_compat/compat-pwd.c ...                                                                                                                                    [ ok ]
 *   Updating nss/nss_compat/compat-spwd.c ...                                                                                                                                   [ ok ]
 *   Updating nss/bug-erange.c ...                                                                                                                                               [ ok ]
 *   Updating nss/nss_files/files-XXX.c ...                                                                                                                                      [ ok ]
 *   Updating nss/nss_files/files-init.c ...                                                                                                                                     [ ok ]
 *   Updating nss/nss_files/files-initgroups.c ...                                                                                                                               [ ok ]
 *   Updating sysdeps/generic/paths.h ...                                                                                                                                        [ ok ]
 *   Updating sysdeps/unix/sysv/linux/paths.h ...                                                                                                                                [ ok ]
 *   Updating sysdeps/posix/system.c ...                                                                                                                                         [ ok ]
 *   Updating nss/db-Makefile ...  

I've done a diff between the original ones and the patched files to find out which paths were prefixified, and got the following:

resolv/netdb.h:
#define _PATH_HEQUIV            "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/hosts.equiv"
#define _PATH_HOSTS             "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/hosts"
#define _PATH_NETWORKS          "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/networks"
#define _PATH_NSSWITCH_CONF     "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/nsswitch.conf"
#define _PATH_PROTOCOLS         "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/protocols"
#define _PATH_SERVICES          "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/services"


resolv/resolv.h: 
#define _PATH_RESCONF        "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/resolv.conf"


elf/rtld.c:
  static const char preload_file[] = "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/ld.so.preload";


nss/nss_compat/compat-grp.c:
      ent->stream = __nss_files_fopen ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group");


nss/nss_compat/compat-initgroups.c:
  ent->stream = __nss_files_fopen ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group");


nss/nss_compat/compat-pwd.c:
      ent->stream = __nss_files_fopen ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/passwd");


nss/nss_compat/compat-spwd.c:
      ent->stream = __nss_files_fopen ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow");


nss/bug-erange.c:
      if (access ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/resolv.conf", R_OK))


nss/nss_files/files-XXX.c:
#define DATAFILE      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/" DATABASE


nss/nss_files/files-init.c:
  register_file (cb, pwddb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/passwd", 0);
  register_file (cb, grpdb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group", 0);
  register_file (cb, hstdb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/hosts", 0);
  register_file (cb, hstdb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/resolv.conf", 1);
  register_file (cb, servdb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/services", 0);
  register_file (cb, netgrdb, "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/netgroup", 0);


nss/nss_files/files-initgroups.c:
  FILE *stream = __nss_files_fopen ("/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group");


sysdeps/generic/paths.h 
#define _PATH_DEFPATH   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin"
#define _PATH_STDPATH  "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/sbin:/sbin"
#define _PATH_BSHELL    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin/sh"
#define _PATH_CSHELL    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin/csh"
#define _PATH_DEVDB     "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/dev.db"
#define _PATH_GSHADOW   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/gshadow"
#define _PATH_LASTLOG   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/lastlog"
#define _PATH_MAILDIR   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/mail"
#define _PATH_MAN       "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/share/man"
#define _PATH_MNTTAB    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/fstab"
#define _PATH_MOUNTED   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/mtab"
#define _PATH_NOLOGIN   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/nologin"
#define _PATH_PRESERVE  "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/lib"
#define _PATH_RWHODIR   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/spool/rwho"
#define _PATH_SENDMAIL  "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/sbin/sendmail"
#define _PATH_SHADOW    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow"
#define _PATH_SHELLS    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shells"
#define _PATH_UTMP      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/utmp"
#define _PATH_UTMP_DB   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/utmp.db"
#define _PATH_VI        "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin/vi"
#define _PATH_WTMP      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/wtmp"
#define _PATH_VARDB     "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/db/"
#define _PATH_VARRUN    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/"
#define _PATH_VARTMP    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/tmp/"


sysdeps/unix/sysv/linux/paths.h
#define _PATH_DEFPATH   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin"
#define _PATH_STDPATH "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin:/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/sbin:/sbin"
#define _PATH_BSHELL    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin/sh"
#define _PATH_CSHELL    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin/csh"
#define _PATH_DEVDB     "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/dev.db"
#define _PATH_GSHADOW   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/gshadow"
#define _PATH_LASTLOG   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/lastlog"
#define _PATH_MAILDIR   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/mail"
#define _PATH_MAN       "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/share/man"
#define _PATH_MNTTAB    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/fstab"
#define _PATH_MOUNTED   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/mtab"
#define _PATH_NOLOGIN   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/nologin"
#define _PATH_PRESERVE  "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/lib"
#define _PATH_RWHODIR   "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/spool/rwho"
#define _PATH_SENDMAIL  "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/sbin/sendmail"
#define _PATH_SHADOW    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow"
#define _PATH_SHELLS    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shells"
#define _PATH_UTMP      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/utmp"
#define _PATH_VI        "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/bin/vi"
#define _PATH_WTMP      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/wtmp"
#define _PATH_VARDB     "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/db/"
#define _PATH_VARRUN    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/"
#define _PATH_VARTMP    "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/tmp/"


sysdeps/posix/system.c: 
#define SHELL_PATH      "/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/bin/sh"  /* Path of the shell.  */


nss/db-Makefile -> noting got prefixified, but it already has a bunch of paths in the original file:
DATABASES = $(wildcard /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/passwd /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/ethers /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/protocols \
                       /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/rpc /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/services /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/gshadow \
                       /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/netgroup)
VAR_DB = /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/db
$(VAR_DB)/passwd.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/passwd
$(VAR_DB)/group.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group
$(VAR_DB)/ethers.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/ethers
$(VAR_DB)/protocols.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/protocols
$(VAR_DB)/rpc.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/rpc
$(VAR_DB)/services.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/services
$(VAR_DB)/shadow.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow
$(VAR_DB)/gshadow.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/gshadow
$(VAR_DB)/netgroup.db: /cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/netgroup

Removing the duplicates and some irrelevant ones (sh, vi, etc), we end up with:

/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/hosts.equiv
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/hosts
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/networks
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/nsswitch.conf
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/protocols
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/services
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/ethers
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/rpc
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/fstab
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/resolv.conf
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/ld.so.preload
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/group
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/netgroup
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/passwd
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shadow
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/gshadow
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/nologin
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/etc/shells

/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/db/
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/lib
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/lastlog
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/log/wtmp
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/mail
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/dev.db
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/mtab
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/utmp
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/run/utmp.db
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/spool/rwho
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/var/tmp/

# not sure about this one, it's not installed in our compat layer, so may make sense to make a symlink?
/cvmfs/software.eessi.io/versions/2026.06/compat/linux/x86_64/usr/sbin/sendmail

So I suppose we could make variant symlinks for all of these, and point them to the corresponding host path by default? For some of them (e.g. var/run/*) we can still choose if we want to have specific, fine-grained ones, or just symlink the entire var/run directory to the host. For var/run it probably makes sense to do that, as we're not running services in the compat layer, and we already used to do that for older versions (see a52c683).