Skip to content

Fix: mulder state failures on CI (jammy PPA, home path, root setup) - #245

Closed
digitalisx wants to merge 4 commits into
teamdfir:masterfrom
digitalisx:fix-mulder-ci
Closed

digitalisx wants to merge 4 commits into
teamdfir:masterfrom
digitalisx:fix-mulder-ci

Conversation

@digitalisx

@digitalisx digitalisx commented Oct 20, 2026 •

Copy link
Copy Markdown

Summary

Fixes the sift.python3-packages.mulder CI failure from #241 (e.g. run 39260879739). Fixing the first problem exposed three more, so this PR fixes all four:

  1. dotnet-sdk-9.0 not found on 22.04. On jammy, ubuntu-universe / ubuntu-multiverse delete /etc/apt/sources.list. When that file is missing, jammy's add-apt-repository still prints Adding deb entry to .../dotnet-ubuntu-backports-jammy.list, but the .list file it leaves is empty (0 bytes) and sources.list gets only the header comment. So the PPA is never really added. A 2x2 probe (add-apt-repository vs salt pkgrepo.managed, with and without sources.list) reproduces it with plain add-apt-repository too, so Salt is not the cause. The fix is to keep an empty (comment-only) /etc/apt/sources.list instead of removing it.
  2. Hardcoded /home/<user>/.local/bin cwd. This path does not exist for root (CI) or for users whose home is somewhere else. The fix resolves the home directory the same way sift/config/user/*.sls already does.
  3. mulder: not found. cwd does not put ~/.local/bin on PATH, so the fix calls the pipx-installed binary by its full path.
  4. mulder setup refuses to run as root (by design). The fix skips the setup step when sift_user is root; pipx ensurepath still runs.

Testing

  • tests.yml on the fork: all mulder, ubuntu-universe and ubuntu-multiverse jobs pass on 22.04/24.04 × Salt 3006/3007.
  • Applied sift.python3-packages.mulder with sift_user = root and sansforensics on 22.04/24.04 (Salt 3007): 0 failures. The non-root runs include a successful mulder setup --yes.

Notes

Coauthored: @ParkJeongKyun @rlo-lo @Ryu-jae-kwong @Refrain4

This work was supported by the Best of the Best (BoB) Program,
a cybersecurity talent development program operated by the
Korea Internet & Security Agency (KISA).

@digitalisx digitalisx changed the title Fix mulder state failures on CI (jammy PPA, home path, root setup) Fix: mulder state failures on CI (jammy PPA, home path, root setup) Oct 20, 2026
@digitalisx

Copy link
Copy Markdown
Author

Hello @calebevans,
I hope this PR will help make testing and deploying mulder smoother.
Please feel free to review it at your convenience.

digitalisx and others added 4 commits October 21, 2026 03:36
add-apt-repository on Ubuntu 22.04 silently writes an empty PPA .list
file when /etc/apt/sources.list is missing, so PPA packages such as
dotnet-sdk-9.0 (ppa:dotnet/backports) cannot be located afterwards.

Co-authored-by: ParkJeongKyun <44119972+ParkJeongKyun@users.noreply.github.com>
Co-authored-by: KIM EUN WOO <148983524+rlo-lo@users.noreply.github.com>
Co-authored-by: Ryu_harudia <145317670+Ryu-jae-kwong@users.noreply.github.com>
Co-authored-by: Donggyun Jo <90193658+Refrain4@users.noreply.github.com>
The setup and ensurepath steps hardcoded /home/<user>/.local/bin, which
does not exist for root (/root) or users with non-default homes.

Co-authored-by: ParkJeongKyun <44119972+ParkJeongKyun@users.noreply.github.com>
Co-authored-by: KIM EUN WOO <148983524+rlo-lo@users.noreply.github.com>
Co-authored-by: Ryu_harudia <145317670+Ryu-jae-kwong@users.noreply.github.com>
Co-authored-by: Donggyun Jo <90193658+Refrain4@users.noreply.github.com>
pipx installs into ~/.local/bin, which is not on PATH for the salt
cmd.run environment, so 'mulder setup' failed with 'not found'.

Co-authored-by: ParkJeongKyun <44119972+ParkJeongKyun@users.noreply.github.com>
Co-authored-by: KIM EUN WOO <148983524+rlo-lo@users.noreply.github.com>
Co-authored-by: Ryu_harudia <145317670+Ryu-jae-kwong@users.noreply.github.com>
Co-authored-by: Donggyun Jo <90193658+Refrain4@users.noreply.github.com>
mulder refuses to run 'setup' as root by design, so applying the state
with sift_user=root (as CI does) always failed.

Co-authored-by: ParkJeongKyun <44119972+ParkJeongKyun@users.noreply.github.com>
Co-authored-by: KIM EUN WOO <148983524+rlo-lo@users.noreply.github.com>
Co-authored-by: Ryu_harudia <145317670+Ryu-jae-kwong@users.noreply.github.com>
Co-authored-by: Donggyun Jo <90193658+Refrain4@users.noreply.github.com>
@calebevans

Copy link
Copy Markdown

@digitalisx thank you! I don't have any permissions in this repo, but I will follow it and reach out to someone if needed.

@ekristen

Copy link
Copy Markdown
Contributor

I prematurely merged in a state we only had done limited testing on. Thank you for submitting this. It is much appreciated and love to see others contributing. We'll get this reviewed as soon as we can. @digitalsleuth can you take a quick look? I have tests running now.

@digitalsleuth

Copy link
Copy Markdown
Collaborator

Will do, just getting in the door and will start shortly.

@digitalsleuth

Copy link
Copy Markdown
Collaborator

Hi @calebevans , did you happen to come across an error which spawned this conversation? I want to make sure I'm reviewing and addressing all of the points of interest before providing a potential solution, which may or may not address your issue.

@digitalisx

Copy link
Copy Markdown
Author

Hello @digitalsleuth, This issue and PR raised by me
While reviewing the SIFT deployment logs, I found that the sift.python3-packages.mulder state was not being tested correctly in CI.

https://github.com/teamdfir/sift-saltstack/actions/runs/39260879739/job/118087938739

After investigating the root cause, I've submitted a fix (PR #245) that also prevents the same class of failure from recurring.

@digitalsleuth

Copy link
Copy Markdown
Collaborator

Hi @digitalisx , Thanks for the heads up. The issues you're experiencing only occur during the CI tests, however don't occur in the actual SIFT environment. The root cause for these issues is that, especially when testing only on the mulder state, the repos which are required for the state to succeed (specifically ubuntu-universe and dotnet\backports) do not get refreshed.

As the sift.repos state does get instantiated during the run of SIFT, this issue doesn't appear except in testing. To prevent this in the future (both with individual states and potential race conditions), I'll be adding some updates to the mulder state, and to other states which may require specific repos be available prior to package installation.

When properly refreshed, the ubuntu-universe state (and the ubuntu-multiverse state) add the necessary ubuntu.sources file before deleting the sources.list file. So the deletion is intended and, upon refresh, Jammy (as well as Noble and Resolute) will create the sources.list placeholder as necessary.

The home and root requirements are not required in the active state, nor the test state, as the state (in practice, during installation) will be run as whichever user initiated the command. When done with cast or with SaltStack via a manual install, this value is provided by the pillar value for 'sift_user'. During testing, the state should be tested with the appropriate sift_user pillar value, and this will not have an issue.

The problem with mulder not being on PATH is two-fold: the first is that the pillar must be accurately set (during testing only, as it is accurately set during cast installation), and the command pipx ensurepath must be run from the appropriate directory. In testing today, it appears that even with the cwd option set in SaltStack, this does not stick, so I'll be introducing a fix which does enforce this by way of adjusting the cmd.run -> name: value.

All in all, this will fix the issues appearing during CI tests, and any issues during installation. I'm working on testing the fixes across both arm64 and amd64 architectures, as well as from 22-26.04 versions of Ubuntu.

@digitalisx

Copy link
Copy Markdown
Author

Thank you for reply @digitalsleuth,
If there’s anything I can help you with in PR right now, please let me know.
I will gladly work on it. Or do you plan to do it yourself?

@digitalsleuth

Copy link
Copy Markdown
Collaborator

There's nothing to do right now, as I'm in the middle of testing some changes, but I'll keep you posted once the tests are finished. Thanks!

@digitalsleuth

Copy link
Copy Markdown
Collaborator

Just as a heads up, the Ubuntu PPA launchpadcontent.net seems to be experiencing some network issues (returning 503 errors), which are preventing my tests from being successful. I'm still monitoring the situation, but I believe I have a viable fix for the issues identified here. More to follow.

@digitalsleuth

Copy link
Copy Markdown
Collaborator

So the issues appear to be stemming from overload, due to the recently identified / disclosed PHP vulnerabilities and people updating their packages. I'll try my testing again later when it should be less busy.

@digitalsleuth

Copy link
Copy Markdown
Collaborator

Hi @digitalisx , while investigating the issue you reported, I also identified a few other areas which needed addressing and, while your changes were effective, the changes which needed to be addressed were updated during my testing. Because of the additional changes, I've created a new PR #247 which encompasses the changes you were looking to make, though in a different way.

@digitalisx

digitalisx commented Oct 23, 2026 •

Copy link
Copy Markdown
Author

Thank you for effort @digitalsleuth,
I checked your PR, and I'm satisfied another method for best way
You can close this PR with merge #247

@digitalsleuth

Copy link
Copy Markdown
Collaborator

#247 has been merged, closing this PR. Thanks again for bringing this to our attention!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants