Skip to content

feat: fallback to /proc/1/environ for missing environment variables - #235

Open
shellyco-code wants to merge 1 commit into
Project-HAMi:mainfrom
shellyco-code:fix-gpu-isolation-2125
Open

feat: fallback to /proc/1/environ for missing environment variables#235
shellyco-code wants to merge 1 commit into
Project-HAMi:mainfrom
shellyco-code:fix-gpu-isolation-2125

Conversation

@shellyco-code

@shellyco-code shellyco-code commented Jul 27, 2026

Copy link
Copy Markdown

Description

This PR fixes the GPU memory isolation bypass vulnerability that occurs when child processes are spawned (e.g., via env -i) or when a user establishes a new SSH session into a HAMi-managed pod.

The Problem:
Currently, HAMi enforces GPU isolation by injecting configuration (like CGROUP_MEMORY_MAX, CUDA_VISIBLE_DEVICES, etc.) via environment variables into the pod's main process. When an SSH session is initiated or environment variables are purged, libvgpu.so fails to read these configurations through getenv(), effectively bypassing all limits.

The Solution:
This PR introduces a thread-safe vgpu_getenv helper function that acts as a drop-in replacement for getenv().
If standard getenv() returns NULL, vgpu_getenv falls back to parsing /proc/1/environ (the environment of the container's initialization process). Because Kubernetes always retains the webhook-injected environment variables in PID 1, libvgpu.so can securely recover its configuration limits regardless of the user's local shell environment.

Changes Made

  • Added vgpu_getenv() implementation in src/utils.c.
  • Replaced all usages of getenv() across libvgpu.c, utils.c, log_utils.c, multiprocess_memory_limit.c, and shrreg_tool.c.

Related Issues

Fixes Project-HAMi/HAMi#2125

Testing

  • Verified code compilation without errors.
  • Verified that deploying a pod and running env -i /bin/bash followed by a GPU workload respects the original memory limits.
  • Verified that an SSH session into a pod enforces the GPU core utilization and memory boundaries.

Checklist

  • I have read the contributing guidelines.
  • My code follows the project's style and conventions.
  • I have tested this solution locally.

Summary by CodeRabbit

  • New Features

    • Added support for reading configuration values from the primary process environment when they are unavailable in the current environment.
    • Applied this behavior across CUDA visibility, logging, memory limits, resource policies, task priorities, shared-region settings, and library selection.
  • Bug Fixes

    • Improved configuration handling in multiprocess and containerized environments by providing consistent environment-variable access.

@hami-robot

hami-robot Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shellyco-code
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot

hami-robot Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Welcome @shellyco-code! It looks like this is your first PR to Project-HAMi/HAMi-core 🎉

@hami-robot hami-robot Bot added the size/M label Jul 27, 2026
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@shellyco-code, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 53 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d6683f5f-e65b-45de-bd42-ee30a1eb655c

📥 Commits

Reviewing files that changed from the base of the PR and between 3324513 and a32d70a.

📒 Files selected for processing (6)
  • src/include/utils.h
  • src/libvgpu.c
  • src/log_utils.c
  • src/multiprocess/multiprocess_memory_limit.c
  • src/multiprocess/shrreg_tool.c
  • src/utils.c
📝 Walkthrough

Walkthrough

Adds vgpu_getenv, which checks the process environment and falls back to /proc/1/environ. CUDA, logging, and multiprocess configuration lookups now use this helper.

Changes

Environment Lookup Fallback

Layer / File(s) Summary
Lookup helper and public contract
src/include/utils.h, src/utils.c
Declares and implements vgpu_getenv, using getenv first and /proc/1/environ as a fallback.
CUDA and logging integration
src/utils.c, src/libvgpu.c, src/log_utils.c
Routes CUDA visibility, redirect-library, and log-level environment reads through vgpu_getenv.
Multiprocess configuration integration
src/multiprocess/*
Routes memory, OOM, utilization, shared-region, task-priority, and kernel-interval reads through vgpu_getenv.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RuntimeCaller
  participant vgpu_getenv
  participant ProcessEnvironment
  participant ProcEnviron
  RuntimeCaller->>vgpu_getenv: request configuration variable
  vgpu_getenv->>ProcessEnvironment: check getenv
  alt variable is unset
    vgpu_getenv->>ProcEnviron: read /proc/1/environ
    ProcEnviron-->>vgpu_getenv: matching value or NULL
  else variable is set
    ProcessEnvironment-->>vgpu_getenv: value
  end
  vgpu_getenv-->>RuntimeCaller: return configuration value
Loading

Suggested labels: enhancement

Suggested reviewers: mesutoezdil

Poem

I’m a rabbit with settings tucked safe in the hay,
vgpu_getenv finds them by night or by day.
CUDA hops forward, logs softly glow,
Multiprocess paths know where to go.
One little helper, a smoother flow!

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning Issue [#2125] is only partially met: the fallback is added, but there are no shown regression tests or documentation for child/SSH scenarios. Add regression tests and user-facing docs proving memory limits and CUDA visibility persist for child, SSH/login-shell, and cleared-env cases.
Docstring Coverage ⚠️ Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding a fallback to /proc/1/environ for missing environment variables.
Out of Scope Changes check ✅ Passed The changes stay within scope by adding vgpu_getenv and swapping related getenv call sites to use it.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai
coderabbitai Bot requested a review from mesutoezdil July 27, 2026 09:53
@coderabbitai coderabbitai Bot added the enhancement New feature or request label Jul 27, 2026
@shellyco-code
shellyco-code force-pushed the fix-gpu-isolation-2125 branch from 3324513 to d745c51 Compare July 27, 2026 09:55

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
src/include/utils.h (1)

8-8: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Document the fallback and pointer-lifetime contract.

This public API should state that getenv() takes precedence, /proc/1/environ is used as a fallback, and fallback results are thread-local and overwritten by later lookups. Document the trust boundary before callers treat it like a normal environment pointer.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/include/utils.h` at line 8, Document the public vgpu_getenv declaration
with its lookup and lifetime contract: use getenv() first, fall back to
/proc/1/environ, and state that fallback results are thread-local and
overwritten by subsequent lookups. Clarify that callers must treat the returned
pointer according to this trust boundary.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/multiprocess/multiprocess_memory_limit.c`:
- Around line 1143-1144: Update the CUDA_TASK_PRIORITY_ENV handling to call
vgpu_getenv once, cache the returned value, and parse it with strtol. Validate
errno, endptr, and the supported priority range before assigning the result to
region->priority; leave the existing priority unchanged for invalid values or
when the variable is absent.

In `@src/utils.c`:
- Around line 29-31: Update the environment-entry parsing around the
thread-local result buffer and single fread so complete values are read without
silent truncation or omission; dynamically handle entries exceeding the fixed
buffers or explicitly reject them, ensuring CUDA_REDIRECT and
MULTIPROCESS_SHARED_REGION_CACHE_ENV are never returned as altered paths. Apply
the same correction to the related logic at the additional occurrence.
- Around line 26-27: Update vgpu_getenv() to read the current process
environment from /proc/self/environ instead of /proc/1/environ. Before copying
any fallback variable into the 256-byte result buffer, detect oversized values
and handle them safely without silent truncation.
- Line 224: Update the logging around the CUDA_VISIBLE_DEVICES lookup in the
relevant utility flow to avoid passing a NULL value to the "%s" formatter:
retain the parsed environment value and log it with a null-safe placeholder when
unset, while preserving the existing behavior when the variable is configured.

---

Nitpick comments:
In `@src/include/utils.h`:
- Line 8: Document the public vgpu_getenv declaration with its lookup and
lifetime contract: use getenv() first, fall back to /proc/1/environ, and state
that fallback results are thread-local and overwritten by subsequent lookups.
Clarify that callers must treat the returned pointer according to this trust
boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 532e8e3e-4c13-4951-b131-27edbeb70a38

📥 Commits

Reviewing files that changed from the base of the PR and between 52f33fc and 3324513.

📒 Files selected for processing (6)
  • src/include/utils.h
  • src/libvgpu.c
  • src/log_utils.c
  • src/multiprocess/multiprocess_memory_limit.c
  • src/multiprocess/shrreg_tool.c
  • src/utils.c

Comment thread src/multiprocess/multiprocess_memory_limit.c Outdated
Comment thread src/utils.c
Comment thread src/utils.c
Comment thread src/utils.c
This resolves an issue where child processes or SSH-started processes lose the inherited environment variables used to pass GPU isolation limits (such as CGROUP_MEMORY_MAX).
By falling back to parsing /proc/1/environ when getenv() fails, libvgpu.so correctly reads the configuration initially passed into the pod by the Mutating Webhook, preserving memory isolation across all processes.

Signed-off-by: shellyco-code <shellychahar57@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[LFX Mentorship] Strengthen GPU memory isolation for child and SSH-started processes

1 participant