[BUG] Changing a GPU desktop's instance type across GPU families leaves the session in ERROR

[dna-oliv]

Changing an existing GPU virtual desktop's instance type to a different GPU family
via the RES UI, for example g4dn/T4 to g6/L4, leaves the session in ERROR after
RESUMING/INITIALIZING. The instance starts and the GPU driver and DCV server come
up, but the DCV console session's X server never initializes because
/etc/X11/xorg.conf is still pinned to the create-time GPU's PCI BusID. The host
bootstrap does re-run on every boot, but the X-config step self-skips when a driver
is already present, so the stale BusID is never refreshed. Fresh desktops created
directly on the target type reach READY, and CPU-only type changes reach READY, so
the issue is specific to GPU-family changes on existing desktops. Reproduced both
g4dn to g6 and g6 to g4dn.

Environment

• RES 2026.06
• Base OS: Ubuntu 24.04 and Amazon Linux 2023. The Linux GPU path is OS-agnostic; see Root cause.
• GPU instances: g4dn (T4), g6 (L4); NVIDIA GRID driver 595.71.05 from s3://ec2-linux-nvidia-drivers/latest/
• Region: us-east-1

Steps to reproduce

1. Create a RES Linux GPU desktop on g4dn.2xlarge. It reaches READY.
2. Stop the desktop.
3. Change its instance type to a different GPU family, e.g. g6.2xlarge.
4. Resume the desktop.

Expected

The desktop resumes on the new GPU type and reaches READY, as a fresh create on that
type does.

Actual

The session goes RESUMING to INITIALIZING to ERROR. On the host the instance is
running, nvidia-smi reports the new GPU and driver, and dcvserver is active, but
no X server is running and the DCV console session never initializes.

Root cause

RES bakes the GPU family into the host bootstrap's install.sh -g <family> argument
at create time, computed from the create-time instance type. The Linux userdata
cloud-config runs on every boot via cloud_final_modules: [scripts-user, always],
and an instance-type change (ModifyInstanceAttribute) followed by resume
(StartInstances) does not rebuild the userdata, so on resume the instance re-runs
the create-time bootstrap.

For a GPU-family change the re-run does not refresh the GPU X configuration.
install_nvidia_grid_drivers in gpu_drivers.sh short-circuits on which nvidia-smi — the pre-existing GRID driver satisfies the check — and returns before
running nvidia-xconfig --preserve-busid, the step that writes
/etc/X11/xorg.conf pinned to the GPU's PCI BusID. So the xorg.conf generated for
the original GPU is never regenerated for the new GPU: X cannot start on the new
GPU's BusID, the DCV console session never initializes, and RES marks the session
ERROR.

A fresh create reaches READY because no driver is present, so the guard passes and
nvidia-xconfig runs for the correct GPU. A CPU-only change reaches READY because
no GPU/X stack is involved.

Code (RES 2026.06):

• GPU family baked into install.sh -g at create time:
  virtual_desktop_controller_utils.py L93-L97, L152
• Userdata cloud-config runs on every boot:
  bootstrap_userdata_linux.sh.jinja2 L11-L12
• Type change performs only ModifyInstanceAttribute; resume performs only StartInstances; userdata is not rebuilt:
  virtual_desktop_controller_utils.py L727-L742,
  virtual_desktop_server_utils.py L100-L113
• X-config step self-skips on which nvidia-smi, so nvidia-xconfig --preserve-busid never re-runs after a GPU change; skip at L34-L36, nvidia-xconfig at L63:
  gpu_drivers.sh L33-L64

gpu_drivers.sh branches by instance family, not OS, and is shared by Ubuntu,
Amazon Linux, and RHEL through the DCV bootstrap, so all Linux distros are expected
to be affected.

Evidence

Same AMI, driver, and kernel across all cases.

• g4dn -> g6: ERROR. Instance running, nvidia-smi shows the L4, dcvserver active, but no "First frame captured" and dcvgltest returns "Unable to open display". Started immediately (no InsufficientInstanceCapacity) and DCV-ready ~2.4 min after start, still ERROR — rules out capacity and readiness-timeout.
• g6 -> g4dn: ERROR. xorg.conf = BusID "PCI:49:0:0" / BoardName "NVIDIA L4"; live GPU Tesla T4 at 00000000:00:1E.0 (bus 0x1E) — BusID mismatch. pgrep Xorg/Xdcv empty, no /var/log/Xorg.0.log; X never started.
• Fresh create on g4dn -> READY. Fresh create on g6 -> READY. Isolates the failure to the type-change path.
• CPU-only c8g -> c9g (Amazon Linux 2023) -> READY. Scopes the bug to GPU desktops.

Suggested fix

Refresh the GPU X configuration when the GPU changes. Options:

• On boot, detect a GPU change (current PCI BusID or GPU model differs from what /etc/X11/xorg.conf was generated for) and re-run nvidia-xconfig --preserve-busid --enable-all-gpus even when the driver is already installed. Do not gate the X-config step behind the which nvidia-smi driver-presence check.
• Or force host GPU/DCV/X re-provisioning on the next resume after a GPU-family change.
• Or block cross-GPU-family instance-type changes on an existing desktop and direct the user to recreate on the target type.

Workaround

Recover an already-ERROR desktop in place, validated. On the host as root:

rm -f /etc/X11/xorg.conf
nvidia-xconfig --enable-all-gpus
# confirm the BusID now matches `nvidia-smi --query-gpu=pci.bus_id`

Then use the RES Reboot action, the only action available from ERROR. It sets the
session to PROVISIONING and re-creates the DCV session. The corrected xorg.conf
persists on the EBS root volume and the which nvidia-smi skip guard leaves it
untouched, so X starts on the correct GPU and the session reaches READY. Verified:
the desktop returned to READY after the reboot and remained READY across a
subsequent stop/start.

Alternatively, recreate the desktop on the target instance type. A fresh create runs
the full create-time bootstrap and configures the GPU display stack for the correct
GPU. User data on shared home storage is preserved.

[junbo75]

Thanks for reporting the issue. We will look into it and get back to you with our findings.

[junbo75]

We will track this issue internally and merge the fix into our future release. Thanks!