(2025.09 - 2026.03) VDC Controller request timeouts under load due to token caching bypass

[junbo75]

Bug description

The VDC Controller's TokenService defaults force_renewal=True for all token acquisition methods, bypassing the in-memory token cache and requesting a new Cognito access token on every API call. Under peak load with many concurrent sessions, the repeated /oauth2/token calls to Cognito add significant latency to each request. This causes the VDC Controller's ALB target response times to climb until requests time out, resulting in "Request timed-out" errors in the web UI when users attempt to list or connect to their virtual desktops.

Mitigation

Tip: We recommend running the following commands in AWS CloudShell in the same account and region where RES is deployed. CloudShell comes with pre-installed dependencies and helps eliminate other local environment conditions that may affect the patching process.

For 2025.09:

1. Set up a Python 3.9 virtual environment and install all dependencies:

sudo yum install -y python3.9 patch
python3.9 -m venv /tmp/patch-env 
source /tmp/patch-env/bin/activate
pip install boto3

2. Download res_tool_2025.09.py and fix_force_renewal_vdc_2025.09.patch by replacing <output-directory> with the directory to download the patch script, <environment-name> with the name of your RES environment in the command below:
   1. The patch applies to RES 2025.09
   2. The patch script requires AWS CLI v2, Python 3.9, and Boto3.
   3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket created by RES.

OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=<environment-name>
RES_VERSION=2025.09

mkdir -p ${OUTPUT_DIRECTORY}
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/res_tool_${RES_VERSION}.py --output ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py
curl https://research-engineering-studio-us-east-1.s3.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/fix_force_renewal_vdc-${RES_VERSION}.patch --output ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch

3. Navigate to the directory where the patch script and patch file are downloaded. Run the following patch command:

python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module virtual-desktop-controller --patch ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch

4. Restart the VDC Controller instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.

INSTANCE_ID=$(aws ec2 describe-instances \
           --filters \
           Name=tag:Name,Values=${ENVIRONMENT_NAME}-vdc-controller \
           Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME} \
           Name=instance-state-name,Values=running \
           --query "Reservations[0].Instances[0].InstanceId" \
           --output text)

aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}

5. Verify the new VDC Controller instance status by checking the activity of the auto scaling group starting with the name vdc-controller-asg. Wait until the new instance is launched successfully.

For 2025.12, 2025.12.01, and 2026.03:

1. Set up a Python 3.12 virtual environment and install all dependencies:

sudo yum install -y python3.12
python3.12 -m venv /tmp/patch-env 
source /tmp/patch-env/bin/activate
pip install boto3

2. Create an S3 bucket with ACLs disabled. If you already have an S3 bucket with ACLs disabled, you can skip this step. Otherwise, create one using either the console or the CLI:

   1. Option A (AWS Console):
      1. Follow the Creating a bucket guide in the Amazon S3 User Guide. When configuring the bucket:
      2. Enter a bucket name and select the region where RES is deployed
      3. Under Object Ownership, keep the default setting ACLs disabled (Bucket owner enforced)
      4. Configure the remaining settings as needed (block public access, versioning, etc.)
      5. Click Create bucket
   2. Option B (CLI):
      1. Replace <BUCKET-NAME> and <res-environment-aws-region> with your values and run the command:

aws s3api create-bucket \
    --bucket <BUCKET-NAME> \
    --region <res-environment-aws-region> \
    --create-bucket-configuration LocationConstraint=<res-environment-aws-region> \
    --object-ownership BucketOwnerEnforced

Note: If your region is us-east-1, omit the --create-bucket-configuration parameter.

3. Download res_tool_<version>.py and idea-virtual-desktop-controller-<version>-<hash>.tar.gz by replacing <environment-name> with the name of your RES environment, <res-environment-aws-region> with the RES deployment region, <bucket-name> with the name of an ACLs disabled S3 bucket under the account/region where RES is deployed, <RES_VERSION> with 2025.12, 2025.12.01, or 2026.03 in the command below, and <partition> with Classic or GovCloud:
   1. The patch applies to 2025.12, 2025.12.01, 2026.03
   2. The patch script requires AWS CLI v2, Python 3.12, and Boto3.
   3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket provided through <bucket-name> and read/write permissions to the cluster settings DynamoDB table.

OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=<environment-name>
RES_VERSION=<RES_VERSION>
BUCKET_NAME=<bucket-name>
PARTITION=<partition>

mkdir -p ${OUTPUT_DIRECTORY}
HASH=$(case $RES_VERSION in "2026.03") echo 20eaec2f;; "2025.12.01") echo ca77c169;; "2025.12") echo ad144beb;; esac)
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/${RES_VERSION}/patch_scripts/res_tool_${RES_VERSION}.py --output ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/${RES_VERSION}/patch_scripts/patches/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --output ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz

4. Run the following patch command:

python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --module virtual-desktop-controller --zip-file ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --s3-bucket ${BUCKET_NAME} --partition ${PARTITION}

5. Restart the VDC Controller instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.

INSTANCE_ID=$(aws ec2 describe-instances \
           --filters \
           Name=tag:Name,Values=${ENVIRONMENT_NAME}-vdc-controller \
           Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME} \
           Name=instance-state-name,Values=running \
           --query "Reservations[0].Instances[0].InstanceId" \
           --output text)

aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}

6. Verify the new VDC Controller instance status by checking the activity of the auto scaling group starting with the name vdc-controller-asg. Wait until the new instance is launched successfully.

Rollback

For 2025.09:

1. Re-run the same patch command to rollback and enter "y" for the prompt. The script detects that the patch has already been applied and reverts the environment to its pre-patch state.

OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=<environment-name>
RES_VERSION=2025.09

python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --res-version ${RES_VERSION} --module virtual-desktop-controller --patch ${OUTPUT_DIRECTORY}/fix_force_renewal_vdc-${RES_VERSION}.patch

2. Terminate the VDC Controller instance to pick up the restored package.

For 2025.12, 2025.12.01, and 2026.03:

1. Re-run the same patch command with argument --rollback. The script restores the original download URL for VDC controller app.

OUTPUT_DIRECTORY=/tmp/res-patch
ENVIRONMENT_NAME=<environment-name>
RES_VERSION=<RES_VERSION>
BUCKET_NAME=<bucket-name>
PARTITION=<partition>

python3 ${OUTPUT_DIRECTORY}/res_tool_${RES_VERSION}.py --environment-name ${ENVIRONMENT_NAME} --module virtual-desktop-controller --zip-file ${OUTPUT_DIRECTORY}/idea-virtual-desktop-controller-${RES_VERSION}-${HASH}.tar.gz --s3-bucket ${BUCKET_NAME} --partition ${PARTITION} --rollback

2. Terminate the VDC Controller instance to pick up the restored download URL for VDC controller app.

[anniekaycruk]

Could this bug create increased Cognito costs? We saw a big increase in costs when we moved to from RES version 2025.03 to 2025.12 ?