Skip to content

fix(toolkit-lib): stricter, opt-in evaluation modes for EvaluateCloudFormationTemplate - #2074

Open
iankhou wants to merge 6 commits into
mainfrom
fix/toolkit-lib-evaluator-strict-mode
Open

iankhou wants to merge 6 commits into
mainfrom
fix/toolkit-lib-evaluator-strict-mode

Conversation

@iankhou

@iankhou iankhou commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First of several stacked PRs, which eventually add an unstable evaluate action to toolkit-lib (#2066).

Extends and improves functionality of our template evaluator.

Description of changes

New opt-in props on EvaluateCloudFormationTemplate. Hotswap doesn't set these props.

  • unavailableParameters: a Ref to one of these parameters (e.g. NoEcho parameters, whose deployed values are masked) fails instead of falling back to the parameter's Default, including where the evaluator uses it internally (Cloud Control identifiers, nested stack parameters).
  • lookups (new SharedLookups class): Cloud Control reads are cached per resource and limited to a number in flight across all evaluators that share it, and nested stack outputs are read from DescribeStacks of the deployed nested stack (by its physical id), instead of evaluated from its template.

Changes that also apply to hotswap:

  • Fn::Select evaluates its index, which can be a Ref.
  • Fn::Sub treats ${!Literal} as literal text.
  • Values are evaluated the way CloudFormation does; for example, an empty string (or 0, false) is a value and only a missing or null value is missing.
  • Unsupported functions throw CfnUnsupportedFunctionException.
  • Failed Cloud Control lookups keep the original error as cause.
  • Fn::GetAtt of a nested stack output when the nested template has no Outputs section is an evaluation failure instead of a TypeError (which failed the hotswap deployment).

Describe any new or updated permissions being added

None.

Description of how you validated changes

Deployed a CDK app containing several resources that test our resolver logic. See code below:

import * as cdk from 'aws-cdk-lib';
import { Fn } from 'aws-cdk-lib';
import * as cognito from 'aws-cdk-lib/aws-cognito';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as sns from 'aws-cdk-lib/aws-sns';
import * as sqs from 'aws-cdk-lib/aws-sqs';
import type { Construct } from 'constructs';

const app = new cdk.App();
const version: string = app.node.tryGetContext('version') ?? '1';
const code = (name: string) => lambda.Code.fromInline(
  `exports.handler = async () => ({ fn: '${name}', version: '${version}', env: process.env });`);

class Nested extends cdk.NestedStack {
  public readonly queue: sqs.Queue;

  constructor(scope: Construct, id: string) {
    super(scope, id);
    this.queue = new sqs.Queue(this, 'Queue');
    new lambda.Function(this, 'NestedFn', {
      runtime: lambda.Runtime.NODEJS_22_X,
      handler: 'index.handler',
      code: code('nested'),
      environment: { VERSION: version, QUEUE_ARN: this.queue.queueArn, QUEUE_NAME: this.queue.queueName },
    });
  }
}

const stack = new cdk.Stack(app, 'Hotswap2074');
const idx = new cdk.CfnParameter(stack, 'Idx', { type: 'Number', default: '1' });
const bucket = new s3.Bucket(stack, 'Bucket', { removalPolicy: cdk.RemovalPolicy.DESTROY });
const topic = new sns.Topic(stack, 'Topic');
const pool = new cognito.UserPool(stack, 'Pool', { removalPolicy: cdk.RemovalPolicy.DESTROY });
const client = pool.addClient('Client', { userPoolClientName: 'hotswap-2074-client' });
const nested = new Nested(stack, 'Nested');

new lambda.Function(stack, 'Fn', {
  runtime: lambda.Runtime.NODEJS_22_X,
  handler: 'index.handler',
  code: code('parent'),
  environment: {
    VERSION: version,
    BUCKET: bucket.bucketName, // Ref
    BUCKET_ARN: bucket.bucketArn, // formatted GetAtt
    TOPIC_NAME: topic.topicName, // GetAtt via Cloud Control
    CLIENT_NAME: (client.node.defaultChild as cognito.CfnUserPoolClient).attrName, // Cloud Control compound identifier (UserPoolId|ClientId)
    SUB: Fn.sub('${AWS::Region}/${B}/v' + version, { B: bucket.bucketName }),
    SELECT: Fn.select(idx.valueAsNumber, ['zero', 'one', 'two']), // Fn::Select with a parameter index
    JOIN: Fn.join(':', [topic.topicArn, pool.userPoolId]),
    NESTED_QUEUE_URL: nested.queue.queueUrl, // nested stack output
  },
});

Ran deploy --hotswap twice (each on a different version of the code that changed lambda code, the VERSION env var, and the SUB variable and found no regressions. These changes fixed a case for hotswap that before these changes, could not handle the SELECT variable:

{
 "Parameters": {
  "Idx": {
   "Type": "Number",
   "Default": "1"
  },
...TRUNCATED...
   
      "SELECT": {
       "Fn::Select": [
        {
         "Ref": "Idx"
        },
        [
         "zero",
         "one",
         "two"
        ]
       ]
      },
...TRUNCATED...

and would fail to evaluate the index, leading to a value of ['zero','one','two'][{ Ref: 'Idx' }] (which is undefined), so the variable gets dropped. This was a hidden bug and did not error out. It had the potential to break a deployed application.

Checklist

  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@iankhou
iankhou deployed to no-approval October 21, 2026 16:36 — with GitHub Actions Active
@codecov-commenter

codecov-commenter commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.05%. Comparing base (95683e3) to head (64ec3ab).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2074      +/-   ##
==========================================
- Coverage   92.08%   92.05%   -0.04%     
==========================================
  Files          83       83              
  Lines       13246    13246              
  Branches     1886     1884       -2     
==========================================
- Hits        12197    12193       -4     
- Misses       1011     1015       +4     
  Partials       38       38              
Flag Coverage Δ
suite.unit 92.05% <ø> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…FormationTemplate

Opt-in props, off for hotswap:
- strictValues: CloudFormation value semantics (empty strings are values, null is missing),
  Fn::Select out of range is an evaluation failure, Fn::Sub handles ${!Literal}
- unavailableParameters: a Ref to one fails instead of falling back to its Default
- lookups (SharedLookups): Cloud Control reads cached per resource and limited across evaluators,
  nested stack outputs read from DescribeStacks

Changes that also apply to hotswap (error handling only):
- unsupported functions throw CfnUnsupportedFunctionException (a CfnEvaluationException, same message)
- failed Cloud Control lookups keep the original error as the cause
- a Cloud Control identifier that can't be evaluated throws its own evaluation error; other errors
  while building it are still wrapped in a CfnEvaluationException
- Fn::GetAtt of a nested stack output when the nested template has no Outputs is an evaluation failure,
  not a TypeError
- parseIntrinsic is exported for reuse
Remove the strictValues option and make its semantics the default:
- an empty string (or 0, false) is a value; only a missing or null value is missing
- Fn::Select with an index out of range is an evaluation failure
- ${!Literal} in Fn::Sub is the literal text ${Literal}

For hotswap, each of these either replaces a wrong value with the correct one (an empty-string parameter
no longer falls back to its Default; Fn::Select out of range no longer becomes an empty string) or turns a
case that fell back to a full deployment into a hotswap with the correct value ("", 0 or false attributes,
${!Literal}). A null attribute is still missing.
…nd evaluate the Fn::Select index

- reading a nested stack's outputs through shared lookups requires its physical id: a nested stack that has
  not been created is an evaluation failure, instead of DescribeStacks without a stack name (which lists
  every stack and returned the first one's outputs)
- Fn::Select evaluates its index (Ref, Fn::FindInMap), accepts numeric strings, and requires an integer
- create the CloudFormation client once per evaluator; one nested-stack-outputs branch; isValue is a
  module-level helper
…ndexes the way CloudFormation does

- A Cloud Control identifier part set to '', 0 or false in the template is used, not replaced by the physical id
- Fn::Sub placeholders, parameters, Cloud Control attributes and deployed nested stack outputs only match their
  own keys, so names like `constructor` and `toString` are evaluation failures instead of functions
- A string Fn::Select index must be a plain non-negative integer ('0x1', '1e0', ' 1 ' are rejected)
- Fix the spacing and spelling of the failed Cloud Control lookup message
- Share the Cloud Control response parsing between the evaluator and SharedLookups, reuse the cached
  CloudFormation client for DescribeType, and compute the lookup scope once
@maintainer-for-aws

Copy link
Copy Markdown

Automated review

A maintainer will still review this — treat the notes below as a starting point.

📌 Commit 64ec3ab

This PR extends EvaluateCloudFormationTemplate (the class that powers hotswap and underpins an upcoming template-evaluation action) with opt-in strictness and a new SharedLookups cache, plus several semantic fixes that align the evaluator with CloudFormation behavior. The two new props — unavailableParameters (surface masked NoEcho parameter values as evaluation failures instead of silently falling back to template defaults) and lookups (a shared, concurrency-limited Cloud Control resource cache and DescribeStacks-based nested-stack output resolution) — default to off, so hotswap is unaffected by them. The remaining changes do reach hotswap: a shared isValue/Object.hasOwn value check that treats empty string, 0, and false as real values; integer/range validation for Fn::Select; Fn::Sub ${!} literal escaping; error cause preservation; and optional chaining so Fn::GetAtt on a nested stack with no Outputs returns undefined rather than crashing. Each change moves toward CloudFormation semantics and toward a safe full-deploy fallback, and each is covered by targeted new tests, including a guard that the nested-stack-not-yet-created path never issues a nameless DescribeStacks. No issues found on this change.

🔴 0 blocking · 🟡 0 recommended · ⚪ 0 optional


Generated automatically. React 👍 or 👎 to tell us whether this review helped, so we can improve these reviews.

@iankhou

iankhou commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

See the output template (account ID redacted)

Hotswap2074.template.json

This branch had an error being deployed

1 failed (outdated), 1 active, and 1 inactive deployments
run-tests — 64ec3ab3 Deployed Oct 28, 2026 by iankhou via integ_cli (cli-integ-tests, 24.19, 11) #7507
no-approval — 64ec3ab3 Deployed Oct 28, 2026 by iankhou via prepare #7507
automation — a8cf53ca Deployed Oct 21, 2026 by iankhou via Triage Pull Requests #2232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants