feat: support newer provider plugin-protocol features via targetVersions (RFC-04)

[so0k]

Implements RFC-04 — provider feature availability: support the newer provider plugin-protocol capability families with targetVersions-aware codegen and synth-time validation. Proposal, dataset, sweep tooling and interactive report live in open-constructs/cdktn-planning → RFCS/04-provider-feature-availability/ (only the merged matrix is vendored here, matching the function-availability split).

Builds directly on the #269 foundation (targetVersions + ValidateFeatureTargetSupport) and the #268 usage-registry pattern.

Guiding decision

Generate the full surface the schema offers; narrow per project at synth time via targetVersions. Generation-time filtering would fork the generated API by project configuration — impossible for prebuilt providers and hostile to caching. Constraints baked into the validations are the per-product >= ranges from the sweep dataset.

Review guide (one commit per RFC rollout item)

1. chore: vendor provider-feature availability matrix — dataset digest + README only, no behavior change.
2. feat(lib): TerraformEphemeralResource — new public base class synthesizing to the top-level ephemeral key (JSON + HCL renderer), refs as ephemeral.<type>.<id>.<attr>, no provisioners/connection/import/move. Constructor registers the target-version validation unconditionally (new API surface — only fires on use, no feature flag per RFC). Internal providerFeatureConstraints map sourced from the vendored matrix.
3. feat(provider-generator): acquire newer provider-protocol schema sections (Phase 0, a bugfix on its own) — commons types for functions / ephemeral_resource_schemas / resource_identity_schemas / write_only; sanitizer walks ephemeral schemas; fetched schemas stamped with the fetching CLI; cache key now includes CLI product+minor (a schema fetched once with an old CLI no longer poisons every later generation); fetch-time warning when the fetching binary structurally cannot emit sections the targets admit.
4. feat(provider-generator): generate ephemeral resource bindings — third schema family mirroring the data_ pipeline (EphemeralRandomPassword in ephemeral-random-password/), config extends TerraformEphemeralMetaArguments, no generateConfigForImport. Zero churn in existing snapshots (ephemeral models append after resources/data sources; class-name dedup is order-dependent).
5. feat(provider-generator): generate provider-defined function bindings — TimeProviderFunctions.rfc3339Parse(ts) → ${provider::time::rfc3339_parse(...)}; namespace defaults to the registry short name with a providerLocalName override (local names change the namespace, aliases don't). Runtime flows through one public jsii chokepoint (TerraformProviderFunction.invoke) feeding a usage registry; TerraformStack validates usage against terraform >=1.8.0 / opentofu >=1.7.0 — note the deliberate asymmetry: OpenTofu language support (1.7.0) predates its schema emission (1.8.0), so generation and validation use different boundaries.
6. feat(provider-generator): deprecate write-only attribute getters, validate usage — providers never persist write-only values (every read is null by protocol contract), so the state-backed getter is a trap: emitted @deprecated now, removal rides the next prebuilt major (JSII-breaking otherwise). Setting one (setter or constructor config) registers usage via a new protected TerraformResource.registerProviderFeatureUsage hook — generated code reaches the validation machinery only by extending base classes.
7. feat(cli): thread targetVersions into cdktn get — GetOptions.targetVersions → ConstructsMaker → readSchema, driving the Phase 0 fetch-time warning; constraints.json gains diagnostic stamps (targetVersions, fetching cli) without affecting the filterAlreadyGenerated staleness logic (targets don't change codegen output, so they must not force regeneration).

Test coverage

• Generator: snapshot tests against real terraform providers schema -json fragments from the sweep (random ephemeral, time functions incl. object returns, vault *_wo), plus a synthetic fixture for variadic/reserved-name mapping branches. Zero churn in pre-existing snapshots.
• Core: validation matrices mirroring validations.test.ts (admit/exclude per product, hint text, the OpenTofu 1.7.0 asymmetry, registry resets).
• Schema: emission-gap logic matrix, cache-key suffix behavior, sanitizer walk over ephemeral schemas; CLI stamps in network snapshots are stubbed (environment-dependent).

Known/deferred

• matchers.test.ts › toPlanSuccessfully fails in this environment before and after these changes (downloads the real docker provider; verified pre-existing via stash on the unmodified tree).
• Deferred per RFC Phase 4: resource identity consumption (schema plumbing already carried), list resources / actions / state stores (Terraform-only today; re-evaluate on Support 'query' CLI and provider 'list' resources as in Terraform 1.14 opentofu/opentofu#3787).
• Follow-ups tracked as issues: Nested write-only attributes don't register targetVersions feature usage #308 (nested write-only registration), In-repo consistency check for the provider-feature availability matrix and its hand-maintained maps #309 (matrix drift check / in-repo tooling), hcl2cdk: convert top-level ephemeral {} blocks #310 (hcl2cdk ephemeral conversion), Edge-provider schema: cross-language compile coverage for ephemeral resources, provider functions, write-only attributes #311 (edge-provider schema cross-language coverage), Integration test: ephemeral resources end-to-end, gated on Terraform >= 1.10 #312 (ephemeral integration test gated on TF >= 1.10), fix(lib): HCL synth corrupts or drops core blocks (terraform{} extras, top-level overrides, lifecycle conditions) #313 (pre-existing HCL renderer defects), lib: add addProviderMeta() helper for terraform { provider_meta } blocks (value stays any) #304 (first-class provider_meta; feat(lib): first-class provider_meta support (aws user_agent module attribution) #314 closed as its duplicate), Remove write-only attribute getters at the next prebuilt-provider major (deprecated in #296) #316 (remove the deprecated write-only getters at the next prebuilt-provider major — the Phase 3 end state).

Review findings addressed (from the demo-harness verification comment)

• Fixed — targetVersions validation bypass: CdktfConfig.targetVersions (the runGetInDir path) now validates via commons validateTargetVersions (warn + ignore), and the emission-gap check treats invalid ranges as not-wanted instead of throwing — a malformed range previously crashed cdktn get via semver.intersects.
• Fixed — ephemeral lifecycle narrowed: new TerraformEphemeralResourceLifecycle (precondition/postcondition only) replaces the full managed-resource lifecycle on the ephemeral API, before it ships as jsii surface.
• Fixed (docs) — provider-fn self-reference cycle: generated provider-function JSDoc and TerraformProviderFunction.invoke now warn against calling a provider's functions inside that same provider's configuration block.
• Documented — ephemeral × write-only: registration deliberately skips ephemeral resources — write-only is a state concept, ephemeral resources have no state, and no schema in the sweep (incl. vault's 16 ephemeral resources) combines the two.
• Follow-up — nested write-only registration (Nested write-only attributes don't register targetVersions feature usage #308): write_only inside nested blocks gets the deprecated getter but skips usage registration; deep config scanning deserves its own PR (a miss degrades to the plan-time error, not silent breakage).
• Pre-existing / by design: the AWS barrel-import OOM predates this PR (lazy-index is the existing mitigation); Fn.ephemeralasnull + sensitive on outputs is Terraform semantics (docs candidate); list resources / actions / resource identity codegen is the RFC Phase 4 deferral noted above.

Second review round addressed

• Fixed — cross-App usage-registry leak: App construction now resets both usage registries (a new App = a new synthesis session). Within an App the process-global registry is by design — every stack resolves the same targetVersions from App context — but usage no longer leaks into later, unrelated Apps in the same process (the reviewer's jest scenario). Regression tests cover the exact sequence; the flag-gated Fn registry had the same latent leak and is reset too.
• Fixed — emission-gap warning on cache hits: the warning moved from readProviderSchema into readSchema, running for cached and fresh schemas using the cli_name/cli_version stamps the schema carries. Regression test: two readSchema calls against a cache dir → one producer call, two warnings.
• Docs — matrix as source of truth: both hand-maintained maps now name features-matrix.json as their source and cross-reference each other; automated drift check tracked in In-repo consistency check for the provider-feature availability matrix and its hand-maintained maps #309.

Third review round addressed (cfncompat/awscc demo findings)

• Fixed — dynamic-typed provider-function returns coerced to string: generated wrappers for dynamic/object/map return types no longer wrap in Token.asString(...); they return the raw invoke() IResolvable, which Tokenization.isResolvable() recognizes — struct-typed attribute assignments (OutputReference.internalValue) no longer vanish silently from synth output.
• Fixed — invoke() dropped literal null positional arguments: arguments are now validated per position (variadic flattening stays in the generated wrappers where the signature is known), and a latent third bug this exposed was fixed too: FunctionCall rendered args via Array.prototype.join, which collapses null entries to empty strings — resolved null/undefined now render as the Terraform null keyword. Regression tests pin the exact provider::cfncompat::condition_if(true, null, {"a" = 1}) rendering.

Resolves Terraform CDK issues

• COMPONENT: Support Terraform ephemeral resources hashicorp/terraform-cdk#3886
• provider-generator: Add support for provider functions hashicorp/terraform-cdk#3686

🤖 Generated with Claude Code

[sakul-learning]

Draft PR #296 review notes / verification results

I built and published a small reusable demo harness against this PR head so the new provider-protocol features can be exercised with real generated AWS bindings:

https://github.com/sakul-learning/cdktn-provider-features-demo

PR head used for the run: 78299b4b9ece16c45ab26a0cd28dec08e8ba57e9.

Verification performed

Using the PR-head packages directly, not npmjs:

• pnpm run assert:prhead-deps confirms cdktn resolves from the PR-head worktree and the CLI comes from the PR-head cdktn-cli bundle.
• Terraform 1.15.7:
  • captured AWS provider schema
  • generated AWS bindings
  • ran schema inspection and generated-feature assertions
  • ran pnpm exec tsc --noEmit
  • ran pnpm run synth:all
  • ran pnpm run plan:all
• OpenTofu 1.12.3:
  • reused the Terraform-generated bindings
  • ran generated-feature assertions
  • ran pnpm exec tsc --noEmit
  • ran pnpm run synth:all
  • ran pnpm run plan:all

The AWS provider schema from Terraform 1.15.7 exposed:

• provider functions: 4
• ephemeral resources: 10
• list resources: 153
• provider actions: 11
• resource identities: 437

Generated bindings confirmed:

• provider functions: yes
• ephemeral resources: yes
• list resources: no first-class generated bindings observed
• provider actions: no first-class generated bindings observed
• resource identities: no first-class generated bindings observed

OpenTofu 1.12.3 successfully planned the examples with the Terraform-generated bindings. A separate OpenTofu schema probe exposed provider functions, ephemeral resources, and resource identities, but reported list_resource_schemas: 0 and action_schemas: 0, so Terraform 1.15.7 is the better generator binary for this comprehensive AWS capability test.

UX / integration notes from the demo

These are non-blocking, but important feedback from trying to use the generated bindings in a real app:

• I switched the examples from the AWS barrel import (./.gen/providers/aws) to direct generated-module imports (provider, provider-functions, and the specific ephemeral resource module). The barrel import path pulled in thousands of generated AWS classes and hit the Node/TypeScript OOM path during synth.
• After that import change, I had to replace the remaining namespace-style references (provider.*, providerFunctions.*, and ephemeral...*) with direct class references like AwsProvider, AwsProviderFunctions, and EphemeralAwsSecretsmanagerRandomPassword.
• The ephemeral output needs both Fn.ephemeralasnull(...) and sensitive: true so Terraform and OpenTofu accept the plan. Without ephemeralasnull, the ephemeral value is used in a non-ephemeral context; without sensitive, the output still fails because it is derived from secret ephemeral data.
• I removed a trial use of AwsProviderFunctions.userAgent(...) inside AwsProvider configuration after OpenTofu reported a provider self-reference/cycle. The mistaken assumption was that provider-defined functions behave like pure static helpers that are safe to use while configuring that same provider. The generated call is still a provider-namespaced function (provider::aws::user_agent), so putting it inside the aws provider block asks Terraform/OpenTofu to evaluate an AWS provider function while the AWS provider is still being configured. The demo now exercises provider functions in outputs instead, after provider configuration exists.

Non-blocking findings / suggested follow-ups

• targetVersions validation bypass in cdktn get path
  targetVersions is validated in parseConfig (packages/@cdktn/commons/src/config.ts), but cdktn get appears to use CdktfConfig.read() plus the raw targetVersions getter (packages/@cdktn/cli-core/src/lib/cdktf-config.ts, then passed through in packages/@cdktn/cli-core/src/lib/get.ts). That path appears to bypass the parseConfig validator, so malformed targetVersions can reach generation.

• Ephemeral resources expose too much lifecycle surface
  TerraformEphemeralMetaArguments currently exposes lifecycle?: TerraformResourceLifecycle. Terraform ephemeral blocks only support lifecycle precondition / postcondition; they do not support the full managed-resource lifecycle surface such as ignore_changes, replace_triggered_by, create_before_destroy, etc. I’d introduce a narrowed lifecycle type for ephemeral resources, e.g. only precondition and postcondition, and use that in TerraformEphemeralMetaArguments.

• Nested write-only attributes do not register feature usage
  The constructor-time write-only registration only iterates top-level assignable attributes in resource.configStruct.assignableAttributes. Provider schemas can put write_only inside nested blocks, not only on top-level resource config attributes, so nested write-only usage can skip registerProviderFeatureUsage("writeOnlyAttributes").

• Generated ephemeral resources skip write-only feature usage registration
  The generator currently limits write-only registration to classes whose parent is TerraformResource; generated ephemeral resources do not get that registration path. Maybe current provider schemas do not combine ephemeral resources with write-only attributes, but since this PR is adding generalized newer-protocol support, I’d either cover the ephemeral path or explicitly document/filter why that combination cannot happen.

Overall: the core provider-function and ephemeral-resource generated bindings worked in real synth/plan flows for Terraform and OpenTofu once the UX issues above were accounted for. The largest current generated-coverage gap is that Terraform exposes AWS list resources, actions, and resource identities, but I did not see first-class generated bindings for those schema sections yet.

[vincenthsh]

not sure if useful to have this in every JSDocs - it's a very limited edge case

[so0k]

deliberately not exposed there because the combination doesn't occur - verified empirically across every provider in the 1.15.6 sweep dump — zero ephemeral attributes carry write_only.

[vincenthsh]

While extending cdktn-provider-features-demo to port real aws-cdk-lib L2 construct logic onto generated awscc bindings + the cfncompat provider's intrinsic-function polyfills, I hit two cdktn-side bugs (not cfncompat bugs) worth flagging here. Full context/examples: examples/l2-kinesis-stream, README "UX notes" section.

1. Provider-function bindings that return a documented "dynamic" type are force-coerced to string, silently dropping struct-typed results

Some provider::*::* functions (e.g. cfncompat's condition_if, matching Fn::If) are documented as returning a dynamic type — either branch of Fn::If can be a string or an object. But the generated TS wrapper unconditionally routes the result through Token.asString(...):

public static conditionIf(condition: any, valueIfTrue: any, valueIfFalse: any, providerLocalName?: string): any {
  return cdktn.Token.asString(cdktn.TerraformProviderFunction.invoke(providerLocalName ?? "cfncompat", "condition_if", [condition, valueIfTrue, valueIfFalse])) as any;
}

That's fine for the common string-return case, but breaks when the result is assigned to a struct-typed resource attribute: Tokenization.isResolvable() doesn't recognize a Token.asString-wrapped value, so the struct's generated setter (e.g. KinesisStreamStreamEncryptionOutputReference.set internalValue) treats it as a plain object with no known keys and the whole attribute silently disappears from synth output — no warning, no error.

Repro:

new KinesisStream(this, "stream", {
  streamEncryption: CfncompatProviderFunctions.conditionIf(
    someCondition, {}, { encryptionType: "KMS", keyId: "alias/aws/kinesis" },
  ),
});

terraform plan shows no stream_encryption key at all in the planned resource.

Workaround: call cdktn.TerraformProviderFunction.invoke("cfncompat", "condition_if", [...]) directly, bypassing the generated wrapper — its return type (IResolvable, no Token.asString) is handled correctly by struct setters.

Suggested fix: only wrap a provider function's result in Token.asString(...) when its schema-declared return type is actually string-shaped; leave functions documented/typed as dynamic (condition_if, find_in_map, select, ...) as a raw IResolvable.

2. TerraformProviderFunction.invoke(...) silently drops literal null positional arguments from any call

TerraformProviderFunction.invoke's runtime validates its whole args array with a single variadic(anyValue) validator — the same one used for genuinely variadic argument lists (e.g. condition_and/condition_or's conditions). variadic's underlying listOf() unconditionally filters out null/undefined entries, which is correct for a variadic list, but wrong for a fixed-arity call: passing null for one positional argument (e.g. condition_if(condition, value_if_true, value_if_false)'s value_if_true) drops it from the argument list entirely instead of preserving its slot.

Repro:

TerraformProviderFunction.invoke("cfncompat", "condition_if", [someCondition, null, { a: 1 }])

terraform plan fails:

Error: Not enough function arguments
while calling provider::cfncompat::condition_if(condition, value_if_true, value_if_false)
Function "provider::cfncompat::condition_if" expects 3 argument(s). Missing value for "value_if_false".

(Terraform actually complains about the third argument going missing, since null — meant for the second slot — got filtered and everything shifted.)

Workaround: avoid passing null; substitute an empty struct ({}) or other schema-appropriate sentinel where possible. Not a great substitute for CloudFormation's Aws.NO_VALUE (property omission) semantics, but the closest expressible result today.

Suggested fix: TerraformProviderFunction.invoke should validate fixed-arity provider functions per-position (preserving null/undefined in their slot) rather than treating the whole args array as one variadic list; the "flatten variadic trailing args" behavior should only apply when a function is genuinely variadic.

[sakul-learning]

I also did a local provider-schema cache/binding-generation simulation across Terraform/OpenTofu CLI versions to sanity-check the cache-busting behavior around the new schema sections.

What I verified:

• Cache entries are keyed by provider plus fetching CLI product/minor version, so a schema fetched with an older binary does not poison later generation with a newer binary.
• Re-running with the same CLI version hits the existing cache entry.
• Moving across version boundaries creates a new cache entry:
  • Terraform 1.7.x → 1.8.x busts and starts seeing provider functions.
  • OpenTofu 1.6.x → 1.7.x busts but still does not see provider functions in tofu providers schema -json.
  • OpenTofu 1.7.x → 1.8.x busts and starts seeing provider functions.
• The simulation also had to account for the OpenTofu registry key shape (registry.opentofu.org/...) rather than assuming Terraform's registry.terraform.io/... prefix.

One documentation point I think is worth making very explicit because it is easy to misread the matrix:

OpenTofu provider-function language support starts at 1.7.0, so synth-time targetVersions validation should allow provider functions for opentofu >=1.7.0. But provider-function schema emission for binding generation starts at OpenTofu 1.8.0, so users need to run cdktn get with tofu >=1.8.0 if they expect provider-function bindings to be generated.

In other words, targeting OpenTofu 1.7 can be valid for using provider functions, but generating those bindings requires an OpenTofu 1.8+ generator binary. The current split between usage constraints and SCHEMA_EMISSION_BOUNDARIES looks intentional; I’d just document the skew prominently near the matrix/README and/or the fetch-time warning text so future maintainers don’t “fix” the apparent mismatch in the wrong direction.

[eduardomourar]

Overall looks good to me. Just a few non-blocking comments.

[sakul-learning]

Approved.