chore(docs): document feature flags and the canonicalAssetHashes scheme

docs/feature-flags.md

@@ -0,0 +1,122 @@
+# Feature Flags
+
+CDK Terrain uses feature flags to introduce behavior we want as the future
+default without breaking existing projects. Flags are plain context keys read
+from `cdktf.json` (or the `App` constructor's `context` option):
+
+```json
+{
+  "context": {
+    "canonicalAssetHashes": "true"
+  }
+}
+```
+
+The lifecycle, defined in
+[`packages/cdktn/src/features.ts`](../packages/cdktn/src/features.ts):
+
+- `cdktn init` writes every current `FUTURE_FLAGS` entry into new projects'
+  `cdktf.json`, so **new projects always get the future behavior**.
+- Existing projects keep the old behavior until they **opt in** by adding the
+  context key themselves.
+- At the next major release the flagged behavior becomes the unconditional
+  default and the flag is removed.
+
+There are deliberately no opt-_out_ flags: once a behavior is the default in
+a major version, the escape hatch is gone, which keeps the code paths from
+accumulating.
+
+## Current flags
+
+| Flag                              | Behavior when enabled                                                                                                   |
+| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
+| `failOnConstructsOutsideOfStacks` | Synthesis throws if a construct is scoped directly to the `App` instead of a `TerraformStack`.                          |
+| `validateFunctionVersions`        | Each stack statically validates that `Fn` functions are supported by the runtime versions declared in `targetVersions`. |
+| `canonicalAssetHashes`            | `TerraformAsset` / `TerraformModuleAsset` hashes use the canonical entry-framed representation described below.         |
+
+## `canonicalAssetHashes`
+
+Introduced in [#323](https://github.com/open-constructs/cdk-terrain/pull/323)
+(design: [#322](https://github.com/open-constructs/cdk-terrain/issues/322)).
+
+### Why the legacy hash is not enough
+
+`TerraformAsset.assetHash` decides the asset's output path, so it must change
+exactly when the emitted artifact changes. The legacy scheme concatenates
+file contents into one digest, which makes it blind to several
+artifact-affecting properties:
+
+- **Renames** — `a.txt` → `b.txt` with identical content hashes identically,
+  but the emitted archive/directory differs.
+- **Entry boundaries** — files `("x", "yz")` and `("xy", "z")` hash
+  identically.
+- **Permissions** — `chmod 0644` → `0755` changes the zip's external
+  attributes (and whether your Lambda binary is executable) but not the
+  legacy hash.
+- **Empty directories** — added or removed, they are materialized by
+  directory assets yet invisible to the legacy hash.
+
+Symlink metadata was folded into the legacy scheme in
+[#321](https://github.com/open-constructs/cdk-terrain/pull/321) under a
+tagged domain (fixing
+[#320](https://github.com/open-constructs/cdk-terrain/issues/320) without
+changing hashes for symlink-free trees), but the blind spots above are
+structural: they cannot be fixed without changing every asset hash, which is
+why the complete scheme is behind a feature flag.
+
+### The canonical representation
+
+The canonical scheme is modeled on git tree objects and Nix NAR
+serialization. Review of the first draft
+([#322](https://github.com/open-constructs/cdk-terrain/issues/322)) made an
+important observation: those formats are canonical **because they serialize a
+complete metadata model, not payload framing alone** — git tree entries carry
+a mode (`100644` / `100755` / `120000`) and represent directories as objects;
+NAR records node types and an executable marker. Borrowing only the
+`(type, path, payload)` framing re-introduces exactly the permission and
+empty-directory blind spots listed above.
+
+The adopted representation therefore frames, per entry, everything that
+affects the emitted artifact:
+
+| entry                   | record                                                     |
+| ----------------------- | ---------------------------------------------------------- |
+| file                    | `F <mode> <relPath>\0<size>\0` + content                   |
+| symlink                 | `L <mode> <relPath>\0<target byte length>\0` + link target |
+| directory (incl. empty) | `D <relPath>\0`                                            |
+
+where:
+
+- `<mode>` is the octal permission mask (`0o7777` bits) — precisely the bits
+  `archiveSync` preserves in zip external attributes. Directory records carry
+  no mode because neither `archiveSync` nor `copySync` preserves directory
+  permissions; the hash only covers what is emitted.
+- Records appear in sorted directory order with `/`-separated relative
+  paths, independent of platform directory-listing order.
+- Symlinks contribute their metadata, never their target's content, so
+  circular symlinks cannot recurse and shared targets are not
+  double-counted. A symlink at the asset root itself is followed, matching
+  how the source path is opened when the artifact is emitted.
+
+### Migration impact
+
+Enabling the flag changes **every** asset hash once, which changes every
+asset output path (`assets/<name>/<hash>/…`) and re-uploads/re-deploys the
+referencing resources (e.g. Lambda functions) on the next apply. Content is
+unchanged; this is a one-time path migration. `TerraformModuleAsset` users
+can pin hashes across the transition with the existing
+`cdktfStaticModuleAssetHash` context key if needed.
+
+### Why adopting it early is recommended
+
+The canonical scheme becomes the unconditional default at the next major
+release, and the legacy scheme is removed then. Opting in early means:
+
+- the one-time re-deploy happens on your schedule, not the major upgrade's;
+- asset hashes become trustworthy change detectors — anything that alters
+  the emitted artifact (content, names, modes, tree shape, symlinks) alters
+  the hash, and nothing else does;
+- combined with the pinned entry timestamps from
+  [#321](https://github.com/open-constructs/cdk-terrain/pull/321), archives
+  are byte-reproducible, so downstream hashing such as `filebase64sha256`
+  stops drifting.

packages/cdktn/src/features.ts

@@ -38,7 +38,26 @@ export const FAIL_ON_CONSTRUCTS_OUTSIDE_OF_STACKS =
  */
 export const VALIDATE_FUNCTION_VERSIONS = "validateFunctionVersions";
 
+/**
+ * When enabled, TerraformAsset and TerraformModuleAsset compute asset hashes
+ * with a canonical entry-framed scheme modeled on git trees and Nix NAR:
+ * every entry contributes its type, permission mask (for files and
+ * symlinks — the bits archiveSync preserves in zip external attributes),
+ * relative path, payload size, and payload, in sorted order, and
+ * directories — including empty ones — contribute explicit records. Renames,
+ * entry-boundary shifts, mode changes, empty-directory changes, and
+ * file-vs-symlink swaps all change the hash.
+ *
+ * When disabled, the legacy content-concatenation hash is kept for
+ * compatibility: trees without symlinks hash byte-identically to earlier
+ * releases, and only symlink-bearing trees (whose hashes had to change when
+ * symlink following was fixed) get a tagged hash that separates symlink
+ * metadata from file content.
+ */
+export const CANONICAL_ASSET_HASHES = "canonicalAssetHashes";
+
 export const FUTURE_FLAGS = {
   [FAIL_ON_CONSTRUCTS_OUTSIDE_OF_STACKS]: "true",
   [VALIDATE_FUNCTION_VERSIONS]: "true",
+  [CANONICAL_ASSET_HASHES]: "true",
 };

packages/cdktn/src/private/fs.ts

@@ -3,11 +3,33 @@
 import * as fs from "fs";
 import * as path from "path";
 import * as crypto from "crypto";
-import { zipSync } from "fflate";
+import { strToU8, zipSync } from "fflate";
+import type { ZipOptions } from "fflate";
 import { assetCanNotCreateZipArchive } from "../errors";
 
 const HASH_LEN = 32;
 
+// Unix file-type bits for zip external attributes (st_mode upper nibble)
+const S_IFLNK = 0o120000;
+const S_IFREG = 0o100000;
+const PERM_MASK = 0o7777;
+// "version made by" host byte; unzip only honors unix mode attrs when set
+const ZIP_OS_UNIX = 3;
+// Pinned so archives are byte-reproducible across synths. fflate encodes
+// DOS dates from local-time getters and rejects years outside 1980-2099,
+// so this must be a local-time 1980 date (a UTC one underflows to 1979 in
+// timezones west of UTC).
+const ZIP_ENTRY_MTIME = new Date(1980, 0, 1);
+
+/**
+ * Zip external attributes for a unix mode: file-type and permission bits
+ * belong in the high 16 bits of the 32-bit field.
+ * @param mode - unix st_mode bits (type | permissions)
+ */
+function zipAttrs(mode: number): number {
+  return (mode << 16) >>> 0;
+}
+
 // Full implementation at https://github.com/jprichardson/node-fs-extra/blob/master/lib/copy/copy-sync.js
 /**
  * Copy a file or directory. The directory can have contents and subfolders.
@@ -21,11 +43,12 @@ export function copySync(src: string, dest: string) {
    */
   function copyItem(p: string) {
     const sourcePath = path.resolve(src, p);
-    const stat = fs.statSync(sourcePath);
-    if (stat.isFile()) {
+    const stat = fs.lstatSync(sourcePath);
+    if (stat.isSymbolicLink()) {
+      fs.symlinkSync(fs.readlinkSync(sourcePath), path.resolve(dest, p));
+    } else if (stat.isFile()) {
       fs.copyFileSync(sourcePath, path.resolve(dest, p));
-    }
-    if (stat.isDirectory()) {
+    } else if (stat.isDirectory()) {
       walkSubfolder(p);
     }
   }
@@ -51,52 +74,177 @@ export function copySync(src: string, dest: string) {
  */
 export function archiveSync(src: string, dest: string) {
   try {
-    const files: Record<string, Uint8Array> = {};
+    const files: Record<string, [Uint8Array, ZipOptions]> = {};
     const walk = (dir: string, prefix: string) => {
       for (const entry of fs.readdirSync(dir)) {
         const full = path.join(dir, entry);
         const zipPath = prefix ? `${prefix}/${entry}` : entry;
-        if (fs.statSync(full).isDirectory()) {
+        const stat = fs.lstatSync(full);
+        if (stat.isSymbolicLink()) {
+          // Store the link target as the entry data with S_IFLNK attrs so
+          // extractors recreate the symlink instead of a copy of the target.
+          // Never recursing through links also makes cycles unreachable.
+          files[zipPath] = [
+            strToU8(fs.readlinkSync(full)),
+            {
+              os: ZIP_OS_UNIX,
+              attrs: zipAttrs(S_IFLNK | (stat.mode & PERM_MASK)),
+              level: 0,
+            },
+          ];
+        } else if (stat.isDirectory()) {
           walk(full, zipPath);
         } else {
-          files[zipPath] = fs.readFileSync(full);
+          files[zipPath] = [
+            fs.readFileSync(full),
+            {
+              os: ZIP_OS_UNIX,
+              attrs: zipAttrs(S_IFREG | (stat.mode & PERM_MASK)),
+            },
+          ];
         }
       }
     };
     walk(src, "");
-    fs.writeFileSync(dest, zipSync(files, { level: 9 }));
+    fs.writeFileSync(
+      dest,
+      zipSync(files, { level: 9, mtime: ZIP_ENTRY_MTIME }),
+    );
   } catch (err: any) {
     throw assetCanNotCreateZipArchive(src, dest, err);
   }
 }
 
+export interface HashPathOptions {
+  /**
+   * Use the canonical entry-framed hash instead of the legacy
+   * content-concatenation hash. Enabled through the `canonicalAssetHashes`
+   * feature flag.
+   */
+  readonly canonical?: boolean;
+}
+
 /**
  * Compute a stable MD5 hash of a file or directory's contents.
- * Directories are hashed by recursively folding each file's contents into
- * the same digest, in directory-listing order.
+ * In both schemes symlinks are hashed by their metadata (path + target)
+ * instead of being followed, so shared targets are not double-counted and
+ * cycles cannot recurse; a symlink at the root itself is followed, matching
+ * how the asset source path is opened when the artifact is emitted.
  * @param src - path to a file or directory to hash
+ * @param options - hash scheme selection, see {@link HashPathOptions}
  * @returns uppercased hex digest, truncated to HASH_LEN characters
  */
-export function hashPath(src: string): string {
-  const hash = crypto.createHash("md5");
+export function hashPath(src: string, options: HashPathOptions = {}): string {
+  const digest = options.canonical
+    ? canonicalHashPath(src)
+    : legacyHashPath(src);
+  return digest.slice(0, HASH_LEN).toUpperCase();
+}
+
+/**
+ * Legacy-compatible hash: file contents fold into one digest in
+ * directory-listing order, exactly as earlier releases did, so trees without
+ * symlinks keep their historical hashes. Symlink metadata goes into a second
+ * digest, and only when symlinks exist are the two combined under a tagged
+ * outer hash — the tag keeps symlink metadata in a separate domain from file
+ * bytes, so a file containing `foo` can never collide with a symlink
+ * targeting `foo`.
+ * @param src - path to a file or directory to hash
+ */
+function legacyHashPath(src: string): string {
+  const content = crypto.createHash("md5");
+  const links = crypto.createHash("md5");
+  let linkCount = 0;
 
   /**
-   * Walk `p`, feeding any file contents into the enclosing hash accumulator.
+   * Walk `p`, feeding file contents and symlink metadata into the enclosing
+   * accumulators.
    * @param p - path to walk
+   * @param relPath - path of `p` relative to the walk root, `/`-separated
+   * @param isRoot - follow a symlink only at the root, matching how the
+   * asset's own path is resolved when it is read
    */
-  function hashRecursion(p: string) {
-    const stat = fs.statSync(p);
-    if (stat.isFile()) {
-      hash.update(fs.readFileSync(p));
+  function hashRecursion(p: string, relPath: string, isRoot = false) {
+    const stat = isRoot ? fs.statSync(p) : fs.lstatSync(p);
+    if (stat.isSymbolicLink()) {
+      links.update(`${relPath}\0${fs.readlinkSync(p)}\0`);
+      linkCount++;
+    } else if (stat.isFile()) {
+      content.update(fs.readFileSync(p));
     } else if (stat.isDirectory()) {
       fs.readdirSync(p).forEach((filename) =>
-        hashRecursion(path.resolve(p, filename)),
+        hashRecursion(
+          path.resolve(p, filename),
+          relPath ? `${relPath}/${filename}` : filename,
+        ),
       );
     }
   }
 
-  hashRecursion(src);
-  return hash.digest("hex").slice(0, HASH_LEN).toUpperCase();
+  hashRecursion(src, "", true);
+  if (linkCount === 0) {
+    return content.digest("hex");
+  }
+  const outer = crypto.createHash("md5");
+  outer.update("cdktn/asset-hash/symlinks/v1\0");
+  outer.update(content.digest("hex"));
+  outer.update(links.digest("hex"));
+  return outer.digest("hex");
+}
+
+/**
+ * Canonical hash, modeled on git trees and Nix NAR: every entry is framed
+ * with everything that affects the emitted artifact —
+ *
+ * - files:      `F <mode> <relPath>\0<size>\0` + content, where `<mode>` is
+ *               the octal permission mask (`0o7777` bits) that archiveSync
+ *               preserves in zip external attributes,
+ * - symlinks:   `L <mode> <relPath>\0<target byte length>\0` + target,
+ * - directories (including empty ones, which copySync materializes):
+ *               `D <relPath>\0`, with no mode — neither archiveSync nor
+ *               copySync preserves directory permissions,
+ *
+ * in sorted directory order with `/`-separated relative paths, so renames,
+ * entry-boundary shifts, permission changes, empty-directory changes, and
+ * file-vs-symlink swaps all change the digest.
+ * @param src - path to a file or directory to hash
+ */
+function canonicalHashPath(src: string): string {
+  const hash = crypto.createHash("md5");
+
+  /**
+   * Walk `p`, framing each entry into the enclosing hash accumulator.
+   * @param p - path to walk
+   * @param relPath - path of `p` relative to the walk root, `/`-separated
+   * @param isRoot - follow a symlink only at the root, matching how the
+   * asset's own path is resolved when the artifact is emitted
+   */
+  function hashRecursion(p: string, relPath: string, isRoot = false) {
+    const stat = isRoot ? fs.statSync(p) : fs.lstatSync(p);
+    const mode = (stat.mode & PERM_MASK).toString(8);
+    if (stat.isSymbolicLink()) {
+      const target = fs.readlinkSync(p);
+      hash.update(`L ${mode} ${relPath}\0${Buffer.byteLength(target)}\0`);
+      hash.update(target);
+    } else if (stat.isFile()) {
+      const data = fs.readFileSync(p);
+      hash.update(`F ${mode} ${relPath}\0${data.length}\0`);
+      hash.update(data);
+    } else if (stat.isDirectory()) {
+      if (relPath) {
+        hash.update(`D ${relPath}\0`);
+      }
+      for (const filename of fs.readdirSync(p).sort()) {
+        hashRecursion(
+          path.resolve(p, filename),
+          relPath ? `${relPath}/${filename}` : filename,
+        );
+      }
+    }
+  }
+
+  hashRecursion(src, "", true);
+  return hash.digest("hex");
 }
 
 /**