Content mappers

[Andrew Branch (andrewbranch)]

Implements microsoft/TypeScript#63800 (comment)

Overview

Content mappers are external integrations that allow TypeScript to include otherwise unsupported file types in a program. They transform a foreign file’s original text into valid TypeScript syntax and provide mappings between the original and transformed content.

Users specify a set of file extensions to be handled by a content mapper package in a tsconfig.json file:

{
  "compilerOptions": {
    // ...
  },
  "contentMappers": [
    {
      "package": "vue-content-mapper",
      "extensions": [".vue"],
      "options": {
        "strictTemplates": true
      }
    }
  ],
  "include": ["src"] // implicitly includes .vue as well as .ts
}

When contentMappers are specified, tsc must be run with --runExternalCode. VS Code passes --runExternalCode to tsc --lsp only in trusted workspaces; otherwise, contentMappers are ignored in the LSP server.

The package field will be resolved as a Node.js module name. The optional options field must be an object and is passed through to the mapper.

The package.json of the content mapper package must specify a typescript top-level field with a nested contentMapper field describing how to spawn the mapper process and what compiler options its transform requires. A mapper that reads additional project-specific configuration from external sources beyond those compiler options can additionally declare dynamicConfig: true:

{
  "name": "vue-content-mapper",
  "version": "1.0.0",
  "typescript": {
    "contentMapper": {
      "exec": ["node", "dist/server.js"],
      "compilerOptions": ["module", "jsx", "jsxImportSource"],
      "dynamicConfig": true
    }
  }
}

Note that the content mapper process need not be run with Node.js or implemented in JavaScript; package resolution serves as a convenient way to associate a content mapper with a versioned identity that can be managed alongside other dependencies in a project, but the exec field can specify any command.

VS Code extensions can also register content mapper integrations with the TypeScript extension. A registration always supplies the extensions that should trigger configured-project discovery, and may additionally provide an inline manifest and options for using the mapper in inferred projects. Configured projects continue to use only the contentMappers declared in their config files; extension-provided inferred-project mappers never modify configured project behavior.

Protocol

When constructing a program for a config that specifies contentMappers, module resolution recognizes file lookups for the specified extensions and requests transformed content from mapper processes over STDIO. Content mappers communicate with TypeScript over JSON-RPC. TypeScript sends all requests; mappers do not send requests or notifications. All mappers handle initialize, openProject, transform, and closeProject. A process may serve multiple projects, each identified by an opaque projectHandle.

type PositionEncoding = "utf-8" | "utf-16";

interface InitializeParams {
    protocolVersion: 1;
    /** The position encodings supported by TypeScript. The mapper must choose one of these encodings. */
    positionEncodings: PositionEncoding[];
    /** BCP 47 locale requested for diagnostics. */
    locale?: string;
}

interface InitializeResult {
    /** Must match the protocolVersion sent in InitializeParams. */
    protocolVersion: 1;
    /** The position encoding the mapper will use for all span mapping positions and diagnostic positions. */
    positionEncoding: PositionEncoding;
    /**
     * The source identifier displayed for mapper-produced diagnostics.
     * Must not be "ts", "tsc", "typescript", or any file extension TypeScript understands.
    */
    diagnosticSource: string;
}

interface OpenProjectParams {
    /** Absolute tsconfig path, or an empty string for a project without a config file. */
    configFileName: string;
    /** Opaque process-local handle assigned by TypeScript. */
    projectHandle: string;
    /** Object from the contentMappers entry, when specified. */
    options?: Record<string, unknown>;
    /** The project's effective compiler options. */
    compilerOptions: CompilerOptions;
}

interface OpenProjectResult {
    /**
     * Stable fingerprint of all dynamically discovered configuration that can affect transforms.
     * Required, and only allowed, when the mapper declares `dynamicConfig: true`.
     */
    configIdentity?: string;
    /**
     * Absolute file names whose changes may alter configIdentity or transform output.
     * May only be returned when the package declares `dynamicConfig: true`. Do not include
     * the files being transformed; those are watched separately.
     */
    watchedFiles?: string[];
    /** Diagnostics for invalid values in this mapper's contentMappers options object. */
    optionDiagnostics?: OptionDiagnostic[];
}

interface OptionDiagnostic {
    /**
     * Property names and nonnegative array indexes relative to the mapper entry's options object.
     * An empty path reports the diagnostic on the options object itself.
     */
    path: (string | number)[];
    messageText: string;
    code?: number;
}

interface TransformParams {
    fileName: string;
    /** Original content of the file to be transformed. */
    content: string;
    /** Project handle supplied in openProject. */
    projectHandle: string;
}

interface MappedOutput {
    /** Valid JS, JSX, TS, TSX, or JSON text that TypeScript can parse. */
    text: string;
    /** The virtual file extension that determines how TypeScript parses this output. */
    extension: ".js" | ".jsx" | ".mjs" | ".cjs" | ".ts" | ".tsx" | ".mts" | ".cts" | ".json";
    /** Mappings between the original and transformed content. */
    mappings?: SpanMapping[];
    /** Framework-specific directives that suppress TypeScript diagnostics in virtual ranges. */
    diagnosticDirectives?: DiagnosticDirectives;
}

enum DiagnosticDirectivePolicy {
    Ignore = 0,
    Expect = 1,
}

interface UnusedExpectDirectiveDiagnostic {
    /** Diagnostic code reported when an `Expect` directive suppresses no diagnostics. */
    code: number;
    /** Diagnostic text reported when an `Expect` directive suppresses no diagnostics. */
    messageText: string;
}

interface DiagnosticDirectives {
    /** Shared diagnostics reported for unused `Expect` directives. */
    unusedExpectDirectiveDiagnostics: UnusedExpectDirectiveDiagnostic[];
    directives: MappedDiagnosticDirective[];
}

/** Positions and lengths are in the specified `positionEncoding`. */
type MappedDiagnosticDirective = [
    /** Location of the framework directive in the original source. */
    originalStart: number,
    originalLength: number,
    /** Region of virtual code affected by the directive. */
    virtualStart: number,
    virtualEnd: number,
    policy: DiagnosticDirectivePolicy,
    /**
     * Index into `unusedExpectDirectiveDiagnostics`. Required for `Expect` directives
     * when the array contains more than one entry.
     */
    unusedExpectDirectiveIndex?: number,
];

interface TransformResult extends MappedOutput {
    /** Parse errors in the original content. */
    diagnostics?: MapperDiagnostic[];
    /** Additional virtual files associated with this input. */
    supplemental?: MappedOutput[];
}

interface CloseProjectParams {
    /** Project handle supplied in openProject. */
    projectHandle: string;
}

/** Positions and lengths are in the specified `positionEncoding`. */
type SpanMapping = [
    virtualStart: number,
    virtualLength: number,
    originalStart: number,
    originalLength: number,
    kind: SpanMapKind,
    features?: SpanMapFeature,
];

enum SpanMapKind {
    /** Verbatim spans in virtual text have the same length and content as their counterparts in original text. */
    Verbatim = 0,
    /** Atom spans in virtual text may have different length and content than their counterparts in the original text. */
    Atom = 1,
    /** Alias spans in virtual text may have different length and content than their counterparts in the original text, but diagnostics display their original text. */
    Alias = 2,
}

/** Controls which TypeScript language service features may use a span. */
enum SpanMapFeature {
    None = 0,
    Hover = 1 << 0,
    SignatureHelp = 1 << 1,
    Completion = 1 << 2,
    Definition = 1 << 3,
    TypeDefinition = 1 << 4,
    Implementation = 1 << 5,
    References = 1 << 6,
    DocumentHighlights = 1 << 7,
    Rename = 1 << 8,
    CallHierarchy = 1 << 9,
    CodeActions = 1 << 10,
    Formatting = 1 << 11,
    InlayHints = 1 << 12,
    SemanticTokens = 1 << 13,
    FoldingRanges = 1 << 14,
    SelectionRanges = 1 << 15,
    LinkedEditing = 1 << 16,
    AutoInsert = 1 << 17,
    DocumentSymbols = 1 << 18,
    CodeLens = 1 << 19,
    /** Enables every language service feature. This is the default when `features` is omitted. */
    All = (CodeLens << 1) - 1,
}

/** Start and length are in the specified `positionEncoding`. */
interface MapperDiagnostic {
    messageText: string;
    start: number;
    length: number;
    code?: number;
}

TypeScript sends openProject before the first transform for a mapper in a project and includes that project handle in every transform request. options and compilerOptions are supplied only in openProject; mappers must store any project-specific state needed by transform keyed by the project handle and release it during closeProject. If project state incorporates any configuration beyond what gets passed to openProject, the mapper must declare "dynamicConfig: true" in its package.json and return a hash of that additional configuration from openProject.

Option diagnostics returned in optionDiagnostics from openProject are reported against the corresponding value in contentMappers[].options. The path identifies nested object properties or array elements structurally, allowing TypeScript to locate the value in the config file:

/*
{
  "path": ["plugins", 0, "name"],
  "messageText": "Plugin option 'name' must be a string",
  "code": 123
}
*/

{
  "contentMappers": [
    {
      "package": "vue-content-mapper",
      "extensions": [".vue"],
      "options": {
        "plugins": [
          {
            "name": true
//                  ^^^^ vue123: Plugin option 'name' must be a string
          }
        ]
      }
    }
  ]
}

A mapper may return supplemental outputs when a file contributes more than one TypeScript or JavaScript file, such as an Astro component containing multiple script blocks. TypeScript automatically includes these outputs in the same program as the canonical output, so they participate in binding and type checking without needing to be imported. Supplemental outputs receive compiler-assigned virtual file names based on their order and extension, but those names are not module resolution targets and cannot be imported directly. Imports written inside supplemental outputs resolve relative to the directory containing the original file.

Span maps

For a content mapper to be useful, it needs to provide a mapping between the transformed output and the original content. In the CLI, these mappings are used to show TypeScript-generated diagnostics in the original, non-TypeScript content. Take a simple example:

// original content:
(+ 1 2 "oops")

// transformed content:
add(1, 2, "oops");

// span mapping:
add(1, 2, "oops");
^^^                 [0, 3)   → [1, 2) + atom
    ^               [4, 5)   → [3, 4) 1 verbatim
       ^            [7, 8)   → [5, 6) 2 verbatim
          ^^^^^^    [10, 16) → [7, 13) "oops" verbatim

TypeScript sees and checks the transformed content, in this case producing a diagnostic for the string literal "oops" because it is not a number. The span mapping allows TypeScript to report the diagnostic in the original content, at the correct location of the string literal ([7, 13), instead of [10, 16)).

In this example, add mapped to + with SpanMapKind.Atom, indicating a correspondence between the two spans, but with different lengths and content. If the name add failed to resolve, the displayed diagnostic range would cover +, but the message would still reference the identifier add:

add.lisp:1:2 - error TS2304: Cannot find name 'add'.

1 (+ 1 2 "oops")
   ~

The mapper can use SpanMapKind.Alias instead of SpanMapKind.Atom to indicate that the virtual and original text name the same entity. When the diagnostic is rendered, the original text of the alias span (+) will be substituted for the virtual text (add) in the diagnostic message:

add.lisp:1:2 - error TS2304: Cannot find name '+'.
1 (+ 1 2 "oops")
   ~

Gaps in the span map are treated as fully synthesized content and cannot be mapped to a location in the original text. Unlike in Volar, diagnostics in unmappable regions are not discarded. In the CLI, they cause a short snippet of the transformed content to be shown with the diagnostic. A common case may be a content mapper that synthesizes an import statement at the top of the file used in scaffolding. If that import fails to resolve, the user will see:

app.vue:1:26 - error TS2307: Cannot find module '@vue/content-mapper-utils' or its corresponding type declarations.
  This location is in code generated by the content mapper '@vue/content-mapper@1.0.0' and has no corresponding location in the original file.

1 import { scaffolding } from "@vue/content-mapper-utils";
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~

Spans in the virtual text must not overlap, but multiple may map to the same span in the original content. In other words, one range in the original content can map to multiple ranges in the transformed content. This can be useful in the language server when combined with SpanMapFeature and SpanMapKind. Broadly speaking, when a language server request is received for a position in a content-mapped file, the handler maps it to every projection whose feature mask includes the requested operation, performs analysis on the transformed content, and maps visible results back through spans that participate in the same feature. This lets a mapper independently select, for example, one projection for hover and another for definitions or references.

The language server currently supports the following features for content-mapped files:

• Diagnostics - always mapped to original content where possible; diagnostics in synthesized regions are collected and reported at the top of the file. Declared-but-not-used diagnostics are automatically suppressed in synthesized regions. Diagnostics are intentionally not represented by a span-map feature flag. Framework-specific semantic diagnostic suppression is instead expressed explicitly through diagnosticDirectives.
• Position-based features - hover, signature help, completions, definitions, type definitions, implementations, source definitions, references, document highlights, rename, call hierarchy, code actions, formatting, linked editing, and auto-insert map incoming positions or ranges through spans participating in their corresponding SpanMapFeature flag.
• Document-wide features - inlay hints, semantic tokens, folding ranges, selection ranges, document symbols, and CodeLens map visible results back only through spans participating in their corresponding flag.
• Text edits - feature participation does not make a mapping edit-safe. Rename, code action, completion, and formatting edits may be written back only through exact, length-preserving SpanMapKind.Verbatim mappings.

Language service requests and visible results can be disabled independently for any span by clearing the corresponding bits, or disabled for all features with SpanMapFeature.None. If features is omitted from the span mapping tuple, it defaults to SpanMapFeature.All, enabling every supported language service feature for that span.

Note

Unlike with Volar, feature participation must be statically determined by the content mapper during transformation. This level of LSP feature mapping is not intended to replace fully custom language servers. TypeScript’s goal in providing language service support for content-mapped files is to support a good editing experience inside <script> blocks or similar verbatim ranges that embed normal TypeScript or JavaScript code without a third-party language server needing to proxy every request unchanged. We expect that ecosystems implementing complex transforms may still want to implement their own language servers alongside TypeScript’s, and either augment or replace TypeScript’s implementation of these language service features. Content mappers provide a baseline editing experience, but they also provide the API foundation for more specialized language servers to build on. Vue tooling, for example, may choose to enable TypeScript features only for selected projections while a separate language server handles the rest, accessing the AST, type, and symbol information of transformed content through an API connection to TypeScript’s language server.

Diagnostic directives

Virtual text can include // @ts-ignore or // @ts-expect-error directives to suppress a TypeScript diagnostic on the next line. However, Vue supports its own diagnostic directives, which have different scope than TypeScript’s:

// Original:
<!-- @vue-expect-error -->
<div
  :id="firstError"
  :title="secondError"
></div>

// Virtual:
__VLS_asFunctionalElement1(
    __VLS_intrinsics.div,
    __VLS_intrinsics.div,
)({
    id: (__VLS_ctx.firstError),
    title: (__VLS_ctx.secondError),
});

The expect-error directive in Vue applies to errors on every line of the <div> element in the original text, which are broken up into multiple lines in the virtual text too. This behavior can’t be replicated with any number of // @ts-expect-error directives. Instead, the content mapper can return diagnosticDirectives in the transform result, which specify a virtual range and a policy of either DiagnosticDirectivePolicy.Expect or DiagnosticDirectivePolicy.Ignore. TypeScript will suppress bind/check diagnostics in that virtual range according to the policy, and report unused Expect directives as diagnostics in the original content. Unused-directive diagnostics are stored once in unusedExpectDirectiveDiagnostics and referenced by index from directive tuples.

Note

In the same way that it’s technically possible for a mapper to put a // @ts-ignore comment between every line of its output, it’s also possible for a mapper to synthesize ignore regions without a corresponding directive in the original content, but this is not recommended. It’s not currently possible to filter diagnostics by code. This is a break from Volar; I want to see if mappers can get around this with different code generation strategies rather than relying on filtering before considering a broad diagnostic filtering feature. As a type checker implementer, we really prefer to report all the type errors we see.

Failure handling

Mappers return diagnostics for unparseable content, and errors in the transformed text itself are handled by TypeScript like any other file. If the mapper fails in an unexpected way (e.g., crashes or doesn’t conform to the protocol), TypeScript reports a localized diagnostic and treats the file as an empty TypeScript file. After five failures in a single project, TypeScript stops attempting to transform files with that content mapper and issues a final diagnostic reporting the failure.

LSP activation

TypeScript’s language server can only know to care about the file extensions registered in contentMappers once the server is running and has discovered a tsconfig.json that specifies them. In the case where a user opens a directory in VS Code and opens a single .vue file, the TypeScript VS Code extension hasn’t even activated, much less spawned a server that knows about a contentMappers registration. To address this, third-party VS Code extensions need to explicitly activate the TypeScript extension and register their content mapper contributions:

const extension = vscode.extensions.getExtension("TypeScriptTeam.native-preview");
const api = await extension?.activate();

const registration = api?.registerContentMappers(
    "publisher.vue-language-features",
    [{ extensions: [".vue"] }],
);

Registering extensions causes TypeScript to inspect matching documents that are already open and discover any configured projects that provide a mapper for them. The returned disposable removes the contribution.

An extension may also provide a mapper for files that do not belong to a configured project by including an inline inferred-project contribution:

const registration = api?.registerContentMappers(
    "publisher.vue-language-features",
    [{
        extensions: [".vue"],
        inferredProjectContribution: {
            options: { strictTemplates: true }, // corresponds to tsconfig.json contentMappers options
            manifest: {                         // corresponds to a content mapper's package.json
                name: "vue-content-mapper",
                version: "1.0.0",
                exec: [process.execPath, mapperEntryPoint],
                cwd: extension.extensionUri,
                compilerOptions: ["module", "jsx", "jsxImportSource"],
                dynamicConfig: true,
            },
        },
    }],
);

It's recommended that extensions always provide an inferredProjectContribution, and to supply a manifest built from resolving the workspace-installed content mapper package, falling back to a bundled version if the package is not installed. But ultimately, the extension is responsible for content mapper resolution and version/fallback policy.

Emit

Content-mapped files are not emitted to JavaScript. When --declaration is enabled, however, declaration files are emitted from the transformed content. The declaration file name for App.svelte is App.d.svelte.ts. Declaration files for supplemental outputs of a file named App.svelte are emitted as App.svelte.0.d.ts, App.svelte.1.d.ts, etc., and are automatically referenced by App.d.svelte.ts. Declaration maps are currently not supported.

Incremental, build, watch, and process consolidation

Content mappers are supported in --incremental, --build, and --watch modes. Each project records sorted mapper transform identities in .tsbuildinfo and compares them during up-to-date checks. Changing an identity forces files handled by that mapper to be transformed again.

For a mapper without dynamicConfig: true, the transform identity is computed without starting its process. It includes the resolved package name and version, the tsconfig entry’s options, and the values of compiler options named by typescript.contentMapper.compilerOptions. Consequently, incremental and solution-build status checks do not spawn processes for mappers with static configuration.

For a mapper declaring dynamicConfig: true, TypeScript uses openProject to obtain configIdentity before an up-to-date decision. The mapper is responsible for changing configIdentity whenever dynamically discovered configuration that can affect transforms changes.

TypeScript watches the absolute paths returned in watchedFiles. A change invalidates only projects that reported that path, closes their current mapper project configuration, obtains a fresh identity and watch set, and performs a normal project rebuild. Other projects using the same mapper package continue using their existing project handles and the shared process.

Note

Modifying a static-config content mapper implementation during local development will not change its identity, so you’ll need to bump the local package.json version, or use --force or --clean to clear cached outputs if testing with --incremental or --build.

In --build mode with project references, and in some instances in the language server, it’s possible to have a project graph with many projects all defining the same content mapper. To avoid excessive spawning of child processes, TypeScript deduplicates content mapper processes by resolved package name and version. One process may have many open project handles. Mappers must isolate project-specific state by projectHandle, accept requests for different projects in any order, and release that state on closeProject. Processes remain alive while any project using that package is retained.

API integration

Content-mapped SourceFiles can be inspected by the JavaScript API. For a content-mapped SourceFile, file.text is the transformed text, file.originalText is the original text, and file.spanMap exposes an API for mapping between the two. Regardless of the positionEncoding used by the content mapper, accessing the span map through the JavaScript API always yields UTF-16 positions.

const mapped = file.spanMap.virtualToOriginalPosition(10);
// { position, fidelity }
// See _packages/native-preview/src/ast/spanMap.ts for details.

If the content mapper provided supplemental outputs for a file, the file names are set on file.supplementalOutputs and can be retrieved with program.getSourceFile(fileName).

Debugging

On the CLI, when the TS_CONTENT_MAPPER_DEBUG environment variable is set, JSON-RPC communication is logged and the mapper process’s STDERR is captured and redirected to tsc’s STDERR.

In the LSP, when the LSP log level is set to Trace, JSON-RPC communication and mapper STDERR flow to the LSP client (you can see the content mapper debug logs in the “TypeScript 7” output channel in VS Code, for example). At lower log levels, mapper STDERR is discarded.

I also have a prototype of a VS Code extension that lets you see the virtual content and span mappings, which I’ll share after this PR is merged.

Later follow-up

• Investigate if declaration maps can work by double-mapping back to original text
• Provide a JavaScript library for implementing the content mapper protocol

[Jake Bailey (jakebailey)]

Some comments, more later

[Jake Bailey (jakebailey)]

No, turns out that was all of my comments

[Remco Haszing (remcohaszing)]

I really appreciate the explanation regarding the security considerations. It felt pointless to me, because tsc typically comes from node_modules anyway. But it makes sense that people should be able to trust the tsc command after they verified it’s good.

I do think quite a lot of people will use this feature if once more language tooling maintainers create content mappers. Maybe it would be nice to add a shorthand option such as -x for eXecute eXternal.

[Jake Bailey (jakebailey)]

LGTM I totally read every line