From 1944c57223c94e50f858130ebca5ba7f4897b1b9 Mon Sep 17 00:00:00 2001 From: Ludovico Fischer Date: Fri, 23 Oct 2026 23:31:34 +0200 Subject: [PATCH 1/4] feat: add token export --- BENCHMARKS.md | 1 + CHANGELOG.md | 7 + README.md | 50 +++ package.json | 5 + scripts/benchmark/benchmark-token-ranges.js | 53 +++ src/reduce-tokens.js | 169 +++++++++ test/helpers/reduce-list.js | 10 + test/integration/package-exports.test.js | 8 + test/property/printer-agreement.test.js | 131 +++++++ test/property/token-edits.test.js | 35 ++ test/property/token-ranges.test.js | 47 +++ test/unit/blocks.test.js | 55 +++ test/unit/create-reducer.test.js | 92 +++++ test/unit/reduceTokens-fusion.test.js | 99 +++++ test/unit/reduceTokens.test.js | 358 ++++++++++++++++++ test/unit/token-core-closers.test.js | 42 +++ test/unit/token-core-contract.test.js | 391 ++++++++++++++++++++ test/unit/token-printer.test.js | 155 ++++++++ tsconfig.json | 7 +- types/reduce-tokens.d.ts | 48 +++ 20 files changed, 1762 insertions(+), 1 deletion(-) create mode 100644 scripts/benchmark/benchmark-token-ranges.js create mode 100644 src/reduce-tokens.js create mode 100644 test/helpers/reduce-list.js create mode 100644 test/property/printer-agreement.test.js create mode 100644 test/property/token-edits.test.js create mode 100644 test/property/token-ranges.test.js create mode 100644 test/unit/blocks.test.js create mode 100644 test/unit/create-reducer.test.js create mode 100644 test/unit/reduceTokens-fusion.test.js create mode 100644 test/unit/reduceTokens.test.js create mode 100644 test/unit/token-core-closers.test.js create mode 100644 test/unit/token-core-contract.test.js create mode 100644 test/unit/token-printer.test.js create mode 100644 types/reduce-tokens.d.ts diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 6a21202..2b10c42 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -400,6 +400,7 @@ infrastructure failures. | `pnpm benchmark:nested-fallbacks` | Deeply nested `var()` fallbacks (depths 16–512) | Regression-gating for recursion and opaque call handling | | `pnpm benchmark:corpus` | Real-world expressions harvested from GitHub vs. `@csstools/css-calc` | Comparative reporting (not a gate) and practical equivalence | | `pnpm benchmark:serialization` | Wide sums, products, and nested calls across serializers | In-process local profiling for serializer changes | +| `pnpm benchmark:token-ranges` | One long token list reduced as many small `[start, end)` ranges | In-process profiling of per-range block matching | | `pnpm test:benchmark` | Unit tests for statistics, bootstrap, and schema validation | Verifying benchmark harness logic | | `pnpm test:benchmark:simulation` | Monte Carlo calibration across adversarial noise scenarios | Verifying bootstrap empirical coverage | | `node scripts/benchmark/benchmark-plugin.js` | PostCSS processing of generated stylesheets (no package script) | In-process local profiling of the adapter | diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b3fd60..951f805 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file. See [commit ## Unreleased +### Features + +- add `postcss-calc/reduce-tokens`, which reduces a `@csstools/css-tokenizer` + token list and returns token replacements. It never reads token source + offsets, so it works on lists edited after tokenization. `postcss-calc/reduce` + shares its implementation and its output is unchanged. + ### Bug fixes - A reduced value that would merge with a neighbouring token is now kept apart diff --git a/README.md b/README.md index 5d65725..b224d37 100755 --- a/README.md +++ b/README.md @@ -87,6 +87,56 @@ const result = reduceCalc('calc(100% + var(--gap))', { Unlike the PostCSS plugin, the standalone reducer does not show warnings by default; provide `onParseError` and/or `onWarn` if you want diagnostics. +### Reduce a token list + +If you already hold the tokens of a value (from `@csstools/css-tokenizer`) and +edit them in place, `postcss-calc/reduce-tokens` reduces them without going +through a string. It reads only each token's type and raw text, never its +source offsets, so the list may have been edited since it was tokenized. Its +other fields must agree: a token's parsed value (`token[4]`) is read along +with its type and raw text, so when you change a token's text, replace the +token with one that `tokenize` produced from the new text. Only the source +offsets may be stale. It returns the replacements that change the text: ordered, non-overlapping +`{ start, end, tokens }` records that replace `tokens.slice(start, end)`. It +does not modify the list; apply the replacements from last to first. + +```js +import { tokenize } from '@csstools/css-tokenizer'; +import reduceTokens from 'postcss-calc/reduce-tokens'; + +const tokens = tokenize({ css: 'calc(1px + 2px) 3px' }); +for (const { start, end, tokens: next } of reduceTokens(tokens).toReversed()) { + tokens.splice(start, end - start, ...next); +} +// tokens now spell 'calc(3px) 3px' +``` + +The replacement tokens are the tokenization of the result text on its own, so +their source offsets (`token[2]`, `token[3]`) describe that text, not their +place in your list. Do not rely on them. Tokens spliced next to unspaced +neighbours can also merge or split when the list is tokenized again, as the +string reducer's output can. + +To skip tokenizing values that cannot contain a math function, check the +string first with the named export `hasPotentialMathFunction(value)`. It is +the quick test `reduceCalc` runs before tokenizing. It may return `true` for a +value with no reducible function, never `false` for one with a reducible +function. + +```js +import { hasPotentialMathFunction } from 'postcss-calc/reduce-tokens'; + +if (hasPotentialMathFunction(value)) { + // tokenize and call reduceTokens +} +``` + +End-of-file tokens in the list (for example from concatenating `tokenize` +results) are ignored. + +It takes the same options as `reduceCalc`. `onParseError` and `onWarn` receive +text built from the current tokens. + ### Standalone reducer options #### `unwrapSingleValue` (default: `false`) diff --git a/package.json b/package.json index bf780cd..4b8c273 100644 --- a/package.json +++ b/package.json @@ -23,6 +23,10 @@ "./reduce": { "types": "./types/reduce.d.ts", "default": "./src/reduce.js" + }, + "./reduce-tokens": { + "types": "./types/reduce-tokens.d.ts", + "default": "./src/reduce-tokens.js" } }, "files": [ @@ -37,6 +41,7 @@ "benchmark:nested-fallbacks": "node scripts/benchmark/benchmark-parser.js nested-fallbacks", "benchmark:corpus": "node scripts/benchmark/benchmark-corpus.js", "benchmark:serialization": "node scripts/benchmark/benchmark-serialization.js", + "benchmark:token-ranges": "node scripts/benchmark/benchmark-token-ranges.js", "test:benchmark": "node --test 'test/unit/benchmark-*.test.js' test/unit/compare-parser-benchmarks.test.js test/unit/corpus-benchmark.test.js", "test:benchmark:simulation": "node test/benchmark/statistical-simulation.js", "benchmark:reanalyze": "node scripts/benchmark/compare-parser-benchmarks.js", diff --git a/scripts/benchmark/benchmark-token-ranges.js b/scripts/benchmark/benchmark-token-ranges.js new file mode 100644 index 0000000..06c7473 --- /dev/null +++ b/scripts/benchmark/benchmark-token-ranges.js @@ -0,0 +1,53 @@ +// Token-range benchmark. One long token list is reduced as many small +// `[start, end)` ranges, the way an editor reduces each declaration in place. +// Each call pays for its own block matching, so the cost should grow with +// the range, not with the length of the whole list. In-process and single +// run: a profiling hint, not a gate (see BENCHMARKS.md). +import { tokenize } from '@csstools/css-tokenizer'; +import { createReducer } from '../../src/reduce-tokens.js'; + +const SIZES = [2_000, 4_000, 8_000, 16_000]; +const RANGE_TEXT = 'calc(1px + 2px)'; +// The tokenizer appends an EOF token, which is not part of any range. +const RANGE_TOKENS = tokenize({ css: RANGE_TEXT }).length - 1; +const REPEATS = 3; + +const reduce = createReducer(); +const target = { warn() {} }; + +/** @param {number} count @return {import('@csstools/css-tokenizer').CSSToken[]} */ +function listOf(count) { + return tokenize({ css: RANGE_TEXT.repeat(count) }); +} + +/** + * Reduce each range in turn and return the best of a few runs, in ms. + * + * @param {import('@csstools/css-tokenizer').CSSToken[]} tokens + * @param {number} count + * @return {number} + */ +function timeRanges(tokens, count) { + let best = Infinity; + for (let repeat = 0; repeat < REPEATS; repeat++) { + const start = performance.now(); + for (let range = 0; range < count; range++) { + const from = range * RANGE_TOKENS; + reduce(tokens, target, from, from + RANGE_TOKENS); + } + best = Math.min(best, performance.now() - start); + } + return best; +} + +console.log('ranges list-tokens ms (best of ' + REPEATS + ') ms per range'); +for (const count of SIZES) { + const tokens = listOf(count); + const ms = timeRanges(tokens, count); + console.log( + String(count).padStart(6), + String(tokens.length).padStart(11), + ms.toFixed(2).padStart(18), + (ms / count).toFixed(5).padStart(14) + ); +} diff --git a/src/reduce-tokens.js b/src/reduce-tokens.js new file mode 100644 index 0000000..b47596a --- /dev/null +++ b/src/reduce-tokens.js @@ -0,0 +1,169 @@ +// Token-level reducer. Like `reduce.js` it has no PostCSS dependency; it is +// for callers that already hold a token list and edit it in place. Source +// offsets in the tokens may be stale and are never read. The other fields +// must be consistent, as `tokenize` produces them. Replacement tokens carry +// fresh offsets that are consistent within the replacement. +import { hasPotentialMathFunction } from './lib/functions.js'; +import { resolveOptions } from './lib/options.js'; +import { reduceTokenList } from './lib/core.js'; + +/** @typedef {import('./reduce.js').ReduceCalcOptions} ReduceCalcOptions */ +/** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ +/** @typedef {import('./lib/serialize/sink.js').TokenSink} TokenSink */ + +/** + * Replace `tokens.slice(start, end)` with `tokens`. + * + * @typedef {object} TokenReplacement + * @property {number} start Index of the first replaced token. + * @property {number} end Index after the last replaced token. + * @property {CSSToken[]} tokens + */ + +/** + * Where a reducer sends its warnings and parse errors. A parse error reaches + * `warn` only when the reducer has no `onParseError` option. + * + * @typedef {object} WarnTarget + * @property {(message: string, plugin?: string) => void} warn + */ + +/** + * Reduce `tokens[start, end)` for the target. Returns the replacements as + * `reduceTokens` does. + * + * @typedef {(tokens: CSSToken[], target: WarnTarget, start?: number, end?: number) => TokenReplacement[]} TokenReducer + */ + +/** + * Check that `[start, end)` names a range of `tokens`. + * + * @param {CSSToken[]} tokens + * @param {number} start + * @param {number} end + * @return {void} + */ +function assertRange(tokens, start, end) { + if ( + !Number.isInteger(start) || + !Number.isInteger(end) || + start < 0 || + end > tokens.length + ) { + throw new RangeError( + `Token range [${start}, ${end}) for ${tokens.length} tokens` + ); + } +} + +/** + * Reduce `tokens[start, end)` with resolved options whose callbacks are set. + * + * @param {import('./reduce.js').ResolvedReduceCalcOptions} options + * @param {CSSToken[]} tokens + * @param {number} start + * @param {number} end + * @return {TokenReplacement[]} + */ +function run(options, tokens, start, end) { + assertRange(tokens, start, end); + /** @type {TokenReplacement[]} */ + const replacements = []; + reduceTokenList( + tokens, + options, + (from, to, printed) => { + // The reducer asks for tokens, so the printed value is a token sink. + const sink = /** @type {TokenSink} */ (printed); + if (sink.identical()) return; + replacements.push({ + start: from, + end: to, + tokens: withOffsets(sink.tokens), + }); + }, + start, + end, + undefined, + true + ); + return replacements; +} + +/** + * The tokens with offsets that locate each one in their own joined text, as + * `tokenize` of that text would give. Reused source tokens are copied, so the + * source list is never shared with a replacement. The detail objects + * (`token[4]`) are shared, so a caller must not edit them in place. + * + * @param {CSSToken[]} tokens + * @return {CSSToken[]} + */ +function withOffsets(tokens) { + let offset = 0; + return tokens.map((token) => { + const from = offset; + offset += token[1].length; + return /** @type {CSSToken} */ ( + /** @type {unknown} */ ([token[0], token[1], from, offset - 1, token[4]]) + ); + }); +} + +/** + * Build a reducer that resolves `opts` once, for a caller that reduces many + * token lists with the same options. Each call sends its warnings to the + * target it is given, unless `opts` has `onWarn`, which takes precedence as + * `onParseError` does. + * + * @param {ReduceCalcOptions} [opts] + * @return {TokenReducer} + */ +function createReducer(opts) { + const options = resolveOptions(opts); + return (tokens, target, start = 0, end = tokens.length) => + run( + { + ...options, + onWarn: + options.onWarn ?? + ((/** @type {string} */ message) => + target.warn(message, 'postcss-calc')), + onParseError: + options.onParseError ?? + ((/** @type {Error} */ error) => + target.warn(error.message, 'postcss-calc')), + }, + tokens, + start, + end + ); +} + +/** A parse error is dropped when the caller gives no `onParseError`. */ +function ignoreParseError() {} + +/** + * Simplify every supported CSS math function in `tokens[start, end)`. Returns + * the replacements that change the text, ordered and non-overlapping, and + * leaves `tokens` untouched. Apply them from last to first, or build a new + * list. Replacement indices are indices into `tokens`, and a range reads as + * the list it names would. The offsets of a replacement's tokens locate them + * in the replacement's own text, counting from 0. An empty or inverted + * range has no replacements. Throws a `RangeError` for a range that is not + * within `tokens`. + * + * @param {CSSToken[]} tokens + * @param {ReduceCalcOptions} [opts] + * @param {number} [start] + * @param {number} [end] + * @return {TokenReplacement[]} + */ +function reduceTokens(tokens, opts, start = 0, end = tokens.length) { + const options = resolveOptions(opts); + options.onParseError ??= ignoreParseError; + return run(options, tokens, start, end); +} + +export default reduceTokens; +export { createReducer, hasPotentialMathFunction }; diff --git a/test/helpers/reduce-list.js b/test/helpers/reduce-list.js new file mode 100644 index 0000000..db0debb --- /dev/null +++ b/test/helpers/reduce-list.js @@ -0,0 +1,10 @@ +export const raw = (tokens) => tokens.map((token) => token[1]).join(''); + +/** Apply `reduceTokens` replacements from last to first, as a caller editing in place would. */ +export function applyReplacements(tokens, replacements) { + const edited = [...tokens]; + for (const { start, end, tokens: next } of replacements.toReversed()) { + edited.splice(start, end - start, ...next); + } + return edited; +} diff --git a/test/integration/package-exports.test.js b/test/integration/package-exports.test.js index c10aed3..df2abd4 100644 --- a/test/integration/package-exports.test.js +++ b/test/integration/package-exports.test.js @@ -50,6 +50,10 @@ test('packed package exposes the standalone reducer with usable types', async () "if (reduceCalc('calc(1px + 2px)') !== 'calc(3px)') throw new Error('reducer failed');", "if (reduceCalc('calc(1 - 2)', { unwrapSingleNegativeNumber: true }) !== '-1') throw new Error('bare negative option failed');", "if (reduceCalc('calc(1 / 2)', { unwrapSingleValue: true }) !== '.5') throw new Error('bare fractional option failed');", + "import reduceTokens from 'postcss-calc/reduce-tokens';", + "import { tokenize } from '@csstools/css-tokenizer';", + "const [replacement] = reduceTokens(tokenize({ css: 'calc(1px + 2px)' }));", + "if (replacement.start !== 0 || replacement.tokens.map((token) => token[1]).join('') !== 'calc(3px)') throw new Error('token reducer failed');", ].join('\n') ); run(process.execPath, [runtime], { cwd: fixture }); @@ -59,6 +63,10 @@ test('packed package exposes the standalone reducer with usable types', async () types, [ "import reduceCalc, { type ReduceCalcOptions } from 'postcss-calc/reduce';", + "import reduceTokens, { type TokenReplacement } from 'postcss-calc/reduce-tokens';", + "import { tokenize } from '@csstools/css-tokenizer';", + "const replacements: TokenReplacement[] = reduceTokens(tokenize({ css: 'calc(1px + 2px)' }), { precision: 2 });", + 'void replacements;', 'const options: ReduceCalcOptions = { precision: false, unwrapSingleNegativeNumber: true, unwrapSingleValue: true };', "const reduced: string = reduceCalc('calc(1px + 2px)', options);", 'void reduced;', diff --git a/test/property/printer-agreement.test.js b/test/property/printer-agreement.test.js new file mode 100644 index 0000000..240b6d4 --- /dev/null +++ b/test/property/printer-agreement.test.js @@ -0,0 +1,131 @@ +// The string reducer and the token reducer print through one emitter. For any +// source, the tokens of the token reducer's edited list must be those of the +// string reducer's output: same types, raw text and parsed values. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import fc from 'fast-check'; +import { TokenType, tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { cssMathSourceArb } from '../helpers/css-math-arbitraries.js'; +import { NUM_RUNS } from '../helpers/laws.js'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const optionsArb = fc.constantFrom( + {}, + { unwrapSingleValue: true }, + { precision: false }, + { precision: 2 } +); + +// Terms the generic math generator never produces: signed zero, degenerate +// keywords, floats, escapes, opaque fallbacks with arbitrary tokens, and units +// that are unknown or contextual. +const edgeTermArb = fc.constantFrom( + '-0', + '-0px', + '0px', + '-1 * 0px', + 'infinity', + '-infinity', + 'NaN', + 'infinity * 1px', + '1.23456789px', + '.5em', + '1e21px', + '1e-7', + '2\\70 x', + '3foo', + '10%', + 'pi', + 'e', + 'var(--a, foo bar)', + 'var(--a, /* c */ 1px)', + 'var(--a,)', + 'var(--a, [x] {y})', + 'v\\61 r(--a)', + 'env(safe-area-inset-top, 2px)', + 'attr(data-x px)' +); +const edgeOperatorArb = fc.constantFrom(' + ', ' - ', ' * ', ' / '); +const edgeExpressionArb = fc.oneof( + edgeTermArb, + fc + .tuple(edgeTermArb, edgeOperatorArb, edgeTermArb) + .map(([a, op, b]) => `${a}${op}${b}`), + fc + .tuple( + edgeTermArb, + edgeOperatorArb, + edgeTermArb, + edgeOperatorArb, + edgeTermArb + ) + .map(([a, op1, b, op2, c]) => `(${a}${op1}${b})${op2}${c}`) +); +const edgeFunctionArb = fc.oneof( + fc + .tuple( + fc.constantFrom('calc', 'CALC', '-webkit-calc', '-moz-calc', 'c\\61 lc'), + edgeExpressionArb + ) + .map(([name, inner]) => `${name}(${inner})`), + fc + .tuple( + fc.constantFrom('min', 'MAX', 'hypot'), + edgeExpressionArb, + edgeExpressionArb + ) + .map(([name, a, b]) => `${name}(${a}, ${b})`), + fc + .tuple(edgeExpressionArb, edgeExpressionArb, edgeExpressionArb) + .map(([a, b, c]) => `clamp(${a}, ${b}, ${c})`), + edgeExpressionArb.map((inner) => `sin(${inner})`) +); +const contextArb = fc.constantFrom( + '$', + 'margin: $', + '$ $', + 'a $ b', + 'translate($, $)', + 'url(calc(1px)) $', + '"calc(1px + 2px)" $', + '$ !important' +); +// One or more functions inside surrounding text, such as a declaration value. +const edgeSourceArb = fc + .tuple(contextArb, fc.array(edgeFunctionArb, { minLength: 1, maxLength: 2 })) + .map(([context, functions]) => { + let index = 0; + return context.replaceAll('$', () => functions[index++ % functions.length]); + }); + +/** Type, raw text and parsed value of each token, without source offsets or EOF. */ +function tokenShapes(tokens) { + return tokens + .filter((token) => token[0] !== TokenType.EOF) + .map((token) => [token[0], token[1], token[4]]); +} + +/** Reduce `source` both ways and compare the results. */ +function assertPrintersAgree(source, options) { + const tokens = tokenize({ css: source }); + const edited = applyReplacements(tokens, reduceTokens(tokens, options)); + const output = reduceCalc(source, options); + assert.equal(raw(edited), output); + assert.deepEqual(tokenShapes(edited), tokenShapes(tokenize({ css: output }))); +} + +describe('string and token printers', () => { + test('the edited token list has the tokens of the string reducer output', () => { + fc.assert(fc.property(cssMathSourceArb, optionsArb, assertPrintersAgree), { + numRuns: NUM_RUNS, + }); + }); + + test('they agree on edge terms, escapes, opaque fallbacks and surrounding text', () => { + fc.assert(fc.property(edgeSourceArb, optionsArb, assertPrintersAgree), { + numRuns: NUM_RUNS * 2, + }); + }); +}); diff --git a/test/property/token-edits.test.js b/test/property/token-edits.test.js new file mode 100644 index 0000000..f921157 --- /dev/null +++ b/test/property/token-edits.test.js @@ -0,0 +1,35 @@ +// Property tests for token lists edited after tokenization. An earlier token +// is replaced with one tokenized from new text, so its source offsets are +// stale while its type, raw text and value agree. The edited list must reduce +// to what the string reducer gives for the edited text. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import fc from 'fast-check'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { cssMathSourceArb } from '../helpers/css-math-arbitraries.js'; +import { NUM_RUNS } from '../helpers/laws.js'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const prefixArb = fc.constantFrom('a', 'bb', 'ccc', 'dddd', 'x\\61 y'); + +describe('reduceTokens: edited token lists', () => { + test('replacing an earlier token matches reduceCalc on the edited text', () => { + fc.assert( + fc.property( + prefixArb, + prefixArb, + cssMathSourceArb, + (before, after, source) => { + const tokens = tokenize({ css: `${before} ${source}` }); + tokens[0] = tokenize({ css: after })[0]; + const edited = raw(applyReplacements(tokens, reduceTokens(tokens))); + const expected = reduceCalc(`${after} ${source}`); + assert.equal(edited, expected); + } + ), + { numRuns: NUM_RUNS } + ); + }); +}); diff --git a/test/property/token-ranges.test.js b/test/property/token-ranges.test.js new file mode 100644 index 0000000..91ea9ec --- /dev/null +++ b/test/property/token-ranges.test.js @@ -0,0 +1,47 @@ +// Property tests for `createReducer` over a sub-range of a token list. The +// range `[start, end)` must reduce to what reducing its joined text gives, +// with the surrounding tokens left as they were. This pins the index +// arithmetic of block matching, which a range does not start at 0. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import fc from 'fast-check'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import { createReducer } from 'postcss-calc/reduce-tokens'; +import { cssMathSourceArb } from '../helpers/css-math-arbitraries.js'; +import { NUM_RUNS } from '../helpers/laws.js'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const sideArb = fc.constantFrom('', 'a', 'x(1px)', '(a, [b])', 'c d'); +const reduce = createReducer(); +const target = { warn() {} }; + +/** Tokens of `css` without its end-of-file token. */ +function contentTokens(css) { + return tokenize({ css }).slice(0, -1); +} + +describe('createReducer: sub-range of a token list', () => { + test('reducing [start, end) matches reducing the joined text of that range', () => { + fc.assert( + fc.property( + sideArb, + cssMathSourceArb, + sideArb, + (before, source, after) => { + const head = contentTokens(`${before} `); + const middle = contentTokens(source); + const tail = contentTokens(` ${after}`); + const tokens = [...head, ...middle, ...tail]; + const start = head.length; + const end = start + middle.length; + const replacements = reduce(tokens, target, start, end); + const edited = raw(applyReplacements(tokens, replacements)); + const expected = `${before} ${reduceCalc(source)} ${after}`; + assert.equal(edited, expected); + } + ), + { numRuns: NUM_RUNS } + ); + }); +}); diff --git a/test/unit/blocks.test.js b/test/unit/blocks.test.js new file mode 100644 index 0000000..66bc770 --- /dev/null +++ b/test/unit/blocks.test.js @@ -0,0 +1,55 @@ +// Block matching over a range of a token list: a function the range does not +// close is unclosed, and an empty or inverted range has nothing to reduce. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceTokens from 'postcss-calc/reduce-tokens'; + +const tokens = tokenize({ css: 'a calc(1px + 2px) b' }); +const opener = tokens.findIndex((token) => token[1] === 'calc('); +const closer = tokens.findIndex((token) => token[1] === ')'); + +describe('reduceTokens: block ranges', () => { + test('a range covering the function replaces it', () => { + const [replacement] = reduceTokens(tokens, {}, opener, closer + 1); + assert.deepEqual( + [replacement.start, replacement.end], + [opener, closer + 1] + ); + }); + + test('a range ending before the closer leaves the function unclosed and unreduced', () => { + const errors = []; + const replacements = reduceTokens( + tokens, + { onParseError: (error) => errors.push(error.message) }, + opener, + closer + ); + assert.deepEqual(replacements, []); + assert.equal(errors.length, 1); + }); + + test('a range starting after the opener has nothing to reduce', () => { + assert.deepEqual(reduceTokens(tokens, {}, opener + 1, tokens.length), []); + }); +}); + +describe('reduceTokens: empty ranges', () => { + test('an empty or inverted range has no replacements', () => { + assert.deepEqual(reduceTokens(tokens, {}, 3, 3), []); + assert.deepEqual(reduceTokens(tokens, {}, 5, 2), []); + }); + + test('a range outside the list or with a fractional bound throws a RangeError', () => { + for (const [start, end] of [ + [0, tokens.length + 1], + [-1, 3], + [0.5, 3], + [0, Number.NaN], + [tokens.length + 1, tokens.length + 2], + ]) { + assert.throws(() => reduceTokens(tokens, {}, start, end), RangeError); + } + }); +}); diff --git a/test/unit/create-reducer.test.js b/test/unit/create-reducer.test.js new file mode 100644 index 0000000..62d6444 --- /dev/null +++ b/test/unit/create-reducer.test.js @@ -0,0 +1,92 @@ +// A reducer from `createReducer` sends its warnings and parse errors to the +// target it is called with, or to `onParseError` when that option is set. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import { createReducer } from 'postcss-calc/reduce-tokens'; + +/** A target that records each `warn(message, plugin)` call. */ +function recordingTarget() { + const calls = []; + return { + calls, + warn(message, plugin) { + calls.push([message, plugin]); + }, + }; +} + +describe('createReducer warnings', () => { + test('an unresolved expression is warned through the target as postcss-calc', () => { + const target = recordingTarget(); + const reduce = createReducer({ warnWhenCannotResolve: true }); + reduce(tokenize({ css: 'calc(1px + var(--a))' }), target); + assert.deepEqual(target.calls, [ + ['Could not reduce expression: calc(1px + var(--a))', 'postcss-calc'], + ]); + }); +}); + +describe('createReducer onWarn', () => { + test('an onWarn option takes precedence over the target', () => { + const target = recordingTarget(); + const messages = []; + const reduce = createReducer({ + warnWhenCannotResolve: true, + onWarn: (message) => messages.push(message), + }); + reduce(tokenize({ css: 'calc(1px + var(--a))' }), target); + assert.deepEqual(messages, [ + 'Could not reduce expression: calc(1px + var(--a))', + ]); + assert.deepEqual(target.calls, []); + }); +}); + +describe('createReducer parse errors', () => { + test('a parse error goes to onParseError and not to the target when it is set', () => { + const target = recordingTarget(); + const errors = []; + const reduce = createReducer({ + onParseError: (error) => errors.push(error.message), + }); + reduce(tokenize({ css: 'calc(1px +)' }), target); + assert.deepEqual(errors, [ + '"+" must be surrounded by whitespace at position 9', + ]); + assert.deepEqual(target.calls, []); + }); + + test('a parse error goes to the target as postcss-calc when onParseError is unset', () => { + const target = recordingTarget(); + const reduce = createReducer(); + reduce(tokenize({ css: 'calc(1px +)' }), target); + assert.deepEqual(target.calls, [ + ['"+" must be surrounded by whitespace at position 9', 'postcss-calc'], + ]); + }); +}); + +describe('createReducer range', () => { + test('a reducer reduces only the tokens of the range it is given', () => { + const target = recordingTarget(); + const reduce = createReducer(); + const tokens = tokenize({ css: 'calc(1px + 1px) calc(2px + 2px)' }); + const replacements = reduce(tokens, target, 0, 7); + assert.deepEqual( + replacements.map((replacement) => [replacement.start, replacement.end]), + [[0, 7]] + ); + }); + + test('a reducer skips the tokens before its range start', () => { + const target = recordingTarget(); + const reduce = createReducer(); + const tokens = tokenize({ css: 'calc(1px + 1px) calc(2px + 2px)' }); + const replacements = reduce(tokens, target, 8); + assert.deepEqual( + replacements.map((replacement) => [replacement.start, replacement.end]), + [[8, 15]] + ); + }); +}); diff --git a/test/unit/reduceTokens-fusion.test.js b/test/unit/reduceTokens-fusion.test.js new file mode 100644 index 0000000..cc8d084 --- /dev/null +++ b/test/unit/reduceTokens-fusion.test.js @@ -0,0 +1,99 @@ +// A replacement must not merge with the token beside it or with a printed +// neighbour: the edited list has to read back as the tokens it holds. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const unwrap = { unwrapSingleValue: true }; + +const retokenized = (tokens) => + tokenize({ css: raw(tokens) }).map((token) => [token[0], token[1]]); + +const cases = [ + [ + 'an escape after a printed dimension', + 'calc(var(--x, calc(1px)\\62) + 1px)', + ], + ['a percent sign after a printed number', 'calc(var(--x, calc(1)%) + 1px)'], + ['an identifier after the replaced function', 'foo(calc(1px + 1px)b)'], + ['an identifier after a negative zero', 'foo(calc(-0)x)'], + ['a percent sign after the replaced function', 'foo(calc(1px + 1px)%)'], + ['an escape after the replaced function', 'foo(calc(1px + 1px)\\62)'], +]; + +describe('reduceTokens: the edited list reads back as its own tokens', () => { + for (const [name, value] of cases) { + test(name, () => { + const tokens = tokenize({ css: value }); + const edited = applyReplacements(tokens, reduceTokens(tokens, unwrap)); + assert.deepEqual( + edited.filter((t) => t[0] !== 'EOF-token').map((t) => [t[0], t[1]]), + retokenized(edited).filter((t) => t[0] !== 'EOF-token') + ); + }); + } +}); + +describe('reduceCalc: unwrapped results keep their neighbours apart', () => { + test('the number before an escape is not read as a unit', () => { + assert.equal( + reduceCalc('calc(var(--x, calc(1px)\\62) + 1px)', unwrap), + 'calc(1px + var(--x, 1px \\62))' + ); + }); + + test('a number before a percent sign is not read as a percentage', () => { + assert.equal( + reduceCalc('calc(var(--x, calc(1)%) + 1px)', unwrap), + 'calc(1px + var(--x, 1 %))' + ); + }); + + test('a dimension before an identifier is not read as a longer unit', () => { + assert.equal(reduceCalc('foo(calc(1px + 1px)b)', unwrap), 'foo(2px b)'); + }); + + test('a zero before an identifier is not read as a dimension', () => { + assert.equal(reduceCalc('foo(calc(-0)x)', unwrap), 'foo(0 x)'); + }); + + test('a replacement after an identifier keeps its own spacing', () => { + assert.equal(reduceCalc('foo(a calc(1px + 1px))', unwrap), 'foo(a 2px)'); + }); +}); + +describe('reduceTokens: separators at the replacement boundary', () => { + test('a space follows the replacement when the next token would fuse', () => { + const tokens = tokenize({ css: 'foo(calc(1px + 1px)b)' }); + const [replacement] = reduceTokens(tokens, unwrap); + assert.deepEqual( + replacement.tokens.map((token) => token[1]), + ['2px', ' '] + ); + }); + + test('a space precedes the replacement when the previous token would fuse', () => { + // A caller can join tokens that were never adjacent in the source text. + const tokens = [ + ...tokenize({ css: 'a' }).slice(0, 1), + ...tokenize({ css: 'calc(1px + 1px)' }), + ]; + const [replacement] = reduceTokens(tokens, unwrap); + assert.deepEqual( + replacement.tokens.map((token) => token[1]), + [' ', '2px'] + ); + }); + + test('no space is added when the neighbours cannot fuse', () => { + const tokens = tokenize({ css: 'foo(calc(1px + 1px), a)' }); + const [replacement] = reduceTokens(tokens, unwrap); + assert.deepEqual( + replacement.tokens.map((token) => token[1]), + ['2px'] + ); + }); +}); diff --git a/test/unit/reduceTokens.test.js b/test/unit/reduceTokens.test.js new file mode 100644 index 0000000..cf4d2d1 --- /dev/null +++ b/test/unit/reduceTokens.test.js @@ -0,0 +1,358 @@ +// Token-list reducer: same results as the string reducer, and no dependence +// on the source offsets stored in the tokens. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { tokenize, TokenType } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens, { + hasPotentialMathFunction, +} from 'postcss-calc/reduce-tokens'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const fullCorpus = process.env.POSTCSS_CALC_FULL_CORPUS === '1'; + +const values = [ + 'calc(1px + 2px)', + 'calc(1px + 2px) calc(3 * 4)', + 'margin calc(100% - 10px) calc(2 * var(--a))', + 'min(1px, 2px) max(3px, 4em)', + 'calc(1px + (2 * 3px))', + 'CALC(1PX + 2PX)', + '-webkit-calc(1px + 2px)', + 'calc(var(--a) + calc(1px + 2px))', + 'calc(1px +', + 'calc(1px + 2)', + 'calc(1 / 3)', + 'translate(calc(1px + 2px), calc(2 * 50%))', + 'url(calc(1+2)) calc(1px + 1px)', + 'calc(1px + 2px)calc(3px + 4px)', + 'calc(1px+2px)', + 'nothing to see', +]; + +describe('reduceTokens: equals reduceCalc', () => { + for (const value of values) { + test(value, () => { + const tokens = tokenize({ css: value }); + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + reduceCalc(value) + ); + }); + } + + test('options', () => { + const value = 'calc(1.23456px + 1px) calc(1 - 2) calc(1 / 3)'; + const tokens = tokenize({ css: value }); + for (const options of [ + { precision: 2 }, + { precision: false }, + { unwrapSingleValue: true }, + { unwrapSingleNegativeNumber: true }, + ]) { + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens, options))), + reduceCalc(value, options) + ); + } + }); + + test('corpus', () => { + const lines = readFileSync( + new URL('../corpus/github-pure.txt', import.meta.url), + 'utf8' + ) + .split('\n') + .filter((line) => line && !line.startsWith('#')) + .slice(0, fullCorpus ? undefined : 2000); + for (const line of lines) { + const tokens = tokenize({ css: line }); + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + reduceCalc(line) + ); + } + }); +}); + +describe('reduceTokens: replacements', () => { + test('are ordered token ranges that change the text', () => { + const tokens = tokenize({ css: 'calc(1px + 2px) 3px calc(2px)' }); + const replacements = reduceTokens(tokens); + assert.equal(replacements.length, 1); + assert.equal(replacements[0].start, 0); + assert.equal(raw(tokens.slice(0, replacements[0].end)), 'calc(1px + 2px)'); + assert.equal(raw(replacements[0].tokens), 'calc(3px)'); + }); + + test('replacement tokens are what tokenizing their text gives', () => { + const [replacement] = reduceTokens(tokenize({ css: 'calc(1px + 2px)' })); + assert.deepEqual( + replacement.tokens.map((token) => token.slice(0, 2)), + tokenize({ css: 'calc(3px)' }) + .slice(0, -1) + .map((token) => token.slice(0, 2)) + ); + }); + + test('unchanged input gives no replacements', () => { + assert.deepEqual(reduceTokens(tokenize({ css: 'calc(2px)' })), []); + assert.deepEqual(reduceTokens(tokenize({ css: '1px solid red' })), []); + }); + + test('values without a math function skip the nesting check', () => { + const errors = []; + const replacements = reduceTokens(tokenize({ css: '('.repeat(5000) }), { + onParseError: (error) => errors.push(error), + }); + assert.deepEqual(replacements, []); + assert.deepEqual(errors, []); + }); + + test('a calculation past the nesting limit still reports', () => { + const errors = []; + const css = `calc(${'('.repeat(2000)}1px${')'.repeat(2000)})`; + reduceTokens(tokenize({ css }), { + onParseError: (error) => errors.push(error.message), + }); + assert.deepEqual(errors, ['Calculation nesting exceeds the limit of 1024']); + }); + + test('reduceCalc skips the nesting check without a math function token', () => { + const errors = []; + const value = 'url(calc(1+2))' + '('.repeat(3000); + assert.equal( + reduceCalc(value, { onParseError: (error) => errors.push(error) }), + value + ); + assert.deepEqual(errors, []); + }); + + test('does not modify the token list', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + const before = JSON.stringify(tokens); + reduceTokens(tokens); + assert.equal(JSON.stringify(tokens), before); + }); +}); + +describe('reduceTokens: edited token lists', () => { + test('ignores stale source offsets after an earlier edit', () => { + const tokens = tokenize({ + css: 'aaaaaaaaaa calc(1px + 2px) bb calc(3px * 2)', + }); + // Shorten the leading ident without touching offsets, as a reducer would. + tokens[0] = tokenize({ css: 'a' })[0]; + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + 'a calc(3px) bb calc(6px)' + ); + }); + + test('ignores stale offsets inside the expression', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + const whitespace = tokens.filter((token) => token[1].startsWith(' ')); + for (const token of whitespace) token[1] = ' '; + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + 'calc(3px)' + ); + }); + + test('inserted tokens take part', () => { + const tokens = tokenize({ css: 'calc(1px + X)' }); + const index = tokens.findIndex((token) => token[1] === 'X'); + tokens.splice(index, 1, ...tokenize({ css: '2px' }).slice(0, -1)); + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + 'calc(3px)' + ); + }); + + test('reports positions in the current text', () => { + const errors = []; + const tokens = tokenize({ css: 'aaaaaaaaaa calc(1px + )' }); + tokens[0] = tokenize({ css: 'a' })[0]; + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.equal(errors.length, 1); + assert.equal(errors[0][1], 'calc(1px + )'); + assert.equal(errors[0][0], 'Unexpected token "" at position 13'); + }); + + test('unclosed calculations report positions in the current text', () => { + const errors = []; + const tokens = tokenize({ css: 'aaaaaaaaaa calc(1px +' }); + tokens[0] = tokenize({ css: 'a' })[0]; + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [['Unclosed calc( at position 2', 'calc(1px +']]); + }); + + test('invalid custom properties report positions in the current text', () => { + const errors = []; + const tokens = tokenize({ css: 'aaaaaaaaaa calc(1px + var(1px))' }); + tokens[0] = tokenize({ css: 'a' })[0]; + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [ + [ + 'Invalid custom property in var() at position 17', + 'calc(1px + var(1px))', + ], + ]); + }); + + test('token lists without a trailing EOF token reduce the same', () => { + for (const value of values) { + const tokens = tokenize({ css: value }); + if (tokens.at(-1)[0] === TokenType.EOF) tokens.pop(); + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + reduceCalc(value) + ); + } + }); + + test('unclosed calculations without a trailing EOF token report', () => { + const errors = []; + const tokens = tokenize({ css: 'calc(1px +' }); + tokens.pop(); + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [['Unclosed calc( at position 0', 'calc(1px +']]); + }); + + test('warnings carry the current text', () => { + const warnings = []; + const tokens = tokenize({ css: 'xxxx calc(100% - var(--a))' }); + tokens[0] = tokenize({ css: 'x' })[0]; + reduceTokens(tokens, { + warnWhenCannotResolve: true, + onWarn: (message) => warnings.push(message), + }); + assert.deepEqual(warnings, [ + 'Could not reduce expression: x calc(100% - var(--a))', + ]); + }); + + test('all parse errors report positions in the current text', () => { + const cases = [ + ['calc(1px+ 2px)', '"+" must be surrounded by whitespace at position 6'], + ['calc(1px 2px)', 'Unexpected token "2px" at position 9'], + ['calc((1px + 2px)', 'Unclosed calc( at position 0'], + ['calc(1px + #)', 'Unexpected character "#" at position 11'], + ]; + for (const [css, message] of cases) { + const errors = []; + const tokens = tokenize({ css: `aaaaaaaaaa ${css}` }); + tokens[0] = tokenize({ css: 'a' })[0]; + reduceTokens(tokens, { + onParseError: (error) => errors.push(error.message), + }); + assert.equal(errors.length, 1, css); + assert.ok( + errors[0].startsWith(message.split(' at position')[0]), + `${css}: ${errors[0]}` + ); + } + }); + + test('end-of-file tokens inside the list are skipped', () => { + const tokens = [ + ...tokenize({ css: 'calc(1px +' }), + ...tokenize({ css: ' 2px)' }), + ]; + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + 'calc(3px)' + ); + }); + + test('lists joined from whole-token pieces match reduceCalc', () => { + const pieces = [ + ['a ', 'calc(1px', ' + ', '2px)', ' b'], + ['calc(', '1px * ', '2', ')calc(3px+', ' 1px)'], + ['min(1px, ', '2px) ', 'max(3px,', ' var(--a))'], + ]; + for (const parts of pieces) { + const tokens = parts.flatMap((part) => tokenize({ css: part })); + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + reduceCalc(raw(tokens)), + raw(tokens) + ); + } + }); +}); + +describe('reduceTokens: concatenated token lists', () => { + test('an end-of-file token inside var() is skipped', () => { + const tokens = [ + ...tokenize({ css: 'calc(1px + var(' }), + ...tokenize({ css: '--a))' }), + ]; + assert.equal( + raw(applyReplacements(tokens, reduceTokens(tokens))), + reduceCalc('calc(1px + var(--a))') + ); + }); +}); + +describe('hasPotentialMathFunction', () => { + test('is true for values reduceTokens can change', () => { + for (const value of values) { + const tokens = tokenize({ css: value }); + if (reduceTokens(tokens).length > 0) { + assert.equal(hasPotentialMathFunction(value), true, value); + } + } + }); + + test('is false for values without a math function', () => { + assert.equal(hasPotentialMathFunction('1px solid red'), false); + assert.equal(hasPotentialMathFunction('rgb(255, 0, 0)'), false); + }); +}); + +describe('reduceTokens: a range of the list', () => { + test('reports replacements at indices of the whole list', () => { + const tokens = tokenize({ css: 'calc(1px + 2px) calc(3px + 4px)' }); + const second = tokens.findLastIndex( + (token) => token[0] === TokenType.Function + ); + const replacements = reduceTokens(tokens, {}, second, tokens.length); + assert.equal(replacements.length, 1); + assert.equal(replacements[0].start, second); + assert.equal(replacements[0].end, tokens.length - 1); + assert.equal( + raw(applyReplacements(tokens, replacements)), + 'calc(1px + 2px) calc(7px)' + ); + }); + + test('ignores the tokens outside its range', () => { + const tokens = tokenize({ css: 'calc(1px + 2px) calc(3px + 4px)' }); + assert.deepEqual( + reduceTokens(tokens, {}, 0, 2), + reduceTokens(tokens.slice(0, 2)) + ); + }); + + test('reports the same errors, at the same positions, as the slice it replaces', () => { + const tokens = tokenize({ css: 'x calc(1px + [2px) y' }); + const errors = []; + const sliceErrors = []; + const onParseError = (error, input) => errors.push([error.message, input]); + reduceTokens(tokens, { onParseError }, 2, tokens.length); + reduceTokens(tokens.slice(2), { + onParseError: (error, input) => sliceErrors.push([error.message, input]), + }); + assert.deepEqual(errors, sliceErrors); + }); +}); diff --git a/test/unit/token-core-closers.test.js b/test/unit/token-core-closers.test.js new file mode 100644 index 0000000..fcbdca5 --- /dev/null +++ b/test/unit/token-core-closers.test.js @@ -0,0 +1,42 @@ +// A closer that does not match the innermost open block is ignored. These +// cases check that rule through the public string and token APIs. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; + +describe('mismatched closers through reduceCalc', () => { + test('a closer that does not match the open block leaves the calc() unclosed', () => { + const errors = []; + const output = reduceCalc('calc(1px + [2px)', { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.equal(output, 'calc(1px + [2px)'); + assert.deepEqual(errors, [ + ['Unclosed calc( at position 0', 'calc(1px + [2px)'], + ]); + }); + + test('a mismatched closer inside a fallback is ignored and the calc() reduces', () => { + const errors = []; + const output = reduceCalc('var(--a, calc(1px ) ] )', { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.equal(output, 'var(--a, calc(1px) ] )'); + assert.deepEqual(errors, []); + }); +}); + +describe('mismatched closers through reduceTokens', () => { + test('a closer that does not match leaves the list unchanged and reports the calc()', () => { + const errors = []; + const replacements = reduceTokens(tokenize({ css: 'calc(1px + [2px) x' }), { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(replacements, []); + assert.deepEqual(errors, [ + ['Unclosed calc( at position 0', 'calc(1px + [2px) x'], + ]); + }); +}); diff --git a/test/unit/token-core-contract.test.js b/test/unit/token-core-contract.test.js new file mode 100644 index 0000000..cd35223 --- /dev/null +++ b/test/unit/token-core-contract.test.js @@ -0,0 +1,391 @@ +// Behaviour the token-core refactor must keep: whitespace around operators, +// range edges, end-of-file tokens, unclosed blocks, nesting limits, negated +// opaque groups and diagnostic positions. Every case goes through the public +// string and token APIs. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +/** Run `reduceCalc` and collect each parse error as `[message, input]`. */ +function reduceWithErrors(value, options = {}) { + const errors = []; + const output = reduceCalc(value, { + ...options, + onParseError: (error, input) => errors.push([error.message, input]), + }); + return { output, errors }; +} + +/** Run `reduceTokens` on a list and return the edited text and errors. */ +function reduceListWithErrors(tokens, options = {}) { + const errors = []; + const replacements = reduceTokens(tokens, { + ...options, + onParseError: (error, input) => errors.push([error.message, input]), + }); + return { output: raw(applyReplacements(tokens, replacements)), errors }; +} + +/** + * Tokenize `css` after a leading ident that is shortened to one character, so + * the later tokens keep stale source offsets. + */ +function staleList(prefix, css) { + const tokens = tokenize({ css: `${prefix} ${css}` }); + tokens[0] = tokenize({ css: 'a' })[0]; + return tokens; +} + +describe('comments and whitespace around operators', () => { + test('a comment counts as whitespace before and after an operator', () => { + assert.deepEqual(reduceWithErrors('calc(1px/**/+ 2px)'), { + output: 'calc(3px)', + errors: [], + }); + assert.deepEqual(reduceWithErrors('calc(1px +/**/2px)'), { + output: 'calc(3px)', + errors: [], + }); + assert.deepEqual(reduceWithErrors('calc(1px/**/+/**/2px)'), { + output: 'calc(3px)', + errors: [], + }); + }); +}); + +describe('operators at range edges', () => { + test('a trailing operator without following whitespace is rejected', () => { + assert.deepEqual(reduceWithErrors('calc(1px +)'), { + output: 'calc(1px +)', + errors: [ + ['"+" must be surrounded by whitespace at position 9', 'calc(1px +)'], + ], + }); + }); + + test('a trailing operator with whitespace before the end is a missing operand', () => { + assert.deepEqual(reduceWithErrors('calc(1px + )'), { + output: 'calc(1px + )', + errors: [['Unexpected token "" at position 11', 'calc(1px + )']], + }); + }); + + test('an operator before a comma needs whitespace after it', () => { + assert.deepEqual(reduceWithErrors('min(1px +, 2px)'), { + output: 'min(1px +, 2px)', + errors: [ + [ + '"+" must be surrounded by whitespace at position 8', + 'min(1px +, 2px)', + ], + ], + }); + }); + + test('an operator at the end of a parenthesized group', () => { + assert.deepEqual(reduceWithErrors('calc((1px +) )'), { + output: 'calc((1px +) )', + errors: [ + [ + '"+" must be surrounded by whitespace at position 10', + 'calc((1px +) )', + ], + ], + }); + }); + + test('a leading operator is not a unary sign', () => { + assert.deepEqual(reduceWithErrors('calc(+ 1px)'), { + output: 'calc(+ 1px)', + errors: [['Unexpected token "+" at position 5', 'calc(+ 1px)']], + }); + }); + + test('a signed number directly after an operand is rejected', () => { + assert.deepEqual(reduceWithErrors('calc(1px+ 2px)'), { + output: 'calc(1px+ 2px)', + errors: [ + [ + '"+" must be surrounded by whitespace at position 8', + 'calc(1px+ 2px)', + ], + ], + }); + assert.deepEqual(reduceWithErrors('calc(1px -2px)'), { + output: 'calc(1px -2px)', + errors: [ + [ + '"-" must be surrounded by whitespace at position 9', + 'calc(1px -2px)', + ], + ], + }); + }); + + test('a double negative after an operator is a valid sign', () => { + assert.deepEqual(reduceWithErrors('calc(1px - -2px)'), { + output: 'calc(3px)', + errors: [], + }); + }); +}); + +describe('end-of-file tokens in a token list', () => { + test('an end-of-file token directly after an operand breaks the whitespace rule', () => { + const tokens = [ + ...tokenize({ css: 'calc(1px' }), + ...tokenize({ css: '' }), + ...tokenize({ css: '+ 2px)' }), + ]; + assert.deepEqual(reduceListWithErrors(tokens), { + output: 'calc(1px+ 2px)', + errors: [ + [ + '"+" must be surrounded by whitespace at position 8', + 'calc(1px+ 2px)', + ], + ], + }); + }); + + test('an end-of-file token between whitespace and an operator is skipped', () => { + const tokens = [ + ...tokenize({ css: 'calc(1px ' }), + ...tokenize({ css: '' }), + ...tokenize({ css: '+ 2px)' }), + ]; + assert.deepEqual(reduceListWithErrors(tokens), { + output: 'calc(3px)', + errors: [], + }); + }); + + test('an end-of-file token inside a var() fallback is skipped', () => { + const tokens = [ + ...tokenize({ css: 'calc(1px + var(--a,' }), + ...tokenize({ css: '' }), + ...tokenize({ css: ' 2px))' }), + ]; + assert.deepEqual(reduceListWithErrors(tokens), { + output: 'calc(1px + var(--a, 2px))', + errors: [], + }); + }); +}); + +describe('unclosed functions', () => { + test('a top-level unclosed calc() reports its position', () => { + assert.deepEqual(reduceWithErrors('calc(1px + 2px'), { + output: 'calc(1px + 2px', + errors: [['Unclosed calc( at position 0', 'calc(1px + 2px']], + }); + }); + + test('an unclosed calc() nested in a var() fallback reports the enclosing range', () => { + assert.deepEqual(reduceWithErrors('calc(1px + var(--a, calc(2px'), { + output: 'calc(1px + var(--a, calc(2px', + errors: [ + ['Unclosed calc( at position 0', 'calc(1px + var(--a, calc(2px'], + ], + }); + }); + + test('an unclosed calc() after a reduced one reports its own position', () => { + assert.deepEqual(reduceWithErrors('calc(1px + 1px) calc('), { + output: 'calc(2px) calc(', + errors: [['Unclosed calc( at position 16', 'calc(']], + }); + }); + + test('an unclosed var() is not a math function and stays silent', () => { + assert.deepEqual(reduceWithErrors('var(--a, 1px'), { + output: 'var(--a, 1px', + errors: [], + }); + }); + + test('an unclosed calc() in a token list reports its position in the list', () => { + const errors = []; + const tokens = staleList('aaaaaaaaaa', 'calc(1px'); + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [['Unclosed calc( at position 2', 'calc(1px']]); + }); +}); + +const LIMIT = 1024; +const nestedParens = (n) => `calc(${'('.repeat(n)}1px${')'.repeat(n)})`; +const nestedFunctions = (n) => + `calc(${'unknown('.repeat(n)}1px${')'.repeat(n)})`; +const nestedSquares = (n) => `calc(var(--a, ${'['.repeat(n)}${']'.repeat(n)}))`; +const nestedCurlies = (n) => `calc(var(--a, ${'{'.repeat(n)}${'}'.repeat(n)}))`; +const nestedVarFunctions = (n) => + `calc(var(--a, ${'unknown('.repeat(n)}${')'.repeat(n)}))`; +const LIMIT_MESSAGE = `Calculation nesting exceeds the limit of ${LIMIT}`; + +describe('nesting limit', () => { + test('parentheses at the limit are accepted and past it are rejected', () => { + // `calc(` plus 1023 parentheses is 1024 openers in total. + assert.equal(reduceCalc(nestedParens(LIMIT - 1)), 'calc(1px)'); + const { output, errors } = reduceWithErrors(nestedParens(LIMIT)); + assert.equal(output, nestedParens(LIMIT)); + assert.deepEqual(errors, [[LIMIT_MESSAGE, nestedParens(LIMIT)]]); + }); + + test('functions at the limit are accepted and past it are rejected', () => { + assert.deepEqual(reduceWithErrors(nestedFunctions(LIMIT - 1)).errors, []); + const { output, errors } = reduceWithErrors(nestedFunctions(LIMIT)); + assert.equal(output, nestedFunctions(LIMIT)); + assert.deepEqual(errors, [[LIMIT_MESSAGE, nestedFunctions(LIMIT)]]); + }); + + test('functions inside a var() fallback count toward the limit', () => { + // `calc(` and `var(` plus 1022 functions is 1024 openers in total. + assert.deepEqual( + reduceWithErrors(nestedVarFunctions(LIMIT - 2)).errors, + [] + ); + const { errors } = reduceWithErrors(nestedVarFunctions(LIMIT - 1)); + assert.deepEqual(errors, [[LIMIT_MESSAGE, nestedVarFunctions(LIMIT - 1)]]); + }); + + test('square brackets inside a var() fallback count toward the limit', () => { + assert.deepEqual(reduceWithErrors(nestedSquares(LIMIT - 2)).errors, []); + const { errors } = reduceWithErrors(nestedSquares(LIMIT - 1)); + assert.deepEqual(errors, [[LIMIT_MESSAGE, nestedSquares(LIMIT - 1)]]); + }); + + test('curly braces inside a var() fallback count toward the limit', () => { + assert.deepEqual(reduceWithErrors(nestedCurlies(LIMIT - 2)).errors, []); + const { errors } = reduceWithErrors(nestedCurlies(LIMIT - 1)); + assert.deepEqual(errors, [[LIMIT_MESSAGE, nestedCurlies(LIMIT - 1)]]); + }); + + test('parentheses inside a url() do not count', () => { + const value = `calc(1px + 2px) url(${'('.repeat(3000)})`; + assert.deepEqual(reduceWithErrors(value), { + output: `calc(3px) url(${'('.repeat(3000)})`, + errors: [], + }); + }); + + test('a bad url() does not count either', () => { + const value = `calc(1px + 2px) url(a b${'('.repeat(3000)})`; + assert.deepEqual(reduceWithErrors(value).errors, []); + }); + + test('a list with a valid calc() and over-limit nesting outside it reports once and changes nothing', () => { + const value = `calc(1px + 1px) x${'('.repeat(1100)}1px${')'.repeat(1100)}`; + const { output, errors } = reduceWithErrors(value); + assert.equal(output, value); + assert.deepEqual(errors, [[LIMIT_MESSAGE, value]]); + }); + + test('the limit error on a token list reports the whole list text', () => { + const css = `calc(1px${'('.repeat(1100)}1px${')'.repeat(1100)})`; + const tokens = staleList('aaaaaaaaaa', css); + const errors = []; + reduceTokens(tokens, { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [[LIMIT_MESSAGE, `a ${css}`]]); + }); +}); + +describe('mismatched closers', () => { + // A closer that does not match the innermost open block is ignored, so an + // enclosing block can still be closed by a later closer. + test('a mismatched closer inside a var() fallback is ignored', () => { + assert.deepEqual(reduceWithErrors('calc(1px + var(--a, [b) 2px] ))'), { + output: 'calc(1px + var(--a, [b) 2px] ))', + errors: [], + }); + }); + + test('a mismatched closer that leaves the calc() open reports it unclosed', () => { + assert.deepEqual(reduceWithErrors('calc(1px + var(--a, [b)) 2px)'), { + output: 'calc(1px + var(--a, [b)) 2px)', + errors: [ + ['Unclosed calc( at position 0', 'calc(1px + var(--a, [b)) 2px)'], + ], + }); + }); + + test('an unclosed block does not stop a later calc() from reducing', () => { + assert.equal( + reduceCalc('var(--a, [b) calc(1px + 2px)'), + 'var(--a, [b) calc(3px)' + ); + }); +}); + +describe('negated grouped sums with opaque terms', () => { + test('a negated group with an opaque term keeps its grouping', () => { + assert.deepEqual(reduceWithErrors('calc(-1 * (1px + var(--a)))'), { + output: 'calc(-1 * (1px + var(--a)))', + errors: [], + }); + }); + + test('a negated group keeps its opaque terms grouped after reduction', () => { + assert.equal( + reduceCalc('calc(-1 * (2px + 1px + var(--a)))'), + 'calc(-1 * (3px + var(--a)))' + ); + assert.equal( + reduceCalc('calc(-1 * (var(--a) + 2px))'), + 'calc(-1 * (2px + var(--a)))' + ); + }); +}); + +describe('diagnostic positions on edited token lists', () => { + // Each case shortens the leading ident after tokenizing, so later tokens + // keep offsets from the longer text. Positions must count the current text, + // which here is `a ${css}`. + const cases = [ + { + name: 'whitespace after an operator', + css: 'calc(1px+ 2px)', + message: '"+" must be surrounded by whitespace at position 10', + }, + { + name: 'unexpected token', + css: 'calc(1px 2px)', + message: 'Unexpected token "2px" at position 11', + }, + { + name: 'unexpected character', + css: 'calc(1px + #)', + message: 'Unexpected character "#" at position 13', + }, + { + name: 'missing closing parenthesis', + css: 'min(1px 2px)', + message: 'Expected ) at position 10, got "2"', + }, + { + name: 'unclosed calc()', + css: 'calc((1px + 2px)', + message: 'Unclosed calc( at position 2', + }, + { + name: 'invalid custom property', + css: 'calc(1px + var(1px))', + message: 'Invalid custom property in var() at position 17', + }, + ]; + for (const { name, css, message } of cases) { + test(name, () => { + const errors = []; + reduceTokens(staleList('aaaaaaaaaa', css), { + onParseError: (error, input) => errors.push([error.message, input]), + }); + assert.deepEqual(errors, [[message, css]]); + }); + } +}); diff --git a/test/unit/token-printer.test.js b/test/unit/token-printer.test.js new file mode 100644 index 0000000..40e4243 --- /dev/null +++ b/test/unit/token-printer.test.js @@ -0,0 +1,155 @@ +// The token printer's output: each replacement from `reduceTokens` holds the +// tokens of its reduced text, for expressions that exercise every emitter. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { TokenType, tokenize } from '@csstools/css-tokenizer'; +import reduceCalc from 'postcss-calc/reduce'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { decodeIdentifier } from '../../src/lib/serialize/sink.js'; + +// Expressions whose printed output exercises each emitter: signed numbers, +// negative zero, degenerate keywords, opaque groups, wrappers and escapes. +const printerCases = [ + 'calc(1px + 2px)', + 'calc(2px - 3px)', + 'calc(-1px + 3px)', + 'calc(1px - 2px - 3px)', + 'calc(1px * -2)', + 'calc(10px / 3)', + 'calc(1.23456px * 1)', + 'calc(1e21px + 1px)', + 'calc(-1 * 0px)', + 'calc(-0px)', + 'calc(1px + -0px)', + 'calc(var(--a) + -0px)', + 'calc(-1 * (var(--a) + 1px))', + 'calc(infinity * 1px)', + 'calc(-infinity * 1px)', + 'calc(-infinity)', + 'calc(NaN + 1px)', + 'calc(-1 * infinity)', + 'min(1px, 2px)', + 'max(-1px, -2px)', + 'clamp(1px, var(--a), 3px)', + 'calc(var(--a) + 2px)', + 'calc(1px + var(--a, calc(2px + 3px)))', + 'calc(1px + 2\\70 x)', + 'CALC(1px + 2px)', + '-webkit-calc(1px + 2px)', +]; +const printerOptions = [{}, { unwrapSingleValue: true }, { precision: false }]; + +/** Type, raw text and parsed value of each token, without source offsets or EOF. */ +function tokenShapes(tokens) { + return tokens + .filter((token) => token[0] !== TokenType.EOF) + .map((token) => [token[0], token[1], token[4]]); +} + +describe('printer tokens', () => { + for (const value of printerCases) { + for (const options of printerOptions) { + const label = `${value} ${JSON.stringify(options)}`; + test(`the replacement tokens of ${label} equal the tokens of its reduced text`, () => { + const output = reduceCalc(value, options); + const replacements = reduceTokens(tokenize({ css: value }), options); + if (output === value) { + assert.deepEqual(replacements, []); + return; + } + assert.equal(replacements.length, 1); + assert.deepEqual( + tokenShapes(replacements[0].tokens), + tokenShapes(tokenize({ css: output })) + ); + }); + } + } +}); + +/** The `[start, end]` offsets of each token, without EOF. */ +const offsets = (tokens) => + tokens + .filter((token) => token[0] !== TokenType.EOF) + .map((token) => [token[2], token[3]]); + +describe('printer token offsets', () => { + for (const value of [ + 'calc(var(--a) + 1px + 1px)', + 'calc(1px + 2px + var(--a, foo bar))', + 'calc(1px + var(--a, calc(2px + 3px)))', + ]) { + test(`the offsets of ${value} locate its tokens in the replacement text`, () => { + const [replacement] = reduceTokens(tokenize({ css: value })); + const text = replacement.tokens.map((token) => token[1]).join(''); + assert.deepEqual( + offsets(replacement.tokens), + offsets(tokenize({ css: text })) + ); + }); + } + + test('a replacement does not change the tokens it was read from', () => { + const tokens = tokenize({ css: 'calc(var(--a) + 1px + 1px)' }); + const before = structuredClone(tokens); + reduceTokens(tokens); + assert.deepEqual(tokens, before); + }); +}); + +describe('printer tokens that would fuse', () => { + // A reduced scalar written next to an adjacent identifier or number must + // stay a separate token, in the text and in the tokens. + const fusing = [ + 'calc(var(--x, calc(1px + 2px)foo) + 1px)', + 'calc(var(--x, calc(1 + 2)foo) + 1px)', + 'calc(var(--x, a calc(1px + 2px)3) + 1px)', + 'calc(var(--x, 1calc(1px + 2px)) + 1px)', + 'calc(var(--x, calc(1px + 2px)calc(3px + 4px)) + 1px)', + ]; + for (const value of fusing) { + test(`${value} keeps its tokens apart`, () => { + const options = { unwrapSingleValue: true }; + const output = reduceCalc(value, options); + const [replacement] = reduceTokens(tokenize({ css: value }), options); + const text = replacement.tokens.map((token) => token[1]).join(''); + assert.equal(text, output); + assert.deepEqual( + tokenShapes(replacement.tokens), + tokenShapes(tokenize({ css: output })) + ); + }); + } + + test('a reduced dimension followed by an identifier reads as two tokens', () => { + assert.equal( + reduceCalc('calc(var(--x, calc(1px + 2px)foo) + 1px)', { + unwrapSingleValue: true, + }), + 'calc(1px + var(--x, 3px foo))' + ); + }); + + test('a wrapped result needs no separator', () => { + assert.equal( + reduceCalc('calc(var(--x, calc(1px + 2px)foo) + 1px)'), + 'calc(1px + var(--x, calc(3px)foo))' + ); + }); +}); + +describe('decodeIdentifier: agrees with the tokenizer', () => { + for (const css of [ + 'a\\', + 'c\\61 lc', + '\\0 x', + '\\d800 x', + '\\110000 ', + 'a\\b', + ]) { + test(JSON.stringify(css), () => { + const [token] = tokenize({ css }); + assert.equal(decodeIdentifier(css), token[4].value); + }); + } +}); diff --git a/tsconfig.json b/tsconfig.json index d229377..4dcad82 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -15,5 +15,10 @@ "forceConsistentCasingInFileNames": true, "strict": true }, - "include": ["src/index.js", "src/reduce.js", "src/lib/**/*.js"] + "include": [ + "src/index.js", + "src/reduce.js", + "src/reduce-tokens.js", + "src/lib/**/*.js" + ] } diff --git a/types/reduce-tokens.d.ts b/types/reduce-tokens.d.ts new file mode 100644 index 0000000..c4dbec4 --- /dev/null +++ b/types/reduce-tokens.d.ts @@ -0,0 +1,48 @@ +import { hasPotentialMathFunction } from './lib/functions.js'; +export type ReduceCalcOptions = import('./reduce.js').ReduceCalcOptions; +export type CSSToken = import('@csstools/css-tokenizer').CSSToken; +export type TokenSink = import('./lib/serialize/sink.js').TokenSink; +export type TokenReplacement = { + /** + * Index of the first replaced token. + */ + start: number; + /** + * Index after the last replaced token. + */ + end: number; + tokens: CSSToken[]; +}; +export type WarnTarget = { + warn: (message: string, plugin?: string) => void; +}; +export type TokenReducer = (tokens: CSSToken[], target: WarnTarget, start?: number, end?: number) => TokenReplacement[]; +/** + * Build a reducer that resolves `opts` once, for a caller that reduces many + * token lists with the same options. Each call sends its warnings to the + * target it is given, unless `opts` has `onWarn`, which takes precedence as + * `onParseError` does. + * + * @param {ReduceCalcOptions} [opts] + * @return {TokenReducer} + */ +declare function createReducer(opts?: ReduceCalcOptions): TokenReducer; +/** + * Simplify every supported CSS math function in `tokens[start, end)`. Returns + * the replacements that change the text, ordered and non-overlapping, and + * leaves `tokens` untouched. Apply them from last to first, or build a new + * list. Replacement indices are indices into `tokens`, and a range reads as + * the list it names would. The offsets of a replacement's tokens locate them + * in the replacement's own text, counting from 0. An empty or inverted + * range has no replacements. Throws a `RangeError` for a range that is not + * within `tokens`. + * + * @param {CSSToken[]} tokens + * @param {ReduceCalcOptions} [opts] + * @param {number} [start] + * @param {number} [end] + * @return {TokenReplacement[]} + */ +declare function reduceTokens(tokens: CSSToken[], opts?: ReduceCalcOptions, start?: number, end?: number): TokenReplacement[]; +export default reduceTokens; +export { createReducer, hasPotentialMathFunction }; From 33c3fbc3b6c2a3590c39796c0a9b71f6ff917b7b Mon Sep 17 00:00:00 2001 From: Ludovico Fischer Date: Fri, 6 Nov 2026 02:54:28 +0100 Subject: [PATCH 2/4] refactor: improve data structures --- CHANGELOG.md | 3 +- README.md | 19 +- src/lib/core.js | 160 ++++++------- src/lib/limits.js | 3 +- src/lib/options.js | 10 +- src/lib/serialize/fuse.js | 110 +++++++++ src/lib/serialize/sink.js | 282 ++++++++++++----------- src/lib/serialize/text-sink.js | 114 +++------ src/lib/serialize/token-values.js | 104 +++++++++ src/reduce-tokens.js | 138 +++++------ src/reduce.js | 40 +++- test/integration/package-exports.test.js | 9 +- test/unit/create-reducer.test.js | 48 +++- test/unit/name-chars.test.js | 37 +++ test/unit/nesting-limit-calc.test.js | 35 +++ test/unit/reduceCalc-core.test.js | 9 + test/unit/reduceTokens-edges.test.js | 84 +++++++ test/unit/reduceTokens-eof.test.js | 42 ++++ test/unit/reduceTokens-fusion.test.js | 92 +++++++- types/lib/core.d.ts | 17 +- types/lib/options.d.ts | 3 + types/lib/serialize/fuse.d.ts | 55 +++++ types/lib/serialize/sink.d.ts | 108 +++++---- types/lib/serialize/text-sink.d.ts | 61 ++--- types/lib/serialize/token-values.d.ts | 40 ++++ types/reduce-tokens.d.ts | 40 ++-- types/reduce.d.ts | 13 +- 27 files changed, 1157 insertions(+), 519 deletions(-) create mode 100644 src/lib/serialize/fuse.js create mode 100644 src/lib/serialize/token-values.js create mode 100644 test/unit/name-chars.test.js create mode 100644 test/unit/nesting-limit-calc.test.js create mode 100644 test/unit/reduceTokens-edges.test.js create mode 100644 test/unit/reduceTokens-eof.test.js create mode 100644 types/lib/serialize/fuse.d.ts create mode 100644 types/lib/serialize/token-values.d.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 951f805..3368bb9 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,8 @@ All notable changes to this project will be documented in this file. See [commit - add `postcss-calc/reduce-tokens`, which reduces a `@csstools/css-tokenizer` token list and returns token replacements. It never reads token source offsets, so it works on lists edited after tokenization. `postcss-calc/reduce` - shares its implementation and its output is unchanged. + shares its implementation; its output is unchanged except as noted under + Bug fixes. ### Bug fixes diff --git a/README.md b/README.md index b224d37..de14ef5 100755 --- a/README.md +++ b/README.md @@ -92,11 +92,14 @@ by default; provide `onParseError` and/or `onWarn` if you want diagnostics. If you already hold the tokens of a value (from `@csstools/css-tokenizer`) and edit them in place, `postcss-calc/reduce-tokens` reduces them without going through a string. It reads only each token's type and raw text, never its -source offsets, so the list may have been edited since it was tokenized. Its -other fields must agree: a token's parsed value (`token[4]`) is read along +source offsets, so the list may have been edited since it was tokenized. + +Its other fields must agree: a token's parsed value (`token[4]`) is read along with its type and raw text, so when you change a token's text, replace the token with one that `tokenize` produced from the new text. Only the source -offsets may be stale. It returns the replacements that change the text: ordered, non-overlapping +offsets may be stale. + +It returns the replacements that change the text: ordered, non-overlapping `{ start, end, tokens }` records that replace `tokens.slice(start, end)`. It does not modify the list; apply the replacements from last to first. @@ -113,9 +116,10 @@ for (const { start, end, tokens: next } of reduceTokens(tokens).toReversed()) { The replacement tokens are the tokenization of the result text on its own, so their source offsets (`token[2]`, `token[3]`) describe that text, not their -place in your list. Do not rely on them. Tokens spliced next to unspaced -neighbours can also merge or split when the list is tokenized again, as the -string reducer's output can. +place in your list. Do not rely on them. A replacement that would merge with +the token beside it in `tokens` starts or ends with a whitespace token. +For a sub-range, this reads the tokens just outside it, so splice +the replacements into the same list you passed. To skip tokenizing values that cannot contain a math function, check the string first with the named export `hasPotentialMathFunction(value)`. It is @@ -132,7 +136,8 @@ if (hasPotentialMathFunction(value)) { ``` End-of-file tokens in the list (for example from concatenating `tokenize` -results) are ignored. +results) are ignored: they never appear in a replacement, and they do not hide +the neighbouring token from that check. It takes the same options as `reduceCalc`. `onParseError` and `onWarn` receive text built from the current tokens. diff --git a/src/lib/core.js b/src/lib/core.js index d36e835..48fba80 100644 --- a/src/lib/core.js +++ b/src/lib/core.js @@ -5,34 +5,29 @@ import { parseRange } from './parser.js'; import { ParseError, positionedError } from './parser/state.js'; import { simplify } from './simplify.js'; import { analyze } from './analyze.js'; -import { printResult, serializeResult } from './serialize.js'; -import { TokenSink } from './serialize/sink.js'; -import { - FUSE_SEPARATOR, - firstCharOf, - lastCharBefore, - wouldFuse, -} from './serialize/text-sink.js'; +import { printResult } from './serialize.js'; +import { firstCharOf, lastCharBefore } from './serialize/fuse.js'; import { closerAt, matchBlocks } from './blocks.js'; import { assertDepth } from './limits.js'; import { hasMathFunctionToken, mathFunctionAt } from './functions.js'; import { joinTokens, offsetLookup } from './offsets.js'; /** @typedef {import('../reduce.js').ResolvedReduceCalcOptions} ResolvedReduceCalcOptions */ +/** @typedef {import('../reduce.js').ReduceDiagnostics} ReduceDiagnostics */ /** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ /** @typedef {import('./blocks.js').Blocks} Blocks */ /** @typedef {import('./functions.js').MathFunction} MathFunction */ +/** @typedef {import('./serialize/sink.js').Sink} Sink */ /** - * Receives one replaced range. `start` and `end` are token indices, `end` - * exclusive. `printed` is the replacement text, or the token sink holding the - * replacement tokens when the caller asked for tokens. + * Where `reduceTokenList` gets its sinks and sends its results. The printer + * writes into the sink that `createSink` makes for the candidate + * `[start, end)`, and `emit` receives each changed range with its sink. * - * @callback EmitReplacement - * @param {number} start - * @param {number} end - * @param {string | TokenSink} printed - * @return {void} + * @template {Sink} S + * @typedef {object} ReplacementTarget + * @property {(start: number, end: number) => S} createSink + * @property {(start: number, end: number, printed: S) => void} emit */ /** @@ -44,9 +39,8 @@ function asError(error) { } /** - * Parse, analyze and simplify one supported math function, then serialize it. - * A throw from parsing or analysis means the range is not a valid calculation. - * A throw from serialization is reported and yields no replacement text. + * Parse, analyze and simplify one supported math function. A throw means the + * range is not a valid calculation. * * @param {CSSToken[]} tokens * @param {Blocks} blocks @@ -54,18 +48,8 @@ function asError(error) { * @param {number} end Index after its closer, or the end of the range when unclosed. * @param {{normalizedName: string, definition: MathFunction}} lookup * @param {ResolvedReduceCalcOptions} options - * @param {boolean} printTokens Print into a token sink instead of a string. - * @return {{text: string | TokenSink | undefined, unresolved: boolean}} */ -function compileCandidate( - tokens, - blocks, - start, - end, - lookup, - options, - printTokens -) { +function compileCandidate(tokens, blocks, start, end, lookup, options) { const name = /** @type {{value: string}} */ (tokens[start][4]).value; const isCalc = lookup.definition.calculation === true; const closer = closerAt(blocks, start); @@ -82,28 +66,44 @@ function compileCandidate( const unresolved = analysis.unresolved; /** @type {'resolved' | 'unresolved'} */ const status = unresolved ? 'unresolved' : 'resolved'; - const result = { - tree, - status, - rootName: lookup.normalizedName, - rootSpelling: tokens[start][1].slice(0, -1), - calculation: isCalc, - }; - const printOptions = { - precision: options.precision, - unwrapSingleNegativeNumber: options.unwrapSingleNegativeNumber, - unwrapSingleValue: options.unwrapSingleValue, + return { + result: { + tree, + status, + rootName: lookup.normalizedName, + rootSpelling: tokens[start][1].slice(0, -1), + calculation: isCalc, + }, + printOptions: { + precision: options.precision, + unwrapSingleNegativeNumber: options.unwrapSingleNegativeNumber, + unwrapSingleValue: options.unwrapSingleValue, + }, + unresolved, }; +} + +/** + * Print a compiled candidate into `sink`. A throw from the printer is reported + * and yields no replacement. + * + * @template {Sink} S + * @param {ReturnType} compiled + * @param {S} sink + * @param {ReduceDiagnostics} diagnostics + * @param {CSSToken[]} tokens + * @param {number} start + * @param {number} end + * @return {S | undefined} + */ +function printCandidate(compiled, sink, diagnostics, tokens, start, end) { try { - if (printTokens) { - const sink = new TokenSink(tokens, start, end); - const printed = printResult(result, printOptions, sink); - return { text: printed ? sink : undefined, unresolved }; - } - return { text: serializeResult(result, printOptions), unresolved }; + return printResult(compiled.result, compiled.printOptions, sink) + ? sink + : undefined; } catch (error) { - options.onParseError?.(asError(error), joinTokens(tokens, start, end)); - return { text: undefined, unresolved }; + diagnostics.onParseError?.(asError(error), joinTokens(tokens, start, end)); + return undefined; } } @@ -115,25 +115,19 @@ function compileCandidate( * The neighbours are the tokens just outside the range, skipping EOF tokens, * because the replacement is spliced in between them. * - * @template {string | TokenSink} T - * @param {T} printed + * @template {Sink} S + * @param {S} printed * @param {CSSToken[]} tokens * @param {number} from * @param {number} to - * @return {T | string} + * @return {S} */ function fitted(printed, tokens, from, to) { - const before = lastCharBefore(tokens, from); - const after = firstCharOf(tokens, to, tokens.length); - if (typeof printed !== 'string') { - printed.fit(before, after); - return printed; - } - return ( - (wouldFuse(before, printed[0]) ? FUSE_SEPARATOR : '') + - printed + - (wouldFuse(printed.at(-1), after) ? FUSE_SEPARATOR : '') + printed.fit( + lastCharBefore(tokens, from), + firstCharOf(tokens, to, tokens.length) ); + return printed; } /** @@ -149,23 +143,24 @@ function fitted(printed, tokens, from, to) { * source offsets (`token[2]`, `token[3]`), so the list may have been edited * since it was tokenized. Emitted indices are indices into `tokens`. * + * @template {Sink} S * @param {CSSToken[]} tokens - * @param {ResolvedReduceCalcOptions} options - * @param {EmitReplacement} emit + * @param {ResolvedReduceCalcOptions} options Shared by every call; never changed. + * @param {ReduceDiagnostics} diagnostics Warnings and parse errors for this call. + * @param {ReplacementTarget} target * @param {number} start * @param {number} end * @param {string} [source] The joined text of the range, when the caller already has it. - * @param {boolean} [printTokens] Emit token sinks instead of text. * @return {void} */ function reduceTokenList( tokens, options, - emit, + diagnostics, + target, start, end, - source, - printTokens = false + source ) { if (!hasMathFunctionToken(tokens, start, end)) return; @@ -176,7 +171,7 @@ function reduceTokenList( assertDepth(matched.maxDepth); blocks = matched.blocks; } catch (error) { - options.onParseError?.( + diagnostics.onParseError?.( asError(error), source ?? joinTokens(tokens, start, end) ); @@ -191,30 +186,37 @@ function reduceTokenList( const candidateEnd = closer === -1 ? end : closer + 1; /** @type {ReturnType | undefined} */ - let result; + let compiled; try { - result = compileCandidate( + compiled = compileCandidate( tokens, blocks, i, candidateEnd, lookup, - options, - printTokens + options ); } catch (error) { - options.onParseError?.( + diagnostics.onParseError?.( positionedError(asError(error), offsetOf), joinTokens(tokens, i, candidateEnd) ); } - if (result !== undefined) { - if (options.warnWhenCannotResolve && result.unresolved) { + if (compiled !== undefined) { + const printed = printCandidate( + compiled, + target.createSink(i, candidateEnd), + diagnostics, + tokens, + i, + candidateEnd + ); + if (options.warnWhenCannotResolve && compiled.unresolved) { wholeText ??= joinTokens(tokens, start, end); - options.onWarn?.('Could not reduce expression: ' + wholeText); + diagnostics.onWarn?.('Could not reduce expression: ' + wholeText); } - if (result.text !== undefined) { - emit(i, candidateEnd, fitted(result.text, tokens, i, candidateEnd)); + if (printed !== undefined) { + target.emit(i, candidateEnd, fitted(printed, tokens, i, candidateEnd)); } } // A supported function is one candidate, so its nested calls are skipped. diff --git a/src/lib/limits.js b/src/lib/limits.js index 0660246..9d7a7d2 100644 --- a/src/lib/limits.js +++ b/src/lib/limits.js @@ -34,8 +34,7 @@ function checkCalculationDepth(node, depth = 0) { // components were nested arrays. for (const part of node.components) { assertDepth(depth + 1); - if (typeof part === 'object' && !Array.isArray(part)) - checkCalculationDepth(part, depth + 2); + if (!Array.isArray(part)) checkCalculationDepth(part, depth + 2); } } } diff --git a/src/lib/options.js b/src/lib/options.js index 617bdd8..cb3a649 100644 --- a/src/lib/options.js +++ b/src/lib/options.js @@ -2,15 +2,17 @@ /** @typedef {import('../reduce.js').ResolvedReduceCalcOptions} ResolvedReduceCalcOptions */ /** + * The options that shape the result. Callbacks are not part of them: they are + * diagnostics, passed to the reducer separately. + * * @param {ReduceCalcOptions} [opts] * @return {ResolvedReduceCalcOptions} */ function resolveOptions(opts) { return { - warnWhenCannotResolve: false, - unwrapSingleNegativeNumber: false, - unwrapSingleValue: false, - ...opts, + warnWhenCannotResolve: opts?.warnWhenCannotResolve ?? false, + unwrapSingleNegativeNumber: opts?.unwrapSingleNegativeNumber ?? false, + unwrapSingleValue: opts?.unwrapSingleValue ?? false, // An explicit `precision: undefined` keeps the default. precision: opts?.precision ?? 5, }; diff --git a/src/lib/serialize/fuse.js b/src/lib/serialize/fuse.js new file mode 100644 index 0000000..f00afec --- /dev/null +++ b/src/lib/serialize/fuse.js @@ -0,0 +1,110 @@ +// Adjacency rules shared by the printer's sinks and by the reducer core. They +// decide when a printed token would merge with the token beside it, and which +// tokens count as CSS at all. + +import { TokenType } from '@csstools/css-tokenizer'; + +/** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ + +/** + * A character that can continue or start an identifier, number or dimension: + * an ASCII letter, digit, `_`, `-` or `.`, or any code point from U+0080 up. + * `/u` makes the class match whole code points. The ASCII ranges are spelled + * out, since `\w` and `\d` are easy to misread as Unicode-aware. + */ +const NAME_CHAR = /[A-Za-z0-9_\-.\u0080-\u{10FFFF}]/u; + +const DIGIT = /[0-9]/; + +/** + * Whether a token ending in `last` and a token starting with `first` would + * read back as one token if written without a separator. The check is + * conservative: it may ask for a separator that is not strictly needed. + * + * @param {string | undefined} last + * @param {string | undefined} first + * @return {boolean} + */ +function wouldFuse(last, first) { + if (last === undefined || first === undefined) return false; + // A digit continues into `%`, and a backslash starts an escape that + // continues the name or unit before it. + if (first === '%') return DIGIT.test(last); + return ( + NAME_CHAR.test(last) && + (NAME_CHAR.test(first) || first === '(' || first === '\\') + ); +} + +/** + * What keeps a replacement apart from a neighbour it would fuse with. A space + * rather than the empty comment CSS Syntax 3 §9 allows, because tools that + * strip comments from values (such as minifiers run after this plugin) would + * fuse the tokens again. + */ +const FUSE_SEPARATOR = ' '; + +/** + * Whether `token` is an end-of-file token. It marks the end of a tokenized + * string, so one inside a list (for example from concatenated `tokenize` + * results) is not part of the CSS and is skipped. + * + * @param {CSSToken | undefined} token + * @return {boolean} + */ +function isEOF(token) { + return token?.[0] === TokenType.EOF; +} + +/** + * The index of the first non-EOF token in `list[i, end)`, or `end` when there + * is none. + * + * @param {CSSToken[]} list + * @param {number} i + * @param {number} end + * @return {number} + */ +function nextCss(list, i, end) { + let j = i; + while (j < end && isEOF(list[j])) j++; + return j; +} + +/** + * The first character of the first non-EOF token in `list[start, end)`. + * + * @param {CSSToken[]} list + * @param {number} start + * @param {number} end + * @return {string | undefined} + */ +function firstCharOf(list, start, end) { + for (let i = start; i < end; i++) { + if (!isEOF(list[i])) return list[i][1][0]; + } + return undefined; +} + +/** + * The last character of the last non-EOF token in `list[0, end)`. + * + * @param {CSSToken[]} list + * @param {number} end + * @return {string | undefined} + */ +function lastCharBefore(list, end) { + for (let i = end - 1; i >= 0; i--) { + if (!isEOF(list[i])) return list[i][1].at(-1); + } + return undefined; +} + +export { + FUSE_SEPARATOR, + firstCharOf, + isEOF, + lastCharBefore, + nextCss, + wouldFuse, +}; diff --git a/src/lib/serialize/sink.js b/src/lib/serialize/sink.js index 2e69a64..bf9cd29 100644 --- a/src/lib/serialize/sink.js +++ b/src/lib/serialize/sink.js @@ -4,136 +4,88 @@ // token has the same type and raw text. import { TokenType } from '@csstools/css-tokenizer'; +import { FUSE_SEPARATOR, firstCharOf, nextCss, wouldFuse } from './fuse.js'; +import { TextSink } from './text-sink.js'; import { - FUSE_SEPARATOR, - TextSink, - firstCharOf, - isEOF, - wouldFuse, -} from './text-sink.js'; + decodeIdentifier, + newToken, + numericDetail, + sameText, +} from './token-values.js'; /** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ /** - * What the printer writes to. `TextSink` and `TokenSink` both implement it, so - * a method added to one must be added to the other. + * What the printer writes to, and what the reducer core calls on the printed + * result. `TextSink` and `TokenSink` both implement it; the `@implements` + * tags make `tsc` check that they keep doing so. * - * @typedef {TextSink | TokenSink} Sink + * @typedef {object} Sink + * @property {() => void} open + * @property {() => void} close + * @property {() => void} comma + * @property {(operator: '+' | '-' | '*' | '/') => void} op + * @property {() => void} minus + * @property {(name: string) => void} fn + * @property {(raw: string) => void} ident + * @property {(text: string) => void} number + * @property {(text: string, unit: string) => void} dimension + * @property {(list: CSSToken[], start: number, end: number) => void} source + * @property {(before: string | undefined, after: string | undefined) => void} fit */ /** - * Whether `raw` is `prefix + first + last`, without building that string. + * A printed token, as a piece of the output. A number is a source token, + * named by its index in the source list, and it is reused as it is. Any other + * piece is a token the printer built, with unspecified offsets. * - * @param {string} raw - * @param {string} prefix - * @param {string} first - * @param {string} last - * @return {boolean} + * @typedef {number | CSSToken} Piece */ -function sameText(raw, prefix, first, last) { - return ( - raw.length === prefix.length + first.length + last.length && - raw.startsWith(prefix) && - raw.startsWith(first, prefix.length) && - raw.endsWith(last) - ); -} /** - * The value of a CSS identifier from its raw text: each escape is replaced by - * the code point it names, so `c\61 lc` reads `calc`. + * The parsed value of a token, copied so a replacement shares no object with + * the source list. * - * @param {string} raw - * @return {string} - */ -function decodeIdentifier(raw) { - if (!raw.includes('\\')) return raw; - let value = ''; - let i = 0; - while (i < raw.length) { - const char = raw[i]; - if (char !== '\\') { - value += char; - i++; - continue; - } - // A backslash at the end of the input is a parse error that reads U+FFFD. - if (i + 1 === raw.length) { - value += '\ufffd'; - break; - } - const hex = /^[0-9a-fA-F]{1,6}/.exec(raw.slice(i + 1, i + 7)); - if (hex === null) { - value += raw[i + 1]; - i += 2; - continue; - } - const codePoint = Number.parseInt(hex[0], 16); - const valid = - codePoint !== 0 && - codePoint <= 0x10ffff && - (codePoint < 0xd800 || codePoint > 0xdfff); - value += String.fromCodePoint(valid ? codePoint : 0xfffd); - i += 1 + hex[0].length; - // One whitespace after a hex escape belongs to it; CRLF counts as one. - if (raw.startsWith('\r\n', i)) i += 2; - else if (/[ \t\n\f\r]/.test(raw[i] ?? '')) i++; - } - return value; -} - -/** - * The detail object of a numeric token, as the tokenizer makes it. + * This is a shallow copy. The numeric details the tokenizer makes are flat, so + * the copy shares nothing with the source. A detail that nests objects would + * need a deep copy here. * - * @param {TokenType} type - * @param {string} number The signed number text. - * @param {string} unit - * @return {object} + * @param {unknown} detail + * @return {unknown} */ -function numericDetail(type, number, unit) { - /** @type {{value: number, signCharacter?: '-', type?: 'integer' | 'number', unit?: string}} */ - // The tokenizer always has the key, `undefined` when the number is unsigned. - const detail = { - value: Number(number), - signCharacter: number.startsWith('-') ? '-' : undefined, - }; - if (type !== TokenType.Percentage) { - detail.type = /^[+-]?\d+$/.test(number) ? 'integer' : 'number'; - } - if (type === TokenType.Dimension) detail.unit = decodeIdentifier(unit); - return detail; +function copyDetail(detail) { + return detail !== null && typeof detail === 'object' ? { ...detail } : detail; } /** - * A printed token. Its offsets are unspecified. + * A separator that keeps two tokens apart. It is always a new token, never a + * source token reused by index, so a replacement that adds one no longer + * matches the source range. All separator sites use this, and only this. * - * @param {TokenType} type - * @param {string} raw - * @param {object | null | undefined} detail * @return {CSSToken} */ -function newToken(type, raw, detail) { - return /** @type {CSSToken} */ ( - /** @type {unknown} */ ([type, raw, 0, 0, detail]) - ); +function fuseSeparator() { + return newToken(TokenType.Whitespace, FUSE_SEPARATOR, undefined); } /** - * The output of a printer that builds tokens. `source[start, end)` is the - * list the printed tokens are compared with: while the printed tokens match - * it in order, each one is the source token itself. Once one differs, the - * rest are new tokens with unspecified offsets. + * The output of a printer that builds tokens. The printed pieces are compared + * with the source range `[start, end)`: when they are that range's non-EOF + * tokens in order, each one a source index, the output is the source range + * unchanged. Tokens are only built, and copied, by `build`. + * + * EOF tokens take no part in matching, because printed output never holds + * them. + * + * @implements {Sink} */ class TokenSink { - /** @type {CSSToken[]} */ - tokens = []; + /** @type {Piece[]} */ + pieces = []; /** @type {boolean} */ minusPending = false; - /** @type {boolean} */ - matching = true; - /** @type {number} */ - cursor; /** @type {CSSToken[]} */ sourceTokens; + /** @type {number} */ start; /** @type {number} */ end; /** @@ -143,22 +95,81 @@ class TokenSink { */ constructor(source, start, end) { this.sourceTokens = source; - this.cursor = start; + this.start = start; this.end = end; } /** - * Whether the printed tokens are the source range, token for token. + * The source index that the next printed token must have to be reused, or + * -1 once the output has diverged. Only the last piece decides: a built + * token ends reuse, and every later token is built too. + * + * @return {number} + */ + expectedIndex() { + const last = this.pieces.at(-1); + if (last === undefined) { + return nextCss(this.sourceTokens, this.start, this.end); + } + if (typeof last !== 'number') return -1; + return nextCss(this.sourceTokens, last + 1, this.end); + } + + /** + * Whether the pieces are the source range, token for token. * * @return {boolean} */ identical() { - return this.matching && !this.minusPending && this.cursor === this.end; + if (this.minusPending) return false; + let expected = nextCss(this.sourceTokens, this.start, this.end); + for (const piece of this.pieces) { + if (piece !== expected) return false; + expected = nextCss(this.sourceTokens, expected + 1, this.end); + } + return expected === this.end; } /** - * Append the source token at the cursor when it has `type` and raw text - * `prefix + first + last`. Otherwise the output has diverged for good. + * The tokens of the output. Each one carries offsets that locate it in the + * output text, counting from 0, and a copied parsed value, so the result + * shares no object with the source list. + * + * @return {CSSToken[]} + */ + build() { + let offset = 0; + return this.pieces.map((piece) => { + const token = + typeof piece === 'number' ? this.sourceTokens[piece] : piece; + const from = offset; + offset += token[1].length; + return /** @type {CSSToken} */ ( + /** @type {unknown} */ ([ + token[0], + token[1], + from, + offset - 1, + copyDetail(token[4]), + ]) + ); + }); + } + + /** + * The text of a piece. + * + * @param {Piece} piece + * @return {string} + */ + textOf(piece) { + return typeof piece === 'number' ? this.sourceTokens[piece][1] : piece[1]; + } + + /** + * Append the source token at the expected index when it has `type` and raw + * text `prefix + first + last`. Returns false otherwise, and the caller + * builds the token. * * @param {TokenType} type * @param {string} prefix @@ -167,15 +178,14 @@ class TokenSink { * @return {boolean} */ reuse(type, prefix, first, last) { - if (this.matching && this.cursor < this.end) { - const token = this.sourceTokens[this.cursor]; + const index = this.expectedIndex(); + if (index !== -1 && index < this.end) { + const token = this.sourceTokens[index]; if (token[0] === type && sameText(token[1], prefix, first, last)) { - this.tokens.push(token); - this.cursor++; + this.pieces.push(index); return true; } } - this.matching = false; return false; } @@ -187,34 +197,27 @@ class TokenSink { * @return {void} */ separate(first) { - if (wouldFuse(this.tokens.at(-1)?.[1].at(-1), first)) - this.push(TokenType.Whitespace, FUSE_SEPARATOR, undefined); + const last = this.pieces.at(-1); + if (last !== undefined && wouldFuse(this.textOf(last).at(-1), first)) + this.pieces.push(fuseSeparator()); } /** * Keep the printed tokens apart from the tokens around them: `before` and - * `after` are the neighbours' adjacent characters. The printed tokens no - * longer match the source range once a separator is added. + * `after` are the neighbours' adjacent characters. Added separators are + * built tokens, so the output no longer matches the source range. * * @param {string | undefined} before * @param {string | undefined} after * @return {void} */ fit(before, after) { - const first = firstCharOf(this.tokens, 0, this.tokens.length); - const last = this.tokens.at(-1)?.[1].at(-1); - if (wouldFuse(before, first)) { - this.tokens.unshift( - newToken(TokenType.Whitespace, FUSE_SEPARATOR, undefined) - ); - this.matching = false; - } - if (wouldFuse(last, after)) { - this.tokens.push( - newToken(TokenType.Whitespace, FUSE_SEPARATOR, undefined) - ); - this.matching = false; - } + const head = this.pieces[0]; + const tail = this.pieces.at(-1); + const first = head === undefined ? undefined : this.textOf(head)[0]; + const last = tail === undefined ? undefined : this.textOf(tail).at(-1); + if (wouldFuse(before, first)) this.pieces.unshift(fuseSeparator()); + if (wouldFuse(last, after)) this.pieces.push(fuseSeparator()); } /** @return {string} The pending sign, or `''`. */ @@ -229,7 +232,7 @@ class TokenSink { if (this.minusPending) { this.minusPending = false; if (!this.reuse(TokenType.Delim, '', '-', '')) - this.tokens.push(newToken(TokenType.Delim, '-', { value: '-' })); + this.pieces.push(newToken(TokenType.Delim, '-', { value: '-' })); } } @@ -241,7 +244,7 @@ class TokenSink { */ push(type, raw, detail) { if (!this.reuse(type, '', raw, '')) { - this.tokens.push(newToken(type, raw, detail)); + this.pieces.push(newToken(type, raw, detail)); } } @@ -285,7 +288,7 @@ class TokenSink { const prefix = this.takeMinus(); this.separate((prefix || name)[0]); if (!this.reuse(TokenType.Function, prefix, name, '(')) { - this.tokens.push( + this.pieces.push( newToken(TokenType.Function, `${prefix}${name}(`, { value: decodeIdentifier(`${prefix}${name}`), }) @@ -298,7 +301,7 @@ class TokenSink { const prefix = this.takeMinus(); this.separate((prefix || raw)[0]); if (!this.reuse(TokenType.Ident, prefix, raw, '')) { - this.tokens.push( + this.pieces.push( newToken(TokenType.Ident, `${prefix}${raw}`, { value: decodeIdentifier(`${prefix}${raw}`), }) @@ -335,15 +338,15 @@ class TokenSink { this.separate((prefix || text)[0]); if (this.reuse(type, prefix, text, unit)) return; const number = `${prefix}${text}`; - this.tokens.push( + this.pieces.push( newToken(type, `${number}${unit}`, numericDetail(type, number, unit)) ); } /** * The raw tokens `list[start, end)` of an opaque component. Source tokens - * are appended by reference, and they keep the cursor only when they start - * exactly where the printed tokens are. + * of the sink's own list are reused by index when the range starts at the + * expected index; any other token is kept by reference as a built token. * * @param {CSSToken[]} list * @param {number} start @@ -354,11 +357,12 @@ class TokenSink { this.flushMinus(); const first = firstCharOf(list, start, end); if (first !== undefined) this.separate(first); - if (this.matching && this.cursor === start) this.cursor = end; - else this.matching = false; + const from = nextCss(list, start, end); + const reusable = + list === this.sourceTokens && this.expectedIndex() === from; // An EOF token is not CSS, so it is never copied into a replacement. - for (let i = start; i < end; i++) { - if (!isEOF(list[i])) this.tokens.push(list[i]); + for (let i = from; i < end; i = nextCss(list, i + 1, end)) { + this.pieces.push(reusable ? i : list[i]); } } } diff --git a/src/lib/serialize/text-sink.js b/src/lib/serialize/text-sink.js index 72d4156..aabb785 100644 --- a/src/lib/serialize/text-sink.js +++ b/src/lib/serialize/text-sink.js @@ -1,95 +1,26 @@ // The text output of the printer: a sink that collects the printed strings. // `sink.js` has the token-building counterpart. -import { TokenType } from '@csstools/css-tokenizer'; +import { FUSE_SEPARATOR, firstCharOf, isEOF, wouldFuse } from './fuse.js'; /** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ +/** @typedef {import('./sink.js').Sink} Sink */ /** The printed text of each binary operator, with the spaces the printer uses. */ const OPERATOR_TEXT = { '+': ' + ', '-': ' - ', '*': ' * ', '/': ' / ' }; -/** A character that can continue or start an identifier, number or dimension. */ -const NAME_CHAR = /[\w\-.\u0080-\uffff]/; - -const DIGIT = /\d/; - -/** - * Whether a token ending in `last` and a token starting with `first` would - * read back as one token if written without a separator. The check is - * conservative: it may ask for a separator that is not strictly needed. - * - * @param {string | undefined} last - * @param {string | undefined} first - * @return {boolean} - */ -function wouldFuse(last, first) { - if (last === undefined || first === undefined) return false; - // A digit continues into `%`, and a backslash starts an escape that - // continues the name or unit before it. - if (first === '%') return DIGIT.test(last); - return ( - NAME_CHAR.test(last) && - (NAME_CHAR.test(first) || first === '(' || first === '\\') - ); -} - -/** - * What keeps a replacement apart from a neighbour it would fuse with. A space - * rather than the empty comment CSS Syntax 3 §9 allows, because tools that - * strip comments from values (such as minifiers run after this plugin) would - * fuse the tokens again. - */ -const FUSE_SEPARATOR = ' '; - -/** - * Whether `token` is an end-of-file token. It marks the end of a tokenized - * string, so one inside a list (for example from concatenated `tokenize` - * results) is not part of the CSS and is skipped. - * - * @param {CSSToken | undefined} token - * @return {boolean} - */ -function isEOF(token) { - return token?.[0] === TokenType.EOF; -} - -/** - * The first character of the first non-EOF token in `list[start, end)`. - * - * @param {CSSToken[]} list - * @param {number} start - * @param {number} end - * @return {string | undefined} - */ -function firstCharOf(list, start, end) { - for (let i = start; i < end; i++) { - if (!isEOF(list[i])) return list[i][1][0]; - } - return undefined; -} - -/** - * The last character of the last non-EOF token in `list[0, end)`. - * - * @param {CSSToken[]} list - * @param {number} end - * @return {string | undefined} - */ -function lastCharBefore(list, end) { - for (let i = end - 1; i >= 0; i--) { - if (!isEOF(list[i])) return list[i][1].at(-1); - } - return undefined; -} - /** * The output of a printer that keeps its printed text as strings. + * + * @implements {Sink} */ class TextSink { /** @type {string[]} */ parts = []; /** @type {boolean} */ signed = false; + /** Text written before `parts`, set by `fit` to avoid shifting them. @type {string} */ + leading = ''; /** * Write a space when a token starting with `first` would fuse with the @@ -181,17 +112,32 @@ class TextSink { } } + /** + * Keep the printed text apart from the text around it: `before` and `after` + * are the neighbours' adjacent characters. + * + * @param {string | undefined} before + * @param {string | undefined} after + * @return {void} + */ + fit(before, after) { + const { parts } = this; + let first; + for (let i = 0; i < parts.length && first === undefined; i++) { + first = parts[i][0]; + } + let last; + for (let i = parts.length - 1; i >= 0 && last === undefined; i--) { + last = parts[i].at(-1); + } + if (wouldFuse(before, first)) this.leading = FUSE_SEPARATOR; + if (wouldFuse(last, after)) parts.push(FUSE_SEPARATOR); + } + /** @return {string} */ join() { - return this.parts.join(''); + return this.leading + this.parts.join(''); } } -export { - FUSE_SEPARATOR, - TextSink, - firstCharOf, - isEOF, - lastCharBefore, - wouldFuse, -}; +export { TextSink }; diff --git a/src/lib/serialize/token-values.js b/src/lib/serialize/token-values.js new file mode 100644 index 0000000..a7b6180 --- /dev/null +++ b/src/lib/serialize/token-values.js @@ -0,0 +1,104 @@ +// Token values for the printer's token sink: the parsed value of an identifier, +// a numeric token's detail, and the raw-text match that lets a source token be reused. +import { TokenType } from '@csstools/css-tokenizer'; + +/** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ + +/** + * Whether `raw` is `prefix + first + last`, without building that string. + * + * @param {string} raw + * @param {string} prefix + * @param {string} first + * @param {string} last + * @return {boolean} + */ +function sameText(raw, prefix, first, last) { + return ( + raw.length === prefix.length + first.length + last.length && + raw.startsWith(prefix) && + raw.startsWith(first, prefix.length) && + raw.endsWith(last) + ); +} + +/** + * The value of a CSS identifier from its raw text: each escape is replaced by + * the code point it names, so `c\61 lc` reads `calc`. + * + * @param {string} raw + * @return {string} + */ +function decodeIdentifier(raw) { + if (!raw.includes('\\')) return raw; + let value = ''; + let i = 0; + while (i < raw.length) { + const char = raw[i]; + if (char !== '\\') { + value += char; + i++; + continue; + } + // A backslash at the end of the input is a parse error that reads U+FFFD. + if (i + 1 === raw.length) { + value += '\ufffd'; + break; + } + const hex = /^[0-9a-fA-F]{1,6}/.exec(raw.slice(i + 1, i + 7)); + if (hex === null) { + value += raw[i + 1]; + i += 2; + continue; + } + const codePoint = Number.parseInt(hex[0], 16); + const valid = + codePoint !== 0 && + codePoint <= 0x10ffff && + (codePoint < 0xd800 || codePoint > 0xdfff); + value += String.fromCodePoint(valid ? codePoint : 0xfffd); + i += 1 + hex[0].length; + // One whitespace after a hex escape belongs to it; CRLF counts as one. + if (raw.startsWith('\r\n', i)) i += 2; + else if (/[ \t\n\f\r]/.test(raw[i] ?? '')) i++; + } + return value; +} + +/** + * The detail object of a numeric token, as the tokenizer makes it. + * + * @param {TokenType} type + * @param {string} number The signed number text. + * @param {string} unit + * @return {object} + */ +function numericDetail(type, number, unit) { + /** @type {{value: number, signCharacter?: '-', type?: 'integer' | 'number', unit?: string}} */ + // The tokenizer always has the key, `undefined` when the number is unsigned. + const detail = { + value: Number(number), + signCharacter: number.startsWith('-') ? '-' : undefined, + }; + if (type !== TokenType.Percentage) { + detail.type = /^[+-]?\d+$/.test(number) ? 'integer' : 'number'; + } + if (type === TokenType.Dimension) detail.unit = decodeIdentifier(unit); + return detail; +} + +/** + * A printed token. Its offsets are unspecified. + * + * @param {TokenType} type + * @param {string} raw + * @param {object | null | undefined} detail + * @return {CSSToken} + */ +function newToken(type, raw, detail) { + return /** @type {CSSToken} */ ( + /** @type {unknown} */ ([type, raw, 0, 0, detail]) + ); +} + +export { decodeIdentifier, newToken, numericDetail, sameText }; diff --git a/src/reduce-tokens.js b/src/reduce-tokens.js index b47596a..2316234 100644 --- a/src/reduce-tokens.js +++ b/src/reduce-tokens.js @@ -6,10 +6,12 @@ import { hasPotentialMathFunction } from './lib/functions.js'; import { resolveOptions } from './lib/options.js'; import { reduceTokenList } from './lib/core.js'; +import { TokenSink } from './lib/serialize/sink.js'; /** @typedef {import('./reduce.js').ReduceCalcOptions} ReduceCalcOptions */ +/** @typedef {import('./reduce.js').ReduceDiagnostics} ReduceDiagnostics */ +/** @typedef {import('./reduce.js').ResolvedReduceCalcOptions} ResolvedReduceCalcOptions */ /** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ -/** @typedef {import('./lib/serialize/sink.js').TokenSink} TokenSink */ /** * Replace `tokens.slice(start, end)` with `tokens`. @@ -24,8 +26,11 @@ import { reduceTokenList } from './lib/core.js'; * Where a reducer sends its warnings and parse errors. A parse error reaches * `warn` only when the reducer has no `onParseError` option. * + * The target has PostCSS's `Result.warn(text, opts)` shape, so a PostCSS + * `Result` can be passed directly. + * * @typedef {object} WarnTarget - * @property {(message: string, plugin?: string) => void} warn + * @property {(message: string, opts?: { plugin?: string }) => void} warn */ /** @@ -36,7 +41,8 @@ import { reduceTokenList } from './lib/core.js'; */ /** - * Check that `[start, end)` names a range of `tokens`. + * Check that `start` and `end` are indices into `tokens`, with `start` and + * `end` each in `[0, tokens.length]`. An inverted range passes this check. * * @param {CSSToken[]} tokens * @param {number} start @@ -48,6 +54,8 @@ function assertRange(tokens, start, end) { !Number.isInteger(start) || !Number.isInteger(end) || start < 0 || + end < 0 || + start > tokens.length || end > tokens.length ) { throw new RangeError( @@ -57,101 +65,91 @@ function assertRange(tokens, start, end) { } /** - * Reduce `tokens[start, end)` with resolved options whose callbacks are set. + * Reduce `tokens[start, end)` with the resolved options and diagnostics of + * one call. * - * @param {import('./reduce.js').ResolvedReduceCalcOptions} options + * @param {ResolvedReduceCalcOptions} options + * @param {ReduceDiagnostics} diagnostics * @param {CSSToken[]} tokens * @param {number} start * @param {number} end * @return {TokenReplacement[]} */ -function run(options, tokens, start, end) { +function run(options, diagnostics, tokens, start, end) { assertRange(tokens, start, end); + // An empty or inverted range names no tokens, so it has no replacements. + if (start >= end) return []; /** @type {TokenReplacement[]} */ const replacements = []; reduceTokenList( tokens, options, - (from, to, printed) => { - // The reducer asks for tokens, so the printed value is a token sink. - const sink = /** @type {TokenSink} */ (printed); - if (sink.identical()) return; - replacements.push({ - start: from, - end: to, - tokens: withOffsets(sink.tokens), - }); + diagnostics, + { + createSink: (from, to) => new TokenSink(tokens, from, to), + emit: (from, to, sink) => { + if (sink.identical()) return; + replacements.push({ + start: from, + end: to, + tokens: sink.build(), + }); + }, }, start, - end, - undefined, - true + end ); return replacements; } -/** - * The tokens with offsets that locate each one in their own joined text, as - * `tokenize` of that text would give. Reused source tokens are copied, so the - * source list is never shared with a replacement. The detail objects - * (`token[4]`) are shared, so a caller must not edit them in place. - * - * @param {CSSToken[]} tokens - * @return {CSSToken[]} - */ -function withOffsets(tokens) { - let offset = 0; - return tokens.map((token) => { - const from = offset; - offset += token[1].length; - return /** @type {CSSToken} */ ( - /** @type {unknown} */ ([token[0], token[1], from, offset - 1, token[4]]) - ); - }); -} - /** * Build a reducer that resolves `opts` once, for a caller that reduces many - * token lists with the same options. Each call sends its warnings to the - * target it is given, unless `opts` has `onWarn`, which takes precedence as - * `onParseError` does. + * token lists with the same options. + * + * Each call sends its warnings and parse errors to the `target` it is given. + * An `onWarn` or `onParseError` option takes precedence over the target. * * @param {ReduceCalcOptions} [opts] * @return {TokenReducer} */ function createReducer(opts) { const options = resolveOptions(opts); - return (tokens, target, start = 0, end = tokens.length) => - run( - { - ...options, - onWarn: - options.onWarn ?? - ((/** @type {string} */ message) => - target.warn(message, 'postcss-calc')), - onParseError: - options.onParseError ?? - ((/** @type {Error} */ error) => - target.warn(error.message, 'postcss-calc')), - }, - tokens, - start, - end - ); + return (tokens, target, start = 0, end = tokens.length) => { + /** @type {ReduceDiagnostics} */ + const diagnostics = { + onWarn: + opts?.onWarn ?? + ((message) => target.warn(message, { plugin: 'postcss-calc' })), + onParseError: + opts?.onParseError ?? + ((error) => target.warn(error.message, { plugin: 'postcss-calc' })), + }; + return run(options, diagnostics, tokens, start, end); + }; } /** A parse error is dropped when the caller gives no `onParseError`. */ function ignoreParseError() {} /** - * Simplify every supported CSS math function in `tokens[start, end)`. Returns - * the replacements that change the text, ordered and non-overlapping, and - * leaves `tokens` untouched. Apply them from last to first, or build a new - * list. Replacement indices are indices into `tokens`, and a range reads as - * the list it names would. The offsets of a replacement's tokens locate them - * in the replacement's own text, counting from 0. An empty or inverted - * range has no replacements. Throws a `RangeError` for a range that is not - * within `tokens`. + * Simplify every supported CSS math function in `tokens[start, end)`. + * + * Returns the replacements that change the text, ordered and non-overlapping. + * `tokens` is left untouched: apply the replacements from last to first, or + * build a new list. + * + * Replacement indices are indices into `tokens`. A range reads as the list it + * names would, with one exception: the token on each side of a replacement is + * read even when it lies outside the range, because a replacement that would + * fuse with it starts or ends with a whitespace token. EOF tokens in the list + * are skipped. + * + * The offsets of a replacement's tokens locate them in the replacement's own + * text, counting from 0. An empty or inverted range has no replacements. + * + * Throws a `RangeError` for a range that is not within `tokens`. A parse error + * is dropped unless `opts.onParseError` is set, as the standalone reducer drops + * warnings by default. `createReducer` sends it to its target instead. * * @param {CSSToken[]} tokens * @param {ReduceCalcOptions} [opts] @@ -161,8 +159,12 @@ function ignoreParseError() {} */ function reduceTokens(tokens, opts, start = 0, end = tokens.length) { const options = resolveOptions(opts); - options.onParseError ??= ignoreParseError; - return run(options, tokens, start, end); + /** @type {ReduceDiagnostics} */ + const diagnostics = { + onWarn: opts?.onWarn, + onParseError: opts?.onParseError ?? ignoreParseError, + }; + return run(options, diagnostics, tokens, start, end); } export default reduceTokens; diff --git a/src/reduce.js b/src/reduce.js index e92cac2..9275a5b 100644 --- a/src/reduce.js +++ b/src/reduce.js @@ -5,6 +5,7 @@ import { tokenize as cssTokenize } from '@csstools/css-tokenizer'; import { hasPotentialMathFunction } from './lib/functions.js'; import { resolveOptions } from './lib/options.js'; import { reduceTokenList } from './lib/core.js'; +import { TextSink as TextSinkClass } from './lib/serialize/text-sink.js'; /** * @typedef {object} ReduceCalcOptions @@ -16,7 +17,16 @@ import { reduceTokenList } from './lib/core.js'; * @property {(message: string) => void} [onWarn] Invoked when `warnWhenCannotResolve` is set and an expression cannot be reduced to a single value. */ -/** @typedef {Required> & Pick} ResolvedReduceCalcOptions */ +/** + * The callbacks of `ReduceCalcOptions`. A reducer call receives them apart from + * the options that shape the result. + * + * @typedef {Pick} ReduceDiagnostics + */ + +/** @typedef {Required>} ResolvedReduceCalcOptions */ + +/** @typedef {import('./lib/serialize/text-sink.js').TextSink} TextSink */ /** * Simplify every supported CSS math function in a component-value string. @@ -32,12 +42,17 @@ function reduceCalc(value, opts) { } const options = resolveOptions(opts); + /** @type {ReduceDiagnostics} */ + const diagnostics = { + onWarn: opts?.onWarn, + onParseError: opts?.onParseError, + }; /** @type {import('@csstools/css-tokenizer').CSSToken[]} */ let tokens; try { tokens = cssTokenize({ css: value }); } catch (error) { - options.onParseError?.( + diagnostics.onParseError?.( error instanceof Error ? error : new Error('Error', { cause: error }), value ); @@ -51,15 +66,18 @@ function reduceCalc(value, opts) { reduceTokenList( tokens, options, - (start, end, printed) => { - // The string caller asks for text, so the printed value is a string. - const text = /** @type {string} */ (printed); - const from = tokens[start][2]; - const to = tokens[end - 1][3] + 1; - if (to - from === text.length && value.startsWith(text, from)) return; - output += value.slice(lastIndex, from) + text; - lastIndex = to; - changed = true; + diagnostics, + { + createSink: () => new TextSinkClass(), + emit: (start, end, printed) => { + const text = printed.join(); + const from = tokens[start][2]; + const to = tokens[end - 1][3] + 1; + if (to - from === text.length && value.startsWith(text, from)) return; + output += value.slice(lastIndex, from) + text; + lastIndex = to; + changed = true; + }, }, 0, tokens.length, diff --git a/test/integration/package-exports.test.js b/test/integration/package-exports.test.js index df2abd4..a0c83d3 100644 --- a/test/integration/package-exports.test.js +++ b/test/integration/package-exports.test.js @@ -63,10 +63,17 @@ test('packed package exposes the standalone reducer with usable types', async () types, [ "import reduceCalc, { type ReduceCalcOptions } from 'postcss-calc/reduce';", - "import reduceTokens, { type TokenReplacement } from 'postcss-calc/reduce-tokens';", + "import reduceTokens, { createReducer, type TokenReplacement } from 'postcss-calc/reduce-tokens';", + '// @ts-expect-error internal types stay out of the public typings', + "import type { TokenSink } from 'postcss-calc/reduce-tokens';", + "import type { Result } from 'postcss';", "import { tokenize } from '@csstools/css-tokenizer';", "const replacements: TokenReplacement[] = reduceTokens(tokenize({ css: 'calc(1px + 2px)' }), { precision: 2 });", 'void replacements;', + 'declare const result: Result;', + 'const reducer = createReducer({ precision: 2 });', + "const viaResult: TokenReplacement[] = reducer(tokenize({ css: 'calc(1px + 2px)' }), result);", + 'void viaResult;', 'const options: ReduceCalcOptions = { precision: false, unwrapSingleNegativeNumber: true, unwrapSingleValue: true };', "const reduced: string = reduceCalc('calc(1px + 2px)', options);", 'void reduced;', diff --git a/test/unit/create-reducer.test.js b/test/unit/create-reducer.test.js index 62d6444..13cfde2 100644 --- a/test/unit/create-reducer.test.js +++ b/test/unit/create-reducer.test.js @@ -3,15 +3,16 @@ import { describe, test } from 'node:test'; import assert from 'node:assert/strict'; import { tokenize } from '@csstools/css-tokenizer'; +import postcss from 'postcss'; import { createReducer } from 'postcss-calc/reduce-tokens'; -/** A target that records each `warn(message, plugin)` call. */ +/** A target that records each `warn(message, opts)` call. */ function recordingTarget() { const calls = []; return { calls, - warn(message, plugin) { - calls.push([message, plugin]); + warn(message, opts) { + calls.push([message, opts]); }, }; } @@ -22,11 +23,34 @@ describe('createReducer warnings', () => { const reduce = createReducer({ warnWhenCannotResolve: true }); reduce(tokenize({ css: 'calc(1px + var(--a))' }), target); assert.deepEqual(target.calls, [ - ['Could not reduce expression: calc(1px + var(--a))', 'postcss-calc'], + [ + 'Could not reduce expression: calc(1px + var(--a))', + { plugin: 'postcss-calc' }, + ], ]); }); }); +describe('createReducer with a PostCSS Result', () => { + test('a warning reaches the Result with its plugin name, after another plugin ran', async () => { + const reduce = createReducer({ warnWhenCannotResolve: true }); + const earlier = { postcssPlugin: 'earlier', Once() {} }; + const probe = { + postcssPlugin: 'probe', + Once(_root, { result }) { + reduce(tokenize({ css: 'calc(1px + var(--a))' }), result); + }, + }; + const result = await postcss([earlier, probe]).process('a{}', { + from: undefined, + }); + assert.deepEqual( + result.warnings().map((warning) => [warning.text, warning.plugin]), + [['Could not reduce expression: calc(1px + var(--a))', 'postcss-calc']] + ); + }); +}); + describe('createReducer onWarn', () => { test('an onWarn option takes precedence over the target', () => { const target = recordingTarget(); @@ -62,7 +86,10 @@ describe('createReducer parse errors', () => { const reduce = createReducer(); reduce(tokenize({ css: 'calc(1px +)' }), target); assert.deepEqual(target.calls, [ - ['"+" must be surrounded by whitespace at position 9', 'postcss-calc'], + [ + '"+" must be surrounded by whitespace at position 9', + { plugin: 'postcss-calc' }, + ], ]); }); }); @@ -89,4 +116,15 @@ describe('createReducer range', () => { [[8, 15]] ); }); + + test('each call sends its warnings to the target it is given', () => { + const reduce = createReducer({ warnWhenCannotResolve: true }); + const first = recordingTarget(); + const second = recordingTarget(); + const tokens = tokenize({ css: 'calc(1px + var(--a))' }); + reduce(tokens, first); + reduce(tokens, second); + assert.equal(first.calls.length, 1); + assert.equal(second.calls.length, 1); + }); }); diff --git a/test/unit/name-chars.test.js b/test/unit/name-chars.test.js new file mode 100644 index 0000000..b7b327b --- /dev/null +++ b/test/unit/name-chars.test.js @@ -0,0 +1,37 @@ +// A replacement that ends beside a name character is separated from it. Name +// characters are ASCII letters, digits, `_`, `-`, `.`, and every code point from +// U+0080 up, so an emoji (a surrogate pair) counts as one. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import reduceCalc from 'postcss-calc/reduce'; + +const unwrap = { unwrapSingleValue: true }; + +describe('a replacement is separated from a neighbouring character before it', () => { + const cases = [ + ['a dot that would make a decimal', '.calc(1px + 1px)', '. 2px'], + ['a dot after a number', '1.calc(1px + 1px)', '1. 2px'], + ['a letter with no space', 'a.calc(1px + 1px)', 'a. 2px'], + ['a quote, which needs no separator', '"a"calc(1px + 1px)', '"a"2px'], + ]; + for (const [name, value, expected] of cases) { + test(name, () => { + assert.equal(reduceCalc(value, unwrap), expected); + }); + } +}); + +describe('a replacement is separated from a neighbouring name character', () => { + const cases = [ + ['an ASCII underscore', 'foo(calc(1px + 1px)_)', 'foo(2px _)'], + ['an ASCII hyphen', 'foo(calc(1px + 1px)-)', 'foo(2px -)'], + ['a Latin-1 letter', 'foo(calc(1px + 1px)é)', 'foo(2px é)'], + ['an emoji', 'foo(calc(1px + 1px)\u{1F600})', 'foo(2px \u{1F600})'], + ['a space', 'foo(calc(1px + 1px) \u{1F600})', 'foo(2px \u{1F600})'], + ]; + for (const [name, value, expected] of cases) { + test(name, () => { + assert.equal(reduceCalc(value, unwrap), expected); + }); + } +}); diff --git a/test/unit/nesting-limit-calc.test.js b/test/unit/nesting-limit-calc.test.js new file mode 100644 index 0000000..9b2acb4 --- /dev/null +++ b/test/unit/nesting-limit-calc.test.js @@ -0,0 +1,35 @@ +// The nesting limit counts a calc() that sits inside a var() fallback, and +// the calc() is reduced there, so its arguments are checked at the same depth. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import reduceCalc from 'postcss-calc/reduce'; + +const LIMIT = 1024; +const LIMIT_MESSAGE = `Calculation nesting exceeds the limit of ${LIMIT}`; +const nestedVarCalc = (n) => + `${'var(--x, '.repeat(n)}calc(1px + 1px)${')'.repeat(n)}`; +const reducedVarCalc = (n) => + `${'var(--x, '.repeat(n)}calc(2px)${')'.repeat(n)}`; + +describe('nesting limit with a calc() inside a var() fallback', () => { + test('a reduced calc() inside a var() fallback counts toward the limit', () => { + // Each `var(` is one opener and the calc() inside the innermost one is + // one more, so 1023 wrappers is 1024 openers in total. + const errors = []; + const output = reduceCalc(nestedVarCalc(LIMIT - 1), { + onParseError: (error) => errors.push(error.message), + }); + assert.deepEqual(errors, []); + assert.equal(output, reducedVarCalc(LIMIT - 1)); + }); + + test('one more wrapper past the limit is rejected and left unchanged', () => { + const input = nestedVarCalc(LIMIT); + const errors = []; + const output = reduceCalc(input, { + onParseError: (error, source) => errors.push([error.message, source]), + }); + assert.equal(output, input); + assert.deepEqual(errors, [[LIMIT_MESSAGE, input]]); + }); +}); diff --git a/test/unit/reduceCalc-core.test.js b/test/unit/reduceCalc-core.test.js index 06a5463..6c2d808 100644 --- a/test/unit/reduceCalc-core.test.js +++ b/test/unit/reduceCalc-core.test.js @@ -46,6 +46,15 @@ describe('reduceCalc: basic pipeline', () => { assert.equal(reduceCalc(String.raw`calc(1P\58 + 2px)`), 'calc(3px)'); }); + test('reduceCalc: a non-hex escape in a function name still names the calculation', () => { + // `\l` escapes a letter that is not a hex digit, so `ca\lc` reads `calc` + // and keeps its escaped spelling in the output. + assert.equal( + reduceCalc(String.raw`ca\lc(1px + 1px)`), + String.raw`ca\lc(2px)` + ); + }); + test('reduceCalc: opaque escaped spellings survive simplification', () => { assert.equal( reduceCalc(String.raw`calc(var(--kendo-spacing-1\.5) + 1px)`), diff --git a/test/unit/reduceTokens-edges.test.js b/test/unit/reduceTokens-edges.test.js new file mode 100644 index 0000000..6ac60f6 --- /dev/null +++ b/test/unit/reduceTokens-edges.test.js @@ -0,0 +1,84 @@ +// Edge cases of the token reducer: range bounds, end-of-file tokens inside a +// function, and parsed values that replacements must not share with the list. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize, TokenType } from '@csstools/css-tokenizer'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const eofToken = () => tokenize({ css: '' })[0]; + +/** The tokens of `css` without its trailing end-of-file token. */ +const bodyOf = (css) => + tokenize({ css }).filter((token) => token[0] !== TokenType.EOF); + +describe('reduceTokens: range bounds', () => { + test('an inverted range has no replacements', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + assert.deepEqual(reduceTokens(tokens, {}, 5, 2), []); + }); + + test('a start past the end of the list throws a RangeError', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + assert.throws( + () => reduceTokens(tokens, {}, tokens.length + 1, 2), + RangeError + ); + }); + + test('a negative end throws a RangeError', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + assert.throws(() => reduceTokens(tokens, {}, 0, -1), RangeError); + }); +}); + +describe('reduceTokens: end-of-file tokens inside a function', () => { + test('an end-of-file token inside an unchanged function gives no replacement', () => { + const tokens = bodyOf('calc(var(--a))'); + const list = [...tokens.slice(0, 3), eofToken(), ...tokens.slice(3)]; + assert.deepEqual(reduceTokens(list), []); + }); + + test('an end-of-file token inside a changed function is skipped', () => { + const tokens = bodyOf('calc(1px + 2px)'); + const list = [...tokens.slice(0, 3), eofToken(), ...tokens.slice(3)]; + assert.equal(raw(applyReplacements(list, reduceTokens(list))), 'calc(3px)'); + }); +}); + +describe('reduceTokens: end-of-file tokens at the edges of a range', () => { + test('an end-of-file token at the start of the range is skipped', () => { + const list = [eofToken(), ...bodyOf('calc(1px + 2px)')]; + assert.equal(raw(applyReplacements(list, reduceTokens(list))), 'calc(3px)'); + }); + + test('an end-of-file token after an unchanged function does not hide the next one', () => { + const list = [ + ...bodyOf('calc(var(--a))'), + eofToken(), + ...bodyOf('calc(1px + 2px)'), + ]; + const replacements = reduceTokens(list); + // The unchanged function and the end-of-file token come first. + const secondCalc = bodyOf('calc(var(--a))').length + 1; + assert.equal(replacements.length, 1); + assert.equal(replacements[0].start, secondCalc); + assert.equal( + raw(applyReplacements(list, replacements)), + 'calc(var(--a))calc(3px)' + ); + }); +}); + +describe('reduceTokens: replacements share no parsed values with the list', () => { + test('a replacement token has its own parsed value', () => { + const tokens = tokenize({ css: 'calc(1px + 2px)' }); + const [replacement] = reduceTokens(tokens); + const sourceValues = new Set( + tokens.map((token) => token[4]).filter((value) => value != null) + ); + for (const token of replacement.tokens) { + if (token[4] != null) assert.equal(sourceValues.has(token[4]), false); + } + }); +}); diff --git a/test/unit/reduceTokens-eof.test.js b/test/unit/reduceTokens-eof.test.js new file mode 100644 index 0000000..7793425 --- /dev/null +++ b/test/unit/reduceTokens-eof.test.js @@ -0,0 +1,42 @@ +// A token list may hold EOF tokens in its middle, as concatenated `tokenize` +// results do. Such tokens are not CSS: they must not be copied into a +// replacement, and they must not stop the tokens after them from being read. +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { tokenize } from '@csstools/css-tokenizer'; +import reduceTokens from 'postcss-calc/reduce-tokens'; +import { applyReplacements, raw } from '../helpers/reduce-list.js'; + +const unwrap = { unwrapSingleValue: true }; +const EOF = ['EOF-token', '', 0, 0, undefined]; + +describe('reduceTokens: EOF tokens in the middle of a list', () => { + test('a later replacement is found and the EOF token is not copied', () => { + // The first part has an opaque fallback that holds a source range, so the + // replacement inside it reuses source tokens by index. + const first = tokenize({ css: 'var(--x, calc(1px + 2px) 3px)' }); + const second = tokenize({ css: ' calc(4px + 5px)' }); + const tokens = [...first.slice(0, -1), EOF, ...second]; + + const replacements = reduceTokens(tokens, unwrap); + + assert.deepEqual( + replacements.map(({ start, end, tokens: next }) => [ + start, + end, + raw(next), + ]), + [ + [4, 11, '3px'], + [16, 23, '9px'], + ] + ); + for (const { tokens: next } of replacements) { + assert.ok(next.every((token) => token[0] !== 'EOF-token')); + } + assert.equal( + raw(applyReplacements(tokens, replacements)), + 'var(--x, 3px 3px) 9px' + ); + }); +}); diff --git a/test/unit/reduceTokens-fusion.test.js b/test/unit/reduceTokens-fusion.test.js index cc8d084..8fa85c4 100644 --- a/test/unit/reduceTokens-fusion.test.js +++ b/test/unit/reduceTokens-fusion.test.js @@ -3,9 +3,11 @@ import { describe, test } from 'node:test'; import assert from 'node:assert/strict'; import { tokenize } from '@csstools/css-tokenizer'; +import fc from 'fast-check'; import reduceCalc from 'postcss-calc/reduce'; import reduceTokens from 'postcss-calc/reduce-tokens'; import { applyReplacements, raw } from '../helpers/reduce-list.js'; +import { cssMathSourceArb } from '../helpers/arbitraries.js'; const unwrap = { unwrapSingleValue: true }; @@ -88,7 +90,7 @@ describe('reduceTokens: separators at the replacement boundary', () => { ); }); - test('no space is added when the neighbours cannot fuse', () => { + test('no separator is added when the neighbours cannot fuse', () => { const tokens = tokenize({ css: 'foo(calc(1px + 1px), a)' }); const [replacement] = reduceTokens(tokens, unwrap); assert.deepEqual( @@ -97,3 +99,91 @@ describe('reduceTokens: separators at the replacement boundary', () => { ); }); }); + +// Concatenated `tokenize` results put an EOF token inside the list. It marks +// the end of a tokenized string and is not CSS, so a replacement never carries +// one, and it does not hide the real neighbour from the separator check. +const join = (...parts) => parts.flatMap((css) => tokenize({ css })); +const isEOF = (token) => token[0] === 'EOF-token'; +const plain = (list) => + list.filter((token) => !isEOF(token)).map((t) => [t[0], t[1]]); + +describe('reduceTokens: EOF tokens inside the list', () => { + test('an opaque range that spans an EOF token does not copy it', () => { + const tokens = join('max(var(--a, 1px', ' x), calc(1px + 1px))'); + const replacements = reduceTokens(tokens, unwrap); + assert.equal(replacements.length, 1); + assert.ok(!replacements[0].tokens.some(isEOF)); + assert.equal( + raw(applyReplacements(tokens, replacements)), + reduceCalc('max(var(--a, 1px x), calc(1px + 1px))', unwrap) + ); + }); + + test('an EOF token after the replacement does not hide a fusing neighbour', () => { + const tokens = join('foo(calc(1px + 1px)', 'b)'); + const [replacement] = reduceTokens(tokens, unwrap); + assert.deepEqual( + replacement.tokens.map((token) => token[1]), + ['2px', ' '] + ); + }); + + test('an EOF token before the replacement does not hide a fusing neighbour', () => { + const tokens = join('a', 'calc(1px + 1px)'); + const [replacement] = reduceTokens(tokens, unwrap); + assert.deepEqual( + replacement.tokens.map((token) => token[1]), + [' ', '2px'] + ); + }); +}); + +// Serialized tokens must read back as the same tokens. +const neighbourArb = fc.constantFrom( + '', + 'a', + 'b1', + '-x', + '%', + '\\62', + '7', + '.5' +); + +describe('reduceTokens: replacements beside arbitrary neighbours', () => { + test('property: the edited list reads back as its own tokens', () => { + fc.assert( + fc.property( + neighbourArb, + cssMathSourceArb, + neighbourArb, + fc.boolean(), + (before, math, after, split) => { + // Splitting the source puts EOF tokens around the math function. + const tokens = split + ? join(before, math, after) + : tokenize({ css: before + math + after }); + // A caller's list that already fuses on its own is out of scope: + // only the replacements' boundaries are under test. + fc.pre( + JSON.stringify(plain(tokens)) === + JSON.stringify(plain(retokenized(tokens))) + ); + const replacements = reduceTokens(tokens, unwrap); + const edited = applyReplacements(tokens, replacements).filter( + (token) => !isEOF(token) + ); + assert.deepEqual( + edited.map((token) => [token[0], token[1]]), + retokenized(edited).filter((token) => token[0] !== 'EOF-token') + ); + for (const { tokens: next } of replacements) { + assert.ok(!next.some(isEOF)); + } + } + ), + { numRuns: 300 } + ); + }); +}); diff --git a/types/lib/core.d.ts b/types/lib/core.d.ts index 7889cfa..3c4635b 100644 --- a/types/lib/core.d.ts +++ b/types/lib/core.d.ts @@ -1,9 +1,13 @@ -import { TokenSink } from './serialize/sink.js'; export type ResolvedReduceCalcOptions = import('../reduce.js').ResolvedReduceCalcOptions; +export type ReduceDiagnostics = import('../reduce.js').ReduceDiagnostics; export type CSSToken = import('@csstools/css-tokenizer').CSSToken; export type Blocks = import('./blocks.js').Blocks; export type MathFunction = import('./functions.js').MathFunction; -export type EmitReplacement = (start: number, end: number, printed: string | TokenSink) => void; +export type Sink = import('./serialize/sink.js').Sink; +export type ReplacementTarget = { + createSink: (start: number, end: number) => S; + emit: (start: number, end: number, printed: S) => void; +}; /** * Reduce every supported math function in `tokens[start, end)`. Each function * is one candidate, so a nested function is never reduced on its own, and an @@ -17,14 +21,15 @@ export type EmitReplacement = (start: number, end: number, printed: string | Tok * source offsets (`token[2]`, `token[3]`), so the list may have been edited * since it was tokenized. Emitted indices are indices into `tokens`. * + * @template {Sink} S * @param {CSSToken[]} tokens - * @param {ResolvedReduceCalcOptions} options - * @param {EmitReplacement} emit + * @param {ResolvedReduceCalcOptions} options Shared by every call; never changed. + * @param {ReduceDiagnostics} diagnostics Warnings and parse errors for this call. + * @param {ReplacementTarget} target * @param {number} start * @param {number} end * @param {string} [source] The joined text of the range, when the caller already has it. - * @param {boolean} [printTokens] Emit token sinks instead of text. * @return {void} */ -declare function reduceTokenList(tokens: CSSToken[], options: ResolvedReduceCalcOptions, emit: EmitReplacement, start: number, end: number, source?: string, printTokens?: boolean): void; +declare function reduceTokenList(tokens: CSSToken[], options: ResolvedReduceCalcOptions, diagnostics: ReduceDiagnostics, target: ReplacementTarget, start: number, end: number, source?: string): void; export { reduceTokenList }; diff --git a/types/lib/options.d.ts b/types/lib/options.d.ts index c5df490..3f40a92 100644 --- a/types/lib/options.d.ts +++ b/types/lib/options.d.ts @@ -3,6 +3,9 @@ export type ReduceCalcOptions = import('../reduce.js').ReduceCalcOptions; export type ResolvedReduceCalcOptions = import('../reduce.js').ResolvedReduceCalcOptions; /** + * The options that shape the result. Callbacks are not part of them: they are + * diagnostics, passed to the reducer separately. + * * @param {ReduceCalcOptions} [opts] * @return {ResolvedReduceCalcOptions} */ diff --git a/types/lib/serialize/fuse.d.ts b/types/lib/serialize/fuse.d.ts new file mode 100644 index 0000000..4b06155 --- /dev/null +++ b/types/lib/serialize/fuse.d.ts @@ -0,0 +1,55 @@ +export type CSSToken = import('@csstools/css-tokenizer').CSSToken; +/** + * Whether a token ending in `last` and a token starting with `first` would + * read back as one token if written without a separator. The check is + * conservative: it may ask for a separator that is not strictly needed. + * + * @param {string | undefined} last + * @param {string | undefined} first + * @return {boolean} + */ +declare function wouldFuse(last: string | undefined, first: string | undefined): boolean; +/** + * What keeps a replacement apart from a neighbour it would fuse with. A space + * rather than the empty comment CSS Syntax 3 §9 allows, because tools that + * strip comments from values (such as minifiers run after this plugin) would + * fuse the tokens again. + */ +declare const FUSE_SEPARATOR = " "; +/** + * Whether `token` is an end-of-file token. It marks the end of a tokenized + * string, so one inside a list (for example from concatenated `tokenize` + * results) is not part of the CSS and is skipped. + * + * @param {CSSToken | undefined} token + * @return {boolean} + */ +declare function isEOF(token: CSSToken | undefined): boolean; +/** + * The index of the first non-EOF token in `list[i, end)`, or `end` when there + * is none. + * + * @param {CSSToken[]} list + * @param {number} i + * @param {number} end + * @return {number} + */ +declare function nextCss(list: CSSToken[], i: number, end: number): number; +/** + * The first character of the first non-EOF token in `list[start, end)`. + * + * @param {CSSToken[]} list + * @param {number} start + * @param {number} end + * @return {string | undefined} + */ +declare function firstCharOf(list: CSSToken[], start: number, end: number): string | undefined; +/** + * The last character of the last non-EOF token in `list[0, end)`. + * + * @param {CSSToken[]} list + * @param {number} end + * @return {string | undefined} + */ +declare function lastCharBefore(list: CSSToken[], end: number): string | undefined; +export { FUSE_SEPARATOR, firstCharOf, isEOF, lastCharBefore, nextCss, wouldFuse, }; diff --git a/types/lib/serialize/sink.d.ts b/types/lib/serialize/sink.d.ts index 28a7d1b..fbae90c 100644 --- a/types/lib/serialize/sink.d.ts +++ b/types/lib/serialize/sink.d.ts @@ -1,57 +1,39 @@ import { TokenType } from '@csstools/css-tokenizer'; import { TextSink } from './text-sink.js'; +import { decodeIdentifier, numericDetail, sameText } from './token-values.js'; export type CSSToken = import('@csstools/css-tokenizer').CSSToken; -export type Sink = TextSink | TokenSink; -/** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ +export type Sink = { + open: () => void; + close: () => void; + comma: () => void; + op: (operator: '+' | '-' | '*' | '/') => void; + minus: () => void; + fn: (name: string) => void; + ident: (raw: string) => void; + number: (text: string) => void; + dimension: (text: string, unit: string) => void; + source: (list: CSSToken[], start: number, end: number) => void; + fit: (before: string | undefined, after: string | undefined) => void; +}; +export type Piece = number | CSSToken; /** - * What the printer writes to. `TextSink` and `TokenSink` both implement it, so - * a method added to one must be added to the other. + * The output of a printer that builds tokens. The printed pieces are compared + * with the source range `[start, end)`: when they are that range's non-EOF + * tokens in order, each one a source index, the output is the source range + * unchanged. Tokens are only built, and copied, by `build`. * - * @typedef {TextSink | TokenSink} Sink - */ -/** - * Whether `raw` is `prefix + first + last`, without building that string. - * - * @param {string} raw - * @param {string} prefix - * @param {string} first - * @param {string} last - * @return {boolean} - */ -declare function sameText(raw: string, prefix: string, first: string, last: string): boolean; -/** - * The value of a CSS identifier from its raw text: each escape is replaced by - * the code point it names, so `c\61 lc` reads `calc`. + * EOF tokens take no part in matching, because printed output never holds + * them. * - * @param {string} raw - * @return {string} - */ -declare function decodeIdentifier(raw: string): string; -/** - * The detail object of a numeric token, as the tokenizer makes it. - * - * @param {TokenType} type - * @param {string} number The signed number text. - * @param {string} unit - * @return {object} - */ -declare function numericDetail(type: TokenType, number: string, unit: string): object; -/** - * The output of a printer that builds tokens. `source[start, end)` is the - * list the printed tokens are compared with: while the printed tokens match - * it in order, each one is the source token itself. Once one differs, the - * rest are new tokens with unspecified offsets. + * @implements {Sink} */ -declare class TokenSink { - /** @type {CSSToken[]} */ - tokens: CSSToken[]; +declare class TokenSink implements Sink { + /** @type {Piece[]} */ + pieces: Piece[]; /** @type {boolean} */ minusPending: boolean; - /** @type {boolean} */ - matching: boolean; - /** @type {number} */ - cursor: number; /** @type {CSSToken[]} */ sourceTokens: CSSToken[]; + /** @type {number} */ start: number; /** @type {number} */ end: number; /** * @param {CSSToken[]} source @@ -60,14 +42,38 @@ declare class TokenSink { */ constructor(source: CSSToken[], start: number, end: number); /** - * Whether the printed tokens are the source range, token for token. + * The source index that the next printed token must have to be reused, or + * -1 once the output has diverged. Only the last piece decides: a built + * token ends reuse, and every later token is built too. + * + * @return {number} + */ + expectedIndex(): number; + /** + * Whether the pieces are the source range, token for token. * * @return {boolean} */ identical(): boolean; /** - * Append the source token at the cursor when it has `type` and raw text - * `prefix + first + last`. Otherwise the output has diverged for good. + * The tokens of the output. Each one carries offsets that locate it in the + * output text, counting from 0, and a copied parsed value, so the result + * shares no object with the source list. + * + * @return {CSSToken[]} + */ + build(): CSSToken[]; + /** + * The text of a piece. + * + * @param {Piece} piece + * @return {string} + */ + textOf(piece: Piece): string; + /** + * Append the source token at the expected index when it has `type` and raw + * text `prefix + first + last`. Returns false otherwise, and the caller + * builds the token. * * @param {TokenType} type * @param {string} prefix @@ -86,8 +92,8 @@ declare class TokenSink { separate(first: string): void; /** * Keep the printed tokens apart from the tokens around them: `before` and - * `after` are the neighbours' adjacent characters. The printed tokens no - * longer match the source range once a separator is added. + * `after` are the neighbours' adjacent characters. Added separators are + * built tokens, so the output no longer matches the source range. * * @param {string | undefined} before * @param {string | undefined} after @@ -139,8 +145,8 @@ declare class TokenSink { numeric(type: TokenType, text: string, unit: string): void; /** * The raw tokens `list[start, end)` of an opaque component. Source tokens - * are appended by reference, and they keep the cursor only when they start - * exactly where the printed tokens are. + * of the sink's own list are reused by index when the range starts at the + * expected index; any other token is kept by reference as a built token. * * @param {CSSToken[]} list * @param {number} start diff --git a/types/lib/serialize/text-sink.d.ts b/types/lib/serialize/text-sink.d.ts index d2e0d47..34fe782 100644 --- a/types/lib/serialize/text-sink.d.ts +++ b/types/lib/serialize/text-sink.d.ts @@ -1,55 +1,17 @@ export type CSSToken = import('@csstools/css-tokenizer').CSSToken; -/** - * Whether a token ending in `last` and a token starting with `first` would - * read back as one token if written without a separator. The check is - * conservative: it may ask for a separator that is not strictly needed. - * - * @param {string | undefined} last - * @param {string | undefined} first - * @return {boolean} - */ -declare function wouldFuse(last: string | undefined, first: string | undefined): boolean; -/** - * What keeps a replacement apart from a neighbour it would fuse with. A space - * rather than the empty comment CSS Syntax 3 §9 allows, because tools that - * strip comments from values (such as minifiers run after this plugin) would - * fuse the tokens again. - */ -declare const FUSE_SEPARATOR = " "; -/** - * Whether `token` is an end-of-file token. It marks the end of a tokenized - * string, so one inside a list (for example from concatenated `tokenize` - * results) is not part of the CSS and is skipped. - * - * @param {CSSToken | undefined} token - * @return {boolean} - */ -declare function isEOF(token: CSSToken | undefined): boolean; -/** - * The first character of the first non-EOF token in `list[start, end)`. - * - * @param {CSSToken[]} list - * @param {number} start - * @param {number} end - * @return {string | undefined} - */ -declare function firstCharOf(list: CSSToken[], start: number, end: number): string | undefined; -/** - * The last character of the last non-EOF token in `list[0, end)`. - * - * @param {CSSToken[]} list - * @param {number} end - * @return {string | undefined} - */ -declare function lastCharBefore(list: CSSToken[], end: number): string | undefined; +export type Sink = import('./sink.js').Sink; /** * The output of a printer that keeps its printed text as strings. + * + * @implements {Sink} */ -declare class TextSink { +declare class TextSink implements Sink { /** @type {string[]} */ parts: string[]; /** @type {boolean} */ signed: boolean; + /** Text written before `parts`, set by `fit` to avoid shifting them. @type {string} */ + leading: string; /** * Write a space when a token starting with `first` would fuse with the * text before it. A sign belongs to the token that follows it. @@ -94,7 +56,16 @@ declare class TextSink { * @return {void} */ source(list: CSSToken[], start: number, end: number): void; + /** + * Keep the printed text apart from the text around it: `before` and `after` + * are the neighbours' adjacent characters. + * + * @param {string | undefined} before + * @param {string | undefined} after + * @return {void} + */ + fit(before: string | undefined, after: string | undefined): void; /** @return {string} */ join(): string; } -export { FUSE_SEPARATOR, TextSink, firstCharOf, isEOF, lastCharBefore, wouldFuse, }; +export { TextSink }; diff --git a/types/lib/serialize/token-values.d.ts b/types/lib/serialize/token-values.d.ts new file mode 100644 index 0000000..69988e3 --- /dev/null +++ b/types/lib/serialize/token-values.d.ts @@ -0,0 +1,40 @@ +import { TokenType } from '@csstools/css-tokenizer'; +export type CSSToken = import('@csstools/css-tokenizer').CSSToken; +/** @typedef {import('@csstools/css-tokenizer').CSSToken} CSSToken */ +/** + * Whether `raw` is `prefix + first + last`, without building that string. + * + * @param {string} raw + * @param {string} prefix + * @param {string} first + * @param {string} last + * @return {boolean} + */ +declare function sameText(raw: string, prefix: string, first: string, last: string): boolean; +/** + * The value of a CSS identifier from its raw text: each escape is replaced by + * the code point it names, so `c\61 lc` reads `calc`. + * + * @param {string} raw + * @return {string} + */ +declare function decodeIdentifier(raw: string): string; +/** + * The detail object of a numeric token, as the tokenizer makes it. + * + * @param {TokenType} type + * @param {string} number The signed number text. + * @param {string} unit + * @return {object} + */ +declare function numericDetail(type: TokenType, number: string, unit: string): object; +/** + * A printed token. Its offsets are unspecified. + * + * @param {TokenType} type + * @param {string} raw + * @param {object | null | undefined} detail + * @return {CSSToken} + */ +declare function newToken(type: TokenType, raw: string, detail: object | null | undefined): CSSToken; +export { decodeIdentifier, newToken, numericDetail, sameText }; diff --git a/types/reduce-tokens.d.ts b/types/reduce-tokens.d.ts index c4dbec4..f0484ec 100644 --- a/types/reduce-tokens.d.ts +++ b/types/reduce-tokens.d.ts @@ -1,7 +1,8 @@ import { hasPotentialMathFunction } from './lib/functions.js'; export type ReduceCalcOptions = import('./reduce.js').ReduceCalcOptions; +export type ReduceDiagnostics = import('./reduce.js').ReduceDiagnostics; +export type ResolvedReduceCalcOptions = import('./reduce.js').ResolvedReduceCalcOptions; export type CSSToken = import('@csstools/css-tokenizer').CSSToken; -export type TokenSink = import('./lib/serialize/sink.js').TokenSink; export type TokenReplacement = { /** * Index of the first replaced token. @@ -14,28 +15,41 @@ export type TokenReplacement = { tokens: CSSToken[]; }; export type WarnTarget = { - warn: (message: string, plugin?: string) => void; + warn: (message: string, opts?: { + plugin?: string; + }) => void; }; export type TokenReducer = (tokens: CSSToken[], target: WarnTarget, start?: number, end?: number) => TokenReplacement[]; /** * Build a reducer that resolves `opts` once, for a caller that reduces many - * token lists with the same options. Each call sends its warnings to the - * target it is given, unless `opts` has `onWarn`, which takes precedence as - * `onParseError` does. + * token lists with the same options. + * + * Each call sends its warnings and parse errors to the `target` it is given. + * An `onWarn` or `onParseError` option takes precedence over the target. * * @param {ReduceCalcOptions} [opts] * @return {TokenReducer} */ declare function createReducer(opts?: ReduceCalcOptions): TokenReducer; /** - * Simplify every supported CSS math function in `tokens[start, end)`. Returns - * the replacements that change the text, ordered and non-overlapping, and - * leaves `tokens` untouched. Apply them from last to first, or build a new - * list. Replacement indices are indices into `tokens`, and a range reads as - * the list it names would. The offsets of a replacement's tokens locate them - * in the replacement's own text, counting from 0. An empty or inverted - * range has no replacements. Throws a `RangeError` for a range that is not - * within `tokens`. + * Simplify every supported CSS math function in `tokens[start, end)`. + * + * Returns the replacements that change the text, ordered and non-overlapping. + * `tokens` is left untouched: apply the replacements from last to first, or + * build a new list. + * + * Replacement indices are indices into `tokens`. A range reads as the list it + * names would, with one exception: the token on each side of a replacement is + * read even when it lies outside the range, because a replacement that would + * fuse with it starts or ends with a whitespace token. EOF tokens in the list + * are skipped. + * + * The offsets of a replacement's tokens locate them in the replacement's own + * text, counting from 0. An empty or inverted range has no replacements. + * + * Throws a `RangeError` for a range that is not within `tokens`. A parse error + * is dropped unless `opts.onParseError` is set, as the standalone reducer drops + * warnings by default. `createReducer` sends it to its target instead. * * @param {CSSToken[]} tokens * @param {ReduceCalcOptions} [opts] diff --git a/types/reduce.d.ts b/types/reduce.d.ts index a8b3dcb..8951361 100644 --- a/types/reduce.d.ts +++ b/types/reduce.d.ts @@ -18,7 +18,9 @@ export type ReduceCalcOptions = { */ onWarn?: (message: string) => void; }; -export type ResolvedReduceCalcOptions = Required> & Pick; +export type ReduceDiagnostics = Pick; +export type ResolvedReduceCalcOptions = Required>; +export type TextSink = import('./lib/serialize/text-sink.js').TextSink; /** * @typedef {object} ReduceCalcOptions * @property {number | false} [precision] @@ -28,7 +30,14 @@ export type ResolvedReduceCalcOptions = Required void} [onParseError] Invoked when parse/simplify throws. * @property {(message: string) => void} [onWarn] Invoked when `warnWhenCannotResolve` is set and an expression cannot be reduced to a single value. */ -/** @typedef {Required> & Pick} ResolvedReduceCalcOptions */ +/** + * The callbacks of `ReduceCalcOptions`. A reducer call receives them apart from + * the options that shape the result. + * + * @typedef {Pick} ReduceDiagnostics + */ +/** @typedef {Required>} ResolvedReduceCalcOptions */ +/** @typedef {import('./lib/serialize/text-sink.js').TextSink} TextSink */ /** * Simplify every supported CSS math function in a component-value string. * Text outside those functions is preserved byte-for-byte. From 891eabc307b2596523406319b9e35ddcc9f5ad8d Mon Sep 17 00:00:00 2001 From: Ludovico Fischer Date: Sat, 14 Nov 2026 17:30:24 +0100 Subject: [PATCH 3/4] fix: improve output for math functions and cases involving infinity --- CHANGELOG.md | 16 ++++++- src/lib/parser.js | 24 +++++++++- src/lib/serialize.js | 14 ++++-- src/lib/serialize/fuse.js | 2 + src/lib/simplify/product.js | 5 +- test/unit/regressions-output.test.js | 68 ++++++++++++++++++++++++++++ test/unit/serialize-result.test.js | 2 +- 7 files changed, 124 insertions(+), 7 deletions(-) create mode 100644 test/unit/regressions-output.test.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 3368bb9..f7a01e8 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,21 @@ All notable changes to this project will be documented in this file. See [commit - A reduced value that would merge with a neighbouring token is now kept apart by a space. For example, `foo(calc(1px + 1px)b)` becomes `foo(2px b)` instead - of `foo(2pxb)`, which reads as a different unit. + of `foo(2pxb)`, which reads as a different unit. This now also applies after + a `+`: `a+calc(25% + 25%)` with `unwrapSingleValue` becomes `a+ 50%` instead + of `a+50%`, which reads as a signed number. +- A bare `clamp()`, `min()`, `max()`, `round()` or other math function that + cannot be folded is no longer wrapped in `calc()`. For example, + `clamp(1rem, 2.5vw, 2rem)` stays as written instead of becoming + `calc(clamp(1rem, 2.5vw, 2rem))`. +- `unwrapSingleValue` no longer removes `calc()` from opaque values. + `calc(var(--a))`, `calc(env(x))` and `calc(attr(x px))` keep their `calc()`, + as the option only applies to fully resolved finite scalars. +- Division by zero is no longer distributed over a sum with mixed units. + `calc((2em - 2px) / 0)` becomes `calc(infinity * (2em - 2px))` instead of a + sum that is always `NaN`. +- A literal `-0` divisor keeps its sign: `calc(1 / -0)` now gives + `calc(-infinity)` instead of `calc(infinity)`. - `reduceCalc` and the plugin no longer report the nesting-limit error for a value nested more than 1024 levels deep that has no supported math function outside `url()`, a string or a comment, such as `url(calc(1+2))((((…`. The diff --git a/src/lib/parser.js b/src/lib/parser.js index 22a6114..8ce1819 100644 --- a/src/lib/parser.js +++ b/src/lib/parser.js @@ -61,6 +61,25 @@ function normalizeSourceZero(value) { return value === 0 ? 0 : value; } +/** + * A literal `-0` divisor keeps its sign: `1 / -0` is `-infinity`, while the + * sign of a zero anywhere else cannot change a result and stays normalized. + * + * @param {ParseState} state + * @param {number} index Token index of the operand. + * @param {Node} node + * @return {Node} + */ +function keepDivisorSign(state, index, node) { + const token = state.tokens[index]; + return node.type === 'Num' && + token !== undefined && + token[0] === CssType.Number && + Object.is(token[4].value, -0) + ? num(-0) + : node; +} + /** * The delimiter at `index` when it is `a` or `b`, else undefined. * @@ -117,9 +136,12 @@ function parseProduct(state, depth) { const op = delimAt(state, i, '*', '/'); if (op === undefined) break; state.pos = i + 1; + const operandIndex = peek(state); + const operand = parseOperand(state, depth); factors.push({ exponent: op === '*' ? 1 : -1, - node: parseOperand(state, depth), + node: + op === '/' ? keepDivisorSign(state, operandIndex, operand) : operand, }); } return factors.length === 1 ? first : mkProduct(factors); diff --git a/src/lib/serialize.js b/src/lib/serialize.js index 5be2781..a6e067f 100644 --- a/src/lib/serialize.js +++ b/src/lib/serialize.js @@ -3,7 +3,7 @@ // operator, or when a finite scalar needs context-sensitive CSS semantics. import { checkCalculationDepth } from './limits.js'; -import { isCalculationFunction } from './functions.js'; +import { isCalculationFunction, lookupMathFunction } from './functions.js'; import { isScalar } from './serialize/precision.js'; import { TextSink } from './serialize/sink.js'; import { @@ -91,8 +91,16 @@ function printResult(result, opts, out) { const wrapper = isCalc ? result.rootSpelling || opts.calcName || 'calc' : 'calc'; - // Under the standard policy a bare Ident/Call result still needs calc(). - if (session.scalarPolicy === 'standard' && !isScalar(node)) { + // A bare Ident/Call result still needs calc() under every policy, unless + // it is a math function (min(), clamp(), ...) that is valid on its own. + if ( + !isScalar(node) && + !( + !isCalc && + node.type === 'Call' && + lookupMathFunction(node.name) !== undefined + ) + ) { emitWrappedRoot(node, session, wrapper); } else { emitMathResult(node, session, wrapper); diff --git a/src/lib/serialize/fuse.js b/src/lib/serialize/fuse.js index f00afec..f36c5a1 100644 --- a/src/lib/serialize/fuse.js +++ b/src/lib/serialize/fuse.js @@ -30,6 +30,8 @@ function wouldFuse(last, first) { // A digit continues into `%`, and a backslash starts an escape that // continues the name or unit before it. if (first === '%') return DIGIT.test(last); + // A `+` sign continues into a following number: `a+` then `50%` reads `+50%`. + if (last === '+') return DIGIT.test(first) || first === '.'; return ( NAME_CHAR.test(last) && (NAME_CHAR.test(first) || first === '(' || first === '\\') diff --git a/src/lib/simplify/product.js b/src/lib/simplify/product.js index 79192b5..be60a25 100644 --- a/src/lib/simplify/product.js +++ b/src/lib/simplify/product.js @@ -83,7 +83,10 @@ function foldFactors(items, start, end, simplify, precision) { ? remainingDims[0] : null; const value = loneDim === null ? coeff : chainValue(scalarChain); - const distributable = remainingDims.length === 0 && isScalarSum(opaque); + // A non-finite coefficient (division by zero) must not be spread over the + // terms: `infinity * 1em + -infinity * 1px` is NaN, not the sign of the sum. + const distributable = + remainingDims.length === 0 && Number.isFinite(coeff) && isScalarSum(opaque); // An inexact quotient stays symbolic when its numerator and denominator // print exactly; otherwise it is folded and rounded at serialization. diff --git a/test/unit/regressions-output.test.js b/test/unit/regressions-output.test.js new file mode 100644 index 0000000..4af5e74 --- /dev/null +++ b/test/unit/regressions-output.test.js @@ -0,0 +1,68 @@ +import { describe, test } from 'node:test'; +import assert from 'node:assert/strict'; +import reduceCalc from '../../src/reduce.js'; + +const unwrap = { unwrapSingleValue: true }; + +describe('math function roots that cannot fold', () => { + for (const input of [ + 'clamp(1rem, 2.5vw, 2rem)', + 'min(100vw - 2rem, 60ch)', + 'max(1rem, 5%)', + 'round(1vw, 1px)', + ]) { + test(`${input} is not wrapped in calc()`, () => { + assert.equal(reduceCalc(input), input); + assert.equal(reduceCalc(input, unwrap), input); + }); + } + + test('an explicit calc() wrapper is kept', () => { + assert.equal( + reduceCalc('calc(clamp(1rem, 2.5vw, 2rem))'), + 'calc(clamp(1rem, 2.5vw, 2rem))' + ); + }); + + test('a function that folds to a scalar still gets calc()', () => { + assert.equal(reduceCalc('abs(1vw)'), 'calc(1vw)'); + }); +}); + +describe('unwrapSingleValue and opaque values', () => { + for (const input of ['calc(var(--a))', 'calc(env(x))', 'calc(attr(x px))']) { + test(`${input} keeps calc()`, () => { + assert.equal(reduceCalc(input, unwrap), input); + }); + } +}); + +describe('division by zero', () => { + test('is not distributed over a mixed-unit sum', () => { + assert.equal( + reduceCalc('calc((2em - 2px) / 0)'), + 'calc(infinity * (2em - 2px))' + ); + }); + + test('a non-finite factor keeps an opaque sum grouped', () => { + assert.equal( + reduceCalc('calc((1px + var(--a)) * (1 / 0))'), + 'calc(infinity * (1px + var(--a)))' + ); + }); + + test('a literal negative zero divisor keeps its sign', () => { + assert.equal(reduceCalc('calc(1 / -0)'), 'calc(-infinity)'); + assert.equal(reduceCalc('calc(1 / 0)'), 'calc(infinity)'); + assert.equal(reduceCalc('calc(-1 / -0)'), 'calc(infinity)'); + assert.equal(reduceCalc('calc(1px / -0)'), 'calc(-infinity * 1px)'); + }); +}); + +describe('unwrapped result after a plus sign', () => { + test('does not fuse into a signed number', () => { + assert.equal(reduceCalc('a+calc(25% + 25%)', unwrap), 'a+ 50%'); + assert.equal(reduceCalc('a+calc(1 / 2)', unwrap), 'a+ .5'); + }); +}); diff --git a/test/unit/serialize-result.test.js b/test/unit/serialize-result.test.js index 0502f9d..2e01d73 100644 --- a/test/unit/serialize-result.test.js +++ b/test/unit/serialize-result.test.js @@ -183,7 +183,7 @@ describe('serializeResult: root planning', () => { assert.equal(serializeResult(result), 'calc(var(--x, calc(1px + 2px)))'); assert.equal( serializeResult(result, { unwrapSingleValue: true }), - 'var(--x, calc(1px + 2px))' + 'calc(var(--x, calc(1px + 2px)))' ); }); From 26b47d37a73b7460ab75a2d6501ccafaca8dd437 Mon Sep 17 00:00:00 2001 From: Ludovico Fischer Date: Sat, 14 Nov 2026 17:39:04 +0100 Subject: [PATCH 4/4] docs: improve tokens API description --- README.md | 63 +++++++++++++++++++++++++++---------------------------- 1 file changed, 31 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index de14ef5..b70e369 100755 --- a/README.md +++ b/README.md @@ -87,21 +87,15 @@ const result = reduceCalc('calc(100% + var(--gap))', { Unlike the PostCSS plugin, the standalone reducer does not show warnings by default; provide `onParseError` and/or `onWarn` if you want diagnostics. -### Reduce a token list +### Reduce a token list (experimental) -If you already hold the tokens of a value (from `@csstools/css-tokenizer`) and -edit them in place, `postcss-calc/reduce-tokens` reduces them without going -through a string. It reads only each token's type and raw text, never its -source offsets, so the list may have been edited since it was tokenized. +> **Experimental:** this API may change in a minor release. -Its other fields must agree: a token's parsed value (`token[4]`) is read along -with its type and raw text, so when you change a token's text, replace the -token with one that `tokenize` produced from the new text. Only the source -offsets may be stale. - -It returns the replacements that change the text: ordered, non-overlapping -`{ start, end, tokens }` records that replace `tokens.slice(start, end)`. It -does not modify the list; apply the replacements from last to first. +`postcss-calc/reduce-tokens` reduces a list of `@csstools/css-tokenizer` +tokens directly, without converting to a string. It returns ordered, +non-overlapping `{ start, end, tokens }` replacements for +`tokens.slice(start, end)` and does not modify the list. Apply them from last +to first: ```js import { tokenize } from '@csstools/css-tokenizer'; @@ -114,18 +108,30 @@ for (const { start, end, tokens: next } of reduceTokens(tokens).toReversed()) { // tokens now spell 'calc(3px) 3px' ``` -The replacement tokens are the tokenization of the result text on its own, so -their source offsets (`token[2]`, `token[3]`) describe that text, not their -place in your list. Do not rely on them. A replacement that would merge with -the token beside it in `tokens` starts or ends with a whitespace token. -For a sub-range, this reads the tokens just outside it, so splice -the replacements into the same list you passed. - -To skip tokenizing values that cannot contain a math function, check the -string first with the named export `hasPotentialMathFunction(value)`. It is -the quick test `reduceCalc` runs before tokenizing. It may return `true` for a -value with no reducible function, never `false` for one with a reducible -function. +`reduceTokens(tokens, options, start, end)` takes the same options as +`reduceCalc`, plus an optional index range to reduce. Things to know: + +- **Edited lists are fine.** Source offsets (`token[2]`, `token[3]`) are + ignored, but the type, raw text, and parsed value (`token[4]`) must agree: + if you change a token's text, replace it with a token `tokenize` produced + from the new text. +- **Replacement offsets are local.** Replacement tokens are tokenized from the + replacement text alone, so their offsets count from 0 within that text, not + from their place in your list. +- **Replacements may add whitespace.** When a replacement would fuse with the + token before or after it, it starts or ends with a whitespace token. For + example, with `unwrapSingleValue: true`, `calc(1px)a` becomes `1px a` + rather than the single token `1pxa`. The tokens checked are the ones + next to the replacement in `tokens`, even outside `start`/`end`, so splice + the replacements into the same list you passed. +- **EOF tokens are ignored.** They never appear in a replacement and are + skipped when finding the token next to one. +- **Diagnostics** passed to `onParseError` and `onWarn` use text built from + the current tokens. + +To skip tokenizing values that cannot contain a math function, check them +first with `hasPotentialMathFunction(value)`. It may return `true` for a value +with nothing to reduce, but never `false` for one that has something. ```js import { hasPotentialMathFunction } from 'postcss-calc/reduce-tokens'; @@ -135,13 +141,6 @@ if (hasPotentialMathFunction(value)) { } ``` -End-of-file tokens in the list (for example from concatenating `tokenize` -results) are ignored: they never appear in a replacement, and they do not hide -the neighbouring token from that check. - -It takes the same options as `reduceCalc`. `onParseError` and `onWarn` receive -text built from the current tokens. - ### Standalone reducer options #### `unwrapSingleValue` (default: `false`)