diff --git a/docs/safecss-filter-attr-token-processor-spec.md b/docs/safecss-filter-attr-token-processor-spec.md new file mode 100644 index 0000000000000..4afa10ea3dab6 --- /dev/null +++ b/docs/safecss-filter-attr-token-processor-spec.md @@ -0,0 +1,251 @@ +# safecss_filter_attr() Token Processor Re-implementation Specification + +## Purpose + +Re-implement `safecss_filter_attr()` in `src/wp-includes/kses.php` on top of +`WP_CSS_Token_Processor`, replacing the current `explode( ';' )` splitting and +regex-based validation with CSS Syntax Level 3 tokenization. + +The function's contract is unchanged: it receives the decoded declaration-list +contents of an HTML `style` attribute and returns a filtered declaration list, +needing HTML escaping before being written back into an attribute. + +Guiding principles, shared with the style attribute processor spec: + +1. follow CSS declaration-list semantics when discovering declarations; +2. preserve authored source text of accepted declarations; +3. validate structurally over decoded tokens, not over raw bytes; +4. when behavior must diverge from the current implementation, diverge only + toward strictness, except where an existing unit test pins the looser + behavior. + +The existing PHPUnit suites are the acceptance gate and must pass unchanged: + +- `data_safecss_filter_attr` +- `data_kses_style_attr_with_url` +- `data_safecss_filter_attr_filtered` +- `data_wp_kses_style_attr_decodes_entities_before_css_filtering` + +## What Is Removed + +- The `explode( ';', $css )` declaration splitting and its `@todo`. +- `preg_match_all( '/url\([^)]+\)/' )` and the url-pieces regex. +- The gradient matching regex. +- The recursive function-stripping regex for `var|calc|min|max|minmax|clamp|repeat`. +- The character denylist check `%[\\\(&=}]|/\*%` and the `$css_test_string` + munging that feeds it. Validation verdicts come from token walks instead. + +## What Is Preserved + +- `_deprecated_argument()` handling of the second parameter. +- Preprocessing: `wp_kses_no_null( $css )` followed by removal of `\n`, `\r`, + and `\t` bytes. This is required byte-for-byte by existing tests (for + example, `url(\n'…')` must serialize as `url('…' )`). The tokenizer therefore + never sees those bytes. +- The `safe_style_css` filter over the allowed property list, including the + `--*` custom property wildcard. +- The url-allowed property list (`background`, `background-image`, `cursor`, + `filter`, `list-style`, `list-style-image`) and gradient-allowed property + list (`background`, `background-image`). +- The early `return $css;` when the filtered allowed-property list is empty. +- The `safecss_filter_attr_allow_css` filter (see below). +- Output assembly: the trimmed authored source of each accepted declaration, + joined with `;`, no trailing semicolon. + +## Declaration Discovery + +A single `WP_CSS_Token_Processor` pass over the preprocessed CSS splits it into +declarations at top-level `semicolon-token`s. Nesting depth is tracked by +counting `function-token`/`(-token`/`[-token`/`{-token` against +`)-token`/`]-token`/`}-token` (never below zero). Semicolons at depth greater +than zero do not split; semicolons inside strings, urls, and comments are +inherently part of those tokens and never seen as `semicolon-token`s. + +Each declaration is represented by its byte range in the preprocessed source. +Empty or whitespace-only segments are skipped. + +Divergence: a declaration left unclosed at EOF (for example +`width: calc(3em; height: 10px`) now swallows the remainder of the input, as +CSS specifies, instead of resurrecting later segments. The whole swallowed +declaration is then rejected by value validation. + +## Property Handling + +The property/value boundary is the first `colon-token` in the declaration +(matching the current `explode( ':', $css_item, 2 )` since a colon cannot +legally occur earlier). The property name is the trimmed raw source before it. + +- Ordinary properties: exact, case-sensitive string match against the allowed + list (`Text-transform` remains rejected). +- Custom properties: raw source must match `/^--[a-zA-Z0-9-_]+$/`; casing is + preserved and semantic. +- Escaped property names (`\63 olor`) do not match and are rejected, since + matching is on raw source. + +A declaration whose property is neither allowed nor a valid custom property is +a **hard rejection**: it is dropped without consulting the +`safecss_filter_attr_allow_css` filter, exactly as today. + +Declarations containing no colon at all keep today's behavior: they undergo +value validation as a bare value (passthrough functions allowed, url and +gradient constructs not allowed) and are kept if they validate, with the +filter consulted. + +## Value Validation + +The declaration's value token stream (everything after the first colon) is +walked once. Validation produces either a **hard rejection** (declaration +dropped, filter not consulted), or a boolean verdict `$allow_css` passed +through the `safecss_filter_attr_allow_css` filter. + +### URL constructs + +A url construct is either a `url-token` (`url(unquoted)`) or a +`function-token` named `url` (ASCII case-insensitive) containing optional +whitespace and exactly one `string-token` before its closing `)-token`. + +Every url construct, at any nesting depth and for any property, has its +decoded value trimmed and checked: + +- empty value, or `wp_kses_bad_protocol()` changing the value → + **hard rejection** of the declaration. This closes the current nesting + loophole where `width: calc(url(javascript:…))` escapes protocol checking. +- valid value → the construct is acceptable only where url constructs are + allowed: values of url-allowed properties and custom properties, at the top + level or nested inside allowed functions. A valid url construct on a + non-url-allowed ordinary property is a soft rejection (`$allow_css = false`), + preserving today's filter-overridable outcome for `color: url(…)`. +- structurally malformed url constructs (a `bad-url-token`, or a `url(` + function that does not close with `)` after its string) are soft rejections. + Even so, an identifiable url payload inside the malformed construct — the + string argument of a malformed `url(` function, or the raw inner text of a + `bad-url-token` (its raw source, stripped of the leading `url(` and a + trailing `)` if present, then trimmed) — is still protocol-checked: an empty + string-argument payload, or either payload failing + `wp_kses_bad_protocol()`, is a **hard rejection**, escalating what would + otherwise be a soft, filter-rescuable outcome. This is a divergence toward + strictness: the old regex-based implementation hard-rejected bad-protocol + payloads in some of these malformed shapes and accepted or soft-rejected + them in others; the token-based implementation now hard-rejects + consistently whenever the payload is identifiable. + +CSS escapes are permitted inside url construct values; the protocol check runs +on the decoded value. + +Divergence: url constructs are recognized ASCII case-insensitively (`URL(…)`), +as CSS specifies. The old lowercase-only regex rejected uppercase variants +outright; they are now accepted when the protocol check passes. + +### Gradient functions + +`function-token`s named `(repeating-)?(linear|radial|conic)-gradient` +(case-sensitive, as today) are allowed in values of gradient-allowed +properties and custom properties. Inside a gradient, in addition to ordinary +component tokens, one level of balanced group nesting is allowed: nested +`function-token`s of any name or bare `(-token` groups, which must close and +must not contain further parenthesis nesting. Url constructs inside gradients +follow the url rules above. A gradient violating these rules, or a gradient +function on a non-gradient property, is a soft rejection. + +Divergence: string tokens containing parentheses inside gradients are now +allowed (strings are atomic tokens); the old regex rejected them. + +### Passthrough functions + +`function-token`s named exactly `var`, `calc`, `min`, `max`, `minmax`, +`clamp`, or `repeat` (case-sensitive, as today) are allowed in any value. +Their contents are permissive, matching today's balanced-anything stripping: +any tokens are accepted inside, at any depth, subject only to: + +- the function (and every nested group) must close before the declaration + ends; unclosed-at-EOF constructs are soft rejections; +- url constructs inside are still subject to the url rules above; +- `bad-string-token` and `bad-url-token` are soft rejections. + +### Top-level token rules + +Outside url constructs, gradients, and passthrough functions, the following +tokens are accepted: `whitespace`, `ident` (without escapes), `number`, +`percentage`, `dimension` (without escapes), `string` (without escapes, and +closed before end of input — with newlines removed during preprocessing, an +unclosed string reaches end of input as a regular string token rather than a +bad-string token, so closure is checked explicitly), `hash`, `comma`, `colon`, +`[-token`, `]-token`, `)-token` (stray closing +parens remain accepted — a pinned quirk), and `delim-token`s other than `\`, +`&`, and `=`. The delim `!` remains accepted, which keeps `!important` +working. + +The following are soft rejections: `comment` tokens, `bad-string-token`, +`bad-url-token`, `{-token`, `}-token`, bare `(-token` groups, +`function-token`s not covered by the allowlists above, `at-keyword-token`, +`CDO-token`, `CDC-token`, and any ident/dimension/string token whose raw +source contains a backslash (escapes stay rejected outside url values, so +`margin-top: \2px` remains rejected and remains rescuable by the filter). + +Divergences at this layer, all toward strictness: `{`, at-keywords, CDO/CDC, +comments nested inside formerly-stripped function spans, bad strings, and bad +urls were previously invisible to the char check in some positions and now +soft-reject. + +## The safecss_filter_attr_allow_css Filter + +The filter is retained. `$allow_css` is now the verdict of token-based +validation instead of the char-check regex. The second argument, +`$css_test_string`, becomes the declaration's trimmed authored source; the +previous value was the declaration with allowed function spans textually +removed, which no longer exists in this design. This is a documented change to +the filter's second argument. + +Hard rejections (unknown property, invalid custom property name, empty or +bad-protocol url values) are dropped before the filter runs, exactly matching +today's `$found = false` paths. Everything else reaches the filter and can be +rescued (`margin-top: \2px`, `height: expression(…)`, `color: rgb(…)`, +`margin-bottom: 2px}` all remain rescuable). + +## Output + +Accepted declarations are emitted as their trimmed authored source bytes from +the preprocessed input (after null/`\n\r\t` removal), joined with `;`. No +normalization, escape decoding, or whitespace collapsing is applied to output. + +## Implementation Shape + +All logic lives in `kses.php`: + +- `safecss_filter_attr()` keeps its signature and orchestrates preprocessing, + splitting, per-declaration validation, the filter, and output assembly. +- Private helper functions (prefixed `_safecss_`, marked `@access private`) + implement declaration splitting and value validation over token streams. The + token stream for a declaration is materialized as a small array of token + records (type, name/value where relevant, byte range) collected during the + single tokenizer pass, so the value walk does not re-tokenize. + +No changes to `WP_CSS_Token_Processor` are expected; if a gap is found (for +example, needing the raw source of a token), prefer using existing getters +(`get_token_start()`, `get_token_length()`, `get_token_value()`, +`get_unnormalized_token()`). + +## Test Requirements + +Existing suites pass unchanged. New cases to add to `data_safecss_filter_attr` +(and the filtered variant where noted): + +- semicolon inside a quoted string no longer splits the declaration; +- semicolon inside `url()` no longer splits the declaration; +- nested bad-protocol url is rejected: `width: calc(url(javascript:alert(1)))`; +- nested valid url on a url-allowed property passes protocol checking; +- unclosed function swallows following declarations (whole tail rejected); +- `url( )` with only whitespace is rejected; +- quoted url function form `url( "…" )` with bad protocol is rejected and is + not rescuable by the filter; +- a string left unclosed at the end of input is rejected; +- `{` in a value is rejected; +- comment in a value is rejected; +- filter receives the declaration source as `$css_test_string` and can rescue + soft rejections but not hard rejections; +- a malformed url function with a bad-protocol string is rejected and is not + rescuable by the filter; +- a bad-url-token with a bad-protocol payload is rejected and is not + rescuable by the filter; +- passthrough and gradient function names match case-sensitively (`CALC(` is + rejected). diff --git a/docs/wp-html-style-attribute-processor-spec.md b/docs/wp-html-style-attribute-processor-spec.md new file mode 100644 index 0000000000000..3728c39f520e0 --- /dev/null +++ b/docs/wp-html-style-attribute-processor-spec.md @@ -0,0 +1,284 @@ +# WP_HTML_Style_Attribute_Processor Specification + +## Purpose + +`WP_HTML_Style_Attribute_Processor` inspects and modifies decoded CSS text from an +HTML `style` attribute. Its input is the declaration-list contents of the +attribute, excluding HTML attribute syntax, entity decoding, and any surrounding +declaration-block braces. + +The processor preserves the CSS declaration-list model. Duplicate declarations +remain distinct. Invalid CSS fragments are skipped or preserved according to CSS +parser semantics unless a requested mutation cannot be proven safe. + +The processor's north star is: + +1. preserve CSS structure; +2. preserve user intent; +3. preserve the semantic CSS value of any existing declaration it rewrites; +4. make minimal edits when doing so does not compromise the first three goals. + +## Construction + +Creation uses: + +```php +WP_HTML_Style_Attribute_Processor::create( $decoded_css_text ) +``` + +The constructor is private. `create()` accepts only string input and returns a +processor instance of the called class. The implementation must remain +compatible with WordPress' supported PHP versions, so the runtime signature may +use PHPDoc and explicit validation instead of PHP syntax unavailable in those +versions. + +Callers must not pass raw `WP_HTML_Tag_Processor::get_attribute( 'style' )` +results directly unless they have already checked that the result is a string. +Missing or boolean attributes are an HTML-layer concern, not accepted CSS text. + +Parsing is lazy. Creating a processor does not scan the style text until cursor +movement, mutation validation, or serialization requires it. + +## Declaration Model + +The processor models each CSS declaration as: + +```text +property-name: component-value-list !important? +``` + +The internal declaration data consists of: + +- decoded property name; +- authored property-name source when available; +- value component list source range; +- `!important` flag; +- nullable source range for the parsed priority span; +- source ranges needed for safe mutation and verification. + +`!important` is not part of the value, including for custom properties. + +## Public API + +Initial public methods: + +```php +public static function create( $decoded_css_text ); +public function next_declaration( ?string $property_name = null ): bool; +public function get_property_name(): ?string; +public function is_important(): ?bool; +public function set_value( string $value, ?bool $important = null ): bool; +public function set_important( bool $important ): bool; +public function remove_declaration(): bool; +public function append_declaration( string $property_name, string $value, bool $important = false ): bool; +public function get_updated_style(): string; +``` + +The first API does not expose `get_value()` or `get_raw_value()`. Raw CSS values +are difficult for callers to use safely, and a higher-level value API needs a +separate design. + +Bookmarks, seek, and rewind are out of scope. Callers that need to rescan should +create a new processor from the updated style text. + +## Cursor Semantics + +`next_declaration()` advances a forward-only cursor over CSS declarations. + +All current-declaration getters return `null` when the cursor is not positioned +on a valid declaration. This includes `is_important()`. + +After `remove_declaration()` succeeds, the cursor becomes invalid until +`next_declaration()` advances it. A second removal on the same invalid cursor +returns `false`. + +After `set_value()` or `set_important()` succeeds without removing the +declaration, the cursor remains on the same logical declaration. + +`set_value( '' )` removes the current declaration. On success it has the same +cursor semantics as `remove_declaration()`. + +`append_declaration()` does not move the cursor. If the cursor was exhausted, it +remains invalid until a later `next_declaration()` advances to the appended +declaration. + +`next_declaration()` flushes pending safe edits before advancing because +advancement must reflect the updated declaration list. + +Current-position getters should reflect successful pending edits without forcing +a full reparse where practical, following HTML API style. + +## Property Names + +All public property-name arguments are decoded/plaintext CSS property names, not +CSS-escaped identifier source. + +Ordinary properties match ASCII case-insensitively. Custom properties match +exactly because casing is semantic. + +`get_property_name()` exposes CSS semantics: + +- ordinary property names are returned lowercase; +- custom property names preserve exact decoded casing. + +Edits preserve authored property-name spelling when the declaration already +exists. Appended ordinary properties serialize lowercase because there is no +authored spelling to preserve. Appended custom properties preserve exact casing. + +Appended property names must be valid plaintext CSS property names. The processor +does not escape arbitrary strings into identifiers. Custom properties follow the +same validity rule with the required `--` prefix. + +## Value Validation + +Mutation inputs accept CSS declaration values, not declaration-list text. + +The supplied value must fit safely in both of these declaration slots, depending +on the property being changed: + +```css +foo: ; +--foo: ; +``` + +Validation checks structure and CSS syntax safety, not property-specific browser +support. The processor should not maintain a browser-style matrix of supported +properties and value grammars. + +Values must be complete and self-contained. Reject inputs that: + +- are empty after CSS whitespace trimming, except `set_value( '' )`, which + removes the current declaration; +- contain top-level declaration separators such as semicolons; +- contain a top-level `!important`, because priority is a separate argument; +- contain bad strings or bad URLs; +- contain unmatched delimiters or constructs that are valid only because CSS + closes them at EOF; +- cannot be proven to occupy only the declaration value slot. + +Nested semicolons and `!important` text inside balanced blocks, functions, +strings, or URLs are allowed when the tokenizer and parser say they are part of +the value. + +## Importance + +`set_value( $value, null )` preserves the existing important flag. +`set_value( $value, true )` sets it. +`set_value( $value, false )` clears it. + +`append_declaration()` receives importance as a separate boolean and rejects a +top-level `!important` in `$value`. + +`set_important()` changes only declaration priority. + +When clearing importance, the processor may remove only the parsed priority span +when that is safe. If minimal removal is unsafe, it may normalize the current +declaration as long as it preserves CSS structure and the existing value's +semantics. If that cannot be proven, it returns `false`. + +When setting importance, the processor adds canonical ` !important` at the +parsed value end or normalizes the declaration if necessary. It must verify that +the updated declaration reparses as important. + +## Mutation Atomicity And Verification + +Invalid or inapplicable mutations return `false` without `_doing_it_wrong()`. + +Failed mutations are atomic: style text, cursor state, and previous successful +edits are preserved. + +A mutation returning `true` guarantees that `get_updated_style()` includes it. + +Every successful mutation must preserve the expected declaration-list structure. +`set_value()` must not let an input value merge, split, hide, or otherwise change +declarations beyond the intended current declaration. + +The public contract describes guarantees, not the exact validation mechanism. +Synthetic declarations, parser comparisons, token-level checks, and reparsing are +implementation details. + +Multiple mutations to the same logical declaration collapse to the final +intended state. Mutations to different declarations in one pass are allowed when +each mutation can be independently verified against the updated declaration +list. + +## Formatting And Normalization + +The processor preserves surrounding text, comments, ignored invalid fragments, +and authored spelling when cheap and safe. + +Minimal edits are a lower priority than safety, correctness, preserved CSS +structure, and semantic preservation. Any normalization required to satisfy +those higher priorities is allowed, but normalization that rewrites an existing +declaration value must be strict: it must serialize from parsed token/component +data with equivalent CSS semantics. If equivalence cannot be proven, the +mutation returns `false`. + +`set_value()` trims supplied values according to CSS whitespace. + +When possible, `set_value()` replaces the current declaration's value and +priority portion rather than normalizing the whole declaration. Correctness wins +over whitespace preservation. + +## Invalid Fragments + +Declaration discovery follows CSS semantics. If CSS would ignore text as a +declaration in a style attribute declaration-list context, the processor does +not expose it as a declaration. + +Ignored invalid fragments are preserved where possible. Mutations touching a +style containing invalid chunks must verify that the declaration-list structure +remains safe. If preservation and a requested mutation are incompatible, the +mutation returns `false` or normalizes only the range necessary to preserve +semantics and structure. + +## Append And EOF Repair + +`append_declaration()` may succeed even when trailing text is malformed, but only +when the processor can preserve existing CSS semantics. + +When trailing declaration text is valid only because CSS closes an unclosed +function or block at EOF, appending must materialize the missing closing +delimiters before inserting the new declaration. + +Example: + +```css +color: var(--x +``` + +Appending `background: white` should produce a structurally safe equivalent such +as: + +```css +color: var(--x); background: white; +``` + +EOF repair is part of a successful append and appears in `get_updated_style()`. +Repair should insert only the specific missing delimiters discovered from parser +or tokenizer state, plus the boundary needed before the appended declaration. If +the processor cannot determine a precise repair, append returns `false`. + +EOF repair is essential for append. `set_value()` replacement values must be +self-contained and do not receive EOF repair. + +## Test Requirements + +Tests should cover: + +- private construction and `create()` behavior; +- decoded string input boundary; +- declaration traversal, duplicate preservation, and property-name matching; +- getter `null` behavior off-cursor; +- absence of public raw value getter in the first API; +- `set_value()` priority preservation, setting, clearing, and empty-value + removal; +- `set_important()` setting, clearing, cursor behavior, and safe normalization; +- rejection of unsafe values and property names with atomic failure; +- append behavior, duplicate appends, exhausted cursor behavior, and empty-value + rejection; +- EOF repair for append and failure when repair cannot be precise; +- invalid fragments, comments, adjacent declarations, and removal ranges; +- preservation of authored spelling where required; +- integration with `WP_HTML_Tag_Processor::set_attribute()` after callers supply + decoded string input. diff --git a/src/wp-includes/css-api/class-wp-css-builder.php b/src/wp-includes/css-api/class-wp-css-builder.php new file mode 100644 index 0000000000000..35250335da661 --- /dev/null +++ b/src/wp-includes/css-api/class-wp-css-builder.php @@ -0,0 +1,261 @@ += 0x80 ) { + $result .= $value[ $i ]; + continue; + } + + // ASCII letters and underscore: always valid in idents. + if ( + ( $byte >= 0x41 && $byte <= 0x5A ) || // A-Z + ( $byte >= 0x61 && $byte <= 0x7A ) || // a-z + 0x5F === $byte // _ + ) { + $result .= $value[ $i ]; + continue; + } + + // Hyphen: valid in idents, but check for hyphen-digit at start. + if ( 0x2D === $byte ) { + // Hyphen at position 0 followed by a digit at position 1: escape the digit. + if ( 0 === $i && $i + 1 < $length && ord( $value[ $i + 1 ] ) >= 0x30 && ord( $value[ $i + 1 ] ) <= 0x39 ) { + $result .= '-'; + ++$i; + $result .= sprintf( '\\%X ', ord( $value[ $i ] ) ); + continue; + } + $result .= '-'; + continue; + } + + // Digits: valid except at position 0. + if ( $byte >= 0x30 && $byte <= 0x39 ) { + if ( 0 === $i ) { + $result .= sprintf( '\\%X ', $byte ); + } else { + $result .= $value[ $i ]; + } + continue; + } + + // Everything else: hex-escape. + $result .= sprintf( '\\%X ', $byte ); + } + + return $result; + } + + /** + * Create a quoted CSS string from a plain PHP string value. + * + * Example: + * $value = 'CSS & a ""; + * + * CSS strings are quoted many characters that are problematic in HTML + * or may be complicated for rudimentary CSS or HTML processors to handle + * are encoded using Unicode escape sequences. + * + * @see https://www.w3.org/TR/css-syntax-3/#escaping + */ + public static function string( string $value ): string { + $value = wp_scrub_utf8( $value ); + $escaped = strtr( + $value, + array( + // Escape existing backslashes to prevent unintentional escapes in result. + '\\' => '\\5C ', + + // Pre-processing replaces NULLs and some newlines. Replace and escape as necessary. + "\0" => "\u{FFFD}", + + // Normalize and replace newlines. https://www.w3.org/TR/css-syntax-3/#input-preprocessing + "\r\n" => '\\A ', + "\r" => '\\A ', + "\f" => '\\A ', + + // Newlines must be escaped in CSS strings. + "\n" => '\\A ', + + // Arbitrary characters for Unicode escaping: + + // HTML syntax may be problematic. + '<' => '\\3C ', + '>' => '\\3E ', + '&' => '\\26 ', + + // CSS syntax may be problematic. + ',' => '\\2C ', + ';' => '\\3B ', + '{' => '\\7B ', + '}' => '\\7D ', + '"' => '\\22 ', + "'" => '\\27 ', + ) + ); + return "\"{$escaped}\""; + } + + public static function normalize_and_escape_css( string $css ): string { + $css = wp_scrub_utf8( $css ); + $processor = WP_CSS_Token_Processor::create( $css ); + if ( null === $processor ) { + return ''; + } + + $normalized_css = ''; + + while ( $processor->next_token() ) { + switch ( $processor->get_token_type() ) { + + // Basic punctuation: + case WP_CSS_Token_Processor::TOKEN_SEMICOLON: + $normalized_css .= ';'; + break; + case WP_CSS_Token_Processor::TOKEN_COMMA: + $normalized_css .= ','; + break; + case WP_CSS_Token_Processor::TOKEN_WHITESPACE: + $normalized_css .= ' '; + break; + case WP_CSS_Token_Processor::TOKEN_COLON: + $normalized_css .= ':'; + break; + + // Paired punctuation: + case WP_CSS_Token_Processor::TOKEN_LEFT_BRACE: + $normalized_css .= '{'; + break; + case WP_CSS_Token_Processor::TOKEN_RIGHT_BRACE: + $normalized_css .= '}'; + break; + case WP_CSS_Token_Processor::TOKEN_LEFT_PAREN: + $normalized_css .= '('; + break; + case WP_CSS_Token_Processor::TOKEN_RIGHT_PAREN: + $normalized_css .= ')'; + break; + case WP_CSS_Token_Processor::TOKEN_LEFT_BRACKET: + $normalized_css .= '['; + break; + case WP_CSS_Token_Processor::TOKEN_RIGHT_BRACKET: + $normalized_css .= ']'; + break; + + // "@" + ident + case WP_CSS_Token_Processor::TOKEN_AT_KEYWORD: + $normalized_css .= '@' . self::ident( $processor->get_token_value() ); + break; + + // ident + "(" + case WP_CSS_Token_Processor::TOKEN_FUNCTION: + $normalized_css .= self::ident( $processor->get_token_value() ) . '('; + break; + + /* + * Hash tokens are not idents but their value can be escaped as such. + * + * ‖→ "#" →─┐ ┌──────────────────────────────┐ ┌─→‖ + * ├─→─┤ a-z A-Z 0-9 _ - or non-ASCII ├─→─┤ + * │ └──────────────────────────────┘ │ + * │ ┌──────────────────────────────┐ │ + * ├─→─┤ escape ├─→─┤ + * │ └──────────────────────────────┘ │ + * └──────────────────←───────────────────┘ + */ + case WP_CSS_Token_Processor::TOKEN_HASH: + $normalized_css .= '#' . self::ident( $processor->get_token_value() ); + break; + + case WP_CSS_Token_Processor::TOKEN_DIMENSION: + $normalized_css .= $processor->get_token_value() . $processor->get_token_unit(); + break; + + case WP_CSS_Token_Processor::TOKEN_PERCENTAGE: + $normalized_css .= "%{$processor->get_token_value()}"; + break; + + case WP_CSS_Token_Processor::TOKEN_NUMBER: + $normalized_css .= $processor->get_token_value(); + break; + + case WP_CSS_Token_Processor::TOKEN_DELIM: + $normalized_css .= $processor->get_token_value(); + break; + + case WP_CSS_Token_Processor::TOKEN_IDENT: + $normalized_css .= self::ident( $processor->get_token_value() ); + break; + + case WP_CSS_Token_Processor::TOKEN_STRING: + var_dump( $processor->get_token_value() ); + $normalized_css .= self::string( $processor->get_token_value() ); + break; + + // Keep or strip comments? + case WP_CSS_Token_Processor::TOKEN_COMMENT: + $normalized_css .= substr( $css, $processor->get_token_start(), $processor->get_token_length() ); + break; + + /** + * A is an open string that reaches a newline. + * + * @see https://www.w3.org/TR/css-syntax-3/#consume-string-token + * + * @see https://www.w3.org/TR/css-syntax-3/#preserved-tokens + * > Note: The tokens <}-token>s, <)-token>s, <]-token>, , and are always parse errors, but they are preserved in the token stream by this specification to allow other specs, such as Media Queries, to define more fine-grained error-handling than just dropping an entire declaration or block. + */ + case WP_CSS_Token_Processor::TOKEN_BAD_STRING: + $normalized_css .= substr( $css, $processor->get_token_start(), $processor->get_token_length() ) . "\n"; + break; + + case WP_CSS_Token_Processor::TOKEN_URL: + case WP_CSS_Token_Processor::TOKEN_BAD_URL: + case WP_CSS_Token_Processor::TOKEN_CDC: + case WP_CSS_Token_Processor::TOKEN_CDO: + default: + throw new Error( 'unhandled token type ' . $processor->get_token_type() . ' with value ' . var_export( $processor->get_token_value(), true ) ); + } + } + + return strtr( + $normalized_css, + array( + ' ' => '␠', + "\t" => "␉\t", + "\n" => "␊\n", + ) + ); + } +} diff --git a/src/wp-includes/css-api/class-wp-css-token-processor.php b/src/wp-includes/css-api/class-wp-css-token-processor.php new file mode 100644 index 0000000000000..74c548dc1ec3a --- /dev/null +++ b/src/wp-includes/css-api/class-wp-css-token-processor.php @@ -0,0 +1,1863 @@ + Replace any U+000D CARRIAGE RETURN (CR) code points, U+000C FORM FEED (FF) + * > code points, or pairs of U+000D CARRIAGE RETURN (CR) followed by U+000A LINE + * > FEED (LF) in input by a single U+000A LINE FEED (LF) code point. + * > Replace any U+0000 NULL or surrogate code points in input with U+FFFD REPLACEMENT + * > CHARACTER (�). + * + * This processor delays normalization as much as possible. That keeps the raw byte + * positions intact for accurate rewrites while still letting consumers ask for a + * normalized token when they need one. + * + * ### No EOF token + * + * The EOF token is a CSS parsing concept, not CSS tokenization concept. Therefore, + * this processor does not produce it. + * + * ### UTF-8 handling + * + * Only UTF-8 strings are supported. Invalid sequences are replaced with U+FFFD (�) + * using the maximal subpart approach described in + * https://www.unicode.org/versions/Unicode9.0.0/ch03.pdf, section 3.9 Best Practices + * for Using U+FFFD. + * + * ## Usage + * + * Basic iteration: + * + * $css = 'width: 10px;'; + * $processor = WP_CSS_Token_Processor::create( $css ); + * while ( $processor->next_token() ) { + * echo $processor->get_normalized_token(); + * } + * // Outputs: + * // width: 10px; + * + * Rewriting a URL while keeping the rest of the stylesheet intact: + * + * $css = 'background: url(old.jpg) center / cover;'; + * $processor = WP_CSS_Token_Processor::create( $css ); + * while ( $processor->next_token() ) { + * if ( WP_CSS_Token_Processor::TOKEN_URL === $processor->get_token_type() ) { + * $processor->set_value( 'uploads/new.jpg' ); + * } + * } + * $result = $processor->get_updated_css(); + * // background: url(uploads/new.jpg) center / cover; + * + * Gathering diagnostics with byte offsets: + * + * $css = "color: red;\ncolor: re\nd;"; + * $processor = WP_CSS_Token_Processor::create( $css ); + * $bad_strings = array(); + * while ( $processor->next_token() ) { + * if ( WP_CSS_Token_Processor::TOKEN_BAD_STRING === $processor->get_token_type() ) { + * $bad_strings[] = array( + * 'start' => $processor->get_token_start(), + * 'length' => $processor->get_token_length(), + * 'value' => $processor->get_unnormalized_token(), + * ); + * } + * } + * + * @see https://www.w3.org/TR/css-syntax-3/#tokenization + */ +class WP_CSS_Token_Processor { + /** + * Token type constants matching the CSS Syntax Level 3 specification. + * + * @see https://www.w3.org/TR/css-syntax-3/#tokenization + */ + public const TOKEN_WHITESPACE = 'whitespace-token'; + public const TOKEN_COMMENT = 'comment'; + public const TOKEN_STRING = 'string-token'; + + /** + * BAD-STRING tokens occur when a string contains an unescaped newline. + * + * Valid strings: "hello", 'world', "line1\Aline2" (escaped newline) + * Invalid (produces bad-string): "hello + * world" (literal newline breaks the string) + * + * The processor stops at the newline and produces a bad-string token for error recovery. + * + * @see https://www.w3.org/TR/css-syntax-3/#typedef-bad-string-token + */ + public const TOKEN_BAD_STRING = 'bad-string-token'; + public const TOKEN_HASH = 'hash-token'; + public const HASH_TOKEN_ID = 'id'; + public const HASH_TOKEN_UNRESTRICTED = 'unrestricted'; + public const TOKEN_DELIM = 'delim-token'; + public const TOKEN_NUMBER = 'number-token'; + public const TOKEN_PERCENTAGE = 'percentage-token'; + public const TOKEN_DIMENSION = 'dimension-token'; + public const TOKEN_AT_KEYWORD = 'at-keyword-token'; + public const TOKEN_COLON = 'colon-token'; + public const TOKEN_SEMICOLON = 'semicolon-token'; + public const TOKEN_COMMA = 'comma-token'; + public const TOKEN_LEFT_PAREN = '(-token'; + public const TOKEN_RIGHT_PAREN = ')-token'; + public const TOKEN_LEFT_BRACKET = '[-token'; + public const TOKEN_RIGHT_BRACKET = ']-token'; + public const TOKEN_LEFT_BRACE = '{-token'; + public const TOKEN_RIGHT_BRACE = '}-token'; + public const TOKEN_FUNCTION = 'function-token'; + + /** + * URL tokens represent unquoted URLs in url() notation. + * + * For example, `url(image.jpg)` is a URL token. + * + * Quoted URLs like `url( "https://example.com" )` are handled as a function + * token, _not_ a URL token. + * + * Bad URL tokens are created when invalid characters are encountered in + * a URL token. + * + * @see https://www.w3.org/TR/css-syntax-3/#typedef-url-token + */ + public const TOKEN_URL = 'url-token'; + + /** + * BAD-URL tokens occur when a URL contains invalid characters. + * + * Invalid characters: quotes ("), apostrophes ('), parentheses (() + * Example invalid: url(image(.jpg) or url(image".jpg) + * + * When detected, the processor consumes everything up to ) or EOF. + * This prevents the bad URL from breaking subsequent tokens. + * + * @see https://www.w3.org/TR/css-syntax-3/#typedef-bad-url-token + */ + public const TOKEN_BAD_URL = 'bad-url-token'; + + /** + * Identifier tokens, such as `color`, `margin-top`, `red`, + * `inherit`, `--my-var`, `\x-escaped`, `über` (Unicode), etc. + * + * There are restrictions on the codepoints that start or are contained in + * an identifier, and identifiers may contain escape sequences. + * + * @see https://www.w3.org/TR/css-syntax-3/#typedef-ident-token + */ + public const TOKEN_IDENT = 'ident-token'; + + /** + * CDC (Comment Delimiter Close) token: --> + * + * Legacy token from when CSS was embedded in HTML + * + * Modern CSS no longer needs these, but they're preserved for compatibility. + * In stylesheets, they're typically treated like whitespace. + * + * @see https://www.w3.org/TR/css-syntax-3/#typedef-CDC-token + */ + public const TOKEN_CDC = 'CDC-token'; + + /** + * CDO (Comment Delimiter Open) token: ) + * + * Comment Delimiter Close - legacy HTML comment syntax in CSS. + * + * @see https://www.w3.org/TR/css-syntax-3/#CDC-token-diagram + */ + if ( + $this->at + 2 < $this->length && + '-' === $this->css[ $this->at + 1 ] && + '>' === $this->css[ $this->at + 2 ] + ) { + // Consume them and return a . + $this->at += 3; + $this->token_type = self::TOKEN_CDC; + $this->token_length = 3; + return true; + } + + // Otherwise, if the input stream starts with an ident sequence, + // reconsume the current input code point, consume an ident-like + // token, and return it. + if ( $this->check_if_3_code_points_start_an_ident_sequence( $this->at ) ) { + return $this->consume_ident_like(); + } + + // Otherwise, return a with its value set to the current input code point. + ++$this->at; + $this->token_type = self::TOKEN_DELIM; + $this->token_length = 1; + return true; + } + + /* + * U+003C LESS-THAN SIGN (<) + * If followed by !--, this is a CDO token (\n", + "tokens": [ + { + "type": "CDC-token", + "raw": "-->", + "startIndex": 0, + "endIndex": 3, + "normalized": "-->", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0001": { + "css": "foo\n", + "tokens": [ + { + "type": "ident-token", + "raw": "foo", + "startIndex": 0, + "endIndex": 3, + "normalized": "foo", + "value": "foo" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0002": { + "css": "--\n", + "tokens": [ + { + "type": "ident-token", + "raw": "--", + "startIndex": 0, + "endIndex": 2, + "normalized": "--", + "value": "--" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 2, + "endIndex": 3, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0003": { + "css": "--0\n", + "tokens": [ + { + "type": "ident-token", + "raw": "--0", + "startIndex": 0, + "endIndex": 3, + "normalized": "--0", + "value": "--0" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0004": { + "css": "-\\\n", + "tokens": [ + { + "type": "delim-token", + "raw": "-", + "startIndex": 0, + "endIndex": 1, + "normalized": "-", + "value": "-" + }, + { + "type": "delim-token", + "raw": "\\", + "startIndex": 1, + "endIndex": 2, + "normalized": "\\", + "value": "\\" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 2, + "endIndex": 3, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0005": { + "css": "-\\ \n", + "tokens": [ + { + "type": "ident-token", + "raw": "-\\ ", + "startIndex": 0, + "endIndex": 3, + "normalized": "- ", + "value": "- " + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0006": { + "css": "--💅\n", + "tokens": [ + { + "type": "ident-token", + "raw": "--💅", + "startIndex": 0, + "endIndex": 6, + "normalized": "--💅", + "value": "--💅" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 6, + "endIndex": 7, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0007": { + "css": "-§\n", + "tokens": [ + { + "type": "ident-token", + "raw": "-§", + "startIndex": 0, + "endIndex": 3, + "normalized": "-§", + "value": "-§" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0008": { + "css": "-×\n", + "tokens": [ + { + "type": "ident-token", + "raw": "-×", + "startIndex": 0, + "endIndex": 3, + "normalized": "-×", + "value": "-×" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 3, + "endIndex": 4, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident/0009": { + "css": "--a𐀀\n", + "tokens": [ + { + "type": "ident-token", + "raw": "--a𐀀", + "startIndex": 0, + "endIndex": 7, + "normalized": "--a𐀀", + "value": "--a𐀀" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 7, + "endIndex": 8, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0001": { + "css": "url(foo)\n", + "tokens": [ + { + "type": "url-token", + "raw": "url(foo)", + "startIndex": 0, + "endIndex": 8, + "normalized": "url(foo)", + "value": "foo" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 8, + "endIndex": 9, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0002": { + "css": "\\75 Rl(foo)\n", + "tokens": [ + { + "type": "url-token", + "raw": "\\75 Rl(foo)", + "startIndex": 0, + "endIndex": 11, + "normalized": "uRl(foo)", + "value": "foo" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 11, + "endIndex": 12, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0003": { + "css": "uR\\6c (foo)\n", + "tokens": [ + { + "type": "url-token", + "raw": "uR\\6c (foo)", + "startIndex": 0, + "endIndex": 11, + "normalized": "uRl(foo)", + "value": "foo" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 11, + "endIndex": 12, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0004": { + "css": "url('foo')\n", + "tokens": [ + { + "type": "function-token", + "raw": "url(", + "startIndex": 0, + "endIndex": 4, + "normalized": "url(", + "value": "url" + }, + { + "type": "string-token", + "raw": "'foo'", + "startIndex": 4, + "endIndex": 9, + "normalized": "'foo'", + "value": "foo" + }, + { + "type": ")-token", + "raw": ")", + "startIndex": 9, + "endIndex": 10, + "normalized": ")", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 10, + "endIndex": 11, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0005": { + "css": "url( 'foo')\n", + "tokens": [ + { + "type": "function-token", + "raw": "url(", + "startIndex": 0, + "endIndex": 4, + "normalized": "url(", + "value": "url" + }, + { + "type": "whitespace-token", + "raw": " ", + "startIndex": 4, + "endIndex": 5, + "normalized": " ", + "value": null + }, + { + "type": "string-token", + "raw": "'foo'", + "startIndex": 5, + "endIndex": 10, + "normalized": "'foo'", + "value": "foo" + }, + { + "type": ")-token", + "raw": ")", + "startIndex": 10, + "endIndex": 11, + "normalized": ")", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 11, + "endIndex": 12, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0006": { + "css": "url( 'foo')\n", + "tokens": [ + { + "type": "function-token", + "raw": "url(", + "startIndex": 0, + "endIndex": 4, + "normalized": "url(", + "value": "url" + }, + { + "type": "whitespace-token", + "raw": " ", + "startIndex": 4, + "endIndex": 6, + "normalized": " ", + "value": null + }, + { + "type": "string-token", + "raw": "'foo'", + "startIndex": 6, + "endIndex": 11, + "normalized": "'foo'", + "value": "foo" + }, + { + "type": ")-token", + "raw": ")", + "startIndex": 11, + "endIndex": 12, + "normalized": ")", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 12, + "endIndex": 13, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0007": { + "css": "url( 'foo')\n", + "tokens": [ + { + "type": "function-token", + "raw": "url(", + "startIndex": 0, + "endIndex": 4, + "normalized": "url(", + "value": "url" + }, + { + "type": "whitespace-token", + "raw": " ", + "startIndex": 4, + "endIndex": 7, + "normalized": " ", + "value": null + }, + { + "type": "string-token", + "raw": "'foo'", + "startIndex": 7, + "endIndex": 12, + "normalized": "'foo'", + "value": "foo" + }, + { + "type": ")-token", + "raw": ")", + "startIndex": 12, + "endIndex": 13, + "normalized": ")", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 13, + "endIndex": 14, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0008": { + "css": "not-url( 'foo')\n", + "tokens": [ + { + "type": "function-token", + "raw": "not-url(", + "startIndex": 0, + "endIndex": 8, + "normalized": "not-url(", + "value": "not-url" + }, + { + "type": "whitespace-token", + "raw": " ", + "startIndex": 8, + "endIndex": 11, + "normalized": " ", + "value": null + }, + { + "type": "string-token", + "raw": "'foo'", + "startIndex": 11, + "endIndex": 16, + "normalized": "'foo'", + "value": "foo" + }, + { + "type": ")-token", + "raw": ")", + "startIndex": 16, + "endIndex": 17, + "normalized": ")", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 17, + "endIndex": 18, + "normalized": "\n", + "value": null + } + ] + }, + "tests/ident-like/0009": { + "css": "url( foo)\n", + "tokens": [ + { + "type": "url-token", + "raw": "url( foo)", + "startIndex": 0, + "endIndex": 11, + "normalized": "url( foo)", + "value": "foo" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 11, + "endIndex": 12, + "normalized": "\n", + "value": null + } + ] + }, + "tests/left-curly-bracket/0001": { + "css": "{\n", + "tokens": [ + { + "type": "{-token", + "raw": "{", + "startIndex": 0, + "endIndex": 1, + "normalized": "{", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 1, + "endIndex": 2, + "normalized": "\n", + "value": null + } + ] + }, + "tests/left-parenthesis/0001": { + "css": "(\n", + "tokens": [ + { + "type": "(-token", + "raw": "(", + "startIndex": 0, + "endIndex": 1, + "normalized": "(", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 1, + "endIndex": 2, + "normalized": "\n", + "value": null + } + ] + }, + "tests/left-square-bracket/0001": { + "css": "[\n", + "tokens": [ + { + "type": "[-token", + "raw": "[", + "startIndex": 0, + "endIndex": 1, + "normalized": "[", + "value": null + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 1, + "endIndex": 2, + "normalized": "\n", + "value": null + } + ] + }, + "tests/less-than/0001": { + "css": "<\n", + "tokens": [ + { + "type": "delim-token", + "raw": "<", + "startIndex": 0, + "endIndex": 1, + "normalized": "<", + "value": "<" + }, + { + "type": "whitespace-token", + "raw": "\n", + "startIndex": 1, + "endIndex": 2, + "normalized": "\n", + "value": null + } + ] + }, + "tests/less-than/0002": { + "css": "