From b9765f4132fab4c9dd83774fff571dee123ca162 Mon Sep 17 00:00:00 2001 From: Glenn Fiedler Date: Sun, 6 Sep 2026 04:22:15 -0700 Subject: [PATCH 1/2] tables: a clamp answers within its bound, and the text runtime takes the wire's own length MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Certification's test (ubuntu-latest) job has been red since 28133b49 (#592) on the length negative control, which removes r.room( len ) from a copy of the emitter and requires the wire fuzzer to go red because the leg DIED on the mutant. On Ubuntu it went red on a report difference instead, and the leg's heap was corrupted: munmap_chunk(): invalid pointer. Mutant 629 of seed root_full spells the string field's L as 0xFFFFFFFFFFFFFFFF. #592 added two text helpers that took that length as a signed count, and (int64_t) 0xFFFFFFFFFFFFFFFF is -1. TableUtf8Clamp opens with "if ( length <= bound ) return length", -1 is under every bound, the caller widens the answer back to SIZE_MAX, and the field's memcpy runs at SIZE_MAX. ASan on the sabotaged leg, replaying that mutant, calls it negative-size-param (size=-1) at the field's memcpy. On this Air that copy faults and the leg dies, which is the red the control names; on glibc it writes outside the destination, corrupts the neighbouring chunk header, returns, and the leg reports before free aborts. A LENGTH IS A 64-BIT NUMBER (docs/SPEC-TABLES.md §3), so TableUtf8Valid and TableUtf8Clamp take it as one, and the clamp's answer is never above its bound. The string field read and the string map key drop the casts that narrowed it. The sites passing a reader's own span are non-negative by construction and are unchanged, and the Go oracle takes a slice and cannot express the defect. Nothing changes in a clean build: room( len ) already bounded every length these helpers see, which is what the negative control exists to prove. Co-Authored-By: Claude Fable 5.1 --- generated/bench/tables/cpp/BenchTableTable.h | 28 +++++++---- internal/codegen/cpptable/codecs.go | 4 +- internal/codegen/cpptable/maps.go | 2 +- internal/codegen/cpptable/text.go | 22 +++++--- test/tables/main.cpp | 26 ++++++++++ testdata/golden/tables/arms/CarryTable.h | 24 ++++++--- testdata/golden/tables/arms/GateTable.h | 24 ++++++--- testdata/golden/tables/arms/NestTable.h | 24 ++++++--- testdata/golden/tables/arms/RingTable.h | 24 ++++++--- testdata/golden/tables/blobs/AssetsTable.h | 32 ++++++++---- testdata/golden/tables/block/PaddedTable.h | 26 +++++++--- testdata/golden/tables/block/RenderTable.h | 22 +++++--- .../golden/tables/blockhome/ConstantsTable.h | 24 ++++++--- testdata/golden/tables/blockhome/DataTable.h | 24 ++++++--- testdata/golden/tables/blockhome/FrameTable.h | 24 ++++++--- .../golden/tables/examples/GuardedTable.h | 28 +++++++---- testdata/golden/tables/examples/KeyedTable.h | 28 +++++++---- testdata/golden/tables/examples/NestedTable.h | 24 ++++++--- testdata/golden/tables/examples/PackTable.h | 36 ++++++++----- testdata/golden/tables/examples/RangesTable.h | 24 ++++++--- testdata/golden/tables/examples/TablesTable.h | 32 ++++++++---- testdata/golden/tables/examples/WideTable.h | 28 +++++++---- testdata/golden/tables/lists/HoldersTable.h | 22 +++++--- testdata/golden/tables/lists/MigrateTable.h | 24 ++++++--- testdata/golden/tables/lists/ReportTable.h | 24 ++++++--- testdata/golden/tables/lists/SaveTable.h | 24 ++++++--- testdata/golden/tables/lists/SharedTable.h | 24 ++++++--- testdata/golden/tables/maps/DepthTable.h | 24 ++++++--- testdata/golden/tables/maps/FleetTable.h | 40 +++++++++------ testdata/golden/tables/maps/RowsTable.h | 36 ++++++++----- .../golden/tables/messages/MessagesTable.h | 48 +++++++++++------- testdata/golden/tables/pointers/GraphTable.h | 50 +++++++++++-------- testdata/golden/tables/pointers/MarksTable.h | 28 +++++++---- testdata/golden/tables/pointers/PartsTable.h | 28 +++++++---- testdata/golden/tables/scalars/ScalarsTable.h | 22 +++++--- testdata/golden/tables/stream/StreamTable.h | 28 +++++++---- 36 files changed, 654 insertions(+), 298 deletions(-) diff --git a/generated/bench/tables/cpp/BenchTableTable.h b/generated/bench/tables/cpp/BenchTableTable.h index d703a02a4..f33360d35 100644 --- a/generated/bench/tables/cpp/BenchTableTable.h +++ b/generated/bench/tables/cpp/BenchTableTable.h @@ -595,13 +595,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -610,7 +614,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -626,9 +630,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -5183,9 +5193,9 @@ BENCHTABLE_TABLE_INLINE bool TableMixedLoadBody( TableReader & r, TableMixed & v if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.player_name[0] = 0; value.player_name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.player_name[0] = 0; value.player_name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 15 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 15 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 15 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 15 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.player_name, r.buffer + r.offset, (size_t) keep ); value.player_name[keep] = 0; value.player_name_length = (int32_t) keep; diff --git a/internal/codegen/cpptable/codecs.go b/internal/codegen/cpptable/codecs.go index 1af24b840..7707fdebf 100644 --- a/internal/codegen/cpptable/codecs.go +++ b/internal/codegen/cpptable/codecs.go @@ -1691,9 +1691,9 @@ func (g *tableGen) emitTableReadField(f *ir.Field, kind int) { g.pf("%suint64_t len = 0;\n", ind) g.pf("%sif ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; }\n", ind) g.pf("%s// ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared\n%s// default, one malformed counts, and the parent reads on past L\n", ind, ind) - g.pf("%sif ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.%s[0] = 0; value.%s_length = 0; r.offset += (int64_t) len; break; }\n", ind, f.Name, f.Name) + g.pf("%sif ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.%s[0] = 0; value.%s_length = 0; r.offset += (int64_t) len; break; }\n", ind, f.Name, f.Name) g.pf("%suint64_t keep = len;\n", ind) - g.pf("%sif ( keep > %d ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, %d ); r.report->clamped++; } // at a code point boundary (§3)\n", ind, f.Type.Size, f.Type.Size) + g.pf("%sif ( keep > %d ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, %d ); r.report->clamped++; } // at a code point boundary (§3)\n", ind, f.Type.Size, f.Type.Size) g.pf("%smemcpy( value.%s, r.buffer + r.offset, (size_t) keep );\n", ind, f.Name) g.pf("%svalue.%s[keep] = 0;\n", ind, f.Name) g.pf("%svalue.%s_length = (int32_t) keep;\n", ind, f.Name) diff --git a/internal/codegen/cpptable/maps.go b/internal/codegen/cpptable/maps.go index 294532637..dd4552471 100644 --- a/internal/codegen/cpptable/maps.go +++ b/internal/codegen/cpptable/maps.go @@ -1033,7 +1033,7 @@ func (g *tableGen) emitMapKeyReader(f *ir.Field) { g.pf(" uint64_t key_len = 0;\n") g.pf(" if ( !r.getleb( key_len ) || !r.room( key_len ) ) { out.malformed = true; return out; }\n") g.pf(" // a key a string value would refuse as malformed makes the MAP malformed (§2.8, §3)\n") - g.pf(" if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) key_len ) ) { out.malformed = true; return out; }\n") + g.pf(" if ( !TableUtf8Valid( r.buffer + r.offset, key_len ) ) { out.malformed = true; return out; }\n") g.pf(" out.key = (const char *) ( r.buffer + r.offset );\n") g.pf(" out.length = (int32_t) key_len;\n") g.pf(" out.over = key_len > %d; // KEYS NEVER CLAMP: the entry is dropped whole\n", key.Type.Size) diff --git a/internal/codegen/cpptable/text.go b/internal/codegen/cpptable/text.go index 842506ee9..6e0799c0f 100644 --- a/internal/codegen/cpptable/text.go +++ b/internal/codegen/cpptable/text.go @@ -16,13 +16,17 @@ const tableTextRuntime = ` // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -31,7 +35,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -47,9 +51,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/test/tables/main.cpp b/test/tables/main.cpp index f9a3bf66d..949508659 100644 --- a/test/tables/main.cpp +++ b/test/tables/main.cpp @@ -935,6 +935,31 @@ static void test_clamping() CHECK( report4.clamped == 1 && out4.name_length == 32 && out4.name[32] == 0 ); } +// A LENGTH IS A 64-BIT NUMBER (docs/SPEC-TABLES.md §3), and the clamp takes it +// as one: the caller turns the answer straight back into the size of the +// field's copy, so the answer has to be within the bound for EVERY length the +// wire can spell, including the ones only a broken reader would still be +// holding. Read as a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every +// bound, and the copy runs at SIZE_MAX. + +static void test_text_clamp_is_bounded() +{ + const uint8_t text[] = "golden-v1"; + + // a payload within the bound is kept whole, and one over it cuts at a code + // point boundary — "é" is two bytes, so a bound of 9 cuts back to 8 + CHECK( tabledemo::TableUtf8Clamp( text, 9, 32 ) == 9 ); + const uint8_t two_byte[] = { 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 0xC3, 0xA9 }; + CHECK( tabledemo::TableUtf8Clamp( two_byte, 10, 9 ) == 8 ); + + // the field read's own three lines, over the length the wire fuzzer's + // length pass plants: the size that reaches memcpy is within the bound + const uint64_t forged = 0xFFFFFFFFFFFFFFFFull; + uint64_t keep = forged; + if ( keep > 32 ) { keep = (uint64_t) tabledemo::TableUtf8Clamp( text, forged, 32 ); } + CHECK( keep <= 32 ); +} + // ---- malformed framing: decode stops, partial result kept, flag raised ---- static void test_malformed() @@ -9047,6 +9072,7 @@ int main() test_flags_are_positional(); test_wide_extents(); test_clamping(); + test_text_clamp_is_bounded(); test_malformed(); test_reflection(); test_cross_file(); diff --git a/testdata/golden/tables/arms/CarryTable.h b/testdata/golden/tables/arms/CarryTable.h index 10810b519..252c8ca71 100644 --- a/testdata/golden/tables/arms/CarryTable.h +++ b/testdata/golden/tables/arms/CarryTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/arms/GateTable.h b/testdata/golden/tables/arms/GateTable.h index d9c6c64c0..e6c15fea6 100644 --- a/testdata/golden/tables/arms/GateTable.h +++ b/testdata/golden/tables/arms/GateTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/arms/NestTable.h b/testdata/golden/tables/arms/NestTable.h index 330df519d..aa1d4da74 100644 --- a/testdata/golden/tables/arms/NestTable.h +++ b/testdata/golden/tables/arms/NestTable.h @@ -654,13 +654,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -669,7 +673,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -685,9 +689,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/arms/RingTable.h b/testdata/golden/tables/arms/RingTable.h index f28d0618d..af48ce1c8 100644 --- a/testdata/golden/tables/arms/RingTable.h +++ b/testdata/golden/tables/arms/RingTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/blobs/AssetsTable.h b/testdata/golden/tables/blobs/AssetsTable.h index c79d0df1f..42d047ecf 100644 --- a/testdata/golden/tables/blobs/AssetsTable.h +++ b/testdata/golden/tables/blobs/AssetsTable.h @@ -628,13 +628,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -643,7 +647,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -659,9 +663,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -3537,9 +3547,9 @@ inline bool AssetLoadBody( TableReader & r, const TableNodeMap & nodes, Asset & if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -4068,9 +4078,9 @@ inline bool CatalogLoadBody( TableReader & r, const TableNodeMap & nodes, Catalo if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; diff --git a/testdata/golden/tables/block/PaddedTable.h b/testdata/golden/tables/block/PaddedTable.h index 6e7a99336..69eb33157 100644 --- a/testdata/golden/tables/block/PaddedTable.h +++ b/testdata/golden/tables/block/PaddedTable.h @@ -614,13 +614,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -629,7 +633,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -645,9 +649,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2680,9 +2690,9 @@ BLOCKDEMO_TABLE_INLINE bool PaddedRowLoadBody( TableReader & r, PaddedRow & valu if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 15 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 15 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 15 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 15 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.label, r.buffer + r.offset, (size_t) keep ); value.label[keep] = 0; value.label_length = (int32_t) keep; diff --git a/testdata/golden/tables/block/RenderTable.h b/testdata/golden/tables/block/RenderTable.h index b6a5f306d..83252e4b9 100644 --- a/testdata/golden/tables/block/RenderTable.h +++ b/testdata/golden/tables/block/RenderTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/blockhome/ConstantsTable.h b/testdata/golden/tables/blockhome/ConstantsTable.h index dc6cba4a6..e15dbf881 100644 --- a/testdata/golden/tables/blockhome/ConstantsTable.h +++ b/testdata/golden/tables/blockhome/ConstantsTable.h @@ -595,13 +595,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -610,7 +614,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -626,9 +630,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/blockhome/DataTable.h b/testdata/golden/tables/blockhome/DataTable.h index 2a706d147..069eb1e74 100644 --- a/testdata/golden/tables/blockhome/DataTable.h +++ b/testdata/golden/tables/blockhome/DataTable.h @@ -595,13 +595,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -610,7 +614,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -626,9 +630,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/blockhome/FrameTable.h b/testdata/golden/tables/blockhome/FrameTable.h index 5c3f6c2f9..6921710f7 100644 --- a/testdata/golden/tables/blockhome/FrameTable.h +++ b/testdata/golden/tables/blockhome/FrameTable.h @@ -596,13 +596,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -611,7 +615,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -627,9 +631,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/examples/GuardedTable.h b/testdata/golden/tables/examples/GuardedTable.h index a0416146d..573fceb2b 100644 --- a/testdata/golden/tables/examples/GuardedTable.h +++ b/testdata/golden/tables/examples/GuardedTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2610,9 +2620,9 @@ TABLEDEMO_TABLE_INLINE bool PatrolLoadBody( TableReader & r, Patrol & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.note[0] = 0; value.note_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.note[0] = 0; value.note_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 8 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 8 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.note, r.buffer + r.offset, (size_t) keep ); value.note[keep] = 0; value.note_length = (int32_t) keep; diff --git a/testdata/golden/tables/examples/KeyedTable.h b/testdata/golden/tables/examples/KeyedTable.h index 387049867..5a96455ed 100644 --- a/testdata/golden/tables/examples/KeyedTable.h +++ b/testdata/golden/tables/examples/KeyedTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2769,9 +2779,9 @@ TABLEDEMO_TABLE_INLINE bool TeamConfigLoadBody( TableReader & r, TeamConfig & va if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.banner[0] = 0; value.banner_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.banner[0] = 0; value.banner_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.banner, r.buffer + r.offset, (size_t) keep ); value.banner[keep] = 0; value.banner_length = (int32_t) keep; diff --git a/testdata/golden/tables/examples/NestedTable.h b/testdata/golden/tables/examples/NestedTable.h index bc482f8bd..efaea7799 100644 --- a/testdata/golden/tables/examples/NestedTable.h +++ b/testdata/golden/tables/examples/NestedTable.h @@ -614,13 +614,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -629,7 +633,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -645,9 +649,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/examples/PackTable.h b/testdata/golden/tables/examples/PackTable.h index 412fa4354..f07d6f56f 100644 --- a/testdata/golden/tables/examples/PackTable.h +++ b/testdata/golden/tables/examples/PackTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2700,9 +2710,9 @@ TABLEDEMO_TABLE_INLINE bool GunnerSettingsLoadBody( TableReader & r, GunnerSetti if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.callsign[0] = 0; value.callsign_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.callsign[0] = 0; value.callsign_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 24 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 24 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 24 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 24 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.callsign, r.buffer + r.offset, (size_t) keep ); value.callsign[keep] = 0; value.callsign_length = (int32_t) keep; @@ -3128,9 +3138,9 @@ TABLEDEMO_TABLE_INLINE bool ShipEntryLoadBody( TableReader & r, ShipEntry & valu if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.display_name[0] = 0; value.display_name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.display_name[0] = 0; value.display_name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.display_name, r.buffer + r.offset, (size_t) keep ); value.display_name[keep] = 0; value.display_name_length = (int32_t) keep; @@ -3891,9 +3901,9 @@ TABLEDEMO_TABLE_INLINE bool GlobalSettingsLoadBody( TableReader & r, GlobalSetti if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.build_note[0] = 0; value.build_note_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.build_note[0] = 0; value.build_note_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 48 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 48 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 48 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 48 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.build_note, r.buffer + r.offset, (size_t) keep ); value.build_note[keep] = 0; value.build_note_length = (int32_t) keep; diff --git a/testdata/golden/tables/examples/RangesTable.h b/testdata/golden/tables/examples/RangesTable.h index 9d7ca6ba5..b132a6a87 100644 --- a/testdata/golden/tables/examples/RangesTable.h +++ b/testdata/golden/tables/examples/RangesTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/examples/TablesTable.h b/testdata/golden/tables/examples/TablesTable.h index 33458285b..673ceb971 100644 --- a/testdata/golden/tables/examples/TablesTable.h +++ b/testdata/golden/tables/examples/TablesTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -4662,9 +4672,9 @@ TABLEDEMO_TABLE_INLINE bool ProfileConfigLoadBody( TableReader & r, ProfileConfi if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -5919,9 +5929,9 @@ TABLEDEMO_TABLE_INLINE bool RootConfigLoadBody( TableReader & r, RootConfig & va if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.version_note[0] = 0; value.version_note_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.version_note[0] = 0; value.version_note_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.version_note, r.buffer + r.offset, (size_t) keep ); value.version_note[keep] = 0; value.version_note_length = (int32_t) keep; diff --git a/testdata/golden/tables/examples/WideTable.h b/testdata/golden/tables/examples/WideTable.h index 04000d2fb..f27248093 100644 --- a/testdata/golden/tables/examples/WideTable.h +++ b/testdata/golden/tables/examples/WideTable.h @@ -613,13 +613,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -628,7 +632,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -644,9 +648,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2488,9 +2498,9 @@ TABLEDEMO_TABLE_INLINE bool WideBlobLoadBody( TableReader & r, WideBlob & value if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 70000 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 70000 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 70000 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 70000 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.label, r.buffer + r.offset, (size_t) keep ); value.label[keep] = 0; value.label_length = (int32_t) keep; diff --git a/testdata/golden/tables/lists/HoldersTable.h b/testdata/golden/tables/lists/HoldersTable.h index d5b1005e9..957d730b6 100644 --- a/testdata/golden/tables/lists/HoldersTable.h +++ b/testdata/golden/tables/lists/HoldersTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/lists/MigrateTable.h b/testdata/golden/tables/lists/MigrateTable.h index ba86450e8..e6cfaadc6 100644 --- a/testdata/golden/tables/lists/MigrateTable.h +++ b/testdata/golden/tables/lists/MigrateTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/lists/ReportTable.h b/testdata/golden/tables/lists/ReportTable.h index 97aa45576..622088b57 100644 --- a/testdata/golden/tables/lists/ReportTable.h +++ b/testdata/golden/tables/lists/ReportTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/lists/SaveTable.h b/testdata/golden/tables/lists/SaveTable.h index 2148035a3..109140e73 100644 --- a/testdata/golden/tables/lists/SaveTable.h +++ b/testdata/golden/tables/lists/SaveTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/lists/SharedTable.h b/testdata/golden/tables/lists/SharedTable.h index 1004e647f..6f98698a2 100644 --- a/testdata/golden/tables/lists/SharedTable.h +++ b/testdata/golden/tables/lists/SharedTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/maps/DepthTable.h b/testdata/golden/tables/maps/DepthTable.h index a64fd0cf2..95b24d57d 100644 --- a/testdata/golden/tables/maps/DepthTable.h +++ b/testdata/golden/tables/maps/DepthTable.h @@ -654,13 +654,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -669,7 +673,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -685,9 +689,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/maps/FleetTable.h b/testdata/golden/tables/maps/FleetTable.h index a8ade56b8..34bb644de 100644 --- a/testdata/golden/tables/maps/FleetTable.h +++ b/testdata/golden/tables/maps/FleetTable.h @@ -653,13 +653,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -668,7 +672,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -684,9 +688,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -4912,7 +4922,7 @@ inline FleetShipsEntryKeyRead FleetShipsEntryReadKey( const uint8_t * body, int6 uint64_t key_len = 0; if ( !r.getleb( key_len ) || !r.room( key_len ) ) { out.malformed = true; return out; } // a key a string value would refuse as malformed makes the MAP malformed (§2.8, §3) - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) key_len ) ) { out.malformed = true; return out; } + if ( !TableUtf8Valid( r.buffer + r.offset, key_len ) ) { out.malformed = true; return out; } out.key = (const char *) ( r.buffer + r.offset ); out.length = (int32_t) key_len; out.over = key_len > 32; // KEYS NEVER CLAMP: the entry is dropped whole @@ -5081,7 +5091,7 @@ inline FleetLoadoutsEntryKeyRead FleetLoadoutsEntryReadKey( const uint8_t * body uint64_t key_len = 0; if ( !r.getleb( key_len ) || !r.room( key_len ) ) { out.malformed = true; return out; } // a key a string value would refuse as malformed makes the MAP malformed (§2.8, §3) - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) key_len ) ) { out.malformed = true; return out; } + if ( !TableUtf8Valid( r.buffer + r.offset, key_len ) ) { out.malformed = true; return out; } out.key = (const char *) ( r.buffer + r.offset ); out.length = (int32_t) key_len; out.over = key_len > 16; // KEYS NEVER CLAMP: the entry is dropped whole @@ -5255,9 +5265,9 @@ MAPDEMO_TABLE_INLINE bool ShipConfigLoadBody( TableReader & r, ShipConfig & valu if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 64 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 64 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -5987,9 +5997,9 @@ MAPDEMO_TABLE_INLINE bool FleetShipsEntryLoadBody( TableReader & r, FleetShipsEn if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.key, r.buffer + r.offset, (size_t) keep ); value.key[keep] = 0; value.key_length = (int32_t) keep; @@ -6772,9 +6782,9 @@ inline bool FleetLoadoutsEntryLoadBody( TableReader & r, const TableNodeMap & no if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.key, r.buffer + r.offset, (size_t) keep ); value.key[keep] = 0; value.key_length = (int32_t) keep; diff --git a/testdata/golden/tables/maps/RowsTable.h b/testdata/golden/tables/maps/RowsTable.h index e98e6c6dc..1a0ae1b3b 100644 --- a/testdata/golden/tables/maps/RowsTable.h +++ b/testdata/golden/tables/maps/RowsTable.h @@ -654,13 +654,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -669,7 +673,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -685,9 +689,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -4726,7 +4736,7 @@ inline RowEntriesEntryKeyRead RowEntriesEntryReadKey( const uint8_t * body, int6 uint64_t key_len = 0; if ( !r.getleb( key_len ) || !r.room( key_len ) ) { out.malformed = true; return out; } // a key a string value would refuse as malformed makes the MAP malformed (§2.8, §3) - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) key_len ) ) { out.malformed = true; return out; } + if ( !TableUtf8Valid( r.buffer + r.offset, key_len ) ) { out.malformed = true; return out; } out.key = (const char *) ( r.buffer + r.offset ); out.length = (int32_t) key_len; out.over = key_len > 8; // KEYS NEVER CLAMP: the entry is dropped whole @@ -4891,7 +4901,7 @@ inline EdgeRowNamesEntryKeyRead EdgeRowNamesEntryReadKey( const uint8_t * body, uint64_t key_len = 0; if ( !r.getleb( key_len ) || !r.room( key_len ) ) { out.malformed = true; return out; } // a key a string value would refuse as malformed makes the MAP malformed (§2.8, §3) - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) key_len ) ) { out.malformed = true; return out; } + if ( !TableUtf8Valid( r.buffer + r.offset, key_len ) ) { out.malformed = true; return out; } out.key = (const char *) ( r.buffer + r.offset ); out.length = (int32_t) key_len; out.over = key_len > 300; // KEYS NEVER CLAMP: the entry is dropped whole @@ -5060,9 +5070,9 @@ MAPDEMO_TABLE_INLINE bool RowEntriesEntryLoadBody( TableReader & r, RowEntriesEn if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 8 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 8 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.key, r.buffer + r.offset, (size_t) keep ); value.key[keep] = 0; value.key_length = (int32_t) keep; @@ -6481,9 +6491,9 @@ MAPDEMO_TABLE_INLINE bool EdgeRowNamesEntryLoadBody( TableReader & r, EdgeRowNam if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.key[0] = 0; value.key_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 300 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 300 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 300 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 300 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.key, r.buffer + r.offset, (size_t) keep ); value.key[keep] = 0; value.key_length = (int32_t) keep; diff --git a/testdata/golden/tables/messages/MessagesTable.h b/testdata/golden/tables/messages/MessagesTable.h index adb46091d..37ce78167 100644 --- a/testdata/golden/tables/messages/MessagesTable.h +++ b/testdata/golden/tables/messages/MessagesTable.h @@ -595,13 +595,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -610,7 +614,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -626,9 +630,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -2652,9 +2662,9 @@ MESSAGEDEMO_TABLE_INLINE bool UserLoadBody( TableReader & r, User & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -2972,9 +2982,9 @@ MESSAGEDEMO_TABLE_INLINE bool ScriptLoadBody( TableReader & r, Script & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 64 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 64 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.path, r.buffer + r.offset, (size_t) keep ); value.path[keep] = 0; value.path_length = (int32_t) keep; @@ -4116,9 +4126,9 @@ MESSAGEDEMO_TABLE_INLINE bool InsertTextLoadBody( TableReader & r, InsertText & if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.text[0] = 0; value.text_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.text[0] = 0; value.text_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 32 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 32 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 32 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.text, r.buffer + r.offset, (size_t) keep ); value.text[keep] = 0; value.text_length = (int32_t) keep; @@ -6628,9 +6638,9 @@ MESSAGEDEMO_TABLE_INLINE bool OpenDocumentLoadBody( TableReader & r, OpenDocumen if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 64 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 64 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.path, r.buffer + r.offset, (size_t) keep ); value.path[keep] = 0; value.path_length = (int32_t) keep; @@ -7062,9 +7072,9 @@ MESSAGEDEMO_TABLE_INLINE bool SaveDocumentLoadBody( TableReader & r, SaveDocumen if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.path[0] = 0; value.path_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 64 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 64 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 64 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.path, r.buffer + r.offset, (size_t) keep ); value.path[keep] = 0; value.path_length = (int32_t) keep; @@ -7773,9 +7783,9 @@ MESSAGEDEMO_TABLE_INLINE bool TransactionLoadBody( TableReader & r, Transaction if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.reason[0] = 0; value.reason_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.reason[0] = 0; value.reason_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.reason, r.buffer + r.offset, (size_t) keep ); value.reason[keep] = 0; value.reason_length = (int32_t) keep; diff --git a/testdata/golden/tables/pointers/GraphTable.h b/testdata/golden/tables/pointers/GraphTable.h index 3c525c056..78ed647ad 100644 --- a/testdata/golden/tables/pointers/GraphTable.h +++ b/testdata/golden/tables/pointers/GraphTable.h @@ -649,13 +649,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -664,7 +668,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -680,9 +684,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -3981,9 +3991,9 @@ GRAPHDEMO_TABLE_INLINE bool MetaLoadBody( TableReader & r, Meta & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.tag[0] = 0; value.tag_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.tag[0] = 0; value.tag_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 8 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 8 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.tag, r.buffer + r.offset, (size_t) keep ); value.tag[keep] = 0; value.tag_length = (int32_t) keep; @@ -4391,9 +4401,9 @@ GRAPHDEMO_TABLE_INLINE bool SettingsLoadBody( TableReader & r, Settings & value if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.label, r.buffer + r.offset, (size_t) keep ); value.label[keep] = 0; value.label_length = (int32_t) keep; @@ -4811,9 +4821,9 @@ inline bool ListNodeLoadBody( TableReader & r, const TableNodeMap & nodes, ListN if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 12 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 12 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -5167,9 +5177,9 @@ inline bool TreeNodeLoadBody( TableReader & r, const TableNodeMap & nodes, TreeN if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 12 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 12 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.label, r.buffer + r.offset, (size_t) keep ); value.label[keep] = 0; value.label_length = (int32_t) keep; @@ -5946,9 +5956,9 @@ inline bool SceneLoadBody( TableReader & r, const TableNodeMap & nodes, Scene & if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 24 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 24 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 24 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 24 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -6780,9 +6790,9 @@ inline bool DepotLoadBody( TableReader & r, const TableNodeMap & nodes, Depot & if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 12 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 12 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 12 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; @@ -7333,9 +7343,9 @@ inline bool AlbumLoadBody( TableReader & r, const TableNodeMap & nodes, Album & if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; diff --git a/testdata/golden/tables/pointers/MarksTable.h b/testdata/golden/tables/pointers/MarksTable.h index b820d638a..ebd9b823e 100644 --- a/testdata/golden/tables/pointers/MarksTable.h +++ b/testdata/golden/tables/pointers/MarksTable.h @@ -646,13 +646,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -661,7 +665,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -677,9 +681,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -3986,9 +3996,9 @@ inline bool MarkerLoadBody( TableReader & r, const TableNodeMap & nodes, Marker if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.label[0] = 0; value.label_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 8 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 8 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.label, r.buffer + r.offset, (size_t) keep ); value.label[keep] = 0; value.label_length = (int32_t) keep; diff --git a/testdata/golden/tables/pointers/PartsTable.h b/testdata/golden/tables/pointers/PartsTable.h index eba54dd36..281a424b6 100644 --- a/testdata/golden/tables/pointers/PartsTable.h +++ b/testdata/golden/tables/pointers/PartsTable.h @@ -646,13 +646,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -661,7 +665,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -677,9 +681,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -3545,9 +3555,9 @@ GRAPHDEMO_TABLE_INLINE bool StampLoadBody( TableReader & r, Stamp & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.tag[0] = 0; value.tag_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.tag[0] = 0; value.tag_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 8 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 8 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 8 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.tag, r.buffer + r.offset, (size_t) keep ); value.tag[keep] = 0; value.tag_length = (int32_t) keep; diff --git a/testdata/golden/tables/scalars/ScalarsTable.h b/testdata/golden/tables/scalars/ScalarsTable.h index dbf712724..41d05e1dc 100644 --- a/testdata/golden/tables/scalars/ScalarsTable.h +++ b/testdata/golden/tables/scalars/ScalarsTable.h @@ -614,13 +614,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -629,7 +633,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -645,9 +649,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) { - if ( length <= bound ) { return length; } + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; diff --git a/testdata/golden/tables/stream/StreamTable.h b/testdata/golden/tables/stream/StreamTable.h index 71c8a89c2..9259a99d6 100644 --- a/testdata/golden/tables/stream/StreamTable.h +++ b/testdata/golden/tables/stream/StreamTable.h @@ -628,13 +628,17 @@ inline double TableWidenF32( uint32_t bits ) // and a code point past U+10FFFF, which is SPEC.md §4.7's rule in this wire's // idiom: the field reads its declared default, one malformed counts, and the // parent reads on past L. -inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) +// +// A LENGTH IS A 64-BIT NUMBER (§3), so it arrives as one: a payload length is +// whatever the wire spelled, and narrowing it to a signed count would read +// 0xFFFFFFFFFFFFFFFF as an empty payload. +inline bool TableUtf8Valid( const uint8_t * bytes, uint64_t length ) { - int64_t i = 0; + uint64_t i = 0; while ( i < length ) { const uint8_t lead = bytes[i]; - int64_t continuations; + uint64_t continuations; uint32_t code_point; if ( lead == 0 ) { return false; } if ( lead < 0x80 ) { i++; continue; } @@ -643,7 +647,7 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) else if ( ( lead & 0xF8 ) == 0xF0 ) { continuations = 3; code_point = lead & 0x07; } else { return false; } if ( i + continuations >= length ) { return false; } - for ( int64_t k = 1; k <= continuations; k++ ) + for ( uint64_t k = 1; k <= continuations; k++ ) { if ( ( bytes[i + k] & 0xC0 ) != 0x80 ) { return false; } code_point = ( code_point << 6 ) | uint32_t( bytes[i + k] & 0x3F ); @@ -659,9 +663,15 @@ inline bool TableUtf8Valid( const uint8_t * bytes, int64_t length ) // A CLAMP CUTS AT A CODE POINT BOUNDARY (§3, §16.2): the last whole code point // that fits within the bound, over a payload the check above already accepted, // so a clamp can never invent ill-formed storage. -inline int64_t TableUtf8Clamp( const uint8_t * bytes, int64_t length, int64_t bound ) -{ - if ( length <= bound ) { return length; } +// +// THE ANSWER IS NEVER ABOVE THE BOUND. The length arrives as the wire's own +// 64-bit number and the caller turns the answer back into the size of a copy, +// so a length no reader could have bounded has to leave here bounded: taken as +// a signed count, 0xFFFFFFFFFFFFFFFF is -1, -1 is under every bound, and the +// copy would run at SIZE_MAX. +inline int64_t TableUtf8Clamp( const uint8_t * bytes, uint64_t length, int64_t bound ) +{ + if ( length <= (uint64_t) bound ) { return (int64_t) length; } int64_t cut = bound; while ( cut > 0 && ( bytes[cut] & 0xC0 ) == 0x80 ) { cut--; } return cut; @@ -3527,9 +3537,9 @@ STREAMDEMO_TABLE_INLINE bool HeaderLoadBody( TableReader & r, Header & value ) if ( !r.getleb( len ) || !r.room( len ) ) { r.report->malformed = true; return false; } // ILL-FORMED TEXT IS DAMAGE (§3, §4): the field reads its declared // default, one malformed counts, and the parent reads on past L - if ( !TableUtf8Valid( r.buffer + r.offset, (int64_t) len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } + if ( !TableUtf8Valid( r.buffer + r.offset, len ) ) { r.report->malformed = true; value.name[0] = 0; value.name_length = 0; r.offset += (int64_t) len; break; } uint64_t keep = len; - if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, (int64_t) len, 16 ); r.report->clamped++; } // at a code point boundary (§3) + if ( keep > 16 ) { keep = (uint64_t) TableUtf8Clamp( r.buffer + r.offset, len, 16 ); r.report->clamped++; } // at a code point boundary (§3) memcpy( value.name, r.buffer + r.offset, (size_t) keep ); value.name[keep] = 0; value.name_length = (int32_t) keep; From 9ae8716845e650456d96b792bb5ac5a11c4090ef Mon Sep 17 00:00:00 2001 From: Glenn Fiedler Date: Sun, 6 Sep 2026 04:26:03 -0700 Subject: [PATCH 2/2] tables: the length control names the red the missing fit check now produces The control removes `r.room( len )` and required the wire fuzzer to go red because the leg DIED on the mutant. With the clamp answering within its bound no leg dies there: the first mutant that exposes the loss is the length pass's 0xFFFFFFFFFFFFFFFF on root_full, mutant 629, where the sabotaged leg takes the payload over bytes the mutant never carried, steps its cursor by a length the body never had, and reports a kind mismatch the oracle does not. That red is an in-bounds deterministic computation, the same on every allocator, where the death it replaces was a wrapped memcpy size that faulted on one libc and corrupted the heap on another. The named reason becomes the report difference, and the comment says what the sabotage now does. Co-Authored-By: Claude Fable 5.1 --- Makefile | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 52e307aad..48004e92d 100644 --- a/Makefile +++ b/Makefile @@ -3525,13 +3525,15 @@ tables-wire-fuzz-negative-control: tables-wire-fuzz-length-negative-control tabl # the string read's `room( len )`. THE ONE CONTENT RULE THE WIRE HAS # (docs/SPEC-TABLES.md §3, §4) reads a kind `12` payload AS IT ARRIVES, over # the whole of `L` and before the reader's own bound, so `room( len )` is what -# says those bytes are there at all: a sabotaged leg walks off the end of the -# buffer on a forged length and DIES on the mutant, where the oracle stops at -# the body and reports. A LENGTH IS A 64-BIT NUMBER (§3), so the check is -# unsigned: cast to int64 first, 0xFFFFFFFFFFFFFFFF reads as -1 and a negative -# length looks like room, which is the length this pass plants. +# says those bytes are there at all. A sabotaged leg believes a forged `L`: it +# takes the payload over bytes the mutant never carried and steps its cursor by +# a length the body never had, so the fields after it decode out of bytes that +# are not their own and the leg reports counters the oracle does not. A LENGTH +# IS A 64-BIT NUMBER (§3) and every reader of one takes it unsigned, so the +# length this pass plants is 0xFFFFFFFFFFFFFFFF, the largest the wire can spell +# and the one no buffer ever has room for. tables-wire-fuzz-length-negative-control: build/conformance-harness - $(call wire_fuzz_control,length,internal/codegen/cpptable/codecs.go,s|if ( !r.getleb( len ) \|\| !r.room( len ) ) { r.report->malformed = true; return false; }|if ( !r.getleb( len ) ) { r.report->malformed = true; return false; } // NEGATIVE CONTROL: the fit check is gone|,the leg died on the mutant) + $(call wire_fuzz_control,length,internal/codegen/cpptable/codecs.go,s|if ( !r.getleb( len ) \|\| !r.room( len ) ) { r.report->malformed = true; return false; }|if ( !r.getleb( len ) ) { r.report->malformed = true; return false; } // NEGATIVE CONTROL: the fit check is gone|,the report differs) # the numbering's `index - 1 >= map.count`: an index past the node table then # reads a directory entry the region does not hold