Feature/binary datapoint ingest

[olavgg]

What this adds

A second datapoint ingest endpoint, POST /timeseries/data/binary, for the high-rate class. It
exists to move the platform's first scaling limit, the API parsing every value, off the API and
onto the client.

The client resolves each series once, checks and sorts the values, writes them as Arrow IPC
frames in the exact schema the ClickHouse table wants, compresses each frame with zstd, and posts
them. The API validates a frame and forwards its bytes unchanged. Frames travel compressed
through a Pulsar topic of their own and the consumer merges them into large ClickHouse blocks.

The JSON path at POST /timeseries/data is untouched and remains the default.

Arrow IPC rather than ClickHouse Native: the two parse at the same speed on these tables once a
frame holds a thousand points or more, so the choice went to the format that also suits the read
path, that a DataFrame round-trips through, and that any Arrow-capable tool can produce. The
frame envelope keeps a codec byte so a Native payload could be added later without a new media
type.

What it measured

100 million float32 points down each path, on a single 32-core development machine running the
services, Pulsar and ClickHouse together:

	JSON	binary
API CPU	287.8 s	6.3 s	46x
Consumer CPU	64.2 s	6.4 s	10x
Bytes on the wire	5.00 GB	346 MB	14.4x
Ingest wall time	52.0 s	30.4 s	1.7x
API peak resident	3.8 GB	5.2 GB	costs more

The API CPU is the result: accepting a hundred million points costs the JSON path 288 seconds of
a core and the binary path 6. Throughput gains less because the client is then the limit.

Scaling the clients out shows the difference more plainly. The JSON path saturates near 1.75
million points a second and adding clients does not move it, because they all queue behind the
same parsing. The binary path keeps scaling until the host runs out of cores:

Concurrent clients	JSON	binary
1	720,022/s	3,839,090/s
4	1,588,183/s	11,318,630/s
8	1,756,275/s	14,558,392/s

Four clients is the balanced point: a billion points went through at 11.3 million a second with
the Pulsar backlog never passing a hundred messages, so ClickHouse absorbed the rows as fast as
the clients produced them. Eight and twelve clients reach higher rates but the backlog grows,
which means they are using Pulsar's burst absorption rather than sustaining.

Two honest caveats, both recorded in SCALABILITY.md: the benchmark sends 10,000 to 40,000
points per collection, which is the favourable shape for the JSON path, and the generated signal
is a slow sine plus noise with one signal per series. Giving every series identical values let
zstd compress across them and reported 0.52 bytes per point against the honest 3.46.

A defect this found

A TEXT timeseries could not receive datapoints through the JSON endpoint at all. The value-type
switch had no TEXT arm, so every text datapoint fell through to the default and threw
"Unsupported value type: TEXT", which the caller saw as a 500. Everything around it already
assumed text worked: the per-collection cap checks it, the quota counts it, and the consumer
stores it. Fixed here, with a unit test.

It was found by the new end-to-end test, which sends the same points down both paths and compares
what ClickHouse stored.

Also in this branch

Consolidations that the new code made visible, each with tests:

• One value-type catalogue. The seven timeseries value types were written out in four places,
  two of which carried a comment asking the reader to keep the copies in sync. DatapointValueType
  is now the single definition; a parity test replays the value-type inserts of every migration
  and compares them with the enum, so an eighth type added to either side fails the build.
• One latest-value cache. The compare-and-set that writes a series' latest value existed twice
  against the same Valkey key. Its rule now has direct tests, which it never had.
• No series metadata cache. The binary path briefly cached what it knows about a series for
  fifteen seconds. The dataset id is an input to the write ACL, so a stale entry authorises
  against the dataset a series has been moved out of, and its invalidation methods were never
  called. Replaced with a projection query that reads four columns instead of hydrating entities.
• A valueType casing fix. The binary path reported the value type lower case to both
  WebSocket paths where the JSON path reports it upper case, so a client could not treat them
  alike. The two tests covering it had locked in the wrong spelling.
• Request logging no longer stringifies a binary body; it records the word binary.
• ClickHouse pinned to 26.8.2.7 in the test fixtures and the development stack.

Testing

• Unit suites across every touched module.

  • datahub-e2e is new: integrationTest drives the platform through the Java SDK only and
    asserts that the same points sent as JSON and as frames land identically. A bulk series is
    compared by an ordered checksum computed inside ClickHouse rather than by row count, so a
    value mangled by one path's encoding cannot pass, and all seven value types are then sent
    both ways and their stored values compared against each other. The assertion is deliberately
    that a reader cannot tell which path wrote the row, not that storage echoes the input back.
  • datahub-e2e:benchmark is the comparison above, re-runnable. datahub-e2e/README.md has the
    setup, the two product limits that have to be off for a run of that size, and the traps.
  • Integration tests against real Pulsar and ClickHouse for the endpoint, the consumer and the
    Arrow round trip.

  Documentation

  The wire contract for third-party producers, the SDK methods and the new limits are documented
  separately in the docs sites; FrameLimits and ArrowSchemaCanon are the normative caps and
  schemas in code. SCALABILITY.md carries the decision and every measurement above, and the
  working notes that preceded it have been folded into it.