convex-db: harden Postgres wire decoder against malformed pre-auth frames

[PrazwalR]

Problem

PgMessageDecoder runs on raw bytes from any client that reaches the Postgres
port, before authentication (default is trust mode, PgServer.java builds the
pipeline as PgMessageDecoder + PgProtocolHandler with no
LengthFieldBasedFrameDecoder and no max frame size). Several wire-supplied
length/count fields were used unchecked:

• Unbounded frame length (decodeStartup, decodeRegular): a client can
  declare a length near Integer.MAX_VALUE and stream bytes, forcing Netty's
  cumulator to buffer up to ~2 GB per connection → heap exhaustion / OOM. A tiny
  length also made length - 4 negative, skipping the "wait for full message"
  guard and mis-parsing the frame.
• Negative counts in PARSE/BIND (paramCount, numParamFormats,
  numParams, numResultFormats, per-param paramLen) were used directly as
  array sizes → NegativeArraySizeException (cheap per-connection crash).
• Unterminated C-string (readCString) scanned with no upper bound past the
  frame → IndexOutOfBoundsException.

This is a pre-authentication DoS surface. Corroborates #555 (convex-db memory
use).

Fix

Validate all wire-supplied lengths/counts in PgMessageDecoder before
allocating or looping. Reject malformed input with CorruptedFrameException,
which Netty routes to the existing PgProtocolHandler.exceptionCaught (closes
the channel cleanly). Changes:

• 32 MB cap (MAX_MESSAGE_LENGTH) plus minimum-length checks on startup and
  regular frames.
• Reject negative paramCount / numParamFormats / numParams /
  numResultFormats, and any paramLen < 0 other than the -1 NULL sentinel
  (also capped at MAX_MESSAGE_LENGTH).
• Bound readCString to the buffer's readable region; throw on a missing NUL.

No changes to PgProtocolHandler or the SQL-rewrite hacks (separate #560). The
CURRENT_DATABASE string concatenation and the absence of a connection limit
are noted follow-ups, out of scope here.

Tests

New PgMessageDecoderTest (Netty EmbeddedChannel, 9 cases): oversized/too-small
length on startup and regular frames, negative numParams / paramLen, negative
PARSE paramCount, unterminated C-string, and a well-formed startup + query
regression. All assert CorruptedFrameException rather than OOM / crash.
mvn -pl convex-db test -Dtest=PgMessageDecoderTest # 9/9 pass
mvn -q -pl convex-db test -Dtest=PgServerTest # no regression

[Copilot]

Pull request overview

This PR hardens convex-db’s PostgreSQL wire-protocol decoder (PgMessageDecoder) against malformed pre-authentication frames to reduce the risk of unauthenticated denial-of-service (OOM / decoder crashes) on the Postgres port.

Changes:

• Add a maximum message size (MAX_MESSAGE_LENGTH, 32MB) plus minimum-length checks for startup and regular frames, rejecting invalid lengths via CorruptedFrameException.
• Validate several wire-supplied counts/lengths (e.g. negative paramCount, numParams, etc.) before allocating arrays / reading parameter values.
• Add a new PgMessageDecoderTest using Netty EmbeddedChannel to assert malformed inputs are rejected with CorruptedFrameException.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
convex-db/src/main/java/convex/db/psql/PgMessageDecoder.java	Adds message-length caps and additional validation / bounds checks to reject malformed frames early.
convex-db/src/test/java/convex/db/psql/PgMessageDecoderTest.java	Adds regression tests for oversized/undersized frames, negative counts/lengths, and unterminated C-strings.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.