Migrate to glaze for all serialisation and deserialisation of trusted sources

[cjen1-msft]

nlohmann is a hardened and secure json parsing library however it ins't very fast (7us vs 100ns for small payloads)
Additionally it requires substantial work to get it to understand the types we want it to serialise.

The idea is that we can use glaze to do all serialisation and also our deserialisation of data from trusted sources (eg to/from the KV).
It also allows us to target different outputs such as CBOR or MsgPack.

[cjen1-msft]

One major constraint we have is that the forward compatibiltiy of glaze is not quite the same as what we have currently, and hence would require careful work to make sure it would be safe.

Here is a write up of some analysis from an AI:

Forwards-compatibility safety: CCF and glaze

Legend: ✅ data preserved, ⚠️ silent value substitution, ❌ runtime error.

---

Matrix 1: CCF DECLARE_JSON_* macros

Approach: declarative per-field policy via macros. Each field is explicitly tagged Required (DECLARE_JSON_REQUIRED_FIELDS) or Optional (DECLARE_JSON_OPTIONAL_FIELDS). On write, optional fields are emitted iff value != default-constructed value. On read, required missing throws ccf::JsonParseError with the field name; optional missing leaves the C++ struct's default in place. Unknown wire keys are silently ignored. Optionality is a property of the declaration, not the field's type.

	Reader: Required	Reader: Optional	Reader: Missing
Writer: Required	✅ value parsed	✅ value parsed	✅ silently ignored
Writer: Optional non-default	✅ value parsed	✅ value parsed	✅ silently ignored
Writer: Optional default	❌ throws	⚠️ reader default substituted	✅ no-op
Writer: Missing	❌ throws	⚠️ reader default substituted	✅ no-op

---

Glaze's levers for "optional"

Glaze gives you several orthogonal knobs. Knowing which combination expresses your intended semantics matters because they default in opposite directions to nlohmann.

Per-call opts — set once per glz::write<opts> / glz::read<opts> invocation, apply to the whole struct:

• error_on_unknown_keys (default true) — whether unknown keys on the wire should error during read. nlohmann silently ignores them; glaze defaults to rejecting.
• error_on_missing_keys (default false) — whether absent keys should error during read. CCF's required-field check is per-field; glaze's is whole-struct via this opt.
• skip_null_members (default true) — whether std::optional<T> members holding nullopt should be omitted on write and exempted from the missing-keys check on read. Two behaviours bundled together.
• skip_default_members (default false) — whether members equal to their hard-coded "natural default" (numeric 0, false, empty string, empty range) should be omitted on write. Doesn't honour user-defined defaults.

Per-field type-driven — encode optionality through C++ types:

• Wrap the field in std::optional<T>. With skip_null_members=true, glaze treats it as nullable: omitted on write when nullopt, exempted from missing-key check on read. Reader can distinguish absence (nullopt) from a valid value (Some(v)) — strictly more informative than CCF.

Per-field meta customisation — attach a glz::meta<T> specialisation:

• meta<T>::requires_key(key, is_nullable) -> bool — per-key compile-time predicate that overrides the default required-set. Returning false waives the missing-key check for that key, even with error_on_missing_keys=true and a non-nullable type. Lets you mark int x as optional.
• meta<T>::skip_if(value, key, mctx) -> bool — per-key runtime predicate called on serialise. Returning true omits the field. Lets you implement "skip if equal to the writer's default-constructed value", matching CCF's WRITE_OPTIONAL byte-for-byte.
• meta<T>::skip(key, mctx) -> bool — compile-time predicate; harder hammer for "drop field entirely from this operation".
• meta<T>::value = glz::object(...) — explicit field list. Required for inherited types (because pure reflection requires aggregate-without-bases) but otherwise optional.

The combinations matter. The same observable semantic can be achieved several ways with different trade-offs:

Goal	Cheapest expression
"Tolerate missing field, give reader nullopt"	Type the field as std::optional<T>
"Tolerate missing field, leave reader's default"	meta::requires_key returning false for that key, plus type stays as plain T
"Omit on write when value equals nullopt"	skip_null_members=true (default) + std::optional<T>
"Omit on write when value equals user-defined default"	meta::skip_if per field
"Allow unknown keys"	error_on_unknown_keys=false
"Error on missing required keys"	error_on_missing_keys=true

---

Reproducing CCF semantics in glaze

The closest cell-for-cell match to CCF that doesn't change field types uses three pieces:

1. Strict opts for the read/write call:

   struct ccf_strict_opts : glz::opts {
     bool error_on_unknown_keys = false;   // unknown keys silently ignored
     bool error_on_missing_keys = true;    // missing required keys error
   };

2. Per-type glz::meta specialisation that uses requires_key (read side) and skip_if (write side) to implement the per-field optional semantics. The skip_if body compares against a static const T t_default{} instance — directly mirroring CCF's WRITE_OPTIONAL macro.

3. A macro to generate (2) so the per-type cost stays comparable to today's DECLARE_JSON_* declarations:

   #define GLZ_DECLARE_OPTIONAL_FIELDS(TYPE, ...) \
     template <> struct glz::meta<TYPE> { \
       using T = TYPE; \
       static bool skip_if(const auto& v, std::string_view k, const glz::meta_context& mctx) { \
         if (mctx.op != glz::operation::serialize) return false; \
         static const T t_default{}; \
         /* unrolled: for each field FIELD, if (k=="FIELD") return v == t_default.FIELD; */ \
         return false; \
       } \
       static constexpr bool requires_key(std::string_view k, bool) { \
         /* unrolled: for each field FIELD, if (k=="FIELD") return false; */ \
         return true; \
       } \
     }

   Per-field expansion uses if constexpr (requires { v == t_default.FIELD; }) to type-gate each comparison so SFINAE ignores branches whose field type doesn't match the current v, allowing the same body to handle every field.

Migration sketch for a struct like JsonWebKey:

// Today:
DECLARE_JSON_TYPE_WITH_OPTIONAL_FIELDS(JsonWebKey);
DECLARE_JSON_REQUIRED_FIELDS(JsonWebKey, kty);
DECLARE_JSON_OPTIONAL_FIELDS(JsonWebKey, kid, x5c);

// With macro-fied glaze meta:
GLZ_DECLARE_OPTIONAL_FIELDS(JsonWebKey, kid, x5c);
// kty automatically required: every field not listed is required under strict opts.

Field types stay unchanged. Caller-side reads use glz::read<ccf_strict_opts>(v, json).

Caveats vs CCF

• Error message granularity: glaze's missing-key error returns error_code::missing_key with only the first missing key in ctx.custom_error_message (the source comment cites "to avoid heap allocations"). CCF reports each missing field by name in a per-call exception. For most uses this is acceptable; for diagnostics-heavy code paths it's a regression.
• Required field cells (Required → Required, etc.): identical behaviour. The match is exact for these cells.
• Inheritance: this macro doesn't yet handle _WITH_BASE. Glaze can't pure-reflect inherited types, so the macro would need to emit an explicit value = glz::object(...) listing both base and derived fields by member-pointer.
• Schema generation: CCF's macros emit fill_json_schema / add_schema_components alongside to_json/from_json. Glaze has separate facilities; if you need schema parity you'd write a parallel macro path.

---

Matrix 2: glaze with macro-fied meta DSL (CCF-equivalent)

Approach: strict opts + GLZ_DECLARE_OPTIONAL_FIELDS(T, fields...) macro generating a glz::meta<T> with requires_key (per-field read-side waiver) and skip_if (per-field write-side compare-to-default). Field types stay as plain non-nullable C++ — int x = 5; can be Optional with default 5 just as in CCF. Surface syntax mirrors the CCF macros one-for-one.

	Reader: Required	Reader: Optional	Reader: Missing
Writer: Required	✅ value parsed	✅ value parsed	✅ silently ignored
Writer: Optional non-default	✅ value parsed	✅ value parsed	✅ silently ignored
Writer: Optional default	❌ error_ctx returned	⚠️ reader default substituted	✅ no-op
Writer: Missing	❌ error_ctx returned	⚠️ reader default substituted	✅ no-op

Cell-for-cell identical to the CCF matrix. The only observable differences are: error reported as error_ctx rather than thrown ccf::JsonParseError, and the missing-key error message names only the first missing field rather than each one.