KeyValue serialization binding ambiguity causes snapshot recovery failure

[fupelaqu]

Problem

app.softnetwork.kv.model.KeyValue extends both ProtobufDomainObject (bound to proto serializer) and KvState which extends State (bound to chill serializer).

When Akka recovers AccountKeys entity snapshots, it finds multiple matching serializers:

Multiple serializers found for [app.softnetwork.kv.model.KeyValue],
choosing first of: [com.twitter.chill.akka.AkkaSerializer, akka.remote.serialization.ProtobufSerializer]

Can't deserialize object of type [app.softnetwork.kv.model.KeyValue]
in [akka.remote.serialization.ProtobufSerializer]

This causes Recovery failed for AccountKeys-dev.

Root Cause

In softnetwork-persistence.conf, the serialization-bindings map both ProtobufDomainObject → proto and State → chill. Since KeyValue implements both traits, Akka's serialization system picks one non-deterministically, leading to serialization/deserialization mismatch.

Fix

Add an explicit binding in the kv module's reference.conf:

akka.actor.serialization-bindings {
  "app.softnetwork.kv.model.KeyValue" = proto
}

A fix has already been prepared in kv/src/main/resources/reference.conf (branch pending). HOCON merges reference.conf files from all JARs, so this binding will be added to the existing ones from softnetwork-persistence.conf.

Workaround

Downstream projects can add the same binding in their own reference.conf (e.g., in a payment or license-server module). This is what softclient4es-license-server does currently.

Impact

Any project using the kv module with snapshot persistence is affected. The error manifests on entity recovery when a KeyValue snapshot exists in the journal.