This is kinda an old topic, but it looks like it might now be time to make a decision, create a plan and start implementing it.
This issue arises now because @elfkuzco came to the conclusion that it begins to be too complex to handle messaging between CMS components through DB and proposed to introduce a real messaging solution in openzim/cms#201
This is an old topic already tracked in openzim/zimfarm#1116 for the Zimfarm, and identified like "years ago".
What I've already identified in terms of requirements:
- throughput is expected to be very moderate, in the order of magnitude of few message per seconds
- we need persistence + backup
- we need at least once delivery guarantee ; up to the producer / consumer to achieve exactly once delivery if needed, typically with idempotent code
- we need a solution with acknowledgement ; consumer acknowledge the message has been received before the broker consider it as delivered
- we want this solution to be used only when the message consumer is a technical component and does not already exposes an API ; it will not replace traditional API communication
- we need some form of security to prevent unauthorized access to the messages
What we probably don't need:
- HA solution (like clustering, ...)
- a solution capable to cope with very big messages (we can ask apps to generate messages in the range of 1K or less, or rely on another protocol)
- a solution accessible by web interfaces (end-user UIs)
I assume we have some requirements linked to our k8s infra but it is very unclear to me. Do we have to require HTTPS (+Websocket) or can we accept any protocol (meaning we will expose it directly on the host port, like we already do for other services)?
In terms of plan, I propose that while we identify a good candidate solution, implement it in the CMS first and from there decide if we continue with this solution or change to another alternative. It could be a good moment to also decide if we deploy a central component or continue with one component per app.
@elfkuzco @rgaudin @kelson42 can you help me better identify requirements?
This is kinda an old topic, but it looks like it might now be time to make a decision, create a plan and start implementing it.
This issue arises now because @elfkuzco came to the conclusion that it begins to be too complex to handle messaging between CMS components through DB and proposed to introduce a real messaging solution in openzim/cms#201
This is an old topic already tracked in openzim/zimfarm#1116 for the Zimfarm, and identified like "years ago".
What I've already identified in terms of requirements:
What we probably don't need:
I assume we have some requirements linked to our k8s infra but it is very unclear to me. Do we have to require HTTPS (+Websocket) or can we accept any protocol (meaning we will expose it directly on the host port, like we already do for other services)?
In terms of plan, I propose that while we identify a good candidate solution, implement it in the CMS first and from there decide if we continue with this solution or change to another alternative. It could be a good moment to also decide if we deploy a central component or continue with one component per app.
@elfkuzco @rgaudin @kelson42 can you help me better identify requirements?