[feature] Transaction-aware command execution via ExecutionMode

[stephanpelikan]

Summary

The Process Engine API currently returns CompletableFutures for all engine interactions, making it fully asynchronous. While this is a clean and engine-agnostic design, it does not address a critical concern in real-world process applications: consistency between application (domain) state and process engine state within a single transaction.

This feature request proposes an ExecutionMode parameter on all command methods that enables transaction-aware execution without changing the API surface — no method duplication, no new interfaces, just an additional optional parameter with a sensible default.

When discussing the need for the Process Engine API on founding the BPM Crafters, we had in mind to migrate VanillaBP once, to work on top of the Process Engine API, as VanillaBP provides a higher level of abstraction. To start migration, kind of TX-support needs to be available at the Process Engine API level — either as first-class features or as well-defined extension points.

Context & Motivation

The Problem

When a @Transactional service method completes a user task, starts a process, or correlates a message, two state changes must be consistent:

1. Domain state — the business entity is updated in the application database.
2. Process engine state — the engine advances to the next step.

If the domain transaction rolls back but the engine command has already been dispatched (or vice versa), the system ends up in an inconsistent state. This is exactly the bug described in process-engine-adapters-camunda-7#156 and process-engine-adapters-cib-seven#30.

Design Principles

1. Composable building blocks. Direct users get a convenient default mode. Higher-order frameworks (like VanillaBP or process-engine-worker) can compose the lower-level modes to implement advanced patterns.
2. Full backward compatibility. Existing code continues to work without changes.
3. Adapter-type transparency. The application code does not need to know whether the adapter is embedded or remote — the adapter interprets each mode according to its own capabilities.
4. Separation of concerns. Transaction awareness is platform-specific and has to be
   done by higher-order implementations (e.g. process-engine-worker or VanillaPB).

Proposed API Change: ExecutionMode

The Enum

enum class ExecutionMode {
    DEFAULT,
    // Placeholder that adapters interpret as ASYNC.
    // Higher-order adapters (e.g. process-engine-worker) can intercept DEFAULT and
    // substitute the appropriate mode based on their own configuration.

    PREFLIGHT_CHECK,
    // Performs only a pre-flight feasibility check, asynchronously.
    // No engine command is executed. No transaction hook is registered.
    // The returned Future completes with success if the command would be
    // feasible, or with an exception if not.
    //
    // Intended as a building block for higher-order frameworks that need
    // to orchestrate multiple adapters or custom transaction strategies.

    ASYNC,
    // Current behaviour. Command is executed asynchronously. No transaction
    // awareness. The returned Future completes when the engine has processed
    // the command.
    //
    // This mode does NOT perform a pre-flight check. The caller is responsible
    // for knowing that the command is likely to succeed (e.g. after a prior
    // PREFLIGHT_CHECK call).

    SYNC,
    // Execute synchronously in the caller's thread and transaction context.
    // The returned Future completes immediately (already resolved).
    //
    // What "execute" means depends on the adapter type:
    //   - Embedded engine: runs the engine command in the current DB transaction.
    //   - Remote engine:   writes an outbox entry in the current DB transaction.
    //                      Actual dispatch happens asynchronously after commit.
    //
    // This mode does NOT perform a pre-flight check. The caller is responsible
    // for knowing that the command is likely to succeed (e.g. after a prior
    // PREFLIGHT_CHECK call). If the command fails, the exception propagates
    // synchronously — allowing the caller's transaction to roll back.
}

Method Signature (example)

fun completeTask(
    cmd: CompleteTaskCmd,
    mode: ExecutionMode = ExecutionMode.DEFAULT
): CompletableFuture<Empty>

All existing command methods (startProcess, completeTask, correlateMessage, sendSignal, etc.) gain this optional parameter. Existing callers are unaffected because DEFAULT resolves to ASYNC, preserving current behavior.

Usage Patterns

Pattern 1: The DEFAULT Mode and Higher-Order Adapters

The developer writes business code; the API handles the rest. If transaction support is needed, the user has
to choose an higher-level adapter which is aware of this (e.g. process-engine-worker).

// Application code
processService.completeTask(cmd);  // mode is DEFAULT (implicit)

// process-engine-worker internally:
//   → interprets DEFAULT according to configuration or @Transactional annotations
// if transactional supported:
//   1. Starts an async pre-flight check (own TX / own connection for
//      embedded engines, remote call for remote engines).
//   2. Registers a beforeCommit hook on the caller's transaction.
//   3. Returns a Future that completes when the pre-flight check completes.
//   4. At beforeCommit: waits for the pre-flight result. On failure → rollback.
//      On success → executes the command synchronously (same semantics as
//      CURRENT_THREAD).

This means DEFAULT is a delegation marker: adapters treat it as ASYNC, but higher-order wrappers can intercept and reinterpret it. The application code doesn't need to choose a mode — the framework decides.

Adapter Responsibilities per Mode:

Mode	Embedded Adapter	Remote Adapter	process-engine-worker
ASYNC	Async engine command (current behaviour)	Async remote call (current behaviour)	Delegate to adapter
SYNC	Engine command in caller's thread/TX	Outbox write in caller's TX, dispatch after commit	Delegate to adapter
PREFLIGHT_CHECK	Like ASYNC but feasibility check in own TX (read-only, separate connection)	Like ASYNC but feasibility check via remote API	Delegate to adapter
DEFAULT	Same as ASYNC	Same as ASYNC	e.g. PREFLIGHT_CHECK + beforeCommit-Hook using SYNC

Key point: SYNC does not perform a pre-flight check. It assumes the caller knows what it's doing (e.g. because it already did a PREFLIGHT_CHECK call). If the command fails, the exception propagates synchronously, which is the desired behaviour — it causes the transaction to roll back.

Implementation Notes

Pre-flight Check Semantics

The pre-flight check is an optimistic fast-fail mechanism, not a guarantee. Between the pre-flight check and the actual execution (in beforeCommit), the engine state can change due to concurrent requests. This is acceptable because:

• For embedded engines, the synchronous execution in beforeCommit will detect conflicts (e.g. optimistic locking) and trigger a rollback.
• For remote engines, eventual consistency is inherent — the outbox entry will be processed, and the remote engine may reject it, requiring compensation.

Connection Pool Considerations

PREFLIGHT_CHECK uses a separate database connection (embedded) or a separate network call (remote). For embedded adapters, this means two connections are held simultaneously during the pre-flight phase. In practice, the pre-flight connection is short-lived and read-mostly. Under extreme connection pool pressure, this should be documented as a known trade-off.

Timeout Configuration

A configurable timeout for waiting on pre-flight results is needed. On timeout, the behaviour should be fail — it's better to fail than to hold a transaction open indefinitely.

Relationship to Existing Work

Artifact	Relevance
process-engine-adapters-camunda-7#156	Documents the state-divergence bug for embedded C7. TRANSACTION_AWARE mode solves this directly.
process-engine-adapters-camunda-7#157	Introduces EngineCommandExecutor for pluggable command dispatch. The ExecutionMode approach is a more general solution at the API level that subsumes this.
process-engine-adapters-cib-seven#30	Same state-divergence bug for CIB 7 embedded.

Acceptance Criteria

• All command methods accept an optional ExecutionMode parameter (default: DEFAULT).
• ASYNC preserves current behaviour (backward compatible).
• SYNC executes synchronously in the caller's thread. For embedded adapters, this participates in the caller's DB transaction. For remote adapters, this may write to an outbox within the caller's transaction.
• PREFLIGHT_CHECK performs an async feasibility check without executing the command.
• DEFAULT behaves as ASYNC at the adapter level and is available for reinterpretation by higher-order frameworks.
• Pre-flight check timeout is configurable.
• The different consistency guarantees for embedded (atomic commit) vs. remote (eventually consistent via after-commit-hook or outbox) are documented.

[zambrovski]

Thank you for the issue. Need to think about the implication. I think we need more use case descriptions here. In some cases the end user uses the API directly (correlation, process start, signalling, deployment, user task completion) and in others not (external task completion). I believe we need to describe this two use cases regarding the parameters and what will be happening for the engines C7-embedded/CIB7-embedded, C7-remote, C8-remote. Let start the description without the outbox pattern. The outbox pattern is in mind but probably another story.

Just a tactical design question: why not put this into an interface TransactionAware and make the commands implement this interface? Would be semantically breaking, but not runtime breaking.

interface TransactionAware {
  fun executionMode(): ExecutionMode = Execution.DEFAULT
}

;-)

[stephanpelikan]

I explicitly didn't call it kind of "transaction-aware" because it is not only about transaction-handling:

1. Transactions are specific to the platform. I'm working on bringing VanillaBP to Quarkus and it uses CDI. Transaction-detection- and commit-hook-handling is totally different. So, transaction-awareness can only be brought in by platform-specific adapters (e.g. process-engine-worker). There might be interfaces be part of the SPI (contract between API-implementation and adapter) but I'm not sure if this is needed.
2. I need this feature also to handle VanillaBP's BPMS-migration functionality. This is not about transactions, just handing over BPMS-calls e.g. in case a task is unknown to an adapter, failing only if the task is unknown to all adapters.

I'm still not sure whether it's feasible to bring VanillaBP onto the Process-Engine-API. There are additional topics VanillaBP currently implementing:

1. VanillaBP supports delegate tasks for C7 by providing a custom expression-resolver. I could bypass this, by automatically transforming them to external-tasks.
2. VanillaBP supports process lifecycle events (receive: created, completed, deleted; send: update) for which no API is there in the Process-Engine-API. For usertask lifecycle events (receive: created, completed, updated, deleted; send: update), the receiving events have to be executed in the right order to the tasks of the process to ensure application data consistency. This is done by using listeners which are not supported by all Process-Engine-adapters (c7-embedded and c7-remote). Additionally, current TaskSubscriptionApi-implementations use APIs not guaranteed to ensure data-consistency (e.g. c8 via Task-API using elastic-search).
3. VanillaBP uses persistence (for business data and internal data like an outbox) and uses kind of a persistence abstraction via interfaces implemented by a VanillaBP itself for defaults or by the business application. This ensures support for a wide range of persistence technologies. Currently, support for JPA and MongoDB is provided.

I don't want to push through a design here if there's no chance that VanillaBP can use the Process-Engine-API anyway. I need your assessment for this. It's difficult to discuss this via individual feature-specific issues, which is why I've posted this concern here.

[zambrovski]

I'm fine to call it differently, I just wanted to understand why do we need to break the command pattern of the API if we could avoid it. If there are no other reasons, I would like to add the ExecutionMode to the command.

I believe your idea is good, but I would like to have a more detailed description how the modes are used in different scenarios (not how they are implemented).

As far I understand for a manual usage of the Process Engine API you never touch / change the mode, but for the high level abstractions like VanillaBP and Process-Engine-Adapter you do... Could you elaborate on the Usage Scenarios please?

I believe it would be a great addition to the process engine api, but I would like to understand the use case more.

[zambrovski]

2. VanillaBP supports process lifecycle events (receive: created, completed, deleted; send: update) for which no API is there in the Process-Engine-API. For usertask lifecycle events (receive: created, completed, updated, deleted; send: update), the receiving events have to be executed in the right order to the tasks of the process to ensure application data consistency. This is done by using listeners which are not supported by all Process-Engine-adapters (c7-embedded and c7-remote). Additionally, current TaskSubscriptionApi-implementations use APIs not guaranteed to ensure data-consistency (e.g. c8 via Task-API using elastic-search).

I believe this is part of the future of process-engine-api. Initially we said that this is not important, but I had to build a light version of it because otherwise the sync was not possible for user tasks.

I believe in a wrong run we need listener support in process-engine-api.

[zambrovski]

3. VanillaBP uses persistence (for business data and internal data like an outbox) and uses kind of a persistence abstraction via interfaces implemented by a VanillaBP itself for defaults or by the business application. This ensures support for a wide range of persistence technologies. Currently, support for JPA and MongoDB is provided.

Yes. If you remember we spoke about this in the one of the first design meetings. Currently, we were able to do a lot without persistence. It is hard but many things are possible. In process-engine-worker we started to use persistence. I believe outbox pattern will need it too. And to be honest - building a task sync without it and without any assumption about the "existing" tasks is just too hard.