External Storage Integration: WorkflowWorker, replay, history by cconstable · Pull Request #3017 · temporalio/sdk-java

cconstable · 2026-08-18T20:40:18Z

What was changed

Integrated external storage into the replay handler and history iterator.
WorkflowWorker handle + send response now use external storage.

Why

Workflow workers should utilize external storage and fetch payloads on replay.

Checklist

Added tests.

… replay handler.

…nal payloads after fetching history.

…ng command.getAttributesCase() + switch for exhaustiveness checking.

…ails.

…ellation gracefully during shutdown

…UTES

…retarget to parent for storage target.

…'t fail

…ead of threading it through and cancel extstore AFTER the poller shutdown flag is set (was previously before).

cconstable · 2026-09-02T18:21:14Z

+    } catch (CancellationException e) {
+      // if the worker is shutting down, extstore will throw a CancellationException and we need to
+      // rethrow it here so the handle() method can decide what to do.
+      throw e;


Rather than absorbing a cancellation exception it's just rethrown. I think this covers all the cases we want and it allows the caller (handle()) to determine whether or not to actually send a task failure to the server... if we are in the middle of a forced shutdown it will see this cancellation error and just absorb it instead of throwing (which would result in a task failure being emitted).

cconstable · 2026-09-02T18:23:31Z

+        || ex instanceof CancellationException
+        || ex.getCause() instanceof CancellationException


During shutdown we shouldn't emit task failures due to things being cancelled because of the shutdown. At the moment, I don't think this actually catches any external storage stuff (since this check happens at a higher level than where the external storage exceptions are caught) but I think it's the "correct" behavior to add regardless and would prevent future issues. Would like some feedback on this @Quinn-With-Two-Ns or @maciejdudko

cconstable · 2026-09-02T18:29:30Z

+            } catch (CancellationException e) {
+              if (!options.getStorageCancellation().isCancellationRequested()) {
+                throw e;
+              }
+              log.trace("Abandoned a workflow task while the worker was shutting down", e);
+              return;


When extstore throws a cancellation, if we are not forcefully shutting down then rethrow the error and report a failure. Otherwise, we should accept that the cancellation is just a normal part of the shutdown. The same logic was added below. It's possible that this cancellation exception might not originate from external storage in the future but I still think this is the correct course of action.

jmaeagle99 · 2026-09-02T22:29:42Z

+                          .setCompletedType(QueryResultType.QUERY_RESULT_TYPE_FAILED)
+                          .setErrorMessage(failure.getMessage())
+                          .setFailure(failure);
+                  sendDirectQueryCompletedResponse(


I think we should send this without offloading to external storage. Recovering from an external storage error to attempt to send a failure message that external storage failed previous and use external storage on that message is likely going to fail again. This would also make it consistent with Go and Python. It should also clear taskFailedCause so that metrics are not emitted for a failure that was not actually reported.

jmaeagle99 · 2026-09-02T22:30:30Z

+                          .setCause(
+                              WorkflowTaskFailedCause
+                                  .WORKFLOW_TASK_FAILED_CAUSE_WORKFLOW_WORKER_UNHANDLED_FAILURE);
+                  sendTaskFailed(


I think we should send this without offloading to external storage. Recovering from an external storage error to attempt to send a failure message that external storage failed previous and use external storage on that message is likely going to fail again. This would also make it consistent with Go and Python. It should also clear taskFailedCause so that metrics are not emitted for a failure that was not actually reported.

jmaeagle99 · 2026-09-02T22:38:59Z

@@ -473,7 +611,10 @@
                          .setErrorMessage(failure.getMessage())
                          .setFailure(failure);
                  sendDirectQueryCompletedResponse(


This should be wrapped in try-catch ExternalStorageTaskFailure and recover by just logging and abandoning sending to the server. See https://github.com/temporalio/sdk-go/blob/57cc5a7d619493f2e67166e2200e8e3758a72ae0/internal/internal_task_pollers.go#L764-L803 as an example. It should also clear taskFailedCause so that metrics are not emitted for a failure that was not actually reported.

jmaeagle99 · 2026-09-02T22:44:51Z

                          .setCause(
                              WorkflowTaskFailedCause
                                  .WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE);
                  sendTaskFailed(


This should be wrapped in try-catch ExternalStorageTaskFailure and recover by just logging and abandoning sending to the server. See https://github.com/temporalio/sdk-go/blob/57cc5a7d619493f2e67166e2200e8e3758a72ae0/internal/internal_task_pollers.go#L764-L803 as an example. It should also clear taskFailedCause so that metrics are not emitted for a failure that was not actually reported.

cconstable force-pushed the extstore/workflow-worker branch from dff9b7f to 3a20dcf Compare August 18, 2026 21:04

cconstable mentioned this pull request Aug 19, 2026

External Storage Integration: Lazy resolving references, general refactoring #3016

Merged

cconstable changed the base branch from main to extstore/foundation August 19, 2026 15:58

cconstable force-pushed the extstore/foundation branch from 831c284 to 460bfbf Compare August 19, 2026 17:40

cconstable changed the base branch from extstore/foundation to main August 19, 2026 17:53

cconstable changed the base branch from main to extstore/foundation August 19, 2026 17:53

cconstable force-pushed the extstore/workflow-worker branch from 3a20dcf to 0e37f9d Compare August 19, 2026 18:12

cconstable changed the title ~~extstore/workflow worker~~ External Storage Integration: WorkflowWorker, replay, history Aug 19, 2026

cconstable force-pushed the extstore/workflow-worker branch from 0e37f9d to c67e0f0 Compare August 19, 2026 18:30

cconstable marked this pull request as ready for review August 19, 2026 18:57

cconstable requested a review from a team as a code owner August 19, 2026 18:57

cconstable force-pushed the extstore/foundation branch from 460bfbf to b3804da Compare August 21, 2026 20:35

cconstable force-pushed the extstore/workflow-worker branch from c67e0f0 to e292757 Compare August 21, 2026 21:57

cconstable force-pushed the extstore/foundation branch from b3804da to ca09b50 Compare August 24, 2026 01:23

cconstable force-pushed the extstore/workflow-worker branch from e292757 to cb05443 Compare August 24, 2026 01:25

cconstable commented Aug 24, 2026

View reviewed changes

Comment thread temporal-sdk/src/main/java/io/temporal/internal/worker/WorkflowWorker.java Outdated

cconstable force-pushed the extstore/workflow-worker branch 5 times, most recently from 6d90f7e to 1538817 Compare August 27, 2026 21:28

cconstable force-pushed the extstore/foundation branch from 2d2d90e to 504467c Compare August 28, 2026 16:10

Base automatically changed from extstore/foundation to main August 28, 2026 17:27

cconstable force-pushed the extstore/workflow-worker branch from 1538817 to d7b9a62 Compare August 28, 2026 21:36

cconstable added 6 commits August 31, 2026 11:47

feature(extstore): integrate into workflow worker pipeline, including…

09573ca

… replay handler.

feat(extstore): make sure sticky cache miss path also retrieves exter…

55b9f7b

…nal payloads after fetching history.

refactor(extstore): refactor the way we derive storage targets by usi…

03c16cd

…ng command.getAttributesCase() + switch for exhaustiveness checking.

Explicitly pass cancellation tokens for external storage methods.

0a91150

more cancellation token threading

4a7ae5e

externalStorage -> externalStorageRunner

8062db7

cconstable force-pushed the extstore/workflow-worker branch from d7b9a62 to 8062db7 Compare August 31, 2026 15:49

fix(extstore): return workflow task failures properly when extstore f…

7be2124

…ails.

cconstable mentioned this pull request Aug 31, 2026

External Storage Integration: NexusWorker #3018

Open

jmaeagle99 reviewed Sep 1, 2026

View reviewed changes

cconstable added 11 commits September 1, 2026 13:39

fix(extstore): don't fail tasks when extstore fails more than once.

7ccf9a0

fix(extstore): add error for worker shutdown and handle extstore canc…

7d02fc0

…ellation gracefully during shutdown

fix(extstore): correctly target SCHEDULE_ACTIVITY_TASK_COMMAND_ATTRIB…

268207f

…UTES

fix(extstore): COMPLETE_WORKFLOW_EXECUTION_COMMAND_ATTRIBUTES should …

f0f193f

…retarget to parent for storage target.

fix(extstore): run extstore inside the try of replay handler.

4f2550e

fix replay tests

2f3b425

add more tests

7ffd079

consolidate test storage drivers.

5a7e291

if the worker is shutting down and extstore retrieve is canceled, don…

c29f141

…'t fail

refactor(extstore): centralize cancellation token in the options inst…

62ce0f3

…ead of threading it through and cancel extstore AFTER the poller shutdown flag is set (was previously before).

only ignore extstore cancelation errors if a shutdown was requested

d673502

cconstable commented Sep 2, 2026

View reviewed changes

jmaeagle99 approved these changes Sep 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

External Storage Integration: WorkflowWorker, replay, history - #3017

External Storage Integration: WorkflowWorker, replay, history#3017
cconstable wants to merge 18 commits into
mainfrom
extstore/workflow-worker

cconstable commented Aug 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cconstable Sep 2, 2026

Uh oh!

cconstable Sep 2, 2026 •

edited

Loading

Uh oh!

cconstable Sep 2, 2026

Uh oh!

jmaeagle99 Sep 2, 2026

Uh oh!

jmaeagle99 Sep 2, 2026

Uh oh!

jmaeagle99 Sep 2, 2026

Uh oh!

jmaeagle99 Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		\|\| ex instanceof CancellationException
		\|\| ex.getCause() instanceof CancellationException

Uh oh!

Conversation

cconstable commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cconstable Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

cconstable Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cconstable Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeagle99 Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeagle99 Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeagle99 Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

jmaeagle99 Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cconstable commented Aug 18, 2026 •

edited

Loading

cconstable Sep 2, 2026 •

edited

Loading