Skip to content

Complete configured Volume transfers through the native backend - #613

Open
aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113
Open

aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan_data/feat/volume-restart-20261113

Conversation

@aakash-saravanan-db

Copy link
Copy Markdown
Collaborator

Feature or gap

Configured SQL Volume operations can return instruction rows without transferring files on the native backend.

What this PR addresses

Invoke shared completion when stagingAllowedLocalPath is supplied, preserve raw opt-out and error mapping, and route cancellation to active transfers. Retain direct results for configured async execution, whose facade already waits for transfer completion.

Dependencies

Shared implementation: kernel change. Requires a compatible published native package and updated platform dependency versions. These release versions remain unselected.

Tests and validation

The full unit suite, affected async/cancellation tests, build and type checks pass. Independent sync/async byte, deletion and error checks pass; five-run comparisons meet ±10%, including changed-byte async overwrite. Windows and published-package proof remain pending.

@github-actions

Copy link
Copy Markdown

Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase (git rebase -i main).

@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from ad49367 to 7b276d7 Compare November 14, 2026 00:40
@github-actions github-actions Bot removed the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
@aakash-saravanan-db aakash-saravanan-db added the kernel-e2e Preview-run the SEA Kernel E2E suite on this PR label Nov 14, 2026
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/volume-restart-20261113 branch from 7b276d7 to 12876fc Compare November 14, 2026 00:52
@aakash-saravanan-db aakash-saravanan-db added kernel-e2e Preview-run the SEA Kernel E2E suite on this PR and removed kernel-e2e Preview-run the SEA Kernel E2E suite on this PR labels Nov 14, 2026
@aakash-saravanan-db
aakash-saravanan-db marked this pull request as ready for review November 14, 2026 02:18

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly sound, with one medium issue: on the kernel path, the new StagingError wrapping can also catch the statement's own execution errors, not just transfer errors. Nits: there is a stray blank line in the CHANGELOG entry, and the constructor comment in KernelOperationBackend still says cancel() "always routes to the cancellable execution". Follow-up (not caused by this PR): executeStatement waits for the transfer before it returns the operation, so a caller can only interrupt an in-flight transfer by closing the session.

this.volumeCompleted = outcome === 'completed';
} catch (err) {
const mapped = decodeNapiKernelError(err);
if (

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium — The try covers getFetchHandle() as well as completeVolumeOperation(). On the cancellable-execution path and the directResults Running arm, getFetchHandle() is where the statement's own execution error comes out (result() / awaitResult() rejecting). Kernel SQL-error envelopes can carry httpStatus (for example a SqlError with 400). In that case the kernelMetadata?.httpStatus branch turns a server SQL failure into a plain StagingError(message), which drops the OperationStateError class, sqlState, kernelMetadata and cause.

This also hits ordinary non-Volume queries. DBSQLSession now calls completeStagingOperation for every kernel statement that has stagingAllowedLocalPath set, so a failing SELECT run with that option gets the wrong error type.

Suggested fix: resolve the handle outside the try, and only reclassify errors that come from completeVolumeOperation() (the transfer's HTTP or path-validation failures). If you still wrap those, consider keeping cause/kernelMetadata on the StagingError.

expect(executionCancelled).to.equal(false);
});

for (const outcome of ['completed', 'not_volume']) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — The new tests cover only the completed and not_volume outcomes and the cancel routing. These new branches in completeStagingOperation have no unit test:

  • the needs_input_stream → StagingError path
  • reclassifying ParameterError and httpStatus errors as StagingError
  • passing other errors through unchanged (for example local fs errors decoded with code/path)

No test checks that DBSQLSession closes the operation when completion fails either. The error-classification branch is where F1 lives, so a test there would have caught it.

This branch was successfully deployed

1 active deployment
azure-prod — 12876fc7 Deployed Nov 14, 2026 by aakash-saravanan-db via run-kernel-e2e #858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel-e2e Preview-run the SEA Kernel E2E suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant