fix(none): Databricks Volumes check reads the configured volume

[paulkarayan]

What & why

Problem: Customers setting up a Databricks Volumes destination got a green connection test for a volume they could not write to, and found out when the job failed at the upload node, after every file had already been partitioned, enriched, embedded and paid for. The check asked whether the principal was awake (current_user.me().active) and never whether it could write, so a volume that doesn't exist, a typo in the path, a principal without WRITE VOLUME, or a catalog or schema it can't USE all passed.

Change: The uploader's precheck now writes a zero-byte unstructured_precheck_<hex> file at the configured volume path through the same files.upload the run uses, then deletes it. Only a 401, 403 or 404 fails the check, and the error names the volume path. Databricks SDK errors are also classified by their HTTP status even when the SDK raises a more specific subclass, which the missing-volume case depends on.

Blast radius: 3/5 -- one connector's precheck, but it now writes (and deletes) a real file in the customer's volume, and the shared wrap_error classification changes for every Databricks Volumes connector.

Linked ticket

none. The source side of this connector got the equivalent fix in #794.

Why a write probe, and why only three status codes

current_user.me() proves the token and the host. It never reads upload_config.path, so it can't see anything Unity Catalog decides per object. The only call that collects the same verdict as the run is the run's own call, so the probe is files.upload at the configured path with overwrite=True, and it goes through the same client.

The fatal set is deliberately narrow. 403 means "you may not write here", 404 means the catalog, schema or volume doesn't exist, and 401 means the workspace rejected the token. A throttle, a Databricks-side 5xx, a dropped socket or an exception the SDK doesn't map is not an answer to the permission question, so it logs a warning and passes: no destination that works today starts failing its connection test.

The probe is deleted afterwards because a Unity Catalog volume is routinely read back by a table, an Auto Loader stream or another ingest job. WRITE VOLUME already covers the delete. If the delete fails anyway, the write question has already been answered, so it warns and names the file it left behind rather than failing the check.

me() still runs first, so a rejected token fails there with no write attempted. Its active flag is no longer asserted, which matches the source check; the write probe answers the question that flag stood in for.

The SDK subclass fix

The Databricks SDK raises the error-code class in preference to the status class, so a 404 carrying RESOURCE_DOES_NOT_EXIST arrives as ResourceDoesNotExist, a subclass of NotFound. Both the probe's status lookup and wrap_error matched the exact type, so that error read as "not a permission answer" and the missing-volume case this PR exists to catch passed anyway. The lookup now walks the MRO, and wrap_error goes through the same helper. The helper change isn't optional: without it the probe's fatal branch would get the raw SDK error back from wrap_error and re-raise its unredacted text.

Impact

• Customers: a Databricks Volumes destination that can't write now fails its connection test with an error naming the volume path, instead of passing and failing the job at the upload node after the paid pipeline work. A destination that works today still passes, including when the probe hits a throttle or a Databricks outage. Each connection test now writes and removes one zero-byte file in the configured path; a consumer watching that path with file notifications can see the create event before the delete.
• Internal: wrap_error is shared by the native, AWS, Azure and GCP Volumes connectors at precheck and run time on both source and destination; SDK error-code subclasses (ResourceDoesNotExist, InvalidParameterValue, RequestLimitExceeded, DataLoss and the rest) that used to escape as raw, unclassified errors now become UserError, RateLimitError or ProviderError like their parent status.
• Wire contract / clients: the uploader precheck can now raise UserAuthError (401/403) or UserError (404) where it used to pass. wrap_error returns a typed error for SDK subclasses it used to return raw. Both are in the CHANGELOG.
• Deployment target considerations: identical across SaaS, DI, in-VPC and on-prem, since the probe talks only to the customer's own Databricks workspace. Not exercised against a live workspace on any target.

Risk / rollback

The main risk is a live Files API answering in a way the mocked SDK doesn't: a missing USE CATALOG grant returning something outside 401/403/404 would warn and pass, which is the safe direction but leaves that case uncaught. Revert the PR to restore the old check.

How it was verified

The new precheck and classification tests were red on the pre-fix code and green after, and the full unit suite passes locally on the rebased branch. Nothing ran against a real Databricks workspace; see the waiver below.

Proof

Proof waived (environment) -- no reachable Databricks workspace or Unity Catalog volume from this machine, so the probe was never run against a live Databricks Files API. What is missing is a Databricks workspace with a Unity Catalog volume and a principal lacking WRITE VOLUME on it; what unblocks it is workspace credentials in the integration-test secret set, after which test/integration/connectors/databricks/test_volumes_native.py is the place to add a live destination-precheck case.

Repro (local). DatabricksVolumesUploader.precheck driven against a faked databricks.sdk WorkspaceClient, with files.upload raising the SDK's 403 PermissionDenied, then its 404 NotFound, for /Volumes/catalog/schema/volume/path. The precheck returned cleanly every time and never called files.upload: its only live call was current_user.me().

Failing tests on the pre-fix code (uv run --locked --no-sync pytest test/unit/connectors/databricks/test_volumes.py -q):

FAILED test_volumes.py::test_uploader_precheck_error_does_not_leak_raw_text_in_traceback
FAILED test_volumes.py::test_uploader_precheck_writes_a_probe_at_the_configured_path_and_removes_it
FAILED test_volumes.py::test_uploader_precheck_probes_the_volume_root_when_no_volume_path_is_set
FAILED test_volumes.py::test_uploader_precheck_raises_when_volume_write_is_not_granted
FAILED test_volumes.py::test_uploader_precheck_raises_when_volume_path_is_missing
FAILED test_volumes.py::test_uploader_precheck_passes_when_the_probe_cannot_be_cleaned_up

The rows that already passed before the fix, and had to stay green, are the no-false-failure guarantees: credentials rejected at me(), and a 429/500/503 or unrecognised probe failure passing with a warning.

Subclass fix, red on the write-probe commit alone:

WARNING  unstructured_ingest:volumes.py:293 skipping write-access precheck for /Volumes/catalog/schema/volume/path: the probe failed for a reason that is not a permission answer (ResourceDoesNotExist)
FAILED test_volumes.py::test_wrap_error_classifies_error_code_subclasses_by_their_status[ResourceDoesNotExist-UserError]
FAILED test_volumes.py::test_wrap_error_classifies_error_code_subclasses_by_their_status[InvalidParameterValue-UserError]
FAILED test_volumes.py::test_wrap_error_classifies_error_code_subclasses_by_their_status[RequestLimitExceeded-RateLimitError]
FAILED test_volumes.py::test_wrap_error_classifies_error_code_subclasses_by_their_status[DataLoss-ProviderError]
FAILED test_volumes.py::test_uploader_precheck_raises_when_a_404_subclass_says_the_volume_is_missing

After: every test in test/unit/connectors/databricks/test_volumes.py passes, and uv run --locked --no-sync pytest -n auto test/unit --ignore test/unit/unstructured -q passes.

Dependencies / merge order

none

🤖 Generated with Claude Code

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="test/unit/connectors/databricks/test_volumes.py">

<violation number="1" location="test/unit/connectors/databricks/test_volumes.py:302">
P2: This assertion does not prove that the probe is placed at the volume root because any descendant path also matches it. Include the required `unstructured_precheck_` filename prefix in the expected prefix so a nested-path regression fails.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: This assertion does not prove that the probe is placed at the volume root because any descendant path also matches it. Include the required unstructured_precheck_ filename prefix in the expected prefix so a nested-path regression fails.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At test/unit/connectors/databricks/test_volumes.py, line 302:

<comment>This assertion does not prove that the probe is placed at the volume root because any descendant path also matches it. Include the required `unstructured_precheck_` filename prefix in the expected prefix so a nested-path regression fails.</comment>

<file context>
@@ -200,3 +226,172 @@ def test_indexed_file_reports_modification_time_in_epoch_seconds(mocker: MockerF
+    _uploader(mocker, client, volume_path="").precheck()
+
+    probe_path = client.files.upload.call_args.kwargs["file_path"]
+    assert probe_path.startswith("/Volumes/catalog/schema/volume/")
+
+
</file context>

Suggested change

	assert probe_path.startswith("/Volumes/catalog/schema/volume/")
	assert probe_path.startswith("/Volumes/catalog/schema/volume/unstructured_precheck_")

[awalker4]

The error-classification fix looks right to me. I checked each of the SDK's error-code subclasses against the locked SDK and they all map to their parent status. Two things before this merges, though. The first is why I'm holding it.

The precheck shouldn't write to the customer's volume

_verify_write_access creates a real file in the destination and then deletes it in a separate call. That means every Test Connection produces a visible create event, and when the delete fails the file stays there with only a warning. The description itself says why that matters for this destination: a Unity Catalog volume is often read by a table, an Auto Loader stream or another ingest job. A file-arrival trigger or a stream without a glob filter can pick up a zero-byte, extension-less file during the window between the upload and the delete. Destination prechecks in this repo have mostly avoided writing. The only prechecks that do write are fsspec's _empty marker, which is never cleaned up, SFTP, and the Delta Table directory marker. Those are older behavior we haven't copied forward, not a precedent. #811 set the standard for exactly this question: the real statement made harmless, with "nothing is written or removed even where the driver is in autocommit."

Unity Catalog can answer the same question without a write. volumes.read("<catalog>.<schema>.<volume>") covers a missing catalog, schema or volume (the 404 this PR exists to catch). grants.get_effective("VOLUME", full_name, principal=<me().user_name>) covers WRITE_VOLUME, plus USE_SCHEMA/USE_CATALOG on the parents. files.get_directory_metadata(path) covers a mistyped volume_path. #811 argued against reading grant tables for SQL because a catalog can drift from what the session is actually allowed to do. Here Unity Catalog is what makes the decision, so that argument doesn't carry over the same way. One thing I haven't verified: whether a principal can always read its own effective grants on a volume it doesn't own. If it can't, a 403 there should get the same warn-and-pass handling this PR already uses.

Aborted stops retrying

Before this change, an unmapped SDK subclass left the plugin as a 500, and the platform retries a 500. With the MRO walk, the 400 and 409 subclasses become UserError (422), and the platform treats a non-429 4xx as terminal. For most of them that's an improvement: a missing volume should fail fast and count against the customer. Aborted is the exception. Databricks uses it for concurrency conflicts, which are usually transient, and the uploader's overwrite=True upload to a shared output path seems like a plausible way to hit one. I couldn't confirm the Files API returns ABORTED for that race without a live workspace, so this is worth a second opinion. If it does, mapping Aborted to ProviderError in wrap_error keeps it retryable without undoing the rest of the fix.

[paulkarayan]

Taken, on both. The write probe is gone; the precheck reads Unity Catalog now, and nothing is created or removed in the volume.

I checked your three calls against the locked SDK (0.88.0) rather than the docs, and all three are there:

• VolumesAPI.read(name) -- the three-level name. Its own docstring is the source of the 403 answer below: it needs READ VOLUME plus USE_CATALOG / USE_SCHEMA on the parents.
• GrantsAPI.get_effective(securable_type: str, full_name: str, *, principal=None) -- returns EffectivePermissionsList.privilege_assignments[].privileges[].privilege, a Privilege enum whose members include WRITE_VOLUME and ALL_PRIVILEGES. securable_type is a plain str in 0.88, so it is "VOLUME".
• FilesAPI.get_directory_metadata(directory_path: str) -- a HEAD on /api/2.0/fs/directories/..., no body.

On the unknown you flagged: yes, a 403 there is warn-and-pass. The answer turned out to be broader than the grants call, though. Databricks' privilege reference gives WRITE VOLUME as "add, modify, or delete files inside a volume" and does not make READ VOLUME a prerequisite the way WRITE FILES requires READ FILES on external locations. So a principal that can write here can be refused any of the reads, and a 403 from any of them is not a write denial. All three treat it as a warning.

That leaves the refusal surface narrower than the probe's, and I would rather say so than paper over it:

• 404 from volumes.read fails the check -- the missing catalog, schema or volume, which is what this PR is for. One guard on it: the grant lookup runs first, and a lookup that resolved the securable proves the volume exists, so a 404 after that is about visibility and only warns.
• Missing WRITE_VOLUME warns, it does not fail. The effective-permissions API is documented as expanding privileges inherited from parent securables; it says nothing about expanding group membership, and a grant held through a group is the normal case here. I could not verify that against a live workspace, so acting on an absent privilege would risk exactly the false failure the rest of the check avoids. It logs the volume and the privilege to grant.
• 404 from get_directory_metadata warns. The uploader writes path/<relative_path>.json and never creates a directory, so the Files API is already relied on to create the parents at upload time; a directory that is not there yet is not a broken destination. A typo is still worth saying out loud.

If a live workspace later shows that get_effective does expand groups, that leg can be promoted to a refusal -- every one of these is set in the direction where a live answer can only let the check refuse more, never less.

On Aborted: agreed, and it is mapped to ProviderError. Reasoning, since it is a classification and not just a status: the audience is the provider (Databricks losing a race with itself, a sequencer check that did not hold -- not the customer's input and not our bug), and it is retryable, which is what ProviderError buys, since the platform retries a 5xx and treats a non-429 4xx as terminal. The other 409 subclasses stay where the MRO walk puts them. I could not confirm against a live Files API that an overwrite=True race actually surfaces as ABORTED either, but the cost of the two mistakes is not symmetric: mapped as a user 4xx a real one drops the write, and mapped as a provider error a phantom one costs a retry.

Also fixed the two cubic P2s: a 429 or 5xx on me() now warns and passes like everything else rather than aborting the check (401/403/404 still refuse), and the probe-placement assertion is gone with the probe -- the reads are asserted on their exact arguments instead.