Skip to content

Fix AsyncToSync error in Databricks deferrable operators by fetching connection asynchronously - #71840

Open
saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-71525-databricks-async-connection
Open

Fix AsyncToSync error in Databricks deferrable operators by fetching connection asynchronously#71840
saitejabandaru-in wants to merge 1 commit into
apache:mainfrom
saitejabandaru-in:fix-71525-databricks-async-connection

Conversation

@saitejabandaru-in

Copy link
Copy Markdown

Closes #71525.

Motivation

When DatabricksRunNowOperator(deferrable=True) is used, the triggerer executes DatabricksExecutionTrigger.run(). Inside the async loop, the trigger polls for status via DatabricksHook.a_get_run_state(), which eventually calls self._endpoint_url(). That function tries to read self.databricks_conn, which is a @cached_property invoking self.get_connection().

In Airflow 3 (TaskSDK), get_connection bridges to a synchronous supervisor call using async_to_sync. Because this happens inside the triggerer's already-running asyncio loop, it crashes with RuntimeError: You cannot use AsyncToSync in the same thread as an async event loop.

Changes

  • Added adatabricks_conn to BaseDatabricksHook which fetches the connection asynchronously via aget_connection.
  • Added ahost, _a_endpoint_url, and _a_get_oidc_token_service_url to prevent async methods from calling self.host (which also evaluates databricks_conn synchronously).
  • Refactored all async def methods in BaseDatabricksHook to use (await self.adatabricks_conn()) and await self._a_endpoint_url().
  • Updated unit tests to mock adatabricks_conn.

This adds `adatabricks_conn` to `BaseDatabricksHook` to allow fetching the connection via TaskSDK asynchronously, resolving an issue where the async triggerer loop crashed when calling the synchronous `databricks_conn` property. All async methods have been refactored to use the async connection path. Fixes apache#71525.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DatabricksRunNowOperator(deferrable=True) crashes in triggerer: AsyncToSync used inside running event loop (Airflow 3.0.6)

2 participants