Backport #33204 to 2.0: [Databricks] Paginate jobs by token so large workspaces ingest in full - #33243
Open
mohittilala wants to merge 2 commits into
Open
Backport #33204 to 2.0: [Databricks] Paginate jobs by token so large workspaces ingest in full#33243mohittilala wants to merge 2 commits into
mohittilala wants to merge 2 commits into
Conversation
… ingest in full (#33204) * fix(databricks): paginate jobs by token so large workspaces ingest in full Databricks caps jobs/list `offset` at 1000, so offset pagination stopped at 1100 jobs. The 400 body is valid JSON and the status was never checked, so the loop read it as an empty page and reported success. Move to API 2.2 with token pagination, raise on a non-200 rather than truncating silently, and follow the per-job has_more to fetch tasks past the 100-element page. 2.1 could not return a job with more than 100 tasks at all, and 2.2 alone would have stopped at one page. * fix(databricks): guard token pagination and cover the paths review found Both token loops could refetch the same page forever if the service handed back the token that produced it, and the shared walk was duplicated in two places. Extract it, stop on a repeated token, and rename the helpers to the verb-first shape the other connector clients use. Fold list_jobs_test_connection onto the same checked GET so a failing GetPipelines step names the URL and status, and cover what was untested: the degrade-not-fail path when one job's task page dies, has_more removal, the loop guard, and both test-connection outcomes. * fix(databricks): close the pagination loop gaps review found Comparing a page token against only the previous one misses a cycle that alternates, and a service handing out fresh tokens forever was not caught at all. Remember every token seen, cap the walk, and raise on either rather than returning, since stopping quietly is one more way to truncate a listing while reporting success. Cover the two contracts that were asserted in prose but never in a test: that get_pipelines_list propagates a listing failure instead of letting mark-deleted work from a short list, and that a failed runs page keeps the runs already read. * test(databricks): make two pagination tests discriminate old from new Both passed against the unpatched client and so proved nothing. The runs limit check allowed an absent `limit`, since a missing value also satisfies "<= 26", and the task-page fallback test never asserted the fallback was attempted, so a client that skipped `jobs/get` entirely still met both assertions. Assert the explicit RUNS_PAGE_SIZE, and record the attempted URLs so the fallback has to actually run. Verified both now fail against the pre-change list_jobs and get_job_runs.
mohittilala
requested review from
stout-yeoman
and removed request for
a team
September 11, 2026 19:30
github-actions
Bot
added
Ingestion
safe to test
Contributor
✅ Playwright Results — workflow succeededValidated commit ✅ 110 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky PerformanceBlocking targets: ✅ met · Optimization targets: 🟡 in progress Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting. 🕒 Full workflow signal wall (to summary) 50m 42s ⏱️ Max setup 3m 3s · max shard execution 11m 54s · max shard-job elapsed before upload 16m 45s · reporting 3s 🌐 210.07 requests/attempt · 1.79 app boots/UI scenario · 0.00% common-shard skew Optimization targets still in progress:
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
Code Review ✅ ApprovedBackport of pagination fix for Databricks jobs API to 2.0 branch. Resolves offset limit by switching to token-based pagination, enabling full ingestion of large workspaces (verified against 2504-job workspace). No issues found. OptionsDisplay: compact → Counting what did not apply, without listing it. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #33204 to 2.0.
Cherry-picked from
66348c975f7dby hand. Two conflicts, both mechanical:ingestion/.ruff-g004-baseline.jsondoes not exist on this branch, so the one-line deletion was dropped.get_job_runsconflicted because 2.0 still types itList[dict]. The superseded body was removed and the replacement taken as-is.The added lines in
client.pyare byte-identical to main, so future backports keep applying cleanly.JOB_API_VERSIONis/api/2.2on this branch, verified after the resolution.Databricks unit tests pass on the branch:
Original description follows.
Databricks caps the
offsetparameter onjobs/listat 1000, so offset pagination stops at 1100 jobs. The 400 body is valid JSON and the status was never checked, so the loop read it as an empty page and reported success. Separately, API 2.1 cannot return a job with more than 100 tasks: it answers 200 withsettings.tasksmissing entirely, so such a job was ingested with no tasks and no lineage.Verified end to end against a 2504-job workspace: before, 1100 jobs ingested and 1370 task definitions lost, reported as
Workflow Success %: 100.0. After, all 2504 jobs with complete task lists.