Skip to content

perf(api): prefetch user capabilities for list pages - #34

Closed
kdelee wants to merge 242 commits into
develfrom
fix/user-capabilities-page-prefetch
Closed

kdelee wants to merge 242 commits into
develfrom
fix/user-capabilities-page-prefetch

Conversation

@kdelee

@kdelee kdelee commented Oct 12, 2026

Copy link
Copy Markdown
Owner

Users API times out due to per-user RBAC capability queries

Problem

For an Organization Admin, GET /api/controller/v2/users/ returns 25 users but evaluates capabilities separately for each user.

Observed:

25 results
362 SQL queries
93 DAB RBAC queries
~27 seconds

The live request exceeds the ~12-second uWSGI timeout and returns HTTP 503.

Slow path:

UserSerializer
→ summary_fields
→ get_user_capabilities()
→ UserAccess.can_change()
→ DAB RoleEvaluation queries

Disabling capability generation reduced the request from ~27 seconds to 0.286 seconds.

WIP branches

  • AWX: branch fix/user-capabilities-page-prefetch ; enables the existing page capability-prefetch framework for UserSerializer and adds a User-specific organization-membership bulk query. Needs parity tests for multi-organization users, orphan users, and team-derived roles.
  • django-ansible-base: no committed code yet. The existing DAB page-prefetch pattern from AAP-86408 is available, but a generic effective-capability API may still be needed for team-role inheritance.

Reproduction on the closer-to-devel aap-devel deployment remains:

13,112 users
25 results
33.36 seconds in shell_plus
HTTP 503 expected through uWSGI after the ~12-second Harakiri timeout

Proposed fix

Prefetch capabilities for only the current page using one/bounded batch query:

request user + page users
→ capability map keyed by object ID
→ serializer consumes cached results

Preserve existing capability fields and authorization behavior.

The mechanism should be reusable by other paginated endpoints that serialize user_capabilities, including projects, inventories, job templates, credentials, and teams.

Scope

Investigate whether DAB should provide a generic batch capability API. AWX should consume it from list serializers without changing authorization semantics.

stevensonmichel and others added 30 commits February 16, 2026 10:14
…sible#16281)

* Added token authentication in logic, arguments, and test
* When the remote host header values contains a comma separated list,
  only consider the first entry. Previously we considered every item in
  the list.
ansible#16289)

* do not add optional survey fields with empty strings that are not backed by extra_vars

* exclude password fields from skipping if not defined
* Revert "[Devel][AAP-65384]Restoration of Token Authentication for AWX CLI (ansible#16281)"

This reverts commit 994a2b3.
* Update PR template to add steps to repro

* updated bottom section to include steps to reproduce to help with
  debugging and triage
* Fix some pytest warnings using Opus 4.6

* Fix review comments

* Use raw-strings and regex markers for matching exception pattern

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

* Make regex work

* Undo always true assertion edit

---------

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
…#16296)

* Add retrieve_workload_identity_jwt to jobs.py and tests

* Apply linting

* Add precondition to client retrieval

* Add test case for client not configured

* Remove trailing period in match string
Added cron schedule for ci jobs
…tcherd (ansible#16300)

  Remove SELECT FOR UPDATE from job dispatch to reduce transaction rollbacks
                                                                                                                                                                                                                                                                                           
  Move status transition from BaseTask.transition_status (which used
  SELECT FOR UPDATE inside transaction.atomic()) into                                                                                                                                                                                                                                      
  dispatch_waiting_jobs. The new approach uses filter().update() which                                                                                                                                                                                                                     
  is atomic at the database level without requiring explicit row locks,
  reducing transaction contention and rollbacks observed in perfscale
  testing.

  The transition_status method was an artifact of the feature flag era
  where we needed to support both old and new code paths. Since
  dispatch_waiting_jobs is already a singleton
  (on_duplicate='queue_one') scoped to the local node, the
  de-duplication logic is unnecessary.

  Status is updated after task submission to dispatcherd, so the job's
  UUID is in the dispatch pipeline before being marked running —
  preventing the reaper from incorrectly reaping jobs during the
  handoff window. RunJob.run() handles the race where a worker picks
  up the task before the status update lands by accepting waiting and
  transitioning it to running itself.

  Signed-off-by: Seth Foster <fosterbseth@gmail.com>
  Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forward port of 5c4653fbbca4b542fea05f070efea0396b034839 from stable-2.6.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…and stable-2.6 (ansible#16315)

* Added worflow dispatch to trigger ci on release_4.6 and stable-2.6

* fix error cascading to subsequent jobs for cron

* made compose_tag resolve to the correct branch name
…6314)

Added HTTP_X_FORWARDED_FOR in Devel for production
* feat: workload identity credentials integration

* feat: cache credentials and add context property to Credential

Assisted-by: Claude

* feat: include safeguard in case feature flag is disabled

* feat: tests to validate workload identity credentials integration

* fix: affected tests by the credential cache mechanism

* feat: remove word cache from variables and comments, use standard library decorators

* fix: reorder tests in correct files

* Use better error catching mechanisms

* Adjust logic to support multiple credential input sources and use internal field

* Remove hardcoded credential type names

* Add tests for the internal field

Assited-by: Claude
Django 5.2 restricts LogoutView to POST only (deprecated in 4.1,
  removed in 5.0+). Without this fix, GET requests to /api/logout/
  return 405 Method Not Allowed.

  Add http_method_names override and a get() method that delegates
  to post() where auth_logout() actually runs
* Pin container versions

* downgrades from coderabbit

* Add notes for future upgrades
…ed hosts (ansible#16318)

* Add new tests for bug saving concurrent facts

* Fix first bug and improve tests

* Fix new bug where concurrent job clears facts from other job in unwanted way

* minor test fixes

* Add in missing playbook

* Fix host reference for constructed inventory

* Increase speed for concurrent fact tests

* Make test a bit faster

* Fix linters

* Add some functional tests

* Remove the sanity test

* Agent markers added

* Address SonarCloud

* Do backdating method, resolving stricter assertions

* Address coderabbit comments

* Address review comment with qs only method

* Delete missed sleep statement

* Add more coverage
* Revert "AAP-58452 Add version fallback for external query files (ansible#16309)"

This reverts commit 0f2692b.

* AAP-58441: Add runtime integration for external query collection (ansible#7208)

Extend build_private_data_files() to copy vendor collections from
/var/lib/awx/vendor_collections/ to the job's private_data_dir,
making external query files available to the indirect node counting
callback plugin in execution environments.

Changes:
- Copy vendor_collections to private_data_dir during job preparation
- Add vendor_collections path to ANSIBLE_COLLECTIONS_PATH in build_env()
- Gracefully handle missing source directory with warning log
- Feature gated by FEATURE_INDIRECT_NODE_COUNTING_ENABLED flag

This enables external query file discovery for indirect node counting
across all deployment types (RPM, Podman, OpenShift, Kubernetes) using
the existing private_data_dir mechanism.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* [stable-2.6] AAP-58451: Add callback plugin discovery for external query files (ansible#7223)

* AAP-58451: Add callback plugin discovery for external query files

Extend the indirect_instance_count callback plugin to discover and load
external query files from the bundled redhat.indirect_accounting collection
when embedded queries are not present in the target collection.

Changes:
- Add external query discovery with precedence (embedded queries first)
- External query path: redhat.indirect_accounting/extensions/audit/
  external_queries/{namespace}.{name}.{version}.yml
- Use self._display.v() for external query messages (visible with -v)
- Use self._display.vv() for embedded query messages (visible with -vv)
- Fix: Change .exists() to .is_file() per Traversable ABC
- Handle missing external query collection gracefully (ModuleNotFoundError)

Note: This implements exact version match only. Version fallback logic
is covered in AAP-58452.

* fix CI error when using Traversable.is_file

* Add minimal implementation for AAP-58451

* Fix formatting

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* AAP-58452 Add version fallback for external query files (ansible#7254)

* AAP-58456 unit test suite for external query handling (ansible#7283)

* Add unit tests for external query handling

* Refactor unit tests for external query handling

* Refactor indirect node counting callback code to improve testing code

* Refactor unit tests for external query handling for improved callback code

* Fix test for majore version boundary check

* Fix weaknesses in some unit tests

* Make callback plugin module self contained, independent from awx

* AAP-58470 integration tests (core) for external queries (ansible#7278)

* Add collection for testing external queries

* Add query files for testing external query file runtime integration

* Add live tests for external query file runtime integration

* Remove redundant wait for events and refactor test data folders

* Fix unit tests: mock flag_enabled to avoid DB access

The AAP-58441 cherry-pick added a flag_enabled() call in
BaseTask.build_private_data_files(), which is called by all task types.
Tests for RunInventoryUpdate and RunJob credentials now hit this code
path and need the flag mocked to avoid database access in unit tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: attempt exact query file match before Version parsing (ansible#7345)

The exact-version filename check does not require PEP440 parsing, but
Version() was called first, causing early return on non-PEP440 version
strings even when an exact file exists on disk. Move the exact file
check before Version parsing so fallback logic only parses when needed.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Do no longer mutate global sys.modules (ansible#7337)

* [stable-2.6] AAP-58452 fix: Add queries_dir guard (ansible#7338)

* Add queries_dir guard

* fix: update unit tests to mock _get_query_file_dir instead of files

The TestVersionFallback tests mocked `files()` with chainable path
mocks, but `find_external_query_with_fallback` now uses
`_get_query_file_dir()` which returns the queries directory directly.
Mock the helper instead for simpler, correct tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove unused EXTERNAL_QUERY_PATH constant (ansible#7336)

The constant was defined but never referenced — the path is constructed
inline via Traversable's `/` operator which requires individual segments,
not a slash-separated string.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore original feature flag state in test fixture (ansible#7347)

The enable_indirect_host_counting fixture unconditionally disabled the
FEATURE_INDIRECT_NODE_COUNTING_ENABLED flag on teardown, even when it
was already enabled before the test (as is the case in development via
development_defaults.py). This caused test_indirect_host_counting to
fail when run after the external query tests, because the callback
plugin was no longer enabled.

Save and restore the original flag state instead.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…ble#16298)

* Address more ignored pytest warnings

* Fix what we can with CI results

* Add new migration file
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
@kdelee
kdelee force-pushed the fix/user-capabilities-page-prefetch branch 5 times, most recently from 1e5a721 to 2b237e3 Compare October 13, 2026 17:30
…sible#16741)

test_bulk_host_create_num_queries budgets 15 queries per request. The
first request in a pytest process pays one extra query for each
database-backed setting it reads (PROXY_IP_ALLOWED_LIST,
REMOTE_HOST_HEADERS, BULK_HOST_MAX_CREATE) until the settings
wrapper's in-memory cache is warm, so the budget only held when an
earlier test in the same xdist worker had already warmed it. Make one
untimed request first so the assertion measures steady state.
@kdelee
kdelee force-pushed the fix/user-capabilities-page-prefetch branch from ceaa114 to 62e03a9 Compare October 14, 2026 21:11
lgalis and others added 4 commits October 14, 2026 17:56
…oints (ansible#16699)

* Add custom last_job_host_summary__failed filter to HostList

The denormalized last_job_host_summary FK was removed (AAP-68024), so
filtering by ?last_job_host_summary__failed=True on /v2/hosts/ would
400 or return stale results. Add custom handling that annotates the
queryset with a Subquery on JobHostSummary (same pattern used by the
dashboard counts) and reserves the param names from FieldLookupBackend
so the generic filter skips them.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Extend last_job_host_summary failed filter to inventory and group host lists

Build on AlanCoding's HostList filter (ansible#16698) by moving the
Subquery-based latest-summary logic into HostRelatedSearchMixin so
InventoryHostsList, GroupHostsList, and GroupAllHostsList honor the same
query params as the global hosts list.

Co-authored-by: Cursor <cursoragent@cursor.com>

* Address review: validate filter values and apply both summary filters

Require True for last_job_host_summary failed query params, apply failed
and not-failed filters independently (AND semantics), add GroupHostsList
coverage, and document mixin helpers.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
…sible#16745)

TestCallbackBrokerWorker::test_duplicate_key_not_saved_twice can crash
with AttributeError: 'FakeRedis' object has no attribute 'hincrby'.

CallbackBrokerWorker.flush() ends with

    if self.subsystem_metrics.should_pipe_execute() is True:
        self.subsystem_metrics.pipe_execute()

and should_pipe_execute() is True once more than
SUBSYSTEM_METRICS_INTERVAL_SAVE_TO_REDIS seconds have passed since the
last pipe execute. On a fast machine that interval never elapses during
the test, so the pipe is never executed and nothing notices; on a slow
one it does, pipe_execute() calls store_value() on every metric, and
IntM/FloatM/SetIntM reach for hincrby/hincrbyfloat/hset, which the
FakeRedis double in awx/main/tests/functional/conftest.py does not have.

Rather than suppress the pipe execute in the one test, make the double
complete enough to execute it: add the hash commands the metric
store_value() implementations use, an execute() for the pipeline
(pipeline() returns self), and a lock() returning a lock that never
acquires, so send_metrics() takes its "another thread holds the lock"
early return instead of trying to emit a websocket notification through
a channel layer that is not available under test.
…ible#16744)

test_bulk_job_launch_queries asserts django_assert_max_num_queries(25),
but the first request in a pytest process pays one extra query per
database-backed setting it reads. The settings wrapper's in-process
memoized TTL cache (awx/conf/settings.py, _awx_conf_memoizedcache)
starts cold and survives the per-test cache.clear() in
awx/main/tests/conftest.py, so cold = 26 queries and warm = 25. The
budget only holds when an earlier test in the same xdist worker already
warmed the cache, which is luck of the -n auto draw.

Make one untimed launch of the same endpoint before the measured loop,
so the assertion measures steady state regardless of which test ran
first in the worker. The budget stays at 25. The warm-up job is a
separate bulk job; the assertions after the task manager run still use
the measured response's data and its own workflow_nodes list, so they
are unaffected.

This is the same fix already applied to the sibling test
test_bulk_host_create_num_queries in ansible#16741.
…ible#16679)

* AAP-94827 Add SSRF egress filtering and allowlist to notification backends

Add url_validation.py to block notification URLs that resolve to
private, loopback, link-local, or reserved addresses. Admins can
allowlist specific IPs, CIDR ranges, or hostnames via the new
NOTIFICATION_IP_ALLOW_LIST system setting.

The allowlist ships together with the filter to avoid breaking
webhook integrations targeting internal (RFC1918) endpoints.

* AAP-94827 Mock validate_url in functional redirect test

The functional test uses a fake hostname ('some.fake.url') that fails
DNS resolution when validate_url tries to resolve it in CI.

* AAP-94163 Add non-overridable deny list for link-local addresses

Link-local addresses (169.254.0.0/16, fe80::/10) are now always blocked
regardless of allowlist settings, protecting cloud metadata endpoints
even if DNS is subverted for an allowlisted hostname.

Requested by Product Security as a condition for approving hostname
support in the NOTIFICATION_IP_ALLOW_LIST setting.

* AAP-94163 Block IPv4-mapped link-local addresses in deny list

Unwrap IPv4-mapped IPv6 addresses (e.g. ::ffff:169.254.169.254) before
checking the link-local deny list, preventing a bypass where the mapped
address doesn't match either the IPv4 or IPv6 deny network directly.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@kdelee
kdelee force-pushed the fix/user-capabilities-page-prefetch branch from 8a099f1 to 77420ae Compare October 21, 2026 17:19
thedoubl3j and others added 11 commits October 22, 2026 09:34
ansible#16751)

* AAP-96178: Fix unified_jobs pagination count to respect RBAC filtering

Non-admin users saw the unfiltered table-wide count instead of the count
matching their RBAC-visible results. Use the same primary-key subquery
approach already proven on the /api/v2/jobs/ endpoint.


* AAP-96178: Add docstrings and mixed-role count test for UnifiedJobPaginator

Clarify the PK subquery approach with expanded docstring and inline
comments per review feedback. Add a functional test verifying that a user
with a mix of JT and inventory roles sees a count matching exactly
their permitted results.


* AAP-96178: Add deduplication test for multiple RBAC access routes

Verify that a job reachable through three overlapping permission paths
(direct JT role, org audit role, and team JT role) appears exactly once
in both results and count.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
…ible#16752)

* fix: do not call .filter() on list before converting to queryset

* fix: optimize slicing of queryset

* fix: preserve existing filters and ordering on the queryset
inventory_import builds its own podman run command and omits --rm, unlike
ansible-runner which always passes it. Each CLI import leaves an exited
container behind, eventually filling disk.

Related: AAPRFE-370
* Stop using floating Python 3.x in docs and label_pr CI

The python-version: '3.x' specifier resolved to Python 3.15.0 on newer
runner images, causing rpds-py to fail to build (no pre-built wheel).
Removing the override lets the setup-python composite action derive the
version from the Makefile, consistent with the rest of CI.


* Fix SonarCloud security findings in docs and label_pr workflows

Pin pip dependencies with --only-binary :all: to prevent execution of
setup scripts, scope write permissions to job level instead of workflow
level, and use full commit SHA for third-party action.


* fix yaml syntax

* update yaml syntax so all run is recongnized correctly

* Add least-privilege permissions to docs workflow


* Disable credential persistence in docs workflow checkout

Prevents tox and dependency code from accessing the GitHub token
stored in .git/config.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
…#16750)

DAB PR ansible#1105 added actor_role_ids() as the public API and deprecated
_actor_role_filter. Update all AWX callers to use the new method.

Assisted-By: Claude Opus 4.6 <noreply@anthropic.com>
* Bump PyJWT to address CVE-2026-102268

Pin PyJWT>=2.14.0 in requirements.in to address CVE-2026-102268
(authentication bypass via improper public key validation).
Regenerated requirements.txt via make docker_update_requirements.

Assisted-By: Claude Code
Signed-off-by: Rodrigo Horie <rodrigo.horie@hotmail.com>

* Use a 32+ byte HMAC key in workload identity tests

PyJWT >= 2.11 emits InsecureKeyLengthWarning for HS256 keys shorter
than 32 bytes, which pytest's filterwarnings=error turns into an
exception that populate_workload_identity_tokens swallows, so the
Gateway request is never made.

Assisted-By: Claude Code
Signed-off-by: Rodrigo Horie <rodrigo.horie@hotmail.com>

---------

Signed-off-by: Rodrigo Horie <rodrigo.horie@hotmail.com>
…lags (ansible#16720)

* [AAP-76923] Honor AWX_TASK_ENV on job pods and ansible-runner event flags

Jobs Extra Environment Variables already reached playbooks via envvars, but
automation-job pods did not list them, and ansible-runner overwrote
RUNNER_OMIT_EVENTS unless omit_event_data was set on the runner params.

Issue: https://redhat.atlassian.net/browse/AAP-76923

Co-authored-by: Cursor <cursoragent@cursor.com>

* [AAP-76923] Keep project-update event metadata when omit flags are set

Project updates still need task_action and scm_version for SCM credential
redaction and revision tracking, so BaseTask.run leaves both runner omission
kwargs off for ProjectUpdate even when AWX_TASK_ENV enables them.

Issue: https://redhat.atlassian.net/browse/AAP-76923

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
CVE-2026-111497 - Bump Brotli to 1.2.1

Add Brotli>=1.2.1 pin to requirements.in to address CVE-2026-111497
(Out-of-Bounds Write in Google Brotli Decoder).
Regenerated requirements.txt via make docker_update_requirements.

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Jake Jackson <24478650+thedoubl3j@users.noreply.github.com>
Use monkeypatch.setitem for awxkit config in tests

awxkit's `config` is a PseudoNamespace, a dict subclass that exposes
its keys as attributes. pytest 9.2.0 changed monkeypatch.setattr to
read the previous value from the target's __dict__ instead of via
getattr, so for a dict-backed namespace it always records the attribute
as "did not exist" and schedules a delattr on undo. Patching the same
key twice (autouse fixture plus test body) then deletes it twice, and
PseudoNamespace.__delattr__ raises KeyError at teardown. This breaks
13 tests in test/cli/test_authentication.py on pytest >= 9.2.0.

monkeypatch.setitem records the old value with dict.get, which sees the
dict-backed storage correctly, so undo restores or removes exactly what
was there on any pytest version. test/api/pages/test_base.py is updated
for the same reason: under 9.2.0 its undo would delete a pre-existing
base_url or credentials value instead of restoring it.


Claude-Session: https://claude.ai/code/session_01SbeYPdRRLaGCk74gJHq9yS

Signed-off-by: Hao Liu <44379968+TheRealHaoLiu@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
ansible#16776)

fix: define asyncio_default_fixture_loop_scope to avoid failing in recent asyncio versions
* fix: update gitpython to 3.2.1

* update comment
@kdelee
kdelee force-pushed the fix/user-capabilities-page-prefetch branch from 77e6983 to e2cba3b Compare November 3, 2026 16:15
Evaluate user capabilities from page-scoped role and permission queries instead of materializing the actor's or target users' complete RBAC assignments. Use correlated existence checks for orphan-user deletion permissions while preserving capability behavior.

Assisted-by: pi coding agent
@kdelee
kdelee force-pushed the fix/user-capabilities-page-prefetch branch from 4760396 to 69246f3 Compare November 3, 2026 19:34
Bind target role lookups to the outer page-user query so orphan delete capabilities are evaluated independently. Add coverage for an orphan holding a permission unavailable to the actor.

Assisted-by: pi coding agent
Evaluate target role permissions for page-scoped orphan users in one joined query instead of a correlated permission subquery per target user.

Assisted-by: pi coding agent
Do not scan role assignments for the requesting user, superusers, or system auditors when evaluating orphan delete permissions; their capability rules are handled separately.

Assisted-by: pi coding agent
@kdelee kdelee closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.