TTL-cache slow read-only CKAN actions (organization_list, group_list)

[anuveyatsu]

Why

organization_list and group_list with all_fields=true are an N+1 in CKAN. _group_or_org_list calls organization_show once per organization, and each of those runs its own Solr query for the dataset count. On oddk-prod that is 51 sequential queries, ~9.3s.

The ODDK theme fetches both in its / handler, so every home page render pays it, and frontend-v2's own /organization route pays it again. Measured on prod:

page	before
/	11.9s
/organization	12.5s

CKAN call	time
organization_list (plain)	0.34s
organization_list?all_fields=true&include_dataset_count=false	1.08s
organization_list?all_fields=true&include_dataset_count=true	9.32s

Not a capacity problem — CKAN sat at 1–23m CPU against a 1000m limit throughout. It is idle, waiting on 51 sequential round-trips.

What this does

Caches results at getJsonResponse, the single funnel for every CKAN call in lib/dms.js, behind an action allowlist. getJsonResponse's signature is unchanged, so no call site moves; the raw request is split out into fetchJsonResponse.

• Off by default (API_CACHE_TTL=0), so other portals on this repo are unaffected. ODDK opts in with API_CACHE_TTL=3600.
• Allowlisted actions only (organization_list,group_list). Anything user- or query-specific — package_search above all — still goes straight to CKAN on every request.
• Single-flight. Entries hold the in-flight promise, not just the value, so concurrent misses collapse into one upstream call rather than each firing its own 51 queries. Without this, TTL expiry under load is a thundering herd.
• Stale-on-error. A failed refresh serves the last good value, so a CKAN blip cannot blank a page that has already rendered. A failure with nothing cached still propagates.
• Param-order-stable keys, so equivalent calls share one entry.
• Bounded at 200 entries.

The cache is in-process, so it is per-pod and cold after a restart. That is a deliberate accepted trade: the point is that the Nth visitor does not pay what the 1st does. Redis was considered and rejected for now — it would need REDIS_URL plumbed into the ODDK frontend env, which is not there today.

Deploying to ODDK

Two steps, and this PR is only the first:

1. Merge this.
2. In datopian/dx-helm-oddk-prod, bump the pinned commit in frontend/Dockerfile (currently git checkout 99f6782e426c8d85d09a555e0b02e96e4839565d) to this branch's merge commit, and set API_CACHE_TTL=3600 in the frontend env.

Until step 2, prod does not pick this up — the Dockerfile pins an exact SHA.

Testing

tests/lib/cache.test.js, 8 tests, all passing: key stability and separation, hit within TTL, refetch after expiry, single-flight collapse, stale-on-error, propagation with nothing cached, and retry after failure.

Integration-checked against DmsModel directly: organization_list served from cache on the second call, both param orderings sharing one entry, package_search never cached, and no caching at all when API_CACHE_TTL is unset.

tests/lib/index.test.js has 2 pre-existing failures on oddk/custom-facet (a package_search nock fixture that does not match the branch's facet params). Verified identical with and without this change by stashing it — not introduced here, and getOrganizations api works passes.

Follow-up

The real fix is upstream: make CKAN's all_fields path use the batched group_list_dictize instead of per-org organization_show, which would take this from 9.3s to ~1.1s and make caching unnecessary. Tracked in datopian/tech-devops#668.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 45495bd0-7af8-4ae7-8eff-a385f1bf7da1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}