TT-17746: Improve resolver performance

[vladzabolotnyi]

Ticket: https://tyktech.atlassian.net/browse/TT-17746

Description

ResolveAll can be called with a huge amount of data and unresolved refs. If cache is disabled it will take a huge amount of time. To improve resolving large amount of data at a single call we will add dedup cache. Its basically one more in-memory cache that lives only for a single request/call. Also we've reworked resolving processing and now its concurrent.

Added 10000 APIs with 50 vault and 50 consul custom stores. Each vault store managing different path(extend gateway gating with single path). Each API contains 100 unresolved kv:// and $kv{} references. When gateway is started, the process should resolve 1_000_000 unresolved references.

With cache turned ON there is no meaningful difference in performance. So its super visible for cases when its disabled.

Benchmarks:
Original logic
NO CACHE
Given 10000 APIs with 50 vault and 50 consul custom stores
With each api has 20 unresolved references(low value to not waiting for 5 min 😄 )
200_000 unresolved references per hot-reload

Took 2min45sec to start the gateway

================================================

New logic
NO CACHE
Given 10000 APIs with 50 vault and 50 consul custom stores
With each api has 200 unresolved references
2_000_000 unresolved references per hot-reload

Took ~2.5 sec

Screen.Recording.2026-07-13.at.18.07.16.mov

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Refactoring or add test (improvements in base code or adds test coverage to functionality)
• Documentation updates or improvements.

Checklist

• I have reviewed the guidelines for contributing to this repository.
• Make sure you are requesting to pull a topic/feature/bugfix branch (right side). If PRing from your fork, don't come from your master!
• Make sure you are making a pull request against our master branch (left side). Also, it would be best if you started your change off our latest master.
• My change requires a change to the documentation.
  • I have manually updated the README(s)/documentation accordingly.
  • If you've changed APIs, describe what needs to be updated in the documentation.
• I have updated the documentation accordingly.
• Modules and vendor dependencies have been updated; run go mod tidy && go mod vendor
• When updating library version must provide reason/explanation for this update.
• I have added tests to cover my changes.
• All new and existing tests passed.
• Check your code additions will not fail linting checks:
  • gofmt -s -w .
  • go vet ./...

[github-actions]

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

Vlad Zabolotnyi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request}

[probelabs]

This PR introduces significant performance improvements to the ResolveAll function by implementing an in-call deduplication cache and concurrent fetching of references. The previous sequential resolution process was inefficient for large configurations with many duplicate references, especially with the primary cache disabled. This change introduces a short-lived memo cache for the duration of a single ResolveAll call, ensuring each unique reference is fetched from the backend only once. Additionally, a new prefetch step gathers all unique references and resolves them concurrently, dramatically reducing latency for bulk operations.

Benchmarks included in the PR description show a reduction in processing time from several minutes to a few seconds for a document with two million unresolved references, highlighting the magnitude of the improvement.

Files Changed Analysis

The changes are well-encapsulated within the kv/internal/resolve package:

• kv/internal/resolve/memo.go (new): Implements the request-scoped memo cache. It uses a context.Context to store resolution results (value or error) for the duration of a single ResolveAll call, preventing redundant backend fetches.
• kv/internal/resolve/refs.go (new): Centralizes the logic for parsing both whole-value (kv://...) and inline ($kv{...}) references into a common refKey struct. It also introduces collectRefs to traverse a document and extract all unique references for the prefetch step.
• kv/internal/resolve/resolver.go (modified): The core ResolveAll function is re-architected. It now orchestrates the new flow: creating the memo, concurrently prefetching all unique references to warm the cache, and then walking the document to substitute values from the memo. The fetchAndExtract function is updated to be memo-aware, and a concurrency limit (maxConcurrentResolves) is introduced for the prefetch phase.
• kv/internal/resolve/refs_test.go (new): Adds unit tests for the new reference parsing and collection logic.
• kv/internal/resolve/resolver_test.go (modified): Adds comprehensive tests to validate the new behavior, including the correctness of memoization (deduplication across syntax forms), the per-call lifecycle of the cache, and the concurrent resolution of distinct references.
• kv/internal/resolve/resolver_bench_test.go (new): Adds new benchmarks to measure the performance improvements with high numbers of both duplicate and distinct references, confirming the effectiveness of the changes.

Architecture & Impact Assessment

• What this PR accomplishes: It transforms the ResolveAll function from a slow, sequential operation into a highly efficient, concurrent process. This is critical for performance-sensitive scenarios like gateway startups and hot-reloads, where large configurations must be resolved quickly.

• Key technical changes introduced:

  1. In-Call Memoization: A temporary, per-call cache (memo) is used to store resolution results, ensuring that each unique reference is fetched from the backend at most once per ResolveAll operation.
  2. Concurrent Prefetching: Before substitution, the resolver scans the document, collects all unique references, and fetches them concurrently using an errgroup, populating the memo cache efficiently.
  3. Refactoring: Reference parsing logic has been extracted and centralized, improving code clarity and maintainability.

• Affected system components: The primary component affected is the KV resolver. Any service or component that uses ResolveAll for processing configurations (e.g., an API Gateway loading API definitions) will experience a significant performance improvement, leading to faster startup times and more responsive configuration updates. The public API remains unchanged, making this a non-breaking enhancement.

Resolution Flow Comparison

graph TD
    subgraph Old Sequential Flow
        A[ResolveAll starts] --> B{Walk JSON};
        B -- Found ref --> C[Fetch from Backend];
        C --> D{Substitute Value};
        D --> B;
        B -- Found same ref again --> E[Fetch from Backend AGAIN];
        E --> F{Substitute Value};
        F --> B;
        B -- Done --> G[Return resolved JSON];
    end

    subgraph New Concurrent Flow
        H[ResolveAll starts] --> I[1. Create Memo Cache];
        I --> J[2. Collect all *unique* refs];
        J --> K[3. Prefetch all refs concurrently];
        K --> L[4. Populate Memo Cache];
        L --> M{5. Walk JSON};
        M -- Found ref --> N[Read from Memo];
        N --> O{Substitute Value};
        O --> M;
        M -- Done --> P[Return resolved JSON];
    end

Loading

Scope Discovery & Context Expansion

The changes are localized to the kv/internal/resolve package, but their impact is systemic. The PR description's focus on gateway startup performance indicates this resolver is on a critical path for service initialization. Any part of the system that resolves KV references in bulk—such as a configuration service loading API definitions or processing dynamic policies—will benefit directly from these improvements.

The performance gain directly translates to faster service availability and more efficient configuration updates. To fully map the impact, a cross-repository search for usages of resolver.ResolveAll would be beneficial to identify all specific system entrypoints that will benefit from this optimization.

Metadata

• Review Effort: 4 / 5
• Primary Label: enhancement

---

Powered by Visor from Probelabs

Last updated: 2026-07-13T16:22:27.798Z | Triggered by: pr_updated | Commit: 878c945

💡 TIP: You can chat with Visor using /visor ask <your question>

[probelabs]

Security Issues (1)

Severity	Location	Issue
🟡 Warning	kv/internal/resolve/resolver.go:184-206	The `ResolveAll` function, through its new `prefetch` helper, collects and attempts to resolve all unique references within an input document. There is no limit on the number of unique references processed. An attacker able to supply a crafted input document containing an extremely large number of unique references could trigger excessive memory allocation to store these references and cause a high volume of concurrent requests to backend data stores. This could lead to a Denial of Service (DoS) condition, either by exhausting memory in the service calling `ResolveAll` or by overwhelming the backend systems (e.g., Vault, Consul). 💡 Suggestion Introduce a configurable limit on the maximum number of unique references that can be processed in a single `ResolveAll` call. If the number of unique references in the input exceeds this limit, the function should return an error immediately instead of attempting to resolve them. This limit can be exposed as a new `Option` for the `Resolver`.

Security Issues (1)

Severity	Location	Issue
🟡 Warning	kv/internal/resolve/resolver.go:184-206	The `ResolveAll` function, through its new `prefetch` helper, collects and attempts to resolve all unique references within an input document. There is no limit on the number of unique references processed. An attacker able to supply a crafted input document containing an extremely large number of unique references could trigger excessive memory allocation to store these references and cause a high volume of concurrent requests to backend data stores. This could lead to a Denial of Service (DoS) condition, either by exhausting memory in the service calling `ResolveAll` or by overwhelming the backend systems (e.g., Vault, Consul). 💡 Suggestion Introduce a configurable limit on the maximum number of unique references that can be processed in a single `ResolveAll` call. If the number of unique references in the input exceeds this limit, the function should return an error immediately instead of attempting to resolve them. This limit can be exposed as a new `Option` for the `Resolver`.

\n\n ### Architecture Issues (1)

Severity	Location	Issue
🟡 Warning	kv/internal/resolve/resolver.go:21	The concurrency limit for prefetching references is hardcoded to 16. While this is a reasonable default, it may not be optimal for all environments or backend stores, which might handle more or fewer concurrent requests. This lack of configurability could limit performance tuning in diverse deployments. 💡 Suggestion Consider making the concurrency limit configurable to provide more flexibility for operators. This could be exposed as an option on the `Resolver`, for example, by adding a `WithConcurrency(n int)` option that modifies a field on the `Resolver` struct. The `prefetch` method would then use this field instead of the constant.

Performance Issues (1)

Severity	Location	Issue
🟡 Warning	kv/internal/resolve/resolver.go:204-211	The prefetch logic launches a goroutine for every unique reference found in the document. While `errgroup.SetLimit` constrains concurrent execution, the `errgroup` itself must manage all scheduled goroutines. If a document contains an exceptionally large number of *unique* references (e.g., hundreds of thousands or millions), this could lead to excessive memory consumption for goroutine stacks and internal `errgroup` state before they are executed. 💡 Suggestion For improved robustness against pathological inputs with a very large number of unique references, consider processing references in batches or using a worker pool pattern with a channel to feed the references. This would limit the total number of goroutines created and managed at any given time, providing more predictable memory usage. However, for the common case of many duplicate references and a manageable number of unique ones, the current implementation is a significant improvement.

---

Powered by Visor from Probelabs

Last updated: 2026-07-13T16:22:15.745Z | Triggered by: pr_updated | Commit: 878c945

💡 TIP: You can chat with Visor using /visor ask <your question>

[vladzabolotnyi]

⚠️ All visor comments are irrelevant!
/visor the module is using Go 1.25. Can you check the Go version before creating a suggestion?

[sonarqubecloud]

Quality Gate passed

Issues
7 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

🚨 Jira Linter Failed

Commit: 878c945
Failed at: 2026-07-14 05:49:24 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details

configuration error: Jira user email is required for API authentication

Next Steps

• Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
• Verify your PR title matches the branch's Jira ticket ID
• Check that the Jira ticket exists and is accessible

---

This comment will be automatically deleted once the linter passes.