TT-17102: Add features to support gateway wiring

[vladzabolotnyi]

Description

Next tickets were reviewed, approved and merged to this branch:
TT-17242
TT-17487
TT-17244
TT-17243
TT-17245
TT-17246

Performance

This PR adds benchmarks covering the epic's performance acceptance criteria.
Run on Apple M4 Pro (darwin/arm64), Go benchmarks with in-process backends
(mocks / httptest), so the numbers isolate library overhead — real network
latency comes on top and is deployment-specific. Reproduce with:
go test -bench=. -run=XXX ./kv/internal/resolve ./kv/internal/store ./kv/registry

1. Whole-document resolution (ResolveAll — the startup path)

References in document	ns/op	per reference
20	23,364	~1.2 µs
50	57,038	~1.1 µs
100	110,233	~1.1 µs

Scaling is linear at ~1.1 µs per reference (parse + walk + substitute +
re-serialize, zero-latency providers). Resolution cost is dominated by the
per-document JSON round trip, not by reference count.

2. Per-request Get through the SecretStore wrapper

Mode	ns/op
cache hit	36 ns
cache disabled	1,844 ns

A cache-resident lookup costs ~36 ns.

3. Full production stack against a co-located Vault (KVv2 over httptest)

NewFromConfig registry → SecretStore → real Vault API client → HTTP:

Mode	ns/op
cache off (every Get = one backend round trip)	44,368 (~44 µs)
cache on (steady state)	35.5 ns

One co-located Vault round trip through the entire stack costs ~44 µs;
the cache collapses that to nanoseconds.

Related Issue

Motivation and Context

Test Coverage For This Change

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Refactoring or add test (improvements in base code or adds test coverage to functionality)
• Documentation updates or improvements.

Checklist

• I have reviewed the guidelines for contributing to this repository.
• Make sure you are requesting to pull a topic/feature/bugfix branch (right side). If PRing from your fork, don't come from your master!
• Make sure you are making a pull request against our master branch (left side). Also, it would be best if you started your change off our latest master.
• My change requires a change to the documentation.
  • I have manually updated the README(s)/documentation accordingly.
  • If you've changed APIs, describe what needs to be updated in the documentation.
• I have updated the documentation accordingly.
• Modules and vendor dependencies have been updated; run go mod tidy && go mod vendor
• When updating library version must provide reason/explanation for this update.
• I have added tests to cover my changes.
• All new and existing tests passed.
• Check your code additions will not fail linting checks:
  • gofmt -s -w .
  • go vet ./...

[github-actions]

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

0 out of 2 committers have signed the CLA.
❌ @vladzabolotnyi
❌ @vlad Zabolotnyi
Vlad Zabolotnyi seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You can retrigger this bot by commenting recheck in this Pull Request}

[probelabs]

This pull request introduces a comprehensive, pluggable Key-Value (KV) secrets management system. This new system supports multiple backends (Vault, Consul, environment variables, files, and inline configuration) and features a sophisticated two-phase initialization process. This allows remote secret stores to be configured using credentials sourced from local providers, resolving a common bootstrap challenge.

Files Changed Analysis

This is a large, primarily additive PR, introducing a new, well-tested subsystem. The changes are concentrated in the kv/ directory.

• New Providers: The core of the feature is the introduction of five new secret providers in kv/providers/: env, file, inline, consul, and vault. Each is self-contained and comes with extensive tests.
• Initialization Logic: The new file kv/registry/from_config.go implements the complex two-phase initialization logic, which is the central architectural pattern of this PR.
• Resolution Enhancements: kv/internal/resolve/resolver.go is modified to include a WithLenientMode. This mode is critical for the two-phase init, as it allows references to not-yet-initialized remote stores to pass through as literals during the first phase.
• Cache Bypass: A new cache-bypass mechanism is introduced via kv/context.go and implemented in kv/internal/store/store.go, allowing for forced re-fetching of secrets.
• Dependency Updates: go.mod and go.sum have been updated to include new HashiCorp client libraries for Vault and Consul integration.
• High Test Coverage: A substantial portion of the 4,700+ new lines consists of tests, indicating a high degree of validation for this new critical subsystem.

Architecture & Impact Assessment

• What this PR accomplishes

  • Decouples secret management from static configuration, allowing secrets to be sourced dynamically from multiple backends.
  • Solves the bootstrap problem where credentials for a remote secret store (e.g., a Vault token) are themselves secrets, by allowing them to be sourced from a local provider (e.g., an environment variable).
  • Introduces security-hardened local providers for file and environment variable access.

• Key technical changes introduced

  1. Two-Phase Initialization: registry.NewFromConfig first initializes local providers (env, file, inline) which have no external dependencies. It then uses a lenient resolver to process the configurations for remote providers (vault, consul), resolving any references to the already-initialized local stores. Finally, it initializes the remote providers with their resolved configurations.
  2. Security-Hardened Local Providers: The file provider is confined to a mandatory, absolute base_path to prevent directory traversal attacks. The env provider requires a prefix to limit the scope of accessible environment variables.
  3. Context-Driven Cache Bypass: The new kv.WithCacheBypass(ctx) function allows callers to instruct the system to ignore any cached secret value, fetch it directly from the source provider, and update the cache. This is useful for operational tasks like manual secret rotation.

• Affected system components

  • The primary impact is on the application's startup and configuration-loading process. All components that consume secrets will now rely on this new KV registry.

• Two-Phase Initialization Flow

graph TD
subgraph phase1 ["Phase 1: Bootstrap & Resolve"]
A[NewFromConfig with raw config] --> B{Parse kv.stores};
B --> C{Categorize Local vs. Remote stores};
C --|Local (env, file, inline)|--> D[Init Temp Registry with Local Stores];
D --> E[Resolve references in Remote Store configs];
E -- "Uses Lenient Resolver" --> F[Resolved Remote Configs];
end

subgraph phase2 ["Phase 2: Final Initialization"]
    G[Create Final Registry] --> H{Init All Stores};
    F -- "Feeds into" --> H;
    C -- "Local configs also feed into" --> H;
    H --> I[Final Registry Ready];
end

A --> G;

## Scope Discovery & Context Expansion
- **Broader Impact**: This is a foundational change that aligns the application with modern deployment practices (e.g., 12-factor app), where configuration and secrets are injected into the environment rather than being hardcoded. It provides a standardized and secure way to manage secrets across the entire application.
- **Further Exploration for Reviewers**:
  - Pay close attention to the logic in `kv/registry/from_config.go`, as it orchestrates the complex initialization flow.
  - Review the security implementations in `kv/providers/file/file.go` and `kv/providers/env/env.go` to validate the confinement mechanisms.
  - Note that the `k8s_files` provider type has been removed from `kv/provider.go`. Its functionality is now covered by the more generic and secure `file` provider, which is designed to handle Kubernetes' `AtomicWriter` symlink patterns for secret updates.
- **Performance**: The PR includes benchmarks demonstrating high performance. Cache hits for secrets are in the nanosecond range (~36ns), while a full round trip to a co-located Vault backend takes ~44µs. Document resolution at startup scales linearly at ~1.1µs per reference.


<details>
  <summary>Metadata</summary>

  - Review Effort: 4 / 5
  - Primary Label: feature


</details>
<!-- visor:section-end id="overview" -->

<!-- visor:thread-end key="TykTechnologies/storage#153@e889a78" -->

---

*Powered by [Visor](https://probelabs.com/visor) from [Probelabs](https://probelabs.com)*

*Last updated: 2026-07-10T11:37:38.340Z | Triggered by: pr_updated | Commit: e889a78*

💡 **TIP:** You can chat with Visor using `/visor ask <your question>`
<!-- /visor-comment-id:visor-thread-overview-TykTechnologies/storage#153 -->

[probelabs]

Security Issues (2)

Severity	Location	Issue
🟡 Warning	kv/providers/consul/consul.go:40	The Consul provider configuration allows disabling TLS certificate validation via `insecure_skip_verify`. When set to `true`, this makes the connection to Consul vulnerable to Man-in-the-Middle (MITM) attacks by allowing it to accept any certificate. While this may be necessary for development or testing with self-signed certificates, its use in production environments introduces a significant security risk. 💡 Suggestion Add a prominent warning in the documentation for this feature, advising against its use in production. Consider logging a warning at application startup if this option is enabled to ensure operators are aware of the insecure configuration.
🟡 Warning	kv/providers/env/env.go:83-92	The `env` provider allows an empty key, which results in a lookup of the environment variable named by the `prefix` itself. This is inconsistent with the `file` and `inline` providers, which both explicitly reject empty keys. This inconsistency can violate the principle of least surprise and may lead to unexpected behavior or misconfigurations. 💡 Suggestion For consistency and to prevent potential confusion, consider rejecting empty keys in the `env` provider, similar to the other providers. This can be done by adding a check at the beginning of the `Get` method. func (ep *envProvider) Get(_ context.Context, key string) (string, error) { if ep.prefix == "" { return "", ErrPrefixRequired } if key == "" { // It's better to use a shared error for this if possible, // but for now we can define one in the env provider. return "", errors.New("env: key must not be empty") } if ep.uppercase { key = strings.ToUpper(key) } return os.Getenv(ep.prefix + key), nil }

Security Issues (2)

Severity	Location	Issue
🟡 Warning	kv/providers/consul/consul.go:40	The Consul provider configuration allows disabling TLS certificate validation via `insecure_skip_verify`. When set to `true`, this makes the connection to Consul vulnerable to Man-in-the-Middle (MITM) attacks by allowing it to accept any certificate. While this may be necessary for development or testing with self-signed certificates, its use in production environments introduces a significant security risk. 💡 Suggestion Add a prominent warning in the documentation for this feature, advising against its use in production. Consider logging a warning at application startup if this option is enabled to ensure operators are aware of the insecure configuration.
🟡 Warning	kv/providers/env/env.go:83-92	The `env` provider allows an empty key, which results in a lookup of the environment variable named by the `prefix` itself. This is inconsistent with the `file` and `inline` providers, which both explicitly reject empty keys. This inconsistency can violate the principle of least surprise and may lead to unexpected behavior or misconfigurations. 💡 Suggestion For consistency and to prevent potential confusion, consider rejecting empty keys in the `env` provider, similar to the other providers. This can be done by adding a check at the beginning of the `Get` method. func (ep *envProvider) Get(_ context.Context, key string) (string, error) { if ep.prefix == "" { return "", ErrPrefixRequired } if key == "" { // It's better to use a shared error for this if possible, // but for now we can define one in the env provider. return "", errors.New("env: key must not be empty") } if ep.uppercase { key = strings.ToUpper(key) } return os.Getenv(ep.prefix + key), nil }

\n\n ### Architecture Issues (2)

Severity	Location	Issue
🟡 Warning	kv/registry/from_config.go:50-53	The documentation for `NewFromConfig` states that 'A store config may reference any local-type store by name', but the implementation in `resolveStoreConfigReferences` only resolves references within remote store configurations. Local store configs are treated as literal and are not resolved. This mismatch could lead to user confusion when trying to create references between local stores (e.g., an `inline` store referencing an `env` store). 💡 Suggestion Update the comment to accurately reflect that only remote store configurations can contain references to local stores. For example: 'A *remote* store config may reference any local-type store by name...'
🟡 Warning	kv/registry/registry.go:307-313	The error messages for provider creation and initialization failures have been changed, removing the provider type from the output (e.g., 'hashicorp_vault'). This information is valuable context for debugging configuration errors, and its removal is a regression in diagnosability. 💡 Suggestion Re-add the provider type to the error messages to improve diagnostics. For example, change `fmt.Errorf("create provider for %q store: %w", name, err)` back to something similar to `fmt.Errorf("create provider for %q store (type: %s): %w", name, storeCfg.Type, err)`.

Performance Issues (1)

Severity	Location	Issue
🟡 Warning	kv/providers/vault/vault.go:245-248	The construction of the physical path for Vault KVv2 secrets involves `strings.Split` and `strings.Join`, which can lead to unnecessary memory allocations on each call. This code is on the hot path for every cache miss to the Vault provider. For keys with multiple path segments, this creates an intermediate slice of strings, which is then discarded. 💡 Suggestion To reduce allocations, consider replacing the `strings.Split`/`strings.Join` combination with `strings.Index` and string slicing/concatenation. This avoids creating an intermediate slice for all path segments. 🔧 Suggested Fix i := strings.Index(key, "/") if i == -1 { return key + "/data", nil } return key[:i] + "/data" + key[i:]

Quality Issues (1)

Severity	Location	Issue
🟡 Warning	kv/registry/registry.go:276-282	The error messages for provider creation and initialization have been simplified, which removes the provider type from the error context. While the store name is preserved in the final error message from the calling `InitStores` function, the type (e.g., 'hashicorp_vault') is lost. This information is valuable for debugging configuration issues, especially when multiple stores of different types are in use. 💡 Suggestion Restore the provider type to the error messages within `buildSingleStore` to improve debuggability. For example, for the provider creation error, change the formatting to include the store name and type, similar to the original version. Original: return nil, fmt.Errorf("failed to create provider %q (type: %s): %w", name, storeCfg.Type, err) Proposed change to restore context: return nil, fmt.Errorf("failed to create provider for store %q (type %s): %w", name, storeCfg.Type, err)

---

Powered by Visor from Probelabs

Last updated: 2026-07-10T11:37:11.203Z | Triggered by: pr_updated | Commit: e889a78

💡 TIP: You can chat with Visor using /visor ask <your question>

[github-actions]

🚨 Jira Linter Failed

Commit: e889a78
Failed at: 2026-07-10 11:35:19 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details

configuration error: Jira user email is required for API authentication

Next Steps

• Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
• Verify your PR title matches the branch's Jira ticket ID
• Check that the Jira ticket exists and is accessible

---

This comment will be automatically deleted once the linter passes.

[sonarqubecloud]

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
93.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud