Skip to content

Stop requesting GSS credential delegation (TGT NOT FORWARDABLE storm)#674

Merged
AlexGodbehere merged 2 commits into
mainfrom
ago/gssapi-no-deleg
Jul 9, 2026
Merged

Stop requesting GSS credential delegation (TGT NOT FORWARDABLE storm)#674
AlexGodbehere merged 2 commits into
mainfrom
ago/gssapi-no-deleg

Conversation

@AlexGodbehere

Copy link
Copy Markdown
Contributor

Summary

Every Factory+ JS service floods the KDC with refused TGT NOT FORWARDABLE requests - one per HTTP token fetch and MQTT connection. This PR removes the cause: the gssapi.js npm module hardcodes GSS_C_DELEG_FLAG in gss_init_sec_context, so libkrb5 asks the KDC for a forwarded TGT on every client context. ACS service principals hold non-forwardable TGTs (straight from keytabs), so the KDC refuses every time:

TGS_REQ (1 etypes {aes256-cts-hmac-sha1-96(18)}) 10.42.0.42: TGT NOT FORWARDABLE:
    authtime ..., sv1configdb@... for krbtgt/..., KDC can't fulfill requested option

libkrb5 silently drops the flag and carries on, so nothing breaks - but every token costs an extra KDC round trip, and under reconnect storms (see #671) this multiplies into thousands of KDC log lines per minute. Nothing in Factory+ consumes delegated credentials; the server side discards them.

Changes

  • lib/js-gssapi: vendored fork of gssapi.js 2.0.1 (MIT, provenance in README) published as @amrc-factoryplus/gssapi 2.1.0. One change: no GSS_C_DELEG_FLAG. Publishable via the existing js-lib release workflow with a release tagged js/gssapi/v2.1.0.
  • js-service-client: deps.js prefers the fork and falls back to gssapi.js; the fork is added to optionalDependencies alongside the existing entries.
  • js-service-api: takes GSS from the service-client re-export instead of its own gssapi.js import, so the native dependency has one owner. service-api now loads on GSS-less platforms and fails at auth time rather than import time.

Rollout

Safe to merge before the npm publish: npm skips unpublished optional dependencies (verified with npm 10.8.2), so installs fall back to upstream gssapi.js and behave exactly as today. Once js/gssapi/v2.1.0 is released to npm, rebuilt images pick the fork up automatically. Services consuming @amrc-factoryplus/service-client from the registry (historian-uns, acs-mcp) get the fix when service-client is next published and bumped.

Verification

A/B tested both modules against a live ACS KDC (fpd-ago, via port-forward) from a Linux container (node:22-bookworm, MIT krb5, cmake - same toolchain as the Docker builds):

module SPNEGO token KDC log
gssapi.js 2.0.1 minted, 812 bytes TGS_REQ ... ISSUE plus TGS_REQ (1 etypes ...) TGT NOT FORWARDABLE
fork minted, 812 bytes single TGS_REQ ... ISSUE, nothing refused

The refused request's signature (single-etype TGS_REQ for krbtgt) exactly matches the storm seen from every sv1 principal in production KDC logs.

Also verified: the fork compiles on Linux and macOS; with the fork installed, service-client's GSS export resolves to it; without it, the fallback to gssapi.js loads and service-api's auth module imports cleanly.

Decisions

  • Left the direct gssapi.js deps in the handful of services that declare one (acs-i3x, uns-ingester-sparkplug, historian-uns): they are the fallback path and become vestigial once service-client ships the fork; removing them is trivial follow-up cleanup.
  • Delegation is removed globally rather than made an option. If a future service genuinely needs delegated credentials it should be an explicit per-context option in the fork.

Fork of gssapi.js 2.0.1 (bitbucket:karoshealth/node-gssapi, MIT).
Upstream passes GSS_C_DELEG_FLAG to gss_init_sec_context
unconditionally, which makes libkrb5 request a forwarded TGT from the
KDC on every client context creation - one per Factory+ HTTP token and
MQTT connection. ACS service principals hold non-forwardable TGTs, so
the KDC refuses every request ('TGT NOT FORWARDABLE ... KDC can't
fulfill requested option') and libkrb5 silently drops the flag: a
wasted KDC round trip and a KDC log line per token, multiplying into
serious noise under reconnect storms.

Nothing in Factory+ consumes delegated credentials, so the fork simply
does not request delegation. No other changes from upstream.

Publish to npm with a release tagged js/gssapi/v2.1.0 (the existing
js-lib workflow handles lib/js-* directories).
service-client tries the delegation-free fork first and falls back to
upstream gssapi.js, so consumers without the fork installed keep
working. The fork is an optionalDependency alongside gssapi.js: npm
skips it while unpublished (installs fall back cleanly) and picks it
up once it is on the registry.

service-api now takes GSS from the service-client re-export instead of
importing gssapi.js directly, so both sides of an ACS service resolve
the same implementation and the native dependency has a single owner.
A side effect is that service-api now loads on platforms with no GSS
support and fails at auth time instead of at import time.
@AlexGodbehere AlexGodbehere force-pushed the ago/gssapi-no-deleg branch from 7d0c28f to 42a88e0 Compare July 9, 2026 11:48
@AlexGodbehere AlexGodbehere merged commit b95110a into main Jul 9, 2026
1 check passed
@AlexGodbehere AlexGodbehere deleted the ago/gssapi-no-deleg branch July 9, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant