Stop requesting GSS credential delegation (TGT NOT FORWARDABLE storm)#674
Merged
Conversation
Fork of gssapi.js 2.0.1 (bitbucket:karoshealth/node-gssapi, MIT).
Upstream passes GSS_C_DELEG_FLAG to gss_init_sec_context
unconditionally, which makes libkrb5 request a forwarded TGT from the
KDC on every client context creation - one per Factory+ HTTP token and
MQTT connection. ACS service principals hold non-forwardable TGTs, so
the KDC refuses every request ('TGT NOT FORWARDABLE ... KDC can't
fulfill requested option') and libkrb5 silently drops the flag: a
wasted KDC round trip and a KDC log line per token, multiplying into
serious noise under reconnect storms.
Nothing in Factory+ consumes delegated credentials, so the fork simply
does not request delegation. No other changes from upstream.
Publish to npm with a release tagged js/gssapi/v2.1.0 (the existing
js-lib workflow handles lib/js-* directories).
service-client tries the delegation-free fork first and falls back to upstream gssapi.js, so consumers without the fork installed keep working. The fork is an optionalDependency alongside gssapi.js: npm skips it while unpublished (installs fall back cleanly) and picks it up once it is on the registry. service-api now takes GSS from the service-client re-export instead of importing gssapi.js directly, so both sides of an ACS service resolve the same implementation and the native dependency has a single owner. A side effect is that service-api now loads on platforms with no GSS support and fails at auth time instead of at import time.
7d0c28f to
42a88e0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every Factory+ JS service floods the KDC with refused
TGT NOT FORWARDABLErequests - one per HTTP token fetch and MQTT connection. This PR removes the cause: thegssapi.jsnpm module hardcodesGSS_C_DELEG_FLAGingss_init_sec_context, so libkrb5 asks the KDC for a forwarded TGT on every client context. ACS service principals hold non-forwardable TGTs (straight from keytabs), so the KDC refuses every time:libkrb5 silently drops the flag and carries on, so nothing breaks - but every token costs an extra KDC round trip, and under reconnect storms (see #671) this multiplies into thousands of KDC log lines per minute. Nothing in Factory+ consumes delegated credentials; the server side discards them.
Changes
lib/js-gssapi: vendored fork of gssapi.js 2.0.1 (MIT, provenance in README) published as@amrc-factoryplus/gssapi2.1.0. One change: noGSS_C_DELEG_FLAG. Publishable via the existing js-lib release workflow with a release taggedjs/gssapi/v2.1.0.js-service-client:deps.jsprefers the fork and falls back togssapi.js; the fork is added tooptionalDependenciesalongside the existing entries.js-service-api: takesGSSfrom the service-client re-export instead of its owngssapi.jsimport, so the native dependency has one owner. service-api now loads on GSS-less platforms and fails at auth time rather than import time.Rollout
Safe to merge before the npm publish: npm skips unpublished optional dependencies (verified with npm 10.8.2), so installs fall back to upstream
gssapi.jsand behave exactly as today. Oncejs/gssapi/v2.1.0is released to npm, rebuilt images pick the fork up automatically. Services consuming@amrc-factoryplus/service-clientfrom the registry (historian-uns, acs-mcp) get the fix when service-client is next published and bumped.Verification
A/B tested both modules against a live ACS KDC (fpd-ago, via port-forward) from a Linux container (node:22-bookworm, MIT krb5, cmake - same toolchain as the Docker builds):
TGS_REQ ... ISSUEplusTGS_REQ (1 etypes ...) TGT NOT FORWARDABLETGS_REQ ... ISSUE, nothing refusedThe refused request's signature (single-etype TGS_REQ for krbtgt) exactly matches the storm seen from every sv1 principal in production KDC logs.
Also verified: the fork compiles on Linux and macOS; with the fork installed,
service-client'sGSSexport resolves to it; without it, the fallback togssapi.jsloads andservice-api's auth module imports cleanly.Decisions
gssapi.jsdeps in the handful of services that declare one (acs-i3x, uns-ingester-sparkplug, historian-uns): they are the fallback path and become vestigial once service-client ships the fork; removing them is trivial follow-up cleanup.