Stop RPC materializer retries when ACL token is not found

[melkor217]

Summary

• Treat ACL not found as a terminal error in RPCMaterializer, matching existing LocalMaterializer and agent/cache behavior
• Stop client-agent streaming health subscriptions from retrying until idle eviction (~20 minutes) after a workload token is deleted
• Notify blocked waiters once before exiting so in-flight queries can complete

Fixes repeated agent.rpcclient.health: subscribe call failed errors reported in #22515 (Nomad workload identity / consul-template).

Test plan

• go test ./agent/submatview/... -count=1
• Deploy patched Consul client agent on a Nomad client node
• Run a job with {{ range service "..." }} template, stop the allocation, confirm no ongoing subscribe call failed / ACL not found errors in Consul logs

Made with Cursor

[hashicorp-cla-app]

All committers have signed the CLA.

[melkor217]

Before the fix (consul 1.22.5)

# journalctl -u consul | grep agent.rpcclient.health | tail -n 3
May 25 09:33:04 hashicorp-test-app02.int.tsum.com consul[1830819]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:33:04.140330+03:00","err":"rpc error: code = Unknown desc = ACL not found","error":"rpc error: code = Unknown desc = ACL not found","failure_count":9,"key":"server-in","topic":2}
May 25 09:34:03 hashicorp-test-app02.int.tsum.com consul[1830819]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:34:03.209473+03:00","err":"rpc error: code = Unknown desc = ACL not found","error":"rpc error: code = Unknown desc = ACL not found","failure_count":10,"key":"server-in","topic":2}
May 25 09:35:34 hashicorp-test-app02.int.tsum.com consul[1830819]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:35:34.466510+03:00","err":"rpc error: code = Unknown desc = ACL not found","error":"rpc error: code = Unknown desc = ACL not found","failure_count":11,"key":"server-in","topic":2}

{
  "@level": "error",
  "@message": "subscribe call failed",
  "@module": "agent.rpcclient.health",
  "@timestamp": "2026-05-25T09:35:34.466510+03:00",
  "err": "rpc error: code = Unknown desc = ACL not found",
  "error": "rpc error: code = Unknown desc = ACL not found",
  "failure_count": 11,
  "key": "server-in",
  "topic": 2
}

[melkor217]

After the fix:

# journalctl -u consul | grep agent.rpcclient.health | tail -n 3
May 25 09:42:28 hashicorp-test-app02.int.tsum.com consul[1830819]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:42:28.860266+03:00","err":"rpc error: code = Canceled desc = grpc: the client connection is closing","error":"rpc error: code = Canceled desc = grpc: the client connection is closing","failure_count":4,"key":"server-in","topic":2}
May 25 09:42:28 hashicorp-test-app02.int.tsum.com consul[1830819]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:42:28.973672+03:00","err":"rpc error: code = Canceled desc = grpc: the client connection is closing","error":"rpc error: code = Canceled desc = grpc: the client connection is closing","failure_count":4,"key":"server-in","topic":2}
May 25 09:45:21 hashicorp-test-app02.int.tsum.com consul[4058732]: {"@level":"error","@message":"subscribe call failed","@module":"agent.rpcclient.health","@timestamp":"2026-05-25T09:45:21.531086+03:00","err":"rpc error: code = Unknown desc = ACL not found","error":"rpc error: code = Unknown desc = ACL not found","failure_count":1,"key":"server-in","topic":2}

Only first error with failure_count=1 appears, no retries.