Skip to content

insane logging by nfs #8119

Open
Open
insane logging by nfs#8119
@calestyo

Description

@calestyo
calestyo
opened on May 25, 2026

Hey.

Last night there was a large scale power cut on our computing centre. The dCache head node (which has a NFS door) survived because it has USV...

Turns out however, that caused some really insane logging outburst by the nfs comain, which fille 400GB in a few hours.

It started with around 4000 lines of:

2026-05-18T17:47:24.242282+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:47:24+02:00 (nfs_lcg-lrz-dcache0) [NIC auto-discovery] Adding [lcg-lrz-dcache0.grid.lrz.de/[2001:4ca0:0:201:a::], lcg-lrz-dcache0.grid.lrz.de/129.187.131.210]
2026-05-18T17:50:02.393014+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:02+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:12.324371+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:12+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:12.404539+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:12+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:22.334309+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:22+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:22.415836+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:22+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:32.354377+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:32+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:32.426347+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:32+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:42.363961+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:42+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-18T17:50:42.470242+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-18 17:50:42+02:00 (nfs_lcg-lrz-dcache0) [door:nfs_lcg-lrz-dcache0@nfs:AAZSGYP50Bg 000013B21E344F474CB695E5A7BDC8D627B3 /[0:0:0:0:0:0:0:1]:774] No write pools online for [net=0:0:0:0:0:0:0:1,protocol=NFS4/4,store=atlas:STATIC@osm,cache=,linkgroup=]
2026-05-25T03:48:07.489413+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:07+02:00 (nfs_lcg-lrz-dcache0) [NIC auto-discovery] Removing [lcg-lrz-dcache0.grid.lrz.de/[2001:4ca0:0:201:a::], lcg-lrz-dcache0.grid.lrz.de/129.187.131.210]
2026-05-25T03:48:14.025293+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:14+02:00 (System) [] Ignoring empty event CONNECTION_SUSPENDED
2026-05-25T03:48:41.431538+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:41+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:48:41.431677+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:48:41.431766+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:48:41.431865+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:48:41.431958+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:48:41.432040+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:48:41.432124+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:48:41.432206+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:48:41.432287+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:48:41.432369+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:48:41.432451+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:48:41.432533+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:48:41.432620+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:48:42.367454+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:42+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:48:54.012632+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:54+02:00 (System) [] Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 40000. Adjusted session timeout ms: 40000
2026-05-25T03:48:54.014666+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:54+02:00 (System) [] Session expired event received
2026-05-25T03:48:54.018641+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:48:54+02:00 (System) [] Ignoring empty event CONNECTION_LOST
2026-05-25T03:49:09.064236+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:49:09+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:49:09.064863+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:49:09.065565+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:49:09.066042+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:49:09.066499+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:49:09.066974+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:49:09.067543+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:49:09.068078+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:49:09.068522+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:49:09.069006+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:49:09.069518+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:49:09.069984+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:49:09.070576+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:49:21.351907+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:49:21+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:49:21.352160+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:49:21.352268+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:49:21.352358+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:49:21.352447+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:49:21.352530+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:49:21.352614+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:49:21.352696+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:49:21.352805+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:49:21.352889+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:49:21.352972+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:49:21.353055+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:49:21.353138+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:49:21.920050+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:49:21+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:49:49.001983+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:49:48+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:49:49.002172+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:49:49.002328+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:49:49.002416+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:49:49.002503+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:49:49.002587+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:49:49.002669+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:49:49.002752+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:49:49.002834+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:49:49.002917+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:49:49.003000+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:49:49.003081+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:49:49.003164+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:49:49.439177+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:49:49+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:50:16.747189+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:50:16+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:50:16.747400+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:50:16.747520+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:50:16.747612+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:50:16.747701+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:50:16.747787+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:50:16.747873+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:50:16.747958+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:50:16.748043+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:50:16.748129+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:50:16.748215+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:50:16.748300+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:50:16.748385+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:50:17.492042+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:50:17+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:50:41.325873+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:50:41+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:50:41.326024+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:50:41.326114+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:50:41.326199+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:50:41.326304+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:50:41.326387+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:50:41.326470+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:50:41.326552+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:50:41.326634+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:50:41.326740+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:50:41.326839+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:50:41.326922+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:50:41.327003+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:50:41.469247+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:50:41+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:51:11.921484+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:51:11+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:51:11.921567+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:51:11.921614+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:51:11.921658+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:51:11.921703+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:51:11.921747+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:51:11.921791+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:51:11.921835+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:51:11.921878+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:51:11.921921+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:51:11.921965+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:51:11.922008+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:51:11.922052+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:51:12.854451+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:51:12+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:51:48.529745+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:51:48+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:51:48.529815+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:51:48.529853+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:51:48.529897+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:51:48.529933+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:51:48.529968+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:51:48.530003+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:51:48.530036+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:51:48.530071+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:51:48.530106+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:51:48.530141+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:51:48.530175+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:51:48.530219+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:51:48.758325+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:51:48+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:52:25.137364+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:52:25+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:52:25.137517+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:52:25.137639+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:52:25.137726+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:52:25.137811+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:52:25.137893+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:52:25.137974+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:52:25.138056+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:52:25.138138+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:52:25.138248+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:52:25.138336+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:52:25.138419+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:52:25.138501+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:52:26.056233+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:52:26+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:53:01.738021+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:53:01+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:53:01.738281+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:53:01.738407+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:53:01.738496+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:53:01.738582+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:53:01.738666+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:53:01.738749+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:53:01.738831+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:53:01.738913+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:53:01.738997+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:53:01.739080+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:53:01.739164+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:53:01.739246+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:53:02.704240+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:53:02+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:53:35.338084+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:53:35+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:53:35.338338+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:53:35.338461+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:53:35.338549+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:53:35.338633+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:53:35.338718+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:53:35.338801+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:53:35.338883+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:53:35.338985+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:53:35.339071+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:53:35.339154+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:53:35.339236+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:53:35.339319+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:53:35.412120+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:53:35+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:54:11.951662+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:54:11+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:54:11.951818+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:54:11.951910+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:54:11.951992+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:54:11.952077+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:54:11.952177+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:54:11.952277+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:54:11.952362+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:54:11.952444+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:54:11.952526+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:54:11.952608+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:54:11.952690+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:54:11.952772+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:54:12.549368+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:54:12+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:54:45.810868+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:54:45+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:54:45.811020+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:54:45.811110+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:54:45.811193+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:54:45.811277+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:54:45.811375+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:54:45.811459+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:54:45.811541+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:54:45.811623+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:54:45.811704+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:54:45.811786+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:54:45.811868+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:54:45.811949+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:54:46.579244+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:54:46+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:55:17.325929+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:55:17+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:55:17.326112+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:55:17.326205+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:55:17.326339+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:55:17.326426+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:55:17.326510+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:55:17.326592+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:55:17.326675+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:55:17.326721+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:55:17.326766+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:55:17.326811+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:55:17.326856+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:55:17.326901+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:55:17.541322+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:55:17+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:55:45.428091+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:55:45+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:55:45.428244+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:55:45.428336+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:55:45.428422+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:55:45.428509+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:55:45.428594+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:55:45.428680+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:55:45.428783+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:55:45.428870+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:55:45.428954+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:55:45.429040+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:55:45.429125+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:55:45.429210+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:55:46.134053+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:55:46+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:56:16.008025+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:56:16+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:56:16.008435+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:56:16.008565+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:56:16.008670+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:56:16.008763+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:56:16.008849+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:56:16.008955+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:56:16.009041+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:56:16.009136+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:56:16.009220+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:56:16.009330+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:56:16.009416+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:56:16.009500+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:56:16.468955+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:56:16+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:56:52.615969+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:56:52+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:56:52.616444+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:56:52.616578+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:56:52.616692+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:56:52.616784+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:56:52.616868+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:56:52.616974+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:56:52.617060+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:56:52.617156+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:56:52.617242+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:56:52.617339+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:56:52.617424+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:56:52.617506+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:56:53.417003+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:56:53+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:57:23.271989+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:57:23+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:57:23.272477+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:57:23.272646+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:57:23.272761+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:57:23.272854+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:57:23.272939+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:57:23.273046+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:57:23.273152+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:57:23.273252+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:57:23.273338+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:57:23.273434+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:57:23.273520+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:57:23.273603+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:57:23.742031+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:57:23+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:57:53.866297+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:57:53+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:57:53.866498+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:57:53.866595+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:57:53.866680+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:57:53.866765+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:57:53.866847+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:57:53.866930+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:57:53.867012+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:57:53.867094+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:57:53.867175+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:57:53.867257+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:57:53.867339+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:57:53.867421+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:57:54.761012+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:57:54+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:58:24.465747+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:58:24+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:58:24.465814+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:58:24.465853+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:58:24.465890+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:58:24.465925+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:58:24.465961+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:58:24.465996+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:58:24.466031+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:58:24.466075+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:58:24.466111+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:58:24.466146+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:58:24.466181+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:58:24.466229+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:58:25.025979+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:58:25+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:58:49.036480+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:58:49+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:58:49.036656+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:58:49.036752+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:58:49.036841+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:58:49.036929+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:58:49.037015+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:58:49.037100+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:58:49.037185+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:58:49.037270+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:58:49.037355+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:58:49.037441+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:58:49.037526+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:58:49.037610+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:58:49.338290+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:58:49+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:59:13.612576+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:59:13+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:59:13.612735+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:59:13.612825+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:59:13.612927+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:59:13.613015+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:59:13.613097+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:59:13.613179+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:59:13.613261+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:59:13.613342+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:59:13.613425+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:59:13.613506+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:59:13.613588+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:59:13.613670+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:59:13.998100+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:59:13+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T03:59:38.189759+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:59:38+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T03:59:38.189924+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T03:59:38.190036+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T03:59:38.190122+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T03:59:38.190227+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T03:59:38.190313+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T03:59:38.190395+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T03:59:38.190476+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T03:59:38.190557+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T03:59:38.190638+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T03:59:38.190720+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T03:59:38.190801+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T03:59:38.190882+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T03:59:38.661179+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 03:59:38+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:02.766042+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:02+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:02.766191+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T04:00:02.766301+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T04:00:02.766386+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T04:00:02.766470+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T04:00:02.766553+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T04:00:02.766635+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T04:00:02.766718+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T04:00:02.766800+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T04:00:02.766881+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T04:00:02.766963+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T04:00:02.767045+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T04:00:02.767127+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T04:00:03.547237+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:03+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:27.345053+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:27+02:00 (System) [] Unable to resolve address: lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:27.345144+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc51.grid.lrz.de: Temporary failure in name resolution
2026-05-25T04:00:27.345202+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T04:00:27.345255+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T04:00:27.345307+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T04:00:27.345358+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T04:00:27.345409+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T04:00:27.345472+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T04:00:27.345525+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T04:00:27.345575+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T04:00:27.345627+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T04:00:27.345678+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T04:00:27.345763+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T04:00:28.007168+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:28+02:00 (System) [] Unable to open socket to lcg-lrz-dc51.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:51.917666+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:51+02:00 (System) [] Unable to resolve address: lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181
2026-05-25T04:00:51.917836+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: java.net.UnknownHostException: lcg-lrz-dc50.grid.lrz.de: Temporary failure in name resolution
2026-05-25T04:00:51.917926+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
2026-05-25T04:00:51.918010+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:52)
2026-05-25T04:00:51.918097+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$PlatformResolver.lookupByName(InetAddress.java:1211)
2026-05-25T04:00:51.918181+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1828)
2026-05-25T04:00:51.918298+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:1139)
2026-05-25T04:00:51.918384+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1818)
2026-05-25T04:00:51.918466+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1688)
2026-05-25T04:00:51.918549+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
2026-05-25T04:00:51.918631+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
2026-05-25T04:00:51.918713+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
2026-05-25T04:00:51.918794+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: #011at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
2026-05-25T04:00:52.001984+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 04:00:52+02:00 (System) [] Unable to open socket to lcg-lrz-dc50.grid.lrz.de/<unresolved>:2181

which I guess is simply fallout from the rest of the computing centre having gone down ;-)

But then comes the outburst:

2026-05-25T05:00:14.014865+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.91 ms, Retrying...
2026-05-25T05:00:14.014895+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.91 ms, Retrying...
2026-05-25T05:00:14.014920+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.92 ms, Retrying...
2026-05-25T05:00:14.014944+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.93 ms, Retrying...
2026-05-25T05:00:14.014969+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.93 ms, Retrying...
2026-05-25T05:00:14.014993+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.94 ms, Retrying...
2026-05-25T05:00:14.015030+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.94 ms, Retrying...
2026-05-25T05:00:14.015055+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.95 ms, Retrying...
2026-05-25T05:00:14.015080+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.95 ms, Retrying...
2026-05-25T05:00:14.015104+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.96 ms, Retrying...
2026-05-25T05:00:14.015129+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.96 ms, Retrying...
2026-05-25T05:00:14.015153+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.97 ms, Retrying...
2026-05-25T05:00:14.015183+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.97 ms, Retrying...
2026-05-25T05:00:14.015209+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.98 ms, Retrying...
2026-05-25T05:00:14.015233+02:00 lcg-lrz-dcache0 dcache@nfs[3456]: 2026-05-25 05:00:13+02:00 (nfs_lcg-lrz-dcache0) [/[0:0:0:0:0:0:0:1]:774 0000080000000013EA84770130] Failed to connect to pool [] within 49.99 ms, Retrying...

millions and millions....

# head -n 100000 nfs.log  | grep --color=auto '^2026-05-25T05:00:14\.' | wc -l
85389

(the head because the file is so large)... around 85k log messages from dCache per second.

There should be some rate limiting ;-)

Cheers,
Chris.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions