Issue setting up S3_PING with infinispan 15

[adrianodsb]

Hello everyone, I've been struggling for a few days on trying to setup S3_PING with infinispan on AWS EC2. I would appreciate any help!

Thanks in advance!

I'm using the following versions:

Infinispan: 15.0.6
OpenJDK-21
jgroups-aws: 3.0.0.Final

Other Libs:

aws-java-sdk-core-1.12.767.jar
aws-java-sdk-s3-1.12.767.jar
commons-logging-1.3.3.jar
httpclient5-5.3.1.jar
httpcore5-5.2.5.jar
joda-time-2.12.7.jar

The EC2 instance has awscli installed and there's a role attached to it, which grants full access to the S3 bucket, which is created at us-west-2 region. I can list the bucket from within the instance using "aws s3 ls" command.

Here's my infinispan config:

<infinispan
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="urn:infinispan:config:15.0 https://infinispan.org/schemas/infinispan-config-15.0.xsd
                            urn:infinispan:server:15.0 https://infinispan.org/schemas/infinispan-server-15.0.xsd"
      xmlns="urn:infinispan:config:15.0"
      xmlns:server="urn:infinispan:server:15.0">

   <jgroups>
      <stack name="bridge" extends="tcp">
        <TCP bind_addr="${jgroups.bind.address,jgroups.tcp.address:SITE_LOCAL}"
             bind_port="${jgroups.bind.port,jgroups.tcp.port:7800}"
             diag.enabled="${jgroups.diag.enabled:false}"
             thread_naming_pattern="pl"
             send_buf_size="640k"
             sock_conn_timeout="300"
             linger="${jgroups.tcp.linger:-1}"
             bundler_type="${jgroups.bundler.type:transfer-queue}"
             bundler.max_size="${jgroups.bundler.max_size:64000}"
             non_blocking_sends="${jgroups.non_blocking_sends:false}"

             thread_pool.min_threads="${jgroups.thread_pool.min_threads:0}"
             thread_pool.max_threads="${jgroups.thread_pool.max_threads:200}"
             thread_pool.keep_alive_time="60000"

             use_virtual_threads="${jgroups.thread.virtual,org.infinispan.threads.virtual:false}"
        />
        <RED/>
        <aws.S3_PING region_name="us-west-2"
                bucket_name="inspn-discovery"
                num_discovery_runs="3"
        />
        <MERGE3 min_interval="10000"
                max_interval="30000"
        />
        <FD_SOCK2 offset="${jgroups.fd.port-offset:50000}"/>
        <FD_ALL3/>
        <VERIFY_SUSPECT2 timeout="1000"/>
        <pbcast.NAKACK2 use_mcast_xmit="false"
                        xmit_interval="200"
                        xmit_table_num_rows="50"
                        xmit_table_msgs_per_row="1024"
                        xmit_table_max_compaction_time="30000"
                        resend_last_seqno="true"
        />
        <UNICAST3 conn_close_timeout="5000"
                  xmit_interval="200"
                  xmit_table_num_rows="50"
                  xmit_table_msgs_per_row="1024"
                  xmit_table_max_compaction_time="30000"
        />
        <pbcast.STABLE desired_avg_gossip="5000"
                       max_bytes="1M"
        />
        <pbcast.GMS print_local_addr="false"
                    join_timeout="${jgroups.join_timeout:2000}"
        />
        <UFC max_credits="${jgroups.max_credits:4m}"
             min_threshold="0.40"
        />
        <MFC max_credits="${jgroups.max_credits:4m}"
             min_threshold="0.40"
        />
        <FRAG4 frag_size="${jgroups.frag_size:60000}"/>
      </stack>
      <stack name="xsite" extends="udp">
         <relay.RELAY2 site="${infinispan.site.name:site1}"
                       max_site_masters="${infinispan.site.max_relay_nodes:1}"
                       xmlns="urn:org:jgroups"/>
         <remote-sites default-stack="bridge">
            <remote-site name="${infinispan.site.name:site1}"/>
            <remote-site name="${infinispan.site.name:site2}"/>
         </remote-sites>
      </stack>
   </jgroups>

   <cache-container name="default" statistics="true">
      <transport cluster="${infinispan.cluster.name:site1}" stack="${infinispan.cluster.stack:xsite}" node-name="${infinispan.node.name:}"/>
      <security>
         <authorization/>
      </security>
      <distributed-cache name="respCache" aliases="0" key-partitioner="org.infinispan.distribution.ch.impl.RESPHashFunctionPartitioner" mode="SYNC" statistics="true">
        <backups>
            <backup site="site2" strategy="ASYNC">
               <state-transfer chunk-size="600"
                            timeout="2400000"
                            max-retries="30"
                            wait-time="2000"
                            mode="AUTO"/>
            </backup>
         </backups>
         <encoding>
                <key media-type="application/octet-stream"/>
                <value media-type="application/octet-stream"/>
         </encoding>
      </distributed-cache>
   </cache-container>

   <server xmlns="urn:infinispan:server:15.0">
      <interfaces>
         <interface name="public">
            <inet-address value="${infinispan.bind.address:0.0.0.0}"/>
         </interface>
      </interfaces>

      <socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
         <socket-binding name="default" port="${infinispan.bind.port:11222}"/>
      </socket-bindings>

      <security>
         <security-realms>
            <security-realm name="default">
               <!-- Uncomment to enable TLS on the realm -->
               <!-- server-identities>
                  <ssl>
                     <keystore path="server.pfx"
                               password="password" alias="server"
                               generate-self-signed-certificate-host="localhost"/>
                  </ssl>
               </server-identities-->
               <properties-realm/>
            </security-realm>
         </security-realms>
      </security>

      <endpoints socket-binding="default" security-realm="default" />
   </server>
</infinispan>

The output:

2024-08-04 22:50:05,878 INFO  [BOOT] JVM OpenJDK 64-Bit Server VM Ubuntu 21.0.4+7-Ubuntu-1ubuntu222.04
2024-08-04 22:50:05,887 INFO  [BOOT] JVM arguments = [-server, --add-exports, java.naming/com.sun.jndi.ldap=ALL-UNNAMED, --add-opens, java.base/java.util=ALL-UNNAMED, --add-opens, java.base/java.util.concurrent=ALL-UNNAMED, -Xlog:gc*:file=/opt/infinispan-server-15.0.6.Final/server/log/gc.log:time,uptimemillis:filecount=5,filesize=3M, -Xms64m, -Xmx512m, -XX:MetaspaceSize=64M, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Djgroups.s3.region_name=us-west-2, -Djgroups.s3.bucket_name=inspn-discovery, -Dvisualvm.display.name=infinispan-server, -Djava.util.logging.manager=org.infinispan.server.loader.LogManager, -Dinfinispan.server.home.path=/opt/infinispan-server-15.0.6.Final, -classpath, :/opt/infinispan-server-15.0.6.Final/boot/infinispan-server-runtime-15.0.6.Final-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap]
2024-08-04 22:50:05,891 INFO  [BOOT] PID = 8031
2024-08-04 22:50:05,947 INFO  [o.i.SERVER] ISPN080000: Infinispan Server 15.0.6.Final starting
2024-08-04 22:50:05,947 INFO  [o.i.SERVER] ISPN080017: Server configuration: infinispan.xml
2024-08-04 22:50:05,948 INFO  [o.i.SERVER] ISPN080032: Logging configuration: /opt/infinispan-server-15.0.6.Final/server/conf/log4j2.xml
2024-08-04 22:50:06,889 INFO  [o.i.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
2024-08-04 22:50:06,890 INFO  [o.i.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
2024-08-04 22:50:06,893 INFO  [o.i.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory'
2024-08-04 22:50:06,895 WARN  [o.i.SERVER] ISPN080059: No script engines are available
2024-08-04 22:50:07,781 INFO  [o.i.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller'
2024-08-04 22:50:08,832 INFO  [o.i.CLUSTER] ISPN000078: Starting JGroups channel `site1` with stack `xsite`
2024-08-04 22:50:08,853 INFO  [o.j.JChannel] local_addr: a4fe24dd-8fc5-410f-81d9-d7444333ff77, name: ip-10-100-1-10-46634
2024-08-04 22:50:08,877 WARN  [o.j.p.UDP] JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2024-08-04 22:50:08,879 WARN  [o.j.p.UDP] JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
2024-08-04 22:50:08,879 WARN  [o.j.p.UDP] JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
2024-08-04 22:50:08,880 WARN  [o.j.p.UDP] JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
2024-08-04 22:50:08,896 INFO  [o.j.p.FD_SOCK2] server listening on *.32627
2024-08-04 22:50:10,905 INFO  [o.j.p.p.GMS] ip-10-100-1-10-46634: no members discovered after 2002 ms: creating cluster as coordinator
2024-08-04 22:50:10,929 INFO  [o.i.CLUSTER] ISPN000094: Received new cluster view for channel site1: [ip-10-100-1-10-46634|0] (1) [ip-10-100-1-10-46634]
2024-08-04 22:50:10,942 ERROR [o.j.p.r.RELAY2] ip-10-100-1-10-46634: failed starting relayer java.lang.NoClassDefFoundError: software/amazon/awssdk/services/s3/model/NoSuchBucketException
        at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3549)
        at java.base/java.lang.Class.getConstructor0(Class.java:3754)
        at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2930)
        at org.jgroups.stack.Configurator.createLayer(Configurator.java:190)
        at org.jgroups.stack.Configurator.createProtocols(Configurator.java:170)
        at org.jgroups.stack.Configurator.createProtocolsAndInitializeAttrs(Configurator.java:104)
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:65)
        at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:55)
        at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:439)
        at org.jgroups.JChannel.init(JChannel.java:916)
        at org.jgroups.JChannel.<init>(JChannel.java:128)
        at org.infinispan.remoting.transport.jgroups.EmbeddedJGroupsChannelConfigurator.createChannel(EmbeddedJGroupsChannelConfigurator.java:82)
        at org.infinispan.remoting.transport.jgroups.EmbeddedJGroupsChannelConfigurator$1.createChannel(EmbeddedJGroupsChannelConfigurator.java:126)
        at org.jgroups.protocols.relay.Relayer2.start(Relayer2.java:45)
        at org.jgroups.protocols.relay.RELAY2.startRelayer(RELAY2.java:476)
        at org.jgroups.protocols.relay.RELAY2.lambda$handleView$2(RELAY2.java:239)
        at org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:332)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ClassNotFoundException: software.amazon.awssdk.services.s3.model.NoSuchBucketException
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:593)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
        ... 21 more

2024-08-04 22:50:11,106 INFO  [o.i.CLUSTER] ISPN000079: Channel `site1` local address is `ip-10-100-1-10-46634`, physical addresses are `[10.100.1.10:48163]`
2024-08-04 22:50:11,115 INFO  [o.i.CONTAINER] ISPN000389: Loaded global state, version=15.0.6.Final timestamp=2024-08-04T22:49:54.310363947Z
2024-08-04 22:50:11,927 INFO  [o.i.CONTAINER] ISPN000104: Using EmbeddedTransactionManager
2024-08-04 22:50:12,123 ERROR [o.j.p.r.RELAY2] JGRP000290: ip-10-100-1-10-46634: no route for message from ip-10-100-1-10-46634:site1 to SiteMaster(site2): dropping message
2024-08-04 22:50:12,268 INFO  [o.i.SERVER] ISPN080018: Started connector Resp (internal)
2024-08-04 22:50:12,294 INFO  [o.i.SERVER] ISPN080018: Started connector Memcached (internal)
2024-08-04 22:50:12,355 INFO  [o.i.SERVER] ISPN080018: Started connector HotRod (internal)
2024-08-04 22:50:12,490 INFO  [o.i.SERVER] ISPN080018: Started connector REST (internal)
2024-08-04 22:50:12,512 INFO  [o.i.SERVER] ISPN005055: Using transport: Epoll
2024-08-04 22:50:12,614 INFO  [o.i.SERVER] ISPN080004: Connector SinglePort (default) listening on 0.0.0.0:11222
2024-08-04 22:50:12,615 INFO  [o.i.SERVER] ISPN080034: Server 'ip-10-100-1-10-46634' listening on http://0.0.0.0:11222
2024-08-04 22:50:12,666 INFO  [o.i.SERVER] ISPN080001: Infinispan Server 15.0.6.Final started in 6715ms

[belaban]

Looks like you're missing the JAR that contains NoBucketException. I suggest add

<dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>

[rhusar]

@adrianodsb What instructions did you follow? Clearly those are wrong and need updating. You are missing the S3 libs...

[adrianodsb]

I've been through tons of different articles, infinispan docs, jgroups docs, Github issues and in fact It's been quite a mess, I lost track of all references I used.

The problem is that Infinispan docs fairly mention this feature (probably because it's a third party lib, i guess) and does not cover this config in details. I think I will clean this up and start from scratch the right way, it's just a study lab.

Could you guys kindly point any link with accurate instructions on how to setup this in newer version of Infinispan?

And thanks for the quick answer!

[rhusar]

I've been through tons of different articles, infinispan docs, jgroups docs, Github issues and in fact It's been quite a mess, I lost track of all references I used.

The problem is that Infinispan docs fairly mention this feature (probably because it's a third party lib, i guess) and does not cover this config in details. I think I will clean this up and start from scratch the right way, it's just a study lab.

Could you guys kindly point any link with accurate instructions on how to setup this in newer version of Infinispan?

And thanks for the quick answer!

Well, I guess the main problem is that they originally did ship the protocol by default and I kept updating it directly in Infinispan, but then they removed it altogether for reasons unknown to me. And never filled in appropriate documentation.

[rhusar]

The tracker for that was - https://issues.redhat.com/browse/ISPN-15719