Ledgers not being offloaded to AWS S3 during testing

[oren-cohen]

Describe the bug
Using 4.1.0 version of this helm chart.
Everything appears correct on the broker's env:

s3ManagedLedgerOffloadServiceEndpoint=https://s3.eu-west-1.amazonaws.com
AWS_DEFAULT_REGION=eu-west-1
AWS_REGION=eu-west-1
AWS_ROLE_ARN=arn:aws:iam::111111111111:role/eks-pulsar
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
managedLedgerOffloadDriver=aws-s3
AWS_STS_REGIONAL_ENDPOINTS=regional
s3ManagedLedgerOffloadRegion=eu-west-1
offloadersDirectory=/pulsar/offloaders
PULSAR_PREFIX_managedLedgerOffloadThresholdInBytes=10
loggerLevelUpdates=org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG
PULSAR_PREFIX_managedLedgerOffloadDeletionLagInMillis=0
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=mybucketishere

To Reproduce
Steps to reproduce the behavior:
Use these broker settings (warning: settings are set to offload fast for testing)

broker:
  podSecurityContext:   # << testing may not be needed. please advise
    enabled: true
    fsGroup: 10000
  service_account:
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::111111111111:role/eks-pulsar
  storageOffload:
    driver: aws-s3
    bucket: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-keys"
    region: "had-no-choice-but-to-hardcode-no-extraenvs-worked-in-these-key"
    managedLedgerOffloadAutoTriggerSizeThresholdBytes: "10"
    managedLedgerOffloadDeletionLagMs: "0"
  configData:
    acknowledgmentAtBatchIndexLevelEnabled: "true"
    brokerDeleteInactiveTopicsEnabled: "false"
    offloadersDirectory: "/pulsar/offloaders". # << may not be needed
    managedLedgerMinLedgerRolloverTimeMinutes: "5" 
    managedLedgerMaxLedgerSize: "256M"
    s3ManagedLedgerOffloadServiceEndpoint: "https://s3.eu-west-1.amazonaws.com"
    loggerLevelUpdates: "org.apache.bookkeeper.mledger.offload=DEBUG,org.jclouds=DEBUG"

Also, extraenvvars for broker did not work for me for anything in the broker key, but I may create another ticket for that as it's much less blocking.

Expected behavior
It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.

Logs

Offload triggered for persistent://public/default/offload-test-3 for messages before 793:0:-1

2025-08-01T04:24:39,760+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Starting offload operation at messageId 793:0:-1

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] Start ledgersOffload. ledgers=[793] totalSize=842208

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/offload-test-3] No ledgers to offload

2025-08-01T04:24:39,761+0000 [pulsar-web-40-5] INFO  org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/offload-test-3] Completed successfully offload operation at messageId 793:0:-1

2025-08-01T04:24:39,762+0000 [pulsar-web-40-5] INFO  org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [01/Aug/2025:04:24:39 +0000] "PUT /admin/v2/persistent/public/default/offload-test-3/offload HTTP/1.1" 204 0 "-" "Pulsar-Java-v4.0.5" 4

Desktop (please complete the following information):

• OS: [e.g. iOS]

Additional context
Would appreciate any help here. I can also definitely contribute to the env vars. IMHO extraenvvars for the entire chart would make it very malleable and repeatable for people with many clusters.

[lhotari]

The topic stats and internal stats would be useful. One possible reason could be that the messages aren't retained.

In Pulsar, messages get discarded if there isn't any subscription or the message retention isn't configured. Have you checked that already?

[oren-cohen]

It looks like the topic is empty. As if it offloaded but nothing send to s3..so...

I have a subscription and I am constantly flooding it with messages now and still no s3 transfers.

Topic stats:

{ "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesInCounter" : 0, "msgInCounter" : 0, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 0, "backlogSize" : 0, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : -1, "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 0, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

Internal Stats:

{ "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesInCounter" : 0, "msgInCounter" : 0, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 0, "backlogSize" : 0, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : -1, "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 0, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

And during flooding, stats:

pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:${PULSAR_ADMIN_TOKEN}" topics stats persistent://public/default/offload-test-5 { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesInCounter" : 3021708, "msgInCounter" : 49000, "systemTopicBytesInCounter" : 0, "bytesOutCounter" : 2932073, "msgOutCounter" : 47504, "bytesOutInternalCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 3021708, "backlogSize" : 3021708, "backlogQuotaLimitSize" : -1, "backlogQuotaLimitTime" : -1, "oldestBacklogMessageAgeSeconds" : 1515, "oldestBacklogMessageSubscriptionName" : "wallah", "publishRateLimitedTimes" : 0, "earliestMsgPublishTimeInBacklogs" : 0, "offloadedStorageSize" : 0, "lastOffloadLedgerId" : 0, "lastOffloadSuccessTimeStamp" : 0, "lastOffloadFailureTimeStamp" : 1754057114270, "ongoingTxnCount" : 0, "abortedTxnCount" : 0, "committedTxnCount" : 0, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "oc" : { "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesOutCounter" : 2932073, "msgOutCounter" : 47504, "msgRateRedeliver" : 0.0, "messageAckRate" : 69.18331873911225, "chunkedMessageRate" : 0.0, "msgBacklog" : 0, "backlogSize" : 0, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 0, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "activeConsumerName" : "Mg6JN", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754058624027, "lastConsumedTimestamp" : 1754058626885, "lastAckedTimestamp" : 1754058626906, "lastMarkDeleteAdvancedTimestamp" : 1754058626906, "consumers" : [ { "appId" : "admin", "msgRateOut" : 68.99998524895315, "msgThroughputOut" : 4284.515750708608, "bytesOutCounter" : 2781984, "msgOutCounter" : 44996, "msgRateRedeliver" : 0.0, "messageAckRate" : 69.18331873911225, "chunkedMessageRate" : 0.0, "consumerName" : "Mg6JN", "availablePermits" : 504, "unackedMessages" : 0, "avgMessagesPerEntry" : 1, "blockedConsumerOnUnackedMsgs" : false, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "address" : "/172.16.124.11:43604", "connectedSince" : "2025-08-01T14:11:01.081608912Z", "clientVersion" : "Pulsar-Java-v4.0.5", "lastAckedTimestamp" : 1754058626906, "lastConsumedTimestamp" : 1754058626885, "lastConsumedFlowTimestamp" : 1754058624027, "metadata" : { }, "lastAckedTime" : "2025-08-01T14:30:26.906Z", "lastConsumedTime" : "2025-08-01T14:30:26.885Z" } ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false }, "wallah" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "messageAckRate" : 0.0, "chunkedMessageRate" : 0.0, "msgBacklog" : 49000, "backlogSize" : 3021708, "earliestMsgPublishTimeInBacklog" : 0, "msgBacklogNoDelayed" : 49000, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "msgInReplay" : 0, "unackedMessages" : 0, "type" : "Exclusive", "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 1754024182067, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "allowOutOfOrderDelivery" : false, "consumersAfterMarkDeletePosition" : { }, "drainingHashesCount" : 0, "drainingHashesClearedTotal" : 0, "drainingHashesUnackedMessages" : 0, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "subscriptionProperties" : { }, "filterProcessedMsgCount" : 0, "filterAcceptedMsgCount" : 0, "filterRejectedMsgCount" : 0, "filterRescheduledMsgCount" : 0, "dispatchThrottledMsgEventsBySubscriptionLimit" : 0, "dispatchThrottledBytesEventsBySubscriptionLimit" : 0, "dispatchThrottledMsgEventsByTopicLimit" : 0, "dispatchThrottledBytesEventsByTopicLimit" : 0, "dispatchThrottledMsgEventsByBrokerLimit" : 0, "dispatchThrottledBytesEventsByBrokerLimit" : 0, "durable" : true, "replicated" : false } }, "replication" : { }, "deduplicationStatus" : "Disabled", "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "delayedMessageIndexSizeInBytes" : 0, "compaction" : { "lastCompactionRemovedEventCount" : 0, "lastCompactionSucceedTimestamp" : 0, "lastCompactionFailedTimestamp" : 0, "lastCompactionDurationTimeInMills" : 0 }, "ownerBroker" : "pulsar-broker-1.pulsar-broker.pulsar.svc.cluster.local:8080" }

Internal stats:

{ "entriesAddedCounter" : 55500, "numberOfEntries" : 55500, "totalSize" : 3426254, "currentLedgerEntries" : 5499, "currentLedgerSize" : 342410, "lastLedgerCreatedTimestamp" : "2025-08-01T14:30:47.189Z", "waitingCursorsCount" : 1, "pendingAddEntriesCount" : 0, "lastConfirmedEntry" : "851:5498", "state" : "LedgerOpened", "ledgers" : [ { "ledgerId" : 843, "entries" : 1, "size" : 59, "offloaded" : false, "underReplicated" : false }, { "ledgerId" : 849, "entries" : 50000, "size" : 3083785, "offloaded" : false, "underReplicated" : false }, { "ledgerId" : 851, "entries" : 0, "size" : 0, "offloaded" : false, "underReplicated" : false } ], "cursors" : { "oc" : { "markDeletePosition" : "851:5498", "readPosition" : "851:5499", "waitingReadOp" : true, "pendingReadOps" : 0, "messagesConsumedCounter" : 55500, "cursorLedger" : 850, "cursorLedgerLastEntry" : 421, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2025-08-01T14:08:09.063Z", "state" : "Open", "active" : true, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : true, "subscriptionHavePendingReplayRead" : false, "properties" : { } }, "wallah" : { "markDeletePosition" : "843:-1", "readPosition" : "843:0", "waitingReadOp" : false, "pendingReadOps" : 0, "messagesConsumedCounter" : 0, "cursorLedger" : -1, "cursorLedgerLastEntry" : -1, "individuallyDeletedMessages" : "[]", "lastLedgerSwitchTimestamp" : "2025-08-01T04:56:22.042Z", "state" : "NoLedger", "active" : false, "numberOfEntriesSinceFirstNotAckedMessage" : 1, "totalNonContiguousDeletedMessagesRange" : 0, "subscriptionHavePendingRead" : false, "subscriptionHavePendingReplayRead" : false, "properties" : { } } }, "schemaLedgers" : [ ], "compactedLedger" : { "ledgerId" : -1, "entries" : -1, "size" : -1, "offloaded" : false, "underReplicated" : false } }

[lhotari]

It looks like the topic is empty. As if it offloaded but nothing send to s3..so...

I have a subscription and I am constantly flooding it with messages now and still no s3 transfers.

The offloading won't happen until the topic's active ledger rolls over to a new one.
Here are some of the related settings: https://github.com/apache/pulsar/blob/dedba1a9bf0becb1b3f80f1d89a779451734fbe6/conf/broker.conf#L1247-L1289
docs: https://pulsar.apache.org/docs/4.0.x/tiered-storage-aws/

[oren-cohen]

Hello, thanks for responding. I tried to manually close the active ledger but still it didn't post any data to s3. What would be a good way to test this if not doing that?

[lhotari]

Hello, thanks for responding. I tried to manually close the active ledger but still it didn't post any data to s3. What would be a good way to test this if not doing that?

Have you seen https://pulsar.apache.org/docs/4.0.x/tiered-storage-aws/#configure-aws-s3-offloader-to-run-manually ?

[oren-cohen]

Yes, I have tried that before. Just did it again. Every time I trigger it I get the same message. So it's not actually offloading anything.

Offload triggered for persistent://public/api/requests for messages before 1064:0:-1

No errors in log and nothing appeared in s3. This is the log I saw when I triggered it:

2025-08-11T16:12:36,718+0000 [pulsar-web-40-4] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [11/Aug/2025:16:12:36 +0000] "PUT /admin/v2/persistent/public/api/requests/offload HTTP/1.1" 307 0 "-" "Pulsar-Java-v4.0.5" 24

Edit:
My config is:

[conf/broker.conf] Applying config managedLedgerOffloadDriver = aws-s3
[conf/broker.conf] Applying config offloadersDirectory = /pulsar/offloaders
[conf/broker.conf] Applying config s3ManagedLedgerOffloadBucket = pulsar-offload-bucket
[conf/broker.conf] Applying config s3ManagedLedgerOffloadRegion = eu-west-1
[conf/broker.conf] Applying config s3ManagedLedgerOffloadServiceEndpoint = https://s3.eu-west-1.amazonaws.com
[conf/broker.conf] Adding config managedLedgerOffloadDeletionLagInMillis = 0
[conf/broker.conf] Adding config managedLedgerOffloadThresholdInBytes = 1

[oren-cohen]

Hello, I have not heard back and this issue is still open? Any suggestions on how to test s3 offloading?

[lhotari]

Hello, I have not heard back and this issue is still open? Any suggestions on how to test s3 offloading?

managedLedgerMinLedgerRolloverTimeMinutes: "5" 
managedLedgerMaxLedgerSize: "256M"

It should offload within 5 minutes when a ledger is closed to my knowledge. I tried running commands to roll it over manually and it never rolled over. I tried consuming and it didn't make a difference.

In addition, you would need to set managedLedgerMaxLedgerRolloverTimeMinutes to be "5" if you want to rollover within 5 minutes.
For testing, you can set managedLedgerMinLedgerRolloverTimeMinutes: "0" so that it's not preventing ledger rollover.

The rules for rollover isn't very obvious, it's explained here: https://github.com/apache/pulsar/blob/0903f45b9cfd97816545f8210d8e6b7dab9cdd39/conf/broker.conf#L1307-L1319

A ledger rollover is triggered after the min rollover time has passed and one of the following conditions is true:

• The max rollover time has been reached
• The max entries have been written to the ledger
• The max ledger size has been written to the ledger

For various commands, you can see how s3 offloading is tested in Pulsar integration tests:
https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/offload/TestS3Offload.java
https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/offload/TestBaseOffload.java#L40

[oren-cohen]

Thanks for the help. It got me to find this error:

KarpenterNodeRole-stg/i-0d5d3f2396b3ca609 is not authorized to perform: s3:PutObject on resource: "arn:aws:s3::offload-bucket

So I realized the Service account it was using is incorrect. When I tried to specify a service account in the broker config in helm it did not work properly FYI:

broker:
  service_account:
    name: pulsar-sa
    annotations:
      eks.amazonaws.com/role-arn:<my role arn>

It still was using this sa which I found in kubernetes: pulsar-broker-acct

Once I added that to my AWS trust policy for my role it finally worked!

Just curious why when I specify a service account for the broker and restart it, it does not pick it up or create the SA in the namespace?