Add PruneRewardAccountingTrigger to delete old reward-accounting data by dfordivam · Pull Request #6874 · canton-network/splice

dfordivam · 2026-08-20T07:21:10Z

Fixes #6136

TODO: add release notes

Pull Request Checklist

Cluster Testing

If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If an upgrade test is required, comment /upgrade_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
If a logical synchronizer upgrade test is required (from canton-3.5), comment /lsu_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

Include any change that might be observable by our partners or affect their deployment in the release notes.
Specify fixed issues with Fixes #n, and mention issues worked on using #n
Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

dfordivam · 2026-08-20T07:37:59Z

+  /** Deletes all reward-accounting data for a single round across all six
+    * tables, in a single transaction.
+    */
+  def deleteRewardAccountingDataForRound(


Indexes being used here for reference

"app_reward_batch_hashes_pkey" PRIMARY KEY, btree (history_id, round_number, batch_level, party_seq_num_begin_incl) "app_reward_root_hashes_pkey" PRIMARY KEY, btree (history_id, round_number) "app_reward_party_totals_pkey" PRIMARY KEY, btree (history_id, round_number, app_provider_party_seq_num) "app_reward_round_totals_pkey" PRIMARY KEY, btree (history_id, round_number) "app_activity_party_totals_pkey" PRIMARY KEY, btree (history_id, round_number, app_provider_party) "app_activity_round_totals_pkey" PRIMARY KEY, btree (history_id, round_number)

dfordivam · 2026-08-20T07:38:36Z

+    }
+
+  // Returns number of rows deleted
+  def pruneArchivedDataForRound(


This will always be a seq scan

dfordivam · 2026-08-20T07:44:15Z

+      summary <- appRewardsStore.deleteRewardAccountingDataForRound(task.roundNumber)
+      deletedArchiveRows <- rewardsReferenceStore.pruneArchivedDataForRound(task.roundNumber)
+    } yield TaskSuccess(
+      s"Pruned reward accounting data for round ${task.roundNumber}: " +


Example log

"Completed processing with outcome: Pruned reward accounting data for round 11: removed 5 archived rows; and RewardAccountingPruneSummary(2,1,2,1,1,1)."

Signed-off-by: Divam <dfordivam@gmail.com>

dfordivam · 2026-08-24T08:16:08Z

+                          Seq(Task(roundNumber))
+                        } else {
+                          logger.debug(
+                            s"Skipping pruning of round $roundNumber as the ingestion's currently active round " +


This is the log, though this may be noisy and perhaps can be removed

"Skipping pruning of round 12 as the ingestion's currently active round 15 opened at 1970-01-01T17:47:20Z, which is not yet past round 13's archivedAt (1970-01-01T17:47:30.000228Z)."

adetokunbo · 2026-08-25T02:16:02Z

      case _ => Future.successful(None)
    }

+  def lookupArchivedAtForOpenMiningRound(


nit: use override here, to be consistent with lookupLowestPrunableArchiveRewardRound below
Also for pruneArchiveDataForRound

adetokunbo · 2026-08-25T02:22:22Z

+        sv2HistoryId,
+        sv2RewardsRefStore,
+        newLowestOpen,
+        timeUntilSuccess = 90.seconds,


The comment above suggests that it can complete in 20s, but is sometimes much longer; is it possible to also document why there is such a large variance as a means of justifying the 90s ?

Also interested in knowing why it takes so long. Is it verdict ingestion taking so long to catch up, or the pruning trigger?

It is verdict ingestion, the cause is some grpc stream issue where it errors out if not consumed, and it need retry by the service.

adetokunbo · 2026-08-25T02:43:54Z

+
+import scala.concurrent.{ExecutionContext, Future}
+
+/** Periodically deletes reward-accounting data that is no longer needed, one


Design question: it's not mentioned in #6136, so asking here: is it confirmed that no retention window is necessary? The data in these tables is useful for debugging. IUC, this prunes them aggressively - should there be a configurable capability to retain them for a bit longer?

Good point! I think a configurable retention window with a default of 1 week (or at least a day) would be very useful for debugging. In case of a high severity issue like a root hash mismatch we might want to inspect the state of the database, and it might take a while to get a data dump from all SVs.

rautenrieth-da

Thanks! Wasn't easy to review, but looking good in general. Left some minor comments.

rautenrieth-da · 2026-08-25T07:20:13Z

+      clue(
+        s"sv2 does not prune rounds below $newLowestOpen while its verdict ingestion is paused"
+      ) {
+        hasUnprunedArchiveDataForRound(sv2RewardsRefStore, newLowestOpen - 1) shouldBe true


Should we explicitly trigger one iteration of PruneRewardAccountingTrigger here? This might mean that we have to pause the trigger for the entire duration of the test and control them manually. Although unlikely, it could be that sv2 has unpruned data simply because the trigger didn't finish yet.

rautenrieth-da · 2026-08-25T07:23:13Z

+        sv2HistoryId,
+        sv2RewardsRefStore,
+        newLowestOpen,
+        timeUntilSuccess = 90.seconds,


Also interested in knowing why it takes so long. Is it verdict ingestion taking so long to catch up, or the pruning trigger?

rautenrieth-da · 2026-08-25T07:36:10Z

+                .lookupActiveOpenMiningRounds(Seq(lastIngestedRecordTime))
+                .flatMap(_.get(lastIngestedRecordTime) match {
+                  case None =>
+                    logger.warn(


Tests are currently hitting this line. Could it be that the verdict ingestion is ahead of the reference store ingestion so that lookupActiveOpenMiningRounds(lastIngestedRecordTime) doesn't have an answer yet?

rautenrieth-da · 2026-08-25T07:40:13Z

+  )(implicit tc: TraceContext): Future[Option[CantonTimestamp]]
+
+  /** The lowest round number that may be safe to prune
+    * The criterion is based on the prescence of `OpenMiningRound`,


Suggested change

* The criterion is based on the prescence of `OpenMiningRound`,

* The criterion is based on the presence of `OpenMiningRound`,

rautenrieth-da · 2026-08-25T07:43:13Z

+  * lower round -- remain active. AND the verdict ingestion has moved past the
+  * `record_time` of OpenMiningRound's archival.
+  */
+class PruneRewardAccountingTrigger(


Can you please add some basic metrics to this trigger? Maybe a Gauge with the latest round eligible for pruning, or some meters for the values in RewardAccountingPruneSummary?

rautenrieth-da · 2026-08-25T07:45:27Z

+                where archived.store_id = $storeId and archived.migration_id = $migrationId
+                  and archived.package_name = ${pqn.packageName}
+                  and archived.template_id_qualified_name = ${pqn.qualifiedName}
+                  and archived.round is not null


Can the round ever be null for OpenMingRound contracts?

not really, I was probably trying to make use of the index, but this can be removed as we have another one to cover this.

"scan_rewards_reference_store_archived_round" btree (store_id, migration_id, round) WHERE round IS NOT NULL "scan_rewards_reference_store_archived_temporal" btree (store_id, migration_id, package_name, template_id_qualified_name, archived_at) INCLUDE (created_at)

rautenrieth-da · 2026-08-26T09:16:48Z

+       * Because both are idempotent operations and re-running either for the
+       * same or an older round is a no-op.
+       * If pruning of either store fails, the task will be retried.


We also need to consider the case where the task fails in between the two pruning operations, and then all retries fail because of intermittent database connection issues.

In that case the task will be abandoned, and a new task eventually created. The logic for creating the new task needs to work even if exactly one of the two independent pruning transactions completed. It looks like this works because we first delete the reward accounting data, and the result of retrieveTasks only depends on the TCS store which is pruned second. I would mention this in the comment though.

rautenrieth-da · 2026-08-26T09:18:05Z

+    *
+    * Returns number of rows deleted.
+    */
+  def pruneArchivedDataForRound(


Suggested change

def pruneArchivedDataForRound(

def pruneArchivedDataBeforeRound(

or similar, "for" sounds like it would only delete data for that particular round

rautenrieth-da · 2026-08-26T09:28:18Z

+
+import scala.concurrent.{ExecutionContext, Future}
+
+/** Periodically deletes reward-accounting data that is no longer needed, one


Good point! I think a configurable retention window with a default of 1 week (or at least a day) would be very useful for debugging. In case of a high severity issue like a root hash mismatch we might want to inspect the state of the database, and it might take a while to get a data dump from all SVs.

rautenrieth-da · 2026-08-26T09:39:35Z

+    )
+  )
+
+  override def retrieveTasks()(implicit tc: TraceContext): Future[Seq[Task]] =


The logic in this method is quite complex. Can we unit test it? If you want to avoid setting up the stores with real contracts, you could factor out the implementation into an independent function and then call it with test values, like we did with AcsSnapshotTrigger.retrieveTaskForCurrentMigration.

dfordivam requested a review from rautenrieth-da August 20, 2026 07:21

dfordivam commented Aug 20, 2026

View reviewed changes

Comment thread ...ralizedtrust/splice/integration/tests/TrafficBasedRewardsSvAppTimeBasedIntegrationTest.scala

dfordivam commented Aug 20, 2026

View reviewed changes

Comment thread ...ain/scala/org/lfdecentralizedtrust/splice/scan/automation/PruneRewardAccountingTrigger.scala Outdated

dfordivam added 4 commits August 24, 2026 08:11

Add PruneRewardAccountingTrigger to delete old reward-accounting data

9cc12ec

Signed-off-by: Divam <dfordivam@gmail.com>

Fix last ingested time check

4d64c4c

Signed-off-by: Divam <dfordivam@gmail.com>

Test fixes

4b88d37

Signed-off-by: Divam <dfordivam@gmail.com>

[ci]

8d8d439

Signed-off-by: Divam <dfordivam@gmail.com>

dfordivam force-pushed the dfordivam/cip-104-prune-rewardaccounting-tables branch from a82c334 to 8d8d439 Compare August 24, 2026 08:11

dfordivam commented Aug 24, 2026

View reviewed changes

adetokunbo reviewed Aug 25, 2026

View reviewed changes

rautenrieth-da reviewed Aug 26, 2026

View reviewed changes


		import scala.concurrent.{ExecutionContext, Future}

		/** Periodically deletes reward-accounting data that is no longer needed, one

	* The criterion is based on the prescence of `OpenMiningRound`,
	* The criterion is based on the presence of `OpenMiningRound`,

	def pruneArchivedDataForRound(
	def pruneArchivedDataBeforeRound(

Uh oh!

Conversation

dfordivam commented Aug 20, 2026

Pull Request Checklist

Cluster Testing

PR Guidelines

Merge Guidelines

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adetokunbo Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rautenrieth-da left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adetokunbo Aug 25, 2026 •

edited

Loading