More effective views cleanup.

[Shubham8287]

Description of Changes

Existing View cleanup code is naive and doesn't anticipate large number of view subscribers, which could easily happen if clients are connecting via http sql call.

As a result I saw this log line:

[c2008d0c308db28e93c1d8785d6aa26cafdc78940da42414310a8a1cdc195fb0] DATABASE: cleared 0 expired views (585700 remaining)

We already have a due ticket to make it configuratable at database level: #3717 but I think that requires some ux decisions to make and little bit more work (I could be wrong).

This patch tries to mitigate it by doing two things:

1. When a cleanup run leaves expired views behind, nowthe loop runs more frequently until the backlog is drained, then returns to the normal interval.

2. (More important) It changes expired view query paths to use iterators and clears expired materialized views in bounded batches, avoiding collection of the full expired-view set before deletion.

API and ABI breaking changes

NA though there could be some noticeable latency sometimes as we are now clearing views more aggresively.

How complicated do you think these changes are? Grade on a scale from 1 to 5,

1.5

Testing

No functional change, existing test should cover.

[gefjon]

This all looks like an improvement, though I still think this code needs a major overhaul. Do you have a sense of what it means that the cleanup cleared 0 expired views with nearly 600k remaining in the previous version? Does that mean that our entire time budget was spent collecting the expired_items vec at line 2910 in the previous code, and didn't get to actually delete any of them?

[Shubham8287]

Does that mean that our entire time budget was spent collecting the expired_items vec at line 2910 in the previous code, and didn't get to actually delete any of them?

Yes.

[gefjon]

Thank you!

[joshua-spacetime]

I can't review these changes and won't block them, but I will just comment that making cleanup run more frequently and even making algorithmic improvements to it won't actually clean up 585700 rows if they're not already being cleaned up (which is the actual issue).

[Shubham8287]

it won't actually clean up 585700 rows if they're not already being cleaned up (which is the actual issue).

Why? All those are expired views.