Excessive partial tile fetching for Static CT logs

[FiloSottile]

Hello! We noticed a significant increase in outbound bandwidth on Tuscolo, investigated a bit, and concluded that Certstream is requesting partial tiles (the ones at the end of the log) disproportionally. I'm wondering if this is something we could work together to improve.

This is a 7d graph of http_requests_total across our logs. with-github is basically all github.com/d-Rickyy-b/certstream-server-go, because the other heavy-hitters all match with-email, with-url, or anonymous.

Filtering for with-github requests, and bucketing them by kind, we see they are almost all partial tiles.

The https://c2sp.org/static-ct-api spec says

A client, such as a Monitor, that “tails” a log SHOULD, as an optimization, avoid fetching partial data tiles when possible. If applying this optimization, the client MUST fetch the corresponding partial “level 0” tile, and use that to verify the current checkpoint. When fetching a subsequent checkpoint, the client MUST verify its consistency with the current checkpoint. If a data tile remains partial for too long (as defined by client policy), the client MUST fetch it to prevent delaying entries from being processed indefinitely.

The filippo.io/sunlight.Client.Entries implementation does that by not fetching the partial at all, unless the log grows by less than a full tile in a cycle.

https://pkg.geomys.dev/filippo.io/torchwood@v0.9.0/tlogclient.go#L139-L148

It looks like Certstream polls every second (which is a very aggressive default) and always fetches the partial.

Did anything change in the logic or defaults that would explain the qps jump?

Would you consider adopting the filippo.io/sunlight logic?

Thank you!

[d-Rickyy-b]

Hi Filippo, thanks for bringing this to my attention. I'll have a closer look at it tomorrow and will definitely try to implement it as best as possible to spec.

Sorry for the inconveniences.

[FiloSottile]

Thank you, no rush at all, this is not causing any operational issues or inconvenience. It's just nice to investigate and if possible optimize when we see one client dominate the graphs, as it helps the whole ecosystem with staying sustainable.

[FiloSottile]

I wasn't really sure how much the filippo.io/{sunlight,torchwood} policy helps, so I built an interactive tool to explore the bandwidth overhead of partial tiles at lower log growth rates. (tuscolo2026h2 is growing at 14.3/s and tuscolo2026h1 at 1.34/s.)

https://htmlpreview.github.io/?https://github.com/FiloSottile/mostly-harmless/blob/main/sunlight-rate-explorer/index.html

Turns out that the Torchwood policy doesn't help much until log growth approaches 256/s. What would help is waiting even a single extra poll cycle before fetching a partial tile.

That means for example

• at poll 0 you fetch a full tile, and skip fetching a partial of 50 entries
• at poll 1 log grew by 100, the partial is 150, wait
• at poll 2 log grew by 70, but it's been two polls since you fetched anything, so you fetch the 220 partial

would save 42.8% bandwidth at 1s polling. Increasing the polling rate to 2s would further reduce bandwidth by 42%. Combining the two cuts bandwidth by 3x. (None of this matters if you implement #105, which cuts bandwidth 20x, but otherwise it'd help.)