Add compression to replicator

[lacklacklack]

Overview

Adds optional gzip compression of outbound request bodies in the replicator.
When enabled, _bulk_docs and _revs_diff request bodies are gzip-compressed
before sending. CouchDB already supports Content-Encoding: gzip on inbound
requests so no server-side changes are needed.

Compression is disabled by default to avoid breaking setups with older or
non-CouchDB targets.

Testing recommendations

Enable compression with a low threshold and run a replication, then check the stats:

# Start CouchDB
./dev/run --admin=admin:admin

# Lower the threshold for testing
curl -X PUT "http://admin:admin@127.0.0.1:15984/_node/_local/_config/replicator/compress_requests" -d '"true"'
curl -X PUT "http://admin:admin@127.0.0.1:15984/_node/_local/_config/replicator/compress_min_size" -d '"10"'

# Create DBs, add docs, replicate
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db
curl -X PUT http://admin:admin@127.0.0.1:25984/target_db
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db/doc1 \
  -H "Content-Type: application/json" \
  -d '{"data":"'$(python3 -c "print('x'*2000)'"}'
curl -X POST http://admin:admin@127.0.0.1:15984/_replicate \
  -H "Content-Type: application/json" \
  -d '{"source":"http://admin:admin@127.0.0.1:15984/source_db","target":"http://admin:admin@127.0.0.1:25984/target_db"}'

# Check stats — value should be > 0
curl -s http://admin:admin@127.0.0.1:15984/_node/_local/_stats/couch_replicator/requests_compressed | python3 -m json.tool

Automated tests: couch_replicator_compression_tests in src/couch_replicator/test/eunit/.

Related Issues or Pull Requests

• https://github.ibm.com/cloudant/dbcore/issues/694 — Enhance replicator to compress HTTP request bodies (this PR)
• https://github.ibm.com/cloudant/dbcore/issues/693 — Enhance replicator to decompress HTTP response bodies (follow-up)

Checklist

• This is my own work, I did not use AI, LLM's or similar technology
• Code is written and works correctly
• Changes are covered by tests
• Any new configurable parameters are documented in rel/overlay/etc/default.ini
• Documentation changes were made in the src/docs folder
• Documentation changes were backported (separated PR) to affected branches

[janl]

Can you please use our PR template?

[nickva]

Nice work @lacklacklack!

A few suggestions / ideas as bullet points:

1. We're trying to do a bit of both: request and response sides; let's keep it simpler at first and focus on the request side (_bulk_docs and _revs_diff). Since we already have server side handling for gzip, let's handle gzip only for the request side. Then we can cleanly test the feature in CI without extra proxies or having to add server side compression sending here too.

2. Let's skip deflate and use gzip only. But add the possibility to add zstd in the future (not this PR though) so keep the configurable compression algorithm. The reason to skip deflate is simplicity (our server handles gzip only), and deflate is a bit of a mess according to https://en.wikipedia.org/wiki/HTTP_compression

Another problem found while deploying HTTP compression on large scale is due to the deflate encoding definition: while HTTP 1.1 defines the deflate encoding as data compressed with deflate (RFC 1951) inside a zlib formatted stream (RFC 1950), Microsoft server and client products historically implemented it as a "raw" deflated stream making its deployment unreliable. For this reason, some software, including the Apache HTTP Server, only implements gzip encoding.

3. As it stands, the most important thing to compress (_bulk_docs body) won't actually be compressed. The body there isn't an iolist or binary but {BodyFun, [prefix | Docs]}}. So that makes me think maybe a better place for this is not in httpc but in api_wrap

4. Do not set AcceptEncodings = config:get("replicator", "accept_encodings", "gzip, deflate, zstd") unless we can always handle these responses and decompress them. If the server then sends us zstd data and we're on OTP 27 we won't be able to handle it and fail the request. For this pr let's just skip setting that altogether

5. Do not enable gzip compression by default. Since that is not a negotiated setting, if the replicator was talking to an older CouchDB or other server not implementing gzip decompression we'd break a customers' setup as soon as they upgrade.

6. Don't forget to fill out the template like Jan suggested

[nickva]

Looks much better!

Added a few more comments with some minor tweaks

One new major bit I thought of is if we can make this a per job overridable. In theory we have https://docs.couchdb.org/en/stable/config/replicator.html#replicator/worker_processes to copy from (and a few others). But if that proves difficult we can punt it for later

[nickva]

Looks good but needs a few more tweaks.

We could also consider gzipping _bulk_get requests. There are not as big as _bulk_docs of course but we're bothering with _revs_diff and _bulk_get is on the order of _revs_diff so if it's good for the goose -- it's good for the gander, as they say.

[nickva]

Very nice improvements. Just added a few tiny style nits then it should be good to go

[rnewson]

For the _bulk_get side, I think we can fairly simply gzip any non-chunked response, the replicator would just have to indicate that it can accept them (Accept-Encoding: gzip).

diff --git a/src/couch/src/couch_httpd.erl b/src/couch/src/couch_httpd.erl
index 0a9e39e7a..fd63ec444 100644
--- a/src/couch/src/couch_httpd.erl
+++ b/src/couch/src/couch_httpd.erl
@@ -1395,10 +1395,23 @@ before_response(Req0, Code0, Headers0, {json, JsonObj}) ->
     {ok, {Req1, Code1, Headers1, Body1}} =
         chttpd_plugin:before_response(Req0, Code0, Headers0, JsonObj),
     Body2 = [start_jsonp(), ?JSON_ENCODE(Body1), end_jsonp(), $\n],
-    {ok, {Req1, Code1, Headers1, Body2}};
+    {Headers2, Body3} = maybe_compress_request_body(Req0, Headers1, Body2),
+    {ok, {Req1, Code1, Headers2, Body3}};
 before_response(Req0, Code0, Headers0, Args0) ->
     chttpd_plugin:before_response(Req0, Code0, Headers0, Args0).

+maybe_compress_request_body(Req, Headers, Body) ->
+    case accepts_gzip(Req) of
+        true ->
+            {[{~"Content-Encoding", ~"gzip"} | Headers],
+             zlib:gzip(Body)};
+        false ->
+            {Headers, Body}
+    end.
+
+accepts_gzip(#httpd{} = Req) ->
+    lists:member("gzip", couch_httpd:accepted_encodings(Req)).
+
 respond_(#httpd{mochi_req = MochiReq} = Req, Code, Headers, Args, Type) ->
     case MochiReq:get(socket) of
         {remote, Pid, Ref} ->

[rnewson]

doing all the chunked responses is a little more involved but worth looking it. we'd need to stash the result of zlib:open somewhere. Ideally in the Resp value but might have to be the process dictionary. Since we have start_chunked_response, send_chunk and last_chunk it should be possible. The only worry I have is for any consumer that would like parse those responses line-by-line, the compression would mess with that.

[lacklacklack]

doing all the chunked responses is a little more involved but worth looking it. we'd need to stash the result of zlib:open somewhere. Ideally in the Resp value but might have to be the process dictionary. Since we have start_chunked_response, send_chunk and last_chunk it should be possible. The only worry I have is for any consumer that would like parse those responses line-by-line, the compression would mess with that.

Thank you for your comments, I'll keep this noted for a later follow-up PR which will include server side compression.

[nickva]

Let's do accept/response gzip encoding in a separate PR just to keep this one simpler.

We'd also want to add some documentation about this feature.

And don't forget to run erlfmt

[nickva]

Good catch by @rnewson that zlib:gzip takes iolists that means we don't have have to create a binary here. The FullBody could just be [Prefix, lists:join(",", Docs), Suffix] just check that we don't do byte_size or is_binary calls on the result