Zstd.compress and Zstd.decompress leak native memory when interrupted (Thread#raise, Timeout, Thread#kill) #160

Hi @SpringMT and @Watson1978,

While reviewing the C extension I found that Zstd.compress and Zstd.decompress leak native memory when a call is interrupted mid-flight.

Both release the GVL around the libzstd call via rb_thread_call_without_gvl. When that call returns, Ruby delivers any pending interrupt by raising from inside the C function. Neither function has an ensure/cleanup path, so the ZSTD_CCtx (compress) or the ZSTD_DCtx plus its 128 KB scratch buffer (decompress) is abandoned rather than freed.

In practice this is reachable via Timeout.timeout, Rack::Timeout, Sidekiq's shutdown handling (which calls Thread#raise on busy job threads), and plain Thread#kill.

Measured on macOS with the leaks tool and RUBY_FREE_AT_EXIT=1, 20 interrupted calls per scenario, current main (a4873ac). Ruby's own baseline for this harness is 1 leak / 192 bytes:

The decompress figures are a regression from my own #156, which released the GVL in Zstd.decompress. Before #156, decompress held the GVL throughout, so an interrupt could only be delivered after the call returned, with nothing left to leak. I rebuilt with just that one line reverted and confirmed it gives 0 leaks for the interrupted-decompress case. Flagging this plainly since it's my own regression to own.

The same root cause — a native context created, then a Ruby exception raised with nothing to free it — also shows up on argument errors raised after the context already exists: Zstd.compress with an unknown keyword, Zstd.compress with a level that doesn't fit in an int (e.g. 2**40), and StreamingCompress.new with an unknown keyword. Each of these leaks about 6 KB per call (21 leaks / 123,072 bytes over 20 calls).

The decompress side is already covered by @Watson1978's open PR #148, which wraps the decompress loop in rb_ensure. It currently has a few conflicts with main after #154 and #156 touched the same lines, but they're mechanical — three hunks. I rebased #148 locally (haven't pushed it anywhere) to check, and it clears every decompress-side leak above, including the interrupted-decompress one, with the suite passing. Happy to share the conflict resolution if that's useful, no pressure either way.

Side note: the Valgrind CI job added in #152 has been failing on main since it merged, and as far as I can tell it's solely down to the rb_decompress DCtx leak that #148 fixes.

I've put together a fix for the compress side in a separate PR.

Reproduction (compress side; decompress is analogous with a large output frame):

require 'zstd-ruby'
src = Random.new(1).bytes(32 * 1024 * 1024)
20.times do
  t = Thread.new { Zstd.compress(src, level: 19) rescue nil }
  sleep 0.02
  t.raise("interrupted")
  t.join
end

Run under RUBY_FREE_AT_EXIT=1 leaks --atExit -- ruby -Ilib repro.rb on macOS, or just watch RSS grow if leaks isn't available.

Thanks for taking a look.