Align vendored ncrypto prime checks and empty SEC1 passphrases with Node f39899d #1094

Node landed the provider-behavior fixes from ncrypto PRs 54 and 55 in nodejs/node#67039 (f39899d). Both standalone PRs were closed without merging. The next node → ncrypto sync is what publishes those fixes to nodejs/ncrypto. Upstream main is still the Node v26.8.0 sync (c954a1a).

What landed

f39899d changes deps/ncrypto/ncrypto.cc only:

• BignumPointer::isPrime(nchecks) repeats BN_check_prime() when the caller asks for more than OpenSSL's minimum (64 rounds, or 128 above 2048 bits). Between repetitions it invokes the callback itself, because trial division on a small prime never does. A null BN_CTX returns -1.
• Encrypted SEC1 PEM with an explicit empty passphrase uses the same traditional-PEM helper as RSA. The SEC1 call passes OSSL_KEYMGMT_SELECT_ALL, so curve parameters stay in the encoding. RSA still passes OSSL_KEYMGMT_SELECT_KEYPAIR.

Explicit EC SPKI handling from ncrypto#54 was left out of the Node port because current Node already does it. The NCRYPTO_EVP_ONLY CMake profile from ncrypto#55 is not in the Node commit. The sync workflow copies include/ncrypto.h, src/ncrypto.cpp, and src/engine.cpp only, so that profile will not arrive through a sync.

What we already have

packages/react-native-quick-crypto/deps/ncrypto is detached at ef5d829 on boorad/ncrypto, one commit past the v26.8.0 sync. That commit already has the repetition loop, the empty-passphrase SEC1 branch, and NCRYPTO_EVP_ONLY. iOS and Android already compile with NCRYPTO_EVP_ONLY=1. HybridPrime and private-key export already call into ncrypto.

Two details from f39899d are missing:

1. Cancellation between prime-check rounds. The vendored loop returns early only when BN_check_prime() itself fails. Node's cctest uses prime 311, which OpenSSL resolves by trial division and never reports through its callback. tests/basic.cpp cancels 65537 at 65 checks, which does hit that callback, so the current test stays green.
2. SEC1 encoder selection. WriteEncryptedTraditionalPEM hardcodes OSSL_KEYMGMT_SELECT_KEYPAIR. The empty-passphrase SEC1 path needs OSSL_KEYMGMT_SELECT_ALL. A named curve can still round-trip. An explicit curve drops its domain parameters on that path. This is the live public case: export({ type: 'sec1', format: 'pem', cipher, passphrase: Buffer.alloc(0) }).

bitLength() and Node's GetBitCount() are both BN_num_bits, so the 64/128 threshold already matches. checkPrime forwards options.checks with no callback, so the count loop is already what callers hit. The cancellation gap shows up only when a callback is passed.

Suggested change

Patch vendored src/ncrypto.cpp inside the existing NCRYPTO_USE_OPENSSL3_PROVIDER blocks. Leave HybridPrime, prime.ts, and KeyObjectData alone.

1. In BignumPointer::isPrime, cancel between rounds:

for (int i = 0; i < repetitions; i++) {
  if (i > 0 && innerCb != nullptr && !innerCb(1, -1)) return -1;
  const int result = BN_check_prime(get(), ctx.get(), cb.get());
  if (result != 1) return result;
}

2. Give WriteEncryptedTraditionalPEM an int selection argument and pass it to OSSL_ENCODER_CTX_new_for_pkey. The PKCS#1 empty-passphrase call keeps OSSL_KEYMGMT_SELECT_KEYPAIR. The SEC1 empty-passphrase call passes OSSL_KEYMGMT_SELECT_ALL, "type-specific", and PEM_STRING_ECPRIVATEKEY.

3. Extend deps/ncrypto/tests/basic.cpp with the two Node cases the current tests miss: cancellation of a repeated Miller-Rabin check (2^127-1, 65 checks) and cancellation of a trial-division prime (311, INT_MAX checks, one callback, result -1). Add an example test that round-trips SEC1 PEM with an empty passphrase and with a passphrase that contains a NUL, then rejects a wrong passphrase and a missing one.

Keep NCRYPTO_EVP_ONLY. After nodejs/ncrypto syncs a Node commit that contains f39899d, rebase the fork onto that sync and re-apply only the EVP-only profile: the CMake and pkg-config option, the header guards, the unused-parameter cleanups, and the provider-side ECKeyHasMissingOid rewrite. The #if NCRYPTO_EVP_ONLY guards live in the three files the sync copies, so they have to be replayed on top.