Repository navigation
test(parquet): pyarrow encryption interop tests - #1439
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
Thanks, this is a useful harness. With pyarrow 26 installed, go test -race -run Pyarrow ./parquet/ passes: Go reads all four pyarrow-written configs, and pyarrow reads the Go-written file. Without pyarrow everything skips cleanly.
- CI: please include the one-line addition to
ci/scripts/cgo_python_test.shin this PR. That job is the only one with pyarrow installed, and without it these tests always skip in CI. TestPyarrowKmsWrittenConfigs: it only skips, so I'd leave it out and add it with the PKMT1 follow-up. #566 already tracks the gap.- Optional: once CI runs these, an env var in that job that turns the "pyarrow not available" skip into a failure would keep an image change from silently disabling them.
- Nit:
plainfooter_directwould also pass with an encrypted footer; asserting the footer is plaintext would make the case check what its name says.
9082c75 to
39bd28a
Compare
|
All four done: |
Cross-language coverage for the direct-key encryption configs: read pyarrow-written files in Go (uniform GCM, AES-GCM-CTR, AAD prefix, plaintext footer) and have pyarrow read an arrow-go-written file. The helper script is embedded in the test binary and driven through python3; tests skip cleanly when python3 or pyarrow isn't installed. Part of apache#566
39bd28a to
7403ca5
Compare
|
Fixed — the script ran go test from the container workdir instead of the repo root. Needs another workflow approval to re-run. |
|
Thanks for the quick turnaround. All four points are addressed. Locally with pyarrow 26.0.1, A few small nits, none blocking:
|
zeroshade
left a comment
There was a problem hiding this comment.
CI is green, including both CGO Python jobs, which now run the interop tests with PARQUET_TEST_REQUIRE_PYARROW=1. Thanks!
New optional parquet/encryption package with a KmsClient interface and a KmsKeyRetriever that plugs into parquet.WithKeyRetriever, so files written by the parquet-cpp/pyarrow CryptoFactory can be read: PKMT1 key metadata is parsed, and DEKs are resolved via the KMS client (single wrapping) or by unwrapping the KEK through the KMS and decrypting the locally wrapped DEK with AES-GCM (double wrapping, AAD = decoded KEK ID bytes, matching parquet-cpp EncryptKeyLocally: nonce || ciphertext || tag). Interop tests against pyarrow CryptoFactory output cover both wrapping modes; the test KMS wraps with plain base64 so no extra python dependencies are needed and the existing cgo python CI job runs them unchanged. Also addresses the review nits from apache#1439 (renamed the requirePyarrow flag, dropped the unused footer key metadata, deduped a doc comment). Part of apache#566
New optional parquet/encryption package with a KmsClient interface and a KmsKeyRetriever that plugs into parquet.WithKeyRetriever, so files written by the parquet-cpp/pyarrow CryptoFactory can be read: PKMT1 key metadata is parsed, and DEKs are resolved via the KMS client (single wrapping) or by unwrapping the KEK through the KMS and decrypting the locally wrapped DEK with AES-GCM (double wrapping, AAD = decoded KEK ID bytes, matching parquet-cpp EncryptKeyLocally: nonce || ciphertext || tag). Interop tests against pyarrow CryptoFactory output cover both wrapping modes; the test KMS wraps with plain base64 so no extra python dependencies are needed and the existing cgo python CI job runs them unchanged. Also addresses the review nits from apache#1439 (renamed the requirePyarrow flag, dropped the unused footer key metadata, deduped a doc comment). Part of apache#566
New optional parquet/encryption package with a KmsClient interface and a KmsKeyRetriever that plugs into parquet.WithKeyRetriever, so files written by the parquet-cpp/pyarrow CryptoFactory can be read: PKMT1 key metadata is parsed, and DEKs are resolved via the KMS client (single wrapping) or by unwrapping the KEK through the KMS and decrypting the locally wrapped DEK with AES-GCM (double wrapping, AAD = decoded KEK ID bytes, matching parquet-cpp EncryptKeyLocally: nonce || ciphertext || tag). Interop tests against pyarrow CryptoFactory output cover both wrapping modes; the test KMS wraps with plain base64 so no extra python dependencies are needed and the existing cgo python CI job runs them unchanged. Also addresses the review nits from apache#1439 (renamed the requirePyarrow flag, dropped the unused footer key metadata, deduped a doc comment). Part of apache#566
…1478) ### Rationale for this change #566 reported that files written by pyarrow's CryptoFactory cannot be read by arrow-go: the footer and column key metadata carry PKMT1 key material with a wrapped DEK, and arrow-go had no parser or unwrapping for it, so footer decryption failed without a hand-written `KeyRetriever`. As agreed in #566, the KMS support lives in the library as an optional component. ### What changes are included in this PR? A new optional `parquet/encryption` package: - `KmsClient` interface mirroring the parquet-cpp/pyarrow contract (`WrapKey`/`UnwrapKey`) - `KeyMaterial`/`ParseKeyMetadata` for PKMT1 key metadata (internal key storage) - `KmsKeyRetriever` implementing `parquet.DecryptionKeyRetriever`: in single wrapping mode the DEK is unwrapped by the `KmsClient`; in double wrapping mode the KEK is unwrapped through the client and the DEK is then decrypted locally with AES-GCM (`nonce || ciphertext || tag`, AAD = the base64 decoded KEK ID bytes), byte-compatible with parquet-cpp's `EncryptKeyLocally` - unwrapped KEKs are cached (10 minute expiry, 64 keys, LRU eviction; concurrent misses share one KMS call), so a double-wrapped file costs one KMS call per KEK instead of one per encrypted column plus the footer It plugs into the existing `parquet.WithKeyRetriever` option; code that does not import the package is unaffected. The package doc and a godoc example (`ExampleKmsKeyRetriever`) show the wiring and state the optional nature explicitly. The interop harness from #1439 gains a `gen-kms` mode: pyarrow's CryptoFactory writes double-wrapped (the default) and single-wrapped files using a test KMS client whose wrapping is plain base64, and Go reads both through the retriever — the footer key and a column key both take that path. No new python dependencies, so the existing cgo python CI job runs them unchanged. The review nits from #1439 are addressed (flag rename, unused footer key metadata dropped, doc comment dedupe). The write side (Go producing PKMT1 key material via a CryptoFactory) is a natural follow-up; this PR is read-only. ### Are these changes tested? Unit tests cover PKMT1 parsing (valid and error cases), single and double unwrap round-trips, and tamper detection. Against pyarrow 26.0.0 locally, with `-race` and `PARQUET_TEST_REQUIRE_PYARROW=1`, both interop modes pass; without pyarrow everything skips. The interop tests also count KMS unwrap calls: three key slots under two master keys resolve in two calls in double wrapping mode, three in single wrapping mode. ### Are there any user-facing changes? New optional public package `parquet/encryption`. No changes to existing APIs.
Rationale for this change
#566 reports encryption and decryption failures between pyarrow and arrow-go. The direct-key half no longer reproduces on main: reading pyarrow-written files in Go works for uniform AES-GCM, AES-GCM-CTR, AAD prefix and plaintext footer, and pyarrow reads an arrow-go-written file. Checked against pyarrow 26; I suspect #596/#1390/#1412 fixed this but haven't bisected. Nothing tests these paths in either direction today, so it could regress unnoticed.
What still fails is KMS. pyarrow's
CryptoFactorywrites PKMT1 key material with a wrapped DEK, arrow-go has no parser or unwrapping for it, so footer decryption fails without a custom retriever. That gap stays tracked in #566; per review the skip-only KMS test was left out of this PR and belongs with the PKMT1 follow-up.What changes are included in this PR?
encryption_interop_pyarrow.pygenerates the encrypted files and reads a Go-written one; it is embedded in the test binary and run throughpython3.encryption_interop_pyarrow_test.goholds the tests. They skip whenpython3orpyarrowisn't installed, so the default test run is unchanged.plainfooter_directalso asserts the footer magic staysPAR1(an encrypted footer would end inPARE1) so the case checks what its name says.CI:
ci/scripts/cgo_python_test.sh, which runs in thedocker-cgo-pythonjob (the only one with pyarrow installed), now runsgo test -run Pyarrowon the package withPARQUET_TEST_REQUIRE_PYARROW=1, which turns the "pyarrow not available" skip into a failure so an image change can't silently disable the tests. Since the install is unpinned, the tests follow the current pyarrow release.Are these changes tested?
The tests are the change. Against pyarrow 26 locally: the four direct-key configs pass and pyarrow reads the Go-written file back correctly. Without python3 or pyarrow everything skips.
Are there any user-facing changes?
No, tests only.