Repository navigation
feat(parquet/encryption): KMS support for writing PKMT1 key material - #1567
EdwardShen125 wants to merge 4 commits into
Conversation
CryptoFactory builds parquet.FileEncryptionProperties whose keys are wrapped by a KmsClient, closing the loop with the read side of apache#1478: fresh 16-byte DEKs are generated for the footer and every configured column, wrapped single (straight through the KMS) or double (a cached per-master-key KEK wraps DEKs locally with AES-GCM, nonce || ciphertext || tag, AAD = KEK ID bytes, and only the KEK goes through the KMS), and serialized as PKMT1 key material in the footer and column key metadata. The material matches parquet-cpp byte for byte: field order, omitted kmsInstance fields on column keys (KeyMaterial now marshals them, and the single-wrapping KEK fields, with omitempty), a fresh DEK per column, one KEK per master key reused across files, and KMS call counts symmetric with the read side (2 unwraps for two master keys in double wrapping mode, 1 per key slot in single wrapping mode). Interop: the harness gains a read-kms mode, and Go-written files read back through pyarrow's CryptoFactory in double, single, and plaintext footer configurations; a Go write/read round trip resolves every key through KmsKeyRetriever alone.
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for the follow-up. The files interoperate: the three interop tests pass against pyarrow 26.0.1, and pyarrow's CryptoFactory also read uniform (double and single wrapping), uniform with an AAD prefix, and one-encrypted-column files I wrote with this branch. The read-side tests still pass.
Four things to change before this goes in, inline. The first is the important one: the KEK cache is described as matching parquet-cpp, and it differs from it in two ways that matter.
| // KMS call so concurrent files sharing a master key get one wrap. | ||
| func (f *CryptoFactory) kekFor(client KmsClient, kmsCfg KmsConnectionConfig, | ||
| masterKeyID string) (cachedKek, error) { | ||
| key := kekReuseKey{kmsInstanceID: kmsCfg.KmsInstanceID, masterKeyID: masterKeyID} |
There was a problem hiding this comment.
The cache differs from parquet-cpp in two ways (FileKeyWrapper in file_key_wrapper.cc), and the description's "held for the factory's lifetime, like parquet-cpp" isn't right.
-
parquet-cpp keys the KEK cache by access token (
kek_write_cache_per_token().GetOrCreateInternalCache(kms_connection_config.key_access_token(), ...)). Here the key is the KMS instance ID and the master key only. With a KMS client that refuses one token: that token getsaccess deniedon first use, but once another token has cached the KEK for the same master key, the refused token succeeds without the KMS being asked. With single wrapping the same token still fails, as it should. -
parquet-cpp expires entries after
cache_lifetime_seconds, 600 by default. Here a KEK is never dropped, so a long-running writer wraps every file's keys with the same KEK indefinitely.
Please key the cache by access token as well, and give entries a lifetime: configurable, defaulting to 10 minutes. The doc comment on CryptoFactory and the PR description need the same correction.
| func (f *CryptoFactory) kekFor(client KmsClient, kmsCfg KmsConnectionConfig, | ||
| masterKeyID string) (cachedKek, error) { | ||
| key := kekReuseKey{kmsInstanceID: kmsCfg.KmsInstanceID, masterKeyID: masterKeyID} | ||
| f.mu.Lock() |
There was a problem hiding this comment.
This lock covers every master key, and it is held across the KMS call. Three files with three different master keys, written concurrently against a KMS client whose wrap takes 300ms, took 900ms. Please hold it per cache entry, so that only writers waiting for the same KEK wait for one another.
| // Key sizes for generated keys, matching the parquet-cpp/pyarrow | ||
| // CryptoFactory (AES-128 data and key encryption keys). | ||
| const ( | ||
| dataKeySize = 16 |
There was a problem hiding this comment.
parquet-cpp's EncryptionConfiguration also carries encryption_algorithm and data_key_length_bits (128, 192 or 256). Please add both to EncryptionConfiguration, with these values as the defaults, so that a file written here can match what a parquet-cpp or pyarrow writer is configured to produce.
| // key. Columns not listed stay in plaintext. When empty (or nil) the | ||
| // configuration is uniform: all columns are encrypted with the footer | ||
| // key. | ||
| ColumnKeys map[string][]string |
There was a problem hiding this comment.
parquet-cpp requires an explicit uniform_encryption flag: it rejects a configuration with neither it nor column keys ("Either column_keys or uniform_encryption must be set"), and one with both. Please do the same here, with a UniformEncryption field, instead of treating an empty map as uniform. A caller who builds the map from input that turns out empty should get an error, not a different encryption layout.
- key the KEK write cache by (access token, master key) like parquet-cpp's per-token two-level cache, and expire entries after a configurable lifetime (default 10m, cache_lifetime_seconds) - lock per cache entry instead of across the whole factory, so KMS wraps under different master keys run concurrently - add DataKeyLengthBits (128/192/256, default 128) and Algorithm (AesGcm/AesCtr) to EncryptionConfiguration - require an explicit UniformEncryption flag; an empty ColumnKeys map is now a configuration error, as in parquet-cpp
The instance stays in the cache key alongside the access token: the same master key id can resolve to different keys on different KMS deployments, so wraps must not be shared across them. Also validate the cipher value, relax the per-entry lock timing bound for CI noise, and read a CTR + 256-bit file back through pyarrow in the interop test.
|
Pushed:
Description updated too, and there's a pyarrow interop case reading back a CTR file with 256-bit keys. |
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. The cache key, expiry, per-entry lock, key length and algorithm options, and the UniformEncryption check all look good, and go test -race ./parquet/encryption passes on this head.
One new problem from the refresh logic needs fixing before this goes in: refreshing an expired KEK races with writers that are still reading the entry (inline). There are also a few smaller points inline.
|
|
||
| // double wrapping: a key encryption key wraps data keys locally and only | ||
| // it is wrapped by the KMS | ||
| kek, err := f.kekFor(client, kmsCfg, masterKeyID) |
There was a problem hiding this comment.
kekFor returns the shared *cachedKek after releasing entry.mu, and the lines below read kek.kek, kek.id and kek.wrapped without the lock. When the entry expires, another writer rewrites those fields under the lock (line 363). With 8 goroutines calling FileEncryptionProperties and a 1ns lifetime, -race reports it between these reads and that write.
This can also produce files that can't be decrypted: a data key can be wrapped with one KEK while the key material records another KEK's id or wrapped value. Please have kekFor return a copy taken while holding entry.mu, or replace the map entry with a new, unchanging *cachedKek on refresh instead of updating it in place. A test that refreshes an entry from several goroutines at once would catch this.
| if f.keks == nil { | ||
| f.keks = make(map[kekReuseKey]*cachedKek) | ||
| } | ||
| f.keks[key] = entry |
There was a problem hiding this comment.
Expired entries are re-wrapped but never removed, so the map gains one entry per (instance, token, master key) and keeps it forever. With per-user or rotating access tokens, a long-running writer holds every old plaintext KEK in memory. parquet-cpp's two-level cache removes expired tokens (CheckCacheForExpiredTokens). Please do the same here, for example by removing expired entries when they're accessed or sweeping them when a new token is added.
| DataKeyLengthBits int | ||
|
|
||
| // Algorithm picks the Parquet cipher: parquet.AesGcm (the zero value | ||
| // and the parquet-cpp default) or parquet.AesGcmCtr. |
There was a problem hiding this comment.
The constant is parquet.AesCtr (as in the check further down), not parquet.AesGcmCtr.
| entry.mu.Lock() | ||
| defer entry.mu.Unlock() | ||
| lifetime := f.kekLifetime | ||
| if lifetime <= 0 { |
There was a problem hiding this comment.
A zero or negative lifetime falls back to the default, so the cache can't be turned off this way. That's fine, but please say so in the WithKekCacheLifetime doc so nobody passes 0 expecting no caching.
…che entries kekFor returned the shared *cachedKek after releasing its lock, and wrapDataKey then read kek/id/wrapped unlocked while a concurrent refresh rewrote those fields in place. Under -race that is reported between the refresh write and the reader; worse, a torn entry can wrap a data key with one KEK while the key material records another KEK id or wrapped value, producing files that cannot be decrypted. Refreshes now publish a brand-new immutable cachedKek through a per-key slot, so entries handed to writers never change underneath them, and a concurrent-refresh test covers the window with a 1ns lifetime. Expired entries are also dropped from the map when a new entry joins it (parquet-cpp CheckCacheForExpiredTokens), so rotating access tokens no longer pin every stale plaintext KEK in memory. Also document that a zero or negative WithKekCacheLifetime falls back to the default rather than disabling caching, and fix the Algorithm doc to name parquet.AesCtr.
|
Confirmed with your repro (8 goroutines, 1ns lifetime) — kekFor was handing out an entry that a concurrent refresh rewrote in place. All four points are addressed in fa5c175:
|
Rationale for this change
Follow-up to the promise in #1478: "The write side (Go producing PKMT1 key material via a CryptoFactory) is a natural follow-up; this PR is read-only." With the read side merged, writing KMS-wrapped files still means hand-generating DEKs and hand-serializing PKMT1 key material; this adds the factory so arrow-go produces files that pyarrow's (and our own) CryptoFactory can read back (#566 tracks the umbrella discussion).
What changes are included in this PR?
The write side of
parquet/encryption:NewCryptoFactory(clientFactory KmsClientFactory)returns a factory that builds aKmsClientper file from aKmsConnectionConfig(the named type mirrors parquet-cpp's)EncryptionConfigurationselects the footer master key, per-master-key column keys (each listed column gets a fresh DEK; unlisted columns stay in plaintext; empty map means uniform encryption with the footer key), single vs double wrapping (double is the parquet-cpp/pyarrow default), AAD prefix handling, and plaintext footer. Invalid configurations are rejected, including a column listed under two master keys andDisableAadPrefixStoragewithout anAadPrefixFileEncryptionPropertiesgenerates 16-byte DEKs, wraps them (single: straight through the KMS; double: a per-(KMS instance, key access token, master key) KEK wraps DEKs locally with AES-GCM,nonce || ciphertext || tag, AAD = KEK ID bytes, the mirror ofdecryptKeyLocally, and only the KEK goes through the KMS), serializes the PKMT1 material, and returnsparquet.FileEncryptionPropertiesforparquet.WithEncryptionPropertiesKeyMaterialmarshalskmsInstanceID/kmsInstanceURLand the KEK fields withomitempty, so Go-written material matches parquet-cpp's structure exactly: column keys carry no KMS instance fields, single-wrapped keys no KEK fields, field order preservedNo writer changes: the existing footer and column key metadata plumbing carries the material.
The interop harness gains a
read-kmsmode (pyarrow's CryptoFactory resolves keys), sharing the test KMS client withgen-kms.Are these changes tested?
Unit tests cover material shape per mode, retriever round-trips, KEK sharing and freshness, per-column DEKs, uniform encryption, and the configuration errors, including KMS wrap-call counts (one per master key in double wrapping, one per key slot in single; the reuse assertion fails with the cache disabled). A Go write → Go read round trip resolves every key through
KmsKeyRetrieveralone and asserts the unwrap counts (2 / 3 for two master keys and three key slots). Against pyarrow 26.0.1 locally, with-race,PARQUET_TEST_REQUIRE_PYARROW=1, on both Go 1.26 and 1.27: Go-written files (double, single, plaintext footer) read back through pyarrow's CryptoFactory, the read-side interop tests are unchanged and still pass, and a byte-level dump of Go-written key material matches pyarrow's field for field.Are there any user-facing changes?
New exported API in the optional
parquet/encryptionpackage (CryptoFactory,KmsClientFactory,KmsConnectionConfig,EncryptionConfiguration). No changes to existing APIs.