Skip to content

feat(parquet/encryption): KMS support for writing PKMT1 key material - #1567

Open
EdwardShen125 wants to merge 4 commits into
apache:mainfrom
EdwardShen125:parquet-kms-write
Open

EdwardShen125 wants to merge 4 commits into
apache:mainfrom
EdwardShen125:parquet-kms-write

Conversation

@EdwardShen125

@EdwardShen125 EdwardShen125 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to the promise in #1478: "The write side (Go producing PKMT1 key material via a CryptoFactory) is a natural follow-up; this PR is read-only." With the read side merged, writing KMS-wrapped files still means hand-generating DEKs and hand-serializing PKMT1 key material; this adds the factory so arrow-go produces files that pyarrow's (and our own) CryptoFactory can read back (#566 tracks the umbrella discussion).

What changes are included in this PR?

The write side of parquet/encryption:

  • NewCryptoFactory(clientFactory KmsClientFactory) returns a factory that builds a KmsClient per file from a KmsConnectionConfig (the named type mirrors parquet-cpp's)
  • EncryptionConfiguration selects the footer master key, per-master-key column keys (each listed column gets a fresh DEK; unlisted columns stay in plaintext; empty map means uniform encryption with the footer key), single vs double wrapping (double is the parquet-cpp/pyarrow default), AAD prefix handling, and plaintext footer. Invalid configurations are rejected, including a column listed under two master keys and DisableAadPrefixStorage without an AadPrefix
  • FileEncryptionProperties generates 16-byte DEKs, wraps them (single: straight through the KMS; double: a per-(KMS instance, key access token, master key) KEK wraps DEKs locally with AES-GCM, nonce || ciphertext || tag, AAD = KEK ID bytes, the mirror of decryptKeyLocally, and only the KEK goes through the KMS), serializes the PKMT1 material, and returns parquet.FileEncryptionProperties for parquet.WithEncryptionProperties
  • in double wrapping mode the factory caches KEKs across files, keyed by (KMS instance, key access token, master key) — parquet-cpp's per-token write cache, additionally scoped to the instance — and expires entries after a configurable lifetime (default 10m, parquet-cpp's cache_lifetime_seconds), so writing many files costs one KMS wrap per master key per token; wrapping locks per cache entry, so files under different master keys wrap concurrently
  • KeyMaterial marshals kmsInstanceID/kmsInstanceURL and the KEK fields with omitempty, so Go-written material matches parquet-cpp's structure exactly: column keys carry no KMS instance fields, single-wrapped keys no KEK fields, field order preserved

No writer changes: the existing footer and column key metadata plumbing carries the material.

The interop harness gains a read-kms mode (pyarrow's CryptoFactory resolves keys), sharing the test KMS client with gen-kms.

Are these changes tested?

Unit tests cover material shape per mode, retriever round-trips, KEK sharing and freshness, per-column DEKs, uniform encryption, and the configuration errors, including KMS wrap-call counts (one per master key in double wrapping, one per key slot in single; the reuse assertion fails with the cache disabled). A Go write → Go read round trip resolves every key through KmsKeyRetriever alone and asserts the unwrap counts (2 / 3 for two master keys and three key slots). Against pyarrow 26.0.1 locally, with -race, PARQUET_TEST_REQUIRE_PYARROW=1, on both Go 1.26 and 1.27: Go-written files (double, single, plaintext footer) read back through pyarrow's CryptoFactory, the read-side interop tests are unchanged and still pass, and a byte-level dump of Go-written key material matches pyarrow's field for field.

Are there any user-facing changes?

New exported API in the optional parquet/encryption package (CryptoFactory, KmsClientFactory, KmsConnectionConfig, EncryptionConfiguration). No changes to existing APIs.

CryptoFactory builds parquet.FileEncryptionProperties whose keys are
wrapped by a KmsClient, closing the loop with the read side of apache#1478:
fresh 16-byte DEKs are generated for the footer and every configured
column, wrapped single (straight through the KMS) or double (a cached
per-master-key KEK wraps DEKs locally with AES-GCM, nonce ||
ciphertext || tag, AAD = KEK ID bytes, and only the KEK goes through
the KMS), and serialized as PKMT1 key material in the footer and
column key metadata.

The material matches parquet-cpp byte for byte: field order, omitted
kmsInstance fields on column keys (KeyMaterial now marshals them, and
the single-wrapping KEK fields, with omitempty), a fresh DEK per
column, one KEK per master key reused across files, and KMS call
counts symmetric with the read side (2 unwraps for two master keys in
double wrapping mode, 1 per key slot in single wrapping mode).

Interop: the harness gains a read-kms mode, and Go-written files read
back through pyarrow's CryptoFactory in double, single, and plaintext
footer configurations; a Go write/read round trip resolves every key
through KmsKeyRetriever alone.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. The files interoperate: the three interop tests pass against pyarrow 26.0.1, and pyarrow's CryptoFactory also read uniform (double and single wrapping), uniform with an AAD prefix, and one-encrypted-column files I wrote with this branch. The read-side tests still pass.

Four things to change before this goes in, inline. The first is the important one: the KEK cache is described as matching parquet-cpp, and it differs from it in two ways that matter.

Comment thread parquet/encryption/crypto_factory.go Outdated
// KMS call so concurrent files sharing a master key get one wrap.
func (f *CryptoFactory) kekFor(client KmsClient, kmsCfg KmsConnectionConfig,
masterKeyID string) (cachedKek, error) {
key := kekReuseKey{kmsInstanceID: kmsCfg.KmsInstanceID, masterKeyID: masterKeyID}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache differs from parquet-cpp in two ways (FileKeyWrapper in file_key_wrapper.cc), and the description's "held for the factory's lifetime, like parquet-cpp" isn't right.

  1. parquet-cpp keys the KEK cache by access token (kek_write_cache_per_token().GetOrCreateInternalCache(kms_connection_config.key_access_token(), ...)). Here the key is the KMS instance ID and the master key only. With a KMS client that refuses one token: that token gets access denied on first use, but once another token has cached the KEK for the same master key, the refused token succeeds without the KMS being asked. With single wrapping the same token still fails, as it should.

  2. parquet-cpp expires entries after cache_lifetime_seconds, 600 by default. Here a KEK is never dropped, so a long-running writer wraps every file's keys with the same KEK indefinitely.

Please key the cache by access token as well, and give entries a lifetime: configurable, defaulting to 10 minutes. The doc comment on CryptoFactory and the PR description need the same correction.

func (f *CryptoFactory) kekFor(client KmsClient, kmsCfg KmsConnectionConfig,
masterKeyID string) (cachedKek, error) {
key := kekReuseKey{kmsInstanceID: kmsCfg.KmsInstanceID, masterKeyID: masterKeyID}
f.mu.Lock()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lock covers every master key, and it is held across the KMS call. Three files with three different master keys, written concurrently against a KMS client whose wrap takes 300ms, took 900ms. Please hold it per cache entry, so that only writers waiting for the same KEK wait for one another.

Comment thread parquet/encryption/crypto_factory.go Outdated
// Key sizes for generated keys, matching the parquet-cpp/pyarrow
// CryptoFactory (AES-128 data and key encryption keys).
const (
dataKeySize = 16

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parquet-cpp's EncryptionConfiguration also carries encryption_algorithm and data_key_length_bits (128, 192 or 256). Please add both to EncryptionConfiguration, with these values as the defaults, so that a file written here can match what a parquet-cpp or pyarrow writer is configured to produce.

// key. Columns not listed stay in plaintext. When empty (or nil) the
// configuration is uniform: all columns are encrypted with the footer
// key.
ColumnKeys map[string][]string

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parquet-cpp requires an explicit uniform_encryption flag: it rejects a configuration with neither it nor column keys ("Either column_keys or uniform_encryption must be set"), and one with both. Please do the same here, with a UniformEncryption field, instead of treating an empty map as uniform. A caller who builds the map from input that turns out empty should get an error, not a different encryption layout.

- key the KEK write cache by (access token, master key) like
  parquet-cpp's per-token two-level cache, and expire entries after
  a configurable lifetime (default 10m, cache_lifetime_seconds)
- lock per cache entry instead of across the whole factory, so KMS
  wraps under different master keys run concurrently
- add DataKeyLengthBits (128/192/256, default 128) and Algorithm
  (AesGcm/AesCtr) to EncryptionConfiguration
- require an explicit UniformEncryption flag; an empty ColumnKeys map
  is now a configuration error, as in parquet-cpp
The instance stays in the cache key alongside the access token: the
same master key id can resolve to different keys on different KMS
deployments, so wraps must not be shared across them. Also validate
the cipher value, relax the per-entry lock timing bound for CI noise,
and read a CTR + 256-bit file back through pyarrow in the interop
test.
@EdwardShen125

EdwardShen125 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Pushed:

  • cache key is (instance, token, master key) now, entries expire via WithKekCacheLifetime (default 10m)
  • the wrap holds only the entry lock, not the map
  • EncryptionConfiguration grew DataKeyLengthBits and Algorithm
  • empty column keys is an error unless UniformEncryption is set

Description updated too, and there's a pyarrow interop case reading back a CTR file with 256-bit keys.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround. The cache key, expiry, per-entry lock, key length and algorithm options, and the UniformEncryption check all look good, and go test -race ./parquet/encryption passes on this head.

One new problem from the refresh logic needs fixing before this goes in: refreshing an expired KEK races with writers that are still reading the entry (inline). There are also a few smaller points inline.


// double wrapping: a key encryption key wraps data keys locally and only
// it is wrapped by the KMS
kek, err := f.kekFor(client, kmsCfg, masterKeyID)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kekFor returns the shared *cachedKek after releasing entry.mu, and the lines below read kek.kek, kek.id and kek.wrapped without the lock. When the entry expires, another writer rewrites those fields under the lock (line 363). With 8 goroutines calling FileEncryptionProperties and a 1ns lifetime, -race reports it between these reads and that write.

This can also produce files that can't be decrypted: a data key can be wrapped with one KEK while the key material records another KEK's id or wrapped value. Please have kekFor return a copy taken while holding entry.mu, or replace the map entry with a new, unchanging *cachedKek on refresh instead of updating it in place. A test that refreshes an entry from several goroutines at once would catch this.

Comment thread parquet/encryption/crypto_factory.go Outdated
if f.keks == nil {
f.keks = make(map[kekReuseKey]*cachedKek)
}
f.keks[key] = entry

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expired entries are re-wrapped but never removed, so the map gains one entry per (instance, token, master key) and keeps it forever. With per-user or rotating access tokens, a long-running writer holds every old plaintext KEK in memory. parquet-cpp's two-level cache removes expired tokens (CheckCacheForExpiredTokens). Please do the same here, for example by removing expired entries when they're accessed or sweeping them when a new token is added.

Comment thread parquet/encryption/crypto_factory.go Outdated
DataKeyLengthBits int

// Algorithm picks the Parquet cipher: parquet.AesGcm (the zero value
// and the parquet-cpp default) or parquet.AesGcmCtr.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constant is parquet.AesCtr (as in the check further down), not parquet.AesGcmCtr.

Comment thread parquet/encryption/crypto_factory.go Outdated
entry.mu.Lock()
defer entry.mu.Unlock()
lifetime := f.kekLifetime
if lifetime <= 0 {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A zero or negative lifetime falls back to the default, so the cache can't be turned off this way. That's fine, but please say so in the WithKekCacheLifetime doc so nobody passes 0 expecting no caching.

…che entries

kekFor returned the shared *cachedKek after releasing its lock, and
wrapDataKey then read kek/id/wrapped unlocked while a concurrent refresh
rewrote those fields in place. Under -race that is reported between the
refresh write and the reader; worse, a torn entry can wrap a data key
with one KEK while the key material records another KEK id or wrapped
value, producing files that cannot be decrypted.

Refreshes now publish a brand-new immutable cachedKek through a per-key
slot, so entries handed to writers never change underneath them, and a
concurrent-refresh test covers the window with a 1ns lifetime.

Expired entries are also dropped from the map when a new entry joins it
(parquet-cpp CheckCacheForExpiredTokens), so rotating access tokens no
longer pin every stale plaintext KEK in memory.

Also document that a zero or negative WithKekCacheLifetime falls back to
the default rather than disabling caching, and fix the Algorithm doc to
name parquet.AesCtr.
@EdwardShen125

Copy link
Copy Markdown
Contributor Author

Confirmed with your repro (8 goroutines, 1ns lifetime) — kekFor was handing out an entry that a concurrent refresh rewrote in place. All four points are addressed in fa5c175:

  • kekFor replaces the map entry with a new, immutable cachedKek on refresh; TestKekRefreshFromConcurrentWriters drives that same repro and additionally checks every generated material still unwraps.
  • Expired entries are swept when a new key joins the map (CheckCacheForExpiredTokens); an internal test asserts it.
  • WithKekCacheLifetime now documents the zero/negative fallback; the Algorithm doc says parquet.AesCtr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants