Skip to content

Implement Spark-compatible aes_encrypt #295

Open

Description

@andygrove

Part of #272 and apache/datafusion-comet#7300.

What the function does

aes_encrypt(expr, key[, mode[, padding[, iv[, aad]]]]) encrypts expr with AES and returns BINARY.
expr, key, iv and aad are BINARY; mode and padding are STRING (collated strings accepted from 4.0).
Through ImplicitCastInputTypes, string arguments for expr, key, iv and aad are cast to binary (their UTF-8
bytes). Defaults: mode = 'GCM', padding = 'DEFAULT', iv = '', aad = ''. Since 3.3.0 (iv and aad since
3.5.0). The result is NULL if any argument is NULL (StaticInvoke with null propagation).

Supported (mode, padding) pairs, both case-insensitive: ECB with PKCS/DEFAULT (AES/ECB/PKCS5Padding),
CBC with PKCS/DEFAULT (AES/CBC/PKCS5Padding), GCM with NONE/DEFAULT (AES/GCM/NoPadding, 128-bit tag).
The key must be 16, 24 or 32 bytes (AES-128/192/256). It is used as is, with no key derivation.

Output layout: ECB is the ciphertext only. CBC is iv (16 bytes) || ciphertext. GCM is
iv (12 bytes) || ciphertext || tag (16 bytes). If iv is empty, a random IV is drawn from SecureRandom, so CBC and
GCM output is nondeterministic. Only ECB, or CBC/GCM with an explicit IV, give fixed results.

Examples

Deterministic ones (from the ExpressionDescription, ExpressionImplUtilsSuite and DataFrameFunctionsSuite):

SELECT base64(aes_encrypt('Spark SQL', '1234567890abcdef', 'ECB', 'PKCS'));
-- 3lmwu+Mw0H3fi5NDvcu9lg==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop', 'ECB'));
-- 4Hv0UKCx6nfUeAoPZo1z+w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678', 'ECB'));
-- NeTYNgA+PCQBN50DA//O2w==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'ECB'));
-- 9J3iZbIxnmaG+OIA9Amd+A==
SELECT base64(aes_encrypt('', 'abcdefghijklmnop', 'ECB'));
-- jmTOhz8XTbskI/zYFFgOFQ==
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT', unhex('00000000000000000000000000000000')));
-- AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg=
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', unhex('000000000000000000000000')));
-- AAAAAAAAAAAAAAAAQiYi+sRNYDAOTjdSEcYBFsAWPL1f
SELECT base64(aes_encrypt('Spark', 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', unhex('000000000000000000000000'), 'This is an AAD mixed into the input'));
-- AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4
SELECT aes_encrypt(cast(null as string), 'abcdefghijklmnop');
-- NULL

Random-IV examples can only be checked by round trip, or by length:

SELECT cast(aes_decrypt(aes_encrypt('Spark', 'abcdefghijklmnop'), 'abcdefghijklmnop') AS STRING);
-- Spark
SELECT length(aes_encrypt('Spark', '0000111122223333'));        -- 33 (12 IV + 5 + 16 tag), by code reading
SELECT length(aes_encrypt('Apache Spark', '1234567890abcdef', 'CBC'));  -- 32 (16 IV + 16), by code reading

Edge cases to test

  • NULL in each of the six arguments gives NULL. String vs binary expr and key give the same bytes.
  • Key length other than 16/24/32 (0, 15, 17 bytes): INVALID_PARAMETER_VALUE.AES_KEY_LENGTH, sqlState 22023,
    "The value of parameter(s) key in aes_encrypt/aes_decrypt is invalid: expects a binary value with 16, 24 or
    32 bytes, but got 17 bytes." The key is checked before the mode.
  • Unsupported pair ('CBC','None', 'ECB','NoPadding', 'GCM','PKCS', 'GSM'): UNSUPPORTED_FEATURE.AES_MODE,
    sqlState 0A000, "The feature is not supported: AES-CBC with the padding None by the aes_encrypt/aes_decrypt
    function." The parameters echo the user's original casing.
  • Non-empty IV with ECB: UNSUPPORTED_FEATURE.AES_MODE_IV (aes_encrypt with AES-ECB does not support
    initialization vectors (IVs).).
  • Wrong IV length (5 bytes for CBC or GCM): INVALID_PARAMETER_VALUE.AES_IV_LENGTH with parameter `iv` and
    "supports 16-byte CBC IVs and 12-byte GCM IVs, but got 5 bytes for CBC."
  • Non-empty AAD with ECB or CBC: UNSUPPORTED_FEATURE.AES_MODE_AAD. Empty iv / aad ('') behave exactly like
    the defaults, so ECB with iv = '' and aad = '' is accepted.
  • Any JCE failure becomes INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR with parameter `expr`, `key` and the JCE
    message as detailMessage. A Rust implementation can't reproduce those messages exactly; match the condition.
  • Empty plaintext: ECB and CBC produce one full padding block, GCM produces IV + tag only (28 bytes).
  • Plaintext longer than one block (This message is longer than a single AES block and should work fine.) and
    multi-byte UTF-8 strings.
  • Literal and column for every argument, including per-row mode, padding, key and iv columns.
  • Determinism: Spark does not mark aes_encrypt nondeterministic. A call where every argument is a literal is
    constant-folded (StaticInvoke is foldable), so every row gets the same IV. With a column argument each row gets a
    fresh IV. In DataFusion the UDF should be Volatile. Tests must round-trip or pass a fixed IV.

Differences across Spark versions

  • 3.4: only ECB and GCM, signature (expr, key[, mode[, padding]]). Only GeneralSecurityException is mapped to
    AES_CRYPTO_ERROR.
  • 3.5: adds CBC (SPARK-43038), iv and aad, UNSUPPORTED_FEATURE.AES_MODE_IV, UNSUPPORTED_FEATURE.AES_MODE_AAD,
    INVALID_PARAMETER_VALUE.AES_IV_LENGTH. IllegalArgumentException from the JCE is also mapped to
    AES_CRYPTO_ERROR.
  • 4.0: mode / padding accept collated strings. No behavior change through 4.2.

Implementation notes

  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala (AesEncrypt, a
    RuntimeReplaceable over StaticInvoke) and
    sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java
    (aesEncrypt, aesInternal, CipherMode). Errors in QueryExecutionErrors.scala.
  • DataFusion core has no AES functions.
  • Suggested crates: RustCrypto aes, aes-gcm, cbc, ecb (with block-padding PKCS7, which matches JCE
    PKCS5Padding for 16-byte blocks). Random IVs from rand (already a dependency) with an OS RNG.
  • Put it in crates/spark/src/function/misc/ (Spark group misc_funcs), sharing a helper module with
    aes_decrypt and try_aes_decrypt for mode/padding parsing, key checks and error construction.
  • Tests: add crates/sqllogictest/test_files/misc/aes_encrypt.slt with the fixed-IV and ECB vectors above and
    round-trip tests through aes_decrypt; document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions