Skip to content

Implement Spark-compatible aes_decrypt #296

Open

Description

@andygrove

Part of #272 and apache/datafusion-comet#7300.

What the function does

aes_decrypt(expr, key[, mode[, padding[, aad]]]) decrypts a value produced by aes_encrypt and returns BINARY.
expr, key and aad are BINARY; mode and padding are STRING (collated strings accepted from 4.0).
ImplicitCastInputTypes casts string arguments for expr, key and aad to their UTF-8 bytes. Defaults:
mode = 'GCM', padding = 'DEFAULT', aad = ''. Since 3.3.0 (aad since 3.5.0). NULL in any argument gives NULL.
There is no iv argument: the IV is read from the input.

Supported (mode, padding) pairs (case-insensitive): ECB + PKCS/DEFAULT, CBC + PKCS/DEFAULT,
GCM + NONE/DEFAULT. The key must be 16, 24 or 32 bytes. Expected input layout: ECB is the ciphertext only;
CBC is iv (16) || ciphertext; GCM is iv (12) || ciphertext || tag (16), with the 128-bit tag verified.
The result is the raw plaintext bytes. Use CAST(... AS STRING) to get text (the ExpressionDescription shows the
bytes as text).

Examples

From the ExpressionDescription, DataFrameFunctionsSuite and MiscFunctionsSuite:

SELECT cast(aes_decrypt(unhex('83F16B2AA704794132802D248E6BFD4E380078182D1544813898AC97E709B28A94'), '0000111122223333') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unhex('6E7CA17BBB468D3084B5744BCA729FB7B2B7BCB8E4472847D02670489D95FA97DBBA7D3210'), '0000111122223333', 'GCM') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('3lmwu+Mw0H3fi5NDvcu9lg=='), '1234567890abcdef', 'ECB', 'PKCS') AS STRING);
-- Spark SQL
SELECT cast(aes_decrypt(unbase64('2NYmDCjgXTbbxGA3/SnJEfFC/JQ7olk2VQWReIAAFKo='), '1234567890abcdef', 'CBC') AS STRING);
-- Apache Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAAAAAAPSd4mWyMZ5mhvjiAPQJnfg='), 'abcdefghijklmnop12345678ABCDEFGH', 'CBC', 'DEFAULT') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('AAAAAAAAAAAAAAAAQiYi+sTLm7KD9UcZ2nlRdYDe/PX4'), 'abcdefghijklmnop12345678ABCDEFGH', 'GCM', 'DEFAULT', 'This is an AAD mixed into the input') AS STRING);
-- Spark
SELECT cast(aes_decrypt(unbase64('jmTOhz8XTbskI/zYFFgOFQ=='), 'abcdefghijklmnop', 'ECB') AS STRING);
-- (empty string)
SELECT cast(aes_decrypt(unbase64('9J3iZbIxnmaG+OIA9Amd+A=='), binary('abcdefghijklmnop12345678ABCDEFGH'), 'ECB') AS STRING);
-- Spark
SELECT aes_decrypt(cast(null as binary), 'abcdefghijklmnop');
-- NULL

Edge cases to test

  • NULL in each argument gives NULL. String vs binary key give the same result.
  • Key length not 16/24/32 (0, 15, 17): INVALID_PARAMETER_VALUE.AES_KEY_LENGTH (sqlState 22023), parameters
    `key`, `aes_encrypt`/`aes_decrypt`, actualLength.
  • Unsupported pairs ('GSM' with the default padding reports padding = DEFAULT; 'GCM','PKCS'; 'ECB','None';
    'CBC','NoPadding'): UNSUPPORTED_FEATURE.AES_MODE (sqlState 0A000).
  • Wrong key for ECB: INVALID_PARAMETER_VALUE.AES_CRYPTO_ERROR, parameter `expr`, `key`, detailMessage
    "Given final block not properly padded. Such issues can arise if a bad key is used during decryption."
  • Corrupted GCM ciphertext or wrong AAD: AES_CRYPTO_ERROR with "Tag mismatch" (JDK may append !).
  • Truncated CBC input: AES_CRYPTO_ERROR with a message starting "Input length". Empty GCM input:
    AES_CRYPTO_ERROR with "Invalid buffer arguments" (input shorter than the IV). The JCE messages are not
    reproducible in Rust; match the error condition and parameters, not detailMessage.
  • AAD with CBC: UNSUPPORTED_FEATURE.AES_MODE_AAD, and the message names `aes_encrypt` even when decrypting.
    AAD with ECB is silently ignored on decrypt (the ECB branch never looks at aad, by code reading).
  • Round trips through aes_encrypt for all modes and key sizes, empty plaintext, plaintext longer than one block,
    and multi-byte UTF-8 (results are bytes, not validated as UTF-8).
  • Literal and column for each argument, including per-row mode / key.
  • Deterministic.

Differences across Spark versions

  • 3.4: only ECB and GCM, no aad. A too-short GCM input throws an unmapped IllegalArgumentException, because
    only GeneralSecurityException maps to AES_CRYPTO_ERROR.
  • 3.5: adds CBC, aad, UNSUPPORTED_FEATURE.AES_MODE_AAD, and maps IllegalArgumentException too.
  • 4.0: collated mode / padding accepted. No behavior change through 4.2.

Implementation notes

  • Spark: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala (AesDecrypt,
    RuntimeReplaceable over StaticInvoke) and
    sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java (aesDecrypt,
    aesInternal). Errors in sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala.
  • DataFusion core has no AES functions.
  • Suggested crates: RustCrypto aes, aes-gcm, cbc, ecb + block-padding (PKCS7).
  • Put it in crates/spark/src/function/misc/ with a helper module shared with aes_encrypt and try_aes_decrypt.
    The helper should return a typed error so try_aes_decrypt can turn every failure into NULL.
  • Tests: add crates/sqllogictest/test_files/misc/aes_decrypt.slt; document with #[user_doc].

This issue was generated by an LLM (Claude) from the Apache Spark source code and tests. Examples come from Spark's ExpressionDescription examples, test suites and .sql.out golden files, but please verify them against Spark before relying on them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions