Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All significant changes to this project will be documented in this file.

### Features

* Add `decoded_len` for exact buffer sizing and `check` for validation without decoded output to all profiles and the `Codec` trait. `decoded_len` checks length and trailing padding shape without fully validating the input; `check` preserves decoding errors and byte offsets.
* Add `policy::PaddedOptional` and `policy::UnpaddedOptional` for Base32 and Base64. Both accept complete padding or none when decoding and emit padded and unpadded output, respectively.
* Add `encode_into_uninit` and `decode_into_uninit` to all profiles and the `Codec` trait. They write into caller-provided `MaybeUninit<u8>` storage and return the initialized prefix.

Expand Down
10 changes: 9 additions & 1 deletion bases/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ For example, use `base64::Base64<base64::Standard, policy::UnpaddedOptional>` fo

## API layers and features

- **Caller buffers (`no_std`):** `encoded_len`, `max_decoded_len`, `encode_into`, `decode_into`, `decode_in_place`, and `decode_array` operate without allocation. Concrete profiles provide `const fn` constructors and sizing methods.
- **Sizing, validation, and caller buffers (`no_std`):** `encoded_len`, `max_decoded_len`, `decoded_len`, `check`, `encode_into`, `decode_into`, `decode_in_place`, and `decode_array` operate without allocation. Concrete profiles provide `const fn` constructors, `encoded_len`, and `max_decoded_len`.
- **Owned output (`alloc`):** `encode` and `decode` return `String` and `Vec<u8>`; `encode_append` and transactional `decode_append` extend existing buffers.
- **Streaming (`no_std` / `std`):** `Encoder<C>` and `Decoder<C>` process arbitrarily chunked input while retaining at most one partial quantum. The `std` feature adds `io::EncoderWriter` and `io::DecoderReader` for `std::io::Write` and `std::io::Read`.
- **Formatting (`no_std`):** `display` (`EncodedDisplay`) and `HexDisplay` (`{:x}` and `{:X}` only) write encoded text through `core::fmt` without an intermediate string.
Expand All @@ -65,6 +65,14 @@ For example, use `base64::Base64<base64::Standard, policy::UnpaddedOptional>` fo
| `std` | Enables `alloc`, I/O adapters, and runtime x86/x86_64 CPU feature detection. |
| `serde` | Adds `serde_core` serialization and fixed-array deserialization, including without an allocator. |

## Decoding buffer sizes and validation

Use `max_decoded_len(input.len())` for an upper bound based only on the encoded length, or `decoded_len(input)` to inspect the input's length and trailing padding shape for exact sizing. For example, `Base64::max_decoded_len(4)` returns `3`, while `Base64::decoded_len("Zg==")` returns `Ok(1)`.

A successful `decoded_len` result guarantees the exact length only if decoding succeeds. It does not fully validate the input: invalid alphabet symbols, misplaced padding, and non-zero unused bits can go undetected. Decoding still performs full validation.

Use `check(input)` when only validation is needed, for example before storing or forwarding an encoded field. It applies the selected profile's complete decoding rules without allocating or writing decoded bytes, and returns the same errors and byte offsets as decoding with sufficient output capacity. `Base64::check("Zg==")` succeeds, while `Base64::check("Zh==")` reports `DecodeError::NonCanonical { index: 1 }`. Calling `check` immediately before decoding normally duplicates validation work.

## Uninitialized output buffers

Use `encode_into_uninit` or `decode_into_uninit` when the destination is an uninitialized stack buffer or a vector's spare capacity. Use the returned slice to access the result. For padded Base64, it can be shorter than the decoding size estimate.
Expand Down
44 changes: 33 additions & 11 deletions bases/src/backend/base16.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,27 @@ pub fn decode_plan(input: &[u8]) -> Result<DecodePlan, DecodeError> {
})
}

/// Checks alphabet symbols after input length has been validated.
#[inline]
pub fn check_with_plan<const UPPERCASE: bool, const STRICT_CASE: bool>(
input: &[u8],
plan: DecodePlan,
) -> Result<(), DecodeError> {
debug_assert_eq!(plan.data_len, input.len());
let lut = select_decode_lut::<UPPERCASE, STRICT_CASE>();
if input.len() < 32 {
for (index, &byte) in input.iter().enumerate() {
decode_value(byte, index, lut)?;
}
} else if let Some(index) = backend::find_invalid(input, lut) {
return Err(DecodeError::InvalidByte {
index,
byte: input[index],
});
}
Ok(())
}

/// Encodes `input` into `output` using Base16, returning the number of bytes written.
#[inline(always)]
pub fn encode<const UPPERCASE: bool>(input: &[u8], output: &mut [MaybeUninit<u8>]) -> usize {
Expand Down Expand Up @@ -206,20 +227,21 @@ fn decode_chunk_8(lut: &[u8; 256], chunk: &[u8; 8]) -> Option<[u8; 4]> {

#[inline]
fn decode_pair(lut: &[u8; 256], b0: u8, b1: u8, index: usize) -> Result<u8, DecodeError> {
let d0 = lut[usize::from(b0)];
if d0 & 0xf0 != 0 {
return Err(DecodeError::InvalidByte { index, byte: b0 });
}
let d1 = lut[usize::from(b1)];
if d1 & 0xf0 != 0 {
return Err(DecodeError::InvalidByte {
index: index + 1,
byte: b1,
});
}
let d0 = decode_value(b0, index, lut)?;
let d1 = decode_value(b1, index + 1, lut)?;
Ok((d0 << 4) | d1)
}

#[inline]
fn decode_value(byte: u8, index: usize, lut: &[u8; 256]) -> Result<u8, DecodeError> {
let value = lut[usize::from(byte)];
if value & 0xf0 == 0 {
Ok(value)
} else {
Err(DecodeError::InvalidByte { index, byte })
}
}

#[inline]
fn decode_scalar_from<const UPPERCASE: bool, const STRICT_CASE: bool>(
input: &[u8],
Expand Down
19 changes: 3 additions & 16 deletions bases/src/backend/base16/arch/x86.rs
Original file line number Diff line number Diff line change
Expand Up @@ -228,12 +228,9 @@ pub fn decode_in_place<const UPPERCASE: bool, const STRICT_CASE: bool>(
(done_32 * 32, done_32 * 16)
}

/// # Safety
///
/// Requires SSSE3 support on the current CPU.
#[inline]
#[target_feature(enable = "ssse3")]
unsafe fn hex_table_128<const UPPERCASE: bool>() -> __m128i {
fn hex_table_128<const UPPERCASE: bool>() -> __m128i {
if UPPERCASE {
_mm_setr_epi8(
b'0' as i8, b'1' as i8, b'2' as i8, b'3' as i8, b'4' as i8, b'5' as i8, b'6' as i8,
Expand Down Expand Up @@ -351,14 +348,9 @@ unsafe fn encode_avx2<const UPPERCASE: bool>(
(input_index, output_index)
}

/// # Safety
///
/// Requires SSSE3 support on the current CPU.
#[inline]
#[target_feature(enable = "ssse3")]
unsafe fn decode_nibbles_128<const UPPERCASE: bool, const STRICT_CASE: bool>(
v: __m128i,
) -> __m128i {
fn decode_nibbles_128<const UPPERCASE: bool, const STRICT_CASE: bool>(v: __m128i) -> __m128i {
// Saturation maps digits to 0-9 and letters to 10-15; all other bytes stay above 15.
let digits = _mm_sub_epi8(
_mm_subs_epu8(
Expand Down Expand Up @@ -443,15 +435,10 @@ unsafe fn decode_ssse3<const UPPERCASE: bool, const STRICT_CASE: bool>(
done
}

/// # Safety
///
/// Requires AVX2 support on the current CPU.
#[cfg(any(feature = "std", target_feature = "avx2"))]
#[inline]
#[target_feature(enable = "avx2")]
unsafe fn decode_nibbles_256<const UPPERCASE: bool, const STRICT_CASE: bool>(
v: __m256i,
) -> __m256i {
fn decode_nibbles_256<const UPPERCASE: bool, const STRICT_CASE: bool>(v: __m256i) -> __m256i {
let digits = _mm256_sub_epi8(
_mm256_subs_epu8(
_mm256_add_epi8(v, _mm256_set1_epi8(!b'9' as i8)),
Expand Down
65 changes: 45 additions & 20 deletions bases/src/backend/base32.rs
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,31 @@ pub fn decode_plan<const PADDED: bool>(input: &[u8]) -> Result<DecodePlan, Decod
}
}

/// Checks alphabet symbols and unused bits using an existing decode plan.
#[inline]
pub fn check_with_plan<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CASE: bool>(
input: &[u8],
plan: DecodePlan,
) -> Result<(), DecodeError> {
let lut = select_decode_lut::<EXTENDED_HEX, UPPERCASE, STRICT_CASE>();
let last = if plan.data_len < 32 {
let mut last = 0;
for (index, &byte) in input[..plan.data_len].iter().enumerate() {
last = decode_value(byte, index, lut)?;
}
last
} else {
if let Some(index) = backend::find_invalid(&input[..plan.data_len], lut) {
return Err(backend::invalid_byte_error(input[index], index));
}
lut[usize::from(input[plan.data_len - 1])]
};
if plan.complete_len != plan.data_len {
check_tail_bits(last, plan.data_len - plan.complete_len, plan.data_len - 1)?;
}
Ok(())
}

/// Encodes `input` into `output` using Base32, returning the number of bytes written.
#[inline]
pub fn encode<const EXTENDED_HEX: bool, const PADDED: bool, const UPPERCASE: bool>(
Expand Down Expand Up @@ -606,11 +631,7 @@ fn decode_tail<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CAS
[b0, b1] => {
let v0 = decode_value(*b0, base_index, lut)?;
let v1 = decode_value(*b1, base_index + 1, lut)?;
if v1 & 0x03 != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 1,
});
}
check_tail_bits(v1, trailing.len(), base_index + 1)?;
output[0].write((v0 << 3) | (v1 >> 2));
Ok(1)
}
Expand All @@ -619,11 +640,7 @@ fn decode_tail<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CAS
let v1 = decode_value(*b1, base_index + 1, lut)?;
let v2 = decode_value(*b2, base_index + 2, lut)?;
let v3 = decode_value(*b3, base_index + 3, lut)?;
if v3 & 0x0f != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 3,
});
}
check_tail_bits(v3, trailing.len(), base_index + 3)?;
output[0].write((v0 << 3) | (v1 >> 2));
output[1].write((v1 << 6) | (v2 << 1) | (v3 >> 4));
Ok(2)
Expand All @@ -634,11 +651,7 @@ fn decode_tail<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CAS
let v2 = decode_value(*b2, base_index + 2, lut)?;
let v3 = decode_value(*b3, base_index + 3, lut)?;
let v4 = decode_value(*b4, base_index + 4, lut)?;
if v4 & 0x01 != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 4,
});
}
check_tail_bits(v4, trailing.len(), base_index + 4)?;
output[0].write((v0 << 3) | (v1 >> 2));
output[1].write((v1 << 6) | (v2 << 1) | (v3 >> 4));
output[2].write((v3 << 4) | (v4 >> 1));
Expand All @@ -652,11 +665,7 @@ fn decode_tail<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CAS
let v4 = decode_value(*b4, base_index + 4, lut)?;
let v5 = decode_value(*b5, base_index + 5, lut)?;
let v6 = decode_value(*b6, base_index + 6, lut)?;
if v6 & 0x07 != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 6,
});
}
check_tail_bits(v6, trailing.len(), base_index + 6)?;
output[0].write((v0 << 3) | (v1 >> 2));
output[1].write((v1 << 6) | (v2 << 1) | (v3 >> 4));
output[2].write((v3 << 4) | (v4 >> 1));
Expand All @@ -667,6 +676,22 @@ fn decode_tail<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CAS
}
}

#[inline]
fn check_tail_bits(value: u8, tail_len: usize, index: usize) -> Result<(), DecodeError> {
let mask = match tail_len {
2 => 0x03,
4 => 0x0f,
5 => 0x01,
7 => 0x07,
_ => unreachable!("decode plan guarantees a valid tail length"),
};
if value & mask == 0 {
Ok(())
} else {
Err(DecodeError::NonCanonical { index })
}
}

fn find_invalid_in_quantum(quantum: &[u8], base_index: usize, lut: &[u8; 256]) -> DecodeError {
for (offset, &byte) in quantum.iter().enumerate() {
if lut[usize::from(byte)] & 0xe0 != 0 {
Expand Down
15 changes: 3 additions & 12 deletions bases/src/backend/base32/arch/x86.rs
Original file line number Diff line number Diff line change
Expand Up @@ -298,12 +298,9 @@ pub fn decode_in_place<const EXTENDED_HEX: bool, const UPPERCASE: bool, const ST
(done_16 * 16, done_16 * 10)
}

/// # Safety
///
/// Requires SSSE3 support on the current CPU.
#[inline]
#[target_feature(enable = "ssse3")]
unsafe fn encode_128_consts<const EXTENDED_HEX: bool, const UPPERCASE: bool>() -> (
fn encode_128_consts<const EXTENDED_HEX: bool, const UPPERCASE: bool>() -> (
__m128i,
__m128i,
__m128i,
Expand Down Expand Up @@ -500,16 +497,10 @@ unsafe fn encode_avx2<const EXTENDED_HEX: bool, const UPPERCASE: bool>(
(input_index, output_index)
}

/// # Safety
///
/// Requires SSSE3 support on the current CPU.
#[inline]
#[target_feature(enable = "ssse3")]
unsafe fn decode_128_tables<
const EXTENDED_HEX: bool,
const UPPERCASE: bool,
const STRICT_CASE: bool,
>() -> (__m128i, __m128i, __m128i, __m128i) {
fn decode_128_tables<const EXTENDED_HEX: bool, const UPPERCASE: bool, const STRICT_CASE: bool>()
-> (__m128i, __m128i, __m128i, __m128i) {
let hi = select_hi_mask::<UPPERCASE, STRICT_CASE>();
let lo = if EXTENDED_HEX {
LO_MASK_HEX
Expand Down
71 changes: 61 additions & 10 deletions bases/src/backend/base64.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ const STANDARD: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwx
const URL_SAFE: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";

const INVALID_SYMBOL: u8 = 0xff;
const CHECK_SIMD_MIN: usize = 16;
const STD_DECODE_LUTS: [[i32; 256]; 4] = make_decode_luts(false);
const URL_DECODE_LUTS: [[i32; 256]; 4] = make_decode_luts(true);
const STD_DECODE_BYTE_LUT: [u8; 256] = make_decode_byte_lut(false);
Expand Down Expand Up @@ -149,6 +150,50 @@ pub fn decode_plan<const PADDED: bool>(input: &[u8]) -> Result<DecodePlan, Decod
}
}

/// Checks alphabet symbols and unused bits using an existing decode plan.
#[inline]
pub fn check_with_plan<const URL_SAFE_ALPHABET: bool>(
input: &[u8],
plan: DecodePlan,
) -> Result<(), DecodeError> {
let lut = if URL_SAFE_ALPHABET {
&URL_DECODE_BYTE_LUT
} else {
&STD_DECODE_BYTE_LUT
};
if plan.complete_len < CHECK_SIMD_MIN {
let mut last = 0;
for (index, &byte) in input[..plan.data_len].iter().enumerate() {
last = decode_value(byte, index, lut)?;
}
if plan.complete_len != plan.data_len {
check_tail_bits(last, plan.data_len - plan.complete_len, plan.data_len - 1)?;
}
return Ok(());
}
decode_value(input[0], 0, lut)?;
let checked = arch::check::<URL_SAFE_ALPHABET>(&input[..plan.complete_len])
.map_err(|index| backend::invalid_byte_error(input[index], index))?;
let remaining = &input[checked..plan.data_len];
let last = if remaining.len() < 32 {
let mut last = 0;
for (index, &byte) in remaining.iter().enumerate() {
last = decode_value(byte, checked + index, lut)?;
}
last
} else {
if let Some(index) = backend::find_invalid(remaining, lut) {
let index = checked + index;
return Err(backend::invalid_byte_error(input[index], index));
}
lut[usize::from(input[plan.data_len - 1])]
};
if plan.complete_len != plan.data_len {
check_tail_bits(last, plan.data_len - plan.complete_len, plan.data_len - 1)?;
}
Ok(())
}

/// Encodes `input` into `output` using Base64, returning the number of bytes written.
#[inline]
pub fn encode<const URL_SAFE_ALPHABET: bool, const PADDED: bool>(
Expand Down Expand Up @@ -611,23 +656,15 @@ fn decode_tail<const URL_SAFE_ALPHABET: bool>(
[first, second] => {
let v0 = decode_value(*first, base_index, lut)?;
let v1 = decode_value(*second, base_index + 1, lut)?;
if v1 & 0x0f != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 1,
});
}
check_tail_bits(v1, trailing.len(), base_index + 1)?;
output[0].write((v0 << 2) | (v1 >> 4));
Ok(1)
}
[first, second, third] => {
let v0 = decode_value(*first, base_index, lut)?;
let v1 = decode_value(*second, base_index + 1, lut)?;
let v2 = decode_value(*third, base_index + 2, lut)?;
if v2 & 0x03 != 0 {
return Err(DecodeError::NonCanonical {
index: base_index + 2,
});
}
check_tail_bits(v2, trailing.len(), base_index + 2)?;
output[0].write((v0 << 2) | (v1 >> 4));
output[1].write((v1 << 4) | (v2 >> 2));
Ok(2)
Expand All @@ -636,6 +673,20 @@ fn decode_tail<const URL_SAFE_ALPHABET: bool>(
}
}

#[inline]
fn check_tail_bits(value: u8, tail_len: usize, index: usize) -> Result<(), DecodeError> {
let mask = match tail_len {
2 => 0x0f,
3 => 0x03,
_ => unreachable!("decode plan guarantees a valid tail length"),
};
if value & mask == 0 {
Ok(())
} else {
Err(DecodeError::NonCanonical { index })
}
}

#[inline]
fn find_invalid_in_slice(slice: &[u8], base_index: usize, lut: &[u8; 256]) -> DecodeError {
for (offset, &byte) in slice.iter().enumerate() {
Expand Down
Loading
Loading