Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/serialization.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ user = User.from_binary(data)

### Options

#### Unknown Fields

When a message is parsed from binary data containing field numbers it doesn't recognize, the unknown fields are stored internally and re-emitted during serialization.
This means a message can pass through an intermediary that doesn't know about newer fields without losing data.

Expand All @@ -37,6 +39,31 @@ data = user.to_binary(write_unknown_fields=False)
user = User.from_binary(data, ignore_unknown_fields=True)
```

#### Allocation Limit

Because protobuf serialization can create very compact binary payloads, it is possible for the memory usage of a
parsed message to differ drastically from the input number of bytes. When parsing untrusted payloads,
such as in an external-facing API server, this can allow malicious users to send small messages that take
a large amount of memory or potentially crashing the server. This is most pronounced in schemas with
repeated fields of message type with a large number of fields.

You can mitigate this using the `allocation_limit` option in `from_binary`. When set, an estimate of the
memory usage of a message is maintained while it is parsed, and if it goes over the limit, the parse fails
immediately before processing the entire payload. The limit is based on the schema and content, not the environment.
Some fixed overhead of objects such as a string can change between Python versions, so the allocation budget should not be
considered a precise value. Such changes are generally relatively small and fixed so an allocation limit determined
for one environment should mostly work fine when i.e., updating Python.

It is recommended to set an allocation limit for applications parsing untrusted payloads based on your target
memory usage. You may need to experiment with values to see the effective memory utilization corresponding to
the limit.

```python
user = User.from_binary(
data, allocation_limit=8 * 1024 * 1024
) # Roughly cap memory usage of parsed message to 8MB
```

## JSON

```python
Expand Down
242 changes: 242 additions & 0 deletions packages/protobuf-py-ext/src/budget.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,242 @@
use pyo3::{Bound, PyResult, exceptions::PyValueError, types::PyString};

use crate::marshaler::MessageMarshaler;

// Approximate sizes of CPython heap allocations, measured with
// `sys.getsizeof` on 64-bit CPython 3.14. The budget guards against
// unbounded allocation from malicious payloads rather than providing exact
// accounting, so small inaccuracies across versions and builds are fine.

/// GC header allocated in front of every GC-tracked object.
const GC_HEAD_SIZE: usize = 16;
/// A float object.
const FLOAT_SIZE: usize = 24;
/// A 64-bit int object. Smaller ints are slightly smaller.
const INT_SIZE: usize = 36;
/// Header of a bytes object.
const BYTES_OVERHEAD: usize = 33;
/// An empty list.
const EMPTY_LIST_SIZE: usize = 56;
/// An empty dict.
const EMPTY_DICT_SIZE: usize = 64;
/// One appended list element: an 8-byte pointer slot.
const LIST_SLOT_SIZE: usize = 8;
/// One inserted dict entry: hash + key + value words plus growth slack.
const DICT_ENTRY_SIZE: usize = 40;
/// A `Oneof` wrapper object: `PyObject` header plus two object pointers.
const ONEOF_SIZE: usize = 32;

// Python strings have two structs, one for ASCII and one for others.
// They also always contain a trailing NUL, so the total overhead is
// the size of the header (40 or 56) + the size of the trailing NUL
// which depends on the width.
const STR_ASCII_OVERHEAD: usize = 41;
const STR_LATIN1_OVERHEAD: usize = 57;
const STR_UCS2_OVERHEAD: usize = 58;
const STR_UCS4_OVERHEAD: usize = 60;

/// The size of a str holding `chars` characters and its largest codepoint.
/// Python picks different representations based on the largest codepoint.
fn str_size_for(chars: usize, max_char: u32) -> usize {
match max_char {
0..0x80 => STR_ASCII_OVERHEAD + chars,
0x80..0x100 => STR_LATIN1_OVERHEAD + chars,
0x100..0x1_0000 => STR_UCS2_OVERHEAD + 2 * chars,
_ => STR_UCS4_OVERHEAD + 4 * chars,
}
}

/// The decoded size of a str.
fn str_size(utf8: &[u8]) -> usize {
let max_byte = utf8.iter().fold(0, |max, &byte| max.max(byte));
if max_byte < 0x80 {
return STR_ASCII_OVERHEAD + utf8.len();
}
let chars = utf8.iter().filter(|&&byte| byte & 0xC0 != 0x80).count();
match max_byte {
// U+0080 to U+00FF
0..=0xC3 => STR_LATIN1_OVERHEAD + chars,
// U+0100 to U+FFFF
0xC4..=0xEF => STR_UCS2_OVERHEAD + 2 * chars,
_ => STR_UCS4_OVERHEAD + 4 * chars,
}
}

/// The size of Python str.
#[cfg(not(any(Py_LIMITED_API, PyPy, GraalPy)))]
fn py_str_size(value: &Bound<'_, PyString>) -> PyResult<usize> {
use pyo3::types::{PyStringMethods as _, string::PyStringData};

// SAFETY - the data is only read, while the str is borrowed.
Ok(match unsafe { value.data()? } {
// An ASCII str has a smaller header than other single byte ones.
PyStringData::Ucs1(data) => {
let max_char = data.iter().fold(0, |max, &ch| max.max(ch));
str_size_for(data.len(), max_char.into())
}
PyStringData::Ucs2(data) => STR_UCS2_OVERHEAD + 2 * data.len(),
PyStringData::Ucs4(data) => STR_UCS4_OVERHEAD + 4 * data.len(),
})
}

/// The size of Python str. The limited API cannot read how a str is stored,
/// so this finds its widest character one at a time.
#[cfg(any(Py_LIMITED_API, PyPy, GraalPy))]
fn py_str_size(value: &Bound<'_, PyString>) -> PyResult<usize> {
use pyo3::{ffi, types::PyAnyMethods as _};

let len = value.len()?;
let mut max_char = 0;
for i in 0..len {
// SAFETY - value is a str and i is within its length.
let ch = unsafe { ffi::PyUnicode_ReadChar(value.as_ptr(), i.cast_signed()) };
max_char = max_char.max(ch);
if max_char > 0xFFFF {
break;
}
}
Ok(str_size_for(len, max_char))
}

/// The size of a new message whose type has the given `__basicsize__` and
/// numbers of repeated and map fields, which start as empty containers.
pub(crate) fn message_size(basic_size: usize, lists: usize, dicts: usize) -> usize {
basic_size + GC_HEAD_SIZE + lists * EMPTY_LIST_SIZE + dicts * EMPTY_DICT_SIZE
}

/// Accounts for the Python objects allocated while parsing a message.
pub(crate) trait Budget {
/// Charges `amount` bytes.
fn charge(&mut self, amount: usize) -> PyResult<()>;

/// Charges for the pre-decoded str.
fn charge_str(&mut self, utf8: &[u8]) -> PyResult<()>;

/// Charges for an existing Python string.
fn charge_py_str(&mut self, value: &Bound<'_, PyString>) -> PyResult<()>;

/// Charges for a new message of the marshaler's type.
fn charge_message(&mut self, marshaler: &MessageMarshaler) -> PyResult<()> {
self.charge(marshaler.alloc_size)
}

/// Charges for an int. Small ints are shared and larger ones vary a
/// little in size, but a flat charge is close enough.
fn charge_int(&mut self) -> PyResult<()> {
self.charge(INT_SIZE)
}

/// Charges for a float.
fn charge_float(&mut self) -> PyResult<()> {
self.charge(FLOAT_SIZE)
}

/// Charges for a bytes of the given length.
fn charge_bytes(&mut self, len: usize) -> PyResult<()> {
self.charge(BYTES_OVERHEAD + len)
}

/// Charges for a materialized unknown enum member.
fn charge_enum(&mut self) -> PyResult<()> {
self.charge(INT_SIZE + GC_HEAD_SIZE)
}

/// Charges for a `Oneof` wrapper.
fn charge_oneof(&mut self) -> PyResult<()> {
self.charge(ONEOF_SIZE)
}

/// Charges for a new list.
fn charge_list(&mut self) -> PyResult<()> {
self.charge(EMPTY_LIST_SIZE)
}

/// Charges for appending an item to a list.
fn charge_list_item(&mut self) -> PyResult<()> {
self.charge(LIST_SLOT_SIZE)
}

/// Charges for a new dict.
fn charge_dict(&mut self) -> PyResult<()> {
self.charge(EMPTY_DICT_SIZE)
}

/// Charges for inserting an entry into a dict.
fn charge_dict_entry(&mut self) -> PyResult<()> {
self.charge(DICT_ENTRY_SIZE)
}
}

/// Tracks the approximate bytes allocated, raising an error once a limit is
/// exceeded.
pub(crate) struct TrackingBudget {
current: usize,
max: usize,
}

impl TrackingBudget {
pub(crate) fn new(limit: usize) -> Self {
Self {
current: 0,
max: limit,
}
}
}

impl Budget for TrackingBudget {
fn charge(&mut self, amount: usize) -> PyResult<()> {
self.current = self.current.saturating_add(amount);
if self.current > self.max {
Err(PyValueError::new_err(format!(
"allocation budget exceeded: needed {} bytes, limit is {}",
self.current, self.max
)))
} else {
Ok(())
}
}

fn charge_str(&mut self, utf8: &[u8]) -> PyResult<()> {
self.charge(str_size(utf8))
}

fn charge_py_str(&mut self, value: &Bound<'_, PyString>) -> PyResult<()> {
self.charge(py_str_size(value)?)
}
}

/// Does no accounting, for parsing without an allocation limit.
pub(crate) struct NoopBudget;

impl Budget for NoopBudget {
fn charge(&mut self, _amount: usize) -> PyResult<()> {
Ok(())
}

fn charge_str(&mut self, _utf8: &[u8]) -> PyResult<()> {
Ok(())
}

fn charge_py_str(&mut self, _value: &Bound<'_, PyString>) -> PyResult<()> {
Ok(())
}
}

/// Evaluates `$body` with `$budget` bound to the budget for an optional
/// allocation limit. The body is compiled for each implementation, so parsing
/// without a limit includes no accounting code.
macro_rules! with_budget {
($limit:expr, |$budget:ident| $body:expr) => {
match $limit {
Some(limit) => {
let $budget = &mut $crate::budget::TrackingBudget::new(limit);
$body
}
None => {
let $budget = &mut $crate::budget::NoopBudget;
$body
}
}
};
}
pub(crate) use with_budget;
3 changes: 3 additions & 0 deletions packages/protobuf-py-ext/src/constants.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ pub(crate) struct ConstantsInner {

/// The string `__new__`.
pub(crate) dunder_new: Py<PyString>,
/// The string `__basicsize__`.
pub(crate) dunder_basicsize: Py<PyString>,

/// Python types.
pub(crate) types: Types,
Expand Down Expand Up @@ -146,6 +148,7 @@ impl Constants {
values: PyString::new(py, "values").unbind(),

dunder_new: PyString::new(py, "__new__").unbind(),
dunder_basicsize: PyString::new(py, "__basicsize__").unbind(),

types: Types {
desc_field: mod_descriptors
Expand Down
Loading
Loading