Skip to content

gh-161767: Fix crash in PyDict_AsFrozenDictAndClear() with inline values - #161768

Open
iRELDevz wants to merge 4 commits into
python:mainfrom
iRELDevz:gh-161767-frozendict-inline-values
Open

iRELDevz wants to merge 4 commits into
python:mainfrom
iRELDevz:gh-161767-frozendict-inline-values

Conversation

@iRELDevz

@iRELDevz iRELDevz commented Nov 10, 2026 •

Copy link
Copy Markdown

Copy the inline values into the new frozendict and clear them on the object, instead of leaving them shared.

@python-cla-bot

python-cla-bot Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

Comment thread Objects/dictobject.c Outdated
}
dictkeys_incref(keys);
clear_common(old);
for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dk_nentries can be modified by another thread. How about using v->capacity instead?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, switched to v->capacity.

class A: ...
a = A()
a.x = 1
f = as_frozendict(a.__dict__)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to check also a just after as_frozendict() call:

Suggested change
f = as_frozendict(a.__dict__)
f = as_frozendict(a.__dict__)
self.assertEqual(vars(a), {})

Comment thread Objects/dictobject.c Outdated
Comment on lines +8699 to +8700
ASSERT_CONSISTENT(old);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it's also worth it to check dict consistency for the if (values && values->embedded) case, so:

Suggested change
ASSERT_CONSISTENT(old);
}
}
ASSERT_CONSISTENT(old);

Comment thread Objects/dictobject.c
}

// Transfer keys and values from dict to frozendict:
new->ma_used = used;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
new->ma_used = used;
// Only transfer keys and values to the new dict
// once the old dict is cleared
new->ma_used = used;

@iRELDevz

Copy link
Copy Markdown
Author

done, applied all three.

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@methane: Do you want to double check the change?

@vstinner

Copy link
Copy Markdown
Member

cc @sobolevn

@sobolevn sobolevn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should disallow values with ->embedded set?
The whole point of this API is to use mutable dict objects to create new frozendict objects efficently.

Using embedded dicts do not really make sense in my opinion.
I would just raise SystemError for this case.

a = A()
a.x = 1
f = as_frozendict(a.__dict__)
self.assertEqual(vars(a), {})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels weird that this mutates instance's __dict__ :(

Comment thread Objects/dictobject.c
}
dictkeys_incref(keys);
clear_common(old);
for (Py_ssize_t i = 0; i < v->capacity; i++) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe invalidate_and_clear_inline_values(v); instead?

Comment thread Objects/dictobject.c
PyErr_NoMemory();
return -1;
}
dictkeys_incref(keys);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: why do we need the incref here?

@vstinner

Copy link
Copy Markdown
Member

Maybe we should disallow values with ->embedded set? The whole point of this API is to use mutable dict objects to create new frozendict objects efficently.

When you call PyDict_AsFrozenDictAndClear(), you may not know in advance if the dict uses inline values or not.

@sobolevn

Copy link
Copy Markdown
Member

I think that all dicts created by PyDict_New don't use inline values, do they? Can you please provide more context on the use-cases when you don't know the dict layout? I am pretty sure that I am missing something :)

@vstinner

Copy link
Copy Markdown
Member

I never understood exactly when inline values are used or not. It seems like it's mostly used for object managed dictionary?

Maybe you're right and raising an exception is just fine in this case. Can it be a ValueError or a RuntimeError? For me, SystemError if you obviously misused the API. Here you pass a dictionary, but the fact that values are inlined is a suble implementation detail.

@sobolevn

Copy link
Copy Markdown
Member

As far as I know, inline values are only used for instance.__dict__ (with Py_TPFLAGS_INLINE_VALUES) and when explicitly asked to create one with _PyObject_InitInlineValues (only in CPython, since this API is not public). But, I might be wrong here.

I doubled checked with DeepWiki: https://deepwiki.com/search/7660ca12-56c2-4868-b54c-a6306dca7990?mode=deep

Снимок экрана 2026-11-17 в 00 38 46

One more thing that I am not quite sure about: will this feature get more usage in other use-cases? Or will it stay as an implementation detail just for this small case?

ValueError seems like a good option :)

@sobolevn

Copy link
Copy Markdown
Member

I created an alternative PR with my proposal: #162139 :)

@vstinner

Copy link
Copy Markdown
Member

I doubled checked with DeepWiki: https://deepwiki.com/search/7660ca12-56c2-4868-b54c-a6306dca7990?mode=deep

Oh, I didn't know this website. Interesting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants