Python: recursively serialize nested container values

[wang ke (CoffeeDrivenCoder)]

Motivation & Context

SerializationMixin.to_dict() handled protocol objects only when they were direct list items or dictionary values. Protocol objects inside combinations such as dict -> list -> dict remained as Python objects, causing the corresponding to_json() call to fail with TypeError.

Description & Review Guide

• What are the major changes? The existing scalar, protocol, list, dictionary, and unsupported-value branches now share a recursive serializer. Regression tests cover deeply nested protocol objects and preserve the existing behavior of skipping unsupported values.
• What is the impact of these changes? Nested session and model state composed from supported serialization objects can be converted to dictionaries and JSON at arbitrary list/dictionary depth.
• What do you want reviewers to focus on? Please review the recursive helper's preservation of existing exclusion and unsupported-value behavior.

Related Issue

Fixes #7788

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[wang ke (CoffeeDrivenCoder)]

@microsoft-github-policy-service agree

[Esan (EsanRAHIMI)]

Thanks for the focused change—the sentinel-based recursion preserves the existing skip behavior cleanly, and the nested protocol regression coverage exercises the original failure well.

One compatibility edge is worth confirming or pinning with tests: moving the (datetime, date, time) branch into the shared helper expands conversion beyond dictionary values. Top-level date/time attributes and date/time values inside lists were previously skipped as non-serializable, but are now emitted as strings. That may be a desirable consistency fix, but it is an observable behavior change beyond nested protocol serialization. Could you either confirm it is intentional and add coverage for those locations, or restrict the conversion to the prior dictionary-value context?

[wang ke (CoffeeDrivenCoder)]

Esan (@EsanRAHIMI) You are right; the broader date/time conversion was not intentional. Addressed in 60d9e33 by moving (datetime, date, time) conversion back into the dictionary-value branch. Top-level date/time attributes and date/time list items are skipped as before, while dictionary values still serialize to strings and the nested protocol recursion remains intact. The regression test now covers all three date/time types in top-level, list-item, and dictionary-value locations. The 485 serialization/session/type tests, Ruff, source Pyright, and test Pyright all pass.

[Esan (EsanRAHIMI)]

The follow-up restores the previous date/time serialization boundary while keeping the nested container recursion intact. The regression test now covers datetime, date, and time across top-level attributes, list items, and dictionary values. This addresses my compatibility concern—thanks!

[wang ke (CoffeeDrivenCoder)]

Evan Mattson (@moonbox3) The follow-up in 8f3b8be addresses both remaining compatibility concerns: nested non-string dictionary keys retain the previous to_dict()/to_json() behavior, and recursive list/dictionary cycles now raise the controlled ValueError while repeated acyclic references remain supported. Regression tests cover both paths, and all 507 relevant tests plus Ruff and Pyright pass locally. A fresh review of the latest commit would be appreciated when convenient.

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework
_serialization.py	179	9	94%	590, 665, 668, 711–712, 716–717, 719, 721
TOTAL	48251	4479	90%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
9824	36 💤	0 ❌	0 🔥	2m 35s ⏱️

[wang ke (CoffeeDrivenCoder)]

Thanks for the suggestion! _SKIP_SERIALIZATION is already defined as a module-level object() sentinel, and all checks use identity comparison with is. If you meant a named sentinel type or a different project convention, I’m happy to adjust it.

[wang ke (CoffeeDrivenCoder)]

Thanks for the clarification. Updated _SKIP_SERIALIZATION to use the project-standard typing_extensions.Sentinel("SKIP_SERIALIZATION") with a Final annotation, while keeping the existing identity checks and behavior unchanged.

The focused serialization tests (39 tests) and Ruff check/format pass locally. The full dependency Pyright run is currently blocked by unrelated missing optional modules in this checkout.

[Eduard van Valkenburg (eavanvalkenburg)]

wang ke (@CoffeeDrivenCoder) there are merge conflicts, please have a look

[wang ke (CoffeeDrivenCoder)]

Resolved the merge conflicts and rebased the PR onto the latest upstream main. The upstream checkpoint/pickle changes are preserved, while the recursive serialization compatibility, circular-container detection, and typing_extensions.Sentinel updates remain applied.

Pushed as commit 4e8b1cf5. The focused serialization test file passes all 40 tests, and Ruff lint/format checks pass.