[release/10.0] Materialize required complex collection absent from JSON document as empty

[philcarbone]

Fixes #38625
Backports #39014

Description

When a complex type is mapped to a JSON column and contains a required complex collection, a stored document that does not contain the collection's key is materialized with that collection set to null. This happens whenever a collection is added to an existing complex type: every row written before the change lacks the key. EF's JSON materializer runs the fixup for each nested property after reading the document, and for a property that never appeared it ran that fixup with null, overwriting whatever the CLR initializer had produced. The fix records whether each nested property was present in the document and, for a required complex collection that was not, assigns an empty collection instead. An explicit JSON null is still materialized as null.

A second, related defect is corrected in the change tracker: when a required complex collection on a complex collection element is null at save time, the validation that reports it looked up the containing complex property on the element's own entry, which threw a misleading "property does not belong to type" exception instead of the intended "required complex property is null" one. That path can only be reached from an already-failing save, so it is not quirked.

Customer impact

Any application that adds a collection to a complex type mapped to JSON finds every pre-existing row unusable: reading the collection throws NullReferenceException, and saving the entity after touching any unrelated property throws InvalidOperationException from SaveChanges.

var order = await context.Orders.FirstAsync();   // row persisted before Tags was added to the type
order.Details.Tags.Count;                        // NullReferenceException
order.Details.Note = "updated";
await context.SaveChangesAsync();                // InvalidOperationException

There is no model-level workaround. The only workaround is application code that coalesces the null collections after every query (for example in a ChangeTracker.Tracked handler), which is neither discoverable nor complete for no-tracking queries. Rewriting the affected rows is not possible through EF because the rows cannot be saved.

How found

User reported on 10.0.2, confirmed on 10.0.10, and reproduced inside EF's own test suite on both release/10.0 and main.

Regression

No. Complex collections mapped to JSON were introduced in 10.0, and this behavior has been present since then.

Testing

Four new tests in the shared relational test class (materialization tracked and no-tracking, explicit null preserved, save after load, and the corrected exception), with SQL baselines for the save case in the SQL Server and SQLite test classes. The main PR also extends a precompiled-query test that does not exist on this branch; that part is omitted here.

Risk

Low. The materializer change only affects a nested property that is absent from the document, which EF itself never writes; documents EF produces always contain the key. A quirk is added: Microsoft.EntityFrameworkCore.Issue38625 restores the previous materialization behavior. The change-tracker change only alters which exception is thrown on a path that already threw.

[github-actions]

Thank you for your contribution! However, this PR targets the release/10.0 branch.

External contributions should not target release branches. This pull request has been closed automatically; please open a new pull request targeting main or another non-release branch.

For more information, see our contribution guidelines.

[Copilot]

🟡 Changes recommended

A critical test setup issue stores malformed JSON and prevents the regression test from exercising the fix.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This release/10.0 backport fixes JSON materialization of absent required complex collections and corrects nested null validation.

Changes:

• Materializes absent required collections as empty while preserving explicit null.
• Adds a compatibility switch and fixes change-tracker validation.
• Adds regression tests and SQL Server/SQLite baselines.

File summaries

File	Summary
test/EFCore.SqlServer.FunctionalTests/Update/ComplexCollectionJsonUpdateSqlServerTest.cs	Adds SQL Server baseline.
test/EFCore.Sqlite.FunctionalTests/Update/ComplexCollectionJsonUpdateSqliteTest.cs	Adds SQLite baseline.
test/EFCore.Relational.Specification.Tests/Update/ComplexCollectionJsonUpdateTestBase.cs	Adds regression tests. Critical issue: doubled braces produce malformed JSON, preventing the materializer test from executing.
src/EFCore/ChangeTracking/Internal/InternalEntryBase.cs	Corrects nested complex-property validation.
src/EFCore.Relational/Query/RelationalShapedQueryCompilingExpressionVisitor.ShaperProcessingExpressionVisitor.cs	Adds absent-collection materialization and compatibility handling.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.