Pass Sync Filter to Model Deserialization for Flexible Foreign Key Handling

[ozer550]

Overview

Enable Morango's deserialize() method to receive sync filter context, allowing syncable models to conditionally handle references based on sync type.

Background & Context

Kolibri's facility data models (like Lesson and Exam) were originally designed for full-facility synchronization.

The partition (dataset_id) meant these models could only sync at facility level i.e ALL data for that facility. There was a need to sync only specific model data and not the entire facility data. The partition structure didn't support fine-grained filtering to individual users.

Historical Solution

To enable learner-specific syncing, Kolibri introduced separate "individual syncable" models:

• IndividualSyncableLesson
• IndividualSyncableExam

These models completely omit created_by, assigned_by, and creator foreign key fields. This was necessary because coach FacilityUser records are never synced to learner devices. Without those records in local database, any foreign key pointing to the coach would trigger validation errors during deserialization. By removing these FK fields entirely and stripping coach references during data copying, these models avoid the validation problem altogether. The trade-off here is maintaining duplicate models for the same concepts (Lesson + IndividualSyncableLesson), requiring conversion logic and making new partition patterns less flexible.

New Prototype

We are now prototyping new partition structures to enable using the same model (Lesson/Exam) for both full-facility and learner-only syncs, eliminating the need for separate IndividualSyncable models.

The solution is to pass the sync filter as a parameter to the deserialize() method during store-to-model-conversion conversion. With this context, models can detect the sync type and conditionally handle foreign keys, models can check if referenced users exist locally and nullify missing foreign keys before validation occurs. This allows a single model to work across all sync types retaining complete data in the Store while adapting the Django model behavior based on context.

Changes Required

A new parameter sync_filter should be added to deserialization methods that:

• is optional with default value None for backward compatibility
• accepts a string representing the current sync filter (e.g., "abc123", "LOD:lesson:123", "abc123:user-ro:alice-uuid")
• gets threaded through the deserialization call chain from orchestration to model-level methods.

The Store._deserialize_store_model method should be updated to:

• accept an optional sync_filter parameter in its signature.
• pass this filter to the deserialize() call.
• maintain backward compatibility when sync_filter is None.

The _deserialize_from_store function should be updated to:

• pass the filter parameter (already available in function)
• thread it through to _deserialize_store_model() calls

The SyncableModel.deserialize classmethod should be updated to:

• accept an optional sync_filter parameter in its signature.
• ignore the parameter in the base implementation (backward compatibility).
• allow subclasses to override and use the filter for conditional logic.

Post validation should also have context of sync_filter and excluded_fields:

• SyncableModel.cached_clean_fields() and SyncableModel.deferred_clean_fields() should accept optional sync_filter and excluded_fields parameters.
• both methods should coalesce excluded_fields using excluded_fields = excluded_fields or [].
• base behavior should remain unchanged when parameters are not provided, but allow overrides to pass additional excluded fields and use sync_filter for conditional validation logic.

Value Add

Eliminates duplicate models (e.g., Lesson + IndividualSyncableLesson), reducing maintenance burden and code duplication. Store maintains complete data with full FK references for better audit trails. Single model adapts to all sync types, enabling flexible partition patterns without architectural changes.