Spec: Deepen interface-family planning and execution

[marcinpsk]

Related to #72. This specification resolves candidate 1 from the full-codebase architecture baseline.

Problem Statement

NetBox now represents both flat breakout interfaces and channelized parent-and-channel interfaces. The plugin applies naming rules to these related interfaces during module installation, prediction, preview, bulk application, conversion, virtual-chassis reapplication, and deferred reconciliation.

The current implementation does not have one authoritative representation of an interface family. Each path reconstructs family identity, topology, intended names, collisions, and outcomes through different combinations of ORM rows, template maps, dictionaries, and tuples. This makes behavior difficult to reason about and creates a material risk that preview, prediction, execution, conversion, and deferred callbacks disagree as NetBox channelization evolves.

The family behavior is also embedded in a large engine that combines rule caching, rule selection, template evaluation, family discovery, mutation, conversion, preview, and reporting. Refactoring this behavior without first recording the existing signal-path performance could introduce N+1 queries, more database work, or slower execution without clear evidence.

Solution

Introduce one deep interface-family module that owns family discovery, planning, execution, conversion, capability handling, and deferred reconciliation. It will model flat breakout families, channelized families, incomplete channelized families, and the existing supported structural conversion. It will derive families from current NetBox rows or prospective templates for each operation. It will not add a persisted family model or cross-request family cache.

The module will expose immutable installed and prospective family plan sets. Each set will contain exactly one family plan per interface family. Installed plans will include validated row snapshots and will be the only executable plans. Prospective plans will support rowless prediction. Execution will lock and revalidate installed rows, preserve the existing family atomicity rules, and return explicit family and member outcomes.

The existing engine entry points will remain as compatibility adapters. Shared rule selection, variable construction, and template evaluation will move below the engine facade so the family module never imports the facade.

Before production behavior changes, add a manually invoked test-suite performance test and record the existing implementation. After the refactor, run the identical scenarios on the same hardware and compare only against that baseline. The comparison will include PostgreSQL work profiles, query scaling, uninstrumented wall time, process CPU time, and raw timing samples. Machine-time comparison will remain outside CI.

User Stories

1. As a NetBox operator, I want a module installation to name every related interface consistently, so that one physical port is represented correctly.
2. As a NetBox operator, I want a flat breakout family to be recognized as one family, so that its sibling channels receive coordinated names.
3. As a NetBox operator, I want a channelized parent and its channels to be recognized as one family, so that parent and channel names cannot drift accidentally.
4. As a NetBox operator, I want an incomplete channelized family to remain a valid channelized family, so that missing channels do not change its topology classification.
5. As a NetBox operator, I want ambiguous or incomplete flat-family discovery to be rejected, so that unrelated interfaces are never grouped by guesswork.
6. As a NetBox operator, I want family membership to be derived from current NetBox data, templates, and the active rule, so that stale cached topology is not applied.
7. As a NetBox operator, I want structural family creation to be atomic, so that a failed operation cannot leave a partial topology.
8. As a NetBox operator, I want the existing supported flat-to-channelized conversion to be atomic, so that failed validation restores the original family.
9. As a NetBox operator, I want a family rename to proceed only after its parent succeeds, so that child names cannot move ahead of a blocked parent.
10. As a NetBox operator, I want a collision on one child to leave only that child unchanged, so that unrelated successful channel renames are retained.
11. As a NetBox operator, I want a blocked family to leave other families unaffected, so that one conflict does not discard useful work across a device or module.
12. As a NetBox operator, I want target-name uniqueness to be enforced by NetBox and PostgreSQL, so that concurrent operations cannot create duplicate device interface names.
13. As a NetBox operator, I want execution to reject a stale installed family plan, so that concurrent edits are not overwritten.
14. As a NetBox operator, I want interactive apply to plan again from live rows, so that an earlier preview is never treated as an executable snapshot.
15. As a NetBox operator, I want preview and execution to use the same planning rules, so that the preview describes the operation that will be attempted.
16. As an integration user, I want rowless prediction to use the same family planner as installed interfaces, so that prospective names match later installation behavior.
17. As an integration user, I want prospective family plans to remain non-executable, so that predictions cannot mutate NetBox accidentally.
18. As a NetBox operator, I want bulk apply to use the same family semantics as automatic installation, so that manual repair does not produce a different topology or name set.
19. As a NetBox operator, I want conversion scans to use the same family membership and naming rules, so that conversion candidates match preview and execution.
20. As a NetBox operator, I want deferred channel-name reconciliation to remain internal to the family operation, so that callback ordering cannot leak into callers.
21. As a NetBox operator, I want virtual-chassis position changes to reapply family rules consistently across modules, so that all affected names reflect the new position.
22. As a NetBox operator, I want older NetBox releases to return an explicit unsupported outcome for channelization, so that the plugin does not attempt an invalid topology.
23. As a plugin maintainer, I want channelization capability detection to have one owner, so that callers do not implement version-specific branches.
24. As a plugin maintainer, I want one immutable family plan per family, so that family identity and intended operations are visible and cannot change during planning.
25. As a plugin maintainer, I want installed plans to contain row snapshots, so that execution can detect stale identity, names, membership, and topology.
26. As a plugin maintainer, I want expected outcomes to be explicit values, so that views, jobs, signals, and tests do not infer meaning from mutable conflict lists.
27. As a plugin maintainer, I want each member outcome to include its relevant facts, so that user-facing adapters can explain changed, unchanged, blocked, stale, unsupported, and failed operations.
28. As a plugin maintainer, I want unexpected programming and infrastructure errors to remain visible, so that database or implementation failures are not mislabeled as ordinary collisions.
29. As a plugin maintainer, I want the family module to own ORM loading and prefetching, so that callers cannot introduce inconsistent query behavior.
30. As a plugin maintainer, I want rule and template data to be loaded once per applicable scope, so that processing does not become an N+1 query path.
31. As a plugin maintainer, I want family discovery and planning to scale linearly with family and member counts, so that larger devices remain practical.
32. As a plugin maintainer, I want the engine to remain a compatibility facade, so that existing public callers can upgrade without adopting private family details.
33. As a plugin maintainer, I want old private family helpers removed when the replacement is complete, so that the codebase has one behavior path.
34. As a plugin maintainer, I want a signal-path performance baseline before refactoring, so that performance changes are measured rather than assumed.
35. As a plugin maintainer, I want query counts and PostgreSQL work recorded for representative families, so that additional scans and database work are visible.
36. As a plugin maintainer, I want before-and-after machine time recorded on the same hardware, so that the refactor can be checked against the existing implementation.
37. As a plugin maintainer, I want raw timing samples retained with summary statistics, so that variance is visible during review.
38. As a plugin maintainer, I want database profiling and machine timing collected in separate passes, so that profiling overhead does not distort the timing comparison.
39. As a plugin maintainer, I want the performance comparison to run manually rather than in CI, so that shared-runner variance does not create unreliable failures.
40. As a reviewer, I want normalized before-and-after artifacts with source and environment revisions, so that the comparison can be reproduced and evaluated.
41. As a reviewer, I want real PostgreSQL integration tests for locking, validation, and collisions, so that fabricated mock behavior cannot hide concurrency defects.
42. As a reviewer, I want the final replacement to preserve installed, prospective, preview, bulk, conversion, and deferred behavior, so that architectural improvement does not remove supported workflows.

Implementation Decisions

• Use interface family as the broad domain term. It includes flat breakout families, channelized families, incomplete channelized families, and the existing supported structural conversion.
• Derive family membership for each operation. Do not add a database model, persisted family identity, or cross-request family cache.
• Identify a channelized family from its parent relationship and channel identifiers.
• Recover a flat breakout family only from the complete expected rule-name set together with raw-template matchers. Reject ambiguous or incomplete recovery.
• Represent planning with immutable requests, family plans, plan sets, and outcomes.
• Keep a common family-plan model, with a distinct installed plan that includes a validated row snapshot.
• Permit execution only for installed plans. Prospective plans support prediction and preview but cannot mutate rows.
• Return one plan set containing exactly one family plan for each family in the operation.
• Let the installed adapter own bulk ORM loading, prefetching, family discovery, and snapshot construction.
• Let the prospective adapter own rowless template inputs and construct the same planning-domain values without pretending that rows exist.
• Let callers provide semantic roots such as the module, device, rule, and intended operation. Do not accept caller-discovered family querysets as the public seam.
• Express rename, structural creation, and the existing structural conversion as operation intents within the same planning architecture.
• Keep conversion discovery, naming, validation, mutation, and reporting inside the family module.
• Preserve real NetBox validation during structural conversion. Validation failures roll back the attempted structural change.
• Keep deferred channel-name reconciliation and its NetBox callback registration inside the family module.
• Keep channelization capability detection inside the family module. Return unsupported when the active NetBox data model cannot represent the topology.
• Do not infer channelization from a pinned beta version. Capability behavior must work against the supported releases and moving NetBox branches.
• Revalidate installed plans against live identity, names, membership, and topology before mutation.
• Reject stale plans. Do not silently rebuild them and do not execute only the members that still match.
• Replan interactive apply from live rows rather than executing the earlier preview object.
• Execute each family in its own database transaction. Do not make an entire module or device batch atomic.
• Lock installed member rows in stable primary-key order, then revalidate the plan.
• Check target names after locking. Use NetBox's device-and-name uniqueness constraint as the final race guard instead of locking every interface on a device.
• Make every structural family change atomic.
• For an existing-family rename, require parent success and use child savepoints so one child collision leaves only that child unchanged.
• Continue with other family plans when one family is blocked.
• Use explicit family statuses: changed, unchanged, blocked, stale, unsupported, and failed.
• Include member-level facts in outcomes so user-facing adapters can render useful explanations.
• Map a known device-and-name uniqueness violation to blocked only when the database constraint identifies that collision.
• Map validation rejection caused by current family state to blocked.
• Propagate unrelated integrity failures, infrastructure errors, and programming errors to the signal or operation boundary for logging.
• Extract shared rule selection, variable construction, and name-template evaluation into lower-level modules.
• Make the existing engine a compatibility facade over the lower-level naming and family modules.
• Prohibit the family module from importing the engine facade.
• Organize the family implementation around a narrow public facade, immutable domain values, installed and prospective adapters, planning, and execution.
• Keep conversion and deferred reconciliation as internal family operations rather than parallel public architectures.
• Remove the old private family helpers when all callers use the new module. Do not retain dual behavior paths or compatibility shims for private functions.
• Preserve the existing public engine entry points as thin adapters for downstream compatibility.
• Preserve rule-selection precedence and name-template language behavior while relocating their shared primitives.
• Add the performance test and record the current implementation before changing production family behavior.
• Keep the performance test in the test suite but outside default test discovery. Invoke it explicitly in the NetBox devcontainer with an isolated test database.
• Record two performance passes. The PostgreSQL work-profile pass collects database work with node timing disabled. The timing pass runs without database profiling instrumentation.
• Record normalized SQL, statement counts, plan shapes, selected indexes, planner cost, actual rows and loops, buffer access, temporary blocks, rows affected, WAL activity, and scaling behavior.
• Record raw wall-time and process-CPU samples, together with median and p95 summaries.
• Run the before and after measurements on the same hardware with the same Python, PostgreSQL, NetBox revision, fixtures, planner settings, and table statistics.
• Compare the refactor only with the recorded existing-implementation baseline. Do not create an absolute machine-time threshold.
• Keep machine-time comparison outside CI. Retain the before-and-after comparison as review evidence.
• Normalize artifacts and exclude SQL parameter values or other environment-specific data.
• Treat performance as an acceptance condition. If the after profile increases database work or machine time beyond observed baseline variance, investigate and optimize before completing the refactor.
• Do not add or modify plugin database schema for this work.

Testing Decisions

• Prefer the complete automatic naming signal path as the highest-value seam. Exercise real module and device saves, NetBox-created interface rows, committed callbacks, the plugin planner and executor, PostgreSQL writes, and final interface rows.
• Use the existing committed-callback capture pattern for end-to-end module installation and virtual-chassis drift tests.
• Keep direct deferred-callback tests as a diagnostic seam. They isolate plugin reload, rule lookup, family planning, validation, collision handling, and writes without measuring NetBox's model-instantiation work.
• Test the public installed/prospective family-plan boundary directly where no higher end-to-end seam exists, especially for rowless prediction and explicit plan outcomes.
• Use real NetBox and Django models with the real PostgreSQL test database. Do not mock ORM rows, transactions, uniqueness constraints, validation, or callbacks.
• Use an isolated test database for every manual performance run and every concurrency-sensitive test.
• Add characterization coverage before replacing existing behavior. Confirm the current installed, preview, bulk, conversion, and deferred outcomes before moving their implementation.
• Add new API behavior tests before implementing each family-plan capability. Confirm that they fail for the missing behavior, then implement until they pass.
• Verify flat-family recovery accepts only a complete, unambiguous expected name set with matching raw templates.
• Verify incomplete channelized families remain channelized.
• Verify each family member appears in exactly one plan and each family appears exactly once in a plan set.
• Verify installed plans capture identity, names, membership, and topology and become stale when any of those facts changes.
• Verify interactive apply constructs a new live plan instead of executing its earlier preview.
• Verify prospective prediction and installed planning produce the same intended names for equivalent inputs.
• Verify prospective plans cannot be executed.
• Verify structural creation and conversion roll back the complete family on collision or validation failure.
• Verify a blocked parent prevents its children from renaming.
• Verify a blocked child leaves only that child unchanged while other valid child renames remain committed.
• Verify one blocked family does not roll back unrelated families in the batch.
• Verify concurrent execution locks family rows in stable order and rejects stale state.
• Verify a concurrent target-name race is classified as blocked only for the expected uniqueness constraint.
• Verify unrelated database integrity failures propagate rather than becoming domain outcomes.
• Verify older NetBox data models produce unsupported without a caller-side channelization branch.
• Verify conversion continues to use real NetBox validation inside a rolled-back transaction when validation fails.
• Verify deferred reconciliation runs in the correct callback order through the real committed-callback seam.
• Verify the virtual-chassis device path pins shared rule loading across multiple modules.
• Verify rule and template loading stays flat as family and module counts grow.
• Verify query scaling for one and eight families and reject per-family or per-member N+1 behavior.
• Baseline at least these signal scenarios: no matching rule, one plain rename, creation of a four-channel family, rename of an installed parent with four channels, deferred channel reconciliation, and virtual-chassis reapplication with one and eight families.
• Capture query SQL outside the committed-callback capture scope so callback queries are included.
• Measure cold and steady-state rule lookup deliberately. Reset or warm the rule cache explicitly rather than depending on test order.
• Use PostgreSQL execution-plan collection around the real test scenario, with JSON plans, actual rows, buffers, WAL, and node timing disabled.
• Capture server plan notices through the database driver so the test suite produces structured artifacts without replaying mutating SQL.
• Keep PostgreSQL work profiling separate from the uninstrumented machine-time pass.
• Warm each timing scenario before collecting repeated wall-time and process-CPU samples. Retain the raw sample distribution in addition to median and p95.
• Fail the manual performance test when required PostgreSQL profiling support is unavailable. Do not silently reduce the measurement.
• Record source revision, NetBox revision, Python version, PostgreSQL version, planner settings, fixture sizes, and relevant database statistics with each artifact.
• Compare the after run only with the recorded before run from the same hardware and controlled software environment.
• Do not run the machine-time comparison in normal or experimental CI jobs.
• Prior art includes the existing full channelized module-install tests, committed-callback virtual-chassis tests, direct deferred-callback tests, query-cache tests, conversion scaling tests, and standard NetBox query-count assertions.

Out of Scope

• Adding a persisted interface-family model, family identifier, or cross-request family cache.
• Adding new database migrations or changing the rule schema.
• Adding a new reverse channelized-to-flat conversion if it is not already supported.
• Changing unrelated plain-interface rename behavior that has no interface-family semantics.
• Changing rule eligibility, precedence, or cache policy beyond extracting the existing behavior behind a lower-level interface.
• Changing the name-template language, accepted variables, or expression-safety policy.
• Replacing all Django signal lifecycle handling outside the family-specific deferred reconciliation path.
• Redesigning foreground views, background jobs, REST, or GraphQL presentation beyond adapting them to explicit outcomes.
• Coordinating atomic name swaps across separate interface families.
• Locking every interface on a device during family execution.
• Persisting or executing an interactive preview plan across requests.
• Pinning an unreleased NetBox beta solely for channelization support.
• Introducing an external benchmark service or a recurring machine-time CI gate.
• Preserving compatibility for private engine helpers after all in-repository callers migrate.

Further Notes

• The implementation must begin with the performance test and existing-implementation baseline. Production family refactoring starts only after that baseline exists.
• The same performance scenarios must generate both the before and after artifacts. Changing a scenario invalidates direct comparison and requires a new before run.
• Planner cost is a PostgreSQL estimate in arbitrary units, not elapsed time. It is useful only with matching PostgreSQL versions, settings, fixtures, and statistics.
• Machine time is valid here as one part of the manual comparison because both measurements run on the same hardware. It is not promoted to a portable or recurring threshold.
• PostgreSQL work metrics and final semantic outcomes remain necessary because an elapsed-time result alone cannot explain a regression.
• Private implementation structure can evolve during the work, but the immutable plan boundary, transaction semantics, outcome contract, dependency direction, and performance acceptance criteria are fixed decisions.