diff --git a/.devcontainer/scripts/setup.sh b/.devcontainer/scripts/setup.sh index 2e9d8ff..df95b10 100755 --- a/.devcontainer/scripts/setup.sh +++ b/.devcontainer/scripts/setup.sh @@ -140,9 +140,9 @@ apt-get update -qq apt-get install -y -qq net-tools git # Dev tools used by the agent loop and pre-commit hooks. Keep in sync with # the `dev` extras in pyproject.toml — at minimum, anything invoked by: -# - test + coverage runs: pytest, pytest-django, pytest-cov +# - test + coverage runs: pytest, pytest-django, pytest-cov, pytest-xdist # - pre-commit hooks (.pre-commit-config.yaml): ruff, pre-commit, reuse -$PIP_CMD install pytest pytest-django pytest-cov ruff pre-commit reuse +$PIP_CMD install pytest pytest-django pytest-cov pytest-xdist ruff pre-commit reuse # Install GitHub CLI if ! command -v gh >/dev/null 2>&1; then diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index c5175ee..a0ab0bb 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -2,24 +2,22 @@ ## Project overview -NetBox plugin that automatically renames interfaces when modules (transceivers, line cards, converters) are installed into device module bays. It hooks into Django's `post_save` signal on `dcim.Module` to apply configurable renaming rules with template variable substitution and arithmetic expression support. +NetBox plugin that automatically renames interfaces when modules (transceivers, line cards, converters) are installed into device module bays. It also reapplies rules when a module type or a device's virtual-chassis position changes. Django signals defer the work until the surrounding transaction commits. -Requires NetBox ≥ 4.2.0 and Python ≥ 3.12. Licensed under Apache-2.0 (REUSE-compliant). +Requires NetBox ≥ 4.3.0 and Python ≥ 3.12. Licensed under Apache-2.0 (REUSE-compliant). ## Architecture This follows the standard [NetBox plugin pattern](https://netboxlabs.com/docs/netbox/en/stable/plugins/development/): -- **`models.py`** — Single model `InterfaceNameRule` linking a module type (nullable FK, required only in exact mode) to a name template, with optional scoping to parent module type, device type, and/or platform. Rules are matched most-specific-first. -- **`signals.py`** — Two signal handlers: - - `post_save` on `dcim.Module` (primary path): fires on `created=True`, defers renaming to `on_commit` so interfaces exist in DB first. This is the **only** path that runs during normal module installation because NetBox creates interfaces via `bulk_create()`. - - `pre_save` on `dcim.Interface` (defence-in-depth): would rename before INSERT, but **does NOT fire** during normal module installation because `bulk_create()` skips `pre_save` signals. Only fires when interfaces are created individually via `Interface.save()` (scripts, custom code, etc.). - - Lazily imports the engine to avoid circular imports during Django startup. -- **`engine.py`** — Core logic: two-tier rule lookup (`_find_matching_rule` first tries an exact FK match across 4 specificity levels, then falls back to regex matching with `re.fullmatch()` across the same 4 levels), template variable building from module bay hierarchy, and interface renaming/breakout creation. The `evaluate_name_template` function supports `{variable}` substitution followed by safe AST-based arithmetic evaluation of remaining brace expressions. -- **`api/`** — DRF REST API using NetBox's `NetBoxModelViewSet` and `NetBoxModelSerializer`. -- **`views.py`, `urls.py`, `tables.py`, `forms.py`, `filters.py`, `navigation.py`** — Standard NetBox UI CRUD views. -- **`utils.py`** — Feature detection for gating (e.g., `{module_path}` token support detected via `dcim.constants.MODULE_PATH_TOKEN` import). +- **`models.py`**: Defines `InterfaceNameRule`. A rule can select an exact module type or a regex pattern, add parent, device, and platform scopes, and describe flat or channelized breakout output. +- **`signals.py`**: Handles `pre_save` and `post_save` for `dcim.Module` and `dcim.Device`. It records prior state, schedules work with `transaction.on_commit()`, and catches failures at the deferred callback boundary. It intentionally does not connect to `dcim.Interface` because NetBox creates module interfaces with `bulk_create()`. It also connects the optional LibreNMS prediction signal when that plugin is installed. +- **`rule_selection.py`**: Loads and fingerprints enabled rules, separates exact and regex candidates, applies scope priority, and pins one cached snapshot across batch work. +- **`naming.py`**: Builds variables from the module-bay hierarchy and evaluates templates. It replaces known variables, parses the remaining integer arithmetic, and evaluates only supported AST nodes. +- **`family/`**: Owns the interface-family domain model, discovery, planning, execution, structural creation, conversion, name collision checks, and NetBox capability detection. +- **`engine.py`**: Orchestrates rule application, prediction, virtual-chassis reapply, preview, and batch operations. It keeps stable entry points while delegating rule selection, naming, and family behavior to their owning modules. +- **`api/` and `graphql/`**: Expose NetBox REST and GraphQL integrations. +- **`views.py`, `urls.py`, `tables.py`, `forms.py`, `filters.py`, `navigation.py`, `jobs.py`**: Provide NetBox UI and background-job integrations. The signal handler → engine import is intentionally lazy to ensure Django models are fully loaded before use. @@ -89,9 +87,9 @@ The `reuse-lint` pre-commit hook validates compliance on every commit. ## Key conventions - All views, forms, serializers, and tables inherit from NetBox's base classes (`NetBoxModel`, `NetBoxModelViewSet`, `NetBoxModelForm`, etc.) — always use these, not raw Django/DRF equivalents. Non-model forms are the exception: NetBox 4.x dropped `BootstrapMixin` and styles every form through its own widget templates (`FORM_RENDERER = TemplatesSetting`), so a plain form subclasses `django.forms.Form`, exactly as NetBox's own `ConfirmationForm`/`BulkRenameForm` do. -- Template variables use Python `str.format()` syntax: `{slot}`, `{bay_position}`, `{bay_position_num}`, `{parent_bay_position}`, `{sfp_slot}`, `{base}`, `{channel}`, `{module_path}` (gated via `utils.supports_module_path()` using import-based feature detection). -- Arithmetic inside braces is evaluated via `ast.parse` with a strict allowlist of AST node types — never use `eval()` directly on user input. +- Template variables use braces: `{slot}`, `{bay_position}`, `{bay_position_num}`, `{parent_bay_position}`, `{sfp_slot}`, `{base}`, `{channel}`, and `{vc_position}`. `naming.py` replaces known variables explicitly before it parses arithmetic. +- Arithmetic inside braces is parsed with `ast.parse` and evaluated recursively for the supported integer operators. Never use `eval()` on user input. - The `tags` field on `InterfaceNameRule` uses `related_name="+"` to avoid reverse accessor clashes with other plugins. -- Rule matching uses **two tiers** within each priority level — exact FK match first, then regex (`re.fullmatch()`) fallback. Priority levels (applied in both tiers): (module_type + parent + device) → (module_type + parent) → (module_type + device) → (module_type only). +- Rule matching uses two tiers. Exact module-type rules take priority over regex rules. Within each tier, `rule_selection.py` applies the parent, device, and platform scope score, then the documented tie breakers. - Add new rules to the appropriate vendor-specific file under `contrib/` (`cisco.yaml`, `juniper.yaml`, `linux.yaml`, `ufispace.yaml`, `ufispace-device-type.yaml`, `converters.yaml`) — keep them updated when adding new rule patterns. - Commits follow [Conventional Commits](https://www.conventionalcommits.org/) format, enforced by pre-commit hook. diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 043a1d8..ba3d962 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -11,11 +11,15 @@ updates: directory: "/" # Location of package manifests schedule: interval: "weekly" + cooldown: + default-days: 7 - package-ecosystem: "github-actions" # See documentation for possible values directory: "/" # Location of package manifests schedule: interval: "weekly" + cooldown: + default-days: 7 groups: github-actions: patterns: @@ -25,8 +29,12 @@ updates: directory: "/.devcontainer" # Location of devcontainer.json schedule: interval: "weekly" + cooldown: + default-days: 7 - package-ecosystem: "docker-compose" # See documentation for possible values directory: "/.devcontainer" # Location of docker-compose.yml schedule: interval: "weekly" + cooldown: + default-days: 7 diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index d92a765..a9e3088 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -63,6 +63,8 @@ jobs: steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false # Add any setup steps before running the `github/codeql-action/init` action. # This includes steps like installing compilers or runtimes (`actions/setup-node` diff --git a/.github/workflows/coverage-badge.yaml b/.github/workflows/coverage-badge.yaml index 5d669bd..6e1df17 100644 --- a/.github/workflows/coverage-badge.yaml +++ b/.github/workflows/coverage-badge.yaml @@ -3,6 +3,8 @@ name: Update coverage report on: + # zizmor: ignore[dangerous-triggers] the job accepts only a successful push from this + # repository's main branch, and never consumes artifacts from pull-request code. workflow_run: workflows: ["Test with supported NetBox and Python versions"] types: [completed] @@ -66,6 +68,8 @@ jobs: fi - name: Checkout gh-pages + # zizmor: ignore[artipacked] this job pushes the coverage report to gh-pages, + # so this checkout has to keep its credential. uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: gh-pages diff --git a/.github/workflows/lint-format.yaml b/.github/workflows/lint-format.yaml index 2858744..a01c64d 100644 --- a/.github/workflows/lint-format.yaml +++ b/.github/workflows/lint-format.yaml @@ -9,9 +9,13 @@ on: jobs: format-and-lint: runs-on: ubuntu-latest + permissions: + contents: read steps: - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false - name: Install uv uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 @@ -22,7 +26,7 @@ jobs: python-version: '3.12' - name: Install dependencies - run: uv pip install --system ruff + run: 'uv pip install --system --only-binary=:all: pre-commit==4.5.1 ruff==0.16.0' - name: Run Ruff linting run: ruff check . @@ -30,5 +34,9 @@ jobs: - name: Run Ruff formatting check run: ruff format --check . + # Runs the pinned hook from .pre-commit-config.yaml, so the version lives in one place. + - name: Audit the workflows + run: pre-commit run --all-files zizmor + - name: Run devcontainer script tests run: bash .devcontainer/scripts/tests/test-debug-toolbar-patches.sh diff --git a/.github/workflows/mkdocs.yaml b/.github/workflows/mkdocs.yaml index a24d5ac..a832057 100644 --- a/.github/workflows/mkdocs.yaml +++ b/.github/workflows/mkdocs.yaml @@ -26,6 +26,8 @@ jobs: contents: write steps: - name: Checkout repository + # zizmor: ignore[artipacked] mkdocs gh-deploy pushes to gh-pages, so this + # checkout has to keep its credential. uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 diff --git a/.github/workflows/pr-title.yaml b/.github/workflows/pr-title.yaml index 04263dc..39dbd9a 100644 --- a/.github/workflows/pr-title.yaml +++ b/.github/workflows/pr-title.yaml @@ -3,7 +3,7 @@ name: Validate PR title on: - pull_request_target: + pull_request: types: [opened, edited, synchronize] permissions: diff --git a/.github/workflows/publish-pypi.yaml b/.github/workflows/publish-pypi.yaml index 64c77d5..d7dc4b7 100644 --- a/.github/workflows/publish-pypi.yaml +++ b/.github/workflows/publish-pypi.yaml @@ -18,13 +18,18 @@ jobs: build: name: Build distribution 📦 runs-on: ubuntu-latest + permissions: + contents: read steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ inputs.tag || github.ref }} + persist-credentials: false - name: Install uv uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 + with: + enable-cache: false - name: Set up Python uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index 5d43e8c..588040c 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -17,6 +17,8 @@ jobs: pull-requests: write steps: - name: Checkout + # zizmor: ignore[artipacked] semantic-release pushes the release commit and tag, + # so this checkout has to keep its credential. uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 0 diff --git a/.github/workflows/test-netbox-main.yaml b/.github/workflows/test-netbox-main.yaml index 85994ac..a7a0fb2 100644 --- a/.github/workflows/test-netbox-main.yaml +++ b/.github/workflows/test-netbox-main.yaml @@ -14,6 +14,8 @@ concurrency: jobs: test-netbox-main: runs-on: ubuntu-latest + permissions: + contents: read strategy: fail-fast: false @@ -49,6 +51,7 @@ jobs: uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: path: netbox-InterfaceNameRules-plugin + persist-credentials: false - name: Install uv uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 @@ -64,12 +67,13 @@ jobs: repository: "netbox-community/netbox" path: netbox ref: main + persist-credentials: false - name: Install NetBox and plugin working-directory: netbox-InterfaceNameRules-plugin run: | uv pip install --system -r ../netbox/requirements.txt - uv pip install --system pytest pytest-django tblib + uv pip install --system --only-binary=:all: pytest==9.0.2 pytest-django==4.12.0 pytest-xdist==3.8.0 tblib==3.2.2 uv pip install --system -e . - name: Set up NetBox configuration @@ -96,8 +100,9 @@ jobs: EOF - name: Run tests - working-directory: netbox/netbox + working-directory: netbox-InterfaceNameRules-plugin env: NETBOX_CONFIGURATION: netbox.configuration + PYTHONPATH: ${{ github.workspace }}/netbox/netbox run: | - python manage.py test netbox_interface_name_rules --verbosity=2 + pytest -n auto netbox_interface_name_rules -o pythonpath=../netbox/netbox diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index 3c3417f..f6d7173 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -18,6 +18,8 @@ concurrency: jobs: test-netbox: runs-on: ubuntu-latest + permissions: + contents: read strategy: fail-fast: false @@ -66,6 +68,7 @@ jobs: uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: path: netbox-InterfaceNameRules-plugin + persist-credentials: false - name: Install uv uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1 @@ -81,12 +84,13 @@ jobs: repository: "netbox-community/netbox" path: netbox ref: ${{ matrix.netbox-version }} + persist-credentials: false - name: Install NetBox and plugin working-directory: netbox-InterfaceNameRules-plugin run: | uv pip install --system -r ../netbox/requirements.txt - uv pip install --system pytest pytest-django tblib coverage + uv pip install --system --only-binary=:all: pytest==9.0.2 pytest-cov==7.0.0 pytest-django==4.12.0 pytest-xdist==3.8.0 tblib==3.2.2 uv pip install --system -e . - name: Set up NetBox configuration @@ -113,10 +117,11 @@ jobs: EOF - name: Run tests with coverage - working-directory: netbox/netbox + working-directory: netbox-InterfaceNameRules-plugin env: NETBOX_CONFIGURATION: netbox.configuration - COVERAGE_RCFILE: ../../netbox-InterfaceNameRules-plugin/pyproject.toml + PYTHONPATH: ${{ github.workspace }}/netbox/netbox + COVERAGE_RCFILE: pyproject.toml # Turns the channelization tests' skipUnless guard into an assertion on the leg that must # have the feature, so a broken probe cannot silently skip the whole file. EXPECT_NETBOX_CHANNELIZATION: ${{ matrix.netbox-version == 'feature' && '1' || '' }} @@ -124,19 +129,19 @@ jobs: # checkout so these cells report plugin compatibility failures instead of baseline drift. UPDATE_QUERY_COUNTS: ${{ matrix.experimental && '1' || '' }} run: | - coverage run manage.py test netbox_interface_name_rules --verbosity=2 - coverage report + pytest -n auto netbox_interface_name_rules --cov=netbox_interface_name_rules --cov-report=term-missing \ + -o pythonpath=../netbox/netbox - name: Generate coverage report if: matrix.python-version == '3.12' && matrix.netbox-version == 'v4.5.3' - working-directory: netbox/netbox + working-directory: netbox-InterfaceNameRules-plugin env: - COVERAGE_RCFILE: ../../netbox-InterfaceNameRules-plugin/pyproject.toml + COVERAGE_RCFILE: pyproject.toml run: | - mkdir -p ../../coverage-report - coverage json -o ../../coverage-report/coverage.json - coverage html -d ../../coverage-report/htmlcov - coverage xml -o ../../coverage-report/coverage.xml + mkdir -p ../coverage-report + coverage json -o ../coverage-report/coverage.json + coverage html -d ../coverage-report/htmlcov + coverage xml -o ../coverage-report/coverage.xml - name: Upload coverage report if: matrix.python-version == '3.12' && matrix.netbox-version == 'v4.5.3' diff --git a/.gitignore b/.gitignore index 8452b5f..6451156 100644 --- a/.gitignore +++ b/.gitignore @@ -22,6 +22,7 @@ venv/ .coverage.* htmlcov/ coverage.xml +performance/baselines/*.json # Ruff / linting .ruff_cache/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 543128c..babd209 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -20,6 +20,12 @@ repos: - id: mixed-line-ending args: [--fix=lf] + # GitHub Actions security audit. The lint workflow runs this same hook. + - repo: https://github.com/zizmorcore/zizmor-pre-commit + rev: v1.29.0 + hooks: + - id: zizmor + # Python code quality - repo: https://github.com/astral-sh/ruff-pre-commit rev: v0.16.0 diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000..c3495de --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,90 @@ + + +# Interface Name Rules + +This context describes naming rules and the related interface topologies they manage. + +## Language + +**Interface family**: +A set of related interfaces that represents one physical port and its breakout channels in either a flat or channelized topology. +_Avoid_: Channel group, breakout set + +**Flat breakout family**: +An interface family whose channels are sibling interfaces without a physical parent relationship. +_Avoid_: Flat channels + +**Channelized family**: +An interface family with one physical parent that declares channel capacity and zero or more channel interfaces bound to it. The family remains channelized when it is incomplete. +_Avoid_: Parented breakout + +**Installed interface family**: +An interface family represented by the current NetBox interface rows. +_Avoid_: Persisted family + +**Prospective interface family**: +An intended interface family described before its interface rows exist. +_Avoid_: Predicted family + +**Family plan**: +An immutable description of one intended interface-family operation, its required live state, and its expected member outcomes. +_Avoid_: Rename instructions + +**Installed family plan**: +A family plan that includes a validated snapshot of installed NetBox rows and can be executed after live-state revalidation. +_Avoid_: Executable prediction + +**Family plan set**: +An immutable batch that contains exactly one family plan for each interface family in an operation. +_Avoid_: Plan list + +**Family rename**: +A change to the names of existing interface-family members that does not change their topology. +_Avoid_: Family conversion + +**Structural family change**: +A change that creates an interface family or changes it between flat and channelized topologies. +_Avoid_: Family rename + +**Flat-to-channelized conversion**: +A structural family change that rebuilds one installed flat breakout family as a channelized family, keeping the physical interface row and moving its logical identity onto the channel that takes its name. It is always an explicit operator action, never a side effect of applying a rule. +_Avoid_: Migration, upgrade + +**Out-of-band rename**: +A change to an interface name made by an actor other than this plugin, such as an operator edit or an import. A family plan is stale when one arrives between planning and execution. +_Avoid_: External rename, manual fix + +**Engine facade**: +The compatibility surface downstream callers import. It selects rules, builds template variables and decides which interfaces an automatic path may touch on a run; it holds no family discovery, planning or mutation. +_Avoid_: Engine layer, core + +**Stored rule pattern**: +An operator-provided RE2 expression saved on an Interface Name Rule. It matches the complete module type model or current device-interface name, depending on the rule mode. +_Avoid_: Python regex, partial regex + +**Unsupported topology**: +An interface-family topology that the active NetBox data model cannot represent. +_Avoid_: Legacy fallback + +**Blocked family operation**: +A valid requested family change that current device state prevents, such as when a required name is already occupied. +_Avoid_: Failed family operation + +**Automatic naming signal path**: +The complete path from a NetBox model save, through the committed callback, to the resulting interface-family rows. +_Avoid_: Signal handler performance + +**Signal-path performance baseline**: +Test-suite measurements of the automatic naming signal path before an implementation change. The baseline includes query counts, PostgreSQL work profiles, scaling behavior, and repeated machine-time samples collected on the hardware used for the after measurement. Shared-runner elapsed time is not a recurring CI metric. +_Avoid_: Runtime limit, CI speed + +**PostgreSQL work profile**: +A test-suite record of the database work performed by one automatic naming scenario. It includes normalized statements, execution plans, rows, loops, buffer access, temporary storage, and WAL activity. +_Avoid_: Standalone query benchmark + +**Implementation performance comparison**: +A one-time comparison produced by running the same test-suite performance scenarios before and after an implementation change on the same hardware. It includes database work, uninstrumented wall time, process CPU time, and raw timing samples. It is review evidence, not a recurring CI gate. +_Avoid_: Performance CI diff --git a/REUSE.toml b/REUSE.toml index 4772834..f48bb26 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -24,6 +24,7 @@ path = [ "README.md", "CONTRIBUTING.md", "docs/**", + "performance/**", ] SPDX-FileCopyrightText = "2025 Marcin Zieba " SPDX-License-Identifier = "Apache-2.0" diff --git a/conftest.py b/conftest.py new file mode 100644 index 0000000..e12d921 --- /dev/null +++ b/conftest.py @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Root pytest configuration for shared-host parallelism.""" + +MAX_PARALLEL_WORKERS = 8 + + +def pytest_xdist_auto_num_workers(config): + """Cap the worker count that pytest-xdist detects for ``-n auto``.""" + from xdist.plugin import pytest_xdist_auto_num_workers as detected_num_workers + + return min(detected_num_workers(config), MAX_PARALLEL_WORKERS) diff --git a/docs/adr/0001-interface-family-operation-atomicity.md b/docs/adr/0001-interface-family-operation-atomicity.md new file mode 100644 index 0000000..f057f85 --- /dev/null +++ b/docs/adr/0001-interface-family-operation-atomicity.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Separate structural atomicity from partial rename handling + +Each family operation owns one database transaction, which isolates it from other families and rolls back unexpected failures. Structural creation and conversion commit or roll back the complete topology together. An installed-family rename uses nested savepoints for expected member-level validation and name collisions: the parent must succeed first, but a blocked channel leaves only that channel unchanged while successful channel renames commit. This preserves useful names without permitting a partial structural topology. diff --git a/docs/adr/0002-revalidate-family-plans-before-execution.md b/docs/adr/0002-revalidate-family-plans-before-execution.md new file mode 100644 index 0000000..507fd74 --- /dev/null +++ b/docs/adr/0002-revalidate-family-plans-before-execution.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Revalidate family plans before execution + +An interface-family plan is never trusted across a change to its interfaces. Execution revalidates identity, names, membership, and topology against live rows, then rejects a stale plan instead of silently constructing a replacement or applying only the members that still match. Interactive apply constructs a fresh plan from live rows rather than executing the earlier preview. This keeps the executed change equal to the selected plan and prevents concurrent edits from being overwritten. diff --git a/docs/adr/0003-profile-database-work-on-the-signal-path.md b/docs/adr/0003-profile-database-work-on-the-signal-path.md new file mode 100644 index 0000000..41dbc85 --- /dev/null +++ b/docs/adr/0003-profile-database-work-on-the-signal-path.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Profile database work on the signal path + +The test suite records a PostgreSQL work profile for each representative automatic naming scenario before the interface-family refactor begins. The profile comes from the real Django signal path and records statement counts, normalized SQL, execution plans, actual rows and loops, buffer access, temporary storage, WAL activity, and scaling behavior. PostgreSQL collects the execution data with node timing disabled. The test suite also runs a separate uninstrumented timing pass and records raw samples, wall time, and process CPU time. Run both passes before the refactor and rerun them afterward on the same hardware with the same PostgreSQL version, NetBox revision, fixtures, planner settings, and statistics. Retain the result as review evidence. CI does not use the machine-time comparison as a recurring regression gate. diff --git a/docs/adr/0004-use-immutable-family-plan-boundaries.md b/docs/adr/0004-use-immutable-family-plan-boundaries.md new file mode 100644 index 0000000..0419804 --- /dev/null +++ b/docs/adr/0004-use-immutable-family-plan-boundaries.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Use immutable family-plan boundaries + +The interface-family module exposes immutable installed and prospective plan sets. Each set contains exactly one plan per family. The installed adapter owns bulk ORM loading, family discovery, and row snapshots. The prospective adapter owns rowless template inputs. Callers provide semantic operation roots instead of querysets or discovered members. Only plans that carry live row snapshots can execute, and execution returns explicit family and member outcomes. Existing engine entry points remain thin adapters to this boundary. diff --git a/docs/adr/0005-execute-each-family-in-its-own-transaction.md b/docs/adr/0005-execute-each-family-in-its-own-transaction.md new file mode 100644 index 0000000..dfcf112 --- /dev/null +++ b/docs/adr/0005-execute-each-family-in-its-own-transaction.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Execute each family in its own transaction + +Each installed family executes in its own database transaction. Execution locks member rows in stable primary-key order and revalidates the plan after locking. Structural changes roll back the complete family. Existing-family renames use child savepoints after the parent succeeds, so one blocked child does not undo unrelated member renames. A blocked family does not roll back other families in the plan set. Target-name checks run after locking, and NetBox's device-and-name uniqueness constraint remains the final collision guard without a device-wide interface lock. diff --git a/docs/adr/0006-make-engine-a-family-facade.md b/docs/adr/0006-make-engine-a-family-facade.md new file mode 100644 index 0000000..5551ce2 --- /dev/null +++ b/docs/adr/0006-make-engine-a-family-facade.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Make the engine a family facade + +The existing engine entry points remain as compatibility adapters. Shared rule selection, variable construction, and template evaluation move into lower-level modules. A dedicated family package owns immutable domain values, installed and prospective adapters, planning, revalidation, locking, execution, conversion, and deferred reconciliation. The family package never imports the engine facade. This dependency direction prevents circular imports and removes the old private family implementation when the replacement is complete. diff --git a/docs/adr/0007-report-an-unsupported-topology-as-a-family-outcome.md b/docs/adr/0007-report-an-unsupported-topology-as-a-family-outcome.md new file mode 100644 index 0000000..745796d --- /dev/null +++ b/docs/adr/0007-report-an-unsupported-topology-as-a-family-outcome.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Report an unsupported topology as a family outcome + +The family package owns the probe that decides whether the active NetBox data model can hold a channelized family, and it probes the Interface model rather than comparing versions. A rule that describes a topology the release cannot hold produces a plan whose precondition is `unsupported` and an outcome that says so. Callers map that outcome to their own reporting; they never branch on the NetBox version and never fall back to a different topology. A caller-side version check would let one entry point build a flat family that another refuses, and silently give the operator a topology the rule did not ask for. diff --git a/docs/adr/0008-plan-prediction-and-preview-prospectively.md b/docs/adr/0008-plan-prediction-and-preview-prospectively.md new file mode 100644 index 0000000..a5203b7 --- /dev/null +++ b/docs/adr/0008-plan-prediction-and-preview-prospectively.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Plan prediction and preview prospectively + +Rowless prediction and the interactive preview build prospective family plans through the same naming rules installed execution follows. One shared target module names every family, so an installed plan, a prospective plan and a structural plan cannot spell the same family differently. A prospective plan carries names rather than row snapshots and the executors reject one by type, so a preview can never become an executable snapshot and prediction can describe a family NetBox has not created yet. Interactive apply ignores the preview object and plans again from live rows, so a change between the two is replanned rather than written over. A plan the live topology blocks previews and predicts no change, because the apply path would build nothing there either. diff --git a/docs/adr/0009-apply-a-rule-batch-family-by-family.md b/docs/adr/0009-apply-a-rule-batch-family-by-family.md new file mode 100644 index 0000000..f64b241 --- /dev/null +++ b/docs/adr/0009-apply-a-rule-batch-family-by-family.md @@ -0,0 +1,13 @@ +--- +status: accepted +--- + +# Apply a rule batch family by family + +Retroactive apply plans every family a rule intends on each module and executes each family on its own. A module's installed families claim their members first; what is left over is planned as the family the rule would build there, so no interface belongs to two plans and two bases that intend one family's names build it once. Execution returns one explicit family result per family, and the Apply view and the background job read their counts and their skips from those results instead of a mutable conflict list. A family that is blocked, stale or unnamable costs the batch only itself: the batch keeps planning and executing the families after it. + +One batch shares what its modules share. The module rows come with the relations template resolution dereferences, and one module type's interface templates are read once however many modules carry it. Retroactive apply also reads every module's interfaces in one query. Virtual-chassis reapplication is a batch of the same kind, but it reaches the families through the installation entry point, so it still reads one module's interfaces at a time and still renames a leftover interface outside the family executor. Contracting that entry point onto the same batch was the remaining step; ADR 0011 completes it. + +Every rename retroactive apply performs goes through the locked, revalidated family executor, which costs a savepoint pair and a row lock per family. That price buys stale-plan rejection and per-family isolation on a path that had neither. + +A module's plain interface count decides whether an earlier flat breakout already expanded it, because converting one sibling into a family parent would strand the others. A channel belongs to the parent that declares it, so it never counts toward that surplus: counting one would stop a module's second port from gaining a family of its own. diff --git a/docs/adr/0010-convert-flat-families-through-conversion-plans.md b/docs/adr/0010-convert-flat-families-through-conversion-plans.md new file mode 100644 index 0000000..964d136 --- /dev/null +++ b/docs/adr/0010-convert-flat-families-through-conversion-plans.md @@ -0,0 +1,15 @@ +--- +status: accepted +--- + +# Convert flat families through conversion plans + +The family package plans and executes one immutable flat-to-channelized conversion plan per family, beside the planners that install and rename families. A plan carries the row snapshots the ch-0 split needs and nothing else: the base row that becomes the parent, the siblings that become channels 2..N, and the parent name the rule resolves. Execution locks the planned rows, compares them against those snapshots, and refuses a family whose identity, names, membership or topology moved since the scan. The scan itself performs each conversion inside a savepoint it always rolls back, so a candidate reports the reason NetBox would refuse the family rather than a guess at its rules. + +Conversion recovers its families through the base names the rename path recovers, so the two cannot drift on which rows belong to which family, a family named before a virtual-chassis renumber included. It identifies a family by its ch-0 row, the way the flat apply that installed it named that row, and then takes whatever the module still carries for the rest of the family. A family with a gap is offered and refused, naming the row it is missing, because dropping it from the page would read as "nothing here to convert" and hide an edit the operator needs to see. A sibling that already belongs to another parent is refused the same way rather than taken from that family. The parent takes the name the rule resolves for the module now, which is the name an apply would give it; the channels keep the names they carry, because converting retypes those rows in place. + +A family the plan can already refuse carries the refusal as a plan precondition, so the scan reports it without a dry run and execution rejects it without locking a row. Everything a snapshot cannot settle stays in the locked preflight beside NetBox's own validation. + +Conversion returns one explicit family outcome per family. The Apply view and the background job read their converted and skipped counts from those outcomes, so no conversion tuple and no mutable conflict list crosses the boundary. The candidate an operator confirms carries its names, roles and blocking reason from the plan; it keeps the live module and ch-0 rows only so the page can link to them. + +The addresses and FHRP group assignments on the ch-0 row move onto the new channel through model saves, one row at a time. A queryset update relocated them with no validation, no signal and no changelog entry, so a device's history showed the family rewritten and the objects on it moved by nobody. They are objects an operator owns and audits, and they are now written to the standard the rest of the family write already meets. diff --git a/docs/adr/0011-contract-the-engine-to-a-family-facade.md b/docs/adr/0011-contract-the-engine-to-a-family-facade.md new file mode 100644 index 0000000..f798ae0 --- /dev/null +++ b/docs/adr/0011-contract-the-engine-to-a-family-facade.md @@ -0,0 +1,13 @@ +--- +status: accepted +--- + +# Contract the engine to a family facade + +The engine no longer holds a family implementation. Automatic installation, prediction, interactive preview and apply, bulk operations, virtual-chassis reapplication, device-level renaming, conversion and deferred reconciliation all reach the family package, and every rename, creation and rewrite goes through one locked, revalidating executor. This completes the direction set in ADR 0006 and the remaining step named in ADR 0009. + +What stays in the engine is what was never about families: which rule wins, how a rule's variables are built, and the raw-name idempotency guard that decides which interfaces an automatic path may touch on this run. The guard runs while it can still see every interface a rule claimed, before two of them that intend one family are collapsed into it, because an ambiguous pair is only visible while both are present. The device-level path keeps its own rule selection and its claim bookkeeping for the same reason, and asks the package for lockstep names directly: a device rule never builds a family, so its channel count says nothing about one it finds. + +Module installation now reads a module's interfaces once and plans the whole module, rather than planning installed families, executing them, and then re-reading what was left. Device-level renaming gained revalidation and row locking it never had. + +The private helpers the engine used are deleted rather than wrapped, so no second implementation can drift from the package. The family package does not import the engine facade, and the shared dependencies point only at rule selection and naming. diff --git a/docs/adr/0012-execute-stored-rule-patterns-with-re2.md b/docs/adr/0012-execute-stored-rule-patterns-with-re2.md new file mode 100644 index 0000000..846adad --- /dev/null +++ b/docs/adr/0012-execute-stored-rule-patterns-with-re2.md @@ -0,0 +1,7 @@ +--- +status: accepted +--- + +# Execute stored rule patterns with RE2 + +Stored rule patterns are operator input that can execute on the automatic naming signal path, so the plugin compiles and executes them only with RE2 and never falls back to Python `re`. This deliberately rejects lookaround, backreferences, atomic groups and other Python-only syntax in exchange for linear matching time and bounded memory use. An upgrade migration audits existing rows for syntax and Unicode semantic differences before they can run under the new engine. Fixed internal expressions remain on Python `re` because they are not stored rule patterns. diff --git a/docs/configuration.md b/docs/configuration.md index 205828d..027193a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -9,7 +9,7 @@ Navigate to **Plugins → Interface Name Rules → Add** or use the REST API. | Field | Required | Description | |-------|----------|-------------| | Module Type | Conditional | The module type that triggers this rule (required when Regex Mode is off) | -| Module Type Pattern | Conditional | Regex pattern matched against module type model name via `re.fullmatch()` (required when Regex Mode is on) | +| Module Type Pattern | Conditional | RE2 pattern matched against the complete module type model name (required when Regex Mode is on) | | Regex Mode | No | When enabled, match by pattern instead of exact module type FK | | Parent Module Type | No | Restrict to modules inside this parent (e.g., converter) | | Device Type | No | Restrict to devices of this type | @@ -28,7 +28,17 @@ When multiple rules could match, the most specific one wins. Exact FK matches al 4. Module type only (universal) **Tier 2 — Regex pattern match (fallback, longest pattern first):** -5–8. Same four specificity levels, but `module_type_pattern` is matched via `re.fullmatch()` against the installed module type's model name. When multiple patterns match at the same level, the longest pattern is preferred. +5–8. Same four specificity levels, but `module_type_pattern` is matched against the complete installed module type model name. When multiple patterns match at the same level, the longest pattern is preferred. + +### RE2 Pattern Syntax + +The plugin compiles and executes every stored rule pattern with +[RE2](https://github.com/google/re2/wiki/syntax). RE2 guarantees bounded memory +use and linear matching time. It does not support Python-only features that +require backtracking, including lookaround, backreferences, and atomic groups. +Use `\z` instead of Python's `\Z` end-of-text escape. The `\d`, `\s`, and `\w` +classes match ASCII characters. Use an RE2 Unicode property such as `\p{L}` when +the rule must match Unicode letters. ## NetBox Module Interface Templates diff --git a/docs/examples.md b/docs/examples.md index adb1a07..863f82b 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -176,22 +176,24 @@ disabled rule converts nothing, on this page or in the background job. Conversion is only ever performed from that page, by an operator who confirmed it: **Apply** renames, it never rewrites a family. Each family gets its own -verdict before anything is written, produced by performing the whole conversion -inside a transaction that is rolled back again — so a family that NetBox would -reject is reported with NetBox's own reason (a cabled sibling, an occupied -parent name, a missing sibling, a sibling already channelized, a sibling that is -already a channel of another parent) instead of being half converted. Selecting +verdict before anything is written. A family the plugin can already refuse from +what it knows, such as a module that no longer carries the whole family, is reported +without touching a row; every other verdict comes from performing the whole +conversion inside a transaction that is rolled back again, so a family NetBox +would reject is reported with NetBox's own reason (a cabled sibling, an occupied +parent name, a sibling already channelized, a sibling that is already a channel +of another parent) instead of being half converted. Selecting a blocked family converts the others and skips that one. What the conversion does to the ch-0 row, per family: | Stays on the physical row (same interface ID) | Moves to the new channel 1 interface | |---|---| -| cable, interface type, module link, `mark_connected` | IP addresses, FHRP group assignments, untagged/tagged VLANs, 802.1Q mode, MTU, description, tags | +| cable, interface type, module link, `mark_connected` | interface VRF, IP addresses, FHRP group assignments, untagged/tagged VLANs, 802.1Q mode, MTU, description, tags | Custom field values are copied to the channel rather than moved, because they can describe either the port or the link. The remaining siblings are retyped in -place, so their own addresses, descriptions and tags — and their interface IDs — +place, so their own addresses, descriptions, tags, and interface IDs survive. The caveat worth reading twice: the ch-0 interface keeps its ID and becomes the @@ -205,7 +207,9 @@ and says so when more are waiting, and it refuses a confirmation naming more tha that many rather than converting part of it. **Convert as Background Job** runs every convertible family of the rule and is not capped. -On NetBox 4.6 and older no family is offered and no conversion section is shown. +On a NetBox release that cannot model channelized interfaces, each installed flat family is shown as +unsupported without a conversion checkbox. A direct conversion request returns the same explicit +unsupported family outcome and changes no row. ### Partial breakout repair diff --git a/docs/index.md b/docs/index.md index 6171ab5..a217a51 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,7 +19,7 @@ automatically apply renaming rules based on configurable templates. - **Template variables** — `{slot}`, `{bay_position}`, `{bay_position_num}`, `{channel}`, etc. - **Arithmetic expressions** — `{8 + ({parent_bay_position} - 1) * 2 + {sfp_slot}}` - **Breakout support** — create multiple channel interfaces from a single port -- **Regex pattern matching** — match module types by regex pattern (e.g., `QSFP-DD-400G-.*`) to cover entire product families with a single rule; exact FK match takes priority over regex +- **Bounded regex pattern matching** — match module types with RE2 patterns (e.g., `QSFP-DD-400G-.*`) to cover entire product families with a single rule; exact FK match takes priority over regex - **Scoping** — rules can target specific device types, parent module types, platforms, or be universal - **Build Rule tester** — interactive form to preview name output and test against installed interfaces before saving - **Apply Rules** — batch rename existing interfaces with live preview and background job support diff --git a/docs/installation.md b/docs/installation.md index 0e80c3c..92dbc35 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -21,6 +21,18 @@ PLUGINS = ["netbox_interface_name_rules"] ## Run Database Migrations +The migration audits every existing nonempty **Module Type Pattern** before the +plugin starts executing stored patterns with RE2. It stops when RE2 cannot +compile a pattern or when a Python shorthand can change its Unicode behavior. +The latter check includes `\d`, `\s`, `\w`, word boundaries, case-insensitive +matching, and POSIX character classes. The migration lists the affected +Interface Name Rule IDs and does not modify any rule. It also stops on Python +counted repeats that RE2 would treat as literal text, such as `{,3}` or `{01}`. +Rewrite those patterns with explicit +[RE2 syntax](https://github.com/google/re2/wiki/syntax), then run the migration +again. For example, use `[0-9]` for ASCII digits, `\p{L}` for Unicode letters, +and `{0,3}` for a repeat with an omitted Python lower bound. + ```bash cd /opt/netbox/netbox python manage.py migrate diff --git a/docs/template-variables.md b/docs/template-variables.md index 56c5412..2911263 100644 --- a/docs/template-variables.md +++ b/docs/template-variables.md @@ -184,19 +184,23 @@ a `parent_name_template` set instead offers each such family for conversion on * Preview & Apply**, where the operator confirms it per family; a blank parent template offers nothing, because a flat family's ch-0 interface is the base and has nowhere else to go. -Every family is preflighted by performing the conversion inside a transaction that is rolled back, -so the verdict carries NetBox's own reason for refusing it — a cabled sibling, an occupied parent -name, a missing sibling — and a refused family is never half converted. +The plugin rejects missing members locally before a conversion transaction starts. Its local +preflight also refuses a stale family, an occupied parent name, a sibling that already belongs to +another channel family, and a cabled sibling. Each of those verdicts is the plugin's own, and no +row is written. Only a family that passes preflight reaches NetBox's own rules. The plugin runs +that rewrite inside a transaction and rolls it back when NetBox refuses a row or a name collides, +so the verdict then carries NetBox's own reason. A preview rolls back a successful rewrite the same +way. The plugin never half converts a refused family. Converting keeps the physical row: the ch-0 interface keeps its interface ID, cable, type, module -link and `mark_connected`, and becomes the parent. Its IP addresses, FHRP group assignments, +link and `mark_connected`, and becomes the parent. Its interface VRF, IP addresses, FHRP group assignments, untagged/tagged VLANs, 802.1Q mode, MTU, description and tags move to a newly created channel 1 interface that takes over its name; custom field values are copied to it. The remaining siblings are retyped in place, keeping their own interface IDs. Automation keyed on the ch-0 interface ID -addresses the parent afterwards, not the channel that carries its name. +addresses the parent afterward, not the channel that carries its name. -On NetBox 4.6 and older no family is offered and conversion reports that this release cannot model -channels. +On a NetBox release that cannot model channels, each flat family is shown as unsupported without a +conversion action, and direct conversion reports the same explicit family outcome. A family installed by a rule that uses `{base}` carries the raw name it was named with, which on a module type using the `{vc_position}` template token may predate a chassis position change (see diff --git a/netbox_interface_name_rules/engine.py b/netbox_interface_name_rules/engine.py index 5c5c4e9..db32ad2 100644 --- a/netbox_interface_name_rules/engine.py +++ b/netbox_interface_name_rules/engine.py @@ -6,255 +6,54 @@ after Django is fully initialised. """ -import ast -import contextlib -import copy import logging -import re -import threading -from collections import defaultdict, namedtuple +from collections import defaultdict from django.core.exceptions import ValidationError -from django.db import IntegrityError, transaction -from django.db.models import Aggregate, F, TextField, Value -from django.db.models.functions import Cast, Coalesce, Concat, Length +from django.db import IntegrityError -from .choices import BreakoutModeChoices +from . import family as family_ops +from . import naming, rule_selection +from .family import targets as family_targets +from .family import template_names as family_template_names +from .regex_safety import compile_module_type_pattern logger = logging.getLogger(__name__) -# In-process cache of the enabled InterfaceNameRule set, keyed by a cheap fingerprint -# of that set. find_matching_rule() is called once per module row when a device's -# module-sync table is rendered, and each call used to run a DB query per scope -# candidate per tier; loading the (small) rule set once and matching in memory removes -# that per-row query storm. The fingerprint (see _enabled_rules_version) is re-read once -# per call and the set reloaded only when it changes, so it self-invalidates on any -# create/delete/edit — including raw bulk ``.update()`` of any matching field and SET_NULL -# cascades that bypass auto_now, and across test transactions — with no signal wiring. -# A reload publishes the new set by rebinding this name to a fresh dict in one atomic assignment -# (see _get_enabled_rules), so a concurrent reader on another worker thread sees either the whole -# old set or the whole new one — never exact/regex/memo torn across two versions. -_RULE_CACHE = {"version": None, "exact": (), "regex": (), "memo": {}} - -# Per-version cap on the find_matching_rule memo. A long-lived worker that sees many distinct -# (module_type, scope) contexts under a stable rule set would otherwise grow it without bound; -# on overflow the memo is cleared wholesale and rebuilt lazily. The cap is far above the number -# of contexts in any single module-sync render, so it never churns mid-render. -_MEMO_MAX = 4096 - -# Sentinel for the memo read. A single ``memo.get(sig, _MEMO_MISS)`` is one atomic dict lookup, so a -# concurrent ``memo.clear()`` at the cap can't wedge between a membership test and the subscript the -# way ``if sig in memo: return memo[sig]`` could — that compound read can raise a sporadic KeyError -# under threaded workers sharing one per-version memo. A real "no rule matched" is memoized as None, -# so the miss sentinel must be a distinct object, not None. -_MEMO_MISS = object() - -# Per-thread pin depth (see pinned_rule_cache). While > 0 on the current thread, _get_enabled_rules -# trusts the loaded set without re-reading the fingerprint, so an internal batch that wraps its loop -# turns N per-call fingerprint queries into one. Thread-local so concurrent requests don't affect -# each other; the depth defaults to 0, so any path that does not pin (the per-row signal handler, -# single applies) re-reads the fingerprint on every call exactly as before. -_pin = threading.local() - - -@contextlib.contextmanager -def pinned_rule_cache(): - """Pin the enabled-rule set for the duration of the block, skipping the per-call fingerprint query. - - find_matching_rule() normally re-reads a cheap fingerprint on every call to detect rule-set - changes. An internal batch that makes many lookups against an unchanging rule set — e.g. - _apply_rules_for_device_deferred(), which re-applies rules to every module on a device after a - virtual-chassis change — can wrap its loop in this context manager so the set is loaded and - fingerprinted once and reused for every call inside the block, turning N fingerprint queries - into one:: - - with pinned_rule_cache(): - for module in modules: - apply_interface_name_rules(module, module.module_bay, force_reapply=True) - - This is an INR-internal helper: only code that owns the batch loop pins, so it never becomes a - cross-plugin dependency. Other plugins integrate through the ``predict_module_interface_names`` - signal, which is dispatched one row at a time — so an external caller neither can nor needs to - pin, and INR exposes no batch API across the plugin boundary. - - Scope is explicit and per-thread: outside the block (and in any other thread/request) the normal - self-invalidating behaviour is unchanged, so there is no global staleness window. Nesting is - safe — only the outermost block manages the pin. Priming is lazy: the first find_matching_rule - inside the block does the one real fingerprint read + reload and captures that rule set into - thread-local state; the rest reuse that *snapshot* — so a block that makes no lookups does no - query at all, and a concurrent request reloading the shared cache on another thread cannot switch - this batch's rule set mid-loop. The loop may rename interfaces, but it must not edit rules: edits - to InterfaceNameRule made *inside* the block are not observed until it exits. - """ - depth = getattr(_pin, "depth", 0) - _pin.depth = depth + 1 - if depth == 0: - _pin.primed = False # the first lookup inside primes the cache (lazily — an empty block stays query-free) - try: - yield - finally: - _pin.depth -= 1 - if _pin.depth == 0: - # Release the per-thread snapshot so the next block re-primes from the live cache. - _pin.primed = False - for attr in ("exact", "regex", "memo"): - _pin.__dict__.pop(attr, None) - - -def _compile_pattern(pattern): - """Compile a regex *pattern* once, returning the compiled object or None for an invalid pattern. - - A None result is skipped at match time, mirroring the previous per-match ``try/except re.error`` - without recompiling the pattern on every lookup. - """ - try: - return re.compile(pattern) - except re.error: - return None - -# Rule columns that affect matching or output, in a fixed order. The enabled-set fingerprint -# (see _enabled_rules_version) is an md5 over these for every enabled rule, so it changes on ANY -# edit that could change a lookup result. ``description`` is excluded — it is operator notes that -# never affect matching. ``id`` anchors each row to its identity, so a compensating swap between -# two rules (which leaves count + column sums unchanged) still changes the hash. -_VERSION_COLUMNS = ( - "id", - "module_type_id", - "module_type_is_regex", - "module_type_pattern", - "parent_module_type_id", - "device_type_id", - "platform_id", - "name_template", - "parent_name_template", - "breakout_mode", - "channel_count", - "channel_start", - "applies_to_device_interfaces", -) - - -class _Md5OrderedStringAgg(Aggregate): - """``md5(string_agg(, ORDER BY id))`` expressed as one ORM aggregate. - - Built through the ORM rather than a hand-formatted SQL string, so the table/column identifiers are - quoted by Django's compiler and the delimiter is a bound parameter — there is no string-interpolated - SQL to audit for injection. The template is ours, so it does not depend on ``StringAgg``'s Python - signature, which differs across the Django 5.x–6.x versions in CI. ``ORDER BY id`` makes string_agg - deterministic — a stable row order yields a stable hash for an unchanged set. - """ +def pinned_rule_cache(): + """Return the lower-level rule cache pinning context.""" + return rule_selection.pinned_rule_cache() - function = "STRING_AGG" - template = "MD5(%(function)s(%(expressions)s ORDER BY id))" - output_field = TextField() +def find_matching_rule(module_type, parent_module_type, device_type, platform=None): + """Delegate rule selection while preserving the engine entry point.""" + return rule_selection.find_matching_rule(module_type, parent_module_type, device_type, platform) -def _version_row_signature(): - """Build the per-rule text signature expression for the fingerprint. - Each matching/output column is cast to text and emitted length-prefixed as ``:``. - That makes the row encoding self-delimiting: distinct column tuples can never serialize to the same - string, even if a text column (name_template / module_type_pattern) contains digits, a colon, or the - control characters a plain separator scheme would rely on being absent. A nullable FK is coalesced to - '' (length 0), so null stays distinct from any value while keeping the column's slot. - """ - empty = Value("", output_field=TextField()) - colon = Value(":", output_field=TextField()) - parts = [] - for column in _VERSION_COLUMNS: - cast = Cast(F(column), output_field=TextField()) - value = Coalesce(cast, empty, output_field=TextField()) if column.endswith("_id") else cast - parts.append(Cast(Length(value), output_field=TextField())) - parts.append(colon) - parts.append(value) - return Concat(*parts, output_field=TextField()) - - -# The signature expression is constant, so build it once and reuse it across calls. -_ROW_SIGNATURE = _version_row_signature() - - -def _enabled_rules_version(): - """Return a deterministic content fingerprint of the enabled-rule set. - - Computed server-side as an md5 over every enabled rule's matching/output columns, row-ordered - by pk, so the fingerprint changes on ANY edit that could change a lookup — including a raw bulk - ``.update()`` of a text field (name_template / module_type_pattern) or a boolean - (module_type_is_regex), which an aggregate of counts/sums cannot see, and a compensating edit - that keeps the column sums constant. It is one query returning a single 32-char hash, so it - stays cheap enough to re-read on every call. - - Each column is length-prefixed (see _version_row_signature), so the per-row encoding is - self-delimiting and rows concatenate unambiguously — distinct rule sets cannot collide even if a - text column contains arbitrary bytes. A nullable FK renders as the empty string (a real id never - does), keeping null distinct from any value. The empty set aggregates to NULL → coalesced to a - stable empty fingerprint, so "no enabled rules" is fixed. - """ - from .models import InterfaceNameRule +def _extract_trailing_digits(value: str) -> str: + """Delegate trailing-digit extraction while preserving the engine helper.""" + return naming._extract_trailing_digits(value) - return InterfaceNameRule.objects.filter(enabled=True).aggregate( - fingerprint=Coalesce( - _Md5OrderedStringAgg(_ROW_SIGNATURE, Value("", output_field=TextField())), - Value("", output_field=TextField()), - ) - )["fingerprint"] +def _resolve_bay_position(module_bay): + """Delegate bay-position resolution while preserving the engine helper.""" + return naming._resolve_bay_position(module_bay) -def _get_enabled_rules(): - """Return ``(exact_rules, regex_rules, memo)``, reloading only when the rule set changes. - ``exact_rules`` are ordered by ``(module_type__model, pk)`` to mirror the model's default - ordering (so an in-memory ``first match`` equals the previous ``.first()``). ``regex_rules`` - is a tuple of ``(compiled_pattern, rule)`` pairs, pre-sorted once by ``(-pattern length, pk)`` - and with each pattern compiled once, so the regex tier neither re-sorts nor recompiles per - call. ``memo`` caches find_matching_rule results for the current rule-set version. +def _resolve_slot(module_bay, bay_position_num, parent_bay_position): + """Delegate slot resolution while preserving the engine helper.""" + return naming._resolve_slot(module_bay, bay_position_num, parent_bay_position) - Inside a ``pinned_rule_cache()`` block on this thread, once the set has been primed (by the first - lookup in the block) the fingerprint query is skipped and the snapshot captured at prime time is - returned — never the live ``_RULE_CACHE``, which another thread may reload mid-block. - """ - global _RULE_CACHE - pinned = getattr(_pin, "depth", 0) > 0 - if pinned and getattr(_pin, "primed", False): - # Serve the snapshot captured when this block primed, not the shared cache: a concurrent - # request on another thread may reload _RULE_CACHE to a different version while we iterate, and - # a pinned batch must match every item against one consistent rule set. - return _pin.exact, _pin.regex, _pin.memo +def build_variables(module_bay, device=None): + """Delegate naming-variable construction while preserving the engine entry point.""" + return naming.build_variables(module_bay, device=device) - from .models import InterfaceNameRule - # Read the module global exactly once. A reload below publishes the new set by rebinding - # _RULE_CACHE to a brand-new dict (a single atomic name assignment) rather than mutating this - # one in place, so this local is a consistent snapshot: exact/regex/memo can never be torn - # across two rule-set versions even if another thread reloads between the reads at the end. - cache = _RULE_CACHE - version = _enabled_rules_version() - if cache["version"] != version: - rules = list(InterfaceNameRule.objects.filter(enabled=True).order_by("module_type__model", "pk")) - exact = tuple(r for r in rules if not r.module_type_is_regex) - regex_rules = sorted( - (r for r in rules if r.module_type_is_regex), - key=lambda r: (-len(r.module_type_pattern or ""), r.pk), - ) - regex = tuple((_compile_pattern(r.module_type_pattern), r) for r in regex_rules) - # Publish the whole new version atomically: a reader either sees the old dict or this one, - # never a mix of the two. Last writer wins; a concurrent reload to the same version just - # rebuilds redundantly, never corrupts. - cache = {"version": version, "exact": exact, "regex": regex, "memo": {}} - _RULE_CACHE = cache - if pinned: - # Pin this thread to the freshly-resolved set for the rest of the block. The tuples are - # immutable and the memo is COPIED into thread-local state — not aliased — so the pinned batch - # neither shares nor races the global memo: another thread clearing the shared memo at the cap - # can't evict our warmed entries (or wedge a KeyError) mid-loop, and entries we add stay private. - _pin.exact = cache["exact"] - _pin.regex = cache["regex"] - _pin.memo = dict(cache["memo"]) - _pin.primed = True - return _pin.exact, _pin.regex, _pin.memo - return cache["exact"], cache["regex"], cache["memo"] +def evaluate_name_template(template: str, variables: dict) -> str: + """Delegate template evaluation while preserving the engine entry point.""" + return naming.evaluate_name_template(template, variables) def _get_parent_module_type(module_bay): @@ -271,32 +70,13 @@ def _get_parent_module_type(module_bay): def supports_channelization(): - """Return True when this NetBox models channelized subinterfaces (NetBox 4.7+). - - Probed from the Interface model rather than a version comparison, so a backport or a - development build is detected by what it actually provides. - """ - from dcim.models import Interface - from django.core.exceptions import FieldDoesNotExist - - try: - Interface._meta.get_field("channel_id") - except FieldDoesNotExist: - return False - return True # pragma: no cover - only reachable on a NetBox that models channelization + """Delegate the channelization capability check while preserving the engine entry point.""" + return family_ops.supports_channelization() def _vc_position_re(): - """Return NetBox's ``{vc_position}`` template-token regex, or None on a release without the token. - - Imported inside the function so the probe reads the module as it stands at call time, matching - ``supports_channelization()``'s style. - """ - try: - from dcim.constants import VC_POSITION_RE - except ImportError: - return None - return VC_POSITION_RE # pragma: no cover - only reachable on a NetBox that resolves the token + """Delegate virtual-chassis token detection to template-name resolution.""" + return family_template_names.vc_position_re() def supports_vc_position_token(): @@ -308,59 +88,6 @@ def supports_vc_position_token(): return _vc_position_re() is not None -def _is_channel_child(iface): - """Return True when *iface* is a channel subinterface bound to a parent's channel. - - Structural, not name-based: a row imported without ``full_clean()`` may carry a ``channel_id`` - without the channel type. ``channel_id`` does not exist before NetBox 4.7, so this is False - on every older release and the family paths below stay dormant there. - """ - return getattr(iface, "channel_id", None) is not None - - -def _is_channelized_parent(iface): - """Return True when *iface* declares a channel count. - - A channelized parent owns a family even when no subinterface is bound yet — this, not the - presence of children, is what disables flat channel creation. - """ - return getattr(iface, "channels", None) is not None - - -def _partition_families(interfaces): - """Split *interfaces* into ``(bases, children_by_parent_pk)``. - - Bases are the interfaces a rule may match on its own: standalone interfaces and channelized - parents. Channel subinterfaces are never independent candidates — they are renamed only by - following their parent, so they are grouped under it instead. - """ - bases = [] - children = defaultdict(list) - for iface in interfaces: - if not _is_channel_child(iface): - bases.append(iface) - continue - children[iface.parent_id].append(iface) # pragma: no cover - requires channelization support - for group in children.values(): # pragma: no cover - no channel children exist without support - group.sort(key=lambda child: child.channel_id) - return bases, children - - -def _child_name_suffix(child_name, parent_name): # pragma: no cover - requires channelization support - """Return the suffix *child_name* adds to *parent_name*, or None when it adds none. - - The first character must be non-alphanumeric so ``et0``/``et01`` is never mistaken for a - family; the punctuation itself is free-form (``:``, ``-``, ``_`` and ``@`` all occur in the - wild), so it is not restricted to a fixed separator. - """ - if not parent_name or not child_name.startswith(parent_name): - return None - suffix = child_name[len(parent_name) :] - if not suffix or suffix[0].isalnum(): - return None - return suffix - - def _unambiguous_claims(candidates, matchers, module): # pragma: no cover - requires vc_position token support """Return the labels of *candidates* that exactly one drifted ``{vc_position}`` template claims. @@ -422,7 +149,7 @@ def _forced_channel_bases(interfaces, raw_names, matchers, module): for i in interfaces: # A channelized parent is its own base: its channels are separate rows, so the name needs no # ":"-splitting to find them. - base = i.name if _is_channelized_parent(i) else i.name.rsplit(":", 1)[0] + base = i.name if family_ops.is_channelized_parent(i) else i.name.rsplit(":", 1)[0] forms = (base, base.rsplit("/", 1)[-1]) if not any(form in raw_names for form in forms) and not any( matcher.pattern.fullmatch(form) for matcher in matchers for form in forms @@ -469,6 +196,31 @@ def _collect_unrenamed(interfaces, rule, raw_names, force_reapply, matchers=(), return _forced_channel_bases(interfaces, raw_names, matchers, module) +def _touches_a_family(plan) -> bool: + """Return whether *plan* acts on a family rather than one standalone interface.""" + if isinstance(plan, family_ops.InstalledFamilyPlan): + return plan.parent_pk is not None or len(plan.members) > 1 + return True + + +def _admitted_installed(plans, rule, raw_names, force_reapply, matchers, module): + """Return the installed families this install path should execute. + + A channelized family is always executed: its parent decides the family's names, and the raw-name + guard describes flat rows. A flat family is executed while the guard still claims a member of it. + """ + flat = [plan for plan in plans if plan.topology == family_ops.FamilyTopology.FLAT] + snapshots = [member.snapshot for plan in flat for member in plan.members] + selected = { + interface.pk for interface in _collect_unrenamed(snapshots, rule, raw_names, force_reapply, matchers, module) + } + return [ + plan + for plan in plans + if plan.topology == family_ops.FamilyTopology.CHANNELIZED or selected.intersection(plan.member_pks) + ] + + def apply_interface_name_rules(module, module_bay, force_reapply=False): """Apply InterfaceNameRule rename after module installation. @@ -480,162 +232,76 @@ def apply_interface_name_rules(module, module_bay, force_reapply=False): ``force_reapply=True`` to skip this check and re-apply rules to ALL module interfaces (used when vc_position or other variables change). - A channelized parent and its channel subinterfaces are processed as one family: the parent - decides, the children follow it (see ``_apply_rule_with_family``). + Every rename and every creation goes through the family package, so this path builds and names + exactly what retroactive apply and the preview describe. Returns: Number of interfaces renamed/created, or 0 if no rule matched. """ - from dcim.models import Interface - device_type = module.device.device_type if module.device else None platform = module.device.platform if module.device else None rule = find_matching_rule(module.module_type, _get_parent_module_type(module_bay), device_type, platform) if not rule: return 0 + # One pin for the module: the raw-name matchers and the family planner resolve its templates once. + with family_ops.pinned_template_cache(): + return _apply_rule_to_module(rule, module, module_bay, force_reapply) - variables = build_variables(module_bay, device=module.device) - interfaces = list(Interface.objects.filter(module=module)) - if not interfaces: - return 0 +def _apply_rule_to_module(rule, module, module_bay, force_reapply): + """Plan and execute every family *rule* intends on *module*; see ``apply_interface_name_rules``.""" + from dcim.models import Interface - # Only bases are rule candidates; the idempotency guard therefore looks at them alone. - bases, children_by_parent = _partition_families(interfaces) - # Determine raw names NetBox assigned from templates; fall back to bay_position. + variables = build_variables(module_bay, device=module.device) raw = _raw_name_matchers(module) raw_names = raw.names or {variables["bay_position"]} - unrenamed = _collect_unrenamed(bases, rule, raw_names, force_reapply, raw.matchers, module) - - if not unrenamed: - return 0 # Already renamed (idempotent guard) - - # A breakout rule on a module that has channelized families processes only those families — - # the same rule the preview and bulk-apply paths follow. - families_only = rule.channel_count > 0 and any(_is_channelized_parent(base) for base in bases) - - renamed = 0 - families_seen = families_only - conflicts: list = [] - for iface in unrenamed: - children = children_by_parent.get(iface.pk, ()) - if families_only and not _is_channelized_parent(iface): # pragma: no cover - see families_only above - logger.debug( - "Interface %r is not channelized; skipping it while rule '%s' breaks out this module's families.", - iface.name, - rule, - ) - continue - families_seen = families_seen or bool(children) or _is_channelized_parent(iface) - try: - count = _apply_rule_with_family(rule, iface, children, variables, module, conflicts) - except (ValueError, ValidationError, IntegrityError): - # The collision pre-check closes the common case, but a concurrent - # insert can still win between that check and the save — surfacing - # here as IntegrityError/ValidationError out of the per-interface - # atomic block (which has already rolled back cleanly). Log and keep - # going so one racing interface never aborts the whole install batch, - # mirroring apply_rule_to_existing(). - logger.exception( - "Failed to apply rule '%s' to interface '%s' (id=%s); skipping.", - rule, - iface.name, - iface.pk, - ) - continue - if count is None: - # A structural skip (unsupported topology, channel-count mismatch) says nothing about the rule. - families_seen = True - continue - renamed += count + interfaces = list( + Interface.objects.using(family_ops.module_db_alias(module)).filter(module_id=module.pk).order_by("pk") + ) + planned = family_ops.plan_module_families( + module, + rule, + variables, + interfaces, + # The guard runs while it can still see every claimed row, before two of them that intend + # one family are collapsed into it. + admit_leftover=lambda plain: _collect_unrenamed(plain, rule, raw_names, force_reapply, raw.matchers, module), + ) + installed = _admitted_installed(planned.installed, rule, raw_names, force_reapply, raw.matchers, module) + leftover = planned.leftover + + outcomes = family_ops.execute_module_families(rule, module, [*installed, *leftover]) + renamed = sum(outcome.changed_count for outcome in outcomes) + blocked = [ + member for outcome in outcomes for member in outcome.members if member.status == family_ops.FamilyStatus.BLOCKED + ] + families_seen = bool(installed) or any(_touches_a_family(plan) for plan in leftover) - if unrenamed and renamed == 0 and not conflicts and not families_seen: + if leftover and renamed == 0 and not blocked and not families_seen: # All interfaces already have the names the rule would produce — flag as # potentially obsolete (e.g., newer NetBox generates correct names natively). # Skipped when the 0-count was caused by name collisions (a different reason # than a no-op rule), so a collision never mislabels the rule as deprecated. - # Skipped for channelized families too: a structural skip, or a family whose parent - # deliberately keeps its raw name, says nothing about the rule being obsolete. + # Skipped for families too: a structural skip, or a family whose parent deliberately + # keeps its raw name, says nothing about the rule being obsolete. _flag_rule_potentially_deprecated(rule) return renamed -def _predicted_channel_name(rule, raw_name, variables, parents, children): # pragma: no cover - channelized only - """Return the name the channel template named *raw_name* takes under *rule*.""" - parent_name, channel_id = children[raw_name] - if rule.channel_count > 0: - if parents.get(parent_name) != rule.channel_count: - return raw_name # channel-count mismatch: the apply path skips the whole family - channel = str(rule.channel_start + channel_id - 1) - return evaluate_name_template(rule.name_template, {**variables, "base": parent_name, "channel": channel}) - # Simple rule: the channel follows its parent, keeping the suffix it adds to the parent's name. - parent_target = evaluate_name_template(rule.name_template, {**variables, "base": parent_name}) - suffix = _child_name_suffix(raw_name, parent_name) - return raw_name if suffix is None else parent_target + suffix - - -def _predicted_family_parent_name(rule, raw_name, variables, parents): # pragma: no cover - channelized only - """Return the name the parent template named *raw_name* takes under *rule*. - - Only a channelized rule that names its parent renames it, and only when the family's channel - count is the one the rule describes — the same two conditions the apply path applies. - """ - if not (_is_channelized_rule(rule) and rule.parent_name_template): - return raw_name - if parents.get(raw_name) != rule.channel_count: - return raw_name # channel-count mismatch: the apply path skips the whole family - return evaluate_name_template(rule.parent_name_template, {**variables, "base": raw_name}) - - -def _predicted_names(rule, raw_name, variables, parents, children, family_blocked=False): - """Return the names *raw_name* predicts to under *rule*. - - A name the module type's templates describe as a channelized parent or channel follows its - family; a channelized rule on a plain name predicts the family it would build there, unless - *family_blocked* says the apply path refuses to build it; anything else keeps the per-name - prediction, expanding once per channel for a breakout rule and once for a simple one. - """ - if raw_name in parents: # pragma: no cover - requires a NetBox that models channelization - if rule.channel_count > 0: - # The rule renames the family's existing channels; only a parent template moves the parent. - return [_predicted_family_parent_name(rule, raw_name, variables, parents)] - return [evaluate_name_template(rule.name_template, {**variables, "base": raw_name})] - if raw_name in children: # pragma: no cover - requires a NetBox that models channelization - return [_predicted_channel_name(rule, raw_name, variables, parents, children)] - if rule.channel_count > 0 and _is_channelized_rule(rule): - if family_blocked or not supports_channelization(): - return [raw_name] # the apply path builds nothing here - parent_name, channels = _channelized_family_names(rule, raw_name, variables) # pragma: no cover - see above - return [parent_name, *(name for _, name in channels)] # pragma: no cover - see above - vars_copy = {**variables, "base": raw_name} - if rule.channel_count > 0: - return [ - evaluate_name_template(rule.name_template, {**vars_copy, "channel": str(rule.channel_start + ch)}) - for ch in range(rule.channel_count) - ] - return [evaluate_name_template(rule.name_template, vars_copy)] - - def predict_rule_output(module, module_bay, raw_names): """Predict the names apply_interface_name_rules would produce for raw_names. - Read-only — saves and mutates nothing. A channelized rule additionally counts the module's - interfaces, because the apply path refuses to convert a module that already carries a flat - breakout family and the prediction has to say the same. Used by external integrations (e.g., + Read-only: saves and mutates nothing. Used by external integrations (e.g. netbox-librenms-plugin) that need to know the post-rename names without applying any rule. - For breakout rules (channel_count > 0), each raw name expands to - channel_count predicted names. For simple renames, one name in → one name - out. Returns raw_names unchanged when no rule matches or evaluation fails. - - A name the module type's interface templates describe as part of a channelized family is - predicted as the apply path treats it instead: the family's channels are renamed in place, so a - breakout rule leaves the parent's name alone and maps each channel through its ``channel_id`` - rather than expanding one name into a flat set. Names no template claims keep the per-name - prediction, so a module type without channelized templates is unaffected. + The names are planned by the family module from the module type's templates, so prediction + describes the same families installed execution builds: a breakout rule expands one plain name + into the family it creates, a name the templates describe as part of a channelized family + follows that family instead, and a family the apply path refuses to touch predicts unchanged. + Returns raw_names unchanged when no rule matches. Precondition: *raw_names* are resolved by the caller at call time. A name captured before the device's virtual-chassis position changed is predicted from itself, not corrected to the name @@ -647,132 +313,113 @@ def predict_rule_output(module, module_bay, raw_names): if not rule: return list(raw_names) - variables = build_variables(module_bay, device=module.device) - parents, children = _template_families(module) - # Costs one count pair, and only where a channelized rule could otherwise predict a family. - family_blocked = ( - rule.channel_count > 0 - and _is_channelized_rule(rule) - and supports_channelization() - and _has_flat_expansion(module) + plan_set = family_ops.plan_prospective_families( + module, + rule, + build_variables(module_bay, device=module.device), + family_ops.describe_module_interfaces(module, raw_names), ) + return [name for raw_name in raw_names for name in plan_set.predicted_names(raw_name)] - output = [] - for raw_name in raw_names: - try: - output.extend(_predicted_names(rule, raw_name, variables, parents, children, family_blocked)) - except (ValueError, TypeError, re.error): - # Template eval failed; apply path would also fail and leave the - # interface alone, so the predicted name is the raw name. - output.append(raw_name) - - return output +def reapply_module_rules(device): + """Re-apply module rules to every module on *device* after its virtual-chassis position changed. -def _try_rename_device_interface(rule, iface, vc_position, device, renamed_pks, conflicts=None): - """Attempt to rename a single device-level interface using *rule*. + The whole device is one batch: its modules match against one enabled-rule snapshot, and one + module type's interface templates are read once however many modules carry it. A failure is + logged and leaves that module behind. Remaining modules continue with the new position. - Returns ``True`` if the interface was successfully renamed, ``False`` otherwise. - Mutates ``renamed_pks`` on success. - - A computed name already taken by another interface on the device is skipped - with a tidy WARNING (no traceback), mirroring the module-install path; pass a - list as *conflicts* to also collect them. ``full_clean()`` remains the - backstop for the rarer cross-member (VC) uniqueness violation. + Returns the number of interfaces renamed across the device's modules. """ - if iface.pk in renamed_pks: - return False # Already renamed by a higher-priority rule - - if rule.module_type_pattern: - try: - if not re.fullmatch(rule.module_type_pattern, iface.name): - return False - except re.error: - return False - - port = iface.name.rsplit("/", 1)[-1] if "/" in iface.name else iface.name - variables = {"vc_position": vc_position, "base": iface.name, "port": port} + from dcim.models import Module - try: - new_name = evaluate_name_template(rule.name_template, variables) - except (ValueError, TypeError, re.error): - logger.exception( - "Failed to evaluate template %r for interface %s (rule %s)", - rule.name_template, - iface.name, - rule.pk, + modules = list( + Module.objects.filter(device=device).select_related( + "module_type", + "device__device_type", + "device__platform", + *family_template_names.BAY_CHAIN_RELATIONS, ) - return False + ) + total = 0 + with pinned_rule_cache(), family_ops.pinned_template_cache(modules): + for module in modules: + if not module.module_bay: + continue + try: + total += apply_interface_name_rules(module, module.module_bay, force_reapply=True) or 0 + except Exception: + logger.exception( + "Failed to re-apply rules for %s in %s after a virtual-chassis change", + module.module_type, + module.module_bay.name, + ) + return total - if new_name == iface.name: - return False - # Pre-check device-scope name uniqueness so an expected collision is a clean - # WARNING + skip instead of an ERROR traceback out of full_clean(). - if _name_exists_on_device(device, new_name, exclude_pk=iface.pk): - _record_conflict(conflicts, device, iface.name, new_name, iface.pk) - return False +def _device_interface_rules(device): + """Return enabled device-interface rules in matching priority order.""" + from django.db.models import Q - old_name = iface.name - iface.name = new_name - try: - iface.full_clean() - except ValidationError as exc: - logger.warning( - "Validation failed renaming device interface %r → %r (rule %s, device %s); skipping: %s", - old_name, - new_name, - rule.pk, - device.pk, - exc, + from .models import InterfaceNameRule + + device_type = getattr(device, "device_type", None) + platform = getattr(device, "platform", None) + rules = list( + InterfaceNameRule.objects.filter( + applies_to_device_interfaces=True, + enabled=True, ) - iface.name = old_name - return False - try: - iface.save() - except (IntegrityError, ValidationError): - logger.exception( - "DB save failed for device interface %s → %s (rule %s, device %s)", - old_name, - new_name, - rule.pk, - device.pk, + .filter(Q(device_type=device_type) | Q(device_type__isnull=True)) + .filter(Q(platform=platform) | Q(platform__isnull=True)) + ) + # Sort Python-side: specificity_score descending, then module_type_pattern length + # descending (for device-interface rules with ties), then pk ascending for stability. + # (InterfaceNameRule has no DB 'priority' field; specificity_score is a property.) + rules.sort( + key=lambda r: ( + -r.specificity_score, + -len(r.module_type_pattern or ""), + r.pk, ) - iface.name = old_name - return False - - renamed_pks.add(iface.pk) - logger.debug("Renamed device interface %s → %s (rule %s, device %s)", old_name, new_name, rule.pk, device.pk) - return True + ) + return rules -def _try_rename_device_family(rule, iface, children, vc_position, device, renamed_pks, conflicts=None): - """Rename a device-level interface with *rule* and carry its channel subinterfaces along. +def _matches_device_interface(rule, interface): + """Return whether one device-interface rule matches one family parent.""" + if not rule.module_type_pattern: + return True + try: + compiled = compile_module_type_pattern(rule.module_type_pattern) + except ValidationError: + return False + return compiled.fullmatch(interface.name) is not None - Returns the number of interfaces renamed. The whole family is claimed in *renamed_pks* the - moment its parent is renamed, so a lower-priority rule can never rename the leftovers of a - family a higher-priority rule already took. - Healing is best-effort here: device-level interfaces have no module template family to recover - a suffix from, so a child that lost its parent's prefix in an earlier run is left alone. - """ - parent_before = iface.name - if not _try_rename_device_interface(rule, iface, vc_position, device, renamed_pks, conflicts): - return 0 - count = 1 - for child, target in _child_target_names( # pragma: no cover - requires channelization support - children, parent_before, iface.name, module=None - ): - renamed_pks.add(child.pk) - if target is None: - logger.warning( - "Cannot derive a name for channel interface %r from parent %r; leaving it unchanged.", - child.name, - iface.name, +def _apply_device_rule_to_families(device, vc_position, rule, families, claimed_pks): + """Apply one rule to each eligible device-interface family.""" + total = 0 + for interface, children in families: + if interface.pk in claimed_pks or not _matches_device_interface(rule, interface): + continue + port = interface.name.rsplit("/", 1)[-1] + variables = {"vc_position": vc_position, "base": interface.name, "port": port} + plan = family_ops.plan_device_interface_rename(device, rule, variables, interface, children) + try: + outcome = family_ops.execute_installed_plan(plan) + except (IntegrityError, ValidationError): + logger.exception( + "Failed to apply rule %s to device interface %r on device %s; skipping.", + rule.pk, + interface.name, + device.pk, ) continue - count += _rename_for_family(child, target, device, conflicts).count - return count + total += outcome.changed_count + if outcome.status in {family_ops.FamilyStatus.CHANGED, family_ops.FamilyStatus.UNCHANGED}: + claimed_pks.update(plan.member_pks) + return total def apply_device_interface_rules(device): @@ -784,7 +431,7 @@ def apply_device_interface_rules(device): Template variables available: ``{vc_position}``, ``{base}`` (full current name), ``{port}`` (segment after the last ``/``, or the full name if no ``/`` present). - Channel subinterfaces are not matched independently — they follow the parent whose family + Channel subinterfaces are not matched independently. They follow the parent whose family a rule wins, so a template like ``eth{vc_position}`` cannot collapse a whole family onto one name. @@ -792,8 +439,6 @@ def apply_device_interface_rules(device): """ from dcim.models import Interface - from .models import InterfaceNameRule - if not getattr(device, "virtual_chassis_id", None): return 0 # Only rename for VC members (vc_position must be set) @@ -801,136 +446,26 @@ def apply_device_interface_rules(device): return 0 # vc_position unset (e.g. VC master before position assigned) vc_position = str(device.vc_position) - device_type = getattr(device, "device_type", None) - platform = getattr(device, "platform", None) - - from django.db.models import Q - - rules = list( - InterfaceNameRule.objects.filter( - applies_to_device_interfaces=True, - enabled=True, - ) - .filter(Q(device_type=device_type) | Q(device_type__isnull=True)) - .filter(Q(platform=platform) | Q(platform__isnull=True)) - ) - # Sort Python-side: specificity_score descending, then module_type_pattern length - # descending (for device-interface rules with ties), then pk ascending for stability. - # (InterfaceNameRule has no DB 'priority' field; specificity_score is a property.) - rules.sort( - key=lambda r: ( - -r.specificity_score, - -(len(r.module_type_pattern or "") if r.applies_to_device_interfaces else 0), - r.pk, - ) - ) - + rules = _device_interface_rules(device) if not rules: return 0 - interfaces = list(Interface.objects.filter(device=device, module=None)) + interfaces = list(Interface.objects.filter(device=device, module=None).order_by("pk")) if not interfaces: return 0 - bases, children_by_parent = _partition_families(interfaces) + families = family_ops.device_interface_families(interfaces) + claimed_pks: set[int] = set() total = 0 - renamed_pks: set[int] = set() for rule in rules: - for iface in bases: - total += _try_rename_device_family( - rule, iface, children_by_parent.get(iface.pk, ()), vc_position, device, renamed_pks - ) + total += _apply_device_rule_to_families(device, vc_position, rule, families, claimed_pks) return total -# The bay chain InterfaceTemplate.resolve_name() dereferences while resolving {module}. -_BAY_CHAIN_RELATIONS = ( - "module_bay", - "module_bay__parent", - "module_bay__module", - "module_bay__module__module_bay", - "module_bay__module__module_bay__parent", - "module_bay__module__module_bay__module", -) - - -def _module_with_bay_chain(module): - """Re-fetch *module* with the bay chain InterfaceTemplate.resolve_name() dereferences. - - Prefetches the module relationships to avoid a per-template query when resolving names. - """ - from dcim.models import Module - - return Module.objects.select_related(*_BAY_CHAIN_RELATIONS).get(pk=module.pk) - - -# NetBox resolves {vc_position} once, at instantiation, so a raw name records the device's VC state -# at that moment while its template keeps resolving to the current one: hence names *and* matchers. -_RawMatcher = namedtuple("_RawMatcher", ("template_name", "resolved", "pattern")) -_RawNames = namedtuple("_RawNames", ("names", "matchers")) - -# Brace-free stand-ins, so NetBox's placeholder pass, this plugin's and re.escape() all leave them be. -_VC_SENTINEL = "InrVcPositionSentinel{}End" -_BASE_SENTINEL = "InrBaseSentinelEnd" - - -def _vc_position_alternatives(fallback): # pragma: no cover - requires vc_position token support - """Return the regex branch covering every value one ``{vc_position}`` occurrence resolves to. - - Any member position and the implicit ``'0'`` are digits; an explicit ``{vc_position:X}`` fallback - adds a branch of its own, since NetBox does not require it to be numeric. - """ - if fallback is None: - return r"\d+" - return f"(?:\\d+|{re.escape(fallback)})" - - -def _raw_name_pattern(tmpl, module, token_re): # pragma: no cover - requires vc_position token support - """Return the matcher for every name *tmpl* has ever resolved to, or None without the token. - - Each token occurrence becomes a sentinel; ``{module}`` is then resolved by NetBox's own code on a - shallow copy carrying that name (its VC pass no-ops on a token-free string), so no placeholder - resolution is reimplemented here. - """ - fallbacks = [] - - def _mark(match): - fallbacks.append(match.group(1)) - return _VC_SENTINEL.format(len(fallbacks) - 1) - - marked = token_re.sub(_mark, tmpl.name) - if not fallbacks: - return None - stub = copy.copy(tmpl) - stub.name = marked - pattern = re.escape(stub.resolve_name(module)) - for index, fallback in enumerate(fallbacks): - pattern = pattern.replace(_VC_SENTINEL.format(index), _vc_position_alternatives(fallback)) - return _compile_pattern(pattern) - - -def _raw_matchers(templates, module): - """Resolve *templates* against *module*: their names now, plus a matcher per token template.""" - token_re = _vc_position_re() - names = set() - matchers = [] - for tmpl in templates: - resolved = tmpl.resolve_name(module) - names.add(resolved) - pattern = None if token_re is None else _raw_name_pattern(tmpl, module, token_re) - if pattern is not None: - matchers.append(_RawMatcher(tmpl.name, resolved, pattern)) # pragma: no cover - token templates only - return _RawNames(names, matchers) - - def _raw_name_matchers(module): - """Return *module*'s raw template names and the drift matchers of its token templates.""" - from dcim.models import InterfaceTemplate - - module_fresh = _module_with_bay_chain(module) - templates = InterfaceTemplate.objects.filter(module_type=module_fresh.module_type) - return _raw_matchers(templates, module_fresh) + """Delegate current and historical raw name resolution.""" + return family_template_names.raw_name_matchers(module) def _get_raw_interface_names(module): @@ -939,124 +474,8 @@ def _get_raw_interface_names(module): def _raw_name_patterns(module): - """Return one compiled matcher per interface template of *module* whose name carries the token. - - Empty for a module type no template of which uses ``{vc_position}``, and on every NetBox release - that does not resolve the token at all. - """ - return [matcher.pattern for matcher in _raw_name_matchers(module).matchers] - - -def _raw_names_by_module(modules): # pragma: no cover - only the conversion scan batches names - """Return ``{module pk: _RawNames}`` for *modules*, in one template query for all of them. - - Raw names are a property of the module type, but ``_raw_name_matchers`` costs a module refetch - and a template query each — a scan over a fleet would pay that per module. *modules* must - already carry ``_BAY_CHAIN_RELATIONS``, since the names are resolved against them in memory. - """ - from dcim.models import InterfaceTemplate - - by_module_type = defaultdict(list) - for tmpl in InterfaceTemplate.objects.filter(module_type__in={module.module_type_id for module in modules}): - by_module_type[tmpl.module_type_id].append(tmpl) - return {module.pk: _raw_matchers(by_module_type[module.module_type_id], module) for module in modules} - - -def _template_families(module): - """Return ``(parents, children)`` describing *module*'s channelized interface templates. - - *parents* maps a channelized parent template's resolved name to its channel count; *children* - maps each channel template's resolved name to ``(parent_name, channel_id)``. Both are empty - where nothing can be channelized, so callers keep their pre-channelization behaviour without - paying for a template scan. - """ - if not supports_channelization(): - return {}, {} - return _resolve_template_families(module) # pragma: no cover - requires channelization support - - -def _resolve_template_families(module): # pragma: no cover - requires a NetBox that models channelization - """Resolve *module*'s interface templates into the channelized families they describe. - - Pairing through ``InterfaceTemplate.parent`` (rather than matching against the flat set of raw - names) keeps ambiguous prefixes like ``xe``/``xe-0`` apart, and a channel template whose parent - declares no channel count is not a family at all. - """ - from dcim.models import InterfaceTemplate - - module_fresh = _module_with_bay_chain(module) - templates = list(InterfaceTemplate.objects.filter(module_type=module_fresh.module_type)) - resolved = {tmpl.pk: tmpl.resolve_name(module_fresh) for tmpl in templates} - parents_by_pk = { - tmpl.pk: (resolved[tmpl.pk], tmpl.channels) for tmpl in templates if getattr(tmpl, "channels", None) is not None - } - children = {} - for tmpl in templates: - channel_id = getattr(tmpl, "channel_id", None) - parent = parents_by_pk.get(getattr(tmpl, "parent_id", None)) - if channel_id is None or parent is None: - continue - parent_name, _channels = parent - children[resolved[tmpl.pk]] = (parent_name, channel_id) - return dict(parents_by_pk.values()), children - - -def _template_channel_suffixes(module): # pragma: no cover - requires a NetBox that models channelization - """Map ``channel_id`` → the set of name suffixes *module*'s interface templates give that channel. - - The suffix comes from the template family itself — each channel template's resolved name minus - its parent template's resolved name — so a child that lost its parent's prefix in an earlier - partial rename can still be repaired. A module type with several families may spell the same - channel differently in each (``et0:2`` vs ``sw0.2``), so the suffixes are collected per channel - rather than overwritten: the recovery only uses one when every family agrees on it. - """ - suffixes = defaultdict(set) - for child_name, (parent_name, channel_id) in _template_families(module)[1].items(): - suffix = _child_name_suffix(child_name, parent_name) - if suffix is not None: - suffixes[channel_id].add(suffix) - return suffixes - - -def _recovered_suffix(child, suffixes): # pragma: no cover - requires channelization support - """Return the template suffix for *child*'s channel, or None when it is not unambiguous. - - Once a parent has been renamed there is no reliable way back from a stranded child to the family - it belongs to, so a channel spelled differently by two families is left alone rather than guessed. - """ - candidates = suffixes.get(child.channel_id) or set() - if len(candidates) == 1: - return next(iter(candidates)) - if candidates: - logger.warning( - "Channel %s is spelled %s by different families of this module type; " - "cannot recover a name for interface %r.", - child.channel_id, - sorted(candidates), - child.name, - ) - return None - - -def _child_target_names(children, parent_before, parent_after, module): - """Pair every child with the name it takes when its parent is renamed to *parent_after*. - - The suffix is read from the child's own name against *parent_before* (the parent's name before - this run's rename); when the child no longer carries that prefix the suffix is recovered from - the module's template family instead. A child that neither shares the prefix nor has an - unambiguous template pairing is returned with a None target — the engine leaves it alone rather - than guessing at a free-form name. - """ - suffixes = None - targets = [] - for child in children: # pragma: no cover - requires channelization support - suffix = _child_name_suffix(child.name, parent_before) - if suffix is None and module is not None: - if suffixes is None: - suffixes = _template_channel_suffixes(module) - suffix = _recovered_suffix(child, suffixes) - targets.append((child, None if suffix is None else parent_after + suffix)) - return targets + """Delegate historical raw-name pattern construction.""" + return family_template_names.raw_name_patterns(module) def _flag_rule_potentially_deprecated(rule): @@ -1087,735 +506,45 @@ def _flag_rule_potentially_deprecated(rule): logger.exception("Failed to flag rule '%s' as potentially-deprecated.", rule) -def _scope_ids(parent_module_type, device_type, platform): - """Map the (parent_module_type, device_type, platform) scope objects to their FK ids. +def _matching_moduletype_pks(module_type_pattern): + """Return PKs of ModuleTypes whose model name matches the given RE2 pattern. - ``None`` (no constraint) maps to ``None`` so it compares equal to a rule's unset scope FK. - Centralises the ``x.pk if x is not None else None`` coalescing used by both match tiers and - the memo key. + Raises ValueError for invalid patterns, mirroring evaluate_name_template's + error-handling convention so callers can treat both as ValueError. """ - return ( - parent_module_type.pk if parent_module_type is not None else None, - device_type.pk if device_type is not None else None, - platform.pk if platform is not None else None, - ) + from dcim.models import ModuleType + try: + compiled = compile_module_type_pattern(module_type_pattern) + except ValidationError as exc: + raise ValueError(exc.messages[0]) from exc + return [mt.pk for mt in ModuleType.objects.only("pk", "model") if compiled.fullmatch(mt.model)] -def _rule_scope_matches(rule, scope_ids): - """Return True when *rule*'s (parent_module_type, device_type, platform) FKs equal *scope_ids*.""" - pmt_id, dt_id, pl_id = scope_ids - return rule.parent_module_type_id == pmt_id and rule.device_type_id == dt_id and rule.platform_id == pl_id +def has_applicable_interfaces(rule) -> bool: + """Check whether applying this rule right now would rename at least one interface. -def _build_candidates(parent_module_type, device_type, platform) -> list: - """Build ordered list of (pmt, dt, pl) tuples from most to least specific. + Calls find_interfaces_for_rule(limit=1) to determine if any currently installed + interface would receive a new name. Returns False when: + - no matching modules/interfaces are installed, OR + - all matching interfaces are already correctly named. - Each argument expands to ``[value, None]`` when provided, or ``[None]`` - when already absent. Deduplication ensures no key appears twice (which - would happen when multiple inputs are None). - """ - seen: set = set() - candidates = [] - pmt_opts = [parent_module_type, None] if parent_module_type else [None] - dt_opts = [device_type, None] if device_type else [None] - pl_opts = [platform, None] if platform else [None] - for pmt in pmt_opts: - for dt in dt_opts: - for pl in pl_opts: - key = (pmt, dt, pl) - if key not in seen: - seen.add(key) - candidates.append(key) - return candidates - - -def _find_exact_match(module_type, candidates, exact_rules=None): - """Tier 1: return the first enabled exact-FK rule in specificity order, or None. - - ``exact_rules`` is the preloaded, ``(module_type__model, pk)``-ordered enabled - exact-rule set (the hot path passes it to avoid a DB query per call); when omitted - it is loaded on demand so direct callers keep working. + This is more expensive than a plain EXISTS query but ensures the Applicable + column in the Apply Rules list accurately reflects "would something change?" + rather than the misleading "do interfaces exist?". """ - if exact_rules is None: - exact_rules, _, _ = _get_enabled_rules() - - # module_type fixed below → the (module_type__model, pk) ordering reduces to pk, so the - # first matching rule equals the previous ``.filter(...).first()``. - scoped = [r for r in exact_rules if r.module_type_id == module_type.pk] - for candidate in candidates: - scope_ids = _scope_ids(*candidate) - for rule in scoped: - if _rule_scope_matches(rule, scope_ids): - return rule - return None + try: + results, _ = find_interfaces_for_rule(rule, limit=1) + return len(results) > 0 + except ValueError: + return False -def _find_regex_match(model_name: str, candidates, regex_rules=None): - """Tier 2: return the first enabled regex rule whose pattern fullmatches *model_name*, or None. +def _build_module_qs(rule): + """Return a Module queryset filtered to the rule's scope (module type, parent, device, platform). - Tries candidates in specificity order; within each level longer patterns are tried first - (more specific). ``regex_rules`` is the preloaded ``(compiled_pattern, rule)`` set — pre-sorted - by ``(-pattern length, pk)`` with each pattern compiled once (a None compile is an invalid - pattern, silently skipped); loaded on demand when omitted. - """ - if regex_rules is None: - _, regex_rules, _ = _get_enabled_rules() - - for candidate in candidates: - scope_ids = _scope_ids(*candidate) - for compiled, rule in regex_rules: - if compiled is not None and _rule_scope_matches(rule, scope_ids) and compiled.fullmatch(model_name): - return rule - return None - - -def find_matching_rule(module_type, parent_module_type, device_type, platform=None): - """Find the most specific InterfaceNameRule matching the context. - - Uses a two-tier strategy: - Tier 1 — Exact FK match (priority order, most specific first): - Iterates all combinations of (parent_module_type, device_type, platform) - from fully-constrained to fully-unconstrained (None = any). - Tier 2 — Regex pattern match (same priority order, longer patterns first): - Same specificity cascade, but module_type_pattern is matched via - re.fullmatch() against module_type.model. Patterns are iterated - from longest to shortest to prefer more specific patterns. - - The enabled rule set is loaded once and matched in memory, and the per-context - result is memoized for the current rule-set version, so repeated calls (e.g. one - per module row in a module-sync render) don't re-query the database. - - Returns the first matching rule, or None if no rule matches. - """ - if module_type is None: - # Module rules are always keyed on a module type; both tiers dereference it - # (module_type.pk / .model), so there is nothing to match without one. - return None - - exact_rules, regex_rules, memo = _get_enabled_rules() - # The regex tier matches against module_type.model (a live string), so the memo must key on - # it too — otherwise a ModuleType.model rename (same pk) would return a stale regex result. - sig = (module_type.pk, module_type.model, *_scope_ids(parent_module_type, device_type, platform)) - # One atomic lookup, not `if sig in memo: return memo[sig]`: another thread sharing this per-version - # memo can clear it at the cap between a membership test and the subscript, raising KeyError. - cached = memo.get(sig, _MEMO_MISS) - if cached is not _MEMO_MISS: - return cached - - candidates = _build_candidates(parent_module_type, device_type, platform) - result = _find_exact_match(module_type, candidates, exact_rules) or _find_regex_match( - module_type.model, candidates, regex_rules - ) - if len(memo) >= _MEMO_MAX: - memo.clear() # bound per-version memory; entries are rebuilt lazily on the next miss - memo[sig] = result - return result - - -def _extract_trailing_digits(s: str) -> str: - r"""Return the trailing digit run of *s* without regex backtracking. - - Pure O(n) string scan — eliminates the polynomial backtracking risk that - arises from using ``re.search(r"(\d+)$", ...)`` on strings ending in a - non-digit character (e.g. ``"1" * n + "x"`` would cause O(n²) steps). - - Returns an empty string when *s* has no trailing digits. - """ - i = len(s) - while i > 0 and s[i - 1].isdigit(): - i -= 1 - return s[i:] - - -def _resolve_bay_position(module_bay): - """Return (bay_position, bay_position_num) from a module bay's position field. - - Handles template expressions like ``{module}`` by extracting the trailing - digit from the bay name. Falls back to ``"0"`` if no digit is found. - """ - bay_position = module_bay.position or "0" - if bay_position.startswith("{"): - digits = _extract_trailing_digits(module_bay.name) - bay_position = digits if digits else "0" - digits = _extract_trailing_digits(bay_position) - bay_position_num = digits if digits else "0" - return bay_position, bay_position_num - - -def _resolve_slot(module_bay, bay_position_num, parent_bay_position): - """Return the ``slot`` variable from the module bay hierarchy. - - When the bay has a parent bay, slot comes from the parent (or grandparent - when two levels of nesting exist). When the bay belongs to an installed - module with its own bay, slot comes from that module's bay position. - Falls back to ``bay_position_num``. - """ - if module_bay.parent: - parent_bay = module_bay.parent - if parent_bay.parent and hasattr(parent_bay.parent, "installed_module"): - return parent_bay.parent.position or parent_bay_position - return parent_bay_position - if hasattr(module_bay, "module") and module_bay.module: - owner_module = module_bay.module - if hasattr(owner_module, "module_bay") and owner_module.module_bay: - return owner_module.module_bay.position or bay_position_num - return bay_position_num - - -def build_variables(module_bay, device=None): - """Build template variable dict from a module bay's position context. - - Extracts numeric and raw position values from the bay and its parent chain, - producing the variables available for name_template substitution. - - Returns a dict with keys: slot, bay_position, bay_position_num, - parent_bay_position, sfp_slot, and optionally vc_position. - - ``vc_position`` is only injected when *device* is a Virtual Chassis member - (device.virtual_chassis_id is set). Templates using ``{vc_position}`` on a - non-VC device will raise ValueError during evaluation — this is intentional. - Note: Juniper VC positions start at 0, so 0 is a valid real-world value and - cannot be used as a "not in VC" sentinel. - """ - bay_position, bay_position_num = _resolve_bay_position(module_bay) - - parent_bay_position = "0" - if module_bay.parent: - parent_bay_position = module_bay.parent.position or "0" - - slot = _resolve_slot(module_bay, bay_position_num, parent_bay_position) - - result = { - "slot": slot, - "bay_position": bay_position, - "bay_position_num": bay_position_num, - "parent_bay_position": parent_bay_position, - "sfp_slot": bay_position_num, - } - if ( - device is not None - and getattr(device, "virtual_chassis_id", None) is not None - and device.vc_position is not None - ): - result["vc_position"] = str(device.vc_position) - return result - - -def _name_exists_on_device(device, name, exclude_pk=None): - """Return True if another interface on *device* already uses *name*. - - Pre-checks the per-device interface-name uniqueness NetBox enforces so a - rename/create that would collide is skipped cleanly instead of raising - mid-transaction. (VC-wide uniqueness is not pre-checked here; full_clean() - remains the authoritative validator for that rarer cross-member case.) - """ - from dcim.models import Interface - - qs = Interface.objects.filter(device=device, name=name) - if exclude_pk is not None: - qs = qs.exclude(pk=exclude_pk) - return qs.exists() - - -def _record_skip(conflicts, device, current_name, attempted_name, interface_pk=None): - """Append a skipped rename to *conflicts* when the caller is collecting them. - - The caller has already logged why it skipped; this only lets the interactive Apply view report - how many renames were dropped. - """ - if conflicts is not None: - conflicts.append( - { - "device": str(device), - "current_name": current_name, - "attempted_name": attempted_name, - "interface_pk": interface_pk, - } - ) - - -def _record_conflict(conflicts, device, current_name, attempted_name, interface_pk=None): - """Log a name collision at WARNING and record it as a skipped rename. - - Collisions are expected during automatic renaming (module install, type - change, VC change) when the computed name is already taken on the device; - they must never abort the batch, so callers skip the rename and carry on. - """ - logger.warning( - "Interface name %r already exists on device %s — skipping rename of %r → %r", - attempted_name, - device, - current_name, - attempted_name, - ) - _record_skip(conflicts, device, current_name, attempted_name, interface_pk) - - -# What a family-aware rename did, so the children can act on their parent's outcome rather than on -# its computed target name (which says nothing about whether the parent actually took it). -_RENAMED = "renamed" -_UNCHANGED = "unchanged" -_COLLISION = "collision" -_ERROR = "error" - -_RenameResult = namedtuple("_RenameResult", ("target_name", "outcome", "count")) - - -def _rename_in_place(iface, new_name, device, conflicts): - """Rename *iface* to *new_name*; return 1 if renamed, 0 if no-op or collision.""" - if new_name == iface.name: - return 0 - if _name_exists_on_device(device, new_name, exclude_pk=iface.pk): - _record_conflict(conflicts, device, iface.name, new_name, iface.pk) - return 0 - iface.name = new_name - iface.full_clean() - iface.save() - return 1 - - -def _rename_for_family(iface, new_name, device, conflicts): # pragma: no cover - requires channelization support - """Rename *iface* as part of a family walk, reporting the outcome instead of raising. - - The save runs in its own savepoint so an unexpected failure on one member leaves the - surrounding transaction usable: family processing is best-effort per interface, and only a - failed *parent* stops the rest of its family. - """ - if new_name == iface.name: - return _RenameResult(new_name, _UNCHANGED, 0) - old_name = iface.name - try: - with transaction.atomic(): - renamed = _rename_in_place(iface, new_name, device, conflicts) - except (ValueError, ValidationError, IntegrityError): - logger.exception("Failed to rename interface %r → %r on device %s; skipping.", old_name, new_name, device) - iface.name = old_name - return _RenameResult(new_name, _ERROR, 0) - return _RenameResult(new_name, _RENAMED, 1) if renamed else _RenameResult(new_name, _COLLISION, 0) - - -def _restore_deferred_channel_names(reconciliations, db_alias): # pragma: no cover - channelization only - """Restore plugin-owned names that NetBox's parent cascade changed after commit.""" - from dcim.models import Interface - - child_pks = [child_pk for child_pk, _final_name, _cascade_name in reconciliations] - with transaction.atomic(using=db_alias): - children = Interface.objects.using(db_alias).select_for_update().select_related("device").in_bulk(child_pks) - for child_pk, final_name, cascade_name in reconciliations: - child = children.get(child_pk) - if child is None or child.name == final_name: - continue - if child.name != cascade_name: - logger.warning( - "Channel interface %s changed to unexpected name %r before deferred reconciliation; " - "leaving it unchanged.", - child_pk, - child.name, - ) - continue - previous_name = child.name - try: - with transaction.atomic(using=db_alias): - child.name = final_name - child.full_clean() - child.save(using=db_alias) - except (ValueError, ValidationError, IntegrityError): - child.name = previous_name - logger.exception( - "Failed to restore channel interface %s from NetBox's deferred name %r to %r; skipping.", - child_pk, - cascade_name, - final_name, - ) - - -def _preserve_names_across_parent_cascade(parent, parent_before, final_names): # pragma: no cover - """Run after NetBox's deferred cascade when a rule intentionally keeps old-parent child names.""" - if parent.name == parent_before: - return - - reconciliations = [] - for child, final_name in final_names: - old_conventional_name = f"{parent_before}:{child.channel_id}" - cascade_name = f"{parent.name}:{child.channel_id}" - if final_name == old_conventional_name and final_name != cascade_name: - reconciliations.append((child.pk, final_name, cascade_name)) - if not reconciliations: - return - - reconciliations = tuple(reconciliations) - db_alias = parent._state.db - transaction.on_commit( - lambda: _restore_deferred_channel_names(reconciliations, db_alias), - using=db_alias, - ) - - -def _rename_channel_children(parent, parent_before, children, module, conflicts): # pragma: no cover - see above - """Carry the parent's new name onto its channel subinterfaces; return how many were renamed.""" - count = 0 - for child, target in _child_target_names(children, parent_before, parent.name, module): - if target is None: - logger.warning( - "Cannot derive a name for channel interface %r from parent %r; leaving it unchanged.", - child.name, - parent.name, - ) - continue - count += _rename_for_family(child, target, module.device, conflicts).count - return count - - -def _apply_simple_rule_to_family(rule, parent, children, variables, module, conflicts): # pragma: no cover - """Rename a channelized family in lockstep with its parent; return the count renamed. - - The children act on the parent's *outcome*, never on its computed target: a parent that - collided or failed to save leaves the whole family untouched, while a parent that already - carries the right name still lets a stale child be repaired. - """ - parent_before = parent.name - new_name = evaluate_name_template(rule.name_template, {**variables, "base": parent.name}) - result = _rename_for_family(parent, new_name, module.device, conflicts) - if result.outcome in (_COLLISION, _ERROR): - logger.debug("Family of %r left unchanged: the parent could not be renamed to %r.", parent.name, new_name) - return result.count - return result.count + _rename_channel_children(parent, parent_before, children, module, conflicts) - - -def _rename_family_parent(rule, parent, variables, module, conflicts): # pragma: no cover - see above - """Rename an existing family's parent per the rule's parent template; return its rename outcome. - - Only a channelized rule that names its parent touches it — a flat rule, or a blank parent - template, leaves the parent the name it already has. - """ - if not (_is_channelized_rule(rule) and rule.parent_name_template): - return _RenameResult(parent.name, _UNCHANGED, 0) - target = evaluate_name_template(rule.parent_name_template, {**variables, "base": parent.name}) - return _rename_for_family(parent, target, module.device, conflicts) - - -def _apply_breakout_rule_to_family(rule, parent, children, variables, module, conflicts): # pragma: no cover - """Rename an already-channelized family; return the count renamed, or None when skipped. - - Nothing is ever created here: the channels the rule describes are rows NetBox already models, - so a breakout rule renames them in place. The parent is renamed only when the rule builds - channelized families and names their parent; the channels' ``{base}`` stays the parent's name - as it was before that rename. A rule whose channel count disagrees with the hardware is a - modelling mismatch — the family is skipped whole rather than renamed into a shape it does not - have, and a parent that could not take its name stops the family the same way a simple rule's - does. - - Every child's name is computed before the first save, so a template that only fails on a later - channel (channel-dependent arithmetic) aborts the family untouched instead of half renaming it. - """ - if getattr(parent, "channels", None) != rule.channel_count: - logger.warning( - "Interface %r provides %s channels but rule '%s' defines %s; skipping the family.", - parent.name, - getattr(parent, "channels", None), - rule, - rule.channel_count, - ) - return None - base_name = parent.name - targets = [ - ( - child, - evaluate_name_template( - rule.name_template, - {**variables, "base": base_name, "channel": str(rule.channel_start + child.channel_id - 1)}, - ), - ) - for child in children - ] - result = _rename_family_parent(rule, parent, variables, module, conflicts) - if result.outcome in (_COLLISION, _ERROR): - logger.debug( - "Family of %r left unchanged: the parent could not be renamed to %r.", base_name, result.target_name - ) - return result.count - count = result.count - final_names = [] - for child, new_name in targets: - previous_name = child.name - child_result = _rename_for_family(child, new_name, module.device, conflicts) - count += child_result.count - final_name = new_name if child_result.outcome in (_RENAMED, _UNCHANGED) else previous_name - final_names.append((child, final_name)) - _preserve_names_across_parent_cascade(parent, base_name, final_names) - return count - - -def _is_channelized_rule(rule): - """Return True when *rule* asks for the channelized topology instead of flat sibling interfaces.""" - return rule.breakout_mode == BreakoutModeChoices.CHANNELIZED - - -def _channelized_family_names(rule, base_name, variables): # pragma: no cover - requires channelization support - """Return ``(parent_name, [(channel_id, name), ...])`` for the family *rule* builds on *base_name*. - - ``{base}`` is the base interface's current name for the parent and every channel; ``{channel}`` - is ``channel_start + channel_id - 1``. A blank parent template leaves the base's name alone. - Takes the name rather than the interface so prediction can reuse it without a row to point at. - """ - family_vars = {**variables, "base": base_name} - parent_name = base_name - if rule.parent_name_template: - parent_name = evaluate_name_template(rule.parent_name_template, family_vars) - channels = [ - ( - channel_id, - evaluate_name_template( - rule.name_template, {**family_vars, "channel": str(rule.channel_start + channel_id - 1)} - ), - ) - for channel_id in range(1, rule.channel_count + 1) - ] - return parent_name, channels - - -def _has_flat_expansion(module): # pragma: no cover - requires channelization support - """Return True when *module* carries more interfaces than its module type's templates describe. - - A flat breakout leaves N-1 rows beyond the templates, so the surplus is the structural mark of a - family an earlier apply installed. Counting templates rather than their resolved names keeps - two templates that resolve to the same string from reading as one. - """ - from dcim.models import Interface, InterfaceTemplate - - templates = InterfaceTemplate.objects.filter(module_type_id=module.module_type_id).count() - return Interface.objects.filter(module=module).count() > templates - - -def _first_taken_name(device, names, exclude_pk): # pragma: no cover - requires channelization support - """Return the first of *names* already used by another interface on *device*, or None.""" - for name in names: - if _name_exists_on_device(device, name, exclude_pk=exclude_pk): - return name - return None - - -def _build_channelized_family(rule, base, variables, module, conflicts): # pragma: no cover - see above - """Turn a plain base interface into a channelized family; return how many rows it changed. - - The whole family is preflighted before anything is written: a module that already carries a flat - family, or a single occupied name — the parent's or any channel's — leaves the base exactly as - it was instead of half converting it. - """ - from dcim.choices import InterfaceTypeChoices - from dcim.models import Interface - - device = module.device - base_name = base.name - parent_name, channels = _channelized_family_names(rule, base_name, variables) - if _has_flat_expansion(module): - # Converting one sibling into a parent would strand the others beside the new family. - logger.warning( - "Module %s already carries a flat breakout family; rule '%s' will not convert interface " - "%r into the channelized parent %r — converting an installed family is a separate, " - "explicit operation. Skipping.", - module, - rule, - base.name, - parent_name, - ) - _record_skip(conflicts, device, base.name, parent_name, base.pk) - return 0 - blocker = _first_taken_name(device, [parent_name, *(name for _, name in channels)], base.pk) - if blocker is not None: - logger.warning( - "Cannot build the channelized family for interface %r on device %s: %r is already taken; skipping.", - base.name, - device, - blocker, - ) - _record_skip(conflicts, device, base.name, blocker, base.pk) - return 0 - - count = 0 - with transaction.atomic(): - created_channels = [] - base.channels = rule.channel_count - if parent_name != base.name: - base.name = parent_name - count += 1 - base.full_clean() - base.save() - for channel_id, name in channels: - channel = Interface( - device=device, - module=module, - name=name, - type=InterfaceTypeChoices.TYPE_CHANNEL, - parent=base, - channel_id=channel_id, - enabled=base.enabled, - ) - channel.full_clean() - channel.save() - created_channels.append((channel, name)) - count += 1 - _preserve_names_across_parent_cascade(base, base_name, created_channels) - return count - - -def _apply_channelized_rule(rule, base, variables, module, conflicts): - """Build the channelized family *rule* describes on a plain base interface. - - Returns None where NetBox cannot model channels: the rule describes a topology this release has - no rows for, and building a flat family instead would silently give the operator another one. - """ - if not supports_channelization(): - logger.warning( - "Rule '%s' builds a channelized family, which this NetBox release cannot model; " - "leaving interface %r unchanged.", - rule, - base.name, - ) - return None - return _build_channelized_family(rule, base, variables, module, conflicts) # pragma: no cover - see above - - -def _apply_rule_with_family(rule, iface, children, variables, module, conflicts): - """Apply *rule* to *iface*, carrying its channel subinterfaces along. - - Returns the number of interfaces renamed/created, or None when a channelized family was - skipped for a structural reason. Interfaces that own no family take the plain path unchanged. - """ - if rule.channel_count > 0 and (_is_channelized_parent(iface) or children): # pragma: no cover - return _apply_breakout_rule_to_family(rule, iface, children, variables, module, conflicts) - if children: # pragma: no cover - requires channelization support - return _apply_simple_rule_to_family(rule, iface, children, variables, module, conflicts) - if rule.channel_count > 0 and _is_channelized_rule(rule): - return _apply_channelized_rule(rule, iface, variables, module, conflicts) - return _apply_rule_to_interface(rule, iface, {**variables, "base": iface.name}, module, conflicts=conflicts) - - -def _create_channel(iface, module, new_name, device, conflicts): - """Create a breakout channel interface *new_name*; return 1 if created, else 0. - - Silently skips when this module already has the channel (idempotent - re-apply); records a conflict when *new_name* is taken by a different - interface on the device. - """ - from dcim.models import Interface - - if Interface.objects.filter(module=module, name=new_name).exists(): - return 0 # idempotent: channel already created on this module - if _name_exists_on_device(device, new_name): - _record_conflict(conflicts, device, iface.name, new_name, iface.pk) - return 0 - breakout_iface = Interface( - device=device, - module=module, - name=new_name, - type=iface.type, - enabled=iface.enabled, - ) - breakout_iface.full_clean() - breakout_iface.save() - return 1 - - -def _apply_rule_to_interface(rule, iface, variables, module, conflicts=None): - """Apply a single rule to an interface, handling breakout channels. - - All saves are wrapped in a transaction so a failure mid-breakout rolls - back any partially created interfaces. A computed name that already exists - on the device is skipped (logged, and recorded in *conflicts* when a list - is passed) instead of raising — so automatic renaming (module install, - module-type change, VC change) never aborts the rest of the batch on a - name collision. - - Returns the number of interfaces renamed/created. - """ - count = 0 - device = module.device - - with transaction.atomic(): - if rule.channel_count > 0: - # Breakout: rename base interface and create additional channel interfaces - for ch in range(rule.channel_count): - variables["channel"] = str(rule.channel_start + ch) - new_name = evaluate_name_template(rule.name_template, variables) - if ch == 0: - count += _rename_in_place(iface, new_name, device, conflicts) - else: - count += _create_channel(iface, module, new_name, device, conflicts) - else: - # Simple rename (converter offset, platform naming, etc.) - new_name = evaluate_name_template(rule.name_template, variables) - count += _rename_in_place(iface, new_name, device, conflicts) - - return count - - -def _find_channel_base(rule, ifaces, variables): - """Find the best 'base' interface for a channel rule on a single module. - - Prefers an interface whose current name already equals the expected ch=0 name - (i.e. it has already been renamed to channel 0 and is safe to re-process). - Falls back to the first interface (alphabetically) so that on first apply, - the template-created base interface becomes channel 0. - - This ensures apply_rule_to_existing / find_interfaces_for_rule call - _apply_rule_to_interface exactly ONCE per module for channel rules, preventing - duplicate-name IntegrityErrors when channels already exist. - """ - if not ifaces: - return None - for iface in ifaces: - vars_copy = dict(variables) - vars_copy["base"] = iface.name - vars_copy["channel"] = str(rule.channel_start) # ch=0 - try: - ch0_name = evaluate_name_template(rule.name_template, vars_copy) - if iface.name == ch0_name: - return iface - except ValueError: - pass - return ifaces[0] - - -def _matching_moduletype_pks(module_type_pattern): - """Return PKs of ModuleTypes whose model name matches the given regex pattern. - - Raises ValueError for invalid regex patterns, mirroring evaluate_name_template's - error-handling convention so callers can treat both as ValueError. - """ - from dcim.models import ModuleType - - try: - compiled = re.compile(module_type_pattern) - except re.error as exc: - raise ValueError(f"Invalid module_type_pattern regex '{module_type_pattern}': {exc}") from exc - return [mt.pk for mt in ModuleType.objects.only("pk", "model") if compiled.fullmatch(mt.model)] - - -def has_applicable_interfaces(rule) -> bool: - """Check whether applying this rule right now would rename at least one interface. - - Calls find_interfaces_for_rule(limit=1) to determine if any currently installed - interface would receive a new name. Returns False when: - - no matching modules/interfaces are installed, OR - - all matching interfaces are already correctly named. - - This is more expensive than a plain EXISTS query but ensures the Applicable - column in the Apply Rules list accurately reflects "would something change?" - rather than the misleading "do interfaces exist?". - """ - try: - results, _ = find_interfaces_for_rule(rule, limit=1) - return len(results) > 0 - except (ValueError, re.error): - return False - - -def _build_module_qs(rule): - """Return a Module queryset filtered to the rule's scope (module type, parent, device, platform). - - Shared by ``find_interfaces_for_rule`` and ``apply_rule_to_existing`` to avoid - duplicating the filtering logic. + Shared by ``find_interfaces_for_rule`` and ``apply_rule_to_existing`` to avoid + duplicating the filtering logic. """ from dcim.models import Module @@ -1832,162 +561,105 @@ def _build_module_qs(rule): return qs -def _name_detail(name, role, channel_id=None) -> dict: - """Describe one previewed name so the UI can render a family as a family. - - *role* is ``interface`` (a plain rename), ``parent`` (the family's physical interface) or - ``channel``; *channel_id* is the parent channel a channel name is bound to, when known. - """ - return {"name": name, "role": role, "channel_id": channel_id} - - -def _family_entry(module, parent, details, children) -> dict | None: - """Build a family preview entry from per-name *details*, or None when nothing would change. - - The entry stays keyed on the parent — the PK the Apply view submits — and lists the family's - names in ``new_names``, so the existing template loop keeps working unchanged. - """ - if [detail["name"] for detail in details] == [parent.name, *(child.name for child in children)]: - return None - return { - "module": module, - "interface": parent, - "current_name": parent.name, - "new_names": [detail["name"] for detail in details], - "name_details": details, - } +_PREVIEW_ROLES = { + family_ops.MemberRole.PARENT: "parent", + family_ops.MemberRole.CHANNEL: "channel", +} -def _evaluate_plain_interface(rule, module, iface, variables, children=()) -> dict | None: - """Return a result dict if *iface* or one of its channels would be renamed by *rule*, else None. +def _member_detail(plan, member) -> family_ops.PlannedName: + """Describe one planned member so the UI can render a family as a family. - A channelized parent is previewed together with its channels, so the Apply page shows the whole - family behind the one PK it submits. + A flat family's members are the channels a breakout rule spells out; a plan that holds only + one of them is a plain rename, not a family. """ - vars_copy = {**variables, "base": iface.name} - try: - new_name = evaluate_name_template(rule.name_template, vars_copy) - except ValueError as exc: - new_name = f"" - if children: # pragma: no cover - requires channelization support - return _family_entry(module, iface, _lockstep_details(new_name, iface, children, module), children) - return _family_entry(module, iface, [_name_detail(new_name, "interface")], ()) - + role = _PREVIEW_ROLES.get(member.role) or ("channel" if len(plan.members) > 1 else "interface") + return family_ops.PlannedName(member.target_name, role, member.channel_id) -def _lockstep_details(new_name, parent, children, module) -> list: # pragma: no cover - see above - """Per-name preview details for a family renamed in lockstep with its parent. - A channel whose suffix cannot be derived previews as unchanged — the same thing the apply path - does with it. - """ - details = [_name_detail(new_name, "parent")] - for child, target in _child_target_names(children, parent.name, new_name, module): - details.append(_name_detail(child.name if target is None else target, "channel", child.channel_id)) - return details +def _plan_details(plan) -> list: + """Describe every name the plan intends, or the error that stopped it from naming them.""" + if plan.precondition_status != family_ops.FamilyStatus.FAILED: + return [_member_detail(plan, member) for member in plan.members] + root = _member_detail(plan, plan.members[0]) + return [family_ops.PlannedName(f"", root.role, root.channel_id)] -def _channelized_family_entry(rule, module, parent, children, variables) -> dict | None: # pragma: no cover - """Preview a breakout rule against an already-channelized family. +def _plan_changes_names(plan, existing_names) -> bool: + """Return whether the planned family would rename or create anything. - Nothing is created — only the existing channels are renamed, plus the parent when the rule - names one — so a family whose channel count disagrees with the rule previews as no change at - all. + A plan that renames members compares intent with the names they carry now. A plan that builds + a family out of one base compares intent with the names the module already holds, so a family + an earlier apply already installed previews as no change. """ - if getattr(parent, "channels", None) != rule.channel_count: - return None - parent_name = parent.name - if _is_channelized_rule(rule) and rule.parent_name_template: - try: - parent_name = evaluate_name_template(rule.parent_name_template, {**variables, "base": parent.name}) - except ValueError as exc: - parent_name = f"" - details = [_name_detail(parent_name, "parent")] - for child in children: - channel = str(rule.channel_start + child.channel_id - 1) - try: - new_name = evaluate_name_template( - rule.name_template, {**variables, "base": parent.name, "channel": channel} - ) - except ValueError as exc: - new_name = f"" - details.append(_name_detail(new_name, "channel", child.channel_id)) - return _family_entry(module, parent, details, children) - - -def _channelized_family_preview(rule, module, base, variables) -> dict | None: # pragma: no cover - see below - """Describe the channelized family a rule would build on a plain base interface.""" - try: - parent_name, channels = _channelized_family_names(rule, base.name, variables) - except ValueError as exc: - return _family_entry(module, base, [_name_detail(f"", "parent")], ()) - details = [_name_detail(parent_name, "parent")] - details.extend(_name_detail(name, "channel", channel_id) for channel_id, name in channels) - return _family_entry(module, base, details, ()) + if plan.base_name is None: # pragma: no cover - requires channelization support + return plan.target_names != plan.source_names + return plan.target_names[0] != plan.base_name or any( + target_name not in existing_names for target_name in plan.target_names + ) -def _channelized_creation_entry(rule, module, bases, variables) -> dict | None: - """Return the preview entry for the family a channelized rule would build, or None for none. +def _plan_entry(module, plan, interface, existing_names) -> dict | None: + """Build the preview entry for one family plan, or None when it would change nothing. - A release that cannot model channels previews nothing, because the apply path builds nothing - there either; neither does a module whose flat family the apply path refuses to convert. + The entry stays keyed on the interface the Apply view submits, and lists the family's names in + ``new_names``, so the existing template loop keeps working unchanged. A plan the live topology + blocks previews nothing, because the apply path would build nothing either. """ - if not supports_channelization(): + failed = plan.precondition_status == family_ops.FamilyStatus.FAILED + if plan.precondition_status is not None and not failed: return None - if _has_flat_expansion(module): # pragma: no cover - requires channelization support + if not failed and not _plan_changes_names(plan, existing_names): return None - return _channelized_family_preview( # pragma: no cover - requires channelization support - rule, module, _find_channel_base(rule, bases, variables), variables - ) - + details = _plan_details(plan) + return { + "module": module, + "interface": interface, + "current_name": interface.name, + "new_names": [detail.name for detail in details], + "name_details": details, + } -def _channel_rule_entries(rule, module, bases, children_by_parent, variables) -> list: - """Return the preview entries a channel rule produces for one module. - A module whose base is already channelized previews per family (renames only); a channelized - rule on a plain base previews the family it would build; anything else keeps the flat breakout - preview of one entry per module. - """ - families = [base for base in bases if _is_channelized_parent(base)] - if families: # pragma: no cover - requires a NetBox that models channelization - entries = [ - _channelized_family_entry(rule, module, parent, children_by_parent.get(parent.pk, ()), variables) - for parent in families - ] - return [entry for entry in entries if entry] - if _is_channelized_rule(rule): - entry = _channelized_creation_entry(rule, module, bases, variables) - return [entry] if entry else [] - entry = _channel_rule_entry(rule, module, bases, variables) - return [entry] if entry else [] - - -def _channel_rule_entry(rule, module, ifaces, variables) -> dict | None: - """Return a result dict if the channel rule would change any name for this module, else None.""" - base_iface = _find_channel_base(rule, ifaces, variables) - if base_iface is None: - return None - vars_copy = {**variables, "base": base_iface.name} - expected_names = [] - try: - for ch in range(rule.channel_count): - expected_names.append( - evaluate_name_template(rule.name_template, {**vars_copy, "channel": str(rule.channel_start + ch)}) - ) - except ValueError as exc: - expected_names = [f""] - existing_names = {i.name for i in ifaces} - # Report if any channel name is missing or the base itself needs renaming - if any(n not in existing_names for n in expected_names) or ( - expected_names and expected_names[0] != base_iface.name - ): - return { - "module": module, - "interface": base_iface, - "current_name": base_iface.name, - "new_names": expected_names, - "name_details": [_name_detail(name, "channel") for name in expected_names], - } - return None +def _plan_root_name(plan) -> str: + """Return the name of the interface a plan is submitted through.""" + return plan.base_name if plan.base_name is not None else plan.members[0].source_name + + +def _preview_plans(rule, plan_set) -> list: + """Return the plans this preview reports. + + A breakout rule on a module that already models channelized families renames those families and + adds none beside them. Anywhere else it builds one family per base, and two bases that intend + the same names are the one family an earlier apply already started, so it is offered once: the + same family the apply path would build. + """ + if rule.channel_count <= 0: + return list(plan_set.plans) + installed = [plan for plan in plan_set.plans if plan.base_name is None] + if installed: # pragma: no cover - requires a NetBox that models channelization + return installed + creations = [plan for plan in plan_set.plans if plan.base_name is not None] + kept = family_targets.one_family_per_name_set([(plan.base_name, plan.target_names) for plan in creations]) + return [creations[index] for index in kept] + + +def _process_module(rule, module, ifaces, variables, limit, results, module_qs, processed_pks): + """Preview one module from its family plans. Returns (checked_count, should_stop).""" + plan_set = family_ops.plan_prospective_families(module, rule, variables, family_ops.describe_interfaces(ifaces)) + checked = len(plan_set.plans) + if not checked: + return 0, False + rows_by_name = {iface.name: iface for iface in ifaces} + existing_names = frozenset(rows_by_name) + for plan in _preview_plans(rule, plan_set): + entry = _plan_entry(module, plan, rows_by_name[_plan_root_name(plan)], existing_names) + if entry is None: + continue + results.append(entry) + if limit is not None and len(results) >= limit: + return checked + _count_remaining_interfaces(module_qs, processed_pks), True + return checked, False def _count_remaining_interfaces(module_qs, processed_pks) -> int: @@ -2000,35 +672,6 @@ def _count_remaining_interfaces(module_qs, processed_pks) -> int: return qs.count() -def _process_channel_module(rule, module, ifaces, variables, limit, results, module_qs, processed_pks): - """Process one module for a channel rule. Returns (checked_count, should_stop).""" - bases, children_by_parent = _partition_families(ifaces) - checked = len(bases) - if not bases: - return checked, False - for entry in _channel_rule_entries(rule, module, bases, children_by_parent, variables): - results.append(entry) - if limit is not None and len(results) >= limit: - return checked + _count_remaining_interfaces(module_qs, processed_pks), True - return checked, False - - -def _process_plain_module(rule, module, ifaces, variables, limit, results, module_qs, processed_pks): - """Process one module for a plain (non-channel) rule. Returns (checked_count, should_stop).""" - bases, children_by_parent = _partition_families(ifaces) - checked = 0 - for iface_idx, iface in enumerate(bases): - checked += 1 - entry = _evaluate_plain_interface(rule, module, iface, variables, children_by_parent.get(iface.pk, ())) - if entry: - results.append(entry) - if limit is not None and len(results) >= limit: - checked += len(bases) - (iface_idx + 1) - checked += _count_remaining_interfaces(module_qs, processed_pks) - return checked, True - return checked, False - - def find_interfaces_for_rule(rule, limit=None): """Find interfaces that would be renamed by applying the given rule retroactively. @@ -2042,7 +685,7 @@ def find_interfaces_for_rule(rule, limit=None): "interface": Interface instance, "current_name": str, "new_names": list[str], # one entry per channel, or single-element - "name_details": list[dict], # {"name", "role", "channel_id"} per new_names entry + "name_details": list[PlannedName], # name, role and channel id per new_names entry } Only includes entries where at least one new_name differs from current_name. @@ -2062,8 +705,6 @@ def find_interfaces_for_rule(rule, limit=None): "module_bay", "module_bay__parent", ) - process_fn = _process_channel_module if rule.channel_count > 0 else _process_plain_module - # Batch-load all interfaces for matching modules to avoid N+1 queries. ifaces_by_module = defaultdict(list) for iface in Interface.objects.filter(module__in=module_qs).order_by("module_id", "name"): @@ -2076,7 +717,7 @@ def find_interfaces_for_rule(rule, limit=None): processed_pks.add(module.pk) variables = build_variables(module.module_bay, device=module.device) ifaces = ifaces_by_module.get(module.pk, []) - checked, stop = process_fn(rule, module, ifaces, variables, limit, results, module_qs, processed_pks) + checked, stop = _process_module(rule, module, ifaces, variables, limit, results, module_qs, processed_pks) total_checked += checked if stop: return results, total_checked @@ -2084,128 +725,37 @@ def find_interfaces_for_rule(rule, limit=None): return results, total_checked -def _apply_channel_rule_to_module(rule, module, ifaces, variables, id_set, conflicts): - """Apply a channel rule to one module via its base interface; return the rename count. - - On a module whose base is already channelized the rule renames the existing channels, once per - family. Otherwise the rule is processed ONCE per module (not per interface) so existing - channel names are not re-created. An unexpected failure (e.g. a save race) is logged and - skipped so it never aborts the surrounding batch. - """ - bases, children_by_parent = _partition_families(ifaces) - if not bases: - return 0 - families = [base for base in bases if _is_channelized_parent(base)] - if families: # pragma: no cover - requires a NetBox that models channelization - count = 0 - for parent in families: - if id_set is not None and parent.pk not in id_set: - continue - count += _apply_family(rule, parent, children_by_parent.get(parent.pk, ()), variables, module, conflicts) - return count - base_iface = _find_channel_base(rule, bases, variables) - if id_set is not None and base_iface.pk not in id_set: - return 0 - vars_copy = dict(variables) - vars_copy["base"] = base_iface.name - try: - if _is_channelized_rule(rule): - return _apply_channelized_rule(rule, base_iface, variables, module, conflicts) or 0 - return _apply_rule_to_interface(rule, base_iface, vars_copy, module, conflicts=conflicts) - except (ValueError, ValidationError, IntegrityError): - logger.exception( - "Failed to apply channel rule '%s' to module '%s' (id=%s); skipping.", - rule, - module, - module.pk, +def _batch_modules(rule): + """Return the rule's modules with every relation planning and template resolution dereference.""" + return list( + _build_module_qs(rule).select_related( + "module_type", + "device__device_type", + "device__platform", + *family_template_names.BAY_CHAIN_RELATIONS, ) - return 0 - - -def _apply_family(rule, iface, children, variables, module, conflicts): - """Apply *rule* to one base interface and its channels, logging (never raising) on failure.""" - try: - return _apply_rule_with_family(rule, iface, children, variables, module, conflicts) or 0 - except (ValueError, ValidationError, IntegrityError): - logger.exception( - "Failed to apply rule '%s' to interface '%s' (id=%s); skipping.", - rule, - iface.name, - iface.pk, - ) - return 0 - - -def _apply_plain_rule_to_module(rule, module, ifaces, variables, id_set, conflicts): - """Apply a non-channel rule to each selected interface on one module; return the rename count. - - Each base interface is independent: an unexpected failure on one is logged and skipped so the - rest of the module (and batch) still process. Channel subinterfaces are not selectable on - their own — they are renamed only as part of the family whose parent was selected. - """ - bases, children_by_parent = _partition_families(ifaces) - count = 0 - for iface in bases: - if id_set is not None and iface.pk not in id_set: - continue - count += _apply_family(rule, iface, children_by_parent.get(iface.pk, ()), variables, module, conflicts) - return count + ) -def apply_rule_to_existing(rule, limit=None, interface_ids=None, conflicts=None): +def apply_rule_to_existing(rule, limit=None, interface_ids=None) -> family_ops.BatchOutcome: """Apply a rule retroactively to all matching installed modules. - Unlike apply_interface_name_rules(), this does not skip already-renamed - interfaces — it re-evaluates every interface on each matching module. - - For channel rules (channel_count > 0), each module is processed as a single - unit using _find_channel_base() to pick the base interface. Calling - _apply_rule_to_interface for every interface in the module would produce - duplicate-name IntegrityErrors when channel interfaces already exist. + Unlike apply_interface_name_rules(), this does not skip already-renamed interfaces: it plans + every family each matching module carries or would gain, and executes each in its own + transaction, so one blocked family costs the batch only that family. - If *interface_ids* is provided (list/set of Interface PKs), only those - interfaces are processed; all others are skipped. For channel rules the - base interface PK is used as the selector. An empty *interface_ids* - collection returns 0 immediately without touching the database. Selecting a - channelized parent brings its channel subinterfaces along; selecting a channel - subinterface on its own does nothing, because it is not an independent candidate. + If *interface_ids* is provided (list/set of Interface PKs), only the families those interfaces + reach are applied; an empty collection touches the database not at all. Selecting a + channelized parent brings its channel subinterfaces along; selecting a channel subinterface on + its own does nothing, because it is not an independent candidate. If *limit* is set the batch + stops after the module that reached that many changed interfaces. - If *conflicts* is a list, each interface skipped because its target name is - already taken on the device is appended to it (and logged) — letting the - caller report how many renames were dropped. Collisions never raise. - - Returns the number of interfaces renamed/created. + Returns the batch outcome: one explicit family result per family it planned. """ - from dcim.models import Interface - id_set = frozenset(interface_ids) if interface_ids is not None else None - if id_set is not None and not id_set: - return 0 - - if not rule.enabled: - return 0 - - module_qs = _build_module_qs(rule) - - # Batch-load interfaces to avoid N+1 queries in the module loop. - ifaces_by_module = defaultdict(list) - for iface in Interface.objects.filter(module__in=module_qs).order_by("module_id", "name"): - ifaces_by_module[iface.module_id].append(iface) - - count = 0 - for module in module_qs.select_related("module_bay", "module_type", "device", "device__virtual_chassis"): - variables = build_variables(module.module_bay, device=module.device) - ifaces = ifaces_by_module.get(module.pk, []) - - if rule.channel_count > 0: - count += _apply_channel_rule_to_module(rule, module, ifaces, variables, id_set, conflicts) - else: - count += _apply_plain_rule_to_module(rule, module, ifaces, variables, id_set, conflicts) - - if limit is not None and count >= limit: - return count - - return count + if not rule.enabled or (id_set is not None and not id_set): + return family_ops.BatchOutcome(families=()) + return family_ops.apply_rule_to_modules(rule, _batch_modules(rule), selected_pks=id_set, limit=limit) # --------------------------------------------------------------------------- @@ -2215,400 +765,28 @@ def apply_rule_to_existing(rule, limit=None, interface_ids=None, conflicts=None) # with N channel subinterfaces. Converting one rewrites rows an operator owns — cables, addresses, # tags — so it is never a side effect of applying a rule: the operator confirms it per family. -# The ch-0 row, the names its family carries now, and the names it would carry once converted. -_ConversionFamily = namedtuple("_ConversionFamily", ("module", "base", "current_names", "parent_name", "channel_names")) +def find_convertible_families(rule, limit=None) -> family_ops.ConversionPreview: + """Return the preview of the flat families *rule* could convert, convertible or not. -def _conversion_offered(rule): - """Return True when *rule* describes a topology an installed flat family could be converted into. - - A disabled rule renames nothing on any apply path, so it converts nothing either. A flat family - has no parent row — its ch-0 interface *is* the base — so without a parent name there is nowhere - for that base to go, and the conversion is not offered at all. - """ - return rule.enabled and _is_channelized_rule(rule) and rule.channel_count > 0 and bool(rule.parent_name_template) - - -def _base_marked_ch0_name(rule, variables): # pragma: no cover - requires channelization support - """Return *rule*'s escaped ch-0 output with ``{base}`` left as a sentinel, or None when it cannot be. - - Evaluated once per rule: the sentinel stands in for ``{base}`` so a raw matcher can be spliced - over it afterwards. - """ - try: - evaluated = evaluate_name_template( - rule.name_template, {**variables, "base": _BASE_SENTINEL, "channel": str(rule.channel_start)} - ) - except (ValueError, TypeError): - # A {base} inside an arithmetic expression cannot take a non-numeric stand-in — see the docs. - logger.debug( - "Rule '%s' evaluates {base} arithmetically, so a base predating a virtual-chassis " - "position change cannot be recovered from its output names; not offering its families.", - rule, - ) - return None - if _BASE_SENTINEL not in evaluated: - return None # the rule's output does not carry the base, so no drift reached it - return re.escape(evaluated) - - -def _recovered_bases(rule, interfaces, variables, matchers): # pragma: no cover - channelization only - """Return the historical ``{base}`` values *rule*'s installed families still spell on this module. - - A flat family carries rule-*output* names, so a raw matcher cannot be run against them directly: - it is spliced into the rule's own ch-0 output as a capture instead — a repeated ``{base}`` becomes - a backreference rather than a second group — and the capture yields the base the family was named - with. Conversion rewrites rows an operator owns, so anything ambiguous (one matcher over two - bases, or two templates recovering the same one) yields nothing at all. - """ - marked = _base_marked_ch0_name(rule, variables) - if marked is None: - return [] - head, _, tail = marked.partition(_BASE_SENTINEL) - tail = tail.replace(_BASE_SENTINEL, "(?P=base)") - recovered = defaultdict(int) - for matcher in matchers: - family_pattern = _compile_pattern(f"{head}(?P{matcher.pattern.pattern}){tail}") - if family_pattern is None: - continue - matches = (family_pattern.fullmatch(iface.name) for iface in interfaces) - bases = {match.group("base") for match in matches if match} - if len(bases) == 1: - recovered[bases.pop()] += 1 - return [base for base, claims in recovered.items() if claims == 1] - - -def _family_on(rule, module, by_name, variables, base_name): # pragma: no cover - channelization only - """Return the family *rule* describes on *base_name*, or None when this module carries none.""" - family_vars = {**variables, "base": base_name} - parent_name = evaluate_name_template(rule.parent_name_template, family_vars) - channel_names = [ - evaluate_name_template(rule.name_template, {**family_vars, "channel": str(rule.channel_start + offset)}) - for offset in range(rule.channel_count) - ] - base = by_name.get(channel_names[0]) - if base is None or _is_channel_child(base) or _is_channelized_parent(base): - return None - return _ConversionFamily( - module=module, - base=base, - current_names=[name for name in channel_names if name in by_name], - parent_name=parent_name, - channel_names=channel_names, - ) - - -def _conversion_family(rule, module, interfaces, variables, raw): # pragma: no cover - channelization only - """Return the flat family *rule* would convert on *module*, or None when it carries none. - - Identification is by name: ``name_template`` is evaluated over the rule's channel range against - each raw template name, and the ch-0 name has to still be a plain interface — a family that was - already converted (its ch-0 name now belongs to a channel row) is therefore never offered twice. - A family named before this device's virtual-chassis position changed spells a base no template - resolves to any more, so those bases are recovered from the family's own names and then - identified exactly the same way. + Each candidate names the ch-0 row the confirm form submits, the family's current names, the + names it would carry, and where the ch-0 row's configuration lands. A family beyond *limit* is + never dry-run; the preview reports that one was left unexamined. """ - by_name = {iface.name: iface for iface in interfaces} - for base_name in sorted(raw.names): - family = _family_on(rule, module, by_name, variables, base_name) - if family is not None: - return family - for base_name in _recovered_bases(rule, interfaces, variables, raw.matchers): - family = _family_on(rule, module, by_name, variables, base_name) - if family is not None: - return family - return None + # Only the cheap half of the guard, so a rule that offers no conversion never reads its modules; + # whether this release can hold a family is the family package's call. + if not family_ops.conversion_offered(rule): + return family_ops.ConversionPreview(candidates=()) + return family_ops.preview_rule_conversions(rule, _batch_modules(rule), limit=limit) -def _conversion_families(rule): # pragma: no cover - requires channelization support - """Yield the flat family each module in *rule*'s scope still carries. - - A flat breakout is applied once per module (see ``_apply_channel_rule_to_module``), so a module - carries at most one such family and the conversion mirrors that. - """ - from dcim.models import Interface - - modules = list(_build_module_qs(rule).select_related("module_type", "device", *_BAY_CHAIN_RELATIONS)) - raw_by_module = _raw_names_by_module(modules) - ifaces_by_module = defaultdict(list) - for iface in Interface.objects.filter(module__in=[module.pk for module in modules]).order_by("module_id", "name"): - ifaces_by_module[iface.module_id].append(iface) - for module in modules: - variables = build_variables(module.module_bay, device=module.device) - ifaces = ifaces_by_module.get(module.pk, []) - family = _conversion_family(rule, module, ifaces, variables, raw_by_module[module.pk]) - if family is not None: - yield family - - -def _validate_or_block(iface, role): # pragma: no cover - requires channelization support - """Run NetBox's own validation on *iface*, restating a rejection as this family's blocking reason.""" - try: - iface.full_clean() - except ValidationError as exc: - raise ValidationError(f"{role} {iface.name!r}: {' '.join(exc.messages)}") from exc - - -def _split_ch0_row(rule, family, base): # pragma: no cover - requires channelization support - """Make *base* the family's parent and move its logical identity onto a new channel-1 child. - - Everything an operator configured on the ch-0 row described a channel, not the cage carrying it, - so addresses, VLANs, MTU, description and tags move; custom fields can mean either thing and are - copied. The physical row keeps its pk, cable, type, module link and mark_connected. - """ - from dcim.choices import InterfaceTypeChoices - from dcim.models import Interface - - carried = { - "description": base.description, - "mtu": base.mtu, - "mode": base.mode, - "untagged_vlan_id": base.untagged_vlan_id, - } - tagged_vlans = list(base.tagged_vlans.all()) - tags = list(base.tags.all()) - - base.name = family.parent_name - base.channels = rule.channel_count - base.description = "" - base.mtu = None - base.mode = "" - base.untagged_vlan = None - _validate_or_block(base, "parent") - base.save() # BaseInterface.save() drops the tagged VLANs of an interface that no longer tags - base.tags.clear() - - channel = Interface( - device=family.module.device, - module=family.module, - name=family.channel_names[0], - type=InterfaceTypeChoices.TYPE_CHANNEL, - parent=base, - channel_id=1, - enabled=base.enabled, - custom_field_data=dict(base.custom_field_data or {}), - **carried, - ) - _validate_or_block(channel, "channel") - channel.save() - channel.tagged_vlans.set(tagged_vlans) - channel.tags.set(tags) - base.ip_addresses.all().update(assigned_object_id=channel.pk) - base.fhrp_group_assignments.all().update(interface_id=channel.pk) - - -def _rewrite_family(rule, family): # pragma: no cover - requires channelization support - """Convert *family* in place, raising ValidationError with the reason when it cannot be converted. - - Only what upstream cannot decide for us is checked here: the parent's name has to be free, every - sibling has to be present, a sibling already bound to another parent's channel is not ours to - take, and a cabled sibling cannot become a channel — TYPE_CHANNEL is nonconnectable but not - virtual, so ``Interface.clean()`` accepts a cable on one. Everything else is left to - ``full_clean()`` on each prospective row, which inherits upstream's rules as they grow. - """ - from dcim.choices import InterfaceTypeChoices - from dcim.models import Interface - - device = family.module.device - # Locked for the transaction: the checks below act on this snapshot, and the saves write it back. - by_name = {iface.name: iface for iface in Interface.objects.select_for_update().filter(module=family.module)} - base = by_name.get(family.channel_names[0]) - if base is None or base.pk != family.base.pk: - raise ValidationError( - f"{family.channel_names[0]!r} is gone or replaced: the family changed since it was scanned" - ) - - if _name_exists_on_device(device, family.parent_name, exclude_pk=base.pk): - raise ValidationError(f"the parent name {family.parent_name!r} is already taken on {device}") - - siblings = [] - for channel_id, name in enumerate(family.channel_names[1:], start=2): - sibling = by_name.get(name) - if sibling is None or sibling.pk == base.pk: - raise ValidationError(f"{name!r} is missing: this module carries no complete flat family") - # Rebinding it validates cleanly, so only this check keeps the other family whole. - if _is_channel_child(sibling): - owner = sibling.parent.name if sibling.parent_id else "another parent" - raise ValidationError( - f"{name!r} is already channel {sibling.channel_id} of {owner}; " - f"converting would take it out of that family" - ) - if sibling.cable_id: - raise ValidationError(f"{name!r} has a cable attached; a channel takes its cable from the parent") - siblings.append((channel_id, sibling)) - - _split_ch0_row(rule, family, base) - for channel_id, sibling in siblings: - sibling.type = InterfaceTypeChoices.TYPE_CHANNEL - sibling.parent = base - sibling.channel_id = channel_id - _validate_or_block(sibling, "channel") - sibling.save() - - -def _convert_family(rule, family, commit): # pragma: no cover - requires channelization support - """Convert *family*; return an empty string on success, or the reason it was refused. - - The whole conversion runs inside one savepoint, so a dry run (*commit* False) and a family that - turns out to be unconvertible both leave every row exactly as it was — the rows are re-read here - too, so a rolled-back dry run cannot hand mutated objects back to the caller. - """ - try: - with transaction.atomic(): - _rewrite_family(rule, family) - if not commit: - transaction.set_rollback(True) - except (ValidationError, IntegrityError, ValueError) as exc: - return "; ".join(getattr(exc, "messages", [str(exc)])) - return "" - - -def _conversion_metadata_note(family): # pragma: no cover - requires channelization support - """Return the sentence the Apply page shows about where the ch-0 row's configuration ends up.""" - return ( - f"The addresses, VLANs, MTU, description and tags on {family.base.name} move to the new " - f"channel 1 interface that takes over that name; custom field values are copied. The physical " - f"row keeps its ID and becomes the parent {family.parent_name}, so automation keyed on that " - f"interface ID will address the parent afterwards." - ) - - -def _conversion_verdict(family, reason): # pragma: no cover - requires channelization support - """Describe what converting *family* would do, and why it cannot be done when it cannot.""" - details = [_name_detail(family.parent_name, "parent")] - details.extend( - _name_detail(name, "channel", channel_id) for channel_id, name in enumerate(family.channel_names, start=1) - ) - return { - "module": family.module, - "interface": family.base, - "current_name": family.base.name, - "current_names": family.current_names, - "new_names": [family.parent_name, *family.channel_names], - "name_details": details, - "convertible": not reason, - "reason": reason, - "metadata_note": _conversion_metadata_note(family), - } - - -def find_convertible_families(rule, limit=None) -> tuple: - """Return ``(verdicts, has_more)`` for the flat families *rule* could convert, convertible or not. - - Nothing is written: every family is converted inside a savepoint that is rolled back again, so - each verdict carries the reason NetBox itself would refuse the family rather than a guess at its - rules. Each verdict names the ch-0 row the confirm form submits, the family's current names, - the names it would carry, and where the ch-0 row's configuration lands. - - That dry run is what the scan costs, and a blocked family costs it too, so *limit* caps the - families examined — one verdict each — rather than the convertible ones among them. A family - beyond the limit is never dry-run; *has_more* reports that one was left unexamined. - """ - if not (_conversion_offered(rule) and supports_channelization()): - return [], False - return _find_convertible_families(rule, limit) # pragma: no cover - requires channelization support - - -def _find_convertible_families(rule, limit): # pragma: no cover - requires channelization support - """Dry-run at most *limit* of *rule*'s flat families; see ``find_convertible_families``.""" - verdicts = [] - for family in _conversion_families(rule): - if limit is not None and len(verdicts) >= limit: - return verdicts, True - verdicts.append(_conversion_verdict(family, _convert_family(rule, family, commit=False))) - return verdicts, False - - -def convert_flat_families(rule, base_pks=None, conflicts=None) -> int: - """Convert *rule*'s installed flat families to the channelized topology; return how many. +def convert_flat_families(rule, base_pks=None) -> family_ops.BatchOutcome: + """Convert *rule*'s installed flat families to the channelized topology. *base_pks* is the set of ch-0 interface pks the operator confirmed: ``None`` converts every - convertible family (the batch the background job runs), an empty collection converts none. A - family that cannot be converted is logged, appended to *conflicts* in the usual skipped-rename - shape and passed over — it is never half converted, and never costs the rest of the batch. - """ - if not supports_channelization(): - logger.warning( - "Rule '%s' converts flat families into the channelized topology, which this NetBox release " - "cannot model; nothing was converted.", - rule, - ) - return 0 - return _convert_flat_families(rule, base_pks, conflicts) # pragma: no cover - see above - - -def _convert_flat_families(rule, base_pks, conflicts): # pragma: no cover - requires channelization support - """Convert the confirmed flat families of *rule*; see ``convert_flat_families``.""" - if not _conversion_offered(rule): - return 0 - selected = None if base_pks is None else frozenset(base_pks) - if selected is not None and not selected: - return 0 + convertible family (the batch the background job runs), an empty collection converts none. - converted = 0 - for family in _conversion_families(rule): - if selected is not None and family.base.pk not in selected: - continue - current_name = family.base.name - reason = _convert_family(rule, family, commit=True) - if reason: - logger.warning( - "Cannot convert the flat family of interface %r on %s into the channelized parent %r: %s. Skipping.", - current_name, - family.module, - family.parent_name, - reason, - ) - _record_skip(conflicts, family.module.device, current_name, family.parent_name, family.base.pk) - continue - converted += 1 - return converted - - -def evaluate_name_template(template: str, variables: dict) -> str: - """Evaluate a name template with variable substitution and safe arithmetic. - - Supports templates like: - "GigabitEthernet{slot}/{8 + ({parent_bay_position} - 1) * 2 + {sfp_slot}}" - - Variables are substituted first, then any brace-enclosed expression - containing arithmetic operators is safely evaluated via AST. True division - (/) is not allowed — use floor division (//) instead. Results are cast to - int to ensure interface names are always whole numbers. + Returns the batch outcome: one explicit family result per family it planned. """ - # First pass: substitute all simple variables - result = template - for key, value in variables.items(): - result = result.replace(f"{{{key}}}", str(value)) - - # Second pass: evaluate any remaining brace-enclosed arithmetic expressions - def _eval_expr(match): - expr = match.group(1).strip() - # Allow digits, arithmetic operators (excluding lone /), parens, whitespace. - # Negative lookahead disallows a single / that is not part of //. - if not re.match(r"^(?!.*(?", "eval")))) # noqa: S307 - except (SyntaxError, TypeError, ZeroDivisionError) as e: - raise ValueError(f"Invalid arithmetic expression '{expr}': {e}") from e - - return re.sub(r"\{([^}]+)\}", _eval_expr, result) + selected = None if base_pks is None else frozenset(base_pks) + return family_ops.convert_rule_families(rule, _batch_modules(rule), selected_pks=selected) diff --git a/netbox_interface_name_rules/family/__init__.py b/netbox_interface_name_rules/family/__init__.py new file mode 100644 index 0000000..50aa43a --- /dev/null +++ b/netbox_interface_name_rules/family/__init__.py @@ -0,0 +1,131 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Plan and execute interface-family operations.""" + +from .batch import ( + BatchOutcome, + ModuleFamilyPlans, + apply_rule_to_modules, + execute_family_plan, + execute_module_families, + plan_module_families, +) +from .capabilities import supports_channelization +from .conversion import ( + conversion_offered, + convert_rule_families, + execute_conversion, + plan_module_conversions, + preview_rule_conversions, +) +from .domain import ( + ConversionCandidate, + ConversionMember, + ConversionPlan, + ConversionPreview, + FamilyOutcome, + FamilyStatus, + FamilyTopology, + FlatCreationPlan, + InstalledFamilyPlan, + InstalledFamilyPlanSet, + InstalledPlanSetOutcome, + InterfaceSnapshot, + MemberOutcome, + MemberRole, + PlannedChannel, + PlannedMember, + PlannedName, + ProspectiveFamilyPlan, + ProspectiveFamilyPlanSet, + ProspectiveMember, + StructuralFamilyPlan, +) +from .execution import execute_installed_plan, execute_installed_plan_set +from .installed import ( + device_interface_families, + interfaces_by_module, + is_channelized_parent, + module_db_alias, + plan_device_interface_rename, + plan_installed_families, + plan_interface_rename, +) +from .prospective import ( + ProspectiveInterface, + describe_interfaces, + describe_module_interfaces, + describe_template_interfaces, + plan_prospective_families, +) +from .structural import ( + execute_flat_family, + execute_structural_family, + has_flat_expansion, + install_channelized_family, + plan_flat_family, + plan_structural_family, +) +from .targets import channelized_family_names, one_family_per_name_set, template_channel_suffixes +from .template_names import pinned_template_cache, resolved_template_names + +__all__ = ( + "BatchOutcome", + "ConversionCandidate", + "ConversionMember", + "ConversionPlan", + "ConversionPreview", + "FamilyOutcome", + "FamilyStatus", + "FamilyTopology", + "FlatCreationPlan", + "InstalledFamilyPlan", + "InstalledFamilyPlanSet", + "InstalledPlanSetOutcome", + "InterfaceSnapshot", + "MemberOutcome", + "MemberRole", + "ModuleFamilyPlans", + "PlannedChannel", + "PlannedMember", + "PlannedName", + "ProspectiveFamilyPlan", + "ProspectiveFamilyPlanSet", + "ProspectiveInterface", + "ProspectiveMember", + "StructuralFamilyPlan", + "apply_rule_to_modules", + "channelized_family_names", + "conversion_offered", + "convert_rule_families", + "describe_interfaces", + "describe_module_interfaces", + "describe_template_interfaces", + "device_interface_families", + "execute_conversion", + "execute_family_plan", + "execute_flat_family", + "execute_installed_plan", + "execute_installed_plan_set", + "execute_module_families", + "execute_structural_family", + "has_flat_expansion", + "install_channelized_family", + "interfaces_by_module", + "is_channelized_parent", + "module_db_alias", + "one_family_per_name_set", + "pinned_template_cache", + "plan_device_interface_rename", + "plan_flat_family", + "plan_installed_families", + "plan_interface_rename", + "plan_module_conversions", + "plan_module_families", + "plan_prospective_families", + "plan_structural_family", + "preview_rule_conversions", + "resolved_template_names", + "supports_channelization", + "template_channel_suffixes", +) diff --git a/netbox_interface_name_rules/family/batch.py b/netbox_interface_name_rules/family/batch.py new file mode 100644 index 0000000..c3517eb --- /dev/null +++ b/netbox_interface_name_rules/family/batch.py @@ -0,0 +1,239 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Apply one rule to a batch of installed modules through family plans. + +Every module in the batch is planned family by family and executed family by family, so a module +that cannot take its names costs the batch only that family. The module rows, their interfaces and +each module type's templates are read once for the whole batch. +""" + +import logging +from dataclasses import dataclass + +from django.core.exceptions import ValidationError + +from ..naming import build_variables +from .domain import ( + FamilyOutcome, + FamilyStatus, + FamilyTopology, + FlatCreationPlan, + InstalledFamilyPlan, + MemberOutcome, + StructuralFamilyPlan, +) +from .execution import execute_installed_plan +from .installed import interfaces_by_module, plan_installed_families, plan_interface_rename +from .structural import execute_flat_family, execute_structural_family, plan_flat_family, plan_structural_family +from .targets import builds_channelized_family, intended_family_names, one_family_per_name_set +from .template_names import pinned_template_cache + +logger = logging.getLogger(__name__) + +# A member left with the name it had for a reason the operator can act on. An unsupported topology +# is not one of them: the release cannot hold the family, so nothing was dropped by this batch. +_SKIPPED_STATUSES = (FamilyStatus.BLOCKED, FamilyStatus.STALE, FamilyStatus.FAILED) + +_EXECUTORS = { + InstalledFamilyPlan: execute_installed_plan, + StructuralFamilyPlan: execute_structural_family, + FlatCreationPlan: execute_flat_family, +} + + +@dataclass(frozen=True, slots=True) +class ModuleFamilyPlans: + """The families a rule intends on one module, split by how each was found. + + *installed* are the families the module already carries; *leftover* are the plans for the + interfaces no installed family claimed, whether the rule renames one or builds a family on it. + """ + + installed: tuple + leftover: tuple + + @property + def plans(self) -> tuple: + """Return every plan, installed families first.""" + return (*self.installed, *self.leftover) + + +@dataclass(frozen=True, slots=True) +class BatchOutcome: + """Every family one batch operation planned, and what happened to it.""" + + families: tuple[FamilyOutcome, ...] + + @property + def changed_count(self) -> int: + """Return the number of interfaces the batch renamed or created.""" + return sum(family.changed_count for family in self.families) + + @property + def skipped_members(self) -> tuple[MemberOutcome, ...]: + """Return every member a collision, a stale plan or a failure left as it was.""" + return tuple( + member for family in self.families for member in family.members if member.status in _SKIPPED_STATUSES + ) + + @property + def changed_families(self) -> tuple[FamilyOutcome, ...]: + """Return every family this batch actually rewrote.""" + return tuple(family for family in self.families if family.status == FamilyStatus.CHANGED) + + @property + def blocked_families(self) -> tuple[FamilyOutcome, ...]: + """Return every family a collision, a stale plan or a failure left as it was.""" + return tuple(family for family in self.families if family.status in _SKIPPED_STATUSES) + + +def execute_family_plan(plan) -> FamilyOutcome: + """Execute one planned family through the executor its plan kind owns. + + A plan carrying no live rows (a prospective plan above all) has no executor, so a preview + object is refused here rather than locking anything. + """ + executor = _EXECUTORS.get(type(plan)) + if executor is None: + raise TypeError(f"{type(plan).__name__} is not an executable family plan") + return executor(plan) + + +def _is_channel(interface) -> bool: + """Return whether *interface* is bound to a parent channel.""" + return getattr(interface, "channel_id", None) is not None + + +def _creation_plan(module, rule, variables, base): + """Return the plan that builds the family *rule* describes on one plain interface.""" + if builds_channelized_family(rule): + return plan_structural_family(module, rule, variables, base) + return plan_flat_family(module, rule, variables, base) + + +def _creation_plans(module, rule, variables, plain): + """Return one creation plan per family, so two bases of one family never build it twice.""" + candidates = [(base, intended_family_names(rule, variables, base.name)) for base in plain] + kept = one_family_per_name_set([(base.name, target_names) for base, target_names in candidates]) + return [_creation_plan(module, rule, variables, candidates[index][0]) for index in kept] + + +def plan_module_families(module, rule, variables, interfaces, admit_leftover=None) -> ModuleFamilyPlans: + """Return one executable plan for every family *rule* intends on *module*. + + Every interface belongs to at most one plan: an installed family claims its members first, and + what is left over is planned as the family the rule would build on it. + + *admit_leftover* filters the interfaces no installed family claimed. It runs before two of + them that intend one family are collapsed into it, so a caller that must not touch one of the + two cannot have it survive the collapse as the row the family is built on. + """ + installed = plan_installed_families(module, rule, variables, interfaces=interfaces) + claimed = installed.member_pks + plain = [interface for interface in interfaces if interface.pk not in claimed and not _is_channel(interface)] + if admit_leftover is not None: + plain = list(admit_leftover(plain)) + if rule.channel_count <= 0: + leftover = tuple(plan_interface_rename(module, rule, variables, interface) for interface in plain) + elif any(plan.topology == FamilyTopology.CHANNELIZED for plan in installed.plans): + # A breakout rule renames the families the module already models; it never adds one beside them. + leftover = () # pragma: no cover - requires channelization support + for interface in plain: # pragma: no cover - see above + logger.debug( + "Interface %r is not channelized; skipping it while rule '%s' breaks out this module's families.", + interface.name, + rule, + ) + else: + leftover = tuple(_creation_plans(module, rule, variables, plain)) + return ModuleFamilyPlans(installed=installed.plans, leftover=leftover) + + +def _selection_pks(plan): + """Return the interface primary keys a selection reaches this family through. + + A channelized family is submitted through its parent alone, because a channel is not an + independent candidate on any path. + """ + if isinstance(plan, InstalledFamilyPlan): + if plan.parent_pk is None: + return plan.member_pks + return (plan.parent_pk,) # pragma: no cover - requires channelization support + return (plan.base.pk,) + + +def _selected(plans, selected_pks): + """Return the plans the operator's interface selection reaches.""" + if selected_pks is None: + return plans + return [plan for plan in plans if selected_pks.intersection(_selection_pks(plan))] + + +def execute_module_families(rule, module, plans): + """Execute each planned family in order, keeping the ones after a failure.""" + return [_execute(rule, module, plan) for plan in plans] + + +def _failed_members(plan, reason): + """Return failed member facts for the live rows an executor left unchanged.""" + return tuple( + MemberOutcome( + interface_pk=member.snapshot.pk, + current_name=member.snapshot.name, + target_name=member.target_name, + status=FamilyStatus.FAILED, + reason=reason, + ) + for member in plan.live_members + ) + + +def _failed_outcome(plan, error): + """Represent an executor exception without dropping the planned family from the batch.""" + reason = f"family execution failed: {error}" + return FamilyOutcome( + family_id=plan.family_id, + topology=plan.topology, + status=FamilyStatus.FAILED, + members=_failed_members(plan, reason), + reason=reason, + ) + + +def _execute(rule, module, plan): + """Execute one family, logging (never raising) so the batch keeps its later families.""" + try: + return execute_family_plan(plan) + except (ValueError, ValidationError) as error: + logger.exception( + "Failed to apply rule '%s' to a family on module '%s' (id=%s); skipping.", rule, module, module.pk + ) + return _failed_outcome(plan, error) + + +def _apply_module(rule, module, interfaces, selected_pks): + """Plan and execute every selected family on one module.""" + variables = build_variables(module.module_bay, device=module.device) + plans = _selected(plan_module_families(module, rule, variables, interfaces).plans, selected_pks) + return execute_module_families(rule, module, plans) + + +def apply_rule_to_modules(rule, modules, selected_pks=None, limit=None) -> BatchOutcome: + """Apply *rule* to every module in *modules*, one family at a time. + + *selected_pks* limits the batch to the families those interfaces reach; *limit* stops it after + the module that reached that many changed interfaces. + """ + if not modules: + return BatchOutcome(families=()) + by_module = interfaces_by_module(modules) + families: list[FamilyOutcome] = [] + changed = 0 + with pinned_template_cache(modules): + for module in modules: + outcomes = _apply_module(rule, module, by_module[module.pk], selected_pks) + families.extend(outcomes) + changed += sum(outcome.changed_count for outcome in outcomes) + if limit is not None and changed >= limit: + break + return BatchOutcome(families=tuple(families)) diff --git a/netbox_interface_name_rules/family/capabilities.py b/netbox_interface_name_rules/family/capabilities.py new file mode 100644 index 0000000..c4738b6 --- /dev/null +++ b/netbox_interface_name_rules/family/capabilities.py @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Probe what the active NetBox data model can represent.""" + +from dcim.models import Interface +from django.core.exceptions import FieldDoesNotExist + + +def supports_channelization() -> bool: + """Return whether this NetBox models channelized subinterfaces (4.7+). + + Probed from the Interface model rather than a version comparison, so a backport or a + development build is detected by what it actually provides. + """ + try: + Interface._meta.get_field("channel_id") + except FieldDoesNotExist: + return False + return True # pragma: no cover - only reachable on a NetBox that models channelization diff --git a/netbox_interface_name_rules/family/conversion.py b/netbox_interface_name_rules/family/conversion.py new file mode 100644 index 0000000..b05f9a8 --- /dev/null +++ b/netbox_interface_name_rules/family/conversion.py @@ -0,0 +1,446 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Convert installed flat breakout families into the channelized topology. + +An earlier flat apply leaves N sibling interfaces where NetBox 4.7+ models a channelized parent +with N channel subinterfaces. Converting one rewrites rows an operator owns (cables, addresses, +tags), so it is never a side effect of applying a rule: the operator confirms it per family. + +A family is identified by its ch-0 row, the way the flat apply that installed it named that row. +A family whose siblings the module no longer carries whole is still offered, and refused with the +row it is missing: silence would read as "nothing here to convert". +""" + +import logging +from dataclasses import replace + +from dcim.choices import InterfaceTypeChoices +from dcim.models import Interface +from django.core.exceptions import ValidationError +from django.db import IntegrityError, transaction + +from ..naming import build_variables +from .batch import BatchOutcome +from .capabilities import supports_channelization +from .domain import ( + ConversionCandidate, + ConversionMember, + ConversionPlan, + ConversionPreview, + FamilyOutcome, + FamilyStatus, + FamilyTopology, + InterfaceSnapshot, + MemberOutcome, +) +from .installed import ( + TemplateNames, + family_names_for, + flat_family_bases, + interfaces_by_module, + is_plain_interface, + module_db_alias, +) +from .names import COLLISION_REASON, is_name_collision, name_is_taken +from .targets import builds_channelized_family, channelized_family_names +from .template_names import pinned_template_cache + +logger = logging.getLogger(__name__) + +STALE_REASON = "the flat family changed after it was scanned" +INCOMPLETE_REASON = "this module carries no complete flat family" +UNSUPPORTED_REASON = "this NetBox release cannot model channelized interfaces" + + +def conversion_offered(rule) -> bool: + """Return whether *rule* describes a topology an installed flat family could be converted into. + + A disabled rule renames nothing on any apply path, so it converts nothing either. A flat + family has no parent row (its ch-0 interface *is* the base), so without a parent name there is + nowhere for that base to go and the conversion is not offered at all. + """ + return rule.enabled and builds_channelized_family(rule) and bool(rule.parent_name_template) + + +# --------------------------------------------------------------------------- +# Planning +# --------------------------------------------------------------------------- + + +def _conversion_plan( + module, db_alias, parent_name, channel_names, rows, channelization_supported +): # pragma: no cover - channelization only + """Build one immutable conversion plan from the rows a module carries for one family.""" + base, *siblings = rows + plan = ConversionPlan( + family_id=f"conversion:{base.pk}", + device_id=module.device_id, + module_id=module.pk, + db_alias=db_alias, + base=InterfaceSnapshot.from_interface(base), + parent_target_name=parent_name, + channel_names=channel_names, + siblings=tuple( + ConversionMember(snapshot=InterfaceSnapshot.from_interface(sibling), channel_id=channel_id) + for channel_id, sibling in siblings + ), + ) + if not channelization_supported: + return replace( + plan, + precondition_status=FamilyStatus.UNSUPPORTED, + precondition_reason=UNSUPPORTED_REASON, + ) + missing = plan.missing_names + if not missing: + return plan + reason = f"{missing[0]!r} is missing: {INCOMPLETE_REASON}" + return replace(plan, precondition_status=FamilyStatus.BLOCKED, precondition_reason=reason) + + +def _family_rows(by_name, channel_names): # pragma: no cover - requires channelization support + """Return the ch-0 row and every sibling row this module still carries, with its channel id. + + A sibling is taken whatever it has become, so a row that now belongs to another parent is + reported against this family instead of quietly leaving a gap where it used to be. + """ + base = by_name.get(channel_names[0]) + if base is None or not is_plain_interface(base): + return None + siblings = [ + (channel_id, by_name[name]) for channel_id, name in enumerate(channel_names[1:], start=2) if name in by_name + ] + return [base, *siblings] + + +def plan_module_conversions( + module, rule, variables, interfaces +) -> tuple[ConversionPlan, ...]: # pragma: no cover - requires channelization support + """Return one plan for every flat family *rule* names on *module*, complete or not. + + The parent takes the name the rule resolves for this module now, so a family named before a + virtual-chassis renumber converts to the parent an apply would give it; the channels keep the + names they already carry, because converting retypes those rows in place. + """ + db_alias = module_db_alias(module) + catalog = TemplateNames(module) + by_name = {interface.name: interface for interface in interfaces} + channelization_supported = supports_channelization() + plans = [] + claimed = set() + for base_name, source_base in flat_family_bases(rule, variables, interfaces, catalog): + names = family_names_for(rule, variables, base_name, source_base) + if names is None: + continue + _target_names, channel_names = names + rows = _family_rows(by_name, channel_names) + if rows is None or rows[0].pk in claimed: + continue + claimed.add(rows[0].pk) + parent_name, _channels = channelized_family_names(rule, base_name, variables) + plans.append( + _conversion_plan( + module, + db_alias, + parent_name, + channel_names, + rows, + channelization_supported, + ) + ) + return tuple(plans) + + +def _planned_families(rule, modules): # pragma: no cover - requires channelization support + """Yield every convertible family in the batch as ``(module, plan, base interface)``. + + The module rows, their interfaces and each module type's templates are read once for the whole + batch, so a second module of the same type costs the scan no extra query. + """ + by_module = interfaces_by_module(modules) + for module in modules: + interfaces = by_module[module.pk] + rows_by_pk = {interface.pk: interface for interface in interfaces} + variables = build_variables(module.module_bay, device=module.device) + for plan in plan_module_conversions(module, rule, variables, interfaces): + yield module, plan, rows_by_pk[plan.base.pk] + + +# --------------------------------------------------------------------------- +# Execution +# --------------------------------------------------------------------------- + + +def _outcome(plan, status, members, reason=""): # pragma: no cover - requires channelization support + """Build the immutable outcome of one conversion.""" + return FamilyOutcome( + family_id=plan.family_id, + topology=FamilyTopology.CHANNELIZED, + status=status, + members=members, + reason=reason, + ) + + +def _refused(plan, status, reason): # pragma: no cover - requires channelization support + """Log why the family was not converted and return an outcome that touched no row.""" + logger.warning( + "Cannot convert the flat family of interface %r (device %s, module %s) into the channelized " + "parent %r: %s. Skipping.", + plan.base.name, + plan.device_id, + plan.module_id, + plan.parent_target_name, + reason, + ) + targets = (plan.parent_target_name, *plan.sibling_target_names) + members = tuple( + MemberOutcome( + interface_pk=pk, + current_name=current_name, + target_name=target_name, + status=status, + reason=reason, + ) + for pk, current_name, target_name in zip(plan.member_pks, plan.current_names, targets, strict=True) + ) + return _outcome(plan, status, members, reason) + + +def _validate_or_block(interface, role): # pragma: no cover - requires channelization support + """Run NetBox's own validation on *interface*, restating a rejection as this family's reason.""" + try: + interface.full_clean() + except ValidationError as error: + raise ValidationError(f"{role} {interface.name!r}: {' '.join(error.messages)}") from error + + +def _locked_family(plan): # pragma: no cover - requires channelization support + """Lock every planned row for the transaction and return the live rows by primary key.""" + return { + interface.pk: interface + for interface in ( + Interface.objects.using(plan.db_alias) + .select_for_update(of=("self",)) # module is nullable, so locking the join is refused + .select_related("device", "module") + .filter(pk__in=plan.member_pks) + .order_by("pk") + ) + } + + +def _is_stale(plan, live) -> bool: # pragma: no cover - requires channelization support + """Return whether live identity, names, membership or topology changed since the scan.""" + planned = {plan.base.pk: plan.base, **{member.snapshot.pk: member.snapshot for member in plan.siblings}} + return planned != {pk: InterfaceSnapshot.from_interface(interface) for pk, interface in live.items()} + + +def _blocking_reason(plan, live) -> str: # pragma: no cover - requires channelization support + """Return why this family cannot become a channelized family, or an empty string. + + Only what upstream cannot decide for us is checked here; everything else is left to + ``full_clean()`` on each prospective row, which inherits upstream's rules as they grow. + """ + if name_is_taken(plan.device_id, plan.parent_target_name, plan.db_alias, exclude_pk=plan.base.pk): + return f"the parent name {plan.parent_target_name!r} is already taken on this device" + for member in plan.siblings: + sibling = live[member.snapshot.pk] + # Rebinding it validates cleanly, so only this check keeps the other family whole. + if sibling.channel_id is not None: + owner = sibling.parent.name if sibling.parent_id else "another parent" + return ( + f"{sibling.name!r} is already channel {sibling.channel_id} of {owner}; " + f"converting would take it out of that family" + ) + # TYPE_CHANNEL is nonconnectable but not virtual, so Interface.clean() accepts a cable on one. + if sibling.cable_id: + return f"{sibling.name!r} has a cable attached; a channel takes its cable from the parent" + return "" + + +def _carry_assignments(plan, base, channel): # pragma: no cover - requires channelization support + """Move the ch-0 row's addresses and first-hop groups onto the channel that took its name. + + Saved one row at a time so each carried object is validated, and recorded in the changelog, + like every other row this conversion writes. A model save writes back every field it read, so + each row is locked first: an edit landing between the read and the save would otherwise be + overwritten by the values this transaction started with. + """ + addresses = base.ip_addresses.using(plan.db_alias).select_for_update().order_by("pk") + for address in addresses: + address.assigned_object = channel + address.full_clean() + address.save(using=plan.db_alias) + assignments = base.fhrp_group_assignments.using(plan.db_alias).select_for_update().order_by("pk") + for assignment in assignments: + assignment.interface = channel + assignment.full_clean() + assignment.save(using=plan.db_alias) + + +def _split_base(plan, base): # pragma: no cover - requires channelization support + """Make *base* the family parent and move its logical identity onto a new channel-1 row. + + Everything an operator configured on the ch-0 row described a channel, not the cage carrying + it, so addresses, VLANs, MTU, description and tags move; custom fields can mean either thing + and are copied. The physical row keeps its pk, cable, type, module link and mark_connected. + """ + carried = { + "description": base.description, + "mtu": base.mtu, + "mode": base.mode, + "untagged_vlan_id": base.untagged_vlan_id, + "vrf_id": base.vrf_id, + } + tagged_vlans = list(base.tagged_vlans.all()) + tags = list(base.tags.all()) + channel_name = base.name + + base.name = plan.parent_target_name + base.channels = plan.channel_count + base.description = "" + base.mtu = None + base.mode = "" + base.untagged_vlan = None + base.vrf = None + _validate_or_block(base, "parent") + base.save(using=plan.db_alias) # BaseInterface.save() drops the tagged VLANs of a row that no longer tags + base.tags.clear() + + channel = Interface( + device=base.device, + module=base.module, + name=channel_name, + type=InterfaceTypeChoices.TYPE_CHANNEL, + parent=base, + channel_id=1, + enabled=base.enabled, + custom_field_data=dict(base.custom_field_data or {}), + **carried, + ) + _validate_or_block(channel, "channel") + channel.save(using=plan.db_alias) + channel.tagged_vlans.set(tagged_vlans) + channel.tags.set(tags) + _carry_assignments(plan, base, channel) + return channel + + +def _rewrite(plan, live): # pragma: no cover - requires channelization support + """Rewrite the whole family and return one outcome per row it wrote.""" + base = live[plan.base.pk] + previous_name = base.name + channel = _split_base(plan, base) + members = [ + MemberOutcome(base.pk, previous_name, plan.parent_target_name, FamilyStatus.CHANGED), + MemberOutcome(channel.pk, channel.name, channel.name, FamilyStatus.CHANGED), + ] + for member in plan.siblings: + sibling = live[member.snapshot.pk] + sibling.type = InterfaceTypeChoices.TYPE_CHANNEL + sibling.parent = base + sibling.channel_id = member.channel_id + _validate_or_block(sibling, "channel") + sibling.save(using=plan.db_alias) + members.append(MemberOutcome(sibling.pk, sibling.name, sibling.name, FamilyStatus.CHANGED)) + return tuple(members) + + +def _convert(plan, commit): # pragma: no cover - requires channelization support + """Convert one family inside a single transaction, or leave every row exactly as it was. + + A dry run (*commit* False) and a family that turns out to be unconvertible both roll the whole + rewrite back, so a family is never half converted and a scan writes nothing at all. + """ + try: + with transaction.atomic(using=plan.db_alias): + live = _locked_family(plan) + if _is_stale(plan, live): + return _refused(plan, FamilyStatus.STALE, STALE_REASON) + reason = _blocking_reason(plan, live) + if reason: + return _refused(plan, FamilyStatus.BLOCKED, reason) + members = _rewrite(plan, live) + if not commit: + transaction.set_rollback(True, using=plan.db_alias) + except ValidationError as error: + return _refused(plan, FamilyStatus.BLOCKED, " ".join(error.messages)) + except IntegrityError as error: + if not is_name_collision(error): + raise + return _refused(plan, FamilyStatus.BLOCKED, COLLISION_REASON) + return _outcome(plan, FamilyStatus.CHANGED, members) + + +def _dry_run(plan) -> FamilyOutcome: # pragma: no cover - requires channelization support + """Report what converting *plan* would do, having written and rolled back every row of it.""" + if plan.precondition_status is not None: + return _refused(plan, plan.precondition_status, plan.precondition_reason) + return _convert(plan, commit=False) + + +def execute_conversion(plan: ConversionPlan) -> FamilyOutcome: # pragma: no cover - requires channelization support + """Convert the planned flat family, or leave every row exactly as it was. + + Only a conversion plan names the family rows to rewrite, so anything else (a prospective plan + above all) is refused before a single row is locked. + """ + if not isinstance(plan, ConversionPlan): + raise TypeError(f"{type(plan).__name__} is not an executable conversion plan") + if plan.precondition_status is not None: + return _refused(plan, plan.precondition_status, plan.precondition_reason) + return _convert(plan, commit=True) + + +# --------------------------------------------------------------------------- +# Batch entry points +# --------------------------------------------------------------------------- + + +def preview_rule_conversions(rule, modules, limit=None) -> ConversionPreview: + """Return what converting each of *rule*'s flat families would do, convertible or not. + + Nothing is written: every family is converted inside a savepoint that is rolled back again, so + each candidate carries the reason NetBox itself would refuse the family rather than a guess at + its rules. That dry run is what the scan costs, and a blocked family costs it too, so *limit* + caps the families examined rather than the convertible ones among them. + """ + if not conversion_offered(rule): + return ConversionPreview(candidates=()) + return _preview(rule, modules, limit) # pragma: no cover - requires channelization support + + +def _preview(rule, modules, limit) -> ConversionPreview: # pragma: no cover - requires channelization support + """Dry-run at most *limit* of the rule's flat families; see ``preview_rule_conversions``.""" + candidates = [] + with pinned_template_cache(modules): + for module, plan, base in _planned_families(rule, modules): + if limit is not None and len(candidates) >= limit: + return ConversionPreview(candidates=tuple(candidates), has_more=True) + outcome = _dry_run(plan) + candidates.append(ConversionCandidate(plan=plan, module=module, interface=base, reason=outcome.reason)) + return ConversionPreview(candidates=tuple(candidates)) + + +def convert_rule_families(rule, modules, selected_pks=None) -> BatchOutcome: + """Convert the confirmed flat families of *rule*; return one explicit outcome per family. + + *selected_pks* is the set of ch-0 interface primary keys the operator confirmed: ``None`` + converts every convertible family (the batch the background job runs), an empty collection + converts none. A family that cannot be converted is reported and passed over, so it is never + half converted and never costs the rest of the batch. + """ + return _convert_families(rule, modules, selected_pks) # pragma: no cover - see above + + +def _convert_families(rule, modules, selected_pks) -> BatchOutcome: # pragma: no cover - channelization only + """Convert the confirmed flat families of *rule*; see ``convert_rule_families``.""" + if not conversion_offered(rule) or (selected_pks is not None and not selected_pks): + return BatchOutcome(families=()) + families = [] + with pinned_template_cache(modules): + for _module, plan, _base in _planned_families(rule, modules): + if selected_pks is not None and plan.base.pk not in selected_pks: + continue + families.append(execute_conversion(plan)) + return BatchOutcome(families=tuple(families)) diff --git a/netbox_interface_name_rules/family/domain.py b/netbox_interface_name_rules/family/domain.py new file mode 100644 index 0000000..4d92061 --- /dev/null +++ b/netbox_interface_name_rules/family/domain.py @@ -0,0 +1,412 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Immutable domain values for interface-family operations.""" + +from dataclasses import dataclass +from enum import StrEnum + + +class FamilyTopology(StrEnum): + """Installed interface-family topology.""" + + FLAT = "flat" + CHANNELIZED = "channelized" + + +class FamilyStatus(StrEnum): + """Result of a family or member operation.""" + + CHANGED = "changed" + UNCHANGED = "unchanged" + BLOCKED = "blocked" + STALE = "stale" + UNSUPPORTED = "unsupported" + FAILED = "failed" + + +class MemberRole(StrEnum): + """A member's role in its installed family.""" + + PARENT = "parent" + CHANNEL = "channel" + FLAT_MEMBER = "flat_member" + + +@dataclass(frozen=True, slots=True) +class InterfaceSnapshot: + """Live interface facts that execution must revalidate.""" + + pk: int + device_id: int + module_id: int | None + name: str + parent_id: int | None + channel_id: int | None + channels: int | None + + @classmethod + def from_interface(cls, interface): + """Capture the protected facts from a NetBox interface row.""" + return cls( + pk=interface.pk, + device_id=interface.device_id, + module_id=interface.module_id, + name=interface.name, + parent_id=getattr(interface, "parent_id", None), + channel_id=getattr(interface, "channel_id", None), + channels=getattr(interface, "channels", None), + ) + + +@dataclass(frozen=True, slots=True) +class PlannedMember: + """One installed member and its intended name.""" + + snapshot: InterfaceSnapshot + target_name: str | None + role: MemberRole + reason: str = "" + + +@dataclass(frozen=True, slots=True) +class InstalledFamilyPlan: + """An executable rename plan for one installed interface family.""" + + family_id: str + topology: FamilyTopology + device_id: int + module_id: int | None + db_alias: str + members: tuple[PlannedMember, ...] + parent_pk: int | None = None + precondition_status: FamilyStatus | None = None + precondition_reason: str = "" + + @property + def member_pks(self) -> tuple[int, ...]: + """Return member primary keys in plan order.""" + return tuple(member.snapshot.pk for member in self.members) + + @property + def live_members(self) -> tuple[PlannedMember, ...]: + """Return the planned members that already have live interface rows.""" + return self.members + + +@dataclass(frozen=True, slots=True) +class InstalledFamilyPlanSet: + """Exactly one immutable plan for each discovered installed family.""" + + module_id: int + plans: tuple[InstalledFamilyPlan, ...] + + @property + def member_pks(self) -> frozenset[int]: + """Return every interface claimed by the plan set.""" + return frozenset(pk for plan in self.plans for pk in plan.member_pks) + + +@dataclass(frozen=True, slots=True) +class PlannedChannel: + """One channel row a structural plan creates under its parent.""" + + channel_id: int + name: str + + +@dataclass(frozen=True, slots=True) +class StructuralFamilyPlan: + """An executable plan that turns one plain interface into a channelized family.""" + + family_id: str + device_id: int + module_id: int + module_type_id: int + db_alias: str + base: InterfaceSnapshot + parent_target_name: str + channel_count: int + channels: tuple[PlannedChannel, ...] + precondition_status: FamilyStatus | None = None + precondition_reason: str = "" + + @property + def topology(self) -> FamilyTopology: + """Return the topology this plan installs.""" + return FamilyTopology.CHANNELIZED + + @property + def target_names(self) -> tuple[str, ...]: + """Return the parent name and every channel name in creation order.""" + return (self.parent_target_name, *(channel.name for channel in self.channels)) + + @property + def live_members(self) -> tuple[PlannedMember, ...]: + """Return the base row this plan rewrites.""" + return (PlannedMember(self.base, self.parent_target_name, MemberRole.PARENT),) + + +@dataclass(frozen=True, slots=True) +class FlatCreationPlan: + """An executable plan that expands one plain interface into a flat breakout family.""" + + family_id: str + device_id: int + module_id: int + db_alias: str + base: InterfaceSnapshot + target_names: tuple[str, ...] + precondition_status: FamilyStatus | None = None + precondition_reason: str = "" + + @property + def topology(self) -> FamilyTopology: + """Return the topology this plan installs.""" + return FamilyTopology.FLAT + + @property + def live_members(self) -> tuple[PlannedMember, ...]: + """Return the base row this plan rewrites.""" + return (PlannedMember(self.base, self.target_names[0], MemberRole.FLAT_MEMBER),) + + +@dataclass(frozen=True, slots=True) +class ProspectiveMember: + """One member of a family planned from names alone.""" + + source_name: str | None + target_name: str + role: MemberRole + channel_id: int | None = None + reason: str = "" + + +@dataclass(frozen=True, slots=True) +class ProspectiveFamilyPlan: + """What a rule intends for one family of named interfaces. + + A plan whose *base_name* is set describes a family the rule builds out of that one name, so the + name expands to every target. A plan without one renames members that already exist. + """ + + family_id: str + topology: FamilyTopology + base_name: str | None + members: tuple[ProspectiveMember, ...] + precondition_status: FamilyStatus | None = None + precondition_reason: str = "" + + @property + def source_names(self) -> tuple[str, ...]: + """Return the name of every member that already exists, in plan order.""" + return tuple(member.source_name for member in self.members if member.source_name is not None) + + @property + def target_names(self) -> tuple[str, ...]: + """Return every intended name, in plan order.""" + return tuple(member.target_name for member in self.members) + + +@dataclass(frozen=True, slots=True) +class ProspectiveFamilyPlanSet: + """Exactly one prospective plan for each family the rule intends on a module.""" + + module_id: int + plans: tuple[ProspectiveFamilyPlan, ...] + + def predicted_names(self, source_name: str) -> tuple[str, ...]: + """Return the names *source_name* becomes, or the name itself when no plan claims it.""" + for plan in self.plans: + if plan.base_name == source_name: + return plan.target_names + for member in plan.members: + if member.source_name == source_name: + return (member.target_name,) + return (source_name,) + + +@dataclass(frozen=True, slots=True) +class MemberOutcome: + """Result facts for one planned family member.""" + + interface_pk: int + current_name: str + target_name: str | None + status: FamilyStatus + reason: str = "" + + +@dataclass(frozen=True, slots=True) +class FamilyOutcome: + """Execution result for one installed family.""" + + family_id: str + topology: FamilyTopology + status: FamilyStatus + members: tuple[MemberOutcome, ...] + reason: str = "" + + @property + def changed_count(self) -> int: + """Return the number of members renamed by this operation.""" + return sum(member.status == FamilyStatus.CHANGED for member in self.members) + + +@dataclass(frozen=True, slots=True) +class InstalledPlanSetOutcome: + """Execution results for an installed family plan set.""" + + families: tuple[FamilyOutcome, ...] + + @property + def changed_count(self) -> int: + """Return the number of members renamed across all families.""" + return sum(family.changed_count for family in self.families) + + +@dataclass(frozen=True, slots=True) +class ConversionMember: # pragma: no cover - requires channelization support + """One installed flat sibling and the channel identifier it takes.""" + + snapshot: InterfaceSnapshot + channel_id: int + + +@dataclass(frozen=True, slots=True) +class ConversionPlan: # pragma: no cover - requires channelization support + """An executable plan that turns one installed flat family into a channelized family. + + The ch-0 row keeps its primary key and becomes the parent, giving up its name to a new + channel-1 row; each sibling is retyped in place as a channel under the name it already has. + + *channel_names* is every name the rule spells for the family, whether or not the module still + carries a row for it, so a family with a gap can still say what it would have become. + """ + + family_id: str + device_id: int + module_id: int + db_alias: str + base: InterfaceSnapshot + parent_target_name: str + channel_names: tuple[str, ...] + siblings: tuple[ConversionMember, ...] + precondition_status: FamilyStatus | None = None + precondition_reason: str = "" + + @property + def topology(self) -> FamilyTopology: + """Return the topology this plan installs.""" + return FamilyTopology.CHANNELIZED + + @property + def channel_count(self) -> int: + """Return how many channels the converted parent declares.""" + return len(self.channel_names) + + @property + def member_pks(self) -> tuple[int, ...]: + """Return the primary key of every row the plan rewrites, in family order.""" + return (self.base.pk, *(sibling.snapshot.pk for sibling in self.siblings)) + + @property + def current_names(self) -> tuple[str, ...]: + """Return the names the family carries now, in family order.""" + return (self.base.name, *(sibling.snapshot.name for sibling in self.siblings)) + + @property + def sibling_target_names(self) -> tuple[str, ...]: + """Return the name each installed sibling keeps as a channel, in family order.""" + return tuple(sibling.snapshot.name for sibling in self.siblings) + + @property + def missing_names(self) -> tuple[str, ...]: + """Return every name the family needs that this module carries no row for.""" + present = {self.base.name, *(sibling.snapshot.name for sibling in self.siblings)} + return tuple(name for name in self.channel_names if name not in present) + + @property + def target_names(self) -> tuple[str, ...]: + """Return the parent name and every channel name, in family order.""" + return (self.parent_target_name, *self.channel_names) + + +@dataclass(frozen=True, slots=True) +class PlannedName: # pragma: no cover - requires channelization support + """One name a family plan intends, and the role it fills in that family.""" + + name: str + role: str + channel_id: int | None = None + + +@dataclass(frozen=True, slots=True) +class ConversionCandidate: # pragma: no cover - requires channelization support + """One flat family an operator may convert, and what converting it would do. + + *module* and *interface* are the live rows the Apply page links to and submits; every name, + role and reason on the candidate comes from the immutable plan instead. + """ + + plan: ConversionPlan + module: object + interface: object + reason: str = "" + + @property + def convertible(self) -> bool: + """Return whether a dry run of this family succeeded.""" + return not self.reason + + @property + def status_label(self) -> str: + """Return the operator-facing result of the conversion preview.""" + if self.convertible: + return "Convertible" + if self.plan.precondition_status == FamilyStatus.UNSUPPORTED: + return "Unsupported" + return "Blocked" + + @property + def current_name(self) -> str: + """Return the name of the ch-0 row the confirm form submits.""" + return self.plan.base.name + + @property + def current_names(self) -> tuple[str, ...]: + """Return the names the family carries now.""" + return self.plan.current_names + + @property + def new_names(self) -> tuple[str, ...]: + """Return the names the family would carry once converted.""" + return self.plan.target_names + + @property + def name_details(self) -> tuple[PlannedName, ...]: + """Describe every intended name so the page can render a family as a family.""" + return ( + PlannedName(self.plan.parent_target_name, "parent"), + *( + PlannedName(name, "channel", channel_id) + for channel_id, name in enumerate(self.plan.channel_names, start=1) + ), + ) + + @property + def metadata_note(self) -> str: + """Return the sentence the Apply page shows about where the ch-0 configuration ends up.""" + return ( + f"The interface VRF, addresses, VLANs, MTU, description and tags on {self.plan.base.name} move to the new " + f"channel 1 interface that takes over that name; custom field values are copied. The physical " + f"row keeps its ID and becomes the parent {self.plan.parent_target_name}, so automation keyed " + f"on that interface ID will address the parent afterwards." + ) + + +@dataclass(frozen=True, slots=True) +class ConversionPreview: + """Every flat family a conversion scan examined, and whether it left any unexamined.""" + + candidates: tuple[ConversionCandidate, ...] + has_more: bool = False diff --git a/netbox_interface_name_rules/family/execution.py b/netbox_interface_name_rules/family/execution.py new file mode 100644 index 0000000..28e2607 --- /dev/null +++ b/netbox_interface_name_rules/family/execution.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Execute installed interface-family plans.""" + +import logging + +from dcim.models import Interface +from django.core.exceptions import ValidationError +from django.db import IntegrityError, transaction +from django.db.models import Q + +from .domain import ( + FamilyOutcome, + FamilyStatus, + InstalledFamilyPlan, + InstalledFamilyPlanSet, + InstalledPlanSetOutcome, + InterfaceSnapshot, + MemberOutcome, + MemberRole, +) +from .names import COLLISION_REASON, is_name_collision, name_is_taken, reconcile_after_parent_cascade + +logger = logging.getLogger(__name__) + +STALE_REASON = "installed family changed after planning" +PARENT_BLOCKED_REASON = "family parent was blocked" + + +def _lock_family(plan: InstalledFamilyPlan): + """Lock and return the current family rows in stable primary-key order.""" + queryset = Interface.objects.using(plan.db_alias) + if plan.parent_pk is None: + queryset = queryset.filter(pk__in=plan.member_pks) + else: # pragma: no cover - requires channelization support + queryset = queryset.filter(Q(pk=plan.parent_pk) | Q(parent_id=plan.parent_pk)) + return list(queryset.select_for_update().order_by("pk")) + + +def _is_stale(plan: InstalledFamilyPlan, interfaces) -> bool: + """Return whether live identity, names, membership, or topology changed.""" + planned = {member.snapshot.pk: member.snapshot for member in plan.members} + live = {interface.pk: InterfaceSnapshot.from_interface(interface) for interface in interfaces} + return planned != live + + +def _member_outcome(member, status, reason=""): + """Build one immutable member outcome from its plan facts.""" + return MemberOutcome( + interface_pk=member.snapshot.pk, + current_name=member.snapshot.name, + target_name=member.target_name, + status=status, + reason=reason, + ) + + +def _blocked_member(member, reason): + """Return a blocked member outcome.""" + return _member_outcome(member, FamilyStatus.BLOCKED, reason) + + +def _rename_member(member, interface, db_alias): + """Apply one planned name inside a savepoint and return its explicit outcome.""" + target_name = member.target_name + if target_name is None: + logger.warning( + "Cannot derive a name for channel interface %r; leaving it unchanged.", + member.snapshot.name, + ) + return _blocked_member(member, member.reason) + if target_name == interface.name: + return _member_outcome(member, FamilyStatus.UNCHANGED) + if name_is_taken(interface.device_id, target_name, db_alias, exclude_pk=interface.pk): + logger.warning( + "Interface name %r already exists on device %s; skipping rename of %r to %r.", + target_name, + interface.device_id, + interface.name, + target_name, + ) + return _blocked_member(member, COLLISION_REASON) + + previous_name = interface.name + try: + with transaction.atomic(using=db_alias): + interface.name = target_name + interface.full_clean() + interface.save(using=db_alias) + except ValidationError as error: + interface.name = previous_name + logger.warning("NetBox rejected rename of %r to %r: %s", previous_name, target_name, error) + return _blocked_member(member, " ".join(error.messages)) + except IntegrityError as error: + interface.name = previous_name + if is_name_collision(error): + logger.warning( + "Interface name %r became occupied while renaming %r; skipping.", + target_name, + previous_name, + ) + return _blocked_member(member, COLLISION_REASON) + raise + return _member_outcome(member, FamilyStatus.CHANGED) + + +def _family_status(members): + """Summarize member outcomes without hiding partial success.""" + statuses = {member.status for member in members} + if FamilyStatus.CHANGED in statuses: + return FamilyStatus.CHANGED + if FamilyStatus.BLOCKED in statuses: + return FamilyStatus.BLOCKED + return FamilyStatus.UNCHANGED + + +def _preserve_names_across_parent_cascade(plan, member_outcomes): # pragma: no cover - channelization only + """Schedule restoration of blocked conventional channel names after a parent cascade.""" + if plan.parent_pk is None: + return + parent_member = next(member for member in plan.members if member.role == MemberRole.PARENT) + parent_outcome = next(outcome for outcome in member_outcomes if outcome.interface_pk == plan.parent_pk) + if parent_outcome.status != FamilyStatus.CHANGED: + return + outcomes_by_pk = {outcome.interface_pk: outcome for outcome in member_outcomes} + channels = tuple( + ( + member.snapshot.pk, + member.snapshot.channel_id, + member.target_name + if outcomes_by_pk[member.snapshot.pk].status in (FamilyStatus.CHANGED, FamilyStatus.UNCHANGED) + else member.snapshot.name, + ) + for member in plan.members + if member.role == MemberRole.CHANNEL + ) + reconcile_after_parent_cascade(parent_member.snapshot.name, parent_member.target_name, channels, plan.db_alias) + + +def _execute_channelized_members(plan, live_by_pk): # pragma: no cover - requires channelization support + """Execute a channelized family after its parent succeeds.""" + parent_member = next(member for member in plan.members if member.role == MemberRole.PARENT) + parent_outcome = _rename_member(parent_member, live_by_pk[parent_member.snapshot.pk], plan.db_alias) + if parent_outcome.status == FamilyStatus.BLOCKED: + return ( + parent_outcome, + *( + _blocked_member(member, PARENT_BLOCKED_REASON) + for member in plan.members + if member.role != MemberRole.PARENT + ), + ) + return ( + parent_outcome, + *( + _rename_member(member, live_by_pk[member.snapshot.pk], plan.db_alias) + for member in plan.members + if member.role != MemberRole.PARENT + ), + ) + + +def _execute_members(plan, live_by_pk): + """Execute members while enforcing parent-first family semantics.""" + if plan.parent_pk is None: + return tuple(_rename_member(member, live_by_pk[member.snapshot.pk], plan.db_alias) for member in plan.members) + return _execute_channelized_members(plan, live_by_pk) # pragma: no cover - channelization only + + +def _stale_outcome(plan: InstalledFamilyPlan) -> FamilyOutcome: + """Return a stale result without applying any planned member.""" + return FamilyOutcome( + family_id=plan.family_id, + topology=plan.topology, + status=FamilyStatus.STALE, + members=tuple(_member_outcome(member, FamilyStatus.STALE, STALE_REASON) for member in plan.members), + reason=STALE_REASON, + ) + + +def execute_installed_plan(plan: InstalledFamilyPlan) -> FamilyOutcome: + """Execute one family plan in its own transaction.""" + with transaction.atomic(using=plan.db_alias): + interfaces = _lock_family(plan) + if _is_stale(plan, interfaces): + return _stale_outcome(plan) + if plan.precondition_status is not None: + logger.warning( + "Installed family of interface %r is %s: %s.", + plan.members[0].snapshot.name, + plan.precondition_status, + plan.precondition_reason, + ) + return FamilyOutcome( + family_id=plan.family_id, + topology=plan.topology, + status=plan.precondition_status, + members=tuple( + _member_outcome(member, plan.precondition_status, plan.precondition_reason) + for member in plan.members + ), + reason=plan.precondition_reason, + ) + live_by_pk = {interface.pk: interface for interface in interfaces} + member_outcomes = _execute_members(plan, live_by_pk) + _preserve_names_across_parent_cascade(plan, member_outcomes) + return FamilyOutcome( + family_id=plan.family_id, + topology=plan.topology, + status=_family_status(member_outcomes), + members=member_outcomes, + ) + + +def execute_installed_plan_set(plan_set: InstalledFamilyPlanSet) -> InstalledPlanSetOutcome: + """Execute each installed family in a separate transaction. + + Only an installed plan set carries the row snapshots execution revalidates, so anything else + (a prospective plan set above all) is refused before a single row is locked. + """ + if not isinstance(plan_set, InstalledFamilyPlanSet): + raise TypeError(f"{type(plan_set).__name__} is not an executable plan set") + return InstalledPlanSetOutcome(families=tuple(execute_installed_plan(plan) for plan in plan_set.plans)) diff --git a/netbox_interface_name_rules/family/installed.py b/netbox_interface_name_rules/family/installed.py new file mode 100644 index 0000000..db365e6 --- /dev/null +++ b/netbox_interface_name_rules/family/installed.py @@ -0,0 +1,373 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Discover and plan installed interface families.""" + +import logging +import re + +from dcim.models import Interface +from django.db import DEFAULT_DB_ALIAS + +from ..choices import BreakoutModeChoices +from ..naming import evaluate_name_template +from .domain import ( + FamilyStatus, + FamilyTopology, + InstalledFamilyPlan, + InstalledFamilyPlanSet, + InterfaceSnapshot, + MemberRole, + PlannedMember, +) +from .targets import ( + channelized_family_targets, + flat_family_names, + lockstep_family_targets, + template_channel_suffixes, +) +from .template_names import resolved_template_names + +logger = logging.getLogger(__name__) + +_BASE_SENTINEL = "InrBaseSentinelEnd" + + +def _db_alias(instance) -> str: + """Return the database alias that *instance* was loaded from.""" + return instance._state.db or DEFAULT_DB_ALIAS + + +def module_db_alias(module) -> str: + """Return the database alias that *module* was loaded from.""" + return _db_alias(module) + + +def is_plain_interface(interface) -> bool: + """Return whether *interface* can be a flat-family member.""" + return ( + getattr(interface, "parent_id", None) is None + and getattr(interface, "channel_id", None) is None + and getattr(interface, "channels", None) is None + ) + + +def is_channelized_parent(interface) -> bool: # pragma: no cover - requires channelization support + """Return whether *interface* declares a channelized family.""" + return getattr(interface, "channels", None) is not None + + +def _is_channel(interface) -> bool: # pragma: no cover - requires channelization support + """Return whether *interface* is bound to a parent channel.""" + return getattr(interface, "channel_id", None) is not None + + +def _historical_bases(rule, variables, template, interfaces): # pragma: no cover - requires VC token support + """Return every historical base claimed by *template*.""" + if template.historical_pattern is None: + return () + try: + marked = evaluate_name_template( + rule.name_template, + {**variables, "base": _BASE_SENTINEL, "channel": str(rule.channel_start)}, + ) + except (TypeError, ValueError): + return () + if _BASE_SENTINEL not in marked: + return () + escaped = re.escape(marked) + head, _, tail = escaped.partition(_BASE_SENTINEL) + tail = tail.replace(_BASE_SENTINEL, "(?P=base)") + pattern = re.compile(f"{head}(?P{template.historical_pattern.pattern}){tail}") + bases = { + match.group("base") for interface in interfaces if (match := pattern.fullmatch(interface.name)) is not None + } + return tuple(sorted(bases)) + + +def _ambiguous_bases(historical_by_template): + """Return every historical base that more than one template could claim.""" + ambiguous = {base_name for bases in historical_by_template.values() if len(bases) > 1 for base_name in bases} + single_claims: dict[str, int] = {} + for bases in historical_by_template.values(): + if len(bases) == 1: # pragma: no cover - historical matchers require VC token support + single_claims[bases[0]] = single_claims.get(bases[0], 0) + 1 + ambiguous.update(base_name for base_name, count in single_claims.items() if count > 1) + return ambiguous + + +def _source_bases(template, historical_bases, ambiguous_bases): + """Return the template's own base and every historical base it alone claims.""" + unambiguous = tuple( + base_name for base_name in historical_bases if len(historical_bases) == 1 and base_name not in ambiguous_bases + ) + return (template.resolved, *unambiguous) + + +def flat_family_bases(rule, variables, interfaces, catalog): + """Return ``(template base, source base)`` for every base a flat family could be named from. + + The template base is the name the rule resolves for this module now; the source base is the one + an installed family still spells, which differs after a virtual-chassis renumber. A historical + base more than one template could claim is dropped: the rows it names are not certainly one + family's, and neither renaming nor converting them is this plugin's guess to make. + """ + if rule.channel_count <= 0: + return () + templates = catalog.get() + historical_by_template = { + template.pk: _historical_bases(rule, variables, template, interfaces) for template in templates + } + ambiguous_bases = _ambiguous_bases(historical_by_template) + return tuple( + (template.resolved, source_base) + for template in templates + for source_base in _source_bases(template, historical_by_template[template.pk], ambiguous_bases) + ) + + +def family_names_for(rule, variables, base_name, source_base): + """Return the names the rule intends for the family and the names it still spells, or None.""" + try: + target_names = flat_family_names(rule, variables, base_name) + source_names = flat_family_names(rule, variables, source_base) + except (TypeError, ValueError): + return None + if len(set(source_names)) != len(source_names): + return None + return target_names, source_names + + +def _singly_claimed(candidates): + """Return only the candidates whose members no other candidate also claims.""" + claims: dict[int, int] = {} + for _base_name, _names, members in candidates: + for member in members: + claims[member.pk] = claims.get(member.pk, 0) + 1 + return [candidate for candidate in candidates if all(claims[member.pk] == 1 for member in candidate[2])] + + +def flat_family_candidates(rule, variables, interfaces, catalog): + """Return complete, unambiguous flat-family candidates on this module. + + A flat family carries the names the rule's channel range spells, and a flat rule and the + channelized rule it later became spell those identically, so the caller decides whether the + rule's current breakout mode makes these families its own to rename or its own to convert. + """ + by_name = {interface.name: interface for interface in interfaces if is_plain_interface(interface)} + if not by_name: + return [] + candidates = [] + for base_name, source_base in flat_family_bases(rule, variables, interfaces, catalog): + names = family_names_for(rule, variables, base_name, source_base) + if names is None: + continue + target_names, source_names = names + if not all(name in by_name for name in source_names): + continue + candidate = (base_name, target_names, tuple(by_name[name] for name in source_names)) + if candidate not in candidates: # pragma: no branch - duplicates require historical matchers + candidates.append(candidate) + return _singly_claimed(candidates) + + +def _flat_candidates(rule, variables, interfaces, catalog): + """Return the flat families a flat-mode rule owns on this module.""" + if rule.breakout_mode != BreakoutModeChoices.FLAT: + return [] + return flat_family_candidates(rule, variables, interfaces, catalog) + + +def _flat_plan(module, db_alias, target_names, interfaces): + """Build one immutable plan from a complete flat-family candidate.""" + members = tuple( + PlannedMember( + snapshot=InterfaceSnapshot.from_interface(interface), + target_name=target_name, + role=MemberRole.FLAT_MEMBER, + ) + for interface, target_name in zip(interfaces, target_names, strict=True) + ) + return InstalledFamilyPlan( + family_id=f"flat:{members[0].snapshot.pk}", + topology=FamilyTopology.FLAT, + device_id=module.device_id, + module_id=module.pk, + db_alias=db_alias, + members=members, + ) + + +def _channelized_plan(device_id, module_id, db_alias, parent, children, targets): # pragma: no cover + """Build one plan for an existing channelized family from the names *targets* intends.""" + members = [ + PlannedMember( + snapshot=InterfaceSnapshot.from_interface(parent), + target_name=targets.parent_name, + role=MemberRole.PARENT, + ) + ] + members.extend( + PlannedMember( + snapshot=InterfaceSnapshot.from_interface(child), + target_name=target, + role=MemberRole.CHANNEL, + reason=reason, + ) + for child, (target, reason) in zip(children, targets.channels, strict=True) + ) + return InstalledFamilyPlan( + family_id=f"channelized:{parent.pk}", + topology=FamilyTopology.CHANNELIZED, + device_id=device_id, + module_id=module_id, + db_alias=db_alias, + members=tuple(members), + parent_pk=parent.pk, + precondition_status=targets.status, + precondition_reason=targets.reason, + ) + + +def _module_family_targets(rule, variables, parent, children, suffixes): # pragma: no cover + """Return the names *rule* intends for a channelized family a module carries.""" + return channelized_family_targets( + rule, + variables, + parent.name, + parent.channels, + tuple((child.name, child.channel_id) for child in children), + suffixes, + ) + + +def _channelized_plans(module, rule, variables, db_alias, interfaces, catalog): # pragma: no cover + """Return one plan for every structurally discovered channelized family.""" + parents = [interface for interface in interfaces if is_channelized_parent(interface)] + if not parents: + return [] + children_by_parent: dict[int, list] = {} + for interface in interfaces: + if _is_channel(interface) and interface.parent_id is not None: + children_by_parent.setdefault(interface.parent_id, []).append(interface) + suffixes = template_channel_suffixes(catalog.get()) + plans = [] + for parent in parents: + children = children_by_parent.get(parent.pk, []) + children.sort(key=lambda child: (child.channel_id, child.pk)) + targets = _module_family_targets(rule, variables, parent, children, suffixes) + plans.append(_channelized_plan(module.device_id, module.pk, db_alias, parent, children, targets)) + return plans + + +class TemplateNames: + """The module type's resolved template names, read only where a plan needs them.""" + + def __init__(self, module): + self._module = module + self._templates = None + + def get(self): + """Return every resolved template name for the module, loading them once.""" + if self._templates is None: + self._templates = resolved_template_names(self._module) + return self._templates + + +def interfaces_by_module(modules): + """Load every interface of a module batch in one query, in stable per-module order.""" + by_module: dict[int, list] = {module.pk: [] for module in modules} + if not by_module: + return by_module + rows = ( + Interface.objects.using(module_db_alias(modules[0])) + .filter(module_id__in=list(by_module)) + .order_by("module_id", "name") + ) + for interface in rows: + by_module[interface.module_id].append(interface) + return by_module + + +def device_interface_families(interfaces): + """Return each device-level base interface with its channel children.""" + children_by_parent: dict[int, list] = {} + for interface in interfaces: + if _is_channel(interface) and interface.parent_id is not None: + children_by_parent.setdefault(interface.parent_id, []).append(interface) + for children in children_by_parent.values(): + children.sort(key=lambda child: (child.channel_id, child.pk)) + return tuple( + (interface, tuple(children_by_parent.get(interface.pk, ()))) + for interface in interfaces + if not _is_channel(interface) + ) + + +def _interface_rename_plan(device_id, module_id, db_alias, rule, variables, interface) -> InstalledFamilyPlan: + """Return a plan that renames one interface which belongs to no family.""" + status, reason, target_name = None, "", interface.name + try: + target_name = evaluate_name_template(rule.name_template, {**variables, "base": interface.name}) + except (TypeError, ValueError) as error: + status, reason = FamilyStatus.FAILED, f"failed to evaluate the interface name: {error}" + return InstalledFamilyPlan( + family_id=f"flat:{interface.pk}", + topology=FamilyTopology.FLAT, + device_id=device_id, + module_id=module_id, + db_alias=db_alias, + members=( + PlannedMember( + snapshot=InterfaceSnapshot.from_interface(interface), + target_name=target_name, + role=MemberRole.FLAT_MEMBER, + ), + ), + precondition_status=status, + precondition_reason=reason, + ) + + +def plan_interface_rename(module, rule, variables, interface) -> InstalledFamilyPlan: + """Return the plan that renames one interface which belongs to no family.""" + return _interface_rename_plan( + module.device_id, + module.pk, + module_db_alias(module), + rule, + variables, + interface, + ) + + +def plan_device_interface_rename(device, rule, variables, interface, children=()) -> InstalledFamilyPlan: + """Return the plan that renames one device-level interface family.""" + db_alias = _db_alias(device) + if not children and not is_channelized_parent(interface): + return _interface_rename_plan(device.pk, None, db_alias, rule, variables, interface) + # A device rule never builds a family, so its channel count says nothing about this one: the + # members keep the suffixes they carry under whatever name the parent takes. A device-level + # interface has no module template family, so there is no suffix to recover from one either. + targets = lockstep_family_targets( + rule, variables, interface.name, tuple((child.name, child.channel_id) for child in children), {} + ) + return _channelized_plan(device.pk, None, db_alias, interface, children, targets) + + +def plan_installed_families(module, rule, variables, interfaces=None) -> InstalledFamilyPlanSet: + """Return immutable plans for the installed families owned by *module*. + + A batch that already holds the module's interface rows passes them in, so planning a fleet + reads them once rather than once per module. + """ + db_alias = module_db_alias(module) + if interfaces is None: + interfaces = list(Interface.objects.using(db_alias).filter(module_id=module.pk).order_by("pk")) + catalog = TemplateNames(module) + plans = _channelized_plans(module, rule, variables, db_alias, interfaces, catalog) + plans.extend( + _flat_plan(module, db_alias, target_names, members) + for _base_name, target_names, members in _flat_candidates(rule, variables, interfaces, catalog) + ) + plans.sort(key=lambda plan: plan.member_pks[0]) + return InstalledFamilyPlanSet(module_id=module.pk, plans=tuple(plans)) diff --git a/netbox_interface_name_rules/family/names.py b/netbox_interface_name_rules/family/names.py new file mode 100644 index 0000000..f98f1f3 --- /dev/null +++ b/netbox_interface_name_rules/family/names.py @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Shared live interface-name primitives for family execution.""" + +import logging + +from dcim.models import Interface +from django.core.exceptions import ValidationError +from django.db import IntegrityError, transaction + +logger = logging.getLogger(__name__) + +COLLISION_REASON = "target name is already in use" +INTERFACE_NAME_CONSTRAINT = "dcim_interface_unique_device_name" + + +def name_is_taken(device_id, target_name, db_alias, exclude_pk) -> bool: + """Return whether an interface other than *exclude_pk* already owns *target_name* on the device.""" + return ( + Interface.objects.using(db_alias).filter(device_id=device_id, name=target_name).exclude(pk=exclude_pk).exists() + ) + + +def is_name_collision(error: IntegrityError) -> bool: + """Return whether PostgreSQL identified NetBox's interface-name constraint.""" + cause = error.__cause__ + diagnostics = getattr(cause, "diag", None) + return getattr(diagnostics, "constraint_name", None) == INTERFACE_NAME_CONSTRAINT + + +def restore_deferred_channel_names(reconciliations, db_alias): + """Restore plugin-owned names that NetBox's parent cascade changed after commit.""" + child_pks = [child_pk for child_pk, _final_name, _cascade_name in reconciliations] + with transaction.atomic(using=db_alias): + children = ( + Interface.objects.using(db_alias) + .select_for_update(of=("self",)) + .select_related("device") + .order_by("pk") + .in_bulk(child_pks) + ) + for child_pk, final_name, cascade_name in reconciliations: + child = children.get(child_pk) + if child is None or child.name == final_name: + continue + if child.name != cascade_name: + logger.warning( + "Channel interface %s changed to unexpected name %r before deferred reconciliation; " + "leaving it unchanged.", + child_pk, + child.name, + ) + continue + previous_name = child.name + try: + with transaction.atomic(using=db_alias): + child.name = final_name + child.full_clean() + child.save(using=db_alias) + except ValidationError: + child.name = previous_name + logger.exception( + "Failed to restore channel interface %s from NetBox's deferred name %r to %r; skipping.", + child_pk, + cascade_name, + final_name, + ) + except IntegrityError as error: + child.name = previous_name + if not is_name_collision(error): + raise + logger.warning( + "Channel interface %s could not reclaim name %r after NetBox's deferred rename; skipping.", + child_pk, + final_name, + ) + + +def reconcile_after_parent_cascade(parent_before, parent_after, channels, db_alias): + """Schedule restoration of the channel names NetBox's deferred parent cascade will overwrite. + + *channels* carries ``(child_pk, channel_id, final_name)`` for every channel the caller settled. + Registration happens on the caller's open transaction so the callback runs after NetBox's own. + """ + if parent_after == parent_before: + return + reconciliations = tuple( + (child_pk, final_name, f"{parent_after}:{channel_id}") + for child_pk, channel_id, final_name in channels + if final_name == f"{parent_before}:{channel_id}" and final_name != f"{parent_after}:{channel_id}" + ) + if not reconciliations: + return + transaction.on_commit( + lambda: restore_deferred_channel_names(reconciliations, db_alias), + using=db_alias, + ) diff --git a/netbox_interface_name_rules/family/prospective.py b/netbox_interface_name_rules/family/prospective.py new file mode 100644 index 0000000..6172fe1 --- /dev/null +++ b/netbox_interface_name_rules/family/prospective.py @@ -0,0 +1,318 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Plan interface families for interfaces described by name alone. + +A prospective plan says what a rule intends, using the same naming rules installed execution +follows. It carries no row snapshot, so it can describe a family NetBox has not created yet and +the executors refuse to take one. +""" + +import logging +from dataclasses import dataclass + +from ..choices import BreakoutModeChoices +from ..naming import evaluate_name_template +from .capabilities import supports_channelization +from .domain import ( + FamilyStatus, + FamilyTopology, + MemberRole, + ProspectiveFamilyPlan, + ProspectiveFamilyPlanSet, + ProspectiveMember, +) +from .names import COLLISION_REASON +from .structural import UNSUPPORTED_REASON, has_flat_expansion +from .targets import channelized_family_names, channelized_family_targets, flat_family_names, template_channel_suffixes +from .template_names import resolved_template_names + +logger = logging.getLogger(__name__) + +FLAT_EXPANSION_REASON = "the module already carries a flat breakout family" + + +@dataclass(frozen=True, slots=True) +class ProspectiveInterface: + """One interface a prospective plan reasons about, described without a row.""" + + name: str + parent_name: str | None = None + channel_id: int | None = None + channels: int | None = None + + +def describe_interfaces(interfaces) -> tuple[ProspectiveInterface, ...]: + """Describe live interface rows for prospective planning.""" + interfaces = tuple(interfaces) + names_by_pk = {interface.pk: interface.name for interface in interfaces} + return tuple( + ProspectiveInterface( + name=interface.name, + parent_name=names_by_pk.get(getattr(interface, "parent_id", None)), + channel_id=getattr(interface, "channel_id", None), + channels=getattr(interface, "channels", None), + ) + for interface in interfaces + ) + + +def describe_template_interfaces(templates, names=()) -> tuple[ProspectiveInterface, ...]: + """Describe the interfaces a module type's templates produce, plus any extra *names*. + + A channel is paired with its parent only where that parent declares a channel count, because a + template family is what the parent's capacity defines. A name no template resolves to is + described as a plain interface. + """ + parents = {template.pk: template for template in templates if template.channels is not None} + described = {} + for template in templates: + parent = parents.get(template.parent_id) if template.channel_id is not None else None + described[template.resolved] = ProspectiveInterface( + name=template.resolved, + parent_name=None if parent is None else parent.resolved, + channel_id=None if parent is None else template.channel_id, + channels=template.channels, + ) + described.update( + {name: ProspectiveInterface(name=name) for name in names if name not in described}, + ) + return tuple(described.values()) + + +def describe_module_interfaces(module, names=()) -> tuple[ProspectiveInterface, ...]: + """Describe the interfaces *module*'s templates produce, plus any extra *names*. + + A release that cannot model channelized families describes the names alone: its templates carry + no family structure, so reading them would buy nothing. + """ + if not supports_channelization(): + return tuple(ProspectiveInterface(name=name) for name in names) + return describe_template_interfaces( # pragma: no cover - requires channelization support + resolved_template_names(module), names + ) + + +def _partition(interfaces): + """Split described interfaces into ``(roots, children_by_parent_name)``. + + A channel is never an independent candidate: it belongs to the interface it names as its + parent, and one whose parent is not described here is planned by nothing at all. + """ + described = {interface.name for interface in interfaces} + roots = [] + children: dict[str, list] = {} + for interface in interfaces: + if interface.channel_id is None: + roots.append(interface) + elif interface.parent_name in described: # pragma: no cover - requires channelization support + children.setdefault(interface.parent_name, []).append(interface) + return roots, children + + +def _is_family_root(rule, interface, children): + """Return whether *interface* owns a family the rule renames as a unit. + + A breakout rule renames only a family NetBox models with a channel count; on anything else it + builds a new family from the interface itself. Any other rule carries whatever channels the + interface has along with it. + """ + if rule.channel_count > 0: + return interface.channels is not None + return bool(children) + + +def _rename_plan(rule, variables, parent, children, suffixes): # pragma: no cover - channelization only + """Plan the rename of a family the templates or rows already describe.""" + children = sorted(children, key=lambda child: child.channel_id) + targets = channelized_family_targets( + rule, + variables, + parent.name, + parent.channels, + tuple((child.name, child.channel_id) for child in children), + suffixes, + ) + members = [ProspectiveMember(source_name=parent.name, target_name=targets.parent_name, role=MemberRole.PARENT)] + members.extend( + ProspectiveMember( + # A channel whose target cannot be derived keeps the name it has. + source_name=child.name, + target_name=child.name if target is None else target, + role=MemberRole.CHANNEL, + channel_id=child.channel_id, + reason=reason, + ) + for child, (target, reason) in zip(children, targets.channels, strict=True) + ) + return ProspectiveFamilyPlan( + family_id=f"channelized:{parent.name}", + topology=FamilyTopology.CHANNELIZED, + base_name=None, + members=tuple(members), + precondition_status=targets.status, + precondition_reason=targets.reason, + ) + + +def _refused_creation(base_name, topology, role, status, reason): + """Plan a family that will not be built, so the base keeps the name it has.""" + return ProspectiveFamilyPlan( + family_id=f"{topology}:{base_name}", + topology=topology, + base_name=base_name, + members=(ProspectiveMember(source_name=base_name, target_name=base_name, role=role),), + precondition_status=status, + precondition_reason=reason, + ) + + +def _simple_plan(rule, variables, base_name): + """Plan the rename of one interface that owns no family.""" + try: + target_name = evaluate_name_template(rule.name_template, {**variables, "base": base_name}) + except (TypeError, ValueError) as error: + return _refused_creation( + base_name, FamilyTopology.FLAT, MemberRole.FLAT_MEMBER, FamilyStatus.FAILED, str(error) + ) + return ProspectiveFamilyPlan( + family_id=f"flat:{base_name}", + topology=FamilyTopology.FLAT, + base_name=base_name, + members=(ProspectiveMember(source_name=base_name, target_name=target_name, role=MemberRole.FLAT_MEMBER),), + ) + + +def _flat_creation_plan(rule, variables, base_name): + """Plan the flat sibling family a breakout rule creates on one plain interface.""" + try: + target_names = flat_family_names(rule, variables, base_name) + except (TypeError, ValueError) as error: + return _refused_creation( + base_name, FamilyTopology.FLAT, MemberRole.FLAT_MEMBER, FamilyStatus.FAILED, str(error) + ) + members = tuple( + ProspectiveMember( + source_name=base_name if offset == 0 else None, + target_name=target_name, + role=MemberRole.FLAT_MEMBER, + ) + for offset, target_name in enumerate(target_names) + ) + return ProspectiveFamilyPlan( + family_id=f"flat:{base_name}", + topology=FamilyTopology.FLAT, + base_name=base_name, + members=members, + ) + + +def _structural_refusal(base_name, flat_expansion, taken_names, target_names): # pragma: no cover - see below + """Return why the planned family cannot be built here, or None when nothing refuses it.""" + if flat_expansion: + # Converting one sibling into a parent would strand the others beside the new family. + return FLAT_EXPANSION_REASON + collisions = [name for name in target_names if name != base_name and name in taken_names] + if collisions: + return f"{COLLISION_REASON}: {collisions[0]}" + return None + + +def _modelled_structural_plan(rule, variables, base_name, context): # pragma: no cover - see below + """Plan the channelized family for a NetBox release that can hold it.""" + try: + parent_name, channels = channelized_family_names(rule, base_name, variables) + except (TypeError, ValueError) as error: + return _refused_creation( + base_name, FamilyTopology.CHANNELIZED, MemberRole.PARENT, FamilyStatus.FAILED, str(error) + ) + target_names = (parent_name, *(name for _channel_id, name in channels)) + refusal = _structural_refusal(base_name, context.flat_expansion, context.taken_names, target_names) + if refusal is not None: + return _refused_creation( + base_name, FamilyTopology.CHANNELIZED, MemberRole.PARENT, FamilyStatus.BLOCKED, refusal + ) + members = ( + ProspectiveMember(source_name=base_name, target_name=parent_name, role=MemberRole.PARENT), + *( + ProspectiveMember(source_name=None, target_name=name, role=MemberRole.CHANNEL, channel_id=channel_id) + for channel_id, name in channels + ), + ) + return ProspectiveFamilyPlan( + family_id=f"channelized:{base_name}", + topology=FamilyTopology.CHANNELIZED, + base_name=base_name, + members=members, + ) + + +def _structural_plan(rule, variables, base_name, context): + """Plan the channelized family a rule builds on one plain interface.""" + if not supports_channelization(): + return _refused_creation( + base_name, FamilyTopology.CHANNELIZED, MemberRole.PARENT, FamilyStatus.UNSUPPORTED, UNSUPPORTED_REASON + ) + return _modelled_structural_plan(rule, variables, base_name, context) # pragma: no cover - see above + + +def _plain_plan(rule, variables, base_name, context): + """Plan the family *rule* intends on one plain interface.""" + if rule.channel_count <= 0: + return _simple_plan(rule, variables, base_name) + if rule.breakout_mode == BreakoutModeChoices.CHANNELIZED: + return _structural_plan(rule, variables, base_name, context) + return _flat_creation_plan(rule, variables, base_name) + + +@dataclass(frozen=True, slots=True) +class _CreationContext: + """The module-wide facts every creation plan on one module shares.""" + + taken_names: frozenset + flat_expansion: bool + + +def _creation_context(module, rule, interfaces, plain): + """Read the module-wide facts once, and only where a creation plan can use them.""" + builds_channels = rule.channel_count > 0 and rule.breakout_mode == BreakoutModeChoices.CHANNELIZED + flat_expansion = bool(plain) and builds_channels and supports_channelization() and has_flat_expansion(module) + return _CreationContext( + taken_names=frozenset(interface.name for interface in interfaces), + flat_expansion=flat_expansion, + ) + + +class _TemplateSuffixes: + """The module type's per-channel name suffixes, read only if a channel name needs recovering.""" + + def __init__(self, module): + self._module = module + self._suffixes = None + + def get(self, channel_id, default=None): # pragma: no cover - requires channelization support + """Return the suffixes the templates give *channel_id*, reading them on first use.""" + if self._suffixes is None: + self._suffixes = template_channel_suffixes(resolved_template_names(self._module)) + return self._suffixes.get(channel_id, default) + + +def plan_prospective_families(module, rule, variables, interfaces) -> ProspectiveFamilyPlanSet: + """Return one plan for each family *rule* intends on the described *interfaces*. + + *interfaces* are ``ProspectiveInterface`` values, so a name NetBox has not created yet is + planned exactly like one it has and no row is written. Collision checking covers the names + described here, because a prospective plan knows only the interfaces it was given. + """ + roots, children = _partition(interfaces) + families = [(root, children.get(root.name, ())) for root in roots] + context = _creation_context( + module, rule, interfaces, [root for root, family in families if not _is_family_root(rule, root, family)] + ) + suffixes = _TemplateSuffixes(module) + plans = tuple( + _rename_plan(rule, variables, root, family, suffixes) + if _is_family_root(rule, root, family) + else _plain_plan(rule, variables, root.name, context) + for root, family in families + ) + return ProspectiveFamilyPlanSet(module_id=module.pk, plans=plans) diff --git a/netbox_interface_name_rules/family/structural.py b/netbox_interface_name_rules/family/structural.py new file mode 100644 index 0000000..f6fa858 --- /dev/null +++ b/netbox_interface_name_rules/family/structural.py @@ -0,0 +1,382 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Plan and execute the creation of interface families.""" + +import logging + +from dcim.choices import InterfaceTypeChoices +from dcim.models import Interface, InterfaceTemplate +from django.core.exceptions import ValidationError +from django.db import IntegrityError, transaction + +from .capabilities import supports_channelization +from .domain import ( + FamilyOutcome, + FamilyStatus, + FamilyTopology, + FlatCreationPlan, + InterfaceSnapshot, + MemberOutcome, + PlannedChannel, + StructuralFamilyPlan, +) +from .installed import module_db_alias +from .names import COLLISION_REASON, is_name_collision, name_is_taken, reconcile_after_parent_cascade +from .targets import channelized_family_names, flat_family_names + +logger = logging.getLogger(__name__) + +UNSUPPORTED_REASON = "this NetBox release cannot model channelized interfaces" +STALE_REASON = "the base interface changed after planning" +MODULE_CHANGED_REASON = "the module's interfaces changed after planning" + + +def _flat_expansion(module_type_id, module_id, db_alias) -> bool: # pragma: no cover - channelization only + """Return whether the module carries more plain interfaces than its templates describe.""" + templates = InterfaceTemplate.objects.using(db_alias).filter(module_type_id=module_type_id).count() + plain = Interface.objects.using(db_alias).filter(module_id=module_id, channel_id__isnull=True) + return plain.count() > templates + + +def has_flat_expansion(module) -> bool: # pragma: no cover - requires channelization support + """Return whether *module* carries more plain interfaces than its module type's templates describe. + + A flat breakout leaves N-1 rows beyond the templates, so the surplus is the structural mark of a + family an earlier apply installed. A channel belongs to the parent that declares it, so it is + never part of that surplus: counting one would stop a second port from gaining its own family. + Counting templates rather than their resolved names keeps two templates that resolve to the same + string from reading as one. + """ + return _flat_expansion(module.module_type_id, module.pk, module_db_alias(module)) + + +def _plan(module, db_alias, base, parent_target_name, channels, status=None, reason=""): + """Build one immutable structural plan for *base*.""" + return StructuralFamilyPlan( + family_id=f"structural:{base.pk}", + device_id=module.device_id, + module_id=module.pk, + module_type_id=module.module_type_id, + db_alias=db_alias, + base=InterfaceSnapshot.from_interface(base), + parent_target_name=parent_target_name, + channel_count=len(channels), + channels=tuple(PlannedChannel(channel_id=channel_id, name=name) for channel_id, name in channels), + precondition_status=status, + precondition_reason=reason, + ) + + +def _modelled_plan(module, rule, variables, db_alias, base): # pragma: no cover - channelization only + """Return the plan for a NetBox release that can hold the family.""" + try: + parent_target_name, channels = channelized_family_names(rule, base.name, variables) + except (TypeError, ValueError) as error: + reason = f"failed to evaluate the family names: {error}" + return _plan(module, db_alias, base, base.name, (), FamilyStatus.FAILED, reason) + if has_flat_expansion(module): + # Converting one sibling into a parent would strand the others beside the new family. + reason = f"module {module} already carries a flat breakout family" + return _plan(module, db_alias, base, parent_target_name, channels, FamilyStatus.BLOCKED, reason) + return _plan(module, db_alias, base, parent_target_name, channels) + + +def plan_structural_family(module, rule, variables, base) -> StructuralFamilyPlan: + """Return the immutable plan for the channelized family *rule* builds on plain interface *base*.""" + db_alias = module_db_alias(module) + if not supports_channelization(): + return _plan(module, db_alias, base, base.name, (), FamilyStatus.UNSUPPORTED, UNSUPPORTED_REASON) + return _modelled_plan(module, rule, variables, db_alias, base) # pragma: no cover - see above + + +def _outcome(plan, status, members, reason=""): + """Build the immutable outcome of one structural family operation.""" + return FamilyOutcome( + family_id=plan.family_id, + topology=FamilyTopology.CHANNELIZED, + status=status, + members=members, + reason=reason, + ) + + +def _refused(plan, status, reason, target_name): + """Log why the family was not built and return an outcome that touched no row.""" + logger.warning("Cannot build a channelized family on interface %r: %s.", plan.base.name, reason) + member = MemberOutcome( + interface_pk=plan.base.pk, + current_name=plan.base.name, + target_name=target_name, + status=status, + reason=reason, + ) + return _outcome(plan, status, (member,), reason) + + +def _locked_base(plan): + """Lock and return the live base row, or None when it is gone.""" + return ( + Interface.objects.using(plan.db_alias) + .select_for_update(of=("self",)) + .select_related("device", "module") + .filter(pk=plan.base.pk) + .first() + ) + + +def _first_taken_name(plan): # pragma: no cover - requires channelization support + """Return the first planned name another interface on the device already owns, or None.""" + for target_name in plan.target_names: + if name_is_taken(plan.device_id, target_name, plan.db_alias, exclude_pk=plan.base.pk): + return target_name + return None + + +def _create_channels(plan, parent): # pragma: no cover - requires channelization support + """Create every planned channel under *parent* and return their outcomes.""" + members = [] + for channel in plan.channels: + row = Interface( + device=parent.device, + module=parent.module, + name=channel.name, + type=InterfaceTypeChoices.TYPE_CHANNEL, + parent=parent, + channel_id=channel.channel_id, + enabled=parent.enabled, + ) + row.full_clean() + row.save(using=plan.db_alias) + members.append( + MemberOutcome( + interface_pk=row.pk, + current_name=channel.name, + target_name=channel.name, + status=FamilyStatus.CHANGED, + ) + ) + return members + + +def _create_family(plan, base): # pragma: no cover - requires channelization support + """Rewrite *base* into the family parent, create its channels, and return every member outcome.""" + parent_status = FamilyStatus.CHANGED if plan.parent_target_name != base.name else FamilyStatus.UNCHANGED + base.channels = plan.channel_count + base.name = plan.parent_target_name + base.full_clean() + base.save(using=plan.db_alias) + parent_member = MemberOutcome( + interface_pk=base.pk, + current_name=plan.base.name, + target_name=plan.parent_target_name, + status=parent_status, + ) + channel_members = _create_channels(plan, base) + reconcile_after_parent_cascade( + plan.base.name, + plan.parent_target_name, + tuple( + (member.interface_pk, channel.channel_id, channel.name) + for member, channel in zip(channel_members, plan.channels, strict=True) + ), + plan.db_alias, + ) + return (parent_member, *channel_members) + + +def _install_family(plan): # pragma: no cover - requires channelization support + """Create the whole family in one transaction, or write nothing at all.""" + try: + with transaction.atomic(using=plan.db_alias): + base = _locked_base(plan) + if base is None or InterfaceSnapshot.from_interface(base) != plan.base: + return _refused(plan, FamilyStatus.STALE, STALE_REASON, plan.parent_target_name) + # A sibling added since planning would be stranded beside the family this plan builds. + if _flat_expansion(plan.module_type_id, plan.module_id, plan.db_alias): + return _refused(plan, FamilyStatus.STALE, MODULE_CHANGED_REASON, plan.parent_target_name) + taken = _first_taken_name(plan) + if taken is not None: + return _refused(plan, FamilyStatus.BLOCKED, f"{COLLISION_REASON}: {taken}", taken) + members = _create_family(plan, base) + except ValidationError as error: + return _refused(plan, FamilyStatus.BLOCKED, " ".join(error.messages), plan.parent_target_name) + except IntegrityError as error: + if not is_name_collision(error): + raise + return _refused(plan, FamilyStatus.BLOCKED, COLLISION_REASON, plan.parent_target_name) + return _outcome(plan, FamilyStatus.CHANGED, members) + + +def execute_structural_family(plan: StructuralFamilyPlan) -> FamilyOutcome: + """Create the planned channelized family, or leave every row exactly as it was. + + Only a structural plan names the base row to rewrite, so anything else (a prospective plan + above all) is refused before a single row is locked. + """ + if not isinstance(plan, StructuralFamilyPlan): + raise TypeError(f"{type(plan).__name__} is not an executable family plan") + if plan.precondition_status is not None: + return _refused(plan, plan.precondition_status, plan.precondition_reason, plan.parent_target_name) + return _install_family(plan) # pragma: no cover - requires channelization support + + +def install_channelized_family(module, rule, variables, base) -> FamilyOutcome: + """Build the channelized family *rule* describes on plain interface *base*.""" + return execute_structural_family(plan_structural_family(module, rule, variables, base)) + + +# --------------------------------------------------------------------------- +# Flat breakout families +# --------------------------------------------------------------------------- +# A flat family is N sibling interfaces on one module: the base takes the first name and the rest +# are new rows. Unlike a channelized family there is no parent to cascade from, so a sibling whose +# name is taken is skipped on its own while the family keeps the names it could take. + + +def _flat_creation_plan(module, db_alias, base, target_names, status=None, reason=""): + """Build one immutable flat-creation plan for *base*.""" + return FlatCreationPlan( + family_id=f"flat:{base.pk}", + device_id=module.device_id, + module_id=module.pk, + db_alias=db_alias, + base=InterfaceSnapshot.from_interface(base), + target_names=target_names, + precondition_status=status, + precondition_reason=reason, + ) + + +def plan_flat_family(module, rule, variables, base) -> FlatCreationPlan: + """Return the immutable plan for the flat breakout family *rule* builds on plain interface *base*.""" + db_alias = module_db_alias(module) + try: + target_names = flat_family_names(rule, variables, base.name) + except (TypeError, ValueError) as error: + reason = f"failed to evaluate the family names: {error}" + return _flat_creation_plan(module, db_alias, base, (base.name,), FamilyStatus.FAILED, reason) + return _flat_creation_plan(module, db_alias, base, target_names) + + +def _flat_outcome(plan, status, members, reason=""): + """Build the immutable outcome of one flat family operation.""" + return FamilyOutcome( + family_id=plan.family_id, + topology=FamilyTopology.FLAT, + status=status, + members=members, + reason=reason, + ) + + +def _flat_refused(plan, status, reason): + """Log why the family was not built and return an outcome that touched no row.""" + logger.warning("Cannot build a flat family on interface %r: %s.", plan.base.name, reason) + member = MemberOutcome( + interface_pk=plan.base.pk, + current_name=plan.base.name, + target_name=plan.target_names[0], + status=status, + reason=reason, + ) + return _flat_outcome(plan, status, (member,), reason) + + +def _rename_flat_base(plan, base): + """Give the base the family's first name, or report why it keeps the one it has.""" + target_name = plan.target_names[0] + if target_name == base.name: + return MemberOutcome(base.pk, base.name, target_name, FamilyStatus.UNCHANGED) + if name_is_taken(plan.device_id, target_name, plan.db_alias, exclude_pk=base.pk): + logger.warning( + "Interface name %r already exists on device %s; skipping rename of %r to %r.", + target_name, + plan.device_id, + base.name, + target_name, + ) + return MemberOutcome(base.pk, base.name, target_name, FamilyStatus.BLOCKED, COLLISION_REASON) + previous_name = base.name + base.name = target_name + base.full_clean() + base.save(using=plan.db_alias) + return MemberOutcome(base.pk, previous_name, target_name, FamilyStatus.CHANGED) + + +def _module_rows(plan): + """Return the family names this module already carries, so a re-apply creates nothing twice.""" + return dict( + Interface.objects.using(plan.db_alias) + .filter(module_id=plan.module_id, name__in=plan.target_names) + .values_list("name", "pk") + ) + + +def _create_sibling(plan, base, name): + """Create one sibling interface beside *base*, or report the name it could not take.""" + if name_is_taken(plan.device_id, name, plan.db_alias, exclude_pk=base.pk): + logger.warning( + "Interface name %r already exists on device %s; skipping the sibling of %r.", + name, + plan.device_id, + base.name, + ) + return MemberOutcome(plan.base.pk, plan.base.name, name, FamilyStatus.BLOCKED, COLLISION_REASON) + row = Interface( + device=base.device, + module=base.module, + name=name, + type=base.type, + enabled=base.enabled, + ) + row.full_clean() + row.save(using=plan.db_alias) + return MemberOutcome(row.pk, name, name, FamilyStatus.CHANGED) + + +def _build_flat_family(plan, base): + """Name the base and create every sibling the module still lacks.""" + members = [_rename_flat_base(plan, base)] + installed = _module_rows(plan) + for name in plan.target_names[1:]: + if name in installed: + members.append(MemberOutcome(installed[name], name, name, FamilyStatus.UNCHANGED)) + continue + members.append(_create_sibling(plan, base, name)) + return tuple(members) + + +def _install_flat_family(plan): + """Build the whole family in one transaction, or write nothing at all.""" + try: + with transaction.atomic(using=plan.db_alias): + base = _locked_base(plan) + if base is None or InterfaceSnapshot.from_interface(base) != plan.base: + return _flat_refused(plan, FamilyStatus.STALE, STALE_REASON) + members = _build_flat_family(plan, base) + except ValidationError as error: + return _flat_refused(plan, FamilyStatus.BLOCKED, " ".join(error.messages)) + except IntegrityError as error: + if not is_name_collision(error): + raise + return _flat_refused(plan, FamilyStatus.BLOCKED, COLLISION_REASON) + return _flat_outcome(plan, _flat_status(members), members) + + +def _flat_status(members): + """Summarize member outcomes without hiding partial success.""" + statuses = {member.status for member in members} + if FamilyStatus.CHANGED in statuses: + return FamilyStatus.CHANGED + if FamilyStatus.BLOCKED in statuses: + return FamilyStatus.BLOCKED + return FamilyStatus.UNCHANGED + + +def execute_flat_family(plan: FlatCreationPlan) -> FamilyOutcome: + """Build the planned flat breakout family, or leave every row exactly as it was.""" + if not isinstance(plan, FlatCreationPlan): + raise TypeError(f"{type(plan).__name__} is not an executable family plan") + if plan.precondition_status is not None: + return _flat_refused(plan, plan.precondition_status, plan.precondition_reason) + return _install_flat_family(plan) diff --git a/netbox_interface_name_rules/family/targets.py b/netbox_interface_name_rules/family/targets.py new file mode 100644 index 0000000..f94e1bf --- /dev/null +++ b/netbox_interface_name_rules/family/targets.py @@ -0,0 +1,214 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Compute the names a rule intends for one interface family. + +Every planner names a family through this module, so an installed plan, a prospective plan and a +structural plan cannot spell the same family differently. Nothing here reads or writes a row: the +inputs are names, channel identifiers and the rule. +""" + +from dataclasses import dataclass + +from ..choices import BreakoutModeChoices +from ..naming import evaluate_name_template +from .domain import FamilyStatus + +AMBIGUOUS_SUFFIX_REASON = "channel suffix is ambiguous or unavailable" + + +@dataclass(frozen=True, slots=True) +class FamilyTargets: + """The names a rule intends for one channelized family, and why it may refuse.""" + + parent_name: str + channels: tuple[tuple[str | None, str], ...] + status: FamilyStatus | None = None + reason: str = "" + + +def child_name_suffix(child_name, parent_name): # pragma: no cover - requires channelization support + """Return the suffix *child_name* adds to *parent_name*, or None when it adds none. + + The first character must be non-alphanumeric so ``et0``/``et01`` is never mistaken for a + family; the punctuation itself is free-form (``:``, ``-``, ``_`` and ``@`` all occur in the + wild), so it is not restricted to a fixed separator. + """ + if not parent_name or not child_name.startswith(parent_name): + return None + suffix = child_name[len(parent_name) :] + if not suffix or suffix[0].isalnum(): + return None + return suffix + + +def template_channel_suffixes(templates): # pragma: no cover - requires channelization support + """Map each channel identifier to the name suffixes the module type's templates give it. + + The suffix comes from the template family itself, so a child that lost its parent's prefix in + an earlier partial rename can still be repaired. A module type with several families may spell + the same channel differently in each (``et0:2`` vs ``sw0.2``), so the suffixes are collected per + channel rather than overwritten: recovery only uses one when every family agrees on it. + """ + parents = {template.pk: template.resolved for template in templates if template.channels is not None} + suffixes: dict[int, set[str]] = {} + for template in templates: + if template.channel_id is None or template.parent_id not in parents: + continue + suffix = child_name_suffix(template.resolved, parents[template.parent_id]) + if suffix is not None: + suffixes.setdefault(template.channel_id, set()).add(suffix) + return suffixes + + +def builds_channelized_family(rule) -> bool: + """Return whether *rule* builds a channelized family instead of flat sibling interfaces.""" + return rule.channel_count > 0 and rule.breakout_mode == BreakoutModeChoices.CHANNELIZED + + +def one_family_per_name_set(candidates): + """Return the index of one candidate base per family, in candidate order. + + *candidates* pairs each base name with the names the rule intends for the family built on it. + Two bases that intend the same names describe one family an earlier apply already started, so + only one of them may build it; the base the family already names is preferred, because renaming + it onto itself is the no-op the other base cannot manage. + """ + kept: dict[tuple[str, ...], int] = {} + for index, (base_name, target_names) in enumerate(candidates): + current = kept.get(target_names) + if current is None or (base_name == target_names[0] and candidates[current][0] != target_names[0]): + kept[target_names] = index + return tuple(sorted(kept.values())) + + +def flat_family_names(rule, variables, base_name): + """Return the names of the flat sibling family that *rule* defines on *base_name*.""" + family_variables = {**variables, "base": base_name} + return tuple( + evaluate_name_template( + rule.name_template, + {**family_variables, "channel": str(rule.channel_start + offset)}, + ) + for offset in range(rule.channel_count) + ) + + +def channelized_family_names(rule, base_name, variables): # pragma: no cover - channelization only + """Return ``(parent_name, ((channel_id, name), ...))`` for the family *rule* builds on *base_name*. + + ``{base}`` is the base interface's current name for the parent and every channel; ``{channel}`` + is ``channel_start + channel_id - 1``. A blank parent template leaves the base's name alone. + Takes the name rather than the interface so prediction can reuse it without a row to point at. + """ + family_variables = {**variables, "base": base_name} + parent_name = base_name + if rule.parent_name_template: + parent_name = evaluate_name_template(rule.parent_name_template, family_variables) + channels = tuple( + ( + channel_id, + evaluate_name_template( + rule.name_template, {**family_variables, "channel": str(rule.channel_start + channel_id - 1)} + ), + ) + for channel_id in range(1, rule.channel_count + 1) + ) + return parent_name, channels + + +def intended_family_names(rule, variables, base_name): + """Return every name *rule* intends for the family it builds on *base_name*. + + A base whose names cannot be evaluated is its own family: it names nothing else, so nothing + else can be grouped with it. + """ + try: + if builds_channelized_family(rule): + parent_name, channels = channelized_family_names(rule, base_name, variables) + return (parent_name, *(name for _channel_id, name in channels)) + return flat_family_names(rule, variables, base_name) + except (TypeError, ValueError): + return (base_name,) + + +def _simple_child_target(child_name, channel_id, parent_name, parent_target, suffixes): # pragma: no cover + """Return a simple rule's child target, or None plus the reason it cannot be derived.""" + suffix = child_name_suffix(child_name, parent_name) + if suffix is None: + candidates = suffixes.get(channel_id, set()) + if len(candidates) == 1: + suffix = next(iter(candidates)) + if suffix is None: + return None, AMBIGUOUS_SUFFIX_REASON + return parent_target + suffix, "" + + +def _failed(parent_name, children, error): # pragma: no cover - requires channelization support + """Return targets that leave the whole family alone because a template could not be evaluated.""" + reason = f"failed to evaluate family targets: {error}" + return FamilyTargets( + parent_name=parent_name, + channels=tuple((child_name, reason) for child_name, _channel_id in children), + status=FamilyStatus.FAILED, + reason=reason, + ) + + +def _breakout_targets(rule, variables, parent_name, parent_channels, children): # pragma: no cover + """Return the names a breakout rule intends for an existing channelized family.""" + if parent_channels != rule.channel_count: + reason = f"installed parent declares {parent_channels} channels but the rule defines {rule.channel_count}" + return FamilyTargets( + parent_name=parent_name, + channels=tuple((child_name, reason) for child_name, _channel_id in children), + status=FamilyStatus.BLOCKED, + reason=reason, + ) + try: + parent_target = parent_name + if rule.breakout_mode == BreakoutModeChoices.CHANNELIZED and rule.parent_name_template: + parent_target = evaluate_name_template(rule.parent_name_template, {**variables, "base": parent_name}) + channels = tuple( + ( + evaluate_name_template( + rule.name_template, + {**variables, "base": parent_name, "channel": str(rule.channel_start + channel_id - 1)}, + ), + "", + ) + for _child_name, channel_id in children + ) + except (TypeError, ValueError) as error: + return _failed(parent_name, children, error) + return FamilyTargets(parent_name=parent_target, channels=channels) + + +def lockstep_family_targets(rule, variables, parent_name, children, suffixes): # pragma: no cover + """Return the names a simple rule intends for a family renamed in lockstep with its parent. + + The rule's channel count says nothing here: this family already exists, and every member keeps + the suffix it carries under whatever name the parent takes. + """ + try: + parent_target = evaluate_name_template(rule.name_template, {**variables, "base": parent_name}) + except (TypeError, ValueError) as error: + return _failed(parent_name, children, error) + channels = tuple( + _simple_child_target(child_name, channel_id, parent_name, parent_target, suffixes) + for child_name, channel_id in children + ) + return FamilyTargets(parent_name=parent_target, channels=channels) + + +def channelized_family_targets( # pragma: no cover - requires channelization support + rule, variables, parent_name, parent_channels, children, suffixes +) -> FamilyTargets: + """Return the names *rule* intends for the channelized family named *parent_name*. + + *children* pairs each channel's current name with its channel identifier, in channel order. + A breakout rule renames the channels it describes; any other rule carries the family along with + its parent, keeping each channel's own suffix. + """ + if rule.channel_count > 0: + return _breakout_targets(rule, variables, parent_name, parent_channels, children) + return lockstep_family_targets(rule, variables, parent_name, children, suffixes) diff --git a/netbox_interface_name_rules/family/template_names.py b/netbox_interface_name_rules/family/template_names.py new file mode 100644 index 0000000..e48b5a2 --- /dev/null +++ b/netbox_interface_name_rules/family/template_names.py @@ -0,0 +1,189 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Resolve current and historical NetBox interface-template names.""" + +import contextlib +import copy +import re +import threading +from collections import namedtuple +from dataclasses import dataclass +from re import Pattern + +from dcim.models import InterfaceTemplate, Module + +from ..rule_selection import _compile_pattern + +BAY_CHAIN_RELATIONS = ( + "device", + "device__virtual_chassis", + "module_bay", + "module_bay__parent", + "module_bay__module", + "module_bay__module__module_bay", + "module_bay__module__module_bay__parent", + "module_bay__module__module_bay__module", +) + +_VC_SENTINEL = "InrVcPositionSentinel{}End" +_VC_SENTINEL_RE = re.compile(r"InrVcPositionSentinel(\d+)End") +# NetBox stores vc_position in a PositiveIntegerField, so ten digits cover every valid value. +_VC_POSITION_DIGITS = r"\d{1,10}" + +RawMatcher = namedtuple("RawMatcher", ("template_name", "resolved", "pattern")) +RawNames = namedtuple("RawNames", ("names", "matchers")) + + +@dataclass(frozen=True, slots=True) +class ResolvedTemplateName: + """One template's current name and optional historical-name matcher.""" + + pk: int + template_name: str + resolved: str + historical_pattern: Pattern[str] | None + parent_id: int | None + channel_id: int | None + channels: int | None + + +def vc_position_re(): + """Return NetBox's virtual-chassis template-token pattern when available.""" + try: + from dcim.constants import VC_POSITION_RE + except ImportError: + return None + return VC_POSITION_RE # pragma: no cover - only available on NetBox releases with the token + + +def _vc_position_alternatives(fallback): # pragma: no cover - requires virtual-chassis token support + """Return every value represented by one virtual-chassis position token.""" + if fallback is None: + return _VC_POSITION_DIGITS + return f"(?:{_VC_POSITION_DIGITS}|{re.escape(fallback)})" + + +def _historical_pattern(template, module, token_re): # pragma: no cover - requires virtual-chassis token support + """Return a matcher for every historical resolution of *template*.""" + fallbacks = [] + + def mark(match): + fallbacks.append(match.group(1)) + return _VC_SENTINEL.format(len(fallbacks) - 1) + + marked = token_re.sub(mark, template.name) + if not fallbacks: + return None + stub = copy.copy(template) + stub.name = marked + parts = _VC_SENTINEL_RE.split(re.escape(stub.resolve_name(module))) + literals = parts[0::2] + indexes = parts[1::2] + # A sentinel-shaped literal in the template name would shift these indexes; refuse to guess. + if indexes != [str(index) for index in range(len(fallbacks))]: + return None + # Adjacent tokens cannot be told apart, and their alternatives would backtrack without bound. + if any(not literal for literal in literals[1:-1]): + return None + pattern = literals[0] + for index, literal in zip(indexes, literals[1:], strict=True): + pattern += _vc_position_alternatives(fallbacks[int(index)]) + literal + return _compile_pattern(pattern) + + +# One batch of modules shares its module chains, template rows and resolved names, thread-locally. +_pin = threading.local() + + +@contextlib.contextmanager +def pinned_template_cache(modules=()): + """Share resolved interface templates across every module in one batch. + + Each module in *modules* must already carry ``BAY_CHAIN_RELATIONS``, so its templates resolve + without a refetch. Modules the block meets later are chained and cached on first use, and one + module type's template rows are read once. Nested blocks share the outermost cache. + """ + depth = getattr(_pin, "depth", 0) + _pin.depth = depth + 1 + try: + # Setting up the cache must sit inside the try, or a raise here strands the depth forever. + if depth == 0: + _pin.chained = {} + _pin.templates = {} + _pin.resolved = {} + _pin.chained.update({module.pk: module for module in modules}) + yield + finally: + _pin.depth -= 1 + if _pin.depth == 0: + for attr in ("chained", "templates", "resolved"): + _pin.__dict__.pop(attr, None) + + +def module_with_bay_chain(module): + """Re-fetch *module* with every relation template name resolution uses.""" + chained = getattr(_pin, "chained", None) + if chained is None: + return Module.objects.select_related(*BAY_CHAIN_RELATIONS).get(pk=module.pk) + if module.pk not in chained: + chained[module.pk] = Module.objects.select_related(*BAY_CHAIN_RELATIONS).get(pk=module.pk) + return chained[module.pk] + + +def _interface_templates(module_type_id): + """Return one module type's interface templates in primary-key order.""" + templates = getattr(_pin, "templates", None) + if templates is None: + return list(InterfaceTemplate.objects.filter(module_type_id=module_type_id).order_by("pk")) + if module_type_id not in templates: + templates[module_type_id] = list(InterfaceTemplate.objects.filter(module_type_id=module_type_id).order_by("pk")) + return templates[module_type_id] + + +def resolve_templates(templates, module) -> tuple[ResolvedTemplateName, ...]: + """Resolve already-loaded interface templates against *module*.""" + token_re = vc_position_re() + return tuple( + ResolvedTemplateName( + pk=template.pk, + template_name=template.name, + resolved=template.resolve_name(module), + historical_pattern=None if token_re is None else _historical_pattern(template, module, token_re), + parent_id=getattr(template, "parent_id", None), + channel_id=getattr(template, "channel_id", None), + channels=getattr(template, "channels", None), + ) + for template in templates + ) + + +def raw_names_from(templates) -> RawNames: + """Return the current names and historical matchers of already-resolved templates.""" + matchers = [ + RawMatcher(template.template_name, template.resolved, template.historical_pattern) + for template in templates + if template.historical_pattern is not None # pragma: no cover - token templates only + ] + return RawNames({template.resolved for template in templates}, matchers) + + +def raw_name_matchers(module): + """Return current and historical raw template names for *module*.""" + return raw_names_from(resolved_template_names(module)) + + +def raw_name_patterns(module): + """Return historical matchers for the module's token templates.""" + return [matcher.pattern for matcher in raw_name_matchers(module).matchers] + + +def resolved_template_names(module) -> tuple[ResolvedTemplateName, ...]: + """Load and resolve every interface template for *module* once.""" + resolved = getattr(_pin, "resolved", None) + if resolved is not None and module.pk in resolved: + return resolved[module.pk] + chained = module_with_bay_chain(module) + names = resolve_templates(_interface_templates(chained.module_type_id), chained) + if resolved is not None: + resolved[module.pk] = names + return names diff --git a/netbox_interface_name_rules/forms.py b/netbox_interface_name_rules/forms.py index 8509915..a806310 100644 --- a/netbox_interface_name_rules/forms.py +++ b/netbox_interface_name_rules/forms.py @@ -1,7 +1,5 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (C) 2025 Marcin Zieba -import re - from dcim.models import DeviceType, ModuleType, Platform from django import forms from django.core.exceptions import ValidationError @@ -39,8 +37,8 @@ class RuleTestForm(forms.Form): ) module_type_pattern = forms.CharField( required=False, - label="Module Type Pattern (regex)", - help_text="Regex pattern matched against ModuleType.model via re.fullmatch()", + label="Module Type Pattern (RE2)", + help_text="RE2 pattern matched against the complete ModuleType.model value", widget=forms.TextInput(attrs={"class": "form-control"}), ) parent_module_type = forms.ModelChoiceField( @@ -139,15 +137,13 @@ def clean(self): if not module_type_pattern: self.add_error("module_type_pattern", "A regex pattern is required when regex mode is enabled.") else: + from .regex_safety import compile_module_type_pattern + try: - re.compile(module_type_pattern) - except re.error as exc: - self.add_error("module_type_pattern", f"Invalid regex: {exc}") - else: - from .models import _REDOS_PATTERN - - if _REDOS_PATTERN.search(module_type_pattern): - self.add_error("module_type_pattern", "Pattern contains potentially unsafe nested quantifiers.") + compile_module_type_pattern(module_type_pattern) + except ValidationError as exc: + for field, messages in exc.message_dict.items(): + self.add_error(field, messages) if module_type: self.add_error("module_type", "Module Type (exact) must be empty when regex mode is enabled.") else: diff --git a/netbox_interface_name_rules/jobs.py b/netbox_interface_name_rules/jobs.py index a2323d1..f8d9282 100644 --- a/netbox_interface_name_rules/jobs.py +++ b/netbox_interface_name_rules/jobs.py @@ -27,16 +27,15 @@ def run(self, *args, **kwargs): self.logger.warning("InterfaceNameRule with pk=%s does not exist; skipping.", rule_id) return - conflicts = [] try: - count = apply_rule_to_existing(rule, conflicts=conflicts) + outcome = apply_rule_to_existing(rule) except Exception as exc: self.logger.exception("Failed to apply rule '%s': %s", rule_id, exc) raise - self.logger.info("Renamed %d interface(s) using rule '%s'", count, rule) - if conflicts: - self.logger.warning("%d interface(s) skipped — the plugin log names each one.", len(conflicts)) + self.logger.info("Renamed %d interface(s) using rule '%s'", outcome.changed_count, rule) + if outcome.skipped_members: + self.logger.warning("%d interface(s) skipped. The plugin log names each one.", len(outcome.skipped_members)) class ConvertFlatFamiliesJob(JobRunner): @@ -61,13 +60,12 @@ def run(self, *args, **kwargs): self.logger.warning("InterfaceNameRule with pk=%s does not exist; skipping.", rule_id) return - conflicts = [] try: - count = convert_flat_families(rule, conflicts=conflicts) + outcome = convert_flat_families(rule) except Exception as exc: self.logger.exception("Failed to convert families for rule '%s': %s", rule_id, exc) raise - self.logger.info("Converted %d interface family(ies) using rule '%s'", count, rule) - if conflicts: - self.logger.warning("%d family(ies) skipped — the plugin log names each one.", len(conflicts)) + self.logger.info("Converted %d interface family(ies) using rule '%s'", len(outcome.changed_families), rule) + if outcome.blocked_families: + self.logger.warning("%d family(ies) skipped. The plugin log names each one.", len(outcome.blocked_families)) diff --git a/netbox_interface_name_rules/migrations/0014_validate_re2_patterns.py b/netbox_interface_name_rules/migrations/0014_validate_re2_patterns.py new file mode 100644 index 0000000..41bf19b --- /dev/null +++ b/netbox_interface_name_rules/migrations/0014_validate_re2_patterns.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba + +from django.db import migrations + +_UNICODE_SHORTHANDS = frozenset("dDsSwW") +_POSIX_CLASSES = frozenset( + { + "alnum", + "alpha", + "ascii", + "blank", + "cntrl", + "digit", + "graph", + "lower", + "print", + "punct", + "space", + "upper", + "word", + "xdigit", + } +) + + +def _is_ascii_decimal(value): + """Return whether a repetition bound contains only ASCII decimal digits.""" + return bool(value) and all("0" <= character <= "9" for character in value) + + +def _counted_repeat_uses_different_semantics(pattern, opening_index): + """Return whether Python accepts a counted repeat that RE2 treats literally.""" + closing_index = pattern.find("}", opening_index + 1) + if closing_index < 0: + return False + bounds = pattern[opening_index + 1 : closing_index] + if "," not in bounds: + return _is_ascii_decimal(bounds) and len(bounds) > 1 and bounds.startswith("0") + if bounds.count(",") != 1: + return False + lower, upper = bounds.split(",", 1) + if (lower and not _is_ascii_decimal(lower)) or (upper and not _is_ascii_decimal(upper)): + return False + return not lower or (len(lower) > 1 and lower.startswith("0")) or (len(upper) > 1 and upper.startswith("0")) + + +def _uses_different_re2_semantics(pattern): + """Return whether Python and RE2 can interpret a legacy construct differently.""" + in_character_class = False + character_class_has_content = False + index = 0 + while index < len(pattern): + character = pattern[index] + if character == "\\" and index + 1 < len(pattern): + shorthand = pattern[index + 1] + if shorthand in _UNICODE_SHORTHANDS or (shorthand in "bB" and not in_character_class): + return True + character_class_has_content = character_class_has_content or in_character_class + index += 2 + continue + if in_character_class: + if pattern.startswith("[:", index): + class_end = pattern.find(":]", index + 2) + if class_end >= 0: + class_name = pattern[index + 2 : class_end].removeprefix("^") + if class_name in _POSIX_CLASSES: + return True + if character == "]" and character_class_has_content: + in_character_class = False + elif character != "^" or character_class_has_content: + character_class_has_content = True + elif character == "[": + in_character_class = True + character_class_has_content = False + elif character == "{" and _counted_repeat_uses_different_semantics(pattern, index): + return True + elif pattern.startswith("(?", index): + flags_end = index + 2 + while flags_end < len(pattern) and pattern[flags_end] in "imsU-": + flags_end += 1 + flag_spec = pattern[index + 2 : flags_end] + enabled_flags = flag_spec.split("-", 1)[0] + if flags_end < len(pattern) and pattern[flags_end] in ":)" and "i" in enabled_flags: + return True + index += 1 + return False + + +def validate_re2_patterns(apps, schema_editor): + """Stop the upgrade before stored patterns can change meaning under RE2.""" + import re + + import re2 + + options = re2.Options() + options.log_errors = False + Rule = apps.get_model("netbox_interface_name_rules", "InterfaceNameRule") + invalid_ids = [] + rules = Rule.objects.using(schema_editor.connection.alias).exclude(module_type_pattern="") + for pk, pattern in rules.values_list("pk", "module_type_pattern").iterator(): + if _uses_different_re2_semantics(pattern): + invalid_ids.append(pk) + continue + try: + re.compile(pattern) + re2.compile(pattern, options=options) + except (OverflowError, re.error, re2.error): + invalid_ids.append(pk) + if invalid_ids: + label = "ID" if len(invalid_ids) == 1 else "IDs" + identifiers = ", ".join(str(pk) for pk in invalid_ids) + raise RuntimeError( + f"Stored patterns require RE2 review for InterfaceNameRule {label}: {identifiers}. " + "Rewrite Python-specific or semantically different shared syntax with explicit RE2 syntax, then retry the migration." + ) + + +class Migration(migrations.Migration): + dependencies = [ + ("netbox_interface_name_rules", "0013_interfacenamerule_breakout_mode_and_more"), + ] + + operations = [ + migrations.RunPython(validate_re2_patterns, migrations.RunPython.noop), + ] diff --git a/netbox_interface_name_rules/models.py b/netbox_interface_name_rules/models.py index ce371da..61b36a1 100644 --- a/netbox_interface_name_rules/models.py +++ b/netbox_interface_name_rules/models.py @@ -11,8 +11,8 @@ from taggit.managers import TaggableManager from .choices import BreakoutModeChoices +from .regex_safety import compile_module_type_pattern -_REDOS_PATTERN = re.compile(r"(\+\*|\*\+|\?\?|\)\s*[\+\*\?]\s*[\+\*\?]|\)\s*\{[^{}]+\}\s*[\+\*\?])") _TEMPLATE_FIELD = re.compile(r"\{([^{}]*)\}") @@ -64,22 +64,6 @@ def _has_unbalanced_braces(template): return depth != 0 -def _validate_module_type_pattern(pattern): - """Compile *pattern* and check for ReDoS-prone constructs. - - Raises ``ValidationError`` targeting ``module_type_pattern`` if the - pattern is syntactically invalid or contains nested quantifiers. - Called from ``InterfaceNameRule.clean()`` to avoid duplicating the same - try/except + ReDoS guard in each branch. - """ - try: - re.compile(pattern) - except re.error as e: - raise ValidationError({"module_type_pattern": f"Invalid regex pattern: {e}"}) - if _REDOS_PATTERN.search(pattern): - raise ValidationError({"module_type_pattern": "Pattern contains potentially unsafe nested quantifiers."}) - - def _validate_breakout_topology(breakout_mode, channel_count, parent_name_template, applies_to_device_interfaces=False): """Check that the mode, the channel count and the parent template describe one topology. @@ -131,7 +115,7 @@ class InterfaceNameRule(NetBoxModel): Module type matching supports two modes: - Exact: FK reference to a specific ModuleType (default) - - Regex: Pattern matched against ModuleType.model via re.fullmatch() + - Regex: RE2 pattern matched against the complete ModuleType.model value Scoping fields (all optional): - parent_module_type: match only when installed inside this module type @@ -153,8 +137,7 @@ class InterfaceNameRule(NetBoxModel): blank=True, default="", verbose_name="Module Type Pattern", - help_text="Regex pattern to match module type model name (e.g. 'QSFP-DD-400G-.*'). " - "Uses Python re.fullmatch() — pattern must match the entire model name.", + help_text="RE2 pattern to match the complete module type model name (e.g. 'QSFP-DD-400G-.*').", ) module_type_is_regex = models.BooleanField( default=False, @@ -256,7 +239,7 @@ def clean(self): raise ValidationError({"module_type": "Module type must be empty for device-level interface rules."}) # module_type_pattern is an optional interface-name filter regex if self.module_type_pattern: - _validate_module_type_pattern(self.module_type_pattern) + compile_module_type_pattern(self.module_type_pattern) # Force regex mode off — module_type_is_regex has no meaning here self.module_type_is_regex = False elif self.module_type_is_regex: @@ -264,7 +247,7 @@ def clean(self): raise ValidationError({"module_type_pattern": "Regex pattern is required when regex mode is enabled."}) if self.module_type: raise ValidationError({"module_type": "Cannot set both module type FK and regex pattern. Choose one."}) - _validate_module_type_pattern(self.module_type_pattern) + compile_module_type_pattern(self.module_type_pattern) else: # Clear any stale pattern so it does not persist when switching modes self.module_type_pattern = "" diff --git a/netbox_interface_name_rules/naming.py b/netbox_interface_name_rules/naming.py new file mode 100644 index 0000000..5471d37 --- /dev/null +++ b/netbox_interface_name_rules/naming.py @@ -0,0 +1,144 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Build naming variables and evaluate interface-name templates.""" + +import ast +import operator +import re + +_BINARY_OPERATORS = { + ast.Add: operator.add, + ast.Sub: operator.sub, + ast.Mult: operator.mul, + ast.FloorDiv: operator.floordiv, +} +_UNARY_OPERATORS = { + ast.UAdd: operator.pos, + ast.USub: operator.neg, +} + + +def _evaluate_arithmetic(node): + """Evaluate one validated integer arithmetic syntax tree without executing code.""" + if isinstance(node, ast.Expression): + return _evaluate_arithmetic(node.body) + if isinstance(node, ast.Constant) and type(node.value) is int: + return node.value + if isinstance(node, ast.BinOp) and type(node.op) in _BINARY_OPERATORS: + return _BINARY_OPERATORS[type(node.op)]( + _evaluate_arithmetic(node.left), + _evaluate_arithmetic(node.right), + ) + if isinstance(node, ast.UnaryOp) and type(node.op) in _UNARY_OPERATORS: + return _UNARY_OPERATORS[type(node.op)](_evaluate_arithmetic(node.operand)) + raise ValueError(f"Unsafe AST node in expression: {type(node).__name__}") + + +def _extract_trailing_digits(value: str) -> str: + r"""Return the trailing digit run of *value* without regex backtracking. + + This O(n) string scan avoids the polynomial backtracking risk from a + trailing-digit regular expression on a long value that ends in a non-digit. + + Returns an empty string when *value* has no trailing digits. + """ + index = len(value) + while index > 0 and value[index - 1].isdigit(): + index -= 1 + return value[index:] + + +def _resolve_bay_position(module_bay): + """Return the raw and numeric positions for *module_bay*. + + A template expression such as ``{module}`` resolves from trailing digits in + the bay name. A missing numeric suffix resolves to zero. + """ + bay_position = module_bay.position or "0" + if bay_position.startswith("{"): + digits = _extract_trailing_digits(module_bay.name) + bay_position = digits if digits else "0" + digits = _extract_trailing_digits(bay_position) + bay_position_num = digits if digits else "0" + return bay_position, bay_position_num + + +def _resolve_slot(module_bay, bay_position_num, parent_bay_position): + """Return the slot value from the module-bay hierarchy. + + A nested bay takes the parent or grandparent position. A bay owned by an + installed module takes that module's bay position. Other bays use their + numeric position. + """ + if module_bay.parent: + parent_bay = module_bay.parent + if parent_bay.parent and hasattr(parent_bay.parent, "installed_module"): + return parent_bay.parent.position or parent_bay_position + return parent_bay_position + if hasattr(module_bay, "module") and module_bay.module: + owner_module = module_bay.module + if hasattr(owner_module, "module_bay") and owner_module.module_bay: + return owner_module.module_bay.position or bay_position_num + return bay_position_num + + +def build_variables(module_bay, device=None): + """Build template variables from a module bay and optional device. + + The result includes slot, bay position, numeric bay position, parent bay + position, and SFP slot. A virtual-chassis position is included only for a + member device that has a position. + + A template that uses ``{vc_position}`` for a non-member device fails during + evaluation because the variable is intentionally absent. Position zero is + retained because it is a valid virtual-chassis position. + """ + bay_position, bay_position_num = _resolve_bay_position(module_bay) + + parent_bay_position = "0" + if module_bay.parent: + parent_bay_position = module_bay.parent.position or "0" + + slot = _resolve_slot(module_bay, bay_position_num, parent_bay_position) + + result = { + "slot": slot, + "bay_position": bay_position, + "bay_position_num": bay_position_num, + "parent_bay_position": parent_bay_position, + "sfp_slot": bay_position_num, + } + if ( + device is not None + and getattr(device, "virtual_chassis_id", None) is not None + and device.vc_position is not None + ): + result["vc_position"] = str(device.vc_position) + return result + + +def evaluate_name_template(template: str, variables: dict) -> str: + """Evaluate a name template with variable substitution and safe arithmetic. + + Variables are substituted before remaining brace-enclosed arithmetic is + evaluated. True division is not allowed. Arithmetic results are converted + to integers so interface names contain whole numbers. + + For example, ``GigabitEthernet{slot}/{8 + {sfp_slot}}`` substitutes the + variables before evaluating the arithmetic expression. + """ + result = template + for key, value in variables.items(): + result = result.replace(f"{{{key}}}", str(value)) + + def _eval_expr(match): + expr = match.group(1).strip() + if not re.match(r"^(?!.*(? +"""Compile stored rule patterns with a bounded regular-expression engine.""" + +import re2 +from django.core.exceptions import ValidationError + +_OPTIONS = re2.Options() +_OPTIONS.log_errors = False + + +def _error_detail(exc): + """Return RE2's parser error as text without its bytes representation.""" + detail = exc.args[0] if exc.args else str(exc) + return detail.decode(errors="replace") if isinstance(detail, bytes) else str(detail) + + +def compile_module_type_pattern(pattern): + """Compile a stored rule pattern with RE2, or raise a field validation error.""" + try: + return re2.compile(pattern, options=_OPTIONS) + except re2.error as exc: + raise ValidationError({"module_type_pattern": f"Invalid RE2 pattern: {_error_detail(exc)}"}) from exc diff --git a/netbox_interface_name_rules/rule_selection.py b/netbox_interface_name_rules/rule_selection.py new file mode 100644 index 0000000..3b7c359 --- /dev/null +++ b/netbox_interface_name_rules/rule_selection.py @@ -0,0 +1,253 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Select an enabled interface-name rule for a module context.""" + +import contextlib +import threading + +from django.core.exceptions import ValidationError +from django.db.models import Aggregate, F, TextField, Value +from django.db.models.functions import Cast, Coalesce, Concat, Length + +from .regex_safety import compile_module_type_pattern + +# Publish each loaded rule set as one new dictionary. Concurrent readers then see +# one complete version rather than a mixture of cache entries from two versions. +_RULE_CACHE = {"version": None, "exact": (), "regex": (), "memo": {}} + +# Bound the number of module and scope contexts retained for one rule-set version. +_MEMO_MAX = 4096 +_MEMO_MISS = object() + +# A pinned batch owns a thread-local rule-set snapshot and a private memo. +_pin = threading.local() + + +@contextlib.contextmanager +def pinned_rule_cache(): + """Pin one enabled-rule snapshot for all selections inside the block. + + The first selection loads and fingerprints the rule set. Later selections in + the same thread skip the fingerprint query. Nested blocks share the snapshot. + The pin is thread-local, and an empty block does not load rules. + """ + depth = getattr(_pin, "depth", 0) + _pin.depth = depth + 1 + if depth == 0: + _pin.primed = False + try: + yield + finally: + _pin.depth -= 1 + if _pin.depth == 0: + _pin.primed = False + for attr in ("exact", "regex", "memo"): + _pin.__dict__.pop(attr, None) + + +def _compile_pattern(pattern): + """Compile a stored pattern once, or return None when RE2 rejects it.""" + try: + return compile_module_type_pattern(pattern) + except ValidationError: + return None + + +# These fields can change either matching or the selected rule's output. The row +# identity prevents compensating edits across two rules from preserving the hash. +_VERSION_COLUMNS = ( + "id", + "module_type_id", + "module_type_is_regex", + "module_type_pattern", + "parent_module_type_id", + "device_type_id", + "platform_id", + "name_template", + "parent_name_template", + "breakout_mode", + "channel_count", + "channel_start", + "applies_to_device_interfaces", +) + + +class _Md5OrderedStringAgg(Aggregate): + """Build ``md5(string_agg(, ORDER BY id))``.""" + + function = "STRING_AGG" + template = "MD5(%(function)s(%(expressions)s ORDER BY id))" + output_field = TextField() + + +def _version_row_signature(): + """Build an unambiguous, length-prefixed signature for one rule row.""" + empty = Value("", output_field=TextField()) + colon = Value(":", output_field=TextField()) + parts = [] + for column in _VERSION_COLUMNS: + cast = Cast(F(column), output_field=TextField()) + value = Coalesce(cast, empty, output_field=TextField()) if column.endswith("_id") else cast + parts.append(Cast(Length(value), output_field=TextField())) + parts.append(colon) + parts.append(value) + return Concat(*parts, output_field=TextField()) + + +_ROW_SIGNATURE = _version_row_signature() + + +def _enabled_rules_version(): + """Return a deterministic content fingerprint of all enabled rules. + + PostgreSQL hashes the matching and output columns in primary-key order. Each + value is length-prefixed, so arbitrary text cannot create field or row boundary + collisions. The empty rule set has a stable empty fingerprint. + """ + from .models import InterfaceNameRule + + return InterfaceNameRule.objects.filter(enabled=True).aggregate( + fingerprint=Coalesce( + _Md5OrderedStringAgg(_ROW_SIGNATURE, Value("", output_field=TextField())), + Value("", output_field=TextField()), + ) + )["fingerprint"] + + +def _get_enabled_rules(): + """Return the exact rules, regex rules, and memo for the current version. + + Exact rules retain model ordering, which reduces to primary-key order for one + module type. Regex rules are compiled once and ordered by decreasing pattern + length, then primary key. A reload publishes one new cache dictionary so a + concurrent reader cannot combine values from two versions. + """ + global _RULE_CACHE + + pinned = getattr(_pin, "depth", 0) > 0 + if pinned and getattr(_pin, "primed", False): + # Return the thread's snapshot. Another thread can replace the shared cache. + return _pin.exact, _pin.regex, _pin.memo + + from .models import InterfaceNameRule + + cache = _RULE_CACHE + version = _enabled_rules_version() + if cache["version"] != version: + rules = list(InterfaceNameRule.objects.filter(enabled=True).order_by("module_type__model", "pk")) + exact = tuple(rule for rule in rules if not rule.module_type_is_regex) + regex_rules = sorted( + (rule for rule in rules if rule.module_type_is_regex), + key=lambda rule: (-len(rule.module_type_pattern or ""), rule.pk), + ) + regex = tuple((_compile_pattern(rule.module_type_pattern), rule) for rule in regex_rules) + cache = {"version": version, "exact": exact, "regex": regex, "memo": {}} + _RULE_CACHE = cache + + if pinned: + # Keep a private memo so another thread cannot clear this batch's entries. + _pin.exact = cache["exact"] + _pin.regex = cache["regex"] + _pin.memo = dict(cache["memo"]) + _pin.primed = True + return _pin.exact, _pin.regex, _pin.memo + + return cache["exact"], cache["regex"], cache["memo"] + + +def _scope_ids(parent_module_type, device_type, platform): + """Map optional scope objects to their foreign-key values.""" + return ( + parent_module_type.pk if parent_module_type is not None else None, + device_type.pk if device_type is not None else None, + platform.pk if platform is not None else None, + ) + + +def _rule_scope_matches(rule, scope_ids): + """Return whether a rule has exactly the requested scope values.""" + parent_module_type_id, device_type_id, platform_id = scope_ids + return ( + rule.parent_module_type_id == parent_module_type_id + and rule.device_type_id == device_type_id + and rule.platform_id == platform_id + ) + + +def _build_candidates(parent_module_type, device_type, platform) -> list: + """Build scope combinations from most specific to least specific.""" + seen: set = set() + candidates = [] + parent_options = [parent_module_type, None] if parent_module_type else [None] + device_options = [device_type, None] if device_type else [None] + platform_options = [platform, None] if platform else [None] + for parent in parent_options: + for device in device_options: + for candidate_platform in platform_options: + candidate = (parent, device, candidate_platform) + if candidate not in seen: + seen.add(candidate) + candidates.append(candidate) + return candidates + + +def _find_exact_match(module_type, candidates, exact_rules=None): + """Return the first enabled exact rule in scope-precedence order.""" + if exact_rules is None: + exact_rules, _, _ = _get_enabled_rules() + + scoped_rules = [rule for rule in exact_rules if rule.module_type_id == module_type.pk] + for candidate in candidates: + scope_ids = _scope_ids(*candidate) + for rule in scoped_rules: + if _rule_scope_matches(rule, scope_ids): + return rule + return None + + +def _find_regex_match(model_name: str, candidates, regex_rules=None): + """Return the first enabled regex rule in scope-precedence order.""" + if regex_rules is None: + _, regex_rules, _ = _get_enabled_rules() + + for candidate in candidates: + scope_ids = _scope_ids(*candidate) + for compiled, rule in regex_rules: + if compiled is not None and _rule_scope_matches(rule, scope_ids) and compiled.fullmatch(model_name): + return rule + return None + + +def find_matching_rule(module_type, parent_module_type, device_type, platform=None): + """Return the most specific enabled rule for a module and scope context. + + Exact module-type rules take priority over regular-expression rules. Within + each tier, parent module type, device type, and platform scopes are tried from + most specific to least specific. + """ + if module_type is None: + return None + + exact_rules, regex_rules, memo = _get_enabled_rules() + signature = ( + module_type.pk, + # Regex matching reads the live model name, so the memo key must include it. + module_type.model, + *_scope_ids(parent_module_type, device_type, platform), + ) + # A single dictionary read cannot race with another thread's memo clear between + # a membership check and a later subscript. + cached = memo.get(signature, _MEMO_MISS) + if cached is not _MEMO_MISS: + return cached + + candidates = _build_candidates(parent_module_type, device_type, platform) + result = _find_exact_match(module_type, candidates, exact_rules) or _find_regex_match( + module_type.model, + candidates, + regex_rules, + ) + if len(memo) >= _MEMO_MAX: + memo.clear() + memo[signature] = result + return result diff --git a/netbox_interface_name_rules/signals.py b/netbox_interface_name_rules/signals.py index 8255850..6a15f45 100644 --- a/netbox_interface_name_rules/signals.py +++ b/netbox_interface_name_rules/signals.py @@ -189,7 +189,7 @@ def _apply_rules_for_device_deferred(device_pk): device_pk: Primary key of the Device to re-apply rules for. """ - from dcim.models import Device, Module + from dcim.models import Device try: device = Device.objects.select_related("virtual_chassis").get(pk=device_pk) @@ -198,35 +198,10 @@ def _apply_rules_for_device_deferred(device_pk): total = 0 - # Re-apply module interface rules - modules = Module.objects.filter(device=device).select_related( - "module_bay", - "module_type", - "device", - "device__device_type", - "device__platform", - "device__virtual_chassis", - ) try: - from .engine import apply_interface_name_rules, pinned_rule_cache - - # Every module on this device matches against the same enabled-rule set, so pin it for the - # loop: the fingerprint is read once when the first lookup primes the cache instead of once - # per module. Safe because the loop only renames interfaces — it never edits rules. - with pinned_rule_cache(): - for module in modules: - module_bay = module.module_bay - if not module_bay: - continue - try: - renamed = apply_interface_name_rules(module, module_bay, force_reapply=True) - total += renamed or 0 - except Exception: - logger.exception( - "Failed to re-apply rules for %s in %s after VC change", - module.module_type, - module_bay.name, - ) + from .engine import reapply_module_rules + + total += reapply_module_rules(device) except Exception: logger.exception("Failed to re-apply module rules for device %s after VC change", device_pk) diff --git a/netbox_interface_name_rules/templates/netbox_interface_name_rules/rule_apply_detail.html b/netbox_interface_name_rules/templates/netbox_interface_name_rules/rule_apply_detail.html index a510628..5446e5a 100644 --- a/netbox_interface_name_rules/templates/netbox_interface_name_rules/rule_apply_detail.html +++ b/netbox_interface_name_rules/templates/netbox_interface_name_rules/rule_apply_detail.html @@ -259,9 +259,9 @@
Convert flat {% if verdict.convertible %} - Convertible + {{ verdict.status_label }} {% else %} - Blocked + {{ verdict.status_label }} {% endif %} @@ -280,16 +280,20 @@
Convert flat {% if can_apply %} -
- - -
+ {% if conversions_available or conversions_have_more %} +
+ {% if conversions_available %} + + {% endif %} + +
+ {% endif %} {% else %}
You do not have permission to apply interface changes. diff --git a/netbox_interface_name_rules/tests/out_of_band.py b/netbox_interface_name_rules/tests/out_of_band.py new file mode 100644 index 0000000..1d005c8 --- /dev/null +++ b/netbox_interface_name_rules/tests/out_of_band.py @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Write interface rows the way an actor outside this plugin would.""" + + +def rename_out_of_band(interface, name): + """Rename *interface* through a real model save and return it. + + This is what an operator's edit, an import or another plugin does to a row this plugin later + finds changed. The plugin registers receivers on Module and Device only, never on Interface, so + a plain save runs NetBox's own machinery and none of ours: it is already the out-of-band write. + A queryset update would skip NetBox's machinery too, and would need a database that permits raw + bulk writes to interface rows. + """ + interface.name = name + interface.save() + return interface diff --git a/netbox_interface_name_rules/tests/signal_performance.py b/netbox_interface_name_rules/tests/signal_performance.py new file mode 100644 index 0000000..b0e5847 --- /dev/null +++ b/netbox_interface_name_rules/tests/signal_performance.py @@ -0,0 +1,1196 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Record the automatic naming signal-path performance baseline. + +This module is intentionally outside Django's default ``test*.py`` discovery. Run it by its full +module label when a before/after implementation comparison is needed. The runner uses the real +NetBox models, PostgreSQL connection, Django signals, and committed callbacks. +""" + +import contextlib +import gc +import hashlib +import json +import math +import os +import platform +import re +import statistics +import subprocess +import time +from collections import Counter +from collections.abc import Callable +from dataclasses import dataclass, replace +from datetime import UTC, datetime +from pathlib import Path +from typing import Any +from unittest.mock import patch + +import django +from dcim.choices import InterfaceTypeChoices +from dcim.models import ( + Device, + DeviceRole, + DeviceType, + Interface, + InterfaceTemplate, + Manufacturer, + Module, + ModuleBay, + ModuleBayTemplate, + ModuleType, + Site, + VirtualChassis, +) +from django.apps import apps +from django.conf import settings +from django.db import DatabaseError, connection, transaction +from django.test import TransactionTestCase +from netbox.constants import RQ_QUEUE_DEFAULT +from utilities.rqworker import any_workers_for_queue + +from netbox_interface_name_rules import __version__ as plugin_version +from netbox_interface_name_rules.choices import BreakoutModeChoices +from netbox_interface_name_rules.engine import supports_channelization, supports_vc_position_token +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.signals import _apply_rules_deferred, _apply_rules_for_device_deferred +from performance.artifact import SCHEMA_VERSION, validate_artifact + +_OUTPUT_VARIABLE = "INTERFACE_FAMILY_PERFORMANCE_OUTPUT" +_SAMPLE_VARIABLE = "INTERFACE_FAMILY_PERFORMANCE_SAMPLES" +_WARMUP_VARIABLE = "INTERFACE_FAMILY_PERFORMANCE_WARMUPS" +_PLUGIN_REVISION_VARIABLE = "INTERFACE_FAMILY_PERFORMANCE_SOURCE_REVISION" +_NETBOX_REVISION_VARIABLE = "NETBOX_PERFORMANCE_SOURCE_REVISION" +_KIND_VARIABLE = "INTERFACE_FAMILY_PERFORMANCE_KIND" +_DEFAULT_KIND = "existing_implementation" +_DEFAULT_SAMPLES = 15 +_DEFAULT_WARMUPS = 3 + +_CHANNEL_TYPE = getattr(InterfaceTypeChoices, "TYPE_CHANNEL", "channel") +_PARENT_TYPE = InterfaceTypeChoices.TYPE_40GE_QSFP_PLUS +_PLAIN_TYPE = InterfaceTypeChoices.TYPE_10GE_SFP_PLUS + +_SPACE_RE = re.compile(r"\s+") +_STRING_LITERAL_RE = re.compile(r"(?i)(?:e|u&)?'(?:''|[^'])*'") +_DOLLAR_LITERAL_RE = re.compile( + r"\$\$.*?\$\$|\$(?P[A-Za-z_][A-Za-z_0-9]*)\$.*?\$(?P=tag)\$", + re.DOTALL, +) +_NUMBER_RE = re.compile(r"(?[\[{].*)", re.DOTALL) + +_PLAN_TIME_KEYS = { + "Actual Startup Time", + "Actual Total Time", + "Execution Time", + "Planning Time", +} +_PLAN_PRIVATE_KEYS = {"Query Parameters", "Query Text"} +_PLAN_EXPRESSION_KEYS = { + "Conflict Filter", + "Filter", + "Function Call", + "Hash Cond", + "Hash Key", + "Index Cond", + "Join Filter", + "Merge Cond", + "Output", + "Recheck Cond", + "Sort Key", + "TID Cond", +} +_WORK_COUNTER_KEYS = ( + "Shared Hit Blocks", + "Shared Read Blocks", + "Shared Dirtied Blocks", + "Shared Written Blocks", + "Local Hit Blocks", + "Local Read Blocks", + "Local Dirtied Blocks", + "Local Written Blocks", + "Temp Read Blocks", + "Temp Written Blocks", + "WAL Records", + "WAL FPI", + "WAL Bytes", +) +_PLAN_IDENTITY_KEYS = { + "Alias", + "Async Capable", + "Cache Key", + "Cache Mode", + "Command", + "Conflict Arbiter Indexes", + "Conflict Filter", + "Conflict Resolution", + "CTE Name", + "Custom Plan Provider", + "Disabled", + "Filter", + "Function Call", + "Function Name", + "Group Key", + "Group Keys", + "Grouping Sets", + "Hash Cond", + "Hash Key", + "Hash Keys", + "Index Cond", + "Index Name", + "Inner Unique", + "Join Filter", + "Join Type", + "Merge Cond", + "Node Type", + "One-Time Filter", + "Operation", + "Output", + "Parallel Aware", + "Parent Relationship", + "Partial Mode", + "Plan", + "Plan Rows", + "Plan Width", + "Plans", + "Presorted Key", + "Recheck Cond", + "Relation Name", + "Relations", + "Remote SQL", + "Repeatable Seed", + "Sampling Method", + "Sampling Parameters", + "Scan Direction", + "Schema", + "SetOp Command", + "Single Copy", + "Sort Key", + "Startup Cost", + "Strategy", + "Subplan Name", + "Table Function Call", + "Target Tables", + "TID Cond", + "Total Cost", + "Workers Planned", +} +_PLANNER_SETTINGS = ( + "block_size", + "cpu_index_tuple_cost", + "cpu_operator_cost", + "cpu_tuple_cost", + "default_statistics_target", + "effective_cache_size", + "effective_io_concurrency", + "jit", + "max_parallel_workers_per_gather", + "plan_cache_mode", + "random_page_cost", + "seq_page_cost", + "shared_buffers", + "work_mem", +) +_PROFILE_MODELS = ( + Device, + DeviceRole, + DeviceType, + Interface, + InterfaceTemplate, + Manufacturer, + Module, + ModuleBay, + ModuleBayTemplate, + ModuleType, + Site, + VirtualChassis, + InterfaceNameRule, +) + + +@dataclass(frozen=True) +class _PreparedScenario: + """One prepared fixture and the operation measured against it.""" + + operation: Callable[[], None] + verify: Callable[[], dict[str, Any]] + cleanup: Callable[[], None] + fixture: dict[str, int] + work_units: int + + +@dataclass(frozen=True) +class _Scenario: + """A stable scenario definition used by both measurement passes.""" + + name: str + description: str + layer: str + prepare: Callable[[], _PreparedScenario] + + +def _positive_integer_from_environment(name: str, default: int, *, allow_zero: bool = False) -> int: + """Read an integer run setting and reject values that cannot produce useful evidence.""" + raw = os.environ.get(name, str(default)) + try: + value = int(raw) + except ValueError as error: + raise AssertionError(f"{name} must be an integer, got {raw!r}.") from error + minimum = 0 if allow_zero else 1 + if value < minimum: + raise AssertionError(f"{name} must be at least {minimum}, got {value}.") + return value + + +def _output_path() -> Path: + """Return the requested artifact path or fail before the expensive measurements start.""" + raw = os.environ.get(_OUTPUT_VARIABLE) + if not raw: + raise AssertionError(f"Set {_OUTPUT_VARIABLE} to an absolute .json artifact path.") + output = Path(raw) + if not output.is_absolute() or output.suffix != ".json": + raise AssertionError(f"{_OUTPUT_VARIABLE} must be an absolute path ending in .json.") + output.parent.mkdir(parents=True, exist_ok=True) + return output + + +def _normalize_sql(sql: str) -> str: + """Remove values and formatting differences from a SQL statement.""" + normalized = _DOLLAR_LITERAL_RE.sub("'?'", str(sql)) + normalized = _STRING_LITERAL_RE.sub("'?'", normalized) + normalized = _SAVEPOINT_RE.sub("s?_x?", normalized) + normalized = _DJANGO_CURSOR_RE.sub(r"_django_curs_?_\1_?", normalized) + normalized = _NUMBER_RE.sub("?", normalized) + return _SPACE_RE.sub(" ", normalized).strip() + + +def _fingerprint(value: str) -> str: + """Return a stable identifier for normalized evidence.""" + return hashlib.sha256(value.encode()).hexdigest() + + +def _percentile(samples: list[int], percentile: float) -> float: + """Return an interpolated percentile without a benchmark dependency.""" + ordered = sorted(samples) + if len(ordered) == 1: + return float(ordered[0]) + rank = (len(ordered) - 1) * percentile + lower = math.floor(rank) + upper = math.ceil(rank) + if lower == upper: + return float(ordered[lower]) + fraction = rank - lower + return ordered[lower] + (ordered[upper] - ordered[lower]) * fraction + + +def _summary(samples: list[int]) -> dict[str, Any]: + """Return raw nanoseconds plus median and p95 values in milliseconds.""" + return { + "samples_ns": samples, + "median_ms": statistics.median(samples) / 1_000_000, + "p95_ms": _percentile(samples, 0.95) / 1_000_000, + } + + +def _optional_summary(samples: list[int]) -> dict[str, Any]: + """Summarize optional warm-up samples while allowing a zero-warm-up smoke run.""" + if not samples: + return {"samples_ns": []} + return _summary(samples) + + +def _sanitize_expression(value: Any) -> Any: + """Remove literal values from one plan expression or expression list.""" + if isinstance(value, str): + return _normalize_sql(value) + if isinstance(value, list): + return [_sanitize_expression(child) for child in value] + return value + + +def _sanitize_plan(value: Any) -> Any: + """Remove query values and node timing from an auto_explain document.""" + if isinstance(value, dict): + sanitized = {} + for key, child in value.items(): + if key in _PLAN_PRIVATE_KEYS or key in _PLAN_TIME_KEYS or key.endswith(" I/O Time"): + continue + if key in _PLAN_EXPRESSION_KEYS or key.endswith((" Cond", " Filter", " Key")): + sanitized[key] = _sanitize_expression(child) + else: + sanitized[key] = _sanitize_plan(child) + return sanitized + if isinstance(value, list): + return [_sanitize_plan(child) for child in value] + return value + + +def _plan_envelope(document: Any) -> dict[str, Any]: + """Return the auto_explain envelope from either accepted JSON shape.""" + if isinstance(document, list) and len(document) == 1 and isinstance(document[0], dict): + return document[0] + if isinstance(document, dict): + return document + raise AssertionError(f"auto_explain returned an unexpected JSON document: {type(document).__name__}.") + + +def _parse_notice(message: str) -> tuple[str, dict[str, Any]] | None: + """Parse one JSON auto_explain notice and discard all original query values.""" + match = _AUTO_EXPLAIN_PLAN_RE.search(message) + if not match: + return None + envelope = _plan_envelope(json.loads(match.group("document"))) + normalized_sql = _normalize_sql(str(envelope.get("Query Text", "unknown statement"))) + return normalized_sql, _sanitize_plan(envelope) + + +def _plan_nodes(plan: dict[str, Any]) -> list[dict[str, Any]]: + """Return every plan node from a sanitized EXPLAIN envelope.""" + root = plan.get("Plan") + if not isinstance(root, dict): + return [] + pending = [root] + nodes = [] + while pending: + node = pending.pop() + nodes.append(node) + pending.extend(child for child in node.get("Plans", ()) if isinstance(child, dict)) + return nodes + + +def _plan_identity_shape(value: Any) -> Any: + """Retain only structural and planner fields that define the chosen plan.""" + if isinstance(value, dict): + return {key: _plan_identity_shape(child) for key, child in value.items() if key in _PLAN_IDENTITY_KEYS} + if isinstance(value, list): + return [_plan_identity_shape(child) for child in value] + return value + + +def _root_work(plan: dict[str, Any]) -> dict[str, float | int]: + """Extract additive work measures from the root plan node.""" + root = plan.get("Plan", {}) + loops = int(root.get("Actual Loops", 0) or 0) + rows = int(root.get("Actual Rows", 0) or 0) + work: dict[str, float | int] = { + "planner_total_cost": float(root.get("Total Cost", 0) or 0), + "planner_rows": int(root.get("Plan Rows", 0) or 0), + "actual_loops": loops, + "actual_rows_times_loops": rows * loops, + } + for key in _WORK_COUNTER_KEYS: + work[key.lower().replace(" ", "_")] = int(root.get(key, 0) or 0) + return work + + +def _add_work(total: dict[str, float | int], work: dict[str, float | int]) -> None: + """Add one plan execution's work to an aggregate.""" + for key, value in work.items(): + total[key] = total.get(key, 0) + value + + +def _group_plans(notices: list[str]) -> list[dict[str, Any]]: + """Group identical sanitized plans and retain one full representative plan.""" + grouped: dict[str, dict[str, Any]] = {} + for message in notices: + parsed = _parse_notice(message) + if parsed is None: + continue + normalized_sql, plan = parsed + serialized = json.dumps(_plan_identity_shape(plan), sort_keys=True, separators=(",", ":")) + identity = _fingerprint(f"{normalized_sql}\0{serialized}") + entry = grouped.setdefault( + identity, + { + "fingerprint": identity, + "statement_fingerprint": _fingerprint(normalized_sql), + "normalized_sql": normalized_sql, + "calls": 0, + "aggregate": {}, + "indexes": set(), + "node_types": Counter(), + "plan": plan, + }, + ) + entry["calls"] += 1 + _add_work(entry["aggregate"], _root_work(plan)) + for node in _plan_nodes(plan): + if index_name := node.get("Index Name"): + entry["indexes"].add(index_name) + if node_type := node.get("Node Type"): + entry["node_types"][node_type] += 1 + plans = [] + for entry in grouped.values(): + entry["indexes"] = sorted(entry["indexes"]) + entry["node_types"] = dict(sorted(entry["node_types"].items())) + plans.append(entry) + return sorted(plans, key=lambda item: item["fingerprint"]) + + +class _StatementRecorder: + """Django execute wrapper that records statement shapes without parameter values.""" + + def __init__(self) -> None: + self._statements: dict[str, dict[str, Any]] = {} + + def __call__(self, execute, sql, params, many, context): + """Record one completed statement and return its normal result.""" + result = execute(sql, params, many, context) + normalized = _normalize_sql(str(sql)) + identity = _fingerprint(normalized) + entry = self._statements.setdefault( + identity, + { + "fingerprint": identity, + "normalized_sql": normalized, + "calls": 0, + "rows_affected": 0, + }, + ) + entry["calls"] += 1 + row_count = context["cursor"].rowcount + if isinstance(row_count, int) and row_count > 0: + entry["rows_affected"] += row_count + return result + + def result(self) -> list[dict[str, Any]]: + """Return statements in stable fingerprint order.""" + return sorted(self._statements.values(), key=lambda item: item["fingerprint"]) + + +@contextlib.contextmanager +def _auto_explain_notices(): + """Capture JSON plans from the active PostgreSQL session or fail explicitly.""" + if connection.vendor != "postgresql": + raise AssertionError(f"Performance profiling requires PostgreSQL, got {connection.vendor!r}.") + connection.ensure_connection() + raw_connection = connection.connection + if not hasattr(raw_connection, "add_notice_handler"): + raise AssertionError("Performance profiling requires psycopg 3 notice handlers.") + + notices: list[str] = [] + + def handle_notice(diagnostic) -> None: + notices.append(str(diagnostic.message_primary)) + + configured = False + raw_connection.add_notice_handler(handle_notice) + try: + try: + with transaction.atomic(), connection.cursor() as cursor: + cursor.execute("LOAD 'auto_explain'") + cursor.execute("SET client_min_messages = notice") + cursor.execute("SET auto_explain.log_min_duration = 0") + cursor.execute("SET auto_explain.log_analyze = on") + cursor.execute("SET auto_explain.log_timing = off") + cursor.execute("SET auto_explain.log_buffers = on") + cursor.execute("SET auto_explain.log_wal = on") + cursor.execute("SET auto_explain.log_nested_statements = on") + cursor.execute("SET auto_explain.log_triggers = on") + cursor.execute("SET auto_explain.log_verbose = off") + cursor.execute("SET auto_explain.log_settings = off") + cursor.execute("SET auto_explain.log_format = json") + cursor.execute("SET auto_explain.log_level = notice") + cursor.execute("SET auto_explain.log_parameter_max_length = 0") + cursor.execute("SET auto_explain.sample_rate = 1") + except DatabaseError as error: + raise AssertionError("PostgreSQL auto_explain profiling is unavailable.") from error + configured = True + yield notices + finally: + if configured: + with contextlib.suppress(DatabaseError), connection.cursor() as cursor: + cursor.execute("SET auto_explain.log_min_duration = -1") + raw_connection.remove_notice_handler(handle_notice) + + +def _git_revision(path: Path) -> str | None: + """Return the source revision for a checkout, if the checkout metadata is present.""" + result = subprocess.run( + ["git", "-C", str(path), "rev-parse", "HEAD"], + check=False, + capture_output=True, + text=True, + ) + return result.stdout.strip() if result.returncode == 0 else None + + +def _cpu_model() -> str: + """Return a stable processor description without recording the host identity.""" + cpuinfo = Path("/proc/cpuinfo") + if cpuinfo.exists(): + for line in cpuinfo.read_text().splitlines(): + if line.lower().startswith("model name"): + return line.partition(":")[2].strip() + return platform.processor() or "unknown" + + +def _planner_environment() -> dict[str, str]: + """Read the PostgreSQL revision and planner settings used by the run.""" + with connection.cursor() as cursor: + cursor.execute("SHOW server_version") + server_version = cursor.fetchone()[0] + cursor.execute("SHOW server_version_num") + server_version_num = cursor.fetchone()[0] + planner = {} + for setting in _PLANNER_SETTINGS: + cursor.execute(f"SHOW {setting}") # noqa: S608 - names come only from the fixed tuple above + planner[setting] = cursor.fetchone()[0] + return { + "server_version": server_version, + "server_version_num": server_version_num, + "planner_settings": planner, + } + + +def _host_load() -> dict[str, float]: + """Return the 1, 5 and 15 minute run-queue averages, which decide whether timing is evidence.""" + one, five, fifteen = os.getloadavg() + return {"one_minute": one, "five_minute": five, "fifteen_minute": fifteen} + + +def _assert_isolated_task_queue() -> None: + """Refuse a run whose search-cache writes can switch between queued and inline paths.""" + if any_workers_for_queue(RQ_QUEUE_DEFAULT): + raise AssertionError( + "The performance run requires a task Redis database with no live RQ workers. " + "Use an isolated Redis database so NetBox search-cache writes stay inline." + ) + + +def _source_environment() -> dict[str, Any]: + """Return source and machine revisions without an environment-specific host name.""" + plugin_root = Path(__file__).resolve().parents[2] + plugin_revision = os.environ.get(_PLUGIN_REVISION_VARIABLE) or _git_revision(plugin_root) + netbox_revision = os.environ.get(_NETBOX_REVISION_VARIABLE) + if not netbox_revision: + netbox_revision = _git_revision(Path(settings.BASE_DIR).resolve().parent) + if not plugin_revision: + raise AssertionError(f"Set {_PLUGIN_REVISION_VARIABLE} when the plugin Git checkout is not readable.") + if not netbox_revision: + raise AssertionError(f"Set {_NETBOX_REVISION_VARIABLE} when NetBox does not run from a Git checkout.") + return { + "plugin_revision": plugin_revision, + "plugin_version": plugin_version, + "netbox_revision": netbox_revision, + "netbox_version": settings.RELEASE.full_version, + "django_version": django.get_version(), + "python_implementation": platform.python_implementation(), + "python_version": platform.python_version(), + "operating_system": platform.system(), + "operating_system_release": platform.release(), + "machine_architecture": platform.machine(), + "cpu_model": _cpu_model(), + } + + +def _work_totals(statements: list[dict[str, Any]], plans: list[dict[str, Any]], work_units: int) -> dict[str, Any]: + """Summarize database work and derive scale-normalized values.""" + totals: dict[str, float | int] = {"statement_calls": sum(item["calls"] for item in statements)} + totals["planned_statement_calls"] = sum(item["calls"] for item in plans) + for plan in plans: + _add_work(totals, plan["aggregate"]) + return { + **totals, + "work_units": work_units, + "statement_calls_per_work_unit": totals["statement_calls"] / work_units, + "planner_total_cost_per_work_unit": totals.get("planner_total_cost", 0) / work_units, + "actual_rows_per_work_unit": totals.get("actual_rows_times_loops", 0) / work_units, + "shared_hit_blocks_per_work_unit": totals.get("shared_hit_blocks", 0) / work_units, + } + + +def _load_sentence(artifact: dict[str, Any]) -> str: + """Report the run-queue length this run competed with, so a reader can judge its timing.""" + load = artifact["environment"].get("host_load") + if not load: + return "Host load during the run was not recorded, so treat the machine time as unverified." + return ( + f"Host load averaged {load['started']['one_minute']:.2f} over the minute before the run and " + f"{load['finished']['one_minute']:.2f} over the minute it ended. Machine time is evidence only " + "when both runs were taken under a comparable load." + ) + + +def _timing_sentence(artifact: dict[str, Any]) -> str: + """Say whether this run measured machine time at all.""" + if artifact["configuration"]["samples"]: + return "This run measured machine time. Statement counts are reproducible and carry the verdict." + return ( + "This run measured database work, including SQL calls, planner cost, and shared hits. It took no " + "machine-time samples, so every timing column reads `not measured` rather than reporting a number " + "nobody should quote." + ) + + +def _markdown_summary(artifact: dict[str, Any]) -> str: + """Render the comparison fields that a reviewer usually needs first.""" + kind = artifact.get("baseline_kind", _DEFAULT_KIND).replace("_", " ") + lines = [ + f"# Automatic naming performance: {kind}", + "", + "This file is generated by `netbox_interface_name_rules.tests.signal_performance`.", + "Machine-time values are evidence for a same-hardware before/after comparison. They are not CI limits.", + "", + _timing_sentence(artifact), + "", + _load_sentence(artifact), + "", + "| Scenario | Layer | SQL calls | Planner cost | Shared hits | Wall median (ms) | Wall p95 (ms) | CPU median (ms) |", + "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |", + ] + for scenario in artifact["scenarios"]: + totals = scenario["database"]["totals"] + machine_time = scenario["machine_time"] + if machine_time is None: + timing = "not measured | not measured | not measured" + else: + wall = machine_time["wall"] + cpu = machine_time["process_cpu"] + timing = f"{wall['median_ms']:.3f} | {wall['p95_ms']:.3f} | {cpu['median_ms']:.3f}" + lines.append( + f"| `{scenario['name']}` | {scenario['layer']} | {totals['statement_calls']} | " + f"{totals.get('planner_total_cost', 0):.3f} | {totals.get('shared_hit_blocks', 0)} | {timing} |" + ) + lines.extend( + [ + "", + f"Plugin revision: `{artifact['environment']['plugin_revision']}`", + "", + f"NetBox revision: `{artifact['environment']['netbox_revision']}`", + "", + f"PostgreSQL: `{artifact['environment']['postgresql']['server_version']}`", + "", + f"Samples per scenario: `{artifact['configuration']['samples']}`", + "", + ] + ) + return "\n".join(lines) + + +class SignalPathPerformanceTest(TransactionTestCase): + """Generate retained evidence for the existing automatic naming implementation.""" + + available_apps = tuple(app_config.name for app_config in apps.get_app_configs()) + maxDiff = None + + def _build_device(self, prefix: str, bay_positions=(), **device_kwargs): + """Create a device and its module bays.""" + slug = prefix.lower() + manufacturer = Manufacturer.objects.create(name=f"{prefix}Mfg", slug=f"{slug}-mfg") + device_type = DeviceType.objects.create( + manufacturer=manufacturer, + model=f"{prefix}-Device", + slug=f"{slug}-device", + ) + for position in bay_positions: + ModuleBayTemplate.objects.create(device_type=device_type, name=f"Bay {position}", position=str(position)) + role = DeviceRole.objects.create(name=f"{prefix}Role", slug=f"{slug}-role") + site = Site.objects.create(name=f"{prefix}Site", slug=f"{slug}-site") + device = Device.objects.create( + name=f"{slug}-device", + device_type=device_type, + role=role, + site=site, + **device_kwargs, + ) + return manufacturer, device + + @staticmethod + def _plain_module_type(manufacturer: Manufacturer, model: str, name: str = "{module}") -> ModuleType: + """Create one module type with one plain interface template.""" + module_type = ModuleType.objects.create(manufacturer=manufacturer, model=model, part_number=model) + InterfaceTemplate.objects.create(module_type=module_type, name=name, type=_PLAIN_TYPE) + return module_type + + @staticmethod + def _channelized_module_type(manufacturer: Manufacturer, model: str) -> ModuleType: + """Create one native parent plus four channel interface templates.""" + module_type = ModuleType.objects.create(manufacturer=manufacturer, model=model, part_number=model) + parent = InterfaceTemplate.objects.create( + module_type=module_type, + name="{module}", + type=_PARENT_TYPE, + channels=4, + ) + for channel_id in range(1, 5): + InterfaceTemplate.objects.create( + module_type=module_type, + name=f"{{module}}:{channel_id}", + type=_CHANNEL_TYPE, + parent=parent, + channel_id=channel_id, + ) + return module_type + + @staticmethod + def _fixture(module_type: ModuleType, device: Device) -> dict[str, int]: + """Record the exact scenario sizes before the measured operation.""" + return { + "devices": 1, + "module_bays": ModuleBay.objects.filter(device=device).count(), + "modules_before": Module.objects.filter(device=device).count(), + "interfaces_before": Interface.objects.filter(device=device).count(), + "interface_templates": InterfaceTemplate.objects.filter(module_type=module_type).count(), + "enabled_rules": InterfaceNameRule.objects.filter(enabled=True).count(), + } + + @staticmethod + def _cleanup_fixture(prefix: str) -> None: + """Delete one committed scenario fixture without changing shared reference data.""" + manufacturer_name = f"{prefix}Mfg" + Device.objects.filter(name=f"{prefix.lower()}-device").delete() + ModuleType.objects.filter(manufacturer__name=manufacturer_name).delete() + DeviceType.objects.filter(manufacturer__name=manufacturer_name).delete() + Manufacturer.objects.filter(name=manufacturer_name).delete() + DeviceRole.objects.filter(name=f"{prefix}Role").delete() + Site.objects.filter(name=f"{prefix}Site").delete() + VirtualChassis.objects.filter(name=f"{prefix}-VC").delete() + + def _module_operation(self, prefix, device, module_type, direct, expected_names) -> _PreparedScenario: + """Prepare either a complete install path or its deferred callback only.""" + bay = ModuleBay.objects.get(device=device, position="3") + holder: dict[str, Module] = {} + if direct: + holder["module"] = Module.objects.create(device=device, module_bay=bay, module_type=module_type) + + def operation(): + _apply_rules_deferred(holder["module"].pk, bay.pk) + + else: + + def operation(): + with transaction.atomic(): + holder["module"] = Module.objects.create( + device=device, + module_bay=bay, + module_type=module_type, + ) + + def verify(): + names = sorted(Interface.objects.filter(module=holder["module"]).values_list("name", flat=True)) + self.assertEqual(names, expected_names) + return {"interface_names": names, "interfaces_after": len(names)} + + return _PreparedScenario( + operation=operation, + verify=verify, + cleanup=lambda: self._cleanup_fixture(prefix), + fixture=self._fixture(module_type, device), + work_units=len(expected_names), + ) + + def _prepare_module(self, prefix: str, kind: str, direct: bool) -> _PreparedScenario: + """Build one module-install scenario.""" + manufacturer, device = self._build_device(prefix, ("3",)) + if kind in {"existing_family", "reconciliation"}: + module_type = self._channelized_module_type(manufacturer, f"{prefix}-Module") + else: + module_type = self._plain_module_type(manufacturer, f"{prefix}-Module") + + rule_options: dict[str, Any] + if kind == "no_matching_rule": + other_type = ModuleType.objects.create( + manufacturer=manufacturer, + model=f"{prefix}-Other", + part_number=f"{prefix}-Other", + ) + rule_module_type = other_type + rule_options = {"name_template": "unused-{bay_position}"} + expected = ["3"] + elif kind == "plain_rename": + rule_module_type = module_type + rule_options = {"name_template": "et-0/0/{bay_position}"} + expected = ["et-0/0/3"] + elif kind == "structural_creation": + rule_module_type = module_type + rule_options = { + "name_template": "xe-0/0/{bay_position}:{channel}", + "parent_name_template": "et-0/0/{bay_position}", + "breakout_mode": BreakoutModeChoices.CHANNELIZED, + "channel_count": 4, + "channel_start": 0, + } + expected = ["et-0/0/3", "xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"] + elif kind == "existing_family": + rule_module_type = module_type + rule_options = {"name_template": "et-0/0/{bay_position}"} + expected = ["et-0/0/3", "et-0/0/3:1", "et-0/0/3:2", "et-0/0/3:3", "et-0/0/3:4"] + elif kind == "reconciliation": + rule_module_type = module_type + rule_options = { + "name_template": "{base}:{channel}", + "parent_name_template": "et-0/0/{bay_position}", + "breakout_mode": BreakoutModeChoices.CHANNELIZED, + "channel_count": 4, + "channel_start": 1, + } + expected = ["3:1", "3:2", "3:3", "3:4", "et-0/0/3"] + else: + raise AssertionError(f"Unknown module scenario {kind!r}.") + prepared = self._module_operation(prefix, device, module_type, direct, expected) + InterfaceNameRule.objects.create(module_type=rule_module_type, **rule_options) + return replace(prepared, fixture=self._fixture(module_type, device)) + + def _prepare_vc(self, prefix: str, module_count: int, direct: bool) -> _PreparedScenario: + """Build a VC reapply scenario with one or eight installed modules.""" + virtual_chassis = VirtualChassis.objects.create(name=f"{prefix}-VC") + manufacturer, device = self._build_device( + prefix, + tuple(str(position) for position in range(1, module_count + 1)), + virtual_chassis=virtual_chassis, + vc_position=1, + ) + module_type = self._plain_module_type( + manufacturer, + f"{prefix}-Module", + name="xe-{vc_position:0}/0/{module}", + ) + for position in range(1, module_count + 1): + bay = ModuleBay.objects.get(device=device, position=str(position)) + Module.objects.create(device=device, module_bay=bay, module_type=module_type) + InterfaceNameRule.objects.create( + module_type=module_type, + name_template="et-{vc_position}/0/{bay_position}", + ) + + if direct: + Device.objects.filter(pk=device.pk).update(vc_position=2) + + def operation(): + _apply_rules_for_device_deferred(device.pk) + + else: + + def operation(): + device.vc_position = 2 + with transaction.atomic(): + device.save() + + expected = [f"et-2/0/{position}" for position in range(1, module_count + 1)] + + def verify(): + names = sorted(Interface.objects.filter(device=device).values_list("name", flat=True)) + self.assertEqual(names, expected) + return {"interface_names": names, "interfaces_after": len(names)} + + return _PreparedScenario( + operation=operation, + verify=verify, + cleanup=lambda: self._cleanup_fixture(prefix), + fixture=self._fixture(module_type, device), + work_units=module_count, + ) + + def _scenarios(self) -> list[_Scenario]: + """Return the stable scenario matrix in report order.""" + scenarios = [] + module_kinds = ( + ("no_matching_rule", "No matching rule"), + ("plain_rename", "Plain interface rename"), + ("structural_creation", "Four-channel structural creation"), + ("existing_family", "Existing parent plus four channels rename"), + ("reconciliation", "Deferred channel reconciliation"), + ) + for kind, description in module_kinds: + for direct in (False, True): + layer = "direct_callback" if direct else "complete_model_save" + name = f"module.{layer}.{kind}" + prefix = "Perf" + _fingerprint(name)[:8] + scenarios.append( + _Scenario( + name=name, + description=description, + layer=layer, + prepare=lambda prefix=prefix, kind=kind, direct=direct: self._prepare_module( + prefix, kind, direct + ), + ) + ) + for module_count in (1, 8): + for direct in (False, True): + layer = "direct_callback" if direct else "complete_model_save" + name = f"vc.{layer}.reapply_{module_count}" + prefix = "Perf" + _fingerprint(name)[:8] + scenarios.append( + _Scenario( + name=name, + description=f"Virtual chassis reapply across {module_count} module(s)", + layer=layer, + prepare=lambda prefix=prefix, module_count=module_count, direct=direct: self._prepare_vc( + prefix, module_count, direct + ), + ) + ) + return scenarios + + def test_direct_vc_fixture_has_target_position(self): + """Keep direct callback setup outside the measured operation.""" + prefix = "PerfDirectVcFixture" + self._prepare_vc(prefix, module_count=1, direct=True) + + device = Device.objects.get(name=f"{prefix.lower()}-device") + self.assertEqual(device.vc_position, 2) + + def test_complete_model_save_callback_runs_after_commit(self): + """Measure the deferred callback outside the model-save transaction.""" + prepared = self._prepare_module("PerfCommittedCallback", "plain_rename", direct=False) + atomic_states = [] + + def record_atomic_state(execute, sql, params, many, context): + result = execute(sql, params, many, context) + atomic_states.append(connection.in_atomic_block) + return result + + with connection.execute_wrapper(record_atomic_state): + prepared.operation() + + self.assertIn(False, atomic_states) + + def test_direct_module_fixture_includes_the_measured_rule(self): + """Record fixture counts after direct-callback setup is complete.""" + prepared = self._prepare_module("PerfDirectFixture", "plain_rename", direct=True) + + self.assertEqual(prepared.fixture["enabled_rules"], 1) + + def test_unmeasured_summary_names_the_recorded_database_work(self): + """Do not describe a planner and buffer profile as statement counts alone.""" + artifact = {"configuration": {"samples": 0}} + + self.assertEqual( + _timing_sentence(artifact), + "This run measured database work, including SQL calls, planner cost, and shared hits. It took no " + "machine-time samples, so every timing column reads `not measured` rather than reporting a number " + "nobody should quote.", + ) + + @patch("netbox_interface_name_rules.tests.signal_performance.any_workers_for_queue", return_value=True) + def test_a_shared_task_queue_is_refused(self, _workers): + """A live worker makes identical saves alternate between queued and inline search indexing.""" + with self.assertRaisesRegex(AssertionError, "isolated Redis database"): + _assert_isolated_task_queue() + + def test_sql_normalization_scrubs_dollar_quoted_literals(self): + """Remove both untagged and tagged PostgreSQL dollar-quoted values.""" + normalized = _normalize_sql("SELECT $$customer-token$$, $audit$second-token$audit$") + + self.assertEqual(normalized, "SELECT '?', '?'") + + def test_profile_closes_the_instrumented_connection(self): + """Make the later timing pass open a session without auto_explain hooks.""" + scenario = self._scenarios()[0] + + self._profile_scenario(scenario) + + self.assertIsNone(connection.connection) + + def test_auto_explain_teardown_preserves_database_error(self): + """Keep the original database diagnosis when profiling aborts.""" + with self.assertRaisesRegex(DatabaseError, "division by zero"), transaction.atomic(): + with _auto_explain_notices(), connection.cursor() as cursor: + cursor.execute("SELECT 1 / 0") + + def test_plan_grouping_ignores_runtime_counters(self): + """Group one SQL plan shape even when its runtime work changes.""" + with transaction.atomic(): + with connection.cursor() as cursor: + cursor.execute("CREATE TEMPORARY TABLE profile_grouping_left (value integer) ON COMMIT DROP") + cursor.execute("CREATE TEMPORARY TABLE profile_grouping_right (value integer) ON COMMIT DROP") + cursor.execute("INSERT INTO profile_grouping_left VALUES (1)") + cursor.execute("INSERT INTO profile_grouping_right VALUES (1)") + cursor.execute("SET LOCAL enable_hashjoin = off") + cursor.execute("SET LOCAL enable_mergejoin = off") + + with _auto_explain_notices() as notices, connection.cursor() as cursor: + cursor.execute( + "SELECT left_side.value FROM profile_grouping_left AS left_side " + "JOIN profile_grouping_right AS right_side ON left_side.value = right_side.value" + ) + cursor.execute("INSERT INTO profile_grouping_left VALUES (2)") + cursor.execute( + "SELECT left_side.value FROM profile_grouping_left AS left_side " + "JOIN profile_grouping_right AS right_side ON left_side.value = right_side.value" + ) + + select_plans = [ + plan + for plan in _group_plans(notices) + if plan["normalized_sql"].startswith("SELECT left_side.value FROM profile_grouping_left") + ] + self.assertEqual(len(select_plans), 1, select_plans) + self.assertEqual(select_plans[0]["calls"], 2) + self.assertEqual(select_plans[0]["aggregate"]["actual_rows_times_loops"], 2) + + @staticmethod + def _analyze_tables() -> None: + """Refresh planner statistics after each fixture is prepared.""" + tables = sorted({model._meta.db_table for model in _PROFILE_MODELS}) + with connection.cursor() as cursor: + for table in tables: + cursor.execute(f"ANALYZE {connection.ops.quote_name(table)}") + + def _warm_scenario(self, scenario: _Scenario) -> None: + """Run and discard the operation once, so the recorded pass never pays a first-use cost. + + NetBox resolves object types, custom fields and its search backend lazily and keeps them for + the process. A cold pass therefore issues statements a warm one does not, and a production + worker answers almost every caller warm. + """ + prepared = None + try: + prepared = scenario.prepare() + prepared.operation() + finally: + if prepared is not None: + prepared.cleanup() + + def _profile_scenario(self, scenario: _Scenario) -> dict[str, Any]: + """Record the operation twice and refuse a scenario whose statement counts do not agree.""" + self._warm_scenario(scenario) + first = self._record_scenario(scenario) + second = self._record_scenario(scenario) + self._assert_same_statements(scenario, first, second) + return second + + def _assert_same_statements(self, scenario: _Scenario, first: dict[str, Any], second: dict[str, Any]) -> None: + """Fail with the differing statements, so an artifact can never record an unreproducible count.""" + counts = [ + {item["normalized_sql"]: item["calls"] for item in run["database"]["statements"]} for run in (first, second) + ] + differing = sorted( + (sql for sql in set(counts[0]) | set(counts[1]) if counts[0].get(sql, 0) != counts[1].get(sql, 0)), + key=lambda sql: abs(counts[1].get(sql, 0) - counts[0].get(sql, 0)), + reverse=True, + ) + if not differing: + return + detail = "\n".join( + f" {counts[0].get(sql, 0)} then {counts[1].get(sql, 0)}: {sql[:120]}" for sql in differing[:8] + ) + self.fail( + f"{scenario.name} issued different statements on two identical runs " + f"({first['database']['totals']['statement_calls']} then " + f"{second['database']['totals']['statement_calls']} calls):\n{detail}" + ) + + def _record_scenario(self, scenario: _Scenario) -> dict[str, Any]: + """Record SQL statements and PostgreSQL work for one operation.""" + prepared = None + try: + prepared = scenario.prepare() + self._analyze_tables() + recorder = _StatementRecorder() + with _auto_explain_notices() as notices, connection.execute_wrapper(recorder): + prepared.operation() + semantic = prepared.verify() + statements = recorder.result() + plans = _group_plans(notices) + if not statements: + self.fail(f"{scenario.name} did not execute any recorded SQL statements.") + if not plans: + self.fail(f"{scenario.name} did not produce any auto_explain plans.") + return { + "fixture": prepared.fixture, + "semantic_result": semantic, + "database": { + "statements": statements, + "plans": plans, + "totals": _work_totals(statements, plans, prepared.work_units), + }, + } + finally: + if prepared is not None: + prepared.cleanup() + connection.close() + + def _time_scenario(self, scenario: _Scenario, samples: int, warmups: int) -> dict[str, Any]: + """Measure the uninstrumented operation with fresh fixtures for every sample.""" + with connection.cursor() as cursor: + cursor.execute("SELECT current_setting('auto_explain.log_min_duration', true)") + if cursor.fetchone()[0] is not None: + self.fail("The timing session has loaded auto_explain instrumentation.") + warmup_wall_samples = [] + warmup_cpu_samples = [] + wall_samples = [] + cpu_samples = [] + for sample_index in range(warmups + samples): + prepared = None + try: + prepared = scenario.prepare() + self._analyze_tables() + gc_enabled = gc.isenabled() + if gc_enabled: + gc.disable() + try: + wall_start = time.perf_counter_ns() + cpu_start = time.process_time_ns() + prepared.operation() + cpu_elapsed = time.process_time_ns() - cpu_start + wall_elapsed = time.perf_counter_ns() - wall_start + finally: + if gc_enabled: + gc.enable() + prepared.verify() + if sample_index < warmups: + warmup_wall_samples.append(wall_elapsed) + warmup_cpu_samples.append(cpu_elapsed) + else: + wall_samples.append(wall_elapsed) + cpu_samples.append(cpu_elapsed) + finally: + if prepared is not None: + prepared.cleanup() + return { + "warmup": { + "wall": _optional_summary(warmup_wall_samples), + "process_cpu": _optional_summary(warmup_cpu_samples), + }, + "wall": _summary(wall_samples), + "process_cpu": _summary(cpu_samples), + } + + def test_record_existing_signal_path_performance(self): + """Write the retained before-refactor database and machine-time evidence.""" + output = _output_path() + samples = _positive_integer_from_environment(_SAMPLE_VARIABLE, _DEFAULT_SAMPLES, allow_zero=True) + warmups = _positive_integer_from_environment(_WARMUP_VARIABLE, _DEFAULT_WARMUPS, allow_zero=True) + self.assertTrue(supports_channelization(), "The baseline requires NetBox channelization support.") + self.assertTrue(supports_vc_position_token(), "The baseline requires NetBox VC position token support.") + _assert_isolated_task_queue() + + load_before = _host_load() + scenario_results = [] + for scenario in self._scenarios(): + profiled = self._profile_scenario(scenario) + scenario_results.append( + { + "name": scenario.name, + "description": scenario.description, + "layer": scenario.layer, + **profiled, + "machine_time": self._time_scenario(scenario, samples, warmups) if samples else None, + } + ) + + artifact = { + "schema_version": SCHEMA_VERSION, + "baseline_kind": os.environ.get(_KIND_VARIABLE, _DEFAULT_KIND), + "generated_at": datetime.now(UTC).isoformat(), + "configuration": { + "samples": samples, + "warmups": warmups, + "postgresql_node_timing": False, + "machine_time_ci_gate": False, + }, + "environment": { + **_source_environment(), + "postgresql": _planner_environment(), + "host_load": {"started": load_before, "finished": _host_load()}, + }, + "scenarios": scenario_results, + } + validate_artifact(artifact, "generated performance artifact") + output.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n") + output.with_suffix(".md").write_text(_markdown_summary(artifact)) diff --git a/netbox_interface_name_rules/tests/test_breakout_mode.py b/netbox_interface_name_rules/tests/test_breakout_mode.py index cb6f73f..adfcd1f 100644 --- a/netbox_interface_name_rules/tests/test_breakout_mode.py +++ b/netbox_interface_name_rules/tests/test_breakout_mode.py @@ -35,7 +35,6 @@ from utilities.testing import APITestCase from netbox_interface_name_rules.engine import ( - _VERSION_COLUMNS, apply_interface_name_rules, apply_rule_to_existing, find_interfaces_for_rule, @@ -46,6 +45,7 @@ from netbox_interface_name_rules.filters import InterfaceNameRuleFilterSet from netbox_interface_name_rules.forms import RuleTestForm from netbox_interface_name_rules.models import InterfaceNameRule, _references_channel +from netbox_interface_name_rules.rule_selection import _VERSION_COLUMNS from netbox_interface_name_rules.tests.test_channelization import ( PARENT_TYPE, PLUGIN_LOGGER, @@ -1063,7 +1063,7 @@ def test_the_bulk_apply_path_skips_it_too(self): with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: built = apply_rule_to_existing(self.rule) - self.assertEqual(built, 0) + self.assertEqual(built.changed_count, 0) self.assertEqual(self._names(module), ["3"]) self.assertTrue(any("channelized" in line.lower() for line in logs.output), logs.output) diff --git a/netbox_interface_name_rules/tests/test_bulk_families.py b/netbox_interface_name_rules/tests/test_bulk_families.py new file mode 100644 index 0000000..de2cffa --- /dev/null +++ b/netbox_interface_name_rules/tests/test_bulk_families.py @@ -0,0 +1,793 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for bulk family application and virtual-chassis reapplication.""" + +from unittest import skipUnless +from unittest.mock import MagicMock, patch + +from dcim.choices import InterfaceTypeChoices +from dcim.models import ( + Device, + DeviceRole, + DeviceType, + Interface, + InterfaceTemplate, + Manufacturer, + Module, + ModuleBay, + ModuleBayTemplate, + ModuleType, + Site, + VirtualChassis, +) +from django.db import IntegrityError, connection +from django.test import TestCase +from django.test.utils import CaptureQueriesContext + +from netbox_interface_name_rules import engine as engine_module +from netbox_interface_name_rules.choices import BreakoutModeChoices +from netbox_interface_name_rules.engine import ( + apply_interface_name_rules, + apply_rule_to_existing, + build_variables, + find_interfaces_for_rule, + supports_channelization, +) +from netbox_interface_name_rules.family import ( + FamilyStatus, + FamilyTopology, + InstalledFamilyPlan, + MemberRole, + PlannedMember, + StructuralFamilyPlan, + describe_interfaces, + execute_family_plan, + execute_flat_family, + pinned_template_cache, + plan_flat_family, + plan_prospective_families, + template_names, +) +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.signals import _apply_rules_for_device_deferred +from netbox_interface_name_rules.tests.out_of_band import rename_out_of_band +from netbox_interface_name_rules.tests.test_channelization import _channelized_module_type + +PLAIN_TYPE = InterfaceTypeChoices.TYPE_10GE_SFP_PLUS +REQUIRES_CHANNELIZATION = "requires a NetBox that models channelized interfaces" + + +class BulkTestCase(TestCase): + """One device with eight bays, so a batch can hold eight modules of one type.""" + + POSITIONS = ("1", "2", "3", "4", "5", "6", "7", "8") + + @classmethod + def setUpTestData(cls): + cls.manufacturer = Manufacturer.objects.create(name="BulkMfg", slug="bulk-mfg") + cls.device_type = DeviceType.objects.create( + manufacturer=cls.manufacturer, + model="BULK-DEVICE", + slug="bulk-device", + ) + for position in cls.POSITIONS: + ModuleBayTemplate.objects.create(device_type=cls.device_type, name=f"Bay {position}", position=position) + cls.role = DeviceRole.objects.create(name="BulkRole", slug="bulk-role") + cls.site = Site.objects.create(name="BulkSite", slug="bulk-site") + cls.device = Device.objects.create( + name="bulk-device-01", + device_type=cls.device_type, + role=cls.role, + site=cls.site, + ) + cls.module_type = cls._module_type("BULK-QSFP", ("{module}",)) + + @classmethod + def _module_type(cls, model, template_names): + """Create a module type whose templates resolve to one interface each.""" + module_type = ModuleType.objects.create(manufacturer=cls.manufacturer, model=model, part_number=model) + for name in template_names: + InterfaceTemplate.objects.create(module_type=module_type, name=name, type=PLAIN_TYPE) + return module_type + + def _install(self, position, module_type=None): + """Install one module without letting the deferred install callback run.""" + return Module.objects.create( + device=self.device, + module_bay=ModuleBay.objects.get(device=self.device, name=f"Bay {position}"), + module_type=module_type or self.module_type, + ) + + @staticmethod + def _module_refetches(queries): + """Return the queries that read one whole module row back by primary key.""" + return [ + query["sql"] + for query in queries + if query["sql"].startswith('SELECT "dcim_module"."id"') and 'WHERE "dcim_module"."id" = ' in query["sql"] + ] + + def _names(self, module): + """Return the module's interface names in stable order.""" + return sorted(Interface.objects.filter(module=module).values_list("name", flat=True)) + + @staticmethod + def _flat_rule(module_type, **kwargs): + """Create a flat breakout rule that names four siblings per port.""" + fields = { + "module_type": module_type, + "name_template": "xe-0/0/{bay_position}:{channel}", + "channel_count": 4, + "channel_start": 0, + } + fields.update(kwargs) + return InterfaceNameRule.objects.create(**fields) + + +class BulkApplyBuildsFlatFamiliesTest(BulkTestCase): + """A retroactive apply builds the family a rule describes on every module it matches.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.modules = [self._install(position) for position in ("1", "2")] + + def test_the_outcome_reports_every_interface_the_batch_renamed_or_created(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 8) + self.assertEqual(self._names(self.modules[0]), ["xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"]) + self.assertEqual(self._names(self.modules[1]), ["xe-0/0/2:0", "xe-0/0/2:1", "xe-0/0/2:2", "xe-0/0/2:3"]) + + def test_the_outcome_carries_one_family_per_module(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(len(outcome.families), 2) + self.assertEqual({family.topology for family in outcome.families}, {FamilyTopology.FLAT}) + self.assertEqual({family.status for family in outcome.families}, {FamilyStatus.CHANGED}) + + def test_no_interface_belongs_to_two_families(self): + outcome = apply_rule_to_existing(self.rule) + + claimed = [member.interface_pk for family in outcome.families for member in family.members] + self.assertEqual(len(claimed), len(set(claimed))) + + def test_applying_the_rule_again_changes_nothing(self): + apply_rule_to_existing(self.rule) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual(self._names(self.modules[0]), ["xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"]) + + def test_a_family_the_batch_completes_is_reported_once(self): + """A half-built family is finished through the base it already named, not through each sibling.""" + apply_rule_to_existing(self.rule) + Interface.objects.filter(module=self.modules[0], name__in=("xe-0/0/1:2", "xe-0/0/1:3")).delete() + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 2) + self.assertEqual(self._names(self.modules[0]), ["xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"]) + + +class BulkApplyRenamesPlainInterfacesTest(BulkTestCase): + """A rule that names no family still reports what it renamed as an explicit outcome.""" + + def setUp(self): + self.rule = InterfaceNameRule.objects.create( + module_type=self.module_type, + name_template="et-{bay_position}/0/0", + ) + self.module = self._install("1") + + def test_a_simple_rename_is_reported_as_a_one_member_family(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 1) + self.assertEqual(self._names(self.module), ["et-1/0/0"]) + self.assertEqual(len(outcome.families), 1) + self.assertEqual(outcome.families[0].members[0].current_name, "1") + self.assertEqual(outcome.families[0].members[0].target_name, "et-1/0/0") + + def test_a_name_another_interface_owns_is_reported_as_a_skip(self): + Interface.objects.create(device=self.device, name="et-1/0/0", type=PLAIN_TYPE) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual([member.status for member in outcome.families[0].members], [FamilyStatus.BLOCKED]) + self.assertEqual(len(outcome.skipped_members), 1) + + def test_every_interface_on_the_module_is_renamed(self): + """A simple rule owns no family, so a second port is a second candidate, not a duplicate.""" + module_type = self._module_type("BULK-DUAL", ("{module}/a", "{module}/b")) + module = self._install("2", module_type=module_type) + InterfaceNameRule.objects.create(module_type=module_type, name_template="et-{base}") + + apply_rule_to_existing(InterfaceNameRule.objects.get(module_type=module_type)) + + self.assertEqual(self._names(module), ["et-2/a", "et-2/b"]) + + +class BulkApplySelectionTest(BulkTestCase): + """The Apply view submits interface primary keys, and only their families are applied.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.modules = [self._install(position) for position in ("1", "2")] + + def test_selecting_one_base_applies_only_its_family(self): + base = Interface.objects.get(module=self.modules[0]) + + outcome = apply_rule_to_existing(self.rule, interface_ids=[base.pk]) + + self.assertEqual(outcome.changed_count, 4) + self.assertEqual(self._names(self.modules[1]), ["2"]) + + def test_an_empty_selection_touches_nothing(self): + outcome = apply_rule_to_existing(self.rule, interface_ids=[]) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual(outcome.families, ()) + + def test_the_limit_stops_the_batch_after_the_module_that_reached_it(self): + outcome = apply_rule_to_existing(self.rule, limit=1) + + self.assertEqual(outcome.changed_count, 4) + self.assertEqual(self._names(self.modules[1]), ["2"]) + + def test_a_disabled_rule_applies_nothing(self): + self.rule.enabled = False + self.rule.save() + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual(outcome.families, ()) + + +class BulkApplyContinuesPastOneBlockedFamilyTest(BulkTestCase): + """One module that cannot take its names must not cost the rest of the batch its own.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.modules = [self._install(position) for position in ("1", "2")] + + def test_a_collision_on_the_first_module_leaves_the_second_renamed(self): + Interface.objects.create(device=self.device, name="xe-0/0/1:0", type=PLAIN_TYPE) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(self._names(self.modules[1]), ["xe-0/0/2:0", "xe-0/0/2:1", "xe-0/0/2:2", "xe-0/0/2:3"]) + self.assertEqual(len(outcome.skipped_members), 1) + self.assertEqual(outcome.skipped_members[0].target_name, "xe-0/0/1:0") + + def test_an_unexpected_family_failure_is_logged_and_the_batch_continues(self): + """The boundary keeps going: one family's failure is visible, never the batch's end.""" + from netbox_interface_name_rules.family import batch + + execute = batch.execute_family_plan + + def fail_on_the_first_module(plan): + if plan.module_id == self.modules[0].pk: + raise ValueError("family boom") + return execute(plan) + + with ( + patch.object(batch, "execute_family_plan", side_effect=fail_on_the_first_module), + self.assertLogs("netbox_interface_name_rules.family.batch", level="ERROR"), + ): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(self._names(self.modules[0]), ["1"]) + self.assertEqual(self._names(self.modules[1]), ["xe-0/0/2:0", "xe-0/0/2:1", "xe-0/0/2:2", "xe-0/0/2:3"]) + self.assertEqual(outcome.changed_count, 4) + self.assertEqual([family.status for family in outcome.families], [FamilyStatus.FAILED, FamilyStatus.CHANGED]) + self.assertEqual([member.target_name for member in outcome.skipped_members], ["xe-0/0/1:0"]) + + def test_an_unrelated_integrity_failure_reaches_the_operation_boundary(self): + def reject_interface_update(execute, sql, params, many, context): + if sql.lstrip().startswith('UPDATE "dcim_interface"'): + raise IntegrityError("injected batch database failure") + return execute(sql, params, many, context) + + with connection.execute_wrapper(reject_interface_update): + with self.assertRaisesMessage(IntegrityError, "injected batch database failure"): + apply_rule_to_existing(self.rule) + + self.assertEqual(self._names(self.modules[0]), ["1"]) + self.assertEqual(self._names(self.modules[1]), ["2"]) + + +class BulkApplyQueryScalingTest(BulkTestCase): + """The Apply button runs this over a fleet, so its cost cannot grow per module or per family.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + + def _apply_queries(self): + """Return the queries one full batch runs.""" + with CaptureQueriesContext(connection) as captured: + apply_rule_to_existing(self.rule) + return captured.captured_queries + + def test_the_batch_reads_the_interface_templates_once_for_the_module_type(self): + for position in ("1", "2", "3", "4"): + self._install(position) + + queries = self._apply_queries() + + template_queries = [query for query in queries if "dcim_interfacetemplate" in query["sql"]] + self.assertEqual(len(template_queries), 1, [query["sql"] for query in queries]) + + def test_the_batch_never_refetches_a_module_it_already_holds(self): + for position in ("1", "2", "3", "4"): + self._install(position) + + queries = self._apply_queries() + + self.assertEqual(self._module_refetches(queries), []) + + def test_eight_same_type_families_cost_no_more_per_module_than_two(self): + for position in ("1", "2"): + self._install(position) + two_modules = len(self._apply_queries()) + for position in ("3", "4", "5", "6", "7", "8"): + self._install(position) + + eight_modules = len(self._apply_queries()) + + per_module = (eight_modules - two_modules) / 6 + self.assertLessEqual(per_module, (two_modules / 2), [per_module, two_modules]) + + +class PinnedTemplateCacheBalanceTest(BulkTestCase): + """The batch template cache has to be released even when setting it up fails. + + A stranded cache outlives the batch that opened it, so every later batch on that thread reads + the failed batch's resolved template names instead of its own. On a long-lived worker that is + a rename against names the modules no longer carry. + """ + + def _setup_failure(self): + """Enter the cache with a batch that raises while it is being pinned.""" + broken = MagicMock() + type(broken).pk = property(lambda _self: (_ for _ in ()).throw(RuntimeError("module went away"))) + return pinned_template_cache([broken]) + + def test_a_failed_setup_releases_the_cache(self): + """Anything left behind here is served to the next batch as its own template names.""" + with self.assertRaises(RuntimeError), self._setup_failure(): + pass # pragma: no cover - the block never runs + + self.assertNotIn("resolved", template_names._pin.__dict__) + self.assertNotIn("chained", template_names._pin.__dict__) + self.assertEqual(getattr(template_names._pin, "depth", 0), 0) + + def test_a_later_batch_still_resolves_its_own_names(self): + """The damage the balance prevents: one batch's names outliving it and serving the next one.""" + self._install("1") + rule = self._flat_rule(self.module_type) + with self.assertRaises(RuntimeError), self._setup_failure(): + pass # pragma: no cover - the block never runs + apply_rule_to_existing(rule) # fills a cache that a stranded pin would keep + + with CaptureQueriesContext(connection) as captured: + apply_rule_to_existing(rule) + + template_queries = [query for query in captured.captured_queries if "dcim_interfacetemplate" in query["sql"]] + self.assertEqual(len(template_queries), 1, [query["sql"] for query in captured.captured_queries]) + + +class VirtualChassisReapplyTestCase(BulkTestCase): + """A position change reaches every module-attached family through the plugin's own signals.""" + + def setUp(self): + self.rule = InterfaceNameRule.objects.create( + module_type=self.module_type, + name_template="et-{vc_position}/0/{bay_position}", + ) + self.virtual_chassis = VirtualChassis.objects.create(name="BulkVC") + + def _install_and_name(self, positions): + """Install one module per bay position and give it the name the rule produces now.""" + modules = [] + for position in positions: + module = self._install(position) + apply_interface_name_rules(module, module.module_bay) + modules.append(module) + return modules + + def _join(self, position): + """Move the device to *position* in the virtual chassis through a real save.""" + with self.captureOnCommitCallbacks(execute=True): + self.device.virtual_chassis = self.virtual_chassis + self.device.vc_position = position + self.device.save() + + +class VirtualChassisReapplyTest(VirtualChassisReapplyTestCase): + """Every module family the position names has to move with it.""" + + def test_every_module_family_follows_the_new_position(self): + modules = self._install_and_name(("1", "2", "3")) + + self._join(4) + + for module, position in zip(modules, ("1", "2", "3"), strict=True): + self.assertEqual(self._names(module), [f"et-4/0/{position}"]) + + def test_a_device_level_rule_still_runs_beside_the_module_families(self): + module = self._install_and_name(("1",))[0] + Interface.objects.create(device=self.device, name="mgmt0", type=PLAIN_TYPE) + InterfaceNameRule.objects.create( + name_template="mgmt-{vc_position}", + applies_to_device_interfaces=True, + module_type_pattern="mgmt0", + ) + + self._join(4) + + self.assertEqual(self._names(module), ["et-4/0/1"]) + self.assertTrue(Interface.objects.filter(device=self.device, module=None, name="mgmt-4").exists()) + + def test_one_failing_module_leaves_the_others_reapplied(self): + modules = self._install_and_name(("1", "2")) + real_apply = engine_module.apply_interface_name_rules + + def fail_on_the_first_module(module, module_bay, force_reapply=False): + if module.pk == modules[0].pk: + raise RuntimeError("module boom") + return real_apply(module, module_bay, force_reapply=force_reapply) + + with ( + patch.object(engine_module, "apply_interface_name_rules", side_effect=fail_on_the_first_module), + self.assertLogs("netbox_interface_name_rules", level="ERROR"), + ): + self._join(4) + + self.assertEqual(self._names(modules[0]), ["1"]) + self.assertEqual(self._names(modules[1]), ["et-4/0/2"]) + + +class VirtualChassisReapplyCostTest(VirtualChassisReapplyTestCase): + """A chassis-wide position change must not read one module type's templates once per module.""" + + def _reapply_queries(self, position): + """Return the queries the deferred reapplication runs for one position change.""" + self.device.virtual_chassis = self.virtual_chassis + self.device.vc_position = position + self.device.save() + with CaptureQueriesContext(connection) as captured: + _apply_rules_for_device_deferred(self.device.pk) + return captured.captured_queries + + def test_the_reapply_reads_the_interface_templates_once_for_the_module_type(self): + self._install_and_name(("1", "2", "3", "4")) + + queries = self._reapply_queries(4) + + template_queries = [query for query in queries if "dcim_interfacetemplate" in query["sql"]] + self.assertEqual(len(template_queries), 1, [query["sql"] for query in queries]) + + def test_the_reapply_never_refetches_a_module_it_already_holds(self): + self._install_and_name(("1", "2", "3", "4")) + + queries = self._reapply_queries(4) + + self.assertEqual(self._module_refetches(queries), []) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class BulkApplyChannelizedFamiliesTest(BulkTestCase): + """Eight channelized families in one batch stay eight families, each renamed as a unit.""" + + @classmethod + def setUpTestData(cls): + super().setUpTestData() + cls.channelized_type = _channelized_module_type(cls.manufacturer, "BULK-CHANNELIZED") + + def setUp(self): + self.rule = self._flat_rule(self.channelized_type, name_template="xe-0/0/{bay_position}:{channel}") + + def _install_families(self, positions): + """Install one channelized module per bay position.""" + return [self._install(position, module_type=self.channelized_type) for position in positions] + + def test_every_installed_family_is_renamed_as_one_family(self): + modules = self._install_families(self.POSITIONS) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(len(outcome.families), len(self.POSITIONS)) + self.assertEqual({family.topology for family in outcome.families}, {FamilyTopology.CHANNELIZED}) + self.assertEqual( + self._names(modules[0]), + ["1", "xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"], + ) + + def test_no_channel_belongs_to_two_families(self): + self._install_families(self.POSITIONS) + + outcome = apply_rule_to_existing(self.rule) + + claimed = [member.interface_pk for family in outcome.families for member in family.members] + self.assertEqual(len(claimed), len(set(claimed))) + + def test_a_family_missing_a_channel_still_names_the_channels_it_has(self): + """A channel is named from its own channel id, so a family that lost one is still repaired.""" + modules = self._install_families(("1",)) + Interface.objects.filter(module=modules[0], channel_id=2).delete() + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(self._names(modules[0]), ["1", "xe-0/0/1:0", "xe-0/0/1:2", "xe-0/0/1:3"]) + self.assertEqual(outcome.skipped_members, ()) + + def _apply_queries(self, positions): + """Return the queries one batch runs over freshly installed families at *positions*.""" + self._install_families(positions) + with CaptureQueriesContext(connection) as captured: + apply_rule_to_existing(self.rule) + return captured.captured_queries + + def test_the_batch_reads_the_interface_templates_once_for_the_module_type(self): + queries = self._apply_queries(self.POSITIONS) + + template_queries = [query for query in queries if "dcim_interfacetemplate" in query["sql"]] + self.assertEqual(len(template_queries), 1, [query["sql"] for query in queries]) + + def test_eight_channelized_families_cost_no_more_per_module_than_the_first(self): + """Renaming a fleet stays linear in its modules: no family pays for the ones beside it.""" + added = self.POSITIONS[1:] + one_module = len(self._apply_queries(("1",))) + + eight_modules = len(self._apply_queries(added)) + + per_module = (eight_modules - one_module) / len(added) + self.assertLessEqual(per_module, one_module, [per_module, one_module, eight_modules]) + + def test_selecting_a_channel_on_its_own_applies_nothing(self): + modules = self._install_families(("1",)) + channel = Interface.objects.filter(module=modules[0], channel_id__isnull=False).first() + + outcome = apply_rule_to_existing(self.rule, interface_ids=[channel.pk]) + + self.assertEqual(outcome.changed_count, 0) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class BulkApplyBuildsChannelizedFamiliesTest(BulkTestCase): + """A channelized rule builds one family per module, and reports a refusal as one outcome.""" + + def setUp(self): + self.rule = self._flat_rule( + self.module_type, + breakout_mode=BreakoutModeChoices.CHANNELIZED, + parent_name_template="et-0/0/{bay_position}", + ) + self.modules = [self._install(position) for position in ("1", "2")] + + def test_each_module_gains_the_family_the_rule_describes(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(len(outcome.families), 2) + self.assertEqual( + self._names(self.modules[0]), + ["et-0/0/1", "xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"], + ) + + def test_a_module_whose_parent_name_is_taken_is_reported_and_the_next_still_builds(self): + Interface.objects.create(device=self.device, name="et-0/0/1", type=PLAIN_TYPE) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(self._names(self.modules[0]), ["1"]) + self.assertEqual( + self._names(self.modules[1]), + ["et-0/0/2", "xe-0/0/2:0", "xe-0/0/2:1", "xe-0/0/2:2", "xe-0/0/2:3"], + ) + self.assertEqual(len(outcome.skipped_members), 1) + + +class FlatFamilyCreationTest(BulkTestCase): + """The rows a flat breakout family adds, and what stops it from adding them.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.module = self._install("1") + self.base = Interface.objects.get(module=self.module) + self.variables = build_variables(self.module.module_bay, device=self.device) + + def _plan(self): + """Plan the flat family the rule builds on this module's only interface.""" + return plan_flat_family(self.module, self.rule, self.variables, self.base) + + def test_a_sibling_whose_name_is_taken_is_skipped_and_the_rest_are_created(self): + """One sibling's collision is that sibling's own; the family keeps the names it can take.""" + Interface.objects.create(device=self.device, name="xe-0/0/1:2", type=PLAIN_TYPE) + + outcome = execute_flat_family(self._plan()) + + self.assertEqual(outcome.status, FamilyStatus.CHANGED) + self.assertEqual(outcome.changed_count, 3) + blocked = [member for member in outcome.members if member.status == FamilyStatus.BLOCKED] + self.assertEqual([member.target_name for member in blocked], ["xe-0/0/1:2"]) + self.assertEqual(self._names(self.module), ["xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:3"]) + + def test_a_base_renamed_after_planning_is_refused_as_stale(self): + """Planning is not a lock: the row it described has to still be the row it finds.""" + plan = self._plan() + rename_out_of_band(Interface.objects.get(pk=self.base.pk), "moved") + + outcome = execute_flat_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.STALE) + self.assertEqual(self._names(self.module), ["moved"]) + + def test_a_name_netbox_rejects_leaves_the_whole_family_unbuilt(self): + """A name the model refuses is not a partial family: nothing is written at all.""" + self.rule.name_template = "x" * 70 + "{channel}" + self.rule.save() + + outcome = execute_flat_family(self._plan()) + + self.assertEqual(outcome.status, FamilyStatus.BLOCKED) + self.assertEqual(self._names(self.module), ["1"]) + + def test_a_template_the_rule_cannot_evaluate_builds_nothing(self): + """The plan carries the failure, so the executor writes nothing and says why.""" + self.rule.name_template = "{undefined_var}:{channel}" + self.rule.save() + + outcome = execute_flat_family(self._plan()) + + self.assertEqual(outcome.status, FamilyStatus.FAILED) + self.assertEqual(self._names(self.module), ["1"]) + + def test_a_simple_rule_the_engine_cannot_evaluate_is_reported_as_a_failure(self): + """A one-interface rename carries its template failure the same way a family does.""" + rule = InterfaceNameRule.objects.create( + module_type=self._module_type("BULK-SIMPLE-FAIL", ("{module}",)), + name_template="{undefined_var}", + ) + module = self._install("2", module_type=rule.module_type) + + outcome = apply_rule_to_existing(rule) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual([member.status for member in outcome.skipped_members], [FamilyStatus.FAILED]) + self.assertEqual(self._names(module), ["2"]) + + +class OnlyLivePlansAreExecutableTest(BulkTestCase): + """A plan without live rows cannot reach an executor, whichever door it arrives at.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.module = self._install("1") + + def _prospective_plan(self): + """Return one prospective plan for this module: the object an interactive preview holds.""" + plan_set = plan_prospective_families( + self.module, + self.rule, + build_variables(self.module.module_bay, device=self.device), + describe_interfaces(Interface.objects.filter(module=self.module)), + ) + return plan_set.plans[0] + + def test_the_batch_executor_refuses_a_prospective_plan(self): + with self.assertRaises(TypeError): + execute_family_plan(self._prospective_plan()) + + self.assertEqual(self._names(self.module), ["1"]) + + def test_the_flat_executor_refuses_a_prospective_plan(self): + with self.assertRaises(TypeError): + execute_flat_family(self._prospective_plan()) + + self.assertEqual(self._names(self.module), ["1"]) + + def test_every_plan_kind_exposes_its_live_members_for_failure_reporting(self): + base = Interface.objects.get(module=self.module) + flat = plan_flat_family( + self.module, + self.rule, + build_variables(self.module.module_bay, device=self.device), + base, + ) + installed = InstalledFamilyPlan( + family_id=f"installed:{base.pk}", + topology=FamilyTopology.FLAT, + device_id=self.device.pk, + module_id=self.module.pk, + db_alias=flat.db_alias, + members=(PlannedMember(flat.base, "installed-target", MemberRole.FLAT_MEMBER),), + ) + structural = StructuralFamilyPlan( + family_id=f"structural:{base.pk}", + device_id=self.device.pk, + module_id=self.module.pk, + module_type_id=self.module_type.pk, + db_alias=flat.db_alias, + base=flat.base, + parent_target_name="parent-target", + channel_count=0, + channels=(), + ) + + self.assertEqual( + [plan.live_members[0].target_name for plan in (installed, structural, flat)], + ["installed-target", "parent-target", "xe-0/0/1:0"], + ) + + +class BulkApplyReportsSkipsToItsCallersTest(BulkTestCase): + """The Apply view and the background job both read their counts from the batch outcome.""" + + def setUp(self): + self.rule = self._flat_rule(self.module_type) + self.module = self._install("1") + Interface.objects.create(device=self.device, name="xe-0/0/1:0", type=PLAIN_TYPE) + for channel in (1, 2, 3): + Interface.objects.create( + device=self.device, module=self.module, name=f"xe-0/0/1:{channel}", type=PLAIN_TYPE + ) + + def test_a_family_that_can_take_no_name_is_reported_blocked(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.changed_count, 0) + self.assertEqual([family.status for family in outcome.families], [FamilyStatus.BLOCKED]) + self.assertEqual([member.target_name for member in outcome.skipped_members], ["xe-0/0/1:0"]) + + def test_the_background_job_warns_about_what_it_skipped(self): + from netbox_interface_name_rules.jobs import ApplyRuleJob + + job = ApplyRuleJob.__new__(ApplyRuleJob) + job.logger = MagicMock() + + job.run(rule_id=self.rule.pk) + + job.logger.info.assert_called_once() + job.logger.warning.assert_called_once() + self.assertEqual(job.logger.warning.call_args.args[1], 1) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class BulkApplyBuildsOneFamilyPerPortTest(BulkTestCase): + """Every port a channelized rule names gets its own family, however many the module carries.""" + + @classmethod + def setUpTestData(cls): + super().setUpTestData() + cls.dual_type = cls._module_type("BULK-DUAL-CH", ("{module}/a", "{module}/b")) + + def setUp(self): + self.rule = self._flat_rule( + self.dual_type, + name_template="{base}:{channel}", + parent_name_template="{base}", + breakout_mode=BreakoutModeChoices.CHANNELIZED, + channel_start=1, + ) + self.module = self._install("1", module_type=self.dual_type) + + def test_both_ports_gain_the_family_the_rule_describes(self): + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual(outcome.skipped_members, ()) + self.assertEqual( + self._names(self.module), + ["1/a", "1/a:1", "1/a:2", "1/a:3", "1/a:4", "1/b", "1/b:1", "1/b:2", "1/b:3", "1/b:4"], + ) + + def test_the_preview_offers_exactly_the_families_the_apply_builds(self): + previewed, _total = find_interfaces_for_rule(self.rule) + + outcome = apply_rule_to_existing(self.rule) + + self.assertEqual([entry["current_name"] for entry in previewed], ["1/a", "1/b"]) + self.assertEqual(len(outcome.families), len(previewed)) + self.assertEqual({family.status for family in outcome.families}, {FamilyStatus.CHANGED}) diff --git a/netbox_interface_name_rules/tests/test_channelization.py b/netbox_interface_name_rules/tests/test_channelization.py index 217f3c3..6c39cdd 100644 --- a/netbox_interface_name_rules/tests/test_channelization.py +++ b/netbox_interface_name_rules/tests/test_channelization.py @@ -35,11 +35,13 @@ apply_device_interface_rules, apply_interface_name_rules, apply_rule_to_existing, + build_variables, find_interfaces_for_rule, has_applicable_interfaces, predict_rule_output, supports_channelization, ) +from netbox_interface_name_rules.family import FamilyStatus, execute_installed_plan_set, plan_installed_families from netbox_interface_name_rules.models import InterfaceNameRule # Resolved defensively so this module still imports on NetBox releases without channelization. @@ -345,7 +347,7 @@ def test_preview_offers_channel_renames_and_no_new_interfaces(self): self.assertEqual(results[0]["current_name"], "3") self.assertEqual(results[0]["new_names"], ["3", "xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"]) self.assertEqual( - [(detail["role"], detail["channel_id"]) for detail in results[0]["name_details"]], + [(detail.role, detail.channel_id) for detail in results[0]["name_details"]], [("parent", None), ("channel", 1), ("channel", 2), ("channel", 3), ("channel", 4)], ) @@ -400,17 +402,20 @@ def test_preview_annotates_the_family_behind_the_parent_entry(self): self.assertEqual(family["new_names"], ["et-3", "et-3:1", "et-3:2", "et-3:3", "et-3:4"]) self.assertEqual( - family["name_details"], + [(detail.name, detail.role, detail.channel_id) for detail in family["name_details"]], [ - {"name": "et-3", "role": "parent", "channel_id": None}, - {"name": "et-3:1", "role": "channel", "channel_id": 1}, - {"name": "et-3:2", "role": "channel", "channel_id": 2}, - {"name": "et-3:3", "role": "channel", "channel_id": 3}, - {"name": "et-3:4", "role": "channel", "channel_id": 4}, + ("et-3", "parent", None), + ("et-3:1", "channel", 1), + ("et-3:2", "channel", 2), + ("et-3:3", "channel", 3), + ("et-3:4", "channel", 4), ], ) self.assertEqual(family["interface"], self._parent(family["module"])) - self.assertEqual(standalone["name_details"], [{"name": "et-3-mgmt", "role": "interface", "channel_id": None}]) + self.assertEqual( + [(detail.name, detail.role, detail.channel_id) for detail in standalone["name_details"]], + [("et-3-mgmt", "interface", None)], + ) def test_has_applicable_interfaces_ignores_a_renamed_family(self): """Children must not keep the rule looking 'applicable' after the family is correctly named.""" @@ -424,12 +429,11 @@ def test_has_applicable_interfaces_ignores_a_renamed_family(self): def test_apply_rule_to_existing_renames_family_without_conflicts(self): """A retroactive apply renames the family through its parent, so no child fights another for a name.""" module, _ = self._install(self.module_type, "3", run_rules=False) - conflicts: list = [] - count = apply_rule_to_existing(self.rule, conflicts=conflicts) + outcome = apply_rule_to_existing(self.rule) - self.assertEqual(count, 6) - self.assertEqual(conflicts, []) + self.assertEqual(outcome.changed_count, 6) + self.assertEqual(outcome.skipped_members, ()) self.assertEqual(self._names(module), ["et-3", "et-3-mgmt", "et-3:1", "et-3:2", "et-3:3", "et-3:4"]) # The rename went through the family, so the structure it hangs on is still intact. parent = self._parent(module) @@ -441,9 +445,9 @@ def test_apply_rule_to_existing_selected_parent_renames_whole_family(self): module, _ = self._install(self.module_type, "3", run_rules=False) parent = self._parent(module) - count = apply_rule_to_existing(self.rule, interface_ids=[parent.pk]) + outcome = apply_rule_to_existing(self.rule, interface_ids=[parent.pk]) - self.assertEqual(count, 5) + self.assertEqual(outcome.changed_count, 5) self.assertEqual(self._names(module), ["3-mgmt", "et-3", "et-3:1", "et-3:2", "et-3:3", "et-3:4"]) def test_apply_rule_to_existing_ignores_a_child_only_selection(self): @@ -451,9 +455,9 @@ def test_apply_rule_to_existing_ignores_a_child_only_selection(self): module, _ = self._install(self.module_type, "3", run_rules=False) child = self._child(module, 1) - count = apply_rule_to_existing(self.rule, interface_ids=[child.pk]) + outcome = apply_rule_to_existing(self.rule, interface_ids=[child.pk]) - self.assertEqual(count, 0) + self.assertEqual(outcome.changed_count, 0) self.assertEqual(self._names(module), ["3", "3-mgmt", "3:1", "3:2", "3:3", "3:4"]) @@ -490,6 +494,20 @@ def test_children_follow_parent_and_are_not_matched_independently(self): self.assertEqual(renamed, 5) self.assertEqual(self._names(), ["eth1", "eth1:1", "eth1:2", "eth1:3", "eth1:4"]) + def test_a_channel_count_on_a_device_rule_does_not_block_the_family(self): + """A device rule never builds a family, so a channel count on one says nothing about this one.""" + InterfaceNameRule.objects.create( + applies_to_device_interfaces=True, + name_template="eth{vc_position}", + channel_count=2, + channel_start=0, + ) + + renamed = apply_device_interface_rules(self.device) + + self.assertEqual(renamed, 5) + self.assertEqual(self._names(), ["eth1", "eth1:1", "eth1:2", "eth1:3", "eth1:4"]) + def test_rule_matching_only_children_renames_nothing(self): """A pattern that hits only channel subinterfaces has no parent to act on.""" InterfaceNameRule.objects.create( @@ -643,6 +661,11 @@ class ChannelizedBreakoutTemplateErrorTest(ChannelizationTestCase): def setUpTestData(cls): manufacturer, cls.device = _build_device("ChanErr", ["4"]) cls.module_type = _channelized_module_type(manufacturer, "ChanErr-QSFP") + InterfaceTemplate.objects.create( + module_type=cls.module_type, + name="mgmt-{module}", + type=PLAIN_TYPE, + ) # Valid for channel 0, divides by zero from channel 1 on. cls.rule = InterfaceNameRule.objects.create( module_type=cls.module_type, @@ -655,10 +678,18 @@ def test_template_failing_on_a_later_channel_renames_no_child(self): """Every child name is computed before the first save, so one bad channel aborts the family.""" module, bay = self._install(self.module_type, "4", run_rules=False) + plan_set = plan_installed_families(module, self.rule, build_variables(bay, device=self.device)) + outcome = execute_installed_plan_set(plan_set) + renamed = apply_interface_name_rules(module, bay) + self.assertEqual(outcome.families[0].status, FamilyStatus.FAILED) + self.assertEqual( + {member.status for member in outcome.families[0].members}, + {FamilyStatus.FAILED}, + ) self.assertEqual(renamed, 0) - self.assertEqual(self._names(module), ["4", "4:1", "4:2", "4:3", "4:4"]) + self.assertEqual(self._names(module), ["4", "4:1", "4:2", "4:3", "4:4", "mgmt-4"]) @skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) diff --git a/netbox_interface_name_rules/tests/test_channelized_mode.py b/netbox_interface_name_rules/tests/test_channelized_mode.py index 3bc4539..e7ae4ae 100644 --- a/netbox_interface_name_rules/tests/test_channelized_mode.py +++ b/netbox_interface_name_rules/tests/test_channelized_mode.py @@ -275,12 +275,11 @@ def test_the_interactive_apply_records_the_collision(self): """The Apply view counts what it skipped, so the collision is reported, not just logged.""" self._occupy("et-0/0/5") self._install(self.module_type, "5", run_rules=False) - conflicts: list = [] - built = apply_rule_to_existing(self.rule, conflicts=conflicts) + built = apply_rule_to_existing(self.rule) - self.assertEqual(built, 0) - self.assertEqual([conflict["attempted_name"] for conflict in conflicts], ["et-0/0/5"]) + self.assertEqual(built.changed_count, 0) + self.assertEqual([member.target_name for member in built.skipped_members], ["et-0/0/5"]) def test_the_family_is_built_once_the_blocker_is_gone(self): """The skip is a state of the device, not a decision about the rule.""" @@ -357,13 +356,12 @@ def test_the_refused_conversion_is_reported(self): def test_the_bulk_apply_path_refuses_it_too(self): """Both entry points share the preflight, so neither can convert a family behind the other's back.""" self._switch_to_channelized() - conflicts: list = [] - changed = apply_rule_to_existing(self.rule, conflicts=conflicts) + changed = apply_rule_to_existing(self.rule) - self.assertEqual(changed, 0) + self.assertEqual(changed.changed_count, 0) self._assert_still_flat() - self.assertTrue(conflicts, "the skipped family was not reported to the Apply view") + self.assertTrue(changed.skipped_members, "the skipped family was not reported to the Apply view") @skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) @@ -414,14 +412,12 @@ def test_force_apply_builds_no_family_beside_the_flat_one(self): def test_the_bulk_apply_path_refuses_it_too(self): """Both entry points share the refusal, so neither can convert a family behind the other's back.""" - conflicts: list = [] - with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: - changed = apply_rule_to_existing(self.rule, conflicts=conflicts) + changed = apply_rule_to_existing(self.rule) - self.assertEqual(changed, 0) + self.assertEqual(changed.changed_count, 0) self._assert_untouched() - self.assertTrue(conflicts, "the skipped module was not reported to the Apply view") + self.assertTrue(changed.skipped_members, "the skipped module was not reported to the Apply view") self.assertTrue(any(str(self.module) in line for line in logs.output), logs.output) def test_the_preview_offers_no_family_it_would_not_build(self): @@ -528,7 +524,7 @@ def test_preview_describes_the_family_it_would_build(self): self.assertEqual(results[0]["current_name"], "3") self.assertEqual(results[0]["new_names"], self.FAMILY_NAMES) self.assertEqual( - [(detail["role"], detail["channel_id"]) for detail in results[0]["name_details"]], + [(detail.role, detail.channel_id) for detail in results[0]["name_details"]], [("parent", None), ("channel", 1), ("channel", 2), ("channel", 3), ("channel", 4)], ) @@ -550,7 +546,7 @@ def test_retroactive_apply_builds_the_family(self): """Applying a rule to already-installed hardware creates the family the preview promised.""" built = apply_rule_to_existing(self.rule) - self.assertEqual(built, 5) + self.assertEqual(built.changed_count, 5) self.assertEqual(self._names(self.module), self.FAMILY_NAMES) self.assertEqual(self._parent(self.module).channels, 4) @@ -560,7 +556,7 @@ def test_applying_the_selected_port_builds_its_family(self): built = apply_rule_to_existing(self.rule, interface_ids=[base.pk]) - self.assertEqual(built, 5) + self.assertEqual(built.changed_count, 5) self.assertEqual(self._names(self.module), self.FAMILY_NAMES) def test_the_rule_test_view_previews_the_family(self): diff --git a/netbox_interface_name_rules/tests/test_conversion.py b/netbox_interface_name_rules/tests/test_conversion.py index 4eb0191..3b5ea8d 100644 --- a/netbox_interface_name_rules/tests/test_conversion.py +++ b/netbox_interface_name_rules/tests/test_conversion.py @@ -9,7 +9,7 @@ would become. The conversion keeps the physical row: the old ch-0 interface becomes the parent (same pk, cable, -type and module link) and its *logical* identity — addresses, VLANs, description, tags — moves to a +type and module link) and its logical identity (VRF, addresses, VLANs, description, and tags) moves to a newly created channel-1 child that takes over its name. Every family is preflighted whole, including a dry-run ``full_clean()`` of the parent and of every prospective child, so a family that cannot become a valid channelized family is reported as blocked instead of half converted. @@ -19,17 +19,17 @@ from unittest import skipIf, skipUnless from unittest.mock import patch -from core.models import Job, ObjectType +from core.models import Job, ObjectChange, ObjectType from dcim.choices import InterfaceModeChoices -from dcim.models import Cable, Interface +from dcim.models import Cable, Interface, Module from django.contrib.auth import get_user_model from django.contrib.messages import get_messages -from django.db import connection +from django.db import connection, transaction from django.test.utils import CaptureQueriesContext from django.urls import reverse from django.utils.html import escape from extras.models import CustomField, Tag -from ipam.models import VLAN, FHRPGroup, FHRPGroupAssignment, IPAddress +from ipam.models import VLAN, VRF, FHRPGroup, FHRPGroupAssignment, IPAddress from netbox_interface_name_rules.engine import ( apply_interface_name_rules, @@ -38,7 +38,11 @@ find_convertible_families, supports_channelization, ) +from netbox_interface_name_rules.family import FamilyStatus, execute_conversion, plan_module_conversions +from netbox_interface_name_rules.family.template_names import BAY_CHAIN_RELATIONS from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.naming import build_variables +from netbox_interface_name_rules.tests.out_of_band import rename_out_of_band from netbox_interface_name_rules.tests.test_breakout_mode import ( CHANNELIZED, FLAT, @@ -92,9 +96,13 @@ def _iface(self, name): return Interface.objects.get(device=self.device, name=name) def _verdicts(self, limit=None): - """Return just the scan's verdicts; the 'more families' flag is asserted where it is the point.""" - verdicts, _ = find_convertible_families(self.rule, limit=limit) - return verdicts + """Return just the scan's candidates; the 'more families' flag is asserted where it is the point.""" + return find_convertible_families(self.rule, limit=limit).candidates + + @staticmethod + def _converted(outcome): + """Return how many families a conversion batch actually rewrote.""" + return len(outcome.changed_families) @staticmethod def _flat_names(position): @@ -133,30 +141,30 @@ def test_the_flat_family_is_offered_for_conversion(self): verdicts = self._verdicts() self.assertEqual(len(verdicts), 1) - self.assertTrue(verdicts[0]["convertible"]) - self.assertEqual(verdicts[0]["reason"], "") + self.assertTrue(verdicts[0].convertible) + self.assertEqual(verdicts[0].reason, "") def test_the_verdict_is_keyed_on_the_ch0_row(self): """The ch-0 interface is the physical row and the pk the confirm form submits.""" verdict = self._verdicts()[0] - self.assertEqual(verdict["interface"].pk, self._iface("xe-0/0/3:0").pk) - self.assertEqual(verdict["current_name"], "xe-0/0/3:0") - self.assertEqual(verdict["module"].pk, self.module.pk) + self.assertEqual(verdict.interface.pk, self._iface("xe-0/0/3:0").pk) + self.assertEqual(verdict.current_name, "xe-0/0/3:0") + self.assertEqual(verdict.module.pk, self.module.pk) def test_the_verdict_names_the_family_as_it_is_and_as_it_would_be(self): """An operator confirms a rewrite of named rows, so both name sets have to be on the page.""" verdict = self._verdicts()[0] - self.assertEqual(verdict["current_names"], self._flat_names("3")) - self.assertEqual(verdict["new_names"], ["et-0/0/3", *self._flat_names("3")]) + self.assertEqual(list(verdict.current_names), self._flat_names("3")) + self.assertEqual(list(verdict.new_names), ["et-0/0/3", *self._flat_names("3")]) def test_the_verdict_describes_the_parent_and_its_channels(self): """The page must show which row becomes the parent and which channel each name binds to.""" verdict = self._verdicts()[0] self.assertEqual( - [(detail["role"], detail["channel_id"]) for detail in verdict["name_details"]], + [(detail.role, detail.channel_id) for detail in verdict.name_details], [("parent", None), ("channel", 1), ("channel", 2), ("channel", 3), ("channel", 4)], ) @@ -164,20 +172,25 @@ def test_the_verdict_says_where_the_ch0_configuration_lands(self): """The ch-0 row keeps its pk but loses its addresses — the operator has to be told which row gets them.""" verdict = self._verdicts()[0] - self.assertIn("xe-0/0/3:0", verdict["metadata_note"]) + self.assertIn("xe-0/0/3:0", verdict.metadata_note) def test_the_verdict_warns_that_the_ch0_interface_id_becomes_the_parent(self): """Automation keyed on the ch-0 interface id will address the parent afterwards, silently.""" verdict = self._verdicts()[0] - self.assertRegex(verdict["metadata_note"], r"(?i)\bid\b") + self.assertRegex(verdict.metadata_note, r"(?i)\bid\b") def test_finding_the_verdicts_converts_nothing(self): """The preflight really performs the conversion to validate it, so the rollback is the feature.""" pks = dict(Interface.objects.filter(module=self.module).values_list("name", "pk")) - self._verdicts() + with patch( + "netbox_interface_name_rules.family.conversion.transaction.set_rollback", + wraps=transaction.set_rollback, + ) as set_rollback: + self._verdicts() + set_rollback.assert_called_once_with(True, using=self.module._state.db) self._assert_still_flat(self.module, "3") self.assertEqual(dict(Interface.objects.filter(module=self.module).values_list("name", "pk")), pks) @@ -185,7 +198,7 @@ def test_a_rule_without_a_parent_template_offers_no_conversion(self): """In a flat family the ch-0 row is the base: without a parent name there is nowhere to put it.""" self._switch_to_channelized(parent_name_template="") - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) def test_a_flat_rule_offers_no_conversion(self): """The rule still describes the flat topology, so its families are not the wrong shape.""" @@ -193,36 +206,53 @@ def test_a_flat_rule_offers_no_conversion(self): self.rule.parent_name_template = "" self.rule.save() - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) def test_a_rule_without_channels_offers_no_conversion(self): """No channel count means no family to identify — there is nothing to convert against.""" self.rule.channel_count = 0 self.rule.save() - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) def test_a_port_the_rule_never_touched_is_not_a_conversion_candidate(self): """A raw port is an ordinary apply, not a conversion: no flat family exists on it yet.""" for iface in Interface.objects.filter(module=self.module).exclude(name="xe-0/0/3:0"): iface.delete() - Interface.objects.filter(pk=self._iface("xe-0/0/3:0").pk).update(name="3") + rename_out_of_band(self._iface("xe-0/0/3:0"), "3") - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) def test_a_disabled_rule_offers_no_conversion(self): """A disabled rule renames nothing; offering it the most invasive rewrite of all would be worse.""" self.rule.enabled = False self.rule.save() - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) + + def test_an_unsupported_release_reports_the_family_explicitly(self): + """Capability refusal is a family result, not an empty scan or batch.""" + base = self._iface("xe-0/0/3:0") + with patch( + "netbox_interface_name_rules.family.conversion.supports_channelization", + return_value=False, + ): + verdicts = self._verdicts() + outcome = convert_flat_families(self.rule, [base.pk]) + + self.assertEqual(len(verdicts), 1) + self.assertFalse(verdicts[0].convertible) + self.assertIn("cannot model channelized interfaces", verdicts[0].reason) + self.assertEqual(len(outcome.families), 1) + self.assertEqual(outcome.families[0].status, FamilyStatus.UNSUPPORTED) + self._assert_still_flat(self.module, "3") def test_a_converted_family_is_not_offered_again(self): """It is a native family now; offering it again would invite a second, duplicate conversion.""" base = self._iface("xe-0/0/3:0") convert_flat_families(self.rule, [base.pk]) - self.assertEqual(self._verdicts(), []) + self.assertEqual(self._verdicts(), ()) @skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) @@ -316,7 +346,7 @@ def test_the_converted_family_validates(self): def test_conversion_reports_how_many_families_it_converted(self): """The Apply view reports the count back to the operator, so it counts families, not rows.""" - self.assertEqual(self._convert(self.base), 1) + self.assertEqual(self._converted(self._convert(self.base)), 1) def test_only_the_selected_family_is_converted(self): """Conversion is per family and confirmed per family; an unselected one must not move.""" @@ -326,13 +356,13 @@ def test_only_the_selected_family_is_converted(self): def test_converting_without_a_selection_converts_every_convertible_family(self): """This is the batch the background job runs after the operator confirms the whole rule.""" - self.assertEqual(self._convert(), 2) + self.assertEqual(self._converted(self._convert()), 2) self.assertEqual(self._names(self.module), self._channelized_names("3")) self.assertEqual(self._names(self.other_module), self._channelized_names("4")) def test_converting_an_empty_selection_does_nothing(self): """An empty confirmation is not 'convert everything' — that would be the worst possible default.""" - self.assertEqual(convert_flat_families(self.rule, []), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [])), 0) self._assert_still_flat(self.module, "3") def test_a_second_conversion_run_leaves_the_family_alone(self): @@ -340,7 +370,7 @@ def test_a_second_conversion_run_leaves_the_family_alone(self): self._convert(self.base) child_pks = dict(Interface.objects.filter(module=self.module).values_list("name", "pk")) - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) self.assertEqual(dict(Interface.objects.filter(module=self.module).values_list("name", "pk")), child_pks) def test_a_disabled_rule_converts_nothing_when_a_family_is_confirmed(self): @@ -348,7 +378,7 @@ def test_a_disabled_rule_converts_nothing_when_a_family_is_confirmed(self): self.rule.enabled = False self.rule.save() - self.assertEqual(self._convert(self.base), 0) + self.assertEqual(self._converted(self._convert(self.base)), 0) self._assert_still_flat(self.module, "3") def test_a_disabled_rule_converts_nothing_in_the_whole_rule_batch(self): @@ -356,7 +386,7 @@ def test_a_disabled_rule_converts_nothing_in_the_whole_rule_batch(self): self.rule.enabled = False self.rule.save() - self.assertEqual(self._convert(), 0) + self.assertEqual(self._converted(self._convert()), 0) self._assert_still_flat(self.module, "3") self._assert_still_flat(self.other_module, "4") @@ -364,7 +394,7 @@ def test_applying_the_rule_afterwards_changes_nothing(self): """The family is native now, so ordinary apply is back to renaming it in place — a no-op here.""" self._convert(self.base) - self.assertEqual(apply_rule_to_existing(self.rule), 0) + self.assertEqual(apply_rule_to_existing(self.rule).changed_count, 0) self.assertEqual(self._names(self.module), self._channelized_names("3")) def test_force_reapply_afterwards_changes_nothing(self): @@ -381,7 +411,7 @@ class Ch0MetadataSplitTest(ConversionTestCase): """The ch-0 row splits in two: the physical port stays put, its logical identity moves to channel 1. Everything an operator configured on ``xe-0/0/3:0`` described a 10G channel, not the QSFP cage - that carries it — so addresses, VLANs, description and tags belong on the new channel-1 child, + that carries it, so its VRF, addresses, VLANs, description, and tags belong on the new channel-1 child, while cable, type, module link and mark_connected stay on the row that is now the parent. """ @@ -396,6 +426,7 @@ def setUpTestData(cls): cls.tagged = VLAN.objects.create(vid=200, name="ConvMeta-200", site=cls.device.site) cls.tag = Tag.objects.create(name="ConvMetaTag", slug="convmeta-tag") cls.fhrp_group = FHRPGroup.objects.create(group_id=71, protocol="vrrp2") + cls.vrf = VRF.objects.create(name="ConvMeta VRF", rd="64512:71") def setUp(self): """Install the flat family and load ch-0 with the configuration an operator would have put there.""" @@ -407,6 +438,7 @@ def setUp(self): self.base.mode = InterfaceModeChoices.MODE_TAGGED self.base.untagged_vlan = self.untagged self.base.mark_connected = True + self.base.vrf = self.vrf self.base.custom_field_data = {"conv_note": "keep me"} self.base.save() self.base.tagged_vlans.set([self.tagged]) @@ -441,6 +473,11 @@ def test_the_vlans_and_the_802_1q_mode_move_to_the_child(self): self.assertIsNone(self.parent.untagged_vlan_id) self.assertEqual(list(self.parent.tagged_vlans.all()), []) + def test_the_vrf_moves_to_the_child(self): + """The VRF belongs to the logical channel interface, not the physical parent.""" + self.assertEqual(self.channel.vrf_id, self.vrf.pk) + self.assertIsNone(self.parent.vrf_id) + def test_the_tags_move_to_the_child(self): """Tags drive saved filters and automation aimed at the channel, not at the physical port.""" self.assertEqual([tag.slug for tag in self.channel.tags.all()], ["convmeta-tag"]) @@ -489,9 +526,9 @@ def setUp(self): def _blocked_verdict(self): """Return the verdict for the family in bay 3, asserting it is blocked.""" - verdicts = {verdict["current_name"]: verdict for verdict in self._verdicts()} + verdicts = {verdict.current_name: verdict for verdict in self._verdicts()} verdict = verdicts["xe-0/0/3:0"] - self.assertFalse(verdict["convertible"], verdict) + self.assertFalse(verdict.convertible, verdict) return verdict def _cable_up(self, iface): @@ -499,6 +536,14 @@ def _cable_up(self, iface): peer = Interface.objects.create(device=self.device, name=f"peer-{iface.name}", type=PLAIN_TYPE) Cable.objects.create(a_terminations=[iface], b_terminations=[peer]) + def _plan_for(self, base): + """Return the conversion plan the scan builds for the family *base* heads.""" + module = Module.objects.select_related(*BAY_CHAIN_RELATIONS).get(pk=base.module_id) + variables = build_variables(module.module_bay, device=module.device) + interfaces = list(Interface.objects.filter(module_id=module.pk).order_by("name")) + plans = plan_module_conversions(module, self.rule, variables, interfaces) + return next(plan for plan in plans if plan.base.pk == base.pk) + def _bind_to_another_family(self, name): """Move the interface called *name* into a second, genuine channelized family on the module.""" parent = Interface.objects.create( @@ -513,29 +558,25 @@ def _bind_to_another_family(self, name): return parent def test_a_base_renamed_between_scan_and_convert_refuses_cleanly(self): - """The ch-0 lookup mirrors the sibling checks: a row that vanished mid-race refuses, never crashes.""" - from netbox_interface_name_rules import engine - - family = next(f for f in engine._conversion_families(self.rule) if f.base.pk == self.base.pk) + """Execution revalidates the snapshot it planned against: a row edited mid-race refuses.""" + plan = self._plan_for(self.base) self.base.name = "renamed-by-a-concurrent-request" self.base.save() - reason = engine._convert_family(self.rule, family, commit=True) + outcome = execute_conversion(plan) - self.assertIn("xe-0/0/3:0", reason) + self.assertEqual(outcome.status, FamilyStatus.STALE) self.assertFalse(Interface.objects.filter(module=self.module, channel_id__isnull=False).exists()) def test_a_base_replaced_between_scan_and_convert_refuses_cleanly(self): - """A same-named replacement is not the scanned row: conversion refuses it instead of rewriting a stranger.""" - from netbox_interface_name_rules import engine - - family = next(f for f in engine._conversion_families(self.rule) if f.base.pk == self.base.pk) + """A same-named replacement is not the scanned row: conversion refuses it rather than rewrite a stranger.""" + plan = self._plan_for(self.base) self.base.delete() Interface.objects.create(device=self.device, module=self.module, name="xe-0/0/3:0", type=PLAIN_TYPE) - reason = engine._convert_family(self.rule, family, commit=True) + outcome = execute_conversion(plan) - self.assertIn("xe-0/0/3:0", reason) + self.assertEqual(outcome.status, FamilyStatus.STALE) self._assert_still_flat(self.module, "3") def test_a_cabled_sibling_blocks_the_family(self): @@ -544,14 +585,14 @@ def test_a_cabled_sibling_blocks_the_family(self): verdict = self._blocked_verdict() - self.assertIn("xe-0/0/3:2", verdict["reason"]) - self.assertIn("cable", verdict["reason"].lower()) + self.assertIn("xe-0/0/3:2", verdict.reason) + self.assertIn("cable", verdict.reason.lower()) def test_a_cabled_sibling_is_not_partially_converted(self): """Blocking after the parent is written would leave the family in neither topology.""" self._cable_up(self._iface("xe-0/0/3:2")) - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) self._assert_still_flat(self.module, "3") def test_an_occupied_parent_target_blocks_the_family(self): @@ -560,13 +601,13 @@ def test_an_occupied_parent_target_blocks_the_family(self): verdict = self._blocked_verdict() - self.assertIn("et-0/0/3", verdict["reason"]) + self.assertIn("et-0/0/3", verdict.reason) def test_an_occupied_parent_target_is_not_partially_converted(self): """Nothing is written before every name the family needs is known to be free.""" Interface.objects.create(device=self.device, name="et-0/0/3", type=PLAIN_TYPE) - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) self._assert_still_flat(self.module, "3") def test_a_missing_sibling_blocks_the_family(self): @@ -575,13 +616,22 @@ def test_a_missing_sibling_blocks_the_family(self): verdict = self._blocked_verdict() - self.assertIn("xe-0/0/3:2", verdict["reason"]) + self.assertIn("xe-0/0/3:2", verdict.reason) + + def test_a_missing_sibling_is_still_shown_with_what_it_would_become(self): + """Dropping the family from the page would read as 'nothing here to convert', which is worse.""" + self._iface("xe-0/0/3:2").delete() + + verdict = self._blocked_verdict() + + self.assertEqual(list(verdict.current_names), ["xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:3"]) + self.assertEqual(list(verdict.new_names), ["et-0/0/3", *self._flat_names("3")]) def test_a_missing_sibling_is_not_partially_converted(self): """Converting three of four rows would silently drop a channel from the family.""" self._iface("xe-0/0/3:2").delete() - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) self.assertFalse(Interface.objects.filter(module=self.module, channels__isnull=False).exists()) self.assertFalse(Interface.objects.filter(module=self.module, channel_id__isnull=False).exists()) @@ -597,8 +647,8 @@ def test_a_sibling_that_cannot_be_a_channel_blocks_the_family(self): verdict = self._blocked_verdict() - self.assertIn("xe-0/0/3:1", verdict["reason"]) - self.assertIn("channel", verdict["reason"].lower()) + self.assertIn("xe-0/0/3:1", verdict.reason) + self.assertIn("channel", verdict.reason.lower()) def test_a_sibling_that_cannot_be_a_channel_is_not_partially_converted(self): """The dry run happens inside a rolled-back transaction, so a late failure still writes nothing.""" @@ -606,7 +656,7 @@ def test_a_sibling_that_cannot_be_a_channel_is_not_partially_converted(self): sibling.channels = 2 sibling.save() - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) self.assertEqual(self._names(self.module), self._flat_names("3")) self.assertIsNone(self._iface("xe-0/0/3:0").channels) @@ -616,14 +666,14 @@ def test_a_sibling_bound_to_another_family_blocks_the_family(self): verdict = self._blocked_verdict() - self.assertIn("xe-0/0/3:1", verdict["reason"]) - self.assertIn("et-0/0/9", verdict["reason"]) + self.assertIn("xe-0/0/3:1", verdict.reason) + self.assertIn("et-0/0/9", verdict.reason) def test_a_sibling_bound_to_another_family_is_not_taken_from_it(self): """Converting past it would silently drop a channel from a family nobody asked about.""" parent = self._bind_to_another_family("xe-0/0/3:1") - self.assertEqual(convert_flat_families(self.rule, [self.base.pk]), 0) + self.assertEqual(self._converted(convert_flat_families(self.rule, [self.base.pk])), 0) foreign_child = self._iface("xe-0/0/3:1") self.assertEqual(foreign_child.parent_id, parent.pk) @@ -634,19 +684,19 @@ def test_a_sibling_bound_to_another_family_is_not_taken_from_it(self): def test_a_blocked_family_is_reported_to_the_caller(self): """The Apply view tells the operator how many families it refused, not just that nothing happened.""" self._cable_up(self._iface("xe-0/0/3:2")) - conflicts: list = [] - convert_flat_families(self.rule, [self.base.pk], conflicts=conflicts) + outcome = convert_flat_families(self.rule, [self.base.pk]) - self.assertEqual(len(conflicts), 1) - self.assertEqual(conflicts[0]["interface_pk"], self.base.pk) - self.assertEqual(conflicts[0]["attempted_name"], "et-0/0/3") + self.assertEqual(len(outcome.blocked_families), 1) + blocked = outcome.blocked_families[0] + self.assertEqual(blocked.members[0].interface_pk, self.base.pk) + self.assertEqual(blocked.members[0].target_name, "et-0/0/3") def test_a_blocked_family_does_not_stop_the_batch(self): """One unconvertible family must not cost the operator every other family in the run.""" self._cable_up(self._iface("xe-0/0/3:2")) - self.assertEqual(convert_flat_families(self.rule), 1) + self.assertEqual(self._converted(convert_flat_families(self.rule)), 1) self._assert_still_flat(self.module, "3") self.assertEqual(self._names(self.other_module), self._channelized_names("4")) @@ -688,7 +738,7 @@ def test_the_page_offers_the_conversion(self): self.assertEqual(response.status_code, 200) verdicts = response.context["conversions"] - self.assertEqual([verdict["current_name"] for verdict in verdicts], ["xe-0/0/3:0"]) + self.assertEqual([verdict.current_name for verdict in verdicts], ["xe-0/0/3:0"]) def test_the_page_renders_a_confirm_action_of_its_own(self): """Conversion rewrites rows; it must never ride along on the ordinary apply button.""" @@ -707,7 +757,7 @@ def test_the_page_shows_the_family_it_would_convert(self): def test_the_page_states_where_the_ch0_configuration_goes(self): """The surprising part of the conversion is the one the page has to spell out.""" response = self.client.get(self._url()) - note = response.context["conversions"][0]["metadata_note"] + note = response.context["conversions"][0].metadata_note self.assertIn(escape(note), response.content.decode()) @@ -856,6 +906,19 @@ def test_the_page_says_more_families_are_waiting(self): self.assertIn(f"at most {self.BATCH_LIMIT} per run", content) self.assertIn("Convert as Background Job", content) + def test_more_families_keep_the_background_action_when_the_page_is_blocked(self): + """Later families can convert even when every family in the capped preview is blocked.""" + for position in ("3", "4"): + sibling = self._iface(f"xe-0/0/{position}:2") + peer = Interface.objects.create(device=self.device, name=f"peer-{position}", type=PLAIN_TYPE) + Cable.objects.create(a_terminations=[sibling], b_terminations=[peer]) + + response = self._capped(self.client.get, self._url()) + + self.assertFalse(response.context["conversions_available"]) + self.assertTrue(response.context["conversions_have_more"]) + self.assertIn('value="convert_background"', response.content.decode()) + def test_a_page_that_shows_every_family_flags_nothing(self): """The hint has to mean something, so it cannot be on when the batch really is the whole fleet.""" response = self.client.get(self._url()) @@ -968,24 +1031,24 @@ def _dry_runs(captured): def test_an_unlimited_scan_examines_every_family(self): """The background job's batch is unbounded by design, and so is a scan without a limit.""" - verdicts, has_more = find_convertible_families(self.rule) + preview = find_convertible_families(self.rule) - self.assertEqual(len(verdicts), 3) - self.assertFalse(has_more) + self.assertEqual(len(preview.candidates), 3) + self.assertFalse(preview.has_more) def test_the_scan_stops_at_the_limit_and_says_more_are_waiting(self): """The page has to distinguish 'that is all of them' from 'that is as far as this run went'.""" - verdicts, has_more = find_convertible_families(self.rule, limit=2) + preview = find_convertible_families(self.rule, limit=2) - self.assertEqual(len(verdicts), 2) - self.assertTrue(has_more) + self.assertEqual(len(preview.candidates), 2) + self.assertTrue(preview.has_more) def test_a_limit_the_families_never_reach_reports_nothing_waiting(self): """A limit above the fleet is not a reason to tell the operator families were left out.""" - verdicts, has_more = find_convertible_families(self.rule, limit=5) + preview = find_convertible_families(self.rule, limit=5) - self.assertEqual(len(verdicts), 3) - self.assertFalse(has_more) + self.assertEqual(len(preview.candidates), 3) + self.assertFalse(preview.has_more) def test_the_families_past_the_limit_are_never_dry_run(self): """The dry run is the scan's whole cost: capping the verdict list alone would buy nothing.""" @@ -999,14 +1062,27 @@ def test_the_families_past_the_limit_are_never_dry_run(self): def test_a_blocked_family_counts_against_the_limit(self): """It costs the same dry run as a convertible one, so it has to bound the scan the same way.""" + for position in ("3", "4", "5"): + sibling = self._iface(f"xe-0/0/{position}:2") + peer = Interface.objects.create(device=self.device, name=f"peer-{sibling.name}", type=PLAIN_TYPE) + Cable.objects.create(a_terminations=[sibling], b_terminations=[peer]) + + preview = find_convertible_families(self.rule, limit=2) + + self.assertEqual(len(preview.candidates), 2) + self.assertFalse(any(candidate.convertible for candidate in preview.candidates), preview) + self.assertTrue(preview.has_more) + + def test_an_incomplete_family_counts_against_the_limit(self): + """A family refused before a row is locked still occupies a slot the operator can see.""" for position in ("3", "4", "5"): self._iface(f"xe-0/0/{position}:2").delete() - verdicts, has_more = find_convertible_families(self.rule, limit=2) + preview = find_convertible_families(self.rule, limit=2) - self.assertEqual(len(verdicts), 2) - self.assertFalse(any(verdict["convertible"] for verdict in verdicts), verdicts) - self.assertTrue(has_more) + self.assertEqual(len(preview.candidates), 2) + self.assertFalse(any(candidate.convertible for candidate in preview.candidates), preview) + self.assertTrue(preview.has_more) def test_the_scan_converts_nothing_whatever_the_limit(self): """A capped scan is still a preview; stopping early must not leave a half-converted fleet.""" @@ -1016,6 +1092,82 @@ def test_the_scan_converts_nothing_whatever_the_limit(self): self._assert_still_flat(module, position) +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class ConversionChangelogTest(ConversionTestCase): + """The rows conversion carries onto the new channel are written the way the family write is. + + Addresses and FHRP group assignments are objects an operator owns and audits. Moving them with + a queryset update would relocate them with no validation, no signal and no changelog entry, so + the device history would show the family rewritten and the addresses on it moved by nobody. + """ + + @classmethod + def setUpTestData(cls): + cls.superuser = User.objects.create_superuser( + username="convlog", password=TEST_PASSWORD, email="convlog@example.com" + ) + manufacturer, cls.device = _build_device("ConvLog", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ConvLog-QSFP") + cls.rule = cls._flat_rule(cls.module_type) + cls.fhrp_group = FHRPGroup.objects.create(group_id=81, protocol="vrrp2") + + def setUp(self): + """Address ch-0, give it a first-hop group, then convert the family from the Apply page.""" + self.module, self.bay = self._install(self.module_type, "3") + self._switch_to_channelized() + self.base = self._iface("xe-0/0/3:0") + self.address = IPAddress.objects.create(address="192.0.2.9/24", assigned_object=self.base) + self.assignment = FHRPGroupAssignment.objects.create(group=self.fhrp_group, interface=self.base, priority=10) + self.client.force_login(self.superuser) + url = reverse("plugins:netbox_interface_name_rules:interfacenamerule_apply_detail", kwargs={"pk": self.rule.pk}) + with self.captureOnCommitCallbacks(execute=True): + self.response = self.client.post(url, {"action": "convert", "convert_ids": [str(self.base.pk)]}) + self.channel = self._child(self.module, 1) + + def _changes_for(self, instance): + """Return the changelog entries recorded for *instance*.""" + return ObjectChange.objects.filter( + changed_object_type=ObjectType.objects.get_for_model(instance), + changed_object_id=instance.pk, + ) + + def test_the_conversion_succeeded(self): + """The changelog assertions below only mean something once the family really converted.""" + self.assertEqual(self.response.status_code, 302) + self.assertEqual(self._names(self.module), self._channelized_names("3")) + + def test_the_carried_address_lands_on_the_channel(self): + """The address was reachable over one 10G channel, and still is.""" + self.address.refresh_from_db() + + self.assertEqual(self.address.assigned_object_id, self.channel.pk) + + def test_the_carried_address_is_recorded_in_the_changelog(self): + """An operator auditing the device has to see who moved the address, and when.""" + changes = self._changes_for(self.address) + + self.assertTrue(changes.exists(), "the carried address left no changelog entry") + self.assertEqual(changes.latest("time").postchange_data["assigned_object_id"], self.channel.pk) + + def test_the_carried_fhrp_assignment_lands_on_the_channel(self): + """The group is a gateway on the addressed interface, which is now the channel.""" + self.assignment.refresh_from_db() + + self.assertEqual(self.assignment.interface_id, self.channel.pk) + + def test_the_carried_fhrp_assignment_is_recorded_in_the_changelog(self): + """Same audit trail: a first-hop gateway must not change interface silently.""" + changes = self._changes_for(self.assignment) + + self.assertTrue(changes.exists(), "the carried FHRP group assignment left no changelog entry") + self.assertEqual(changes.latest("time").postchange_data["interface_id"], self.channel.pk) + + def test_the_family_rows_are_recorded_in_the_changelog(self): + """The carried rows are held to the standard the family write already meets.""" + self.assertTrue(self._changes_for(self.channel).exists()) + self.assertTrue(self._changes_for(self._parent(self.module)).exists()) + + @skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) class ConversionJobTest(ConversionTestCase): """The background job runs the batch an operator confirmed for the whole rule.""" @@ -1112,7 +1264,7 @@ def test_the_operator_is_offered_the_conversion_after_switching_the_rule(self): verdicts = self._verdicts() self.assertEqual(len(verdicts), 1) - self.assertTrue(verdicts[0]["convertible"], verdicts[0]["reason"]) + self.assertTrue(verdicts[0].convertible, verdicts[0].reason) self._assert_still_flat(self.module, "3") def test_the_conversion_produces_the_juniper_family(self): @@ -1167,7 +1319,7 @@ def test_the_converted_family_validates(self): @skipIf(supports_channelization(), REQUIRES_NO_CHANNELIZATION) class ConversionWithoutSupportTest(ConversionTestCase): - """Where NetBox has no channel model there is no topology to convert to, so nothing is offered.""" + """Where NetBox has no channel model each flat family reports an unsupported conversion.""" @classmethod def setUpTestData(cls): @@ -1184,25 +1336,33 @@ def setUp(self): self._switch_to_channelized() self.base = self._iface("xe-0/0/3:0") - def test_no_family_is_offered_for_conversion(self): - """There is no channelized topology on this release, so no family can be converted into one.""" - self.assertEqual(self._verdicts(), []) + def test_family_is_reported_as_unsupported(self): + """The scan distinguishes unsupported topology from no family found.""" + verdicts = self._verdicts() + + self.assertEqual(len(verdicts), 1) + self.assertFalse(verdicts[0].convertible) + self.assertIn("cannot model channelized interfaces", verdicts[0].reason) def test_converting_refuses_gracefully_and_says_so(self): """A refusal has to be visible: a silent zero looks exactly like 'nothing needed converting'.""" with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: - converted = convert_flat_families(self.rule, [self.base.pk]) + outcome = convert_flat_families(self.rule, [self.base.pk]) - self.assertEqual(converted, 0) + self.assertEqual(self._converted(outcome), 0) + self.assertEqual(len(outcome.families), 1) + self.assertEqual(outcome.families[0].status, FamilyStatus.UNSUPPORTED) self.assertEqual(self._names(self.module), self._flat_names("3")) self.assertTrue(any("channeliz" in line.lower() for line in logs.output), logs.output) - def test_the_apply_page_has_no_conversion_section(self): - """Offering an action this release cannot perform is worse than not mentioning it.""" + def test_the_apply_page_reports_the_unsupported_family_without_an_action(self): + """The page explains the refusal but does not offer a conversion checkbox.""" self.client.force_login(self.superuser) url = reverse("plugins:netbox_interface_name_rules:interfacenamerule_apply_detail", kwargs={"pk": self.rule.pk}) response = self.client.get(url) - self.assertFalse(response.context["conversions"]) + self.assertEqual(len(response.context["conversions"]), 1) + self.assertContains(response, "Unsupported") + self.assertContains(response, "cannot model channelized interfaces") self.assertNotIn('value="convert"', response.content.decode()) diff --git a/netbox_interface_name_rules/tests/test_device_rules.py b/netbox_interface_name_rules/tests/test_device_rules.py index 80e7bdb..733e0fc 100644 --- a/netbox_interface_name_rules/tests/test_device_rules.py +++ b/netbox_interface_name_rules/tests/test_device_rules.py @@ -113,6 +113,15 @@ def test_no_interfaces_returns_zero(self): result = apply_device_interface_rules(self.device1) self.assertEqual(result, 0) + def test_python_only_filter_written_without_validation_is_skipped(self): + """A legacy filter that RE2 cannot compile must not run through Python's engine.""" + self._make_rule("Gi{vc_position}/{port}", pattern=r"(?=Gi)Gi\d+/\d+") + interface = self._make_interface("Gi0/1") + + self.assertEqual(apply_device_interface_rules(self.device1), 0) + interface.refresh_from_db() + self.assertEqual(interface.name, "Gi0/1") + # ------------------------------------------------------------------ # Successful renames # ------------------------------------------------------------------ @@ -157,12 +166,12 @@ def test_validation_failure_is_logged_and_skipped(self): # assertLogs asserts the warning branch actually fired — without it, "no rule matched" would # also yield result==0 and a preserved name, so the test would pass without covering the path. - with self.assertLogs("netbox_interface_name_rules.engine", level="WARNING") as logs: + with self.assertLogs("netbox_interface_name_rules.family.execution", level="WARNING") as logs: result = apply_device_interface_rules(self.device1) self.assertEqual(result, 0) # validation failed → nothing renamed self.assertTrue( - any("Validation failed renaming device interface" in line for line in logs.output), + any("NetBox rejected rename" in line for line in logs.output), logs.output, ) iface.refresh_from_db() @@ -250,6 +259,18 @@ def test_first_matching_rule_wins(self): iface.refresh_from_db() self.assertEqual(iface.name, "Gi1/1") + def test_first_matching_rule_wins_when_name_is_already_correct(self): + """A no-op from the first rule prevents a less specific rule from renaming the interface.""" + self._make_rule("Gi{vc_position}/{port}", device_type=self.device_type) + self._make_rule("OVERWRITTEN{vc_position}/{port}") + iface = self._make_interface("Gi1/1") + + result = apply_device_interface_rules(self.device1) + + self.assertEqual(result, 0) + iface.refresh_from_db() + self.assertEqual(iface.name, "Gi1/1") + def test_disabled_rule_skipped(self): """Disabled rules are not applied.""" InterfaceNameRule.objects.create( @@ -309,7 +330,7 @@ def test_global_rule_applies_to_any_device_type(self): class DeviceInterfaceEdgeCaseNamesTest(TestCase): - """Test _try_rename_device_interface with edge-case interface names.""" + """Test device-level rules with edge-case interface names.""" @classmethod def setUpTestData(cls): diff --git a/netbox_interface_name_rules/tests/test_documentation.py b/netbox_interface_name_rules/tests/test_documentation.py new file mode 100644 index 0000000..d819546 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_documentation.py @@ -0,0 +1,87 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Test documentation statements that define plugin behavior.""" + +import re +import unittest +from pathlib import Path + +_PROJECT_ROOT = Path(__file__).resolve().parents[2] + + +# _blocking_reason() and the staleness check decide these before _rewrite() runs, so NetBox never sees them. +_PREFLIGHT_REASONS = ( + "missing member", + "stale family", + "occupied parent name", + "another channel family", + "cabled sibling", +) + + +def _conversion_sentences(): + """Return the conversion section as lowercased, whitespace-collapsed sentences.""" + guide = (_PROJECT_ROOT / "docs" / "template-variables.md").read_text() + section = guide.split("### Converting an installed flat family", 1)[1].split("### Converter Offset", 1)[0] + return [sentence for sentence in " ".join(section.lower().split()).split(". ") if sentence] + + +class ConversionDocumentationTest(unittest.TestCase): + """Keep the conversion preflight description consistent with its implementation.""" + + def test_missing_members_are_not_described_as_a_rollback_rejection(self): + normalized = ". ".join(_conversion_sentences()) + + self.assertIn("rejects missing members locally before a conversion transaction starts", normalized) + self.assertNotIn("a missing sibling", normalized) + + def test_preflight_reasons_are_credited_to_the_plugin(self): + sentences = _conversion_sentences() + local = " ".join(s for s in sentences if "preflight" in s or "locally" in s) + + for reason in _PREFLIGHT_REASONS: + with self.subTest(reason=reason): + self.assertIn(reason, local) + + def test_no_preflight_reason_is_credited_to_netbox(self): + for sentence in _conversion_sentences(): + if "netbox" not in sentence: + continue + for reason in _PREFLIGHT_REASONS: + with self.subTest(reason=reason, sentence=sentence): + self.assertNotIn(reason, sentence) + + +class PerformanceDocumentationTest(unittest.TestCase): + """Keep the performance narrative consistent with the committed comparison.""" + + def test_statement_attribution_matches_the_comparison(self): + comparison = (_PROJECT_ROOT / "performance" / "comparisons" / "family-package-vs-existing.md").read_text() + attribution = comparison.split("### Where those statements come from", 1)[1] + changes_by_scenario = {} + for line in attribution.splitlines(): + cells = [cell.strip().strip("`") for cell in line.strip().strip("|").split("|")] + if len(cells) != 5 or not cells[0].startswith("module."): + continue + change = int(cells[4]) + if change: + changes_by_scenario.setdefault(cells[0], {})[cells[1]] = change + + readme = (_PROJECT_ROOT / "performance" / "README.md").read_text() + result = readme.split("## Result of the interface-family comparison", 1)[1] + readme_changes = {} + for line in result.split("Count the statements", 1)[0].splitlines(): + cells = [cell.strip().strip("`") for cell in line.strip().strip("|").split("|")] + if len(cells) == 4 and re.fullmatch(r"[-+]?\d+", cells[3]): + readme_changes[cells[0]] = int(cells[3]) + + narrative = readme.split("No shared-buffer reads were observed", 1)[1].split("That revalidation", 1)[0].lower() + sources = {source for changes in changes_by_scenario.values() for source in changes} + + self.assertTrue(changes_by_scenario) + for source in sorted(sources): + with self.subTest(source=source): + self.assertIn(source.lower(), narrative) + for scenario, changes in changes_by_scenario.items(): + with self.subTest(scenario=scenario): + self.assertEqual(readme_changes.get(scenario.rsplit(".", 1)[-1]), sum(changes.values())) diff --git a/netbox_interface_name_rules/tests/test_engine.py b/netbox_interface_name_rules/tests/test_engine.py index e260b8b..51db6d3 100644 --- a/netbox_interface_name_rules/tests/test_engine.py +++ b/netbox_interface_name_rules/tests/test_engine.py @@ -3,6 +3,7 @@ """Tests for the template evaluation engine (pure functions, no DB).""" from unittest import TestCase +from unittest.mock import patch from netbox_interface_name_rules.engine import evaluate_name_template @@ -39,10 +40,11 @@ def test_channel_variable(self): self.assertEqual(result, "xe-0/0/2:3") def test_arithmetic_expression(self): - result = evaluate_name_template( - "GigabitEthernet{slot}/{8 + 3}", - {"slot": "2", "bay_position": "0", "bay_position_num": "0"}, - ) + with patch("builtins.eval", side_effect=AssertionError("template arithmetic called eval")): + result = evaluate_name_template( + "GigabitEthernet{slot}/{8 + 3}", + {"slot": "2", "bay_position": "0", "bay_position_num": "0"}, + ) self.assertEqual(result, "GigabitEthernet2/11") def test_arithmetic_with_variables(self): diff --git a/netbox_interface_name_rules/tests/test_engine_advanced.py b/netbox_interface_name_rules/tests/test_engine_advanced.py index cc7ab15..f164103 100644 --- a/netbox_interface_name_rules/tests/test_engine_advanced.py +++ b/netbox_interface_name_rules/tests/test_engine_advanced.py @@ -1,8 +1,9 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (C) 2025 Marcin Zieba """Tests for advanced engine functions: find_interfaces_for_rule, apply_rule_to_existing, -has_applicable_interfaces, _find_channel_base, _matching_moduletype_pks, build_variables edges.""" +has_applicable_interfaces, _matching_moduletype_pks, build_variables edges.""" +from types import SimpleNamespace from unittest.mock import patch from dcim.models import ( @@ -24,7 +25,6 @@ from netbox_interface_name_rules.engine import ( _extract_trailing_digits, - _find_channel_base, _matching_moduletype_pks, apply_interface_name_rules, apply_rule_to_existing, @@ -33,6 +33,8 @@ find_interfaces_for_rule, has_applicable_interfaces, ) +from netbox_interface_name_rules.family import FamilyStatus +from netbox_interface_name_rules.family.names import INTERFACE_NAME_CONSTRAINT from netbox_interface_name_rules.models import InterfaceNameRule @@ -214,7 +216,7 @@ def test_platform_filter_scopes_modules(self): def test_template_error_shown_in_new_names(self): """A template that raises ValueError produces '' in new_names.""" - # {vc_position} is undefined for a non-VC device → ValueError at eval time + # {vc_position} is undefined for a non-VC device, so evaluation raises ValueError. rule = InterfaceNameRule.objects.create( module_type=self.module_type, name_template="xe-{vc_position}/0/{bay_position}", @@ -255,7 +257,7 @@ def test_empty_interface_ids_returns_zero(self): name_template="et-0/0/{bay_position}", ) result = apply_rule_to_existing(rule, interface_ids=[]) - self.assertEqual(result, 0) + self.assertEqual(result.changed_count, 0) def test_disabled_rule_returns_zero(self): """Disabled rule returns 0 immediately.""" @@ -267,7 +269,7 @@ def test_disabled_rule_returns_zero(self): module = Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) iface = Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") result = apply_rule_to_existing(rule) - self.assertEqual(result, 0) + self.assertEqual(result.changed_count, 0) # Verify interface name is unchanged iface.refresh_from_db() self.assertEqual(iface.name, "0") @@ -282,7 +284,7 @@ def test_renames_matching_interface(self): iface = Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") result = apply_rule_to_existing(rule) - self.assertEqual(result, 1) + self.assertEqual(result.changed_count, 1) iface.refresh_from_db() self.assertEqual(iface.name, "et-0/0/0") @@ -298,14 +300,14 @@ def test_interface_ids_filter(self): iface1 = Interface.objects.create(device=self.device, module=module1, name="1", type="10gbase-x-sfpp") result = apply_rule_to_existing(rule, interface_ids=[iface0.pk]) - self.assertEqual(result, 1) + self.assertEqual(result.changed_count, 1) iface0.refresh_from_db() iface1.refresh_from_db() self.assertEqual(iface0.name, "et-0/0/0") self.assertEqual(iface1.name, "1") # untouched def test_channel_rule_applies_once_per_module(self): - """Channel rule calls _apply_rule_to_interface once per module, not per interface.""" + """A channel rule plans one family per module, not one per interface.""" rule = InterfaceNameRule.objects.create( module_type=self.module_type, name_template="xe-0/0/{bay_position}:{channel}", @@ -316,7 +318,7 @@ def test_channel_rule_applies_once_per_module(self): Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") result = apply_rule_to_existing(rule) - self.assertEqual(result, 4) + self.assertEqual(result.changed_count, 4) names = sorted(Interface.objects.filter(module=module).values_list("name", flat=True)) self.assertEqual(names, ["xe-0/0/0:0", "xe-0/0/0:1", "xe-0/0/0:2", "xe-0/0/0:3"]) @@ -332,7 +334,7 @@ def test_limit_respected(self): Interface.objects.create(device=self.device, module=module1, name="1", type="10gbase-x-sfpp") result = apply_rule_to_existing(rule, limit=1) - self.assertEqual(result, 1) + self.assertEqual(result.changed_count, 1) def test_regex_rule_applies_to_matching_module_types(self): """Regex rule's module_qs includes all matching module types.""" @@ -345,7 +347,7 @@ def test_regex_rule_applies_to_matching_module_types(self): iface = Interface.objects.create(device=self.device, module=module, name="0", type="100gbase-x-qsfp28") result = apply_rule_to_existing(rule) - self.assertEqual(result, 1) + self.assertEqual(result.changed_count, 1) iface.refresh_from_db() self.assertEqual(iface.name, "Hu0/0/0/0") @@ -412,45 +414,47 @@ def test_invalid_regex_raises_value_error(self): _matching_moduletype_pks("[invalid(") -class FindChannelBaseTest(EngineAdvancedFixtures): - """Test _find_channel_base chooses the right interface for channel rules.""" +class ChannelFamilyBaseTest(EngineAdvancedFixtures): + """Two bases that intend one family's names build it once, through the base that already names it.""" - def test_prefers_already_renamed_ch0_interface(self): - """_find_channel_base prefers an interface already named as channel 0.""" - rule = InterfaceNameRule.objects.create( + def _breakout_rule(self): + """Return a four-channel flat breakout rule whose names ignore the base.""" + return InterfaceNameRule.objects.create( module_type=self.module_type, name_template="xe-0/0/{bay_position}:{channel}", channel_count=4, channel_start=0, ) + + def test_a_half_built_family_is_completed_through_its_own_first_member(self): + """The interface already named channel 0 owns the family; the raw port is left where it is.""" + rule = self._breakout_rule() module = Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) - # One interface is already the ch=0 name, another is still raw - iface_ch0 = Interface.objects.create( - device=self.device, module=module, name="xe-0/0/0:0", type="10gbase-x-sfpp" - ) - iface_raw = Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") + Interface.objects.create(device=self.device, module=module, name="xe-0/0/0:0", type="10gbase-x-sfpp") + Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") - variables = build_variables(self.bay0) - ifaces = sorted([iface_ch0, iface_raw], key=lambda i: i.name) - result = _find_channel_base(rule, ifaces, variables) - self.assertEqual(result, iface_ch0) + outcome = apply_rule_to_existing(rule) - def test_falls_back_to_first_interface_alphabetically(self): - """When no interface matches ch=0, returns first alphabetically (via caller-sorted list).""" - rule = InterfaceNameRule.objects.create( - module_type=self.module_type, - name_template="xe-0/0/{bay_position}:{channel}", - channel_count=4, - channel_start=0, + self.assertEqual(outcome.changed_count, 3) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["0", "xe-0/0/0:0", "xe-0/0/0:1", "xe-0/0/0:2", "xe-0/0/0:3"], ) + + def test_with_no_member_named_yet_the_first_port_builds_the_family(self): + """Nothing names the family, so its first candidate in module order builds it.""" + rule = self._breakout_rule() module = Module.objects.create(device=self.device, module_bay=self.bay1, module_type=self.module_type) - iface_a = Interface.objects.create(device=self.device, module=module, name="1", type="10gbase-x-sfpp") - iface_b = Interface.objects.create(device=self.device, module=module, name="2", type="10gbase-x-sfpp") + Interface.objects.create(device=self.device, module=module, name="1", type="10gbase-x-sfpp") + Interface.objects.create(device=self.device, module=module, name="2", type="10gbase-x-sfpp") + + outcome = apply_rule_to_existing(rule) - variables = build_variables(self.bay1) - ifaces = sorted([iface_a, iface_b], key=lambda i: i.name) - result = _find_channel_base(rule, ifaces, variables) - self.assertEqual(result, iface_a) + self.assertEqual(outcome.changed_count, 4) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["2", "xe-0/0/1:0", "xe-0/0/1:1", "xe-0/0/1:2", "xe-0/0/1:3"], + ) class BuildVariablesEdgesTest(TestCase): @@ -545,7 +549,7 @@ def test_syntax_error_in_expression_raises_value_error(self): evaluate_name_template("{1 + }", {}) def test_unsafe_ast_node_rejected(self): - """AST node not in allowlist (e.g., Name lookup — not a call) raises ValueError.""" + """The arithmetic evaluator rejects a name lookup before it can execute.""" with self.assertRaises(ValueError): evaluate_name_template("{__import__}", {}) @@ -672,7 +676,7 @@ def test_no_op_rename_adds_deprecated_tag(self): name_template="et-0/0/{bay_position}", ) module = Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) - # Create with the final correct name (so _apply_rule_to_interface returns 0 for it) + # Create with the final correct name (so the rule renames nothing for it) # But the name "et-0/0/0" is NOT in raw_names (raw = "0"), so this won't trigger deprecated. # Instead, set force_reapply so it's in unrenamed but produces 0 renames. iface = Interface.objects.create(device=self.device, module=module, name="et-0/0/0", type="10gbase-x-sfpp") @@ -695,12 +699,12 @@ class EngineEvaluateTemplateUnsafeASTTest(TestCase): """Test evaluate_name_template raises for unsafe AST node types (defense-in-depth).""" def test_unsafe_ast_node_raises_valueerror(self): - """An exponent expression passes the char guard but fails the AST allowlist. + """An exponent expression passes the character guard but fails the AST evaluator. ``{2**3}`` is built only from characters the guard regex permits - (digits and ``*``), so it slips past the cheap char check — but - ``ast.parse`` yields a ``Pow`` node, which is not in the arithmetic - allowlist, so the AST walk raises ValueError for real (no mock needed). + (digits and ``*``), so it slips past the cheap character check. The + parser yields a ``Pow`` node, which the recursive arithmetic + evaluator rejects with ValueError. """ from netbox_interface_name_rules.engine import evaluate_name_template @@ -709,12 +713,12 @@ def test_unsafe_ast_node_raises_valueerror(self): # --------------------------------------------------------------------------- -# engine.py — _find_regex_match re.error path (lines 355-356) +# rule_selection.py, invalid regex path # --------------------------------------------------------------------------- class EngineFindRegexMatchErrorTest(TestCase): - """Test that _find_regex_match silently skips rules with invalid regex patterns.""" + """Test that rule selection silently skips stored patterns RE2 cannot compile.""" @classmethod def setUpTestData(cls): @@ -734,14 +738,13 @@ def setUpTestData(cls): cls.bad_regex_rule.save() def test_invalid_regex_pattern_is_skipped_not_raised(self): - """_find_regex_match silently skips rules whose module_type_pattern is not valid regex. + """_find_regex_match silently skips rules whose module_type_pattern is not valid RE2. The bad-pattern rule is inserted directly via save() to bypass clean() - validation. _find_regex_match wraps re.fullmatch in a try/except re.error - so an invalid pattern just moves on to the next candidate instead of - propagating the exception to the caller (lines 355-356). + validation. Rule loading compiles through the RE2 seam, so an invalid + pattern moves on to the next candidate instead of propagating. """ - from netbox_interface_name_rules.engine import _find_regex_match + from netbox_interface_name_rules.rule_selection import _find_regex_match candidates = [(None, None, None)] result = _find_regex_match("RegX-GOOD", candidates) @@ -769,7 +772,7 @@ def test_invalid_regex_rule_returns_false(self): A rule with an invalid ``module_type_pattern`` (inserted via save() to bypass clean() validation) makes ``_matching_moduletype_pks`` raise a real ValueError inside ``find_interfaces_for_rule``; has_applicable_interfaces - catches (ValueError, re.error) and returns False — no mock required. + catches ValueError and returns False. No mock is required. """ from netbox_interface_name_rules.engine import has_applicable_interfaces @@ -787,12 +790,12 @@ def test_invalid_regex_rule_returns_false(self): # --------------------------------------------------------------------------- -# engine.py — _find_channel_base ValueError path (lines 535-536) +# engine.py: a breakout rule whose family names cannot be evaluated # --------------------------------------------------------------------------- -class EngineFindChannelBaseValueErrorTest(TestCase): - """Test _find_channel_base skips ValueError from template evaluation (lines 535-536).""" +class BreakoutTemplateValueErrorTest(TestCase): + """A breakout template that cannot be evaluated leaves every candidate base where it is.""" @classmethod def setUpTestData(cls): @@ -807,30 +810,29 @@ def setUpTestData(cls): cls.device = Device.objects.create(name="chanx-dev-01", device_type=cls.device_type, role=role, site=site) cls.bay = ModuleBay.objects.get(device=cls.device, name="CXBay 0") - def test_find_channel_base_valueerror_skips_to_fallback(self): - """_find_channel_base catches a real template ValueError and falls back to ifaces[0]. - - ``{undefined_var}`` is not in the variable dict, so evaluate_name_template - raises ValueError for every interface; _find_channel_base swallows each one - and returns the first interface as the base — no mock required. - """ - from netbox_interface_name_rules.engine import _find_channel_base - - rule = InterfaceNameRule( + def test_an_unevaluable_template_builds_nothing_and_renames_nothing(self): + """``{undefined_var}`` is not a naming variable, so every family reports the failure.""" + rule = InterfaceNameRule.objects.create( module_type=self.module_type, - name_template="{undefined_var}:{channel}", # undefined_var → real ValueError on eval + name_template="{undefined_var}:{channel}", # The undefined variable raises ValueError. channel_count=2, channel_start=0, ) module = Module.objects.create(device=self.device, module_bay=self.bay, module_type=self.module_type) - iface0 = Interface.objects.create(device=self.device, module=module, name="Eth0", type="100gbase-x-qsfp28") - iface1 = Interface.objects.create(device=self.device, module=module, name="Eth1", type="100gbase-x-qsfp28") - ifaces = [iface0, iface1] - variables = {"bay_position": "0", "slot": "0", "sfp_slot": "0"} + Interface.objects.create(device=self.device, module=module, name="Eth0", type="100gbase-x-qsfp28") + Interface.objects.create(device=self.device, module=module, name="Eth1", type="100gbase-x-qsfp28") + + outcome = apply_rule_to_existing(rule) - result = _find_channel_base(rule, ifaces, variables) - # Falls back to ifaces[0] after ValueError on every interface - self.assertEqual(result, iface0) + self.assertEqual(outcome.changed_count, 0) + self.assertEqual( + {member.status for member in outcome.skipped_members}, + {FamilyStatus.FAILED}, + ) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["Eth0", "Eth1"], + ) # --------------------------------------------------------------------------- @@ -913,7 +915,7 @@ def test_channel_rule_no_interfaces_skips(self): # Module with NO interfaces Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) count = apply_rule_to_existing(rule) - self.assertEqual(count, 0) + self.assertEqual(count.changed_count, 0) def test_channel_rule_id_set_filters_base(self): """Channel rule with id_set skips when base_iface.pk not in id_set (line 753).""" @@ -929,7 +931,7 @@ def test_channel_rule_id_set_filters_base(self): iface = Interface.objects.create(device=self.device, module=module, name="Eth99", type="100gbase-x-qsfp28") # Pass an id_set that does NOT include iface.pk count = apply_rule_to_existing(rule, interface_ids=[iface.pk + 9999]) - self.assertEqual(count, 0) + self.assertEqual(count.changed_count, 0) # Interface should be unchanged iface.refresh_from_db() self.assertEqual(iface.name, "Eth99") @@ -1003,12 +1005,12 @@ def test_full_clean_exception_skips_interface(self): # --------------------------------------------------------------------------- -# engine.py — _channel_rule_entry ValueError path (lines 618-619) +# engine.py: preview of a breakout rule whose template cannot be evaluated # --------------------------------------------------------------------------- -class EngineChannelRuleEntryValueErrorTest(TestCase): - """Test _channel_rule_entry handles ValueError from template evaluation (lines 618-619).""" +class PreviewTemplateErrorTest(TestCase): + """A breakout rule the preview cannot evaluate reports the error instead of a name.""" @classmethod def setUpTestData(cls): @@ -1023,42 +1025,32 @@ def setUpTestData(cls): cls.device = Device.objects.create(name="chrulex-dev-01", device_type=device_type, role=role, site=site) cls.bay = ModuleBay.objects.get(device=cls.device, name="CRBay 0") - def test_valueerror_in_template_sets_error_name(self): - """_channel_rule_entry stores an ```` placeholder when the template can't be evaluated. - - A template referencing an undefined variable raises ValueError in - evaluate_name_template for real (no mock), so the loop that builds - expected_names catches it and collapses to a single error-placeholder - entry (lines 618-619). The result dict is returned because the - placeholder is not in existing_names. - """ - from netbox_interface_name_rules.engine import _channel_rule_entry - - rule = InterfaceNameRule( + def test_an_unevaluable_template_previews_one_error_placeholder(self): + """An undefined variable raises for real, so the family previews as a single placeholder.""" + rule = InterfaceNameRule.objects.create( module_type=self.module_type, - name_template="{base}:{channel}:{undefined_var}", # undefined_var → real ValueError on eval + name_template="{base}:{channel}:{undefined_var}", # The undefined variable raises ValueError. channel_count=2, channel_start=0, ) module = Module.objects.create(device=self.device, module_bay=self.bay, module_type=self.module_type) - iface = Interface.objects.create(device=self.device, module=module, name="Eth0", type="100gbase-x-qsfp28") - variables = {"bay_position": "0", "slot": "0", "sfp_slot": "0"} + Interface.objects.create(device=self.device, module=module, name="Eth0", type="100gbase-x-qsfp28") - result = _channel_rule_entry(rule, module, [iface], variables) + results, total = find_interfaces_for_rule(rule) - # The ValueError collapses expected_names to one "" placeholder. - self.assertIsNotNone(result) - self.assertEqual(len(result["new_names"]), 1) - self.assertTrue(result["new_names"][0].startswith("= 2: + raise ValueError("forced failure on the second family") + return execute(plan) + return _side_effect + + def test_a_failing_plain_rename_leaves_the_earlier_module_renamed(self): rule = InterfaceNameRule.objects.create( module_type=self.module_type, name_template="et-0/0/{bay_position}", @@ -1245,27 +1209,19 @@ def test_exception_on_second_call_is_swallowed(self): iface0 = Interface.objects.create(device=self.device, module=module0, name="0", type="10gbase-x-sfpp") iface1 = Interface.objects.create(device=self.device, module=module1, name="1", type="10gbase-x-sfpp") - call_count = [0] - - def _side_effect(*args, **kwargs): - call_count[0] += 1 - if call_count[0] >= 2: - raise ValueError("forced failure on second call") - return _real_fn(*args, **kwargs) - - with patch("netbox_interface_name_rules.engine._apply_rule_to_interface", side_effect=_side_effect): + with patch( + "netbox_interface_name_rules.family.batch.execute_family_plan", + side_effect=self._failing_after_the_first_family(), + ): count = apply_rule_to_existing(rule) - self.assertEqual(count, 1) + self.assertEqual(count.changed_count, 1) iface0.refresh_from_db() iface1.refresh_from_db() self.assertEqual(iface0.name, "et-0/0/0") # first was renamed self.assertEqual(iface1.name, "1") # second was skipped - def test_exception_on_channel_call_is_swallowed(self): - """Exception inside the channel-rule branch (lines 756-764) is logged and loop continues.""" - from netbox_interface_name_rules.engine import _apply_rule_to_interface as _real_fn - + def test_a_failing_breakout_family_leaves_the_earlier_module_built(self): rule = InterfaceNameRule.objects.create( module_type=self.module_type, name_template="xe-0/0/{bay_position}:{channel}", @@ -1277,21 +1233,16 @@ def test_exception_on_channel_call_is_swallowed(self): iface0 = Interface.objects.create(device=self.device, module=module0, name="0", type="10gbase-x-sfpp") Interface.objects.create(device=self.device, module=module1, name="1", type="10gbase-x-sfpp") - call_count = [0] - - def _side_effect(*args, **kwargs): - call_count[0] += 1 - if call_count[0] >= 2: - raise ValueError("forced channel failure") - return _real_fn(*args, **kwargs) - - with patch("netbox_interface_name_rules.engine._apply_rule_to_interface", side_effect=_side_effect): + with patch( + "netbox_interface_name_rules.family.batch.execute_family_plan", + side_effect=self._failing_after_the_first_family(), + ): count = apply_rule_to_existing(rule) - # First module produced channels; second raised and was skipped - self.assertGreaterEqual(count, 1) + self.assertEqual(count.changed_count, 2) iface0.refresh_from_db() self.assertEqual(iface0.name, "xe-0/0/0:0") + self.assertEqual(Interface.objects.get(module=module1).name, "1") # --------------------------------------------------------------------------- @@ -1400,12 +1351,12 @@ def test_tag_getorcreate_exception_is_swallowed(self): # --------------------------------------------------------------------------- -# engine.py — DB save exception rollback in _try_rename_device_interface (lines 159-168) +# Device interface save exception rollback # --------------------------------------------------------------------------- class DeviceInterfaceSaveExceptionTest(TestCase): - """Test _try_rename_device_interface rolls back name on iface.save() failure (lines 159-168).""" + """Test the device rule path rolls back the name when an interface save fails.""" @classmethod def setUpTestData(cls): @@ -1423,11 +1374,11 @@ def setUpTestData(cls): vc_position=1, ) - def test_save_exception_restores_name_and_returns_false(self): - """When iface.save() raises, name is rolled back to old_name and False returned (lines 159-168).""" - from netbox_interface_name_rules.engine import _try_rename_device_interface + def test_save_exception_restores_name_and_returns_zero(self): + """Restore the old name and return zero when an interface save fails.""" + from netbox_interface_name_rules.engine import apply_device_interface_rules - rule = InterfaceNameRule.objects.create( + InterfaceNameRule.objects.create( applies_to_device_interfaces=True, name_template="xe-{vc_position}/{port}", ) @@ -1436,9 +1387,10 @@ def test_save_exception_restores_name_and_returns_false(self): from django.db import IntegrityError with patch.object(Interface, "save", side_effect=IntegrityError("disk full")): - result = _try_rename_device_interface(rule, iface, "1", self.device, set()) + result = apply_device_interface_rules(self.device) - self.assertFalse(result) + iface.refresh_from_db() + self.assertEqual(result, 0) self.assertEqual(iface.name, "Gi0/1") # rolled back @@ -1545,26 +1497,6 @@ def test_longer_pattern_wins_over_shorter(self): self.assertEqual(matched, rule_long) -# --------------------------------------------------------------------------- -# engine.py — _find_channel_base with empty interface list -# --------------------------------------------------------------------------- - - -class FindChannelBaseEmptyIfacesTest(TestCase): - """Test _find_channel_base handles empty interface list gracefully.""" - - def test_empty_ifaces_returns_none(self): - """_find_channel_base returns None when ifaces is empty.""" - rule = InterfaceNameRule( - name_template="port{bay_position}:{channel}", - channel_start=0, - channel_count=2, - ) - variables = {"bay_position": "0", "bay_position_num": "0", "slot": "0"} - result = _find_channel_base(rule, [], variables) - self.assertIsNone(result) - - # --------------------------------------------------------------------------- # engine.py — find_interfaces_for_rule uses set for processed_pks # --------------------------------------------------------------------------- @@ -1597,18 +1529,18 @@ def test_multiple_modules_all_counted(self): class BreakoutTransactionRollbackTest(EngineAdvancedFixtures): - """Test that _apply_rule_to_interface rolls back on mid-breakout failure.""" + """The install path writes a breakout family whole or not at all.""" def test_partial_breakout_rolls_back(self): """If channel 2 fails validation, channels 0–1 are rolled back too.""" - rule = InterfaceNameRule.objects.create( + InterfaceNameRule.objects.create( module_type=self.module_type, name_template="Hu0/0/0/{bay_position}:{channel}", channel_count=4, channel_start=0, ) module = Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) - iface = Interface.objects.create(device=self.device, module=module, name="0", type="100gbase-x-qsfp28") + Interface.objects.create(device=self.device, module=module, name="0", type="100gbase-x-qsfp28") original_full_clean = Interface.full_clean call_count = [0] @@ -1621,18 +1553,11 @@ def failing_full_clean(self_iface, *args, **kwargs): raise ValidationError("simulated failure") return original_full_clean(self_iface, *args, **kwargs) - from netbox_interface_name_rules.engine import _apply_rule_to_interface - - variables = build_variables(module.module_bay, device=module.device) - variables["base"] = iface.name - - from django.core.exceptions import ValidationError - with patch.object(Interface, "full_clean", failing_full_clean): - with self.assertRaises(ValidationError): - _apply_rule_to_interface(rule, iface, variables, module) + renamed = apply_interface_name_rules(module, module.module_bay) - # Transaction rolled back — only the original interface remains + # The transaction rolled back. Only the original interface remains with its original name. + self.assertEqual(renamed, 0) iface_names = list(Interface.objects.filter(module=module).values_list("name", flat=True)) self.assertEqual(iface_names, ["0"]) @@ -1796,7 +1721,7 @@ def test_install_path_skips_collision_without_raising(self): self.assertFalse(rule.tags.filter(slug="potentially-deprecated").exists()) def test_apply_rule_to_existing_records_conflict_and_continues(self): - """A taken target name is recorded in *conflicts* and skipped; the batch keeps going.""" + """A taken target name is reported as a skipped member; the batch keeps going.""" rule = InterfaceNameRule.objects.create( module_type=self.module_type, name_template="et-0/0/{bay_position}", @@ -1805,12 +1730,10 @@ def test_apply_rule_to_existing_records_conflict_and_continues(self): module = Module.objects.create(device=self.device, module_bay=self.bay0, module_type=self.module_type) iface = Interface.objects.create(device=self.device, module=module, name="0", type="10gbase-x-sfpp") - conflicts: list = [] - count = apply_rule_to_existing(rule, conflicts=conflicts) + count = apply_rule_to_existing(rule) - self.assertEqual(count, 0) - self.assertEqual(len(conflicts), 1) - self.assertEqual(conflicts[0]["attempted_name"], "et-0/0/0") + self.assertEqual(count.changed_count, 0) + self.assertEqual([member.target_name for member in count.skipped_members], ["et-0/0/0"]) iface.refresh_from_db() self.assertEqual(iface.name, "0") @@ -1841,11 +1764,9 @@ def test_install_path_save_race_skips_one_interface_without_aborting_batch(self) """A post-check save race on one interface is logged + skipped; the rest still rename. The collision pre-check closes the common case, but a concurrent insert can - still win between the check and the save — a true race we cannot reproduce - deterministically, so the IntegrityError is injected on the first save only - (everything else is a real rule / real interfaces / the real install path). - Without the per-interface guard in apply_interface_name_rules this exception - propagates and aborts the whole batch. + still win between the check and the save, a true race we cannot reproduce + deterministically. The first save injects the PostgreSQL constraint diagnostic + that identifies this race. Everything else uses the real install path. """ from django.db import IntegrityError @@ -1863,7 +1784,9 @@ def test_install_path_save_race_skips_one_interface_without_aborting_batch(self) def flaky_save(self_iface, *args, **kwargs): calls["n"] += 1 if calls["n"] == 1: # first interface loses the race - raise IntegrityError("duplicate key value violates unique constraint") + cause = Exception("duplicate key value violates unique constraint") + cause.diag = SimpleNamespace(constraint_name=INTERFACE_NAME_CONSTRAINT) + raise IntegrityError("duplicate key value violates unique constraint") from cause return real_save(self_iface, *args, **kwargs) with patch.object(Interface, "save", flaky_save): @@ -1891,8 +1814,7 @@ def test_idempotent_breakout_reapply_records_no_conflict(self): first = apply_interface_name_rules(module, self.bay0) self.assertEqual(first, 4) - conflicts: list = [] - second = apply_rule_to_existing(rule, conflicts=conflicts) - self.assertEqual(second, 0) - self.assertEqual(conflicts, []) # its own existing channels are NOT conflicts + second = apply_rule_to_existing(rule) + self.assertEqual(second.changed_count, 0) + self.assertEqual(second.skipped_members, ()) # its own existing channels are NOT conflicts self.assertEqual(Interface.objects.filter(module=module).count(), 4) diff --git a/netbox_interface_name_rules/tests/test_installed_families.py b/netbox_interface_name_rules/tests/test_installed_families.py new file mode 100644 index 0000000..8927eb4 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_installed_families.py @@ -0,0 +1,782 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for installed interface-family plans.""" + +from dataclasses import FrozenInstanceError, replace +from unittest import skipUnless +from unittest.mock import patch + +from dcim.choices import InterfaceTypeChoices +from dcim.models import ( + Device, + DeviceRole, + DeviceType, + Interface, + InterfaceTemplate, + Manufacturer, + Module, + ModuleBay, + ModuleBayTemplate, + ModuleType, + Site, + VirtualChassis, +) +from django.db import DEFAULT_DB_ALIAS, IntegrityError, connection, connections +from django.test import TestCase +from django.test.utils import CaptureQueriesContext + +from netbox_interface_name_rules.choices import BreakoutModeChoices +from netbox_interface_name_rules.engine import ( + apply_interface_name_rules, + build_variables, + supports_channelization, + supports_vc_position_token, +) +from netbox_interface_name_rules.family import ( + FamilyStatus, + FamilyTopology, + execute_installed_plan_set, + plan_installed_families, +) +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.naming import evaluate_name_template +from netbox_interface_name_rules.tests.out_of_band import rename_out_of_band + +CHANNEL_TYPE = getattr(InterfaceTypeChoices, "TYPE_CHANNEL", "channel") +PARENT_TYPE = InterfaceTypeChoices.TYPE_40GE_QSFP_PLUS +PLAIN_TYPE = InterfaceTypeChoices.TYPE_10GE_SFP_PLUS +REQUIRES_CHANNELIZATION = "requires a NetBox that models channelized interfaces" + + +class InstalledFlatFamilyPlanningTest(TestCase): + """Plan installed flat breakout families from real NetBox rows.""" + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="FamilyPlanMfg", slug="family-plan-mfg") + device_type = DeviceType.objects.create( + manufacturer=manufacturer, + model="FAMILY-PLAN-DEVICE", + slug="family-plan-device", + ) + ModuleBayTemplate.objects.create(device_type=device_type, name="Bay 7", position="7") + cls.module_type = ModuleType.objects.create( + manufacturer=manufacturer, + model="FAMILY-PLAN-QSFP", + part_number="FAMILY-PLAN-QSFP", + ) + InterfaceTemplate.objects.create( + module_type=cls.module_type, + name="{module}", + type="100gbase-x-qsfp28", + ) + role = DeviceRole.objects.create(name="FamilyPlanRole", slug="family-plan-role") + site = Site.objects.create(name="FamilyPlanSite", slug="family-plan-site") + cls.device = Device.objects.create( + name="family-plan-device-01", + device_type=device_type, + role=role, + site=site, + ) + cls.bay = ModuleBay.objects.get(device=cls.device, name="Bay 7") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{base}:{channel}", + channel_count=2, + channel_start=0, + ) + + def test_complete_flat_family_has_one_immutable_plan_with_every_snapshot(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + self.assertEqual(apply_interface_name_rules(module, self.bay), 2) + + plan_set = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + + self.assertEqual(len(plan_set.plans), 1) + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.FLAT) + self.assertEqual( + [member.snapshot.name for member in plan.members], + ["xe-0/0/7:0", "xe-0/0/7:1"], + ) + self.assertEqual( + {member.snapshot.pk for member in plan.members}, + set(Interface.objects.filter(module=module).values_list("pk", flat=True)), + ) + with self.assertRaises(FrozenInstanceError): + plan.module_id = 0 + + def _current_family_plan(self): + """Install and plan one complete flat family without version-specific tokens.""" + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + self.assertEqual(apply_interface_name_rules(module, self.bay), 2) + plan_set = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + return module, plan_set + + @staticmethod + def _retarget(plan_set, prefix): + """Return *plan_set* with deterministic new member targets.""" + plans = tuple( + replace( + plan, + members=tuple( + replace(member, target_name=f"{prefix}-{plan_index}-{member_index}") + for member_index, member in enumerate(plan.members) + ), + ) + for plan_index, plan in enumerate(plan_set.plans) + ) + return replace(plan_set, plans=plans) + + def test_current_flat_family_execution_locks_and_renames_every_member(self): + module, plan_set = self._current_family_plan() + plan_set = self._retarget(plan_set, "renamed") + + with CaptureQueriesContext(connection) as queries: + result = execute_installed_plan_set(plan_set) + + self.assertEqual(result.families[0].status, FamilyStatus.CHANGED) + self.assertEqual(result.changed_count, 2) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["renamed-0-0", "renamed-0-1"], + ) + lock_queries = [query["sql"] for query in queries.captured_queries if "FOR UPDATE" in query["sql"]] + self.assertTrue(lock_queries) + self.assertTrue(all("ORDER BY" in query for query in lock_queries)) + + def test_current_flat_family_execution_propagates_unrelated_integrity_failure(self): + module, plan_set = self._current_family_plan() + plan_set = self._retarget(plan_set, "renamed") + + def reject_interface_update(execute, sql, params, many, context): + if sql.lstrip().startswith('UPDATE "dcim_interface"'): + raise IntegrityError("injected current-family database failure") + return execute(sql, params, many, context) + + with connection.execute_wrapper(reject_interface_update): + with self.assertRaisesMessage(IntegrityError, "injected current-family database failure"): + execute_installed_plan_set(plan_set) + + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["xe-0/0/7:0", "xe-0/0/7:1"], + ) + + def test_current_flat_family_collision_blocks_each_occupied_target(self): + module, plan_set = self._current_family_plan() + plan_set = self._retarget(plan_set, "occupied") + for member in plan_set.plans[0].members: + Interface.objects.create( + device=self.device, + name=member.target_name, + type=PLAIN_TYPE, + ) + + result = execute_installed_plan_set(plan_set) + + self.assertEqual(result.families[0].status, FamilyStatus.BLOCKED) + self.assertEqual( + {member.status for member in result.families[0].members}, + {FamilyStatus.BLOCKED}, + ) + self.assertEqual(result.changed_count, 0) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["xe-0/0/7:0", "xe-0/0/7:1"], + ) + + def test_failed_precondition_returns_member_facts_without_changes(self): + module, plan_set = self._current_family_plan() + plan = replace( + plan_set.plans[0], + precondition_status=FamilyStatus.FAILED, + precondition_reason="failed to evaluate family targets", + ) + + result = execute_installed_plan_set(replace(plan_set, plans=(plan,))) + + self.assertEqual(result.families[0].status, FamilyStatus.FAILED) + self.assertEqual( + {member.status for member in result.families[0].members}, + {FamilyStatus.FAILED}, + ) + self.assertEqual(result.changed_count, 0) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["xe-0/0/7:0", "xe-0/0/7:1"], + ) + + def test_incomplete_current_flat_family_is_not_planned(self): + module, plan_set = self._current_family_plan() + Interface.objects.filter(pk=plan_set.plans[0].members[1].snapshot.pk).delete() + + replanned = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + + self.assertEqual(replanned.plans, ()) + + def test_simple_rule_has_no_flat_family_plan(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + self.rule.channel_count = 0 + + plan_set = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + + self.assertEqual(plan_set.plans, ()) + + def test_automatic_reapplication_propagates_planner_failures(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + + with ( + patch( + "netbox_interface_name_rules.engine.family_ops.plan_module_families", + side_effect=TypeError("planner defect"), + ), + self.assertRaisesMessage(TypeError, "planner defect"), + ): + apply_interface_name_rules(module, self.bay, force_reapply=True) + + def test_changed_member_makes_the_plan_stale_without_partial_execution(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + apply_interface_name_rules(module, self.bay) + plan_set = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + changed_pk = plan_set.plans[0].members[1].snapshot.pk + rename_out_of_band(Interface.objects.get(pk=changed_pk), "operator-change") + + result = execute_installed_plan_set(plan_set) + + self.assertEqual(result.families[0].status, FamilyStatus.STALE) + self.assertEqual( + {member.status for member in result.families[0].members}, + {FamilyStatus.STALE}, + ) + self.assertEqual(result.changed_count, 0) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["operator-change", "xe-0/0/7:0"], + ) + + def test_deleted_member_makes_the_complete_family_plan_stale(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + apply_interface_name_rules(module, self.bay) + plan_set = plan_installed_families( + module, + self.rule, + build_variables(self.bay, device=self.device), + ) + Interface.objects.filter(pk=plan_set.plans[0].members[1].snapshot.pk).delete() + + result = execute_installed_plan_set(plan_set) + + self.assertEqual(result.families[0].status, FamilyStatus.STALE) + self.assertEqual(result.changed_count, 0) + self.assertEqual( + list(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["xe-0/0/7:0"], + ) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_complete_historical_family_maps_to_current_raw_template_names(self): + _module, plan_set = self._historical_family_plan() + + self.assertEqual(len(plan_set.plans), 1) + self.assertEqual( + [member.snapshot.name for member in plan_set.plans[0].members], + ["brk-xe-1/0/7:0", "brk-xe-1/0/7:1"], + ) + self.assertEqual( + [member.target_name for member in plan_set.plans[0].members], + ["brk-xe-2/0/7:0", "brk-xe-2/0/7:1"], + ) + + def _historical_family_plan(self): + """Install a flat family at VC position 1 and plan it at position 2.""" + module, rule = self._historical_family_state() + plan_set = plan_installed_families( + module, + rule, + build_variables(module.module_bay, device=module.device), + ) + return module, plan_set + + def _historical_family_state(self, name_template="brk-{base}:{channel}"): + """Return a flat family installed at VC position 1 after moving to position 2.""" + virtual_chassis = VirtualChassis.objects.create(name="family-plan-vc") + self.device.virtual_chassis = virtual_chassis + self.device.vc_position = 1 + self.device.save() + token_type = ModuleType.objects.create( + manufacturer=self.module_type.manufacturer, + model="FAMILY-PLAN-VC-QSFP", + part_number="FAMILY-PLAN-VC-QSFP", + ) + InterfaceTemplate.objects.create( + module_type=token_type, + name="xe-{vc_position:0}/0/{module}", + type="100gbase-x-qsfp28", + ) + rule = InterfaceNameRule.objects.create( + module_type=token_type, + name_template=name_template, + channel_count=2, + channel_start=0, + ) + module = Module.objects.create(device=self.device, module_bay=self.bay, module_type=token_type) + apply_interface_name_rules(module, self.bay) + expected = [ + evaluate_name_template(name_template, {"base": "xe-1/0/7", "channel": str(channel)}) for channel in range(2) + ] + self.assertEqual(sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), expected) + self.device.vc_position = 2 + self.device.save() + module = Module.objects.select_related("device", "module_bay").get(pk=module.pk) + return module, rule + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_execution_locks_and_renames_the_complete_flat_family(self): + module, plan_set = self._historical_family_plan() + + with CaptureQueriesContext(connection) as queries: + result = execute_installed_plan_set(plan_set) + + self.assertEqual(result.families[0].status, FamilyStatus.CHANGED) + self.assertEqual( + [member.status for member in result.families[0].members], + [FamilyStatus.CHANGED, FamilyStatus.CHANGED], + ) + self.assertEqual(result.changed_count, 2) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["brk-xe-2/0/7:0", "brk-xe-2/0/7:1"], + ) + lock_queries = [query["sql"] for query in queries.captured_queries if "FOR UPDATE" in query["sql"]] + self.assertTrue(lock_queries) + self.assertTrue(all("ORDER BY" in query for query in lock_queries)) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_unrelated_integrity_failure_propagates_and_rolls_back_the_family(self): + module, plan_set = self._historical_family_plan() + + def reject_interface_update(execute, sql, params, many, context): + if sql.lstrip().startswith('UPDATE "dcim_interface"'): + raise IntegrityError("injected non-uniqueness database failure") + return execute(sql, params, many, context) + + with connection.execute_wrapper(reject_interface_update): + with self.assertRaisesMessage(IntegrityError, "injected non-uniqueness database failure"): + execute_installed_plan_set(plan_set) + + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["brk-xe-1/0/7:0", "brk-xe-1/0/7:1"], + ) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_automatic_reapplication_uses_the_installed_family_executor(self): + module, _rule = self._historical_family_state("{base}:{channel}") + + with CaptureQueriesContext(connection) as queries: + renamed = apply_interface_name_rules(module, module.module_bay, force_reapply=True) + + self.assertEqual(renamed, 2) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["xe-2/0/7:0", "xe-2/0/7:1"], + ) + self.assertTrue(any("FOR UPDATE" in query["sql"] for query in queries.captured_queries)) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_forced_reapplication_preserves_a_wrapped_historical_family(self): + module, _rule = self._historical_family_state() + + renamed = apply_interface_name_rules(module, module.module_bay, force_reapply=True) + + self.assertEqual(renamed, 0) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["brk-xe-1/0/7:0", "brk-xe-1/0/7:1"], + ) + + @skipUnless( + supports_channelization() and supports_vc_position_token(), + "requires NetBox channelization and virtual-chassis position templates", + ) + def test_normal_reapplication_preserves_a_wrapped_historical_family(self): + module, _rule = self._historical_family_state() + + renamed = apply_interface_name_rules(module, module.module_bay) + + self.assertEqual(renamed, 0) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["brk-xe-1/0/7:0", "brk-xe-1/0/7:1"], + ) + + def _two_historical_family_state(self): + """Return two complete historical flat families and their current rule context.""" + virtual_chassis = VirtualChassis.objects.create(name="family-plan-two-vc") + self.device.virtual_chassis = virtual_chassis + self.device.vc_position = 1 + self.device.save() + module_type = ModuleType.objects.create( + manufacturer=self.module_type.manufacturer, + model="FAMILY-PLAN-TWO-QSFP", + part_number="FAMILY-PLAN-TWO-QSFP", + ) + for suffix in ("a", "b"): + InterfaceTemplate.objects.create( + module_type=module_type, + name=f"xe-{{vc_position:0}}/0/{{module}}-{suffix}", + type="100gbase-x-qsfp28", + ) + rule = InterfaceNameRule.objects.create( + module_type=module_type, + name_template="brk-{base}:{channel}", + channel_count=2, + channel_start=0, + ) + module = Module.objects.create(device=self.device, module_bay=self.bay, module_type=module_type) + self.assertEqual(apply_interface_name_rules(module, self.bay), 4) + self.device.vc_position = 2 + self.device.save() + module = Module.objects.select_related("device", "module_bay").get(pk=module.pk) + return module, rule + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_incomplete_flat_family_is_rejected_without_hiding_a_complete_family(self): + module, rule = self._two_historical_family_state() + Interface.objects.filter(module=module, name="brk-xe-1/0/7-a:1").delete() + + plan_set = plan_installed_families( + module, + rule, + build_variables(module.module_bay, device=module.device), + ) + + self.assertEqual(len(plan_set.plans), 1) + self.assertEqual( + [member.snapshot.name for member in plan_set.plans[0].members], + ["brk-xe-1/0/7-b:0", "brk-xe-1/0/7-b:1"], + ) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_blocked_member_does_not_stop_an_unrelated_flat_family(self): + module, rule = self._two_historical_family_state() + Interface.objects.create( + device=self.device, + name="brk-xe-2/0/7-a:1", + type=PLAIN_TYPE, + ) + plan_set = plan_installed_families( + module, + rule, + build_variables(module.module_bay, device=module.device), + ) + + result = execute_installed_plan_set(plan_set) + + self.assertEqual(len(result.families), 2) + self.assertEqual(result.changed_count, 3) + self.assertEqual( + [member.status for member in result.families[0].members], + [FamilyStatus.CHANGED, FamilyStatus.BLOCKED], + ) + self.assertEqual( + [member.status for member in result.families[1].members], + [FamilyStatus.CHANGED, FamilyStatus.CHANGED], + ) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + [ + "brk-xe-1/0/7-a:1", + "brk-xe-2/0/7-a:0", + "brk-xe-2/0/7-b:0", + "brk-xe-2/0/7-b:1", + ], + ) + + @skipUnless(supports_vc_position_token(), "requires NetBox virtual-chassis position templates") + def test_overlapping_historical_matchers_reject_every_ambiguous_flat_family(self): + virtual_chassis = VirtualChassis.objects.create(name="family-plan-ambiguous-vc") + self.device.virtual_chassis = virtual_chassis + self.device.vc_position = 1 + self.device.save() + module_type = ModuleType.objects.create( + manufacturer=self.module_type.manufacturer, + model="FAMILY-PLAN-AMBIGUOUS-QSFP", + part_number="FAMILY-PLAN-AMBIGUOUS-QSFP", + ) + for name in ("xe-{vc_position}/0/{module}", "xe-1/{vc_position}/{module}"): + InterfaceTemplate.objects.create( + module_type=module_type, + name=name, + type="100gbase-x-qsfp28", + ) + rule = InterfaceNameRule.objects.create( + module_type=module_type, + name_template="brk-{base}:{channel}", + channel_count=2, + channel_start=0, + ) + module = Module.objects.create(device=self.device, module_bay=self.bay, module_type=module_type) + self.assertEqual(apply_interface_name_rules(module, self.bay), 4) + self.device.vc_position = 2 + self.device.save() + module = Module.objects.select_related("device", "module_bay").get(pk=module.pk) + + plan_set = plan_installed_families( + module, + rule, + build_variables(module.module_bay, device=module.device), + ) + + self.assertEqual(plan_set.plans, ()) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class InstalledChannelizedFamilyTest(TestCase): + """Plan and execute existing channelized families through real NetBox rows.""" + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="InstalledChanMfg", slug="installed-chan-mfg") + device_type = DeviceType.objects.create( + manufacturer=manufacturer, + model="INSTALLED-CHAN-DEVICE", + slug="installed-chan-device", + ) + for position in ("3", "4", "5", "6"): + ModuleBayTemplate.objects.create( + device_type=device_type, + name=f"Bay {position}", + position=position, + ) + cls.module_type = ModuleType.objects.create( + manufacturer=manufacturer, + model="INSTALLED-CHAN-QSFP", + part_number="INSTALLED-CHAN-QSFP", + ) + parent = InterfaceTemplate.objects.create( + module_type=cls.module_type, + name="{module}", + type=PARENT_TYPE, + channels=4, + ) + for channel_id in (1, 3): + InterfaceTemplate.objects.create( + module_type=cls.module_type, + name=f"{{module}}:{channel_id}", + type=CHANNEL_TYPE, + parent=parent, + channel_id=channel_id, + ) + role = DeviceRole.objects.create(name="InstalledChanRole", slug="installed-chan-role") + site = Site.objects.create(name="InstalledChanSite", slug="installed-chan-site") + cls.device = Device.objects.create( + name="installed-chan-device-01", + device_type=device_type, + role=role, + site=site, + ) + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + channel_count=4, + channel_start=0, + ) + + def _raw_module(self, position): + """Install one module without executing the plugin's committed callback.""" + bay = ModuleBay.objects.get(device=self.device, name=f"Bay {position}") + module = Module.objects.create(device=self.device, module_bay=bay, module_type=self.module_type) + return module, bay + + def _plan(self, module, bay): + """Plan the installed family on *module*.""" + return plan_installed_families(module, self.rule, build_variables(bay, device=self.device)) + + def test_incomplete_channelized_family_is_one_plan_with_only_installed_members(self): + module, bay = self._raw_module("3") + + plan_set = self._plan(module, bay) + + self.assertEqual(len(plan_set.plans), 1) + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertEqual( + [(member.role.value, member.snapshot.channel_id) for member in plan.members], + [("parent", None), ("channel", 1), ("channel", 3)], + ) + self.assertEqual( + [member.target_name for member in plan.members], + ["3", "xe-0/0/3:0", "xe-0/0/3:2"], + ) + + def test_automatic_installation_uses_locked_family_execution(self): + bay = ModuleBay.objects.get(device=self.device, name="Bay 4") + + with CaptureQueriesContext(connection) as queries, self.captureOnCommitCallbacks(execute=True): + module = Module.objects.create(device=self.device, module_bay=bay, module_type=self.module_type) + + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["4", "xe-0/0/4:0", "xe-0/0/4:2"], + ) + self.assertTrue(any("FOR UPDATE" in query["sql"] for query in queries.captured_queries)) + + def test_blocked_parent_prevents_every_child_rename(self): + self.rule.breakout_mode = BreakoutModeChoices.CHANNELIZED + self.rule.parent_name_template = "et-0/0/{bay_position}" + self.rule.save() + Interface.objects.create(device=self.device, name="et-0/0/5", type=PLAIN_TYPE) + module, bay = self._raw_module("5") + + result = execute_installed_plan_set(self._plan(module, bay)) + + self.assertEqual(result.families[0].status, FamilyStatus.BLOCKED) + self.assertEqual( + {member.status for member in result.families[0].members}, + {FamilyStatus.BLOCKED}, + ) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["5", "5:1", "5:3"], + ) + + def test_blocked_child_leaves_only_that_child_unchanged(self): + Interface.objects.create(device=self.device, name="xe-0/0/6:2", type=PLAIN_TYPE) + module, bay = self._raw_module("6") + + result = execute_installed_plan_set(self._plan(module, bay)) + + self.assertEqual(result.families[0].status, FamilyStatus.CHANGED) + self.assertEqual( + [member.status for member in result.families[0].members], + [FamilyStatus.UNCHANGED, FamilyStatus.CHANGED, FamilyStatus.BLOCKED], + ) + self.assertEqual( + sorted(Interface.objects.filter(module=module).values_list("name", flat=True)), + ["6", "6:3", "xe-0/0/6:0"], + ) + + +class InstalledFamilyDatabaseAliasTest(TestCase): + """Automatic renaming reads leftover interfaces from the alias the module row came from. + + A second alias onto the same test database gets its own connection, so it cannot see the rows + this test created inside its own open transaction. Binding the module to that alias therefore + leaves the leftover query with nothing to rename, unless the query still runs on the default + connection, which is the split the review found. + """ + + ALIAS = "family_alias" + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="AliasMfg", slug="alias-mfg") + device_type = DeviceType.objects.create(manufacturer=manufacturer, model="ALIAS-DEV", slug="alias-dev") + ModuleBayTemplate.objects.create(device_type=device_type, name="Bay 7", position="7") + cls.module_type = ModuleType.objects.create( + manufacturer=manufacturer, + model="ALIAS-QSFP", + part_number="ALIAS-QSFP", + ) + InterfaceTemplate.objects.create(module_type=cls.module_type, name="{module}", type="100gbase-x-qsfp28") + role = DeviceRole.objects.create(name="AliasRole", slug="alias-role") + site = Site.objects.create(name="AliasSite", slug="alias-site") + cls.device = Device.objects.create( + name="alias-device-01", + device_type=device_type, + role=role, + site=site, + ) + cls.bay = ModuleBay.objects.get(device=cls.device, name="Bay 7") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{base}:{channel}", + channel_count=2, + channel_start=0, + ) + + def _use_second_alias(self): + """Point a second alias at the live test database and let this test reach it.""" + connections.settings[self.ALIAS] = dict(connections[DEFAULT_DB_ALIAS].settings_dict) + declared = type(self).databases + type(self).databases = frozenset({*declared, self.ALIAS}) + self.addCleanup(connections.settings.pop, self.ALIAS, None) + self.addCleanup(connections[self.ALIAS].close) + self.addCleanup(setattr, type(self), "databases", declared) + return self.ALIAS + + @staticmethod + def _names(module): + return sorted(Interface.objects.filter(module=module).values_list("name", flat=True)) + + def test_leftover_interfaces_are_read_from_the_module_alias(self): + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + module._state.db = self._use_second_alias() + + with CaptureQueriesContext(connections[self.ALIAS]) as aliased: + renamed = apply_interface_name_rules(module, self.bay) + + self.assertEqual(renamed, 0) + self.assertEqual(self._names(module), ["7"]) + self.assertTrue([query for query in aliased.captured_queries if 'FROM "dcim_interface"' in query["sql"]]) + + def test_the_same_module_on_the_default_alias_breaks_out_the_family(self): + """The control: only the alias makes the leftover pass find no interface to rename.""" + module = Module.objects.create( + device=self.device, + module_bay=self.bay, + module_type=self.module_type, + ) + + self.assertEqual(apply_interface_name_rules(module, self.bay), 2) + self.assertEqual(self._names(module), ["xe-0/0/7:0", "xe-0/0/7:1"]) diff --git a/netbox_interface_name_rules/tests/test_misc.py b/netbox_interface_name_rules/tests/test_misc.py index 8022bbe..84c1916 100644 --- a/netbox_interface_name_rules/tests/test_misc.py +++ b/netbox_interface_name_rules/tests/test_misc.py @@ -1,7 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (C) 2025 Marcin Zieba -"""Tests for jobs, model properties, and API serializer edge-cases.""" +"""Tests for jobs, model properties, family contracts, and API serializer edge-cases.""" +from typing import get_args, get_type_hints from unittest.mock import MagicMock, patch from dcim.models import DeviceType, Manufacturer, ModuleType, Platform @@ -10,6 +11,21 @@ from netbox_interface_name_rules.models import InterfaceNameRule + +class FamilyTargetContractTest(TestCase): + """The target contract describes values the planners can produce.""" + + def test_channel_targets_allow_an_unresolved_name(self): + """Lockstep naming can leave one channel unresolved without invalidating the tuple.""" + from netbox_interface_name_rules.family.targets import FamilyTargets + + channels = get_type_hints(FamilyTargets)["channels"] + channel_pair = get_args(channels)[0] + target_name = get_args(channel_pair)[0] + + self.assertIn(type(None), get_args(target_name)) + + # --------------------------------------------------------------------------- # jobs.py # --------------------------------------------------------------------------- @@ -262,12 +278,11 @@ def test_device_iface_rule_forces_regex_false(self): rule.clean() self.assertFalse(rule.module_type_is_regex) - def test_exact_rule_with_redos_pattern_fails(self): - """Regex rule with ReDoS-prone pattern (nested quantifiers) fails validation.""" - # "(ab)+*" has ")+*" which triggers the nested-quantifier guard + def test_exact_rule_with_python_only_pattern_fails(self): + """A regex rule using syntax outside RE2 fails validation.""" rule = InterfaceNameRule( module_type_is_regex=True, - module_type_pattern="(ab)+*", + module_type_pattern=r"(?=QSFP)QSFP-.*", name_template="port{bay_position}", ) with self.assertRaises(ValidationError) as ctx: @@ -484,13 +499,9 @@ def test_regex_mode_invalid_pattern_adds_error(self): form.is_valid() self.assertIn("module_type_pattern", form.errors) - def test_regex_mode_redos_pattern_adds_error(self): - """RuleTestForm.clean() adds error when pattern contains nested quantifiers (line 125). - - (a)+? compiles OK (valid lazy quantifier syntax) but triggers the ReDoS guard - because )+? matches \\)\\s*[\\+\\*\\?]\\s*[\\+\\*\\?] in _REDOS_PATTERN. - """ - form = self._make_form({"module_type_is_regex": True, "module_type_pattern": "(a)+?"}) + def test_regex_mode_python_only_pattern_adds_error(self): + """RuleTestForm.clean() adds an error when RE2 does not support the pattern.""" + form = self._make_form({"module_type_is_regex": True, "module_type_pattern": r"(?=QSFP)QSFP-.*"}) form.is_valid() self.assertIn("module_type_pattern", form.errors) @@ -663,30 +674,6 @@ def test_specificity_label_all_scopes(self): self.assertIn("platform", label) -# --------------------------------------------------------------------------- -# models.py — _validate_module_type_pattern ReDoS guard (line 29) -# --------------------------------------------------------------------------- - - -class ModelValidatePatternReDoSTest(TestCase): - """Test _validate_module_type_pattern raises for valid-but-ReDoS-prone patterns.""" - - def test_valid_regex_with_nested_quantifiers_raises(self): - """A valid regex with nested quantifiers e.g. (a)+? raises ValidationError (line 29). - - (a)+? compiles without error but contains )+? which matches - \\)\\s*[\\+\\*\\?]\\s*[\\+\\*\\?] in _REDOS_PATTERN. - """ - from django.core.exceptions import ValidationError - - from netbox_interface_name_rules.models import _validate_module_type_pattern - - with self.assertRaises(ValidationError) as ctx: - _validate_module_type_pattern("(a)+?") - self.assertIn("module_type_pattern", ctx.exception.message_dict) - self.assertIn("nested quantifiers", str(ctx.exception)) - - # --------------------------------------------------------------------------- # jobs.py — ApplyRuleJob.run() success + exception paths (lines 30-36) # --------------------------------------------------------------------------- diff --git a/netbox_interface_name_rules/tests/test_naming.py b/netbox_interface_name_rules/tests/test_naming.py new file mode 100644 index 0000000..dbb24be --- /dev/null +++ b/netbox_interface_name_rules/tests/test_naming.py @@ -0,0 +1,74 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for the lower-level naming seam.""" + +from dcim.models import ( + Device, + DeviceRole, + DeviceType, + Manufacturer, + Module, + ModuleBay, + ModuleBayTemplate, + ModuleType, + Site, + VirtualChassis, +) +from django.test import TestCase + +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.naming import build_variables, evaluate_name_template + + +class NamingTest(TestCase): + """Exercise naming through its lower-level public interface.""" + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="NamingMfg", slug="naming-mfg") + device_type = DeviceType.objects.create( + manufacturer=manufacturer, + model="NAMING-DEVICE", + slug="naming-device", + ) + ModuleBayTemplate.objects.create(device_type=device_type, name="Slot 7", position="7") + module_type = ModuleType.objects.create( + manufacturer=manufacturer, + model="NAMING-SFP", + part_number="NAMING-SFP", + ) + role = DeviceRole.objects.create(name="NamingRole", slug="naming-role") + site = Site.objects.create(name="NamingSite", slug="naming-site") + virtual_chassis = VirtualChassis.objects.create(name="naming-vc") + cls.device = Device.objects.create( + name="naming-device-01", + device_type=device_type, + role=role, + site=site, + virtual_chassis=virtual_chassis, + vc_position=3, + ) + cls.bay = ModuleBay.objects.get(device=cls.device, name="Slot 7") + cls.module = Module.objects.create(device=cls.device, module_bay=cls.bay, module_type=module_type) + cls.rule = InterfaceNameRule.objects.create( + module_type=module_type, + name_template="xe-{vc_position}/{slot}/{bay_position_num}", + ) + + def test_real_module_context_builds_and_evaluates_the_rule_name(self): + variables = build_variables(self.module.module_bay, device=self.module.device) + + name = evaluate_name_template(self.rule.name_template, variables) + + self.assertEqual( + variables, + { + "slot": "7", + "bay_position": "7", + "bay_position_num": "7", + "parent_bay_position": "0", + "sfp_slot": "7", + "vc_position": "3", + }, + ) + self.assertEqual(name, "xe-3/7/7") diff --git a/netbox_interface_name_rules/tests/test_performance_compare.py b/netbox_interface_name_rules/tests/test_performance_compare.py new file mode 100644 index 0000000..9384de9 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_performance_compare.py @@ -0,0 +1,274 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Tests for the performance artifact comparison script.""" + +import fnmatch +import importlib.util +import tomllib +import unittest +from pathlib import Path +from unittest.mock import patch + +_PROJECT_ROOT = Path(__file__).resolve().parents[2] +_COMPARE_PATH = _PROJECT_ROOT / "performance" / "compare.py" +_spec = importlib.util.spec_from_file_location("performance_compare", _COMPARE_PATH) +compare = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(compare) + + +def _artifact(planner_settings, load=None): + """Return the smallest artifact the environment table reads.""" + return { + "schema_version": 1, + "baseline_kind": "test", + "generated_at": "2026-08-30T00:00:00+00:00", + "environment": { + "host_load": load, + "plugin_revision": "abc123", + "netbox_revision": "def456", + "netbox_version": "4.7", + "cpu_model": "Test CPU", + "operating_system_release": "test-release", + "postgresql": {"server_version": "18.3", "planner_settings": planner_settings}, + }, + "configuration": {"samples": 15, "warmups": 3}, + } + + +def _timed_artifact(machine_time): + """Return an artifact whose single scenario carries *machine_time*.""" + artifact = _artifact({"work_mem": "4MB"}) + artifact["scenarios"] = [ + { + "name": "module.direct_callback.plain_rename", + "layer": "direct_callback", + "database": {"statements": [], "totals": {"statement_calls": 31}}, + "machine_time": machine_time, + } + ] + return artifact + + +_MACHINE_TIME = {"wall": {"median_ms": 2.0, "p95_ms": 3.0}, "process_cpu": {"median_ms": 1.0}} + + +class PerformancePackageTest(unittest.TestCase): + """The repository-only performance harness does not ship as a generic package.""" + + def test_package_discovery_excludes_the_performance_tools(self): + configuration = tomllib.loads((_PROJECT_ROOT / "pyproject.toml").read_text()) + patterns = configuration["tool"]["setuptools"]["packages"]["find"]["include"] + + self.assertFalse(any(fnmatch.fnmatchcase("performance", pattern) for pattern in patterns)) + + def test_pytest_adds_the_repository_checkout_to_pythonpath(self): + configuration = tomllib.loads((_PROJECT_ROOT / "pyproject.toml").read_text()) + + self.assertIn(".", configuration["tool"]["pytest"]["ini_options"]["pythonpath"]) + + def test_recorded_direct_callbacks_reach_zero_shared_reads(self): + comparison = (_PROJECT_ROOT / "performance" / "comparisons" / "family-package-vs-existing.md").read_text() + expected_scenarios = { + "module.direct_callback.no_matching_rule", + "module.direct_callback.plain_rename", + "module.direct_callback.structural_creation", + "module.direct_callback.existing_family", + "module.direct_callback.reconciliation", + "vc.direct_callback.reapply_1", + "vc.direct_callback.reapply_8", + } + rows = [ + [cell.strip().strip("`") for cell in line.strip("|\n").split("|")] + for line in comparison.splitlines() + if line.startswith("|") + ] + direct_reads = [row for row in rows if ".direct_callback." in row[0] and row[1] == "Shared reads"] + structural_save = next( + row + for row in rows + if row[0] == "module.complete_model_save.structural_creation" and row[1] == "Shared reads" + ) + + self.assertEqual(len(direct_reads), 7) + self.assertEqual({row[0] for row in direct_reads}, expected_scenarios) + self.assertEqual({row[3] for row in direct_reads}, {"0"}) + self.assertEqual(structural_save[3], "1") + + def test_readme_does_not_equate_shared_reads_with_all_disk_io(self): + readme = (_PROJECT_ROOT / "performance" / "README.md").read_text() + + self.assertNotIn("never goes to disk", readme) + + +class XdistWorkerCapTest(unittest.TestCase): + """The shared host gets a bounded number of automatic pytest workers.""" + + def test_auto_worker_count_is_capped_at_eight(self): + configuration_path = _PROJECT_ROOT / "conftest.py" + spec = importlib.util.spec_from_file_location("project_conftest", configuration_path) + configuration = importlib.util.module_from_spec(spec) + spec.loader.exec_module(configuration) + + with patch("xdist.plugin.pytest_xdist_auto_num_workers", return_value=32): + self.assertEqual(configuration.pytest_xdist_auto_num_workers(object()), 8) + + +class ArtifactValidationTest(unittest.TestCase): + """The comparison refuses an artifact it cannot interpret completely.""" + + def test_the_current_artifact_shape_is_accepted(self): + compare.validate_artifact(_timed_artifact(_MACHINE_TIME), "before artifact") + + def test_a_newer_schema_version_is_rejected(self): + artifact = _timed_artifact(_MACHINE_TIME) + artifact["schema_version"] = 2 + + with self.assertRaisesRegex(ValueError, "schema_version"): + compare.validate_artifact(artifact, "before artifact") + + def test_a_missing_required_environment_field_is_rejected(self): + artifact = _timed_artifact(_MACHINE_TIME) + del artifact["environment"]["plugin_revision"] + + with self.assertRaisesRegex(ValueError, "plugin_revision"): + compare.validate_artifact(artifact, "before artifact") + + def test_a_non_numeric_database_metric_is_rejected(self): + artifact = _timed_artifact(_MACHINE_TIME) + artifact["scenarios"][0]["database"]["totals"]["planner_total_cost"] = "not a number" + + with self.assertRaisesRegex(ValueError, "planner_total_cost"): + compare.validate_artifact(artifact, "before artifact") + + def test_a_negative_database_metric_is_rejected(self): + artifact = _timed_artifact(_MACHINE_TIME) + artifact["scenarios"][0]["database"]["totals"]["planner_total_cost"] = -1.0 + + with self.assertRaisesRegex(ValueError, "non-negative"): + compare.validate_artifact(artifact, "before artifact") + + def test_a_fractional_statement_call_count_is_rejected(self): + artifact = _timed_artifact(_MACHINE_TIME) + artifact["scenarios"][0]["database"]["statements"] = [{"normalized_sql": "SELECT * FROM example", "calls": 1.5}] + + with self.assertRaisesRegex(ValueError, "calls"): + compare.validate_artifact(artifact, "before artifact") + + +class TimeTableTest(unittest.TestCase): + """A comparison must not invent machine time that a run did not measure.""" + + def test_unmeasured_machine_time_is_reported_not_crashed(self): + rows = compare._time_table( + {"module.direct_callback.plain_rename": _timed_artifact(None)["scenarios"][0]}, + {"module.direct_callback.plain_rename": _timed_artifact(_MACHINE_TIME)["scenarios"][0]}, + ) + + self.assertEqual( + rows[-3:], + [ + "| `module.direct_callback.plain_rename` | Wall median (ms) | not measured | 2 | n/a | n/a |", + "| `module.direct_callback.plain_rename` | Wall p95 (ms) | not measured | 3 | n/a | n/a |", + "| `module.direct_callback.plain_rename` | CPU median (ms) | not measured | 1 | n/a | n/a |", + ], + ) + + def test_measured_machine_time_is_still_compared(self): + rows = compare._time_table( + {"module.direct_callback.plain_rename": _timed_artifact(_MACHINE_TIME)["scenarios"][0]}, + {"module.direct_callback.plain_rename": _timed_artifact(_MACHINE_TIME)["scenarios"][0]}, + ) + + self.assertIn("Wall median (ms)", "".join(rows)) + + def test_an_after_only_scenario_reports_its_missing_baseline(self): + scenario = _timed_artifact(_MACHINE_TIME)["scenarios"][0] + + rows = compare._time_table({}, {scenario["name"]: scenario}) + + self.assertIn( + "| `module.direct_callback.plain_rename` | Wall median (ms) | missing baseline | 2 | n/a | n/a |", + rows, + ) + + +class DatabaseTableTest(unittest.TestCase): + """A comparison must show every database metric present on either side.""" + + def test_a_metric_missing_from_one_artifact_is_reported(self): + before_scenario = _timed_artifact(None)["scenarios"][0] + after_scenario = _timed_artifact(None)["scenarios"][0] + before_scenario["database"]["totals"]["planner_total_cost"] = 12.5 + + rows, _regressions = compare._database_table( + {before_scenario["name"]: before_scenario}, + {after_scenario["name"]: after_scenario}, + ) + + self.assertIn( + "| `module.direct_callback.plain_rename` | Planner cost | 12.50 | n/a | n/a | n/a |", + rows, + ) + + def test_an_after_only_scenario_reports_its_missing_baseline(self): + scenario = _timed_artifact(None)["scenarios"][0] + + rows, _regressions = compare._database_table({}, {scenario["name"]: scenario}) + + self.assertIn( + "| `module.direct_callback.plain_rename` | SQL calls | missing baseline | 31 | n/a | n/a |", + rows, + ) + + def test_a_statement_increase_is_reported_as_a_regression(self): + before_scenario = _timed_artifact(None)["scenarios"][0] + after_scenario = _timed_artifact(None)["scenarios"][0] + after_scenario["database"]["totals"]["statement_calls"] = 32 + + _rows, regressions = compare._database_table( + {before_scenario["name"]: before_scenario}, + {after_scenario["name"]: after_scenario}, + ) + + self.assertEqual(regressions, [(before_scenario["name"], "SQL calls", 31, 32)]) + + +class EnvironmentTableTest(unittest.TestCase): + """Exercise the environment table the comparison writes.""" + + def _planner_row(self, before, after): + rows = compare._environment_table(before, after) + return next(row for row in rows if row.startswith("| planner settings |")) + + def test_identical_planner_settings_fill_both_columns(self): + settings = {"work_mem": "4MB"} + + row = self._planner_row(_artifact(settings), _artifact(dict(settings))) + + self.assertEqual(row, "| planner settings | `identical` | `identical` |") + + def test_changed_planner_settings_fill_both_columns(self): + row = self._planner_row(_artifact({"work_mem": "4MB"}), _artifact({"work_mem": "64MB"})) + + self.assertEqual(row, "| planner settings | `CHANGED` | `CHANGED` |") + + def _load_row(self, before, after): + rows = compare._environment_table(before, after) + return next(row for row in rows if row.startswith("| host load")) + + def test_recorded_host_load_is_reported_for_both_runs(self): + settings = {"work_mem": "4MB"} + quiet = {"started": {"one_minute": 0.5}, "finished": {"one_minute": 0.9}} + busy = {"started": {"one_minute": 31.0}, "finished": {"one_minute": 44.25}} + + row = self._load_row(_artifact(settings, busy), _artifact(settings, quiet)) + + self.assertEqual(row, "| host load (1 min, start to end) | `31.00 to 44.25` | `0.50 to 0.90` |") + + def test_unrecorded_host_load_is_named_rather_than_left_blank(self): + settings = {"work_mem": "4MB"} + quiet = {"started": {"one_minute": 0.5}, "finished": {"one_minute": 0.9}} + + row = self._load_row(_artifact(settings, None), _artifact(settings, quiet)) + + self.assertEqual(row, "| host load (1 min, start to end) | `not recorded` | `0.50 to 0.90` |") diff --git a/netbox_interface_name_rules/tests/test_prospective_families.py b/netbox_interface_name_rules/tests/test_prospective_families.py new file mode 100644 index 0000000..99ca2f3 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_prospective_families.py @@ -0,0 +1,612 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for prospective (rowless) interface-family plans. + +A prospective plan says what a rule intends for interfaces described by name alone. It is the +planner behind rowless prediction and the interactive preview, and it is never executable: the +same planning rules that installed execution follows, without a row to mutate. +""" + +from unittest import skipIf, skipUnless + +from dcim.models import Interface, InterfaceTemplate, Module +from django.test import SimpleTestCase + +from netbox_interface_name_rules.choices import BreakoutModeChoices +from netbox_interface_name_rules.engine import build_variables, predict_rule_output, supports_channelization +from netbox_interface_name_rules.family import ( + FamilyStatus, + FamilyTopology, + MemberRole, + ProspectiveFamilyPlan, + ProspectiveFamilyPlanSet, + ProspectiveMember, + describe_interfaces, + describe_template_interfaces, + execute_installed_plan_set, + execute_structural_family, + plan_installed_families, + plan_prospective_families, + resolved_template_names, +) +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.tests.test_breakout_mode import CHANNELIZED, _plain_module_type +from netbox_interface_name_rules.tests.test_channelization import ( + CHANNEL_TYPE, + PLAIN_TYPE, + REQUIRES_CHANNELIZATION, + ChannelizationTestCase, + _build_device, + _channelized_module_type, +) + +REQUIRES_NO_CHANNELIZATION = "requires a NetBox that cannot model channelized interfaces (4.6 and older)" + + +def _projection(plan): + """Reduce one plan to the facts installed and prospective planning must agree on.""" + return ( + plan.topology, + tuple(member.target_name for member in plan.members), + plan.precondition_status, + plan.precondition_reason, + ) + + +def _installed_projection(plan): + """Reduce one installed plan to the same facts, reading targets from its snapshots.""" + return ( + plan.topology, + tuple(member.target_name or member.snapshot.name for member in plan.members), + plan.precondition_status, + plan.precondition_reason, + ) + + +class ProspectivePlanSetLookupTest(SimpleTestCase): + """The plan set answers for a name with what its family plan intends for that name.""" + + PLAN = ProspectiveFamilyPlan( + family_id="channelized:et0", + topology=FamilyTopology.CHANNELIZED, + base_name=None, + members=( + ProspectiveMember(source_name="et0", target_name="xe0", role=MemberRole.PARENT), + ProspectiveMember(source_name="et0:1", target_name="xe0:1", role=MemberRole.CHANNEL, channel_id=1), + ), + ) + + def test_a_member_of_an_existing_family_maps_to_its_own_target(self): + plan_set = ProspectiveFamilyPlanSet(module_id=1, plans=(self.PLAN,)) + + self.assertEqual(plan_set.predicted_names("et0"), ("xe0",)) + self.assertEqual(plan_set.predicted_names("et0:1"), ("xe0:1",)) + self.assertEqual(plan_set.predicted_names("et0:2"), ("et0:2",)) + + def test_the_plan_lists_the_names_its_members_carry_now(self): + self.assertEqual(self.PLAN.source_names, ("et0", "et0:1")) + self.assertEqual(self.PLAN.target_names, ("xe0", "xe0:1")) + + +class DescribeInterfacesTest(SimpleTestCase): + """Live-row adaptation accepts every iterable promised by its public boundary.""" + + def test_a_one_shot_iterable_retains_every_interface(self): + parent = Interface(pk=1, name="3") + channel = Interface(pk=2, name="3:0", parent_id=1) + channel.channel_id = 0 + interfaces = iter((parent, channel)) + + described = describe_interfaces(interfaces) + + self.assertEqual( + [(interface.name, interface.parent_name) for interface in described], + [("3", None), ("3:0", "3")], + ) + + +class ProspectivePlanTestCase(ChannelizationTestCase): + """Plan the families a rule intends for a module, from its templates alone.""" + + def _prospective(self, module, bay, rule, names=()): + """Plan the families *rule* intends for the template names of *module*.""" + templates = resolved_template_names(module) + return plan_prospective_families( + module, + rule, + build_variables(bay, device=self.device), + describe_template_interfaces(templates, names), + ) + + +class ProspectiveFlatPlanTest(ProspectivePlanTestCase): + """A flat breakout rule expands one plain name into the sibling family it creates.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspFlat", ["3", "4"]) + cls.module_type = _plain_module_type(manufacturer, "ProspFlat-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + channel_count=4, + channel_start=0, + ) + + def test_a_plain_name_plans_the_flat_family_the_rule_creates(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + plan_set = self._prospective(module, bay, self.rule) + + self.assertEqual(len(plan_set.plans), 1) + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.FLAT) + self.assertEqual(plan.base_name, "3") + self.assertEqual( + plan.target_names, + ("xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"), + ) + self.assertEqual({member.role for member in plan.members}, {MemberRole.FLAT_MEMBER}) + self.assertEqual(plan_set.predicted_names("3"), plan.target_names) + + def test_a_simple_rule_plans_one_member(self): + rule = InterfaceNameRule(module_type=self.module_type, name_template="et-0/0/{bay_position}") + module, bay = self._install(self.module_type, "4", run_rules=False) + + plan_set = self._prospective(module, bay, rule) + + self.assertEqual([plan.target_names for plan in plan_set.plans], [("et-0/0/4",)]) + self.assertEqual(plan_set.predicted_names("4"), ("et-0/0/4",)) + + def test_an_unevaluable_template_fails_the_plan_and_keeps_the_name(self): + rule = InterfaceNameRule( + module_type=self.module_type, + name_template="xe-{vc_position}/0/{bay_position}", + ) + module, bay = self._install(self.module_type, "4", run_rules=False) + + plan_set = self._prospective(module, bay, rule) + + plan = plan_set.plans[0] + self.assertEqual(plan.precondition_status, FamilyStatus.FAILED) + self.assertEqual(plan.target_names, ("4",)) + self.assertEqual(plan_set.predicted_names("4"), ("4",)) + + def test_a_name_no_plan_claims_predicts_to_itself(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + plan_set = self._prospective(module, bay, self.rule) + + self.assertEqual(plan_set.predicted_names("unclaimed"), ("unclaimed",)) + + +@skipIf(supports_channelization(), REQUIRES_NO_CHANNELIZATION) +class ProspectiveUnsupportedTopologyTest(ProspectivePlanTestCase): + """A release that cannot model channels plans an explicit unsupported family.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspUnsup", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspUnsup-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + parent_name_template="et-0/0/{bay_position}", + breakout_mode=CHANNELIZED, + channel_count=4, + channel_start=0, + ) + + def test_the_plan_reports_unsupported_and_the_name_is_unchanged(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + plan_set = self._prospective(module, bay, self.rule) + + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertEqual(plan.precondition_status, FamilyStatus.UNSUPPORTED) + self.assertEqual(plan.target_names, ("3",)) + self.assertEqual(plan_set.predicted_names("3"), ("3",)) + + def test_prediction_leaves_the_name_alone(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + self.assertEqual(predict_rule_output(module, bay, ["3"]), ["3"]) + + +class ProspectivePlansAreNotExecutableTest(ProspectivePlanTestCase): + """A prospective plan describes intent; the executors refuse to take one.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspExec", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspExec-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + channel_count=2, + channel_start=0, + ) + + def test_the_installed_executor_refuses_a_prospective_plan_set(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + plan_set = self._prospective(module, bay, self.rule) + + with self.assertRaises(TypeError): + execute_installed_plan_set(plan_set) + + self.assertEqual(self._names(module), ["3"]) + + def test_the_structural_executor_refuses_a_prospective_plan(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + plan = self._prospective(module, bay, self.rule).plans[0] + + with self.assertRaises(TypeError): + execute_structural_family(plan) + + self.assertEqual(self._names(module), ["3"]) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class ProspectiveChannelizedPlanTest(ProspectivePlanTestCase): + """A rule on a channelized module type plans the family the templates already describe.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspChan", ["3", "5", "8", "9"]) + cls.breakout_type = _channelized_module_type(manufacturer, "ProspChan-BRK") + cls.incomplete_type = _channelized_module_type(manufacturer, "ProspChan-INC", child_channel_ids=(1, 2, 3)) + cls.mismatch_type = _channelized_module_type(manufacturer, "ProspChan-MM", channels=8) + cls.simple_type = _channelized_module_type( + manufacturer, + "ProspChan-SMP", + child_names={1: "{module}:1", 2: "{module}:2", 3: "{module}:3", 4: "mgmt-chan"}, + ) + for module_type in (cls.breakout_type, cls.incomplete_type, cls.mismatch_type): + InterfaceNameRule.objects.create( + module_type=module_type, + name_template="xe-0/0/{bay_position}:{channel}", + channel_count=4, + channel_start=0, + ) + InterfaceNameRule.objects.create(module_type=cls.simple_type, name_template="et-0/0/{bay_position}") + + def _rule(self, module_type): + return InterfaceNameRule.objects.get(module_type=module_type) + + def test_an_existing_family_is_renamed_in_place(self): + module, bay = self._install(self.breakout_type, "3", run_rules=False) + + plan_set = self._prospective(module, bay, self._rule(self.breakout_type)) + + self.assertEqual(len(plan_set.plans), 1) + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertIsNone(plan.base_name) + self.assertEqual( + plan.target_names, + ("3", "xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"), + ) + self.assertEqual(plan_set.predicted_names("3"), ("3",)) + self.assertEqual(plan_set.predicted_names("3:2"), ("xe-0/0/3:1",)) + + def test_an_incomplete_family_stays_channelized(self): + module, bay = self._install(self.incomplete_type, "5", run_rules=False) + + plan_set = self._prospective(module, bay, self._rule(self.incomplete_type)) + + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertIsNone(plan.precondition_status) + self.assertEqual(plan.target_names, ("5", "xe-0/0/5:0", "xe-0/0/5:1", "xe-0/0/5:2")) + + def test_a_channel_count_mismatch_blocks_the_family(self): + module, bay = self._install(self.mismatch_type, "8", run_rules=False) + + plan_set = self._prospective(module, bay, self._rule(self.mismatch_type)) + + plan = plan_set.plans[0] + self.assertEqual(plan.precondition_status, FamilyStatus.BLOCKED) + self.assertIn("8 channels", plan.precondition_reason) + self.assertEqual(plan.target_names, ("8", "8:1", "8:2", "8:3", "8:4")) + + def test_an_ambiguous_channel_suffix_leaves_that_member_alone(self): + module, bay = self._install(self.simple_type, "9", run_rules=False) + + plan_set = self._prospective(module, bay, self._rule(self.simple_type)) + + plan = plan_set.plans[0] + self.assertEqual( + plan.target_names, + ("et-0/0/9", "et-0/0/9:1", "et-0/0/9:2", "et-0/0/9:3", "mgmt-chan"), + ) + stranded = [member for member in plan.members if member.source_name == "mgmt-chan"] + self.assertEqual(stranded[0].reason, "channel suffix is ambiguous or unavailable") + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class ProspectiveStructuralPlanTest(ProspectivePlanTestCase): + """A channelized rule on a plain name plans the family it would build there.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspStruct", ["3", "4"]) + cls.module_type = _plain_module_type(manufacturer, "ProspStruct-QSFP") + cls.collision_type = _plain_module_type(manufacturer, "ProspStruct-COL") + InterfaceTemplate.objects.create(module_type=cls.collision_type, name="et-0/0/{module}", type=PLAIN_TYPE) + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + parent_name_template="et-0/0/{bay_position}", + breakout_mode=CHANNELIZED, + channel_count=4, + channel_start=0, + ) + cls.collision_rule = InterfaceNameRule.objects.create( + module_type=cls.collision_type, + name_template="xe-0/0/{bay_position}:{channel}", + parent_name_template="et-0/0/{bay_position}", + breakout_mode=CHANNELIZED, + channel_count=4, + channel_start=0, + ) + + def test_a_plain_name_plans_the_parent_and_every_channel(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + plan_set = self._prospective(module, bay, self.rule) + + plan = plan_set.plans[0] + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertEqual(plan.base_name, "3") + self.assertEqual( + plan.target_names, + ("et-0/0/3", "xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"), + ) + self.assertEqual([member.role for member in plan.members][0], MemberRole.PARENT) + self.assertEqual(plan_set.predicted_names("3"), plan.target_names) + + def test_a_planned_name_another_interface_owns_blocks_the_family(self): + """The structural executor refuses a family whose names are taken, so the plan says so first.""" + module, bay = self._install(self.collision_type, "4", run_rules=False) + + plan_set = self._prospective(module, bay, self.collision_rule) + + blocked = [plan for plan in plan_set.plans if plan.base_name == "4"] + self.assertEqual(blocked[0].precondition_status, FamilyStatus.BLOCKED) + self.assertIn("et-0/0/4", blocked[0].precondition_reason) + self.assertEqual(plan_set.predicted_names("4"), ("4",)) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class ProspectiveMatchesInstalledPlanningTest(ProspectivePlanTestCase): + """Equivalent installed and prospective inputs describe the same intended family.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspSame", ["3", "5"]) + cls.breakout_type = _channelized_module_type(manufacturer, "ProspSame-BRK") + cls.mismatch_type = _channelized_module_type(manufacturer, "ProspSame-MM", channels=8) + for module_type in (cls.breakout_type, cls.mismatch_type): + InterfaceNameRule.objects.create( + module_type=module_type, + name_template="xe-0/0/{bay_position}:{channel}", + channel_count=4, + channel_start=0, + ) + + def _compare(self, module_type, position): + """Plan the same module twice, from live rows and from its templates alone.""" + module, bay = self._install(module_type, position, run_rules=False) + rule = InterfaceNameRule.objects.get(module_type=module_type) + variables = build_variables(bay, device=self.device) + installed = plan_installed_families(module, rule, variables) + interfaces = list(Interface.objects.filter(module=module).order_by("pk")) + prospective = plan_prospective_families(module, rule, variables, describe_interfaces(interfaces)) + return installed, prospective + + def test_a_channelized_family_plans_the_same_names_either_way(self): + installed, prospective = self._compare(self.breakout_type, "3") + + self.assertEqual( + [_installed_projection(plan) for plan in installed.plans], + [_projection(plan) for plan in prospective.plans], + ) + + def test_a_blocked_family_gives_the_same_reason_either_way(self): + installed, prospective = self._compare(self.mismatch_type, "5") + + self.assertEqual( + [_installed_projection(plan) for plan in installed.plans], + [_projection(plan) for plan in prospective.plans], + ) + self.assertEqual(prospective.plans[0].precondition_status, FamilyStatus.BLOCKED) + + +class ProspectivePreviewIsNotAppliedTest(ChannelizationTestCase): + """Interactive apply replans from live rows instead of executing an earlier preview.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspReplan", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspReplan-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="et-0/0/{bay_position}", + ) + + def test_a_rename_between_preview_and_apply_is_replanned(self): + from netbox_interface_name_rules.engine import apply_rule_to_existing, find_interfaces_for_rule + + module, _bay = self._install(self.module_type, "3", run_rules=False) + preview, _checked = find_interfaces_for_rule(self.rule) + self.assertEqual(preview[0]["current_name"], "3") + + interface = Interface.objects.get(module=module) + interface.name = "renamed-by-someone-else" + interface.save() + + self.assertEqual(apply_rule_to_existing(self.rule, interface_ids=[interface.pk]).changed_count, 1) + interface.refresh_from_db() + self.assertEqual(interface.name, "et-0/0/3") + + def test_a_previewed_interface_that_disappeared_is_not_recreated(self): + from netbox_interface_name_rules.engine import apply_rule_to_existing, find_interfaces_for_rule + + module, _bay = self._install(self.module_type, "3", run_rules=False) + preview, _checked = find_interfaces_for_rule(self.rule) + previewed_pk = preview[0]["interface"].pk + + Interface.objects.filter(pk=previewed_pk).delete() + + self.assertEqual(apply_rule_to_existing(self.rule, interface_ids=[previewed_pk]).changed_count, 0) + self.assertEqual(Interface.objects.filter(module=module).count(), 0) + + +class ProspectivePlanningIsReadOnlyTest(ChannelizationTestCase): + """Planning a prospective family never writes a row.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspRead", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspRead-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + breakout_mode=BreakoutModeChoices.FLAT, + channel_count=4, + channel_start=0, + ) + + def test_planning_leaves_every_interface_exactly_as_it_was(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + before = list(Interface.objects.filter(module=module).values_list("pk", "name")) + + plan_prospective_families( + module, + self.rule, + build_variables(bay, device=self.device), + describe_template_interfaces(resolved_template_names(module), ["3"]), + ) + + self.assertEqual(list(Interface.objects.filter(module=module).values_list("pk", "name")), before) + self.assertEqual(Module.objects.filter(pk=module.pk).count(), 1) + + +class PreviewComesFromTheFamilyPlanTest(ChannelizationTestCase): + """The interactive preview reports the plan's intended names and writes nothing.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspPrev", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspPrev-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="xe-0/0/{bay_position}:{channel}", + breakout_mode=BreakoutModeChoices.FLAT, + channel_count=4, + channel_start=0, + ) + + def test_the_preview_names_are_the_planned_names(self): + from netbox_interface_name_rules.engine import find_interfaces_for_rule + + module, bay = self._install(self.module_type, "3", run_rules=False) + interfaces = list(Interface.objects.filter(module=module).order_by("pk")) + plan = plan_prospective_families( + module, + self.rule, + build_variables(bay, device=self.device), + describe_interfaces(interfaces), + ).plans[0] + + preview, checked = find_interfaces_for_rule(self.rule) + + self.assertEqual(checked, 1) + self.assertEqual(preview[0]["new_names"], list(plan.target_names)) + self.assertEqual([detail.role for detail in preview[0]["name_details"]], ["channel"] * 4) + + def test_the_preview_changes_no_interface(self): + from netbox_interface_name_rules.engine import find_interfaces_for_rule + + module, _bay = self._install(self.module_type, "3", run_rules=False) + before = list(Interface.objects.filter(module=module).values_list("pk", "name")) + + find_interfaces_for_rule(self.rule) + + self.assertEqual(list(Interface.objects.filter(module=module).values_list("pk", "name")), before) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class PreviewFollowsTheApplyClassificationTest(ChannelizationTestCase): + """A channel row is never an independent candidate, however its parent is modelled.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspClass", ["3"]) + cls.module_type = _plain_module_type(manufacturer, "ProspClass-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="et-0/0/{bay_position}", + ) + + def _family(self): + """Install a module and bind a channel to a parent that declares no channel count.""" + module, _bay = self._install(self.module_type, "3", run_rules=False) + parent = Interface.objects.get(module=module) + child = Interface.objects.create( + device=self.device, module=module, name="3:1", type=CHANNEL_TYPE, parent=parent, channel_id=1 + ) + return module, parent, child + + def test_such_a_channel_is_previewed_with_its_parent_and_never_on_its_own(self): + """The apply path carries such a child along with its parent, so the preview must too.""" + from netbox_interface_name_rules.engine import find_interfaces_for_rule + + _module, parent, _child = self._family() + + preview, checked = find_interfaces_for_rule(self.rule) + + self.assertEqual(checked, 1) + self.assertEqual([entry["interface"].pk for entry in preview], [parent.pk]) + self.assertEqual(preview[0]["new_names"], ["et-0/0/3", "et-0/0/3:1"]) + + def test_applying_to_the_channel_alone_changes_nothing(self): + """A previewed name always belongs to a parent, so selecting the channel is not a candidate.""" + from netbox_interface_name_rules.engine import apply_rule_to_existing + + module, _parent, child = self._family() + + self.assertEqual(apply_rule_to_existing(self.rule, interface_ids=[child.pk]).changed_count, 0) + self.assertEqual(self._names(module), ["3", "3:1"]) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class PreviewReadsNoTemplatesForDerivableSuffixesTest(ChannelizationTestCase): + """A channel whose own name carries its parent's prefix needs no template lookup.""" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device("ProspQuery", ["3"]) + cls.module_type = _channelized_module_type(manufacturer, "ProspQuery-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template="et-0/0/{bay_position}", + ) + + def test_previewing_a_channelized_family_reads_no_interface_templates(self): + from django.db import connection + from django.test.utils import CaptureQueriesContext + + from netbox_interface_name_rules.engine import find_interfaces_for_rule + + self._install(self.module_type, "3", run_rules=False) + + with CaptureQueriesContext(connection) as queries: + preview, _checked = find_interfaces_for_rule(self.rule) + + self.assertEqual(len(preview), 1) + template_reads = [ + query["sql"] for query in queries.captured_queries if "dcim_interfacetemplate" in query["sql"] + ] + self.assertEqual(template_reads, []) diff --git a/netbox_interface_name_rules/tests/test_regex.py b/netbox_interface_name_rules/tests/test_regex.py index cf2ae7f..11808a1 100644 --- a/netbox_interface_name_rules/tests/test_regex.py +++ b/netbox_interface_name_rules/tests/test_regex.py @@ -15,11 +15,97 @@ Site, ) from django.core.exceptions import ValidationError +from django.db import connection +from django.db.migrations.executor import MigrationExecutor from django.test import TestCase -from netbox_interface_name_rules.engine import apply_interface_name_rules, find_matching_rule +from netbox_interface_name_rules.engine import apply_interface_name_rules, find_matching_rule, has_applicable_interfaces from netbox_interface_name_rules.models import InterfaceNameRule +_SUPPORTED_PATTERNS = ( + "QSFP-.*", + "QSFP28-(100G|40G)", + r"^ge-\d+/\d+/\d+$", + r"[A-Z]+-\d+", + r"Ethernet\d+", + "^(10|25|100)G$", + "(?:QSFP|SFP)+", + r"(?i)^qsfp-.*$", + "^(a+)+$", + "(?:a|aa)+", + "^" + "(a|aa)" * 40 + "$", +) + +_PYTHON_ONLY_PATTERNS = ( + r"(?=QSFP)QSFP", + r"(?<=QSFP-)100G", + r"(QSFP)\1", + r"(?>QSFP)", + r"(?a:QSFP)", + r"QSFP\Z", +) + + +class RegexPatternEngineTest(TestCase): + """Compile and execute stored patterns with RE2's bounded engine.""" + + def _rule(self, pattern): + return InterfaceNameRule( + module_type_is_regex=True, + module_type_pattern=pattern, + name_template="port{bay_position}", + ) + + def test_backtracking_shape_is_accepted_and_matches_through_rule_selection(self): + rule = self._rule("^(a+)+$") + rule.clean() + rule.save() + manufacturer = Manufacturer.objects.create(name="Linear Regex", slug="linear-regex") + module_type = ModuleType.objects.create(manufacturer=manufacturer, model="aaaa", part_number="LINEAR") + + self.assertEqual(find_matching_rule(module_type, None, None), rule) + + def test_supported_patterns_are_accepted(self): + for pattern in _SUPPORTED_PATTERNS: + with self.subTest(pattern=pattern): + self._rule(pattern).clean() + + def test_python_only_patterns_are_refused(self): + for pattern in _PYTHON_ONLY_PATTERNS: + with self.subTest(pattern=pattern), self.assertRaises(ValidationError) as ctx: + self._rule(pattern).clean() + self.assertIn("module_type_pattern", ctx.exception.message_dict) + self.assertIn("Invalid RE2 pattern", ctx.exception.messages[0]) + + def test_device_level_rule_refuses_python_only_syntax(self): + rule = InterfaceNameRule( + applies_to_device_interfaces=True, + module_type_pattern=r"(?=Gi)Gi\d+/\d+", + name_template="port{bay_position}", + ) + + with self.assertRaises(ValidationError) as ctx: + rule.clean() + + self.assertIn("module_type_pattern", ctx.exception.message_dict) + + def test_rule_tester_refuses_python_only_syntax(self): + from netbox_interface_name_rules.forms import RuleTestForm + + form = RuleTestForm( + { + "name_template": "et-0/0/{bay_position}", + "channel_count": "0", + "channel_start": "0", + "module_type_is_regex": True, + "module_type_pattern": r"(?=QSFP)QSFP-.*", + } + ) + + self.assertFalse(form.is_valid()) + self.assertIn("module_type_pattern", form.errors) + self.assertIn("Invalid RE2 pattern", form.errors["module_type_pattern"][0]) + class RegexModelValidationTest(TestCase): """Test model clean() validation for regex fields.""" @@ -119,6 +205,27 @@ def test_regex_does_not_match_unrelated(self): ) self.assertIsNone(find_matching_rule(self.sfp_10g, None, None)) + def test_python_only_pattern_written_without_validation_is_not_evaluated(self): + """Rule selection skips a legacy RE2-incompatible row and returns the next match.""" + fallback = InterfaceNameRule.objects.create( + module_type_is_regex=True, + module_type_pattern="abab", + name_template="fallback/{bay_position}", + ) + incompatible = InterfaceNameRule.objects.create( + module_type_is_regex=True, + module_type_pattern="unused", + name_template="incompatible/{bay_position}", + ) + InterfaceNameRule.objects.filter(pk=incompatible.pk).update(module_type_pattern=r"(?=abab)abab") + module_type = ModuleType.objects.create( + manufacturer=self.sfp_lr4.manufacturer, + model="abab", + part_number="CASE-OVERLAP", + ) + + self.assertEqual(find_matching_rule(module_type, None, None), fallback) + def test_exact_takes_priority_over_regex(self): """Exact FK match is preferred over regex pattern match.""" InterfaceNameRule.objects.create( @@ -154,7 +261,7 @@ def test_regex_with_device_type_specificity(self): self.assertEqual(result, generic) def test_regex_fullmatch_not_partial(self): - """re.fullmatch requires the entire model name to match.""" + """A regex rule requires the entire model name to match.""" InterfaceNameRule.objects.create( module_type_is_regex=True, module_type_pattern="400G", # Partial — should NOT match @@ -232,3 +339,128 @@ def test_same_regex_rule_different_module_type(self): self.assertEqual(renamed, 1) iface.refresh_from_db() self.assertEqual(iface.name, "FourHundredGigE0/0/0/1") + + def test_python_only_pattern_written_without_validation_is_not_applicable(self): + """The applicability scan must not execute a legacy pattern outside RE2 syntax.""" + rule = InterfaceNameRule.objects.create( + module_type_is_regex=True, + module_type_pattern=r"(?=QSFP)QSFP-DD-400G-.*", + name_template="FourHundredGigE0/0/0/{bay_position}", + ) + bay = ModuleBay.objects.get(device=self.device, name="Transceiver 0") + module = Module.objects.create(device=self.device, module_bay=bay, module_type=self.mt_lr4) + Interface.objects.create(device=self.device, module=module, name="0", type="400gbase-x-osfp") + + self.assertFalse(has_applicable_interfaces(rule)) + + +class RegexEngineMigrationTest(TestCase): + """Audit stored rule patterns before the plugin starts executing them with RE2.""" + + APP = "netbox_interface_name_rules" + BEFORE = "0013_interfacenamerule_breakout_mode_and_more" + + def _migrate(self, target): + executor = MigrationExecutor(connection) + executor.loader.build_graph() + return executor.migrate([(self.APP, target)]) + + def _latest_migration(self): + loader = MigrationExecutor(connection).loader + return sorted(name for app_label, name in loader.graph.leaf_nodes(self.APP))[-1] + + def test_upgrade_refuses_a_stored_pattern_outside_re2_syntax(self): + latest = self._latest_migration() + old_state = self._migrate(self.BEFORE) + Rule = old_state.apps.get_model(self.APP, "InterfaceNameRule") + rule = Rule.objects.create( + applies_to_device_interfaces=True, + module_type_pattern=r"(?=Gi)Gi\d+/\d+", + name_template="Gi{vc_position}/{port}", + ) + connection.check_constraints() + + try: + with self.assertRaisesRegex(RuntimeError, rf"InterfaceNameRule ID: {rule.pk}\b"): + self._migrate(latest) + finally: + Rule.objects.filter(pk=rule.pk).delete() + self._migrate(latest) + + def test_upgrade_refuses_a_stored_pattern_with_different_re2_semantics(self): + latest = self._latest_migration() + old_state = self._migrate(self.BEFORE) + Rule = old_state.apps.get_model(self.APP, "InterfaceNameRule") + rules = [ + Rule.objects.create( + applies_to_device_interfaces=True, + module_type_pattern=pattern, + name_template="Gi{vc_position}/{port}", + ) + for pattern in ( + r"\w+", + r"\d+", + r"\s+", + r"\bMóduló\b", + r"(?i)i", + r"[[:alpha:]]+", + r"[[:^alpha:]]+", + r"a{,3}", + r"a{01}", + r"a{1,03}", + ) + ] + connection.check_constraints() + + try: + with self.assertRaises(RuntimeError) as ctx: + self._migrate(latest) + for rule in rules: + self.assertRegex(str(ctx.exception), rf"\b{rule.pk}\b") + finally: + Rule.objects.filter(pk__in=[rule.pk for rule in rules]).delete() + self._migrate(latest) + + def test_upgrade_preserves_unambiguous_shared_syntax(self): + latest = self._latest_migration() + old_state = self._migrate(self.BEFORE) + Rule = old_state.apps.get_model(self.APP, "InterfaceNameRule") + patterns = (r"\\w+", r"a{1,3}") + rules = [ + Rule.objects.create( + applies_to_device_interfaces=True, + module_type_pattern=pattern, + name_template="Gi{vc_position}/{port}", + ) + for pattern in patterns + ] + connection.check_constraints() + + try: + new_state = self._migrate(latest) + migrated_patterns = new_state.apps.get_model(self.APP, "InterfaceNameRule").objects.filter( + pk__in=[rule.pk for rule in rules] + ) + self.assertEqual(set(migrated_patterns.values_list("module_type_pattern", flat=True)), set(patterns)) + finally: + Rule.objects.filter(pk__in=[rule.pk for rule in rules]).delete() + self._migrate(latest) + + def test_upgrade_preserves_a_backtracking_pattern_that_re2_can_execute(self): + latest = self._latest_migration() + old_state = self._migrate(self.BEFORE) + Rule = old_state.apps.get_model(self.APP, "InterfaceNameRule") + rule = Rule.objects.create( + applies_to_device_interfaces=True, + module_type_pattern="^(a+)+$", + name_template="Gi{vc_position}/{port}", + ) + connection.check_constraints() + + try: + new_state = self._migrate(latest) + migrated_rule = new_state.apps.get_model(self.APP, "InterfaceNameRule").objects.get(pk=rule.pk) + self.assertEqual(migrated_rule.module_type_pattern, "^(a+)+$") + finally: + Rule.objects.filter(pk=rule.pk).delete() + self._migrate(latest) diff --git a/netbox_interface_name_rules/tests/test_rule_selection.py b/netbox_interface_name_rules/tests/test_rule_selection.py new file mode 100644 index 0000000..d9b6f12 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_rule_selection.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for the lower-level rule-selection seam.""" + +from dcim.models import DeviceType, Manufacturer, ModuleType +from django.test import TestCase + +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.rule_selection import find_matching_rule + + +class RuleSelectionTest(TestCase): + """Exercise rule selection through its lower-level public interface.""" + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="SelectorMfg", slug="selector-mfg") + cls.module_type = ModuleType.objects.create( + manufacturer=manufacturer, + model="SELECTOR-SFP", + part_number="SELECTOR-SFP", + ) + cls.device_type = DeviceType.objects.create( + manufacturer=manufacturer, + model="SELECTOR-DEVICE", + slug="selector-device", + ) + + def test_exact_rule_prefers_matching_device_scope(self): + InterfaceNameRule.objects.create( + module_type=self.module_type, + name_template="generic{bay_position}", + ) + scoped_rule = InterfaceNameRule.objects.create( + module_type=self.module_type, + device_type=self.device_type, + name_template="scoped{bay_position}", + ) + + selected = find_matching_rule(self.module_type, None, self.device_type) + + self.assertEqual(selected, scoped_rule) diff --git a/netbox_interface_name_rules/tests/test_rules.py b/netbox_interface_name_rules/tests/test_rules.py index 528dd42..73f48b5 100644 --- a/netbox_interface_name_rules/tests/test_rules.py +++ b/netbox_interface_name_rules/tests/test_rules.py @@ -293,16 +293,16 @@ def setUpTestData(cls): cls.device_type = DeviceType.objects.create(manufacturer=mfr, model="C-DEV", slug="c-dev") def setUp(self): - # These tests mutate find_matching_rule()'s module-level _RULE_CACHE / _pin, which TestCase does + # These tests mutate rule_selection's module-level _RULE_CACHE / _pin, which TestCase does # NOT roll back (only the DB is). Reset them before each method so the class is order-independent # and a stale snapshot from a prior method can't be reused for a same-content rule set. - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection - engine._RULE_CACHE.update({"version": None, "exact": (), "regex": (), "memo": {}}) - engine._pin.depth = 0 - engine._pin.primed = False + rule_selection._RULE_CACHE.update({"version": None, "exact": (), "regex": (), "memo": {}}) + rule_selection._pin.depth = 0 + rule_selection._pin.primed = False for attr in ("exact", "regex", "memo"): - engine._pin.__dict__.pop(attr, None) + rule_selection._pin.__dict__.pop(attr, None) def test_repeated_calls_do_not_re_query_rules(self): """A second identical call issues at most the cheap fingerprint query — no per-candidate lookups.""" @@ -320,23 +320,23 @@ def test_repeated_calls_do_not_re_query_rules(self): def test_memoized_result_is_not_recomputed(self): """A repeat call with the same context returns the memoized rule without re-running tier matching.""" - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection InterfaceNameRule.objects.create(module_type=self.module_type, name_template="port{bay_position}") first = find_matching_rule(self.module_type, None, self.device_type) # warm, populates the memo calls = [] - real_find_exact = engine._find_exact_match + real_find_exact = rule_selection._find_exact_match def counting_find_exact(*args, **kwargs): calls.append(1) return real_find_exact(*args, **kwargs) - engine._find_exact_match = counting_find_exact + rule_selection._find_exact_match = counting_find_exact try: second = find_matching_rule(self.module_type, None, self.device_type) finally: - engine._find_exact_match = real_find_exact + rule_selection._find_exact_match = real_find_exact self.assertEqual(second, first) self.assertEqual(calls, [], "second call must be served from the memo, not re-matched") @@ -424,7 +424,7 @@ def test_compensating_fk_swap_changes_fingerprint(self): A Sum()-based fingerprint collides here (the summed ids are identical after the swap, count and last_updated unchanged); the per-row, pk-anchored content hash distinguishes them. """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection mfr = Manufacturer.objects.create(name="SwapMfg", slug="swapmfg") mt_a = ModuleType.objects.create(manufacturer=mfr, model="SWAP-A", part_number="SWAP-A") @@ -434,12 +434,12 @@ def test_compensating_fk_swap_changes_fingerprint(self): rule_a = InterfaceNameRule.objects.create(module_type=mt_a, device_type=dt_x, name_template="a{bay_position}") rule_b = InterfaceNameRule.objects.create(module_type=mt_b, device_type=dt_y, name_template="b{bay_position}") - before = engine._enabled_rules_version() + before = rule_selection._enabled_rules_version() # Swap device_type between the two rules via bulk .update(): SUM(device_type) is unchanged # (x+y == y+x), count unchanged, last_updated unbumped. Only the per-rule pairing differs. InterfaceNameRule.objects.filter(pk=rule_a.pk).update(device_type=dt_y) InterfaceNameRule.objects.filter(pk=rule_b.pk).update(device_type=dt_x) - after = engine._enabled_rules_version() + after = rule_selection._enabled_rules_version() self.assertNotEqual(before, after, "fingerprint collided on a compensating FK swap (Sum-aggregate weakness)") @@ -467,7 +467,7 @@ def test_none_module_type_returns_none(self): def test_find_exact_match_loads_rules_on_demand(self): """_find_exact_match loads the rule set itself when called without preloaded rules (direct-caller path).""" - from netbox_interface_name_rules.engine import _find_exact_match + from netbox_interface_name_rules.rule_selection import _find_exact_match rule = InterfaceNameRule.objects.create(module_type=self.module_type, name_template="p{bay_position}") # No exact_rules argument → the function loads them via _get_enabled_rules() rather than a passed-in set. @@ -475,10 +475,10 @@ def test_find_exact_match_loads_rules_on_demand(self): def test_memo_is_bounded(self): """The per-version memo is capped, so it can't grow without bound under a stable rule set.""" - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection - original_max = engine._MEMO_MAX - engine._MEMO_MAX = 2 + original_max = rule_selection._MEMO_MAX + rule_selection._MEMO_MAX = 2 try: mfr = Manufacturer.objects.create(name="MemoMfg", slug="memomfg") module_types = [ @@ -492,20 +492,20 @@ def test_memo_is_bounded(self): # would slip past an end-of-loop `<=` snapshot (which only sees the post-clear size) but # trips here on the very insert that crosses the cap. self.assertLessEqual( - len(engine._RULE_CACHE["memo"]), - engine._MEMO_MAX, + len(rule_selection._RULE_CACHE["memo"]), + rule_selection._MEMO_MAX, f"memo exceeded the cap mid-insertion after {i} contexts", ) # And the eviction actually fired: the final size is below the number of distinct contexts, # so the test isn't vacuously green on a memo that simply never reached the cap. self.assertLess( - len(engine._RULE_CACHE["memo"]), + len(rule_selection._RULE_CACHE["memo"]), len(module_types), "memo never evicted — the cap was never exercised", ) finally: - engine._MEMO_MAX = original_max + rule_selection._MEMO_MAX = original_max def test_pinned_rule_cache_skips_per_call_fingerprint(self): """Inside pinned_rule_cache() the per-call fingerprint query is elided; outside it still runs.""" @@ -560,7 +560,7 @@ def test_pinned_block_holds_snapshot_across_concurrent_cache_reload(self): """ import threading - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection from netbox_interface_name_rules.engine import pinned_rule_cache universal = InterfaceNameRule.objects.create(module_type=self.module_type, name_template="a{bay_position}") @@ -572,8 +572,13 @@ def concurrent_reload(): # inherits the main thread's active pin. It then publishes a different rule-set version the # way _get_enabled_rules() does: one atomic rebind of the module global. (No DB access — a # separate thread has its own connection and cannot see this TestCase's uncommitted rows.) - worker_pin_depth.append(getattr(engine._pin, "depth", 0)) - engine._RULE_CACHE = {"version": "concurrent-reload-other-version", "exact": (), "regex": (), "memo": {}} + worker_pin_depth.append(getattr(rule_selection._pin, "depth", 0)) + rule_selection._RULE_CACHE = { + "version": "concurrent-reload-other-version", + "exact": (), + "regex": (), + "memo": {}, + } try: with pinned_rule_cache(): @@ -586,7 +591,7 @@ def concurrent_reload(): # The worker did not inherit our pin — proves _pin is per-thread, not a shared global. self.assertEqual(worker_pin_depth, [0], "pin leaked across threads — _pin is not thread-local") # ...and its reload really did replace the shared cache. - self.assertEqual(engine._RULE_CACHE["version"], "concurrent-reload-other-version") + self.assertEqual(rule_selection._RULE_CACHE["version"], "concurrent-reload-other-version") # Still inside the pin: must serve the snapshot captured at entry, not the worker's reload. self.assertEqual( @@ -597,7 +602,7 @@ def concurrent_reload(): finally: # This test deliberately rebinds the module global from another thread; restore a clean # sentinel so the simulated reload can't leak into sibling tests even if setUp() is weakened. - engine._RULE_CACHE = {"version": None, "exact": (), "regex": (), "memo": {}} + rule_selection._RULE_CACHE = {"version": None, "exact": (), "regex": (), "memo": {}} def test_reload_publishes_a_fresh_cache_dict_atomically(self): """A version change rebinds _RULE_CACHE to a new dict instead of mutating the old one in place. @@ -607,12 +612,12 @@ def test_reload_publishes_a_fresh_cache_dict_atomically(self): with memo from V2. We assert the published dict is a *new object* and that a reference captured before the reload is left untouched — the in-place mutation the previous code did would fail both. """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection InterfaceNameRule.objects.create(module_type=self.module_type, name_template="a{bay_position}") find_matching_rule(self.module_type, None, self.device_type) # prime version 1 - snap = engine._RULE_CACHE + snap = rule_selection._RULE_CACHE snap_version = snap["version"] snap_exact = snap["exact"] @@ -623,7 +628,9 @@ def test_reload_publishes_a_fresh_cache_dict_atomically(self): find_matching_rule(self.module_type, None, self.device_type) # reload to version 2 self.assertIsNot( - engine._RULE_CACHE, snap, "reload mutated the cache dict in place instead of publishing a new one" + rule_selection._RULE_CACHE, + snap, + "reload mutated the cache dict in place instead of publishing a new one", ) self.assertEqual(snap["version"], snap_version, "a reload mutated a previously-published cache dict") self.assertEqual(snap["exact"], snap_exact, "the captured exact snapshot changed under a concurrent reload") @@ -638,7 +645,7 @@ def test_find_matching_rule_survives_concurrent_memo_clear(self): itself, then assert find_matching_rule still returns the rule. The sentinel ``.get()`` read is a single atomic lookup, so it never performs the separate membership test the race needs. """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection rule = InterfaceNameRule.objects.create(module_type=self.module_type, name_template="p{bay_position}") self.assertEqual(find_matching_rule(self.module_type, None, self.device_type), rule) # warm the memo @@ -652,7 +659,7 @@ def __contains__(self, key): return present # Same version → no reload; the racing memo is what find_matching_rule reads next. - engine._RULE_CACHE["memo"] = _RacingMemo(engine._RULE_CACHE["memo"]) + rule_selection._RULE_CACHE["memo"] = _RacingMemo(rule_selection._RULE_CACHE["memo"]) self.assertEqual( find_matching_rule(self.module_type, None, self.device_type), @@ -663,7 +670,7 @@ def __contains__(self, key): def test_pinned_block_uses_a_private_memo_copy(self): """A pinned batch gets its own memo copy, so another thread clearing the shared memo can't evict the batch's warmed entries (or wedge a KeyError) mid-loop.""" - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection from netbox_interface_name_rules.engine import pinned_rule_cache rule = InterfaceNameRule.objects.create(module_type=self.module_type, name_template="p{bay_position}") @@ -672,13 +679,13 @@ def test_pinned_block_uses_a_private_memo_copy(self): find_matching_rule(self.module_type, None, self.device_type) # primes + memoizes into the copy self.assertIsNot( - engine._pin.memo, - engine._RULE_CACHE["memo"], + rule_selection._pin.memo, + rule_selection._RULE_CACHE["memo"], "pinned memo aliases the shared cache memo instead of holding a private copy", ) # An unpinned thread clearing the shared memo at the cap must not disturb the pinned batch. - engine._RULE_CACHE["memo"].clear() + rule_selection._RULE_CACHE["memo"].clear() self.assertEqual( find_matching_rule(self.module_type, None, self.device_type), rule, @@ -693,7 +700,7 @@ def test_fingerprint_resists_separator_injection_in_text_fields(self): encoding produced for a two-rule set — two distinct rule sets hashing the same, silently missing a cache invalidation. Length-prefixed fields make the encoding self-delimiting, so they differ. """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection field_sep, row_sep = "\x1f", "\x1e" @@ -706,7 +713,7 @@ def test_fingerprint_resists_separator_injection_in_text_fields(self): r2 = InterfaceNameRule.objects.create( applies_to_device_interfaces=True, module_type_pattern="p2", name_template="b" ) - fp_two_rules = engine._enabled_rules_version() + fp_two_rules = rule_selection._enabled_rules_version() # Forge a one-rule set whose name_template embeds r1's trailing columns, a row separator, and # r2's columns up to its name_template. Column order: id, module_type_id, is_regex, pattern, @@ -716,7 +723,7 @@ def test_fingerprint_resists_separator_injection_in_text_fields(self): forged_name_template = field_sep.join(["a", "0", "0", "true"]) + row_sep + r2_cells_through_name InterfaceNameRule.objects.filter(pk=r2.pk).delete() InterfaceNameRule.objects.filter(pk=r1.pk).update(name_template=forged_name_template) - fp_forged_one_rule = engine._enabled_rules_version() + fp_forged_one_rule = rule_selection._enabled_rules_version() self.assertNotEqual( fp_two_rules, @@ -731,14 +738,14 @@ class EnabledRuleFingerprintColumnsTest(TestCase): def test_version_columns_account_for_every_concrete_column(self): """Every concrete InterfaceNameRule column must be fingerprinted or deliberately excluded. - The enabled-rule fingerprint (engine._enabled_rules_version) hashes exactly - engine._VERSION_COLUMNS. If a future field that affects matching is added to the model but not + The enabled-rule fingerprint (rule_selection._enabled_rules_version) hashes exactly + rule_selection._VERSION_COLUMNS. If a future field that affects matching is added to the model but not to that hand-maintained tuple, edits to it won't change the fingerprint and find_matching_rule will keep serving a stale cached rule. This guard fails the moment a new concrete column appears that hasn't been consciously classified — forcing the author to either add it to _VERSION_COLUMNS (so the cache invalidates on its edits) or list it below with a reason it can't affect a match. """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection from netbox_interface_name_rules.models import InterfaceNameRule # Columns intentionally NOT in the fingerprint, each with the reason it cannot change a match: @@ -756,11 +763,11 @@ def test_version_columns_account_for_every_concrete_column(self): for f in InterfaceNameRule._meta.get_fields() if getattr(f, "concrete", False) and not f.many_to_many and f.column } - classified = set(engine._VERSION_COLUMNS) | excluded + classified = set(rule_selection._VERSION_COLUMNS) | excluded self.assertEqual( concrete_columns, classified, "InterfaceNameRule has concrete columns that are neither fingerprinted nor excluded. " - "Classify each: add match-affecting columns to engine._VERSION_COLUMNS so the rule cache " + "Classify each: add match-affecting columns to rule_selection._VERSION_COLUMNS so the rule cache " "invalidates when they change, or add audit/notes columns to this test's `excluded` set.", ) diff --git a/netbox_interface_name_rules/tests/test_signals.py b/netbox_interface_name_rules/tests/test_signals.py index b3d1c78..eb5501f 100644 --- a/netbox_interface_name_rules/tests/test_signals.py +++ b/netbox_interface_name_rules/tests/test_signals.py @@ -253,7 +253,7 @@ def test_deferred_device_reapply_pins_rule_cache_across_modules(self): module's interface (real work) while reading the enabled-rule fingerprint a single time for the whole device, proving the pin is engaged on INR's own batch path (no cross-plugin API). """ - from netbox_interface_name_rules import engine + from netbox_interface_name_rules import rule_selection InterfaceNameRule.objects.create(module_type=self.module_type, name_template="et-0/0/{bay_position}") @@ -264,17 +264,17 @@ def test_deferred_device_reapply_pins_rule_cache_across_modules(self): iface1 = Interface.objects.create(device=self.device, module=module1, name="old-1", type="10gbase-x-sfpp") calls = [] - real_version = engine._enabled_rules_version + real_version = rule_selection._enabled_rules_version def counting_version(*args, **kwargs): calls.append(1) return real_version(*args, **kwargs) - engine._enabled_rules_version = counting_version + rule_selection._enabled_rules_version = counting_version try: _apply_rules_for_device_deferred(self.device.pk) finally: - engine._enabled_rules_version = real_version + rule_selection._enabled_rules_version = real_version iface0.refresh_from_db() iface1.refresh_from_db() diff --git a/netbox_interface_name_rules/tests/test_structural_families.py b/netbox_interface_name_rules/tests/test_structural_families.py new file mode 100644 index 0000000..7e08845 --- /dev/null +++ b/netbox_interface_name_rules/tests/test_structural_families.py @@ -0,0 +1,419 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Integration tests for structural (channelized) interface-family plans. + +A structural plan turns one plain port into the topology NetBox models: the base row becomes the +physical parent and N channel subinterfaces are created under it. The whole family is one unit. +It is installed completely or not at all. The family module, not the engine, decides whether +the active NetBox release can hold it. +""" + +from unittest import skipIf, skipUnless + +from dcim.models import Device, DeviceRole, DeviceType, Interface, Manufacturer, Site +from django.db import IntegrityError, connection +from django.test import TestCase +from django.test.utils import CaptureQueriesContext + +from netbox_interface_name_rules.engine import ( + apply_interface_name_rules, + build_variables, + supports_channelization, +) +from netbox_interface_name_rules.family import ( + FamilyStatus, + FamilyTopology, + execute_structural_family, + install_channelized_family, + plan_structural_family, +) +from netbox_interface_name_rules.family import names as family_names +from netbox_interface_name_rules.models import InterfaceNameRule +from netbox_interface_name_rules.tests.out_of_band import rename_out_of_band +from netbox_interface_name_rules.tests.test_breakout_mode import CHANNELIZED, _plain_module_type +from netbox_interface_name_rules.tests.test_channelization import ( + PLAIN_TYPE, + PLUGIN_LOGGER, + REQUIRES_CHANNELIZATION, + ChannelizationTestCase, + _build_device, +) + +REQUIRES_NO_CHANNELIZATION = "requires a NetBox that cannot model channelized interfaces (4.6 and older)" + + +class StructuralFamilyTestCase(ChannelizationTestCase): + """One device, one plain module type, and the channelized rule under test.""" + + NAME_TEMPLATE = "xe-0/0/{bay_position}:{channel}" + PREFIX = "Struct" + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device(cls.PREFIX, ["3", "4"]) + cls.module_type = _plain_module_type(manufacturer, f"{cls.PREFIX}-QSFP") + cls.rule = InterfaceNameRule.objects.create( + module_type=cls.module_type, + name_template=cls.NAME_TEMPLATE, + parent_name_template="et-0/0/{bay_position}", + breakout_mode=CHANNELIZED, + channel_count=4, + channel_start=0, + ) + + def _plan(self, position="3"): + """Install a raw-named module and return its module, bay and structural plan.""" + module, bay = self._install(self.module_type, position, run_rules=False) + base = Interface.objects.get(module=module) + plan = plan_structural_family(module, self.rule, build_variables(bay, device=self.device), base) + return module, bay, plan + + +@skipIf(supports_channelization(), REQUIRES_NO_CHANNELIZATION) +class StructuralFamilyWithoutChannelizationTest(StructuralFamilyTestCase): + """Where NetBox has no channel model the family module says so, in the outcome itself.""" + + PREFIX = "StructNoSup" + + def test_the_plan_reports_unsupported_and_describes_no_channels(self): + _module, _bay, plan = self._plan() + + self.assertEqual(plan.precondition_status, FamilyStatus.UNSUPPORTED) + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertEqual(plan.channels, ()) + self.assertEqual(plan.channel_count, 0) + self.assertEqual(plan.target_names, ("3",)) + self.assertEqual(plan.base.name, "3") + + def test_executing_an_unsupported_plan_creates_nothing(self): + module, _bay, plan = self._plan() + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.UNSUPPORTED) + self.assertEqual(outcome.changed_count, 0) + self.assertEqual(self._names(module), ["3"]) + self.assertEqual(len(logs.records), 1, logs.output) + self.assertIn("channelized", logs.output[0].lower()) + + def test_the_outcome_names_the_base_interface_it_left_alone(self): + module, _bay, plan = self._plan() + base = Interface.objects.get(module=module) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING"): + outcome = execute_structural_family(plan) + + self.assertEqual(len(outcome.members), 1) + member = outcome.members[0] + self.assertEqual(member.interface_pk, base.pk) + self.assertEqual(member.current_name, "3") + self.assertEqual(member.status, FamilyStatus.UNSUPPORTED) + self.assertTrue(member.reason) + + def test_the_install_entry_point_reaches_the_same_outcome(self): + module, bay = self._install(self.module_type, "4", run_rules=False) + base = Interface.objects.get(module=module) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING"): + outcome = install_channelized_family(module, self.rule, build_variables(bay, device=self.device), base) + + self.assertEqual(outcome.status, FamilyStatus.UNSUPPORTED) + self.assertEqual(self._names(module), ["4"]) + + +class DeferredChannelNameReconciliationTest(TestCase): + """The family module restores the channel names NetBox's deferred parent cascade overwrites. + + The reconciliation is plain name arithmetic over committed rows, so it is exercised here on + ordinary interfaces: no release-specific channel model is involved in the decision it makes. + """ + + @classmethod + def setUpTestData(cls): + manufacturer = Manufacturer.objects.create(name="ReconMfg", slug="recon-mfg") + device_type = DeviceType.objects.create(manufacturer=manufacturer, model="RECON-DEVICE", slug="recon-device") + role = DeviceRole.objects.create(name="ReconRole", slug="recon-role") + site = Site.objects.create(name="ReconSite", slug="recon-site") + cls.device = Device.objects.create(name="recon-device-01", device_type=device_type, role=role, site=site) + + def _interface(self, name): + """Create one plain interface on the shared device.""" + return Interface.objects.create(device=self.device, name=name, type=PLAIN_TYPE) + + @staticmethod + def _cascade(child, name): + """Rename *child* the way NetBox's parent cascade does, without running the plugin.""" + rename_out_of_band(child, name) + + def test_the_intended_name_is_restored_after_the_cascade(self): + child = self._interface("et-0/0/3:1") + + with self.captureOnCommitCallbacks(execute=True) as callbacks: + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "xe-0/0/3", ((child.pk, 1, "et-0/0/3:1"),), child._state.db + ) + self._cascade(child, "xe-0/0/3:1") + + self.assertTrue(callbacks, "no deferred reconciliation was registered") + child.refresh_from_db() + self.assertEqual(child.name, "et-0/0/3:1") + + def test_a_parent_that_kept_its_name_registers_no_callback(self): + child = self._interface("et-0/0/3:1") + + with self.captureOnCommitCallbacks(execute=True) as callbacks: + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "et-0/0/3", ((child.pk, 1, "et-0/0/3:1"),), child._state.db + ) + + self.assertEqual(callbacks, []) + + def test_a_channel_the_cascade_will_not_touch_registers_no_callback(self): + child = self._interface("ge-0/0/3-1") + + with self.captureOnCommitCallbacks(execute=True) as callbacks: + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "xe-0/0/3", ((child.pk, 1, "ge-0/0/3-1"),), child._state.db + ) + + self.assertEqual(callbacks, []) + + def test_a_channel_the_cascade_left_alone_keeps_its_intended_name(self): + child = self._interface("et-0/0/3:1") + + with self.captureOnCommitCallbacks(execute=True): + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "xe-0/0/3", ((child.pk, 1, "et-0/0/3:1"),), child._state.db + ) + + child.refresh_from_db() + self.assertEqual(child.name, "et-0/0/3:1") + + def test_a_channel_moved_to_an_unexpected_name_is_left_alone(self): + child = self._interface("et-0/0/3:1") + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + with self.captureOnCommitCallbacks(execute=True): + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "xe-0/0/3", ((child.pk, 1, "et-0/0/3:1"),), child._state.db + ) + self._cascade(child, "someone-else-renamed-it") + + child.refresh_from_db() + self.assertEqual(child.name, "someone-else-renamed-it") + self.assertTrue(any("unexpected name" in line for line in logs.output), logs.output) + + def test_a_name_taken_since_the_cascade_is_not_reclaimed(self): + child = self._interface("et-0/0/3:1") + + with self.assertLogs(PLUGIN_LOGGER, level="ERROR"), self.captureOnCommitCallbacks(execute=True): + family_names.reconcile_after_parent_cascade( + "et-0/0/3", "xe-0/0/3", ((child.pk, 1, "et-0/0/3:1"),), child._state.db + ) + self._cascade(child, "xe-0/0/3:1") + occupant = self._interface("et-0/0/3:1") + + child.refresh_from_db() + occupant.refresh_from_db() + self.assertEqual(child.name, "xe-0/0/3:1") + self.assertEqual(occupant.name, "et-0/0/3:1") + + def test_the_reconciliation_locks_only_interfaces_in_primary_key_order(self): + """A joined device row must not be locked, and overlapping runs must agree on lock order.""" + children = [self._interface("et-0/0/3:1"), self._interface("et-0/0/3:2")] + reconciliations = tuple( + (child.pk, child.name, f"xe-0/0/3:{index}") for index, child in enumerate(children, start=1) + ) + + with CaptureQueriesContext(connection) as queries: + family_names.restore_deferred_channel_names(reconciliations, children[0]._state.db) + + locking = [query["sql"] for query in queries.captured_queries if "FOR UPDATE" in query["sql"]] + self.assertEqual(len(locking), 1, locking) + self.assertIn('FOR UPDATE OF "dcim_interface"', locking[0]) + self.assertNotIn("dcim_device", locking[0].split("FOR UPDATE")[1]) + self.assertIn('ORDER BY "dcim_interface"."id" ASC', locking[0]) + + def test_an_unrelated_integrity_failure_propagates(self): + interface = self._interface("cascade-name") + + def reject_interface_update(execute, sql, params, many, context): + if sql.lstrip().startswith('UPDATE "dcim_interface"'): + raise IntegrityError("injected deferred database failure") + return execute(sql, params, many, context) + + with connection.execute_wrapper(reject_interface_update): + with self.assertRaisesMessage(IntegrityError, "injected deferred database failure"): + family_names.restore_deferred_channel_names( + ((interface.pk, "final-name", "cascade-name"),), + interface._state.db, + ) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class StructuralFamilyPlanTest(StructuralFamilyTestCase): + """The plan describes the whole family before a single row is written.""" + + PREFIX = "StructPlan" + + def test_the_plan_describes_the_parent_capacity_channels_and_snapshot(self): + module, _bay, plan = self._plan() + base = Interface.objects.get(module=module) + + self.assertIsNone(plan.precondition_status) + self.assertEqual(plan.topology, FamilyTopology.CHANNELIZED) + self.assertEqual(plan.module_id, module.pk) + self.assertEqual(plan.device_id, self.device.pk) + self.assertEqual(plan.parent_target_name, "et-0/0/3") + self.assertEqual(plan.channel_count, 4) + self.assertEqual([channel.channel_id for channel in plan.channels], [1, 2, 3, 4]) + self.assertEqual( + plan.target_names, + ("et-0/0/3", "xe-0/0/3:0", "xe-0/0/3:1", "xe-0/0/3:2", "xe-0/0/3:3"), + ) + self.assertEqual(plan.base.pk, base.pk) + self.assertEqual(plan.base.name, "3") + self.assertIsNone(plan.base.channels) + + def test_planning_writes_nothing(self): + module, _bay, _plan = self._plan() + + self.assertEqual(self._names(module), ["3"]) + + def test_execution_creates_the_complete_family(self): + module, _bay, plan = self._plan() + + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.CHANGED) + self.assertEqual(outcome.changed_count, 5) + self.assertEqual(sorted(self._names(module)), sorted(plan.target_names)) + parent = self._parent(module) + self.assertEqual(parent.pk, plan.base.pk) + self.assertEqual(parent.channels, 4) + self.assertEqual( + sorted( + Interface.objects.filter(parent=parent).values_list("channel_id", "name"), + ), + [(1, "xe-0/0/3:0"), (2, "xe-0/0/3:1"), (3, "xe-0/0/3:2"), (4, "xe-0/0/3:3")], + ) + + def test_a_base_that_changed_after_planning_is_stale_and_untouched(self): + module, _bay, plan = self._plan() + rename_out_of_band(Interface.objects.get(pk=plan.base.pk), "renamed-by-someone-else") + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.STALE) + self.assertEqual(self._names(module), ["renamed-by-someone-else"]) + self.assertTrue(any("changed after planning" in line for line in logs.output), logs.output) + + def test_a_sibling_added_after_planning_is_stale(self): + """A row added after planning would be stranded beside the family, so the plan is refused.""" + module, _bay, plan = self._plan() + Interface.objects.create(device=self.device, module=module, name="added-after-planning", type=PLAIN_TYPE) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.STALE) + self.assertEqual(self._names(module), ["3", "added-after-planning"]) + self.assertTrue(any("module's interfaces changed" in line for line in logs.output), logs.output) + + def test_a_base_deleted_after_planning_is_stale(self): + module, _bay, plan = self._plan() + Interface.objects.filter(pk=plan.base.pk).delete() + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING"): + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.STALE) + self.assertEqual(self._names(module), []) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class StructuralFamilyRollbackTest(StructuralFamilyTestCase): + """A family that cannot be completed leaves the port exactly as NetBox instantiated it.""" + + PREFIX = "StructRoll" + # Every channel resolves to one name, so the second insert hits NetBox's own uniqueness index. + NAME_TEMPLATE = "xe-0/0/{bay_position}" + + def _assert_untouched(self, module): + """Assert the port is still one plain, raw-named row with no channel anywhere on it.""" + self.assertEqual(Interface.objects.filter(module=module).count(), 1) + base = Interface.objects.get(module=module) + self.assertEqual(base.name, "3") + self.assertIsNone(base.channels) + + def test_a_second_channel_with_the_same_name_rolls_the_family_back(self): + module, _bay, plan = self._plan() + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.BLOCKED) + self.assertEqual(outcome.changed_count, 0) + self.assertTrue(outcome.reason) + self._assert_untouched(module) + self.assertTrue(any("already exists" in line for line in logs.output), logs.output) + + def test_the_install_entry_point_reports_the_rollback(self): + module, bay = self._install(self.module_type, "3", run_rules=False) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING"): + built = apply_interface_name_rules(module, bay) + + self.assertEqual(built, 0) + self._assert_untouched(module) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class StructuralFamilyValidationTest(StructuralFamilyTestCase): + """NetBox's own validation runs against the real rows, inside the structural transaction.""" + + PREFIX = "StructVal" + # 64 characters is NetBox's interface-name limit, so every channel name here is rejected. + NAME_TEMPLATE = "xe-0/0/{bay_position}:{channel}" + "x" * 64 + + def test_a_channel_netbox_rejects_rolls_the_whole_family_back(self): + module, _bay, plan = self._plan() + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + outcome = execute_structural_family(plan) + + self.assertEqual(outcome.status, FamilyStatus.BLOCKED) + self.assertEqual(outcome.changed_count, 0) + self.assertEqual(Interface.objects.filter(module=module).count(), 1) + base = Interface.objects.get(module=module) + self.assertEqual(base.name, "3") + self.assertIsNone(base.channels) + self.assertTrue(any("64 characters" in line for line in logs.output), logs.output) + + +@skipUnless(supports_channelization(), REQUIRES_CHANNELIZATION) +class StructuralFamilyTemplateFailureTest(StructuralFamilyTestCase): + """A rule whose channel template cannot be evaluated builds nothing and says why.""" + + PREFIX = "StructTpl" + NAME_TEMPLATE = "xe-0/0/{bay_position}:{missing_variable}" + + def test_the_plan_carries_the_template_failure_instead_of_raising(self): + _module, _bay, plan = self._plan() + + self.assertEqual(plan.precondition_status, FamilyStatus.FAILED) + self.assertEqual(plan.channels, ()) + self.assertIn("missing_variable", plan.precondition_reason) + + def test_the_install_creates_nothing_and_does_not_read_the_rule_as_obsolete(self): + module, bay = self._install(self.module_type, "4", run_rules=False) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + built = apply_interface_name_rules(module, bay) + + self.assertEqual(built, 0) + self.assertEqual(self._names(module), ["4"]) + self.assertFalse(self.rule.tags.filter(slug="potentially-deprecated").exists()) + self.assertTrue(any("missing_variable" in line for line in logs.output), logs.output) diff --git a/netbox_interface_name_rules/tests/test_vc_drift.py b/netbox_interface_name_rules/tests/test_vc_drift.py index 8271883..07d0c61 100644 --- a/netbox_interface_name_rules/tests/test_vc_drift.py +++ b/netbox_interface_name_rules/tests/test_vc_drift.py @@ -56,6 +56,7 @@ ) from netbox_interface_name_rules.models import InterfaceNameRule from netbox_interface_name_rules.signals import _apply_rules_deferred +from netbox_interface_name_rules.tests.out_of_band import rename_out_of_band from netbox_interface_name_rules.tests.test_channelization import ( CHANNEL_TYPE, PARENT_TYPE, @@ -307,7 +308,7 @@ def test_the_apply_page_path_never_consulted_raw_names(self): self._leave() rule = InterfaceNameRule.objects.create(module_type=self.module_type, name_template="et-0/0/{bay_position}") - self.assertEqual(apply_rule_to_existing(rule), 1) + self.assertEqual(apply_rule_to_existing(rule).changed_count, 1) self.assertEqual(self._names(module), ["et-0/0/5"]) @@ -340,7 +341,7 @@ def test_a_matcher_that_claims_two_interfaces_renames_neither(self): module, bay = self._install_on(self.device, self.decoy_type, "3") self.assertEqual(self._names(module), ["mgmt-3", "xe-1/0/3"]) # An earlier rename left the plain template's interface sitting on the token template's fallback name. - Interface.objects.filter(module=module, name="mgmt-3").update(name="xe-0/0/3") + rename_out_of_band(Interface.objects.get(module=module, name="mgmt-3"), "xe-0/0/3") self._renumber(2) InterfaceNameRule.objects.create(module_type=self.decoy_type, name_template="et-{base}") @@ -362,7 +363,7 @@ def test_a_forced_re_apply_does_not_break_out_an_ambiguous_pair(self): downstream refuses the second one: the guard has to be the thing that stops it. """ module, _ = self._install_on(self.device, self.decoy_type, "3") - Interface.objects.filter(module=module, name="mgmt-3").update(name="xe-0/0/3") + rename_out_of_band(Interface.objects.get(module=module, name="mgmt-3"), "xe-0/0/3") InterfaceNameRule.objects.create( module_type=self.decoy_type, name_template="et-{base}:{channel}", channel_count=2, channel_start=0 ) @@ -377,6 +378,30 @@ def test_a_forced_re_apply_does_not_break_out_an_ambiguous_pair(self): for candidate in ("xe-0/0/3", "xe-1/0/3"): self.assertIn(candidate, output) + def test_a_forced_re_apply_does_not_break_out_an_ambiguous_pair_with_one_target(self): + """The same claim where both bases intend one family, so nothing downstream tells them apart. + + A breakout rule without ``{base}`` gives every base on the module the same names, so the two + claimed rows collapse into one family before anything is written. The guard has to refuse + the pair while it can still see both of them. + """ + module, _ = self._install_on(self.device, self.decoy_type, "3") + rename_out_of_band(Interface.objects.get(module=module, name="mgmt-3"), "xe-0/0/3") + InterfaceNameRule.objects.create( + module_type=self.decoy_type, + name_template="et-0/0/{bay_position}:{channel}", + channel_count=2, + channel_start=0, + ) + + with self.assertLogs(PLUGIN_LOGGER, level="WARNING") as logs: + self._renumber(2) + + self.assertEqual(self._names(module), ["xe-0/0/3", "xe-1/0/3"]) + output = "\n".join(logs.output) + for candidate in ("xe-0/0/3", "xe-1/0/3"): + self.assertIn(candidate, output) + def test_an_interface_claimed_by_two_templates_renames_nothing(self): """Two matchers over one name is the same failure seen from the other side; both claims are dropped.""" module, bay = self._install_on(self.device, self.overlap_type, "4") @@ -403,6 +428,55 @@ def test_two_token_templates_that_do_not_overlap_both_rename(self): self.assertEqual(self._names(module), ["et-xe-1/5/4", "et-xe-5/0/4"]) +class VcPositionAdjacentTokenTest(VcDriftTestCase): + """Tokens with nothing between them cannot be told apart, so the template builds no matcher. + + Back-to-back tokens expand to back-to-back numeric alternatives, which backtrack for an + unbounded time on a name that does not match. Every remaining token matches only as many + digits as ``vc_position`` can hold. + """ + + @classmethod + def setUpTestData(cls): + manufacturer, cls.device = _build_device( + "VcAdj", ["3", "4"], virtual_chassis=VirtualChassis.objects.create(name="vcadj-vc"), vc_position=1 + ) + cls.adjacent_type = _token_module_type(manufacturer, "VcAdj-QSFP", "xe-{vc_position}{vc_position}/0/{module}") + cls.separated_type = _token_module_type(manufacturer, "VcAdj-SFP", "xe-{vc_position}/{vc_position}/{module}") + # A template name may legally spell the marker the matcher builder inserts for itself. + cls.sentinel_type = _token_module_type( + manufacturer, "VcAdj-QSFP28", "xe-InrVcPositionSentinel1End-{vc_position}/{module}" + ) + + @skipUnless(supports_vc_position_token(), REQUIRES_VC_POSITION_TOKEN) + def test_adjacent_tokens_build_no_matcher_at_all(self): + module, _ = self._install_on(self.device, self.adjacent_type, "3") + + self.assertEqual(self._names(module), ["xe-11/0/3"]) + self.assertEqual(_raw_name_patterns(module), []) + + @skipUnless(supports_vc_position_token(), REQUIRES_VC_POSITION_TOKEN) + def test_a_separated_token_matches_only_a_storable_position(self): + module, _ = self._install_on(self.device, self.separated_type, "4") + self.assertEqual(self._names(module), ["xe-1/1/4"]) + + patterns = _raw_name_patterns(module) + + self.assertEqual(len(patterns), 1) + self.assertNotIn(r"\d+", patterns[0].pattern) + self.assertTrue(patterns[0].fullmatch("xe-1/1/4")) + self.assertTrue(patterns[0].fullmatch("xe-2147483647/0/4")) # the largest position NetBox stores + self.assertIsNone(patterns[0].fullmatch("xe-12345678901/0/4")) + + @skipUnless(supports_vc_position_token(), REQUIRES_VC_POSITION_TOKEN) + def test_a_sentinel_shaped_literal_builds_no_matcher_instead_of_raising(self): + module, bay = self._install_on(self.device, self.sentinel_type, "3") + + self.assertEqual(self._names(module), ["xe-InrVcPositionSentinel1End-1/3"]) + self.assertEqual(_raw_name_patterns(module), []) + self.assertEqual(apply_interface_name_rules(module, bay), 0) + + # --------------------------------------------------------------------------- # Controls: no token, and no token support # --------------------------------------------------------------------------- @@ -470,7 +544,7 @@ def test_matching_takes_the_exact_only_path_without_the_constant(self): """Byte-identical pre-4.6 behaviour: a name only a matcher could claim is not a candidate.""" module, bay = self._install_on(self.device, self.module_type, "3") # Named explicitly, so the interface sits on a position variant whatever the release resolved. - Interface.objects.filter(module=module).update(name="xe-1/0/3") + rename_out_of_band(Interface.objects.get(module=module), "xe-1/0/3") self._renumber(2) InterfaceNameRule.objects.create(module_type=self.module_type, name_template="et-0/0/{bay_position}") @@ -664,12 +738,12 @@ def test_a_renumbered_family_is_identified_through_its_wrapped_base(self): self.assertEqual(self._names(module), [f"brk-xe-1/0/3:{channel}" for channel in range(4)]) self._switch_to_channelized(rule) - verdicts, _ = find_convertible_families(rule) + candidates = find_convertible_families(rule).candidates - self.assertEqual(len(verdicts), 1) - self.assertTrue(verdicts[0]["convertible"], verdicts[0]["reason"]) - self.assertEqual(verdicts[0]["current_names"], [f"brk-xe-1/0/3:{channel}" for channel in range(4)]) - self.assertEqual(verdicts[0]["new_names"][0], "et-0/0/3") + self.assertEqual(len(candidates), 1) + self.assertTrue(candidates[0].convertible, candidates[0].reason) + self.assertEqual(list(candidates[0].current_names), [f"brk-xe-1/0/3:{channel}" for channel in range(4)]) + self.assertEqual(candidates[0].new_names[0], "et-0/0/3") def test_a_template_that_repeats_the_base_is_recovered_through_a_backreference(self): """One capture group and a backreference — a repeated ``{base}`` must not become a second group.""" @@ -679,11 +753,11 @@ def test_a_template_that_repeats_the_base_is_recovered_through_a_backreference(s self._renumber(2) self._switch_to_channelized(rule) - verdicts, _ = find_convertible_families(rule) + candidates = find_convertible_families(rule).candidates - self.assertEqual(len(verdicts), 1) - self.assertTrue(verdicts[0]["convertible"], verdicts[0]["reason"]) - self.assertEqual(verdicts[0]["new_names"][0], "et-0/0/4") + self.assertEqual(len(candidates), 1) + self.assertTrue(candidates[0].convertible, candidates[0].reason) + self.assertEqual(candidates[0].new_names[0], "et-0/0/4") def test_a_base_inside_an_arithmetic_expression_is_skipped_cleanly(self): """A non-numeric sentinel cannot go through the arithmetic validator; the family is simply not offered.""" @@ -693,10 +767,10 @@ def test_a_base_inside_an_arithmetic_expression_is_skipped_cleanly(self): self._renumber(2) self._switch_to_channelized(rule) - verdicts, has_more = find_convertible_families(rule) + preview = find_convertible_families(rule) - self.assertEqual(verdicts, []) - self.assertFalse(has_more) + self.assertEqual(preview.candidates, ()) + self.assertFalse(preview.has_more) self.assertEqual(self._names(module), [f"p16:{channel}" for channel in range(4)]) def test_a_family_matcher_that_captures_two_bases_is_not_offered(self): @@ -716,9 +790,7 @@ def test_a_family_matcher_that_captures_two_bases_is_not_offered(self): self._join(VirtualChassis.objects.create(name="vcconv-vc2"), 5, device=standalone) self._switch_to_channelized(rule) - verdicts, _ = find_convertible_families(rule) - - self.assertEqual(verdicts, []) + self.assertEqual(find_convertible_families(rule).candidates, ()) def test_a_rule_without_a_base_is_identified_after_a_renumber(self): """Drift-immune by construction — asserted, not assumed, so the fix cannot regress it.""" @@ -728,8 +800,8 @@ def test_a_rule_without_a_base_is_identified_after_a_renumber(self): self._renumber(2) self._switch_to_channelized(rule, parent_name_template="pe-0/0/{bay_position}") - verdicts, _ = find_convertible_families(rule) + candidates = find_convertible_families(rule).candidates - self.assertEqual(len(verdicts), 1) - self.assertTrue(verdicts[0]["convertible"], verdicts[0]["reason"]) - self.assertEqual(verdicts[0]["new_names"][0], "pe-0/0/7") + self.assertEqual(len(candidates), 1) + self.assertTrue(candidates[0].convertible, candidates[0].reason) + self.assertEqual(candidates[0].new_names[0], "pe-0/0/7") diff --git a/netbox_interface_name_rules/tests/test_views.py b/netbox_interface_name_rules/tests/test_views.py index 1c69587..25d095c 100644 --- a/netbox_interface_name_rules/tests/test_views.py +++ b/netbox_interface_name_rules/tests/test_views.py @@ -206,7 +206,7 @@ def test_a_conversion_scan_failure_keeps_the_apply_preview(self): self.assertEqual(response.status_code, 200) self.assertEqual(len(response.context["preview"]), 1) # apply preview survives - self.assertEqual(response.context["conversions"], []) + self.assertEqual(response.context["conversions"], ()) msgs = [str(m) for m in get_messages(response.wsgi_request)] self.assertTrue(any("conversion" in m.lower() for m in msgs), msgs) diff --git a/netbox_interface_name_rules/views.py b/netbox_interface_name_rules/views.py index 226ec01..5919d16 100644 --- a/netbox_interface_name_rules/views.py +++ b/netbox_interface_name_rules/views.py @@ -301,7 +301,7 @@ def _evaluate_template_preview(self, cd): Each row carries the role the DB preview uses — ``parent``, ``channel`` or ``interface`` — so a channelized rule shows the parent it creates alongside the channels under it. """ - from .engine import evaluate_name_template + from .naming import evaluate_name_template name_template = cd["name_template"] channel_count = cd.get("channel_count") or 0 @@ -432,6 +432,7 @@ class RuleApplyDetailView(generic.ObjectView): def get(self, request, **kwargs): """Render a preview of all interfaces that would be renamed by this rule.""" from .engine import find_convertible_families, find_interfaces_for_rule, supports_channelization + from .family import ConversionPreview rule = self.get_object(**kwargs) try: @@ -447,23 +448,24 @@ def get(self, request, **kwargs): # A conversion-scan failure must not blank the unrelated apply preview above. try: # Each family scanned costs a dry-run conversion, so the scan takes the same batch cap. - conversions, conversions_have_more = find_convertible_families(rule, limit=APPLY_BATCH_LIMIT) + preview_conversions = find_convertible_families(rule, limit=APPLY_BATCH_LIMIT) except (re.error, ValueError) as exc: logger.exception("Failed to compute the conversion preview for rule %s: %s", rule, exc) messages.error(request, f"Failed to compute the conversion preview: {exc}") - conversions, conversions_have_more = [], False + preview_conversions = ConversionPreview(candidates=()) except Exception as exc: logger.exception("Unexpected error computing the conversion preview for rule %s: %s", rule, exc) messages.error(request, f"Failed to compute the conversion preview: {type(exc).__name__}") - conversions, conversions_have_more = [], False + preview_conversions = ConversionPreview(candidates=()) return render( request, self.template_name, { "rule": rule, "preview": preview, - "conversions": conversions, - "conversions_have_more": conversions_have_more, + "conversions": preview_conversions.candidates, + "conversions_available": any(candidate.convertible for candidate in preview_conversions.candidates), + "conversions_have_more": preview_conversions.has_more, "total_checked": total_checked, "batch_limit": APPLY_BATCH_LIMIT, "has_more": len(preview) >= APPLY_BATCH_LIMIT, @@ -507,16 +509,18 @@ def _convert(self, request, rule): f"Use Convert as Background Job to convert all of them.", ) return - conflicts = [] try: - count = convert_flat_families(rule, convert_ids, conflicts=conflicts) + outcome = convert_flat_families(rule, convert_ids) except Exception as e: logger.exception("Failed to convert families for rule %s: %s", rule, e) messages.error(request, f"Failed to convert families: {type(e).__name__}") return - messages.success(request, f"Converted {count} interface family(ies) to the channelized topology.") - if conflicts: - messages.warning(request, f"{len(conflicts)} family(ies) skipped — the plugin log names each one.") + converted = len(outcome.changed_families) + messages.success(request, f"Converted {converted} interface family(ies) to the channelized topology.") + if outcome.blocked_families: + messages.warning( + request, f"{len(outcome.blocked_families)} family(ies) skipped. The plugin log names each one." + ) def post(self, request, **kwargs): """Apply or convert (foreground batch or background job) and redirect back.""" @@ -542,15 +546,12 @@ def post(self, request, **kwargs): if not interface_ids: messages.warning(request, "No interfaces selected; nothing was applied.") else: - conflicts = [] - count = apply_rule_to_existing( - rule, limit=APPLY_BATCH_LIMIT, interface_ids=interface_ids, conflicts=conflicts - ) - messages.success(request, f"Applied rule: {count} interface(s) renamed.") - if conflicts: + outcome = apply_rule_to_existing(rule, limit=APPLY_BATCH_LIMIT, interface_ids=interface_ids) + messages.success(request, f"Applied rule: {outcome.changed_count} interface(s) renamed.") + if outcome.skipped_members: messages.warning( request, - f"{len(conflicts)} interface(s) skipped — the plugin log names each one.", + f"{len(outcome.skipped_members)} interface(s) skipped. The plugin log names each one.", ) except Exception as e: logger.exception("Failed to apply rule %s: %s", rule, e) diff --git a/performance/README.md b/performance/README.md new file mode 100644 index 0000000..598610a --- /dev/null +++ b/performance/README.md @@ -0,0 +1,113 @@ +# Automatic naming performance evidence + +The performance runner records the current implementation before an interface-family refactor. It +uses real NetBox models, signals, committed callbacks, and PostgreSQL. It is not part of default test +discovery or CI. + +Start in this plugin checkout, and use a fixed NetBox `feature` source checkout that supports +channelized interfaces. Use the same NetBox revision, PostgreSQL version, planner settings, sample +counts, and hardware for the after-run. NetBox `main` does not contain the channelization feature +while its development takes place on `feature`. + +Point NetBox's task queue at an isolated Redis database with no live RQ workers. NetBox chooses +between queued and inline search-cache writes based on worker availability. Mixing those paths +changes the SQL profile between identical saves. The runner checks this condition before it starts. + +The command below records a measured run with the retained configuration of 15 timing samples and +3 warmups per scenario. + +```bash +export PYTHONPATH="$PWD/.devcontainer/config" +export TEST_DB_NAME="inr_performance_74" +export INTERFACE_FAMILY_PERFORMANCE_SOURCE_REVISION="$(git rev-parse HEAD)" +export NETBOX_PERFORMANCE_SOURCE_REVISION="$(git -C /path/to/netbox rev-parse HEAD)" +export INTERFACE_FAMILY_PERFORMANCE_OUTPUT="$PWD/performance/baselines/existing-feature.json" +export INTERFACE_FAMILY_PERFORMANCE_SAMPLES="15" +export INTERFACE_FAMILY_PERFORMANCE_WARMUPS="3" +export INTERFACE_FAMILY_PERFORMANCE_KIND="existing_implementation" + +cd /path/to/netbox/netbox +python manage.py test \ + netbox_interface_name_rules.tests.signal_performance \ + --settings=isolated_test_settings \ + --verbosity=2 \ + --noinput +``` + +The temporary JSON artifact contains normalized statements, PostgreSQL plans, rows and loops, buffer +and WAL work, fixture sizes, planner settings, and raw wall and process CPU samples. Do not retain it +in Git. Retain only its generated Markdown summary in `performance/baselines/`. PostgreSQL node +timing stays disabled. The runner observes the statements issued by the operation and does not replay +mutating SQL. + +The complete model-save scenarios include NetBox model creation, plugin signal scheduling, and the +committed callback. Direct-callback scenarios isolate the deferred callback for diagnosis. Compare +the complete model-save scenarios first when deciding whether the refactor changed production-path +performance. + +## Comparing two runs + +`performance/compare.py BEFORE.json AFTER.json OUT.md` writes a readable comparison of database +work and machine time, and breaks down by table any scenario whose statement count rose. + +Set `INTERFACE_FAMILY_PERFORMANCE_KIND` to name what a run measured, such as `family_package` for +an after run. It labels the artifact and titles its summary. + +## Result of the interface-family comparison + +`comparisons/family-package-vs-existing.md` compares the pre-refactor baseline with the family +package. Both sides were measured with the same runner, against the same NetBox revision, on the +same host, and each recorded count reproduced twice before it was written. + +The refactor moves the committed callback's database work as follows: + +| Scenario | Before | After | Change | +| --- | ---: | ---: | ---: | +| `no_matching_rule` | 7 | 7 | 0 | +| `plain_rename` | 36 | 38 | +2 | +| `structural_creation` | 113 | 113 | 0 | +| `existing_family` | 254 | 168 | -86 | +| `reconciliation` | 255 | 257 | +2 | +| `vc.reapply_1` | 37 | 36 | -1 | +| `vc.reapply_8` | 254 | 239 | -15 | + +One scenario improves substantially: naming a module into a family that already exists costs 86 +fewer statements, a third of the callback's work. Virtual-chassis reapplication is slightly cheaper. +Two scenarios cost two statements more. + +Count the statements, but read the work. The direct-callback database and process-CPU changes are: + +| Scenario | SQL calls | Planner cost | Shared hits | CPU median | +| --- | ---: | ---: | ---: | ---: | +| `plain_rename` | +5.6% | +3.1% | -46.0% | -9.1% | +| `structural_creation` | 0.0% | -23.3% | -29.6% | -3.1% | +| `existing_family` | -33.9% | -59.9% | -41.2% | -34.2% | +| `reconciliation` | +0.8% | +2.4% | +35.8% | +3.1% | +| `vc.reapply_1` | -2.7% | -6.6% | +13.5% | +6.7% | +| `vc.reapply_8` | -5.9% | -18.6% | -7.2% | -4.0% | + +No shared-buffer reads were observed in any direct-callback scenario. + +The net change of two statements has four parts: `SAVEPOINT` (+1), `RELEASE` (+1), +`dcim_interface` (+1) and `dcim_moduletype` (-1), which sum to +2. The savepoint pair comes from the +transaction the executor opens per family. On `dcim_interface`, two new reads replace one: a read of +the module's interfaces ordered by interface ID, and the `... WHERE id IN (...) FOR UPDATE` read +that locks and revalidates the planned rows. The read they replace joined `dcim_device` and ordered +by a collated name, which is why the two together cost less planner work than the one. The module +type is now read once instead of twice. The plugin buys stale-plan revalidation and per-family +rollback for two round trips and 108 bytes of WAL on a rename. + +That revalidation is the part to keep. This callback runs after commit, so another actor, including +another plugin, can rename an interface between the moment a family is planned and the moment it is +written. The locked re-read is what detects that. + +Read SQL deltas at the `direct_callback` layer. Every scenario's `complete_model_save` SQL delta +equals its direct-callback delta, which says the changed statement count belongs to this plugin. +`no_matching_rule` is the control: the plugin returns before doing anything and issues the same 7 +statements either way. + +Both runs recorded 15 machine-time samples after 3 warmups on the same hardware. Process CPU +supports the large existing-family improvement: its median falls 34.2%. The smaller changes are +diagnostic, not pass/fail limits. Host load was not comparable: the before run moved from 25.02 to +9.59, and the after run moved from 9.24 to 16.28. Treat wall time, especially p95, as contextual +evidence and use the deterministic SQL profile as the verdict. diff --git a/performance/__init__.py b/performance/__init__.py new file mode 100644 index 0000000..ca23833 --- /dev/null +++ b/performance/__init__.py @@ -0,0 +1,3 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Performance evidence tools.""" diff --git a/performance/artifact.py b/performance/artifact.py new file mode 100644 index 0000000..c7bfba0 --- /dev/null +++ b/performance/artifact.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Define and validate the performance artifact boundary.""" + +from collections.abc import Mapping +from math import isfinite +from typing import Any + +SCHEMA_VERSION = 1 +DATABASE_METRICS = ( + ("statement_calls", "SQL calls"), + ("planner_total_cost", "Planner cost"), + ("shared_hit_blocks", "Shared hits"), + ("shared_read_blocks", "Shared reads"), + ("wal_bytes", "WAL bytes"), +) + + +def _invalid(source: str, path: str, expectation: str) -> ValueError: + """Return a consistent artifact validation error.""" + return ValueError(f"{source}: {path} {expectation}") + + +def _mapping(value: Any, source: str, path: str) -> Mapping[str, Any]: + """Require one mapping value.""" + if not isinstance(value, Mapping): + raise _invalid(source, path, "must be an object") + return value + + +def _list(value: Any, source: str, path: str) -> list[Any]: + """Require one list value.""" + if not isinstance(value, list): + raise _invalid(source, path, "must be a list") + return value + + +def _required(mapping: Mapping[str, Any], key: str, source: str, path: str) -> Any: + """Return a required field from one mapping.""" + if key not in mapping: + raise _invalid(source, f"{path}.{key}", "is required") + return mapping[key] + + +def _string(value: Any, source: str, path: str) -> str: + """Require one non-empty string value.""" + if not isinstance(value, str) or not value: + raise _invalid(source, path, "must be a non-empty string") + return value + + +def _finite_number(value: Any, source: str, path: str) -> int | float: + """Require one non-negative finite JSON number value.""" + if isinstance(value, bool) or not isinstance(value, int | float) or not isfinite(value) or value < 0: + raise _invalid(source, path, "must be a non-negative finite number") + return value + + +def _non_negative_integer(value: Any, source: str, path: str) -> int: + """Require one non-negative integer value.""" + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise _invalid(source, path, "must be a non-negative integer") + return value + + +def _validate_host_load(environment: Mapping[str, Any], source: str) -> None: + """Validate optional host-load evidence.""" + host_load = environment.get("host_load") + if host_load is None: + return + host_load = _mapping(host_load, source, "environment.host_load") + for phase in ("started", "finished"): + load = _mapping( + _required(host_load, phase, source, "environment.host_load"), + source, + f"environment.host_load.{phase}", + ) + _finite_number( + _required(load, "one_minute", source, f"environment.host_load.{phase}"), + source, + f"environment.host_load.{phase}.one_minute", + ) + + +def _validate_environment(artifact: Mapping[str, Any], source: str) -> None: + """Validate the environment fields read by comparisons.""" + environment = _mapping(_required(artifact, "environment", source, "artifact"), source, "environment") + for field in ( + "plugin_revision", + "netbox_revision", + "netbox_version", + "cpu_model", + "operating_system_release", + ): + _string(_required(environment, field, source, "environment"), source, f"environment.{field}") + postgresql_path = "environment.postgresql" + postgresql = _mapping(_required(environment, "postgresql", source, "environment"), source, postgresql_path) + _string( + _required(postgresql, "server_version", source, postgresql_path), + source, + f"{postgresql_path}.server_version", + ) + _mapping( + _required(postgresql, "planner_settings", source, postgresql_path), + source, + f"{postgresql_path}.planner_settings", + ) + _validate_host_load(environment, source) + + +def _validate_machine_time(value: Any, source: str, path: str) -> None: + """Validate optional wall-clock and process-time evidence.""" + if value is None: + return + machine_time = _mapping(value, source, path) + wall = _mapping(_required(machine_time, "wall", source, path), source, f"{path}.wall") + process_cpu = _mapping( + _required(machine_time, "process_cpu", source, path), + source, + f"{path}.process_cpu", + ) + for field in ("median_ms", "p95_ms"): + _finite_number(_required(wall, field, source, f"{path}.wall"), source, f"{path}.wall.{field}") + _finite_number( + _required(process_cpu, "median_ms", source, f"{path}.process_cpu"), + source, + f"{path}.process_cpu.median_ms", + ) + + +def _validate_scenarios(artifact: Mapping[str, Any], source: str) -> None: + """Validate scenario identities and the database evidence used by comparisons.""" + scenarios = _list(_required(artifact, "scenarios", source, "artifact"), source, "scenarios") + names: set[str] = set() + for index, value in enumerate(scenarios): + path = f"scenarios[{index}]" + scenario = _mapping(value, source, path) + name = _string(_required(scenario, "name", source, path), source, f"{path}.name") + if name in names: + raise _invalid(source, f"{path}.name", f"duplicates {name!r}") + names.add(name) + _string(_required(scenario, "layer", source, path), source, f"{path}.layer") + + database = _mapping(_required(scenario, "database", source, path), source, f"{path}.database") + totals = _mapping( + _required(database, "totals", source, f"{path}.database"), + source, + f"{path}.database.totals", + ) + for field, _label in DATABASE_METRICS: + if field == "statement_calls": + _non_negative_integer( + _required(totals, field, source, f"{path}.database.totals"), + source, + f"{path}.database.totals.{field}", + ) + elif field in totals: + _finite_number(totals[field], source, f"{path}.database.totals.{field}") + statements = _list( + _required(database, "statements", source, f"{path}.database"), + source, + f"{path}.database.statements", + ) + for statement_index, statement_value in enumerate(statements): + statement_path = f"{path}.database.statements[{statement_index}]" + statement = _mapping(statement_value, source, statement_path) + _string( + _required(statement, "normalized_sql", source, statement_path), + source, + f"{statement_path}.normalized_sql", + ) + _non_negative_integer( + _required(statement, "calls", source, statement_path), + source, + f"{statement_path}.calls", + ) + + _validate_machine_time(_required(scenario, "machine_time", source, path), source, f"{path}.machine_time") + + +def validate_artifact(value: Any, source: str = "performance artifact") -> None: + """Reject a performance artifact that this code cannot interpret.""" + artifact = _mapping(value, source, "artifact") + version = _required(artifact, "schema_version", source, "artifact") + if version != SCHEMA_VERSION: + raise _invalid(source, "artifact.schema_version", f"must be {SCHEMA_VERSION}, got {version!r}") + _string(_required(artifact, "baseline_kind", source, "artifact"), source, "artifact.baseline_kind") + _string(_required(artifact, "generated_at", source, "artifact"), source, "artifact.generated_at") + + configuration = _mapping( + _required(artifact, "configuration", source, "artifact"), + source, + "configuration", + ) + for field in ("samples", "warmups"): + _non_negative_integer( + _required(configuration, field, source, "configuration"), + source, + f"configuration.{field}", + ) + + _validate_environment(artifact, source) + _validate_scenarios(artifact, source) diff --git a/performance/baselines/existing-feature.md b/performance/baselines/existing-feature.md new file mode 100644 index 0000000..c846541 --- /dev/null +++ b/performance/baselines/existing-feature.md @@ -0,0 +1,33 @@ +# Automatic naming performance: existing implementation + +This file is generated by `netbox_interface_name_rules.tests.signal_performance`. +Machine-time values are evidence for a same-hardware before/after comparison. They are not CI limits. + +This run measured machine time. Statement counts are reproducible and carry the verdict. + +Host load averaged 25.02 over the minute before the run and 9.59 over the minute it ended. Machine time is evidence only when both runs were taken under a comparable load. + +| Scenario | Layer | SQL calls | Planner cost | Shared hits | Wall median (ms) | Wall p95 (ms) | CPU median (ms) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `module.complete_model_save.no_matching_rule` | complete_model_save | 77 | 1268.440 | 435 | 205.417 | 275.813 | 134.672 | +| `module.direct_callback.no_matching_rule` | direct_callback | 7 | 13.440 | 13 | 22.119 | 36.044 | 15.709 | +| `module.complete_model_save.plain_rename` | complete_model_save | 106 | 1620.800 | 438 | 266.112 | 356.490 | 170.104 | +| `module.direct_callback.plain_rename` | direct_callback | 36 | 325.070 | 215 | 95.032 | 111.400 | 63.090 | +| `module.complete_model_save.structural_creation` | complete_model_save | 183 | 2552.290 | 814 | 430.604 | 554.378 | 270.455 | +| `module.direct_callback.structural_creation` | direct_callback | 113 | 1446.050 | 706 | 235.461 | 262.178 | 164.085 | +| `module.complete_model_save.existing_family` | complete_model_save | 436 | 6291.550 | 1987 | 828.146 | 1008.849 | 593.035 | +| `module.direct_callback.existing_family` | direct_callback | 254 | 3500.530 | 1146 | 493.059 | 541.989 | 351.462 | +| `module.complete_model_save.reconciliation` | complete_model_save | 437 | 6098.280 | 2156 | 806.415 | 887.028 | 582.722 | +| `module.direct_callback.reconciliation` | direct_callback | 255 | 3200.990 | 944 | 471.063 | 511.525 | 330.303 | +| `vc.complete_model_save.reapply_1` | complete_model_save | 62 | 607.180 | 326 | 125.342 | 143.462 | 88.441 | +| `vc.direct_callback.reapply_1` | direct_callback | 37 | 339.110 | 155 | 85.037 | 91.612 | 59.026 | +| `vc.complete_model_save.reapply_8` | complete_model_save | 279 | 2654.980 | 1061 | 572.069 | 629.913 | 391.765 | +| `vc.direct_callback.reapply_8` | direct_callback | 254 | 2575.540 | 1033 | 518.475 | 553.082 | 359.993 | + +Plugin revision: `ccadd0bb5b2852ff81f2747769d61746254173e1` + +NetBox revision: `ab9bd6f5b69a4d359fcb20dcbf6280af39fc77d6` + +PostgreSQL: `18.3 (Debian 18.3-1.pgdg13+1)` + +Samples per scenario: `15` diff --git a/performance/baselines/family-package.md b/performance/baselines/family-package.md new file mode 100644 index 0000000..3d1f0f3 --- /dev/null +++ b/performance/baselines/family-package.md @@ -0,0 +1,33 @@ +# Automatic naming performance: family package + +This file is generated by `netbox_interface_name_rules.tests.signal_performance`. +Machine-time values are evidence for a same-hardware before/after comparison. They are not CI limits. + +This run measured machine time. Statement counts are reproducible and carry the verdict. + +Host load averaged 9.24 over the minute before the run and 16.28 over the minute it ended. Machine time is evidence only when both runs were taken under a comparable load. + +| Scenario | Layer | SQL calls | Planner cost | Shared hits | Wall median (ms) | Wall p95 (ms) | CPU median (ms) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| `module.complete_model_save.no_matching_rule` | complete_model_save | 77 | 1363.830 | 227 | 177.015 | 195.255 | 127.728 | +| `module.direct_callback.no_matching_rule` | direct_callback | 7 | 8.440 | 8 | 18.897 | 21.972 | 14.825 | +| `module.complete_model_save.plain_rename` | complete_model_save | 108 | 1698.830 | 342 | 229.476 | 259.763 | 163.679 | +| `module.direct_callback.plain_rename` | direct_callback | 38 | 335.020 | 116 | 77.968 | 86.425 | 57.358 | +| `module.complete_model_save.structural_creation` | complete_model_save | 183 | 2775.510 | 636 | 396.451 | 406.980 | 278.442 | +| `module.direct_callback.structural_creation` | direct_callback | 113 | 1109.230 | 497 | 220.266 | 247.762 | 158.976 | +| `module.complete_model_save.existing_family` | complete_model_save | 350 | 3513.550 | 1483 | 694.012 | 738.334 | 499.153 | +| `module.direct_callback.existing_family` | direct_callback | 168 | 1403.760 | 674 | 314.685 | 352.797 | 231.229 | +| `module.complete_model_save.reconciliation` | complete_model_save | 439 | 5680.630 | 2420 | 839.687 | 904.039 | 603.197 | +| `module.direct_callback.reconciliation` | direct_callback | 257 | 3279.230 | 1282 | 470.856 | 510.018 | 340.615 | +| `vc.complete_model_save.reapply_1` | complete_model_save | 61 | 527.750 | 299 | 121.766 | 138.959 | 86.539 | +| `vc.direct_callback.reapply_1` | direct_callback | 36 | 316.570 | 176 | 89.928 | 108.948 | 62.991 | +| `vc.complete_model_save.reapply_8` | complete_model_save | 264 | 2494.160 | 1276 | 506.132 | 536.683 | 362.128 | +| `vc.direct_callback.reapply_8` | direct_callback | 239 | 2097.740 | 959 | 464.611 | 520.050 | 345.583 | + +Plugin revision: `636bebc5f823146b818c86bae1905eceb813dd6e` + +NetBox revision: `ab9bd6f5b69a4d359fcb20dcbf6280af39fc77d6` + +PostgreSQL: `18.3 (Debian 18.3-1.pgdg13+1)` + +Samples per scenario: `15` diff --git a/performance/compare.py b/performance/compare.py new file mode 100644 index 0000000..04a6b19 --- /dev/null +++ b/performance/compare.py @@ -0,0 +1,250 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (C) 2025 Marcin Zieba +"""Compare two automatic naming performance artifacts and write a readable summary. + +Usage: python performance/compare.py BEFORE.json AFTER.json OUT.md + +Database work is deterministic, so its deltas are evidence on any machine. Machine time is only +evidence when both runs were taken on the same hardware under a comparable load; the summary says +which numbers it is reporting and leaves the judgement to the reader. +""" + +import json +import re +import sys +from collections import Counter +from pathlib import Path + +_ROOT = Path(__file__).resolve().parent.parent +if str(_ROOT) not in sys.path: + sys.path.insert(0, str(_ROOT)) + +from performance.artifact import DATABASE_METRICS, validate_artifact # noqa: E402 + +_TIME_METRICS = ( + ("wall", "median_ms", "Wall median (ms)"), + ("wall", "p95_ms", "Wall p95 (ms)"), + ("process_cpu", "median_ms", "CPU median (ms)"), +) + + +def _artifact_path(raw, must_exist): + """Resolve *raw* inside this repository, refusing a path that points outside it.""" + candidate = Path(raw) + path = (candidate if candidate.is_absolute() else _ROOT / candidate).resolve() + if not path.is_relative_to(_ROOT): + raise SystemExit(f"{raw} is outside the repository") + if must_exist and not path.is_file(): + raise SystemExit(f"{raw} is not a readable artifact") + return path + + +def _statement_table(sql): + """Return the table a normalized statement reads or writes, for attribution.""" + match = re.search(r'(?:FROM|INTO|UPDATE)\s+"([a-z_]+)"', sql) + return match.group(1) if match else sql.split()[0][:24] + + +def _calls_by_table(scenario): + """Return how many statements the scenario issued against each table.""" + counts: Counter = Counter() + for entry in scenario["database"]["statements"]: + counts[_statement_table(entry["normalized_sql"])] += entry["calls"] + return counts + + +def _attribution(name, before_scenario, after_scenario): + """Return the per-table statement deltas behind one scenario's change.""" + before_calls, after_calls = _calls_by_table(before_scenario), _calls_by_table(after_scenario) + rows = [] + for table in sorted(set(before_calls) | set(after_calls)): + change = after_calls[table] - before_calls[table] + if change: + rows.append(f"| `{name}` | `{table}` | {before_calls[table]} | {after_calls[table]} | {change:+d} |") + return rows + + +def _scenarios(artifact): + """Return the artifact's scenarios keyed by name.""" + return {scenario["name"]: scenario for scenario in artifact["scenarios"]} + + +def _scenario_names(before, after): + """Return the scenario-name union in stable artifact order.""" + return (*before, *(name for name in after if name not in before)) + + +def _delta(before, after): + """Return the absolute and percentage change, or None when there is no baseline to divide by.""" + change = after - before + if not before: + return change, None + return change, change / before * 100 + + +def _format(value): + """Return a compact fixed-point rendering of one metric value.""" + if isinstance(value, float) and not value.is_integer(): + return f"{value:.2f}" + return f"{int(value)}" + + +def _comparison_cells(before, after, before_unavailable, after_unavailable): + """Return rendered comparison cells for one optional metric.""" + if before is None or after is None: + before_value = before_unavailable if before is None else _format(before) + after_value = after_unavailable if after is None else _format(after) + return before_value, after_value, "n/a", "n/a" + change, percent = _delta(before, after) + share = "n/a" if percent is None else f"{percent:+.1f}%" + return _format(before), _format(after), _format(change), share + + +def _metric_row(name, label, before, after, before_unavailable, after_unavailable): + """Render one database-work or machine-time comparison row.""" + before_value, after_value, change, share = _comparison_cells( + before, + after, + before_unavailable, + after_unavailable, + ) + return f"| `{name}` | {label} | {before_value} | {after_value} | {change} | {share} |" + + +def _database_scenario(name, before_scenario, after_scenario): + """Return metric rows and an optional statement regression for one scenario.""" + before_unavailable = "missing baseline" if before_scenario is None else "n/a" + after_unavailable = "missing" if after_scenario is None else "n/a" + rows = [] + regression = None + for key, label in DATABASE_METRICS: + old = None if before_scenario is None else before_scenario["database"]["totals"].get(key) + new = None if after_scenario is None else after_scenario["database"]["totals"].get(key) + rows.append(_metric_row(name, label, old, new, before_unavailable, after_unavailable)) + if key == "statement_calls" and old is not None and new is not None and new > old: + regression = (name, label, old, new) + return rows, regression + + +def _database_table(before, after): + """Return the per-scenario database-work comparison.""" + lines = ["| Scenario | Metric | Before | After | Change | Share |", "| --- | --- | ---: | ---: | ---: | ---: |"] + regressions = [] + for name in _scenario_names(before, after): + scenario_lines, regression = _database_scenario(name, before.get(name), after.get(name)) + lines.extend(scenario_lines) + if regression is not None: + regressions.append(regression) + return lines, regressions + + +def _time_value(scenario, section, key): + """Return one timing value, or None when the scenario was not measured.""" + if scenario is None: + return None + machine_time = scenario["machine_time"] + return None if machine_time is None else machine_time[section][key] + + +def _time_table(before, after): + """Return the per-scenario machine-time comparison.""" + lines = ["| Scenario | Metric | Before | After | Change | Share |", "| --- | --- | ---: | ---: | ---: | ---: |"] + for name in _scenario_names(before, after): + before_scenario = before.get(name) + after_scenario = after.get(name) + before_unavailable = "missing baseline" if before_scenario is None else "not measured" + after_unavailable = "missing" if after_scenario is None else "not measured" + for section, key, label in _TIME_METRICS: + old = _time_value(before_scenario, section, key) + new = _time_value(after_scenario, section, key) + lines.append(_metric_row(name, label, old, new, before_unavailable, after_unavailable)) + return lines + + +def _load_span(artifact): + """Return the run-queue length a run started and finished under.""" + load = artifact["environment"].get("host_load") + if not load: + return "not recorded" + return f"{load['started']['one_minute']:.2f} to {load['finished']['one_minute']:.2f}" + + +def _environment_table(before, after): + """Return the revisions and settings each run was taken under.""" + rows = ["| Field | Before | After |", "| --- | --- | --- |"] + for key in ("plugin_revision", "netbox_revision", "netbox_version", "cpu_model", "operating_system_release"): + rows.append(f"| {key} | `{before['environment'].get(key)}` | `{after['environment'].get(key)}` |") + for key in ("samples", "warmups"): + rows.append(f"| {key} | `{before['configuration'].get(key)}` | `{after['configuration'].get(key)}` |") + postgres_before = before["environment"].get("postgresql", {}).get("server_version") + postgres_after = after["environment"].get("postgresql", {}).get("server_version") + rows.append(f"| postgresql | `{postgres_before}` | `{postgres_after}` |") + settings_before = before["environment"].get("postgresql", {}).get("planner_settings") + settings_after = after["environment"].get("postgresql", {}).get("planner_settings") + planner_status = "identical" if settings_before == settings_after else "CHANGED" + rows.append(f"| planner settings | `{planner_status}` | `{planner_status}` |") + load_before = _load_span(before) + load_after = _load_span(after) + rows.append(f"| host load (1 min, start to end) | `{load_before}` | `{load_after}` |") + return rows + + +def main(argv): + """Write the comparison of two artifacts to a Markdown file.""" + if len(argv) != 4: + raise SystemExit(__doc__) + before = json.loads(_artifact_path(argv[1], must_exist=True).read_text()) + after = json.loads(_artifact_path(argv[2], must_exist=True).read_text()) + validate_artifact(before, argv[1]) + validate_artifact(after, argv[2]) + destination = _artifact_path(argv[3], must_exist=False) + before_scenarios, after_scenarios = _scenarios(before), _scenarios(after) + + database_lines, regressions = _database_table(before_scenarios, after_scenarios) + report = [ + "# Automatic naming performance comparison", + "", + "Generated by `performance/compare.py` from the before and after artifacts of the " + "interface-family refactor. Database work is deterministic and is the evidence this " + "comparison rests on. Machine time is reported for completeness and is only evidence when " + "both runs were taken on the same otherwise-idle hardware.", + "", + "## Environment", + "", + *_environment_table(before, after), + "", + "## Database work", + "", + *database_lines, + "", + "## Machine time", + "", + *_time_table(before_scenarios, after_scenarios), + "", + "## Statement-count regressions", + "", + ] + if regressions: + report.extend(f"- `{name}` {label}: {_format(old)} to {_format(new)}" for name, label, old, new in regressions) + report += [ + "", + "### Where those statements come from", + "", + "Each raised scenario is broken down by the table its statements touch, so the work this " + "plugin drives can be told apart from the per-save bookkeeping NetBox does for the object " + "types and custom fields the database happens to hold.", + "", + "| Scenario | Table | Before | After | Change |", + "| --- | --- | ---: | ---: | ---: |", + ] + for name, _label, _old, _new in regressions: + report.extend(_attribution(name, before_scenarios[name], after_scenarios[name])) + else: + report.append("None. No scenario issues more statements than the baseline.") + report.append("") + destination.write_text("\n".join(report)) + print(f"wrote {destination} ({len(regressions)} statement-count regressions)") + + +if __name__ == "__main__": + main(sys.argv) diff --git a/performance/comparisons/family-package-vs-existing.md b/performance/comparisons/family-package-vs-existing.md new file mode 100644 index 0000000..1c5fa0e --- /dev/null +++ b/performance/comparisons/family-package-vs-existing.md @@ -0,0 +1,170 @@ +# Automatic naming performance comparison + +Generated by `performance/compare.py` from the before and after artifacts of the interface-family refactor. Database work is deterministic and is the evidence this comparison rests on. Machine time is reported for completeness and is only evidence when both runs were taken on the same otherwise-idle hardware. + +## Environment + +| Field | Before | After | +| --- | --- | --- | +| plugin_revision | `ccadd0bb5b2852ff81f2747769d61746254173e1` | `636bebc5f823146b818c86bae1905eceb813dd6e` | +| netbox_revision | `ab9bd6f5b69a4d359fcb20dcbf6280af39fc77d6` | `ab9bd6f5b69a4d359fcb20dcbf6280af39fc77d6` | +| netbox_version | `4.7.0-beta2` | `4.7.0-beta2` | +| cpu_model | `Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz` | `Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz` | +| operating_system_release | `6.12.96+deb13-amd64` | `6.12.96+deb13-amd64` | +| samples | `15` | `15` | +| warmups | `3` | `3` | +| postgresql | `18.3 (Debian 18.3-1.pgdg13+1)` | `18.3 (Debian 18.3-1.pgdg13+1)` | +| planner settings | `identical` | `identical` | +| host load (1 min, start to end) | `25.02 to 9.59` | `9.24 to 16.28` | + +## Database work + +| Scenario | Metric | Before | After | Change | Share | +| --- | --- | ---: | ---: | ---: | ---: | +| `module.complete_model_save.no_matching_rule` | SQL calls | 77 | 77 | 0 | +0.0% | +| `module.complete_model_save.no_matching_rule` | Planner cost | 1268.44 | 1363.83 | 95.39 | +7.5% | +| `module.complete_model_save.no_matching_rule` | Shared hits | 435 | 227 | -208 | -47.8% | +| `module.complete_model_save.no_matching_rule` | Shared reads | 6 | 0 | -6 | -100.0% | +| `module.complete_model_save.no_matching_rule` | WAL bytes | 2843 | 2843 | 0 | +0.0% | +| `module.direct_callback.no_matching_rule` | SQL calls | 7 | 7 | 0 | +0.0% | +| `module.direct_callback.no_matching_rule` | Planner cost | 13.44 | 8.44 | -5 | -37.2% | +| `module.direct_callback.no_matching_rule` | Shared hits | 13 | 8 | -5 | -38.5% | +| `module.direct_callback.no_matching_rule` | Shared reads | 0 | 0 | 0 | n/a | +| `module.direct_callback.no_matching_rule` | WAL bytes | 0 | 0 | 0 | n/a | +| `module.complete_model_save.plain_rename` | SQL calls | 106 | 108 | 2 | +1.9% | +| `module.complete_model_save.plain_rename` | Planner cost | 1620.80 | 1698.83 | 78.03 | +4.8% | +| `module.complete_model_save.plain_rename` | Shared hits | 438 | 342 | -96 | -21.9% | +| `module.complete_model_save.plain_rename` | Shared reads | 31 | 0 | -31 | -100.0% | +| `module.complete_model_save.plain_rename` | WAL bytes | 5106 | 4988 | -118 | -2.3% | +| `module.direct_callback.plain_rename` | SQL calls | 36 | 38 | 2 | +5.6% | +| `module.direct_callback.plain_rename` | Planner cost | 325.07 | 335.02 | 9.95 | +3.1% | +| `module.direct_callback.plain_rename` | Shared hits | 215 | 116 | -99 | -46.0% | +| `module.direct_callback.plain_rename` | Shared reads | 0 | 0 | 0 | n/a | +| `module.direct_callback.plain_rename` | WAL bytes | 1891 | 1999 | 108 | +5.7% | +| `module.complete_model_save.structural_creation` | SQL calls | 183 | 183 | 0 | +0.0% | +| `module.complete_model_save.structural_creation` | Planner cost | 2552.29 | 2775.51 | 223.22 | +8.7% | +| `module.complete_model_save.structural_creation` | Shared hits | 814 | 636 | -178 | -21.9% | +| `module.complete_model_save.structural_creation` | Shared reads | 10 | 1 | -9 | -90.0% | +| `module.complete_model_save.structural_creation` | WAL bytes | 21024 | 12897 | -8127 | -38.7% | +| `module.direct_callback.structural_creation` | SQL calls | 113 | 113 | 0 | +0.0% | +| `module.direct_callback.structural_creation` | Planner cost | 1446.05 | 1109.23 | -336.82 | -23.3% | +| `module.direct_callback.structural_creation` | Shared hits | 706 | 497 | -209 | -29.6% | +| `module.direct_callback.structural_creation` | Shared reads | 1 | 0 | -1 | -100.0% | +| `module.direct_callback.structural_creation` | WAL bytes | 9841 | 9841 | 0 | +0.0% | +| `module.complete_model_save.existing_family` | SQL calls | 436 | 350 | -86 | -19.7% | +| `module.complete_model_save.existing_family` | Planner cost | 6291.55 | 3513.55 | -2778.00 | -44.2% | +| `module.complete_model_save.existing_family` | Shared hits | 1987 | 1483 | -504 | -25.4% | +| `module.complete_model_save.existing_family` | Shared reads | 0 | 0 | 0 | n/a | +| `module.complete_model_save.existing_family` | WAL bytes | 27863 | 26507 | -1356 | -4.9% | +| `module.direct_callback.existing_family` | SQL calls | 254 | 168 | -86 | -33.9% | +| `module.direct_callback.existing_family` | Planner cost | 3500.53 | 1403.76 | -2096.77 | -59.9% | +| `module.direct_callback.existing_family` | Shared hits | 1146 | 674 | -472 | -41.2% | +| `module.direct_callback.existing_family` | Shared reads | 0 | 0 | 0 | n/a | +| `module.direct_callback.existing_family` | WAL bytes | 11531 | 10024 | -1507 | -13.1% | +| `module.complete_model_save.reconciliation` | SQL calls | 437 | 439 | 2 | +0.5% | +| `module.complete_model_save.reconciliation` | Planner cost | 6098.28 | 5680.63 | -417.65 | -6.8% | +| `module.complete_model_save.reconciliation` | Shared hits | 2156 | 2420 | 264 | +12.2% | +| `module.complete_model_save.reconciliation` | Shared reads | 0 | 0 | 0 | n/a | +| `module.complete_model_save.reconciliation` | WAL bytes | 33945 | 34393 | 448 | +1.3% | +| `module.direct_callback.reconciliation` | SQL calls | 255 | 257 | 2 | +0.8% | +| `module.direct_callback.reconciliation` | Planner cost | 3200.99 | 3279.23 | 78.24 | +2.4% | +| `module.direct_callback.reconciliation` | Shared hits | 944 | 1282 | 338 | +35.8% | +| `module.direct_callback.reconciliation` | Shared reads | 0 | 0 | 0 | n/a | +| `module.direct_callback.reconciliation` | WAL bytes | 17466 | 18160 | 694 | +4.0% | +| `vc.complete_model_save.reapply_1` | SQL calls | 62 | 61 | -1 | -1.6% | +| `vc.complete_model_save.reapply_1` | Planner cost | 607.18 | 527.75 | -79.43 | -13.1% | +| `vc.complete_model_save.reapply_1` | Shared hits | 326 | 299 | -27 | -8.3% | +| `vc.complete_model_save.reapply_1` | Shared reads | 1 | 0 | -1 | -100.0% | +| `vc.complete_model_save.reapply_1` | WAL bytes | 4808 | 4829 | 21 | +0.4% | +| `vc.direct_callback.reapply_1` | SQL calls | 37 | 36 | -1 | -2.7% | +| `vc.direct_callback.reapply_1` | Planner cost | 339.11 | 316.57 | -22.54 | -6.6% | +| `vc.direct_callback.reapply_1` | Shared hits | 155 | 176 | 21 | +13.5% | +| `vc.direct_callback.reapply_1` | Shared reads | 0 | 0 | 0 | n/a | +| `vc.direct_callback.reapply_1` | WAL bytes | 1857 | 1967 | 110 | +5.9% | +| `vc.complete_model_save.reapply_8` | SQL calls | 279 | 264 | -15 | -5.4% | +| `vc.complete_model_save.reapply_8` | Planner cost | 2654.98 | 2494.16 | -160.82 | -6.1% | +| `vc.complete_model_save.reapply_8` | Shared hits | 1061 | 1276 | 215 | +20.3% | +| `vc.complete_model_save.reapply_8` | Shared reads | 0 | 0 | 0 | n/a | +| `vc.complete_model_save.reapply_8` | WAL bytes | 17718 | 18878 | 1160 | +6.5% | +| `vc.direct_callback.reapply_8` | SQL calls | 254 | 239 | -15 | -5.9% | +| `vc.direct_callback.reapply_8` | Planner cost | 2575.54 | 2097.74 | -477.80 | -18.6% | +| `vc.direct_callback.reapply_8` | Shared hits | 1033 | 959 | -74 | -7.2% | +| `vc.direct_callback.reapply_8` | Shared reads | 0 | 0 | 0 | n/a | +| `vc.direct_callback.reapply_8` | WAL bytes | 14856 | 15719 | 863 | +5.8% | + +## Machine time + +| Scenario | Metric | Before | After | Change | Share | +| --- | --- | ---: | ---: | ---: | ---: | +| `module.complete_model_save.no_matching_rule` | Wall median (ms) | 205.42 | 177.02 | -28.40 | -13.8% | +| `module.complete_model_save.no_matching_rule` | Wall p95 (ms) | 275.81 | 195.25 | -80.56 | -29.2% | +| `module.complete_model_save.no_matching_rule` | CPU median (ms) | 134.67 | 127.73 | -6.94 | -5.2% | +| `module.direct_callback.no_matching_rule` | Wall median (ms) | 22.12 | 18.90 | -3.22 | -14.6% | +| `module.direct_callback.no_matching_rule` | Wall p95 (ms) | 36.04 | 21.97 | -14.07 | -39.0% | +| `module.direct_callback.no_matching_rule` | CPU median (ms) | 15.71 | 14.82 | -0.88 | -5.6% | +| `module.complete_model_save.plain_rename` | Wall median (ms) | 266.11 | 229.48 | -36.64 | -13.8% | +| `module.complete_model_save.plain_rename` | Wall p95 (ms) | 356.49 | 259.76 | -96.73 | -27.1% | +| `module.complete_model_save.plain_rename` | CPU median (ms) | 170.10 | 163.68 | -6.42 | -3.8% | +| `module.direct_callback.plain_rename` | Wall median (ms) | 95.03 | 77.97 | -17.06 | -18.0% | +| `module.direct_callback.plain_rename` | Wall p95 (ms) | 111.40 | 86.42 | -24.98 | -22.4% | +| `module.direct_callback.plain_rename` | CPU median (ms) | 63.09 | 57.36 | -5.73 | -9.1% | +| `module.complete_model_save.structural_creation` | Wall median (ms) | 430.60 | 396.45 | -34.15 | -7.9% | +| `module.complete_model_save.structural_creation` | Wall p95 (ms) | 554.38 | 406.98 | -147.40 | -26.6% | +| `module.complete_model_save.structural_creation` | CPU median (ms) | 270.45 | 278.44 | 7.99 | +3.0% | +| `module.direct_callback.structural_creation` | Wall median (ms) | 235.46 | 220.27 | -15.20 | -6.5% | +| `module.direct_callback.structural_creation` | Wall p95 (ms) | 262.18 | 247.76 | -14.42 | -5.5% | +| `module.direct_callback.structural_creation` | CPU median (ms) | 164.08 | 158.98 | -5.11 | -3.1% | +| `module.complete_model_save.existing_family` | Wall median (ms) | 828.15 | 694.01 | -134.13 | -16.2% | +| `module.complete_model_save.existing_family` | Wall p95 (ms) | 1008.85 | 738.33 | -270.52 | -26.8% | +| `module.complete_model_save.existing_family` | CPU median (ms) | 593.03 | 499.15 | -93.88 | -15.8% | +| `module.direct_callback.existing_family` | Wall median (ms) | 493.06 | 314.69 | -178.37 | -36.2% | +| `module.direct_callback.existing_family` | Wall p95 (ms) | 541.99 | 352.80 | -189.19 | -34.9% | +| `module.direct_callback.existing_family` | CPU median (ms) | 351.46 | 231.23 | -120.23 | -34.2% | +| `module.complete_model_save.reconciliation` | Wall median (ms) | 806.42 | 839.69 | 33.27 | +4.1% | +| `module.complete_model_save.reconciliation` | Wall p95 (ms) | 887.03 | 904.04 | 17.01 | +1.9% | +| `module.complete_model_save.reconciliation` | CPU median (ms) | 582.72 | 603.20 | 20.47 | +3.5% | +| `module.direct_callback.reconciliation` | Wall median (ms) | 471.06 | 470.86 | -0.21 | -0.0% | +| `module.direct_callback.reconciliation` | Wall p95 (ms) | 511.53 | 510.02 | -1.51 | -0.3% | +| `module.direct_callback.reconciliation` | CPU median (ms) | 330.30 | 340.62 | 10.31 | +3.1% | +| `vc.complete_model_save.reapply_1` | Wall median (ms) | 125.34 | 121.77 | -3.58 | -2.9% | +| `vc.complete_model_save.reapply_1` | Wall p95 (ms) | 143.46 | 138.96 | -4.50 | -3.1% | +| `vc.complete_model_save.reapply_1` | CPU median (ms) | 88.44 | 86.54 | -1.90 | -2.2% | +| `vc.direct_callback.reapply_1` | Wall median (ms) | 85.04 | 89.93 | 4.89 | +5.8% | +| `vc.direct_callback.reapply_1` | Wall p95 (ms) | 91.61 | 108.95 | 17.34 | +18.9% | +| `vc.direct_callback.reapply_1` | CPU median (ms) | 59.03 | 62.99 | 3.96 | +6.7% | +| `vc.complete_model_save.reapply_8` | Wall median (ms) | 572.07 | 506.13 | -65.94 | -11.5% | +| `vc.complete_model_save.reapply_8` | Wall p95 (ms) | 629.91 | 536.68 | -93.23 | -14.8% | +| `vc.complete_model_save.reapply_8` | CPU median (ms) | 391.76 | 362.13 | -29.64 | -7.6% | +| `vc.direct_callback.reapply_8` | Wall median (ms) | 518.48 | 464.61 | -53.86 | -10.4% | +| `vc.direct_callback.reapply_8` | Wall p95 (ms) | 553.08 | 520.05 | -33.03 | -6.0% | +| `vc.direct_callback.reapply_8` | CPU median (ms) | 359.99 | 345.58 | -14.41 | -4.0% | + +## Statement-count regressions + +- `module.complete_model_save.plain_rename` SQL calls: 106 to 108 +- `module.direct_callback.plain_rename` SQL calls: 36 to 38 +- `module.complete_model_save.reconciliation` SQL calls: 437 to 439 +- `module.direct_callback.reconciliation` SQL calls: 255 to 257 + +### Where those statements come from + +Each raised scenario is broken down by the table its statements touch, so the work this plugin drives can be told apart from the per-save bookkeeping NetBox does for the object types and custom fields the database happens to hold. + +| Scenario | Table | Before | After | Change | +| --- | --- | ---: | ---: | ---: | +| `module.complete_model_save.plain_rename` | `RELEASE` | 3 | 4 | +1 | +| `module.complete_model_save.plain_rename` | `SAVEPOINT` | 3 | 4 | +1 | +| `module.complete_model_save.plain_rename` | `dcim_interface` | 8 | 9 | +1 | +| `module.complete_model_save.plain_rename` | `dcim_moduletype` | 3 | 2 | -1 | +| `module.direct_callback.plain_rename` | `RELEASE` | 1 | 2 | +1 | +| `module.direct_callback.plain_rename` | `SAVEPOINT` | 1 | 2 | +1 | +| `module.direct_callback.plain_rename` | `dcim_interface` | 5 | 6 | +1 | +| `module.direct_callback.plain_rename` | `dcim_moduletype` | 2 | 1 | -1 | +| `module.complete_model_save.reconciliation` | `RELEASE` | 19 | 20 | +1 | +| `module.complete_model_save.reconciliation` | `SAVEPOINT` | 19 | 20 | +1 | +| `module.complete_model_save.reconciliation` | `dcim_interface` | 71 | 72 | +1 | +| `module.complete_model_save.reconciliation` | `dcim_moduletype` | 3 | 2 | -1 | +| `module.direct_callback.reconciliation` | `RELEASE` | 17 | 18 | +1 | +| `module.direct_callback.reconciliation` | `SAVEPOINT` | 17 | 18 | +1 | +| `module.direct_callback.reconciliation` | `dcim_interface` | 40 | 41 | +1 | +| `module.direct_callback.reconciliation` | `dcim_moduletype` | 2 | 1 | -1 | diff --git a/pyproject.toml b/pyproject.toml index 824d8f2..e08a69f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -24,7 +24,9 @@ keywords = [ "transceiver", "automation", ] -dependencies = [] +dependencies = [ + "google-re2>=1.1.20251105", +] classifiers = [ "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.12", @@ -56,6 +58,7 @@ dev = [ "pytest", "pytest-django", "pytest-cov>=6.0", + "pytest-xdist>=3.8", "python-semantic-release", "django>=5.1,<7.0", ] @@ -65,7 +68,9 @@ docs = [ ] [tool.pytest.ini_options] -pythonpath = ["/opt/netbox/netbox"] +pythonpath = ["/opt/netbox/netbox", "."] +DJANGO_SETTINGS_MODULE = "netbox.settings" +addopts = "-n auto --dist loadscope" [tool.ruff] line-length = 120