Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ node_modules/*
.envrc
dist/
.infrahub-sync-cache/
# invoke bench.run artifacts (default to repo root)
# Local benchmark artifacts
bench-results.csv
.bench-filtered-config.yml
# spec-kit transient download cache (extensions/presets)
Expand Down
1 change: 1 addition & 0 deletions .vale/styles/spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ kbps
Keycloak
LibreNMS
liveness
loopback
Loopbacks
markdownlint
max_count
Expand Down
31 changes: 19 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Built on the [`diffsync`](https://github.com/networktocode/diffsync) framework,
## Prerequisites

- A running [Infrahub](https://github.com/opsmill/infrahub) instance
- A running Sync API and worker
- Python 3.10–3.13
- Credentials and network access for the source and destination systems

Expand All @@ -51,16 +52,19 @@ infrahub-sync --help

## Example: NetBox → Infrahub

The repository includes a NetBox project at `examples/netbox_to_infrahub/`, named
`from-netbox`. It reads the changing public NetBox demo (`demo.netbox.dev`) and targets a
local Infrahub at `localhost:8000`.
The repository includes a NetBox configuration package at
`examples/netbox_to_infrahub/package.yml`. Register it once, then address the immutable
configuration version returned by the service:

```bash
# Preview what the sync would change
infrahub-sync diff --name from-netbox --directory ./examples

# Review the saved plan without contacting either system
infrahub-sync diff --name from-netbox --directory ./examples --from-plan <run-id>
export INFRAHUB_SYNC_API_URL=https://sync.example.com
export INFRAHUB_SYNC_API_TOKEN=<token>

infrahub-sync configs register examples/netbox_to_infrahub/package.yml \
--reason "register NetBox import"
infrahub-sync diff --config-id <config-id> --version <version> \
--reason "review NetBox import"
infrahub-sync runs plan <run-id> --detail
```

Before applying the plan, read the
Expand All @@ -75,7 +79,9 @@ For setup and a complete walkthrough, see the

## Day 2 Operations

**Scheduling.** Infrahub Sync runs as a CLI, so it plugs into the scheduling tooling your team already uses — cron, CI jobs, Prefect, or any workflow engine. This keeps the footprint small and lets you control sync cadence, observability, and failure handling through systems you already trust.
**Scheduling.** The CLI submits runs to the Sync API. The service owns admission, durable
run records, and worker execution, so a caller can disconnect after using `--no-wait` and
inspect the same run later.

**Observability.** Sync runs emit lifecycle and adapter logs through Python logging. Public
Python API lifecycle records include structured attributes such as the run identifier,
Expand Down Expand Up @@ -118,16 +124,17 @@ may change.

- **Sync engine.** Built on `diffsync` with three sync flags (`SKIP_UNMATCHED_DST` default, `SKIP_UNMATCHED_SRC`, `SKIP_MODIFIED`) and optional Redis-backed store for stateful sync.
- **Declarative YAML configuration.** Per-field mapping with 14 filter operations (including `regex` and `is_ip_within`), per-field transforms, custom Jinja filters, and ordered cross-reference resolution.
- **Typer-based CLI.** Five commands — `list`, `diff` (read-only), `generate`, `sync`, `apply` (write a previously reviewed saved plan to the destination without re-extracting the source).
- **Typer-based CLI.** Register and inspect configuration packages, create plan or sync
runs, review service-owned plans, and apply a reviewed checksum through the Sync API.
- **Custom adapters and certificates.** Load custom adapters from filesystem paths, Python module paths, or installed entry points (`INFRAHUB_SYNC_ADAPTER_PATHS`); custom CA certificate support for internal PKI.
- **Example library.** Sample YAML configurations under `examples/` cover every pre-configured adapter plus additional targets (Device42, PeeringDB) and a custom adapter template. Review each example's adapter, schema, credential, and destination prerequisites before running it.

### Execution surfaces

| Surface | Use it for | Runtime requirements |
|---|---|---|
| CLI | Interactive and scheduled `list`, `generate`, `diff`, `sync`, and reviewed-plan `apply` operations | Base installation plus adapter dependencies |
| Python API | Typed in-process plan, verify, apply, and confirmed sync operations | Base installation plus adapter dependencies |
| CLI | Configuration registration, plan review, run admission, and reviewed-plan apply | Base installation and Sync API access |
| Python client | Typed access to every shipped Sync API resource | Base installation and Sync API access |
| Direct Prefect deployment | Starting and observing one plan or confirmed sync through Prefect's API | `prefect` extra and a Prefect server |
| Managed Sync HTTP API | Authenticated remote runs, durable records and artifacts, reviewed apply, idempotency, and cancellation | `managed` extra, Prefect, a work pool, a worker, and shared durable storage |

Expand Down
2 changes: 1 addition & 1 deletion changelog/+cli-core-migration.changed.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
Existing `diff`, `sync`, saved-plan review, and `apply` commands now share the same execution core. Their command names, options, output, and destination behavior are unchanged; a conflicting same-sync CLI run now emits an early holder-naming warning, waits for the bounded lock timeout, then refuses with a typed one-line error, with credential-shaped values redacted from reported errors.
The CLI now uses the Sync HTTP client for configuration registration, `diff`, `sync`, saved-plan review, and `apply`. Run commands address an immutable registered configuration version, and plan review and apply address the service run ID as a positional argument. The service owns execution, bounded waiting, and terminal records; credential-shaped values remain redacted from reported errors.
2 changes: 1 addition & 1 deletion changelog/+continue-on-error-source-references.changed.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
`diff` and `sync` now refuse a run in which a source record references a peer the loaded source cannot resolve, naming the peer, the referring field and the kinds that were searched. The reference used to be dropped silently by the destination write path, which wrote the object with the relationship absent; the plan a run derives has to be a faithful record of what it would write, so an unresolvable reference now fails the run before anything reaches the destination. `--continue-on-error` does not soften this — it covers peer identifiers missing while a system is loaded, not references the source itself cannot resolve. Add the peer's kind to your configuration so it is loaded, or remove the relationship from the schema mapping.
`diff` and `sync` now refuse a run in which a source record references a peer the loaded source cannot resolve, naming the peer, the referring field, and the kinds that were searched. The reference used to be dropped silently by the destination write path, which wrote the object with the relationship absent. A plan must be a faithful record of what it would write, so an unresolvable source reference now fails the run before anything reaches the destination. Add the peer's kind to the registered configuration so it is loaded, or remove the relationship from the schema mapping.
2 changes: 1 addition & 1 deletion changelog/+saved-plan-review-and-apply.added.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
`diff` and `sync` now save the changes they computed as a plan artifact under the run directory, `diff --from-plan <run-id>` reviews that plan without contacting either system, and `apply --run-id <run-id>` writes it to the destination without re-extracting the source. A plan is verified against its own checksum before the first write, and the bytes verified are the bytes applied. Review depth is `--detail` (one record per operation, with the payload and relationship peers it would write, credential-looking fields redacted) and `--kind` to narrow it. Two flags bind an approval to what was approved: `apply --expected-checksum <value>` refuses unless the stored plan still hashes to the value the review printed, and an apply refuses when the live destination endpoint or branch differs from the one recorded in the plan unless `--allow-destination-change` is passed. Once a run holds a committed plan, re-planning into the same run id is refused — a re-plan means a new run id. Deletes are recorded in a plan and counted in the run record but are never executed. Two limits remain: a kind whose convergence key crosses a relationship is applied with a warning, and convergence is not verified for this path, and a plan cannot clear a cardinality-one relationship.
`diff` and `sync` now submit service-owned runs that retain the computed plan. Review a saved plan with `runs plan <run-id>` and add `--detail` for one record per operation; `--kind` narrows that detailed view. Apply the reviewed plan with `apply <run-id> --expected-checksum <value>`. The service verifies the checksum before the first write, binds execution to the registered configuration and destination context, and refuses stale schema fingerprints. Once a run holds a committed plan, re-planning requires a new run. Deletes are recorded and counted but are never executed. Two limits remain: convergence is not verified for kinds whose identity crosses a relationship, and a plan cannot clear a cardinality-one relationship.
7 changes: 4 additions & 3 deletions docs/docs/adapters/aci.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -135,12 +135,13 @@ This transform:
- Builds a mapping of node IDs to device names
- The filter performs a lookup with a fallback to the original node ID if not found

## Generating the models
## Registering the package

Use the generate command to produce models from the schema mapping and examples:
Register the example package before creating a run:

```bash
uv run infrahub-sync generate --name from-cisco-aci --directory examples/
uv run infrahub-sync configs register examples/aci_to_infrahub/package.yml \
--reason "register Cisco ACI import"
```

## Common issues and troubleshooting
Expand Down
7 changes: 4 additions & 3 deletions docs/docs/adapters/infrahub.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -191,12 +191,13 @@ This allows you to:
| Only `owner` specified | CoreAccount (source name) | CoreAccountGroup |
| Both specified | CoreAccountGroup | CoreAccountGroup |

## Generating the models
## Registering the package

Use the generate command to produce Python models from your configuration:
Register the example package before creating a run:

```bash
uv run infrahub-sync generate --name from-netbox --directory examples/
uv run infrahub-sync configs register examples/netbox_to_infrahub/package.yml \
--reason "register NetBox import"
```

## Common issues and troubleshooting
Expand Down
8 changes: 5 additions & 3 deletions docs/docs/adapters/prometheus.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,14 @@ schema_mapping:
expression: "{{ lookup('node_uname_info', {}, 'labels.nodename') or 'unknown' }}"
```

## Generating the models
## Registering the package

Use the generate command to produce Python models from your configuration:
Register the example package before creating a run:

```bash
uv run infrahub-sync generate --name from-node-exporter --directory examples/
uv run infrahub-sync configs register \
"examples/prometheus_to_infrahub (node_exporter)/package.yml" \
--reason "register Prometheus import"
```

## Common issues and troubleshooting
Expand Down
6 changes: 3 additions & 3 deletions docs/docs/contributing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ This installs all runtime and development dependencies defined in `pyproject.tom

```bash
uv run infrahub-sync --help
uv run infrahub-sync list --directory examples/
uv run infrahub-sync configs --help
```

## Development workflow
Expand All @@ -60,8 +60,8 @@ After making changes, verify the CLI still works:

```bash
uv run infrahub-sync --help
uv run infrahub-sync list --directory examples/
uv run infrahub-sync generate --name from-netbox --directory examples/
uv run infrahub-sync configs --help
uv run infrahub-sync runs plan --help
```

### Running tests
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/creating-a-sync-project.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ These flags govern the live comparison that `diff` prints and that `sync` writes
govern the saved plan artifact: a plan records a `delete` for every destination object missing from
the source regardless of `SKIP_UNMATCHED_DST`. Recording one is not executing it — `apply` never
executes a delete, and completes successfully while reporting how many it skipped. See
[Deletes are recorded, never executed](./running-a-sync#deletes-are-recorded-never-executed).
[Deletes in a plan](./reference/cache-layout#deletes-in-a-plan).

:::

Expand Down
19 changes: 10 additions & 9 deletions docs/docs/migrating-from-netbox-or-nautobot.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,11 @@ The Infrahub schema does not have to mirror the source system one-to-one. Migrat

For a concrete NetBox starting point, follow the
[NetBox demo tutorial](./tutorials/netbox-demo-to-infrahub.mdx). It provisions a
fresh Infrahub instance, loads the matching schema-library revision, creates a
current NetBox demo token, installs the adapter dependency, and generates the
`from-netbox` sync code. Use `examples/nautobot_to_infrahub/` as a configuration
reference for Nautobot, and replace its model and field names with the equivalents
in your Infrahub schema.
fresh Infrahub instance, loads the matching schema-library revision, configures the
Sync API and service worker, creates a current NetBox demo token, installs the adapter
dependency, and registers the `from-netbox` configuration package. Use
`examples/nautobot_to_infrahub/` as a configuration reference for Nautobot, and replace
its model and field names with the equivalents in your Infrahub schema.

## Phase 2 — Build the sync project

Expand Down Expand Up @@ -67,10 +67,11 @@ For each model, define how source fields map to Infrahub fields. The schema mapp

## Phase 3 — Seed Infrahub with initial data

Run `infrahub-sync diff` to verify what the sync will do before applying changes:
Register the package, then use its immutable identity to verify what the sync will do:

```shell
infrahub-sync diff --name <project> --directory <dir>
infrahub-sync configs register <package.yml> --reason "register migration"
infrahub-sync diff --config-id <config-id> --version <version> --reason "review migration"
```

:::warning Current NetBox saved-plan limits
Expand Down Expand Up @@ -102,7 +103,7 @@ The diff prints the proposed changes to the terminal without modifying Infrahub.
When the diff looks correct, run the sync:

```shell
infrahub-sync sync --name <project> --directory <dir>
infrahub-sync sync --config-id <config-id> --version <version> --reason "seed Infrahub"
```

The sync applies changes in the order defined by the project's `order` key. Independent models are created first, then dependent models, then models that reference earlier ones.
Expand All @@ -123,7 +124,7 @@ By default, the `SKIP_UNMATCHED_DST` flag is enabled — Infrahub objects withou

To change this behavior for `sync` — for example, removing Infrahub objects when they're deleted in the source system — adjust the `diffsync_flags` in the sync configuration. See [Sync instance configuration](./reference/config.mdx).

The saved-plan path behaves differently and the flags do not change it. A plan **records** a delete for every Infrahub object missing from the source, but `apply` never executes one: it applies the non-deletes, completes successfully, and reports how many deletes it skipped. Reviewing a plan against a destination holding Infrahub-only data will therefore show a non-zero delete count that no apply will act on. See [Deletes are recorded, never executed](./running-a-sync#deletes-are-recorded-never-executed).
The saved-plan path behaves differently and the flags do not change it. A plan **records** a delete for every Infrahub object missing from the source, but `apply` never executes one: it applies the non-deletes, completes successfully, and reports how many deletes it skipped. Reviewing a plan against a destination holding Infrahub-only data will therefore show a non-zero delete count that no apply will act on. See [Deletes in a plan](./reference/cache-layout#deletes-in-a-plan).

### Decide on conflict policy early

Expand Down
4 changes: 3 additions & 1 deletion docs/docs/orchestration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@ Any tool that can run a CLI command on a schedule and capture its output works.

### Cron

The most direct option. A cron entry runs `infrahub-sync sync` on a defined schedule. Output and exit codes are captured by your normal cron logging.
The most direct option. A cron entry submits `infrahub-sync sync --config-id <id>
--version <version> --reason "scheduled sync"` on a defined schedule. Output and exit
codes are captured by your normal cron logging.

**When this works well:**

Expand Down
Loading
Loading