diff --git a/src/migrate/AGENTS.md b/src/migrate/AGENTS.md new file mode 100644 index 00000000000..8eddac45e0a --- /dev/null +++ b/src/migrate/AGENTS.md @@ -0,0 +1,93 @@ +# Principal Engineer Review Mode — migrate extension + +This file is a standing instruction for **any** agent doing coding work anywhere under +`src/migrate/`. Treat every change as if you are the **principal engineer** who must approve +the pull request. Do not merely make code work — make it the code a principal engineer would +sign off on. + +## Non-negotiable review discipline + +Before finishing ANY migrate task, rigorously self-review against these criteria and reject +your own work if it fails: + +1. **Simplicity** — Is this the simplest solution that fully solves the problem? Remove any + complexity that does not earn its place. +2. **Reuse first** — Prefer existing helpers, patterns, and abstractions over new ones. Search + before you write. (`shared/`, `runbook/`, existing `ArmClient`/`files` patterns.) +3. **Architecture fit** — The change must match the established structure (REST via `ArmClient`, + `shared/files.py` for archive/IO, `runbook/cmds/*` for command logic, `transformers.py` for + table shaping). No parallel or competing mechanisms. +4. **No speculative code** — Do not add constants, parameters, branches, or error handling for + cases that cannot occur or are unproven. Validate only at real system boundaries. +5. **No duplicate logic** — Collapse repeated iterate/parse/classify/format loops into a single + source of truth. Duplication is a defect. +6. **Root-cause fixes only** — Fix the underlying cause, never paper over a symptom. State the + root cause explicitly in your summary. +7. **Net code growth** — Prefer changes that remove more than they add. Justify every new + abstraction with a concrete, present-day need and a net-complexity benefit. +8. **Security by design** — Prefer designing hazards out (e.g. flatten to basename to eliminate + zip-slip) over runtime guards. Keep the OWASP Top 10 in mind for every I/O boundary. +9. **Maintenance score** — Rate the resulting code 1–10 for maintainability. Do not ship below + **9**. If below 9, keep simplifying. +10. **PR approval test** — Ask: "Would I approve this PR as principal engineer?" If not, revise. + +## Mandatory concluding deliverable + +Every non-trivial migrate change MUST end with a **10-point engineering review** covering: + +1. Selected design and why it won. +2. Alternatives considered and why they were rejected. +3. What existing code was reused. +4. What was refactored/consolidated. +5. Duplicate logic removed. +6. New abstractions introduced and their justification. +7. Net lines added vs. removed. +8. Remaining technical debt (with explicit `TODO(confirm)` where behavior is unverified). +9. Maintenance score (1–10) with rationale. +10. Why this is the simplest correct solution. + +## Verification gate (always run before declaring done) + +- `python -m pytest migrate/azext_migrate/tests/latest/runbook/test_runbook_unit.py -q` +- `python -m azdev style migrate` +- `python -m azdev linter migrate` (the trailing `ERROR: invalid git repo: None` is harmless) + +## Tests move with the code — never leave a reconciliation gap + +Code and its tests are ONE change. A task is not done until the tests that cover the changed +behavior are updated in the SAME change and the suite is green. + +- **Every code change updates its tests in lockstep.** If you change a contract (request body, + command signature, transformer columns, file/archive handling, action verb, call kwargs), update + the covering unit/scenario tests in the same edit. Never defer test updates to "later" or to a + separate reconciliation pass. +- **Green-before-done.** Run the unit suite (see Verification gate) and confirm it passes before + declaring any change complete. A change that leaves failing tests is an unfinished change. +- **Tests must load the source under `src/migrate/azext_migrate/`, not build artifacts.** Run + pytest with `cwd = src/migrate`. A stale `build/lib/azext_migrate` copy can shadow/merge with the + source package and mask source/test drift (a suite may appear to pass against the stale copy). + If collection counts look inflated or failures vanish inexplicably, delete `src/migrate/build/` + (a regenerable artifact) and clear `__pycache__`, then re-run against source. +- **Root cause of drift:** code advanced while its tests were not updated in the same change. Do not + recreate that state. When source and tests disagree, the source is authoritative only because it + was reviewed — still confirm the current behavior is intended before aligning the test to it. + +## Domain facts to preserve + +- Downloaded runbook archive members: + - `runbook.json` → the **definition** (`{"runbookSpec": {...}}`). + - `user-input(s).json` → the **parameters** (`{"runbookInputs": {...}}`). `definition download` + writes this alongside the definition (per-step `configurationStatus` is derived from it), but + table/CLI output (`show`, `visualize` grid) still renders the definition only. + - `derived-input(s).json` → same shape as user-inputs; **never downloaded/rendered** by any CLI. + It is distinguishable from user-inputs ONLY by filename, so it is excluded by name. +- Archive members are classified by **content**, not filename suffix (member naming varies across + services, e.g. `rb--spec.json` vs `runbook.json`). See `shared/files.py::_classify_archive` + as the single source of truth. +- **UpdateStep/AddStep `dependsOn` write contract (verified against live service):** each entry is a + System.Text.Json polymorphic `RunbookStepDependency`. The discriminator property is the verbatim + (non-camelCased) `"Mode"` whose value is the integer enum ordinal (`0` = step gate, + `1` = migration-entity gate), and it must appear first. A `--depends-on ` maps to + `{"Mode": 0, "stepId": ""}`. See `models.py::_depends_on_refs`. NOTE: the GET (read) model + differs — it emits `{"step": "", "mode": "migrationEntity"}` (property `step`, string `mode`). + Read/write are NOT symmetric; do not assume round-trip. diff --git a/src/migrate/HISTORY.rst b/src/migrate/HISTORY.rst index bd2edf06b3f..e74ea6005c1 100644 --- a/src/migrate/HISTORY.rst +++ b/src/migrate/HISTORY.rst @@ -2,6 +2,22 @@ Release History =============== +3.0.0b6 ++++++++++++++++ +* Add ``az migrate runbook`` commands (generate, show, list, update, + regenerate, delete, wait). +* Add ``az migrate runbook definition`` commands (show, download, + visualize). +* Add ``az migrate runbook definition step`` commands (add, update, + remove) and ``az migrate runbook definition workstream`` commands + (split, merge). +* Add ``az migrate runbook parameter`` and + ``az migrate runbook execution parameter`` commands (download, upload). +* Add ``az migrate runbook execution`` commands (start, show, list, + pause, resume, cancel, visualize). +* Add ``az migrate runbook execution step`` commands (retry, approve, + complete). + 3.0.0b5 +++++++++++++++ * Change migrate command parameter name. diff --git a/src/migrate/azext_migrate/__init__.py b/src/migrate/azext_migrate/__init__.py index 65abfbf53f6..bf5e7a56370 100644 --- a/src/migrate/azext_migrate/__init__.py +++ b/src/migrate/azext_migrate/__init__.py @@ -36,11 +36,16 @@ def load_command_table(self, args): args=args ) load_command_table(self, args) + from azext_migrate.runbook.commands import ( + load_runbook_command_table) + load_runbook_command_table(self) return self.command_table def load_arguments(self, command): from azext_migrate._params import load_arguments load_arguments(self, command) + from azext_migrate.runbook.params import load_runbook_arguments + load_runbook_arguments(self, command) COMMAND_LOADER_CLS = MigrateCommandsLoader diff --git a/src/migrate/azext_migrate/_help.py b/src/migrate/azext_migrate/_help.py index 38607ab9b11..d62d69703a3 100644 --- a/src/migrate/azext_migrate/_help.py +++ b/src/migrate/azext_migrate/_help.py @@ -4,6 +4,7 @@ # -------------------------------------------------------------------------------------------- # coding=utf-8 from knack.help_files import helps # pylint: disable=unused-import +from azext_migrate.runbook import _help as _runbook_help # noqa: F401 helps['migrate'] = """ diff --git a/src/migrate/azext_migrate/_params.py b/src/migrate/azext_migrate/_params.py index a9569f42295..1f33cd324fb 100644 --- a/src/migrate/azext_migrate/_params.py +++ b/src/migrate/azext_migrate/_params.py @@ -12,7 +12,7 @@ def load_arguments(self, _): project_name_type = CLIArgumentType( - options_list=['--project-name'], + options_list=['--project-name', '-p'], help='Name of the Azure Migrate project.', id_part='name' ) diff --git a/src/migrate/azext_migrate/runbook/__init__.py b/src/migrate/azext_migrate/runbook/__init__.py new file mode 100644 index 00000000000..34913fb394d --- /dev/null +++ b/src/migrate/azext_migrate/runbook/__init__.py @@ -0,0 +1,4 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- diff --git a/src/migrate/azext_migrate/runbook/_help.py b/src/migrate/azext_migrate/runbook/_help.py new file mode 100644 index 00000000000..ee80dd4a994 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/_help.py @@ -0,0 +1,469 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +# coding=utf-8 +from knack.help_files import helps # pylint: disable=unused-import + + +helps['migrate runbook'] = """ + type: group + short-summary: Manage Azure Migrate runbooks. + long-summary: | + Commands to generate and manage Azure Migrate runbooks and their + executions. This command group is in preview and under active + development; additional subgroups and commands are added + incrementally. +""" + + +helps['migrate runbook show'] = """ + type: command + short-summary: Get the details of a runbook. + examples: + - name: Show the details of a runbook. + text: | + az migrate runbook show -g myRg --project-name myProject \\ + -n myRunbook +""" + + +helps['migrate runbook generate'] = """ + type: command + short-summary: Generate a runbook for a migration wave. + examples: + - name: Generate a runbook scoped to a wave. + text: | + az migrate runbook generate -g myRg --project-name myProject \\ + -n myRunbook --wave-name myWave + - name: Generate a runbook and return immediately. + text: | + az migrate runbook generate -g myRg --project-name myProject \\ + -n myRunbook --wave-name myWave --no-wait +""" + + +helps['migrate runbook list'] = """ + type: command + short-summary: List runbooks in a migrate project. + examples: + - name: List all runbooks in a project. + text: | + az migrate runbook list -g myRg --project-name myProject + - name: List runbooks filtered by wave and status. + text: | + az migrate runbook list -g myRg --project-name myProject \\ + --wave-name myWave --status InExecution +""" + + +helps['migrate runbook delete'] = """ + type: command + short-summary: Delete a runbook. + examples: + - name: Delete a runbook. + text: | + az migrate runbook delete -g myRg --project-name myProject \\ + -n myRunbook +""" + + +helps['migrate runbook update'] = """ + type: command + short-summary: Update editable runbook metadata. + examples: + - name: Update a runbook description. + text: | + az migrate runbook update -g myRg --project-name myProject \\ + -n myRunbook --description "Wave 1 cutover runbook" +""" + + +helps['migrate runbook regenerate'] = """ + type: command + short-summary: Regenerate a runbook from its current scope. + examples: + - name: Regenerate a runbook. + text: | + az migrate runbook regenerate -g myRg \\ + --project-name myProject -n myRunbook + - name: Regenerate a runbook and return immediately. + text: | + az migrate runbook regenerate -g myRg \\ + --project-name myProject -n myRunbook --no-wait +""" + + +helps['migrate runbook definition'] = """ + type: group + short-summary: View and download the contents of a runbook definition. +""" + + +helps['migrate runbook definition show'] = """ + type: command + short-summary: Show the definition (contents) of a runbook. + examples: + - name: Show a runbook definition. + text: | + az migrate runbook definition show -g myRg \\ + --project-name myProject -n myRunbook + - name: Show a single workstream in a runbook definition. + text: | + az migrate runbook definition show -g myRg \\ + --project-name myProject -n myRunbook \\ + --workstream-id myWorkstream +""" + + +helps['migrate runbook definition download'] = """ + type: command + short-summary: Download the runbook definition and documentation files. + examples: + - name: Download a runbook definition to the current directory. + text: | + az migrate runbook definition download -g myRg \\ + --project-name myProject -n myRunbook + - name: Download a runbook definition to a specific directory. + text: | + az migrate runbook definition download -g myRg \\ + --project-name myProject -n myRunbook \\ + --destination ./runbooks +""" + + +helps['migrate runbook definition visualize'] = """ + type: command + short-summary: Render the runbook definition as a self-contained HTML page. + examples: + - name: Visualize a runbook definition and open it in the browser. + text: | + az migrate runbook definition visualize -g myRg \\ + --project-name myProject -n myRunbook --open + - name: Visualize from a local definition file. + text: | + az migrate runbook definition visualize \\ + --from-file ./definition.json --file ./definition.html +""" + + +helps['migrate runbook definition step'] = """ + type: group + short-summary: Manage individual steps in a runbook definition. +""" + + +helps['migrate runbook definition step add'] = """ + type: command + short-summary: Add a step to the runbook definition. + examples: + - name: Add a manual step to a workstream. + text: | + az migrate runbook definition step add -g myRg \\ + --project-name myProject -n myRunbook \\ + --step-type Manual --step-name "Verify cutover" \\ + --workstream-id workstream-0 + - name: Add an approval step that depends on another step. + text: | + az migrate runbook definition step add -g myRg \\ + --project-name myProject -n myRunbook \\ + --step-type Approval --step-name "Change approval" \\ + --workstream-id workstream-0 --depends-on step0 + - name: Add a manual step scoped to specific migration entities. + text: | + az migrate runbook definition step add -g myRg \\ + --project-name myProject -n myRunbook \\ + --step-type Manual --step-name "Post checks" \\ + --workstream-id workstream-0 \\ + --migration-entity-ids entity1 entity2 +""" + + +helps['migrate runbook definition step update'] = """ + type: command + short-summary: Update a step in the runbook definition. + examples: + - name: Rename a step and change its dependencies. + text: | + az migrate runbook definition step update -g myRg \\ + --project-name myProject -n myRunbook \\ + --step-id step1 --step-name "New name" \\ + --depends-on step0 +""" + + +helps['migrate runbook definition step remove'] = """ + type: command + short-summary: Remove a step from the runbook definition. + examples: + - name: Remove a step by id. + text: | + az migrate runbook definition step remove -g myRg \\ + --project-name myProject -n myRunbook --step-id step1 +""" + + +helps['migrate runbook definition workstream'] = """ + type: group + short-summary: Manage workstreams in a runbook definition. +""" + + +helps['migrate runbook definition workstream split'] = """ + type: command + short-summary: Split a workstream into two workstreams. + examples: + - name: Move steps into a new workstream. + text: | + az migrate runbook definition workstream split -g myRg \\ + --project-name myProject -n myRunbook \\ + --source-workstream-id ws1 \\ + --new-workstream-name "Database tier" \\ + --step-ids step1 step2 +""" + + +helps['migrate runbook definition workstream merge'] = """ + type: command + short-summary: Merge two or more workstreams into a single workstream. + examples: + - name: Merge two workstreams. + text: | + az migrate runbook definition workstream merge -g myRg \\ + --project-name myProject -n myRunbook \\ + --source-workstream-ids ws1 ws2 \\ + --new-workstream-name "Combined tier" +""" + + +helps['migrate runbook execution'] = """ + type: group + short-summary: Manage runbook executions. +""" + + +helps['migrate runbook execution start'] = """ + type: command + short-summary: Start a new execution of a runbook. + examples: + - name: Start a runbook execution. + text: | + az migrate runbook execution start -g myRg \\ + --project-name myProject --runbook-name myRunbook + - name: Start without waiting for completion. + text: | + az migrate runbook execution start -g myRg \\ + --project-name myProject --runbook-name myRunbook --no-wait +""" + + +helps['migrate runbook execution show'] = """ + type: command + short-summary: Show (or watch) the status of a runbook execution. + examples: + - name: Show an execution's status. + text: | + az migrate runbook execution show -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution + - name: Show the status of a single step. + text: | + az migrate runbook execution show -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --step-id step1 + - name: Auto-refresh the status table until it completes. + text: | + az migrate runbook execution show -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --watch +""" + + +helps['migrate runbook execution list'] = """ + type: command + short-summary: List the executions of a runbook. + examples: + - name: List a runbook's executions. + text: | + az migrate runbook execution list -g myRg \\ + --project-name myProject --runbook-name myRunbook +""" + + +helps['migrate runbook execution pause'] = """ + type: command + short-summary: Pause an in-progress runbook execution. + examples: + - name: Pause an execution. + text: | + az migrate runbook execution pause -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution +""" + + +helps['migrate runbook execution resume'] = """ + type: command + short-summary: Resume a paused runbook execution. + examples: + - name: Resume an execution. + text: | + az migrate runbook execution resume -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution +""" + + +helps['migrate runbook execution cancel'] = """ + type: command + short-summary: Cancel an in-progress or paused runbook execution. + long-summary: > + Cancellation is terminal; a cancelled execution cannot be + resumed. Start a new execution instead. + examples: + - name: Cancel an execution. + text: | + az migrate runbook execution cancel -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution +""" + + +helps['migrate runbook wait'] = """ + type: command + short-summary: Wait until a runbook reaches a desired state. + examples: + - name: Wait until execution completes. + text: | + az migrate runbook wait -g myRg --project-name myProject \\ + -n myRunbook --custom "properties.state=='ExecutionSucceeded'" + - name: Wait until the runbook exists. + text: | + az migrate runbook wait -g myRg --project-name myProject \\ + -n myRunbook --created +""" + + +helps['migrate runbook execution visualize'] = """ + type: command + short-summary: Render an execution's status as a self-contained HTML graph. + examples: + - name: Visualize an execution's status and open it in the browser. + text: | + az migrate runbook execution visualize -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --open + - name: Regenerate the snapshot on an interval until it completes. + text: | + az migrate runbook execution visualize -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --watch +""" + + +helps['migrate runbook execution step'] = """ + type: group + short-summary: Act on a single step within a runbook execution. +""" + + +helps['migrate runbook execution step retry'] = """ + type: command + short-summary: Restart the execution of a failed step. + examples: + - name: Retry a failed step. + text: | + az migrate runbook execution step retry -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --step-id step1 +""" + + +helps['migrate runbook execution step approve'] = """ + type: command + short-summary: Provide approval for an approval-type step during execution. + examples: + - name: Approve a Full approval step. + text: | + az migrate runbook execution step approve -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --step-id step1 + - name: Approve specific ready entities for a Partial approval step. + text: | + az migrate runbook execution step approve -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --step-id step1 \\ + --entities vm1 vm2 +""" + + +helps['migrate runbook execution step complete'] = """ + type: command + short-summary: Mark a manual step as complete during execution. + examples: + - name: Complete a manual step with a comment. + text: | + az migrate runbook execution step complete -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --step-id step1 \\ + --comment "Verified manually" +""" + + +helps['migrate runbook parameter'] = """ + type: group + short-summary: Download and upload a runbook's parameters (inputs) file. +""" + + +helps['migrate runbook parameter download'] = """ + type: command + short-summary: Download the runbook's parameters file. + examples: + - name: Download the parameters file to the current directory. + text: | + az migrate runbook parameter download -g myRg \\ + --project-name myProject --runbook-name myRunbook +""" + + +helps['migrate runbook parameter upload'] = """ + type: command + short-summary: Upload a new parameters file and validate it. + examples: + - name: Upload a parameters file. + text: | + az migrate runbook parameter upload -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --file ./params.json +""" + + +helps['migrate runbook execution parameter'] = """ + type: group + short-summary: Download and upload an execution's input-parameters file. +""" + + +helps['migrate runbook execution parameter download'] = """ + type: command + short-summary: Download an execution's input-parameters file. + examples: + - name: Download the execution input file. + text: | + az migrate runbook execution parameter download -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution +""" + + +helps['migrate runbook execution parameter upload'] = """ + type: command + short-summary: Upload an execution's input-parameters file. + examples: + - name: Upload the execution input file. + text: | + az migrate runbook execution parameter upload -g myRg \\ + --project-name myProject --runbook-name myRunbook \\ + --execution-id myExecution --file ./input.json +""" diff --git a/src/migrate/azext_migrate/runbook/cmds/__init__.py b/src/migrate/azext_migrate/runbook/cmds/__init__.py new file mode 100644 index 00000000000..34913fb394d --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/__init__.py @@ -0,0 +1,4 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- diff --git a/src/migrate/azext_migrate/runbook/cmds/definition.py b/src/migrate/azext_migrate/runbook/cmds/definition.py new file mode 100644 index 00000000000..630c9df6317 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/definition.py @@ -0,0 +1,227 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook definition command implementations (show/download).""" + +import os + +from knack.log import get_logger +from azure.cli.core.azclierror import CLIInternalError +from azure.cli.core.commands.client_factory import get_subscription_id + +from azext_migrate.shared import arm_ids, files +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.shared.constants import ARTIFACTS_API_VERSION +from azext_migrate.runbook import config_status, models +from azext_migrate.runbook.constants import ( + RUNBOOK_ARTIFACT_DOWNLOAD_AS_ZIP, + RUNBOOK_DEFINITION_FILE, + ARTIFACT_DOWNLOAD_MODE_FILE, + ARTIFACT_DOWNLOAD_MODE_DIRECTORY, +) +from azext_migrate.runbook.visualize import graph as graph_mod +from azext_migrate.runbook.visualize import renderer +from azext_migrate.runbook.visualize import viewmodel + +logger = get_logger(__name__) + + +def _runbook_id(cmd, resource_group_name, project_name, runbook_name): + subscription_id = get_subscription_id(cmd.cli_ctx) + project = arm_ids.migrate_project_id( + subscription_id, resource_group_name, project_name) + return arm_ids.runbook_id(project, runbook_name) + + +def _artifact_id(cmd, resource_group_name, project_name, artifact): + """Resolve an artifact name or full ARM id to a full artifact ARM id. + + ``properties.artifactId`` may be either a bare artifact name or a full + ARM id. If it is already an ARM id, use it as-is (case-insensitive); + otherwise compose the id under the migrate project. + """ + if isinstance(artifact, str) and artifact.strip().lower().startswith( + '/subscriptions/'): + return artifact.strip() + subscription_id = get_subscription_id(cmd.cli_ctx) + project = arm_ids.migrate_project_id( + subscription_id, resource_group_name, project_name) + return arm_ids.artifact_id(project, artifact) + + +def _runbook_artifact_id(cmd, resource_group_name, project_name, + runbook_name): + """Read a runbook's ``artifactId`` and resolve it to a full ARM id.""" + runbook = ArmClient(cmd).get( + _runbook_id(cmd, resource_group_name, project_name, runbook_name)) + artifact = ((runbook or {}).get('properties') or {}).get('artifactId') + if not artifact: + raise CLIInternalError( + 'The runbook has no associated artifact to download.') + return _artifact_id( + cmd, resource_group_name, project_name, artifact) + + +def _download_url(cmd, resource_group_name, project_name, runbook_name, + path=RUNBOOK_DEFINITION_FILE): + """Return a SAS URL for a file within the runbook's definition artifact. + + Resolve the runbook's ``artifactId`` and request the latest version. + By default the single ``path`` blob is fetched in file mode (e.g. + ``runbook.json`` for the definition, ``input.json`` for the inputs); + when the service packages the whole artifact as a ZIP, flip + ``RUNBOOK_ARTIFACT_DOWNLOAD_AS_ZIP`` to fetch it all in directory mode. + """ + artifact_id = _runbook_artifact_id( + cmd, resource_group_name, project_name, runbook_name) + if RUNBOOK_ARTIFACT_DOWNLOAD_AS_ZIP: + body = models.build_artifact_download_url_body( + path="/", mode=ARTIFACT_DOWNLOAD_MODE_DIRECTORY) + else: + body = models.build_artifact_download_url_body( + path=path, mode=ARTIFACT_DOWNLOAD_MODE_FILE) + # Artifact LRO is polled at its own async-operation URI (do not rewrite + # the api-version the way the waveOperations runbook LRO requires). + client = ArmClient( + cmd, api_version=ARTIFACTS_API_VERSION, + rewrite_poll_api_version=False) + result = client.post_action( + artifact_id, 'generateDownloadUrl', body, return_final_poll=True) + url = files.extract_sas_url(result) + if not url: + raise CLIInternalError( + 'The service did not return a runbook download URL.') + return url + + +def _project_definition(definition, workstream_id, step_id): + """Filter the definition to a workstream and/or a single step.""" + workstreams = definition.get('workstreams', []) or [] + if workstream_id: + workstreams = [ + w for w in workstreams if w.get('id') == workstream_id] + if step_id: + for workstream in workstreams: + for step in workstream.get('steps', []) or []: + if step_id in (step.get('id'), step.get('stepId')): + return step + return {} + if workstream_id: + return workstreams[0] if workstreams else {} + return definition + + +def _definition_has_steps(definition): + """True when any workstream (or the flat step list) has at least a step.""" + if not isinstance(definition, dict): + return False + for workstream in definition.get('workstreams') or []: + if isinstance(workstream, dict) and workstream.get('steps'): + return True + return bool(definition.get('steps')) + + +def _load_definition(cmd, resource_group_name, project_name, runbook_name): + """Download the runbook archive and return an annotated definition. + + The archive holds both the definition (``runbookSpec``) and the + parameters (``runbookInputs``); the latter is used to stamp each step + with its computed ``configurationStatus`` so downstream table/grid/graph + rendering can show configuration readiness without re-fetching. + """ + zip_bytes = files.download_bytes(_download_url( + cmd, resource_group_name, project_name, runbook_name)) + spec = files.read_spec_json(zip_bytes) + if spec is None: + raise CLIInternalError( + 'The downloaded runbook artifact did not contain a definition ' + '(runbook.json). If the runbook was just generated, wait for it ' + 'to finish and try again.') + definition = spec.get('runbookSpec', spec) + runbook_inputs = files.read_parameters_json(zip_bytes) + config_status.annotate(definition, runbook_inputs) + if not _definition_has_steps(definition): + logger.warning( + 'The runbook definition has no steps yet (its workstreams are ' + 'empty). Nothing to display for this runbook.') + return definition + + +def _load_definition_from_file(spec_file, parameters_file=None): + """Load and annotate a runbook definition from local JSON files. + + ``spec_file`` is a runbook spec JSON (optionally wrapping the definition + under ``runbookSpec``). ``parameters_file`` is an optional parameters + JSON (a ``runbookInputs`` body, or a document that wraps it) used to + compute each step's ``configurationStatus``. Enables offline + rendering/testing without contacting the service. + """ + spec = files.read_json_file(spec_file) or {} + definition = spec.get('runbookSpec', spec) + runbook_inputs = None + if parameters_file: + params = files.read_json_file(parameters_file) + if isinstance(params, dict) and isinstance( + params.get('runbookInputs'), dict): + runbook_inputs = params['runbookInputs'] + else: + runbook_inputs = params + config_status.annotate(definition, runbook_inputs) + return definition + + +def show(cmd, resource_group_name, project_name, runbook_name, + workstream_id=None, step_id=None): + """Show the definition (contents) of a runbook.""" + definition = _load_definition( + cmd, resource_group_name, project_name, runbook_name) + return _project_definition(definition, workstream_id, step_id) + + +def download(cmd, resource_group_name, project_name, runbook_name, + destination=None): + """Download the runbook definition/documentation files to disk.""" + destination = destination or os.getcwd() + zip_bytes = files.download_bytes(_download_url( + cmd, resource_group_name, project_name, runbook_name)) + paths = files.extract_definition_files(zip_bytes, destination) + result = [] + for path in paths: + lower = os.path.basename(path).lower() + if lower.endswith('.md'): + kind = 'documentation' + elif 'input' in lower: + kind = 'parameters' + else: + kind = 'definition' + logger.warning( + 'Runbook %s file downloaded and saved to %s', kind, path) + result.append({'kind': kind, 'path': path}) + return result + + +def visualize(cmd, resource_group_name=None, project_name=None, + runbook_name=None, file=None, open_file=False, + from_file=None, parameters_file=None): + """Render the runbook definition as a self-contained HTML page.""" + if from_file: + definition = _load_definition_from_file(from_file, parameters_file) + name = runbook_name or os.path.splitext( + os.path.basename(from_file))[0] + else: + definition = _load_definition( + cmd, resource_group_name, project_name, runbook_name) + name = runbook_name + title = 'Runbook definition: %s' % name + dag = graph_mod.build_definition_graph(definition, title=title) + view = viewmodel.build_definition_view(definition, title=title) + html_text = renderer.render(dag, view=view) + target = files.resolve_output_path( + file, 'runbook-%s-definition.html' % name) + path = files.write_text(target, html_text) + logger.warning( + 'Runbook definition visualization saved to %s', path) + if open_file: + files.open_in_browser(path) + return {'path': path} diff --git a/src/migrate/azext_migrate/runbook/cmds/definition_step.py b/src/migrate/azext_migrate/runbook/cmds/definition_step.py new file mode 100644 index 00000000000..1a33acec189 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/definition_step.py @@ -0,0 +1,38 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook definition step commands (add/update/remove).""" + +from azext_migrate.runbook import models +from azext_migrate.runbook.cmds.definition import _runbook_id +from azext_migrate.shared.arm_client import ArmClient + + +def add(cmd, resource_group_name, project_name, runbook_name, step_type, + step_name, workstream_id, step_description=None, depends_on=None, + migration_entity_ids=None): + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_add_step_body( + step_type, step_name, workstream_id, + step_description=step_description, depends_on=depends_on, + migration_entity_ids=migration_entity_ids) + return ArmClient(cmd).post_action(resource_id, 'AddStep', body) + + +def update(cmd, resource_group_name, project_name, runbook_name, step_id, + step_name=None, step_description=None, depends_on=None): + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_update_step_body( + step_id, step_name=step_name, step_description=step_description, + depends_on=depends_on) + return ArmClient(cmd).post_action(resource_id, 'UpdateStep', body) + + +def remove(cmd, resource_group_name, project_name, runbook_name, step_id): + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_delete_step_body(step_id) + return ArmClient(cmd).post_action(resource_id, 'DeleteStep', body) diff --git a/src/migrate/azext_migrate/runbook/cmds/definition_workstream.py b/src/migrate/azext_migrate/runbook/cmds/definition_workstream.py new file mode 100644 index 00000000000..12ef3522cab --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/definition_workstream.py @@ -0,0 +1,27 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook definition workstream commands (split/merge).""" + +from azext_migrate.runbook import models +from azext_migrate.runbook.cmds.definition import _runbook_id +from azext_migrate.shared.arm_client import ArmClient + + +def split(cmd, resource_group_name, project_name, runbook_name, + source_workstream_id, new_workstream_name, step_ids): + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_split_workstream_body( + source_workstream_id, new_workstream_name, step_ids) + return ArmClient(cmd).post_action(resource_id, 'SplitWorkstream', body) + + +def merge(cmd, resource_group_name, project_name, runbook_name, + source_workstream_ids, new_workstream_name=None): + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_merge_workstreams_body( + source_workstream_ids, new_workstream_name) + return ArmClient(cmd).post_action(resource_id, 'MergeWorkstreams', body) diff --git a/src/migrate/azext_migrate/runbook/cmds/execution.py b/src/migrate/azext_migrate/runbook/cmds/execution.py new file mode 100644 index 00000000000..f0f16d66dc8 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/execution.py @@ -0,0 +1,258 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook execution command implementations (start/show/list + +pause/resume/cancel).""" + +import time + +from knack.log import get_logger +from azure.cli.core.azclierror import CLIInternalError, ManualInterrupt +from azure.cli.core.commands.client_factory import get_subscription_id + +from azext_migrate.shared import arm_ids +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.shared import files +from azext_migrate.runbook import models, transformers +from azext_migrate.runbook.models import ExecutionAction +from azext_migrate.runbook.constants import EXECUTION_TERMINAL_STATES +from azext_migrate.runbook.visualize import graph as graph_mod +from azext_migrate.runbook.visualize import renderer +from azext_migrate.runbook.visualize import viewmodel + +logger = get_logger(__name__) + + +def _runbook_id(cmd, resource_group_name, project_name, runbook_name): + subscription_id = get_subscription_id(cmd.cli_ctx) + project = arm_ids.migrate_project_id( + subscription_id, resource_group_name, project_name) + return arm_ids.runbook_id(project, runbook_name) + + +def _execution_resource_id(cmd, resource_group_name, project_name, + runbook_name, execution): + runbook = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + return arm_ids.execution_id(runbook, execution) + + +def _status_download_url(cmd, resource_id): + body = ArmClient(cmd).post_action(resource_id, 'GenerateDownloadUrl') + url = files.extract_sas_url(body) + if not url: + raise CLIInternalError( + 'The service did not return an execution status download URL.') + return url + + +def _fetch_status(cmd, resource_id): + """Download and parse the per-execution ``status.json`` via SAS. + + Raises :class:`CLIInternalError` when no status document exists yet + (for example a not-yet-run execution whose download archive contains + only the input parameters); the parameters blob is never returned as a + status document. + """ + return files.read_status_json( + files.download_bytes(_status_download_url(cmd, resource_id))) + + +def start(cmd, resource_group_name, project_name, runbook_name, + no_wait=False): + """Start a new execution of a runbook.""" + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + body = models.build_start_execution_body() + client = ArmClient(cmd) + result = client.post_action( + resource_id, 'execute', body, no_wait=no_wait) + execution_id = None + if isinstance(result, dict): + execution_id = result.get('name') or ( + result.get('properties') or {}).get('executionId') + if execution_id: + logger.warning( + "Runbook execution started. Execution id: %s", execution_id) + else: + logger.warning("Runbook execution started.") + if no_wait or not execution_id: + return result + # Re-read the execution child resource so callers render the latest + # status instead of the initial (stale) accepted response body. + return client.get(arm_ids.execution_id(resource_id, execution_id)) + + +def list_(cmd, resource_group_name, project_name, runbook_name): + """List the executions of a runbook.""" + collection_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + '/executions' + return ArmClient(cmd).list(collection_id) + + +def show(cmd, resource_group_name, project_name, runbook_name, + execution_id, step_id=None, watch=False, interval=5): + """Show (optionally watch) a runbook execution's status.""" + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + if watch: + return _watch(cmd, resource_id, execution_id, step_id, interval) + return _project(_fetch_status(cmd, resource_id), step_id) + + +def pause(cmd, resource_group_name, project_name, runbook_name, + execution_id): + """Pause an in-progress execution.""" + return _perform( + cmd, resource_group_name, project_name, runbook_name, + execution_id, ExecutionAction.PAUSE) + + +def resume(cmd, resource_group_name, project_name, runbook_name, + execution_id): + """Resume a paused execution.""" + return _perform( + cmd, resource_group_name, project_name, runbook_name, + execution_id, ExecutionAction.RESUME) + + +def cancel(cmd, resource_group_name, project_name, runbook_name, + execution_id): + """Cancel an in-progress or paused execution.""" + return _perform( + cmd, resource_group_name, project_name, runbook_name, + execution_id, ExecutionAction.CANCEL) + + +def _perform(cmd, resource_group_name, project_name, runbook_name, + execution_id, action): + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + body = models.build_perform_action_body(action) + return ArmClient(cmd).post_action(resource_id, 'PerformAction', body) + + +def _project(execution, step_id): + """Filter an execution status to a single step when requested.""" + if not step_id or not isinstance(execution, dict): + return execution + status = execution.get('properties', execution) or {} + workstreams = status.get('workstreams') or [] + for workstream in workstreams: + for step in workstream.get('steps', []) or []: + if step_id in (step.get('id'), step.get('stepId')): + return step + for step in status.get('steps', []) or []: + if step_id in (step.get('id'), step.get('stepId')): + return step + return execution + + +def _terminal(execution): + status = execution.get('properties', execution) or {} + state = status.get('state') or status.get('status') + return bool(state) and state.lower() in EXECUTION_TERMINAL_STATES + + +def _watch(cmd, resource_id, execution_id, step_id, interval): + """Re-render the execution status table until a terminal state.""" + logger.warning( + "Watching execution '%s' (interval: %ss). Press Ctrl+C to stop.", + execution_id, interval) + try: + while True: + execution = _fetch_status(cmd, resource_id) + _render(execution) + if _terminal(execution): + logger.warning( + "Execution '%s' reached a terminal state.", + execution_id) + return _project(execution, step_id) + time.sleep(interval) + except KeyboardInterrupt: + raise ManualInterrupt('Watch cancelled by user.') + + +def _render(execution): + rows = transformers.execution_table(execution) + if not rows: + logger.warning("No step status available yet.") + return + for row in rows: + logger.warning( + "%s | %s | %s | %s | %s", + row.get('Step Id'), row.get('Step Name'), row.get('Step Status'), + row.get('Depends On'), row.get('Workload Progress')) + + +def visualize(cmd, resource_group_name=None, project_name=None, + runbook_name=None, execution_id=None, file=None, + open_file=False, watch=False, interval=5, from_file=None): + """Render an execution's status as a self-contained HTML graph.""" + name = runbook_name or 'runbook' + exec_label = execution_id or 'local' + target = files.resolve_output_path( + file, 'runbook-%s-execution-%s.html' % (name, exec_label)) + if from_file: + path = _write_visualization( + files.read_json_file(from_file), name, exec_label, target) + logger.warning( + 'Runbook execution visualization saved to %s', path) + if open_file: + files.open_in_browser(path) + return {'path': path} + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + if watch: + return _watch_visualize( + cmd, resource_id, runbook_name, execution_id, target, + interval, open_file) + path = _write_visualization( + _fetch_status(cmd, resource_id), name, exec_label, target) + logger.warning( + 'Runbook execution visualization saved to %s', path) + if open_file: + files.open_in_browser(path) + return {'path': path} + + +def _write_visualization(execution, runbook_name, execution_id, target, + refresh_interval=None): + title = 'Runbook execution: %s / %s' % (runbook_name, execution_id) + dag = graph_mod.build_execution_graph(execution, title=title) + view = viewmodel.build_execution_view(execution, title=title) + return files.write_text( + target, + renderer.render(dag, view=view, refresh_interval=refresh_interval)) + + +def _watch_visualize(cmd, resource_id, runbook_name, execution_id, target, + interval, open_file): + """Regenerate the HTML snapshot on an interval until a terminal state.""" + logger.warning( + "Watching execution '%s' (interval: %ss). Press Ctrl+C to stop.", + execution_id, interval) + opened = False + try: + while True: + execution = _fetch_status(cmd, resource_id) + terminal = _terminal(execution) + # While running, bake an auto-reload tag so the browser refreshes + # itself; on the final (terminal) snapshot omit it so it stops. + path = _write_visualization( + execution, runbook_name, execution_id, target, + refresh_interval=None if terminal else interval) + logger.warning( + 'Runbook execution visualization saved to %s', path) + if open_file and not opened: + files.open_in_browser(path) + opened = True + if terminal: + logger.warning( + "Execution '%s' reached a terminal state.", + execution_id) + return {'path': path} + time.sleep(interval) + except KeyboardInterrupt: + raise ManualInterrupt('Watch cancelled by user.') diff --git a/src/migrate/azext_migrate/runbook/cmds/execution_parameter.py b/src/migrate/azext_migrate/runbook/cmds/execution_parameter.py new file mode 100644 index 00000000000..9b43e38b9b8 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/execution_parameter.py @@ -0,0 +1,85 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook execution input-parameters commands (download, upload). + +Per-execution inputs are served by the execution resource's own +``GenerateInputDownloadUrl`` / ``GenerateInputUploadUrl`` endpoints (not the +Artifact Service). Downloads fetch the input blob via SAS; uploads PUT the +file to blob storage. +""" + +import os + +from knack.log import get_logger +from azure.cli.core.azclierror import ( + CLIInternalError, + InvalidArgumentValueError, +) + +from azext_migrate.shared import files +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.runbook.cmds.execution import _execution_resource_id +from azext_migrate.runbook.constants import RUNBOOK_INPUT_FILE + +logger = get_logger(__name__) + + +def _download_url(cmd, resource_id): + body = ArmClient(cmd).post_action( + resource_id, 'GenerateInputDownloadUrl') + url = files.extract_sas_url(body) + if not url: + raise CLIInternalError( + 'The service did not return an execution input download URL.') + return url + + +def _upload_url(cmd, resource_id): + body = ArmClient(cmd).post_action( + resource_id, 'GenerateInputUploadUrl') + url = files.extract_sas_url(body) + if not url: + raise CLIInternalError( + 'The service did not return an execution input upload URL.') + return url + + +def download(cmd, resource_group_name, project_name, runbook_name, + execution_id, file=None): + """Download an execution's input-parameters file to disk.""" + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + blob = files.download_bytes(_download_url(cmd, resource_id)) + found = files.extract_parameters_file(blob) + if found: + default_name, data = found + else: + # File mode returns the raw input blob directly (not a ZIP). + default_name, data = RUNBOOK_INPUT_FILE, blob + target = files.resolve_output_path(file, default_name) + parent = os.path.dirname(target) + if parent: + os.makedirs(parent, exist_ok=True) + with open(target, 'wb') as handle: + handle.write(data) + logger.warning( + 'Execution input file downloaded and saved to %s', target) + return {'path': target} + + +def upload(cmd, resource_group_name, project_name, runbook_name, + execution_id, file): + """Upload an execution's input-parameters file.""" + source = os.path.abspath(file) + if not os.path.isfile(source): + raise InvalidArgumentValueError( + 'The parameters file was not found: {}'.format(source)) + with open(source, 'rb') as handle: + data = handle.read() + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + files.upload_bytes(_upload_url(cmd, resource_id), data) + logger.warning('Execution input file uploaded to Azure Migrate.') + return {'status': 'uploaded'} diff --git a/src/migrate/azext_migrate/runbook/cmds/execution_step.py b/src/migrate/azext_migrate/runbook/cmds/execution_step.py new file mode 100644 index 00000000000..698c692aef9 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/execution_step.py @@ -0,0 +1,61 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook execution step action commands (retry/approve/complete). + +Each command acts on a single step within an in-progress execution: + +* ``retry`` -> ``PerformAction`` with the integer ``RETRY`` (4) code. +* ``approve`` -> ``ProvideApproval`` with the ``"Approve"`` action string. +* ``complete`` -> ``UpdateStepStatus`` with the ``"Complete"`` action + string (a comment is required). +""" + +from knack.log import get_logger + +from azext_migrate.runbook import models +from azext_migrate.runbook.cmds.execution import _execution_resource_id +from azext_migrate.shared.arm_client import ArmClient + +logger = get_logger(__name__) + + +def retry(cmd, resource_group_name, project_name, runbook_name, + execution_id, step_id): + """Restart the execution of a failed step.""" + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + body = models.build_retry_step_body(step_id) + logger.warning("Step retry started.") + return ArmClient(cmd).post_action(resource_id, 'PerformAction', body) + + +def approve(cmd, resource_group_name, project_name, runbook_name, + execution_id, step_id, entities=None, all_ready=False): + """Provide approval for an approval-type step during execution. + + A Full approval step approves the whole step (no entities). A Partial + approval step approves either the supplied ``entities`` or, when + ``all_ready`` is set, every currently ready entity (an empty entity + list, which the service treats as approve-all-ready for the step). + """ + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + if all_ready: + entities = None + logger.info( + "Approving every ready entity for step '%s'.", step_id) + body = models.build_approve_step_body(step_id, entity_ids=entities) + logger.warning("Step approval recorded.") + return ArmClient(cmd).post_action(resource_id, 'ProvideApproval', body) + + +def complete(cmd, resource_group_name, project_name, runbook_name, + execution_id, step_id, comment): + """Mark a manual step as complete during execution.""" + resource_id = _execution_resource_id( + cmd, resource_group_name, project_name, runbook_name, execution_id) + body = models.build_complete_step_body(step_id, comment) + logger.warning("Step marked as complete.") + return ArmClient(cmd).post_action(resource_id, 'UpdateStepStatus', body) diff --git a/src/migrate/azext_migrate/runbook/cmds/parameter.py b/src/migrate/azext_migrate/runbook/cmds/parameter.py new file mode 100644 index 00000000000..866c7630cf1 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/parameter.py @@ -0,0 +1,73 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook parameters-file commands (download, upload). + +The parameters file is delivered inside the same SAS-protected ZIP that +``GenerateDownloadUrl`` returns for the runbook definition; the archive +contains both the definition/spec document and the parameters document. +Uploads use ``GenerateInputUploadUrl`` to obtain a SAS URL, PUT the file to +blob storage, and then run ``ValidateInput`` on the runbook. +""" + +import os + +from knack.log import get_logger +from azure.cli.core.azclierror import ( + CLIInternalError, + InvalidArgumentValueError, +) + +from azext_migrate.shared import files +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.runbook.cmds.definition import _download_url, _runbook_id +from azext_migrate.runbook.constants import RUNBOOK_INPUT_FILE + +logger = get_logger(__name__) + + +def download(cmd, resource_group_name, project_name, runbook_name, + file=None): + """Download the runbook parameters (inputs) file to disk.""" + blob = files.download_bytes(_download_url( + cmd, resource_group_name, project_name, runbook_name, + path=RUNBOOK_INPUT_FILE)) + found = files.extract_parameters_file(blob) + if found: + default_name, data = found + else: + # File mode returns the raw input blob directly (not a ZIP). + default_name, data = RUNBOOK_INPUT_FILE, blob + target = files.resolve_output_path(file, default_name) + parent = os.path.dirname(target) + if parent: + os.makedirs(parent, exist_ok=True) + with open(target, 'wb') as handle: + handle.write(data) + logger.warning('Parameters file downloaded and saved to %s', target) + return {'path': target} + + +def _upload_url(cmd, resource_id): + body = ArmClient(cmd).post_action(resource_id, 'GenerateInputUploadUrl') + url = files.extract_sas_url(body) + if not url: + raise CLIInternalError( + 'The service did not return a parameters upload URL.') + return url + + +def upload(cmd, resource_group_name, project_name, runbook_name, file): + """Upload a parameters file and report its validation status.""" + source = os.path.abspath(file) + if not os.path.isfile(source): + raise InvalidArgumentValueError( + 'The parameters file was not found: {}'.format(source)) + with open(source, 'rb') as handle: + data = handle.read() + resource_id = _runbook_id( + cmd, resource_group_name, project_name, runbook_name) + files.upload_bytes(_upload_url(cmd, resource_id), data) + logger.warning('Parameters file uploaded to Azure Migrate.') + return ArmClient(cmd).post_action(resource_id, 'ValidateInput') diff --git a/src/migrate/azext_migrate/runbook/cmds/runbook.py b/src/migrate/azext_migrate/runbook/cmds/runbook.py new file mode 100644 index 00000000000..3d398402e88 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/cmds/runbook.py @@ -0,0 +1,187 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook command implementations (generate/show/list/update/ +regenerate/delete/wait).""" + +import time + +from knack.log import get_logger +from knack.util import CLIError +from azure.cli.core.commands.client_factory import get_subscription_id +from azext_migrate.shared import arm_ids +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.runbook import models + +logger = get_logger(__name__) + + +def _project_id(cmd, resource_group_name, project_name): + subscription_id = get_subscription_id(cmd.cli_ctx) + return arm_ids.migrate_project_id( + subscription_id, resource_group_name, project_name) + + +def _matches(runbook, wave_name, status): + props = runbook.get('properties', {}) or {} + if status and props.get('status') != status: + return False + if wave_name: + scope = props.get('scope', {}) or {} + wid = (scope.get('waveId', '') or '').rstrip('/') + if not wid.endswith('/waves/' + wave_name): + return False + return True + + +def generate(cmd, resource_group_name, project_name, runbook_name, + wave_name, no_wait=False): + """Generate (create) a runbook scoped to a wave.""" + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + body = models.build_generate_body(models.wave_id(project, wave_name)) + return ArmClient(cmd).put(resource_id, body, no_wait=no_wait) + + +def show(cmd, resource_group_name, project_name, runbook_name): + """Get a single runbook.""" + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + return ArmClient(cmd).get(resource_id) + + +def list_(cmd, resource_group_name, project_name, wave_name=None, + status=None): + """List runbooks, optionally filtered by wave and/or status.""" + project = _project_id(cmd, resource_group_name, project_name) + collection_id = project + '/runbooks' + items = ArmClient(cmd).list(collection_id) + return [rb for rb in items if _matches(rb, wave_name, status)] + + +def delete(cmd, resource_group_name, project_name, runbook_name, + no_wait=False): + """Delete a runbook.""" + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + return ArmClient(cmd).delete(resource_id, no_wait=no_wait) + + +def update(cmd, resource_group_name, project_name, runbook_name, + description=None): + """Update editable runbook metadata (e.g. description).""" + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + body = models.build_update_body(description) + return ArmClient(cmd).patch(resource_id, body) + + +def regenerate(cmd, resource_group_name, project_name, runbook_name, + no_wait=False): + """Regenerate a runbook: delete it, then re-create it from its scope. + + The service has no Regenerate action, so the CLI reads the runbook's + current scope (wave), deletes the runbook, and re-generates it with the + same scope. + """ + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + client = ArmClient(cmd) + existing = client.get(resource_id) + scope = ((existing or {}).get('properties') or {}).get('scope') or {} + wave_id = scope.get('waveId') + if not wave_id: + raise CLIError( + 'Cannot regenerate: the runbook has no wave scope to ' + 'regenerate from.') + logger.warning( + "Regenerating runbook '%s': deleting and re-creating from its " + "wave scope.", runbook_name) + client.delete(resource_id) + body = models.build_generate_body(wave_id) + return client.put(resource_id, body, no_wait=no_wait) + + +def _provisioning_state(runbook): + props = (runbook or {}).get('properties', {}) or {} + return runbook.get('provisioningState') or \ + props.get('provisioningState') + + +def wait(cmd, resource_group_name, project_name, runbook_name, + created=False, updated=False, deleted=False, exists=False, + custom=None, interval=30, timeout=3600): + """Poll a runbook until a wait condition is met, logging progress.""" + from azure.cli.core.commands.arm import verify_property + from azure.cli.core.azclierror import ( + InvalidArgumentValueError, AzureResponseError) + + if not any([created, updated, deleted, exists, custom]): + raise InvalidArgumentValueError( + 'incorrect usage: --created | --updated | --deleted | ' + '--exists | --custom JMESPATH') + + project = _project_id(cmd, resource_group_name, project_name) + resource_id = arm_ids.runbook_id(project, runbook_name) + client = ArmClient(cmd) + + active = ', '.join(name for name, on in ( + ('created', created), ('updated', updated), ('deleted', deleted), + ('exists', exists), ('custom', custom)) if on) + logger.warning( + "Waiting for runbook '%s' [condition: %s, interval: %ss, " + "timeout: %ss].", runbook_name, active, interval, timeout) + + start = time.monotonic() + attempt = 0 + for _ in range(0, timeout, interval): + attempt += 1 + elapsed = int(time.monotonic() - start) + instance = client.get_or_none(resource_id) + + if instance is None: + if deleted: + logger.warning( + "Runbook '%s' is deleted (elapsed %ss, %s poll(s)).", + runbook_name, elapsed, attempt) + return None + logger.warning( + "Runbook '%s' not found yet (elapsed %ss, poll #%s, " + "next check in %ss).", + runbook_name, elapsed, attempt, interval) + time.sleep(interval) + continue + + if exists: + logger.warning( + "Runbook '%s' exists (elapsed %ss, %s poll(s)).", + runbook_name, elapsed, attempt) + return None + + state = _provisioning_state(instance) + norm = state.lower() if state else None + if norm == 'failed': + raise AzureResponseError( + "Runbook '%s' provisioning failed " + "(provisioningState=Failed)." % runbook_name) + if custom and bool(verify_property(instance, custom)): + logger.warning( + "Custom condition '%s' met (elapsed %ss, %s poll(s)).", + custom, elapsed, attempt) + return None + if (created or updated) and norm == 'succeeded': + logger.warning( + "Runbook '%s' provisioningState=Succeeded " + "(elapsed %ss, %s poll(s)).", + runbook_name, elapsed, attempt) + return None + + logger.warning( + "Still waiting for runbook '%s': provisioningState=%s " + "(elapsed %ss, poll #%s, next check in %ss).", + runbook_name, state or '(none)', elapsed, attempt, interval) + time.sleep(interval) + + raise CLIError( + 'Wait operation timed out after %ss.' % timeout) diff --git a/src/migrate/azext_migrate/runbook/commands.py b/src/migrate/azext_migrate/runbook/commands.py new file mode 100644 index 00000000000..3eee01d8d4b --- /dev/null +++ b/src/migrate/azext_migrate/runbook/commands.py @@ -0,0 +1,122 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Command registration for the ``migrate runbook`` feature package.""" + +from azure.cli.core.commands import CliCommandType +from azext_migrate.runbook.transformers import ( + runbook_table, + definition_table, + execution_table, + executions_table, +) + + +def load_runbook_command_table(self): + runbook_cmds = CliCommandType( + operations_tmpl='azext_migrate.runbook.cmds.{}') + + with self.command_group( + 'migrate runbook', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command( + 'generate', 'runbook#generate', + supports_no_wait=True, + table_transformer=runbook_table) + g.custom_show_command( + 'show', 'runbook#show', + table_transformer=runbook_table) + g.custom_command( + 'list', 'runbook#list_', + table_transformer=runbook_table) + g.custom_command( + 'update', 'runbook#update', + table_transformer=runbook_table) + g.custom_command( + 'regenerate', 'runbook#regenerate', + supports_no_wait=True, + table_transformer=runbook_table) + g.custom_command( + 'delete', 'runbook#delete', + supports_no_wait=True, + confirmation=True) + g.custom_command('wait', 'runbook#wait') + + with self.command_group( + 'migrate runbook definition', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_show_command( + 'show', 'definition#show', + table_transformer=definition_table) + g.custom_command('download', 'definition#download') + g.custom_command('visualize', 'definition#visualize') + + with self.command_group( + 'migrate runbook definition step', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command( + 'add', 'definition_step#add', + table_transformer=definition_table) + g.custom_command( + 'update', 'definition_step#update', + table_transformer=definition_table) + g.custom_command( + 'remove', 'definition_step#remove', + confirmation=True) + + with self.command_group( + 'migrate runbook definition workstream', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command( + 'split', 'definition_workstream#split', + table_transformer=definition_table) + g.custom_command( + 'merge', 'definition_workstream#merge', + table_transformer=definition_table) + + with self.command_group( + 'migrate runbook parameter', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command('download', 'parameter#download') + g.custom_command('upload', 'parameter#upload') + + with self.command_group( + 'migrate runbook execution', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command( + 'start', 'execution#start', + supports_no_wait=True) + g.custom_show_command( + 'show', 'execution#show', + table_transformer=execution_table) + g.custom_command( + 'list', 'execution#list_', + table_transformer=executions_table) + g.custom_command('pause', 'execution#pause') + g.custom_command('resume', 'execution#resume') + g.custom_command( + 'cancel', 'execution#cancel', + confirmation=True) + g.custom_command('visualize', 'execution#visualize') + + with self.command_group( + 'migrate runbook execution parameter', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command('download', 'execution_parameter#download') + g.custom_command('upload', 'execution_parameter#upload') + + with self.command_group( + 'migrate runbook execution step', + custom_command_type=runbook_cmds, + is_preview=True) as g: + g.custom_command('retry', 'execution_step#retry') + g.custom_command('approve', 'execution_step#approve') + g.custom_command('complete', 'execution_step#complete') diff --git a/src/migrate/azext_migrate/runbook/config_status.py b/src/migrate/azext_migrate/runbook/config_status.py new file mode 100644 index 00000000000..a60e4513921 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/config_status.py @@ -0,0 +1,133 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Compute the configuration status of a runbook definition step. + +A runbook is *shipped* as two documents inside the same download archive: + +* the **definition** (``runbookSpec``) — the immutable step graph, and +* the **parameters** (``runbookInputs``) — the per-step input *schema* plus + the customer-supplied *values*. + +A step is only runnable once every *required* input has a value. Required +inputs come in two scopes: + +* ``Appliance`` — one shared value stored at ``stepInputs[stepId][field]``. +* ``Entity`` — one value **per migration entity**, stored at + ``stepInputs[stepId].workloadOverrides[entityId][field]``. An entity-scope + field is only "set" when *every* entity on the step has a value for it. + +The status is one of: + +* ``Configured`` — all required inputs have values (or none are required), +* ``Partial (n/m)`` — some but not all required inputs have values, +* ``NotConfigured`` — no required input has a value, and +* ``Unknown`` — the step is not tracked in the parameters document + (``stepInputs`` has no entry for it), or no parameters are available. + +Steps that need no inputs (e.g. approval gates, cutover, cleanup) are emitted +as ``stepInputs[stepId] = {}`` with no ``schema`` entry; because they are +tracked and have no required inputs, they are :data:`CONFIGURED`. +""" + +CONFIGURED = 'Configured' +NOT_CONFIGURED = 'NotConfigured' +UNKNOWN = 'Unknown' + + +def _is_empty(value): + """Return True when ``value`` counts as "not provided".""" + if value is None: + return True + if isinstance(value, str): + return value.strip() == '' + if isinstance(value, (list, dict, tuple, set)): + return len(value) == 0 + return False + + +def _field_is_set(field, meta, step, step_inputs): + """Return True when a required input ``field`` has a value on the step.""" + scope = (meta.get('scope') if isinstance(meta, dict) else None) or \ + 'Appliance' + if scope == 'Entity': + overrides = step_inputs.get('workloadOverrides') or {} + entities = step.get('entities') or [] + if not entities: + return False + for entity_id in entities: + entity_values = overrides.get(entity_id) or {} + if _is_empty(entity_values.get(field)): + return False + return True + return not _is_empty(step_inputs.get(field)) + + +def compute(step, runbook_inputs): + """Return the configuration status string for a definition ``step``. + + ``runbook_inputs`` is the ``runbookInputs`` object from the parameters + document (with ``schema`` and ``stepInputs``). Returns :data:`UNKNOWN` + only when the step is not tracked under ``stepInputs`` (or no parameters + are available). A step tracked with an empty inputs object and no schema + entry has no required inputs and is therefore :data:`CONFIGURED`. + """ + step = step or {} + if not isinstance(runbook_inputs, dict): + return UNKNOWN + step_ref = step.get('stepRef') + step_id = step.get('stepId') or step.get('id') + # Presence under ``stepInputs`` (even as an empty object) is the signal + # that the parameters document tracks this step. A missing entry means + # the step is untracked -> Unknown. + step_inputs = (runbook_inputs.get('stepInputs') or {}).get(step_id) + if not isinstance(step_inputs, dict): + return UNKNOWN + # The schema entry may be absent for steps that need no inputs; treat a + # missing/invalid schema as "no required inputs". + schema = (runbook_inputs.get('schema') or {}).get(step_ref) + if not isinstance(schema, dict): + schema = {} + + required = [ + (field, meta) for field, meta in schema.items() + if isinstance(meta, dict) and meta.get('required')] + if not required: + return CONFIGURED + + set_count = sum( + 1 for field, meta in required + if _field_is_set(field, meta, step, step_inputs)) + total = len(required) + if set_count == 0: + return NOT_CONFIGURED + if set_count == total: + return CONFIGURED + return 'Partial (%d/%d)' % (set_count, total) + + +def annotate(definition, runbook_inputs): + """Stamp ``configurationStatus`` onto every step of ``definition``. + + Mutates and returns ``definition`` in place so the table transformer and + the graph/grid can read ``step['configurationStatus']`` without needing + the parameters document threaded through them. + """ + if not isinstance(definition, dict): + return definition + workstreams = definition.get('workstreams') + if isinstance(workstreams, list): + for workstream in workstreams: + if isinstance(workstream, dict): + _annotate_steps(workstream.get('steps'), runbook_inputs) + _annotate_steps(definition.get('steps'), runbook_inputs) + return definition + + +def _annotate_steps(steps, runbook_inputs): + if not isinstance(steps, list): + return + for step in steps: + if isinstance(step, dict): + step['configurationStatus'] = compute(step, runbook_inputs) diff --git a/src/migrate/azext_migrate/runbook/constants.py b/src/migrate/azext_migrate/runbook/constants.py new file mode 100644 index 00000000000..34591d11e8a --- /dev/null +++ b/src/migrate/azext_migrate/runbook/constants.py @@ -0,0 +1,150 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Runbook-feature constants (extends the shared api-version registry).""" + +from enum import Enum + +# Scope type used by CreateRunbook (generate). +SCOPE_TYPE_WAVE = "Wave" + +# Wave ARM id template (relative to the migrate project id). +WAVE_ID_TEMPLATE = "{project_id}/waves/{wave_name}" + +# Step types accepted by ``definition step add``. +STEP_TYPE_MANUAL = "Manual" +STEP_TYPE_APPROVAL = "Approval" +STEP_TYPE_VALUES = [ + STEP_TYPE_MANUAL, + STEP_TYPE_APPROVAL, +] + +# String action codes sent by the execution-step action endpoints. +# ``PerformAction`` (retry) sends the integer ``ExecutionAction`` code, +# but ``ProvideApproval`` / ``UpdateStepStatus`` send these PascalCase +# strings. +STEP_ACTION_APPROVE = "Approve" +STEP_ACTION_COMPLETE = "Complete" + +# ``DownloadMode`` for the Artifact Service GenerateDownloadUrl request: +# a single file or a whole directory (subtree) of the artifact. +ARTIFACT_DOWNLOAD_MODE_FILE = "file" +ARTIFACT_DOWNLOAD_MODE_DIRECTORY = "directory" + +# Runbook definition artifact fetch strategy. The service currently returns +# individual blobs (file mode, raw JSON). Set this True once the service +# packages the whole artifact as a downloadable ZIP so the CLI switches to +# directory mode without touching call sites. +RUNBOOK_ARTIFACT_DOWNLOAD_AS_ZIP = True +RUNBOOK_DEFINITION_FILE = "runbook.json" +RUNBOOK_INPUT_FILE = "input.json" + +# ``stepRef`` value the AddStep body binds per step type. These correlate +# the CLI step with the partner runbook step used for execution. +STEP_REF_BY_TYPE = { + STEP_TYPE_MANUAL: "common.manual", + STEP_TYPE_APPROVAL: "common.approval", +} + +# A step dependency in the AddStep/UpdateStep write model (service +# ``RunbookStepDependency``) is ``{"mode": , "stepId": }``. +# ``mode`` is the ``RunbookStepDependencyMode`` enum (string values). The +# CLI ``--depends-on`` takes step ids only and maps each to a Step gate. +STEP_DEPENDENCY_MODE_STEP = "Step" +STEP_DEPENDENCY_MODE_MIGRATION_ENTITY = "MigrationEntity" + + +class RunbookStatus(str, Enum): + """Runbook lifecycle status values (GetRunbook properties.status).""" + + GENERATING = "Generating" + NOT_CONFIGURED = "NotConfigured" + READY_TO_START = "ReadyToStart" + IN_EXECUTION = "InExecution" + PAUSED = "Paused" + COMPLETED = "Completed" + FAILED = "Failed" + + +# Ordered choices for the ``--status`` filter on ``runbook list``. +RUNBOOK_STATUS_VALUES = [member.value for member in RunbookStatus] + + +class RunbookExecutionStatus(str, Enum): + """Execution ARM resource status (GetRunbookExecution properties.status). + + Source of truth: service enum ``RunbookExecutionStatus`` + (Microsoft.Azure.Migrate.MgmtSvcs.Constants). + """ + + QUEUED = "Queued" + IN_PROGRESS = "InProgress" + COMPLETED = "Completed" + FAILED = "Failed" + PAUSING = "Pausing" + PAUSED = "Paused" + RESUMING = "Resuming" + CANCELLING = "Cancelling" + CANCELLED = "Cancelled" + + +class ExecutionState(str, Enum): + """Per-node state in the execution ``status.json`` document. + + Source of truth: service enum ``ExecutionState`` + (MigrationOrchestrator.Engine.Models.ExecutionStatus). Coordinator + nodes (runbook, workstream) use ``Completed``; steps report + ``Succeeded``/``PartiallySucceeded``. + """ + + NOT_STARTED = "NotStarted" + IN_PROGRESS = "InProgress" + AWAITING_USER_ACTION = "AwaitingUserAction" + COMPLETED = "Completed" + SUCCEEDED = "Succeeded" + PARTIALLY_SUCCEEDED = "PartiallySucceeded" + FAILED = "Failed" + CANCELLED = "Cancelled" + PAUSED = "Paused" + PAUSING = "Pausing" + RESUMING = "Resuming" + CANCELLING = "Cancelling" + SKIPPED = "Skipped" + + +# Telemetry fault types for this feature. +RUNBOOK_ARM_ERROR = "RUNBOOK_ARM_ERROR" +RUNBOOK_VALIDATION_ERROR = "RUNBOOK_VALIDATION_ERROR" +RUNBOOK_FILE_ERROR = "RUNBOOK_FILE_ERROR" +RUNBOOK_VISUALIZE_ERROR = "RUNBOOK_VISUALIZE_ERROR" + +# Terminal execution states that stop a ``--watch`` polling loop, compared +# case-insensitively. Sourced from the two authoritative service enums: +# * RunbookExecutionStatus (execution ARM resource properties.status) --> +# Completed / Failed / Cancelled. +# * ExecutionState (status.json node state) --> adds the step-level finals +# Succeeded / PartiallySucceeded / Skipped. +# Confirmed against the live API. +_EXECUTION_TERMINAL_MEMBERS = ( + RunbookExecutionStatus.COMPLETED, + RunbookExecutionStatus.FAILED, + RunbookExecutionStatus.CANCELLED, + ExecutionState.COMPLETED, + ExecutionState.SUCCEEDED, + ExecutionState.PARTIALLY_SUCCEEDED, + ExecutionState.FAILED, + ExecutionState.CANCELLED, + ExecutionState.SKIPPED, +) +EXECUTION_TERMINAL_STATES = frozenset( + member.value.lower() for member in _EXECUTION_TERMINAL_MEMBERS) + +# Per-entity status values that count as successfully finished when +# summarizing a step's workload progress ("n/m completed"). The status.json +# schema reports entity success as ``Succeeded``; ``Completed`` is kept for +# backward compatibility with the earlier ``state`` field. +ENTITY_COMPLETED_STATES = frozenset({ + ExecutionState.SUCCEEDED.value.lower(), + ExecutionState.COMPLETED.value.lower(), +}) diff --git a/src/migrate/azext_migrate/runbook/deps.py b/src/migrate/azext_migrate/runbook/deps.py new file mode 100644 index 00000000000..e033de31414 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/deps.py @@ -0,0 +1,113 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Shared dependency-resolution helpers for runbook definition/execution. + +A runbook step declares its upstream steps through two sibling lists: + +* ``prerequisite`` — hard ordering constraints, and +* ``dependsOn`` — soft/gate dependencies (e.g. approval gates). + +Both are lists of objects shaped like ``{"step": "", "mode": ...}`` +(and, defensively, plain id strings). The dependency graph and the table +projection must consider *both* lists, so the merge logic lives here in one +place rather than being duplicated (and drifting) across modules. +""" + + +def _dep_id(dep): + """Extract the referenced step id from one dependency entry.""" + if isinstance(dep, dict): + return dep.get('step') or dep.get('stepId') + if dep: + return str(dep) + return None + + +def merged_dep_ids(step): + """Return the ordered, de-duplicated upstream step ids for ``step``. + + Merges the ``prerequisite`` and ``dependsOn`` lists (in that order), + dropping blanks and duplicates while preserving first-seen order. + """ + step = step or {} + ids = [] + seen = set() + for key in ('prerequisite', 'dependsOn'): + for dep in step.get(key) or []: + dep_id = _dep_id(dep) + if dep_id and dep_id not in seen: + seen.add(dep_id) + ids.append(dep_id) + return ids + + +def _step_id(step): + return step.get('stepId') or step.get('id') or step.get('name') + + +def _step_name(step): + return (step.get('displayName') or step.get('name') + or step.get('stepName') or _step_id(step) or 'step') + + +def _iter_ws_steps(document): + """Yield ``(workstream_name, step)`` for every step in a definition or + execution document. + + Unwraps an execution ``properties`` envelope and covers both the grouped + ``workstreams[].steps[]`` and flat ``steps[]`` shapes. Steps that live + outside any workstream yield a ``None`` workstream name. + """ + root = document or {} + if isinstance(root, dict) and isinstance(root.get('properties'), dict): + merged = dict(root) + merged.update(root['properties']) + root = merged + if not isinstance(root, dict): + return + workstreams = root.get('workstreams') + if isinstance(workstreams, list) and workstreams: + for workstream in workstreams: + if not isinstance(workstream, dict): + continue + name = (workstream.get('displayName') or workstream.get('name') + or workstream.get('id')) + for step in workstream.get('steps') or []: + if isinstance(step, dict): + yield name, step + return + for step in root.get('steps') or []: + if isinstance(step, dict): + yield None, step + + +def build_dep_labels(document): + """Map each step id to a readable ``"Workstream:Step name"`` label. + + A dependency is stored as a step id, which is opaque to a reader. This + builds a single lookup (one per document) so every surface -- the + ``--output table`` views and the visualize grid -- can render dependency + references as ``workstream:step name`` instead of the raw id. Steps + outside any workstream map to just their display name; ids not present + here (e.g. dangling references) fall back to the raw id via + :func:`label_deps`. + """ + labels = {} + for ws_name, step in _iter_ws_steps(document): + step_id = _step_id(step) + if not step_id: + continue + name = _step_name(step) + labels[step_id] = '%s:%s' % (ws_name, name) if ws_name else name + return labels + + +def label_deps(step, labels): + """Return ``step``'s merged dependency ids mapped through ``labels``. + + Ids missing from ``labels`` (dangling references, or a single-step + projection with no sibling context) fall back to the raw id. + """ + return [labels.get(dep_id, dep_id) for dep_id in merged_dep_ids(step)] diff --git a/src/migrate/azext_migrate/runbook/models.py b/src/migrate/azext_migrate/runbook/models.py new file mode 100644 index 00000000000..06c62121ab8 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/models.py @@ -0,0 +1,227 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Request/response body builders for the runbook feature.""" + +from enum import Enum + +from azext_migrate.runbook.constants import ( + SCOPE_TYPE_WAVE, + WAVE_ID_TEMPLATE, + STEP_TYPE_APPROVAL, + STEP_REF_BY_TYPE, + STEP_DEPENDENCY_MODE_STEP, + STEP_ACTION_APPROVE, + STEP_ACTION_COMPLETE, + ARTIFACT_DOWNLOAD_MODE_FILE, +) + + +class ExecutionAction(str, Enum): + """Service ``RunbookExecutionAction`` enum (string values). + + ``PerformAction`` / ``ProvideApproval`` / ``UpdateStepStatus`` all send + the string member value. + """ + + START = "Start" + PAUSE = "Pause" + RESUME = "Resume" + CANCEL = "Cancel" + RETRY = "Retry" + COMPLETE = "Complete" + FAIL = "Fail" + SKIP = "Skip" + APPROVE = "Approve" + REJECT = "Reject" + + +def wave_id(project_id, wave_name): + """Build a wave ARM id relative to the migrate project id.""" + return WAVE_ID_TEMPLATE.format( + project_id=project_id, wave_name=wave_name) + + +def build_generate_body(wave_resource_id): + """Build the CreateRunbook (PUT) body scoped to a wave. + + The scope is a polymorphic type on the service; its discriminator + property (``scopeType``) is matched case-sensitively, so the payload + must use camelCase keys (``scopeType``/``waveId``) to bind to the + concrete wave scope. The GET read model echoes the same camelCase + values. + """ + return { + "properties": { + "scope": { + "scopeType": SCOPE_TYPE_WAVE, + "waveId": wave_resource_id, + } + } + } + + +def build_update_body(description=None): + """Build the runbook update (PATCH) body for editable metadata.""" + properties = {} + if description is not None: + properties["description"] = description + return {"properties": properties} + + +def _depends_on_refs(depends_on): + """Map CLI ``--depends-on`` entries to write-model dependency objects. + + The AddStep/UpdateStep write model expects a list of + ``RunbookStepDependency`` objects ``{"mode": , "stepId": }``. + ``mode`` is the ``RunbookStepDependencyMode`` string; a plain + ``--depends-on `` maps to a Step gate. Entries that are already + dicts (e.g. carrying an ``entityMap``) are passed through unchanged. + """ + refs = [] + for entry in depends_on or []: + if isinstance(entry, dict): + refs.append(entry) + else: + refs.append( + {"mode": STEP_DEPENDENCY_MODE_STEP, "stepId": entry}) + return refs + + +def build_add_step_body(step_type, step_name, workstream_id, + step_description=None, depends_on=None, + migration_entity_ids=None): + """Build the AddStep POST body for a single definition step. + + Mirrors the service ``RunbookStepAddRequest``. ``step_type`` selects + the ``stepRef`` binding (Approval -> ``common.approval``, Manual -> + ``common.manual``); the step is added to ``workstream_id``. + ``migrationEntityIds`` is only carried by the Approval step variant. + """ + body = { + "workstreamId": workstream_id, + "displayName": step_name, + "description": step_description or "", + "stepRef": STEP_REF_BY_TYPE.get(step_type, step_type), + "dependsOn": _depends_on_refs(depends_on), + } + if step_type == STEP_TYPE_APPROVAL: + body["migrationEntityIds"] = migration_entity_ids or [] + return body + + +def build_update_step_body(step_id, step_name=None, step_description=None, + depends_on=None): + """Build the UpdateStep POST body; only provided fields are sent.""" + body = {"stepId": step_id} + if step_name is not None: + body["displayName"] = step_name + if step_description is not None: + body["description"] = step_description + if depends_on is not None: + body["dependsOn"] = _depends_on_refs(depends_on) + return body + + +def build_delete_step_body(step_id): + """Build the DeleteStep POST body.""" + return {"stepId": step_id} + + +def build_split_workstream_body(source_workstream_id, new_workstream_name, + step_ids): + """Build the SplitWorkstream POST body. + + ``step_ids`` are the steps moved from the source workstream into the + new one. Mirrors service ``RunbookWorkstreamSplitRequest`` + (sourceWorkstreamId / stepIds / newWorkstreamName). + """ + return { + "sourceWorkstreamId": source_workstream_id, + "stepIds": step_ids or [], + "newWorkstreamName": new_workstream_name, + } + + +def build_merge_workstreams_body(source_workstream_ids, + new_workstream_name): + """Build the MergeWorkstreams POST body. + + ``source_workstream_ids`` serializes as the ``workstreamIds`` array and + ``new_workstream_name`` as ``newWorkstreamName``; both are required by + the service ``RunbookWorkstreamsMergeRequest``. + """ + return { + "workstreamIds": source_workstream_ids or [], + "newWorkstreamName": new_workstream_name, + } + + +def build_start_execution_body(): + """Build the StartRunbookExecution (PUT) body.""" + return {"properties": {}} + + +def build_artifact_download_url_body( + path="runbook.json", mode=ARTIFACT_DOWNLOAD_MODE_FILE, + include_metadata=True): + """Build the Artifact Service GenerateDownloadUrl request body. + + Omitting ``version``/``versionId`` requests the latest committed + version. File mode targets a single blob within the artifact by + ``path``. + """ + body = {"mode": mode, "path": path} + if include_metadata is not None: + body["includeMetadata"] = include_metadata + return body + + +def build_perform_action_body(action, target_id=None, entity_ids=None): + """Build the PerformAction POST body (string action value).""" + return { + "action": action.value if isinstance(action, ExecutionAction) + else action, + "targetId": target_id or "", + "migrationEntityIds": entity_ids or [], + } + + +def build_retry_step_body(step_id, entity_ids=None): + """Build the PerformAction POST body to retry a failed step. + + Retry reuses ``PerformAction`` with the ``Retry`` action and the step + id as the ``targetId``. + """ + return build_perform_action_body( + ExecutionAction.RETRY, target_id=step_id, entity_ids=entity_ids) + + +def build_approve_step_body(step_id, entity_ids=None): + """Build the ProvideApproval POST body for an approval step. + + ``ProvideApproval`` sends the PascalCase ``"Approve"`` action string. + ``migrationEntityIds`` carries the per-entity approvals for a Partial + step; a Full step (or ``--all-ready``) sends an empty list. + """ + return { + "action": STEP_ACTION_APPROVE, + "targetId": step_id, + "migrationEntityIds": entity_ids or [], + } + + +def build_complete_step_body(step_id, comment, entity_ids=None): + """Build the UpdateStepStatus POST body to complete a manual step. + + ``UpdateStepStatus`` sends the PascalCase ``"Complete"`` action string. + ``comment`` is required by the service to record who/why the step was + completed (captured in ``status.json``). + """ + return { + "action": STEP_ACTION_COMPLETE, + "targetId": step_id, + "migrationEntityIds": entity_ids or [], + "comment": comment, + } diff --git a/src/migrate/azext_migrate/runbook/params.py b/src/migrate/azext_migrate/runbook/params.py new file mode 100644 index 00000000000..ca64195bed9 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/params.py @@ -0,0 +1,315 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Argument declarations for the ``migrate runbook`` command group.""" + +from azure.cli.core.commands.parameters import ( + resource_group_name_type, + get_enum_type, +) +from azext_migrate.runbook.constants import ( + RUNBOOK_STATUS_VALUES, + STEP_TYPE_VALUES, +) +from azext_migrate.runbook.validators import ( + validate_generate, +) + + +def load_runbook_arguments(self, _): + with self.argument_context('migrate runbook') as c: + c.argument('resource_group_name', resource_group_name_type) + c.argument( + 'project_name', + options_list=['--project-name', '-p'], + help='Name of the Azure Migrate project.') + c.argument( + 'runbook_name', + options_list=['--name', '-n', '--runbook-name'], + help='Name of the runbook.') + + with self.argument_context('migrate runbook generate') as c: + c.argument( + 'wave_name', + options_list=['--wave-name'], + validator=validate_generate, + help='Name of the wave to generate the runbook from ' + '(required).') + + with self.argument_context('migrate runbook update') as c: + c.argument( + 'description', + options_list=['--description'], + help='Updated description for the runbook.') + + with self.argument_context('migrate runbook list') as c: + c.argument( + 'wave_name', + options_list=['--wave-name'], + help='Filter runbooks by wave name.') + c.argument( + 'status', + options_list=['--status'], + arg_type=get_enum_type(RUNBOOK_STATUS_VALUES), + help='Filter runbooks by lifecycle status.') + + with self.argument_context('migrate runbook definition show') as c: + c.argument( + 'workstream_id', + options_list=['--workstream-id'], + help='Limit the output to a single workstream.') + c.argument( + 'step_id', + options_list=['--step-id'], + help='Limit the output to a single step.') + + with self.argument_context('migrate runbook definition download') as c: + c.argument( + 'destination', + options_list=['--destination'], + help='Directory to save the downloaded files to ' + '(default: current directory).') + + with self.argument_context( + 'migrate runbook definition step add') as c: + c.argument( + 'step_type', options_list=['--step-type'], required=True, + arg_type=get_enum_type(STEP_TYPE_VALUES), + help='Kind of step to add.') + c.argument( + 'step_name', options_list=['--step-name'], required=True, + help='Display name for the step.') + c.argument( + 'workstream_id', options_list=['--workstream-id'], + required=True, + help='Id of the workstream to add the step to.') + c.argument( + 'step_description', options_list=['--step-description'], + help='Optional description for the step.') + c.argument( + 'depends_on', options_list=['--depends-on'], nargs='*', + help='Space-separated step ids this step depends on.') + c.argument( + 'migration_entity_ids', + options_list=['--migration-entity-ids'], nargs='*', + help='Space-separated migration entity ids to associate ' + 'with the step.') + + with self.argument_context( + 'migrate runbook definition step update') as c: + c.argument( + 'step_id', options_list=['--step-id'], required=True, + help='Id of the step to update.') + c.argument( + 'step_name', options_list=['--step-name'], + help='Updated display name for the step.') + c.argument( + 'step_description', options_list=['--step-description'], + help='Updated description for the step.') + c.argument( + 'depends_on', options_list=['--depends-on'], nargs='*', + help='Space-separated step ids this step depends on.') + + with self.argument_context( + 'migrate runbook definition step remove') as c: + c.argument( + 'step_id', options_list=['--step-id'], required=True, + help='Id of the step to remove.') + + with self.argument_context( + 'migrate runbook definition workstream split') as c: + c.argument( + 'source_workstream_id', + options_list=['--source-workstream-id'], required=True, + help='Id of the workstream to split.') + c.argument( + 'new_workstream_name', + options_list=['--new-workstream-name'], required=True, + help='Display name for the new workstream.') + c.argument( + 'step_ids', options_list=['--step-ids'], + nargs='+', required=True, + help='Space-separated step ids to move into the new ' + 'workstream.') + + with self.argument_context( + 'migrate runbook definition workstream merge') as c: + c.argument( + 'source_workstream_ids', + options_list=['--source-workstream-ids'], nargs='+', + required=True, + help='Space-separated ids of the workstreams to merge.') + c.argument( + 'new_workstream_name', + options_list=['--new-workstream-name'], required=True, + help='Display name for the merged workstream.') + + with self.argument_context( + 'migrate runbook execution show') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'step_id', options_list=['--step-id'], + help='Limit the output to a single step.') + c.argument( + 'watch', options_list=['--watch'], action='store_true', + help='Re-render the status table on an interval until the ' + 'execution reaches a terminal state.') + c.argument( + 'interval', options_list=['--interval'], type=int, + help='Refresh interval in seconds for --watch (default: 5).') + + with self.argument_context( + 'migrate runbook execution pause') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + + with self.argument_context( + 'migrate runbook execution resume') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + + with self.argument_context( + 'migrate runbook execution cancel') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + + with self.argument_context('migrate runbook wait') as c: + c.argument( + 'created', options_list=['--created'], action='store_true', + help='Wait until provisioningState reaches Succeeded.') + c.argument( + 'updated', options_list=['--updated'], action='store_true', + help='Wait until provisioningState reaches Succeeded after ' + 'an update.') + c.argument( + 'deleted', options_list=['--deleted'], action='store_true', + help='Wait until the runbook no longer exists.') + c.argument( + 'exists', options_list=['--exists'], action='store_true', + help='Wait until the runbook exists.') + c.argument( + 'custom', options_list=['--custom'], + help="Wait until a JMESPath condition is met, e.g. " + "\"properties.state=='ExecutionSucceeded'\".") + c.argument( + 'interval', options_list=['--interval'], type=int, + help='Polling interval in seconds (default: 30).') + c.argument( + 'timeout', options_list=['--timeout'], type=int, + help='Maximum wait time in seconds (default: 3600).') + + with self.argument_context( + 'migrate runbook definition visualize') as c: + c.argument( + 'file', options_list=['--file'], + help='Output path for the generated HTML file ' + '(default: current directory).') + c.argument( + 'open_file', options_list=['--open'], action='store_true', + help='Open the generated HTML file in the default browser.') + c.argument( + 'from_file', options_list=['--from-file'], + help='Render from a local runbook definition JSON file ' + 'instead of fetching from the service.') + c.argument( + 'parameters_file', options_list=['--parameters-file'], + help='Optional local parameters JSON file to merge when ' + 'rendering from --from-file.') + + with self.argument_context( + 'migrate runbook execution visualize') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], + help='Id of the runbook execution to visualize.') + c.argument( + 'file', options_list=['--file'], + help='Output path for the generated HTML file ' + '(default: current directory).') + c.argument( + 'open_file', options_list=['--open'], action='store_true', + help='Open the generated HTML file in the default browser.') + c.argument( + 'from_file', options_list=['--from-file'], + help='Render from a local execution status JSON file ' + 'instead of fetching from the service.') + c.argument( + 'watch', options_list=['--watch'], action='store_true', + help='Regenerate the HTML snapshot on an interval until the ' + 'execution reaches a terminal state.') + c.argument( + 'interval', options_list=['--interval'], type=int, + help='Refresh interval in seconds for --watch (default: 5).') + + with self.argument_context( + 'migrate runbook execution step retry') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'step_id', options_list=['--step-id'], required=True, + help='Id of the step to retry.') + + with self.argument_context( + 'migrate runbook execution step approve') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'step_id', options_list=['--step-id'], required=True, + help='Id of the approval step to approve.') + c.argument( + 'entities', options_list=['--entities'], nargs='*', + help='Space-separated entity ids to approve (partial approval ' + 'steps only).') + c.argument( + 'all_ready', options_list=['--all-ready'], action='store_true', + help='Approve every currently ready entity (partial approval ' + 'steps only).') + + with self.argument_context( + 'migrate runbook execution step complete') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'step_id', options_list=['--step-id'], required=True, + help='Id of the manual step to complete.') + c.argument( + 'comment', options_list=['--comment'], required=True, + help='Comment recording who/why the step was completed.') + + with self.argument_context('migrate runbook parameter download') as c: + c.argument( + 'file', options_list=['--file'], + help='Output path for the parameters file ' + '(default: current directory).') + + with self.argument_context('migrate runbook parameter upload') as c: + c.argument( + 'file', options_list=['--file'], required=True, + help='Path to the parameters JSON file to upload.') + + with self.argument_context( + 'migrate runbook execution parameter download') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'file', options_list=['--file'], + help='Output path for the input-parameters file ' + '(default: current directory).') + + with self.argument_context( + 'migrate runbook execution parameter upload') as c: + c.argument( + 'execution_id', options_list=['--execution-id'], required=True, + help='Id of the runbook execution.') + c.argument( + 'file', options_list=['--file'], required=True, + help='Path to the input-parameters JSON file to upload.') diff --git a/src/migrate/azext_migrate/runbook/transformers.py b/src/migrate/azext_migrate/runbook/transformers.py new file mode 100644 index 00000000000..89014d94ad2 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/transformers.py @@ -0,0 +1,169 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Table (``--output table``) transformers for runbook commands.""" + +from collections import OrderedDict + +from azext_migrate.runbook import deps as dep_utils +from azext_migrate.runbook.constants import ENTITY_COMPLETED_STATES + +# Placeholder shown for the "Applications" column, which the runbook +# definition does not currently model (steps carry entities, not apps). +_APPLICATIONS_PLACEHOLDER = '-' + + +def runbook_table(result): + """Project a runbook (or a list of runbooks) into table rows.""" + if isinstance(result, list): + return [_runbook_row(item) for item in result] + return _runbook_row(result) + + +def _runbook_row(item): + item = item or {} + props = item.get('properties', {}) if isinstance(item, dict) else {} + return OrderedDict([ + ('Name', item.get('name')), + ('State', props.get('state')), + ('ProvisioningState', props.get('provisioningState')), + ]) + + +def definition_table(result): + """Project a runbook definition into one step row per definition step.""" + labels = dep_utils.build_dep_labels(result) + rows = [] + if isinstance(result, dict) and result.get('workstreams') is not None: + for workstream in result.get('workstreams') or []: + rows.extend(_step_rows(workstream, labels)) + elif isinstance(result, dict) and result.get('steps') is not None: + rows.extend(_step_rows(result, labels)) + elif isinstance(result, dict) and _looks_like_step(result): + rows.append(_step_row(result, labels=labels)) + return rows + + +def _looks_like_step(step): + """True when a dict carries step-identifying keys (single-step show).""" + return any(step.get(key) for key in + ('stepId', 'id', 'displayName', 'stepName')) + + +def _step_rows(workstream, labels): + workstream = workstream or {} + workstream_id = workstream.get('id') + steps = workstream.get('steps', []) or [] + # Keep an empty workstream visible with a single placeholder row. + if not steps: + return [_empty_workstream_row(workstream_id)] + return [_step_row(step, workstream_id, labels) for step in steps] + + +def _empty_workstream_row(workstream_id): + return OrderedDict([ + ('Workstream Id', workstream_id), + ('Step Id', ''), + ('Step Name', '(no steps)'), + ('Depends On', ''), + ('Configuration Status', ''), + ('Workloads', ''), + ('Applications', _APPLICATIONS_PLACEHOLDER), + ]) + + +def _step_row(step, workstream_id=None, labels=None): + step = step or {} + return OrderedDict([ + ('Workstream Id', workstream_id), + ('Step Id', step.get('stepId') or step.get('id')), + ('Step Name', step.get('displayName') or step.get('stepName')), + ('Depends On', '\n'.join(dep_utils.label_deps(step, labels or {}))), + ('Configuration Status', step.get('configurationStatus')), + ('Workloads', len(step.get('entities') or [])), + ('Applications', _APPLICATIONS_PLACEHOLDER), + ]) + + +def executions_table(result): + """Project a list of runbook executions into one row per execution.""" + items = result if isinstance(result, list) else [result] + return [_execution_row(item) for item in items if item] + + +def _execution_row(item): + item = item or {} + props = item.get('properties', {}) if isinstance(item, dict) else {} + return OrderedDict([ + ('Name', item.get('name')), + ('Status', props.get('status') or props.get('state')), + ('ProvisioningState', props.get('provisioningState')), + ('StartTime', props.get('startTime') or props.get('jobStartTime')), + ('EndTime', props.get('endTime')), + ]) + + +def execution_table(result): + """Project a runbook execution status into one row per step.""" + rows = [] + status = _execution_status(result) + labels = dep_utils.build_dep_labels(status) + if isinstance(status, dict) and status.get('workstreams') is not None: + for workstream in status.get('workstreams') or []: + rows.extend(_exec_step_rows(workstream, labels)) + elif isinstance(status, dict) and status.get('steps') is not None: + rows.extend(_exec_step_rows(status, labels)) + elif isinstance(status, dict) and status: + rows.append(_exec_step_row(status, labels=labels)) + return rows + + +def _execution_status(result): + """Unwrap an execution resource to its status document.""" + if (isinstance(result, dict) + and result.get('properties') is not None + and result.get('workstreams') is None + and result.get('steps') is None): + return result.get('properties') or {} + return result or {} + + +def _exec_step_rows(workstream, labels): + workstream = workstream or {} + workstream_id = workstream.get('id') + return [_exec_step_row(step, workstream_id, labels) + for step in workstream.get('steps', []) or []] + + +def _exec_step_row(step, workstream_id=None, labels=None): + step = step or {} + return OrderedDict([ + ('Workstream Id', workstream_id), + ('Step Id', step.get('id') or step.get('stepId')), + ('Step Name', step.get('displayName') or step.get('stepName')), + ('Step Status', + step.get('status') or step.get('stepStatus') or step.get('state')), + ('Depends On', '\n'.join(dep_utils.label_deps(step, labels or {}))), + ('Workload Progress', _workload_progress(step)), + ]) + + +def _workload_progress(step): + """Summarize per-entity progress from ``entityExecutions``. + + Falls back to an explicit ``workloadProgress`` scalar when present. + """ + progress = step.get('workloadProgress') + if progress is not None: + return progress + entities = step.get('entityExecutions') + if not entities: + return None + total = len(entities) + completed = 0 + for entity in entities: + value = (entity or {}).get('status') or (entity or {}).get('state') + if str(value or '').lower() in ENTITY_COMPLETED_STATES: + completed += 1 + return '%d/%d completed' % (completed, total) diff --git a/src/migrate/azext_migrate/runbook/validators.py b/src/migrate/azext_migrate/runbook/validators.py new file mode 100644 index 00000000000..e6436547f7e --- /dev/null +++ b/src/migrate/azext_migrate/runbook/validators.py @@ -0,0 +1,70 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Command validators for the runbook feature.""" + +from azure.cli.core.azclierror import ( + RequiredArgumentMissingError, + InvalidArgumentValueError, +) + + +def validate_generate(namespace): + """Ensure ``runbook generate`` has a source wave.""" + if not getattr(namespace, 'wave_name', None): + raise RequiredArgumentMissingError( + "--wave-name is required to generate a runbook.") + + +def validate_step_approve(namespace): + """Enforce the ``execution step approve`` parameter-set rules. + + ``--entities`` (per-entity approval) and ``--all-ready`` (approve every + ready entity) are mutually exclusive. Both apply only to Partial + approval steps; the service rejects them for Full/non-approval steps. + """ + entities = getattr(namespace, 'entities', None) + all_ready = getattr(namespace, 'all_ready', None) + if entities and all_ready: + raise InvalidArgumentValueError( + "--entities and --all-ready cannot be used together.") + + +def validate_step_complete(namespace): + """Ensure ``execution step complete`` has the required comment.""" + if not getattr(namespace, 'comment', None): + raise RequiredArgumentMissingError( + "--comment is required to complete a manual step.") + + +def _require_runbook_identity(namespace): + """Raise unless the full runbook identity (rg/project/name) is set.""" + missing = [] + if not getattr(namespace, 'resource_group_name', None): + missing.append('--resource-group/-g') + if not getattr(namespace, 'project_name', None): + missing.append('--project-name/-p') + if not getattr(namespace, 'runbook_name', None): + missing.append('--name/-n') + if missing: + raise RequiredArgumentMissingError( + "%s required unless --from-file is used." + % ', '.join(missing)) + + +def validate_definition_visualize(namespace): + """Require the runbook identity unless rendering a local ``--from-file``.""" + if getattr(namespace, 'from_file', None): + return + _require_runbook_identity(namespace) + + +def validate_execution_visualize(namespace): + """Require execution identity unless rendering a local ``--from-file``.""" + if getattr(namespace, 'from_file', None): + return + _require_runbook_identity(namespace) + if not getattr(namespace, 'execution_id', None): + raise RequiredArgumentMissingError( + "--execution-id is required unless --from-file is used.") diff --git a/src/migrate/azext_migrate/runbook/visualize/__init__.py b/src/migrate/azext_migrate/runbook/visualize/__init__.py new file mode 100644 index 00000000000..b3f4ff5bfad --- /dev/null +++ b/src/migrate/azext_migrate/runbook/visualize/__init__.py @@ -0,0 +1,11 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Client-side (non-REST) rendering of runbook definitions/executions. + +The visualize package turns a runbook definition or execution JSON +document into a single, self-contained, offline HTML dependency graph. +It is stdlib-only and performs no I/O beyond the caller writing the +returned HTML to disk. +""" diff --git a/src/migrate/azext_migrate/runbook/visualize/graph.py b/src/migrate/azext_migrate/runbook/visualize/graph.py new file mode 100644 index 00000000000..87fa8d991b8 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/visualize/graph.py @@ -0,0 +1,197 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Transform a runbook definition/execution JSON document into a DAG model. + +This module is **data only** — it contains no HTML and performs no I/O. It +parses the runbook JSON into an in-memory directed acyclic graph (nodes + +edges), validates it (cycle detection, dangling ``dependsOn`` handling) and +computes a stable topological layering so the renderer can lay the graph out +deterministically. Keeping the graph shape free of HTML makes it trivial to +unit-test. +""" + +from knack.log import get_logger +from azure.cli.core.azclierror import InvalidArgumentValueError + +from azext_migrate.runbook import deps as dep_utils + +logger = get_logger(__name__) + +NODE_TYPE_STEP = 'step' + + +class Node: + """A single step in the runbook dependency graph.""" + + # pylint: disable=too-few-public-methods,too-many-arguments,too-many-instance-attributes + def __init__(self, node_id, name, node_type=NODE_TYPE_STEP, + group=None, status=None, layer=0, ref=None, + group_id=None): + self.id = node_id + self.name = name + self.type = node_type + self.group = group + self.group_id = group_id + self.status = status + self.layer = layer + self.ref = ref + + def __repr__(self): + return ( + "Node(id=%r, name=%r, group=%r, status=%r, layer=%r)" + % (self.id, self.name, self.group, self.status, self.layer)) + + +class Edge: + """A ``dependsOn`` dependency: ``source`` must finish before ``target``.""" + + # pylint: disable=too-few-public-methods + def __init__(self, source, target): + self.source = source + self.target = target + + def __repr__(self): + return "Edge(source=%r, target=%r)" % (self.source, self.target) + + +class Graph: + """A layered DAG of runbook steps.""" + + # pylint: disable=too-few-public-methods + def __init__(self, title, nodes, edges, group_order=None): + self.title = title + self.nodes = nodes + self.edges = edges + # Workstream groups in source-document order (a list of + # ``(name, group_id)`` pairs). The renderer orders swimlanes by this + # so the diagram matches the grid; ``nodes`` is separately sorted by + # dependency layer for column layout. + self.group_order = group_order or [] + + @property + def layer_count(self): + return (max((n.layer for n in self.nodes), default=-1) + 1) + + +def _step_id(step): + return step.get('id') or step.get('stepId') or step.get('name') + + +def _step_name(step): + return (step.get('displayName') or step.get('name') + or step.get('stepName') or _step_id(step) or 'step') + + +def _step_status(step): + status = step.get('status') or step.get('state') + if isinstance(status, dict): + status = status.get('state') or status.get('status') + # Definition steps carry no execution state; fall back to the computed + # configuration status so the definition DAG colours by readiness. + return status or step.get('configurationStatus') + + +def _iter_steps(document): + """Yield ``(step, workstream_name, workstream_id)`` triples. + + Handles both the ``workstreams[].steps[]`` shape and a flat + ``steps[]`` shape, and unwraps an execution ``properties`` envelope. + """ + root = document + if isinstance(root, dict) and isinstance(root.get('properties'), dict): + merged = dict(root) + merged.update(root['properties']) + root = merged + if not isinstance(root, dict): + return + workstreams = root.get('workstreams') or [] + for workstream in workstreams: + if not isinstance(workstream, dict): + continue + ws_id = workstream.get('id') + ws_name = (workstream.get('displayName') + or workstream.get('name') or ws_id) + for step in workstream.get('steps', []) or []: + if isinstance(step, dict): + yield step, ws_name, ws_id + for step in root.get('steps', []) or []: + if isinstance(step, dict): + yield step, None, None + + +def _build_graph(document, title): + nodes = [] + node_by_id = {} + group_order = [] + seen_groups = set() + for step, ws_name, ws_id in _iter_steps(document): + node_id = _step_id(step) + if not node_id or node_id in node_by_id: + continue + node = Node( + node_id, _step_name(step), group=ws_name, group_id=ws_id, + status=_step_status(step), ref=step.get('stepRef')) + nodes.append(node) + node_by_id[node_id] = node + group_key = ws_name or 'Ungrouped' + if group_key not in seen_groups: + seen_groups.add(group_key) + group_order.append((group_key, ws_id)) + + edges = [] + dependencies = {node.id: [] for node in nodes} + for step, _, _ in _iter_steps(document): + node_id = _step_id(step) + if node_id not in node_by_id: + continue + for dep_id in dep_utils.merged_dep_ids(step): + if dep_id not in node_by_id: + logger.warning( + "Step '%s' depends on unknown step '%s'; ignoring the " + "dangling dependency.", node_id, dep_id) + continue + edges.append(Edge(dep_id, node_id)) + dependencies[node_id].append(dep_id) + + _assign_layers(nodes, node_by_id, dependencies) + return Graph(title, nodes, edges, group_order=group_order) + + +def _assign_layers(nodes, node_by_id, dependencies): + """Compute a topological layering (Kahn's algorithm), raising on cycles.""" + remaining = dict(dependencies) + resolved = set() + order = list(node_by_id) + progressed = True + while remaining and progressed: + progressed = False + ready = [ + node_id for node_id in order + if node_id in remaining + and all(dep in resolved for dep in remaining[node_id])] + for node_id in ready: + deps = dependencies[node_id] + layer = 0 + for dep in deps: + layer = max(layer, node_by_id[dep].layer + 1) + node_by_id[node_id].layer = layer + resolved.add(node_id) + del remaining[node_id] + progressed = True + if remaining: + raise InvalidArgumentValueError( + 'The runbook dependency graph contains a cycle involving steps: ' + + ', '.join(sorted(remaining))) + nodes.sort(key=lambda n: (n.layer, n.id)) + + +def build_definition_graph(definition, title='Runbook definition'): + """Build the DAG for a runbook definition document.""" + return _build_graph(definition or {}, title) + + +def build_execution_graph(execution, title='Runbook execution'): + """Build the DAG for a runbook execution document (status-annotated).""" + return _build_graph(execution or {}, title) diff --git a/src/migrate/azext_migrate/runbook/visualize/renderer.py b/src/migrate/azext_migrate/runbook/visualize/renderer.py new file mode 100644 index 00000000000..6bef1a6f0c0 --- /dev/null +++ b/src/migrate/azext_migrate/runbook/visualize/renderer.py @@ -0,0 +1,538 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Render a runbook view into a self-contained, offline HTML file. + +This is the security-critical stage: **every** user-controlled value (step +names, workstream names, statuses, dependency labels) is passed through +:func:`html.escape` before it is substituted into the markup, which is the +mandatory XSS guard. The emitted HTML embeds all styles/scripts inline and +references no external/CDN resources, so it is fully offline and makes no +outbound network calls. + +The document shows two views of the same runbook: + +* a **grid** (portal-style, grouped by workstream) — the default view, and +* a **diagram** (SVG dependency DAG) — an optional view toggled in-page. + +The tiny inline toggle script runs entirely locally (no network), preserving +the offline/air-gapped guarantee. +""" + +import datetime +import html +import os +import re +import string + +from azext_migrate.runbook.visualize import viewmodel + +_TEMPLATE_PATH = os.path.join( + os.path.dirname(__file__), 'templates', 'runbook.html.tmpl') + +# SVG layout constants (deterministic, so output is stable for tests). +_MARGIN = 24 +_NODE_W = 220 +_NODE_H = 56 +_COL_GAP = 80 +_ROW_GAP = 24 +# Workstream swimlane band metrics. +_BAND_LABEL_H = 26 +_BAND_GAP = 18 + +# Execution ``state`` vocabulary, in legend display order. +_EXECUTION_LEGEND = ( + ('Completed', '#6bb700'), + ('InProgress', '#2899f5'), + ('Blocked', '#ffaa44'), + ('Failed', '#d13438'), + ('NotStarted', '#8a8886'), +) + +# Definition configuration-status vocabulary, in legend display order. +_DEFINITION_LEGEND = ( + ('Configured', '#6bb700'), + ('Partial', '#ffaa44'), + ('NotConfigured', '#d13438'), + ('Unknown', '#8a8886'), +) + + +def _esc(value): + """HTML-escape a possibly-``None`` user value (the XSS guard).""" + if value is None: + return '' + return html.escape(str(value), quote=True) + + +def _id_badge(value, title='id'): + """Render an id as a small, greyish, monospace inline badge (HTML). + + Ids are surfaced so users can copy them into the CLI commands that + address workstreams/steps by id (e.g. ``workstream split``/``merge``). + Returns escaped markup, so callers must NOT re-escape it. + """ + if not value: + return '' + return '%s' % ( + _esc(title), _esc(value)) + + +def _id_tspan(value): + """Render an id as a small, muted ```` inside an SVG text run.""" + if not value: + return '' + return ' %s' % _esc(value) + + +def _status_class(status): + """Map a status to a CSS-safe class suffix (first word, alnum only).""" + if not status: + return '' + token = str(status).strip().split(' ', 1)[0] + token = re.sub(r'[^A-Za-z0-9_-]', '', token) + return ' status-%s' % token if token else '' + + +def _workstream_order(graph): + """Group nodes into workstream swimlanes in source-document order. + + Bands follow the runbook's workstream order (the same order the grid + uses) so the diagram is not reversed relative to the grid. Within a + band, nodes keep their layer-sorted order for column layout. + """ + by_ws = {} + for node in graph.nodes: + by_ws.setdefault(node.group or 'Ungrouped', []).append(node) + ordered = [] + for name, ws_id in graph.group_order: + nodes = by_ws.pop(name, None) + if nodes: + ordered.append((name, ws_id, nodes)) + # Any group not present in the recorded order (defensive) keeps a stable + # first-appearance fallback. + for name, nodes in by_ws.items(): + ordered.append((name, nodes[0].group_id, nodes)) + return ordered + + +def _layout(graph): + """Lay steps out in workstream swimlanes with dependency columns. + + The horizontal axis is the (global) dependency layer so ``dependsOn`` + edges always flow left-to-right; the vertical axis groups steps into + per-workstream bands. Returns ``(positions, bands, width, height)`` where + ``bands`` is a list of ``(name, top, height, count)`` tuples. + """ + positions = {} + bands = [] + width = _MARGIN * 2 + graph.layer_count * _NODE_W \ + + max(graph.layer_count - 1, 0) * _COL_GAP + y = _MARGIN + for name, ws_id, nodes in _workstream_order(graph): + band_top = y + content_top = band_top + _BAND_LABEL_H + rows_per_layer = {} + max_rows = 0 + for node in nodes: + row = rows_per_layer.get(node.layer, 0) + rows_per_layer[node.layer] = row + 1 + node_x = _MARGIN + node.layer * (_NODE_W + _COL_GAP) + node_y = content_top + row * (_NODE_H + _ROW_GAP) + positions[node.id] = (node_x, node_y) + max_rows = max(max_rows, row + 1) + band_height = _BAND_LABEL_H + max_rows * (_NODE_H + _ROW_GAP) + bands.append((name, ws_id, band_top, band_height, len(nodes))) + y = band_top + band_height + _BAND_GAP + height = y + _MARGIN - _BAND_GAP + return positions, bands, width, height + + +def _svg(graph): + if not graph.nodes: + return '

This runbook has no steps to display.

' + + positions, bands, width, height = _layout(graph) + parts = [ + '' + % (width, height, width, height)] + + for name, ws_id, top, band_height, count in bands: + parts.append( + '' + '' + 'Workstream: %s%s (%d)' + % (_MARGIN / 2, top, width - _MARGIN, band_height, + _MARGIN / 2 + 12, top + 16, + _esc(name or 'Ungrouped'), _id_tspan(ws_id), count)) + + for edge in graph.edges: + if edge.source not in positions or edge.target not in positions: + continue + sx, sy = positions[edge.source] + tx, ty = positions[edge.target] + x1, y1 = sx + _NODE_W, sy + _NODE_H / 2 + x2, y2 = tx, ty + _NODE_H / 2 + midx = (x1 + x2) / 2 + parts.append( + '' + % (x1, y1, midx, y1, midx, y2, x2, y2)) + + for node in graph.nodes: + x, y = positions[node.id] + status_class = _status_class(node.status) + sub = node.ref or node.status + parts.append('' % status_class) + parts.append( + '' + % (x, y, _NODE_W, _NODE_H)) + parts.append( + '%s' + % (x + 12, y + 24, _esc(node.name))) + if sub: + parts.append( + '%s' + % (x + 12, y + 42, _esc(sub))) + parts.append('') + + parts.append('') + return '\n'.join(parts) + + +def _legend(graph, view): + """Render the status legend appropriate to the view kind.""" + if view is not None and view.kind == viewmodel.KIND_DEFINITION: + statuses = _DEFINITION_LEGEND + else: + statuses = _EXECUTION_LEGEND + if not any(node.status for node in graph.nodes): + return '' + items = ''.join( + '%s' % (color, _esc(label)) + for label, color in statuses) + return '
%s
' % items + + +# --------------------------------------------------------------------------- +# Portal-style grid +# --------------------------------------------------------------------------- + +def _summary_cards(view): + if view is None or not view.summary: + return '' + stats = ''.join( + '
%s' + '%s
' % (_esc(value), _esc(label)) + for label, value in view.summary) + return '
%s
' % stats + + +def _grid_row(kind, index, step): + """Render one step as a clickable grid row (portal-style). + + The status/last-column semantics differ by view kind: a definition row + shows its configuration status and entity count, while an execution row + shows its live step status and workload progress. + """ + ref = ('%s' % _esc(step.step_ref) + if step.step_ref else '') + dep = ', '.join(step.deps) if step.deps else '-' + if kind == viewmodel.KIND_EXECUTION: + status = step.status or 'NotStarted' + count = step.workload_progress or '-' + else: + status = step.status or 'Unknown' + count = step.workloads + return ( + '
' + '
' + '▸' + '%s%s
' + '
%s
' + '
%s
' + '
%s
' + '
' + % (index, _esc(step.name), ref, + _status_class(status), _esc(status), _esc(dep), _esc(count))) + + +def _iter_indexed_steps(view): + """Yield ``(index, workstream_name, step)`` in stable grid order.""" + index = 0 + for workstream in view.workstreams: + for step in workstream.steps: + yield index, workstream.name, step + index += 1 + + +def _portal_grid(view): + """Render the runbook as a portal-style grid: header + grouped rows.""" + if view.kind == viewmodel.KIND_EXECUTION: + status_head, count_head = 'Step status', 'Workload progress' + else: + status_head, count_head = 'Configuration status', 'Entities' + parts = [ + '
' + '
Steps
' + '
%s
' + '
Step dependency
' + '
%s
' % (status_head, count_head)] + index = 0 + for workstream in view.workstreams: + head = 'Workstream: %s%s (%d)' % ( + _esc(workstream.name or 'Ungrouped'), + _id_badge(workstream.id, 'Workstream id'), + len(workstream.steps)) + parts.append('
') + parts.append('%s' % head) + if not workstream.steps: + parts.append('
' + 'No steps in this workstream.
') + for step in workstream.steps: + parts.append(_grid_row(view.kind, index, step)) + index += 1 + parts.append('
') + return '
%s
' % ''.join(parts) + + +def _field(label, value): + text = value if value not in (None, '') else '-' + return ('
%s
' + '
%s
' + % (_esc(label), _esc(text))) + + +def _chip_field(label, values): + if not values: + body = '
-
' + else: + body = '
%s
' % ''.join( + '%s' % _esc(value) for value in values) + return ('
%s
%s
' + % (_esc(label), body)) + + +def _status_field(label, status, default): + """Render a labelled status pill field for the detail pane.""" + return ('
%s
' + '
%s
' + '
' + % (_esc(label), _status_class(status), _esc(status or default))) + + +def _detail_html(workstream_name, step, kind): + """Build the step detail-pane markup (shown in the side drawer).""" + if kind == viewmodel.KIND_EXECUTION: + entities = ['%s (%s)' % (entity.name, entity.status) + if entity.status else entity.name + for entity in step.entities] + body = ( + _field('Step ID', step.id) + + _status_field('Step status', step.status, 'NotStarted') + + _field('Workload progress', step.workload_progress) + + _chip_field('Entities (%d)' % len(entities), entities) + + _chip_field('Depends on', step.deps)) + else: + entities = step.entity_names + body = ( + _field('Step type', step.step_ref) + + _field('Step ID', step.id) + + _status_field('Configuration status', step.status, 'Unknown') + + _chip_field('Entities (%d)' % len(entities), entities) + + _chip_field('Pre-requisites', step.prereqs) + + _chip_field('Depends on', step.dep_details)) + return ( + '
' + '
' + '

%s

' + '
Workstream: %s
' + '
' + '
%s
' + % (_esc(step.name), _esc(workstream_name or 'Ungrouped'), body)) + + +def _grid_details(view): + """Emit hidden per-step detail blocks that the drawer clones on click.""" + if view is None: + return '' + blocks = ''.join( + '' + % (index, _detail_html(ws_name, step, view.kind)) + for index, ws_name, step in _iter_indexed_steps(view)) + return '' % blocks + + +# CLI cmdlet help chips (static; shown above the definition grid). +_HELP_CHIPS = ( + ('▶', 'Start execution', + 'Runs the wave and streams live progress in the execution view.', + 'az migrate runbook execution start --resource-group ' + '--project-name --runbook-name '), + ('＋', 'Add a step', + 'Adds a step to a workstream in the runbook definition.', + 'az migrate runbook definition step add --resource-group ' + '--project-name --runbook-name ' + '--step-type --step-name --workstream-id '), + ('⇉', 'Merge workstreams', + 'Combines two workstreams into a single track.', + 'az migrate runbook definition workstream merge --resource-group ' + '--project-name --runbook-name ' + '--source-workstream-ids --new-workstream-name '), + ('▱', 'Split a workstream', + 'Splits a workstream into parallel tracks.', + 'az migrate runbook definition workstream split --resource-group ' + '--project-name --runbook-name ' + '--source-workstream-id --new-workstream-name ' + '--entities-to-move '), + ('↻', 'Refresh this view', + 'Regenerates the HTML from the latest runbook definition.', + 'az migrate runbook definition visualize --resource-group ' + '--project-name --runbook-name '), +) + + +# CLI cmdlet help chips shown above the execution grid. +_EXEC_HELP_CHIPS = ( + ('❙❙', 'Pause execution', + 'Pauses the in-progress execution so it can be resumed later.', + 'az migrate runbook execution pause --resource-group ' + '--project-name --runbook-name ' + '--execution-id '), + ('▶', 'Resume execution', + 'Resumes a paused execution from where it left off.', + 'az migrate runbook execution resume --resource-group ' + '--project-name --runbook-name ' + '--execution-id '), + ('✕', 'Cancel execution', + 'Cancels an in-progress or paused execution.', + 'az migrate runbook execution cancel --resource-group ' + '--project-name --runbook-name ' + '--execution-id '), + ('↻', 'Refresh this view', + 'Regenerates the HTML from the latest execution status.', + 'az migrate runbook execution visualize --resource-group ' + '--project-name --runbook-name ' + '--execution-id '), +) + + +def _help_bar(view): + """Render the static CLI cmdlet help chips (kind-aware).""" + if view is None: + return '' + chips_src = (_HELP_CHIPS if view.kind == viewmodel.KIND_DEFINITION + else _EXEC_HELP_CHIPS) + chips = ''.join( + '' + % (_esc(title), _esc(desc), _esc(cmd), ico, _esc(title)) + for ico, title, desc, cmd in chips_src) + return ( + '
This is a ' + 'read-only view — actions run from the Azure CLI. Pick one to ' + 'see the command:
%s
' + % chips) + + +def _meta_block(view): + """Render the header metadata strip (any view that carries meta).""" + if view is None or not view.meta: + return '' + fields = '' + for label, value in view.meta: + display = _format_generated(value) if label == 'Generated' else value + fields += ( + '
%s' + '%s
' + % (_esc(label), _esc(display), _esc(display))) + return '
%s
' % fields + + +def _grid(view): + if view is None: + return '' + if not view.workstreams or view.step_count == 0: + return '

This runbook has no steps to display.

' + return _portal_grid(view) + + +# --------------------------------------------------------------------------- +# Document assembly +# --------------------------------------------------------------------------- + +def render(graph, view=None, refresh_interval=None): + """Return a complete, self-contained HTML document for the runbook. + + ``graph`` drives the SVG dependency diagram; the optional ``view`` + (:class:`~.viewmodel.RunbookView`) drives the default portal-style grid + and enables the in-page grid/diagram toggle. When ``refresh_interval`` is + a positive number of seconds, an offline ```` + tag is embedded so a browser viewing the file auto-reloads it from disk on + that cadence (used by ``--watch`` so the user never has to refresh + manually). Reloading a local file makes no network call, preserving the + offline/air-gapped guarantee. + """ + with open(_TEMPLATE_PATH, encoding='utf-8') as handle: + template = string.Template(handle.read()) + title = graph.title if view is None else view.title + step_count = len(graph.nodes) if view is None else view.step_count + summary = '%d step%s' % (step_count, '' if step_count == 1 else 's') + generated = _format_generated( + None if view is None else view.generated) + grid_html = _grid(view) + return template.substitute( + title=_esc(title), + summary=_esc(summary), + generated=_esc(generated), + meta=_meta_block(view), + help=_help_bar(view), + details=_grid_details(view), + legend=_legend(graph, view), + refresh=_refresh_meta(refresh_interval), + summary_cards=_summary_cards(view), + toggle=_toggle(grid_html), + grid=grid_html, + grid_hidden='' if grid_html else ' hidden', + diagram_hidden=' hidden' if grid_html else '', + svg=_svg(graph)) + + +def _format_generated(declared): + """Format the generation timestamp for the header. + + Prefer the document's own ``generatedAt`` (ISO 8601) when present, + normalising it to ``YYYY-MM-DD HH:MM UTC``; otherwise fall back to the + current render time. + """ + if declared: + text = str(declared).replace('T', ' ') + text = text.split('.', 1)[0].rstrip('Z').strip() + return '%s UTC' % text if text else text + return datetime.datetime.now(datetime.timezone.utc).strftime( + '%Y-%m-%d %H:%M UTC') + + +def _toggle(grid_html): + """Render the grid/diagram tab buttons (only when a grid exists).""" + if not grid_html: + return '' + return ( + '
' + '' + '
') + + +def _refresh_meta(refresh_interval): + """Build an offline auto-reload ```` tag, or '' when disabled.""" + try: + seconds = int(refresh_interval) + except (TypeError, ValueError): + return '' + if seconds <= 0: + return '' + return '\n' % seconds diff --git a/src/migrate/azext_migrate/runbook/visualize/templates/runbook.html.tmpl b/src/migrate/azext_migrate/runbook/visualize/templates/runbook.html.tmpl new file mode 100644 index 00000000000..8960fb9919b --- /dev/null +++ b/src/migrate/azext_migrate/runbook/visualize/templates/runbook.html.tmpl @@ -0,0 +1,460 @@ + + + + + +$refresh$title + + + +
+ + Azure Migrate Runbook Viewer +
+
+

$title

+
$summary · generated $generated
+
+$meta +$help +$toggle +$summary_cards +$legend +
+$grid +
+
+
+$svg +
+
+$details + +
+ + + + diff --git a/src/migrate/azext_migrate/runbook/visualize/viewmodel.py b/src/migrate/azext_migrate/runbook/visualize/viewmodel.py new file mode 100644 index 00000000000..7c28072296a --- /dev/null +++ b/src/migrate/azext_migrate/runbook/visualize/viewmodel.py @@ -0,0 +1,286 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Build the *grid* view model for a runbook definition/execution. + +This is the data model behind the portal-style grid (the default view in the +generated HTML), grouped by workstream. Like :mod:`graph`, this module is +**data only** — it contains no HTML and performs no I/O, so it is trivially +unit-testable. The renderer turns this model into offline, XSS-escaped markup. +""" + +from azext_migrate.runbook import deps as dep_utils +from azext_migrate.runbook.constants import ENTITY_COMPLETED_STATES + +KIND_DEFINITION = 'definition' +KIND_EXECUTION = 'execution' + + +class EntityProgress: + """Per-entity execution state shown under an execution step.""" + + # pylint: disable=too-few-public-methods + def __init__(self, name, status): + self.name = name + self.status = status + + +class StepRow: + """One row in the grid (a single runbook step).""" + + # pylint: disable=too-few-public-methods,too-many-arguments + # pylint: disable=too-many-instance-attributes + def __init__(self, step_id, name, deps=None, status=None, + workloads=None, workload_progress=None, entities=None, + step_ref=None, entity_names=None, prereqs=None, + dep_details=None): + self.id = step_id + self.name = name + self.deps = deps or [] + self.status = status + self.workloads = workloads + self.workload_progress = workload_progress + self.entities = entities or [] + self.step_ref = step_ref + # Detail-pane fields (definition): resolved entity display names and + # the prerequisite / dependsOn edges labelled with their mode. + self.entity_names = entity_names or [] + self.prereqs = prereqs or [] + self.dep_details = dep_details or [] + + +class Workstream: + """A named group of step rows.""" + + # pylint: disable=too-few-public-methods + def __init__(self, name, steps, ws_id=None): + self.name = name + self.steps = steps + self.id = ws_id + + +class RunbookView: + """The full grid view model for one runbook.""" + + # pylint: disable=too-few-public-methods,too-many-arguments + def __init__(self, title, kind, workstreams, summary, + meta=None, generated=None): + self.title = title + self.kind = kind + self.workstreams = workstreams + self.summary = summary + # ``meta`` is a list of (label, value) header fields (e.g. step + # library version, wave id); ``generated`` is the source-declared + # generation timestamp, if the document carried one. + self.meta = meta or [] + self.generated = generated + + @property + def step_count(self): + return sum(len(ws.steps) for ws in self.workstreams) + + +def _unwrap(document): + """Return the root object, unwrapping an execution ``properties`` envelope.""" + root = document or {} + if isinstance(root, dict) and isinstance(root.get('properties'), dict): + merged = dict(root) + merged.update(root['properties']) + return merged + return root if isinstance(root, dict) else {} + + +def _step_id(step): + return step.get('stepId') or step.get('id') or step.get('name') + + +def _step_name(step): + return (step.get('displayName') or step.get('name') + or step.get('stepName') or _step_id(step) or 'step') + + +def _iter_workstreams(root): + """Yield ``(name, ws_id, [steps])`` triples, covering grouped/flat shapes.""" + workstreams = root.get('workstreams') + if isinstance(workstreams, list) and workstreams: + for workstream in workstreams: + if not isinstance(workstream, dict): + continue + ws_id = workstream.get('id') + name = (workstream.get('displayName') or workstream.get('name') + or ws_id or 'Workstream') + steps = [s for s in workstream.get('steps') or [] + if isinstance(s, dict)] + yield name, ws_id, steps + return + flat = [s for s in root.get('steps') or [] if isinstance(s, dict)] + if flat: + yield None, None, flat + + +def _step_name_map(root): + """Map every step id to its display name (for dependency labels).""" + names = {} + for _, _, steps in _iter_workstreams(root): + for step in steps: + names[_step_id(step)] = _step_name(step) + return names + + +def _entity_name_map(root): + """Map every entity id to its display name (for step detail panes).""" + names = {} + for entity in root.get('entities') or []: + if isinstance(entity, dict): + entity_id = entity.get('id') or entity.get('name') + if entity_id: + names[entity_id] = entity.get('displayName') or entity_id + return names + + +def _dep_entries(raw, id_to_name): + """Label a prerequisite/dependsOn list with resolved names and mode.""" + entries = [] + for dep in raw or []: + if isinstance(dep, dict): + dep_id = dep.get('step') or dep.get('stepId') + mode = dep.get('mode') + else: + dep_id, mode = dep, None + if not dep_id: + continue + name = id_to_name.get(dep_id, dep_id) + entries.append('%s (%s)' % (name, mode) if mode else name) + return entries + + +def build_definition_view(document, title): + """Build the grid view model for a runbook definition document.""" + root = _unwrap(document) + id_to_name = _step_name_map(root) + dep_labels = dep_utils.build_dep_labels(root) + entity_map = _entity_name_map(root) + workstreams = [] + status_counts = {} + for name, ws_id, steps in _iter_workstreams(root): + rows = [] + for step in steps: + status = step.get('configurationStatus') + if status: + key = str(status).split(' ', 1)[0] + status_counts[key] = status_counts.get(key, 0) + 1 + entity_ids = step.get('entities') or [] + rows.append(StepRow( + step_id=_step_id(step), + name=_step_name(step), + deps=dep_utils.label_deps(step, dep_labels), + status=status, + workloads=len(entity_ids), + step_ref=step.get('stepRef'), + entity_names=[entity_map.get(eid, eid) + for eid in entity_ids], + prereqs=_dep_entries(step.get('prerequisite'), id_to_name), + dep_details=_dep_entries(step.get('dependsOn'), id_to_name))) + workstreams.append(Workstream(name, rows, ws_id)) + + step_total = sum(len(ws.steps) for ws in workstreams) + summary = [('Workstreams', len(workstreams)), ('Steps', step_total), + ('Entities', len(root.get('entities') or []))] + summary.extend(sorted(status_counts.items())) + meta, generated = _definition_meta(root) + return RunbookView(title, KIND_DEFINITION, workstreams, summary, + meta=meta, generated=generated) + + +def _definition_meta(root): + """Extract header metadata from a runbook definition document. + + Returns ``(meta, generated)`` where ``meta`` is a list of + ``(label, value)`` header fields and ``generated`` is the + source-declared generation timestamp (or ``None``). + """ + metadata = root.get('metadata') if isinstance( + root.get('metadata'), dict) else {} + generated = metadata.get('generatedAt') + meta = [] + versions = root.get('stepLibraryVersions') + if isinstance(versions, dict) and versions: + meta.append(('Runbook version', ', '.join( + '%s %s' % (name, ver) for name, ver in sorted(versions.items())))) + if generated: + meta.append(('Generated', generated)) + meta.append(('Data source', 'runbook.json')) + resource_id = root.get('runbookResourceId') + if resource_id: + meta.append(('Runbook resource id', resource_id)) + wave_id = metadata.get('waveId') + if wave_id: + meta.append(('Wave id', wave_id)) + return meta, generated + + +def _entity_status(entity): + status = entity.get('status') or entity.get('state') + if isinstance(status, dict): + status = status.get('state') or status.get('status') + return status + + +def _progress_text(step): + """Return a "n/m completed" summary from ``entityExecutions``.""" + explicit = step.get('workloadProgress') + if explicit is not None: + return str(explicit) + entities = step.get('entityExecutions') + if not isinstance(entities, list) or not entities: + return None + completed = sum( + 1 for entity in entities + if isinstance(entity, dict) + and str(_entity_status(entity) or '').lower() + in ENTITY_COMPLETED_STATES) + return '%d/%d completed' % (completed, len(entities)) + + +def _exec_status(step): + return (step.get('status') or step.get('stepStatus') + or step.get('state')) + + +def build_execution_view(document, title): + """Build the grid view model for a runbook execution status document.""" + root = _unwrap(document) + dep_labels = dep_utils.build_dep_labels(root) + workstreams = [] + status_counts = {} + for name, ws_id, steps in _iter_workstreams(root): + rows = [] + for step in steps: + status = _exec_status(step) + if status: + status_counts[str(status)] = \ + status_counts.get(str(status), 0) + 1 + entity_execs = [e for e in step.get('entityExecutions') or [] + if isinstance(e, dict)] + entities = [ + EntityProgress( + e.get('displayName') or e.get('entityId') or e.get('name'), + _entity_status(e)) + for e in entity_execs] + rows.append(StepRow( + step_id=_step_id(step), + name=_step_name(step), + deps=dep_utils.label_deps(step, dep_labels), + status=status, + workload_progress=_progress_text(step), + entities=entities)) + workstreams.append(Workstream(name, rows, ws_id)) + + summary = [] + overall = root.get('state') or root.get('status') + if overall: + summary.append(('State', overall)) + summary.extend(sorted(status_counts.items())) + meta = [('Data source', 'status.json')] + return RunbookView(title, KIND_EXECUTION, workstreams, summary, meta=meta) diff --git a/src/migrate/azext_migrate/shared/__init__.py b/src/migrate/azext_migrate/shared/__init__.py new file mode 100644 index 00000000000..34913fb394d --- /dev/null +++ b/src/migrate/azext_migrate/shared/__init__.py @@ -0,0 +1,4 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- diff --git a/src/migrate/azext_migrate/shared/arm_client.py b/src/migrate/azext_migrate/shared/arm_client.py new file mode 100644 index 00000000000..5a8d3cc9fc1 --- /dev/null +++ b/src/migrate/azext_migrate/shared/arm_client.py @@ -0,0 +1,271 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Single cross-feature ARM REST surface built on send_raw_request.""" + +import json as _json +import re as _re +import time as _time + +from knack.log import get_logger + +from azure.cli.core.util import send_raw_request + +from azext_migrate.shared import arm_ids, errors +from azext_migrate.shared.constants import ( + RUNBOOKS_API_VERSION, + WAVE_OPERATIONS_API_VERSION, +) + +logger = get_logger(__name__) + +# Async status polling tuning. +_DEFAULT_POLL_DELAY = 5 +_MAX_POLL_DELAY = 60 +_TERMINAL_SUCCESS = 'succeeded' +_TERMINAL_FAILURE = ('failed', 'canceled', 'cancelled') + + +def _poll_delay(response): + """Seconds to wait before the next poll, honoring Retry-After.""" + retry_after = response.headers.get('Retry-After') + if retry_after and retry_after.isdigit(): + return min(int(retry_after), _MAX_POLL_DELAY) + return _DEFAULT_POLL_DELAY + + +def _rewrite_poll_api_version(url): + """Point an async-operation status URL at the wave-operations API era. + + Runbook create/delete are served by RUNBOOKS_API_VERSION, but the + Azure-AsyncOperation/Location status endpoint (migrateProjects/ + waveOperations) is only registered at WAVE_OPERATIONS_API_VERSION and + returns NoRegisteredProviderFound otherwise. The header echoes the + request's api-version, so swap only that value, preserving the signed + token that follows. + """ + if 'api-version=' not in url.lower(): + return url + ('&' if '?' in url else '?') + \ + 'api-version=' + WAVE_OPERATIONS_API_VERSION + return _re.sub( + r'([?&]api-version=)[^&]+', + r'\g<1>' + WAVE_OPERATIONS_API_VERSION, + url, count=1) + + +class ArmClient: + """Thin, generic wrapper over send_raw_request for ARM resources. + + Every migrate feature package routes its REST calls through this one + surface so authentication, endpoint selection, serialization, paging + and error mapping live in exactly one place. + """ + + def __init__(self, cmd, api_version=RUNBOOKS_API_VERSION, + rewrite_poll_api_version=True): + self.cmd = cmd + self.api_version = api_version + # Runbook create/delete LROs are polled via the waveOperations type + # at a different api-version; artifact LROs are polled at their own + # async-operation URI as-is, so callers can opt out of the rewrite. + self.rewrite_poll_api_version = rewrite_poll_api_version + + def _url(self, resource_id): + endpoint = self.cmd.cli_ctx.cloud.endpoints.resource_manager + return endpoint.rstrip('/') + arm_ids.with_api_version( + resource_id, self.api_version) + + def _send(self, method, resource_id, body=None): + kwargs = {} + if body is not None: + kwargs['body'] = _json.dumps(body) + response = send_raw_request( + self.cmd.cli_ctx, method, self._url(resource_id), **kwargs) + if response.status_code >= 400: + errors.raise_for_arm_error(response) + return response + + @staticmethod + def _json_or_none(response): + if not getattr(response, 'content', None): + return None + try: + return response.json() + except ValueError: + return None + + def _finalize(self, result, final_get_id): + """Return the settled resource state after a successful LRO. + + Standard Azure CLI behaviour is for create/update commands to + render the *final* resource (a GET issued once the operation + completes), not the initial 201/202 accepted body -- which still + shows ``provisioningState: InProgress``. ``final_get_id`` is the + resource to re-read; it is ``None`` for operations with nothing to + fetch (e.g. delete), in which case the initial body is returned + unchanged. If the follow-up GET 404s (resource gone), the initial + body is kept as a safe fallback. + """ + if not final_get_id: + return result + refreshed = self.get_or_none(final_get_id) + return refreshed if refreshed is not None else result + + def _poll_until_done(self, response, method, resource_id, + final_get_id=None, return_final_poll=False): + """Follow an Azure LRO to completion and return the final state. + + Create/delete on runbooks are long-running: they return 201/202 + with an Azure-AsyncOperation (or Location) header. Poll that URL + until the operation reaches a terminal state. On success, when + ``final_get_id`` is set, re-read that resource so the caller sees + the settled representation (see :meth:`_finalize`); otherwise the + original response body is returned. When ``return_final_poll`` is + set (and no ``final_get_id``), the terminal operation-status body + is returned instead -- used by actions whose result (e.g. a SAS + URL) is carried in the async operation status rather than the + initial accepted body. + """ + header_name = ('Azure-AsyncOperation' + if response.headers.get('Azure-AsyncOperation') + else 'Location') + poll_url = response.headers.get(header_name) + result = self._json_or_none(response) + if response.status_code not in (201, 202) or not poll_url: + logger.info( + "%s '%s' completed synchronously (HTTP %s).", + method, resource_id, response.status_code) + return self._finalize(result, final_get_id) + if self.rewrite_poll_api_version: + poll_url = _rewrite_poll_api_version(poll_url) + op_ref = poll_url.split('?', 1)[0] + delay = _poll_delay(response) + logger.warning( + "%s '%s' is a long-running operation (HTTP %s). Tracking via " + "'%s' header: %s. First status check in %ss " + "(use --no-wait to skip).", + method, resource_id, response.status_code, header_name, + op_ref, delay) + logger.info("Full async-operation poll URL: %s", poll_url) + start = _time.monotonic() + attempt = 0 + while True: + _time.sleep(delay) + attempt += 1 + poll = send_raw_request(self.cmd.cli_ctx, 'GET', poll_url) + if poll.status_code >= 400: + errors.raise_for_arm_error(poll) + body = self._json_or_none(poll) or {} + status = body.get('status') or '' + elapsed = int(_time.monotonic() - start) + norm = status.lower() + if poll.status_code in (200, 204) and not status: + logger.warning( + "%s '%s' completed (elapsed %ss, %s poll(s)).", + method, resource_id, elapsed, attempt) + if return_final_poll and not final_get_id: + return body + return self._finalize(result, final_get_id) + if norm == _TERMINAL_SUCCESS: + logger.warning( + "%s '%s' succeeded (elapsed %ss, %s poll(s)).", + method, resource_id, elapsed, attempt) + if return_final_poll and not final_get_id: + return body + return self._finalize(result, final_get_id) + if norm in _TERMINAL_FAILURE: + errors.raise_for_async_operation(body) + delay = _poll_delay(poll) + logger.warning( + "%s '%s' still running: status=%s (elapsed %ss, " + "poll #%s, next check in %ss).", + method, resource_id, status or '(none)', elapsed, + attempt, delay) + + def _begin(self, method, resource_id, body=None, no_wait=False, + final_get_id=None, return_final_poll=False): + response = self._send(method, resource_id, body) + if no_wait: + logger.warning( + "%s '%s' accepted; --no-wait set, not polling for " + "completion.", method, resource_id) + return self._json_or_none(response) + return self._poll_until_done( + response, method, resource_id, final_get_id, return_final_poll) + + def get(self, resource_id): + """GET a resource and return its JSON body.""" + return self._send('GET', resource_id).json() + + def get_or_none(self, resource_id): + """GET a resource, returning None on 404 (existence check).""" + from azure.cli.core.azclierror import HTTPError + try: + response = send_raw_request( + self.cmd.cli_ctx, 'GET', self._url(resource_id)) + except HTTPError as ex: + if getattr(ex.response, 'status_code', None) == 404: + return None + raise + if response.status_code >= 400: + errors.raise_for_arm_error(response) + return response.json() + + def list(self, collection_id): + """GET a collection, following nextLink pagination.""" + items = [] + url = self._url(collection_id) + while url: + response = send_raw_request(self.cmd.cli_ctx, 'GET', url) + if response.status_code >= 400: + errors.raise_for_arm_error(response) + body = response.json() + items.extend(body.get('value', [])) + url = body.get('nextLink') + return items + + def put(self, resource_id, body=None, no_wait=False): + """PUT (create/generate/start) a resource, awaiting any LRO. + + On success the settled resource is re-read (a final GET on the same + id) so callers render the final state (e.g. ``provisioningState: + Succeeded``) rather than the initial accepted body. + """ + return self._begin( + 'PUT', resource_id, body, no_wait, final_get_id=resource_id) + + def patch(self, resource_id, body=None): + """PATCH (update) a resource.""" + return self._json_or_none(self._send('PATCH', resource_id, body)) + + def delete(self, resource_id, no_wait=False): + """DELETE a resource, awaiting any LRO. + + Returns None: a completed delete has no resource to render (the + initial 202 accepted body still shows the resource as InProgress, + which is misleading), matching standard Azure CLI delete behaviour. + """ + self._begin('DELETE', resource_id, no_wait=no_wait) + + def post_action(self, resource_id, action_name, body=None, + no_wait=False, final_get=False, + return_final_poll=False): + """POST {resourceId}/{action_name} with an optional JSON body. + + This is the workhorse for every action endpoint (AddStep, + PerformAction, ProvideApproval, GenerateDownloadUrl, ...). + + Set ``final_get=True`` only for actions whose settled state is the + parent resource itself (e.g. Regenerate), so the LRO result is a + fresh GET of ``resource_id`` rather than the initial InProgress body. + Set ``return_final_poll=True`` for actions whose result is carried + in the async operation status (e.g. an artifact download SAS URL). + Most actions return their own payload (SAS URL, validation result) + or mutate state exposed elsewhere, so they leave both False. + """ + action_id = f"{resource_id}/{action_name}" + final_get_id = resource_id if final_get else None + return self._begin( + 'POST', action_id, body, no_wait, final_get_id, + return_final_poll) diff --git a/src/migrate/azext_migrate/shared/arm_ids.py b/src/migrate/azext_migrate/shared/arm_ids.py new file mode 100644 index 00000000000..348ea3a7ea3 --- /dev/null +++ b/src/migrate/azext_migrate/shared/arm_ids.py @@ -0,0 +1,40 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Generic ARM resource-id and URL builders shared across features.""" + +from azext_migrate.shared import constants + + +def migrate_project_id(subscription_id, resource_group, project_name): + """Build a migrate-project ARM id.""" + return constants.MIGRATE_PROJECT_ID_TEMPLATE.format( + subscription_id=subscription_id, + resource_group=resource_group, + project_name=project_name, + ) + + +def runbook_id(project_id, runbook_name): + """Build a runbook ARM id from a project id.""" + return constants.RUNBOOK_ID_TEMPLATE.format( + project_id=project_id, runbook_name=runbook_name) + + +def execution_id(runbook_resource_id, execution): + """Build a runbook-execution ARM id from a runbook id.""" + return constants.EXECUTION_ID_TEMPLATE.format( + runbook_id=runbook_resource_id, execution_id=execution) + + +def artifact_id(project_id, artifact_name): + """Build an artifact ARM id from a project id.""" + return constants.ARTIFACT_ID_TEMPLATE.format( + project_id=project_id, artifact_name=artifact_name) + + +def with_api_version(resource_id, api_version): + """Append the api-version query parameter to a resource id/url.""" + joiner = '&' if '?' in resource_id else '?' + return f"{resource_id}{joiner}api-version={api_version}" diff --git a/src/migrate/azext_migrate/shared/constants.py b/src/migrate/azext_migrate/shared/constants.py new file mode 100644 index 00000000000..477cf2cb64b --- /dev/null +++ b/src/migrate/azext_migrate/shared/constants.py @@ -0,0 +1,34 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Constants shared across all migrate feature packages.""" + +# Resource provider namespace (canonical casing). +PROVIDER_NAMESPACE = "Microsoft.Migrate" + +# API version for the Runbooks resource provider surface. +RUNBOOKS_API_VERSION = "2020-06-01-preview" + +# The runbook definition/parameters archive is delivered by the separate +# Artifact Service (Microsoft.Migrate artifacts), which is versioned +# independently of the runbooks surface. +ARTIFACTS_API_VERSION = "2020-06-01-preview" + +# Runbook create/delete are long-running operations whose async status +# is served by the migrateProjects/waveOperations type. That type is NOT +# registered at RUNBOOKS_API_VERSION; the async-operation status URL must +# be polled at this newer API version instead. +WAVE_OPERATIONS_API_VERSION = "2025-03-30-preview" + +# Canonical ARM ID templates (camelCase per the confirmed spec): +# /subscriptions/{s}/resourceGroups/{rg}/providers/Microsoft.Migrate +# /migrateProjects/{p}/runbooks/{n}/executions/{e} +MIGRATE_PROJECT_ID_TEMPLATE = ( + "/subscriptions/{subscription_id}" + "/resourceGroups/{resource_group}" + "/providers/Microsoft.Migrate/migrateProjects/{project_name}" +) +RUNBOOK_ID_TEMPLATE = "{project_id}/runbooks/{runbook_name}" +EXECUTION_ID_TEMPLATE = "{runbook_id}/executions/{execution_id}" +ARTIFACT_ID_TEMPLATE = "{project_id}/artifacts/{artifact_name}" diff --git a/src/migrate/azext_migrate/shared/errors.py b/src/migrate/azext_migrate/shared/errors.py new file mode 100644 index 00000000000..5c2d1c904b4 --- /dev/null +++ b/src/migrate/azext_migrate/shared/errors.py @@ -0,0 +1,59 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Map ARM error responses to typed azclierror exceptions.""" + +from azure.cli.core.azclierror import ( + AzureResponseError, + ClientRequestError, + CLIInternalError, + ForbiddenError, + InvalidArgumentValueError, + ResourceNotFoundError, +) + + +def raise_for_arm_error(response): + """Raise a typed CLI error for an ARM response with status >= 400.""" + status = response.status_code + code, message = _extract_error(response) + detail = f"{code}: {message}" if code else message + if status == 404: + raise ResourceNotFoundError(detail) + if status == 400: + raise InvalidArgumentValueError(detail) + if status == 403: + raise ForbiddenError(detail) + if status == 409: + raise ClientRequestError(detail) + if status >= 500: + raise CLIInternalError(detail) + raise ClientRequestError(detail) + + +def raise_for_async_operation(status_body): + """Raise a typed CLI error for a failed async operation status body.""" + status = status_body.get('status', 'Failed') + error = status_body.get('error') + if isinstance(error, dict): + code = error.get('code') + message = error.get('message', '') + detail = f"{code}: {message}" if code else message + else: + detail = '' + if not detail: + detail = f"The operation completed with status '{status}'." + raise AzureResponseError(detail) + + +def _extract_error(response): + """Extract (code, message) from an ARM error body when present.""" + try: + body = response.json() + except ValueError: + return None, response.text + error = body.get('error') if isinstance(body, dict) else None + if isinstance(error, dict): + return error.get('code'), error.get('message', response.text) + return None, response.text diff --git a/src/migrate/azext_migrate/shared/files.py b/src/migrate/azext_migrate/shared/files.py new file mode 100644 index 00000000000..e5ce0beb378 --- /dev/null +++ b/src/migrate/azext_migrate/shared/files.py @@ -0,0 +1,373 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Download and safely extract SAS-protected ZIP artifacts. + +Several runbook endpoints (``GenerateDownloadUrl``) return a pre-signed +blob SAS URL that points at a ZIP. The SAS is self-authorizing, so the +blob is fetched with a plain HTTPS GET (no ARM token) and then extracted +with guards against path traversal (zip-slip) and decompression bombs. +""" + +import io +import json +import os +import tempfile +import zipfile +from urllib.request import Request, urlopen + +from knack.log import get_logger + +from azure.cli.core.azclierror import ( + CLIInternalError, + InvalidArgumentValueError, +) + +logger = get_logger(__name__) + +# Guard rails for the untrusted archive we extract. +_MAX_TOTAL_UNCOMPRESSED = 256 * 1024 * 1024 +_MAX_MEMBERS = 1000 + +# Keys a GenerateDownloadUrl / GenerateInputUploadUrl response may use for +# the SAS URL, checked both at the top level and under ``properties``. +_SAS_URL_KEYS = ( + 'uploadUrl', 'uploadUri', 'downloadUrl', 'downloadUri', + 'sasUrl', 'sasUri', 'url', 'uri') + +# Derived/computed inputs the CLI must never surface or download. This +# document shares the 'runbookInputs' shape with the user parameters, so it +# can only be distinguished by name (content classification is not enough). +_DERIVED_INPUTS_NAMES = ('derived-input.json', 'derived-inputs.json') + +# File name of the execution status document fetched via a per-execution +# SAS download (GenerateDownloadUrl on the execution resource). Confirmed +# against the live API: the blob may be either the raw status.json bytes or +# a ZIP that contains it; read_status_json handles both. +_STATUS_SUFFIX = 'status.json' + +# Local ZIP file magic; a SAS download may be a ZIP archive or a raw blob. +_ZIP_MAGIC = b'PK\x03\x04' + + +def _is_zip(data): + """True when ``data`` starts with the local ZIP file signature.""" + return isinstance(data, (bytes, bytearray)) and data[:4] == _ZIP_MAGIC + + +def extract_sas_url(response_body): + """Return the download URL from a GenerateDownloadUrl response body.""" + if not isinstance(response_body, dict): + return None + for source in (response_body, response_body.get('properties')): + if not isinstance(source, dict): + continue + for key in _SAS_URL_KEYS: + value = source.get(key) + if isinstance(value, str) and value: + return value + return None + + +def download_bytes(url): + """HTTP GET a self-authorizing https URL and return the raw bytes.""" + if not isinstance(url, str) or not url.lower().startswith('https://'): + raise InvalidArgumentValueError( + 'The download URL must be an absolute https URL.') + request = Request(url, method='GET') + # The URL is a pre-signed blob SAS returned by ARM and validated above + # to be https; no ARM token is attached. + with urlopen(request) as response: # nosec B310 + return response.read() + + +def upload_bytes(url, data): + """HTTP PUT raw bytes to a self-authorizing https blob SAS URL.""" + if not isinstance(url, str) or not url.lower().startswith('https://'): + raise InvalidArgumentValueError( + 'The upload URL must be an absolute https URL.') + request = Request( + url, data=data, method='PUT', + headers={'x-ms-blob-type': 'BlockBlob'}) + # The URL is a pre-signed blob SAS returned by ARM and validated above + # to be https; no ARM token is attached. The SAS query string carries + # the signature, so only the blob path (before '?') is logged. + logger.debug( + 'Uploading %d bytes to blob %s', len(data), url.split('?', 1)[0]) + with urlopen(request) as response: # nosec B310 + logger.debug('Blob upload completed (HTTP %s).', response.status) + return None + + +def _looks_like_spec(parsed): + """True when a parsed JSON document is a runbook definition/spec.""" + return isinstance(parsed, dict) and ( + 'runbookSpec' in parsed or 'workstreams' in parsed) + + +def _looks_like_parameters(parsed): + """True when a parsed JSON document is a runbook parameters file.""" + return isinstance(parsed, dict) and ( + 'runbookInputs' in parsed + or 'stepInputs' in parsed + or 'schema' in parsed) + + +def _looks_like_status(parsed): + """True when a parsed JSON document is an execution status document. + + The per-execution download archive can also carry the definition + (wrapped in ``runbookSpec``) and the input parameters (``runbookInputs`` + / ``schema`` / ``stepInputs``); neither is a status document. A status + document has a bare ``workstreams`` / ``steps`` / ``state`` shape, so it + is anything that is a dict and is not the definition wrapper or the + parameters file. + """ + return (isinstance(parsed, dict) + and 'runbookSpec' not in parsed + and not _looks_like_parameters(parsed)) + + +def _classify_archive(zip_bytes): + """Sort the GenerateDownloadUrl archive members by role. + + Returns ``{'definition': (name, bytes) | None, + 'parameters': (name, bytes) | None, 'docs': [(name, bytes), ...]}``. + + The archive ships the definition (``runbookSpec``), the user parameters + (``runbookInputs``), the ``derived-input.json`` computed inputs, and a + documentation markdown. ``derived-input.json`` shares the parameters + shape and is distinguished only by name, so it is skipped here; every + other member is classified by content. This is the single source of + truth for the archive layout. + """ + definition = parameters = None + docs = [] + with zipfile.ZipFile(io.BytesIO(zip_bytes)) as archive: + _guard_archive(archive.infolist()) + for info in archive.infolist(): + if info.is_dir(): + continue + name = os.path.basename(info.filename.replace('\\', '/')) + lower = name.lower() + if lower in _DERIVED_INPUTS_NAMES: + continue + data = archive.read(info) + if lower.endswith('.md'): + docs.append((name, data)) + continue + if not lower.endswith('.json'): + continue + try: + parsed = json.loads(data.decode('utf-8')) + except ValueError: + continue + if definition is None and _looks_like_spec(parsed): + definition = (name, data) + elif parameters is None and _looks_like_parameters(parsed): + parameters = (name, data) + return { + 'definition': definition, 'parameters': parameters, 'docs': docs} + + +def read_spec_json(zip_bytes): + """Return the parsed runbook definition (``runbookSpec``) or None. + + Accepts either a ZIP archive (definition classified out of it) or a raw + ``runbook.json`` blob (file-mode download), returning the parsed JSON. + """ + if not _is_zip(zip_bytes): + try: + return json.loads(zip_bytes.decode('utf-8')) + except ValueError: + return None + found = _classify_archive(zip_bytes)['definition'] + return json.loads(found[1].decode('utf-8')) if found else None + + +def extract_parameters_file(zip_bytes): + """Return ``(filename, raw_bytes)`` for the user parameters, or None. + + ``derived-input.json`` (which shares the parameters shape) is never + returned; see :func:`_classify_archive`. A raw (non-ZIP) blob carries + no separate parameters file, so None is returned. + """ + if not _is_zip(zip_bytes): + return None + return _classify_archive(zip_bytes)['parameters'] + + +def read_parameters_json(zip_bytes): + """Return the parsed ``runbookInputs`` object from the ZIP, or None.""" + found = extract_parameters_file(zip_bytes) + if not found: + return None + parsed = json.loads(found[1].decode('utf-8')) + if isinstance(parsed, dict) and isinstance( + parsed.get('runbookInputs'), dict): + return parsed['runbookInputs'] + return parsed + + +def read_status_json(raw_bytes): + """Return the parsed execution status document from a SAS download. + + The per-execution SAS blob may be either the raw ``status.json`` bytes + or a ZIP archive that contains it. A not-yet-run execution's download + archive ships only the input parameters (``runbookInputs``) and/or the + definition (``runbookSpec``); those are NOT a status document and are + rejected here so callers can fall back to the execution resource. Raises + :class:`CLIInternalError` when no status document is present. + """ + if raw_bytes[:4] == b'PK\x03\x04': + with zipfile.ZipFile(io.BytesIO(raw_bytes)) as archive: + _guard_archive(archive.infolist()) + named = typed = None + for info in archive.infolist(): + if info.is_dir(): + continue + base = os.path.basename(info.filename.replace('\\', '/')) + lower = base.lower() + if lower in _DERIVED_INPUTS_NAMES \ + or not lower.endswith('.json'): + continue + try: + parsed = json.loads(archive.read(info).decode('utf-8')) + except ValueError: + continue + if not _looks_like_status(parsed): + continue + if lower.endswith(_STATUS_SUFFIX): + named = parsed + break + if typed is None: + typed = parsed + status = named if named is not None else typed + if status is None: + raise CLIInternalError( + 'The downloaded archive did not contain an execution ' + 'status file.') + return status + parsed = json.loads(raw_bytes.decode('utf-8')) + if not _looks_like_status(parsed): + raise CLIInternalError( + 'The download did not contain an execution status document.') + return parsed + + +def read_json_file(path): + """Read and parse a local JSON file, returning the parsed object. + + Enables offline rendering/testing of the visualize commands from + definition/parameters/status JSON files without contacting the service. + The file is read as bytes so ``json.loads`` can auto-detect the encoding + (UTF-8/16/32, with or without a BOM); this tolerates files saved by + Windows PowerShell redirection, which default to UTF-16. + """ + with open(path, 'rb') as handle: + return json.loads(handle.read()) + + +def resolve_output_path(file, default_name): + """Resolve a user ``--file`` value to an absolute output file path. + + ``file`` may be ``None`` (write ``default_name`` into the current + directory), a directory (write ``default_name`` inside it), or a full + file path. A value that names a not-yet-created directory -- one that + ends with a path separator or has no file extension -- is treated as a + directory so ``default_name`` is written inside it (rather than becoming + an extensionless output file). The result is always an absolute, + normalized path. + """ + if not file: + return os.path.join(os.getcwd(), default_name) + looks_like_dir = ( + file.endswith(('/', '\\')) or os.path.splitext(file)[1] == '') + target = os.path.abspath(file) + if os.path.isdir(target): + return os.path.join(target, default_name) + if looks_like_dir and not os.path.isfile(target): + return os.path.join(target, default_name) + return target + + +def write_text(path, text): + """Write ``text`` (UTF-8) to ``path`` atomically, creating parent dirs. + + The content is written to a temporary file in the same directory and then + atomically moved into place with :func:`os.replace`. This guarantees a + reader (e.g. a browser auto-reloading the file during ``--watch``) never + observes a partially written file. + """ + absolute = os.path.abspath(path) + parent = os.path.dirname(absolute) + if parent: + os.makedirs(parent, exist_ok=True) + fd, tmp = tempfile.mkstemp( + prefix='.runbook-', suffix='.tmp', dir=parent or None) + try: + with os.fdopen(fd, 'w', encoding='utf-8') as handle: + handle.write(text) + os.replace(tmp, absolute) + except BaseException: + try: + os.remove(tmp) + except OSError: + pass + raise + return absolute + + +def open_in_browser(path): + """Best-effort open a local file in the default browser.""" + import webbrowser + try: + webbrowser.open('file://' + os.path.abspath(path)) + except OSError as ex: # pragma: no cover - environment dependent + logger.warning('Could not open the file in a browser: %s', ex) + + +def extract_definition_files(zip_bytes, destination): + """Write the runbook definition, its parameters and docs to disk. + + Writes the definition (``runbookSpec``), the user parameters + (``runbookInputs``) and any ``.md`` docs; the redundant + ``derived-input.json`` is skipped (see :func:`_classify_archive`). The + parameters file is downloaded because the definition's per-step + ``configurationStatus`` is computed from it, but callers still render only + the definition in table/CLI output. Member names are flattened to their + base name, so a hostile archive path cannot escape ``destination`` + (zip-slip is designed out rather than checked at write time). Returns the + absolute paths written. + """ + destination = os.path.abspath(destination) + os.makedirs(destination, exist_ok=True) + if not _is_zip(zip_bytes): + target = os.path.join(destination, 'runbook.json') + with open(target, 'wb') as handle: + handle.write(zip_bytes) + return [target] + classified = _classify_archive(zip_bytes) + selected = list(classified['docs']) + if classified['definition']: + selected.insert(0, classified['definition']) + if classified['parameters']: + selected.append(classified['parameters']) + written = [] + for name, data in selected: + target = os.path.join(destination, os.path.basename(name)) + with open(target, 'wb') as handle: + handle.write(data) + written.append(target) + return written + + +def _guard_archive(infos): + """Reject archives with too many members or an implausible size.""" + if len(infos) > _MAX_MEMBERS: + raise CLIInternalError( + 'Downloaded archive has too many entries.') + if sum(i.file_size for i in infos) > _MAX_TOTAL_UNCOMPRESSED: + raise CLIInternalError( + 'Downloaded archive is unexpectedly large.') diff --git a/src/migrate/azext_migrate/shared/telemetry.py b/src/migrate/azext_migrate/shared/telemetry.py new file mode 100644 index 00000000000..db46d5f2f48 --- /dev/null +++ b/src/migrate/azext_migrate/shared/telemetry.py @@ -0,0 +1,37 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- +"""Best-effort telemetry wrappers. These never raise on failure.""" + +from knack.log import get_logger + +logger = get_logger(__name__) + + +def record_exception(ex, fault_type, summary=None): + """Record a handled exception as a telemetry fault.""" + try: + from azure.cli.core import telemetry + telemetry.set_exception( + exception=ex, fault_type=fault_type, summary=summary) + except Exception as tex: # pylint: disable=broad-except + logger.debug('telemetry record_exception failed: %s', tex) + + +def set_user_fault(summary=None): + """Flag the current failure as a user fault (4xx/validation).""" + try: + from azure.cli.core import telemetry + telemetry.set_user_fault(summary=summary) + except Exception as tex: # pylint: disable=broad-except + logger.debug('telemetry set_user_fault failed: %s', tex) + + +def add_event(name, properties=None): + """Emit a lightweight extension telemetry event (no PII).""" + try: + from azure.cli.core import telemetry + telemetry.add_extension_event(name, properties or {}) + except Exception as tex: # pylint: disable=broad-except + logger.debug('telemetry add_event failed: %s', tex) diff --git a/src/migrate/azext_migrate/tests/latest/runbook/__init__.py b/src/migrate/azext_migrate/tests/latest/runbook/__init__.py new file mode 100644 index 00000000000..34913fb394d --- /dev/null +++ b/src/migrate/azext_migrate/tests/latest/runbook/__init__.py @@ -0,0 +1,4 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- diff --git a/src/migrate/azext_migrate/tests/latest/runbook/recordings/test_runbook_show_and_list.yaml b/src/migrate/azext_migrate/tests/latest/runbook/recordings/test_runbook_show_and_list.yaml new file mode 100644 index 00000000000..11c61c141ea --- /dev/null +++ b/src/migrate/azext_migrate/tests/latest/runbook/recordings/test_runbook_show_and_list.yaml @@ -0,0 +1,98 @@ +interactions: +- request: + body: null + headers: + Accept: + - '*/*' + Accept-Encoding: + - gzip, deflate + CommandName: + - migrate runbook show + Connection: + - keep-alive + ParameterSetName: + - -g --project-name -n + User-Agent: + - python/3.12.10 (Windows-11-10.0.26100-SP0) AZURECLI/2.88.0 + method: GET + uri: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/BP_AE_Can/providers/Microsoft.Migrate/migrateProjects/BP-AE-Can-Proj/runbooks/testrunbook1?api-version=2020-06-01-preview + response: + body: + string: '{"properties":{"scope":{"ScopeType":"Wave","scopeType":"Wave","waveId":"/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/bp_ae_can/providers/microsoft.migrate/migrateprojects/bp-ae-can-proj/waves/wave66"},"artifactId":"rb-testrunbook1","state":"ExecutionSucceeded","provisioningState":"Succeeded"},"id":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/BP_AE_Can/providers/Microsoft.Migrate/MigrateProjects/BP-AE-Can-Proj/Runbooks/testrunbook1","name":"testrunbook1","type":"Microsoft.Migrate/MigrateProjects/Runbooks","location":"","eTag":null}' + headers: + cache-control: + - no-cache + content-length: + - '581' + content-type: + - application/json; charset=utf-8 + date: + - Fri, 17 Jul 2026 12:59:16 GMT + expires: + - '-1' + pragma: + - no-cache + strict-transport-security: + - max-age=31536000; includeSubDomains + x-cache: + - CONFIG_NOCACHE + x-content-type-options: + - nosniff + x-ms-operation-identifier: + - tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47,objectId=92634de3-03f6-4092-b41b-20616b11a464/australiaeast/949f8bc7-0670-49f8-b169-38c414445645 + x-ms-ratelimit-remaining-subscription-global-reads: + - '16499' + x-msedge-ref: + - 'Ref A: BEF09BF87170414D8B447BE86681E59F Ref B: MAA211070113031 Ref C: 2026-07-17T12:59:16Z' + status: + code: 200 + message: OK +- request: + body: null + headers: + Accept: + - '*/*' + Accept-Encoding: + - gzip, deflate + CommandName: + - migrate runbook list + Connection: + - keep-alive + ParameterSetName: + - -g --project-name + User-Agent: + - python/3.12.10 (Windows-11-10.0.26100-SP0) AZURECLI/2.88.0 + method: GET + uri: https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/BP_AE_Can/providers/Microsoft.Migrate/migrateProjects/BP-AE-Can-Proj/runbooks?api-version=2020-06-01-preview + response: + body: + string: '{"value":[{"properties":{"scope":{"ScopeType":"Wave","scopeType":"Wave","waveId":"/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/bp_ae_can/providers/microsoft.migrate/migrateprojects/bp-ae-can-proj/waves/wave66"},"artifactId":"rb-testrunbook1","state":"ExecutionSucceeded","provisioningState":"Succeeded"},"id":"/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/BP_AE_Can/providers/Microsoft.Migrate/MigrateProjects/BP-AE-Can-Proj/Runbooks/testrunbook1","name":"testrunbook1","type":"Microsoft.Migrate/MigrateProjects/Runbooks","location":"","eTag":null}]}' + headers: + cache-control: + - no-cache + content-length: + - '593' + content-type: + - application/json; charset=utf-8 + date: + - Fri, 17 Jul 2026 12:59:23 GMT + expires: + - '-1' + pragma: + - no-cache + strict-transport-security: + - max-age=31536000; includeSubDomains + x-cache: + - CONFIG_NOCACHE + x-content-type-options: + - nosniff + x-ms-operation-identifier: + - tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47,objectId=92634de3-03f6-4092-b41b-20616b11a464/australiaeast/0cd37251-8846-462a-9b05-1914819ff3e2 + x-ms-ratelimit-remaining-subscription-global-reads: + - '16499' + x-msedge-ref: + - 'Ref A: 0EAA51D6728E49589A758C63707E3BC2 Ref B: MAA211070116025 Ref C: 2026-07-17T12:59:22Z' + status: + code: 200 + message: OK +version: 1 diff --git a/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_recording.py b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_recording.py new file mode 100644 index 00000000000..ae159d233bc --- /dev/null +++ b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_recording.py @@ -0,0 +1,232 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- + +"""Recorded ScenarioTests for the ``az migrate runbook`` commands. + +Runbooks require a pre-existing migrate project + wave (a +``ResourceGroupPreparer`` cannot provision those), so these scenarios target +fixed, pre-provisioned resources. Override the defaults at RECORD time with +env vars (see below); the subscription id and SAS/cert secrets are scrubbed, +so PLAYBACK needs no live resources or credentials. + + AZURE_MIGRATE_TEST_RG resource group of the migrate project + AZURE_MIGRATE_TEST_PROJECT migrate project name + AZURE_MIGRATE_TEST_WAVE wave to generate runbooks from + AZURE_MIGRATE_TEST_RUNBOOK an existing runbook (read-only scenarios) + AZURE_MIGRATE_TEST_EXECUTION an existing execution id (read scenarios) + +Recording model: +* Pure-ARM commands (generate/show/list/update/delete/regenerate, + execution list) record to a cassette and replay offline in CI. +* Commands that fetch/put a SAS blob (definition show/download, parameter + and execution-parameter download/upload, execution show/visualize) do + their blob I/O via ``urllib`` -- which the CLI test recorder does NOT + intercept -- so they are marked ``@live_only`` and are never replayed. +""" + +import os +import re +import unittest + +from azure.cli.testsdk import ScenarioTest, live_only +from azure.cli.testsdk.scenario_tests import RecordingProcessor + +TEST_DIR = os.path.abspath(os.path.join(os.path.abspath(__file__), '..')) + +RUNBOOK_TYPE = 'Microsoft.Migrate/MigrateProjects/Runbooks' + + +def _cfg(name, default): + return os.environ.get(name, default) + + +PROJECT_RG = _cfg('AZURE_MIGRATE_TEST_RG', 'wave-runbooks-rg') +PROJECT_NAME = _cfg('AZURE_MIGRATE_TEST_PROJECT', 'wave-runbooks-project') +WAVE_NAME = _cfg('AZURE_MIGRATE_TEST_WAVE', 'wave-runbook-01') +RUNBOOK_NAME = _cfg('AZURE_MIGRATE_TEST_RUNBOOK', 'runbook-01') +EXECUTION_ID = _cfg('AZURE_MIGRATE_TEST_EXECUTION', 'exec-01') + + +class _SasScrubber(RecordingProcessor): + """Redact SAS signatures, async-op signing blobs and storage hosts. + + The runbook LRO poll URLs (Location/Azure-AsyncOperation) carry large + ``c=``/``s=``/``h=`` signing blobs, and download responses carry a + ``sasUrl`` with a live ``sig=`` token; none of these must land in a + committed cassette. + """ + + _BLOB_HOST = re.compile( + r'https://[a-z0-9]+\.blob\.core\.windows\.net') + + def _scrub(self, text): + if not text: + return text + text = re.sub(r'(sig=)[^&"\s\\]+', r'\1REDACTED', text) + text = re.sub(r'([?&](?:c|s|h)=)[^&"\s\\]+', r'\1REDACTED', text) + text = self._BLOB_HOST.sub( + 'https://mockstorage.blob.core.windows.net', text) + return text + + def process_request(self, request): + request.uri = self._scrub(request.uri) + if isinstance(request.body, bytes): + try: + request.body = self._scrub( + request.body.decode('utf-8')).encode('utf-8') + except UnicodeDecodeError: + pass + elif isinstance(request.body, str): + request.body = self._scrub(request.body) + return request + + def process_response(self, response): + body = (response.get('body') or {}).get('string') + if body: + response['body']['string'] = self._scrub(body) + return response + + +class _RunbookScenario(ScenarioTest): + """Base ScenarioTest that installs the SAS/cert scrubber.""" + + def __init__(self, method_name): + super().__init__( + method_name, recording_processors=[_SasScrubber()]) + + +# TODO(runbook): record cassettes against a live migrate project + wave, +# then drop @live_only so these replay offline in CI. Until a cassette +# exists the framework would run them live and fail on CI's empty sub. +@live_only() +class RunbookReadScenario(_RunbookScenario): + + def test_runbook_show_and_list(self): + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'name': RUNBOOK_NAME, + }) + self.cmd( + 'migrate runbook show -g {rg} --project-name {project} ' + '-n {name}', + checks=[ + self.check('name', '{name}'), + self.check('type', RUNBOOK_TYPE), + ]) + self.cmd( + 'migrate runbook list -g {rg} --project-name {project}', + checks=[self.check("length([?name=='{name}'])", 1)]) + + +@live_only() +class RunbookCrudScenario(_RunbookScenario): + + def test_runbook_generate_update_delete(self): + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'wave': WAVE_NAME, + 'name': self.create_random_name('cli-rb-', 20), + }) + self.cmd( + 'migrate runbook generate -g {rg} --project-name {project} ' + '-n {name} --wave-name {wave}', + checks=[ + self.check('name', '{name}'), + self.check('type', RUNBOOK_TYPE), + ]) + self.cmd( + 'migrate runbook show -g {rg} --project-name {project} ' + '-n {name}', + checks=[self.check('name', '{name}')]) + self.cmd( + 'migrate runbook list -g {rg} --project-name {project}', + checks=[self.check("length([?name=='{name}'])", 1)]) + self.cmd( + 'migrate runbook update -g {rg} --project-name {project} ' + '-n {name} --description "recorded by scenario test"', + checks=[self.check( + 'properties.description', 'recorded by scenario test')]) + self.cmd( + 'migrate runbook delete -g {rg} --project-name {project} ' + '-n {name} --yes') + self.cmd( + 'migrate runbook list -g {rg} --project-name {project}', + checks=[self.check("length([?name=='{name}'])", 0)]) + + +@live_only() +class RunbookExecutionReadScenario(_RunbookScenario): + + def test_execution_list(self): + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'runbook': RUNBOOK_NAME, + }) + self.cmd( + 'migrate runbook execution list -g {rg} ' + '--project-name {project} --runbook-name {runbook}') + + +# --------------------------------------------------------------------------- +# Live-only scenarios: these fetch/put a SAS blob (or run a real migration), +# which the CLI test recorder does not intercept, so they cannot be replayed +# from a cassette. Run them with --live against a prepared subscription. +# --------------------------------------------------------------------------- + +@live_only() +class RunbookArtifactLiveScenario(ScenarioTest): + """definition show/download + parameter download go through a SAS blob.""" + + def test_definition_show_and_download(self): + import tempfile + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'name': RUNBOOK_NAME, + 'dest': tempfile.mkdtemp(), + }) + self.cmd( + 'migrate runbook definition show -g {rg} ' + '--project-name {project} -n {name}') + self.cmd( + 'migrate runbook definition download -g {rg} ' + '--project-name {project} -n {name} --destination {dest}') + + def test_parameter_download(self): + import tempfile + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'runbook': RUNBOOK_NAME, + 'dest': tempfile.mkdtemp(), + }) + self.cmd( + 'migrate runbook parameter download -g {rg} ' + '--project-name {project} --runbook-name {runbook} ' + '--file {dest}') + + +@live_only() +class RunbookExecutionLiveScenario(ScenarioTest): + """Execution status needs live, running state and a SAS status blob.""" + + def test_execution_show(self): + self.kwargs.update({ + 'rg': PROJECT_RG, + 'project': PROJECT_NAME, + 'runbook': RUNBOOK_NAME, + 'execution': EXECUTION_ID, + }) + self.cmd( + 'migrate runbook execution show -g {rg} ' + '--project-name {project} --runbook-name {runbook} ' + '--execution-id {execution}') + + +if __name__ == '__main__': + unittest.main() diff --git a/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_scenario.py b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_scenario.py new file mode 100644 index 00000000000..a16b2dfa141 --- /dev/null +++ b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_scenario.py @@ -0,0 +1,137 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- + +import unittest +from unittest import mock + +from azext_migrate.runbook.cmds import runbook as runbook_cmds +from azext_migrate.runbook.cmds import execution_step as execution_step_cmds + +SUB = "00000000-0000-0000-0000-000000000000" +RG = "myRg" +PROJECT = "myProject" +RUNBOOK = "myRunbook" +WAVE = "myWave" + +PROJECT_ID = ( + f"/subscriptions/{SUB}/resourceGroups/{RG}/providers/" + f"Microsoft.Migrate/migrateProjects/{PROJECT}") +RUNBOOK_ID = f"{PROJECT_ID}/runbooks/{RUNBOOK}" +EXECUTION_ID = f"{RUNBOOK_ID}/executions/exec1" + + +def _mock_cmd(): + cmd = mock.Mock() + cmd.cli_ctx.cloud.endpoints.resource_manager = ( + "https://management.azure.com") + return cmd + + +class RunbookCrudScenarioTest(unittest.TestCase): + """Exercises the runbook generate/show/list/delete orchestration.""" + + def setUp(self): + self.cmd = _mock_cmd() + sub_patch = mock.patch( + 'azext_migrate.runbook.cmds.runbook.get_subscription_id', + return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + + client_patch = mock.patch( + 'azext_migrate.runbook.cmds.runbook.ArmClient') + self.addCleanup(client_patch.stop) + self.client_cls = client_patch.start() + self.client = self.client_cls.return_value + + def test_runbook_crud(self): + generated = { + "id": RUNBOOK_ID, "name": RUNBOOK, + "properties": {"status": "Generating", "scope": { + "scopeType": "Wave", + "waveId": f"{PROJECT_ID}/waves/{WAVE}"}}} + self.client.put.return_value = generated + + result = runbook_cmds.generate( + self.cmd, RG, PROJECT, RUNBOOK, WAVE) + + self.assertEqual(result, generated) + put_id, put_body = self.client.put.call_args[0] + self.assertEqual(put_id, RUNBOOK_ID) + self.assertEqual( + put_body["properties"]["scope"]["waveId"], + f"{PROJECT_ID}/waves/{WAVE}") + + self.client.get.return_value = generated + shown = runbook_cmds.show(self.cmd, RG, PROJECT, RUNBOOK) + self.assertEqual(shown["name"], RUNBOOK) + self.client.get.assert_called_once_with(RUNBOOK_ID) + + self.client.list.return_value = [ + generated, + {"name": "other", "properties": { + "status": "NotConfigured", "scope": { + "waveId": f"{PROJECT_ID}/waves/otherWave"}}}] + filtered = runbook_cmds.list_( + self.cmd, RG, PROJECT, wave_name=WAVE) + self.assertEqual([r["name"] for r in filtered], [RUNBOOK]) + self.client.list.assert_called_once_with( + f"{PROJECT_ID}/runbooks") + + by_status = runbook_cmds.list_( + self.cmd, RG, PROJECT, status="NotConfigured") + self.assertEqual([r["name"] for r in by_status], ["other"]) + + self.client.delete.return_value = None + runbook_cmds.delete(self.cmd, RG, PROJECT, RUNBOOK) + self.client.delete.assert_called_once_with( + RUNBOOK_ID, no_wait=False) + + +class ExecutionStepScenarioTest(unittest.TestCase): + """Exercises the retry/approve/complete step-action orchestration.""" + + def setUp(self): + self.cmd = _mock_cmd() + sub_patch = mock.patch( + 'azext_migrate.runbook.cmds.execution.get_subscription_id', + return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + + client_patch = mock.patch( + 'azext_migrate.runbook.cmds.execution_step.ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def test_execution_step_actions(self): + self.client.post_action.return_value = {"stepId": "step1"} + + execution_step_cmds.retry( + self.cmd, RG, PROJECT, RUNBOOK, "exec1", "step1") + resource_id, action, body = self.client.post_action.call_args[0] + self.assertEqual(resource_id, EXECUTION_ID) + self.assertEqual(action, 'PerformAction') + self.assertEqual(body["action"], "Retry") + self.assertEqual(body["targetId"], "step1") + + execution_step_cmds.approve( + self.cmd, RG, PROJECT, RUNBOOK, "exec1", "step1", + entities=["ent1"]) + _, action, body = self.client.post_action.call_args[0] + self.assertEqual(action, 'ProvideApproval') + self.assertEqual(body["action"], "Approve") + self.assertEqual(body["migrationEntityIds"], ["ent1"]) + + execution_step_cmds.complete( + self.cmd, RG, PROJECT, RUNBOOK, "exec1", "step1", "done") + _, action, body = self.client.post_action.call_args[0] + self.assertEqual(action, 'UpdateStepStatus') + self.assertEqual(body["action"], "Complete") + self.assertEqual(body["comment"], "done") + + +if __name__ == '__main__': + unittest.main() diff --git a/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_unit.py b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_unit.py new file mode 100644 index 00000000000..5e14cf205d8 --- /dev/null +++ b/src/migrate/azext_migrate/tests/latest/runbook/test_runbook_unit.py @@ -0,0 +1,2112 @@ +# -------------------------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for license information. +# -------------------------------------------------------------------------------------------- + +import io +import json +import os +import copy +import tempfile +import unittest +import zipfile +from types import SimpleNamespace +from unittest import mock + +from azure.cli.core.azclierror import ( + AzureResponseError, + CLIInternalError, + InvalidArgumentValueError, + RequiredArgumentMissingError, +) +from knack.util import CLIError + +from azext_migrate.shared import arm_ids +from azext_migrate.shared import arm_client as arm_client_mod +from azext_migrate.shared import files +from azext_migrate.shared.arm_client import ArmClient +from azext_migrate.shared.constants import WAVE_OPERATIONS_API_VERSION +from azext_migrate.runbook import models, transformers +from azext_migrate.runbook import deps as deps_mod +from azext_migrate.runbook import config_status as config_status_mod +from azext_migrate.runbook.cmds import runbook as runbook_cmds +from azext_migrate.runbook.cmds import definition as definition_cmds +from azext_migrate.runbook.cmds import definition_step as step_cmds +from azext_migrate.runbook.cmds import ( + definition_workstream as workstream_cmds, +) +from azext_migrate.runbook.cmds import execution as execution_cmds +from azext_migrate.runbook.cmds import execution_step as execution_step_cmds +from azext_migrate.runbook.cmds import parameter as parameter_cmds +from azext_migrate.runbook.cmds import ( + execution_parameter as execution_parameter_cmds) +from azext_migrate.runbook.visualize import graph as visualize_graph +from azext_migrate.runbook.visualize import renderer as visualize_renderer +from azext_migrate.runbook.visualize import viewmodel as visualize_viewmodel +from azext_migrate.runbook.constants import ( + SCOPE_TYPE_WAVE, + RUNBOOK_STATUS_VALUES, +) +from azext_migrate.runbook.models import ExecutionAction +from azext_migrate.runbook.validators import ( + validate_generate, + validate_step_approve, + validate_step_complete, +) + +SUB = "00000000-0000-0000-0000-000000000000" +RG = "myRg" +PROJECT = "myProject" +RUNBOOK = "myRunbook" +ARTIFACT = "rb-art-1" +WAVE = "myWave" + + +class RunbookArmIdTests(unittest.TestCase): + + def test_migrate_project_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + self.assertEqual( + project, + f"/subscriptions/{SUB}/resourceGroups/{RG}/providers/" + f"Microsoft.Migrate/migrateProjects/{PROJECT}") + + def test_runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + self.assertEqual( + arm_ids.runbook_id(project, RUNBOOK), + f"{project}/runbooks/{RUNBOOK}") + + def test_with_api_version_no_query(self): + self.assertEqual( + arm_ids.with_api_version("/a/b", "2020-06-01-preview"), + "/a/b?api-version=2020-06-01-preview") + + def test_with_api_version_existing_query(self): + self.assertEqual( + arm_ids.with_api_version("/a/b?x=1", "2020-06-01-preview"), + "/a/b?x=1&api-version=2020-06-01-preview") + + +class RunbookModelTests(unittest.TestCase): + + def test_wave_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + self.assertEqual( + models.wave_id(project, WAVE), + f"{project}/waves/{WAVE}") + + def test_build_generate_body(self): + body = models.build_generate_body("/waves/myWave") + self.assertEqual( + body, + {"properties": {"scope": { + "scopeType": SCOPE_TYPE_WAVE, + "waveId": "/waves/myWave"}}}) + + def test_build_update_body_empty(self): + self.assertEqual( + models.build_update_body(), {"properties": {}}) + + def test_build_update_body_with_description(self): + self.assertEqual( + models.build_update_body(description="new desc"), + {"properties": {"description": "new desc"}}) + + +class RunbookTransformerTests(unittest.TestCase): + + def test_single_runbook(self): + item = {"name": RUNBOOK, "properties": {"state": "Ready"}} + row = transformers.runbook_table(item) + self.assertEqual(row["Name"], RUNBOOK) + self.assertEqual(row["State"], "Ready") + + def test_runbook_list(self): + items = [ + {"name": "r1", "properties": {"state": "Ready"}}, + {"name": "r2", "properties": {"state": "Generating"}}, + ] + rows = transformers.runbook_table(items) + self.assertEqual([r["Name"] for r in rows], ["r1", "r2"]) + + def test_missing_properties(self): + row = transformers.runbook_table({"name": "r1"}) + self.assertIsNone(row["State"]) + + +class RunbookValidatorTests(unittest.TestCase): + + def test_validate_generate_ok(self): + validate_generate(SimpleNamespace(wave_name=WAVE)) + + def test_validate_generate_missing_wave(self): + with self.assertRaises(RequiredArgumentMissingError): + validate_generate(SimpleNamespace(wave_name=None)) + + +class RunbookStatusChoiceTests(unittest.TestCase): + + def test_status_values(self): + self.assertEqual( + RUNBOOK_STATUS_VALUES, + ["Generating", "NotConfigured", "ReadyToStart", "InExecution", + "Paused", "Completed", "Failed"]) + + +def _fake_response(status_code, headers=None, body=None): + resp = mock.Mock() + resp.status_code = status_code + resp.headers = headers or {} + resp.content = b'{}' if body is not None else b'' + resp.json.return_value = body if body is not None else {} + return resp + + +def _arm_client(): + cmd = mock.Mock() + cmd.cli_ctx.cloud.endpoints.resource_manager = ( + "https://management.azure.com") + return ArmClient(cmd) + + +class ArmClientPollApiVersionTests(unittest.TestCase): + + def test_rewrites_waveoperations_api_version(self): + url = ("https://management.azure.com/subscriptions/s/providers/" + "Microsoft.Migrate/migrateProjects/p/WaveOperations/op" + "?api-version=2020-06-01-preview&c=SIG&s=SIG2") + rewritten = arm_client_mod._rewrite_poll_api_version(url) + self.assertIn( + "api-version=" + WAVE_OPERATIONS_API_VERSION, rewritten) + self.assertNotIn("api-version=2020-06-01-preview", rewritten) + # Signed token must be preserved untouched. + self.assertIn("&c=SIG&s=SIG2", rewritten) + + def test_rewrites_any_poll_url_api_version(self): + url = "https://x/operationstatus/o?api-version=2020-06-01-preview" + rewritten = arm_client_mod._rewrite_poll_api_version(url) + self.assertEqual( + rewritten, + "https://x/operationstatus/o?api-version=" + + WAVE_OPERATIONS_API_VERSION) + + def test_appends_api_version_when_missing(self): + url = "https://x/operationstatus/o" + rewritten = arm_client_mod._rewrite_poll_api_version(url) + self.assertEqual( + rewritten, + "https://x/operationstatus/o?api-version=" + + WAVE_OPERATIONS_API_VERSION) + + +class ArmClientLroTests(unittest.TestCase): + + def setUp(self): + sleep_patch = mock.patch.object(arm_client_mod._time, 'sleep') + self.addCleanup(sleep_patch.stop) + sleep_patch.start() + send_patch = mock.patch.object( + arm_client_mod, 'send_raw_request') + self.addCleanup(send_patch.stop) + self.send = send_patch.start() + + def test_delete_polls_until_succeeded(self): + async_url = ("https://management.azure.com/.../WaveOperations/op" + "?api-version=2020-06-01-preview") + accepted = _fake_response( + 201, headers={'Azure-AsyncOperation': async_url}, + body={"properties": {"state": "ExecutionSucceeded"}}) + running = _fake_response(200, body={"status": "Running"}) + done = _fake_response(200, body={"status": "Succeeded"}) + self.send.side_effect = [accepted, running, done] + + result = _arm_client().delete("/runbooks/r") + + # A completed delete renders nothing (not the stale accepted body). + self.assertIsNone(result) + # Initial DELETE + two status polls. + self.assertEqual(self.send.call_count, 3) + polled_url = self.send.call_args_list[1][0][2] + self.assertIn( + "api-version=" + WAVE_OPERATIONS_API_VERSION, polled_url) + + def test_poll_uses_async_uri_as_is_when_rewrite_disabled(self): + async_url = ("https://management.azure.com/.../operationStatuses/op" + "?api-version=2026-06-01-preview") + accepted = _fake_response( + 202, headers={'Azure-AsyncOperation': async_url}, body={}) + done = _fake_response( + 200, body={"status": "Succeeded", + "properties": {"sasUrl": "https://blob/x"}}) + self.send.side_effect = [accepted, done] + + cmd = mock.Mock() + cmd.cli_ctx.cloud.endpoints.resource_manager = ( + "https://management.azure.com") + client = ArmClient(cmd, rewrite_poll_api_version=False) + result = client.post_action( + "/artifacts/a", 'generateDownloadUrl', {}, + return_final_poll=True) + + self.assertEqual( + (result.get("properties") or {}).get("sasUrl"), + "https://blob/x") + polled_url = self.send.call_args_list[1][0][2] + self.assertIn("api-version=2026-06-01-preview", polled_url) + self.assertNotIn(WAVE_OPERATIONS_API_VERSION, polled_url) + + def test_delete_raises_on_failed_operation(self): + async_url = ("https://management.azure.com/.../WaveOperations/op" + "?api-version=2020-06-01-preview") + accepted = _fake_response( + 202, headers={'Azure-AsyncOperation': async_url}, body={}) + failed = _fake_response(200, body={ + "status": "Failed", + "error": {"code": "BadThing", "message": "it broke"}}) + self.send.side_effect = [accepted, failed] + + with self.assertRaises(AzureResponseError): + _arm_client().delete("/runbooks/r") + + def test_delete_no_wait_skips_polling(self): + async_url = "https://x/WaveOperations/op?api-version=x" + accepted = _fake_response( + 202, headers={'Azure-AsyncOperation': async_url}, body={}) + self.send.side_effect = [accepted] + + _arm_client().delete("/runbooks/r", no_wait=True) + + self.assertEqual(self.send.call_count, 1) + + +class RunbookWaitTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + runbook_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + sleep_patch = mock.patch.object(runbook_cmds.time, 'sleep') + self.addCleanup(sleep_patch.stop) + sleep_patch.start() + client_patch = mock.patch.object(runbook_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _wait(self, **kwargs): + return runbook_cmds.wait( + mock.Mock(), RG, PROJECT, RUNBOOK, **kwargs) + + def test_requires_a_condition(self): + with self.assertRaises(InvalidArgumentValueError): + self._wait() + + def test_created_returns_when_succeeded(self): + self.client.get_or_none.return_value = { + "properties": {"provisioningState": "Succeeded"}} + self.assertIsNone(self._wait(created=True)) + self.assertEqual(self.client.get_or_none.call_count, 1) + + def test_deleted_returns_when_absent(self): + self.client.get_or_none.return_value = None + self.assertIsNone(self._wait(deleted=True)) + + def test_exists_returns_when_present(self): + self.client.get_or_none.return_value = {"properties": {}} + self.assertIsNone(self._wait(exists=True)) + + def test_failed_provisioning_raises(self): + self.client.get_or_none.return_value = { + "properties": {"provisioningState": "Failed"}} + with self.assertRaises(AzureResponseError): + self._wait(created=True) + + def test_custom_condition_met(self): + self.client.get_or_none.return_value = { + "properties": {"state": "ExecutionSucceeded"}} + self.assertIsNone(self._wait( + custom="properties.state=='ExecutionSucceeded'")) + + def test_times_out_when_never_satisfied(self): + self.client.get_or_none.return_value = { + "properties": {"provisioningState": "InProgress"}} + with self.assertRaises(CLIError): + self._wait(created=True, interval=1, timeout=2) + + +class RunbookUpdateRegenerateTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + runbook_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(runbook_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_update_calls_patch_with_body(self): + self.client.patch.return_value = {"ok": True} + result = runbook_cmds.update( + mock.Mock(), RG, PROJECT, RUNBOOK, description="d") + self.assertEqual(result, {"ok": True}) + self.client.patch.assert_called_once_with( + self._runbook_id(), {"properties": {"description": "d"}}) + + def test_regenerate_deletes_then_regenerates(self): + wave_id = (arm_ids.migrate_project_id(SUB, RG, PROJECT) + + '/waves/wave-1') + self.client.get.return_value = { + "properties": {"scope": { + "scopeType": "Wave", "waveId": wave_id}}} + self.client.put.return_value = {"ok": True} + result = runbook_cmds.regenerate( + mock.Mock(), RG, PROJECT, RUNBOOK, no_wait=True) + self.assertEqual(result, {"ok": True}) + self.client.get.assert_called_once_with(self._runbook_id()) + self.client.delete.assert_called_once_with(self._runbook_id()) + self.client.put.assert_called_once_with( + self._runbook_id(), + models.build_generate_body(wave_id), + no_wait=True) + + def test_regenerate_raises_without_scope(self): + self.client.get.return_value = {"properties": {}} + with self.assertRaises(CLIError): + runbook_cmds.regenerate(mock.Mock(), RG, PROJECT, RUNBOOK) + + +class RunbookDefinitionTransformerTests(unittest.TestCase): + + def test_workstreams_flattened_to_steps(self): + definition = {"workstreams": [ + {"id": "w1", "steps": [ + {"stepId": "s1", "displayName": "Step One", + "prerequisite": [{"step": "b"}], + "dependsOn": [{"step": "a"}], + "configurationStatus": "Configured", + "entities": ["e1", "e2"]}]}, + {"id": "w2", "steps": [{"stepId": "s2"}]}, + ]} + rows = transformers.definition_table(definition) + self.assertEqual([r["Step Id"] for r in rows], ["s1", "s2"]) + self.assertEqual([r["Workstream Id"] for r in rows], ["w1", "w2"]) + self.assertEqual(rows[0]["Step Name"], "Step One") + self.assertEqual(rows[0]["Depends On"], "b\na") + self.assertEqual(rows[0]["Configuration Status"], "Configured") + self.assertEqual(rows[0]["Workloads"], 2) + self.assertEqual(rows[0]["Applications"], "-") + + def test_single_workstream(self): + rows = transformers.definition_table( + {"id": "w1", "steps": [{"id": "s1"}]}) + self.assertEqual([r["Step Id"] for r in rows], ["s1"]) + self.assertEqual(rows[0]["Workstream Id"], "w1") + + def test_single_step(self): + rows = transformers.definition_table({"stepId": "s9"}) + self.assertEqual(rows[0]["Step Id"], "s9") + + def test_empty_workstream_still_shows_a_row(self): + rows = transformers.definition_table({"workstreams": [ + {"id": "w1", "steps": [{"stepId": "s1"}]}, + {"id": "w-empty", "displayName": "Unmapped", "steps": []}, + ]}) + self.assertEqual( + [r["Workstream Id"] for r in rows], ["w1", "w-empty"]) + empty = rows[1] + self.assertEqual(empty["Step Id"], "") + self.assertEqual(empty["Step Name"], "(no steps)") + + def test_empty_definition(self): + self.assertEqual(transformers.definition_table({}), []) + + def test_parameters_document_does_not_fabricate_row(self): + # A parameters/inputs document must never be rendered as a single + # bogus step row (regression: -o table showed one empty 3-column + # row when the parameters file was mis-selected as the definition). + params = {"runbookInputs": { + "schema": {"vm.agentless.setup": {}}, + "stepInputs": {"vm.agentless.setup-1": {}}}} + self.assertEqual(transformers.definition_table(params), []) + + +class DefinitionProjectionTests(unittest.TestCase): + + def setUp(self): + self.definition = {"workstreams": [ + {"id": "w1", "steps": [{"id": "s1"}, {"stepId": "s2"}]}, + {"id": "w2", "steps": [{"id": "s3"}]}, + ]} + + def test_full_definition_returned(self): + result = definition_cmds._project_definition( + self.definition, None, None) + self.assertEqual(result, self.definition) + + def test_filter_by_workstream(self): + result = definition_cmds._project_definition( + self.definition, "w2", None) + self.assertEqual(result["id"], "w2") + + def test_filter_by_step_id(self): + result = definition_cmds._project_definition( + self.definition, None, "s2") + self.assertEqual(result["stepId"], "s2") + + def test_step_id_no_match(self): + result = definition_cmds._project_definition( + self.definition, None, "missing") + self.assertEqual(result, {}) + + def test_workstream_no_match(self): + result = definition_cmds._project_definition( + self.definition, "missing", None) + self.assertEqual(result, {}) + + +def _make_zip(members): + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, 'w') as archive: + for name, content in members.items(): + archive.writestr(name, content) + return buffer.getvalue() + + +class FilesTests(unittest.TestCase): + + def test_extract_sas_url_top_level(self): + self.assertEqual( + files.extract_sas_url({"downloadUrl": "https://x"}), + "https://x") + + def test_extract_sas_url_in_properties(self): + self.assertEqual( + files.extract_sas_url( + {"properties": {"sasUri": "https://y"}}), + "https://y") + + def test_extract_sas_url_none(self): + self.assertIsNone(files.extract_sas_url({"other": 1})) + + def test_read_spec_json_prefers_spec_suffix(self): + zip_bytes = _make_zip({ + "extra.json": '{"a": 1}', + "rb-x-spec.json": '{"runbookSpec": {"id": "r"}}', + }) + spec = files.read_spec_json(zip_bytes) + self.assertEqual(spec, {"runbookSpec": {"id": "r"}}) + + def test_read_status_json_raw_status_doc(self): + raw = json.dumps({"state": "InProgress"}).encode('utf-8') + self.assertEqual( + files.read_status_json(raw), {"state": "InProgress"}) + + def test_read_status_json_raw_rejects_parameters(self): + raw = json.dumps({"runbookInputs": {"schema": {}}}).encode('utf-8') + with self.assertRaises(CLIInternalError): + files.read_status_json(raw) + + def test_read_status_json_zip_prefers_status_member(self): + zip_bytes = _make_zip({ + "rb-x-spec.json": '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": '{"runbookInputs": {"schema": {}}}', + "status.json": '{"workstreams": [{"steps": []}]}', + }) + self.assertEqual( + files.read_status_json(zip_bytes), + {"workstreams": [{"steps": []}]}) + + def test_read_status_json_zip_inputs_only_raises(self): + # A not-yet-run execution archive (definition + parameters, no + # status) must not be mistaken for a status document. + zip_bytes = _make_zip({ + "rb-x-spec.json": '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": '{"runbookInputs": {"schema": {}}}', + "derived-input.json": '{"runbookInputs": {"schema": {}}}', + }) + with self.assertRaises(CLIInternalError): + files.read_status_json(zip_bytes) + + def test_extract_definition_files_flattens_hostile_path(self): + # Zip-slip is designed out: a member with a traversal path is + # written by its base name, staying inside the destination. + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {}}', + "../../evil.md": "# bad", + }) + with tempfile.TemporaryDirectory() as tmp: + written = files.extract_definition_files(zip_bytes, tmp) + for path in written: + self.assertEqual( + os.path.commonpath([tmp, os.path.abspath(path)]), tmp) + self.assertTrue(os.path.isfile(os.path.join(tmp, "evil.md"))) + + def test_extract_definition_files_round_trip(self): + zip_bytes = _make_zip({ + "rb-x-spec.json": '{"runbookSpec": {"workstreams": []}}', + "rb-x-input.json": '{"runbookInputs": {"a": 1}}', + "docs/readme.md": "# hello", + }) + with tempfile.TemporaryDirectory() as tmp: + written = files.extract_definition_files(zip_bytes, tmp) + self.assertEqual(len(written), 3) + for path in written: + self.assertTrue(os.path.isfile(path)) + self.assertEqual( + sorted(os.path.basename(p) for p in written), + ["rb-x-input.json", "rb-x-spec.json", "readme.md"]) + with open(os.path.join(tmp, "readme.md")) as handle: + self.assertEqual(handle.read(), "# hello") + + def test_extract_definition_files_flattens_paths(self): + zip_bytes = _make_zip({ + "../evil-spec.json": '{"runbookSpec": {}}', + "../../notes.md": "# n", + }) + with tempfile.TemporaryDirectory() as tmp: + written = files.extract_definition_files(zip_bytes, tmp) + self.assertEqual(len(written), 2) + for path in written: + self.assertTrue(os.path.isfile(path)) + self.assertEqual( + os.path.dirname(os.path.abspath(path)), + os.path.abspath(tmp)) + + def test_extract_parameters_file_by_content(self): + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {}}', + "user-inputs.json": '{"runbookInputs": {"stepInputs": {}}}', + }) + name, data = files.extract_parameters_file(zip_bytes) + self.assertEqual(name, "user-inputs.json") + self.assertIn(b"runbookInputs", data) + + def test_extract_parameters_file_none_when_only_spec(self): + zip_bytes = _make_zip({"rb-x-spec.json": '{"runbookSpec": {}}'}) + self.assertIsNone(files.extract_parameters_file(zip_bytes)) + + def test_read_spec_json_selects_spec_by_content(self): + # Real service names the members runbook.json / user-inputs.json, + # neither of which carries a -spec.json suffix. Selection must fall + # back to content so the parameters file is never returned as the + # definition. + for members in ( + {"user-inputs.json": '{"runbookInputs": {"schema": {}}}', + "runbook.json": + '{"runbookSpec": {"workstreams": []}}'}, + {"runbook.json": + '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": '{"runbookInputs": {"schema": {}}}'}): + spec = files.read_spec_json(_make_zip(members)) + self.assertIn("runbookSpec", spec) + self.assertIn("workstreams", spec["runbookSpec"]) + + def test_read_spec_json_none_when_only_parameters(self): + zip_bytes = _make_zip({ + "user-inputs.json": + '{"runbookInputs": {"stepInputs": {}}}'}) + self.assertIsNone(files.read_spec_json(zip_bytes)) + + def test_extract_parameters_selects_inputs_by_content(self): + # Mirror of the spec test: the params file must win over the spec + # regardless of member ordering or non-standard names. + for members in ( + {"runbook.json": + '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": + '{"runbookInputs": {"schema": {}}}'}, + {"user-inputs.json": + '{"runbookInputs": {"schema": {}}}', + "runbook.json": + '{"runbookSpec": {"workstreams": []}}'}): + name, data = files.extract_parameters_file(_make_zip(members)) + self.assertEqual(name, "user-inputs.json") + self.assertIn("runbookInputs", json.loads(data.decode())) + + def test_read_parameters_json_by_content(self): + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": + '{"runbookInputs": {"stepInputs": {"s1": {}}}}'}) + params = files.read_parameters_json(zip_bytes) + self.assertEqual(params, {"stepInputs": {"s1": {}}}) + + def test_parameters_excludes_derived_input(self): + # The real archive ships runbook.json (spec), user-inputs.json and + # derived-input.json (both runbookInputs-shaped). Only user-inputs + # is the parameters file; derived-input must never be selected. + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {"workstreams": []}}', + "derived-input.json": + '{"runbookInputs": {"stepInputs": {"d": {}}}}', + "user-inputs.json": + '{"runbookInputs": {"stepInputs": {"u": {}}}}'}) + name, data = files.extract_parameters_file(zip_bytes) + self.assertEqual(name, "user-inputs.json") + self.assertEqual( + files.read_parameters_json(zip_bytes), {"stepInputs": {"u": {}}}) + self.assertNotIn("derived", data.decode()) + + def test_read_spec_ignores_input_documents(self): + zip_bytes = _make_zip({ + "derived-input.json": '{"runbookInputs": {"schema": {}}}', + "user-inputs.json": '{"runbookInputs": {"schema": {}}}', + "runbook.json": + '{"runbookSpec": {"workstreams": [{"id": "w1"}]}}'}) + spec = files.read_spec_json(zip_bytes) + self.assertIn("runbookSpec", spec) + + def test_extract_definition_files_includes_inputs_not_derived(self): + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {"workstreams": []}}', + "user-inputs.json": '{"runbookInputs": {}}', + "derived-input.json": '{"runbookInputs": {}}', + "runbook.md": "# docs", + }) + with tempfile.TemporaryDirectory() as tmp: + written = files.extract_definition_files(zip_bytes, tmp) + names = sorted(os.path.basename(p) for p in written) + self.assertEqual( + names, ["runbook.json", "runbook.md", "user-inputs.json"]) + self.assertTrue( + os.path.exists(os.path.join(tmp, "user-inputs.json"))) + self.assertFalse( + os.path.exists(os.path.join(tmp, "derived-input.json"))) + + def test_read_spec_json_accepts_raw_blob(self): + raw = b'{"runbookSpec": {"workstreams": [{"id": "w1"}]}}' + self.assertEqual( + files.read_spec_json(raw), + {"runbookSpec": {"workstreams": [{"id": "w1"}]}}) + + def test_extract_parameters_file_none_for_raw_blob(self): + self.assertIsNone( + files.extract_parameters_file(b'{"runbookSpec": {}}')) + + def test_extract_definition_files_writes_raw_blob(self): + raw = b'{"runbookSpec": {"workstreams": []}}' + with tempfile.TemporaryDirectory() as tmp: + written = files.extract_definition_files(raw, tmp) + self.assertEqual( + [os.path.basename(p) for p in written], ["runbook.json"]) + with open(written[0], "rb") as handle: + self.assertEqual(handle.read(), raw) + + +class DefinitionCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + definition_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(definition_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_show_projects_runbook_spec(self): + self.client.get.return_value = { + "properties": {"artifactId": ARTIFACT}} + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + zip_bytes = _make_zip({ + "rb-x-spec.json": + '{"runbookSpec": {"workstreams": ' + '[{"id": "w1", "steps": []}]}}'}) + with mock.patch.object( + definition_cmds.files, 'download_bytes', + return_value=zip_bytes) as dl: + result = definition_cmds.show( + mock.Mock(), RG, PROJECT, RUNBOOK, workstream_id="w1") + dl.assert_called_once_with("https://blob/x") + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + self.client.post_action.assert_called_once_with( + arm_ids.artifact_id(project, ARTIFACT), 'generateDownloadUrl', + {"mode": "directory", "path": "/", "includeMetadata": True}, + return_final_poll=True) + self.assertEqual(result["id"], "w1") + + def test_show_raises_without_download_url(self): + self.client.post_action.return_value = {"expiresAt": "t"} + with self.assertRaises(CLIInternalError): + definition_cmds.show(mock.Mock(), RG, PROJECT, RUNBOOK) + + def test_show_raises_when_no_definition_in_archive(self): + self.client.get.return_value = { + "properties": {"artifactId": ARTIFACT}} + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + zip_bytes = _make_zip({"user-inputs.json": '{"runbookInputs": {}}'}) + with mock.patch.object( + definition_cmds.files, 'download_bytes', + return_value=zip_bytes): + with self.assertRaises(CLIInternalError): + definition_cmds.show(mock.Mock(), RG, PROJECT, RUNBOOK) + + def test_show_uses_full_artifact_arm_id_as_is(self): + full_id = ( + "/subscriptions/other/resourceGroups/rg2/providers" + "/Microsoft.Migrate/migrateProjects/p2/artifacts/art9") + self.client.get.return_value = { + "properties": {"artifactId": full_id}} + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + with mock.patch.object( + definition_cmds.files, 'download_bytes', + return_value=_make_zip({ + "s.json": '{"runbookSpec": {"workstreams": []}}'})): + definition_cmds.show(mock.Mock(), RG, PROJECT, RUNBOOK) + called_id = self.client.post_action.call_args[0][0] + self.assertEqual(called_id, full_id) + + def test_download_writes_files(self): + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + with mock.patch.object( + definition_cmds.files, 'download_bytes', + return_value=b'zip'), \ + mock.patch.object( + definition_cmds.files, 'extract_definition_files', + return_value=["/tmp/runbook.json", "/tmp/readme.md"]) as ex: + result = definition_cmds.download( + mock.Mock(), RG, PROJECT, RUNBOOK, destination="/tmp") + ex.assert_called_once_with(b'zip', "/tmp") + self.assertEqual(result, [ + {"kind": "definition", "path": "/tmp/runbook.json"}, + {"kind": "documentation", "path": "/tmp/readme.md"}, + ]) + + +class StepModelTests(unittest.TestCase): + + def test_build_add_step_body_manual(self): + body = models.build_add_step_body("Manual", "Step 1", "ws1") + self.assertEqual(body, { + "workstreamId": "ws1", + "displayName": "Step 1", + "description": "", + "stepRef": "common.manual", + "dependsOn": [], + }) + + def test_build_add_step_body_approval(self): + body = models.build_add_step_body( + "Approval", "Approve", "ws1", + depends_on=["s0"], step_description="desc", + migration_entity_ids=["e1", "e2"]) + self.assertEqual(body, { + "workstreamId": "ws1", + "displayName": "Approve", + "description": "desc", + "stepRef": "common.approval", + "dependsOn": [{"mode": "Step", "stepId": "s0"}], + "migrationEntityIds": ["e1", "e2"], + }) + + def test_build_update_step_body_minimal(self): + self.assertEqual( + models.build_update_step_body("s1"), {"stepId": "s1"}) + + def test_build_update_step_body_full(self): + body = models.build_update_step_body( + "s1", step_name="New", step_description="d", + depends_on=["s0"]) + self.assertEqual(body, { + "stepId": "s1", "displayName": "New", + "description": "d", + "dependsOn": [{"mode": "Step", "stepId": "s0"}]}) + + def test_build_delete_step_body(self): + self.assertEqual( + models.build_delete_step_body("s1"), {"stepId": "s1"}) + + def test_build_split_workstream_body(self): + body = models.build_split_workstream_body( + "ws1", "new", ["e1", "e2"]) + self.assertEqual(body, { + "sourceWorkstreamId": "ws1", + "stepIds": ["e1", "e2"], + "newWorkstreamName": "new"}) + + def test_build_merge_workstreams_body(self): + body = models.build_merge_workstreams_body(["w1", "w2"], "merged") + self.assertEqual(body, { + "workstreamIds": ["w1", "w2"], + "newWorkstreamName": "merged"}) + + def test_build_merge_workstreams_body_requires_name(self): + with self.assertRaises(TypeError): + models.build_merge_workstreams_body(["w1", "w2"]) + + +class StepCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + definition_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(step_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_add_posts_add_step(self): + self.client.post_action.return_value = {"ok": True} + result = step_cmds.add( + mock.Mock(), RG, PROJECT, RUNBOOK, "Manual", "Step 1", "ws1") + self.assertEqual(result, {"ok": True}) + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'AddStep', + models.build_add_step_body("Manual", "Step 1", "ws1")) + + def test_update_posts_update_step(self): + self.client.post_action.return_value = {"ok": True} + step_cmds.update( + mock.Mock(), RG, PROJECT, RUNBOOK, "s1", step_name="New") + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'UpdateStep', + {"stepId": "s1", "displayName": "New"}) + + def test_remove_posts_delete_step(self): + self.client.post_action.return_value = {"ok": True} + step_cmds.remove(mock.Mock(), RG, PROJECT, RUNBOOK, "s1") + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'DeleteStep', {"stepId": "s1"}) + + +class WorkstreamCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + definition_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(workstream_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_split_posts_split_workstream(self): + self.client.post_action.return_value = {"ok": True} + workstream_cmds.split( + mock.Mock(), RG, PROJECT, RUNBOOK, "ws1", "new", ["e1"]) + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'SplitWorkstream', + models.build_split_workstream_body("ws1", "new", ["e1"])) + + def test_merge_posts_merge_workstreams(self): + self.client.post_action.return_value = {"ok": True} + workstream_cmds.merge( + mock.Mock(), RG, PROJECT, RUNBOOK, ["w1", "w2"], "merged") + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'MergeWorkstreams', + {"workstreamIds": ["w1", "w2"], + "newWorkstreamName": "merged"}) + + +class ExecutionModelTests(unittest.TestCase): + + def test_start_body_is_empty_properties(self): + self.assertEqual( + models.build_start_execution_body(), {"properties": {}}) + + def test_build_artifact_download_url_body(self): + self.assertEqual( + models.build_artifact_download_url_body(), + {"mode": "file", "path": "runbook.json", + "includeMetadata": True}) + self.assertEqual( + models.build_artifact_download_url_body( + path="reports/r.xlsx", mode="directory", + include_metadata=False), + {"mode": "directory", "path": "reports/r.xlsx", + "includeMetadata": False}) + + def test_action_enum_values(self): + self.assertEqual(ExecutionAction.START.value, "Start") + self.assertEqual(ExecutionAction.PAUSE.value, "Pause") + self.assertEqual(ExecutionAction.RESUME.value, "Resume") + self.assertEqual(ExecutionAction.CANCEL.value, "Cancel") + self.assertEqual(ExecutionAction.RETRY.value, "Retry") + + def test_perform_action_body_shape(self): + body = models.build_perform_action_body(ExecutionAction.PAUSE) + self.assertEqual( + body, + {"action": "Pause", "targetId": "", + "migrationEntityIds": []}) + self.assertIsInstance(body["action"], str) + + def test_perform_action_body_with_target(self): + body = models.build_perform_action_body( + ExecutionAction.RESUME, target_id="t1", entity_ids=["e1"]) + self.assertEqual( + body, + {"action": "Resume", "targetId": "t1", + "migrationEntityIds": ["e1"]}) + + +class ExecutionTransformerTests(unittest.TestCase): + + def test_flattens_workstream_steps(self): + result = { + "workstreams": [ + {"steps": [ + {"id": "s1", "displayName": "Step 1", + "status": "Running", + "workloadProgress": "1/2"}, + ]}, + ], + } + rows = transformers.execution_table(result) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["Step Id"], "s1") + self.assertEqual(rows[0]["Step Status"], "Running") + self.assertEqual(rows[0]["Workload Progress"], "1/2") + + def test_unwraps_properties(self): + result = { + "properties": { + "steps": [ + {"stepId": "s2", "stepName": "Step 2", + "stepStatus": "Succeeded"}, + ], + }, + } + rows = transformers.execution_table(result) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["Step Id"], "s2") + self.assertEqual(rows[0]["Step Status"], "Succeeded") + + +class ExecutionsListTransformerTests(unittest.TestCase): + + def test_lists_one_row_per_execution(self): + result = [ + {"name": "e1", "properties": { + "status": "Completed", + "provisioningState": "Succeeded", + "startTime": "2026-01-01T10:00:00Z", + "endTime": "2026-01-01T10:05:00Z"}}, + {"name": "e2", "properties": {"status": "InProgress"}}, + ] + rows = transformers.executions_table(result) + self.assertEqual(len(rows), 2) + self.assertEqual(rows[0]["Name"], "e1") + self.assertEqual(rows[0]["Status"], "Completed") + self.assertEqual(rows[0]["ProvisioningState"], "Succeeded") + self.assertEqual(rows[0]["StartTime"], "2026-01-01T10:00:00Z") + self.assertEqual(rows[0]["EndTime"], "2026-01-01T10:05:00Z") + self.assertEqual(rows[1]["Name"], "e2") + self.assertEqual(rows[1]["Status"], "InProgress") + + def test_single_execution_dict(self): + rows = transformers.executions_table( + {"name": "e9", "properties": {"state": "Queued"}}) + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["Name"], "e9") + self.assertEqual(rows[0]["Status"], "Queued") + + +class ExecutionCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + execution_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(execution_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_start_posts_execute_action(self): + self.client.post_action.return_value = {"ok": True} + result = execution_cmds.start(mock.Mock(), RG, PROJECT, RUNBOOK) + self.assertEqual(result, {"ok": True}) + self.client.post_action.assert_called_once_with( + self._runbook_id(), 'execute', {"properties": {}}, + no_wait=False) + + def test_start_no_wait(self): + self.client.post_action.return_value = {"ok": True} + execution_cmds.start( + mock.Mock(), RG, PROJECT, RUNBOOK, no_wait=True) + _, kwargs = self.client.post_action.call_args + self.assertTrue(kwargs.get('no_wait')) + + def test_start_final_get_reads_execution(self): + # After the execute action settles, start re-reads the created + # execution resource so the caller sees the latest status rather + # than the initial (stale) accepted body. + self.client.post_action.return_value = {"name": "e5"} + fresh = {"name": "e5", "properties": {"status": "InProgress"}} + self.client.get.return_value = fresh + result = execution_cmds.start(mock.Mock(), RG, PROJECT, RUNBOOK) + self.assertEqual(result, fresh) + self.client.get.assert_called_once_with( + arm_ids.execution_id(self._runbook_id(), "e5")) + + def test_start_no_wait_skips_final_get(self): + self.client.post_action.return_value = {"name": "e5"} + execution_cmds.start( + mock.Mock(), RG, PROJECT, RUNBOOK, no_wait=True) + self.client.get.assert_not_called() + + def test_list_calls_executions_collection(self): + self.client.list.return_value = [] + execution_cmds.list_(mock.Mock(), RG, PROJECT, RUNBOOK) + self.client.list.assert_called_once_with( + self._runbook_id() + '/executions') + + def test_show_returns_execution(self): + self.client.post_action.return_value = {"downloadUrl": "https://b/x"} + status = {"state": "InProgress", "id": "e1"} + with mock.patch.object( + execution_cmds.files, 'download_bytes', + return_value=json.dumps(status).encode('utf-8')) as dl: + result = execution_cmds.show( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1") + self.assertEqual(result, status) + self.client.post_action.assert_called_once_with( + arm_ids.execution_id(self._runbook_id(), "e1"), + 'GenerateDownloadUrl') + dl.assert_called_once_with("https://b/x") + + def test_show_projects_step(self): + status = { + "workstreams": [ + {"steps": [{"id": "s1"}, {"id": "s2"}]}, + ], + } + self.client.post_action.return_value = {"downloadUrl": "https://b/x"} + with mock.patch.object( + execution_cmds.files, 'download_bytes', + return_value=json.dumps(status).encode('utf-8')): + result = execution_cmds.show( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", step_id="s2") + self.assertEqual(result, {"id": "s2"}) + + def test_show_raises_for_inputs_only_archive(self): + # A not-yet-run execution's download archive contains only the input + # parameters (no status.json); show must raise rather than return the + # inputs blob as if it were a status document. + self.client.post_action.return_value = {"downloadUrl": "https://b/x"} + inputs_zip = _make_zip({ + "user-inputs.json": '{"runbookInputs": {"schema": {}}}'}) + with mock.patch.object( + execution_cmds.files, 'download_bytes', + return_value=inputs_zip): + with self.assertRaises(CLIInternalError): + execution_cmds.show( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1") + self.client.get.assert_not_called() + + def test_pause_posts_perform_action(self): + self.client.post_action.return_value = {"ok": True} + execution_cmds.pause(mock.Mock(), RG, PROJECT, RUNBOOK, "e1") + self.client.post_action.assert_called_once_with( + arm_ids.execution_id(self._runbook_id(), "e1"), + 'PerformAction', + {"action": "Pause", "targetId": "", + "migrationEntityIds": []}) + + def test_resume_posts_perform_action(self): + self.client.post_action.return_value = {"ok": True} + execution_cmds.resume(mock.Mock(), RG, PROJECT, RUNBOOK, "e1") + _, args, _ = self.client.post_action.mock_calls[0] + self.assertEqual(args[2]["action"], "Resume") + + def test_cancel_posts_perform_action(self): + self.client.post_action.return_value = {"ok": True} + execution_cmds.cancel(mock.Mock(), RG, PROJECT, RUNBOOK, "e1") + _, args, _ = self.client.post_action.mock_calls[0] + self.assertEqual(args[2]["action"], "Cancel") + + +class ExecutionStepModelTests(unittest.TestCase): + + def test_build_retry_step_body(self): + body = models.build_retry_step_body("step1") + self.assertEqual(body, { + "action": "Retry", "targetId": "step1", + "migrationEntityIds": []}) + self.assertIsInstance(body["action"], str) + + def test_build_approve_step_body_full(self): + body = models.build_approve_step_body("step1") + self.assertEqual(body, { + "action": "Approve", "targetId": "step1", + "migrationEntityIds": []}) + + def test_build_approve_step_body_partial(self): + body = models.build_approve_step_body( + "step1", entity_ids=["e1", "e2"]) + self.assertEqual(body["migrationEntityIds"], ["e1", "e2"]) + + def test_build_complete_step_body(self): + body = models.build_complete_step_body("step1", "done") + self.assertEqual(body, { + "action": "Complete", "targetId": "step1", + "migrationEntityIds": [], "comment": "done"}) + + +class ExecutionStepValidatorTests(unittest.TestCase): + + def test_approve_full_ok(self): + validate_step_approve( + SimpleNamespace(entities=None, all_ready=False)) + + def test_approve_entities_ok(self): + validate_step_approve( + SimpleNamespace(entities=["e1"], all_ready=False)) + + def test_approve_all_ready_ok(self): + validate_step_approve( + SimpleNamespace(entities=None, all_ready=True)) + + def test_approve_entities_and_all_ready_rejected(self): + with self.assertRaises(InvalidArgumentValueError): + validate_step_approve( + SimpleNamespace(entities=["e1"], all_ready=True)) + + def test_complete_requires_comment(self): + with self.assertRaises(RequiredArgumentMissingError): + validate_step_complete(SimpleNamespace(comment=None)) + + def test_complete_ok_with_comment(self): + validate_step_complete(SimpleNamespace(comment="done")) + + +class ExecutionStepCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + execution_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(execution_step_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _execution_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + runbook = arm_ids.runbook_id(project, RUNBOOK) + return arm_ids.execution_id(runbook, "e1") + + def test_retry_posts_perform_action(self): + self.client.post_action.return_value = {"ok": True} + result = execution_step_cmds.retry( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", "step1") + self.assertEqual(result, {"ok": True}) + self.client.post_action.assert_called_once_with( + self._execution_id(), 'PerformAction', + {"action": "Retry", "targetId": "step1", + "migrationEntityIds": []}) + + def test_approve_posts_provide_approval(self): + self.client.post_action.return_value = {"ok": True} + execution_step_cmds.approve( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", "step1", + entities=["ent1"]) + self.client.post_action.assert_called_once_with( + self._execution_id(), 'ProvideApproval', + {"action": "Approve", "targetId": "step1", + "migrationEntityIds": ["ent1"]}) + + def test_complete_posts_update_step_status(self): + self.client.post_action.return_value = {"ok": True} + execution_step_cmds.complete( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", "step1", "done") + self.client.post_action.assert_called_once_with( + self._execution_id(), 'UpdateStepStatus', + {"action": "Complete", "targetId": "step1", + "migrationEntityIds": [], "comment": "done"}) + + +class ParameterCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + definition_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object(definition_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _runbook_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + return arm_ids.runbook_id(project, RUNBOOK) + + def test_download_writes_parameters_file(self): + self.client.get.return_value = { + "properties": {"artifactId": ARTIFACT}} + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + zip_bytes = _make_zip({ + "runbook.json": '{"runbookSpec": {}}', + "user-inputs.json": '{"runbookInputs": {"stepInputs": {}}}', + }) + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object( + parameter_cmds.files, 'download_bytes', + return_value=zip_bytes) as dl: + result = parameter_cmds.download( + mock.Mock(), RG, PROJECT, RUNBOOK, file=tmp) + dl.assert_called_once_with("https://blob/x") + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + self.client.post_action.assert_called_once_with( + arm_ids.artifact_id(project, ARTIFACT), + 'generateDownloadUrl', + {"mode": "directory", "path": "/", + "includeMetadata": True}, + return_final_poll=True) + expected = os.path.join(tmp, "user-inputs.json") + self.assertEqual(result, {"path": expected}) + with open(expected) as handle: + self.assertEqual( + handle.read(), '{"runbookInputs": {"stepInputs": {}}}') + + def test_download_writes_raw_input_blob(self): + self.client.get.return_value = { + "properties": {"artifactId": ARTIFACT}} + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + raw = b'{"runbookInputs": {"stepInputs": {}}}' + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object( + parameter_cmds.files, 'download_bytes', + return_value=raw): + result = parameter_cmds.download( + mock.Mock(), RG, PROJECT, RUNBOOK, file=tmp) + expected = os.path.join(tmp, "input.json") + self.assertEqual(result, {"path": expected}) + with open(expected, "rb") as handle: + self.assertEqual(handle.read(), raw) + + +class ExecutionParameterCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + execution_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + client_patch = mock.patch.object( + execution_parameter_cmds, 'ArmClient') + self.addCleanup(client_patch.stop) + self.client = client_patch.start().return_value + + def _execution_id(self): + project = arm_ids.migrate_project_id(SUB, RG, PROJECT) + runbook = arm_ids.runbook_id(project, RUNBOOK) + return arm_ids.execution_id(runbook, "e1") + + def test_download_writes_input_file(self): + self.client.post_action.return_value = { + "downloadUrl": "https://blob/x"} + raw = b'{"runbookInputs": {"stepInputs": {}}}' + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object( + execution_parameter_cmds.files, 'download_bytes', + return_value=raw) as dl: + result = execution_parameter_cmds.download( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", file=tmp) + dl.assert_called_once_with("https://blob/x") + self.client.post_action.assert_called_once_with( + self._execution_id(), 'GenerateInputDownloadUrl') + expected = os.path.join(tmp, "input.json") + self.assertEqual(result, {"path": expected}) + with open(expected, "rb") as handle: + self.assertEqual(handle.read(), raw) + + def test_upload_puts_input_file(self): + self.client.post_action.return_value = { + "uploadUrl": "https://blob/u"} + with tempfile.TemporaryDirectory() as tmp: + src = os.path.join(tmp, "input.json") + with open(src, "wb") as handle: + handle.write(b'{"runbookInputs": {}}') + with mock.patch.object( + execution_parameter_cmds.files, 'upload_bytes') as up: + result = execution_parameter_cmds.upload( + mock.Mock(), RG, PROJECT, RUNBOOK, "e1", src) + self.client.post_action.assert_called_once_with( + self._execution_id(), 'GenerateInputUploadUrl') + up.assert_called_once_with( + "https://blob/u", b'{"runbookInputs": {}}') + self.assertEqual(result, {"status": "uploaded"}) + + +_DEFINITION_DOC = { + "workstreams": [ + { + "id": "ws1", + "displayName": "Web tier", + "steps": [ + {"id": "s1", "displayName": "Prepare", "dependsOn": []}, + {"id": "s2", "displayName": "Migrate", + "dependsOn": [{"stepId": "s1"}]}, + {"id": "s3", "displayName": "Cutover", + "dependsOn": ["s1", "s2"]}, + ], + } + ] +} + + +# Mirrors the execution status.json shape: top-level state/workstreams, +# steps keyed by stepId/displayName/state, dependsOn as objects with a +# "step" key, and per-entity progress under entityExecutions. +_STATUS_DOC = { + "state": "InProgress", + "workstreams": [ + { + "id": "ws1", + "displayName": "Web tier", + "steps": [ + {"stepId": "setup", "displayName": "Setup", + "state": "Completed", "dependsOn": []}, + {"stepId": "network", "displayName": "Network", + "state": "Failed", "dependsOn": []}, + {"stepId": "dataSync", "displayName": "Data sync", + "state": "InProgress", + "dependsOn": [ + {"step": "setup", "mode": "step"}, + {"step": "network", "mode": "perEntity"}, + ], + "entityExecutions": [ + {"entityId": "e1", "state": "Completed"}, + {"entityId": "e2", "state": "InProgress"}, + ]}, + {"stepId": "cutover", "displayName": "Cutover", + "state": "Blocked", + "dependsOn": [{"step": "dataSync", "mode": "step"}]}, + ], + } + ], +} + + +class ExecutionStatusParsingTests(unittest.TestCase): + + def test_read_status_json_raw_bytes(self): + parsed = files.read_status_json( + json.dumps(_STATUS_DOC).encode('utf-8')) + self.assertEqual(parsed["state"], "InProgress") + + def test_read_status_json_from_zip(self): + zip_bytes = _make_zip({"status.json": json.dumps(_STATUS_DOC)}) + parsed = files.read_status_json(zip_bytes) + self.assertEqual(parsed["state"], "InProgress") + + def test_execution_table_renders_without_crash(self): + rows = transformers.execution_table(_STATUS_DOC) + by_id = {row['Step Id']: row for row in rows} + self.assertEqual(by_id['setup']['Step Status'], 'Completed') + self.assertEqual(by_id['network']['Step Status'], 'Failed') + self.assertEqual(by_id['cutover']['Step Status'], 'Blocked') + + def test_execution_table_formats_depends_on(self): + rows = transformers.execution_table(_STATUS_DOC) + by_id = {row['Step Id']: row for row in rows} + self.assertEqual( + by_id['dataSync']['Depends On'], + 'Web tier:Setup\nWeb tier:Network') + self.assertEqual(by_id['setup']['Depends On'], '') + + def test_execution_table_workload_progress(self): + rows = transformers.execution_table(_STATUS_DOC) + by_id = {row['Step Id']: row for row in rows} + self.assertEqual( + by_id['dataSync']['Workload Progress'], '1/2 completed') + self.assertIsNone(by_id['setup']['Workload Progress']) + + def test_execution_graph_edges_from_step_key(self): + graph = visualize_graph.build_execution_graph(_STATUS_DOC) + edges = {(e.source, e.target) for e in graph.edges} + self.assertIn(('setup', 'dataSync'), edges) + self.assertIn(('network', 'dataSync'), edges) + self.assertIn(('dataSync', 'cutover'), edges) + by_id = {n.id: n for n in graph.nodes} + self.assertEqual(by_id['network'].status, 'Failed') + self.assertEqual(by_id['cutover'].status, 'Blocked') + + +class VisualizeGraphTests(unittest.TestCase): + + def test_build_definition_graph_nodes_and_edges(self): + graph = visualize_graph.build_definition_graph(_DEFINITION_DOC) + self.assertEqual({n.id for n in graph.nodes}, {"s1", "s2", "s3"}) + self.assertEqual(len(graph.edges), 3) + by_id = {n.id: n for n in graph.nodes} + self.assertEqual(by_id["s1"].name, "Prepare") + self.assertEqual(by_id["s1"].group, "Web tier") + + def test_topological_layering(self): + graph = visualize_graph.build_definition_graph(_DEFINITION_DOC) + layer = {n.id: n.layer for n in graph.nodes} + self.assertEqual(layer["s1"], 0) + self.assertEqual(layer["s2"], 1) + self.assertEqual(layer["s3"], 2) + + def test_cycle_detection_raises(self): + doc = {"steps": [ + {"id": "a", "dependsOn": ["b"]}, + {"id": "b", "dependsOn": ["a"]}, + ]} + with self.assertRaises(InvalidArgumentValueError): + visualize_graph.build_definition_graph(doc) + + def test_dangling_dependency_is_dropped(self): + doc = {"steps": [ + {"id": "a", "dependsOn": ["missing"]}, + ]} + graph = visualize_graph.build_definition_graph(doc) + self.assertEqual(len(graph.nodes), 1) + self.assertEqual(graph.edges, []) + self.assertEqual(graph.nodes[0].layer, 0) + + def test_execution_graph_carries_status(self): + doc = {"properties": {"steps": [ + {"id": "a", "displayName": "A", "status": "Succeeded"}, + {"id": "b", "displayName": "B", "status": "Running", + "dependsOn": ["a"]}, + ]}} + graph = visualize_graph.build_execution_graph(doc) + by_id = {n.id: n for n in graph.nodes} + self.assertEqual(by_id["a"].status, "Succeeded") + self.assertEqual(by_id["b"].status, "Running") + + +class VisualizeRendererTests(unittest.TestCase): + + def test_escapes_malicious_step_name(self): + doc = {"steps": [ + {"id": "s1", "displayName": ""}, + ]} + graph = visualize_graph.build_definition_graph(doc) + html_text = visualize_renderer.render(graph) + self.assertNotIn("", html_text) + self.assertIn("<script>alert(1)</script>", html_text) + + def test_output_is_self_contained(self): + graph = visualize_graph.build_definition_graph(_DEFINITION_DOC) + html_text = visualize_renderer.render(graph) + self.assertNotIn("http://", html_text) + self.assertNotIn("https://", html_text) + self.assertNotIn("src=", html_text) + self.assertIn("', html_text) + # The auto-reload must stay offline (no URL to fetch). + self.assertNotIn("http://", html_text) + self.assertNotIn("https://", html_text) + + def test_auto_reload_omitted_for_non_positive_interval(self): + graph = visualize_graph.build_definition_graph(_DEFINITION_DOC) + for value in (0, -1, None, "x"): + html_text = visualize_renderer.render( + graph, refresh_interval=value) + self.assertNotIn("http-equiv", html_text) + + +class VisualizeCommandTests(unittest.TestCase): + + def setUp(self): + sub_patch = mock.patch.object( + definition_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(sub_patch.stop) + sub_patch.start() + exec_sub_patch = mock.patch.object( + execution_cmds, 'get_subscription_id', return_value=SUB) + self.addCleanup(exec_sub_patch.stop) + exec_sub_patch.start() + + def test_definition_visualize_writes_html(self): + zip_bytes = _make_zip({ + "rb-x-spec.json": json.dumps( + {"runbookSpec": _DEFINITION_DOC}), + }) + with tempfile.TemporaryDirectory() as tmp: + with mock.patch.object(definition_cmds, 'ArmClient') as client, \ + mock.patch.object( + definition_cmds.files, 'download_bytes', + return_value=zip_bytes): + client.return_value.post_action.return_value = { + "downloadUrl": "https://blob/x"} + result = definition_cmds.visualize( + mock.Mock(), RG, PROJECT, RUNBOOK, file=tmp) + path = result['path'] + self.assertTrue(os.path.isfile(path)) + with open(path, encoding='utf-8') as handle: + self.assertIn("' + 'workstream-0 (1)', html_text) + self.assertIn( + 'Workstream: waveapp' + '' + 'workstream-1 (3)', html_text) + self.assertIn("NotConfigured", html_text) + self.assertIn('data-view="grid"', html_text) + self.assertIn('data-view="diagram"', html_text) + self.assertNotIn("http://", html_text) + self.assertNotIn("https://", html_text) + self.assertNotIn("src=", html_text) + + def test_definition_grid_has_portal_columns_and_stepref(self): + view = self._definition_view() + graph = visualize_graph.build_definition_graph( + _REAL_DEFINITION, title="Def") + html_text = visualize_renderer.render(graph, view=view) + # Column header row matches the portal grid titles. + self.assertIn('class="grid__head"', html_text) + self.assertIn(">Steps<", html_text) + self.assertIn(">Configuration status<", html_text) + self.assertIn(">Step dependency<", html_text) + self.assertIn(">Entities<", html_text) + # Steps render as grid rows carrying the stepRef badge. + self.assertIn('class="row__ref"', html_text) + self.assertIn("vm.agentless.setup", html_text) + + def test_definition_metadata_header_renders(self): + document = { + "runbookResourceId": "/subscriptions/s/rb/testrunbook", + "metadata": { + "waveId": "/subscriptions/s/waves/testwave", + "generatedAt": "2026-07-25T06:51:42.6972548Z", + }, + "stepLibraryVersions": {"vm.agentless": "1.0"}, + "workstreams": [ + {"id": "w0", "displayName": "Init", "steps": [ + {"stepId": "s1", "displayName": "Setup"}]}, + ], + } + view = visualize_viewmodel.build_definition_view( + document, title="Def") + graph = visualize_graph.build_definition_graph(document, title="Def") + html_text = visualize_renderer.render(graph, view=view) + self.assertIn('class="tab-meta"', html_text) + self.assertIn("Runbook version", html_text) + self.assertIn("vm.agentless 1.0", html_text) + self.assertIn("Runbook resource id", html_text) + self.assertIn("/subscriptions/s/rb/testrunbook", html_text) + self.assertIn("Wave id", html_text) + # generatedAt drives the header timestamp (normalised, offline) + # and also appears as a dedicated metadata field. + self.assertIn("2026-07-25 06:51:42 UTC", html_text) + self.assertIn(">Generated<", html_text) + + def test_definition_rows_open_detail_drawer(self): + document = { + "runbookResourceId": "/subscriptions/s/rb/tr", + "metadata": {"generatedAt": "2026-01-01T00:00:00Z"}, + "entities": [{"id": "e1", "displayName": "VM-App01"}], + "workstreams": [ + {"id": "w0", "displayName": "Init", "steps": [ + {"stepId": "s1", "displayName": "Prepare", + "stepRef": "vm.prep"}, + {"stepId": "s2", "displayName": "Migrate", + "stepRef": "vm.migrate", "entities": ["e1"], + "prerequisite": [{"step": "s1", "mode": "Blocking"}], + "dependsOn": [{"step": "s1", "mode": "Soft"}]}]}, + ], + } + view = visualize_viewmodel.build_definition_view( + document, title="Def") + graph = visualize_graph.build_definition_graph(document, title="Def") + html_text = visualize_renderer.render(graph, view=view) + # Rows are keyboard-accessible buttons wired to hidden detail blocks. + self.assertIn('data-step="0"', html_text) + self.assertIn('data-step="1"', html_text) + self.assertIn('role="button"', html_text) + self.assertIn('id="detail-1"', html_text) + self.assertIn('class="drawer"', html_text) + # The detail block carries the step's full context. + self.assertIn("Step type", html_text) + self.assertIn("vm.migrate", html_text) + self.assertIn("VM-App01", html_text) + self.assertIn("Prepare (Blocking)", html_text) + self.assertIn("Prepare (Soft)", html_text) + # Everything stays offline/self-contained. + self.assertNotIn("https://", html_text) + self.assertNotIn("http://", html_text) + + def test_definition_renders_brand_bar_and_cli_help(self): + document = { + "workstreams": [ + {"id": "w0", "displayName": "Init", "steps": [ + {"stepId": "s1", "displayName": "Setup"}]}, + ], + } + view = visualize_viewmodel.build_definition_view( + document, title="Def") + graph = visualize_graph.build_definition_graph(document, title="Def") + html_text = visualize_renderer.render(graph, view=view) + self.assertIn("Azure Migrate Runbook Viewer", html_text) + self.assertIn('class="how-bar"', html_text) + self.assertIn( + "az migrate runbook execution start", html_text) + self.assertIn( + "az migrate runbook definition step add", html_text) + + def test_diagram_uses_workstream_swimlanes(self): + graph = visualize_graph.build_definition_graph( + _REAL_DEFINITION, title="Def") + html_text = visualize_renderer.render(graph) + # The SVG groups steps into labelled workstream bands and keeps + # the dependency edges + per-step info (stepRef sub-label). + self.assertIn('class="lane"', html_text) + self.assertIn( + 'Workstream: Initialization' + ' workstream-0 (1)', html_text) + self.assertIn( + 'Workstream: waveapp' + ' workstream-1 (3)', html_text) + self.assertIn('class="edge"', html_text) + self.assertIn("vm.agentless.migration", html_text) + + def test_diagram_swimlanes_follow_document_order(self): + # Even when dependency-layer/id sorting would reverse them, the + # diagram bands must follow the source workstream order so the + # diagram is not reversed relative to the grid. Here both steps are + # layer 0, and the second workstream's step id ('aaa') sorts before + # the first ('zzz'); band order must still be Setup then Cleanup. + document = { + "workstreams": [ + {"id": "w0", "displayName": "Setup", "steps": [ + {"stepId": "zzz", "displayName": "Prepare"}]}, + {"id": "w1", "displayName": "Cleanup", "steps": [ + {"stepId": "aaa", "displayName": "Cleanup step"}]}, + ], + } + graph = visualize_graph.build_definition_graph(document, title="Def") + html_text = visualize_renderer.render(graph) + setup_at = html_text.find('Workstream: Setup') + cleanup_at = html_text.find('Workstream: Cleanup') + self.assertNotEqual(setup_at, -1) + self.assertNotEqual(cleanup_at, -1) + self.assertLess(setup_at, cleanup_at) + + def test_execution_grid_shows_progress_and_groups(self): + view = visualize_viewmodel.build_execution_view( + _STATUS_DOC, title="Exec") + graph = visualize_graph.build_execution_graph( + _STATUS_DOC, title="Exec") + html_text = visualize_renderer.render(graph, view=view) + self.assertIn( + 'Workstream: Web tier' + '' + 'ws1 (4)', html_text) + self.assertIn("1/2 completed", html_text) + self.assertNotIn("https://", html_text) + + +class VisualizeWorkstreamIdTests(unittest.TestCase): + """The visualize output must surface the workstream id so users can + copy it into ``runbook split`` / ``runbook merge``.""" + + def _render(self, document): + from azext_migrate.runbook.visualize import ( + graph as graph_mod, + renderer as renderer_mod, + viewmodel as viewmodel_mod, + ) + graph = graph_mod.build_definition_graph(document) + view = viewmodel_mod.build_definition_view(document, 't') + return renderer_mod.render(graph, view) + + def test_workstream_id_in_visualize_header(self): + document = {'workstreams': [{ + 'id': 'ws-abc123', + 'displayName': 'Migration', + 'steps': [{'id': 's1', 'displayName': 'Cutover', + 'configurationStatus': 'Configured'}], + }]} + html = self._render(document) + # Grid header shows the workstream id as a greyish badge. + self.assertIn( + 'ws-abc123', + html) + # SVG band shows the same id as a muted tspan. + self.assertIn(' ws-abc123', html) + + def test_workstream_id_and_name_are_html_escaped(self): + document = {'workstreams': [{ + 'id': '', + 'displayName': 'ws', + 'steps': [{'id': 's1', 'displayName': 'a'}], + }]} + html = self._render(document) + self.assertNotIn('', html) + self.assertIn('<script>x</script>', html) + + +class _RecordingGroup: + """Minimal stand-in for an Azure CLI command group context manager.""" + + def __init__(self, name, recorded): + self._name = name + self._recorded = recorded + + def __enter__(self): + return self + + def __exit__(self, *_args): + return False + + def custom_command(self, name, *_args, **_kwargs): + self._recorded.append('%s %s' % (self._name, name)) + + def custom_show_command(self, name, *_args, **_kwargs): + self._recorded.append('%s %s' % (self._name, name)) + + +class _RecordingLoader: + def __init__(self): + self.recorded = [] + + def command_group(self, name, **_kwargs): + return _RecordingGroup(name, self.recorded) + + +class CommandRegistrationTests(unittest.TestCase): + def _registered_commands(self): + from azext_migrate.runbook import commands as runbook_commands + loader = _RecordingLoader() + runbook_commands.load_runbook_command_table(loader) + return loader.recorded + + def test_visualize_and_step_commands_registered(self): + commands = self._registered_commands() + for expected in ( + 'migrate runbook definition visualize', + 'migrate runbook execution visualize', + 'migrate runbook execution step retry', + 'migrate runbook execution step approve', + 'migrate runbook execution step complete', + 'migrate runbook parameter download', + 'migrate runbook parameter upload', + 'migrate runbook execution parameter download', + 'migrate runbook execution parameter upload'): + self.assertIn(expected, commands) + + +if __name__ == '__main__': + unittest.main() diff --git a/src/migrate/docs/runbook-cli-design.md b/src/migrate/docs/runbook-cli-design.md new file mode 100644 index 00000000000..930c81bfc11 --- /dev/null +++ b/src/migrate/docs/runbook-cli-design.md @@ -0,0 +1,1118 @@ +# Azure Migrate Runbooks CLI — Implementation & Architecture Plan + +> Status: **Design only. No implementation code is produced in this phase.** +> Scope boundary: **all new code lives under `src/migrate/azext_migrate/`.** +> `k8s-extension` is studied as a *reference pattern only* — nothing is imported from it. + +--- + +## 0. Agent execution prerequisites (read first) + +This section makes the plan runnable by an autonomous agent. Everything below is *execution fact*, not +architecture — do these before Phase 0 (§15). + +### 0.1 Environment bootstrap (one-time) + +```pwsh +# activate the existing venv (root: C:\virtual\environment) +& C:\virtual\environment\Scripts\Activate.ps1 # or: python -m venv to create a new one +pip install azdev +# from the repo root (azure-cli-extensions): +azdev setup --repo . # wires azure-cli + this ext repo +azdev extension add migrate # make the migrate ext importable/dev-installed +az extension list # confirm 'migrate' is a dev extension +``` + +Everything the `Verify` gates run (`azdev style|linter|test migrate`) requires this to be done once. If +`azdev` is unavailable, STOP and escalate — do not hand-roll pytest. + +### 0.2 Concrete constants to pin (fill before writing `constants.py`) + +The design references these but the **literal values must be read from the spec** (`spec/Runbooks/*`, +`spec/RunbookExecutions/*`) or confirmed with the service owner — do NOT invent them: + +| Constant | Where it lives | Source of truth | Status | +| --- | --- | --- | --- | +| `RUNBOOKS_API_VERSION` = `"2020-06-01-preview"` | `shared/constants.py` | `api-version=` in the spec YAMLs | **confirmed** | +| Provider namespace + type-segment casing (`Microsoft.Migrate` / `migrateProjects` / `runbooks` / `executions`) | `shared/constants.py` | resource IDs + returned `type` in the spec YAMLs | **confirmed** — canonical ID `/subscriptions/{s}/resourceGroups/{rg}/providers/Microsoft.Migrate/migrateProjects/{p}/runbooks/{n}/executions/{e}` (camelCase `migrateProjects`). Note: existing `local`/replication code uses lowercase `migrateprojects` (`helpers/_server.py`); ARM path is case-insensitive so both work, but new runbook code pins this canonical casing for stable recordings + `id` parsing | +| `scopeType` value (`"Wave"`) + `waveId` template | `runbook/constants.py`, `models.build_generate_body` | CreateRunbook body in spec | confirmed (§1.3.3) — scope is polymorphic; discriminator `scopeType` is matched **case-sensitively**, so the payload uses camelCase (`scopeType`/`waveId`), verified live against the service | +| `RunbookExecutionAction` codes | `runbook/models.py` | service enum | confirmed (§1.3.2) | +| `stepRef` values per step type | `runbook/models.py` | AddStep body in spec / service | **placeholder for now** — stub in `models.py` with `# TODO(confirm): stepRef per step type`; fill from spec/service before Phase 3 (`step add`) | +| Runbook status terminal states (for `wait`/`--watch`/polling) | `runbook/constants.py` | GetRunbook `properties.status` enum | **confirmed** — `--watch` terminal set (`EXECUTION_TERMINAL_STATES`) derived from two authoritative service enums: `RunbookExecutionStatus` (execution ARM resource `properties.status`: Queued/InProgress/Completed/Failed/Pausing/Paused/Resuming/Cancelling/Cancelled) and `ExecutionState` (status.json node state: adds Succeeded/PartiallySucceeded/Skipped etc.). Terminal = Completed/Failed/Cancelled ∪ Succeeded/PartiallySucceeded/Skipped. Validated live | + +Any row marked **CONFIRM** is a Phase-0 blocker for the module that needs it; if unavailable, stub the +constant with a `# TODO(confirm): ` and gate the dependent live test with `@live_only()` until known. + +### 0.3 Test-harness recipe (migrate has none today) + +Phase 1 introduces the first recordings. Create a shared `ScenarioTest` base once so every scenario test +inherits scrubbing (there is no existing base to copy in migrate): + +- Put it at `tests/latest/_test_base.py` with a `MigrateScenarioTest(ScenarioTest)` that registers, in + `__init__`/`setUp`: subscription-id replacer → `00000000-0000-0000-0000-000000000000`, and processors that + strip `sig=`/`Bearer`/SAS query strings from **both** request and response bodies/URIs (see §14). +- Unit tests use plain `unittest.TestCase` with a mocked `cmd` (copy the `_create_mock_cmd` shape from the + existing `test_migrate_commands.py`). +- Test method name == recording file name (§14). Keep fixtures in `runbook/data/`. + +### 0.4 Per-command parameter contract + +The exact `options_list`, enum choices, and required/optional flags per command are **already fully +enumerated in the CLI spec** (`spec/az migrate runbooks cli - spec.md`) — every command has its own +"Arguments" table listing each flag, type, required/optional, and allowed values. This doc deliberately does +**not** duplicate them; the spec's per-command Arguments tables **are** the parameter contract. The agent's +job is to transcribe them 1:1 into `runbook/params.py` (`options_list` = the spec's literal `--flag` +spelling, enums = the spec's allowed-value lists, `required=` per the spec's column), and to encode the +cross-argument constraints (approval / custom-script parameter-sets, mutual exclusions) in +`runbook/validators.py` per §5/§12. Where a name is ambiguous, the spec's literal `--flag` spelling wins. +Keep the finalized arg table in `runbook/params.py` docstrings so it stays the single source in code. + +--- + +## 1. Repository observations + +### 1.1 The `migrate` extension today + +- **Package:** `src/migrate/azext_migrate/`, extension name `migrate`, version `3.0.0b4`, + `azext.minCliCoreVersion = 2.75.0`, `azext.isPreview = true`. +- **Command loader:** `MigrateCommandsLoader(AzCommandsLoader)` in + [azext_migrate/__init__.py](../azext_migrate/__init__.py). It: + - registers `custom_command_type = azext_migrate.custom#{}`, + - calls `load_aaz_command_table` (the `aaz/` package is currently empty — just `__init__.py`), + - then calls `load_command_table` from `commands.py` and `load_arguments` from `_params.py`. +- **Existing command groups** (all `is_preview=True`) in + [azext_migrate/commands.py](../azext_migrate/commands.py): + - `migrate get-discovered-server` + - `migrate local replication {init,new,list,get,remove,get-job}` + - `migrate local start-migration` +- **REST already uses raw requests.** [azext_migrate/helpers/_utils.py](../azext_migrate/helpers/_utils.py) + already wraps `azure.cli.core.util.send_raw_request` with: + - `send_get_request(cmd, uri)` + - `get_resource_by_id(cmd, resource_id, api_version)` (returns `None` on 404) + - `create_or_update_resource(cmd, resource_id, api_version, properties)` (handles 202/empty) + - `delete_resource(cmd, resource_id, api_version)` + - `validate_arm_id_format(arm_id, template)` + - centralized `APIVersion` enum, `IdFormats`, `RoleDefinitionIds` constants. + **This is the precedent the spec's Open Question #1/#2 refers to — we extend it, not replace it.** +- **Logging:** `knack.log.get_logger(__name__)`; **errors:** `knack.util.CLIError`. +- **Helpers layout:** `helpers/_utils.py`, `helpers/_server.py`, `helpers/migration/start/`, + `helpers/replication/{get,init,job,list,new,remove}/` — i.e. **one folder per verb** with logic split + out of `custom.py`. Commands in `custom.py` are thin and import their implementation lazily. +- **Tests:** [azext_migrate/tests/latest/test_migrate_commands.py](../azext_migrate/tests/latest/test_migrate_commands.py) + uses `azure.cli.testsdk.ScenarioTest` but with **`unittest.mock` patched HTTP** — there is currently + **no `recordings/` folder** in the migrate extension. + +### 1.2 The `k8s-extension` reference (patterns to replicate, not import) + +- Clean separation: `commands.py`, `_params.py`, `_validators.py`, `custom.py`, `_format.py` (table + transformers), `_help.py`, `action.py` (custom argparse actions), `consts.py`, `utils.py`. +- `commands.py` uses `command_group` / `custom_command` / `custom_show_command`, with + `supports_no_wait=True`, `table_transformer=`, `confirmation=`. +- Centralized `consts.py` for API versions, RP namespaces, fault types. +- `utils.py` centralizes logging (`knack.log`), **telemetry** (`azure.cli.core.telemetry.set_exception / + set_user_fault / add_extension_event`), and reusable error mapping (`HttpResponseError` → + `azure.cli.core.azclierror.*`). +- Uses typed CLI errors from `azure.cli.core.azclierror` (`InvalidArgumentValueError`, + `RequiredArgumentMissingError`, `MutuallyExclusiveArgumentError`, `ResourceNotFoundError`, etc.). +- **Recording tests:** `tests/latest/` with `ScenarioTest`, `recordings/.yaml` (one YAML per + test), `data/` for fixtures, `MockClasses.py` for pure unit tests. YAML filename == test method name. +- **Difference from migrate:** k8s-extension uses **vendored SDK clients**. We deliberately **do not** + copy that; per the spec's Overall Objective we invoke ARM directly via `send_raw_request()`. + +### 1.3 Spec observations (source of truth) + +Command tree from `spec/az migrate runbooks cli - spec.md` (Appendix A) and REST files under +`spec/Runbooks/` and `spec/RunbookExecutions/`: + +| Command group | Verbs | Backing | +| --- | --- | --- | +| `migrate runbook` | `generate`, `show`, `list`, `update`, `delete`, `wait` | ARM resource `.../migrateProjects/{p}/runbooks/{n}` | +| `migrate runbook definition` | `show`, `download`, `visualize` | definition document / files / client-side | +| `migrate runbook definition step` | `add`, `update`, `remove` | sub-objects of definition (read-modify-write) | +| `migrate runbook definition workstream` | `split`, `merge` | sub-objects of definition | +| `migrate runbook parameter` | `download`, `upload` | parameters file (SAS blob) | +| `migrate runbook execution` | `start`, `pause`, `resume`, `cancel`, `show`, `list`, `visualize` | ARM child `.../runbooks/{n}/executions/{id}` | +| `migrate runbook execution step` | `retry`, `approve`, `complete` | action on a step within an execution | +| `migrate wave` | `show`, `list` | supporting reads | +| `migrate project` | `show`, `list` | supporting reads | +| `migrate workload` | `show`, `list`, `update-target-settings` | supporting reads + update | + +Confirmed REST endpoints (all `https://management.azure.com{MigrateProjectResourceId}/...`, +`api-version={RunbooksAPIVersion}`, `auth: inherit`): + +**Runbook resource & sub-object actions** (`spec/Runbooks/`): + +- `PUT .../runbooks/{n}` — CreateRunbook (`generate`); body `properties.scope = {scopeType:Wave, waveId}` +- `GET .../runbooks/{n}` — GetRunbook (`show`) +- `GET .../runbooks` — ListRunbooks (`list`) +- `DELETE .../runbooks/{n}` — DeleteRunbook (`delete`) +- `POST .../runbooks/{n}/Regenerate` — RegenerateRunbook (re-`generate`; no body) +- `POST .../runbooks/{n}/AddStep` — body `{stepName, displayName, stepRef, migrationEntityIds, dependsOn[]}` +- `POST .../runbooks/{n}/UpdateStep` — body `{stepId, displayName, dependsOn[]}` +- `POST .../runbooks/{n}/DeleteStep` — body `{stepId}` +- `POST .../runbooks/{n}/SplitWorkstream` — body `{sourceWorkstreamId, stepIds[], newWorkstreamName}` (moves the given steps into the new workstream; no `migrationEntityIds`) +- `POST .../runbooks/{n}/MergeWorkstreams` — body `{workstreamId[], newWorkstreamName?}` (`newWorkstreamName` optional — service defaults to the first workstream's name) +- `POST .../runbooks/{n}/GenerateDownloadUrl` — download definition/spec + parameters; returns a **SAS URL to a ZIP** + +**Execution resource & actions** (`spec/RunbookExecutions/`): + +- `GET .../runbooks/{n}/executions` — ListRunbookExecutions (`execution list`) +- `GET .../runbooks/{n}/executions/{e}` — GetRunbookExecution (`execution show`) +- `PUT .../runbooks/{n}/executions/{e}` — StartRunbookExecution (`execution start`); body `{properties:{}}` +- `PATCH .../runbooks/{n}/executions/{e}` — PatchRunbookExecution; body `{status}` +- `POST .../runbooks/{n}/executions/{e}/PerformAction` — body `{action:, targetId, migrationEntityIds[]}` → **pause=1/resume=2/cancel=3/retry=4** (differ only by the integer `RunbookExecutionAction` code) +- `POST .../runbooks/{n}/executions/{e}/ProvideApproval` — body `{action:"Approve", targetId:, migrationEntityIds[]}` (`execution step approve`; `Reject` also available) +- `POST .../runbooks/{n}/executions/{e}/UpdateStepStatus` — body `{action:"Complete", targetId:, migrationEntityIds[]}` (`execution step complete`; `Fail`/`Skip` also available) +- `POST .../runbooks/{n}/executions/{e}/GenerateDownloadUrl` — download status.json; returns a **SAS URL to a ZIP** + +> **Dominant pattern:** ~13 of the ~20 endpoints are **`POST {resourceId}/{ActionName}` with a small JSON +> body**. This is the single most important architectural signal — it validates one generic +> `post_action(resource_id, action_name, body)` client method that every edit/action command reuses +> (see §6). + +**Inconsistencies / assumptions (clarified with the spec owner):** + +1. **~~Swapped verbs~~ — RESOLVED.** The execution YAMLs now correctly declare `GET` for + `GetRunbookExecution` and `PUT` for `StartRunbookExecution`. +2. **`action` field — codes now CONFIRMED.** All execution actions come from one service enum + `RunbookExecutionAction` (0-based ordinal): `Start=0, Pause=1, Resume=2, Cancel=3, Retry=4, Complete=5, + Fail=6, Skip=7, Approve=8, Reject=9`. `PerformAction` uses the **integer** code (e.g. `"action": 1` for + pause); `ProvideApproval`/`UpdateStepStatus` send the **string** member name (`"Approve"`/`"Complete"`). + `models.py` mirrors this one enum + body builders. **Nothing about these codes is open anymore** — a + value change would be a one-line edit. +3. **Start execution needs no scope.** `StartRunbookExecution` body is `{properties:{}}`; it only needs the + **runbook ARM id in the URL**. Scope (`{scopeType:"Wave", waveId}`) applies **only to `generate`** + (CreateRunbook). `waveId` is derived as `{project_resource_id}/waves/{wave_name}` in + `models.build_generate_body`; the extra leading slash in the YAML example is an artifact, not part of the + value. No casing/scope concern remains. +4. **`GenerateDownloadUrl` returns a SAS URL to a blob, not the file.** Flow: `POST .../GenerateDownloadUrl` + → response carries a **pre-signed blob SAS URL** → `files.py` does a plain HTTP GET on that URL + (**no ARM token** — the SAS is self-authorizing) to fetch a **ZIP** → extract with Python `zipfile`. + Used by definition download, parameter download, and execution status download. +5. **CLI params → REST body mapping for steps.** The CLI surfaces + `--step-type/--step-name/--step-description/--depends-on` plus type-specific `--approval-type` (Approval) + and `--run-mode/--execution-target` (CustomScript). `models.py` maps these onto the `AddStep`/`UpdateStep` + body (`{stepName, displayName, stepRef, migrationEntityIds, dependsOn, …}`). **The same parameter-set + treatment applies to `step update` (§2.3), not only `step add`**, and additional approval/custom-script + properties the service adds later are absorbed by this mapping layer. +6. **`MergeWorkstreams` takes an array of workstreams.** Body key `workstreamId` is a **list**; the CLI + surfaces it as `--source-workstream-ids` (plural). Mapping handled in `models.py`. +7. **The YAMLs are a Swagger substitute only.** `auth: inherit`, `settings`, `runtime`/`.bru` scripts, and + `seq` are **Bruno-client artifacts and are ignored** by the implementation — only **method + URL + body** + are contractually meaningful. (`auth: inherit` just means "use the caller's ARM token", which + `send_raw_request` supplies automatically.) +8. **Steps/workstreams are not ARM resources** and are edited via dedicated **POST action endpoints** + (AddStep/UpdateStep/DeleteStep/SplitWorkstream/MergeWorkstreams) — no read-modify-write PATCH; + `cmds/definition_step.py`/`cmds/definition_workstream.py` call `post_action` directly. + +--- + +## 2. Proposed folder structure + +All new code is organized as **self-contained feature packages** inside `azext_migrate` (starting with +`runbook/`), plus a **cross-feature `shared/` access layer** every group reuses. New command paths never +collide with existing `get-discovered-server` / `local` commands and can be reviewed/shipped independently. + +```text +src/migrate/ +├── azext_migrate/ +│ ├── __init__.py # (edit) also load runbook command table + args + help +│ ├── commands.py # (unchanged existing) +│ ├── custom.py # (unchanged) existing local/replication commands +│ ├── _params.py # (unchanged) +│ ├── _help.py # (unchanged) + import feature _help modules +│ ├── helpers/ # existing shared helpers (send_raw_request wrappers) — reused +│ │ └── _utils.py +│ ├── shared/ # NEW cross-feature access layer (used by EVERY migrate feature group) +│ │ ├── __init__.py +│ │ ├── arm_client.py # ArmClient — generic ARM REST wrapper (send_raw_request) +│ │ ├── arm_ids.py # generic ID/URL builders, --ids parsing, api-version join +│ │ ├── constants.py # provider namespace, base ID templates, api-version registry +│ │ ├── errors.py # ARM error → azclierror mapping +│ │ ├── polling.py # LRO / --no-wait / wait / --watch helpers +│ │ ├── files.py # GenerateDownloadUrl SAS→zip + safe local file IO +│ │ └── telemetry.py # azure.cli.core.telemetry wrappers +│ └── runbook/ # FEATURE package: `migrate runbook` (+ its nested subgroups) +│ ├── __init__.py +│ ├── commands.py # load_runbook_command_table(self) — registers group + all subgroups +│ ├── params.py # load_runbook_arguments(self) +│ ├── _help.py # helps[...] for every runbook command +│ ├── constants.py # runbook-specific enums, ID templates, fault types +│ ├── validators.py # runbook argument-constraint validators +│ ├── transformers.py # runbook `--output table` transformers +│ ├── models.py # runbook request-body builders + enums +│ ├── cmds/ # business logic — ONE module per (sub)group; filename = subgroup path +│ │ ├── __init__.py +│ │ ├── runbook.py # `migrate runbook` generate/show/list/update/delete/wait/regenerate +│ │ ├── definition.py # `migrate runbook definition` show/download/visualize +│ │ ├── definition_step.py # `migrate runbook definition step` add/update/remove (nested subgroup) +│ │ ├── definition_workstream.py # `migrate runbook definition workstream` split/merge (nested subgroup) +│ │ ├── parameter.py # `migrate runbook parameter` download/upload +│ │ ├── execution.py # `migrate runbook execution` start/show/list/pause/resume/cancel/visualize +│ │ └── execution_step.py # `migrate runbook execution step` retry/approve/complete (nested subgroup) +│ └── visualize/ # client-side (non-REST) rendering +│ ├── __init__.py +│ ├── graph.py # definition/execution JSON -> DAG model +│ ├── renderer.py # DAG model -> self-contained HTML (HTML-escaped, inline JS/CSS) +│ └── templates/ +│ └── runbook.html.tmpl # inline template (no CDN references) +│ # future peer groups (`migrate wave`/`project`/`workload`) — when implemented, each gets its OWN +│ # feature package (wave/, project/, workload/) beside runbook/, reusing shared/ +└── azext_migrate/tests/latest/ + ├── __init__.py + ├── shared/ # tests for the cross-feature shared/ layer + │ ├── test_arm_ids_unit.py # id/--ids parsing, api-version join + │ └── test_errors_unit.py # ARM status → azclierror mapping + └── runbook/ # ALL runbook tests live under one per-feature folder + ├── __init__.py + ├── test_runbook_scenario.py # ScenarioTest recording/playback tests + ├── test_runbook_unit.py # pure unit tests (models, validators, transformers, graph, renderer) + ├── recordings/ # one YAML per test method (filename == method name) + │ ├── test_runbook_crud.yaml + │ ├── test_runbook_execution.yaml + │ └── ... + └── data/ # fixtures scoped to runbook tests + ├── sample_definition.json + └── sample_parameters.json + # future peer groups add their OWN test folder here too (tests/latest/wave/, .../project/, …) +``` + +**Why feature packages + a shared layer:** it guarantees namespace isolation from the existing flat +`custom.py`/`_params.py`, keeps each (sub)group's logic in its own small module, and lets each feature +surface be enabled/disabled by **one loader call** (`load_runbook_command_table(self)`) added to +`__init__.py` (§4). The runbook feature therefore **does not touch the existing `custom.py` at all** — its +`CliCommandType(operations_tmpl='azext_migrate.runbook.cmds.{}')` routes every command into `runbook/cmds/*` +instead, so `cmds/` plays the role `custom.py` plays for the existing commands. Crucially this makes the +pattern **extensible**: future top-level groups (`migrate wave`, `migrate project`, `migrate workload`, and +anything added later) are new **peer packages**, each enabled by its own one-line +`load__command_table(self)` hook, that reuse the same `shared/` layer instead of duplicating it. + +### 2.1 Avoiding utility duplication across subgroups (the core concern) + +**Concern:** if each subgroup gets its own folder with its own `command`/`init`/`util`, common helpers get +copy-pasted into every folder. This is a real anti-pattern — and it is exactly what the existing migrate +`helpers/replication/{get,init,job,list,new,remove}/` **verb-per-folder** layout risks. + +**How other multi-subgroup extensions actually organize (reference evidence):** + +- **`containerapp`** — the largest multi-subgroup extension in this repo (`containerapp`, `containerapp env`, + `containerapp job`, `containerapp session`, `containerapp auth`, `containerapp env storage`, …). It does + **NOT** use folder-per-subgroup. It has **one flat shared layer at the package root**: + `_utils.py`, `_clients.py`, `_client_factory.py`, `_constants.py`, `_models.py`, `_transformers.py`, + `_validators.py`, `_help.py`, `_params.py`, a **single** `commands.py`, a **single** `custom.py`. + Per-subgroup *logic* lives in **one file per subgroup** (`containerapp_env_decorator.py`, + `containerapp_job_decorator.py`, …) that all consume the shared root layer. Cross-cutting helpers are + further split **by concern, not by subgroup** (`_ssh_utils.py`, `_dapr_utils.py`, `_archive_utils.py`). +- **`dataprotection`** keeps generated commands in `aaz/` and hand-written logic in a single `manual/` + package + one `custom.py` — again, shared helpers are centralized, not per-subgroup. + +**Conclusion / rule for migrate:** subgroups differ only in **business logic**, never in utilities. +Therefore: + +1. **A cross-feature `shared/` access layer** — `arm_client.py`, `arm_ids.py`, `errors.py`, `polling.py`, + `files.py`, `telemetry.py`, and base `constants.py` are defined **exactly once** under + `azext_migrate/shared/` and imported by **every feature group** (runbook, wave, project, workload, and + anything added later). Feature-specific `models.py`/`transformers.py`/`validators.py`/`constants.py` live + in each feature package and are shared by that feature's subgroups. +2. **`cmds/` contains only per-(sub)group business logic** — each module is a thin orchestrator + (resolve → validate → build body → `arm_client.post_action/put/get` → transform). **No `arm_client.py`, + `_utils.py`, or `constants.py` is ever created inside `cmds/`.** A lint/review rule enforces that + `cmds/*` and feature packages may import from `shared/`, but `shared/` never imports from any feature. +3. Because ~13 endpoints are the same `POST {resourceId}/{action}` shape, the *entire* edit/action surface + (steps, workstreams, execution actions, approvals, completes, regenerate) is served by **one** + `arm_client.post_action` + a handful of body builders in `models.py`. There is essentially nothing left + to duplicate — `cmds/definition_step.py:add()` is ~5 lines: build body, call + `post_action(runbook_id, 'AddStep', body)`. + +This directly resolves the duplication worry: `commands`/`params`/`_help` are split per subgroup for +readability, but **all common code lives in a single shared layer**, mirroring how `containerapp` scales to +dozens of subgroups without duplicating utilities. + +### 2.2 Reference patterns for logic placement (why a sub-package, not top-level `custom.py`) + +Two in-repo patterns were studied: + +- **`migrate` (existing `local`/replication):** a **thin top-level `custom.py`** whose functions **lazily + import** their implementation from `helpers/replication//…`. This works and keeps `custom.py` small, + but as the surface grows `custom.py` becomes a catch-all and the per-verb folders tempt utility + duplication. +- **`containerapp` (large extension):** a single large `custom.py` that delegates to **decorator classes** + (one per feature, e.g. `ContainerAppJobDecorator`) over a **centralized** `_utils.py`/`_clients.py`. + +Both centralize utilities and keep the command layer thin. The runbook feature adopts the **self-contained +`runbook/` sub-package** with `cmds/*.py` (plain functions, no decorator state machine — raw REST +doesn't need one) over a single shared access layer. This matches containerapp's centralized-utility +principle while staying isolated inside the `migrate` boundary. **The existing `local` commands are left +as-is** for now (isolation requirement); §2.4 describes how they *can* later adopt this same pattern +without any user-facing change. + +### 2.3 Naming, nested subgroups, and peer groups (extensibility rules) + +These rules make the layout scale to arbitrary future `migrate` groups/subgroups: + +- **Logic folder is `cmds/`** (not `operations/`), and **files are named after the subgroup path** below the + feature root — no `_ops` suffix. Examples inside `runbook/cmds/`: `runbook.py`, `definition.py`, + `parameter.py`, `execution.py`. +- **Nested / doubly-nested subgroups** encode their full path with underscores, so depth is unlimited and + two different `step` subgroups never collide: + - `migrate runbook definition step` → `cmds/definition_step.py` + - `migrate runbook execution step` → `cmds/execution_step.py` + A third level (e.g. a hypothetical `migrate runbook definition step approval`) is simply + `cmds/definition_step_approval.py`. The registration string `'#'` maps 1:1 to the file. +- **Peer groups are separate feature packages, never nested under `runbook/`.** Future groups like + `migrate wave`, `migrate project`, and `migrate workload` would each be a **sibling** feature package in + its **own folder named after the group** (`wave/`, `project/`, `workload/`) — not a shared/generic + `supporting/` folder, and not nested under `runbook/`. (They are **out of scope for now** — see §15 + Future TODO.) +- **Every feature package reuses `shared/`** for REST/IDs/errors/polling/files/telemetry, so adding a group + never duplicates the access layer. Enablement is **one loader hook per feature package** in + `__init__.py` (§4). + +### 2.4 Future: migrating the existing `local` commands to the feature-package pattern + +The current `local`/replication commands (`migrate get-discovered-server`, `migrate local replication …`, +`migrate local start-migration`) **can** be moved onto this pattern later as a pure refactor — **command +paths stay identical**, so there is no user-facing break. This is **out of scope now** (don't touch working +commands), but nothing in the design blocks it. The mapping: + +| Today (`local`, verb-per-folder) | Future (feature-package pattern) | +| --- | --- | +| thin `custom.py` funcs that lazily import helpers | `local/cmds/*` functions bound via `operations_tmpl` | +| `helpers/replication/{get,init,job,list,new,remove}/` (folder per verb) | `local/cmds/replication.py` (one module per subgroup) | +| `helpers/migration/start/` | `local/cmds/start_migration.py` (or a `local start-migration` module) | +| `helpers/_utils.py` `send_raw_request` wrappers | folded into `shared/arm_client.py` / `shared/arm_ids.py` | +| registration in root `commands.py` | `local/commands.py::load_local_command_table(self)` | +| args in root `_params.py` | `local/params.py::load_local_arguments(self)` | + +Migration steps (mechanical, incremental): + +1. Create a `local/` **peer package** (sibling of `runbook/`); move the replication/migration logic into + `local/cmds/*`, collapsing the six verb-folders into per-subgroup modules. +2. Point a new `CliCommandType(operations_tmpl='azext_migrate.local.cmds.{}')` at them. +3. Add `load_local_command_table(self)` / `load_local_arguments(self)` hooks in `__init__.py` and **remove** + the old root `commands.py` / `_params.py` registration for those commands. +4. Retire `helpers/replication/*` once its logic lives in `local/cmds/*` + `shared/`. + +Why it's safe / low-risk: paths are unchanged (`migrate local …`); it's the same REST, now routed through +`shared/arm_client` instead of `helpers/_utils`. Best done as its **own phase after** the runbook feature is +stable, so `shared/` is already proven before `local` depends on it. Caveat: the existing +`unittest.mock`-patched HTTP tests would be re-pointed at `shared/arm_client` (and ideally gain recordings, +per §14). + +--- + +## 3. Module responsibilities + +| Module | Responsibility | Depends on | +| --- | --- | --- | +| `runbook/commands.py` | Register command groups/subgroups, wire `supports_no_wait`, `confirmation`, `table_transformer`, `custom_show_command`. | knack/azcli, `transformers`, `constants` | +| `runbook/params.py` | Declare arguments, `options_list`, `get_enum_type`, `get_three_state_flag`, attach `validators`. | `constants`, `validators` | +| `runbook/_help.py` | `helps[...]` YAML docstrings + **examples** (release requirement per spec Open Q#4). | — | +| `runbook/constants.py` | Runbook-specific enums, ID templates, fault types (extends the shared api-version registry). | `shared/constants` | +| `runbook/validators.py` | Enforce the spec's **Argument constraints** (Azure CLI has no parameter-sets). | `constants`, azclierror | +| `runbook/transformers.py` | `--output table` transformers (definition, execution status, list). | — | +| `shared/arm_client.py` | **Single, cross-feature** ARM REST surface: `get/list/put/patch/delete/post_action` via `send_raw_request`; header/body serialization; delegates error mapping to `errors`. | `errors`, `arm_ids`, azcli.util | +| `shared/arm_ids.py` | Generic resource-ID/URL builders; join `api-version`; parse `--ids`. | `constants` | +| `shared/polling.py` | Poll GET until terminal status; back `generate`/`execution start` `--no-wait`; power `wait`; `--watch`. | `arm_client`, `constants` | +| `shared/errors.py` | Map ARM error bodies + status codes to `azclierror` types with actionable messages. | azclierror, telemetry | +| `shared/telemetry.py` | `record_exception`, `set_user_fault`, `add_event` wrappers (no-op safe). | azcli.telemetry | +| `shared/files.py` | `POST GenerateDownloadUrl` → SAS blob URL → HTTP GET (no ARM token) → unzip; safe local path handling. | `arm_client` | +| `shared/constants.py` | Provider namespace, base ID templates, api-version registry — shared by all feature groups. | — | +| `runbook/models.py` | Request-body builders (e.g. `build_generate_body(wave_id)`), enum definitions, response projections. | `constants` | +| `runbook/cmds/*` | Business logic per (sub)group; **one module per subgroup path**; orchestrate validate → build request → call client → transform. | shared layer + models/validators/transformers | +| `runbook/visualize/*` | Pure client-side transform of JSON → DAG → **self-contained, HTML-escaped** HTML file. | stdlib only | + +**Dependency direction (strictly one-way):** +`commands/params` → `cmds` → (`models`, `transformers`, `validators`, feature `constants`) → `shared/` +(`arm_client`, `arm_ids`, `polling`, `files`, `errors`, `telemetry`, base `constants`). No cycles. +`cmds` never imports `commands`; `shared/` never imports any feature package. + +--- + +## 4. Command registration strategy + +Keep existing registration untouched; **append** a single call so new commands are additive and +conflict-free. + +- [azext_migrate/__init__.py](../azext_migrate/__init__.py) `load_command_table`: after the existing + `load_command_table(self, args)`, add `from azext_migrate.runbook.commands import + load_runbook_command_table; load_runbook_command_table(self)`. +- `load_arguments`: after existing `load_arguments`, add + `from azext_migrate.runbook.params import load_runbook_arguments; load_runbook_arguments(self)`. +- Help: `azext_migrate/_help.py` adds `from azext_migrate.runbook import _help # noqa` (import side-effect + registers `helps`). No changes to existing help entries. + +Inside `runbook/commands.py`, register with a **custom command type** scoped to the `cmds` package so +resolution is unambiguous. **This is why the feature needs no `custom.py`:** the existing commands resolve +through `custom_command_type = azext_migrate.custom#{}` (into `custom.py`), whereas the runbook group binds +its own `CliCommandType(operations_tmpl='azext_migrate.runbook.cmds.{}')`, so `'runbook#generate'` loads +`azext_migrate.runbook.cmds.runbook.generate`. The `runbook/cmds/*` modules **are** this feature's command +implementations — `custom.py` is left untouched. Nested subgroups get their own `command_group` block and +their own `cmds` module: + +```python +# runbook/commands.py +runbook_cmds = CliCommandType(operations_tmpl='azext_migrate.runbook.cmds.{}') + +with self.command_group('migrate runbook', runbook_cmds, is_preview=True) as g: + g.custom_command('generate', 'runbook#generate', supports_no_wait=True) + g.custom_show_command('show', 'runbook#show', table_transformer='...') + g.custom_command('list', 'runbook#list_', table_transformer='...') + g.custom_command('update', 'runbook#update') + g.custom_command('delete', 'runbook#delete', confirmation=True) + g.custom_wait_command('wait', 'show') # native predicates over custom show (spec §7) + +# nested subgroup: `migrate runbook definition` +with self.command_group('migrate runbook definition', runbook_cmds) as g: + g.custom_show_command('show', 'definition#show', table_transformer='...') + g.custom_command('download', 'definition#download') + +# doubly-nested subgroup: `migrate runbook definition step` +with self.command_group('migrate runbook definition step', runbook_cmds) as g: + g.custom_command('add', 'definition_step#add') + g.custom_command('update', 'definition_step#update') + g.custom_command('remove', 'definition_step#remove', confirmation=True) +``` + +`operations_tmpl` resolves `'#'` to `azext_migrate.runbook.cmds..`, so the +module name is exactly the subgroup path (`definition_step`, `execution_step`) — **nesting depth is +unlimited and two different `step` subgroups never collide.** + +**Peer groups** (added later) would live in their **own** feature package named after the group and be +registered the same way (they are *siblings* of `migrate runbook`, not children). For example, a future +`migrate wave` group: + +```python +# wave/commands.py (added only when wave/project/workload are implemented — each in its own folder) +wave_cmds = CliCommandType(operations_tmpl='azext_migrate.wave.cmds.{}') +with self.command_group('migrate wave', wave_cmds, is_preview=True) as g: + g.custom_show_command('show', 'wave#show') + g.custom_command('list', 'wave#list_') +``` + +This mirrors the k8s-extension registration style while staying self-contained per feature. + +**Isolation guarantees:** new command paths all start with `migrate runbook` (and, when added, future peer +groups like `migrate wave`) — none overlap existing `migrate get-discovered-server` or `migrate local ...`. +No global argument context is modified except adding new contexts. + +--- + +## 5. CLI command organization + +- **`commands.py`** — *only* command registration + transformer/confirmation/no-wait wiring. No logic. +- **`params.py`** — *only* argument declarations. Shared arg types defined once: + `project_name`, `runbook_name` (`--runbook-name`; and `-n/--name` where the runbook is the primary + resource), `execution_id`, `step_id`, `workstream_id`, `wave_name`, enum types (status, step-type, + approval-type, run-mode, execution-target). `--ids` is auto-provided by Azure CLI when `id_part` is set + on the name arguments, satisfying the spec's `--ids` addressing. +- **`cmds/*.py`** — thin command functions (target ~15–40 lines each). Each function: + 1. resolves identity (`arm_ids`), 2. validates (`validators`, already partly enforced declaratively), + 3. builds request (`models`), 4. calls `arm_client`, 5. transforms/returns. +- **`validators.py`** — cross-argument constraints the spec spells out per command (e.g. `--approval-type` + required when `--step-type Approval`; `--run-mode`/`--execution-target` only for `--step-type CustomScript`; + `--entities`/`--all-ready` only for `Partial` and mutually exclusive; `--comment` required for `complete`). + **`step update` reuses the same parameter-set validators as `step add`** (approval / custom-script + conditionals), inferring the step's type from the target step when `--step-type` is omitted. +- **`shared/arm_client.py` + `shared/arm_ids.py`** — the client/helper modules; the only place + `send_raw_request` and URL/ID assembly live (reused by every feature group). +- **`transformers.py`** — the "default table view" tables named in the spec (definition table: + `Id, Step Name, Depends On, Configuration Status, Workloads, Applications`; execution table: + `Id, Step Name, Step Status, Depends On, Workload Progress`). + +--- + +## 6. REST client architecture + +A single `ArmClient` (in `shared/arm_client.py`) centralizes every ARM interaction (spec Objective + +Reusability §8). It wraps +the existing proven `send_raw_request` helpers rather than reinventing them. + +```text +cmds/* ──► ArmClient (shared/) ──► send_raw_request(cli_ctx, method, url, body, headers) + │ (auth, token, cloud endpoint handled by Az CLI) + ├─ arm_ids: build resource id + ?api-version= + └─ errors: status/body → azclierror +``` + +Responsibilities of `ArmClient` (all take `cmd`): + +- `get(resource_id)` / `get_or_none(resource_id)` — GET; 404→None variant for existence checks. +- `list(collection_id)` — GET with automatic **`nextLink` pagination** aggregation (runbooks, executions). +- `put(resource_id, body)` — create/generate/start; returns body or async handle. +- `patch(resource_id, body)` — `runbook update`, `PatchRunbookExecution` (`{status}`). +- `delete(resource_id)` — delete. +- `post_action(resource_id, action_name, body=None)` — **the workhorse.** Serves every + `POST {resourceId}/{action_name}` endpoint: `Regenerate`, `AddStep`, `UpdateStep`, `DeleteStep`, + `SplitWorkstream`, `MergeWorkstreams`, `GenerateDownloadUrl`, `PerformAction`, `ProvideApproval`, + `UpdateStepStatus`. ~13 commands share this one method. +- All methods send `Content-Type=application/json`, serialize dict→JSON, parse JSON→dict, and route errors + through `errors.raise_for_arm_error(response)`. + +**Action-code abstraction (removes duplication for execution actions):** `PerformAction` handles +`pause/resume/cancel/retry` via an integer `action` code from the confirmed `RunbookExecutionAction` enum +(`Start=0, Pause=1, Resume=2, Cancel=3, Retry=4, Complete=5, Fail=6, Skip=7, Approve=8, Reject=9`). +`models.py` owns an `ExecutionAction` enum (member → int) and a +single `build_perform_action_body(action, target_id=None, entity_ids=None)` builder, so those four commands +are one-liners differing only by enum member. `approve`/`complete` reuse the **same** enum but serialize the +**string member name** for the `ProvideApproval`/`UpdateStepStatus` bodies via +`build_step_action_body(action, target_id, entity_ids)`. + +**Download/unzip:** `GenerateDownloadUrl` returns a SAS URL to a **ZIP**; `files.py` downloads then +extracts with Python `zipfile` (definition+docs, parameters, or execution status.json). + +**Cloud/endpoint handling:** URLs are always +`cmd.cli_ctx.cloud.endpoints.resource_manager + resource_id + '?api-version=' + RUNBOOKS_API_VERSION`, +so sovereign clouds work automatically (matches existing `_utils.py` pattern). + +**Reuse decision:** `arm_client.py` internally calls the existing +`helpers/_utils.py` functions where they already fit (`get_resource_by_id`, `create_or_update_resource`, +`delete_resource`) and adds runbook-specific methods (`list` with paging, `post_action`, `patch`). This +honors "reuse migrate utilities where appropriate" without importing anything from other extensions. + +--- + +## 7. Utility layer design + +- **`arm_ids.py`** + - `project_id(sub, rg, project)`, `runbook_id(project_id, name)`, + `execution_id(runbook_id, execution_id)` from ID templates in `constants.py`. + - `resolve_ids(cmd, namespace)` — when `--ids` is supplied, parse it into + (sub, rg, project, runbook, execution); otherwise assemble from discrete args. Central place for the + spec's "Execution identity" addressing rules. + - `with_api_version(resource_id)` — append `?api-version=`. +- **`polling.py`** — `poll_until(cmd, get_fn, predicate, interval, timeout)` used by `--watch`, + and terminal-status detection for `generate`/`execution start`. The public `wait` command uses the CLI's + native `custom_wait_command` bound to the custom `show`, so `wait` gets `--created/--updated/--deleted/` + `--exists/--custom/--interval/--timeout` for free (no hand-written polling for `wait` itself). +- **`files.py`** — `download_sas_zip(sas_url, dest)` (plain HTTP GET on the pre-signed blob URL — **no ARM + token** — then unzip with Python `zipfile`), `save_json(obj, path)`, + `generate_and_download(cmd, resource_id, dest)` = `POST GenerateDownloadUrl` → read SAS URL from response + → `download_sas_zip`. Validates/normalizes destination paths (prevents path traversal), defaults to CWD + per spec. +- **`errors.py`** — `raise_for_arm_error(response)`; maps `404→ResourceNotFoundError`, + `400→InvalidArgumentValueError`/`BadRequestError`, `403→ForbiddenError`, `409→ClientRequestError`, + `5xx→CLIInternalError`; extracts `error.code`/`error.message`; appends remediation hints. +- **`telemetry.py`** — `record_exception(ex, fault_type, summary)`, `set_user_fault()`, + `add_event(name, props)`; all guarded so telemetry never breaks a command. +- **`transformers.py`** — pure functions `dict → list[OrderedDict]` for the named table views + (feature-specific; lives in `runbook/`). + +The access modules above (`arm_client`, `arm_ids`, `polling`, `files`, `errors`, `telemetry`, base +`constants`) are **cross-feature** and live in `shared/`, so `runbook/` and any future feature group +(e.g. `wave/`, `project/`, `workload/`) reuse them without duplication. Feature-specific +`models`/`transformers`/`validators`/`constants` live in each feature package. Only if a `shared/` helper +proves useful to the existing `local`/`replication` commands would it also be surfaced via `helpers/` (per +spec §11 "introduce new shared utilities only when they provide value across multiple command groups"). + +### 7.1 Visualize pipeline (`graph.py` → `renderer.py` → `runbook.html.tmpl`) + +`migrate runbook definition visualize` and `migrate runbook execution visualize` turn a runbook JSON +document into a **single, self-contained, offline HTML file** (a dependency graph the user can open in a +browser). It is **purely client-side** — the only network call is the one read (`arm_client.get`) that +fetches the JSON; everything after that is local, stdlib-only transformation. The three files split the job +into **parse → render → template** so each stage is independently unit-testable: + +1. **`visualize/graph.py` — JSON → DAG model (data only, no HTML).** + - Input: the definition document (steps + `dependsOn` + workstreams) or an execution document (steps + + per-step `status`/progress). + - Output: an in-memory **DAG** — a list of **nodes** (one per step/workstream, carrying id, name, + type, and for executions the status) and **edges** (one per `dependsOn` link). + - Also: validates the graph (detects cycles / dangling `dependsOn` references) and computes a stable + **topological layering** (which nodes sit in which "row") so the renderer can lay them out + deterministically. This module contains **no HTML and no I/O** — just dict/list → dataclasses, which + makes it trivial to unit-test the graph shape. + +2. **`visualize/renderer.py` — DAG model → HTML string (the security-critical stage).** + - Takes the DAG model and produces the final HTML by filling the template. + - **HTML-escapes every user-controlled value** (step names, descriptions, workstream names) with + `html.escape` **before** substitution — this is the mandatory **XSS guard** (spec §5): a step named + `` must render as inert text, never execute. This is the single most + important line in the feature and is asserted by a unit test. + - Emits nodes/edges as data the template's inline script draws (e.g. a small vanilla-JS layout, or + precomputed SVG/positioned `
`s from the layering in step 1). Status colors for execution graphs + are applied here. + +3. **`visualize/templates/runbook.html.tmpl` — the static shell.** + - A plain HTML skeleton with placeholders plus **inline `