feat(scripts): add support-bundle log collector and HTML report (partial #429) - #1946
Open
valentin-pf9 wants to merge 2 commits into
Open
feat(scripts): add support-bundle log collector and HTML report (partial #429)#1946valentin-pf9 wants to merge 2 commits into
valentin-pf9 wants to merge 2 commits into
Conversation
Adds scripts/support-bundle/ with two tools for post-mortem migration analysis: - collect-logs.sh: gathers controller logs, v2v-helper pod logs, migration CRDs, related CRDs (MigrationPlan, mappings, etc.), Kubernetes events, and the vjailbreak-settings ConfigMap into a single .tar.gz. Credentials (vmwarecreds, openstackcreds, arraycreds, esxisshcreds) are piped through jq before tarring; any field whose key matches password|secret|token|key|credential is replaced with ***REDACTED***. Supports four destinations: local, gdrive (rclone), sftp (scp), s3 (rclone). - parse-logs.py: takes a bundle and produces a self-contained report.html. Extracts migration health (phase, parsed duration, suggested root cause + remediation), per-disk copy timeline (size, throughput, CBT iteration count), categorised errors (DNS, vCenter, NBD/VDDK, virt-v2v, OpenStack, Kubernetes, Auth, Network), pod restart count, network/storage mapping tables, sortable Kubernetes events table, and a vanilla-JS search box. - README.md: install/usage/limitations. deploy/installer.yaml and deploy/00crds.yaml were regenerated by the pre-commit make build-installer hook (current branch name in image tags). Partially addresses platform9#429 (the 'Expose through logs' approach). Does not replace a live in-UI dashboard; intended as a support-engineer tool. Verified against the real CRD types in k8s/migration/api/v1alpha1/.
20 unittest cases covering: - parse_disk_copies: basic two-disk flow, regression for the 0-based-vs- 1-based index bug, CBT attribution to the correct disk, empty input. - parse_cbt_iterations: iteration extraction with disk/duration/progress. - categorize_errors: DNS, vCenter, NBD/VDDK buckets; non-error lines excluded; unknown patterns fall through to Other. - suggest_root_cause: DNS and VDDK heuristics fire; clean logs yield None. - parse_pod_describe: extracts name, restart count, image, exit reason. - parse_events: disk.copy.start, migration.complete, Go log timestamp. - extract_mapping_entries: network mapping rows; empty YAML safe. Run from scripts/support-bundle/: python3 -m unittest test_parse_logs.py -v
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
scripts/support-bundle/with two tools for post-mortem migration analysis:collect-logs.sh— gathers controller logs, everyv2v-helper-*pod's logs inmigration-system, the Migration CRD, related CRDs (MigrationPlan, MigrationTemplate, NetworkMapping, StorageMapping, VjailbreakNode, VMwareMachine/Cluster/Host, RDMDisk), credentials (redacted), Kubernetes events from both namespaces, and thevjailbreak-settingsConfigMap into a single.tar.gz.parse-logs.py— takes a bundle and produces a self-containedreport.htmlwith: migration health card (phase, parsed duration, suggested root cause + remediation), per-disk copy table, event timeline, CBT iteration table, errors categorized by subsystem (DNS, vCenter, NBD/VDDK, virt-v2v, OpenStack, Kubernetes, Auth, Network), pod restart count / exit reason, network & storage mappings as tables, sortable Kubernetes events, and a vanilla-JS search box.README.md— install, usage, limitations.test_parse_logs.py— 20 unittest cases covering the parser (run viapython3 -m unittest test_parse_logs.py -v).Scope
Partial address of #429 — specifically the "Expose through logs" approach. Does not replace a live in-UI dashboard; it's a support-engineer post-mortem tool, not an end-user one. Happy to follow up with a Prometheus-metrics output mode (
--format prometheus) if that's wanted (per @roopakparikh's comment on #429).Credentials safety
Before anything is written to the tarball, credentials CRDs (
vmwarecreds,openstackcreds,arraycreds,esxisshcreds) are piped throughjq. Any field whose key matchespassword|secret|token|key|credential(case-insensitive) is replaced with***REDACTED***. Tarballs are safe to share / upload.Verified against the real codebase
vjailbreak.k8s.pf9.iov1alpha1 —k8s/migration/api/v1alpha1/groupversion_info.go:29migration-systemandvjailbreakv2v-helper-<vmk8sname>-<sha>—pkg/utils/migrationplanutils.go:GetJobNameForVMNamemigration_types.go:28have explicit CSS stylingv2v-helper/migrate/migrate.goTest plan
python3 -m unittest scripts/support-bundle/test_parse_logs.py -v— 20 tests, all pass (0.005s)collect-logs.sh --migration <name> --dest localagainst akindcluster with planted Migration CRDs and fake v2v-helper pods produces a valid tarball;parse-logs.pyon that tarball produces a populatedreport.htmlwith non-empty health card, disk table, CBT iterations, event timeline, categorized errors, and root-cause suggestion.What this does not do (yet, happy to follow up)