Feat/add tools part 1 - #1
Merged
Merged
Conversation
Introduced a new ui package that provides consistent terminal styling for lab commands. This includes styles for titles, subtitles, and various message types (OK, Warn, Err), as well as functions for rendering sections, steps, tables, and boxed content. The implementation enhances the visual output of the CLI, improving user experience.
Added a new package for managing language runtimes using mise. This includes functionality for installing, listing, and activating specific versions of languages. The implementation supports known language aliases and ensures mise is installed before performing operations, enhancing the CLI's capability to manage development environments.
Introduced a new scaffold package that facilitates the creation of new projects from predefined templates. This includes functionality for copying project-initiator templates based on specified languages (Go, Python, Rust, TypeScript) into a designated directory, enhancing project setup efficiency. The package also provides a method to list supported template kinds.
Added functionality for discovering bootable images from Ubuntu and Fedora mirrors, including support for LTS and interim releases. Implemented tests for image resolution and regex matching for ISO filenames. Introduced a method for creating bootable USB drives, including ISO download and checksum verification. This enhances the CLI's capabilities for managing bootable environments.
Introduced a new SSH package that provides functionality for connecting to remote hosts via SSH and synchronizing local homelab directories with remote servers using rsync. The package includes methods for establishing SSH connections with configurable options and handling home directory expansion for identity files, enhancing the CLI's capabilities for remote management.
Introduced a new services package that provides functionality for managing Docker and Podman compose stacks. This includes a Runner type for starting, stopping, and logging stacks, as well as methods for listing known stack names. The implementation enhances the CLI's capabilities for orchestrating containerized applications in a homelab environment.
Introduced deploy and remote packages to facilitate server management in the homelab CLI. The deploy package includes functionality for syncing the homelab to a server and executing various deployment modes (sync, provision, compose, full). The remote package provides methods for establishing SSH connections and executing commands on remote servers, enhancing the CLI's capabilities for remote operations and deployment workflows.
Introduced a new GitLabBackup function in the repos package, enabling the backup of GitLab accounts. This implementation includes environment variable handling for authentication, customizable backup directories, and support for parallel job execution. The addition enhances the CLI's capabilities for managing GitLab repositories and automating backup workflows.
Introduced new functionality for managing PostgreSQL instances, including configuration loading and applying desired states for databases and users. The implementation includes methods for ensuring user and database existence, granting permissions, and handling environment variables for authentication. This enhances the CLI's capabilities for managing PostgreSQL environments.
Introduced a new platform package that detects the operating system and identifies the package manager in use. The implementation includes constants for OS types and packagers, a struct for machine information, and methods for detecting the environment and checking for command availability. Additionally, unit tests were added to verify the detection logic for different operating systems, enhancing the CLI's capabilities for environment awareness and package management.
…d detection Added a new packager package that provides functionality for installing system packages using native backends (brew, apt, dnf). The implementation includes a Manager struct for managing package installations, methods for ensuring package presence, and checking if packages are installed. Unit tests were also added to verify the installation logic and handle edge cases, enhancing the CLI's capabilities for package management across different platforms.
Introduced a new mlstack package that provides functionality to ensure the ML compose stack is running. The implementation includes a method for executing `podman-compose up -d`, checking for necessary files and commands, and printing service URLs for easy access. This enhances the CLI's capabilities for managing machine learning environments in a homelab setup.
Eliminated the exclusion of the yt_playlist_downloads directory from the rsync command in the remote package. This change allows the directory to be included in the synchronization process, enhancing the flexibility of the rsync functionality.
Modified the ListTracked method in the packager package to streamline the list of common lab dependencies by removing 'ffmpeg' and 'yt-dlp'. This change simplifies the dependency management process and focuses on essential tools.
Introduced a new media package for converting HEIC files to JPEG format, including the implementation of the ConvertHEIC function. Added unit tests to validate the conversion process, error handling for non-directory inputs, and dry run functionality to skip existing JPEG files. This enhances the CLI's capabilities for image format conversion.
Introduced a new homelabroot package that provides functionality to resolve the path to a personal homelab repository. The implementation includes a Resolve function that checks various locations and environment variables to find the repository. Additionally, unit tests were added to validate the path resolution and error handling for missing repositories, enhancing the CLI's capabilities for managing homelab environments.
Introduced new functions for installing ClickHouse, Milvus, and Qdrant on Linux bare metal environments. Each function checks for necessary dependencies, handles installation via package managers, and configures service settings. This enhances the CLI's capabilities for managing data services in a homelab setup.
…unctionality Introduced a new bootstrap package that includes functionality for loading and executing setup profiles. The package supports embedded YAML profiles for various environments, such as laptop and server configurations. Added a Runner struct to manage the execution of profile steps, including package installation and toolchain setup. Unit tests were also created to validate the loading and execution processes, enhancing the CLI's capabilities for automated environment setup.
Updated CLI tests to include a dry run for the bootstrap laptop command, ensuring no errors occur during execution. Added tests for the bootstrap list and media help commands, verifying their output. Refactored import paths to use the correct module path, improving code organization and clarity. Introduced a new appctx package to manage per-invocation state, enhancing the context handling in the CLI.
… server management Added new commands for managing bare metal installations, including support for installing Qdrant, Milvus, and ClickHouse. Implemented media conversion utilities for HEIC to JPEG. Introduced PostgreSQL provisioning from YAML configurations, allowing for idempotent database and user management. Added server command for remote operations, including running commands and deploying homelab configurations. Enhanced CLI functionality with dry run support across all new commands, improving usability and testing capabilities.
Introduced new SSH and server configuration structures to the config package, enabling management of SSH host inventory and server connection details. Added methods for retrieving sorted SSH host names and default values for homelab and server configurations. This enhances the configuration capabilities for managing remote operations in a homelab environment.
… and dry-run support Introduced a new executil package that provides a Runner struct for executing shell commands with consistent logging and optional dry-run functionality. The package includes methods for running commands, checking command existence, and capturing output, enhancing the CLI's capabilities for managing external processes in a homelab environment.
Replaced the placeholder module path with the actual GitHub import path for the internal CLI package, ensuring correct package resolution and improving code clarity.
Replaced the placeholder module path with the actual GitHub import path in .goreleaser.yaml and Makefile. Updated Go version to 1.25.0 in go.mod, added new dependencies for pgx and yaml, and adjusted existing dependencies to their latest versions. This enhances the project's structure and ensures compatibility with the latest libraries.
…dates Added new commands for managing server operations, PostgreSQL provisioning, and bare metal installations, including support for Qdrant, Milvus, and ClickHouse. Enhanced media handling with HEIC conversion utilities. Updated configuration management to include a starter YAML file and improved documentation across various guides. Removed deprecated commands and streamlined existing functionalities for better usability and clarity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the first wave of real functionality in
homelab-cli(lab): bootstrap profiles, package/toolchain management, homelab compose stacks, remote server workflows (SSH/rsync/deploy), PostgreSQL provisioning, bare-metal DB installers, dynamic bootable USB creation, templates, and HEIC conversion — with orchestration in Go and documented external binaries only.Replaces placeholder stubs and homelab shell wrappers for the migrated workflows. Full English documentation refresh included.
Motivation
The personal homelab repo accumulated one-off scripts (
sync-to-server.sh,deploy-and-compose.sh,apply.py, bare-metal installers, etc.). This PR consolidates the common paths into a single CLI with shared config, dry-run, and terminal UI — without calling homelab.sh/.pyfor migrated features.YouTube playlist download was briefly implemented in Go, then removed from this CLI (stays in homelab
tools/media/if needed).What's new
Foundation
lab bootstrap laptop|server|profile|listpkg,toolchain, optionalscript:)lab pkg install|ensure|listinternal/packagerlab toolchain install|list|uselab services up|down|list|logs|ensureml-stack)Infra & server
lab server run,lab server deploy [provision|compose|full]lab postgres apply --configlab baremetal install qdrant|milvus|clickhouselab system usb list,lab system usblab ssh connect,lab ssh syncserver.*Workflow
lab templates list|newproject-initiators/lab media heicheif-convertlab repos backupCross-cutting
internal/ui— lipgloss sections/tablesinternal/executil— command runner with--dry-runinternal/homelabroot— resolve homelab repo pathinternal/cli/appctx— session on context (config, styles, dry-run)--homelab-root,--dry-run,--no-colorgithub.com/bartrosa/homelab-clipgx/v5for PostgreSQLStill stubbed (unchanged intent)
cluster,gpu,models,data,notebooks,mlops,vector,pipelines,agents,obs,logs,mcp, most ofrepos(clone/sync/status/list).Design rules
.sh/.pyfor migrated commands (exceptions documented below).docs/external-binaries.md.Interim homelab dependencies
lab repos backuptools/gitlab/backup_account.pyscript:stepsscripts/install-server-deps.shon server profileConfiguration
New/extended keys:
homelab.root/LAB_HOMELAB_ROOT/--homelab-rootserver.host,server.user,server.port,server.pathssh.hostsfor named connectionsExample:
docs/config.example.yamlRemoved:
media.*(YouTube cookies) — no longer applicable.Documentation
README.mddocs/README.mddocs/commands.mddocs/configuration.mddocs/architecture.mddocs/external-binaries.mddocs/homelab-migration.mdlabmappingCHANGELOG.mdNotable exclusions
lab media playlist— not part of this CLI; use homelab scripts oryt-dlpdirectly.run-ml-stack-setup-on-server.sh— not wired into bootstrap yet.Test plan
make cipasses locallylab bootstrap list/lab bootstrap laptop --dry-runlab pkg liston macOS and Linuxlab services listwith validhomelab.rootlab ssh sync --dry-runwithserver.*configuredlab server deploy --dry-run(sync only)lab postgres apply --config … --dry-run(or against test PG)lab system usb list(network; verifies Ubuntu/Fedora discovery)lab media heic .with sample.HEIC+heif-convertinstalledlab templates listwithhomelab.rootsetlab repos backupwithGITLAB_TOKEN(optional)