Skip to content

tech.md

Latest commit

 

History

History
718 lines (523 loc) · 25.5 KB

tech.md

File metadata and controls

718 lines (523 loc) · 25.5 KB
Raw

Technical Documentation

Table of Contents

Architecture Overview

┌─────────────────────────────────────────┐
│  CLI Frontend (build.py)                │  ← Argument parsing, orchestration
├─────────────────────────────────────────┤
│  Config Layer (lib/config.py)           │  ← JSON loading, conf.d merging
├─────────────────────────────────────────┤
│  Build Engine (lib/builder.py)          │  ← Preset processing, cloud-init generation
├─────────────────────────────────────────┤
│  Resolution Layer (lib/scripts.py)      │  ← Script matching, include handling
│  Network Layer (lib/network.py)         │  ← v1/v2 config generation
├─────────────────────────────────────────┤
│  Output Layer (lib/iso.py)              │  ← YAML writing, ISO creation
└─────────────────────────────────────────┘

Key principle: The CLI frontend is thin. All logic lives in lib/ for testability and reuse.

Design Decisions

Why NoCloud ISO instead of Proxmox CloudInit Drive?

Proxmox's built-in cloud-init only supports: user, password, DNS, IP, SSH key, package upgrade. Real provisioning needs custom packages, files, commands, routing policies, etc.

NoCloud ISO is a regular CD-ROM with cidata label. Cloud-init auto-detects it. Proxmox sees it as a normal ISO.

Trade-offs:

  • ✅ Works offline, no network dependencies
  • ✅ Compatible with any hypervisor
  • ✅ Full control over cloud-init modules
  • ❌ No dynamic metadata (instance identity tokens)
  • ❌ Static files only

Why shell scripts instead of cloud-init modules?

Cloud-init modules vary across distros:

  • packages module: works on apt/dnf, broken on apk (Alpine)
  • runcmd with inline commands: universal but unreadable for complex logic
  • write_files + runcmd: universal, readable, debuggable

Decision: Write scripts to /opt/provision/ via write_files, then execute via runcmd.

Benefits:

  1. Cross-distro compatibility (bash works everywhere)
  2. Scripts exist on disk after provisioning — easy debugging
  3. No dependency on cloud-init module availability
  4. Version-controllable provisioning logic

Why network-config v1 as default?

Format Supported by Features
v1 All distros Static, DHCP, bonds, bridges, VLANs, routes, passthrough
v2 (Netplan) Ubuntu/Debian only Policy routing, tables, bonds, VLANs

v1 is translated by cloud-init to NetworkManager, ifupdown, or sysconfig depending on the distro. v2 requires Netplan.

Decision: v1 is default for universal compatibility. v2 is opt-in via "network_version": 2.

Unified interface format: Both v1 and v2 accept the same interfaces list. The builder translates type: "dhcp" and type: "static" to the appropriate format, while passing all other fields verbatim.

Why JSON for configuration?

YAML has implicit type conversion: yes → True, 10:30 → time object, off → False. JSON is explicit and universally parseable.

Trade-off: Slightly more verbosity, zero ambiguity.

Why multi-ID conflict resolution?

Scenario: base setup (default) + network override (2nets). Instead of duplicating all scripts, specify ids: ["default", "2nets"]. The builder collects all matching scripts and resolves same-priority conflicts by ID position (later wins).

This enables composition over inheritance.

Why conf.d/ for modular builds?

As the number of presets grows, main.json becomes unwieldy. Splitting into conf.d/*.json allows:

  • Team members to own different preset files
  • Git-friendly diffs
  • Easy environment-specific overrides

Files are sorted alphabetically and appended after main.json builds.

Why global override in conf.d?

Environment-specific settings (timezone, DNS, postbuild) often need to be applied across all builds in a conf.d file. Instead of duplicating them in every build definition, a global section in conf.d/*.json can override or extend the main global config.

Merge strategy:

  • params and default_include_permissions: deep-merge (later files override earlier ones)
  • All other keys: shallow merge (later files overwrite earlier ones)

Why passthrough for Netplan v2?

Netplan is actively developed. New features appear regularly. Instead of maintaining a mapping of every Netplan field, the builder copies interface definitions verbatim (with minimal conversion for name and type).

This means users can use any Netplan feature immediately, without waiting for a builder update.

Why dynamic user-data generation?

Early versions hardcoded users, ssh_pwauth, disable_root, growpart, package_update, package_upgrade into every user-data. This caused issues:

  • Unwanted user creation when the admin wanted custom user management
  • package_update running before runcmd and breaking dpkg state
  • Bloated configs for minimal deployments

Decision: Only include cloud-init modules when explicitly configured in params.

Why CLI-first inline builds?

Not every environment has a persistent main.json. CI/CD pipelines, one-off tests, and quick provisioning often need ad-hoc ISO generation. The builder supports full inline parameter passing via CLI, including:

  • All cloud-init modules (users, packages, timezone, growpart, etc.)
  • Network configuration (v1 and v2)
  • Custom scripts and includes
  • JSON config via --config-json for ephemeral environments

This makes the builder usable as a standalone tool without any configuration files.

Module Reference

build.py

Role: CLI frontend and orchestrator.

Responsibilities:

  • Parse CLI arguments
  • Load main.json via lib.config.load_config()
  • Merge conf.d/ via lib.config.merge_conf_d()
  • Instantiate lib.builder.Builder
  • Iterate over builds, apply filters (--preset, --profile, --conf-d-build)
  • Handle inline builds (no config)
  • Handle --config-json (ephemeral config)
  • Auto-clean output directory
  • Print summary

No build logic here. All heavy lifting is delegated to lib/builder.py.

# Entry point flow
main()
  ├── print_help()          # if no args
  ├── argparse              # parse CLI
  ├── load_config()         # load main.json (unless --no-config or --config-json)
  ├── merge_conf_d()        # merge conf.d/*.json (or single file for --conf-d-build)
  ├── Builder()             # create builder instance
  ├── builder.build_preset()  # for each build
  └── print summary

CLI modes:

  • build.py (no args) → help
  • build.py -a → build all from config
  • build.py --preset X,Y → build specific preset(s)
  • build.py --conf-d-build PATH → build from specific conf.d file/dir
  • build.py --config-json '{...}' → ephemeral config
  • build.py --no-config --profile X --ids Y ... → pure inline build

Inline parameter parsing:

  • _parse_inline_params(args) converts CLI flags to params dict
  • Supports all cloud-init modules: hostname, ip, gateway, dns, domain, interface, ciuser, password, ssh_key, ssh_pwauth, disable_root, packages, timezone, growpart, package_update, package_upgrade, network_version, interfaces
  • --script can be specified multiple times for extra scripts
  • --include can be specified multiple times for extra includes

lib/config.py

Role: Configuration loading and merging.

Functions:

load_config(path)

Loads main.json and returns (global_cfg, builds_list).

global_cfg, builds = load_config('main.json')
# global_cfg: dict with output_dir, iso_label, etc.
# builds: list of build definitions

merge_conf_d(conf_d_path, global_cfg, builds)

Merges JSON files from conf.d/ (or a single file) into the global config and builds list.

  • Accepts a directory or a single file path
  • Files are sorted alphabetically
  • Each file can contain:
    • A list of build definitions
    • A single build definition (dict)
    • A dict with global and/or builds keys
  • global sections are merged: deep-merge for params and default_include_permissions, shallow for others
  • builds are appended after main.json builds
# Merge entire conf.d directory
global_cfg, builds = merge_conf_d('conf.d', global_cfg, builds)

# Merge a single file (for --conf-d-build)
global_cfg, builds = merge_conf_d('conf.d/production.json', global_cfg, builds)

lib/scripts.py

Role: Script resolution and include handling.

parse_script_filename(filename)

Parses {priority}-{name}_{id}.sh into a dict:

{
    'priority': 10,
    'name': 'network',
    'id': 'router',
    'filename': '10-network_router.sh'
}

Returns None for invalid filenames.

resolve_scripts(samples_dir, profile_name, ids, ignore_missing=False)

Finds scripts matching any of the given ids.

Search order:

  1. samples/{profile_name}/
  2. samples/common/

Conflict resolution:

  • Group by priority
  • If multiple scripts share a priority, the one whose id appears later in ids wins
  • Prints [CONFLICT] message when resolving

Returns: (resolved_scripts, missing_ids)

scripts, missing = resolve_scripts('samples', 'ubuntu', ['default', 'router'])

resolve_includes(samples_dir, profile_name, ids, include_manifest, default_perms, include_dest)

Resolves include files based on manifest and/or auto-discovery.

Parameters:

  • samples_dir — path to samples directory
  • profile_name — profile subdirectory name
  • ids — list of IDs to match
  • include_manifest — list of include directives (strings or dicts), optional
  • default_perms — dict of default_include_permissions, optional
  • include_dest — default destination path, default /opt/provision/include

Returns: (entries, missing_ids)

  • entries: list of dicts for write_files entries
  • missing_ids: set of IDs with no includes found

Resolution flow:

  1. Parse filenames via parse_include_filename() — first _ is separator
  2. Process manifest entries in order:
    • String: find source by src_name across IDs (last ID wins), copy to include_dest
    • Object: extract src, dest, mode, owner, tree, preserve_mode
    • If tree=True or source is directory: recurse via _collect_tree_files()
  3. Auto-discover remaining files not in manifest, applying conflict resolution (last ID wins)
  4. Resolve permissions per file via _resolve_include_permissions():
    • Explicit mode/owner from manifest entry
    • preserve_mode=True → mode from filesystem
    • Exact filename match in default_perms
    • Extension match in default_perms
    • * fallback

Permission format:

  • String value: "0755" → mode only
  • Dict value: {"mode": "0755", "owner": "0:0"} → both
  • Owner is always uid:gid string, passed to cloud-init write_files.owner

lib/network.py

Role: Network configuration generation for cloud-init.

Key design: Both v1 and v2 use the same unified interfaces list. The builder translates type: "dhcp" and type: "static" to the appropriate format, while passing all other fields verbatim.

cidr_to_netmask(cidr)

Converts CIDR prefix (e.g., 24) to dotted netmask (255.255.255.0).

cidr_to_netmask(24)  # -> '255.255.255.0'

_parse_interfaces(params)

Parses and normalizes the unified interfaces list from params. Returns a list of dicts.

_v1_entry_from_interface(iface)

Converts a unified interface dict to a v1 config entry.

Translation rules:

  • type: "dhcp" → type: "physical", subnets: [{"type": "dhcp"}]
  • type: "static" → type: "physical", subnets: [{"type": "static", "address", "netmask", "gateway"}]
  • type: "physical"/"bond"/"bridge"/"vlan" → passthrough type as-is
  • All other fields copied verbatim
  • routes inside interface: wrapped into subnets[0] with network/netmask/gateway normalization

_v2_entry_from_interface(iface)

Converts a unified interface dict to a v2 (Netplan) ethernet config.

Translation rules:

  • type: "dhcp" → dhcp4: true
  • type: "static" → addresses, gateway4 (if address/gateway provided)
  • type: "physical"/"bond"/"bridge"/"vlan" → passthrough as-is
  • All other fields copied verbatim
  • dns/domain → nameservers.addresses/nameservers.search

generate_network_config_v1(params)

Generates network-config version 1 (universal format).

Modes:

  1. Raw passthrough: params['network_config'] with version: 1 → returned as-is.
  2. Unified interfaces: params['interfaces'] list → translated via _v1_entry_from_interface().
  3. Legacy single interface: params['ip']/params['interface']/params['gateway'].

Additional features:

  • params['routes'] → top-level type: route entries
  • params['extra_config_entries'] → appended verbatim
  • Nameserver generated only if params['dns'] is explicitly set (and no_auto_nameserver is not True)

Output:

{
    'version': 1,
    'config': [
        {'type': 'physical', 'name': 'ens18', 'subnets': [...]},
        {'type': 'route', 'network': '0.0.0.0', 'netmask': '0.0.0.0', 'gateway': '10.0.0.1'},
        {'type': 'nameserver', 'address': ['8.8.8.8'], 'search': ['local']}
    ]
}

generate_network_config_v2(params)

Generates network-config version 2 (Netplan, Ubuntu/Debian only).

Modes:

  1. Unified interfaces: params['interfaces'] list → translated via _v2_entry_from_interface().
  2. Raw Netplan dict: params['interfaces'] as dict with ethernets key → returned as-is.
  3. Fallback: no interfaces → delegates to generate_network_config_v1().

Output:

{
    'version': 2,
    'ethernets': {
        'ens18': {'dhcp4': True, ...},
        'ens19': {'addresses': ['10.0.0.10/24'], ...}
    }
}

lib/iso.py

Role: YAML writing and ISO creation.

write_yaml_file(path, data, header=None)

Writes a Python dict to YAML.

  • Uses yaml.dump() with default_flow_style=False for readability
  • allow_unicode=True for non-ASCII support
  • sort_keys=False to preserve insertion order
  • Optional header (e.g., #cloud-config )

create_iso(preset_name, user_data, meta_data, network_config, rendered_scripts, includes, output_dir, iso_label)

Creates the output directory structure and ISO file.

Steps:

  1. Create output/{preset}/ directory
  2. Write user-data, meta-data, network-config as YAML
  3. Write rendered scripts to output/{preset}/scripts/
  4. Write includes to output/{preset}/scripts/include/
  5. Generate ISO using genisoimage or mkisofs

ISO command:

genisoimage \
  -output output/{preset}.iso \
  -volid cidata \
  -joliet -rock \
  -input-charset utf-8 \
  -graft-points \
  user-data=output/{preset}/user-data \
  meta-data=output/{preset}/meta-data \
  network-config=output/{preset}/network-config \
  scripts/=output/{preset}/scripts/

Fallback: If genisoimage/mkisofs is not found, files are still written to disk with instructions for manual ISO creation.

lib/builder.py

Role: Core build engine.

Builder class

Constructor:

Builder(samples_dir, output_dir, iso_label, default_ssh_key, global_cfg=None)

Methods:

render_template(content, params)

Replaces {{key}} placeholders in script content with values from params.

content = "echo {{hostname}}"
render_template(content, {'hostname': 'web-01'})
# -> "echo web-01"
load_ssh_key(key_path)

Loads SSH public key from file. Expands ~ to home directory.

run_postbuild(script_path, env_vars)

Executes a postbuild script with environment variables set.

generate_cloudinit(preset_name, profile_name, scripts, includes, params, ssh_key_content)

Generates all cloud-init artifacts.

Steps:

  1. Render each script with render_template()
  2. Build write_files entries for scripts and includes
    • Includes are base64-encoded to preserve binary data exactly
  3. Build runcmd entries with set -e wrapping and logging
  4. Construct user_data dict dynamically:
    • users block only if ciuser is set
    • password attached to user entry if set
    • hostname/fqdn/manage_etc_hosts if hostname is set
    • growpart, package_update, package_upgrade, timezone, packages, ntp only if explicitly configured
    • write_files and runcmd always included
  5. Construct meta_data:
    • instance-id: {preset_name} (no prefix)
    • local-hostname: params.hostname or vm-{preset_name}
  6. Generate network_config via v1 or v2

Returns: (user_data, meta_data, network_config, rendered_scripts)

build_preset(preset_def, args)

Orchestrates a single preset build.

Steps:

  1. Extract preset, profile, ids from preset_def
  2. Resolve scripts via resolve_scripts() with ignore_missing=True (always)
  3. Resolve includes via resolve_includes()
  4. Handle missing IDs:
    • IDs with includes but no scripts → [INFO] log, build continues
    • IDs with neither scripts nor includes → [BLOCK] if ignore_missing_ids=false
  5. Apply overrides (disable, extra_files)
  6. Merge default_include_permissions: global → build (build wins)
  7. Resolve includes via resolve_includes() with manifest and default permissions
  8. Merge params: global.params → build.params
  9. Load SSH key if ciuser is set
  10. Generate cloud-init configs via generate_cloudinit()
  11. Write files or create ISO via create_iso()
  12. Run postbuild if configured and not disabled by --no-postbuild

Postbuild priority: CLI --postbuild > build.postbuild > global.postbuild. --no-postbuild disables all.

Returns: True on success, False on failure.

Data Flow

main.json + conf.d/*.json
        │
        ▼
┌─────────────────┐
│  load_config()  │  → (global_cfg, builds)
└─────────────────┘
        │
        ▼
┌─────────────────┐
│ merge_conf_d() │  → (global_cfg, builds) extended
└─────────────────┘
        │
        ▼
┌─────────────────┐     ┌──────────────────┐
│  build_preset() │────→│ resolve_scripts()│ → scripts[]
│                 │     └──────────────────┘
│                 │     ┌──────────────────┐
│                 │────→│ resolve_includes()│ → includes[]
│                 │     └──────────────────┘
│                 │     ┌──────────────────────┐
│                 │────→│ generate_cloudinit() │ → user_data, meta_data, network_config
│                 │     └──────────────────────┘
│                 │     ┌──────────────────┐
│                 │────→│   create_iso()   │ → .iso file
│                 │     └──────────────────┘
└─────────────────┘

Cloud-init Integration

How the ISO is detected

  1. VM boots with ISO mounted as CD-ROM
  2. Cloud-init scans for volume label cidata
  3. Reads user-data, meta-data, network-config from ISO root
  4. Processes modules in order:
    • write_files → creates files on disk
    • runcmd → executes commands

user-data structure

#cloud-config
# [optional] users block (only if ciuser set)
users:
  - name: admin
    groups: sudo
    shell: /bin/bash
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    ssh_authorized_keys: ["ssh-rsa ..."]

# [optional] hostname block (only if hostname set)
hostname: router1
fqdn: router1
manage_etc_hosts: true

# [optional] cloud-init modules (only if explicitly set)
timezone: UTC
package_update: true
package_upgrade: true

# [always] scripts and includes
write_files:
  - path: /opt/provision/01-disk.sh
    content: "#!/bin/bash\n..."
    permissions: '0755'
  - path: /opt/provision/include/motd
    content: "V2VsY29tZSEK"
    encoding: b64
    permissions: '0644'

# [always] execution
runcmd:
  - bash -c 'set -e; echo "[cloud-init] Running 01-disk.sh"; bash /opt/provision/01-disk.sh; echo "[cloud-init] 01-disk.sh done"'

Why set -e wrapping?

Without set -e, bash continues after errors. A failed apt install might leave the system in an inconsistent state. With set -e, the script stops on first error, and cloud-init logs the failure.

The wrapper pattern:

bash -c 'set -e; echo "[cloud-init] Running {script}"; bash {path}; echo "[cloud-init] {script} done"'

Provides:

  • Fail-fast behavior
  • Console logging for traceability
  • Clear error attribution

Script execution order

Scripts execute in priority order (numeric). The builder sorts by priority before generating runcmd.

Why base64 for includes?

Include files are base64-encoded in write_files to ensure exact byte reproduction. This prevents corruption of:

  • SSH private keys (PEM format with specific line endings)
  • Compiled binaries
  • Archives (tar, zip)
  • Any file with non-ASCII or non-UTF-8 content

Cloud-init decodes base64 content before writing to disk, preserving the original file exactly.

Cross-Profile Strategy

Profile Init Packages Network Manager cloud-init Support
Ubuntu/Debian systemd apt systemd-networkd / Netplan v1 + v2
Rocky/Alma/RHEL systemd dnf/yum NetworkManager v1 only
Alpine OpenRC apk ifupdown v1 only (limited modules)
Arch systemd pacman systemd-networkd v1 + v2

Key insight: Shell scripts in runcmd work on all distros. Network v1 works on all distros. v2 requires Netplan (Ubuntu/Debian only).

Profile-specific adaptations:

  • sudo group is wheel on Alpine, sudo everywhere else
  • apt-get preferred over apt in scripts (stable CLI)
  • || true used for service commands that may fail on non-systemd systems

Security Considerations

Passwords in config

Passwords are stored in main.json as plain text. This is intentional — the builder generates plain-text cloud-init configs, and the ISO is not encrypted.

Mitigations:

  • Use SSH keys instead of passwords when possible
  • Restrict file permissions on main.json (chmod 600)
  • Do not commit main.json with real passwords to version control
  • Use environment-specific conf.d/ files for sensitive data

SSH keys

The builder reads the SSH public key from default_ssh_key path and embeds it in user-data. The private key never leaves your machine.

ISO contents

The generated ISO contains:

  • user-data with all configuration (including passwords if set)
  • meta-data with instance ID and hostname
  • network-config with network settings
  • scripts/ with provisioning scripts

Treat the ISO as sensitive. Do not distribute it publicly if it contains passwords.

disable_root

Setting disable_root: true in params prevents root login. Recommended for production.

ssh_pwauth

Setting ssh_pwauth: false disables password authentication for SSH. Recommended when using SSH keys.

Extending the Builder

Adding a new profile

  1. Create samples/{profile_name}/
  2. Add scripts with appropriate {id} values
  3. Add builds to main.json or conf.d/

Adding a new cloud-init module

Edit lib/builder.py, method generate_cloudinit(). Add a conditional block:

if 'my_module' in params:
    user_data['my_module'] = params['my_module']

Customizing ISO creation

Edit lib/iso.py, function create_iso(). The ISO is created via genisoimage with -graft-points for flexible file placement.

Adding CLI arguments

  1. Edit build.py, add argument to argparse
  2. Pass through to Builder or build_preset()
  3. Document in docs/usage.md

Testing

The builder is designed for unit testing:

from lib.scripts import resolve_scripts, parse_script_filename
from lib.network import generate_network_config_v1, generate_network_config_v2
from lib.builder import Builder

# Test script resolution
scripts, missing = resolve_scripts('samples', 'ubuntu', ['default'])
assert len(scripts) > 0

# Test network generation
config = generate_network_config_v1({'ip': '10.0.0.10/24', 'gateway': '10.0.0.1'})
assert config['version'] == 1

# Test builder
builder = Builder('samples', './output', 'cidata', '~/.ssh/id_rsa.pub')
# ...