OSAC Installer

This repository contains Kubernetes/OpenShift deployment configurations for the OSAC platform, providing a fulfillment service framework for clusters and virtual machines.

Note: Throughout this guide, <project-name> refers to your unique OSAC installation name, which is used as the namespace. Replace it with your chosen project name (e.g., user1, team-a, etc.).

Overview

OSAC (Open Sovereign AI Cloud) provides a streamlined, self-service framework for provisioning and managing OpenShift clusters and virtual machines. This installer repository contains the Kubernetes/OpenShift deployment configurations needed to deploy OSAC components on your infrastructure.

For detailed architecture, workflows, and design documentation, please refer to the OSAC documentation repository.

The OSAC platform provides:

Self-service provisioning for clusters and virtual machines through a governed API
Template-based automation using Red Hat Ansible Automation Platform
Multi-hub support allowing multiple infrastructure hubs to be managed by a single fulfillment service
API access via both gRPC and REST interfaces for integration with custom tools

This installer uses Helm to manage deployments via an umbrella chart (charts/osac/) with per-environment values files under values/.

OSAC Components

The OSAC platform relies on four core components to deliver governed self-service:

Fulfillment Service: The API and frontend entry point used to manage user requests and map them to specific templates.
OSAC Operator: An OpenShift operator residing on the Hub cluster (ACM/OCP-Virt). It orchestrates the lifecycle of clusters and VMs by coordinating between the Fulfillment Service and the automation backend.
Console Proxy: A Kubernetes aggregated API server that provides serial and VNC console access to ComputeInstance VMs. Deployed alongside the operator on each hub.
Automation Backend (AAP): Leverages the Red Hat Ansible Automation Platform to store and execute the custom template logic required for provisioning.
Bare Metal Fulfillment Operator: Kubernetes operator for managing bare-metal host pools. It watches BareMetalPool custom resources and reconciles them to their desired state by provisioning pools of bare-metal hosts organized by host type (e.g., GPU nodes, worker nodes). Uses profile templates to configure workflows and apply configuration parameters to selected hosts. It also contains a Kubernetes controller that manages bare-metal hosts via OpenStack Ironic. It watches BareMetalInstance CRs (defined by the Bare Metal Fulfillment Operator) and reconciles power state with Ironic.

Prerequisites & Setup

System Requirements This solution requires the following platforms to be installed and operational:

Red Hat OpenShift Advanced Cluster Management (RHACM)

Red Hat OpenShift Virtualization (OCP-Virt) - Optional: Only required for VM as a Service (VMaaS) support

Red Hat Ansible Automation Platform (AAP)

A network backend for bare metal provisioning: either ESI (Elastic System Infrastructure) or Netris (see Network Backend Configuration)

Configuration Manifests

The /prerequisites directory contains additional manifests required to configure the target Hub cluster.

Warning: Cluster-Wide Impact If you are using a shared cluster or are not the primary administrator, do not apply these manifests without consultation. These files modify cluster-wide settings. Please coordinate with the appropriate cluster administrators before proceeding.

Prerequisites Summary

Category	Requirement	Notes / Details
Platform	Red Hat OpenShift Container Platform (OCP) 4.17 or later	Must have cluster admin access to the hub cluster.
Operators	Red Hat Advanced Cluster Management (RHACM) 2.18+ Red Hat OpenShift Virtualization (OCP-Virt) 4.17+ Red Hat Ansible Automation Platform (AAP) 2.5+	These must be installed and running prior to OSAC installation.
CLI Tools	`oc` (OpenShift CLI) v4.17+ `helm` v3.x `git`	Ensure all CLIs are available in your `PATH`.
Container Registry Access	`registry.redhat.io` and `quay.io`	Verify credentials and pull secrets are valid in the target cluster namespace.
Network / DNS	Ingress route configured for OSAC services	Required for external access to fulfillment API and AAP UI.
Authentication / IDM	Organization Identity Provider (e.g., Keycloak, LDAP, RH-SSO)	Used for tenant and user identity mapping.
Storage	Dynamic storage class available (e.g., `ocs-storagecluster-cephfs`, `lvms-storage`)	Required for persistence of operator and AAP components.
Permissions	Cluster-admin access to deploy operators and create CRDs	Limited access users can only deploy into namespaces configured by the admin.
License Files	`license.zip` (AAP subscription)	Must be placed in your values directory (e.g., `values/<env>/license.zip`).
Internet Access	Outbound access to GitHub (for fetching submodules, releases)	Required during installation and updates.

Installation

Obtaining an AAP License (Subscription Manifest)

The AAP license is a Subscription Manifest (a .zip file), not a key file. To obtain it:

Log in to the Red Hat Customer Portal.
Navigate to Subscriptions > Subscription Allocations.
Create or select an allocation: If you haven't created one, click "New Subscription Allocation" and set the type (usually "Satellite 6.x").
Add entitlements: Click on your allocation, go to the Subscriptions tab, and add your Ansible Automation Platform subscriptions.
Download: Click Export Manifest to download the .zip file.

Place the downloaded license.zip file in your values directory (e.g., values/development/license.zip).

Pre-Installation Steps

1. Initialize Submodules

The OSAC installer uses Git submodules for version tracking:

$ git submodule update --init --recursive

2. Populate Local Secrets

Ensure your values directory contains the necessary secret files:

AAP License: Place license.zip in values/<env>/
Pull Secret: Place pull-secret.json in values/<env>/

Installation

OSAC installs in three phases via make install. Each phase's outputs are the next phase's inputs — this is required because Helm validates all templates before applying any, and later phases depend on CRDs and secrets that earlier phases create. See docs/helm-deployment-guide.md for a detailed explanation.

# Full install (all 3 phases)
make install VALUES_FILE=values/<env>/values.yaml

# Or run phases individually:
make install-operators VALUES_FILE=values/<env>/values.yaml   # Phase 1: OLM operators
make install-prereqs VALUES_FILE=values/<env>/values.yaml     # Phase 2: CRD instances, Keycloak, certs
make install-osac VALUES_FILE=values/<env>/values.yaml        # Phase 3: OSAC application

Variable	Default	Description
`INSTALLER_NAMESPACE`	`osac`	Target namespace for the OSAC deployment
`VALUES_FILE`	`values/development/values.yaml`	Helm values file to use

Configure Values

Copy and customize a values file for your environment:

mkdir -p values/<project-name>
cp values/development/values.yaml values/<project-name>/values.yaml
# Edit to match your cluster (see values file comments for guidance)

Prerequisites (cert-manager, AAP, LVMS, MetalLB, CNV, MCE) are installed automatically by Phase 1. Each is gated by a values toggle (e.g., certManager.enabled: true). See prerequisites/README.md for details on what each prerequisite provides.

AAP Configuration

AAP instance groups carry backend credentials for provisioning jobs. Configure via Helm values under aap.instanceGroups.* in your values file.

See docs/aap-configuration.md for details.

Network Backend Configuration (CaaS)

By default the network backend is ESI. To switch to Netris, set the Netris-specific values in your values file under aap.instanceGroups.clusterFulfillment.

See docs/network-backend.md for Netris-specific variables and the NETRIS_RESOURCE_CLASS_MAP format.

DNS Backend Configuration (CaaS)

DNS record management uses a pluggable backend. The default is AWS Route 53.

See docs/dns-backend.md for backend details, the interface contract, and how to add a new provider.

Verify

helm status osac -n <project-name>
oc get pods -n <project-name>
oc logs -f job/osac-aap-bootstrap -n <project-name>

Upgrading

helm upgrade osac charts/osac/ \
  --namespace <project-name> \
  --values values/<project-name>/values.yaml \
  --timeout 40m \
  --wait

The post-upgrade hook re-publishes cluster templates automatically. This is idempotent - existing templates are updated via PATCH while new templates are created via POST.

Uninstalling

make uninstall

Note: CRDs are preserved after uninstall (they have the helm.sh/resource-policy: keep annotation). To remove them manually: oc delete crd -l app.kubernetes.io/part-of=osac

Makefile Targets

make install             # Full install (all 3 phases)
make install-operators   # Phase 1: OLM operators
make install-prereqs     # Phase 2: Prerequisites
make install-osac        # Phase 3: OSAC application
make uninstall           # Full uninstall (reverse order)
make helm-deps           # Build chart dependencies
make helm-lint           # Lint all charts
make helm-template       # Dry-run render all templates
make helm-validate       # Lint + template (full validation)
make sync-charts         # Update submodules + rebuild dependencies

Monitor Progress

# Monitor pod creation and startup
$ watch oc get -n <project-name> pods

Once helm install completes successfully (including all post-install hooks), OSAC is ready for use. Cluster templates are published automatically by a post-install hook that runs after the AAP bootstrap job.

OSAC CLI: Setup & Usage

To install the CLI and register a hub, follow these steps:

1. Install the Binary

Download the latest release and make it executable.

# Adjust URL for the latest version as needed
$ curl -L -o osac \
    https://github.com/osac-project/fulfillment-service/releases/latest/download/osac_Linux_x86_64
$ chmod +x osac

# Optional: Move to your path
$ sudo mv osac /usr/local/bin/

2. Log in to the Service

Authenticate with the fulfillment API. You will need the route address and a valid token generation script.

$ osac login \
    --address <your-fulfillment-route-url> \
    --token-script "oc create token fulfillment-controller -n <project-name> \
    --duration 1h --as system:admin" \
    --insecure

Tip: Retrieve your route URL using: oc get routes -n <project-name>

3. Register the Hub

To allow the OSAC operator to communicate with the fulfillment service, you must obtain the kubeconfig and register the hub. The script located at scripts/create-hub-access-kubeconfig.sh demonstrates how to generate the kubeconfig for a hub.

# Generate the kubeconfig
$ ./scripts/create-hub-access-kubeconfig.sh

# Register the Hub
$ osac create hub \
    --kubeconfig=kubeconfig.hub-access \
    --id <hub-name> \
    --namespace <project-name>

4. Use the CLI

Once configured, you can use the OSAC CLI to manage clusters and virtual machines. For detailed usage instructions and command reference, see the OSAC CLI documentation.

Accessing Ansible Automation Platform

After deployment, you can access the AAP web interface to monitor jobs and manage automation:

Get the AAP URL

$ oc get route -n <project-name> | grep osac-aap

Note: The main AAP URL will be something like: https://osac-aap-<project-name>.apps.your-cluster.com

Get the AAP Admin Password

# Extract the admin password
$ oc extract secret/osac-aap-admin-password -n <project-name> --to -

Login to AAP

Open the AAP controller URL in your browser
Username: admin
Password: (from the previous step)

AAP API Token

The OSAC operator requires an API token to communicate with AAP. The create-api-token Helm hook creates this automatically during make install-osac. The token is stored in the osac-aap-api-token Secret.

Tearing Down OSAC

To completely remove an OSAC deployment and all its prerequisites, use the teardown script:

# Using defaults (namespace: osac)
$ ./scripts/teardown.sh

# Or specify your namespace
$ INSTALLER_NAMESPACE=<project-name> ./scripts/teardown.sh

# Include all optional services in teardown (must match what was used during setup)
$ EXTRA_SERVICES=true INSTALLER_NAMESPACE=<project-name> ./scripts/teardown.sh

The script removes resources in reverse order:

OSAC CRs (while operator is running for finalizer processing)
Helm release and project namespace
Keycloak
AAP operator
Multicluster Engine and AgentServiceConfig (if MCE_SERVICE=true)
OpenShift Virtualization (if VIRT_SERVICE=true)
LVMS storage service (if STORAGE_SERVICE=true)
MetalLB ingress service (if INGRESS_SERVICE=true)
CA issuer, trust-manager, and cert-manager
Stale API services and CRD cleanup

Warning: This removes all prerequisite operators and their namespaces. If other workloads on the cluster depend on these operators (e.g., cert-manager, MetalLB), do not run this script. Instead, manually uninstall:
$ helm uninstall osac -n <project-name>
$ oc delete namespace <project-name>

Troubleshooting

Common Issues

cert-manager not ready: Ensure cert-manager operator is installed and running
Certificate issues: Check cert-manager logs and certificate status
ImagePullBackOff errors: Verify registry credentials and image string
Cluster templates missing or incomplete: Cluster templates are published automatically by a Helm post-install hook (osac-publish-templates). If osac get clustertemplates returns missing or unexpected results, check the hook pod logs: oc logs job/osac-publish-templates -n <project-name>. See docs/helm-deployment-guide.md for details.

Debug Commands

# Check certificate status
$ oc describe certificate -n <project-name>

# Check pod events
$ oc describe pod -n <project-name> <pod-name>

# Check service endpoints
$ oc get endpoints -n <project-name>

# View component logs
$ oc logs -n <project-name> deployment/fulfillment-service -c server --tail=100

# Get all events in namespace
$ oc get events -n <project-name> --sort-by=.metadata.creationTimestamp

Support

For issues and questions:

Check the troubleshooting section above
Review component logs for error messages
Verify prerequisites are properly installed
Open issues in the respective component repositories

License

This project is licensed under the Apache License 2.0.

Name		Name	Last commit message	Last commit date
Latest commit History 659 Commits
.ai-bot		.ai-bot
.github		.github
base		base
charts		charts
docs		docs
prerequisites		prerequisites
scripts		scripts
values		values
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.gitmodules		.gitmodules
.pre-commit-config.yaml		.pre-commit-config.yaml
.yamllint.yaml		.yamllint.yaml
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
Makefile		Makefile
OSAC-CLI-HOWTO.md		OSAC-CLI-HOWTO.md
OWNERS		OWNERS
README.md		README.md
ct.yaml		ct.yaml

Folders and files

Latest commit

History

Repository files navigation

OSAC Installer

Overview

OSAC Components

Prerequisites & Setup

Prerequisites Summary

Installation

Obtaining an AAP License (Subscription Manifest)

Pre-Installation Steps

1. Initialize Submodules

2. Populate Local Secrets

Installation

Configure Values

AAP Configuration

Network Backend Configuration (CaaS)

DNS Backend Configuration (CaaS)

Verify

Upgrading

Uninstalling

Makefile Targets

Monitor Progress

OSAC CLI: Setup & Usage

1. Install the Binary

2. Log in to the Service

3. Register the Hub

4. Use the CLI

Accessing Ansible Automation Platform

Get the AAP URL

Get the AAP Admin Password

Login to AAP

AAP API Token

Tearing Down OSAC

Troubleshooting

Common Issues

Debug Commands

Support

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages