SOLTI Containers - Lightweight Development Services

A comprehensive Ansible collection for deploying containerized development and testing services using Podman, Quadlets, and systemd integration.

Part of the SOLTI Ansible Collections Suite - This collection provides containerized testing and development infrastructure services.

📖 Quick Reference

Attribute	Value
Collection	jackaltx.solti_containers
Container Runtime	Podman (rootless)
Service Management	Systemd user services via Quadlets
Network	ct-net (shared container network with DNS)
Management Scripts	manage-svc.sh (lifecycle), svc-exec.sh (operations)
Deployment Targets	localhost, remote hosts via SSH
Configuration Files	inventory.yml, ansible.cfg
Orchestrator	../mylab/ (site-specific deployment automation)
Related Collections	solti-monitoring, solti-ensemble

🎯 Project Philosophy

Modern development requires lightweight, ephemeral services that can be quickly deployed, tested, and removed. Virtual machines are too heavy for rapid iteration cycles. This collection addresses the need for:

• Consistent deployment patterns across different services
• Lightweight testing environments using containers instead of VMs
• Easy service lifecycle management (prepare → deploy → verify → remove)
• Standardized configuration with security best practices
• Rapid iteration for development and testing workflows

I always use the latest version of the container. This can be painful sometimes. I did not intend for these to be long running services. They are here to satisfy a need for rapid development on the developer machine.

You may ask why not a Debian distro? That is because Podman is RHEL-focused and the versions on Debian are behind. Debian distributions are generally better at Docker. I prefer to have non-privileged containers.

🚀 Quick Start

There are two places I test: localhost and a Fedora server VM named podman.

There are two scripts that manage container lifecycle:

1. manage-svc.sh - Service lifecycle management:

   • prepare: Create data directories and apply SELinux contexts (RHEL/Fedora)
   • deploy: Deploy pod and containers using Podman quadlets
   • remove: Stop and remove containers (preserves data by default)

2. svc-exec.sh - Task execution (verify, check_upgrade, configure)

Local Deployment

# Deploy container stack
./manage-svc.sh <service> prepare && ./manage-svc.sh <service> deploy

# Verify the container stack
./svc-exec.sh <service> verify

# Check for updated container
./svc-exec.sh <service> check_upgrade

# Clean up (preserves data)
./manage-svc.sh <service> remove

# Full removal using env variables
DELETE_DATA=true DELETE_IMAGES=true ./manage-svc.sh <service> remove

Available services:

Core Infrastructure:

• traefik - SSL reverse proxy with automatic Let's Encrypt
• hashivault - Secrets management

Data Stores:

• redis - Key-value cache and message broker
• elasticsearch - Search and analytics engine
• minio - S3-compatible object storage
• mongodb - Document database
• influxdb3 - Time-series metrics database

Applications:

• mattermost - Team communication platform
• grafana - Metrics visualization dashboards
• gitea - Lightweight Git hosting
• obsidian - Note-taking and knowledge management

Legacy/Disabled:

• wazuh - Security monitoring (disabled - container issues)

Remote Host Deployment

# Deploy to remote host (e.g., podma)
./manage-svc.sh -h podma -i inventory/podma.yml redis prepare
./manage-svc.sh -h podma -i inventory/podma.yml redis deploy

# Verify remote service
./svc-exec.sh -h podma -i inventory/podma.yml redis verify

# Clean up remote host
./manage-svc.sh -h podma -i inventory/podma.yml redis remove

Note: manage-svc.sh will prompt for your sudo password. This is required because containers create files with elevated ownership that your user cannot modify without privileges.

📋 Service Catalog

Infrastructure Services

Service	Purpose	Ports	SSL Domain	Role Path
Traefik	HTTP reverse proxy with SSL termination	8080, 8443, 9999	*.domain.com	roles/traefik
HashiVault	Secrets management and credential storage	8200, 8201	vault.domain.com	roles/hashivault

Data Storage Services

Service	Purpose	Ports	SSL Domain	Role Path
Redis	Key-value cache and message broker	6379, 8081	redis-ui.domain.com	roles/redis
Elasticsearch	Search and analytics engine for logs	9200, 8088	elasticsearch.domain.com	roles/elasticsearch
MinIO	S3-compatible object storage	9000, 9001	minio.domain.com	roles/minio
MongoDB	NoSQL document database	27017	mongodb.domain.com	roles/mongodb
InfluxDB3	Time-series database for metrics	8086	influxdb.domain.com	roles/influxdb3

Application Services

Service	Purpose	Ports	SSL Domain	Role Path
Mattermost	Team communication and notifications	8065	mattermost.domain.com	roles/mattermost
Grafana	Metrics visualization and dashboards	3001	grafana.domain.com	roles/grafana
Gitea	Lightweight Git hosting service	3000, 2222	gitea.domain.com	roles/gitea
Obsidian	Note-taking and knowledge management	TBD	obsidian.domain.com	roles/obsidian

Planned Services

Service	Status	Purpose
Jepson	📋 Planned	Fuzzing framework for security testing
Trivy	📋 Planned	Container vulnerability scanner

🏗️ Architecture Overview

Relationship with SOLTI Orchestrator

This collection is designed to work both standalone and integrated with the SOLTI orchestrator:

• Standalone Mode: Use manage-svc.sh and svc-exec.sh directly in this repository
• Orchestrator Mode: The ../mylab/ orchestrator provides:
  • Centralized inventory management across all SOLTI collections
  • Workflow automation (e.g., deploy-fleur-workflow.sh)
  • Site-specific credentials and tokens (kept out of this collection)
  • Cross-collection service coordination

Key Principle: This collection contains generic, reusable roles. Site-specific data lives in ../mylab/.

The SOLTI Pattern

All services follow a consistent pattern based on the _base role:

┌─────────────────────────────────────────────────────────────────┐
│                        Service Layer                           │
├─────────────────┬─────────────────┬─────────────────┬─────────────┤
│     Redis       │  Elasticsearch  │   Mattermost    │   MinIO     │
│   (Testing)     │   (Analytics)   │  (Communication)│  (Storage)  │
└─────────────────┴─────────────────┴─────────────────┴─────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                      Infrastructure Layer                       │
├─────────────────┬─────────────────┬─────────────────┬─────────────┤
│    Traefik      │   HashiVault    │     _base       │   Quadlets  │
│  (SSL Proxy)    │   (Secrets)     │   (Common)      │  (systemd)  │
└─────────────────┴─────────────────┴─────────────────┴─────────────┘
                              │
┌─────────────────────────────────────────────────────────────────┐
│                       Platform Layer                           │
├─────────────────┬─────────────────┬─────────────────┬─────────────┤
│     Podman      │     systemd     │    SELinux      │   Network   │
│  (Containers)   │   (Services)    │  (Security)     │   (ct-net)  │
└─────────────────┴─────────────────┴─────────────────┴─────────────┘

Service Integration Flow

graph TD
    A[Developer] --> B[manage-svc.sh]
    A --> C[svc-exec.sh]
    
    B --> D[prepare]
    B --> E[deploy] 
    B --> F[remove]
    
    C --> G[verify]
    C --> H[configure]
    C --> I[backup]
    
    D --> J[_base/prepare]
    E --> K[_base/containers]
    F --> L[_base/cleanup]
    
    J --> M[Directories]
    J --> N[SELinux]
    K --> O[Quadlets]
    K --> P[systemd]
    
    subgraph "Service Layer"
        Q[Redis]
        R[Elasticsearch] 
        S[Mattermost]
        T[MinIO]
    end
    
    subgraph "Infrastructure"
        U[Traefik SSL]
        V[HashiVault]
        W[Container Network]
    end
    
    O --> Q
    O --> R
    O --> S
    O --> T
    
    Q --> U
    R --> U
    S --> U
    T --> U

Loading

🛠️ Management Interface

Primary Commands

Service Lifecycle (manage-svc.sh)

# System preparation (one-time per service)
./manage-svc.sh <service> prepare

# Deploy and start service
./manage-svc.sh <service> deploy

# Remove service (preserves data by default)
./manage-svc.sh <service> remove

Note: Requires sudo password - containers create files with elevated ownership requiring privilege escalation for prepare/deploy/remove operations.

Service Operations (svc-exec.sh)

# Execute verification tasks
./svc-exec.sh <service> verify

# Run service-specific tasks
./svc-exec.sh <service> configure
./svc-exec.sh <service> backup
./svc-exec.sh <service> initialize

# Use sudo for privileged operations
./svc-exec.sh -K <service> <task>

Script Features

• Dynamic playbook generation - Creates Ansible playbooks on-the-fly
• Inventory integration - Uses your inventory variables and defaults
• Error handling - Preserves generated playbooks on failure for debugging
• Cleanup automation - Removes successful temporary playbooks
• Flexible task execution - Any role task file can be executed independently

🌐 SSL and Networking

Automatic SSL with Traefik

When Traefik is deployed, all services automatically get SSL termination:

# Deploy Traefik first
./manage-svc.sh traefik prepare
./manage-svc.sh traefik deploy

# Now all other services get automatic SSL
./manage-svc.sh redis deploy     # → https://redis-ui.yourdomain.com
./manage-svc.sh mattermost deploy # → https://mattermost.yourdomain.com

DNS Requirements

Point wildcard DNS to your development machine:

*.yourdomain.com → 192.168.1.100

Container Networking

All services use a common network with consistent DNS:

service_network: "ct-net"
service_dns_servers: ["1.1.1.1", "8.8.8.8"]
service_dns_search: "yourdomain.com"

🔧 System Requirements

Minimum Requirements

• OS: RHEL 9+, CentOS 9+, Debian 12+, Ubuntu 22.04+
• Podman: 4.x or later
• systemd: User services enabled (loginctl enable-linger $USER)
• Memory: 4GB RAM minimum, 8GB recommended
• Storage: 20GB free space for service data

Recommended Setup

• CPU: 4+ cores for multiple concurrent services
• Memory: 16GB for full service stack
• Storage: SSD storage for better I/O performance
• Network: Stable internet for Let's Encrypt certificates

Prerequisites Installation

# RHEL/CentOS/Rocky Linux
sudo dnf install podman ansible-core python3-pip
pip3 install --user containers.podman

# Debian/Ubuntu  
sudo apt install podman ansible python3-pip
pip3 install --user containers.podman

# Enable user services
loginctl enable-linger $USER

📁 Project Structure

solti-containers/
├── roles/                          # Service role definitions
│   ├── _base/                      # Common functionality
│   │   ├── tasks/
│   │   │   ├── prepare.yml         # Directory and permission setup
│   │   │   ├── networks.yml        # Container networking
│   │   │   └── cleanup.yml         # Service removal
│   │   └── defaults/main.yml       # Common defaults
│   │
│   ├── redis/                      # Redis key-value store
│   ├── elasticsearch/              # Search and analytics
│   ├── hashivault/                 # Secrets management
│   ├── mattermost/                 # Team communication
│   ├── traefik/                    # SSL reverse proxy
│   └── minio/                      # S3-compatible storage
│
├── inventory.yml                   # Service configuration
├── ansible.cfg                     # Ansible settings
├── manage-svc.sh                   # Service lifecycle management
├── svc-exec.sh                     # Task execution wrapper
└── README.md                       # This file

Role Structure Pattern

Each service role follows this structure:

roles/<service>/
├── defaults/main.yml               # Default variables
├── handlers/main.yml               # Service restart handlers
├── meta/main.yml                   # Role metadata
├── tasks/
│   ├── main.yml                   # Role entry point
│   ├── prepare.yml                # System preparation
│   ├── prerequisites.yml          # Configuration setup
│   ├── quadlet_rootless.yml       # Container deployment
│   ├── verify.yml                 # Health verification
│   └── <service-specific>.yml     # Custom tasks
└── templates/
    ├── <service>.conf.j2          # Service configuration
    └── <service>.env.j2           # Environment variables

🔐 Security Model

Container Security

• Rootless containers - All services run without root privileges
• SELinux integration - Proper security contexts on RHEL systems
• Network isolation - Services communicate via dedicated container network
• Resource limits - Memory and CPU constraints prevent resource exhaustion

Access Control

• Localhost binding - Services bind to 127.0.0.1 by default
• Password protection - All services require authentication
• SSL/TLS encryption - Traefik provides automatic HTTPS
• API tokens - Role-based access where supported

Data Protection

• Volume encryption - Data stored in user directories with proper permissions
• Backup integration - Services support data backup and restore
• Secrets management - HashiVault integration for credential storage

🧪 Development Workflows

Testing New Features

# Create isolated test environment
./manage-svc.sh redis deploy
./manage-svc.sh elasticsearch deploy

# Run your tests against the services
pytest tests/ --redis-url=localhost:6379 --es-url=localhost:9200

# Analyze results
./svc-exec.sh redis verify
./svc-exec.sh elasticsearch verify

# Clean up
./manage-svc.sh redis remove
./manage-svc.sh elasticsearch remove

Service Development

# Test single service changes
./manage-svc.sh myservice prepare
./manage-svc.sh myservice deploy
./svc-exec.sh myservice verify

# Make changes to role
vim roles/myservice/tasks/main.yml

# Redeploy with changes
./manage-svc.sh myservice deploy
./svc-exec.sh myservice verify

Iterative Development with Data Persistence

The -K flag combined with data preservation enables rapid "iterate until you get it right" workflow:

# Initial deployment
./manage-svc.sh elasticsearch deploy

# Test and discover issues
./svc-exec.sh elasticsearch verify

# Remove container (data preserved by default)
./manage-svc.sh elasticsearch remove

# Modify role: edit tasks, templates, configuration
vim roles/elasticsearch/tasks/prerequisites.yml

# Redeploy with your changes - data still intact
./manage-svc.sh elasticsearch deploy

# Your test data, indices, and configurations persist!
# Repeat this cycle until working correctly

Key Benefits:

• Data persists across cycles: Elasticsearch indices, Mattermost channels, database records remain intact
• Faster iteration: No need to recreate test data after each change
• True testing: Work with realistic data throughout development

Data-Centric Services (benefit from persistence):

• elasticsearch (indices, mappings)
• mattermost (channels, messages, users)
• minio (buckets, objects)
• hashivault (secrets, policies)

Stateless Services (less critical):

• redis (just cache)
• traefik (just proxy configuration)

When to Reset Data:

# Set in inventory.yml for full cleanup
elasticsearch_delete_data: true
./manage-svc.sh elasticsearch remove  # Removes data directories

Integration Testing

# Deploy full stack
for service in traefik redis elasticsearch mattermost; do
    ./manage-svc.sh $service prepare
    ./manage-svc.sh $service deploy
done

# Run integration tests
./svc-exec.sh traefik verify
for service in redis elasticsearch mattermost; do
    ./svc-exec.sh $service verify
done

# Test cross-service communication
./test-integration.sh

🔄 Data Management

Backup Strategy

Each service supports backup operations:

# Backup service data
./svc-exec.sh <service> backup

# Backup with compression
./svc-exec.sh <service> backup --compress

# Backup to specific location
./svc-exec.sh <service> backup --dest /backup/location

Data Persistence

• Data preservation - remove command preserves data by default
• Complete cleanup - Set <SERVICE>_DELETE_DATA=true to remove all data
• Volume management - Data stored in ~/service-data/ directories
• Migration support - Data directories can be moved/copied between systems

Disaster Recovery

# Backup critical service data
for service in hashivault mattermost; do
    ./svc-exec.sh $service backup
done

# Restore from backup
./svc-exec.sh <service> restore --from /backup/location

# Verify restored service
./svc-exec.sh <service> verify

🎛️ Monitoring and Observability

Health Monitoring

# Check all service status
systemctl --user status | grep -E "(redis|elasticsearch|mattermost)"

# Resource utilization
podman stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# Network connectivity
./svc-exec.sh traefik verify

Log Aggregation

# Centralized logging via Elasticsearch
./manage-svc.sh elasticsearch deploy

# Configure log forwarding (example)
for service in redis mattermost; do
    ./svc-exec.sh $service configure-logging
done

# Search logs via Elasticvue
open https://elasticsearch.yourdomain.com:8088

Performance Monitoring

# Service-specific metrics
./svc-exec.sh redis info
./svc-exec.sh elasticsearch stats
./svc-exec.sh mattermost metrics

# Container resource usage
podman stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}\t{{.BlockIO}}"

🔧 Troubleshooting

Common Issues

Services won't start

# Check systemd status
systemctl --user status <service>-pod

# Check container logs
podman logs <service>-svc

# Verify directory permissions
ls -la ~/<service>-data/

SSL certificates not working

# Check Traefik status
./svc-exec.sh traefik verify

# Verify DNS configuration
dig +short *.yourdomain.com

# Check certificate logs
podman logs traefik-svc | grep -i acme

Network connectivity issues

# Check container network
podman network inspect ct-net

# Test inter-service communication
podman exec redis-svc ping elasticsearch-svc

# Verify port bindings
ss -tlnp | grep -E "(6379|9200|8065)"

Debug Mode

# Enable debug logging
export SOLTI_DEBUG=1

# Run with verbose output
./manage-svc.sh <service> deploy -vvv

# Check generated playbooks
ls -la tmp/<service>-*.yml

🤝 Contributing

Adding New Services

1. Follow the Service Template
2. Implement the standard task files
3. Add Traefik integration labels
4. Include comprehensive verification tasks
5. Update management scripts as needed

Development Guidelines

1. Consistency - Follow established patterns
2. Documentation - Include comprehensive README
3. Testing - Add verification tasks
4. Security - Implement proper access controls
5. Integration - Support Traefik SSL and HashiVault secrets

Testing Changes

# Test role syntax
ansible-playbook --syntax-check roles/<service>/tasks/main.yml

# Test deployment
./manage-svc.sh <service> prepare
./manage-svc.sh <service> deploy
./svc-exec.sh <service> verify

# Test cleanup
./manage-svc.sh <service> remove

Molecule Testing

Test services across multiple platforms (Debian 12, Rocky 9, Ubuntu 24) using nested containers:

# Test service on all platforms
./run-podman-tests.sh --services redis

# Test specific platform
./run-podman-tests.sh --platform uut-deb12 --services redis

# Test multiple services
./run-podman-tests.sh --services "redis,traefik,hashivault"

# View test results
tail -f verify_output/latest_test.out
cat verify_output/debian/consolidated_test_report.md

See molecule/README.md for comprehensive testing documentation.

📚 Additional Resources

Documentation

• Service Template - Template for creating new services
• Traefik Integration Guide - SSL setup and configuration
• Security Best Practices - Security recommendations
• Performance Tuning - Optimization guidelines

External References

• Podman Documentation
• Quadlets Guide
• Ansible Collections
• Traefik Documentation

📄 License

MIT License - See LICENSE file for details.

👨‍💻 Maintained By

Jackaltx - Created for development testing workflows with significant assistance from Claude AI for pattern development and documentation.

Project Goals

This project aims to provide:

• Lightweight alternatives to heavy VM-based development environments
• Consistent patterns for containerized service deployment
• Easy-to-use management interfaces for rapid iteration
• Production-ready security and monitoring capabilities
• Educational examples of modern container orchestration

Support

• Issues: Report bugs or request features via GitHub issues
• Documentation: Comprehensive README files in each role
• Community: Share your service implementations and improvements

---

🤖 Metadata for AI Tools

This section provides structured information for AI documentation tools and code assistants.

Collection Metadata

collection_name: jackaltx.solti_containers
collection_type: ansible_collection
namespace: jackaltx
version: 1.0.0
container_runtime: podman
service_manager: systemd
deployment_mode: rootless
network_model: shared_bridge

Key Files Reference

File	Purpose	Used By
manage-svc.sh	Service lifecycle (prepare/deploy/remove)	Human operators, orchestrator
svc-exec.sh	Task execution (verify/configure)	Human operators, CI/CD
inventory.yml	Service configuration variables	Ansible playbooks
ansible.cfg	Ansible settings and vault config	Ansible engine
roles/_base/	Common functionality for all services	All service roles
CLAUDE.md	AI assistant context and patterns	Claude Code, AI tools

Service Role Structure

All service roles follow this standard structure:

roles/<service>/
├── defaults/main.yml          # Default variables
├── handlers/main.yml          # systemd restart handlers
├── tasks/
│   ├── main.yml              # Entry point (includes other tasks)
│   ├── prepare.yml           # Directory/permission setup
│   ├── prerequisites.yml     # Service-specific config
│   ├── quadlet_rootless.yml  # Container deployment
│   └── verify.yml            # Health checks
├── templates/
│   ├── <service>.conf.j2     # Configuration files
│   └── <service>.env.j2      # Environment variables
└── README.md                 # Service-specific documentation

Common Ansible Variables

# Network Configuration
service_network: "ct-net"
service_dns_servers: ["1.1.1.1", "8.8.8.8"]
service_dns_search: "{{ domain }}"
domain: "example.com"

# Service Paths
<service>_data_dir: "{{ ansible_env.HOME }}/<service>-data"
<service>_config_dir: "{{ <service>_data_dir }}/config"

# Container Lifecycle
<service>_delete_data: false    # Preserve data on remove
<service>_delete_images: false  # Preserve images on remove

# Security
ansible_become: true            # Required for prepare/deploy/remove
secure_logging: true            # Hide credentials in logs

Related Documentation

• Parent Project: ../README.md - SOLTI Collections Suite overview
• Orchestrator: ../mylab/README.md - Deployment automation
• Monitoring: ../solti-monitoring/README.md - Metrics and logging
• Ensemble: ../solti-ensemble/README.md - Shared infrastructure services

---

Happy containerizing! 🐋