Skip to content
Techwasti-InfotechPublic

Repository files navigation

OpenMimic ⚡

The 100% Free & Open-Source, Visual-First Synthetic Data & Traffic Generation Platform.
Effortlessly generate high-throughput streaming datasets, simulate production traffic patterns, and test distributed architectures across Kafka, PostgreSQL, S3, and standard output.

License Go Report Card React Buy Me A Coffee GitHub Sponsors


🌟 Support Open-Source & Buy Us a Coffee

OpenMimic is 100% free, open-source software released under the Apache 2.0 license. Anyone can freely use, modify, distribute, and run OpenMimic for personal, educational, or commercial production workloads with zero licensing fees.

If OpenMimic saves your team time, speeds up your testing, or helps validate your data infrastructure, please consider supporting continued development:


⚡ Key Features

  • 🎯 High-Performance Engine (Go 1.23+): Multithreaded streaming engine generating tens of thousands of records per second directly to sinks.
  • 🎨 Mimic Studio IDE: Modern React 18 + Mantine visual interface for designing schemas, viewing live 10-event streaming previews, inspecting statistical fidelity, and tracking real-time metrics.
  • 🤖 Mimic Copilot: FastAPI + LangChain AI assistant that converts plain-English prompts into validated Protocol Buffers configurations.
  • ⚖️ Predicate Constraints Solver: Define cross-field business logic (amount > 50000 && fraud_flag == 1). Engine samples and evaluates candidate events with bounded retries.
  • 🔬 Statistical Fidelity & Quality Gates: Built-in two-sample Kolmogorov-Smirnov test (KolmogorovSmirnovTest) and Pearson cross-column correlation (PearsonCorrelation) to compare synthetic streams against baseline data.
  • 🛡️ Privacy & PII Scanner: Proactive detector highlighting unmasked sensitive data (email, ssn, phone, card) with recommended masking transforms.
  • 💥 Anomaly Injection Playbooks: Inject real-world edge cases (null field injection, corrupt payload bytes, burst spikes) at configurable rates.
  • 🌊 Realistic Traffic Shaper:
    • Diurnal (Day/Night) Cycles: Sine/cosine waves simulating daily traffic curves.
    • Poisson Arrivals: Stochastic inter-arrival times ($-\ln(U)/\lambda$).
    • Burst Spikes: Periodic surges simulating flash sales or load testing.
  • 🔄 Relational Entity Pools & Markov Chains:
    • Share customer IDs across multiple topics with bounded entity pools (EntityPool).
    • Model multi-step user funnels through probabilistic transition matrices (MarkovChain).
  • 📥 SQL DDL Schema Auto-Import: Paste any CREATE TABLE statement to auto-generate a matching generator configuration.
  • 🔌 Universal Sinks: Apache Kafka (with Confluent Schema Registry TopicRecordNameStrategy & EOS Transactions), Apache Iceberg V2 ACID Table Sink, Delta Lake ACID Log Sink, PostgreSQL (pgx), Amazon S3 / MinIO (with Snappy/GZIP/ZSTD Columnar Parquet), Snowflake Streaming, Google BigQuery, GraphQL mutations, gRPC streams, Redis, ClickHouse, and NDJSON Stdout.
  • 📦 Enterprise Security & Identity: Single Sign-On (SSO) with Okta, Microsoft Entra ID (Azure AD), Keycloak, and SAML 2.0; automatic user & group provisioning via RFC 7643/7644 SCIM 2.0.
  • 🧠 Model Context Protocol (MCP) Server: Native Go MCP server (mimic mcp / mimic-mcp) enabling LLMs (Claude Desktop, Cursor, Antigravity, Copilot) to generate preview events, validate schemas, convert SQL DDL, and profile sample records autonomously via standard JSON-RPC 2.0.

OpenMimic MCP Architecture

  • ⚡ Advanced Streaming: Complex Event Processing (CEP) window rollups (sum, avg, min, max, count), synthetic camera video broadcaster (real-time MJPEG frames with motion tracking and OCR timecodes), PCM WAV audio synthesizers, visual DAG canvas, and schema version control with visual diff.

🖥️ One-Click Local Setup Guide

We provide one-click scripts for Windows, macOS, and Linux that launch the Engine, Copilot, and Studio UI simultaneously with a single command!

Option 1: One-Click Launcher

Windows (PowerShell):

.\run-local.ps1

Windows (Command Prompt / Batch):

run-local.bat

macOS / Linux:

chmod +x run-local.sh
./run-local.sh

The launcher will verify prerequisites, build the Go binaries, and start:

  • Mimic Studio: http://localhost:3000
  • Mimic Engine REST/Bridge: http://localhost:8080
  • Engine gRPC: localhost:50051
  • Mimic Copilot Service: http://localhost:8000

Option 2: Docker Compose

  1. Clone the repository:

    git clone https://github.com/Techwasti/OpenMimic.git
    cd OpenMimic
  2. Start the Engine, Copilot, and Redpanda Kafka broker:

    docker compose up -d
  3. Launch Mimic Studio:

    cd studio
    npm install
    npm run dev

    Open http://localhost:3000 in your browser.


🛠️ Mimic GitOps CLI (mimic-cli)

OpenMimic includes a full-featured GitOps command-line tool for local testing, automation, and CI/CD pipelines.

Build CLI Binary

cd engine
go build -o bin/mimic-cli.exe ./cmd/mimic-cli

CLI Subcommands

  • Validate Schema & Constraints:
    .\engine\bin\mimic-cli.exe validate --config sample_order_flow.json --constraints
  • Stream Event Preview to Terminal:
    .\engine\bin\mimic-cli.exe preview --config sample_order_flow.json --count 5
  • Dry-Run Event Generation (Null Sink):
    .\engine\bin\mimic-cli.exe run --config sample_order_flow.json --dry-run
  • Export Config as Starter Template:
    .\engine\bin\mimic-cli.exe export --config sample_order_flow.json --to-template
  • Auto-Discover from SQL DDL:
    .\engine\bin\mimic-cli.exe discover --sql "CREATE TABLE users (id UUID, email VARCHAR, age INT);"

📋 Sample Configuration (mimic.json)

{
  "id": "gen-signups-traffic",
  "name": "User Signups with Bursts & Constraints",
  "seed": 42,
  "event_limit": 1000,
  "rate_limit_per_second": 20,
  "sink": {
    "kafka": {
      "brokers": ["localhost:9092"],
      "topic": "user.signups",
      "client_id": "openmimic-engine"
    }
  },
  "constraints": [
    {
      "id": "c_amount",
      "name": "Positive Amount",
      "expression": "amount > 0",
      "max_resample_retries": 100
    }
  ],
  "playbooks": [
    {
      "name": "Corrupt Payload Edge Case",
      "anomaly_rate": 0.05,
      "injection_type": "corrupt_payload"
    }
  ],
  "traffic_profile": {
    "burst": {
      "normal_rate": 10,
      "burst_rate": 100,
      "interval_seconds": 20,
      "duration_seconds": 5
    }
  },
  "fields": [
    { "name": "user_id", "generator": { "entity_pool": { "pool_size": 100, "entity_prefix": "usr_" } } },
    { "name": "email", "generator": { "string_faker": { "category": "email" } } },
    { "name": "amount", "generator": { "distribution": { "normal": { "mean": 120.0, "std_dev": 25.0 } } } },
    { "name": "signup_timestamp", "generator": { "now": { "format": "RFC3339" } } }
  ]
}

🤝 Contributing

We welcome community contributions! Please check out CONTRIBUTING.md to get started with our development guidelines, issue templates, and pull request workflows.


📄 License

OpenMimic is 100% open-source software licensed under the Apache License 2.0. See the LICENSE file for details.