Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AnomalyShield: Multi-Tenant Distributed ML-Powered WAF

AnomalyShield is an internal agentic Web Application Firewall (WAF) designed to analyze normal API traffic baseline characteristics per tenant/API key and automatically isolate anomaly events (e.g. volumetric spam, payload size outliers, missing client headers) in real-time.

The system features a decoupled, multi-tenant architecture:

  1. Control Plane (Next.js): Developer dashboard for provisioning API keys, tracking rates, and configuring tenant client access.
  2. Multi-Tenant ML Engine (Python/FastAPI): High-speed engine with per-tenant IsolationForest models dynamically loaded via an LRU cache, async batch traffic logging, background model retraining, and a global baseline cold-start fallback.
  3. Data Plane (Go Middleware): Lightweight middleware that wraps standard Go routers (net/http) to intercept client requests, extract metadata, query the ML engine, and handle connection hijacking or fail-open fallbacks.

System Architecture

graph TD
    Client[Client Browser/API] -->|HTTP Request + X-API-Key| GoWAF[Go Middleware Data Plane]
    GoWAF -->|1. Extract Features| MetricParser[Request Metrics: Size, Rate, Headers]
    GoWAF -->|2. Evaluate Traffic (25ms timeout)| FastAPI[Python ML Engine]
    
    subgraph Multi-Tenant ML Engine
        FastAPI -->|Resolve Tenant| KeyCheck[API Key Authentication & Cache]
        KeyCheck -->|Fetch Model| Registry[TenantModelRegistry LRU Cache]
        Registry -->|Model Hit| TenantModel[Dedicated Tenant IsolationForest]
        Registry -->|Cold Start| BaseModel[Global Baseline IsolationForest]
        FastAPI -->|Log Metrics| Buffer[Async TrafficLogBuffer]
        Buffer -->|Batch Flush| DB[(Shared Database - TrafficLog)]
        Worker[Background Trainer Thread] -->|Read Logs & Retrain| DB
        Worker -->|Save Trained Binary| Disk[models/tenant_id.joblib]
    end

    GoWAF -->|Pass or Block| App[Protected App Handler]
    NextJS[Next.js Dashboard Control Plane] -->|Manage Users & Keys| DB
    
    style GoWAF fill:#0a305e,stroke:#38bdf8,stroke-width:2px,color:#fff
    style FastAPI fill:#3b2057,stroke:#a855f7,stroke-width:2px,color:#fff
    style Registry fill:#1f2937,stroke:#a855f7,stroke-width:2px,color:#fff
    style NextJS fill:#0c3e30,stroke:#10b981,stroke-width:2px,color:#fff
    style DB fill:#1f2937,stroke:#94a3b8,stroke-width:2px,color:#fff
Loading

Directory Structure

WAFproject/
├── anomalyshield/          # Go Middleware Module
│   ├── anomalyshield.go    # Interceptor and rate limit tracker
│   └── anomalyshield_test.go # Test cases (blocking, timeout, fail-open)
├── anomaly_shield_waf/     # Python Multi-Tenant ML Engine (FastAPI)
│   ├── config.py           # Configuration environment variables & thresholds
│   ├── database.py         # DB connection, API keys, TrafficLog & TenantModelMeta
│   ├── model.py            # TenantModelRegistry (LRU), TrafficLogBuffer & BackgroundTrainer
│   ├── main.py             # FastAPI HTTP server & management endpoints
│   └── test_service.py     # Multi-tenant test suite (Phases 1-5)
├── frontend/               # Next.js Control Plane (Dashboard)
│   ├── prisma/             # Schema definitions (SQLite + Prisma models)
│   └── src/                # Authentication, components & layout
├── waf_demo/               # Demo Application Wrapper
│   ├── main.go             # Go server wrapping WAF middleware
│   └── go.mod              # Demo configuration pointing to local middleware
├── provision_test_key.py   # DB helper script to seed mock dev credentials
└── verify.sh               # Verification test suite runner

Key Features & Production-Ready Architecture

  • Phase 1: Metric Logging & Storage: Async batch logging buffers request metrics in memory (TrafficLogBuffer) and flushes to TrafficLog table in batches to eliminate per-request DB write overhead.
  • Phase 2: Automated Background Trainer: A background thread (BackgroundTrainer) periodically scans tenants and trains dedicated IsolationForest models for tenants with $\ge 1,000$ traffic records.
  • Phase 3: Model Management & LRU Cache: TenantModelRegistry provides a thread-safe LRU cache (default max 100 models in RAM) with memory eviction and dynamic disk loading (models/tenant_<user_id>.joblib).
  • Phase 4: Cold-Start & Fallback: Tenants with insufficient data dynamically fall back to a pre-trained global baseline model.
  • Phase 5: Management Endpoints: FastAPI exposes API routes for tenant status (GET /tenants/{user_id}/status), manual retraining (POST /tenants/{user_id}/train), cache status (GET /cache), and buffer flushing (POST /logs/flush).

Setup & Installation

Prerequisite Checklist

  • Go Compiler (v1.18+)
  • Python (v3.10+) & pip
  • Node.js (v18+) & npm

1. Install Dependencies

# Install Python ML server dependencies
pip3 install fastapi uvicorn scikit-learn joblib anyio

# Install Next.js frontend dependencies
cd frontend
npm install
cd ..

2. Synchronize Database Schema

cd frontend
npx prisma db push
cd ..

3. Provision the Test API Key

python3 provision_test_key.py

Running the Services

Terminal 1: Start the Multi-Tenant Python WAF Engine

python3 -m anomaly_shield_waf.main

Terminal 2: Start the Next.js Dashboard

cd frontend
npm run dev

Terminal 3: Start the Go Demo Web Server

cd waf_demo
go run main.go

Running Verification Tests

Run the full multi-tenant test suite:

python3 -m anomaly_shield_waf.test_service

Or run the complete verification script:

./verify.sh

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages