CNNFOLE: Co-Evolutionary Neural Network Intrusion Detection System

Overview

CNNFOLE is an advanced co-evolutionary Intrusion Detection System that combines deep learning with genetic algorithms to detect and adapt to evolving network attacks.

Key Features

CNN-based Feature Extraction: 64-dimensional feature vectors from network traffic
RNN Pattern Recognition: Temporal anomaly detection using LSTM
Hybrid Decision Engine: Multi-factor threat assessment
Co-Evolutionary Training: Attacker evolves evasions, IDS learns defenses
Real-time Validation: FP/FN detection and auto-correction
High Performance: 24,668 events/sec with C++ backend
Production Ready: 93%+ accuracy with < 7% FPR

Quick Start

Installation

# Clone repository
git clone https://github.com/patchyevolve/AI-IDS-pipeline.git
cd AI-IDS-pipeline

# Install dependencies
pip install -r ai-architecture/requirements.txt

# Build C++ backend (optional)
python ai-architecture/cpp/build.py

Training with Validation

cd ai-architecture
python start_training_with_validation.bat

This will:

Start IDS with real-time validation
Start Attacker with co-evolutionary mutations
Track FP/FN automatically
Auto-correct database mistakes
Generate validation report

Individual Component Testing

# Test CNN feature extraction
python Stage_1_Foundation_CNN/examples/test_cnn.py

# Test RNN pattern detection
python Stage_2_Pattern_Recognition_RNN/examples/test_rnn.py

# Test decision engine
python Stage_3_Decision_Engine/examples/test_decoder.py

# Test validation system
python tests/test_validation_metrics.py

Architecture

7-Tier System Architecture

┌─────────────────────────────────────────────────────────────┐
│ Tier 7: Visualization & UI                                  │
│ (Dashboard, CLI, Tkinter)                                   │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 6: Threat Intelligence                                 │
│ (MITRE ATT&CK, Campaign Correlation, Behavioral Analysis)   │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 5: Attack Simulation                                   │
│ (Attack Engine, Mutator, Genetic Algorithm)                 │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 4: Validation & Learning                               │
│ (Training Validator, Metrics Tracker, Auto-Corrector)       │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 3: Network & Capture                                   │
│ (IDS Bridge, Firewall Enforcer, Network Config)             │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 2: Data & Storage                                      │
│ (Database Engine, Mutation Predictor, Vector Store)         │
└─────────────────────────────────────────────────────────────┘
                              ↑
┌─────────────────────────────────────────────────────────────┐
│ Tier 1: Core Engines                                        │
│ (CNN Gate, RNN Temporal, Hybrid Decoder)                    │
└─────────────────────────────────────────────────────────────┘

Packet Processing Flow

Network Packet
    ↓
ids_bridge.IDSBridge._packet_callback()
    ├─ Extract Features (source, dest, port, protocol, etc)
    ↓
cnn.cnn_engine.forward()
    ├─ Gate Classifier: is_attack_prob
    ├─ Autoencoder: anomaly_score
    └─ Feature Vector: 64 dimensions
    ↓
rnn.rnn_engine.forward()
    ├─ Temporal Analysis: anomaly_trend
    ├─ Drift Detection: drift_score
    └─ State Update: SSM state
    ↓
database.db_engine.retrieve_memory()
    ├─ Vector Search: similarity matching
    └─ Retrieved: Top-K similar signatures
    ↓
decoder.decoder_engine.decode()
    ├─ Attention Pooling: token fusion
    ├─ Score Fusion: multi-factor decision
    ├─ Threshold Matching: adaptive thresholds
    └─ Decision: Block/Alert/Log/Escalate
    ↓
decoder.mutation_predictor.predict_mutations()
    ├─ Evasion Tactics: predicted mutations
    └─ Mutation Scores: likelihood of evasion
    ↓
threat_intelligence.threat_intelligence_engine.process_attack()
    ├─ MITRE Mapping: tactics & techniques
    ├─ Campaign Correlation: multi-stage attacks
    ├─ Behavioral Analysis: anomaly detection
    └─ Threat Actor: attribution
    ↓
Decision & Action
    ├─ firewall_enforcer.block_ip() (if Block)
    ├─ database.db_engine.log_prediction() (store for learning)
    └─ event_bus.emit("decoder_output") (notify listeners)

Validation & Learning Loop

Decoder Output
    ↓
validation.training_validator._on_decoder_output()
    ├─ Compare with Ground Truth (from attacker metadata)
    ├─ Detect FN (missed attack) or FP (blocked benign)
    ↓
_correct_false_negative() OR _correct_false_positive()
    ├─ Create correction record (confidence=0.95)
    ├─ database.db_engine.log_prediction() (store)
    ├─ database.db_engine.export_ids_signatures() (export)
    └─ event_bus.emit("db_updated") (notify)
    ↓
decoder.mutation_predictor._on_db_updated()
    ├─ mutation_predictor.learn_from_database()
    ├─ Reload patterns immediately
    └─ Patterns reloaded for next packet
    ↓
attacker.attack_engine._receive_feedback()
    ├─ attacker.mutator.fitness_score() (evaluate evasion)
    ├─ attacker.mutator.mutate() (evolve next generation)
    └─ Attacker evolves based on IDS decisions
    ↓
Loop Back to Packet Processing

Cloud Synchronization

Local Database (Instance 1)
    ↓ (Every 5 minutes)
database.db_engine.sync_batch() → Pinecone
    ↓
Cloud Database (Pinecone Vector Store)
    ↓ (Broadcast to all instances)
Local Database (Instance 2)
Local Database (Instance 3)
Local Database (Instance N)
    ↓
All instances have latest signatures

Performance

Metric	Value
Accuracy	93%+
Precision	92%+
Recall	92%+
False Positive Rate	< 7%
False Negative Rate	< 8%
Latency (Python)	100-500 events/sec
Latency (C++)	24,668 events/sec
Per-Event Latency	2-7 µs

Training Results

After training with 22,000+ records:

Database Statistics:
  - ids_signatures.jsonl: 1,685 records (known signatures)
  - refined_threats.jsonl: 22,385 records (learned patterns)
  - synthetic_from_datasets.jsonl: 500 records (base data)

Decision Distribution:
  - Log: 55% (low confidence threats)
  - Escalate: 35% (medium confidence)
  - Alert: 6% (high confidence)
  - Block: 4% (immediate action)

Attack Classes Learned:
  - DoS/DDoS: 952 patterns
  - PortScan: 103 patterns
  - BruteForce: 200+ patterns
  - C2/Exfiltration: 134 patterns
  - Unknown High Severity: 7,689 patterns

Documentation

Main Documentation

README.md - This file (project overview)
PRODUCTION_DEPLOYMENT_ARCHITECTURE.md - Production deployment guide
graphify/GRAPH_CONTEXT.md - Complete codebase architecture reference
graphify/README.md - Codebase graph analysis guide
ai-architecture/README.md - AI-architecture guide

Component Documentation

ai-architecture/attacker/README.md - Attack engine guide
ai-architecture/network/ - Network capture documentation
ai-architecture/validation/README.md - Validation system guide

Training & Quick Start

ai-architecture/train.bat - Main training script
ai-architecture/test_5min.bat - 5-minute test
ai-architecture/TRAINING_GUIDE.bat - Training guide

Components

Tier 1: Core Engines (ML Models)

CNN Engine (ai-architecture/cnn/cnn_engine.py)
- Gate Classifier: attack vs normal
- Autoencoder: anomaly detection
- Feature extraction: 64-dimensional vectors
RNN Engine (ai-architecture/rnn/rnn_engine.py)
- State Space Model: temporal analysis
- Anomaly trend detection
- Drift score calculation
Hybrid Decoder (ai-architecture/decoder/decoder_engine.py)
- Multi-factor fusion (CNN + RNN + DB)
- Attention-based token pooling
- Adaptive thresholds per source IP
- Correlation engine for multi-stage attacks

Tier 2: Data & Storage

Database Engine (ai-architecture/database/db_engine.py)
- Vector Graph Store: local threat signatures
- Pinecone integration: cloud vector database
- Real-time signature export
- Cloud synchronization (every 5 min)
Mutation Predictor (ai-architecture/decoder/mutation_predictor.py)
- Evasion tactic prediction
- Pattern learning from database
- Real-time pattern reload on DB updates

Tier 3: Network & Capture

IDS Bridge (ai-architecture/network/ids_bridge.py)
- Scapy-based packet capture
- Synthetic mode (no network needed)
- Live mode (real network traffic)
- Remote mode (API-based)
Firewall Enforcer (ai-architecture/network/firewall_enforcer.py)
- Real-time packet blocking
- IP reputation management
- Rule optimization
Network Config (ai-architecture/network/net_config.py)
- Interface discovery
- BPF filter configuration
- Network setup UI

Tier 4: Validation & Learning

Training Validator (ai-architecture/validation/training_validator.py)
- Real-time FN/FP detection
- Automatic database correction
- Confidence=0.95 for corrections
- Immediate signature export
Metrics Tracker (ai-architecture/validation/metrics_tracker.py)
- Accuracy, Precision, Recall tracking
- FPR/FNR calculation
- Per-event logging
- Report generation
Auto-Corrector (ai-architecture/validation/auto_corrector.py)
- FN correction: add missed attacks
- FP correction: add blocked benign traffic
- Database persistence

Tier 5: Attack Simulation

Attack Engine (ai-architecture/attacker/attack_engine.py)
- Attack generation and scheduling
- Feedback reception from IDS
- Population management
- Generation evolution
Mutator (ai-architecture/attacker/mutator.py)
- Genetic algorithm implementation
- Evasion-first fitness scoring
- Mutation and crossover operations
- Population diversity management
Attack Profiles (ai-architecture/attacker/attack_profiles.py)
- DoS/DDoS attacks
- C2 Beacon communication
- BruteForce attempts
- PortScan reconnaissance
- DNS Tunneling
- Data Exfiltration

Tier 6: Threat Intelligence

Threat Intelligence Engine (ai-architecture/threat_intelligence/threat_intelligence_engine.py)
- Attack processing and enrichment
- Decision enhancement
- Threat level assessment
MITRE Mapper (ai-architecture/threat_intelligence/mitre_mapper.py)
- ATT&CK tactic mapping
- Technique identification
- Framework alignment
Campaign Correlator (ai-architecture/threat_intelligence/campaign_correlator.py)
- Multi-stage attack detection
- Campaign tracking
- Threat actor correlation
Behavioral Baseline (ai-architecture/threat_intelligence/behavioral_baseline.py)
- Baseline establishment
- Anomaly detection
- Behavioral profiling

Tier 7: Visualization & UI

Dashboard (ai-architecture/visualizer/dashboard.py)
- Pygame-based real-time visualization
- Threat metrics display
- Attack timeline
Tkinter Dashboard (ai-architecture/visualizer/dashboard_tk.py)
- Cross-platform GUI
- Event bus integration
- Live metrics update
CLI Dashboard (ai-architecture/visualizer/fast_cli.py)
- Terminal-based visualization
- Lightweight monitoring
- Remote access support

Supporting Systems

Event Bus (ai-architecture/event_bus.py)
- Pub/sub event system
- Thread-safe communication
- Event filtering and routing
Main Pipeline (ai-architecture/run.py)
- System orchestration
- Component initialization
- Pipeline execution
C++ Backend (ai-architecture/cpp/)
- ids_pipeline.cpp: Python bindings
- ids_mutation_predictor.cpp: Mutation detection
- ids_ebpf.cpp: eBPF packet capture
- 247x performance improvement

Usage Examples

Basic IDS Usage

from ai_architecture.cnn.cnn_engine import CNNEngine
from ai_architecture.rnn.rnn_engine import RNNEngine
from ai_architecture.decoder.decoder_engine import HybridDecoder
from ai_architecture.database.db_engine import DatabaseEngine
from ai_architecture.event_bus import EventBus

# Initialize
bus = EventBus()
cnn = CNNEngine(bus)
rnn = RNNEngine(bus)
db = DatabaseEngine(bus)
decoder = HybridDecoder(bus)

# Process packet
event = {
    "source": "192.168.1.100",
    "destination": "10.0.0.1",
    "port_dst": 80,
    "protocol": 6,
    "flags": 0x02,
    "payload_size": 1024,
    "rate_hz": 100.0,
}

# Get decision
cnn_out = cnn.process_event(event)
rnn_out = rnn.process_features(cnn_out)
db_matches = db.retrieve_memory(embedding=cnn_out["feature_vector"])
decision = decoder.decode(cnn_out, rnn_out, db_matches["retrieved"])

print(f"Decision: {decision['decision']}")
print(f"Confidence: {decision['confidence']:.2f}")

Standalone Component Usage

# Use CNN independently
features = cnn.process_event(event)
print(f"Feature vector: {features['feature_vector']}")

# Use RNN independently
patterns = rnn.process_features(features)
print(f"Anomaly score: {patterns['anomaly_score']}")

# Use database independently
matches = db.retrieve_memory(embedding=features['feature_vector'])
print(f"Similar threats: {len(matches['retrieved'])}")

Training

Co-Evolutionary Training

cd ai-architecture
python start_training_with_validation.bat

Extended Training (1 hour)

cd ai-architecture
python start_extended_training.bat

Custom Training

from ai_architecture.run import main
import sys

sys.argv = ['run.py', '--synthetic', '--validate']
main()

Validation & Metrics

The system tracks:

Accuracy: Overall correctness
Precision: How many alerts are real attacks?
Recall: How many attacks are detected?
F1 Score: Balanced metric
FPR: False Positive Rate
FNR: False Negative Rate

Reports saved to:

validation/validation_report.json - Final metrics
validation/metrics_timeline.jsonl - Per-event log

Codebase Analysis

Codebase Statistics

Total Modules: 204 Python files
Total Dependencies: 196 import relationships
Total Classes: 50+ core classes
Total Functions: 500+ functions
Lines of Code: 50,000+

Module Breakdown

ai-architecture/
├── cnn/                    (CNN implementation)
├── rnn/                    (RNN implementation)
├── decoder/                (Decision engine)
├── database/               (Threat storage)
├── attacker/               (Attack generation)
├── validation/             (FP/FN detection)
├── network/                (Packet capture)
├── threat_intelligence/    (Threat analysis)
├── visualizer/             (Dashboard & UI)
├── cpp/                    (C++ backend)
├── tests/                  (Test suite)
└── run.py                  (Main entry point)

Dependency Graph

The complete codebase dependency graph is available in the graphify/ folder:

graphify/output/codebase_graph.json - Complete graph data (JSON)
graphify/output/codebase_graph.dot - GraphViz format
graphify/GRAPH_CONTEXT.md - Detailed architecture reference
graphify/README.md - Graph analysis guide

Generate fresh graph:

python graphify/generate_codebase_graph.py

Key Dependencies

run.py (Main Pipeline)
├── cnn.cnn_engine (Gate + Autoencoder)
├── rnn.rnn_engine (Temporal analysis)
├── decoder.decoder_engine (Decision making)
├── decoder.mutation_predictor (Evasion prediction)
├── database.db_engine (Signature storage)
├── network.ids_bridge (Packet capture)
├── validation.training_validator (FN/FP correction)
├── attacker.attack_engine (Attack simulation)
├── threat_intelligence.threat_intelligence_engine (Threat analysis)
└── visualizer.dashboard (Real-time visualization)

Event Bus Communication

Key events flowing through the system:

decoder_output - Decoder emits decisions
db_updated - Database emits updates
db_retrieved - Database emits retrieved records
attack_feedback - Firewall emits feedback
threat_intelligence - TI engine emits enrichment

Testing

# Run all tests
python -m pytest tests/

# Run specific test
python tests/test_cpp_ids.py
python tests/test_validation_metrics.py
python tests/test_auto_correction.py

# Run 5-minute co-evolution test
python test_coevo_5min.py

Performance Optimization

Python Backend

Latency: 100-500 events/sec
Memory: 500 MB - 1 GB
Best for: Development, research

C++ Backend

Latency: 24,668 events/sec (247x faster)
Memory: 50-100 MB
Best for: Production deployment

Build C++ backend:

python ai-architecture/cpp/build.py

Deployment

Production Deployment

cd ai-architecture
python run.py --synthetic --cpp

Live Packet Capture

cd ai-architecture
python run.py  # Requires admin/root

Remote Attacker

cd ai-architecture
python attacker/run_attacker.py --remote <IDS_IP>

Architecture Highlights

Co-Evolution

Attacker evolves mutations based on IDS decisions
IDS learns from attacker's evolved patterns
Both systems improve iteratively
Arms race drives continuous improvement

Validation & Auto-Correction

Real-time FP/FN detection
Automatic database correction
Prevents same mistakes from repeating
Metrics guide co-evolution

High Performance

CNN: < 1 µs per packet
RNN: 1-5 µs per sequence
Decoder: 2-10 µs per decision
C++: 2-7 µs total latency

Production Ready

100% feature parity (Python ↔ C++)
Real-time packet capture
Dashboard visualization
Comprehensive logging

Repository Structure

AI-IDS-pipeline/
├── README.md (this file)
├── PRODUCTION_DEPLOYMENT_ARCHITECTURE.md (deployment guide)
├── .gitignore
│
├── graphify/
│   ├── generate_codebase_graph.py (graph generator)
│   ├── README.md (graphify guide)
│   ├── GRAPH_CONTEXT.md (architecture reference)
│   └── output/
│       ├── codebase_graph.json (complete graph)
│       └── codebase_graph.dot (GraphViz format)
│
├── ai-architecture/
│   ├── cnn/ (CNN implementation)
│   │   └── cnn_engine.py
│   ├── rnn/ (RNN implementation)
│   │   └── rnn_engine.py
│   ├── decoder/ (Decision engine)
│   │   ├── decoder_engine.py
│   │   └── mutation_predictor.py
│   ├── database/ (Threat storage)
│   │   └── db_engine.py
│   ├── attacker/ (Attack generation)
│   │   ├── attack_engine.py
│   │   ├── mutator.py
│   │   ├── attack_profiles.py
│   │   └── README.md
│   ├── validation/ (FP/FN detection)
│   │   ├── training_validator.py
│   │   ├── metrics_tracker.py
│   │   └── auto_corrector.py
│   ├── network/ (Packet capture)
│   │   ├── ids_bridge.py
│   │   ├── firewall_enforcer.py
│   │   ├── net_config.py
│   │   └── setup_screen.py
│   ├── threat_intelligence/ (Threat analysis)
│   │   ├── threat_intelligence_engine.py
│   │   ├── mitre_mapper.py
│   │   ├── campaign_correlator.py
│   │   └── behavioral_baseline.py
│   ├── visualizer/ (Dashboard & UI)
│   │   ├── dashboard.py
│   │   ├── dashboard_tk.py
│   │   └── fast_cli.py
│   ├── cpp/ (C++ backend)
│   │   ├── ids_pipeline.cpp
│   │   ├── ids_mutation_predictor.cpp
│   │   ├── ids_ebpf.cpp
│   │   └── build.py
│   ├── tests/ (Test suite)
│   │   ├── test_validation_integration.py
│   │   ├── test_cpp_ids.py
│   │   ├── test_mutation_predictor.py
│   │   └── ... (20+ test files)
│   ├── run.py (Main entry point)
│   ├── train.bat (Training script)
│   ├── test_5min.bat (5-minute test)
│   ├── TRAINING_GUIDE.bat (Training guide)
│   ├── test_coevo_5min.py (Co-evolution test)
│   ├── event_bus.py (Event system)
│   ├── requirements.txt (Dependencies)
│   └── README.md (AI-architecture guide)
│
└── real_datasets/ (Training data)
    └── 19 CSV files from ISCX/NSL-KDD

Requirements

Python 3.9+
NumPy, SciPy, Scikit-learn
PyTorch or TensorFlow
Scapy (for packet capture)
Pinecone (for vector database)
C++17 compiler (for C++ backend)

See ai-architecture/requirements.txt for full list.

Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Submit a pull request

License

MIT License - See LICENSE file for details

Citation

If you use CNNFOLE in your research, please cite:

@software{cnnfole2026,
  title={CNNFOLE: Co-Evolutionary Neural Network Intrusion Detection System},
  author={AI-IDS-Pipeline Contributors},
  year={2026},
  url={https://github.com/patchyevolve/AI-IDS-pipeline}
}

Contact

GitHub: patchyevolve/AI-IDS-pipeline
Issues: GitHub Issues

Acknowledgments

ISCX/NSL-KDD datasets for training data
PyTorch/TensorFlow communities
Scapy for packet processing
Pinecone for vector database

Last Updated: April 22, 2026 Version: 1.0.0

Name		Name	Last commit message	Last commit date
ai-architecture		ai-architecture
graphify		graphify
include		include
.gitignore		.gitignore
ADVANCED_THREAT_INTELLIGENCE.md		ADVANCED_THREAT_INTELLIGENCE.md
PRODUCTION_DEPLOYMENT_ARCHITECTURE.md		PRODUCTION_DEPLOYMENT_ARCHITECTURE.md
README.md		README.md
SYSTEM_STATUS.md		SYSTEM_STATUS.md
clean_separator_comments.py		clean_separator_comments.py

Latest commit

History

Folders and files

Repository files navigation