MIA - Market Intelligence Agent

Market Intelligence Agent (MIA) - A production-ready, event-driven trading intelligence platform designed to detect market anomalies, evaluate trading signals through professional frameworks, and route execution decisions to multiple venues.

Quick Start • Architecture • Configuration • API Reference

Overview

MIA (Market Intelligence Agent) is a layered, event-driven platform that combines:

Real-time market data ingestion from multiple cryptocurrency exchanges (KuCoin, Coinbase, Kraken, Binance, ByBit, CoinGecko)
Six professional signal frameworks for signal evaluation and validation
Structural + indicator-based signal detection with dynamic confidence scaling
Position-aware execution with feedback loops preventing cascade trades
Multi-adapter distribution supporting SSE, WebSocket, Database, and Webhook outputs

Target Use Cases

Automated Trading Systems - MirrorCore-X integration
Governance Triggers - MtaaDAO participation
Risk Management - Real-time market monitoring
Market Analysis - Live dashboard and reporting
External Agents - REST API data access

Key Features

🎯 Signal Detection & Validation

Information Ratio Framework - Quantifies signal edge and alpha generation
Regime-First Analysis - Market context awareness (trending, ranging, volatile)
Liquidity Stack Framework - Order book depth analysis
Lead-Lag Causality - Cross-symbol correlation and dependency detection
Capital Flow Funnel - Volume and momentum analysis
Asymmetric Risk Framework - Downside protection evaluation

🔄 Multi-Exchange Data Integration

Real-time WebSocket feeds for 6+ exchanges
REST API fallback with adaptive retry logic
Unified symbol normalization across all venues
Gap detection and data integrity validation
Intelligent rate limiting with circuit breakers

📊 Advanced Analysis

Market Graph Construction - Symbol correlation and relationship mapping
Timeframe Synthesis - Multi-timeframe analysis from single candle stream
Regime Classification - Automatic market condition detection
Structural Confidence Scoring - Dynamic signal weighting

🔌 Event-Driven Architecture

Real-time event propagation < 100-500ms latency
Redis event fabric for distributed processing
Signal consumer with multi-adapter distribution
Webhook integration for external systems
Position feedback loops for intelligent execution

📈 Data Persistence

PostgreSQL (Neon) for durable state
Redis caching for high-speed access
Schema versioning and migration support

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Market Intelligence Agent (MIA)          │
└─────────────────────────────────────────────────────────────┘

┌─ Layer 1: Market Discovery ──────────────────────────────┐
│  • Market Selector                                       │
│  • Symbol Registry & Normalization                       │
│  • Exchange Market Enumeration                           │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 2: Data Ingestion ────────────────────────────────┐
│  • Multi-Exchange Adapters (KuCoin, Coinbase, Kraken)   │
│  • WebSocket Manager & Connection Pooling               │
│  • Batch Fetcher & Fallback Logic                        │
│  • Adaptive Load Balancer & Circuit Breaker              │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 3: Data Integrity ────────────────────────────────┐
│  • Gap Detection & Validation                            │
│  • Timeframe Synthesis                                   │
│  • Candle Schema Validation                              │
│  • Data Quality Scoring                                  │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 3.5: Market Graph Construction ─────────────────┐
│  • Graph-based Structural Analysis                      │
│  • Lead-Lag Relationships                               │
│  • Correlation Breaking Detection                       │
│  • Sector Rotation Analysis                             │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 4: Signal Evaluation ─────────────────────────────┐
│  • 6 Professional Analysis Frameworks                    │
│  • Regime Classification                                │
│  • Indicator Analysis (RSI, MACD, Bollinger, ADX)       │
│  • Confidence Scaling & Edge Quantification              │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 5: Event Routing ─────────────────────────────────┐
│  • Event Engine & Fabric                                │
│  • Signal Consumer & Distribution                       │
│  • Multi-Adapter Output (SSE, WS, DB, Webhook)          │
│  • Position Feedback Integration                        │
└──────────────────────────────────────────────────────────┘
                          ↓
┌─ Layer 6: Execution Gateway ─────────────────────────────┐
│  • Position Management                                  │
│  • Risk Enforcement                                     │
│  • Execution Routing                                    │
│  • Position Feedback Loop                               │
└──────────────────────────────────────────────────────────┘

┌─ Supporting Services ────────────────────────────────────┐
│  • PostgreSQL (Neon) Data Store                         │
│  • Redis Event Fabric & Caching                         │
│  • MCP Server for Agent Integration                     │
│  • REST/WebSocket APIs for External Access              │
└──────────────────────────────────────────────────────────┘

For detailed architectural documentation, see MIA_SYSTEM_ARCHITECTURE.md.

Prerequisites

Python 3.10+
PostgreSQL 14+ (or Neon database)
Redis 6.0+
pip or poetry for dependency management

System Requirements

Minimum 2GB RAM (4GB+ recommended)
Internet connectivity for exchange API access
Modern CPU (2+ cores recommended)

API Keys Required

You'll need API keys from the exchanges you want to use:

KuCoin - For real-time crypto market data
Coinbase - For US exchange data
Kraken - For European coverage
Binance - For global exchange data (optional)
ByBit - For derivatives data (optional)
CoinGecko - For on-chain metrics (optional, has free tier)

Installation

1. Clone the Repository

git clone https://github.com/yourusername/mia.git
cd mia

2. Create Virtual Environment

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install Dependencies

pip install -r requirements.txt

Core Dependencies:

fastapi - Web framework
uvicorn - ASGI server
ccxt - Crypto exchange integration
aiohttp - Async HTTP client
asyncpg - PostgreSQL async driver
redis - Redis client
pydantic - Data validation
python-dotenv - Environment configuration

4. Set Up Databases

PostgreSQL Setup

# Create database
createdb mia_db

# Run migrations (if available)
python -m alembic upgrade head

Redis Setup

# Using Docker
docker run -d -p 6379:6379 redis:latest

# Or local installation
redis-server

Configuration

Environment Variables

Create a .env file in the project root:

# PostgreSQL Configuration
DATABASE_URL=postgresql://user:password@localhost/mia_db
DB_HOST=localhost
DB_PORT=5432
DB_NAME=mia_db
DB_USER=postgres
DB_PASSWORD=your_password

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0
REDIS_PASSWORD=

# Exchange API Keys
KUCOIN_API_KEY=your_key
KUCOIN_API_SECRET=your_secret
COINBASE_API_KEY=your_key
KRAKEN_API_KEY=your_key
BINANCE_API_KEY=your_key
BYBIT_API_KEY=your_key
COINGECKO_API_KEY=your_key  # Optional, has free tier

# System Configuration
LOG_LEVEL=INFO
ENVIRONMENT=development
DEBUG=false

# Signal Configuration
SIGNAL_IR_THRESHOLD=1.0          # Information Ratio threshold
SIGNAL_CONFIDENCE_MIN=0.65       # Minimum confidence for signals
POSITION_LIMIT_PER_SYMBOL=100000 # Maximum position size
MAX_CONCURRENT_SIGNALS=10        # Max concurrent signal processing

# Webhook Configuration (optional)
WEBHOOK_URL=https://your-domain.com/webhook
WEBHOOK_SECRET=your_secret_key

Configuration Files

config/signal_detection_config.py - Signal parameters
docs/CONFIG_GUIDE.md - Detailed configuration guide
docs/ENVIRONMENT_SETUP.md - Environment setup steps

Quick Start

Step 1: Verify System Health

python -m tests.diagnose_db_connection
python -m tests.check_table_schema

Step 2: Start Market Data Collection

Terminal 1:

python main.py

This will:

Initialize exchange adapters
Connect to WebSocket feeds
Start market data ingestion
Begin signal detection
Emit events to Redis fabric

Expected Output:

INFO:__main__:Starting Market Data Collection...
INFO:adapter_manager:Initializing KuCoinAdapter
INFO:adapter_manager:WebSocket connected to KuCoin
INFO:event_engine:Event engine initialized
INFO:market_integration:Market graph constructed

Step 3: Start Data Access API

Terminal 2:

python -m uvicorn api.intelligence_api:app --host 0.0.0.0 --port 8001 --reload

This provides:

REST API endpoints
WebSocket real-time feed
Live dashboard UI

Step 4: View Live Dashboard

Terminal 3:

# Open in your browser:
http://localhost:8001

Or via curl:

# Get current signals
curl http://localhost:8001/api/signals

# Get market intelligence
curl http://localhost:8001/api/markets

# Get arbitrage opportunities
curl http://localhost:8001/api/arbitrage

Step 5: Query Data from Your Scripts

Terminal 4:

# Example: Get all active signals
curl http://localhost:8001/api/signals

# Example: Get market data for specific symbol
curl http://localhost:8001/api/markets/BTC-USDT

# Example: Stream real-time updates
curl -N http://localhost:8001/stream/signals

Project Structure

mia/
├── main.py                          # Entry point - Market data collection
├── metrics.py                       # System metrics and monitoring
├── requirements.txt                 # Python dependencies
│
├── analysis/                        # Signal Analysis Frameworks
│   ├── indicators.py               # Technical indicators (RSI, MACD, etc)
│   ├── regime_classifier.py        # Market regime detection
│   ├── structural_confidence.py    # Confidence scoring
│   ├── asymmetric_risk.py          # Risk framework
│   ├── causality_validator.py      # Lead-lag analysis
│   ├── liquidity_stack.py          # Liquidity analysis
│   └── edge_calculation.py         # Information ratio & edge metrics
│
├── ingestion/                       # Exchange Data Ingestion
│   ├── adapter_manager.py          # Adapter orchestration
│   ├── adapter_cluster.py          # Optimized adapter cluster
│   ├── kucoin_adapter.py           # KuCoin integration
│   ├── coinbase_adapter.py         # Coinbase integration
│   ├── kraken_adapter.py           # Kraken integration
│   ├── binance_adapter.py          # Binance integration
│   ├── bybit_adapter.py            # ByBit integration
│   ├── websocket_manager.py        # WebSocket connection pooling
│   ├── batch_fetcher.py            # REST API fallback
│   ├── circuit_breaker.py          # Load balancing & circuit breaking
│   ├── gap_detector.py             # Data integrity validation
│   ├── timeframe_synthesizer.py    # Multi-timeframe synthesis
│   └── multi_exchange_intelligence.py  # Cross-exchange analysis
│
├── events/                          # Event-Driven Architecture
│   ├── event_engine.py             # Core event processing
│   ├── event_fabric.py             # Redis-based event distribution
│   ├── signal_consumer.py          # Signal consumption & routing
│   ├── signal_quality.py           # Signal quality assessment
│   ├── execution_gateway.py        # Position management & execution
│   ├── notifier.py                 # Notification system
│   └── adapters/                   # Output adapters
│       ├── redis_consumer_adapter.py
│       ├── websocket_adapter.py
│       ├── database_adapter.py
│       └── webhook_adapter.py
│
├── market/                          # Market Analysis
│   ├── market_graph.py             # Symbol relationship graph
│   ├── market_selector.py          # Market filtering & selection
│   ├── market_integration.py       # Market integration layer
│   ├── position_manager.py         # Position tracking
│   └── capital_flow_funnel.py      # Capital flow analysis
│
├── api/                             # API Layer
│   ├── intelligence_api.py         # Main REST API
│   ├── data_access_api.py          # Data retrieval API
│   ├── query_api.py                # Query builder
│   └── web_interface.py            # Dashboard UI
│
├── cache/                           # Caching Layer
│   └── redis_layer.py              # Redis client & operations
│
├── config/                          # Configuration
│   └── signal_detection_config.py  # Signal parameters
│
├── normalization/                   # Symbol Normalization
│   ├── normalizer.py               # Symbol normalization logic
│   ├── symbol_registry.py          # Symbol registry
│   ├── symbol_resolver.py          # Symbol resolution
│   └── symbol_utils.py             # Utility functions
│
├── mia_mcp/                         # MCP Server Integration
│   └── mcp_server.py               # Model Context Protocol server
│
├── neondb/                          # Database Schema
│   └── [migration files]
│
├── redis/                           # Redis Configuration
│   └── [scripts and config]
│
├── tests/                           # Testing & Diagnostics
│   ├── diagnose_db_connection.py
│   ├── diagnose_data_flow.py
│   ├── check_tables.py
│   └── [other test utilities]
│
├── docs/                            # Documentation
│   ├── README_STARTUP.md           # Startup guide
│   ├── MIA_SYSTEM_ARCHITECTURE.md  # Architecture details
│   ├── API_GUIDE.md                # API documentation
│   ├── CONFIG_GUIDE.md             # Configuration guide
│   ├── ENVIRONMENT_SETUP.md        # Environment setup
│   ├── EVENT_FABRIC_GUIDE.md       # Event system guide
│   └── [other documentation]
│
└── .env                             # Environment configuration (git-ignored)

API Reference

REST Endpoints

Signals API

# Get all active signals
GET /api/signals

# Get signals for specific symbol
GET /api/signals?symbol=BTC-USDT

# Get signals with minimum confidence
GET /api/signals?confidence=0.75

# Get signal history
GET /api/signals/history?limit=100&offset=0

Markets API

# Get all available markets
GET /api/markets

# Get market data for specific symbol
GET /api/markets/{symbol}

# Get market metrics
GET /api/markets/{symbol}/metrics

# Get market correlations
GET /api/markets/correlations?symbols=BTC-USDT,ETH-USDT

Arbitrage API

# Get arbitrage opportunities
GET /api/arbitrage

# Get arbitrage for specific symbol
GET /api/arbitrage?symbol=BTC-USDT

# Get arbitrage by exchange pair
GET /api/arbitrage?exchange1=kucoin&exchange2=coinbase

Health & Status

# System health check
GET /health

# Get system status
GET /api/status

# Get metrics
GET /api/metrics

WebSocket Streams

# Connect to real-time signal stream
ws://localhost:8001/ws/signals

# Connect to market data stream
ws://localhost:8001/ws/markets

# Connect to all events stream
ws://localhost:8001/ws/events

For complete API documentation, see API_GUIDE.md.

Development

Running Tests

# Run all tests
pytest

# Run specific test file
pytest tests/test_analysis.py

# Run with coverage
pytest --cov=.

Database Diagnostics

# Check database connection
python -m tests.diagnose_db_connection

# Verify table schema
python -m tests.check_table_schema

# Diagnose data flow
python -m tests.diagnose_data_flow

Code Style

# Format code
black .

# Lint code
flake8 .

# Type checking
mypy .

Adding New Exchange

Create adapter in ingestion/new_exchange_adapter.py
Inherit from base adapter class
Implement required methods (connect, subscribe, parse)
Register in ingestion/adapter_manager.py
Add to main.py initialization

See ADAPTER_OPTIMIZATION_GUIDE.md for details.

Monitoring & Metrics

System Metrics

The system tracks:

Async task counts - Active coroutines
Batch flush times - Data pipeline latency
Event throughput - Events per second
Processing latencies - End-to-end latency
Queue depths - Signal backlog monitoring
Exchange connectivity - Feed health status

Accessing Metrics

# Via API
curl http://localhost:8001/api/metrics

# Via logs
grep METRIC main.log | tail -20

# Via dashboard
http://localhost:8001#metrics

Performance Tuning

Adjust batch sizes in config/signal_detection_config.py
Configure circuit breaker thresholds in ingestion/circuit_breaker.py
Tune Redis connection pool in cache/redis_layer.py

Troubleshooting

Common Issues

1. Database Connection Failed

# Check connectivity
python -m tests.diagnose_db_connection

# Verify credentials in .env
cat .env | grep DATABASE_URL

# Test connection manually
psql $DATABASE_URL -c "SELECT 1"

2. Redis Connection Refused

# Check Redis is running
redis-cli ping

# Start Redis if needed
redis-server

# Or with Docker
docker run -d -p 6379:6379 redis:latest

3. Exchange API Rate Limiting

The system includes adaptive retry logic with exponential backoff. Check:

API key rate limits with your exchange
RATE_LIMIT_BACKOFF_MS in config
Circuit breaker thresholds

# From ingestion/circuit_breaker.py
BACKOFF_BASE = 1000  # 1 second initial backoff
MAX_RETRIES = 5

4. WebSocket Connection Drops

Enable reconnection logic in your adapter:

# Auto-reconnect with exponential backoff
await adapter.connect_with_backoff(max_retries=5)

5. Signal Detection Not Working

Verify:

# 1. Data is flowing
curl http://localhost:8001/api/markets

# 2. Market graph is built
python -c "from market.market_graph import MarketGraph; print(MarketGraph().symbols_count())"

# 3. Event engine is running
tail -f main.log | grep "EventEngine"

Enable Debug Logging

# In .env
LOG_LEVEL=DEBUG

# Or in Python
import logging
logging.basicConfig(level=logging.DEBUG)

Get Detailed Diagnostic Report

python tests/diagnose_data_flow.py

Documentation

README_STARTUP.md - Quick startup guide
MIA_SYSTEM_ARCHITECTURE.md - Complete architecture
API_GUIDE.md - REST API reference
CONFIG_GUIDE.md - Configuration options
ENVIRONMENT_SETUP.md - Environment setup
EVENT_FABRIC_GUIDE.md - Event system
ADAPTER_OPTIMIZATION_GUIDE.md - Exchange adapters
ROBUSTNESS_IMPROVEMENTS.md - Reliability features
NETWORK_TROUBLESHOOTING.md - Network issues

Performance Characteristics

Metric	Target	Status
Signal Latency (candle → signal)	< 500ms	✅
Event Propagation	< 100ms	✅
Exchange Connection Setup	< 2s	✅
Database Query (95th percentile)	< 100ms	✅
Redis Operations	< 10ms	✅
Max Concurrent Signals	10+	✅
Data Integrity Score	> 95%	✅

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Make your changes
Add tests for new functionality
Commit with clear messages (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For issues, questions, or suggestions:

📧 Email: gichurejames001@gmail.com
💬 Issues: GitHub Issues
📚 Documentation: docs/
🐛 Bug Reports: Include system info, .env (sanitized), and logs

Roadmap

Multi-timeframe regime detection enhancements
Additional exchange integrations (Deribit, OKX)
Advanced position sizing algorithms
Machine learning signal filtering
Risk management enhancements
Kubernetes deployment templates
Advanced metrics dashboard

Acknowledgments

Built with FastAPI
Exchange data via CCXT
Event streaming with Redis
Data storage with PostgreSQL

MIA - Bringing Professional-Grade Market Intelligence to Crypto Trading

Made with ❤️ for traders and developers

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
analysis		analysis
api		api
cache		cache
config		config
docs		docs
events		events
ingestion		ingestion
market		market
mia_mcp		mia_mcp
normalization		normalization
tests		tests
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
main.py		main.py
metrics.py		metrics.py
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

MIA - Market Intelligence Agent

Table of Contents

Overview

Target Use Cases

Key Features

🎯 Signal Detection & Validation

🔄 Multi-Exchange Data Integration

📊 Advanced Analysis

🔌 Event-Driven Architecture

📈 Data Persistence

System Architecture

Prerequisites

System Requirements

API Keys Required

Installation

1. Clone the Repository

2. Create Virtual Environment

3. Install Dependencies

4. Set Up Databases

PostgreSQL Setup

Redis Setup

Configuration

Environment Variables

Configuration Files

Quick Start

Step 1: Verify System Health

Step 2: Start Market Data Collection

Step 3: Start Data Access API

Step 4: View Live Dashboard

Step 5: Query Data from Your Scripts

Project Structure

API Reference

REST Endpoints

Signals API

Markets API

Arbitrage API

Health & Status

WebSocket Streams

Development

Running Tests

Database Diagnostics

Code Style

Adding New Exchange

Monitoring & Metrics

System Metrics

Accessing Metrics

Performance Tuning

Troubleshooting

Common Issues

1. Database Connection Failed

2. Redis Connection Refused

3. Exchange API Rate Limiting

4. WebSocket Connection Drops

5. Signal Detection Not Working

Enable Debug Logging

Get Detailed Diagnostic Report

Documentation

Performance Characteristics

Contributing

License

Support

Roadmap

Acknowledgments

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages