ScreenPlay is a screen-perception-driven AI co-pilot system that enables intelligent agents to participate in FPS games purely through visual and audio capture — no game source code, no SDK hooks, no memory reading — making it fully game-agnostic and compliant with anti-cheat boundaries.
- Non-invasive Perception: DXGI desktop duplication + MSS fallback for high-FPS screen capture with zero kernel-level hooks
- Real-time Object Detection: YOLOv8 + TensorRT / ONNX Runtime inference pipeline detecting enemies, teammates, items, and HUD elements
- HUD Parsing Engine: JSON-profile-driven parser for health, ammo, minimap, compass, and teammate status — fully configurable per game title
- Spatial Audio Perception: WASAPI binaural capture with ILD/ITD-based azimuth estimation and event classification (footsteps, shots, explosions)
- Three-layer Decision Architecture:
- Reactive Layer — sub-10 ms rule-based reflexes (emergency fire, crouch, reload, grenade dodge)
- Tactical Layer — PPO-trained reinforcement learning policy network (53-dim state → 12 action categories)
- Strategic Layer — LLM-driven high-level planning via llama.cpp with structured JSON output
- Three-layer Memory System:
- Short-term — sliding-window episode buffer (game state frames)
- Spatial — 2-D occupancy grid with enemy / teammate heat-maps
- Long-term — ChromaDB vector store for strategic knowledge retrieval
- Autonomous Mode: Full AI loop from perception → decision → action
- Co-Pilot Mode: Human plays, AI provides voice alerts, threat assessment overlay, and tactical hints
- Transparent Overlay: pygame + win32gui borderless transparent HUD overlay
┌─────────────────────────────────────────────────────────────────┐
│ ScreenPlay System │
├─────────────────────────────────────────────────────────────────┤
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Perception Engine │ │
│ │ Screen Capture → Object Detector → HUD Parser │ │
│ │ Audio Perception Engine → Game State Builder │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ GameState │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Memory System │ │
│ │ Short-term Buffer | Spatial Grid | Long-term VectorDB │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ Context │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Decision Engine │ │
│ │ Reactive Layer → Tactical Layer → Strategic Layer │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ ActionCommand │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Action Engine │ │
│ │ SendInput Wrapper → Human-like Mouse Controller │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────▼───────────────────────────────┐ │
│ │ Co-Pilot / Overlay (optional) │ │
│ │ TTS Voice Alerts | Transparent Overlay HUD │ │
│ └──────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
| Component | Minimum | Recommended |
|---|---|---|
| CPU | Intel i5-10th / Ryzen 5 3600 | Intel i9 / Ryzen 9 |
| GPU | NVIDIA GTX 1660 (6 GB VRAM) | NVIDIA RTX 3080+ |
| RAM | 16 GB | 32 GB |
| Storage | 20 GB SSD | 50 GB NVMe SSD |
- OS: Windows 10/11 (64-bit) — DXGI capture and DirectInput require Windows
- Python: 3.10 or 3.11
- CUDA: 11.8 or 12.1 (for GPU inference)
- TensorRT: 8.6+ (optional, for maximum inference speed)
- Visual C++ Redistributable 2022
git clone https://github.com/korose523/AI-collaborative-game.git
cd AI-collaborative-game
pip install -r requirements.txt# Copy default config and edit to match your setup
copy config\default_config.yaml config\local_config.yamlEdit config/local_config.yaml — set monitor_index, game_window_title, etc.
# Train YOLOv8 detector on your labelled screenshots
python training/train_detector.py --data training/detection_classes.yaml --epochs 100
# Or place a pre-trained .onnx model into models/python main.py --mode copilot --config config/local_config.yamlpython main.py --mode autonomous --config config/local_config.yamlAI-collaborative-game/
├── config/ # YAML configs + per-game HUD profiles
├── screenplay/ # Core Python package
│ ├── perception/ # Screen / audio perception
│ ├── memory/ # Three-layer memory system
│ ├── decision/ # Three-layer decision engine
│ ├── action/ # Input injection
│ ├── copilot/ # Co-pilot mode + overlay
│ └── utils/ # Shared utilities
├── training/ # Training scripts
├── models/ # Model weights (gitignored except .gitkeep)
├── data/ # Datasets (gitignored except .gitkeep)
├── docs/ # Architecture & quickstart docs
└── main.py # System entry point
The AI takes full control of keyboard and mouse. The system runs the complete perception → memory → decision → action loop at configurable frame rates. Suitable for testing in offline/LAN matches or custom servers.
python main.py --mode autonomous
The human player retains full control. ScreenPlay overlays real-time threat indicators, enemy direction arrows, health warnings, and reload reminders. The TTS engine announces tactical advisories ("Enemy 9 o'clock!", "Low ammo!"). No mouse/keyboard injection in this mode.
python main.py --mode copilot
| Script | Purpose |
|---|---|
training/train_detector.py |
Fine-tune YOLOv8 on labelled game screenshots |
training/data_recorder.py |
Record screen + input pairs for behavioural cloning |
training/reward_engine.py |
Compute shaped rewards for RL training from HUD delta |
Recommended training pipeline:
- Record ~5 000 frames with
data_recorder.py - Label using CVAT or LabelImg, export to YOLO format
- Train detector with
train_detector.py - Export to ONNX:
yolo export model=runs/detect/best.pt format=onnx - RL fine-tune the tactical policy using
reward_engine.pyrewards
- AI research on game-playing agents without simulator access
- FPS accessibility tools for players with motor disabilities (Co-Pilot mode)
- Automated game-play recording for content creation
- Synthetic benchmark generation for computer vision datasets
- Multi-agent cooperation research in partial-information environments
MIT License — see LICENSE for details.
Ethics Notice: Use only on private/offline servers or in modes permitted by the game's Terms of Service. The authors are not responsible for any account bans or ToS violations arising from use of this software.