🧠 MACVS — Memory-Augmented Cognitive Vision System with Signal-Guided Reasoning

MACVS is an advanced cognitive vision framework that bridges real-time computer vision perception with episodic memory and large language model (LLM) reasoning. Rather than treating each video frame in isolation, MACVS builds a continuous temporal memory of the environment, detects behavioral & scene signals, and reasons over both present observations and historical contexts.

---

🌟 Key Features

• Multi-Modal Perception Layer:
  • Object Detection: Fast, lightweight real-time detection via YOLOv8.
  • Scene Captioning: Dense contextual descriptions using BLIP (Salesforce/blip-image-captioning-base).
• Episodic Vector Memory:
  • Encodes scene representations into high-dimensional semantic embeddings via sentence-transformers/all-MiniLM-L6-v2.
  • Fast similarity retrieval powered by FAISS (Facebook AI Similarity Search) to recall past relevant states and contexts.
• Signal-Guided Temporal Tracking:
  • Continuous cosine-distance tracking across sliding temporal windows to detect sudden scene transitions, object arrival/departure, and activity shifts.
• Cognitive Reasoning Engine:
  • Graph-based reasoning chains connecting perceptions, signals, and retrieved memories.
  • Signal-guided reasoning utilizing local LLMs through Ollama (llama3.2:1b) or cloud LLMs (Google Gemini).
• Interactive Streamlit Dashboard:
  • Premium dark-themed UI with real-time video feed annotations, detection HUD, signal telemetry, memory match exploration, and interactive visual Q&A.
• Rigorous Evaluation Suite:
  • 20 pre-built benchmark scenarios testing normal scenes, sudden anomalies, and complex reasoning queries.

---

🏗️ Architecture Pipeline

[ Camera / Video Input ]
           │
           ▼
┌────────────────────────────────────────┐
│           Perception Layer             │
│  • YOLOv8 (Object Detection)           │
│  • BLIP (Scene Captioning)             │
└──────────────────┬─────────────────────┘
                   │
                   ▼
┌────────────────────────────────────────┐
│         Context Representation         │
│  • Entity Aggregation                  │
│  • Sentence-Transformers Embedding     │
└──────────┬──────────────────┬──────────┘
           │                  │
           ▼                  ▼
┌───────────────────┐  ┌─────────────────┐
│ Temporal & Signal │  │ Episodic Memory │
│     Detection     │  │  (FAISS Vector  │
│  • Scene shifts   │  │     Store)      │
│  • Event triggers │  │ • Top-K Recall  │
└──────────┬────────┘  └────────┬────────┘
           │                    │
           └──────────┬─────────┘
                      ▼
┌────────────────────────────────────────┐
│           Reasoning Engine             │
│  • Reasoning Graph Construction        │
│  • LLM Inference (Ollama / Gemini)     │
└──────────────────┬─────────────────────┘
                   │
                   ▼
┌────────────────────────────────────────┐
│        Streamlit Dashboard & UI        │
│  • Live Feed Overlays • Q&A Inspector  │
│  • Memory Matches     • Signal Telemetry│
└────────────────────────────────────────┘

---

📂 Project Structure

.
├── app.py                      # Streamlit web application & interface
├── config.py                   # Central system configuration & hyperparameters
├── pipeline.py                 # Core vision pipeline orchestrator
├── run.bat                     # Windows quick-launch script
├── requirements.txt            # Python dependencies
├── evaluation.py               # 20-scenario reasoning evaluation framework
├── evaluation_results.json     # Benchmark evaluation outputs
├── create_ppt.py               # Presentation deck generator
├── create_ppt_clean.py         # Refined presentation generator
├── generate_diagrams.py        # System architecture diagram generator
├── core/
│   ├── context_builder.py      # Context aggregation and formatting
│   ├── memory.py               # FAISS episodic vector memory store
│   ├── reasoner.py             # LLM reasoning integration (Ollama / Gemini)
│   ├── reasoning_graph.py      # Structured reasoning graph construction
│   ├── signals.py              # Behavioral & perceptual signal detectors
│   └── temporal.py             # Sliding-window temporal tracking
├── vision/
│   ├── captioner.py            # BLIP image captioning wrapper
│   ├── capture.py              # OpenCV video capture abstraction
│   └── detector.py             # YOLOv8 object detection wrapper
├── utils/
│   └── drawing.py              # Visual annotation and bounding box overlays
└── images/                     # Sample test images

---

🚀 Getting Started

1. Clone the Repository

git clone https://github.com/sam4010/Memory-Augmented-Cognitive-Vision-System.git
cd Memory-Augmented-Cognitive-Vision-System

2. Set Up Virtual Environment

python -m venv .venv

# On Windows:
.venv\Scripts\activate

# On Linux/macOS:
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

4. Setup LLM Backend (Optional but Recommended)

MACVS supports local inference via Ollama:

1. Download and install Ollama.
2. Pull the default reasoning model:

   ollama pull llama3.2:1b

(Alternatively, configure Google Gemini API key in evaluation.py or environment variables for cloud inference).

---

💻 Running the Application

Launch the Streamlit dashboard using:

streamlit run app.py

Or simply double-click / run:

run.bat

Open your browser at http://localhost:8501 to access the live dashboard.

---

🧪 Running Evaluations

Evaluate the system's reasoning accuracy across 20 built-in test scenarios:

# Run with Gemini API
python evaluation.py --key YOUR_GEMINI_API_KEY

# Run in offline fallback mode
python evaluation.py --offline

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.