Skip to content

About

MSK is a local-first Project Intelligence and Privacy layer for software engineering. It analyzes your project locally, builds a living knowledge model of the software system, and gives AI only the minimum context required to answer a question—without exposing the entire repository.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

36 Commits

Folders and files

Repository files navigation

MSK

The project stays private. AI gets only what it needs.

MSK is a local-first Project Intelligence, Privacy, and Security layer for software engineering.

Warning

Alpha software (v0.1.0-alpha.1). APIs and CLI may change between releases.


What is MSK?

A modern software project often contains sensitive business logic, database models, internal network topologies, and proprietary algorithms. Traditional AI assistants demand sending large portions (or entirety) of source repositories to external cloud providers.

MSK establishes a controlled, local-first intelligence boundary. It analyzes your software project locally on your machine, constructs an offline Project Knowledge Model (stored in SQLite), and determines the minimum relevant context required when interacting with AI systems.

                    GIT (Source of truth)
                             │
                             ▼
              ┌─────────────────────────────┐
              │             MSK             │
              │                             │
              │  Local Project Intelligence │
              │  Knowledge Graph (SQLite)   │
              │  Security & Privacy Gateway │
              │  Minimum Context Engine     │
              └──────────────┬──────────────┘
                             │
                             ▼ (Policy-approved minimum context)
                            AI

Key Principles

  1. Simple Outside. Complex Inside: Familiar, Git-like CLI (msk init, msk status, msk update, msk doctor).
  2. Git is Source of Truth: MSK adds intelligence on top of Git; it never replaces or tampers with your Git history.
  3. Offline-First & Local Analysis: Core project scanning, Tree-sitter parsing, and knowledge model creation run 100% locally with zero internet dependency.
  4. Regeneratable Knowledge: The .msk/ directory contains metadata, fingerprints, symbols, and relationship graphs—never redundant source code duplicates. It can be recreated from scratch at any time.
  5. Security & Boundary Enforcement: Secrets, credentials, and sensitive data are classified and blocked from leaving your boundary.

Installation

From source (recommended for alpha)

# Clone the repository
git clone https://github.com/Omkar-git-hub/msk-project-intelligence.git
cd msk-project-intelligence

# Create a virtual environment
python -m venv .venv

# Activate it
# Linux/macOS:
source .venv/bin/activate
# Windows:
.venv\Scripts\activate

# Install MSK
pip install -e .

# Or with development dependencies
pip install -e ".[dev]"

From wheel

pip install dist/msk-0.1.0a1-py3-none-any.whl

Quick Start

# 1. Check your environment is ready
msk doctor

# 2. Initialize project analysis (creates .msk/ directory)
msk init

# 3. View project intelligence summary
msk status

# 4. Incrementally update after code changes
msk update

# 5. Analyze impact before making code modifications
msk impact --file src/payment/service.py
msk impact --symbol process_payment --file src/payment/service.py

# 6. Select ranked, privacy-safe context for a question, file, or error
msk context --question "How does payment processing work?"
msk context --file src/payment/service.py --budget 1500
msk context --error "ValueError: transaction failed in process_payment"

# 7. Force a full re-analysis
msk init --force

CLI Reference

Command Description
msk init Analyze the project and build the Project Knowledge Model
msk status Display project intelligence: files, symbols, dependencies, security
msk update Incrementally update the knowledge model based on changed files
msk impact Analyze downstream impact of modifying a file, symbol, or dependency
msk context Select and rank relevant, privacy-safe project context for queries
msk provider Configure, list, and test local/cloud AI providers and models
msk ask Query project intelligence and AI with policy-approved context
msk explain Explain files, code symbols, or diagnose runtime error logs
msk runtime Local-first runtime intelligence, log parsing, error correlation
msk doctor Diagnose environment: Python, Tree-sitter parsers, Git, database

Global Options

Option Description
--verbose Enable verbose log output
--debug Show full debug tracebacks
--help Show help and exit

Project Knowledge Model

MSK builds a structured model of your project stored in SQLite (.msk/msk.db), with JSON exports for debugging and interoperability (.msk/exports/).

What MSK captures

Entity Description
Files All source files with language, type (source/test/config), hash
Symbols Classes, functions, methods extracted via Tree-sitter AST parsing
Dependencies External package dependencies (requirements.txt, package.json, etc.)
Relationships File→Symbol, Symbol→Symbol calls, imports, test targets
Tests Test classes and functions with their target symbols
API Endpoints Flask/FastAPI route decorators with HTTP methods
Infrastructure Dockerfile, docker-compose, GitHub Actions, Kubernetes, Terraform
Security Secret/credential detection (privacy-preserving: no raw values)

Supported Languages

Language Parsing Symbols Imports Call Graph
Python ✅ ✅ ✅ ✅
Java ✅ ✅ ✅ ✅
TypeScript ✅ ✅ ✅ ✅
JavaScript ✅ ✅ ✅ ✅

Relationship Types

Type Example
FILE_CONTAINS_SYMBOL controller.py → handle_payment()
SYMBOL_IMPORTS_SYMBOL service.py imports PaymentRepository
SYMBOL_CALLS_SYMBOL handle_payment() calls process_payment()
FILE_DEPENDS_ON_FILE controller.py depends on service.py
TEST_TARGETS_SYMBOL test_process_payment → process_payment

.msk/ Directory Layout

.msk/
├── msk.db          # SQLite knowledge database (git-ignored)
├── project.json    # Deterministic project config
├── policy.json     # Privacy/security policy
├── exports/        # JSON exports for debugging/interop
│   ├── structure.json
│   ├── symbols.json
│   ├── graph.json
│   ├── tests.json
│   ├── apis.json
│   ├── infrastructure.json
│   └── security.json
└── cache/          # Temporary cache (git-ignored)

The .msk/ directory is fully regeneratable from source. You can delete it and run msk init to rebuild everything.

Incremental Change Engine

When your codebase evolves, msk update incrementally updates the knowledge model:

  1. Cryptographic Change Detection: Compares current files against SQLite records using streaming SHA-256 content hashes (not relying on volatile modification timestamps alone).
  2. Selective AST Analysis: Analyzes only newly added, modified, or renamed files. Unchanged files are skipped, preserving existing knowledge records without re-parsing.
  3. Rename Detection: Unambiguous file moves/renames are detected by matching content hashes across deleted and added paths.
  4. Stale Record Cleanup & Cascades: Deleted or modified files cascade-delete old symbols, tests, APIs, and clean up orphaned relationship edges.
  5. Knowledge Graph Reconciliation: Synchronizes FILE_CONTAINS_SYMBOL, SYMBOL_IMPORTS_SYMBOL, SYMBOL_CALLS_SYMBOL, FILE_DEPENDS_ON_FILE, and TEST_TARGETS_SYMBOL across all entities.
  6. Interruption-Safe Atomic Writes: Database updates execute within a single atomic SQLite transaction with WAL mode. JSON exports are written via temporary .tmp files and swapped atomically using os.replace.

Impact Analysis Engine

MSK answers: "If this file, function, class, module, or dependency changes, what other parts of the project might be affected, and why?"

The engine traverses verified relationships in the local knowledge graph (call graphs, imports, test-targets, containment, manifest dependencies) without external network calls or cloud dependencies:

  1. File Impact (msk impact --file <path>):
    • Identifies contained symbols, dependent files importing this file, direct callers of functions in this file, tests targeting this file's symbols, and associated API endpoints or infrastructure.
  2. Symbol Impact (msk impact --symbol <name> [--file <path>]):
    • Identifies direct callers, member functions / parent classes, dependent files, and tests targeting the specific symbol.
    • Disambiguates duplicate symbol names across different files using --file.
  3. Dependency Impact (msk impact --dependency <package>):
    • Identifies manifest files declaring the dependency and source files directly importing the package.
  4. Direct vs. Transitive Analysis:
    • Direct impact: Items connected through verified depth-1 relationships.
    • Transitive impact: Multi-hop callers and dependent files reachable up to --depth <N> (default: 3). Use --direct for depth 1 only.
    • Cycle prevention and deduplication ensure deterministic results even in recursive or circular codebases.
  5. Machine-Readable JSON Output (--json):
    • Emits structured JSON suitable for CI pipelines, commit hooks, or developer tooling.

CLI Usage Examples

# Analyze impact of modifying a service file (transitive up to depth 3)
msk impact --file src/payment/service.py

# Analyze direct impact only
msk impact --file src/payment/service.py --direct

# Analyze impact of a specific function
msk impact --symbol process_payment --file src/payment/service.py

# Bounded traversal depth
msk impact --file src/payment/service.py --depth 2

# Output machine-readable JSON
msk impact --file src/payment/service.py --json

Interpreting Results & Incomplete Knowledge

  • Missing Knowledge: If a queried file or symbol is not indexed, MSK informs you and suggests running msk update (if changes are recent) or msk init (if the project has not been initialized).
  • Empty Results: An item with 0 affected items indicates that it is indexed, but no other project files, callers, or tests depend on it.
  • Evidence-Based Explanations: Every result item includes the explicit relationship type, traversal depth, and causation path.

Smart Context Engine (Phase 5)

MSK answers: "What is the minimum, most relevant, privacy-safe project context an AI or developer needs to answer this question or debug this error?"

Traditional AI assistants dump entire codebases or blind file selections into cloud prompts. The Smart Context Engine operates 100% offline and locally, querying your verified project knowledge model to select, rank, and budget the most relevant context while scrubbing secrets and excluding sensitive files.

Note

Context Preparation Only: Phase 5 prepares and budgets context locally. It does not connect to an external LLM, make remote API calls, or modify source code. Pluggable local and cloud LLM gateways are scheduled for Phase 6.

Key Capabilities

  1. Multi-Modal Query Input (msk context):
    • Natural-language developer questions (--question "How does payment processing work?")
    • Target focus files (--file src/payment/service.py)
    • Target code symbols (--symbol process_payment)
    • Error logs or stack traces (--error "ValueError: transaction failed" or --error-file trace.log)
    • Configurable token budget (--budget 2000, default: 2,000 tokens)
  2. Explainable Relevance Ranking:
    • Exact Target Match (100.0 pts): Explicitly queried files or symbols.
    • Stack Trace Frames (95.0 pts): Files, symbols, and line numbers parsed from Python, Java, or TS/JS error traces.
    • Containment (85.0 pts): Symbols declared in target files, or files declaring target symbols.
    • Direct Callers & Tests (75.0 pts): Depth-1 callers, callees, and test targets.
    • Direct File Dependencies (70.0 pts): Files importing or imported by the target file.
    • Query Keyword Matches (60.0 pts): Exact and partial identifier matches from natural language descriptions.
    • Transitive Impact (25.0 - 40.0 pts): Multi-hop callers and dependent files reachable via graph paths.
    • APIs, Infra & Security (25.0 - 45.0 pts): Associated route handlers, Docker/k8s configs, and security findings.
    • Deterministic Tie-Breaking: Items with equal score break ties stably by (-score, type_priority, file_path, name, id).
  3. Token Budgeting & Selection:
    • Uses a documented, deterministic token estimator approximating BPE tokenizers without heavy external dependencies.
    • Deduplicates redundant representations across files and symbols.
    • Transparently records all candidates that were omitted due to budget constraints, including their scores and required tokens.
  4. Local Privacy & Data Minimization:
    • Secret Scrubbing: Automatically scrubs passwords, bearer tokens, AWS keys, GitHub tokens, Slack tokens, and private keys from queries and stack traces.
    • Sensitive File Exclusion: Excludes credentials, certificates, and keys (.env, *.pem, *.key, id_rsa) from context.
    • Code Minimization: Surfacing symbol signatures, line ranges, docstrings, and relationships rather than raw full-file dumps.
    • Transparency & Limits: Reports redactions and notes that heuristic pattern matching cannot guarantee 100% detection of custom or obfuscated secrets.

CLI Usage Examples

# Query context with a developer question
msk context --question "How does process_payment work?"

# Focus context on a specific file within a 1,500 token budget
msk context --file src/payment/service.py --budget 1500

# Focus context on a specific symbol and file
msk context --symbol process_payment --file src/payment/service.py

# Diagnose a runtime error or stack trace
msk context --error "ValueError: Invalid currency in process_payment"

# Provide a stack trace from a log file
msk context --error-file runtime_error.log

# Export machine-readable context in JSON format for future LLM integration
msk context --file src/payment/service.py --json

JSON Output Format

When invoked with --json, msk context emits a structured, machine-readable payload suitable for developer tooling and future LLM gateways:

{
  "project_name": "payment_service",
  "analyzed_at": "2026-10-20T22:20:00Z",
  "request_summary": {
    "question": "How does payment processing work?",
    "file": "src/payment/service.py",
    "symbol": null,
    "has_error": false,
    "budget_tokens": 2000
  },
  "budget_metrics": {
    "token_budget": 2000,
    "estimated_tokens_used": 640,
    "utilization_pct": 32.0,
    "total_candidates": 14,
    "selected_count": 8,
    "omitted_count": 6
  },
  "privacy_summary": {
    "local_only": true,
    "redactions_count": 0,
    "excluded_sensitive_files": [],
    "notes": [...]
  },
  "selected_items": [
    {
      "id": "file:f_123",
      "name": "src/payment/service.py",
      "item_type": "file",
      "file_path": "src/payment/service.py",
      "relevance_score": 100.0,
      "reasons": ["Explicitly requested file 'src/payment/service.py'"],
      "snippet": "...",
      "estimated_tokens": 85,
      "metadata": {...}
    }
  ],
  "relationships": [...],
  "related_tests": ["test_process_payment (tests/test_payment_service.py)"],
  "omitted_items": [...],
  "warnings": [],
  "llm_context": "# Project Context: payment_service\n..."
}

Pluggable AI Gateway & Provider Management (Phase 8)

MSK enables developers to connect local or cloud AI models without sacrificing privacy or writing custom code. All AI provider communications are routed through the fail-closed PolicyGatedGateway to ensure that the project stays private and AI gets only what it needs.

Provider CLI Reference (msk provider)

  • List Providers: Inspect all configured providers, default election, and credential status.
    msk provider list
  • Add Provider: Add a local or external provider dynamically without code changes.
    # Local Ollama daemon
    msk provider add ollama --name "Local Ollama" --type ollama --endpoint http://localhost:11434 --model llama3 --auth-type none
    
    # External OpenAI provider using environment variable reference
    msk provider add openai --name "OpenAI Cloud" --type openai --model gpt-4o --auth-type bearer --credential-ref env:OPENAI_API_KEY
  • Set Default Provider:
    msk provider set-default ollama
  • Connection Testing Framework: Validate configuration, connectivity, authentication, model availability, and latency without sending project code:
    msk provider test ollama
    msk provider test openai --model gpt-4o

Developer AI Commands (Phase 8)

1. Ask Questions (msk ask)

Ask natural-language questions about your codebase. MSK selects relevant, privacy-scrubbed context, verifies external transmission permissions against your policy, and presents answers with citations:

# Query with default provider
msk ask "How does the payment processing flow handle negative amounts?"

# Target external provider explicitly
msk ask "Explain database migration strategy" --target external

# Specify budget and emit JSON
msk ask "List all authentication endpoints" --budget 1500 --json

2. Explain Code & Incidents (msk explain)

Diagnose runtime exceptions, explain architectural components, or inspect symbols:

# Explain a specific file
msk explain --file src/payment/service.py

# Explain a symbol and its callers
msk explain --symbol process_payment --file src/payment/service.py

# Diagnose a runtime error trace
msk explain --error "ValueError: Invalid currency in process_payment"

# Diagnose an incident from a log file
msk explain --error-file logs/incident_trace.log --json


Runtime Intelligence & Correlation (Phase 9)

MSK Phase 9 introduces local-first runtime intelligence, correlating empirical observations (application logs, exceptions, stack traces, API requests, and performance metrics) directly with static project intelligence:

1. Ingest Application Logs (msk runtime logs)

Parse multi-format logs (structured JSON, standard timestamped logs, syslog) with automatic secret scrubbing:

# Ingest application logs from a file
msk runtime logs --file logs/app.log

# Ingest and display in JSON format
msk runtime logs --file logs/app.log --limit 50 --json

# Test parse raw log text without persisting (dry-run)
msk runtime logs --raw "[ERROR] Connection timed out to cache" --no-save

2. Analyze Runtime Errors & Blast Radius (msk runtime analyze)

Analyze an active stack trace or incident file, identify affected code symbols, map downstream callers and dependent modules, and produce an actionable investigation checklist:

# Analyze a stack trace or error log file
msk runtime analyze --file crash.log

# Analyze raw traceback text
msk runtime analyze --raw "Traceback (most recent call last):\n  File \"src/order.py\", line 42, in process_order\n    raise ValueError(\"invalid amount\")\nValueError: invalid amount"

# Emit complete empirical facts and static inferences in JSON
msk runtime analyze --file crash.log --json

3. Error Tracking & Deduplication (msk runtime errors)

Inspect recurring errors grouped by normalized fingerprints, occurrence count, first/last seen timestamps, and resolution status:

# List all recorded runtime errors
msk runtime errors

# Show only unresolved errors
msk runtime errors --unresolved

# Mark an error as resolved
msk runtime errors --resolve rterr_abc123

4. Correlate Runtime Incidents with Knowledge Graph (msk runtime correlate)

Map a runtime failure directly onto affected source files, enclosing functions, callers/callees, related tests, and user-facing API endpoints:

# Correlate the latest recorded error
msk runtime correlate --latest

# Correlate a specific error ID or fingerprint
msk runtime correlate rterr_abc123

5. Timeline History & Retention (msk runtime history, msk runtime prune)

Inspect chronological events and manage storage footprint locally:

# View recent event timeline
msk runtime history --limit 30 --level ERROR

# Prune older events beyond 30 days retention
msk runtime prune --days 30 --max-events 1000

# Clear all runtime observations for the project
msk runtime prune --clear

Privacy & Security

MSK is designed with privacy as a core architectural concern:

  • Local-only analysis: All scanning, parsing, impact analysis, and context selection happen 100% locally. No data leaves your machine.
  • No raw secrets stored or sent: Secret scanner and context privacy filters detect credentials and redact them with [REDACTED_SECRET].
  • Sensitive file exclusion: Files matching patterns like .env, *.key, *.pem, id_rsa are automatically excluded from context.
  • No source code copies: The .msk/ directory never stores source code copies—only structural metadata.
  • Git-ignored databases: SQLite databases and caches are git-ignored by default.

Architecture

msk/
├── common/        # Logging, hashing, path handling, errors
├── config/        # Project and policy configuration loaders
├── project/       # Root detection, streaming file scanner, ignore handling
├── analyzer/      # Tree-sitter AST parsing, language adapters, symbols, imports
│   ├── security.py       # Privacy-preserving secret scanner
│   └── infrastructure.py # Infrastructure config detector
├── storage/       # SQLite schema (V2) and transactional repository
├── knowledge/     # Knowledge graph models, builder, JSON serializers
├── graph/         # In-memory and SQL graph relationship queries
├── changes/       # SHA-256 content and structural fingerprinting, Git integration
├── impact/        # Impact analysis engine, graph traversal, deterministic reporting
├── context/       # Smart Context Engine: ranking, token budgeting, privacy scrubbing
├── policy/        # Privacy policy models, auditor, and PolicyGatedGateway
├── gateway/       # Pluggable AI adapters (Mock, Ollama, Local OpenAI, OpenAI, Anthropic, Gemini)
├── runtime/       # Multi-format log parser, error grouping, static-to-runtime correlation & impact
└── cli/           # Typer CLI commands with Rich terminal UI

Development

Prerequisites

  • Python ≥ 3.11
  • Git

Setup

git clone https://github.com/Omkar-git-hub/msk-project-intelligence.git
cd msk-project-intelligence
python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -e ".[dev]"

Running Tests

pytest

Linting

ruff check src/ tests/

Building

python -m build

Produces dist/msk-0.1.0a1-py3-none-any.whl and dist/msk-0.1.0a1.tar.gz.

Security Checks

bandit -r src/ -c pyproject.toml
pip-audit

Alpha Limitations

This is v0.1.0-alpha.1. The following limitations apply:

  • Pluggable AI Gateways: MSK Phase 8 supports local and external AI models with strict privacy boundaries and zero third-party vendor SDK bloat.
  • Limited language support: Python, Java, TypeScript, and JavaScript parsers currently enabled.
  • No remote/cloud features: Everything runs 100% locally on your machine.
  • Schema evolution: Non-destructive migrations keep SQLite schema backwards compatible across updates.
  • Windows note: Tested on Windows 11 with Python 3.14. Tree-sitter AST nodes use safe byte-offset extraction.

Roadmap

  • Phase 1: Local Project Intelligence & CLI
  • Phase 2: Hardened Knowledge Model (tests, APIs, infrastructure, security)
  • Phase 3: Incremental Change Engine & Knowledge Model Consistency
  • Phase 4: Impact Analysis Engine
  • Phase 5: Smart Context Engine & Local Privacy Boundary
  • Phase 6: Privacy Policy Enforcement
  • Phase 7: Policy Integration & Autonomous Enforcement
  • Phase 8: Pluggable AI Gateway, Provider Management, Model Configuration & Developer AI Commands
  • Phase 9: Local-First Runtime Intelligence & Static-Runtime Correlation (msk runtime)
  • Phase 10: Error & Incident Autonomous Remediation (msk fix)

License

Apache-2.0

About

MSK is a local-first Project Intelligence and Privacy layer for software engineering. It analyzes your project locally, builds a living knowledge model of the software system, and gives AI only the minimum context required to answer a question—without exposing the entire repository.

Topics

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages