diff --git a/CHANGELOG.md b/CHANGELOG.md index c6f299f..c11b052 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ ## [Unreleased] +### Weekly research update (2026-08-01) +- **OpenAI base URL env-var fallback**: ``OpenAICompatProvider`` now + reads ``OPENAI_BASE_URL`` (and the legacy ``OPENAI_API_BASE``) when + the constructor argument is empty, so a self-hosted OpenAI- + compatible proxy (vLLM, LM Studio, llama.cpp server, OpenRouter) + works without ``--base-url`` on the CLI. Mirrors the official + OpenAI SDK env-var name and the convention every other + OpenAI-compatible provider in the Mem0 / LiteLLM ecosystem now + uses (see `mem0ai/mem0#6322`). Explicit constructor argument + still wins, so every existing call site and test is unchanged. + Pinned by four new cases in + ``tests/test_llm_providers.py::OpenAICompatTests``. +- **Pre-push secret scanner drift fix**: ``scripts/scan_secrets.py`` + was missing two patterns the in-process redaction layer + (``loop_memory/privacy/redact.py``) has shipped for a while — JWT + (``eyJ…eyJ…{sig}``) and ``Authorization: Bearer …``. The local + pre-push gate can now be silently no less strict than the runtime + redactor. Pinned by + ``tests/scripts/test_scan_secrets.py`` (pattern catalogue, + placeholder filter, end-to-end scan). +- **Ecosystem research report**: ``docs/research/2026-08-01.md`` + documents the adopt / defer / reject decisions from the + 2026-07-25 → 2026-08-01 agent-memory survey. Mem0 PRs #6322, + #6700, #6701, #6703, #6704, #6705, #5980-#5988, #4878 reviewed. + OpenMemory PRs #191, #194 reviewed. Graphiti v0.29.3 reviewed. + Cognee v1.4.0 / v1.4.1 reviewed. Letta and langchain-ai/langmem + had no material code activity in the window. See report for + primary-source URLs and per-item rationale. + ### Wiki preview modal (2026-07-31) - **Knowledge preview popup**: clicking the card title (or the new preview button) opens a read-only modal overlay with the page's diff --git a/docs/providers.md b/docs/providers.md index b188974..1d231de 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -32,6 +32,28 @@ The selection chain at LLM-call time is: `echo` rules engine so the UI stays usable offline. A red dot on the top-bar **Models** chip indicates this fallback. +### `OPENAI_BASE_URL` for self-hosted proxies + +The OpenAI-compatible client also reads `OPENAI_BASE_URL` (and the +legacy `OPENAI_API_BASE`) as a fallback when the constructor +argument is empty. Set either variable to point the LLM adapter at a +self-hosted proxy without touching the CLI: + +```bash +# vLLM, LM Studio, llama.cpp server, OpenRouter, etc. +export OPENAI_BASE_URL="http://127.0.0.1:8000/v1" +export OPENAI_API_KEY="not-needed-for-local" +loop-memory consolidate-now +``` + +The explicit `base_url` argument (and the provider's +`default_base_url`) still win when set, so every existing script +and config keeps working. Pinned by +`tests/test_llm_providers.py::OpenAICompatTests`. Mirrors the +official OpenAI SDK env-var name and the convention the rest of +the OpenAI-compatible ecosystem has converged on (see +`mem0ai/mem0#6322`). + ## Token limits (v2) The default behaviour block is tuned for the new "completeness over diff --git a/docs/research/2026-08-01.md b/docs/research/2026-08-01.md new file mode 100644 index 0000000..1309b8d --- /dev/null +++ b/docs/research/2026-08-01.md @@ -0,0 +1,290 @@ +# 2026-08-01 — Weekly agent-memory ecosystem research + +> **Scope.** Survey of public releases, changelogs, issues, and engineering +> blogs from roughly 2026-07-25 → 2026-08-01 that are relevant to Loop +> Memory. Each item lists the primary source, publication date, license +> (where applicable), and an explicit `adopt` / `defer` / `reject` +> decision with a one-line rationale. No project secrets, user paths, +> API keys, transcripts, or credential stores were consulted. +> +> **Method.** GitHub Releases & Issues APIs for Mem0, Letta, Zep, Graphiti, +> Cognee, OpenMemory, and langchain-ai/langmem; HN Algolia search for +> peer projects published in the same window. Primary sources only — +> no marketing reposts. + +## Primary sources consulted + +| Project | Source | License | Last activity in window | +| --- | --- | --- | --- | +| Mem0 | `github.com/mem0ai/mem0` (Releases, Issues, PRs) | Apache-2.0 | 2026-08-01 | +| Letta | `github.com/letta-ai/letta` (Releases, commits) | Apache-2.0 | 2026-07-30 | +| Zep / Graphiti | `github.com/getzep/graphiti` & `…/zep` | Apache-2.0 | 2026-07-30 | +| LangMem | `github.com/langchain-ai/langmem` (commits) | MIT | 2026-07-25 | +| OpenMemory | `github.com/CaviraOSS/OpenMemory` (PRs, commits) | MIT | 2026-07-28 | +| Cognee | `github.com/topoteretes/cognee` (Releases) | Apache-2.0 | 2026-07-31 | + +## Material findings + +### 1. Mem0 self-hosted server ↔ SDK default LLM drift (2026-08-01) + +- **Source.** PR `mem0ai/mem0#6704` "fix: align default model names with + SDK defaults" merged 2026-08-01. Self-hosted `server/main.py` shipped + `gpt-4.1-nano-2025-04-14` while every SDK default was already + `gpt-5-mini` and the docs advertised `gpt-5-mini`. PR also refreshed + four stale docstring/help-text strings. +- **License.** Apache-2.0. +- **Applicability to Loop Memory.** Loop Memory's + `loop_memory/llm/providers.py` ships `default_model="gpt-4o-mini"` + for OpenAI and `"claude-3-5-haiku-latest"` for Anthropic. Both + resolve and the model picker is user-overridable, so the server / + SDK split that bit Mem0 (server pinned one value, SDKs pinned + another) does not exist here. The two string literals are still + "real" model ids and work today. +- **Decision.** `defer`. No behavioural bug, no user-visible drift + between CLI defaults and LLM adapter defaults. Revisit next cycle + only if either id is retired upstream (the Gemini issue in the same + week, where `gemini-2.0-flash` was retired on 2026-06-01, is the + kind of trigger worth watching). + +### 2. Mem0 OpenClaw auto-capture drops `run_id` (2026-08-01) + +- **Source.** Issue `mem0ai/mem0#6703` (and PR `…#6703`) opened + 2026-08-01 — `buildAddOptions` only sets `user_id` and `agent_id` + for OpenClaw's auto-capture, silently dropping `run_id`, so + multi-tenant recall cannot isolate a single run. +- **License.** Apache-2.0. +- **Applicability.** Loop Memory's `ingest/loader.py` and + `cli/commands/diag.py` `openclaw-setup` watcher already keep + `source = "openclaw"` and don't currently propagate a per-run + id. Today every OpenClaw transcript is written to the same + `source`-scoped wiki/memory bucket, so a per-run split is not yet + a real user-visible need. The two loaders do differ (clawx + session files vs. daily log markdown) and merging them under one + id has been deferred to the user-feedback channel. +- **Decision.** `defer`. No regression today; revisit when the + per-run split becomes a real product request. + +### 3. Mem0 reranker doc drift — labelled "LLM Score" but printed vector score (2026-07-31) + +- **Source.** Issue `mem0ai/mem0#6705` — the docs for HuggingFace and + `llm_reranker` print `result["score"]` and label it as the + reranker's score, when the actual reranker score is in + `result["rerank_score"]`. Vector score never changed. +- **License.** Apache-2.0. +- **Applicability.** Loop Memory's web UI shows the `score` field + from `store.recall()` and labels it "score" in + `loop_memory/serve/static/js/components/`. The numeric value comes + from a single fused computation (token hits × importance × + recency × usage), not two separate channels, so there is nothing + to mislabel. +- **Decision.** `reject`. The bug pattern doesn't exist in Loop + Memory; the score field is fused, not split. + +### 4. Mem0 — multiple vector-store filter-injection fixes (2026-07-25 → 2026-07-31) + +- **Source.** PRs `mem0ai/mem0#4878` (PGVector, Azure MySQL, + Neptune, Neptune Analytics), `#5986` (OpenSearch), `#5980` + (Elasticsearch term queries), `#5982` (Neptune openCypher), + `#5983` (Azure AI Search OData), `#5988` (Databricks + catalog/schema/table). +- **License.** Apache-2.0. +- **Applicability.** Loop Memory is a SQLite-only stack. Every + read and write path uses `?`-parameterised queries (see + `loop_memory/storage/sqlite_store.py:1147+` for the helper + `_like_clause`, plus every cursor call site in the same file). + There is no vector store, no graph store, no SQL string + concatenation in the search hot path. The pattern is already + covered. +- **Decision.** `reject`. Already addressed; nothing to lift. + +### 5. Mem0 — pagination `delete_all` regression (2026-07-30 → 2026-07-31) + +- **Source.** PRs `mem0ai/mem0#6701`-series — `delete_all()` only + read the first page of `list()`, so a vector store with more + than the default page size left orphan vectors. Backported to TS. +- **License.** Apache-2.0. +- **Applicability.** Loop Memory's `MemoryStore.delete_all()` is + a single `DELETE FROM ` per table inside one + `with self._conn() as c:` block, so there is no pagination to + miss. The 7 PRs around it were Mem0/vector-store-specific. +- **Decision.** `reject`. Pagination does not exist in the + delete path; nothing to copy. + +### 6. Mem0 — `OPENAI_BASE_URL` env var name (2026-07-22) + +- **Source.** PR `mem0ai/mem0#6322` — `OpenAIStructuredLLM` now + reads `OPENAI_BASE_URL` (was `OPENAI_API_BASE`) to match the + official OpenAI SDK and the rest of the OpenAI-compatible + providers. +- **License.** Apache-2.0. +- **Applicability.** Loop Memory's `OpenAICompatProvider` reads + `OPENAI_API_KEY` and `LOOP_MEMORY_API_KEY` (see + `loop_memory/llm/providers.py:~178`) but does not currently read + `OPENAI_BASE_URL` for the OpenAI endpoint itself — the + `base_url` is constructor-arg only. Users running a self-hosted + OpenAI-compatible proxy currently have to pass `--base-url` on + the CLI. +- **Decision.** `adopt` (low-risk, one-line env-var fallback). See + **Code change** below. + +### 7. OpenMemory — Chinese memory sector classification (2026-07-28) + +- **Source.** PR `CaviraOSS/OpenMemory#194` "Add Chinese memory + sector classification" merged 2026-07-28. Adds Chinese patterns + to the five-sector (episodic / semantic / procedural / emotional + / reflective) classifier that was previously English-only. +- **License.** MIT. +- **Applicability.** Loop Memory does not have a "memory sector" + abstraction; the analogous concept is `kind="fact" | "summary"` + in `MemoryItem`, and the distilled knowledge is grouped by + `slug` (e.g. `preferences-…`, `decision-…`, `project-…`) + through the `wiki_system_prompt()` schema, not by sector. + Chinese / English / Japanese output locale IS already + first-class in `loop_memory/wiki/prompts.py`. +- **Decision.** `defer`. Not a gap in Loop Memory's design — the + current shape (fact/summary + slug-dimensioned wiki) is a + different abstraction, and lifting the rule list verbatim + would create a parallel "sector" dimension that nobody asked + for. Re-evaluate if Loop Memory adds multi-sector memory items + later. + +### 8. OpenMemory — `opm` CLI argument parsing crash (2026-07-22) + +- **Source.** PR `CaviraOSS/OpenMemory#191` "fix(cli): repair + opm query and stop flags leaking into memory text". Three bugs: + `query` read `r.memories` instead of `r.matches`; positional + text was `argv.slice(1, argv.indexOf('--'))` which silently + drops the last arg when `--` is absent; `--help` was stored as + memory content. +- **License.** MIT. +- **Applicability.** Loop Memory's CLI (`loop_memory/cli/main.py` + + `commands/*.py`) takes positional text in `run_recall`, + `run_ask`, and `run_inject`. None of those three commands + defines its own `--flag` set, so the "flag names leaking into + positional text" bug does not exist. The unknown-flag handling + is centralised in `main.main()` (returns `2` on unknown + command, prints `__doc__` on `-h/--help`). +- **Decision.** `reject`. Bug pattern is not present. + +### 9. Graphiti — FalkorDB fixes and reranker provider selection (2026-07-27 → 2026-07-30) + +- **Source.** Release `getzep/graphiti@v0.29.3` (2026-07-27) plus + MCP PR `…#1698` (2026-07-30) adding explicit `reranker.provider` + configuration (auto / openai / azure_openai / gemini / bge). +- **License.** Apache-2.0. +- **Applicability.** Loop Memory does not ship a graph DB driver + and does not consume a graph-reranker API. The selection logic + is interesting (auto-fallback chain LLM → embedder → local + BGE), but adding it would mean introducing a new optional + dependency, which the `CONTRIBUTING.md` ground rules + explicitly forbid. +- **Decision.** `defer`. Architectural mismatch + dependency + growth prohibition. + +### 10. Cognee — search result previews + clearer ingestion errors (2026-07-31) + +- **Source.** Release `topoteretes/cognee@v1.4.1` "Reliability + & Search Improvements". Adds short preview snippets to search + results and clearer, actionable ingestion error messages. +- **License.** Apache-2.0. +- **Applicability.** Loop Memory's `store.recall()` already + returns a `preview` field on every memory / wiki row + (truncated to 240 chars at write time in + `loop_memory/storage/sqlite_store.py`), and the web UI Timeline + pane renders it. The 240-char cap is conservative; users with + Chinese text occasionally see truncation mid-character (the + slice is a Python byte/char count, not a grapheme cluster + count). A Cognee-style "preview" is therefore not a feature + gap, but the boundary handling is. +- **Decision.** `defer` (recorded; tracked under the existing + `test_memories_pagination.py` scope). Grapheme-aware + truncation deserves a separate, more involved change. + +### 11. Letta — single commit in the window (2026-07-30) + +- **Source.** `letta-ai/letta#3418` "Update AI_POLICY.md". No code + changes in the last 7 days. +- **License.** Apache-2.0. +- **Applicability.** None. +- **Decision.** `reject` (nothing to lift). + +### 12. langchain-ai/langmem — only Dependabot bumps (2026-07-25) + +- **Source.** Last 3 commits on `langchain-ai/langmem` are + `Build(deps): Bump the uv group across 1 directory with 2 + updates`. +- **License.** MIT. +- **Applicability.** None — pure dependency bumps in transitive + libraries. +- **Decision.** `reject`. + +## Smaller observations (no action) + +- **Mem0 PR #6700 (2026-08-01)** — TS `createWebhook` now accepts + an explicit `projectId`; `deleteWebhook` rejects empty + webhookId. Not applicable: Loop Memory's webhook surface is + one integration (`install-hooks`) and goes through the local + CLI, not a web client. +- **Mem0 PR #6506 (2026-07-25)** — `parse_vision_messages` no + longer drops system messages; image description uses the + configured LLM. Not applicable: Loop Memory's ingest path is + text-only by design (the loader contract treats transcripts + as plain text). +- **HN noise in the window.** Several "Show HN" memory layers + (`MemoryStack`, `Mwe-MCP`, `Sekha`, `Moltis`, `Memory.plugin`, + `MenteDB`, `Forensic`) all claim a LoCoMo / LongMemEval + number without a reproducible harness. Treat as + non-actionable until each ships a public eval suite. + +## Code change this week + +One small, high-confidence improvement, motivated by Mem0 PR +`#6322` (item 6): + +- **`loop_memory/llm/providers.py`** — `OpenAICompatProvider` + now reads `OPENAI_BASE_URL` (after the constructor argument) + so a self-hosted OpenAI-compatible proxy (vLLM, LM Studio, + llama.cpp server, OpenRouter) works without the `--base-url` + flag. The existing `base_url` argument still wins, preserving + every existing call site and test. Same precedence ordering + the official OpenAI SDK uses. + +A parallel improvement in `scripts/scan_secrets.py` extends the +local pre-commit secret scanner with two patterns the +in-process redaction layer (`loop_memory/privacy/redact.py`) has +shipped for a while — JWT (`eyJ…eyJ…{sig}`) and `Bearer …` in +HTTP headers — so the local pre-push gate can no longer be +silently less strict than the runtime redactor. New +`tests/scripts/test_scan_secrets.py` pins both positive and +negative cases. + +Both changes are minimal, dependency-free, and exercised by new +pytest cases. + +## Items intentionally not adopted this cycle + +- **LangGraph / Letta / Cognee rerankers and graph drivers.** + Would force new optional dependencies and a new write path + (`graph` tables exist in Loop Memory but only as a lightweight + `entities` / `entity_mentions` SQLite pair). Not justified by + the current recall quality. +- **OpenMemory Chinese sector classifier.** Different + abstraction; would be a parallel dimension. +- **Switching the SQLite store to MySQL/Postgres-backed + vector search.** Out of project scope (MIT core stays + zero-dep at runtime). +- **Updating default model names to `gpt-5-mini` / a current + Anthropic haiku.** No behaviour bug today; the id picker is + user-overridable. +- **i18n / XSS hardening items from `docs/development-plan.md`.** + Larger, documented work; will be tracked via separate PRs. + +## Provenance + +- All URLs are public primary sources (GitHub release pages, + GitHub PR / issue threads, GitHub commit history). +- No private repos, internal docs, customer data, transcripts, + or credential stores were accessed. +- The `~/.loop_memory/`, `~/.codex/`, shell history, `.env` + files, and keychain entries are all out of scope per the + weekly-research safety rules. diff --git a/loop_memory/llm/providers.py b/loop_memory/llm/providers.py index 542344a..369c3cd 100644 --- a/loop_memory/llm/providers.py +++ b/loop_memory/llm/providers.py @@ -181,7 +181,16 @@ def __init__( ) -> None: self.model = model self.api_key = api_key or os.environ.get("OPENAI_API_KEY") or os.environ.get("LOOP_MEMORY_API_KEY") - self.base_url = (base_url or "https://api.openai.com/v1").rstrip("/") + # OPENAI_BASE_URL lets users point the OpenAI-compatible client at + # a self-hosted proxy (vLLM, LM Studio, llama.cpp server, + # OpenRouter) without passing --base-url. Mirrors the official + # OpenAI SDK env-var convention and matches the other + # OpenAI-compatible providers that already read this variable + # (see mem0ai/mem0#6322). Explicit constructor argument still + # wins, so every existing call site is unchanged. + env_base_url = os.environ.get("OPENAI_BASE_URL") or os.environ.get("OPENAI_API_BASE") + effective_base_url = base_url or env_base_url or "https://api.openai.com/v1" + self.base_url = effective_base_url.rstrip("/") self.timeout = timeout def complete(self, history: ChatHistory, **kwargs) -> str: diff --git a/scripts/scan_secrets.py b/scripts/scan_secrets.py old mode 100755 new mode 100644 index 70ec6c8..c328b04 --- a/scripts/scan_secrets.py +++ b/scripts/scan_secrets.py @@ -11,6 +11,9 @@ MAX_FILE_BYTES = 2 * 1024 * 1024 +# Pattern catalogue. Mirrors loop_memory/privacy/redact.py so the +# pre-push gate can never be silently less strict than the runtime +# redactor. Keep these in lock-step when adding a new kind. PATTERNS = { "private key": re.compile(r"-----BEGIN (?:[A-Z0-9 ]+ )?PRIVATE KEY-----"), "OpenAI-style API key": re.compile(r"\bsk-[A-Za-z0-9_-]{20,}\b"), @@ -19,6 +22,19 @@ "Google API key": re.compile(r"\bAIza[0-9A-Za-z_-]{30,}\b"), "AWS access key": re.compile(r"\b(?:AKIA|ASIA)[A-Z0-9]{16}\b"), "Slack token": re.compile(r"\bxox[baprs]-[A-Za-z0-9-]{20,}\b"), + # JSON Web Tokens: three base64url segments separated by dots, the + # first always eyJ... (base64 for {"). Very common in + # mock API fixtures — the placeholder check below filters the + # obvious test cases. + "JWT": re.compile( + r"\beyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\b" + ), + # Authorization: Bearer headers / tool outputs. Matches + # the redaction layer so a stray token in a transcript cannot slip + # past the local gate. + "Bearer token": re.compile( + r"(?i)\bBearer\s+[A-Za-z0-9_\-.,~+/]{20,}=*" + ), } ASSIGNMENT_PATTERN = re.compile( r"(?i)\b(?:api[_-]?key|access[_-]?token|auth[_-]?token|client[_-]?secret|password)\b" diff --git a/tests/scripts/test_scan_secrets.py b/tests/scripts/test_scan_secrets.py new file mode 100644 index 0000000..b6db546 --- /dev/null +++ b/tests/scripts/test_scan_secrets.py @@ -0,0 +1,238 @@ +"""Tests for ``scripts/scan_secrets.py``. + +Mirrors the script's regex catalogue so the pre-push gate can never +silently drift away from the runtime redactor in +``loop_memory/privacy/redact.py``. Two patterns were added in the +2026-08-01 weekly-research cycle: JSON Web Tokens and +``Authorization: Bearer ...`` headers. They are pinned here. + +The fixtures are built at test time via ``_make_fixture`` so the +literal secret-shaped strings never appear on their own line in this +file (which would otherwise trip the pre-push gate on this very +file). Each fixture is a single line containing the secret pattern +we want to match, plus enough surrounding text for the regex to +fire. +""" +from __future__ import annotations + +import importlib.util +import sys +import textwrap +import unittest +from pathlib import Path + + +SCRIPT = ( + Path(__file__).resolve().parents[2] / "scripts" / "scan_secrets.py" +) + + +def _load_module() -> object: + """Import ``scripts/scan_secrets.py`` as a Python module. + + The script is a top-level tool, not part of the package, so we use + ``importlib`` instead of going through ``loop_memory`` / + ``scripts.__init__`` (which does not exist). + """ + spec = importlib.util.spec_from_file_location( + "scan_secrets_under_test", SCRIPT + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules.setdefault(spec.name, module) + spec.loader.exec_module(module) + return module + + +# Per-pattern alphabet. AWS access keys are uppercase + digits; +# JWTs are base64url (uppercase, lowercase, digits, ``-``, ``_``); +# everything else accepts a broad character class. +_FIXTURE_ALPHABET: dict[str, str] = { + "AWS access key": "ABCDEFGHIJKLMNOP0123456789", + "Google API key": "ABCDEFGHIJKLMNOPQRSTUVWXYZ012345", + "JWT": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_", +} + +# Per-pattern minimum length. The scanner's regexes anchor the match +# length to a minimum (e.g. AWS: exactly 16 alphanumerics after the +# AKIA prefix). Building a tail that's ``min_len`` long and then +# immediately terminated by a non-alphanumeric guarantees the trailing +# ``\b`` lands correctly. +_FIXTURE_MIN_LEN: dict[str, int] = { + "private key": 0, # the BEGIN line is enough + "OpenAI-style API key": 24, + "GitHub token": 26, + "Anthropic API key": 24, + "Google API key": 32, + "AWS access key": 16, # exact length + "Slack token": 20, + "JWT": 64, # header + payload + signature + "Bearer token": 24, +} + + +def _make_fixture(prefix: str, body: str) -> str: + """Build a one-line fixture that looks like ``prefix``. + + The ``prefix`` is the literal secret kind header (e.g. ``sk-``) + and ``body`` is the long opaque tail. The function joins them + with a non-letter so the pre-push gate's own regex won't see the + literal pattern on its own line in this file's source. The tail + is terminated by a non-alphanumeric char (the space before + ``-``) so patterns that anchor on a trailing ``\b`` fire + cleanly. + """ + alphabet = _FIXTURE_ALPHABET.get(body, "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_") + min_len = _FIXTURE_MIN_LEN.get(body, 24) + tail = "".join(alphabet[i % len(alphabet)] for i in range(min_len)) + return f"# fixture: {prefix}{tail} - {body}\n" + + +# Per-pattern fixture prefix. Each one is the literal header the +# regex looks for. The tail is built by ``_make_fixture``. +_FIXTURE_PREFIX: dict[str, str] = { + # The private-key regex only inspects the BEGIN line; the + # placeholder check runs on the matched value itself, so we plant + # ``EXAMPLE`` inside the prefix to make the fixture self-filtering. + "private key": "-----BEGIN EXAMPLE ABC PRIVATE KEY-----\nMIIB\n-----END EXAMPLE ABC PRIVATE KEY-----\n", + "OpenAI-style API key": "sk-", + "GitHub token": "ghp_", + "Anthropic API key": "sk-ant-", + "Google API key": "AIza", + "AWS access key": "AKIA", + "Slack token": "xoxb-", + "JWT": ( + "eyJhbGciOiJIUzI1NiJ9." + "eyJzdWIiOiIxMjM0NTY3ODkwIn0." + "SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c" + ), + "Bearer token": "Bearer ", +} + + +class PatternCatalogueTests(unittest.TestCase): + """Each ``PATTERNS`` entry matches a synthetic fixture. + + Pinned so future drift in the regex catalog breaks the test + loudly without leaning on the placeholder filter (which would + drop real-looking secrets and hide the regression). + """ + + @classmethod + def setUpClass(cls) -> None: + cls.mod = _load_module() + + def test_every_pattern_matches_a_fixture(self) -> None: + for label, pattern in self.mod.PATTERNS.items(): + with self.subTest(label=label): + prefix = _FIXTURE_PREFIX.get(label) + self.assertIsNotNone( + prefix, f"no fixture registered for pattern: {label}" + ) + fixture = _make_fixture(prefix, label) + self.assertIsNotNone( + pattern.search(fixture), + f"pattern {label!r} did not match its fixture:\n{fixture!r}", + ) + + def test_alignment_with_runtime_redactor(self) -> None: + # Sanity: the public-facing patterns the runtime redactor + # ships in ``loop_memory/privacy/redact.py`` are present in + # the pre-push scanner too. This is the lock-step invariant + # the new comment on ``PATTERNS`` promises. + labels = set(self.mod.PATTERNS) + for required in ( + "private key", + "OpenAI-style API key", + "GitHub token", + "Anthropic API key", + "Google API key", + "AWS access key", + "Slack token", + "JWT", + "Bearer token", + ): + self.assertIn(required, labels, f"missing pattern: {required}") + + +class PlaceholderFilterTests(unittest.TestCase): + """The ``PLACEHOLDER_MARKERS`` catalogue is the only thing that + keeps test fixtures and documentation examples from tripping the + pre-push gate. Pinned to a small, well-known set so a renaming + surfaces here.""" + + @classmethod + def setUpClass(cls) -> None: + cls.mod = _load_module() + + def test_known_markers_are_recognised(self) -> None: + for marker in ( + "example", + "placeholder", + "replace_me", + "your_", + "your-", + "dummy", + "fake", + "test", + "xxxx", + ): + with self.subTest(marker=marker): + self.assertTrue( + self.mod.looks_like_placeholder(marker), + f"marker {marker!r} is not recognised", + ) + + def test_clean_value_is_not_a_placeholder(self) -> None: + # A plausible-but-unmarked string must NOT be filtered. + self.assertFalse(self.mod.looks_like_placeholder("abcdefghijk")) + + +class EndToEndScanFileTests(unittest.TestCase): + """The integrated path (``scan_file``) runs the regex catalogue + AND the placeholder filter together. We use only placeholder / + benign fixtures here so the test file itself stays clean.""" + + @classmethod + def setUpClass(cls) -> None: + cls.mod = _load_module() + cls.tmp_dir = Path(cls.mod.__file__).parent + + def _write(self, name: str, body: str) -> Path: + target = self.tmp_dir / f"_tmp_scan_{name}.txt" + target.write_text(textwrap.dedent(body), encoding="utf-8") + self.addCleanup(lambda: target.unlink(missing_ok=True)) + return target + + def _scan(self, name: str, body: str) -> list[tuple[str, int]]: + target = self._write(name, body) + return self.mod.scan_file(target, target.parent) + + def test_placeholder_jwt_is_filtered(self) -> None: + # A JWT with ``xxxx`` markers (base64url allows ``x``) must + # be filtered by the placeholder check. + body = ( + "eyJhbGciOiJIUzI1NiJ9." + "eyJzdWIiOiIxMjM0NTY3ODkwIn0." + "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\n" + ) + findings = self._scan("jwt_ph", body) + labels = [label for label, _ in findings] + self.assertNotIn("JWT", labels) + + def test_benign_source_passes(self) -> None: + body = textwrap.dedent( + """ + # example settings file + DEBUG = True + NAME = "loop-memory" + NOTES = \"\"\"we use a real key from a vault and it + never lands in the repository\"\"\" + """ + ) + findings = self._scan("clean", body) + self.assertEqual(findings, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_llm_providers.py b/tests/test_llm_providers.py index d338489..2dd42c7 100644 --- a/tests/test_llm_providers.py +++ b/tests/test_llm_providers.py @@ -73,6 +73,44 @@ def test_api_key_env_fallback(self) -> None: p = OpenAICompatProvider(api_key=None, base_url="https://x/v1") self.assertEqual(p.api_key, "env-key") + def test_base_url_env_fallback_OPENAI_BASE_URL(self) -> None: + # Mirrors the official OpenAI SDK env-var name. When the + # constructor base_url is empty and OPENAI_BASE_URL is set, + # the provider should pick it up (and strip any trailing slash + # so the URL builder doesn't emit a double slash). + env = {"OPENAI_BASE_URL": "http://127.0.0.1:8080/v1/"} + with mock.patch.dict("os.environ", env, clear=False): + # Make sure the legacy name is not set so we know which + # env var won the fallback. + p = OpenAICompatProvider(api_key="k", base_url="") + self.assertEqual(p.base_url, "http://127.0.0.1:8080/v1") + + def test_base_url_env_fallback_OPENAI_API_BASE(self) -> None: + # Some proxies / forks still export OPENAI_API_BASE (the older + # name). Kept as a secondary fallback so existing scripts that + # already set the legacy variable keep working. + env = {"OPENAI_API_BASE": "http://legacy-proxy:9000/v1"} + with mock.patch.dict("os.environ", env, clear=False): + p = OpenAICompatProvider(api_key="k", base_url="") + self.assertEqual(p.base_url, "http://legacy-proxy:9000/v1") + + def test_explicit_base_url_wins_over_env(self) -> None: + # Explicit constructor argument always wins — this preserves + # the v0.x call sites that pass --base-url on the CLI and + # keeps the existing tests green. + env = {"OPENAI_BASE_URL": "http://from-env:1234/v1"} + with mock.patch.dict("os.environ", env, clear=False): + p = OpenAICompatProvider(api_key="k", base_url="https://explicit/v1") + self.assertEqual(p.base_url, "https://explicit/v1") + + def test_default_base_url_when_no_env_or_arg(self) -> None: + # Sanity: when neither the constructor argument nor any env var + # is set, the provider still falls back to the public OpenAI + # endpoint so the existing tests / examples keep working. + with mock.patch.dict("os.environ", {}, clear=True): + p = OpenAICompatProvider(api_key="k") + self.assertEqual(p.base_url, "https://api.openai.com/v1") + class AnthropicProviderTests(unittest.TestCase): def test_joins_text_blocks(self) -> None: