Cleanup: format, lint, prune comments, fix tests, add CI

.github/workflows/ci.yml

@@ -0,0 +1,53 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install
+        run: pip install -e '.[dev]'
+
+      - name: Black
+        run: python -m black --check pyagent tests
+
+      - name: Ruff
+        run: python -m ruff check pyagent tests
+
+      - name: Smoke tests
+        run: |
+          set +e
+          fail=0
+          failed=""
+          for t in tests/test_*.py; do
+            name=$(basename "$t" .py)
+            echo "::group::$name"
+            python -m "tests.$name"
+            rc=$?
+            echo "::endgroup::"
+            if [ $rc -ne 0 ]; then
+              fail=$((fail+1))
+              failed="$failed $name"
+            fi
+          done
+          if [ $fail -gt 0 ]; then
+            echo "FAILED ($fail tests):$failed"
+            exit 1
+          fi
+          echo "All tests passed"

.pre-commit-config.yaml

@@ -0,0 +1,16 @@
+# Run `pre-commit install` once after cloning so these fire on every
+# commit. CI re-runs them as a safety net via `pre-commit run --all-files`.
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.12
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+        # `ruff format` is left disabled — black is the source of truth.
+        stages: [manual]
+
+  - repo: https://github.com/psf/black-pre-commit-mirror
+    rev: 26.3.1
+    hooks:
+      - id: black

README.md

@@ -1,24 +1,21 @@
 # Pyagent
 
-A multi-provider agent framework with plugin, skill, and subagent systems.
-Designed to explore how agent loops, tool calling, memory, and orchestration
-can be structured outside of large frameworks.
+A multi-provider agent framework with plugin, skill, subagent, and
+memory systems. Designed to explore how agent loops, tool calling,
+and orchestration can be structured outside of large frameworks.
 
-## Features
-
-- **Chat**: `pyagent` for a chat that can read files, run shell
-  commands, search the web, and call any tools you've added.
-- **Plugins, skills, memory**: opt into what you need.
-- **Subagents**: Subagents work independently and can communicate with their parent.
-- **Configurable**: Pyagent can be configured at the workspace and user level.
-- **Library**: `from pyagent import Agent, auto_client` to embed the
-  agent loop in your own app, notebook, or service.
-- **Bring your own model**: Anthropic, OpenAI, Gemini, or local Ollama.
+Use it two ways: `pyagent` for a terminal chat that can read files,
+run shell commands, search the web, and call any tool you've added,
+or `from pyagent import Agent, auto_client` to embed the same loop
+in your own app, notebook, or service. Bring your own model —
+Anthropic, OpenAI, Gemini, or local Ollama.
 
 [![asciicast](https://asciinema.org/a/5FvXNO6wzrSkKVwd.svg)](https://asciinema.org/a/5FvXNO6wzrSkKVwd)
 
 ## Quick start
 
+Not on PyPI yet — install from GitHub:
+
 ```bash
 pip install git+https://github.com/derekwisong/pyagent.git
 export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY / GEMINI_API_KEY
@@ -32,43 +29,47 @@ like `--model`, `--resume`, and `--prompt-dump` are documented in
 ## As a library
 
 ```python
-from pyagent import Agent, get_client
+from pyagent import Agent, auto_client
 
 def add(a: int, b: int) -> int:
     """Add two integers."""
     return a + b
 
-client = get_client("ollama/llama3.2:latest")   # local, no API key
-# client = get_client("anthropic/claude-sonnet-4-6")
-# client = get_client("openai/gpt-4o-mini")
-
-agent = Agent(client=client, system="You are a helpful calculator.")
+agent = Agent(client=auto_client(), system="You are a helpful calculator.")
 agent.add_tool("add", add)
 print(agent.run("What is 17 + 25?"))
 ```
 
+`auto_client()` picks a provider from the first env-var key it finds
+(`ANTHROPIC_API_KEY` → `OPENAI_API_KEY` → `GEMINI_API_KEY`). To pin a
+specific model, use `get_client("anthropic/claude-sonnet-4-6")` or
+`get_client("ollama/llama3.2:latest")` instead.
+
 Type hints and docstrings become the tool schema — no hand-written
-schemas.
+schemas. A runnable version of this snippet lives at
+[examples/quickstart.py](examples/quickstart.py).
 
 ## What's inside
 
-At first, this was a simple `Agent` class, written by hand as an exercise to learn
-how to build a flexible tool-calling agent.
-
-I used Claude Code to expand and build more of the features needed for
-a complete agent.
+The interesting design choices: tools are plain Python functions
+(no schema authoring), the system prompt is split into stable +
+volatile halves to keep provider caches warm across turns, and the
+plugin / skill / subagent boundary keeps each piece swappable. No
+LangChain, no agent-DSL — just an explicit `Agent.run()` loop that
+you can read top to bottom. See [docs/architecture.md](docs/architecture.md)
+for diagrams.
 
-| | |
+| Feature | What it does |
 |---|---|
-| **System Prompt Builder** | Build the system prompt through [SOUL.md](pyagent/defaults/SOUL.md) [PRIMER.md](pyagent/defaults/PRIMER.md) and more; leverages provider caching |
+| **System prompt builder** | Builds the prompt from [SOUL.md](pyagent/defaults/SOUL.md), [PRIMER.md](pyagent/defaults/PRIMER.md), and TOOLS sections; stable / volatile split keeps provider caches warm. |
 | **Tool calling** | Plain Python functions become tools — type hints + docstrings drive the schema. |
-| **Sessions** | Resumable conversations. |
-| **Plugins** | Write [plugins](docs/plugins.md) in Python to provide tools and hooks to extend Pyagent |
+| **Sessions** | Resumable conversations with on-disk JSONL transcripts. |
+| **Plugins** | [Plugins](docs/plugins.md) register tools, prompt sections, and lifecycle hooks. |
 | **Skills** | [Skills](docs/skills.md) are lazy-loaded "how do I" docs the agent reads on demand. |
-| **Subagents** | Spawn focused child agents; bidirectional comms (`ask_parent`, `tell_subagent`, async fan-out). |
-| **Memory** | USER ledger + MEMORY index, with semantic vector search recall via fastembed. Auto-loaded into the prompt. |
-| **Multi-model** | Switch mid-session with `/model`, define named roles per subagent in `config.toml`. |
-| **CLI** | Basic CLI REPL to converse with the agent |
+| **Subagents** | Spawn focused child agents; bidirectional comms (`ask_parent`, `tell_subagent`) and parallel calls (`call_subagent_async` + `wait_for_subagents`). |
+| **Memory** | USER ledger + [MEMORY](docs/design.md#memory) index with semantic recall via fastembed. Auto-loaded into the prompt. |
+| **Multi-model** | Switch mid-session with `/model`; define named roles per subagent in `config.toml`. |
+| **CLI** | Rich-rendered REPL with input queue, status footer, and slash commands. |
 
 ## Environment variables
 
@@ -82,6 +83,21 @@ a complete agent.
 
 Local Ollama needs no key — just the daemon running and a model pulled.
 
+## Tests
+
+Smoke tests live under `tests/` as standalone scripts (no pytest):
+
+```bash
+pip install -e '.[dev]'
+pre-commit install                       # one-time: run black + ruff on every commit
+python -m tests.test_token_meter        # run one
+for f in tests/test_*.py; do python -m tests.$(basename "$f" .py); done
+```
+
+Three recall sub-tests in `test_plugins.py` are gated on
+`PYAGENT_HEAVY_TESTS=1` because they download a ~130MB embedding
+model. CI runs the same loop on every push and pull request.
+
 ## License
 
 MIT — see [LICENSE](LICENSE).

docs/architecture.md

@@ -0,0 +1,101 @@
+# Architecture
+
+Three diagrams. See [design.md](design.md) for more detail.
+
+## System overview
+
+```mermaid
+flowchart TB
+    CLI[CLI or library code]
+    AGENT["<b>Agent.run()</b><br/>turn loop"]
+    PROMPT["System prompt<br/>SOUL · TOOLS · PRIMER<br/>+ plugin sections"]
+    TOOLS["Tools<br/>built-in + plugin-registered"]
+    PLUGINS["Plugins<br/>tools · hooks · prompt sections"]
+    LLM["LLM<br/>Anthropic / OpenAI / Gemini / Ollama"]
+    SESSION["Session<br/>conversation.jsonl + attachments/"]
+    SUB["Subagent processes<br/>multiprocessing.spawn"]
+
+    CLI --> AGENT
+    AGENT --> PROMPT
+    AGENT --> TOOLS
+    AGENT --> PLUGINS
+    AGENT --> LLM
+    AGENT <--> SESSION
+    AGENT <-->|duplex pipe| SUB
+```
+
+Notes:
+
+- Subagents are separate OS processes, not threads. Parent and child
+  talk over a duplex pipe carrying the event protocol in
+  [`pyagent/protocol.py`](../pyagent/protocol.py).
+- Anthropic, OpenAI, and Gemini ship as built-in clients in
+  `pyagent.llms`. Ollama is added by a bundled plugin via
+  `api.register_provider("ollama", ...)`. Third-party plugins can
+  register providers the same way.
+- The permissions gate only covers the built-in filesystem and shell
+  tools. Plugin tools and your own `add_tool`s don't go through it
+  unless they call it themselves.
+
+## The turn cycle
+
+```mermaid
+sequenceDiagram
+    participant U as User
+    participant A as Agent.run()
+    participant P as Plugins
+    participant L as LLM
+    participant T as Tool
+
+    U->>A: prompt
+    loop until no tool_calls
+        A->>L: respond(conversation, system, tools)
+        L-->>A: text + optional tool_calls
+        opt tool_calls
+            loop each call
+                A->>P: before_tool_call (may block / mutate)
+                A->>T: execute
+                T-->>A: result
+                A->>P: after_tool_call (may replace)
+            end
+        end
+    end
+    A-->>U: final text
+```
+
+Notes:
+
+- `before_tool_call` fires before the permissions prompt, so a plugin
+  can block a call before the human is asked to approve it.
+- The plugin set is rescanned at the top of every turn. A plugin the
+  agent just authored (via the `write-plugin` skill) is callable on
+  its next turn without restarting.
+
+## System prompt assembly
+
+```mermaid
+flowchart LR
+    subgraph CACHED["cached prefix (stable)"]
+        SOUL[SOUL]
+        T[TOOLS]
+        PR[PRIMER]
+        PLG_S[plugin sections]
+    end
+    BP{{breakpoint}}
+    subgraph FRESH["fresh every turn"]
+        VOL["volatile plugin sections<br/>· skills catalog · live state"]
+    end
+
+    SOUL --> T --> PR --> PLG_S --> BP --> VOL
+```
+
+The prefix is cached by the provider; anything past the breakpoint is
+sent fresh each turn. Plugin prompt sections pick a side via
+`volatile=True/False` on `register_prompt_section`. Anything that
+changes turn-to-turn (memory recall, skills catalog, live checklist)
+goes on the volatile side so it doesn't invalidate the cached prefix.
+
+---
+
+See [design.md](design.md) for more detail and
+[plugin-design.md](plugin-design.md) for the plugin author API.

docs/configuration.md

@@ -29,6 +29,7 @@ built_in_plugins_enabled = [
   "memory", "html-tools",
   "web-search", "reddit-search", "hn-search",
   "code-mapper", "claude-code-cli", "ollama", "py-dev-toolkit",
+  "echo-plugin",
 ]
 
 [subagents]

docs/design.md

@@ -1,7 +1,8 @@
 # Pyagent design
 
 How the agent loop is wired, where tools come from, how the system
-prompt is assembled, how memory works.
+prompt is assembled, how memory works. For diagrams, see
+[architecture.md](architecture.md).
 
 ## The agent loop
 
@@ -15,6 +16,19 @@ The loop lives in `Agent.run()` and does the same thing every turn:
   message, and loop back to the model.
 
 The model decides when it's done by simply not asking for any more tools.
+In code, that's:
+
+```python
+from pyagent import Agent, auto_client
+
+agent = Agent(client=auto_client(), system="You are helpful.")
+agent.add_tool("add", lambda a, b: a + b)
+print(agent.run("What is 17 + 25?"))
+```
+
+`run()` blocks until the model stops asking for tools; the
+concatenated assistant text is returned. Call it again to continue
+the conversation — history lives on `agent.conversation`.
 
 ## Tools
 
@@ -38,12 +52,12 @@ Built-in tools shipped with pyagent:
 | `fetch_url` | HTTP GET. Always saves the raw body to a session attachment; by default also returns markdown of the article body inline. |
 
 HTML tooling (`html_select`) comes from the bundled
-`html-tools` plugin and operate on saved attachments (or any local
-HTML file). Memory tools (`add_memory`, `read_memory`, `write_memory`,
-`write_user`, `set_memory_description`, `recall_memory`) come from the
-bundled `memory` plugin. Other bundled plugins (`code-mapper`,
-`web-search`, `reddit-search`, `hn-search`, `claude-code-cli`,
-`doc-tools`) register additional tools — see
+`html-tools` plugin and operates on saved attachments (or any local
+HTML file). Memory tools (`create_memory`, `read_memory`,
+`update_memory`, `delete_memory`, `write_user`, `recall_memory`)
+come from the bundled `memory` plugin. Other bundled plugins
+(`code-mapper`, `web-search`, `reddit-search`, `hn-search`,
+`claude-code-cli`, `doc-tools`) register additional tools — see
 [docs/plugins.md](plugins.md) for the full list.
 
 File tools resolve paths and refuse anything outside the workspace unless the
@@ -92,10 +106,10 @@ Long-term memory is provided by the bundled **`memory`** plugin.
   `MEMORY.md` index (also auto-loaded), and a `memories/` directory
   of individual entries (loaded on demand). Each body carries a
   `created_at` YAML frontmatter the read tools strip on the way out.
-- **Tools:** `add_memory` writes the body and updates the index in
-  one atomic call. `read_memory(file)` fetches a body. `write_memory`
-  and `write_user` overwrite an existing body or USER. `set_memory_description`
-  retunes one bullet line in MEMORY.md without rewriting the index.
+- **Tools:** `create_memory` writes a body and updates the index in
+  one atomic call. `read_memory(file)` fetches a body. `update_memory`
+  edits an existing body and refreshes the bullet line in MEMORY.md;
+  `write_user` overwrites USER.md. `delete_memory` is role-only.
 - **Recall:** `recall_memory(query, ...)` runs cosine search over an
   L2-normalized vector index built with fastembed (BGE-small-en-v1.5).
   Index files (`vectors.npy`, `index.json`) live alongside MEMORY.md,