Repository navigation
feat: unified operations-core architecture (0.3.0) + MCP extra - #40
Merged
Merged
Conversation
The Readwise export API returns integer `nextPageCursor` values, but
`parse_pagination_cursor` called `.startswith("http")` on the raw
value, crashing with `AttributeError` on non-string cursors. Add
`str()` coercion before the string check.
This was previously worked around by a monkey-patch in highlight_helper.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fold the standalone readwise-mcp server into readwise-plus as an optional interface, alongside the existing [cli] extra. - New readwise_sdk.mcp package (server + console entry point) exposing 9 agent-friendly tools over the SDK: documents (v3), highlights + books (v2). - New `mcp` optional-dependency and `readwise-mcp` console script (also runnable as `python -m readwise_sdk.mcp`). - Tests under tests/mcp/ (28 unit tests, mocked HTTP; live tests gated on READWISE_API_KEY). Graceful collect_ignore + importorskip when the extra isn't installed, mirroring the cli pattern. mcp added to the dev group so CI exercises it; ty excludes the FastMCP module like it does for cli. - README: MCP Server section, install extra, ToC entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design (Codex gpt-5.6 oracle) for the unified operations-core restructure, plus the standing Working-Effectively-With-Legacy-Code contract every stage must follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stage 1 of the 0.3.0 restructure (Feathers step 1: cover before you move). 69 characterization tests pinning today's observable behavior before any code moves — public exports, client/context-manager contracts, the three pagination cursor styles, manager return shapes, sync-state JSON schemas, CLI stdout/stderr/exit codes, and all nine MCP tools' exact JSON envelopes. Includes tests that pin known-buggy behavior AS-IS (e.g. save_to_reader drops `category`) so a later, separate commit can fix it deliberately. Tests only; no src/ changes. `just fc` green (605 passed). Generated by Codex gpt-5.6-sol under REFACTORING_DISCIPLINE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Adds immutable config.py (ClientConfig) and canonical errors.py; exceptions.py re-exports the same classes (identity-preserving, verified `exceptions.ReadwiseError is errors.ReadwiseError`). Sync/async clients delegate to ClientConfig with all defaults carried over verbatim (timeout 30.0, retries 3, backoff 0.5, v2/v3 base URLs). No behavior change; characterization + full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Moves retry classification/backoff and the pagination
engine out of client.py/_utils.py into transport/{retry,pagination,errors}.py,
with endpoint-aware page decoders (standard v2 `next`, v2 export integer
cursors, v3 `nextPageCursor`). Retry semantics (max_retries+1 attempts,
retry_backoff * 2**attempt, same retryable conditions) and cursor behavior
carried over verbatim. ReadwiseClient/AsyncReadwiseClient.paginate() remain
compatibility wrappers with unchanged signatures. Characterization + full
suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Adds canonical async transport (transport/async_.py) and a resources/ layer (v2 highlights/books/tags/export/review, v3 documents/tags) that build endpoint requests and decode models with no local search/format/ workflow logic. AsyncReadwiseV2Client/AsyncReadwiseV3Client now delegate to resources with unchanged public methods, signatures, and return types. Sync clients untouched. New tests cover the resources/transport surface; existing characterization suite unchanged. Non-live gate green (605 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New models/ package: operation input models (search/create/update inputs owning limit bounds + normalization), shared read/summary models (DocumentSummary/HighlightSummary/BookSummary matching current adapter projections), and result types (DocumentSearchResult, generic BulkResult, OperationFailure, unified SyncCheckpoint/SyncResult). Not yet wired into any call site. New unit tests for validation/normalization. Nothing existing changed; gate green (621 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New operations/documents.py + service wiring: canonical document operations (search, get, save, update, delete, move, tag add/remove/set, inbox/archive/later views, statistics) built on resources/ + stage-5 models. Semantics ported verbatim from DocumentManager, ReadingInbox, DocumentImporter, the MCP document tools, and CLI Reader commands — quirks carried as-is (list_documents single-tag; save_to_reader category still dropped). Operations own intent only (no formatting/protocol). MCP/CLI/managers NOT rewired yet. New operation tests; characterization unchanged; gate green (634 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive; completes the operations core. New operations/highlights.py and operations/books.py (wired into service): highlight search/list/get/create/ update/delete, bulk tag/untag, export, truncation + partial-failure policy; book search/list/get, book-with-highlights, reading stats. Semantics ported verbatim from the sync/async managers, HighlightPusher, CLI, and MCP tools (text-substring search, 1-200/1-100 limit clamps, text[:8191] truncation, BookCategory validation incl. supplementals, grouped export). BulkResult retains original exceptions to reproduce legacy dict/list/bool returns later. MCP/CLI/managers not rewired yet. Gate green (657 passed); characterization unchanged; uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving adapter rewire. Each MCP tool now does: arg adaptation → one operations service call → existing JSON rendering → existing error translation. Presenters (_doc_summary/_highlight_summary/_book_summary/ _json_result + error rendering) moved to mcp/output.py, behavior unchanged. The MCP characterization contract (test_mcp_contracts.py, 17 tests) passes UNCHANGED — byte-for-byte output preserved (tool names, compact JSON, None omission, not-found text, invalid-enum envelopes, limit clamps, lenient datetime, and the still-dropped save_to_reader category). No CLI/manager/ operation changes. Gate green (657 passed); uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New sdk/async_.py: AsyncReadwise async context manager composed from the operations service, exposing concept attributes (.documents/.highlights/ .books) plus a .raw.v2/.raw.v3 low-level escape hatch. Exported from the package root alongside the unchanged AsyncReadwiseClient and all existing imports. Facade tests cover construction, lifecycle, delegation, and .raw. Nothing existing changed; gate green (665 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New sdk/sync.py: Readwise sync context manager that runs the async operations service through a dedicated AnyIO blocking portal on its own event-loop thread, mirroring AsyncReadwise's surface and returning bounded results. The facade owns and closes its portal + async client. Adds anyio>=4.0 as a direct dependency. Exported from the package root; ReadwiseClient and all existing imports unchanged. Tests cover ordinary use, use from a thread with a running event loop (no deadlock), lifecycle, repeated ops, and exceptions crossing the portal. Gate green (677 passed); characterization unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cli extra (typer/rich) was not in the dev environment, so the CLI characterization + shell tests were silently skipped and the CLI wasn't type-checked in CI. Add them to the dev group (as done for mcp) so the CLI is a genuinely exercised surface during the restructure. No runtime deps change (typer/rich remain the optional [cli] extra for end users). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ore (stage 11)
Behavior-preserving adapter rewire. cli/main.py split into cli/commands/
{documents,highlights,books}.py + cli/context.py (service lifecycle + credential
discovery) + cli/output.py (renderer). Each migrated command calls exactly one
operation. Adds a global --output table|json|jsonl whose default reproduces
today's Rich output byte-for-byte (machine data → stdout, notices → stderr).
Existing command names/options/output/exit codes unchanged; test_cli_contracts.py
and test_cli.py pass unchanged. cli/ stays ty-excluded until PR17. Gate green
(767 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive. New commands (each one operation): document get/update/delete/move/ tag; highlight create/update/delete/tag/untag. Adds --output jsonl streaming and global --no-color/--quiet; JSON/JSONL output stays free of color codes and notices. Deprecated `reader ...` aliases forward to `documents ...` with a stderr-only deprecation notice. Existing command behavior/output/exit codes unchanged (test_cli_contracts.py + test_cli.py green). digest custom / sync status+reset deferred to stages 13-14 (their operations don't exist yet). Gate green (807 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e 13) Behavior-preserving. New operations/tags.py + operations/digests.py own tag and digest data/selection/grouping; new presenters/ (json/text/markdown/csv) own digest rendering. TagWorkflow and DigestBuilder become thin shims delegating to them with unchanged public methods, return types, and output — their existing workflow tests pass unchanged. AsyncReadwise/Readwise now expose .tags and .digests; new CLI `digest custom` command added. Characterization + workflow tests unchanged; gate green (821 passed); uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. New operations/sync.py (full/incremental/poll-once/batch/ background scheduling) + state/ (StateStore protocol + JsonFileStateStore). SyncManager, BatchSync, and BackgroundPoller become shims delegating to the unified sync operation, preserving public methods, return types, callback (and exception-suppression) semantics, and — critically — the on-disk JSON formats (state-contract tests green). New writes carry a version field; migration readers load all legacy formats; malformed state is never silently reset. .sync exposed on both facades; new `sync status`/`sync reset` commands (existing sync full/incremental unchanged). Characterization unchanged; gate green (835 passed); uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…uplication (stage 15) Behavior-preserving. HighlightManager/BookManager/DocumentManager, async_managers, HighlightPusher, DocumentImporter, and ReadingInbox now delegate to the operations layer; ~1,770 net lines of duplicated legacy implementation (incl. the ~800-line parallel async manager copy) deleted. Legacy return shapes preserved exactly — lists, lazy iterators, dict[id,bool], dataclasses, None, formatted output, broad exception suppression, and partial-failure behavior (BulkResult converted back to legacy boolean maps). No direct .v2/.v3 calls remain in these layers. Manager/contrib/async + characterization tests pass UNCHANGED; gate green (835 passed); uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Behavior-preserving. Model definitions moved to models/readwise.py (v2 concepts) and models/reader.py (v3 concepts); v2/models.py and v3/models.py become identity-preserving re-export shims (verified v2.models.HighlightCreate is models.readwise.HighlightCreate, and the v3 equivalent). Internal imports across resources/operations/sdk and mcp/server.py now use canonical paths. Every legacy model import path still resolves to the same class object. Characterization unchanged; gate green (835 passed); deps/uv.lock/Justfile untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the cli/ and mcp/ (and stale test) ty exclusions — the `type` recipe is now `uv run ty check .`, checking every module; the restructure left cli/ and mcp/ type-clean, so zero fixes were needed (verified ty is not no-opping). Adds a core-import isolation test proving `import readwise_sdk` pulls no typer/rich/mcp (subprocess + meta_path blocker to simulate a core-only install past httpx's own optional-CLI fallback). Adds py.typed (shipped in the wheel) and .gitattributes (eol=lf). Wheel builds with py.typed + both console scripts (readwise, readwise-mcp). Gate green (836 passed); deps/uv.lock unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(stage 18) Rewrites README around the new Readwise/AsyncReadwise facade (.documents/ .highlights/.books/.tags/.digests/.sync + .raw escape hatch), with accurate CLI (expanded tree + --output) and MCP sections and a Compatibility APIs section for the still-supported ReadwiseClient/managers/workflows/contrib. Adds MIGRATION.md (old→new mapping + deprecation timeline to 1.0) and a short docs/architecture.md contributor note; refreshes llms.txt/llms-full.txt. Every snippet was executed under respx mocks, which also corrected several pre-existing doc inaccuracies. Removes the refactor working docs (ARCHITECTURE_PLAN.md, REFACTORING_DISCIPLINE.md). Docs only; gate green (836 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
save_to_reader accepted a `category` argument but never forwarded it — the MCP tool didn't parse it, and the document save operation deliberately stripped it to preserve the pre-refactor bug. Now the tool validates category (returning an "Invalid category" envelope on bad input) and passes it through; the save operation forwards it unchanged. This is a deliberate, isolated behavior change: the three tests that pinned the dropped-category behavior are flipped to assert the fix (plus a new invalid-category rejection test). Gate green (837 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_typer_parse_error_is_exit_two_and_routed_to_stderr asserted a plain substring, but typer/rich colorizes its error box when color is forced (GitHub Actions enables it; a plain terminal doesn't), which split "--bad-option" with escape codes and failed only on CI's 3.14 job. Strip ANSI before the content assertions — the exit code, empty stdout, and stderr routing contract is unchanged; only the incidental coloring is normalized. Gate green (837 passed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restructures readwise-plus around a single async operations core with the SDK, CLI, and MCP server as thin adapters over it, following Michael Feathers' Working Effectively with Legacy Code discipline throughout (characterization tests first; every stage behavior-preserving and
just fc-green; public contract preserved via shims).Highlights
Readwise/AsyncReadwisewith.documents/.highlights/.books/.tags/.digests/.sync+.raw.v2/.raw.v3. OldReadwiseClient/.v2/.v3/managers/workflows/contrib still work (compat shims, unchanged behavior).readwise-plus[mcp]extra (9 tools; byte-for-byte identical output, proven by characterization contract tests).--output table|json|jsonl,readerdeprecation aliases.ty),py.typedshipped, core import pulls no typer/rich/mcp.See
MIGRATION.mdfor the old→new mapping and deprecation timeline (removal at 1.0). Full stage-by-stage history in the commits.🤖 Generated with Claude Code