Skip to content

feat: claude-science + claude-tag product launches + archive pipeline fixes #1215

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
claude-yolo merged 5 commits into from
Aug 25, 2026
Merged

feat: claude-science + claude-tag product launches + archive pipeline fixes #1215

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9176562
fix(fetcher): close the archive's three silent-drift gaps
lroolle Aug 25, 2026
e553a55
fix(fetcher): stop writing redirect targets back to the old path
lroolle Aug 25, 2026
01a17aa
Merge origin/main into docs/archive-integrity-20260824
lroolle Aug 25, 2026
e8548e9
docs: full sync 2026-08-25 — 3,809/3,810 live docs
lroolle Aug 25, 2026
c86de1e
feat(fetcher): make discovery a tracked file, and stop the metadata l…
lroolle Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
26 changes: 26 additions & 0 deletions .github/workflows/fetch-claude-docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ jobs:
- `content/blog/` - FROZEN archive (anthropic.com is HTML-only; no longer fetched)
- `content/github/` - GitHub repos (cookbooks, skills, plugins, courses)
- `content/support/` - Support articles (sitemap + .md)
- `content/claude/` - Product docs (Claude Tag, Cowork, office agents, connectors)

## YOUR WORKFLOW

Expand All @@ -119,9 +120,34 @@ jobs:
- Breaking changes
- Security-related updates
- New MCP spec versions
- ANY deleted files (`git status` shows `D`). The fetcher reaps
pages that died upstream. A page disappearing is a real editorial
event and a human should see it, so deletions are never `minor`
no matter which directory they are in. Say what went and why it
looks gone (moved? renamed? product sunset?).
- New entries in `tombstones.json` (pages newly gone upstream), or
entries removed from it (pages that came BACK). Both are
editorial news. `git diff tombstones.json` shows exactly what
changed - read it, do not just note that the file changed.
- ANY change to `discovery.json`. This is the archive finding
something it does not yet hold, and it is the highest-value
thing a run can produce - a source we are missing is worse
than a doc we are missing. `git diff discovery.json` and say
which of these happened:
* a domain appeared under `domains` with status `UNKNOWN` -
upstream is redirecting somewhere we have never probed
* a `review` line appeared - a domain now serves .md and
could be archived today but is not in `sources.json`
* `serves_markdown` flipped false -> true on a `not-fetched`
domain (e.g. academy.claude.com getting .md variants)
* a repo appeared under `github_org_anthropics.not_fetched` -
Anthropic published a new public repo
Name the domain or repo in the title. Adding the source is a
human decision; your job is to make sure it reaches one.

**minor** (WHY: archive freshness, zero human value - PR is self-merged instantly):
- Only github/ mirror or support/ content changed
- Additions and modifications only - never use `minor` if anything was deleted
- Only .metadata.json / timestamps changed
- Typo fixes, formatting tweaks, dead links, minor wording

Expand Down
77 changes: 66 additions & 11 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,67 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Repository Purpose

Comprehensive archive of everything Anthropic publishes for building with
Claude. 2,900+ docs from 11 sources; active sources auto-updated four
Claude. 3,900+ docs from 12 sources; active sources auto-updated four
times daily (blog archive frozen, see Fetcher).

## Fetcher

`scripts/fetcher.py` -- single-file multi-source fetcher.

Sources: code.claude.com, platform.claude.com, modelcontextprotocol.io,
support.claude.com (sitemap + .md), github.com/anthropics/* (10 repos).
anthropic.com blog (engineering/research/news) is a FROZEN archive as of
2026-07: the site is HTML-only and the jina.ai proxy path was removed.
Sources: code.claude.com, platform.claude.com, claude.com/docs,
modelcontextprotocol.io, support.claude.com (sitemap + .md),
github.com/anthropics/* (10 repos). anthropic.com blog
(engineering/research/news) is a FROZEN archive as of 2026-07: the site is
HTML-only and the jina.ai proxy path was removed.

Five rules keep the archive honest, all learned the hard way:

- **Discovery surfaces are incomplete.** Sitemaps and llms.txt undercount what
a site serves, so every full run also refetches what is already on disk.
Without this, 1,560 de-indexed-but-live API pages sat stale for seven weeks.
- **The archive must be able to shrink.** Full runs reap files whose URL is
gone (404/410 or HTML shell). Only markup is deleted automatically; real
markdown that died upstream is reported for a human. A >200-file reap is
refused outright as an upstream outage.
- **A redirect means the content belongs elsewhere.** If a fetch lands on a
different path, the body in hand is the *target's*; writing it back to the
requested path misattributes it. That is how the 471KB
`release-notes/system-prompts.md` briefly became the 3.7KB overview it now
redirects to, after upstream split it into per-model pages.
- **A known failure must not look like a new one.** `tombstones.json` records
every URL confirmed gone, so a page that died once is counted quietly on
later runs instead of re-reported. Only *new* deaths and *resurrections*
are printed loudly, and the success rate is computed over live docs — 123
standing failures pinning it at 96.9% would hide the next real breakage.
- **A discovery only counts once it is a file.** `discovery.json` records what
exists upstream that `sources.json` does not: unknown redirect targets, new
anthropics repos, and whether each domain serves `.md` at all. Printing it
was not enough — the pipeline had logged
`support.claude.com -> academy.claude.com` for weeks into an Actions log
with no reader, and a human found those 725 pages by chasing a dead article.
The decision agent reads `git status`, so a new source now arrives as a diff.

```bash
uv run scripts/fetcher.py # Fetch everything
uv run scripts/fetcher.py --section mcp # Single section
uv run scripts/fetcher.py --tree # Show sources
uv run scripts/fetcher.py --discover # Probe for new sources
uv run scripts/fetcher.py --no-reap # Report dead pages, delete none
```

Sections: `claude-code`, `api`, `platform`, `mcp`, `github`, `support`, `all`
A full run refreshes `discovery.json` on its own; `--discover` is the manual
probe and writes the same file. Neither adds a source — that stays a human
decision. `discovery.json.review` is the actionable list: domains reachable
and serving markdown that nothing fetches yet. Empty is the healthy state.

Sections: `claude-code`, `api`, `platform`, `mcp`, `github`, `support`,
`products`, `all`

Source registry: `sources.json`
Confirmed-dead URLs: `tombstones.json` (self-maintaining; an entry disappears
if the URL answers again)
Sources we do NOT hold: `discovery.json` (self-maintaining; regenerated by
every full run)
Architecture: `REFACTOR.md`

When adding new sections:
Expand Down Expand Up @@ -100,6 +139,20 @@ Use these paths to reference documentation when helping users:
- `content/en/managed-agents/` - Managed agents API
- `content/en/test-and-evaluate/` - Testing and evaluation

#### Product Docs (from claude.com/docs)
- `content/claude/claude-tag/` - Claude Tag / Claude in Slack (65)
- `content/claude/government/` - Government offerings (38)
- `content/claude/connectors/` - Connectors, building + publishing (33)
- `content/claude/claude-science/` - Claude for Science (29)
- `content/claude/third-party/` - Bedrock, Vertex, Foundry desktop setups (28)
- `content/claude/office-agents/` - Claude for Excel, Word, PowerPoint, Outlook (12)
- `content/claude/cowork/` - Claude Cowork (6)

#### Release Notes (from platform.claude.com)
- `content/en/release-notes/system-prompts/` - Claude.ai system prompts, one
page per model (split from a single file upstream on 2026-08-24)
- `content/en/models/` - Per-model overviews and "what's new" pages

#### MCP Protocol (from modelcontextprotocol.io)
- `content/mcp/docs/` - Getting started, build client/server
- `content/mcp/specification/` - Protocol spec versions
Expand All @@ -123,15 +176,16 @@ Use these paths to reference documentation when helping users:
## Repository Structure

```
content/ 2,900+ files
en/docs/claude-code/ Claude Code + Agent SDK (141)
en/api/ API reference (1,500+)
content/ 3,900+ files
en/docs/claude-code/ Claude Code + Agent SDK (198)
en/api/ API reference (1,900+)
en/build-with-claude/ Platform features
en/agents-and-tools/ Tool use, agent skills
mcp/ MCP protocol spec (203)
claude/ Product docs (215)
mcp/ MCP protocol spec (373)
blog/ Engineering, research, news
github/ 10 repos (718 files)
support/ Help articles
support/ Help articles (365)
scripts/
fetcher.py Multi-source fetcher
sources.json Source registry
Expand All @@ -143,3 +197,4 @@ sources.json Source registry
- https://code.claude.com/docs/en/overview
- https://platform.claude.com/docs/en/home
- https://modelcontextprotocol.io
- https://claude.com/docs/llms.txt
85 changes: 70 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Claude Code Docs

> Comprehensive, auto-updating archive of everything Anthropic publishes
> for building with Claude. 2,900+ docs from 11 sources.
> for building with Claude. 3,900+ docs from 12 sources.

[![fetch](https://github.com/thevibeworks/claude-code-docs/actions/workflows/fetch-claude-docs.yml/badge.svg)](https://github.com/thevibeworks/claude-code-docs/actions/workflows/fetch-claude-docs.yml)
[![review](https://github.com/thevibeworks/claude-code-docs/actions/workflows/claude-review.yml/badge.svg)](https://github.com/thevibeworks/claude-code-docs/actions/workflows/claude-review.yml)
[![license](https://img.shields.io/github/license/thevibeworks/claude-code-docs)](LICENSE)
[![docs](https://img.shields.io/badge/docs-2900%2B-blue)](#content)
[![docs](https://img.shields.io/badge/docs-3900%2B-blue)](#content)

Clone this repo and point Claude Code at it. Every doc, tutorial, cookbook,
skill, and engineering post Anthropic has published -- searchable, version-
Expand All @@ -32,14 +32,15 @@ claude "how do I set up hooks in the Agent SDK?"

| Source | Section | Files | What |
|--------|---------|------:|------|
| code.claude.com | `--section claude-code` | 141 | Claude Code + Agent SDK docs |
| platform.claude.com | `--section api` | 1,541 | API reference, build guides |
| modelcontextprotocol.io | `--section mcp` | 203 | MCP spec, SDKs, governance |
| code.claude.com | `--section claude-code` | 198 | Claude Code + Agent SDK docs |
| platform.claude.com | `--section api` | 1,993 | API reference, build guides |
| claude.com/docs | `--section products` | 215 | Claude Tag, Cowork, office agents, connectors |
| modelcontextprotocol.io | `--section mcp` | 373 | MCP spec, SDKs, governance |
| anthropic.com | `--section engineering` | 25 | "Building Effective Agents", context engineering, tool use |
| anthropic.com | `--section research` | 118 | Research papers |
| anthropic.com | `--section news` | ~76 | Model releases, announcements |
| github.com/anthropics | `--section github` | 718 | Cookbooks, skills, plugins, courses, SDK docs |
| support.claude.com | `--section support` | 343 | Help articles |
| support.claude.com | `--section support` | 365 | Help articles |

```
content/
Expand All @@ -48,6 +49,7 @@ content/
en/build-with-claude/ Platform features
en/agents-and-tools/ Tool use, agent skills
en/manage-claude/ Admin, billing, managed agents
claude/ Product docs (Claude Tag, Cowork, office agents)
mcp/ MCP protocol spec + community
blog/
engineering/ Building Effective Agents, context engineering, ...
Expand All @@ -62,12 +64,12 @@ content/
code-action/ GitHub Actions for Claude Code
sdk-python/ Python SDK reference
sdk-typescript/ TypeScript SDK reference
support/ 343 help articles
support/ 365 help articles
```

## Fetching

Auto-updates every 6 hours via GitHub Actions. To fetch manually:
Auto-updates four times daily via GitHub Actions. To fetch manually:

```bash
# Requires: uv (https://docs.astral.sh/uv/)
Expand All @@ -80,22 +82,75 @@ uv run scripts/fetcher.py --discover # Probe domains for new sources
```

GitHub repo fetching needs `GITHUB_TOKEN` or `GH_TOKEN` in the environment.
Blog, research, and support articles use [jina.ai](https://jina.ai) for
HTML-to-markdown conversion (rate-limited at ~10 req/s).
Every fetched source serves a `.md` variant of each page, so nothing is
converted from HTML. That is also why `content/blog/` is frozen: anthropic.com
is HTML-only and the jina.ai proxy path it used was removed in July 2026.

See [`sources.json`](sources.json) for the complete machine-readable source
registry.

## Source Discovery

The fetcher doesn't just download from hardcoded URLs. `--discover` probes
every known Anthropic domain for `robots.txt`, `sitemap.xml`, `llms.txt`,
and `llms-full.txt`, then compares against what we already archive.
The fetcher doesn't just download from hardcoded URLs. It probes every known
Anthropic domain for `robots.txt`, `sitemap.xml`, `llms.txt`, and — the
question that decides everything — whether the domain serves `.md` variants at
all. It also enumerates `github.com/anthropics` and watches the `Location`
header on every redirect it follows.

Known domains: `anthropic.com`, `platform.claude.com`, `code.claude.com`,
`support.claude.com`, `modelcontextprotocol.io`, `claude.ai`, `claude.com`
`support.claude.com`, `modelcontextprotocol.io`, `claude.ai`, `claude.com`,
`academy.claude.com`

**The result is a file, not a log line.** Every full run rewrites
[`discovery.json`](discovery.json) with what exists upstream that
`sources.json` does not. This matters because printing it did not work: the
pipeline had been logging `support.claude.com -> academy.claude.com` four
times a day for weeks, into an Actions log nobody opens, and those 725 pages
were eventually found by a human chasing a dead support article. A discovery
that isn't a diff doesn't reach anyone.

So a new domain, a new `anthropics` repo, or a domain that starts serving
markdown now shows up as a tracked change, gets classified as high-signal, and
opens a PR — the same path a new doc takes. `discovery.json.review` is the
actionable list: reachable, serves markdown, nothing fetches it. Empty is
healthy. Adding a source stays a human decision.

Run `--discover` periodically to catch new sources before they go stale.
```bash
uv run scripts/fetcher.py --discover # manual probe; writes the same file
```

**Sitemaps are treated as incomplete, not authoritative.** Upstream de-indexes
pages it still serves: in July 2026 platform.claude.com dropped every
per-language SDK reference page from both its sitemap and its `llms.txt` while
continuing to edit them, and the archive quietly stopped refreshing 1,560 files
for seven weeks. So every full run also refetches what is already on disk, and
pages that really died are removed by the reaper below rather than by absence
from an index.

### Reaping

A page removed upstream used to live here forever — the fetcher only ever added
or overwrote. Full runs now delete archived files whose URL returns 404/410 or
the site's HTML shell, with two guardrails:

- **Only markup is deleted automatically.** A file holding real markdown whose
URL has died is content Anthropic removed and we may hold the only copy; it is
reported for a human instead of destroyed by a job that merges its own PRs.
- **A mass-deletion circuit breaker.** More than 200 pages vanishing at once
means an upstream outage, not 200 real deletions — nothing is deleted and the
run fails loudly.

`--no-reap` reports what would go without touching anything.

### Tombstones

`tombstones.json` records every URL confirmed gone upstream, with the date and
reason. It exists so that a page which died once does not report as a fresh
failure on every subsequent run — 123 standing failures would pin the success
rate at 96.9% and bury the one new breakage that matters. Later runs count
known deaths quietly and print only what changed: pages newly gone, and pages
that came *back* (whose tombstone is then removed automatically). The success
rate is computed over live docs, so it means something.

## Automation

Expand Down
Loading
Loading