Skip to content

docs: add docs/fork/ feature guide + README/AGENTS pointers - #61

Closed
Piggidragon wants to merge 4 commits into
GenerelSchwerz:llama/devfrom
Piggidragon:docs/llama-dev-guide
Closed

docs: add docs/fork/ feature guide + README/AGENTS pointers#61
Piggidragon wants to merge 4 commits into
GenerelSchwerz:llama/devfrom
Piggidragon:docs/llama-dev-guide

Conversation

@Piggidragon

@Piggidragon Piggidragon commented Sep 2, 2026

Copy link
Copy Markdown

Establishes docs/fork/ as the home for docs specific to this maintained fork,
with the first page a user-facing guide to everything the branch adds over
upstream master.

Files

  • docs/fork/README.md - the guide (browsing to docs/fork/ shows it).
  • README.md - a 6-line note at the top linking to docs/fork/. llama/dev's
    README is otherwise byte-identical to upstream; nothing fork-specific was in
    it to remove.
  • AGENTS.md - one line under "Useful Resources": docs/fork/ is where fork
    docs go, upstream docs stay in docs/, and the guide tracks user-facing flag
    changes.

What the guide covers

  • New CLI flags - what each does, when to use it, how, and the LLAMA_ARG_*
    env var: --kv-cpu-pinned, --recurrent-state-offload, --kv-gpu-layers,
    --phase-aware-workspace, --live-context-workspace,
    --spec-draft-ubatch-size, --spec-draft-kv-gpu-layers, --spec-mtp-rs-planes.
  • Optimizations that run with no flag - compact causal attention masks, the
    quantization-preserving host KV copy, shared target/MTP workspaces.
  • llama-bench additions and the extra environment variables.
  • Ready-to-use recipes (single GPU large context, hybrid model, two asymmetric GPUs).

Material from branches not yet on llama/dev, marked inline in the guide:

  • ggml-meta : split a host-resident KV cache by head #57 - -as / --attn-split, the -sm tensor + --no-kv-offload
    correctness fix, the --kv-gpu-layers slowest-link-first ordering.
  • sched: pipeline the delivery of a host-resident KV cache #39 - --kv-pipeline-depth / --kv-pipeline-budget, the llama-bench
    -kvcp / -rso / -kvpd / -kvpb switches, GGML_SCHED_TRANSPORT_DEBUG.
  • branch moe-cache-drafting - the CUDA MoE expert cache:
    --moe-expert-cache-size, --moe-expert-cache-l2-pinned-mb,
    --experimental-logs, plus the automatic grouped-decode / prefetch / bias
    residency behaviour and the layer-split-only and speculative caveats.

Notes for reviewers

  • Docs only, no code change. Style follows docs/multi-gpu.md, ASCII only per
    AGENTS.md. All internal links checked.
  • #39 (docs/kv-transport-pipelining.md, docs/repro/) and
    moe-cache-drafting (README feature section, commit eb7f74e) carry their own
    docs. When they rebase/land, those should move into docs/fork/ and the
    README section should collapse to the pointer added here, so there is a single
    source.
  • Draft while ggml-meta : split a host-resident KV cache by head #57, sched: pipeline the delivery of a host-resident KV cache #39 and moe-cache-drafting are unmerged; the flag names and
    defaults for that material match those branches as of now.

AI usage disclosure: drafted with Claude Code, see the commit trailers.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP

Collect the user-facing additions this branch carries over upstream master:
the host-resident KV cache flags (--kv-cpu-pinned, --recurrent-state-offload,
--kv-gpu-layers), the compute workspace flags, the speculative draft-context
flags, and the optimizations that run with no flag. Includes the open PR GenerelSchwerz#57
(--attn-split, -sm tensor + -nkvo fix) and PR GenerelSchwerz#39 (--kv-pipeline-depth /
--kv-pipeline-budget, llama-bench switches), marked as such.

Assisted-by: Claude Sonnet 5
Claude-Session: https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 2, 2026
From the moe-cache-drafting branch: --moe-expert-cache-size,
--moe-expert-cache-l2-pinned-mb, --experimental-logs, the automatic grouped
decode / prefetch / bias residency behaviour, and the layer-split-only and
speculative interactions. Marked as coming from an unmerged branch, like the
PR GenerelSchwerz#57 and GenerelSchwerz#39 material.

Assisted-by: Claude Sonnet 5
Claude-Session: https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP
Short pointer to docs/llama-dev-guide.md instead of spelling the fork's
additions out in the README. Keeps the rest of the README as upstream's.

Assisted-by: Claude Sonnet 5
Claude-Session: https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP
@Piggidragon Piggidragon changed the title docs: user guide for the llama/dev fork features docs: user guide for the llama/dev fork features + README pointer Sep 2, 2026
Give fork-specific docs their own directory so they do not mix with the
upstream docs/ tree. The guide becomes docs/fork/README.md, so browsing to
docs/fork/ shows it. Update the README pointer and add a note to AGENTS.md
that docs/fork/ is the home for fork docs and the guide tracks user-facing
flag changes.

Assisted-by: Claude Sonnet 5
Claude-Session: https://claude.ai/code/session_013Bs926e6SdsxnkrqqKq5HP
@Piggidragon Piggidragon changed the title docs: user guide for the llama/dev fork features + README pointer docs: add docs/fork/ feature guide + README/AGENTS pointers Sep 2, 2026
@Piggidragon Piggidragon closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant