Skip to content

Latest commit

 

History

History
199 lines (168 loc) · 11.7 KB

git-protocol-and-namespaces.md

File metadata and controls

199 lines (168 loc) · 11.7 KB

Git Smart HTTP, Namespaces & LFS

TL;DR: besides the JSON API on api.{org}.code.storage, the backend must serve standard git smart HTTP on the storage host {org}.code.storage, so plain git clone/push/fetch/pull work.

Remote URL scheme

https://t:<jwt>@{org}.code.storage/{repo_id}.git              # persistent namespace
https://t:<jwt>@{org}.code.storage/{repo_id}+ephemeral.git    # ephemeral namespace
https://t:<jwt>@{org}.code.storage/{repo_id}+import.git       # import remote (push only)
  • Basic auth: the username is the fixed literal t; the password is the JWT.
  • JWT requirements match the HTTP API: the repo claim must match the repo; git:read for fetch/clone, git:write for push; refs/ops policies enforced per ref during receive-pack.

Endpoints to implement (git http-protocol, smart only)

GET  /{repo}.git/info/refs?service=git-upload-pack     # advertise refs (fetch/clone)
POST /{repo}.git/git-upload-pack                        # pack negotiation + packfile
GET  /{repo}.git/info/refs?service=git-receive-pack     # advertise refs (push)
POST /{repo}.git/git-receive-pack                        # receive pack, update refs

These are standard git paths; the SDKs only build the base remote URL and hand it to git. Serving via git http-backend or a git library (go-git, libgit2, gitoxide, JGit) against the repo storage is sufficient. The dumb protocol is not required.

The receive path must:

  • Enforce JWT ref policies per ref update (no-push, no-force-push, verify-sig), first match wins, reporting per-ref ng results with a policy message.
  • Refuse a push command that sets a branch under refs/heads/ to an object that is not a commit (a blob, a tree or an annotated tag), with a per-ref ng, as git does. Git checks only after the hooks have run, and the server's ref store has taken the value by then, so the hooks refuse it first. The refusal comes before the ref policy check. A policy denial and this refusal take the same push rules: an --atomic push that carries one lands nothing, and a push whose objects only refused commands need is declined whole, with no pack stored. A branch delete and a new value that the server does not have are not refused here. Other namespaces, such as tags, take any object.
  • Map namespaces: +ephemeral remotes see and update only the ephemeral ref space; persistent remotes must not see ephemeral refs (git namespaces or separate ref stores). open.storage gives the +ephemeral remote its own storage key. The HTTP API, though, stores its ephemeral branches under the base repository's key as refs/namespaces/ephemeral/refs/heads/<name>, so the persistent remote hides refs/namespaces and every ref under it: fetch, clone and ls-remote never list one, a fetch of one by name fails (couldn't find remote ref), and a push that updates or deletes one gets its own ng (deny updating a hidden ref or deny deleting a hidden ref). An --atomic push that names one lands nothing. A push without --atomic still lands its other refs when one of them needs the objects that the push sends. When none does (only the hidden refs need them, or the other refs move to commits the server already has), pre-receive declines the whole push. A push that only deletes refs sends no objects and keeps per-ref results. The +ephemeral remote hides nothing. An import push writes the repository's own refs, so these push rules apply to it too. The persistent remote serves no commit that only a hidden ref reaches, whatever the git config says about wants of unadvertised commits. When the repository's default branch is a hidden ref, it points HEAD at the same name under refs/heads/, which normally does not exist, so it advertises no HEAD (PATCH /repo sets a visible default). One exception: git still resolves a hidden ref that a fetch names in --shallow-exclude, and cuts the history at its commit, so a client with git:read can tell whether such a ref exists. The API's branch list shows the ephemeral branches to the same token.
  • On pushes to persistent refs, emit push webhooks (before/after SHAs per ref) and schedule upstream sync for synced repos.
  • Accept large histories on +import pushes (see Import remote).

Import remote

The +import remote follows upstream's import page:

  • Push only. A clone, fetch or ls-remote through {repo_id}+import.git gets 403 with a text/plain body, and git prints it as remote: reads are disabled for +import remotes; use the repository remote for clone/fetch. Upstream names no status code: 403 is open.storage's choice. Use the normal remote to read. LFS batch requests on the import URL keep working in both directions.
  • An import push writes normal branches and tags. The push goes to the repository's own refs, exactly like a push to {repo_id}.git: the same ref policies, quota, push webhooks and LFS object space. A normal clone shows every branch and tag you pushed, and the REST branch list shows every branch.
  • A synced repository refuses an import push. When the repository has a sync base, the pre-receive hook refuses the whole push before it publishes any pack: import push refused: the repository has a sync base; .... A push to the normal remote still works.
  • Upstream moves an imported repository to cold storage at once. open.storage keeps it on the normal storage path, which is a functional equivalent.

Upgrading from 0.20.0 or earlier

Versions up to 0.20.0 kept +import pushes in a separate ref space, under the storage key <repo_id>+import, that only the +import remote could read. The new version deletes nothing there, but no git path reads it any more, and only a 0.20.0 server can still serve it. Copy that data to the normal remote with the current version before every replica runs the new one:

  1. Stop sending pushes to +import remotes. On a fleet, an import push that reaches a replica that still runs 0.20.0 lands under the old key again.

  2. Check the pack quota. The copy stores the objects a second time under the repository's own key, and both copies count against the pack quota: the old packs keep counting until the repository is deleted. When the repository's stored bytes plus the import's size pass OPENSTORAGE_PACK_QUOTA_BYTES, raise that limit (or set it to 0) for the copy and for the pushes after it.

  3. Copy every ref that you pushed through +import (branches, tags, notes and any other ref):

    git clone --mirror https://t:<jwt>@{org}.code.storage/{repo_id}+import.git import.git
    git -C import.git push https://t:<jwt>@{org}.code.storage/{repo_id}.git 'refs/*:refs/*' '^refs/namespaces/*'

    The ^refs/namespaces/* pattern leaves out refs under refs/namespaces/: after the upgrade the normal remote hides them, and git can no longer list or delete them there (see the end of this section).

    The push refuses a ref that already exists on the normal remote and does not fast-forward, such as a branch with other history or a tag with another value. Push such a ref under a new name, for example refs/heads/main:refs/heads/imported/main or refs/tags/v1:refs/tags/imported/v1.

    Do not copy into a repository that has a sync base: when the link forwards pushes, a copied ref reaches the connected upstream. Create a repository without a sync base and copy into that one instead.

  4. Upgrade every replica, then send import pushes again. A replica's cache can keep a <repo_id>+import.git directory that no git request uses: 0.20.0 made it when it served the old key, and repack and LFS reclamation jobs for the old key can make it again, on any replica that runs them. Check every replica. The cache is rebuildable, so you can delete that directory, but only while the replica is drained and stopped, as for any cache cleanup (cache directory).

If a replica already runs the new version and you did not copy the data, the data is still in storage. Stop every replica of the new version, start 0.20.0 on the same stores, and follow steps 1 to 4. Do this only when no release after 0.20.0, up to and including the version you upgraded to, lists a change to the stored formats in its release notes. Otherwise do not start 0.20.0 on those stores: keep the new version and leave the old data in place.

Before open.storage hid them, a push to the persistent remote could write refs under refs/namespaces/. Those refs stay after an upgrade, and git can no longer list or delete them. Delete every one you do not want before you upgrade: list them with git ls-remote <url> 'refs/namespaces/*' (the pattern also lists refs such as refs/heads/refs/namespaces/x or refs/tags/refs/namespaces/t, which stay visible), and delete each one with git push <url> --delete <ref>. Do not count on the API to delete them later: it deletes only ephemeral branches whose names it accepts. In a fleet, a node that runs an earlier version still lists, serves and takes such refs until it is upgraded.

Git LFS

The storage host must also expose the LFS Batch API at the standard discovery path:

POST https://{org}.code.storage/{repo_id}.git/info/lfs/objects/batch

Same JWT basic-auth remote; no separate LFS server or config. Implementing the batch API plus object upload/download (e.g. via S3-presigned or local storage) covers the documented behavior ("no separate LFS server, no extra configuration").

A batch body that does not decode answers 400 with the LFS error body {"message":"bad batch request: operation has the wrong JSON type"}, which names the field of the wrong JSON type (objects.0.size for a nested one), with bad batch request: request body too large for a body over 4 MiB (a gzip body counts against that limit twice: its compressed bytes and its decoded bytes), or with bad batch request for any other body that does not decode (invalid JSON, or a value of the wrong top-level type). The body is decoded before the request is authenticated, so the message never holds the text of Go's JSON decoder, which names Go types and structs.

A batch that is not rejected as a whole answers 200 with one entry per requested object, in request order, and each entry names its object by the oid. An entry whose oid is not 64 lowercase hex digits carries the per-object error 422 oid must be 64 lowercase hex digits (sha256). It repeats an invalid oid of up to 64 bytes unchanged, and a longer one cut to its first 64 bytes at a character boundary and ended with …, so the answer stays small whatever the oid holds. The 64 bytes count the oid as decoded from the request, before JSON escapes it, with each byte of invalid UTF-8 read as U+FFFD (3 bytes). git-lfs builds an oid from the content's SHA-256, but git lfs push --object-id sends the oid it is given; git-lfs reports the error of an entry without matching its oid to the request.

Interop invariant

Objects and refs written through the HTTP API (commit-pack, merge, tags, notes) and through git push must land in the same object database and ref store. The consumer freely mixes git clone with API-created commits, and integration tests poll GET /repos/branches right after pushes.

Notes on refs via git

  • Notes refs (refs/notes/*) and tags are fetchable and pushable via standard git refspecs (e.g. git fetch origin 'refs/notes/*:refs/notes/*'), so the smart-HTTP layer must not filter them.
  • The API's branch listing shows created_at per branch, so the backend needs per-ref creation metadata beyond what bare git stores (reflog or DB column).