The ref authority. RefStore holds a repo's refs and registered packs; refs change only through compare-and-swap (CASRef), and packs register only after their bytes are durable. Three backends implement it: Postgres (PG), SQLite (SQLite), and S3WAL, which keeps each base repo's whole state in one manifest object advanced by conditional PUT (see the s3wal child package). Around that core the package holds the pack-quota ledger (reservations, usage, retained bytes, adoption claims, restore covers), the LFS object registry and reclamation rows, repo purge phases, absence-cover rows, job scan state, and the IdempotentCAS decorator that recovers ambiguous CAS outcomes.
ZeroSHA,ValidSHA,ErrNotFound,ErrRepoPurged,ErrRefNameNotUTF8[ref.go:ZeroSHA] [purge.go:ErrRepoPurged]Unlockable— whether git can never lock a ref name (a part over 255 bytes, or a last part over 250), shared by the push-delete gate andopenstorage repair-refs[unlockable.go:Unlockable]- Types:
Pack,RefInfo,RefEntry,HydrateSnapshot[ref.go:Pack] - Contracts:
RefStore,Store(addsInitSchema,Close),HydrateSnapshotter,ScanStateStore,RestoreCoverStore[ref.go:RefStore] [scanstate.go:ScanStateStore] Open— dispatchpostgres/postgresql,sqlite, ors3walURLs; never runsInitSchema[ref.go:Open]- Backends:
PG/OpenPG,SQLite/OpenSQLite,S3WAL/NewS3WAL(plusS3WAL.ConditionalStore) [ref.go:OpenPG] [sqlite.go:OpenSQLite] [s3walstore.go:NewS3WAL] - Ref and pack methods:
ListRefs,ListRefsWithMeta,ListRefsAfter,GetRef,CASRef,CASRefGen,RegisterPacks,ReplacePacks,ListPacks,HasPack,PackRegisteredBytes,GetGeneration,HydrateSnapshot,DFConflicts,RefValues,CountRefs[ref.go:CASRef] [dfconflicts.go:DFConflicts] [refvalues.go:RefValues] WithSyncClaimFence— attach a sync-claim holder fence to a CAS context [ref.go:WithSyncClaimFence]WithCASTxHook— attach aCASTxHookto a CAS context: on PG,InTxwrites in the transaction of a swap that changed a row, andCommittedruns once that transaction is known to have committed [ref.go:WithCASTxHook] [ref.go:CASTxHook]IdempotentCAS— decorator that re-reads the ref after an ambiguous or lost CAS [idempotent.go:IdempotentCAS]- Quota ledger:
NewReservationOwner,ReserveNoFloorGen,ReserveDecision,ReservePackBytes,RetirePackAttempts,SweepExpiredPackReservations,IsRetainedPack,ReclaimRetainedPack,QuotaLedgerState,PackQuotaLedgerState,IncomingBytes,Admitted,AddSat[ledger.go:ReservePackBytes] - Adoption and covers:
AdoptionClaimSuffix,AdoptionClaimTTL,TryAdoptionClaim,RenewAdoptionClaim,ReleaseAdoptionClaim,RepairCoverSuffix,ReserveRestoreCover[ledger.go:TryAdoptionClaim] - LFS registry:
MaxLFSObjectsPerRepo,ErrLFSObjectCapExceeded,RegisterLFSObject,RegisterAdoptedLFSObject,ReconcileLegacyLFSObjects,GuardedLFSDelete,BumpLFSAdoptionGen,LFSAdoptionGen,RecordLFSMismatch,LFSMismatchFingerprint,ClearLFSMismatch,CountLFSObjects,HasLFSObject,TouchLFSObject,LFSObjectRegisteredAt[ledger.go:RegisterLFSObject] - LFS reclamation:
LFSObjectRow,LFSReposWithObjects,ListRepoLFSObjects,MarkLFSObjectsUnreferenced,ClearLFSObjectsUnreferenced,RetireUnreferencedLFSObject,SettleLFSRefunds[reclaim.go:RetireUnreferencedLFSObject] - Purge:
FenceRepoPurge,IsRepoPurged,DropRepoRefs,RetireRepoPacks,DropRepoLFSObjects,CountRepoRows,PurgeRepoLedger[purge.go:FenceRepoPurge] - Absence covers:
AbsenceCoverPrefix,AbsenceFoldedOwner,AbsenceCoverRow,AbsenceLFSRow,AbsencePackRow,ListFleetLFSObjects,ListFleetPacks,ListAbsenceCovers,AbsenceCoverHead,RenewAbsenceCovers,LapsedAbsenceCoverExists,CASAbsenceCoverOwner,DeleteAbsenceCover[absence.go:ListAbsenceCovers] - Scan state:
JobScanState,ReapWatermark,DecodeReapWatermark,ErrReapWatermarkStale,ErrLegacyReapUnarmed,LFSRefundBucket,SettledLFSRefunds,ScanJobLFSReclaim,ScanJobAbsence,ScanJobSweep,ScanScope*builders, plusGetJobScanState,PutJobScanState,PutJobScanStateIf,ErrScanStateConflict,DeleteJobScanState,ParkRepoJobScanState,ListJobScanStateScopes,JobScanStateAge,BeginLFSReapEpoch,RecordLFSReapComplete,ArmAbsenceAdmissionFence[scanstate.go:JobScanState] - S3WAL sync-claim fence:
S3WAL.SetSyncClaim,S3WAL.ClearSyncClaim[s3walstore_syncclaim.go:SetSyncClaim] - SQL tables
refs,packs,repo_generations,pack_reservations,pack_retained,pack_usage,lfs_objects,lfs_object_counts,lfs_adoption_gens,lfs_mismatches,purged_repos,job_scan_state[ref.go:initSchemaTx] [sqlite.go:InitSchema] - Child package:
s3wal— conditional-object CAS client, instance lease, v1 manifest schema,MemCASfake, and boot probe used by theS3WALbackend
assertfpanics withref assertion failedon bad arguments [ref.go:assertf]casRefasserts valid SHAs, rejects zero-to-zero, returnsErrRepoPurgedfor a fenced repo before the swap, and reports won only when one row changed [ref.go:casRef] [sqlite.go:casRef]- Fenced CAS requires the upstream's
sync_claim_holderto match the fence holder [sqlite.go:casRef] RegisterPacksasserts non-empty packs and fails for purge-fenced repos;ReplacePacksreturns won=false when the old set is not fully present [ref.go:RegisterPacks] [ref.go:ReplacePacks]- Stored SHAs are asserted valid on every read;
ListRefsAfterorders by byte order [ref.go:ListRefsAfter] IdempotentCASnever recovers a delete, a definitive error, or a clean create conflict; PG SQLSTATE 40003 and 08007 count as ambiguous [idempotent.go:reconcileLoss] [idempotent.go:ambiguousError]- PG
casRefresolves a failed COMMIT of a swap that changed a row withtxid_status, waiting at mostcommitStatusTimeout(5 s): committed is a win, aborted is definitive, and an unknown outcome (the status query failed, or the transaction was still in progress) is ambiguous, whatever pgx's ownSafeToRetrytag says [ref.go:casRef] [ref.go:commitStatus] [idempotent.go:commitUnknownError] ReservePackBytesdenies withFloorStaleon a floor-generation mismatch,KeyCapExceededpast 10,000 active rows,RenewalDegradedwhile the absence fence is armed, and on quota overflow [ledger.go:ReservePackBytes] [ledger_sqlite.go:ReservePackBytes]addSatsaturates at MaxInt64, andadmitteddenies a saturated total [ledger.go:admitted]- LFS registration rolls back above
MaxLFSObjectsPerRepoand refuses purge-fenced repos [ledger.go:registerLFSObjectTx] RetireUnreferencedLFSObjectrefuses withErrReapWatermarkStaleorErrLegacyReapUnarmed, and skips moved generations, live adoption claims, and rows inside grace [reclaim.go:RetireUnreferencedLFSObject] [s3walstore_purge.go:RetireUnreferencedLFSObject]- Refunds settle only under a strictly newer sweep epoch [reclaim.go:SettleLFSRefunds]
DFConflictscounts only strict prefix or descendant refs as conflicts [dfconflicts.go:dfConflictsFromSorted]- S3WAL
mutatemaps exhausted contention toErrCASContention, resolves ambiguous writes by writer nonce, and refuses growing manifests above the hard size cap [s3walstore.go:mutate] [s3walstore.go:checkManifestSize] - S3WAL
InitSchemarefuses to serve without a fresh store Date witness after the probe; with a nil probe (every store built byNewS3WAL) it returns nil without checking [s3walstore.go:InitSchema] [s3walstore.go:NewS3WAL] - S3WAL reservation expiry needs elapsed time plus a skew guard, and leans live when drift or Date checks fail [s3walstore_quota.go:s3walReservationLive]
- S3WAL single-writer scan rows treat a 412 lost race as dual-serve evidence; the shared fleet-global lapse marker treats it as contention and retries with jitter [s3walstore_scanstate.go:scanMutateWith] [s3walstore_scanstate.go:scanMutateShared] [s3walstore_quota.go:s3walRecordLapse]
- S3WAL scan rows stamped more than
s3walScanFutureTolerance(2 s) ahead read as stale; a stamp inside that tolerance clamps to age 0 [s3walstore_scanstate.go:scanRowAge] [s3walstore_scanstate.go:s3walScanFutureTolerance] GuardedLFSDeletenever reports a delete when its S3WAL lease was lost [s3walstore_lfs.go:GuardedLFSDelete]
PG.InitSchemaruns DDL under advisory locks0x5350494458/0x5350494B45, installs generation triggers, and builds indexes concurrently [ref.go:InitSchema]SQLite.InitSchemacreates tables and generation triggers; writes useBEGIN IMMEDIATE[sqlite.go:InitSchema]- Ledger and purge paths take the per-base advisory lock and write the ledger tables listed above [ledger.go:ReservePackBytes] [purge.go:FenceRepoPurge]
GuardedLFSDeleteinvokes the caller's delete callback inside its guard [ledger.go:GuardedLFSDelete]IdempotentCASissues a 5 s point read on a detached context [idempotent.go:reconcileLoss]- PG
casRefreadstxid_current()in each swap that changed a row, and after a failed COMMIT of such a swap pollstxid_statuson a detached context for up to 5 s [ref.go:commitStatus] - S3WAL reads and writes manifests under
_wal/repo/and scan rows under_wal/scan/through conditional GET and PUT, with per-base in-process locks and jittered retries [s3walstore.go:mutate] [s3walstore_scanstate.go:s3walScanPrefix] - S3WAL fleet walks list manifests 512 keys per page [s3walstore_quota.go:s3walEachBase]
- S3WAL increments
metrics.S3WALManifestSizeGuardandmetrics.S3WALContentionExhausted, and logs size and skew warnings [s3walstore.go:countContention] NewReservationOwnerreadscrypto/rand[ledger.go:NewReservationOwner]
- Run:
go test ./internal/ref/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m - Needs Docker: the backend fan-out's
/pgsubtests start Postgres and its/s3walsubtests start SeaweedFS throughs3waltest