Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
name: Bug report
about: Report a bug in mmapforge
title: ''
labels: bug
assignees: ''
---

## Description

A clear description of what the bug is.

## Steps to reproduce

1.
2.
3.

## Expected behavior

What you expected to happen.

## Actual behavior

What actually happened. Include error messages or stack traces if available.

## Environment

- OS: (e.g. macOS 15.3, Ubuntu 24.04)
- Go version: (e.g. 1.24)
- mmapforge version/commit:
23 changes: 23 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
name: Feature request
about: Suggest a feature or improvement
title: ''
labels: enhancement
assignees: ''
---

## Problem

What problem does this solve? What use case does it enable?

## Proposed solution

Describe how you'd like this to work.

## Alternatives considered

Any other approaches you've thought about.

## Additional context

Benchmarks, links, or anything else relevant.
34 changes: 34 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Changelog

## v0.1.0 (Unreleased)

Initial release.

### Features

- Zero-copy, mmap-backed typed record store
- Code generator (`mmapforge`) parses structs with `mmap` tags and generates fully typed stores
- Read/write support for all Go primitive types, strings, and byte slices
- Per-record seqlock protocol for lock-free concurrent reads
- Automatic file growth with stable base address (MAP_FIXED remapping)
- Header with magic bytes, format version, and schema hash validation
- Layout engine with proper alignment and deterministic schema hashing
- Crash recovery: stuck seqlock counters auto-reset on OpenStore

### Performance

- ~2ns per field read, ~2ns per field write (Apple M4 Pro)
- Zero heap allocations on reads
- 179x faster than os.File ReadAt, 337x faster than os.File WriteAt

### Testing

- 100% code coverage across all packages
- Race detector clean
- Fuzz testing on header parser (21M+ executions, zero crashes)

### Documentation

- README with install, usage, benchmarks, and crash safety docs
- Godoc comments on all exported types and functions
- CONTRIBUTING.md
72 changes: 72 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Contributing to mmapforge

## Prerequisites

- Go 1.24+
- macOS or Linux (unix build tag required)

## Getting started

```bash
git clone https://github.com/CreditWorthy/mmapforge.git
cd mmapforge
go test ./...
```

## Running tests

```bash
# all tests with race detector
go test -race -count=1 ./...

# with coverage
go test -coverprofile=cover.out ./...
go tool cover -html=cover.out

# fuzz the header parser
go test -fuzz=FuzzDecodeHeader -fuzztime=30s

# benchmarks
go test ./... -bench=. -benchmem
```

## Code generation

The `mmapforge` binary is the code generator. To regenerate the example store:

```bash
go build -o mmapforge ./cmd/mmapforge
go generate ./example/...
```

## Project structure

```
mmapforge/
common.go - shared constants (Magic, HeaderSize, etc.)
errors.go - sentinel errors
header.go - binary header encode/decode
layout.go - field layout engine and schema hashing
mmap_unix.go - memory-mapped Region (Map, Grow, Close, Sync)
store.go - Store (CreateStore, OpenStore, Append, grow)
store_seq.go - per-record seqlock protocol
store_read.go - typed field readers (ReadUint64, ReadString, etc.)
store_write.go - typed field writers (WriteUint64, WriteString, etc.)
cmd/mmapforge/ - code generator CLI
internal/codegen/ - struct parser and code generator
example/ - generated MarketCap store with tests and benchmarks
```

## Style

- Run `golangci-lint run` before submitting. The repo has a `.golangci.yml` config.
- Every exported type and function needs a godoc comment.
- Keep 100% test coverage. If you add code, add tests.
- Use the mockable function vars (`mmapFixedFunc`, `madviseFunc`, etc.) for testing syscall paths.

## Submitting a PR

1. Fork the repo and create a branch from `main`.
2. Make your changes. Add tests.
3. Run `go test -race ./...` and `golangci-lint run`.
4. Open a PR with a clear description of what changed and why.
5 changes: 5 additions & 0 deletions common.go
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
package mmapforge

// Magic is the 4-byte file signature written at the start of every mmapforge file.
var Magic = [4]byte{'M', 'M', 'F', 'G'}

// MagicString is the string form of Magic for display purposes.
const MagicString = "MMFG"

// Version is the current binary format version.
const Version uint32 = 1

// HeaderSize is the fixed size of the file header in bytes.
const HeaderSize = 64

// StoreReserveVA is the default virtual address reservation for Store files (1 GB).
const StoreReserveVA = 1 << 30
1 change: 1 addition & 0 deletions errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package mmapforge

import "errors"

// Sentinel errors returned by Store and Region operations.
var (
ErrSchemaMismatch = errors.New("mmapforge: schema hash mismatch")
ErrOutOfBounds = errors.New("mmapforge: index out of bounds")
Expand Down
1 change: 1 addition & 0 deletions header.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
)

// Header is the 64-byte metadata block at the start of every mmapforge file.
type Header struct {
Magic [4]byte
FormatVersion uint32
Expand Down
11 changes: 11 additions & 0 deletions store_read.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"unsafe"
)

// ReadBool reads a bool from record idx at the given byte offset.
func (s *Store) ReadBool(idx int, offset uint32) (bool, error) {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -17,6 +18,7 @@ func (s *Store) ReadBool(idx int, offset uint32) (bool, error) {
return b[0] == 1, nil
}

// ReadInt8 reads an int8 from record idx at the given byte offset.
func (s *Store) ReadInt8(idx int, offset uint32) (int8, error) {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -25,6 +27,7 @@ func (s *Store) ReadInt8(idx int, offset uint32) (int8, error) {
return int8(b[0]), nil
}

// ReadUint8 reads a uint8 from record idx at the given byte offset.
func (s *Store) ReadUint8(idx int, offset uint32) (uint8, error) {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -33,6 +36,7 @@ func (s *Store) ReadUint8(idx int, offset uint32) (uint8, error) {
return b[0], nil
}

// ReadInt16 reads an int16 from record idx at the given byte offset.
func (s *Store) ReadInt16(idx int, offset uint32) (int16, error) {
b, err := s.fieldSlice(idx, offset, 2)
if err != nil {
Expand All @@ -41,6 +45,7 @@ func (s *Store) ReadInt16(idx int, offset uint32) (int16, error) {
return *(*int16)(unsafe.Pointer(&b[0])), nil
}

// ReadUint16 reads a uint16 from record idx at the given byte offset.
func (s *Store) ReadUint16(idx int, offset uint32) (uint16, error) {
b, err := s.fieldSlice(idx, offset, 2)
if err != nil {
Expand All @@ -49,6 +54,7 @@ func (s *Store) ReadUint16(idx int, offset uint32) (uint16, error) {
return binary.LittleEndian.Uint16(b), nil
}

// ReadInt32 reads an int32 from record idx at the given byte offset.
func (s *Store) ReadInt32(idx int, offset uint32) (int32, error) {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -57,6 +63,7 @@ func (s *Store) ReadInt32(idx int, offset uint32) (int32, error) {
return *(*int32)(unsafe.Pointer(&b[0])), nil
}

// ReadUint32 reads a uint32 from record idx at the given byte offset.
func (s *Store) ReadUint32(idx int, offset uint32) (uint32, error) {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -65,6 +72,7 @@ func (s *Store) ReadUint32(idx int, offset uint32) (uint32, error) {
return binary.LittleEndian.Uint32(b), nil
}

// ReadInt64 reads an int64 from record idx at the given byte offset.
func (s *Store) ReadInt64(idx int, offset uint32) (int64, error) {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand All @@ -73,6 +81,7 @@ func (s *Store) ReadInt64(idx int, offset uint32) (int64, error) {
return *(*int64)(unsafe.Pointer(&b[0])), nil
}

// ReadUint64 reads a uint64 from record idx at the given byte offset.
func (s *Store) ReadUint64(idx int, offset uint32) (uint64, error) {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand All @@ -81,6 +90,7 @@ func (s *Store) ReadUint64(idx int, offset uint32) (uint64, error) {
return binary.LittleEndian.Uint64(b), nil
}

// ReadFloat32 reads a float32 from record idx at the given byte offset.
func (s *Store) ReadFloat32(idx int, offset uint32) (float32, error) {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -89,6 +99,7 @@ func (s *Store) ReadFloat32(idx int, offset uint32) (float32, error) {
return math.Float32frombits(binary.LittleEndian.Uint32(b)), nil
}

// ReadFloat64 reads a float64 from record idx at the given byte offset.
func (s *Store) ReadFloat64(idx int, offset uint32) (float64, error) {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand Down
11 changes: 11 additions & 0 deletions store_write.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"unsafe"
)

// WriteBool writes a bool to record idx at the given byte offset.
func (s *Store) WriteBool(idx int, offset uint32, val bool) error {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -22,6 +23,7 @@ func (s *Store) WriteBool(idx int, offset uint32, val bool) error {
return nil
}

// WriteInt8 writes an int8 to record idx at the given byte offset.
func (s *Store) WriteInt8(idx int, offset uint32, val int8) error {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -31,6 +33,7 @@ func (s *Store) WriteInt8(idx int, offset uint32, val int8) error {
return nil
}

// WriteUint8 writes a uint8 to record idx at the given byte offset.
func (s *Store) WriteUint8(idx int, offset uint32, val uint8) error {
b, err := s.fieldSlice(idx, offset, 1)
if err != nil {
Expand All @@ -40,6 +43,7 @@ func (s *Store) WriteUint8(idx int, offset uint32, val uint8) error {
return nil
}

// WriteInt16 writes an int16 to record idx at the given byte offset.
func (s *Store) WriteInt16(idx int, offset uint32, val int16) error {
b, err := s.fieldSlice(idx, offset, 2)
if err != nil {
Expand All @@ -49,6 +53,7 @@ func (s *Store) WriteInt16(idx int, offset uint32, val int16) error {
return nil
}

// WriteUint16 writes a uint16 to record idx at the given byte offset.
func (s *Store) WriteUint16(idx int, offset uint32, val uint16) error {
b, err := s.fieldSlice(idx, offset, 2)
if err != nil {
Expand All @@ -58,6 +63,7 @@ func (s *Store) WriteUint16(idx int, offset uint32, val uint16) error {
return nil
}

// WriteInt32 writes an int32 to record idx at the given byte offset.
func (s *Store) WriteInt32(idx int, offset uint32, val int32) error {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -67,6 +73,7 @@ func (s *Store) WriteInt32(idx int, offset uint32, val int32) error {
return nil
}

// WriteUint32 writes a uint32 to record idx at the given byte offset.
func (s *Store) WriteUint32(idx int, offset uint32, val uint32) error {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -76,6 +83,7 @@ func (s *Store) WriteUint32(idx int, offset uint32, val uint32) error {
return nil
}

// WriteInt64 writes an int64 to record idx at the given byte offset.
func (s *Store) WriteInt64(idx int, offset uint32, val int64) error {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand All @@ -85,6 +93,7 @@ func (s *Store) WriteInt64(idx int, offset uint32, val int64) error {
return nil
}

// WriteUint64 writes a uint64 to record idx at the given byte offset.
func (s *Store) WriteUint64(idx int, offset uint32, val uint64) error {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand All @@ -94,6 +103,7 @@ func (s *Store) WriteUint64(idx int, offset uint32, val uint64) error {
return nil
}

// WriteFloat32 writes a float32 to record idx at the given byte offset.
func (s *Store) WriteFloat32(idx int, offset uint32, val float32) error {
b, err := s.fieldSlice(idx, offset, 4)
if err != nil {
Expand All @@ -103,6 +113,7 @@ func (s *Store) WriteFloat32(idx int, offset uint32, val float32) error {
return nil
}

// WriteFloat64 writes a float64 to record idx at the given byte offset.
func (s *Store) WriteFloat64(idx int, offset uint32, val float64) error {
b, err := s.fieldSlice(idx, offset, 8)
if err != nil {
Expand Down
Loading