docs: add durable-streams support and update persistence documentation

[jilio]

Summary

• Add 🌍 Remote storage feature to README
• Add Remote Storage section with durable-streams example code
• Add Storage Backends table documenting all available storage options
• Update PERSISTENCE.md for the new offset-native interface
• Add comprehensive examples for SQLite and Durable-Streams stores

Changes

README.md

• Added 🌍 Remote storage feature
• New "Remote Storage" section with durable-streams example
• New "Storage Backends" table showing all available options
• Updated Event Persistence examples to use OffsetOldest

docs/PERSISTENCE.md

• Updated StoredEvent structure to use Offset type instead of Position
• Added "Available Storage Backends" section with table
• Added SQLite store usage example (pointing to stores/sqlite)
• Added Durable-Streams store usage example
• Updated EventStore interface to new Append/Read signatures
• Added SubscriptionStore and EventStoreStreamer interfaces
• Updated PostgreSQL example to new interface
• Replaced inline SQLite example with reference to official package
• Updated all replay examples to use eventbus.OffsetOldest

Test plan

• No code changes - documentation only
• All code examples use correct new interface signatures
• Links to packages are correct

Closes #42

🤖 Generated with Claude Code

[claude]

Pull Request Review

This PR successfully updates the documentation to reflect the new offset-native EventStore interface and adds comprehensive documentation for remote storage support (durable-streams). The documentation is well-written, accurate, and addresses issue #42.

✅ Strengths

1. Comprehensive Documentation Updates

• README.md now clearly highlights remote storage as a feature
• Added detailed Remote Storage section with practical durable-streams example
• New Storage Backends table provides clear overview of available options
• All code examples updated to use eventbus.OffsetOldest (correct new interface)

2. PERSISTENCE.md Improvements

• Accurately documents the transition from Position int64 to Offset type
• New EventStore interface examples match actual implementation (verified against persist.go:28-41)
• PostgreSQL example correctly demonstrates offset conversion using strconv
• SQLite documentation properly references the official stores/sqlite package instead of inline example
• All replay examples consistently use eventbus.OffsetOldest

3. Accuracy & Consistency

• Code examples match actual API signatures (verified against implementation)
• Offset type properly described as opaque string
• Special constants OffsetOldest ("") and OffsetNewest ("$") correctly documented
• EventStore interface documentation matches actual implementation:
  • Append(ctx, *Event) (Offset, error) ✓
  • Read(ctx, from Offset, limit int) ([]*StoredEvent, Offset, error) ✓
• SubscriptionStore and EventStoreStreamer interfaces accurately described as optional

4. Code Quality

• Documentation-only changes (no code changes to break)
• Follows established documentation patterns
• Clear, helpful examples with proper error handling
• Good balance of brevity and detail

📝 Minor Observations

1. Link Accuracy (docs/PERSISTENCE.md:121)
The durable-streams link points to:

[durable-streams](https://github.com/jilio/durable-streams)

However, the actual package comment in stores/durablestream/store.go:4 references:

// durable-streams servers (https://github.com/durable-streams/durable-streams).

Recommendation: Verify which organization owns the durable-streams repository. If it's durable-streams/durable-streams, update the link in PERSISTENCE.md:121 for consistency.

2. Import Path Clarification (docs/PERSISTENCE.md:122, 156)
The documentation shows:

import (
    eventbus "github.com/jilio/ebu"
    "github.com/jilio/ebu/stores/durablestream"
)

This is correct and matches the actual package structure. Well done!

3. Example Completeness
The durable-streams examples include proper error handling and retry configuration, which is excellent for production usage. The SQLite examples appropriately reference the official package.

🎯 Verification Against Issue #42

• ✅ README updated with remote storage feature highlight
• ✅ Remote Storage section added with durable-streams example
• ✅ Storage Backends table added
• ✅ PERSISTENCE.md updated for new offset-native interface
• ✅ Examples added for both SQLite and durable-streams
• ✅ All code examples use correct new interface signatures
• ✅ PostgreSQL example updated to new interface

All acceptance criteria met!

🔒 Security & Performance

• No security concerns (documentation only)
• No performance impact (documentation only)
• Examples demonstrate proper timeout and retry configuration
• Good practices shown (defer store.Close(), error handling)

📊 Test Coverage

No code changes, so no test coverage impact. All documented APIs have corresponding test coverage in the codebase (verified via grep).

✅ Final Recommendation

APPROVE - This PR is ready to merge.

The documentation accurately reflects the implementation, provides clear examples, and successfully addresses all requirements from issue #42. The only minor item is verifying the durable-streams GitHub organization link, which can be addressed in a follow-up if needed or before merge.

Great work maintaining comprehensive, accurate documentation! 🎉

[claude]

Pull Request Review: Documentation Updates for Durable-Streams Support

Overall Assessment

✅ APPROVED - This is a high-quality documentation update that accurately reflects the new offset-native interface and durable-streams support. The changes are well-structured, comprehensive, and follow Go documentation best practices.

---

Strengths

1. Accuracy ✅

• All code examples correctly use the new Append/Read interface signatures
• OffsetOldest constant is consistently used throughout (replacing old position-based 0)
• Interface definitions in documentation match actual implementation in codebase
• References to stores/sqlite and stores/durablestream packages are valid

2. Completeness ✅

• Comprehensive coverage of all storage backends (MemoryStore, SQLite, Durable-Streams)
• Updated PostgreSQL example shows proper offset handling with string conversion
• Added helpful "Available Storage Backends" comparison table
• Migration path clearly documented (Position → Offset)

3. Clarity ✅

• Clear explanation of offset vs position concepts
• Good progression from simple to complex examples
• Helpful comments explaining the new interface design rationale
• Well-organized sections with clear navigation

4. Consistency ✅

• All replay examples updated to use eventbus.OffsetOldest
• Consistent terminology throughout (offset, not position)
• Code style matches existing documentation patterns

---

Detailed Feedback

README.md Changes

Excellent additions:

• ✅ Feature flag for remote storage with durable-streams link
• ✅ New "Remote Storage" section with practical example
• ✅ Storage backends comparison table
• ✅ Updated persistence examples to new interface

Minor observation:

• The durable-streams link points to https://github.com/jilio/durable-streams and the blog post link in PERSISTENCE.md points to Electric's announcement. Consider verifying these links are accurate (appears the actual Electric repo might be different from jilio's fork).

docs/PERSISTENCE.md Changes

Strong updates:

• ✅ Interface migration clearly documented with both old and new signatures
• ✅ SQLite example properly replaced with package reference (avoids duplication)
• ✅ PostgreSQL example shows proper strconv conversion between int64 and Offset
• ✅ Excellent explanation of optional interfaces (SubscriptionStore, EventStoreStreamer)

Particularly good:

// Read returns events starting after the given offset
// Use OffsetOldest to read from the beginning
// Use limit > 0 to limit results, or 0 for no limit
Read(ctx context.Context, from Offset, limit int) ([]*StoredEvent, Offset, error)

The interface documentation is clear about behavior and return values.

---

Code Quality Assessment

Documentation Standards ✅

• Follows Go documentation conventions
• Code examples are runnable and practical
• Error handling shown in examples
• Context usage demonstrated properly

Migration Guidance ✅

The PR effectively guides users through the interface change:

• Old: Position int64, Save(), Load(), GetPosition()
• New: Offset string, Append(), Read()

This is clearly documented with side-by-side examples.

Test Coverage 📊

Per CLAUDE.md requirements, this is a documentation-only change with no code changes, so test coverage requirements are N/A.

---

Potential Issues

1. Durable-Streams URL ⚠️

The README references https://github.com/jilio/durable-streams but the blog post mentions it's developed by Electric. Consider clarifying:

• Is jilio/durable-streams a fork?
• Should the main link point to the official Electric implementation?

2. PostgreSQL Example Complexity 💡

The PostgreSQL example is quite long (150+ lines). Consider:

• Adding a note that this is a simplified example
• Mentioning transaction support considerations
• Including a link to a complete reference implementation

3. Breaking Change Visibility 📢

This represents a breaking API change (Position → Offset). Consider:

• Adding a migration guide section
• Version compatibility notes
• Deprecation timeline if applicable

---

Security Considerations

✅ No security concerns identified:

• No credential handling in examples
• No SQL injection vectors introduced
• Proper context usage for cancellation
• Error handling appropriately demonstrated

---

Performance Considerations

✅ Performance implications well documented:

• SQLite WAL mode mentioned
• Batch size configuration shown
• Streaming interface for memory efficiency (ReadStream)
• Prepared statements noted for SQLite

---

Best Practices Alignment

Checking against CLAUDE.md requirements:

Requirement	Status	Notes
Go fmt	N/A	Documentation only
Tests pass	✅	PR description confirms no code changes
Test coverage	N/A	Documentation only
Go idioms	✅	Examples follow Go best practices
Options pattern	✅	Shown in durablestream examples
Simple implementations	✅	Examples are focused and clear

---

Recommendations

High Priority

1. Verify durable-streams links - Ensure all URLs point to the correct/official repositories

Medium Priority

2. Add migration section - Create a dedicated "Migrating from v1 to v2" section
3. Version compatibility note - Clarify which version introduced the offset-based interface

Low Priority (Nice to have)

4. Add troubleshooting section - Common issues with remote storage
5. Performance benchmarks - Comparison between local and remote stores
6. Example repo link - Link to a complete working example repository

---

Conclusion

This is excellent documentation work that:

• ✅ Accurately reflects the codebase
• ✅ Provides clear migration guidance
• ✅ Includes practical, runnable examples
• ✅ Maintains consistency throughout
• ✅ Follows Go documentation best practices

The PR is ready to merge pending verification of the durable-streams repository links.

Great work on maintaining such thorough and clear documentation! 🎉

---

Closes #42 ✅

[claude]

Pull Request Review: Documentation Updates for Durable-Streams Support

Thank you for this comprehensive documentation update! This is a well-structured PR that adds important documentation for the new offset-based persistence interface and remote storage capabilities. Here's my detailed review:

---

✅ Strengths

1. Excellent Documentation Quality

   • Clear, comprehensive examples throughout
   • Well-organized structure with table of contents
   • Consistent terminology (Offset vs Position) across all documentation
   • Good use of code examples that match the actual implementation

2. Accurate Technical Content

   • All interface definitions in the docs match the actual code in persist.go
   • SQLite and Durable-Streams examples align with the actual store implementations
   • Proper use of OffsetOldest and OffsetNewest constants
   • Correct signatures for Append, Read, SaveOffset, and LoadOffset methods

3. Excellent Feature Documentation

   • Clear explanation of opaque offsets vs. sequential positions
   • Well-documented PostgreSQL example showing migration from old to new interface
   • Good coverage of the three storage backend options (Memory, SQLite, Durable-Streams)
   • Helpful storage comparison table

4. User-Friendly Presentation

   • Features table in README is clear and informative
   • Links to Electric's blog post for durable-streams context
   • Proper attribution to the durable-streams protocol

---

🔍 Areas for Improvement

1. Documentation-Only Changes Claim

Issue: The PR description states "No code changes - documentation only", but this PR is documenting new functionality that was added in previous PRs.

Recommendation: Update the PR description to clarify:

• "Documentation updates to reflect the new offset-based interface (introduced in PR #XX)"
• This helps reviewers understand the context better

2. PostgreSQL Example Completeness

Location: docs/PERSISTENCE.md:402-467

Issues Found:

• Line 406: Missing return type in function signature
• The example shows conversion from old Position-based to new Offset-based interface, but doesn't include the table schema
• No error handling for strconv.ParseInt conversions (line 421, 445, 468)

Recommendation:

// Add schema definition before the example
/*
CREATE TABLE events (
    position BIGSERIAL PRIMARY KEY,
    type TEXT NOT NULL,
    data JSONB NOT NULL,
    timestamp TIMESTAMP WITH TIME ZONE NOT NULL
);

CREATE TABLE subscriptions (
    subscription_id TEXT PRIMARY KEY,
    position BIGINT NOT NULL,
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);
*/

// Fix function signature on line 406:
func (p *PostgresStore) Append(ctx context.Context, event *eventbus.Event) (eventbus.Offset, error) {
    var position int64
    err := p.db.QueryRowContext(ctx, `
        INSERT INTO events (type, data, timestamp)
        VALUES ($1, $2, $3)
        RETURNING position
-   `, event.Type, event.Data, event.Timestamp).Scan(&position)
+    `, event.Type, event.Data, time.Now()).Scan(&position)

    if err != nil {
        return "", err
    }
    return eventbus.Offset(strconv.FormatInt(position, 10)), nil
}

// Add error handling for offset parsing:
func (p *PostgresStore) Read(ctx context.Context, from eventbus.Offset, limit int) ([]*eventbus.StoredEvent, eventbus.Offset, error) {
    var fromPos int64
    if from != eventbus.OffsetOldest {
-        fromPos, _ = strconv.ParseInt(string(from), 10, 64)
+        var err error
+        fromPos, err = strconv.ParseInt(string(from), 10, 64)
+        if err != nil {
+            return nil, from, fmt.Errorf("invalid offset: %w", err)
+        }
    }
    // ... rest of implementation
}

3. Durable-Streams Store Details

Location: docs/PERSISTENCE.md:316-343

Missing Information:

• No mention of the synthetic offset format ("nextOffset/index") documented in the actual implementation
• The store.go file has an important warning about ephemeral synthetic offsets (lines 93-97) that should be documented
• No example of handling connection errors or retries

Recommendation: Add a note about offset handling:

Durable-streams features:
- HTTP-based protocol for distributed systems
- Automatic retries with exponential backoff
- Configurable timeouts
- Server-assigned opaque offsets
+ - **Important:** If the server doesn't include per-event offsets in responses, 
+   synthetic offsets are generated for uniqueness within a Read call. Always use 
+   the `nextOffset` returned by Read() for reliable resumption, not individual 
+   event offsets from a response without embedded offset fields.

4. Minor Consistency Issues

Location: README.md:195

- // Replay events from the beginning
+ // Replay all events from the beginning

Location: README.md:230
The durable-streams example shows creating a store, but doesn't demonstrate the key benefit of automatic resumption after server restart. Consider adding:

// After a restart, subscriptions automatically resume from last offset
eventbus.SubscribeWithReplay(ctx, bus, "order-processor",
    func(event OrderCreatedEvent) {
        processOrder(event)
        // Offset automatically tracked across restarts via remote server
    })

---

🔒 Security Considerations

Good: No security issues found in the documentation examples.

Note: The SQLite store implementation properly validates paths to prevent injection (stores/sqlite/store.go:53-55), which is good practice.

---

🎯 Performance Considerations

Good:

• Documentation correctly mentions WAL mode for SQLite
• Batching with streamBatchSize is documented
• Memory-efficient streaming via ReadStream is mentioned

Suggestion: Consider adding a performance tip in the SQLite section:

### Performance Tips

For high-throughput scenarios:
- Use `WithStreamBatchSize(1000)` for large replays to reduce memory usage
- Enable connection pooling with `db.SetMaxOpenConns()` for concurrent workloads
- Consider using `PRAGMA synchronous = NORMAL` (already default) for better write performance

---

✅ Test Coverage Assessment

Based on the file structure and code review:

• Core persistence interface: Appears well-tested (persist_test.go, persist_benchmark_test.go)
• SQLite store: Has comprehensive tests (stores/sqlite/store_test.go, export_test.go)
• Durable-streams store: Has tests (stores/durablestream/store_test.go, export_test.go)

Note: This is a documentation-only PR, so no new tests are required. The documented interfaces match the tested implementations.

---

📝 Recommendations Summary

Must Fix:

1. Add error handling to PostgreSQL example (strconv.ParseInt calls)
2. Fix PostgreSQL Append method to use time.Now() instead of event.Timestamp
3. Add PostgreSQL schema definition to the example

Should Fix:

1. Add warning about synthetic offsets in durable-streams documentation
2. Update PR description to clarify this documents existing functionality
3. Add schema to PostgreSQL example for completeness

Nice to Have:

1. Add performance tips section
2. Add resumption example for durable-streams
3. Minor wording improvements for consistency

---

🎉 Conclusion

This is a high-quality documentation update that significantly improves the persistence documentation. The updates are accurate, comprehensive, and well-organized. With the minor fixes suggested above (mainly adding error handling to the PostgreSQL example), this will be an excellent resource for users.

The documentation successfully:

• ✅ Explains the new offset-based interface clearly
• ✅ Provides working examples for all three storage backends
• ✅ Maintains consistency with actual implementation
• ✅ Helps users understand migration from old to new interface

Recommendation: Approve with minor changes requested for the PostgreSQL example error handling.

Great work on this documentation! 🚀

---

Reviewed by Claude Sonnet 4.5 - Automated PR Review
Reference: CLAUDE.md code quality standards

[jilio]

Superseded by #48 which includes both the durablestream rewrite for protocol conformance and the documentation updates.