Skip to content

OPERATIONS.md

Latest commit

 

History

History
133 lines (94 loc) · 6.88 KB

OPERATIONS.md

File metadata and controls

133 lines (94 loc) · 6.88 KB
Raw

Operations Guide

This guide covers the operational aspects of running SOP in production, including failover handling, connection management, and backup strategies.

Connection Management

SOP relies on long-lived (pooled) connections to Redis and Cassandra (for incfs).

Note: If running in Standalone Mode (using sop.InMemory cache), Redis is not required, and this section can be ignored.

Note: SOP does not require Redis data persistence (RDB/AOF). Redis is used for ephemeral locking and caching. If Redis restarts, SOP detects the change and recovers safely.

Redis

  • Pool: SOP uses the standard go-redis client.
  • Configuration: Ensure your redis.Options includes:
    • PoolSize: Set according to your concurrency needs (e.g., 10 * CPU cores).
    • MinIdleConns: Keep some connections warm to avoid latency spikes.
    • ReadTimeout / WriteTimeout: Tune these to avoid premature timeouts during heavy load.

Cassandra (incfs)

  • Consistency: SOP typically uses LOCAL_QUORUM for strong consistency.
  • Keyspace: Ensure the keyspace is created with an appropriate replication strategy (e.g., NetworkTopologyStrategy) for your cluster.

Failover Logic

SOP includes sophisticated logic to handle storage failures transparently.

"Failover Qualified" I/O Errors

Not all errors trigger a failover. SOP distinguishes between transient errors (retryable) and permanent hardware/filesystem failures.

Triggers for Failover:

  • syscall.EIO (Input/output error)
  • syscall.EROFS (Read-only file system)
  • syscall.ENOSPC (No space left on device)
  • Specific Linux error codes for media failure (e.g., EUCLEAN).

Behavior:

  1. Detection: When a "Qualified" error occurs during a write.
  2. Switch: The system automatically marks the current storage path as "Passive" and switches to the configured "Active" standby path.
  3. Recovery: Operations continue on the new path. The failed path requires manual intervention or auto-repair (if configured).

Monitoring

  • Logs: Watch for [Failover] tagged logs.
  • Metrics: Monitor the sop_failover_count metric (if you have instrumented your application).

Backup & Restore

Hybrid Backend (incfs)

Backing up a hybrid system requires coordination.

  1. Snapshot Registry (Cassandra):
    • Use nodetool snapshot to capture the state of the Cassandra keyspace.
  2. Snapshot Blob Store (Filesystem):
    • Use filesystem snapshots (e.g., ZFS, LVM, or cloud volume snapshots) to capture the data directory.
  3. Consistency:
    • If possible, pause writes during the snapshot window to ensure the Registry and Blob Store are aligned.
    • If zero-downtime is required, snapshot Cassandra first, then the Filesystem. SOP's Copy-On-Write nature means old blobs (referenced by the older Cassandra snapshot) will still exist on disk, ensuring a consistent point-in-time restore.

Restore Procedure

  1. Stop the application.
  2. Restore the Cassandra keyspace.
  3. Restore the Filesystem data.
  4. Start the application.
  5. Run Integrity Check: Use SOP's internal tools to verify that all Registry entries point to valid blobs.

Data Management Suite (SOP Web UI)

SOP includes an HTTP Server and Web UI that functions as a database management suite. It allows you to:

  • Perform Full CRUD: Create, Read, Update, and Delete records in any B-Tree.
  • Manage Any Database: Works with any SOP store, regardless of the language or data type used.
  • Leverage Rich Indexing: Utilize IndexSpecification to define and search on compound indexes with multiple fields and custom sort orders, giving you RDBMS-like power.
  • Debug & Inspect: Visually verify data integrity and structure.

Note on Architecture: This tool is not a central database server. In SOP's masterless architecture, this UI is another client node. You can run it locally to manage a remote production cluster, or deploy it as a sidecar. It connects directly to the storage layer and respects the same ACID guarantees as other SOP clients.

Running the Management Suite

You can run the tool directly from the source:

# Point to your SOP registry folder
go run ./tools/httpserver -registry /path/to/your/sop/data

Access the UI at http://localhost:8080.

Key Features

  • Store Listing: See all B-Trees in your registry.
  • Rich Search: Search by key prefix or complex multi-field queries (if using IndexSpecification).
  • JSON Editor: View and edit complex value objects as formatted JSON.
  • Bulk Friendly: Pagination and efficient loading for large datasets.
  • Safe Schema Updates: Admins can securely unlock and modify Index/CEL expressions on live stores.

For more details, see the SOP Data Manager Documentation.

Troubleshooting & Best Practices

Cluster reboot procedure

When rebooting an entire cluster running applications that use SOP, follow this order to avoid stale locks and ensure clean recovery:

  1. Gracefully stop all apps that use SOP across the cluster.
  2. Stop the Redis service(s) used by these SOP apps.
  3. Reboot hosts if needed (or proceed directly if not).
  4. Start the Redis service(s) first and verify they are healthy.
  5. Start the apps that use SOP.

Notes:

  • SOP relies on Redis for coordination (locks and recovery bookkeeping). Bringing Redis up before SOP apps prevents unnecessary failovers or stale-lock handling during app startup.
  • If any node was force-killed, SOP’s stale-lock and rollback paths will repair on next write; starting Redis first ensures the required state is available.

Clustered Mode: Data Deletion

When running SOP in Clustered Mode (using Redis + Disk Storage), it is critical to maintain synchronization between the persistent data on disk and the ephemeral locks/cache in Redis.

Recommended Practice:

  • Always use the SOP API (e.g., RemoveBtree, DeleteDatabase, or RemoveStore) to delete data.
  • These methods automatically handle the cleanup of both the physical files on disk and the corresponding entries in Redis (cache keys, locks, registry entries).

Manual Deletion (Development Only): If you must delete the data files manually (e.g., during local development or a hard reset):

  1. Stop all Applications: Ensure no SOP processes (services, CLI, Data Manager) are running.
  2. Delete Files: Remove the store folders/files from the disk.
  3. Flush Redis: You MUST run FLUSHALL on your Redis instance immediately after deleting the files. 
    redis-cli flushall

Why this is necessary: Redis maintains locks and cached metadata with a default timeout (typically 15 minutes). If you delete the files but leave the Redis keys active, any new application instance you start will see "ghost" locks or stale metadata, preventing it from recreating the stores or acquiring locks until the timeout expires.