Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,8 @@ and `docker.io/blockossreleases/spectre`.

## SPECTRE server

`spectre serve` collects the heartbeats that mirrors send. Each heartbeat
carries the mirror's latest state: its candidate revision, SPECTRE version,
uptime, and quarantines. The server also records each quarantine as a
`spectre serve` collects the states that mirrors stream to it: each mirror's
candidate revision, SPECTRE version, uptime, and quarantines. The server also records each quarantine as a
divergence that outlives the mirror run. It listens on `127.0.0.1:50070` by
default and keeps everything in memory, so a restart loses it. To keep them in Redis instead, pass a Redis URL to `--storage`, such as
`--storage=redis://localhost:6379/0`. For a Redis Cluster, use the
Expand All @@ -86,14 +85,20 @@ pass the socket as `--server-socket` and the server's mesh address as
`--server-url`. Every call dials the socket, and the mesh routes it by the URL's
host, such as `--server-url=http://spectre.mesh --server-socket=@egress.sock`.

A mirror sends a heartbeat when it starts, every `--heartbeat-interval` (30s by
default), and as soon as it quarantines. A failed heartbeat is only logged.
A mirror sends everything to the server over one stream, which begins with its
whole state. It then sends each quarantine as it begins, and a keepalive every
`--server-keepalive-interval` (30s by default) when nothing else is sent.
Keep the interval below the idle timeout of any proxy in front of the server;
Caddy, for one, ends a request stream after 60s without data. When the stream
fails, the mirror logs it and reconnects with backoff, from 1s up to 1m. It also
replaces the stream after `--server-stream-max-age` (10m by default, with
jitter), so streams spread across server replicas.

List the reporting mirrors and recorded divergences with
`spectre status --server-url=URL`. Filter with `--service`,
`--environment`, `--revision`, or `--node`, and print JSON with `--json`. A
mirror stays listed until it misses heartbeats for the server's `--state-ttl`
(90s by default), so keep that above the heartbeat interval. A mirror stays
mirror stays listed until it sends nothing for the server's `--state-ttl`
(90s by default), so keep that above the keepalive interval. A mirror stays
quarantined until it restarts.

The server keeps the latest divergence of each direction of a service
Expand Down
6 changes: 4 additions & 2 deletions cli_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ func expectedReporting() reporter.Config {
Service: "users",
Environment: "production",
Revision: "abc123",
HeartbeatInterval: 30 * time.Second,
KeepaliveInterval: 30 * time.Second,
StreamMaxAge: 10 * time.Minute,
}
}

Expand Down Expand Up @@ -89,7 +90,8 @@ func TestSuperviseCommandReadsFlagsFromEnvironment(t *testing.T) {
Service: "invoices",
Environment: "production-us-west-2",
Revision: "3f9c2a1",
HeartbeatInterval: 30 * time.Second,
KeepaliveInterval: 30 * time.Second,
StreamMaxAge: 10 * time.Minute,
}
assert.Equal(t, expected, command.Supervise.Supervisor.Server)
}
Expand Down
2 changes: 1 addition & 1 deletion demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ Base your image on the same pattern as `demo/Dockerfile`:
(`--scripts-dir`), TypeScript schema (`--schema-dir`), and protobuf
descriptors (`--descriptors-dir`).
3. Run the supervisor with your two commands, pointed at a SPECTRE server
(`spectre serve`) that it sends heartbeats to.
(`spectre serve`) that it streams its state to.

The supervisor runs each service command through `/bin/sh -c` and exports two
environment variables to it:
Expand Down
2 changes: 1 addition & 1 deletion demo/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ spectre-sample-forecasts \
--listen="unix:$sockets/forecasts.sock" \
--weather=/spectre/testdata/weather.json &

# Mirrors send heartbeats to the SPECTRE server, so run one alongside.
# Mirrors stream their states to the SPECTRE server, so run one alongside.
spectre serve --listen=127.0.0.1:50070 &

service="spectre-sample --listen=\$SPECTRE_LISTEN --forecasts=\$SPECTRE_EGRESS --data=/spectre/testdata/users.json --weather=/spectre/testdata/weather.json"
Expand Down
2 changes: 1 addition & 1 deletion internal/cli/commandline.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ type CommandLine struct {
Version kong.VersionFlag `env:"-" help:"Print the version and exit."`

Supervise superviseCommand `cmd:"" help:"Supervise a candidate and a reference behind the embedded proxies, splitting resources between them."`
Serve serveCommand `cmd:"" help:"Run the SPECTRE server, which mirrors send their heartbeats to."`
Serve serveCommand `cmd:"" help:"Run the SPECTRE server, which mirrors stream their states to."`
Gen genCommand `cmd:"" help:"Generate TypeScript schema declarations and the script API."`
Status statusCommand `cmd:"" help:"List the mirrors reporting to the SPECTRE server and the divergences they reported, or clear divergences."`
State stateCommand `cmd:"" help:"Describe a service's migration state."`
Expand Down
3 changes: 2 additions & 1 deletion internal/cli/status_internal_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,9 @@ func TestStatusCommands(t *testing.T) {
Uptime: 12*time.Minute + 300*time.Millisecond,
}
for _, state := range []model.State{quarantined, healthy} {
assert.NoError(t, spectre.Update(t.Context(), state))
assert.NoError(t, store.Put(t.Context(), state, time.Now()))
}
assert.NoError(t, store.Record(t.Context(), model.Divergence{Service: "users", Environment: "production", Revision: "def456", Direction: model.Ingress, Quarantine: quarantined.Quarantines[model.Ingress]}))

var text strings.Builder
list := statusListCommand{}
Expand Down
45 changes: 45 additions & 0 deletions internal/httpserver/middleware/endearly/handler.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Package endearly lets a server end a client stream before the client does.
package endearly

import (
"io"
"net/http"
"time"

"github.com/alecthomas/errors"
)

// New lets next end a client stream before the client does. Otherwise an
// HTTP/1 server reads the rest of the request before it replies.
func New(next http.Handler) http.Handler {
return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
request.Body = newUnendedBody(request.Body, http.NewResponseController(writer))
next.ServeHTTP(writer, request)
})
}

// unendedBody is a request body that the client may not have ended.
type unendedBody struct {
io.ReadCloser
controller *http.ResponseController
ended bool
}

func newUnendedBody(body io.ReadCloser, controller *http.ResponseController) *unendedBody {
return &unendedBody{ReadCloser: body, controller: controller}
}

func (b *unendedBody) Read(data []byte) (int, error) {
n, err := b.ReadCloser.Read(data)
b.ended = b.ended || errors.Is(err, io.EOF)
return n, err //nolint:wrapcheck // Readers must return io.EOF itself.
}

// Close stops reading a body the client has not ended, so the server replies
// at once and then closes the connection, which tells the client.
func (b *unendedBody) Close() error {
if !b.ended {
_ = b.controller.SetReadDeadline(time.Now()) //nolint:errcheck // Without a deadline, closing only takes longer.
}
return errors.WithStack(b.ReadCloser.Close())
}
18 changes: 16 additions & 2 deletions internal/model/state.go
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
package model

import "time"
import (
"maps"
"time"
)

// State is what the mirror of a service environment runs and holds, as its
// latest heartbeat reported it.
// latest update reported it.
type State struct {
Service string `json:"service"`
Environment string `json:"environment"`
Expand All @@ -16,3 +19,14 @@ type State struct {
// Quarantines holds the quarantine of each direction whose proxy has one.
Quarantines map[Direction]Quarantine `json:"quarantines,omitempty"`
}

func (State) update() {}

// Quarantined returns the state with the quarantine of the proxy in direction.
func (s State) Quarantined(direction Direction, quarantine Quarantine) State {
quarantines := make(map[Direction]Quarantine, len(s.Quarantines)+1)
maps.Copy(quarantines, s.Quarantines)
quarantines[direction] = quarantine
s.Quarantines = quarantines
return s
}
15 changes: 15 additions & 0 deletions internal/model/update.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
package model

// Update is one message of the stream a mirror sends the SPECTRE server. Every
// stream begins with the mirror's whole State.
//
//sumtype:decl
type Update interface{ update() }

// QuarantineUpdate adds the quarantine of the proxy in Direction to the state.
type QuarantineUpdate struct {
Direction Direction
Quarantine Quarantine
}

func (QuarantineUpdate) update() {}
46 changes: 35 additions & 11 deletions internal/server/client/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import (
"context"
"net"
"net/http"
"time"

"connectrpc.com/connect/v2"
"connectrpc.com/connect/v2/connecthttp"
Expand All @@ -17,9 +16,6 @@ import (
"github.com/block/spectre/spectrepb/spectrepbconnect"
)

// heartbeatTimeout bounds one heartbeat, so a slow server never delays the next.
const heartbeatTimeout = 10 * time.Second

// Config locates the SPECTRE server.
type Config struct {
URL string `name:"server-url" default:"${server_url}" placeholder:"URL" help:"Base URL of the SPECTRE server."`
Expand Down Expand Up @@ -56,13 +52,14 @@ func New(config Config) *Client {
return &Client{rpc: spectrepbconnect.NewSpectreServiceClient(connect.NewClient(connecthttp.NewTransport(httpClient, config.URL)))}
}

// Update sends a mirror's latest state. It does not retry, because the
// next heartbeat replaces it.
func (c *Client) Update(ctx context.Context, state model.State) error {
ctx, cancel := context.WithTimeout(ctx, heartbeatTimeout)
defer cancel()
_, err := c.rpc.Update(ctx, &pb.UpdateRequest{State: wire.StateMessage(state)})
return errors.Wrap(err, "send heartbeat")
// OpenUpdates opens a stream of a mirror's updates, which lasts until ctx ends
// or the stream is closed.
func (c *Client) OpenUpdates(ctx context.Context) (*Updates, error) {
stream, err := c.rpc.Update(ctx)
if err != nil {
return nil, errors.Wrap(err, "open update stream")
}
return newUpdates(stream), nil
}

// ListDivergences returns the states of the reporting mirrors and the
Expand All @@ -82,3 +79,30 @@ func (c *Client) ClearDivergences(ctx context.Context, service, environment stri
_, err := c.rpc.ClearDivergences(ctx, request)
return errors.Wrap(err, "clear divergences")
}

// Updates streams changes to a mirror's state to the SPECTRE server.
type Updates struct {
stream spectrepbconnect.SpectreServiceUpdateClientStream
}

func newUpdates(stream spectrepbconnect.SpectreServiceUpdateClientStream) *Updates {
return &Updates{stream: stream}
}

// Send sends update. It fails once the server has ended the stream, and
// CloseAndReceive then returns why it did.
func (u *Updates) Send(update model.Update) error {
return errors.Wrap(u.stream.Send(wire.UpdateMessage(update)), "send update")
}

// Keepalive shows the server that a quiet mirror is still connected.
func (u *Updates) Keepalive() error {
keepalive := &pb.UpdateRequest{Update: &pb.UpdateRequest_Keepalive{Keepalive: &pb.Keepalive{}}}
return errors.Wrap(u.stream.Send(keepalive), "send keepalive")
}

// CloseAndReceive ends the stream and returns the error the server ended it with.
func (u *Updates) CloseAndReceive() error {
_, err := u.stream.CloseAndReceive()
return errors.Wrap(err, "close update stream")
}
6 changes: 3 additions & 3 deletions internal/server/config.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Package server is the SPECTRE server, which collects the heartbeats that
// mirrors send. It keeps their states and divergences in memory or in Redis.
// Package server is the SPECTRE server, which collects the states that mirrors
// stream. It keeps their states and divergences in memory or in Redis.
package server

import (
Expand All @@ -12,7 +12,7 @@ import (
type ServeConfig struct {
Listen string `default:"127.0.0.1:50070" help:"Address for the SPECTRE server: host:port, or unix:<path|@abstract>."`
Storage storage.Config `embed:""`
StateTTL time.Duration `default:"90s" help:"How long a mirror's state is listed after its last heartbeat. Keep it above the heartbeat interval."`
StateTTL time.Duration `default:"90s" help:"How long a mirror's state is listed after its last update. Keep it above the mirrors' keepalive interval."`
}

// ClearConfig names the service environment whose divergences to clear.
Expand Down
Loading
Loading