Skip to content

Double check nats is shut down gracefully in docker #2282

New issue
New issue
Open
Bug
Open
Double check nats is shut down gracefully in docker#2282
Bug
Labels
Priority:p2-highHigh priorityType:Bug
@butonic

Description

@butonic
butonic
opened on Feb 5, 2026

I saw a corrupted KV_ids-storage-users stream. opencloud would consume all available CPU and memory. AFAICT this only happens when nats is killed while writing to disk. This is unrelated to the Jepson report and more of our own doing I fear. My guess is that we are still not gracefully shutting down nats when the opencloud process receives the SIGTERM from docker. In kubernetes nats is deployed as a dedicated service so this is not as much of a problem there.

We did look into the graceful shutdown but there are still corner cases.

Maybe there is still an exit(0) somewhere that kills nats while it is writing the stream.

👀

well it is not an exit(0) but the suture runtime does not seem to wait for all services. I have patched the main.go to panic on exit and wait 15 seconds when the command has finished. I also wanted to know when nats is being shut down gracefully:

diff --git a/opencloud/cmd/opencloud/main.go b/opencloud/cmd/opencloud/main.go
index 7968efc1d87..9c2022982fc 100644
--- a/opencloud/cmd/opencloud/main.go
+++ b/opencloud/cmd/opencloud/main.go
@@ -3,13 +3,22 @@ package main
 import (
        "fmt"
        "os"
+       "time"
 
        "github.com/opencloud-eu/opencloud/opencloud/pkg/command"
 )
 
+var osExit = os.Exit
+
 func main() {
+       osExit = func(code int) {
+               panic(fmt.Sprintf("os.Exit(%d) called", code))
+       }
        if err := command.Execute(); err != nil {
                fmt.Fprintln(os.Stderr, err.Error())
                os.Exit(1)
        }
+       fmt.Println("waiting 15 sec")
+       time.Sleep(15 * time.Second)
+       fmt.Println("program exit")
 }
diff --git a/services/nats/pkg/command/server.go b/services/nats/pkg/command/server.go
index 342036167e8..9deec13b3af 100644
--- a/services/nats/pkg/command/server.go
+++ b/services/nats/pkg/command/server.go
@@ -5,6 +5,7 @@ import (
        "crypto/tls"
        "fmt"
        "os/signal"
+       "time"
 
        "github.com/opencloud-eu/opencloud/pkg/config/configlog"
        pkgcrypto "github.com/opencloud-eu/opencloud/pkg/crypto"
@@ -91,7 +92,10 @@ func Server(cfg *config.Config) *cobra.Command {
                        gr.Add(runner.New(cfg.Service.Name+".svc", func() error {
                                return natsServer.ListenAndServe()
                        }, func() {
+                               logger.Info().Msg("gracefully stopping nats server")
                                natsServer.Shutdown()
+                               time.Sleep(5 * time.Second)
+                               logger.Info().Msg("nats server gracefully stopped")
                        }))
 
                        grResults := gr.Run(ctx)

and this is the output when sending SIGTERM:

...
2026-02-05T17:12:56+01:00 INF http server listening at http://127.0.0.1:9140 '' '' host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:124 pkg=rhttp service=frontend
2026-02-05T17:12:56+01:00 WRN core access token not set host.name=wrecker line=/home/jfd/Repositories/reva/internal/http/interceptors/auth/auth.go:248 pkg=rhttp service=frontend traceid=290edc069b0d5925f452194e8f67c4ab
2026-02-05T17:12:56+01:00 WRN http end="05/Feb/2026:17:12:56 +0100" host=127.0.0.1 host.name=wrecker line=/home/jfd/Repositories/reva/internal/http/interceptors/log/log.go:112 method=GET pkg=rhttp proto=HTTP/1.1 service=frontend size=0 start="05/Feb/2026:17:12:56 +0100" status=401 time_ns=111219 traceid=290edc069b0d5925f452194e8f67c4ab uri=/ url=/
2026-02-05T17:13:02+01:00 INF context cancelled line=/home/jfd/Repositories/opencloud/services/frontend/pkg/command/events.go:102 service=frontend
2026-02-05T17:13:02+01:00 INF stopping event handler line=/home/jfd/Repositories/opencloud/services/frontend/pkg/command/server.go:94 service=frontend
2026-02-05T17:13:02+01:00 WRN received signal line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm signal=15
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=app-provider
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=frontend
2026-02-05T17:13:02+01:00 INF clean server shutdown start line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:02+01:00 INF gracefully stopping nats server line=/home/jfd/Repositories/opencloud/services/nats/pkg/command/server.go:95 service=nats
2026-02-05T17:13:02+01:00 INF service "appprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=app-provider
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=app-provider
2026-02-05T17:13:02+01:00 INF context cancelled line=/home/jfd/Repositories/opencloud/services/graph/pkg/service/v0/service.go:581 service=graph
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=ocm
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=storage-system
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=sharing
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-app
2026-02-05T17:13:02+01:00 INF Context canceled. Shutting down event handler line=/home/jfd/Repositories/opencloud/services/storage-users/pkg/event/service.go:51 service=storage-users
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-basic
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=storage-system
2026-02-05T17:13:02+01:00 INF service "ocm" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-app
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=app-registry
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-machine
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9216 network.transport=tcp protocol=http service=storage-system
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=groups
2026-02-05T17:13:02+01:00 INF service "gateway" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-machine
2026-02-05T17:13:02+01:00 INF service "userprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "applicationauth" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-app
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=gateway
2026-02-05T17:13:02+01:00 INF service "app" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "publicstorageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=auth-machine
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=users
2026-02-05T17:13:02+01:00 INF service "appregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=app-registry
2026-02-05T17:13:02+01:00 INF waiting for services to exit line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=http service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9246 network.transport=tcp protocol=grpc service=auth-app
2026-02-05T17:13:02+01:00 INF service "authregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "userprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9242 network.transport=tcp protocol=grpc service=app-registry
2026-02-05T17:13:02+01:00 INF service "" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=users
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=auth-service
2026-02-05T17:13:02+01:00 INF service "sciencemesh" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=ocm
2026-02-05T17:13:02+01:00 INF service "groupprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=groups
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=ocm
2026-02-05T17:13:02+01:00 INF service "permissions" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "publicshareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=sharing
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-service
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9178 network.transport=tcp protocol=grpc service=storage-publiclink
2026-02-05T17:13:02+01:00 INF service "storageregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-system
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=storage-users
2026-02-05T17:13:02+01:00 INF Shutting down server line=/home/jfd/Repositories/opencloud/services/activitylog/pkg/command/server.go:142 server=activitylog service=activitylog transport=http
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=auth-service
2026-02-05T17:13:02+01:00 INF service "archiver" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF starting graceful shutdown line=/home/jfd/Repositories/opencloud/opencloud/pkg/runtime/service/service.go:526 service=opencloud
2026-02-05T17:13:02+01:00 INF service "data" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9280 network.transport=tcp protocol=http service=ocm
2026-02-05T17:13:02+01:00 INF service "storageregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9158 network.transport=tcp protocol=http service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9160 network.transport=tcp protocol=grpc service=groups
2026-02-05T17:13:02+01:00 INF service "usershareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=sharing
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-users
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "gateway" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF service "ocminvitemanager" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "ocmproviderauthorizer" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "authprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=auth-basic
2026-02-05T17:13:02+01:00 INF graceful shutdown done duration=0.065834 line=/home/jfd/Repositories/opencloud/opencloud/pkg/runtime/service/service.go:587 service=opencloud
2026-02-05T17:13:02+01:00 INF service "authregistry" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=gateway
2026-02-05T17:13:02+01:00 INF service "ocs" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "ocmshareprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF service "ocmcore" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
waiting 15 sec
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9142 network.transport=tcp protocol=grpc service=gateway
2026-02-05T17:13:02+01:00 INF service "" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rhttp/rhttp.go:150 pkg=rhttp service=frontend
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9215 network.transport=tcp protocol=grpc service=storage-system
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9146 network.transport=tcp protocol=grpc service=auth-basic
2026-02-05T17:13:02+01:00 INF gracefully stopping reva server host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:108 protocol=grpc service=storage-shares
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9157 network.transport=tcp protocol=grpc service=storage-users
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9150 network.transport=tcp protocol=grpc service=sharing
2026-02-05T17:13:02+01:00 INF service "storageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=ocm
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9140 network.transport=tcp protocol=http service=frontend
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 network.local.address=127.0.0.1:9282 network.transport=tcp protocol=grpc service=ocm
2026-02-05T17:13:02+01:00 INF service "sharesstorageprovider" correctly closed host.name=wrecker line=/home/jfd/Repositories/reva/pkg/rgrpc/rgrpc.go:275 pkg=rgrpc service=storage-shares
2026-02-05T17:13:02+01:00 INF reva server gracefully stopped host.name=wrecker line=/home/jfd/Repositories/reva/cmd/revad/runtime/drivenserver.go:128 protocol=grpc service=storage-shares
2026-02-05T17:13:02+01:00 INF Initiating Shutdown... line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF Initiating JetStream Shutdown... line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF JetStream Shutdown line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:02+01:00 INF Server Exiting.. line=/home/jfd/Repositories/opencloud/services/nats/pkg/logging/nats.go:21 service=nats
2026-02-05T17:13:03+01:00 INF clean server shutdown complete, exiting line=/home/jfd/Repositories/opencloud/pkg/log/logrus_wrapper.go:50 service=idm
2026-02-05T17:13:07+01:00 INF nats server gracefully stopped line=/home/jfd/Repositories/opencloud/services/nats/pkg/command/server.go:98 service=nats
2026-02-05T17:13:10+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/search/pkg/service/event/service.go:216 service=search
2026-02-05T17:13:10+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/postprocessing/pkg/service/service.go:439 service=postprocessing
2026-02-05T17:13:15+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/search/pkg/service/event/service.go:216 service=search
2026-02-05T17:13:15+01:00 ERR failed to get consumer error="context deadline exceeded" line=/home/jfd/Repositories/opencloud/services/postprocessing/pkg/service/service.go:439 service=postprocessing
{"level":"error","pid":330462,"error":"eu.opencloud.api.gateway: service not found","time":"2026-02-05T17:13:17+01:00","caller":"/usr/lib/go/src/runtime/asm_amd64.s:1693","message":"error registering app provider: could not get gateway client"}
program exit

we definitely are not waiting for all services to shut down 😞

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority:p2-highHigh priorityType:Bug

    Type

    Bug

    Projects

    OpenCloud Team Board

    Status

    Prio 2

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions