It seems that it can happen that the controller wants to start sampling, but the nodes aren't ready yet. I encountered this in a local (minikube) deployment using a Stan model. The time it takes Stan to compile the model and thus for the user code container to be ready to receive gRPC calls is shorter than the time it takes for the controller to do its stuff and eventually kick off the MPI sampling. The nodes then crash, because they send a gRPC requests to the not-ready-yet user code containers.
We thus need some kind of reliable health check for the user code container which is visible to the controller. For now, I implemented a dirty hack, namely just having the controller wait two minutes before starting to sample (#382). This seems to make sampling from a simple Stan model work locally.
It seems that it can happen that the controller wants to start sampling, but the nodes aren't ready yet. I encountered this in a local (minikube) deployment using a Stan model. The time it takes Stan to compile the model and thus for the user code container to be ready to receive gRPC calls is shorter than the time it takes for the controller to do its stuff and eventually kick off the MPI sampling. The nodes then crash, because they send a gRPC requests to the not-ready-yet user code containers.
We thus need some kind of reliable health check for the user code container which is visible to the controller. For now, I implemented a dirty hack, namely just having the controller wait two minutes before starting to sample (#382). This seems to make sampling from a simple Stan model work locally.