I encountered a really, really annoying issue, just the night before the Chainsail release: after nodes were created and sampling was supposed to start, MPI crashed, saying something to the extent that nodes are not reachable. I investigated further and if I believe correctly, it turns out from the logs that pip install was failing - it wasn't able to connect to files.pythonhosted.org to download the dependencies. I googled around and stumbled upon some SO thread or GitHub issue I can't find anymore saying that there can be DNS-related bugs after running helm upgrades. So while fixing stuff, I regularly ran helm uninstall, terraform destroy etc. and eventually somehow it worked.
I'm sorry for not having kept track of this properly, but we should investigate this and try to reproduce it in a dev deployment.
I encountered a really, really annoying issue, just the night before the Chainsail release: after nodes were created and sampling was supposed to start, MPI crashed, saying something to the extent that nodes are not reachable. I investigated further and if I believe correctly, it turns out from the logs that
pip installwas failing - it wasn't able to connect tofiles.pythonhosted.orgto download the dependencies. I googled around and stumbled upon some SO thread or GitHub issue I can't find anymore saying that there can be DNS-related bugs after runninghelm upgrades. So while fixing stuff, I regularly ranhelm uninstall,terraform destroyetc. and eventually somehow it worked.I'm sorry for not having kept track of this properly, but we should investigate this and try to reproduce it in a dev deployment.