Installation Error message (inside corporate firewall)

[vcoisne]

Here is the error message:

error while getting docker-compose container: error downloading https://github.com/docker/compose/releases/download/1.24.0/run.sh: Get https://github.com/docker/compose/releases/download/1.24.0/run.sh: dial tcp: lookup github.com on 171.172.3.251:53: no such host

Comments from @creachadair on this topic:

It makes me realize, though, that we should make it possible to build our stuff without having to pierce the firewall. I think right now they have to import fully-built images, because building would require them to fetch stuff from GH.
I was thinking they could just build everything inside—but our build scripts have a bunch of special cases. At least for Babelfish, and I suspect other teams too.
fixable, but that's the current state and it's probably worth looking into

[carlosms]

In this case the error is happening because the system does not have docker-compose installed, and sourced-ce tries to install a containerized alternative.
The solution is to simply install docker-compose following the instructions here: https://docs.docker.com/compose/install/

But sourced-ce will fail again when it tries to download the docker-compose.yml file from this repository. If opening the firewall is not an option, the user will have to download it manually:

• Somehow download https://raw.githubusercontent.com/src-d/sourced-ce/v0.15.1/docker-compose.yml
• Place it in ~/.sourced/compose-files/v0.15.1/docker-compose.yml
• Enable it with sourced compose set v0.15.1

Of course, without access to github, sourced-ce will not be able to download any organization metadata. Only sourced init local will work, sourced init orgs will be pointless.

[dpordomingo]

@carlosms @vcoisne Do you think we should document something like:

How to use source{d} offline?
Explaining what you wrote here.

I think it could be a recurrent problem in some scenarios (if I'm not wrong, there was no regular access to the Internet in other enterprises we tried)

[carlosms]

I'd strongly prefer not to, to be honest. We use docker images that need to be downloaded, and the main use case is going to download git repositories and metadata...

If there are firewalls in place, our error messages are clear enough to understand which URLs we are trying to access.

[vcoisne]

What about @creachadair's comment. Should we create a separate issue to discuss alternatives for setup behind firewalls ?

[carlosms]

Ok, let's create an issue (in backlog? feature-idea?) if you think this a use case we will need to support.
There are some things we could do in sourced-ce to avoid downloads (like embedding files blobs in the binary and creating necessary files without internet access).
But to be honest, building the whole sourced-ce with dependencies without internet access is something I would not know where to start with. For example to build the sourced-ui image we download the base docker images, python packages, npm packages, debian packages...

[creachadair]

I think for this case, operating completely locally would be OK: The goal here is to test CE inside the corp firewall, where direct GitHub access is not possible—and as I understand it, he only needs to target local repositories. The problem, though, is that he wasn't able to start up the tool even for local repos, because of missing dependencies.

It sounds like fetching more containers explicitly might help in this case. More generally, I feel like we probably should have a way to do completely-local testing after some initial setup. It's fine if the user has to fetch some stuff (or maybe we give them a script to do it), but I think we ought to have some point after which local access should "just work" without further chatting to GH.

[vcoisne]

cc @rpau @eiso What do you think ?

[carlosms]

To be clear, I'm against adding any possible deployment mode by default to the docs, without further discussion on what we really want to support and to what extent.
But I'm not against supporting this use case specifically, if we think it's needed.

As it is, the code will work locally with the differences mentioned above:

• docker-compose is not optional, it's a requirement.
• The docker-compose.yml file has to be placed manually in ~/sourced/..., and enabled with sourced compose set
• The docker images need to be available.

After this is done, either manually or by running sourced init local once with internet access, the user can run init local offline from that point.

(This is off the top of my head, somebody please correct me if I'm wrong!)

Now, the question is: if we want to support it, to what extent? Should this be just an entry in the FAQ, and we consider it a workaround? Or should this be a first-class use case?

If it's the latter, it may have some implications beyond the docs.
For example we may want to add tests for it.
Also, as I commented, there are ways to embed the docker-compose.yml file into the released sourced-ce binary, to avoid the download.

[creachadair]

As I understand it, he was using docker-compose, had fetched the images locally (by separate arrangement), and had set up the docker-compose.yml as you describe. I believe the trouble was not in fetching the images, but that some part of the startup tried to grab a copy of run.sh directly from GitHub.

Unfortunately I don't have all the details of the transaction, but broadly the issue is not that he wants to avoid using Docker at all, but that pulling in data from outside the corp network requires special permissions (and I believe he had to fetch and/or build the images manually via some other side channel and import them). Once all those pieces are in place, it would be good if CE could be started up without additional pulls from remote sources.

Edit: Also, I have in my notes that he tried fetching run.sh separately, but could not figure out where to put it, so that it would work correctly with the other files.

[carlosms]

Once all those pieces are in place, it would be good if CE could be started up without additional pulls from remote sources.

This is already the case if the requirements I described above are met. Most probably their system did not have docker-compose installed. Following the official Docker docs, docker and docker-compose are installed separately.
The run.sh file is an alternative we use when sourced-ce fails to find the docker-compose binary installed. This is why we say in our docs docker-compose is an optional dependency, and why they probably didn't install it in their system.

[creachadair]

The run.sh file is an alternative we use when sourced-ce fails to find the docker-compose binary installed. This is why we say in our docs docker-compose is an optional dependency, and why they probably didn't install it in their system.

Aha, thanks for the clarification. I thought this had come up while running docker-compose. So if he installs the docker-compose tool (and has the other files in place as described), it should be self-contained?

[smacker]

srcd-ce makes calls to github to check new version on every run but shouldn't fail if gh is unavailable, it will just print a warning.

[carlosms]

So if he installs the docker-compose tool (and has the other files in place as described), it should be self-contained?

Yes.

srcd-ce makes calls to github to check new version on every run but shouldn't fail if gh is unavailable, it will just print a warning.

👍 I've checked the code just in case. We don't even print a warning, just ignore it silently.

[dpordomingo]

About the error getting run.sh, which caused this issue, I think it could be more user friendly, and point to proper docs about what was needed, why, and how to solve it.