chore(operations): Kubernetes test procedure

[MOZGIII]

This PR outlines the initial test procedure we agreed upon at #2222.

It only includes minikube tests at CI, and it should be very easy to add EKS or GKE one we set them up.

Closes #2223.

[MOZGIII]

Should I address sharing the vector binary at this PR, or postpone it to some time later?

[binarylogic]

I assume you mean sharing it across all of our CI jobs? I'd postpone it. I'm also not sure that's possible given that we build Vector with different features for different targets, right?

[MOZGIII]

Each of the kubernetes infrastructure tests (1.18, 1.17, etc) currently builds a full release build, then a .deb, and then builds a docker image and pushes it top a repo so that it's accessible to k8s cluster. It's enough to do all those steps once, and reuse the docker image.
Those can be shared not just across the Kubernetes tests, but also among other kinds of tests - for instance - .deb files and also be used at the Test Harness action.

After the docker image is built - we also build the tests (with the kubernetes-specific features) - this is what you mentioned. This part we can't share among non-k8s tests - but across the k8s tests against multiple clusters we can share them too.

Kubernetes tests are designed in a way that we deploy Vector into k8s - this is why we need to build not just the tests, but also the full vector binary (and docker image).

[MOZGIII]

Seems like docker push fails:

+ docker push ***:test-1588264400-w2r8937o-alpine
Successfully tagged ***:test-1588264400-w2r8937o-alpine
The push refers to repository [docker.io/***]
4ed201ced5b4: Preparing
7096a04c7355: Preparing
60c64a536e00: Preparing
1b3ee35aacca: Preparing
denied: requested access to the resource is denied

Login step was ok, so the token must not have access to push into timberiodev/vector-ci-builds.

[binarylogic]

I just added that user to the timberiodev org, it should be fixed now.

[MOZGIII]

It's still failing ☹️

[binarylogic]

Docker Hub's permissions are confusing. It should work now 😄 .

[MOZGIII]

For some time, I was baffled with how we'd merge this without the Kubernetes implementation present. At the same time, I stumble upon the need to have this functionality at my implementation PR. I think I found the solution.

I'll change the k8s related tests as "allow failed" for now (or do something similar), and we merge this. The downside is people will still see those jobs - but they won't affect build success.
When I start actually implementing the tests - I'll reenable them.

An alternative would be to close this PR and for me to take these changes in my impl PR, but that would needlessly bloat that other PR, complicate the reviews and limit the PR atomicity.

[binarylogic]

I’m fine with that