K8s: implement k8s healthcheck API at Vector for load balancing

[MOZGIII]

We already have a topology-level health check, which is different.

Kubernetes offers a health check system for pods. It's a contract between the Kubernetes load balancing system and the software running is a Pod - in short, when health check for a Pod fails, the Pod is removed from balancing and the traffic is distributed to other endpoints of a Service that the failing Pod is part of.

It is explained in more detail at #2222.

It doesn't seem like directly exposing the topology healthiness status makes sense as the Kubernetes health check.

I suggest we discuss the possible options here.

In essence, it seems like we don't want to ever remove the Vector Pod from balancing in some use cases (generally speaking - it's always the case in a DaemonSet deployment scenario), but in other cases - like in a Serivce deployment strategy (or, generally, with a Deployment resource) - we might want to remove Vector Pods from balancing because in that use case they probably are acting as a set of processes behind a load balancer.

So, things in question are:

• do we report health status differently depending on the deployment strategy?
• how do we detect/specify which mode of health status reporting to use?
• do we even bother, or just report "healthy" unconditionally and rely on internal protocol-level load balancing?

The decision we make here will likely have implications beyond just k8s, since the health check interface and semantics are pretty much standard at this point. Among others, nginx and HAProxy can use healthiness data if exposed via HTTP interface.

[lukesteensen]

It doesn't seem like directly exposing the topology healthiness status makes sense as the Kubernetes health check.

Agreed.

just report "healthy" unconditionally and rely on internal protocol-level load balancing

This would be my choice of starting point. It's not obvious how much meaningful health information we'll have to expose, so it makes sense to start with the simplest possible thing.

[MOZGIII]

This would be my choice of starting point. It's not obvious how much meaningful health information we'll have to expose, so it makes sense to start with the simplest possible thing.

I think it's a good approach.

Should we even care to do anything then? Not exposing health check API is effectively the same as implementing it as always healthy (well, there are some minor differences).

One benefit of implementing an always-ok health check is that we'll have a placeholder for the future, more advanced implementation - but that doesn't seem like what we'd want to do.

I'm perfectly fine with considering this question resolved for now and closing this issue without any further work.

[lukesteensen]

Should we even care to do anything then? Not exposing health check API is effectively the same as implementing it as always healthy (well, there are some minor differences).

If there's no benefit to be gained, we shouldn't do any work 😄

[MOZGIII]

Ok! We've officially covered readiness checks (for now)!