Fix server ip collision

[anyasabo]

Description

Hey friends long time no see. We had an incident a little bit ago where we removed a consul server for maintenance, renamed it, and when we rejoined it it looked okay! But it wasn't. You'll see things like

Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 500 (Raft leader not found in server lookup mapping)

To my knowledge there's a couple options if you do find yourself in this situation:

• restart every server in the cluster so the in memory data is cleared
• be sad because there is no other workaround

I tried to add tests demonstrating how it can happen.

This happened while I was out, but it looks like it's still live so hopefully this helps. Happy to share our incident notes if it's helpful but tried to extract enough to have this make sense.

Happy to also move this to an issue to discuss further, I was just curious how looking at the code would go.

Testing & Reproduction steps

• Remove consul server from cluster
• Change name of said server
• Re-join to cluster
• Make sure this server is the leader somehow
• On a server that has been up (and so has the original hostname of the now-leader in memory), run a query that hits the leader
• Get error

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

• If applicable, I've documented the impact of any changes to security controls.

  Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}