ci: DH-19408: PNAP Integration

[stanbrub]

Wrote an integration with Phoenix NAP (our new bare metal provider).

• Used the PNAP ReST API (Tried terraform, open tofu, and cloud init and could not get working even with support help)
• We now run as a user instead of root
• Upgraded to using Ubuntu 24.04 from 22.04
• Most changes have to do with the change in providers, but some are because of Ubuntu 24.04
• We now have two wait states to check after deploy
  • Wait for the server to be up and ssh accessible
  • Wait for APT to be unlocked (done updating) after first login
• We now turn off all auto-updaters for both auto-provisioned and always-on servers
• We now turn off ASLR in an attempt to minimize benchmark run variability
• Purging expired servers is now a function performed by us (Equinix had a way to set expiration on server create)
  • We expire ephemeral servers every 24 hours that have been alive at least 24 hours
  • Given that this only affects auto-provisioned servers, and those servers are used about 10% of the time compared to the nightly always-on server of 9hrs, and the rate at which Github Actions outages would affect cleanup (always()) routines was ~27 incidents last year, the probability of the purge-expired function having an impact is 0.28%. (Gotta love Copilot)
  • It's still necessary to have the fallback
• Updated the secrets document (The public one is here, and the actual keys are in the private vault)

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 20 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cpwright]

A test -z does seem like a reasonable safety check here.