diff --git a/modules/flashbox/flashbox-l1/readme.md b/modules/flashbox/flashbox-l1/readme.md index aa8cb5c9..655126a4 100644 --- a/modules/flashbox/flashbox-l1/readme.md +++ b/modules/flashbox/flashbox-l1/readme.md @@ -16,13 +16,15 @@ Using Intel TDX, Flashbots has built a way for searchers to trustlessly backrun - [Developer Notes](#developer-notes) - [Security](#security) +See also: [Flashbox documentation site](https://flashbox.flashbots.dev) + TDX Mental Model ------------------------ First, searchers verify that the TDX operator, in this case Flashbots, cannot access or observe code inside a TDX machine through a process called **attestation:** - Searchers audit the TDX image that only their public SSH key can access the machine, and then build and measure (hash) the image locally. -- Then, searchers request and verify the measurement from the cloud provider (in our case Azure) is the same to confirm the exact image they audited is running on the TDX machine they will upload their code to. +- Then, searchers request and verify the measurement from the cloud provider (in our case Google Cloud Platform) is the same to confirm the exact image they audited is running on the TDX machine they will upload their code to. But, in the TDX image, the searcher is also restricted to its own user group without root privileges. This allows us to implement sandboxing and log delays on the host safely, without the searcher being able to override or interfere with these restrictions. So, even though the searcher is given SSH access to the machine, the searcher is sufficiently restricted to guarantee they would not be able to expose any sensitive data submitted by the order flow provider. @@ -132,7 +134,7 @@ Attestation Walkthrough Once searchers receive the IP for their TDX Machine deployed by Flashbots, they should first perform the process of attestation. -*At a high level, searchers will audit the minimal VM image prepared by Flashbots does not introduce malicious code and contains the right SSH configuration. Builders will audit the firewall rules and log delay. Then they will confirm that exact image is running on the TDX VM Flashbots deployed by “measuring” the image (by hashing its files) and comparing their local measurement to that measured by Azure.* +*At a high level, searchers will audit the minimal VM image prepared by Flashbots does not introduce malicious code and contains the right SSH configuration. Builders will audit the firewall rules and log delay. Then they will confirm that exact image is running on the TDX VM Flashbots deployed by “measuring” the image (by hashing its files) and comparing their local measurement to that measured by GCP.* ### 1. build the VM image @@ -188,165 +190,13 @@ On the first startup, after the searcher's SSH key is received and stored, the s **This design ensures that only the searcher whose SSH key initialized the disk can decrypt it across image reboots.** -### 2. audit and run the local measurement software - -Under the hood, Intel TDX attestation relies on a process called measured-boot. - -[Measured boot]((https://docs.edgeless.systems/constellation/2.10/architecture/images#measured-boot)) uses a Trusted Platform Module (TPM) to measure every part of the boot process: - -edgeless-measured-boot - -*[https://docs.edgeless.systems/constellation/2.10/architecture/images#measured-boot](https://docs.edgeless.systems/constellation/2.10/architecture/images#measured-boot)* - -Azure’s vTPM “hash-chains” each stage of the boot process, ensuring the integrity of the entire boot chain up to the root file system. - -azure-measured-boot - -*[https://learn.microsoft.com/en-us/azure/security/fundamentals/measured-boot-host-attestation#measured-boot](https://learn.microsoft.com/en-us/azure/security/fundamentals/measured-boot-host-attestation#measured-boot)* +### 2. audit and run the local measurement and remote attestation software In order to leverage attestation, Flashbots: 1. uses MKOSI to ensure reproducible builds, such that each time anyone builds the image, the measurement will be the same, even on different hardware 2. packages the entire image inside the initramfs, such that any change to the image will result in a different measurement -Flashbots has adapted Edgeless Constellation’s [measured-boot](https://github.com/edgelesssys/constellation/tree/ffde0ef7b7d3277c63f3c67ee666237f5863c744/image/measured-boot) library to simulate the measurements locally, which dissects the .efi image and measures the initramfs and unified kernel PE sections. - -Only [PCR 4, 9, and 11](https://constellation-docs.netlify.app/constellation/2.2/architecture/attestation#runtime-measurements) are meaningful, since the other PCR’s in Azure’s vTPM are not reproducible due to their proprietary closed-source implementations. But, these 3 measurements are enough to ensure Flashbots does not have access to the searcher VM, as any change in the image will generate different PCR 4, 9, and 11 measurements! You can test and verify this claim yourself by changing a line of code, building the new image, and running the measurement software again. - -```bash -cd flashbots-images - -# assuming you've run make build IMAGE=flashbox-l1 -make measure -``` - -
-Expected Output - - ``` - ubuntu@builder:~/flashbots-images$ make measure - EFI Boot Stages: - Stage 1 - Unified Kernel Image (UKI): 320af1bf8257b6fd1a47b8fa865bdde7bdfdbf235894804b6b15b676296b1ba4 - Stage 2 - Linux : eb1a69b12b47b6b3d4716bad94323d27173cba5f4285b918a2bf59ea5cb3c9ea - Linux LOAD_FILE2 protocol: - cmdline: "console=tty0 console=ttyS0,115200n8 mitigations=auto,nosmt spec_store_bypass_disable=on nospectre_v2\x00" - initrd (digest 0cc531c70b473425e513310dfb4cbcfd5161444a07d318b4d5b816f557d589a6) - UKI sections: - Section 1 - .linux ( 5829632 bytes): 0da293e37ad5511c59be47993769aacb91b243f7d010288e118dc90e95aaef5a, 7439b377dbba898b0db23928be49fb906aa5551cfc01395bc37b8bd50d8f5530 - Section 2 - .osrel ( 308 bytes): 3fb9e4e3cc810d4326b5c13cef18aee1f9df8c5f4f7f5b96665724fa3b846e08, 94e5e922dec19c3ab3e3c85b5d30dbb563098a430418a70c11a5b729721fae39 - Section 3 - .cmdline ( 101 bytes): 461203a89f23e36c3a4dc817f905b00484d2cf7e7d9376f13df91c41d84abe46, 5b20d03fb990ccafdcfa1ddb37feff37141e728776ed89f335798f3c3899a135 - Section 4 - .initrd ( 166037465 bytes): 15ee37e75f1e8d42080e91fdbbd2560780918c81fe3687ae6d15c472bbdaac75, 0cc531c70b473425e513310dfb4cbcfd5161444a07d318b4d5b816f557d589a6 - Section 5 - .uname ( 7 bytes): da7a6d941caa9d28b8a3665c4865c143db8f99400ac88d883370ae3021636c30, 2200d673ad92228af377b9573ed86e7a4e36a87a2a9a08d8c1134aca3ddb021c - Section 6 - .sbat ( 309 bytes): ff552fd255be18a3d61c0da88976fc71559d13aad12d1dfe1708cf950cc4b74c, eae67f3a8f5614d71bd75143feeecbb3c12cd202192e2830f0fb1c6df0f4a139 - Section 7 - .data : not measured - Section 8 - .reloc : not measured - Section 9 - .rodata : not measured - Section 10 - .sdmagic: not measured - Section 11 - .text : not measured - PCR[ 4]: 176543f594059b26292565a3c07b5eaa34122cf2ce7f53b149b6fb85c3046d30 - PCR[ 9]: 817c80c72f0a42bd72d4c7130f0d48c39a6d3ac6def92da085dec16feb822518 - PCR[11]: 46a1b5dd625d967205699242a2de2815e539424c3132306b91f31bcda442693f - PCR[12]: 0000000000000000000000000000000000000000000000000000000000000000 - PCR[13]: 0000000000000000000000000000000000000000000000000000000000000000 - PCR[15]: 0000000000000000000000000000000000000000000000000000000000000000 - ``` -
- -Then, copy and paste PCR 4, 9, and 11 into the following format and save as `measurements.json` - -**The image built locally, and as measured by Azure, should match the following hashes!** -```bash -[ - { - "measurement_id": "azure-tdx-example-01", - "attestation_type": "azure-tdx", - "measurements": { - "4": { - "expected": "176543f594059b26292565a3c07b5eaa34122cf2ce7f53b149b6fb85c3046d30" - }, - "9": { - "expected": "817c80c72f0a42bd72d4c7130f0d48c39a6d3ac6def92da085dec16feb822518" - }, - "11": { - "expected": "46a1b5dd625d967205699242a2de2815e539424c3132306b91f31bcda442693f" - } - } - } -] -``` - -> Note: at the time of the writing, those measurements were acquired by building from commit ef5dd2727ba4569d530c67822dc96778f54a295a, if you're viewing this from main branch please ensure to build from the same commit to get the same measurements. - -> Note: at the time of the writing, compiling flashbox-l1 image is not reproducible if building under ARM mac with Rosetta. Please use x86_64 Linux for now. - -### 3. audit and run the remote attestation software which requests the measurement from Azure’s vTPM - -Flashbots again leverages Edgeless Constellation’s [attested TLS](https://docs.edgeless.systems/constellation/architecture/attestation#attested-tls-atls) and other attestation primitives to interact with Azure’s attestation service. attested-tls-proxy fetches Azure's vTPM measurement and compares it with the locally supplied measurement. - -```bash -# download remote attestation tool -git clone https://github.com/flashbots/attested-tls-proxy.git -cd attested-tls-proxy - -# This will run the client proxy that is listening on port 8080 -# and use the server reverse proxy on the deployed image as a target, -# marshalling the measurements.json for validation of the attestation. -cargo run -- client \ - --listen-addr 127.0.0.1:8080 \ - --allow-self-signed \ - --measurements-file ./measurements.json \ - --log-debug \ - :8745 - -# To trigger remote attestation, open a new terminal and run this command: -curl http://127.0.0.1:8080 - -# Bind the attested host keys to known_hosts. -# This ensures the ssh server the searcher connects to is the one running on the -# attested machine. The attested :8745 channel and the host (dropbear) control-plane -# key are available at boot — before you run `initialize` — so the very first SSH -# is verified rather than trust-on-first-use. /pubkey returns whatever host keys -# are currently available; the container (data-plane) key joins once the disk is -# unlocked and the container is up, so re-run this then to bind it too. -git clone https://github.com/flashbots/ssh-pubkey-server - -./ssh-pubkey-server/cmd/cli/add_to_known_hosts.sh \ -http://127.0.0.1:8080 \ - -``` - -
-Example Output - - ```bash - # Start the proxy client - $ cargo run -- client --listen-addr 127.0.0.1:8080 --allow-self-signed --measurements-file ./measurements.json --log-debug 35.255.95.67:8745 - Compiling attested-tls-proxy v1.1.1 (/home/pumkin/src/flashbots/attested-tls-proxy) - Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.17s - Running `target/debug/attested-tls-proxy client --listen-addr '127.0.0.1:8080' --allow-self-signed --measurements-file ./measurements.json --log-debug '35.255.95.67:8745'` - 2026-06-22T07:07:38.553942Z DEBUG attested_tls_proxy: [proxy-client] Connected to proxy server with measurements: Some(DCAP({MRTD: "feb7486608382c1ff0e15b4648ddc0acea6ca974eb53e3529f4c4bd5ffbaa20bf335cb75965cea65fe473aed9647c162", RTMR0: "e1d0235496f93f9475bf0b26d33da5c15831cfc94104d6bea7ab82db027c5f1e917d47dda6953eefae7dcb20ab6f75c4", RTMR1: "4ea5a990afef023f89e11fc32d99103d0adc91d5734664542eb980cdabc88224e1fd206d1d3b2eda71f713fdf8308a2b", RTMR2: "c42ba4fb83f99e4b90bc2a3aa9a2e81c5ac578e9a439c23b457ff7dd0d0eae958760616eff05d827289e47608e757547", RTMR3: "000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"})) - at src/lib.rs:674 - - 2026-06-22T07:07:38.554036Z DEBUG attested_tls_proxy::http_version: [client] Negotiated ALPN Some("flashbots-ratls/1+h2"), chosen protocol Http2 - at src/http_version.rs:39 - - 2026-06-22T07:07:44.356260Z DEBUG attested_tls_proxy: proxy-client accepted connection - at src/lib.rs:594 - - 2026-06-22T07:07:44.356578Z DEBUG attested_tls_proxy: [proxy-client] Read incoming request from source client: Request { method: GET, uri: /pubkey, version: HTTP/1.1, headers: {"host": "127.0.0.1:8080", "user-agent": "curl/8.19.0", "accept": "*/*"}, body: Body(Empty) } - at src/lib.rs:494 - - 2026-06-22T07:07:44.639534Z DEBUG attested_tls_proxy: [proxy-client] Read response from proxy-server: Response { status: 200, version: HTTP/2.0, headers: {"date": "Mon, 22 Jun 2026 07:07:44 GMT", "content-length": "80", "content-type": "text/plain; charset=utf-8"}, body: Body(Streaming) } - at src/lib.rs:498 - - $ curl http://127.0.0.1:8080/pubkey | less - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 100 80 100 80 0 0 280 0 0 - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHqPgQoc6yLlMsRUvEeV+6oUsvCMV1sr+b1uKTLzu4e9 - ``` -
- -If attested-tls-proxy client is able to successfully make a connection to the proxy server, the searcher has now verified that they SSH into a genuine TDX VM, running the exact same image as the one they audited locally. In doing so, the searcher has also verified that no one else has access to the container or host, and they can safely upload your arbitrage bot inside ✨🚀 +Instructions for measuring OS images and checking they match remote attestation from the deployment are on the [Flashbox documentation site](https://flashbox.flashbots.dev/build-and-attest). Order Flow APIs ------------------------