Support for starting confidential containers by ambarve · Pull Request #2495 · microsoft/hcsshim

ambarve · 2025-08-08T19:50:09Z

This PR adds the support for running confidential UVMs/Pods and then running containers inside such pods. It also brings in some additional fixes related to confidential pods.

Support for starting confidential pods
This commit allows running confidential pods/UtilityVMs by including a non-empty security
policy in the pod config annotations. The files (VHDs & guest statef file) required for
starting such a confidential UVM MUST be available in a directory named
WindowsBootFiles/confidential that is located next to the shim binary.
Support for starting confidential containers
Adds support for mounting block CIM based container image layers inside the UVM.
(ResourceTypeWCOWBlockCims is only supported by sidecar GCS at the moment. Inbox GCS will
add this support soon.)
Include policy digest in the host data for confidential UVM
When the policy digest is included in the host data field of the UVM config, the SNP
hardware is able to directly access that and include that in the attestation report.
Attach EFI VHD in read-only mode by default
EFI VHDs should always be attached as read-only by default to block UVMs from writing to
it and corrupting its contents. A new annotation is added to allow attaching EFI VHDs in
writable mode when debugging boot failures and such. When this annotation is included a
copy of the EFI VHD is made next to the scratch VHD. This is based on the assumption that
generally the scratch of the UVM would be stored in its own snapshot directory so adding
another VHD in there shouldn't be a problem. It should get cleaned up when the snapshot is
removed. This commit also adds the code to always grant VM group access to the VHDs and guest state
files to avoid access denied failures.
format container scratch in superfloppy mode

ambarve · 2025-08-26T15:09:31Z

@KenGordon commit d324057 in this PR handles passing policy through host data, in case you want to review the change.

We already added support for starting confidential UVMs. This commit adds support for running confidential pods (via containerd) by including a non-empty security policy in the pod config annotations. The files (VHDs & guest statef file) required for starting such a confidential UVM MUST be available in a directory named `WindowsBootFiles/confidential` that is located next to the shim binary. Since some LCOW & WCOW annotations have conflicting/confusing names, existing LCOW annotation vars are renamed to have the "LCOW" prefix. Signed-off-by: Amit Barve <ambarve@microsoft.com>

Adds support for mounting block CIM based container image layers inside the UVM. (ResourceTypeWCOWBlockCims is only supported by sidecar GCS at the moment. Inbox GCS will add this support soon.) Signed-off-by: Amit Barve <ambarve@microsoft.com>

When the policy digest is included in the host data field of the UVM config, the SNP hardware is able to directly access that and include that in the attestation report. Signed-off-by: Amit Barve <ambarve@microsoft.com>

EFI VHDs should always be attached as read-only by default to block UVMs from writing to it and corrupting its contents. A new annotation is added to allow attaching EFI VHDs in writable mode when debugging boot failures and such. When this annotation is included a copy of the EFI VHD is made next to the scratch VHD. This is based on the assumption that generally the scratch of the UVM would be stored in its own snapshot directory so adding another VHD in there shouldn't be a problem. It should get cleaned up when the snapshot is removed. This commit also adds the code to always grant VM group access to the VHDs and guest state files to avoid access denied failures. Signed-off-by: Amit Barve <ambarve@microsoft.com>

Signed-off-by: Amit Barve <ambarve@microsoft.com>

KenGordon · 2025-08-28T09:44:21Z

@KenGordon commit d324057 in this PR handles passing policy through host data, in case you want to review the change.

Done. Looks good, I did mention maybe taking the comment wrt the purpose of the host data.

KenGordon · 2025-08-28T10:10:42Z

Sorry, pressed the wrong button when commenting. Opps.

KenGordon

I have added some comments about changes but there is nothing I would say should get in the way of merging it so we can get the rest merged and the demo building.

KenGordon · 2025-08-28T10:28:37Z


+// handleWCOWSecurityPolicy handles parsing confidential pods related options and setting
+// implied options from the results for WCOW.
+func handleWCOWSecurityPolicy(ctx context.Context, a map[string]string, wopts *uvm.OptionsWCOW) error {


You should take a look at handleSecurityPolicy in the LCOW path. There is a flag:

// allow actual isolated boot etc to be ignored if we have no hardware. Required for dev // this is not a security issue as the attestation will fail without a genuine report noSecurityHardware := ParseAnnotationsBool(ctx, a, annotations.NoSecurityHardware, false)

and if it is set then SecurityPolicyEnabled is false. This there CAN BE a opts.SecurityPolicy when SecurityPolicyEnabled is false. That opts.SecurityPolicy will be sent to the UVM by SetConfidentialUVMOptions at the first opportunity (see
func (uvm *UtilityVM) Start(ctx context.Context) (err error) {)

The point of all this complexity is to allow us to run the Confidential stack on non confidential hardware, as nobody has a laptop/desktop that is SNP enabled. You have a choice about how the boot behaves. In the LCOW case the non SNP "confidential" mode boots exactly like regular ContainerPlat, so the hcs json is the same. In a WCOW version of this you might or might not choose to use the full new-style boot. I suggest you do use that boot and that everything is identical except that the PSP driver will tell the guest agent there is no PSP. This lets us test/experiment with the whole stack without SNP hardware or VM.

BTW, yes the SecurityPolicyEnabled flag would be better named something like EncryptedHardwareEnabled or something like it.

I see there is another flag to disable/enable secure boot. We ought to double check that we get a different measurement when it is disabled.

In the current CWCOW setup, if you use the WCOWIsolationType annotation to set it to use "VBS" isolation and use the WCOWGuestStateFile annotation to pass the appropriate VMGS file for VBS (instead of the default SNP one), it will actually let you boot the confidential pod on a non-SNP hardware. That's how I have been testing it thus far. Would that be enough for us?

Yes if you disable secure boot but are still booting with a VBS or SNP VMGS, it does fail to boot. So we won't even get to the measurement part.

Is it easy to update a binary inside the UVM with this scheme? Or will secure boot enforce code integrity and insist on proper signing (or signing that is hard to do on a local build).

KenGordon · 2025-08-28T10:44:47Z

+		return fmt.Errorf("allow overcommit MUST be false for confidential pods")
+	}
+
+	wopts.SecurityPolicyEnforcer = ParseAnnotationsString(a, annotations.WCOWSecurityPolicyEnforcer, wopts.SecurityPolicyEnforcer)


TBC next week when Tingmao is back but you probably do not need this enforcer type now. It used to be used to switch between json and rego policies and was just a hangover but Tingmao touched that last and can say if this can be ignored.

KenGordon · 2025-08-28T10:49:19Z


-	// GuestStateFile specifies the path of the vmgs file to use if required. Only applies in SNP mode.
-	GuestStateFile = "io.microsoft.virtualmachine.lcow.gueststatefile"
+	// LCOWGuestStateFile specifies the path of the vmgs file to use if required. Only applies in SNP mode.


For purely internal annotations, eg lcow.gueststatefile, where we control the client or where no client ever sets it (TBC with ACI) then I suggest not having wcow.gueststatefile and lcow.gueststatefile but just the one for both. Otherwise there is quite a bit of unnecessary duplication. Some of our tests do have pod.json files that set these explicitly but I can fix those to use a shared one. Again, this is not needed immediately.

Yeah, I will handle this as a part of a separate PR to reduce churn on this PR.

MahatiC · 2025-08-29T12:37:38Z

 			}
 			return nil

-		case guestresource.ResourceTypeCWCOWCombinedLayers:


@ambarve As discussed, this was added intentionally. It is just logging in this commit, but my changes need containerID and is being used for enforcement. Please revert this commit.

containerID is required for all gcs requests in confidential mode. Most of the requests already include it but CombineLayers and MountBlockCIM request types don't. This commit adds that to these request types. Signed-off-by: Amit Barve <ambarve@microsoft.com>

MahatiC · 2025-08-29T15:10:25Z

-			wcowBlockCimMounts := modifyGuestSettingsRequest.Settings.(*guestresource.WCOWBlockCIMMounts)
+			if modifyGuestSettingsRequest.RequestType == guestrequest.RequestTypeRemove {
+				return fmt.Errorf("not implemented")
+			}


@ambarve Thanks for the changes. What about RequestTypeAdd? Is it because the default Request type is Add? I understood from our discussion that you would add a no-op (for now) for both types.

The types are either Add or Remove. We do want the current handling to happen for Add. When I add the implementation for Remove (in a separate PR), I will restructure the code a bit better for to have a switch case on the type of request.

Ok. I think you want the current handling only on RequestTypeAdd type and not on RequestTypeRemove. So a switch case is necessary - a separate PR sounds good.

KenGordon · 2025-08-29T16:13:58Z

 	// It may be an error to have a security policy but not expose it to the
 	// container as in that case it can never be checked as correct by a verifier.
-	if oci.ParseAnnotationsBool(ctx, settings.OCISpecification.Annotations, annotations.UVMSecurityPolicyEnv, true) {
+	if oci.ParseAnnotationsBool(ctx, settings.OCISpecification.Annotations, annotations.LCOWSecurityPolicyEnv, true) {


There will need to be code exactly like this in the WCOW case too. As I mentioned elsewhere, this is likely a local scope annotation and we should have the same for both and not have duplicate code.

(also not a blocker)

ambarve force-pushed the vcim_boot branch 2 times, most recently from f10bd37 to bc466fc Compare August 20, 2025 16:29

ambarve changed the title ~~Vcim boot~~ Support for starting confidential containers Aug 20, 2025

ambarve force-pushed the vcim_boot branch from bc466fc to bb4e798 Compare August 20, 2025 16:41

ambarve marked this pull request as ready for review August 20, 2025 16:46

ambarve requested a review from a team as a code owner August 20, 2025 16:46

msscotb assigned kiashok and helsaawy Aug 20, 2025

kiashok approved these changes Aug 25, 2025

View reviewed changes

ambarve force-pushed the vcim_boot branch from bd2300d to 106b161 Compare August 25, 2025 19:59

helsaawy requested changes Aug 26, 2025

View reviewed changes

Comment thread internal/layers/wcow_parse.go Outdated

Comment thread internal/oci/uvm.go

Comment thread pkg/annotations/annotations.go

Comment thread internal/uvm/cimfs.go

Comment thread internal/uvm/combine_layers.go Outdated

Comment thread internal/uvm/cimfs.go

ambarve added 5 commits August 27, 2025 11:23

Include policy digest in the host data for confidential UVM

61b1bd4

When the policy digest is included in the host data field of the UVM config, the SNP hardware is able to directly access that and include that in the attestation report. Signed-off-by: Amit Barve <ambarve@microsoft.com>

format container scratch in superfloppy mode

e48451f

Signed-off-by: Amit Barve <ambarve@microsoft.com>

ambarve force-pushed the vcim_boot branch from 106b161 to e48451f Compare August 27, 2025 15:27

KenGordon closed this Aug 28, 2025

KenGordon reopened this Aug 28, 2025

KenGordon approved these changes Aug 28, 2025

View reviewed changes

MahatiC suggested changes Aug 29, 2025

View reviewed changes

ambarve force-pushed the vcim_boot branch from bdb413f to 11fd4bd Compare August 29, 2025 14:54

MahatiC suggested changes Aug 29, 2025

View reviewed changes

KenGordon reviewed Aug 29, 2025

View reviewed changes

helsaawy approved these changes Sep 2, 2025

View reviewed changes

ambarve merged commit 034954b into microsoft:main Sep 2, 2025
16 of 17 checks passed

Uh oh!

Conversation

ambarve commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ambarve commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KenGordon commented Aug 28, 2025

Uh oh!

KenGordon commented Aug 28, 2025

Uh oh!

KenGordon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MahatiC Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ambarve commented Aug 8, 2025 •

edited

Loading

MahatiC Aug 29, 2025 •

edited

Loading