AMD SEV configuration in YAOOK

YAOOK includes backports and additions for OpenStack that enable usage of AMD SEV-SNP for virtual machines in Nova starting with 2024.2.

This guide will describe how to configure a YAOOK infrastructure to offer the AMD SEV feature for virtual machines to users.

Important caveats

Property matching between flavors and images

Please note that the following SEV-related properties may appear on both Glance images and Nova flavors with slightly different naming:

Image property	Flavor property	Purpose
hw_mem_encryption	hw:mem_encryption	Enables or disables SEV usage
hw_mem_encryption_model	hw:mem_encryption_model	Selects between SEV, SEV-ES and SEV-SNP

These properties on images and flavors are optional, they may each be specified on either, both or none of them. If specified on both, they must not have conflicting values. If combining images and flavors with conflicting values, the VM creation will fail.

Please consider the following implications when deciding where to distribute the properties:

• When omitting these properties from flavors, end users may combine SEV features with any flavor by using those of the image instead.

• When omitting these properties from images, affected images may be used for SEV and non-SEV instances alike.

• Setting these properties on both image and flavors will restrict SEV usage to specific image and flavor combinations, i.e., those which do not have conflicting values.

End users may trigger SEV using image properties

As long as end users are able to upload images or modify image properties on their own, they may be able to set hw_mem_encryption/hw_mem_encryption_model properties on images, which might lead to a SEV-enabled VM instance even if the chosen flavor does not specify those properties.

As a result, end users may be able to create SEV-enabled instances even if no flavor with these properties was configured for them by the cloud provider.

To prevent this, set hw:mem_encryption=false on every non-SEV flavor in the OpenStack infrastructure explicitly to cause conflict with such image property. Alternatively, set up host aggregates and flavors in a way that prevents unintended usage of SEV-capable host by tenants.

SEV firmware updates inhibit SEV-ES usage

Applying recent firmware updates on SEV-SNP-enabled processors will prevent SEV-ES usage altogether. This is an intended change by AMD [1].

Note that SEV firmware updates are applied at boot time and are not persistent. When removing the update files and rebooting, the original firmware will start up.

Configuring Nova compute nodes

Choosing the virtualized CPU model

In order to offer AMD SEV-SNP for virtual machines, the applicable compute nodes must be configured to a vCPU model that matches the SEV feature set.

This can be implemented in one of two ways:

1. Use host-passthrough or host-model as the cpu_mode setting in the Nova libvirt configuration. This will mimic the same CPU model as the physical compute host has as the vCPU in virtual machines.

2. Use custom as the cpu_mode setting in the Nova libvirt configuration and add SEV-capable CPU models to the cpu_models setting.

The first approach will expose CPU model details of the physical host to virtualized guests. Furthermore, the resulting CPU model might vary between compute nodes when those use different CPU models.

As such, it is recommended to decide on a standardized CPU model offering the desired SEV features and setting it using the custom mode setting instead. To discover available CPU model names, look into the XML files at /usr/share/libvirt/cpu_map/*.xml and look for the “model name” attribute. In case of YAOOK, you can find this folder structure in the nova-compute image. AMD SEV-SNP is available starting with the EPYC line of processors.

Setting the nova.conf configuration options

The following nova.conf settings must be applied to each applicable compute node:

• [DEFAULT] ram_allocation_ratio: 1.0 to prevent overcommit which conflicts with the memory encryption

• [libvirt] virt_type: kvm

• [libvirt] hw_machine_type: x86_64=q35

• [libvirt] cpu_mode and [libvirt] cpu_models as described above

To implement this in YAOOK on a per-node basis, use node labels and an appropriate spec.compute.configTemplates section in the NovaDeployment as described in the following handbook section: Enable AMD SEV-SNP for nova compute.

Preparing Nova flavors for SEV-SNP

Note: When VM instances are created in Nova, the mem_encryption and mem_encryption_model properties are retrieved from both the flavor and image. If image and flavor specify conflicting values, the creation of the instance will fail.

Use the following template and fill in the variables as desired:

openstack flavor create \
    --ram $RAM_SIZE_MB --disk $DISK_SIZE_GB --vcpus $VCPU_COUNT \
    --public \
    --property hw:mem_encryption=true \
    --property hw:mem_encryption_model='amd-sev-snp' \
    --id $FLAVOR_ID \
    $FLAVOR_NAME

If the flavor should be project-scoped instead, replace --public by --project $PROJECT_ID.

Also consider further segmenting the SEV-capable nodes using flavors that are bound to host aggregates as described in the next section.

Adjusting the scheduling of SEV-capable compute hosts

Nova will automatically limit the scheduling of SEV virtual machines, as indicated by their mem_encryption and mem_encryption_model flavor or image properties, to nodes which actually provide the desired hardware extension.

However, as a cloud provider it might be desirable to further restrict and influence the selection of compute hosts.

This can be implemented using host aggregates in Nova with the AggregateInstanceExtraSpecsFilter scheduler filter and appropriate flavor attributes.

Enabling the scheduler filter

This is very important. Make sure that the AggregateInstanceExtraSpecsFilter is enabled in Nova.

In the NovaDeployment:

spec:
  novaConfig:
    ...
    filter_scheduler:
      enabled_filters:
        - "ComputeFilter"
        - "ComputeCapabilitiesFilter"
        - "ImagePropertiesFilter"
        - "ServerGroupAntiAffinityFilter"
        - "ServerGroupAffinityFilter"
        - "AggregateInstanceExtraSpecsFilter"

Creating a host aggregate

Create the aggregate:

AGG_NAME="sev-hosts-1"
openstack aggregate create "$AGG_NAME"

Next, set the properties on the aggregate, which can later be filtered for in a flavor’s properties. The names and values can be chosen freely according to your use case.

openstack aggregate set \
    --property hwsecurity=amd-sev-snp \
    "$AGG_NAME"

Add the desired hosts to the aggregate. You may use openstack hypervisor list to discover the names.

openstack aggregate add host $AGG_NAME $HOST_NAME

Creating aggregate-bound SEV flavors

Making an existing flavor aggregate-bound:

openstack flavor set \
    --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \
    $EXISTING_FLAVOR_NAME_OR_ID

Creating a new aggregate-bound public flavor for all tenants:

openstack flavor create --ram 2048 --disk 20 --vcpus 2 --public \
    --property hw:mem_encryption=true \
    --property hw:mem_encryption_model=amd-sev-snp \
    --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \
    --id sev-small SEV-2G-2Core

Creating a new aggregate-bound private flavor for a specific tenant:

TENANT_PROJECT_ID=f58bbef5-55b4-4b75-8c1b-1e76ce76aa9b

openstack flavor create --ram 2048 --disk 20 --vcpus 2 --private \
    --project $TENANT_PROJECT_ID \
    --property hw:mem_encryption=true \
    --property hw:mem_encryption_model=amd-sev-snp \
    --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \
    --id sev-small-t1 SEV-2G-2Core-t1

Using any of these flavors will enable SEV-SNP for the VM and choose only the hosts which are in an aggregate with the aggregate property hwsecurity=amd-sev-snp set.

Enabling the visibility of attestation metadata properties in Nova

The SEV-SNP extensions added to Nova by YAOOK also introduce an optional new metadata attribute to the output of the server show API which displays information about the vCPU model setting of a Nova instance. This is useful for calculating a reference attestation measurement outside of the virtual machine or infrastructure for attestation validation purposes. Enabling the visibility of this attribute makes the value available upfront and avoids the necessety of connecting to a running virtual machine to retrieve it.

To not introduce any API behavior regression for infrastructures, the visibility of this attribute is disabled by default. To enable this feature for users, the Nova API policy for os_compute_api:visible-attestation-server-attributes must be adjusted.

If enabled, the value will be visible as ATTESTATION:vcpu_model attribute within the “properties” field of a server.

For example, to enable the visibility of this attribute for both admin and member roles, add the following policy definition to the spec of the NovaDeployment:

spec:
  ...
  policy:
     "os_compute_api:visible-attestation-server-attributes": "role:admin or role:member"

Preparing SEV-SNP-capable Glance images

In order to make proper use of AMD SEV-SNP, the virtual machine images must fulfill certain requirements and uploaded with specific properties.

Refer to the image uploading section of the YAOOK SEV user guide for instructions.

SEV firmware

Note: SEV firmware updates are non-persistent. When the update files are removed from the host system and the system is rebooted, the firmware will be reset to its delivery condition.

Checking the firmware version

The firmware version of AMD SEV can be checked using various tools but is also present in the kernel log during bootup.

By parsing the kernel log:

sudo dmesg | grep ccp

[    4.496365] ccp 0000:43:00.1: SEV API:1.55 build:31
[    4.496377] ccp 0000:43:00.1: SEV-SNP API:1.55 build:31

Using sevctl:

sudo sevctl show version

1.55.31

Using snphost:

sudo snphost show tcb

Reported TCB: TCB Version:
  Microcode:   219
  SNP:         28
  TEE:         0
  Boot Loader: 4
  FMC:         None
Platform TCB: TCB Version:
  Microcode:   219
  SNP:         28
  TEE:         0
  Boot Loader: 4
  FMC:         None

(the values returned by snphost are more “raw” values than human-readable version numbers and often also appear as part of attestation measurements)

Upgrading the SEV firmware

Note: This only concerns the SEV firmware. The PSP bootloader can only be updated by applying a BIOS update.

On most Linux systems, the related firmware files reside within /lib/firmware/amd/ and are applied on bootup.

There are two options for retrieving updated firmware for AMD SEV:

1. Updating the amd64-microcode package from the distribution’s repositories and rebooting.

2. Downloading the latest firmware from AMD and placing them in /lib/firmware/amd/.

In case of the latter, follow the instructions below.

Downloading and applying SEV firmware updates from AMD

1. Make a backup of the /lib/firmware/amd/ directory (recommended).

2. Download the correct firmware for the CPU family from amd.com/en/developer/sev.html.

   • Be mindful of the naming convention of the files.

3. Replace the applicable amd_sev_fam*_model*.sbin file(s) in /lib/firmware/amd/ with the downloaded one(s).

4. Reboot the system or reload the ccp kernel module*.

* as an alternative to a full system reboot, the ccp kernel module can be reloaded to apply the update if no virtual machine is currently running:

sudo rmmod kvm_amd
sudo rmmod ccp
sudo modprobe ccp
sudo modprobe kvm_amd

Notes:

• Reloading the ccp module only works for upgrading the firmware version, a downgrade has to be done via reboot (see below).

• Somewhere between firmware versions 1.55.31 and 1.55.36 a change was introduced that disallows using SEV-ES while SEV-SNP is enabled on a host. This is likely a permanent limitation and an intended change by AMD [1].

Downgrading the SEV firmware

For downgrading the SEV firmware, simply replace the updated files in /lib/firmware/amd/ by older versions (e.g. from a backup) or remove them (to restore factory state) and then reboot the system. As SEV firmware updates are non-persistent, they are gone after a power-cycle and whatever firmware files are picked up by the ccp module are reapplied during host boot.

Note: Downgrading the firmware does not work by reloading the ccp module and always requires a full reboot (after replacing/removing the updated firmware files again).

[1] (1,2)

https://github.com/AMDESE/AMDSEV/issues/282#issuecomment-3554492101

Tempest Testing

The YAOOK Tempest image contains a custom plugin that provides a test suite for AMD SEV.

Preparing the Nova flavor

Create a SEV-enabled flavor:

openstack flavor create \
    --ram 1024 --disk 20 --vcpus 1 \
    --public --property hw:mem_encryption=true \
    --id tempest-sev TEST-SEV

Its ID must later match the spec.flavor_ref_sev value of the TempestJob.

Note: Make sure to not set hw:mem_encryption_model on the flavor as the Tempest plugin relies on changing that through image properties!

TempestJob manifest for SEV tests

Below is an example TempestJob template for executing the AMD SEV test suite using the YAOOK Tempest build on SEV-SNP-able environments.

Please also note the following:

• For machines supporting AMD SEV-SNP, the SEV-ES testing will not work as SEV-SNP makes SEV-ES unavailable in recent firmwares, hence do not use sev-es and sev-snp together in enabled_sev_modes (sev is unaffected).

  • The example below is written for simultaneous SEV and SEV-SNP testing. You may remove sev from enabled_sev_modes if not needed.

• Do not change any of the sev_img_* values, they are designed to match image files already included in the YAOOK Tempest image.

• Make sure to adjust spec.region correctly to make Tempest pick the correct internal API endpoints when talking to services!

• If test_console_log_only …

  • is set to true, successful SEV boot will be verified by parsing a VM’s console log only, not connecting to it directly.

  • is set to false, Tempest will attempt to connect directly to the VM using SSH and parse the kernel log this way; this might not work from the TempestJob pod in some environments.

• It is recommended to set test_attestation_metadata to true if your Nova API policies are configured to allow os_compute_api:visible-attestation-server-attributes for project members as per Enabling the visibility of attestation metadata properties in Nova.

---
apiVersion: yaook.cloud/v1
kind: TempestJob
metadata:
  name: amd-sev-tests
  namespace: yaook
spec:
  keystoneRef:
    name: keystone
  region: MyRegion
  targetRelease: 41.0.0
  target:
    regex: "tests.scenario.test_amd_sev"
  serial: true
  account_cleanup: false
  tempestConfig:
    DEFAULT:
      debug: true
    amd_sev:
      enabled: True
      test_direct_kernel_boot: true
      test_console_log_only: true
      test_attestation_metadata: false
      enabled_sev_modes: [sev, sev-snp]
      flavor_ref_sev: tempest-sev
      sev_img_file: /home/tempest/debian-cloud-amd64.qcow2
      sev_img_container_format: bare
      sev_img_disk_format: qcow2
      sev_img_file_kernel: /home/tempest/debian-cloud-kernel-amd64.vmlinuz
      sev_img_file_initrd: /home/tempest/debian-cloud-initrd-amd64.initrd
      sev_img_cmdline: "root=/dev/vda1 console=ttyS0"
      sev_img_ssh_username: debian
    compute:
      endpoint_type: internal
      max_microversion: latest
    identity:
      endpoint_type: internal
    image:
      endpoint_type: internal
    network:
      endpoint_type: internal
    placement:
      endpoint_type: internal
    key_manager:
      endpoint_type: internal
    service_available:
      glance: true
      neutron: true
      nova: true
      cinder: true
      barbican: false
      swift: false
      horizon: false
    validation:
      run_validation: false
    volume:
      endpoint_type: internal
      max_microversion: latest