.. _amd-sev-configuration: AMD SEV configuration in YAOOK ============================== YAOOK includes backports and additions for OpenStack that enable usage of AMD SEV-SNP for virtual machines in Nova starting with 2024.2. This guide will describe how to configure a YAOOK infrastructure to offer the AMD SEV feature for virtual machines to users. Important caveats ----------------- Property matching between flavors and images ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please note that the following SEV-related properties may appear on both Glance images and Nova flavors with slightly different naming: +-----------------------------+-----------------------------+-----------------------+ | Image property | Flavor property | Purpose | +=============================+=============================+=======================+ | ``hw_mem_encryption`` | ``hw:mem_encryption`` | Enables or disables | | | | SEV usage | +-----------------------------+-----------------------------+-----------------------+ | ``hw_mem_encryption_model`` | ``hw:mem_encryption_model`` | Selects between SEV, | | | | SEV-ES and SEV-SNP | +-----------------------------+-----------------------------+-----------------------+ These properties on images and flavors are optional, they may each be specified on either, both or none of them. If specified on both, they must not have conflicting values. If combining images and flavors with conflicting values, the VM creation will fail. Please consider the following implications when deciding where to distribute the properties: - When omitting these properties from flavors, end users may combine SEV features with any flavor by using those of the image instead. - When omitting these properties from images, affected images may be used for SEV and non-SEV instances alike. - Setting these properties on both image and flavors will restrict SEV usage to specific image and flavor combinations, i.e., those which do not have conflicting values. End users may trigger SEV using image properties ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ As long as end users are able to upload images or modify image properties on their own, they may be able to set ``hw_mem_encryption``/``hw_mem_encryption_model`` properties on images, which might lead to a SEV-enabled VM instance even if the chosen flavor does not specify those properties. As a result, end users may be able to create SEV-enabled instances even if no flavor with these properties was configured for them by the cloud provider. To prevent this, set ``hw:mem_encryption=false`` on *every* non-SEV flavor in the OpenStack infrastructure explicitly to cause conflict with such image property. Alternatively, set up host aggregates and flavors in a way that prevents unintended usage of SEV-capable host by tenants. SEV firmware updates inhibit SEV-ES usage ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Applying recent firmware updates on SEV-SNP-enabled processors will prevent SEV-ES usage altogether. This is an intended change by AMD [1]_. Note that SEV firmware updates are applied at boot time and are not persistent. When removing the update files and rebooting, the original firmware will start up. Configuring Nova compute nodes ------------------------------ Choosing the virtualized CPU model ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to offer AMD SEV-SNP for virtual machines, the applicable compute nodes must be configured to a vCPU model that matches the SEV feature set. This can be implemented in one of two ways: 1. Use ``host-passthrough`` or ``host-model`` as the ``cpu_mode`` setting in the Nova libvirt configuration. This will mimic the same CPU model as the physical compute host has as the vCPU in virtual machines. 2. Use ``custom`` as the ``cpu_mode`` setting in the Nova libvirt configuration and add SEV-capable CPU models to the ``cpu_models`` setting. The first approach will expose CPU model details of the physical host to virtualized guests. Furthermore, the resulting CPU model might vary between compute nodes when those use different CPU models. As such, it is recommended to decide on a standardized CPU model offering the desired SEV features and setting it using the ``custom`` mode setting instead. To discover available CPU model names, look into the XML files at ``/usr/share/libvirt/cpu_map/*.xml`` and look for the “``model name``” attribute. In case of YAOOK, you can find this folder structure in the ``nova-compute`` image. AMD SEV-SNP is available starting with the EPYC line of processors. Setting the ``nova.conf`` configuration options ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following ``nova.conf`` settings must be applied to each applicable compute node: - ``[DEFAULT] ram_allocation_ratio: 1.0`` to prevent overcommit which conflicts with the memory encryption - ``[libvirt] virt_type: kvm`` - ``[libvirt] hw_machine_type: x86_64=q35`` - ``[libvirt] cpu_mode`` and ``[libvirt] cpu_models`` as described above To implement this in YAOOK on a per-node basis, use node labels and an appropriate ``spec.compute.configTemplates`` section in the NovaDeployment as described in the following handbook section: :ref:`enable-amd-sev-snp-nova-compute`. Preparing Nova flavors for SEV-SNP ---------------------------------- **Note:** When VM instances are created in Nova, the ``mem_encryption`` and ``mem_encryption_model`` properties are retrieved from both the flavor and image. If image and flavor specify conflicting values, the creation of the instance will fail. Use the following template and fill in the variables as desired: .. code:: sh openstack flavor create \ --ram $RAM_SIZE_MB --disk $DISK_SIZE_GB --vcpus $VCPU_COUNT \ --public \ --property hw:mem_encryption=true \ --property hw:mem_encryption_model='amd-sev-snp' \ --id $FLAVOR_ID \ $FLAVOR_NAME If the flavor should be project-scoped instead, replace ``--public`` by ``--project $PROJECT_ID``. Also consider further segmenting the SEV-capable nodes using flavors that are bound to host aggregates as described in the next section. Adjusting the scheduling of SEV-capable compute hosts ----------------------------------------------------- Nova will automatically limit the scheduling of SEV virtual machines, as indicated by their ``mem_encryption`` and ``mem_encryption_model`` flavor or image properties, to nodes which actually provide the desired hardware extension. However, as a cloud provider it might be desirable to further restrict and influence the selection of compute hosts. This can be implemented using host aggregates in Nova with the ``AggregateInstanceExtraSpecsFilter`` scheduler filter and appropriate flavor attributes. Enabling the scheduler filter ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is very important. Make sure that the ``AggregateInstanceExtraSpecsFilter`` is enabled in Nova. In the ``NovaDeployment``: .. code:: yaml spec: novaConfig: ... filter_scheduler: enabled_filters: - "ComputeFilter" - "ComputeCapabilitiesFilter" - "ImagePropertiesFilter" - "ServerGroupAntiAffinityFilter" - "ServerGroupAffinityFilter" - "AggregateInstanceExtraSpecsFilter" Creating a host aggregate ~~~~~~~~~~~~~~~~~~~~~~~~~ Create the aggregate: .. code:: sh AGG_NAME="sev-hosts-1" openstack aggregate create "$AGG_NAME" Next, set the properties on the aggregate, which can later be filtered for in a flavor’s properties. The names and values can be chosen freely according to your use case. .. code:: sh openstack aggregate set \ --property hwsecurity=amd-sev-snp \ "$AGG_NAME" Add the desired hosts to the aggregate. You may use ``openstack hypervisor list`` to discover the names. .. code:: sh openstack aggregate add host $AGG_NAME $HOST_NAME Creating aggregate-bound SEV flavors ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Making an existing flavor aggregate-bound: .. code:: sh openstack flavor set \ --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \ $EXISTING_FLAVOR_NAME_OR_ID Creating a new aggregate-bound public flavor for all tenants: .. code:: sh openstack flavor create --ram 2048 --disk 20 --vcpus 2 --public \ --property hw:mem_encryption=true \ --property hw:mem_encryption_model=amd-sev-snp \ --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \ --id sev-small SEV-2G-2Core Creating a new aggregate-bound private flavor for a specific tenant: .. code:: sh TENANT_PROJECT_ID=f58bbef5-55b4-4b75-8c1b-1e76ce76aa9b openstack flavor create --ram 2048 --disk 20 --vcpus 2 --private \ --project $TENANT_PROJECT_ID \ --property hw:mem_encryption=true \ --property hw:mem_encryption_model=amd-sev-snp \ --property aggregate_instance_extra_specs:hwsecurity=amd-sev-snp \ --id sev-small-t1 SEV-2G-2Core-t1 Using any of these flavors will enable SEV-SNP for the VM and choose only the hosts which are in an aggregate with the aggregate property ``hwsecurity=amd-sev-snp`` set. .. _amd-sev-enable-attestation-metadata-visibility: Enabling the visibility of attestation metadata properties in Nova ------------------------------------------------------------------ The SEV-SNP extensions added to Nova by YAOOK also introduce an optional new metadata attribute to the output of the server show API which displays information about the vCPU model setting of a Nova instance. This is useful for :ref:`calculating a reference attestation measurement outside of the virtual machine or infrastructure ` for attestation validation purposes. Enabling the visibility of this attribute makes the value available upfront and avoids the necessety of connecting to a running virtual machine to retrieve it. To not introduce any API behavior regression for infrastructures, the visibility of this attribute is disabled by default. To enable this feature for users, the Nova API policy for ``os_compute_api:visible-attestation-server-attributes`` must be adjusted. If enabled, the value will be visible as ``ATTESTATION:vcpu_model`` attribute within the "properties" field of a server. For example, to enable the visibility of this attribute for both admin and member roles, add the following policy definition to the spec of the NovaDeployment: .. code:: yaml spec: ... policy: "os_compute_api:visible-attestation-server-attributes": "role:admin or role:member" Preparing SEV-SNP-capable Glance images --------------------------------------- In order to make proper use of AMD SEV-SNP, the virtual machine images must fulfill certain requirements and uploaded with specific properties. Refer to the :ref:`image uploading ` section of the YAOOK SEV user guide for instructions. SEV firmware ------------ **Note:** SEV firmware updates are non-persistent. When the update files are removed from the host system and the system is rebooted, the firmware will be reset to its delivery condition. Checking the firmware version ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The firmware version of AMD SEV can be checked using various tools but is also present in the kernel log during bootup. By parsing the kernel log: :: sudo dmesg | grep ccp [ 4.496365] ccp 0000:43:00.1: SEV API:1.55 build:31 [ 4.496377] ccp 0000:43:00.1: SEV-SNP API:1.55 build:31 Using `sevctl `__: :: sudo sevctl show version 1.55.31 Using `snphost `__: :: sudo snphost show tcb Reported TCB: TCB Version: Microcode: 219 SNP: 28 TEE: 0 Boot Loader: 4 FMC: None Platform TCB: TCB Version: Microcode: 219 SNP: 28 TEE: 0 Boot Loader: 4 FMC: None (the values returned by ``snphost`` are more “raw” values than human-readable version numbers and often also appear as part of attestation measurements) Upgrading the SEV firmware ~~~~~~~~~~~~~~~~~~~~~~~~~~ **Note:** This only concerns the SEV firmware. The PSP bootloader can only be updated by applying a BIOS update. On most Linux systems, the related firmware files reside within ``/lib/firmware/amd/`` and are applied on bootup. There are two options for retrieving updated firmware for AMD SEV: 1. Updating the ``amd64-microcode`` package from the distribution’s repositories and rebooting. 2. Downloading the latest firmware from AMD and placing them in ``/lib/firmware/amd/``. In case of the latter, follow the instructions below. Downloading and applying SEV firmware updates from AMD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. Make a backup of the ``/lib/firmware/amd/`` directory (recommended). 2. Download the correct firmware for the CPU family from `amd.com/en/developer/sev.html `__. - Be mindful of the `naming convention `__ of the files. 3. Replace the applicable ``amd_sev_fam*_model*.sbin`` file(s) in ``/lib/firmware/amd/`` with the downloaded one(s). 4. Reboot the system or reload the ``ccp`` kernel module\*. \* as an alternative to a full system reboot, the ``ccp`` kernel module can be reloaded to apply the update if no virtual machine is currently running: .. code:: sh sudo rmmod kvm_amd sudo rmmod ccp sudo modprobe ccp sudo modprobe kvm_amd Notes: - Reloading the ``ccp`` module only works for upgrading the firmware version, a downgrade has to be done via reboot (see below). - Somewhere between firmware versions 1.55.31 and 1.55.36 a change was introduced that disallows using SEV-ES while SEV-SNP is enabled on a host. This is likely a permanent limitation and an intended change by AMD [1]_. Downgrading the SEV firmware ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For downgrading the SEV firmware, simply replace the updated files in ``/lib/firmware/amd/`` by older versions (e.g. from a backup) or remove them (to restore factory state) and then reboot the system. As SEV firmware updates are non-persistent, they are gone after a power-cycle and whatever firmware files are picked up by the ``ccp`` module are reapplied during host boot. **Note:** Downgrading the firmware does not work by reloading the ``ccp`` module and always requires a full reboot (after replacing/removing the updated firmware files again). .. [1] https://github.com/AMDESE/AMDSEV/issues/282#issuecomment-3554492101 Tempest Testing --------------- The YAOOK Tempest image contains a custom plugin that provides a test suite for AMD SEV. Preparing the Nova flavor ~~~~~~~~~~~~~~~~~~~~~~~~~ Create a SEV-enabled flavor: .. code:: sh openstack flavor create \ --ram 1024 --disk 20 --vcpus 1 \ --public --property hw:mem_encryption=true \ --id tempest-sev TEST-SEV Its ID must later match the ``spec.flavor_ref_sev`` value of the TempestJob. **Note:** Make sure to *not* set ``hw:mem_encryption_model`` on the flavor as the Tempest plugin relies on changing that through image properties! TempestJob manifest for SEV tests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Below is an example TempestJob template for executing the AMD SEV test suite using the YAOOK Tempest build on SEV-SNP-able environments. Please also note the following: - For machines supporting AMD SEV-SNP, the SEV-ES testing will not work as SEV-SNP makes SEV-ES unavailable in recent firmwares, hence do not use ``sev-es`` and ``sev-snp`` together in ``enabled_sev_modes`` (``sev`` is unaffected). - The example below is written for simultaneous SEV and SEV-SNP testing. You may remove ``sev`` from ``enabled_sev_modes`` if not needed. - Do not change any of the ``sev_img_*`` values, they are designed to match image files already included in the YAOOK Tempest image. - Make sure to adjust ``spec.region`` correctly to make Tempest pick the correct internal API endpoints when talking to services! - If ``test_console_log_only`` ... - is set to ``true``, successful SEV boot will be verified by parsing a VM's console log only, not connecting to it directly. - is set to ``false``, Tempest will attempt to connect directly to the VM using SSH and parse the kernel log this way; this might not work from the TempestJob pod in some environments. - It is recommended to set ``test_attestation_metadata`` to ``true`` if your Nova API policies are configured to allow ``os_compute_api:visible-attestation-server-attributes`` for project members as per :ref:`amd-sev-enable-attestation-metadata-visibility`. .. literalinclude:: amd-sev-tempestjob.yaml