AMD SEV usage in YAOOK ====================== Preface: flavor and image properties for SEV -------------------------------------------- The following properties related to AMD SEV features can be set on Nova flavors and/or Glance images: +-----------------------------+-------------+-----------+----------------------+ | Property name | Flavor | Image | Comment | +=============================+=============+===========+======================+ | ``hw_machine_type`` | | x | Must be | | | | | ``q35``. | +-----------------------------+-------------+-----------+----------------------+ | ``hw_mem_encryption`` | x | x | Must be | | | | | ``true``. | +-----------------------------+-------------+-----------+----------------------+ | ``hw_mem_encryption_model`` | x | x | Choices: | | | | | ``amd-sev``, | | | | | ``amd-sev-es``, | | | | | ``amd-sev-snp``. | +-----------------------------+-------------+-----------+----------------------+ | ``hw_firmware_type`` | | x | Must be | | | | | ``uefi``. | +-----------------------------+-------------+-----------+----------------------+ | ``hw_firmware_stateless`` | | x | For SEV-SNP, | | | | | must be set | | | | | to ``true``. | +-----------------------------+-------------+-----------+----------------------+ .. **Note:** In case of flavors, the property names are prefixed with ``hw:`` instead of ``hw_``, e.g., ``hw:mem_encryption``. When creating a virtual machine in OpenStack, these properties may appear on either flavor or image or - for some properties - on both. If the same property is specified on both flavor and image they must not have conflicting values. When selecting/preparing image and flavor for a virtual machine make sure that all necessary values exist. Creating a virtual machine with AMD SEV-SNP ------------------------------------------- To create a virtual machine with AMD SEV-SNP enabled, the following steps are necessary: 1. Select a suitable flavor with the necessary attributes. 2. Select or upload a suitable image with the necessary attributes. 3. Create a Nova instance using the selected flavor and image. Choosing a SEV-SNP-compatible flavor ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Select a flavor that fulfills the following requirements: - in the flavor’s properties: - ``hw:mem_encryption`` is either not specified or set to ``true`` - ``hw:mem_encryption_model`` is either not specified or set to ``amd-sev-snp`` Use the following process to select such flavor: 1. Use ``openstack flavor list`` to enumerate the flavors available to your project. 2. Use ``openstack flavor show $NAME_OR_ID`` to inspect an individual flavor. 3. In the output of the ``openstack flavor show`` command, inspect the ``properties`` row and look for the property names mentioned above. Choosing a SEV-SNP-compatible image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The image must fulfill the following requirements: - in the image’s properties: - ``hw_machine_type`` must be set to ``q35`` - ``hw_mem_encryption`` must\* be set to ``true`` - ``hw_mem_encryption_model`` must\* be set to ``amd-sev-snp`` - ``hw_firmware_type`` must be set to ``uefi`` - ``hw_firmware_stateless`` must be set to ``true`` - the operating system contained in the image: - must support UEFI boot - use Linux kernel version 5.19 or later \* If ``hw:mem_encryption`` and/or ``hw:mem_encryption_model`` are specified on the chosen flavor, their image property counterparts may be omitted. However, it is strongly recommended to always set those properties on the image to expose conflicting settings in the flavor. Regarding the operating system requirements (UEFI boot, kernel version), this includes but is not limited to: - Ubuntu 24.04 or later - Debian 13 or later - RHEL 9.3 or later - SLES 15 SP4 or later An image adhering to these requirements can either be selected from the pool of existing images (if any) or uploaded. a) Selecting an existing image ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Use the following process to select an image: 1. Use ``openstack image list`` to enumerate the images available to your project. 2. Use ``openstack image show $NAME_OR_ID`` to inspect an individual image. 3. In the output of the ``openstack image show`` command, inspect the ``properties`` row and look for the property names mentioned above. 4. Make sure that the operating system contained in the image fulfills the requirements by either deducing the version from the image name (if applicable), booting and inspecting the image once or asking the owner of the image. .. _amd-sev-usage-upload-compatible-image: b) Uploading a compatible image ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A SEV-SNP-capable image can be prepared in one of two ways: 1. As a single unified image, containing the kernel and init ramdisk. 2. Three separate images, where the kernel and init ramdisk are dedicated and immutable images for direct kernel boot. The second choice is especially useful for specifc security and attestation purposes as the booted ramdisk and kernel will be immutable and deterministic, improving the attestation measurement scope. Furthermore, the kernel cannot be replaced at runtime. The following examples will address both cases based on a generic Debian 13 cloud image in QCOW2 format. Uploading a single image is straightforward: .. code:: sh openstack image create $IMAGE_NAME \ --disk-format qcow2 \ --container-format bare \ --property hw_firmware_type=uefi \ --property hw_machine_type=q35 \ --property hw_mem_encryption=true \ --property hw_mem_encryption_model=amd-sev-snp \ --property hw_firmware_stateless=true \ --file debian-13-genericcloud-amd64-20250814-2204.qcow2 (the values for ``--file`` and ``--disk-format`` will vary depending on the source image file; replace the ``$IMAGE_NAME`` variable as desired) To upload a split image for direct kernel boot first prepare the dedicated kernel and init ramdisk image file. Refer to the :ref:`extracting-kernel-and-initrd-images` appendix section for instructions on how to extract these from an existing image. Next, upload the kernel and init ramdisk images first, determine their resulting image IDs and finally specify them as ``kernel_id`` and ``ramdisk_id`` properties on the main image. Furthermore, an ``os_command_line`` property must be added to the main image specifying the kernel boot arguments to ensure proper boot of the correct device. Example: .. code:: sh # kernel image openstack image create $IMAGE_NAME-kernel \ --disk-format raw \ --container-format bare \ --file vmlinuz-6.12.41+deb13-cloud-amd64 # ramdisk image openstack image create $IMAGE_NAME-initrd \ --disk-format raw \ --container-format bare \ --file initrd.img-6.12.41+deb13-cloud-amd64 # retrieve resulting image IDs export KERNEL_IMG_UUID=$( openstack image list -f value -c ID -c Name \ | grep "$IMAGE_NAME-kernel" | cut -d' ' -f1 ) export INITRD_IMG_UUID=$( openstack image list -f value -c ID -c Name \ | grep "$IMAGE_NAME-initrd" | cut -d' ' -f1 ) # specify appropriate kernel arguments export KERNEL_ARGUMENTS="root=/dev/vda1 console=ttyS0" openstack image create $IMAGE_NAME \ --disk-format qcow2 \ --container-format bare \ --property hw_firmware_type=uefi \ --property hw_machine_type=q35 \ --property hw_mem_encryption=true \ --property hw_mem_encryption_model=amd-sev-snp \ --property hw_firmware_stateless=true \ --property kernel_id=$KERNEL_IMG_UUID \ --property ramdisk_id=$INITRD_IMG_UUID \ --property os_command_line="$KERNEL_ARGUMENTS" \ --file debian-13-genericcloud-amd64-20250814-2204.qcow2 (the values for ``--file`` and ``--disk-format`` will vary depending on the source image file; replace the ``$IMAGE_NAME`` variable as desired) Starting the virtual machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To start a SEV-SNP-enabled virtual machine, simply create a server instance in Nova and specify the flavor and image as selected or prepared above respectively. As long as the flavor and image specify the correct properties, the virtual machine will automatically boot with SEV-SNP enabled. No further actions are necessary. Verify AMD SEV-SNP state of a virtual machine --------------------------------------------- SEV-SNP capability can be confirmed via the kernel log. It should be something like this: :: user@vm:~$ sudo dmesg | grep SEV [ 1.151097] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP [ 1.153063] SEV: Status: SEV SEV-ES SEV-SNP [ 1.286999] SEV: APIC: wakeup_secondary_cpu() replaced with wakeup_cpu_via_vmgexit() [ 1.808122] SEV: Using SNP CPUID table, 33 entries present. [ 1.808678] SEV: SNP running at VMPL0. [ 2.766601] SEV: SNP guest platform device initialized. [ 7.209154] sev-guest sev-guest: Initialized SEV guest driver (using VMPCK0 communication key) [ 7.555961] kvm_amd: KVM is unsupported when running as an SEV guest In case SSH access is not available, ``openstack console log show`` can be used instead: :: openstack console log show $VM_ID_OR_NAME | grep SEV If connecting to the virtual machine is possible, the state can also be confirmed in a more elaborate and robust fashion using the ``snpguest`` tool (see :ref:`getting-the-snpguest-tool`): :: user@vm:~$ sudo snpguest ok [ PASS ] - SEV: ENABLED [ PASS ] - SEV-ES: ENABLED [ PASS ] - SNP: ENABLED [ PASS ] - Optional Features statuses: [...] Attestation procedures for SEV-SNP ---------------------------------- Attestation of a virtual machine booted with SEV-SNP can help proving that the SEV feature is genuine and that the confidentiality is ensured. This includes measurement of the guest system and the SEV interface, which can be matched against precalculated or previously recorded values. This usually consists of two parts: 1. Executing measurement and generating an attestation report from within the virtual machine. 2. Calculating reference measurement values outside of the virtual machine for comparison. The latter is optional and provides an additional layer of security for ensuring guest system integrity. Generating an attestation report from within a virtual machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Requirements: - ``msr`` kernel module - ``sev-guest`` kernel module, often part of ``linux-modules-extra-*-generic`` - ``snpguest`` tool, see :ref:`getting-the-snpguest-tool` Refer to the corresponding for acquiring the ``snpguest`` tool. Make sure the kernel modules are loaded: .. code:: sh # check if loaded lsmod | grep "sev\|msr" # load any missing modules sudo modprobe msr sudo modprobe sev-guest Then proceed with generating the attestation report. First, check the status of the guest: .. code:: sh snpguest ok Then, generate the report: .. code:: sh snpguest report report.bin request-file.txt -r The report can be displayed with: .. code:: sh snpguest display report report.bin Verifying an attestation report from within a virtual machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The attestation report depends on a chain of certificates. To verify the certificate trust chain, some certificates must be retrived first: 1. The AMD certificate chain consisting of ARK (AMD Root Key) and ASK (AMD SEV Key). 2. The VCEK (Versioned Chip Endorsement Key). The AMD certificate chain can be retrieved from the KDS (Key Distribution System) interface: .. code:: sh snpguest fetch ca -r report.bin pem ./ This reads the ``report.bin`` to identify the correct CPU model and retrieve the corresponding certificate. Alternatively, ``snpguest fetch ca pem ./ milan`` may be used without a report by specifying a CPU family (``milan`` in this case). The VCEK (Versioned Chip Endorsement Key) is embedded into the processor and can also be retrieved from the KDS (Key Distribution System) interface: .. code:: sh snpguest fetch vcek pem ./ ./report.bin Using both the CA chain and the VCEK, the full chain can now be verified: .. code:: sh snpguest verify certs ./ The AMD ARK was self-signed! The AMD ASK was signed by the AMD ARK! The VCEK was signed by the AMD ASK! The above example shows the expected output. The ARK (AMD Root Key) is self-signed as it is the root key. It signs the ASK (AMD SEV Key), which in turn signs the VCEK. Lastly, the attestation report is signed by the VCEK. Finally, the attestation report itself can be verified: .. code:: sh snpguest verify attestation ./ ./report.bin Reported TCB Boot Loader from certificate matches the attestation report. Reported TCB TEE from certificate matches the attestation report. Reported TCB SNP from certificate matches the attestation report. Reported TCB Microcode from certificate matches the attestation report. VEK signed the Attestation Report! Notes: - The measurement depends on the CPU-type (EPYC, EPYC-Milan, etc.), but subtypes don’t matter (e.g. EPYC-Milan-v1 vs EPYC-Milan-v2). - Repeating the measurement within a different virtual machine under the same conditions (vCPU count, images used, physical host) will lead to the same result values and as such is deterministic. .. _amd-sev-attestation-calculate-offline-measurement: Calculating reference measurement outside of a virtual machine ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To verify that the guest system of a virtual machine has not been tampered with and the expected kernel is running with the correct boot arguments, an offline measurement can be calculated outside of the virtual machine for the purpose of comparing the resulting values. The expected measurement calculation should be done on another secure system. To calculate the measurement, the following assets and values must be acquired beforehand: - CPU model family, example: ``EPYC-Milan`` - can be retrieved from within the virtual machine via ``lscpu`` or ``/proc/cpuinfo`` - if enabled by the infrastructure provider, it may be visible as the ``ATTESTATION:vcpu_model`` attribute [1]_ within the "properties" field of ``openstack server show`` (this requires a specific YAOOK configuration as per :ref:`amd-sev-enable-attestation-metadata-visibility`) - number of vCPU cores - can be retrieved from the OpenStack flavor using ``openstack flavor show`` - OVMF firmware file - in case of YAOOK, it can be extracted from the ``nova-compute`` container image, see :ref:`amd-sev-extracting-ovmf-from-compute-image` If direct kernel boot is used, the following is additionally required: - kernel image file - can be retrieved using ``openstack image save`` on the image with the ID referenced as ``kernel_id`` on the main image - initrd image file - can be retrieved using ``openstack image save`` on the image with the ID referenced as ``ramdisk_id`` on the main image - kernel boot parameters (cmdline) - for direct kernel boot images, can be retrieved from the ``os_command_line`` property of the main image .. **Note:** This kind of attestation is primarily useful for direct kernel boot scenarios where kernel and initrd images are immutable separate from the main guest image. This ensures that the kernel cannot be changed from within the guest and an attestation measurement always matches the precalculated one. Using the tool `sev-snp-measure `__, a measurement value can be calculated offline independent of the SEV virtual machine or host, by supplying the correct data: .. code:: sh ./sev-snp-measure.py --mode snp \ --vcpus=2 \ --vcpu-type=EPYC-Milan \ --ovmf OVMF.amdsev.fd \ --kernel vmlinuz-6.12.41+deb13-cloud-amd64.vmlinuz \ --initrd initrd.img-6.12.41+deb13-cloud-amd64.initrd \ --append "root=/dev/vda1 console=ttyS0" (omit the parameters ``--kernel``, ``--initrd`` and ``--append`` if direct kernel boot is not used) The resulting measurement value is a long string in hexadecimal encoding. The representation of the measurement value via ``snpguest display report`` from within the virtual machine uses a different encoding. In order to properly compare the values, the following can be used from within the virtual machine: .. code:: sh snpguest display report report.bin \ | grep -A3 Measurement | tail -n3 | \ tr -d '\n ' | tr '[:upper:]' '[:lower:]'; \ echo This will output the same format as the ``sev-snp-measure`` command and the resulting strings should be identical. This way, a measurement value can be calculated in a secure environment and compared against the measurement results of an actual virtual machine. Appendix -------- .. _extracting-kernel-and-initrd-images: Extracting kernel and init ramdisk from an image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To extract the kernel and initrd images contained in a regular Linux image, the ``guestmount`` tool may be used, which is often part of the “libguestfs-tools” package or a dedicated package called “guestmount”. For example: .. code:: sh guestmount --ro -a debian-13-genericcloud-amd64-20250814-2204.qcow2 -i /mnt # discover the filenames ls /mnt/boot/ # copy the files cp /mnt/boot/vmlinuz-6.12.41+deb13-cloud-amd64 . cp /mnt/boot/initrd.img-6.12.41+deb13-cloud-amd64 . guestunmount /mnt (``guestmount`` will likely require root privileges) The above example uses a specific Debian 13 cloud image as an example. The exact names and paths to the kernel and ramdisk files will differ based on the Linux image used. .. _getting-the-snpguest-tool: Getting the snpguest tool ~~~~~~~~~~~~~~~~~~~~~~~~~ Download the binary from https://github.com/virtee/snpguest For example: .. code:: sh wget https://github.com/virtee/snpguest/releases/download/v0.9.2/snpguest sudo mv snpguest /usr/local/bin/snpguest sudo chmod +x /usr/local/bin/snpguest Or build the binary from source. Note: Version 0.9.1 has a bug breaking signature verification, make sure to use >= 0.9.2. .. _amd-sev-extracting-ovmf-from-compute-image: Extracting the OVMF firmware file from YAOOK ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The AMD SEV OVMF file is built directly into the nova-compute image of YAOOK. Since YAOOK is open source, the file can be extracted from the downloaded container image: .. code:: sh docker pull registry.yaook.cloud/yaook/nova-compute-2024.2-ubuntu:$VERSION docker run --name temp-ovmf-extract registry.yaook.cloud/yaook/nova-compute-2024.2-ubuntu:$VERSION docker cp temp-ovmf-extract:/usr/share/OVMF/OVMF_AMDSEV_4M.fd ./OVMF_AMDSEV_4M.fd docker rm temp-ovmf-extract For determining ``$VERSION``, please refer to `pinned_version.yml `__ of the appropriate YAOOK release. .. [1] Note that in case a Nova compute host is configured with host or host- passthrough mode for the virtualized CPU model, the value of ``ATTESTATION:vcpu_model`` will simply be ``host`` and the actual CPU model name must be retrieved via other means. This is a limitation of Nova not enumerating the model for passthrough modes.