"""""""""""""""""
vGPU Support
"""""""""""""""""

In this guide we will show you how to set up Yaook with vGPU Support?

Requirements
--------------------
To proceed this guide make sure that the following prerequisites are met:

- INTEL CPU / AMD CPU
- NVIDIA GPU which supports vGPU

A complete list can be found :doc:`here<vgpu_types>`


Architecture Graph
--------------------
.. image:: figures/yaook_vgpu_simplified.drawio.svg

1. If you want to deploy your kubernetes cluster via ``yaook/k8s`` you need to enable the vgpu support. The following variables must be set in ``config.toml``. The template ``config.template.toml`` is located under ``yaook/k8s/templates/config.template.toml``.
Note: ``Yaook/k8s`` is not a strict requirement for ``yaook/operator``.

        .. code-block:: bash

            # vGPU Support
            [nvidia.vgpu]
            driver_blob_url = #vGPU Manager location
            manager_filename = #vGPU Manager file

2. In Openstack, the Nova compute configuration must be customized. This requires the name of the folder with the selected vGPU configuration, in which the UUID (universally unique identifier) for the vGPU is created. A distinction must be made between 2 cases.


    a) GPU without SR-IOV support

        1. Physical GPUs supporting virtual GPUs propose mediate device types (mdev). You still need to get the right PCI port, in which the GPU is plugged in. A list with  all vGPU types can be found :doc:`here<vgpu_types>`.

            .. code-block:: bash

                $ lspci | grep NVIDIA
                82:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)

        2. Change to the ``mdev_supported_types`` directory on which you want to create the vGPU and find the subdirectory, that contains your chosen vGPU configuration. Replace ``"vgpu-type"`` with your chosen vGPU configuration.
            .. code-block:: bash

                $ grep -l "vgpu-type" nvidia-*/name

    b) GPU with SR-IOV support (Ampere architecture and newer)

        1. Obtain the bus , domain, slot and function of the available virtual funcitons on the GPU

            .. code-block:: bash

                $ ls -l /sys/bus/pci/devices/domain\:bus\:slot.function/ | grep virtfn

        2. This example shows the output of this command for a physical GPU with the slot 00, bus 82, domain 0000 and function 0.

            .. code-block:: bash

                $ ls -l /sys/bus/pci/devices/0000:82:00.0/ | grep virtfn
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn0 -> ../0000:82:00.4
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn1 -> ../0000:82:00.5
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn10 -> ../0000:82:01.6
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn11 -> ../0000:82:01.7
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn12 -> ../0000:82:02.0
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn13 -> ../0000:82:02.1
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn14 -> ../0000:82:02.2
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn15 -> ../0000:82:02.3
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn2 -> ../0000:82:00.6
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn3 -> ../0000:82:00.7
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn4 -> ../0000:82:01.0
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn5 -> ../0000:82:01.1
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn6 -> ../0000:82:01.2
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn7 -> ../0000:82:01.3
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn8 -> ../0000:82:01.4
                lrwxrwxrwx 1 root root           0 Jul 25 07:57 virtfn9 -> ../0000:82:01.5

        3. Choose the virtual function on which you want to create the vGPU. A list with  all vGPU types can be found :doc:`here<vgpu_types>`.

        .. code-block:: bash

            $ cd /sys/class/mdev_bus/0000\:82\:00.4/mdev_supported_types/
            $ grep -l "vgpu-type" nvidia-*/name

3. The most important variable to set is the `enable_vgpu_types` in the `nova.yaml`. The file is located under `/docs/examples/nova.yaml`. Here, you decide the given vGPU configuration considering the acquired license. Replace ``nvidia-233`` with the folder name containing your chosen vGPU configuration.

    .. code-block:: bash

        compute:
          configTemplates:
          - nodeSelectors:
            - matchLabels: {}
            novaComputeConfig:
              DEFAULT:
                debug: True
              devices:
                enabled_vgpu_types:
                - nvidia-233

4. The last step is to configure a flavor to request one virtual GPU.

    .. code-block:: bash

          $ openstack flavor set <FLAVOR_VGPU> --property "resources:VGPU=1"