Releases and Versioning ======================= .. note:: This is our current concept and not yet fully implemented. The dedicated tests for ``release`` and ``stable`` builds are missing. In order to ensure that we do not ship broken things we need to test all changes before releasing them. Requirements ------------ * Guarantee stability by testing the entire system before releasing it * Support new OpenStack versions quickly after their release * Release new features and fixes in a regular and fast cycle * Be able to hotfix old stable releases (this does not mean that we support every old release forever) * Be able to fix critical bugs/security issues within a few hours while not sacrificing stability Versioning Overview ------------------- Yaook is developed and versioned according to cycles. Each development cycle should last about one to two weeks. During each cycle contributors can merge changes to the main branch of the Operator repository. When the cycle ends all changes on the main branch are merged to a ``rolling`` branch. The goal of the ``rolling`` branch is to have a chance to test changes in a stable and controlled way. Ideally no changes are made to the ``rolling`` branch anymore. Only necessary hotfixes are allowed. The ``rolling`` branch is kept like that until the next cycle ends at which point it merges to ``stable`` and gets a final version number. Images are automatically versioned with distinct version numbers for each merged MR. The connection between the version of the image and the Operator that deploys this image is handled in the Operator repository. The Operator defines for each image the tag which should be used. Detailed branching model for the Operator ***************************************** Each (pre-)release of Yaook :math:`R` supports :math:`N` consecutive releases of OpenStack denoted as :math:`V_1 \ldots V_N` which are effecting the tests below. The repositories are structured in 4 branches: devel The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. External Dependencies (like our Docker images) are pinned to specific versions. When developing on Yaook this should be your base branch. You should not run any useful deployment from here. rolling Roughly every one to two weeks the current state of devel is merged to the rolling branch. During this merge and after any change to the branch the Operator is built and versioned with a rolling version. To be merged to this branch the code has to pass the following tests across all repositories: * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` must pass a full Tempest run * for each :math:`k = 1, \ldots, N-1` do a full deployment with OpenStack release :math:`V_k` and upgrade it to release :math:`V_{k+1}`. The installation must pass a full Tempest test. * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` from rolling version :math:`R-1` is upgraded to version :math:`R` and must then pass a full Tempest run This version should already be stable enough that it is usable for non-prod use cases. stable The rolling branch is merged to the stable branch at the end of each one to two week period (directly before rolling is set to devel). To be merged to stable we run the following tests: * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` on bare-metal must pass a full Tempest run * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` form stable version :math:`R-1` is upgraded to version :math:`R` and must then pass a full Tempest run This version is basically the same as the rolling one but accounts for two additional things: * its not realistic for everyone to upgrade every day. So we need a version with a little longer release cycle * it allows for some manual tests by users running a local rolling testcluster stable-$Major.$Minor This branch is created when a change is merged from ``rolling`` to ``stable``. It points to the ``stable`` commit for this version and is used to build the Operator image and build hotfixes against old versions. Detailed branching model for the images *************************************** The image repositories are using a single branch by default: devel The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. When developing on Yaook this should be your base branch. The Operator is refering to specific versions/commits on this branch when using an image. If a hotfix would need to be created, an additional branch can be created based on the specific commit in devel. Versioning specification ************************ In combination with the above we define the following versioning scheme following the SemVer concept (https://semver.org/). ``stable`` is hereby using the plain ``Major.Minor.Patch`` versioning: * We increment Major if we have a incompatible change. This number is defined to be 0 during our current development. * We increment Minor every time we merge ``rolling`` into ``stable``. This number is the releasedate in the format "YYYYMMDD". * We increment Patch for every hotfix we merge (see below). It starts from 0 for each normal release. For the following examples we define ``stable`` as having the version ``X.Y.Z``. As ``rolling`` will become the next version it will use ``X.(Y+1).0-rolling.W``. Hereby ``W`` is the build number being incremented each time during the nightly builds. As ``devel`` is always containing the current development version we just use ``devel`` as a version. nova-compute images can get an additional "evacuation serial number" that allows the Operator to determine if a given hotfix needs to evacuate all VMs before applying it. The number is set to 0 on each new Minor release and incremented each time a hotfix is merged that requires a evacuation. The Operator can then determine if the previous serial is different from the new one and issue an evacuation if this is the case. The serial is added to the image as a build metadata, e.g. ``1.0.2+evacuation1``. If the serial is missing it is assumed to require evacuation. Hotfix process for Operator *************************** We might from time to time need to build hotfixes for the Operator. To do this we follow the following process: 1. Create a branch of the merge-base of ``stable`` and ``devel`` into ``hotfix/base/$name`` and create the fix. 2. Create a branch of ``devel`` named ``hotfix/devel/$name`` and merge ``hotfix/base/$name`` into there. Create and merge a MR to ``devel``. 3. Create a branch of ``rolling`` named ``hotfix/rolling/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``rolling``. We will bump ``W`` automatically. 4. Create a branch of ``stable`` named ``hotfix/stable/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``stable``. We will bump ``Z`` automatically. 5. For each old release needing this: Create a branch of ``stable-`` named ``hotfix/stable-/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``stable``. We will bump ``Z`` automatically. Each commit must contain a reference to the original issue using `Hotfix-For: #$issueid` to help with transparency. This process ensures that each hotfix has run through the normal validation pipeline and we can consider it stable. .. note:: If a hotfix is only relevant for an old version, then create a MR again ``stable-`` directly and skip the other steps. Hotfix process for images ************************* As the images are pinned in the Operator a change on the image side does not directly affect the Operators. To create a hotfix follow the following process: 1. Create a branch of the relevant commit at the image repository. 2. Update the Dockerfile of the image to refer to the existing image tag. 3. Develop the bugfix. 4. Push the changes and let the image build run. At this point the image can already be used if the user sets ``YAOOK_OP_VERSIONS_OVERRIDE``. 5. Potentially also merge the changes to ``devel`` if the fix is also needed for the current version. 6. Build a hotfix to the Operator repo as described above and updated the version pins. Graphical example ***************** Below you can find a overview over the process for normal releases and for hotfixes. .. image:: figures/release_operator_versions.drawio.svg Practical implementation ------------------------ The following describes the practical implementation of these concepts across Yaook. Operator implementation *********************** The release pipeline of the Operator repositories are following these steps: * create a ``stable-preparation-`` branch based off ``stable`` * merge ``rolling`` to this branch * create a MR from this branch to ``stable`` and set it to merge when pipeline succeeds * create a ``rolling-preparation-`` branch based off ``rolling`` * merge ``devel`` to this branch * create a MR from this branch to ``rolling`` and set it to merge when pipeline succeeds The MR pipelines just grabs the code of the branch, builds it and runs the tests defined above. The pipeline for the ``stable`` branch does the following: * Build the Operator, tag it with ``$Major.$Minor.0-base`` and push it to the repository * Checkout a new branch named ``stable-`` * Pin the Operator dependencies and add the ``yaook/assets/pinned_version.yml`` to git * Create a ``stable-Dockerfile`` using the newly pushed version as a base and adding a layer to copy the ``yaook/assets/pinned_version.yml`` to the image * Commit this change to the new branch and push The pipeline for ``stable-`` does the following: * find the appropriate version tag by searching for previous versions * Build the Operator using the ``stable-Dockerfile``, tag it with the appropriate version and push it The pipeline for ``rolling`` does the following: * find the appropriate version tag by searching for previous versions * Pin the Operator dependencies * Update the Dockerfile to copy ``yaook/assets/pinned_version.yml`` to the image * build the Operator, tag it with the version tag found in the first step and push it Docker image implementation *************************** The image pipeline for ``devel`` does the following: * build the image with the version specified in the ``version`` file * bump the version and commit the new version to the repo If the version change in the ``version`` file does not fit your requirements you can update the file during your normal MR. Updating -------- The Operator generally needs to be updated for each Major and Minor version change. Updating hereby means to use whatever method we define to release Yaook to obtain a newer version and push it to the Kubernetes cluster. The user is hereby responsible to only follow supported upgrade paths (only one Minor version jump at a time). For Patch releases it might be possible in the future to enable auto-updating. This causes the Operator to regularly check for upstream image updates and apply them automatically. Changes in this category normaly include critical security updates. Alternatively the user can choose to not auto-update but follow the normal update process to get the change at a user defined time. In all cases the Operator takes care that resources are properly evacuated before updating (if necessary). When only updating the Patch release of images the Operator will do the update without evacuating services (if possible, e.g. in the case of VMs). This allows to rollout to complete a lot faster than a traditional update but is only suitable for small and precise changes.