Releases and Versioning ======================= .. note:: This is our current concept and not yet fully implemented. The dedicated tests for ``release`` and ``stable`` builds are missing. In order to ensure that we do not ship broken things we need to test all changes before releasing them. Requirements ------------ * Guarantee stability by testing the entire system before releasing it * Support new OpenStack versions quickly after their release * Release new features and fixes in a regular and fast cycle * Be able to hotfix old stable releases (this does not mean that we support every old release forever) * Be able to fix critical bugs/security issues within a few hours while not sacrificing stability Versioning Overview ------------------- Yaook is developed and versioned according to cycles. Each development cycle should last about one to two weeks. During each cycle contributors can merge changes to the main branch (``devel``) of the Operator repository. When the cycle ends all changes on the main branch are merged to a ``release-prepare/v$Major.$Minor.$Patch`` branch. The goal of the ``release-prepare/v$Major.$Minor.$Patch`` branch is to have a chance to test changes in a stable and controlled way. Ideally no changes are made to the ``release-prepare/v$Major.$Minor.$Patch`` branch anymore. Only necessary hotfixes are allowed. The ``release-prepare/v$Major.$Minor.$Patch`` branch merges to ``stable`` and finally gets released after the pipeline finished. Images are automatically versioned with distinct version numbers for each merged MR. The connection between the version of the image and the Operator that deploys this image is handled in the Operator repository. The Operator defines for each image the tag which should be used. Operator information ******************** Detailed branching model ~~~~~~~~~~~~~~~~~~~~~~~~ Each (pre-)release of Yaook :math:`R` supports :math:`N` consecutive releases of OpenStack denoted as :math:`V_1 \ldots V_N` which are effecting the tests below. The repository is structured in four branch-types: ``devel`` The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. External Dependencies (like our Docker images) are pinned to specific versions. When developing on Yaook this should be your base branch. You should not run any useful deployment from here. ``feature`` All changes are developed in feature or fix-branches (the name is free), branching off and merging back to devel and therefore passing the beforementioned tests. ``release-prepare/v$Major.$Minor.$Patch`` Roughly every one to two weeks the current state of ``devel`` is pushed into a new ``release-prepare/v$Major.$Minor.$Patch``-branch. During this process and after any change to the branch the Operator is built and versioned as a pre-release. On this branch the code has to pass the following tests across all repositories: * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` must pass a full Tempest run * for each :math:`k = 1, \ldots, N-1` do a full deployment with OpenStack release :math:`V_k` and upgrade it to release :math:`V_{k+1}`. The installation must pass a full Tempest test. * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` from rolling version :math:`R-1` is upgraded to version :math:`R` and must then pass a full Tempest run This pre-release version should already be stable enough that it is usable for non-prod use cases. ``stable`` The ``release-prepare/v$Major.$Minor.$Patch``-branch is merged to the ``stable`` branch after at most one week. To be merged to stable we run the following tests: * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` on bare-metal must pass a full Tempest run * a full deployment of all supported OpenStack releases :math:`V_1 \ldots V_N` form stable version :math:`R-1` is upgraded to version :math:`R` and must then pass a full Tempest run This version is basically the same as the pre-release one but accounts for two additional things: * its not realistic for everyone to upgrade every day. So we need a version with a little longer release cycle * it allows for some manual tests by users running a local pre-release testcluster .. TODO: add image Versioning specification ~~~~~~~~~~~~~~~~~~~~~~~~ In combination with the above we define the following versioning scheme following the SemVer concept (https://semver.org/). ``stable`` is hereby using the plain ``$Major.$Minor.$Patch`` versioning: * We increment ``$Major`` by 1 if we have a incompatible change. * We increment ``$Minor`` by 1 every time we added at least one new feature in the release. It starts from 0 for each major release. * We increment ``$Patch`` by 1 for every release not introducing a new feature. It starts from 0 for each normal release. .. TODO: additionally: prereleases, alpha, dev (helm, images)? Hotfix process ~~~~~~~~~~~~~~ .. note:: This section needs adjustments to our changed release process. We might from time to time need to build hotfixes for the Operator. To do this we follow the following process: 1. Create a branch of the merge-base of ``stable`` and ``devel`` into ``hotfix/base/$name`` and create the fix. 2. Create a branch of ``devel`` named ``hotfix/devel/$name`` and merge ``hotfix/base/$name`` into there. Create and merge a MR to ``devel``. 3. Create a branch of ``rolling`` named ``hotfix/rolling/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``rolling``. We will bump ``W`` automatically. 4. Create a branch of ``stable`` named ``hotfix/stable/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``stable``. We will bump ``Z`` automatically. 5. For each old release needing this: Create a branch of ``stable-`` named ``hotfix/stable-/$name`` and cherry pick ``hotfix/base/$name`` into there. Create and merge a MR to ``stable``. We will bump ``Z`` automatically. Each commit must contain a reference to the original issue using `Hotfix-For: #$issueid` to help with transparency. This process ensures that each hotfix has run through the normal validation pipeline and we can consider it stable. .. note:: If a hotfix is only relevant for an old version, then create a MR again ``stable-`` directly and skip the other steps. .. _release-and-versioning.operator-implementation: Practical implementation ~~~~~~~~~~~~~~~~~~~~~~~~ .. TODO: check whether the old information is still correct The release pipeline (in rdm) of the yaook/operator repository is following these steps: - create a ``release-prepare/v$Major.$Minor.$Patch`` branch based off ``stable`` - merge ``devel`` to this branch - on the branch do the following: - calculate the next version number based on the provided releasenote-files since the last release and write it to the version-file - generate the changelog using towncrier and remove the old releasenote-files The pipeline for the ``release-prepare/v$Major.$Minor.$Patch``-branch does the following: - finds the appropriate version tag by searching for previous versions - pins the Operator dependencies - updates the Dockerfile to copy ``yaook/assets/pinned_version.yml`` to the image - builds the Operator - creates a delayed job (one week) which - merges to ``stable`` - triggers a MR back to ``devel`` The pipeline for the ``stable``-branch - builds the Operator, tag it with ``$Major.$Minor.$Patch`` and push it to the repository - pins the Operator dependencies and add the ``yaook/assets/pinned_version.yml`` to git - creates a ``stable-Dockerfile`` using the newly pushed version as a base and adding a layer to copy the ``yaook/assets/pinned_version.yml`` to the image - commits this change to the new branch and push - tags the release with ``v$Major.$Minor.$Patch`` and - creates a `Gitlab Release `__. .. TODO: The pipeline on ``devel`` does ... The MR pipelines just grab the code of the branches, build it and run the tests defined above. Image information ***************** Detailed branching model ~~~~~~~~~~~~~~~~~~~~~~~~ The image repositories are using a single branch by default: devel The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. When developing on Yaook this should be your base branch. The Operator is refering to specific versions/commits on this branch when using an image. If a hotfix would need to be created, an additional branch can be created based on the specific commit in devel. Versioning specification ~~~~~~~~~~~~~~~~~~~~~~~~ The versions for the images also follow (https://semver.org/) mainly using the plain ``$Major.$Minor.$Patch``. nova-compute images can get an additional "evacuation serial number" that allows the Operator to determine if a given hotfix needs to evacuate all VMs before applying it. The number is set to 0 on each new Minor release and incremented each time a hotfix is merged that requires a evacuation. The Operator can then determine if the previous serial is different from the new one and issue an evacuation if this is the case. The serial is added to the image as a build metadata, e.g. ``1.0.2+evacuation1``. If the serial is missing it is assumed to require evacuation. Hotfix process ~~~~~~~~~~~~~~ As the images are pinned in the Operator a change on the image side does not directly affect the Operators. To create a hotfix follow the following process: 1. Create a branch of the relevant commit at the image repository. 2. Update the Dockerfile of the image to refer to the existing image tag. 3. Develop the bugfix. 4. Push the changes and let the image build run. At this point the image can already be used if the user sets ``YAOOK_OP_VERSIONS_OVERRIDE``. 5. Potentially also merge the changes to ``devel`` if the fix is also needed for the current version. 6. Build a hotfix to the Operator repo as described above and updated the version pins. Docker image implementation ~~~~~~~~~~~~~~~~~~~~~~~~~~~ The image pipeline for ``devel`` does the following: * build the image with the version specified in the ``version`` file * bump the version and commit the new version to the repo If the version change in the ``version`` file does not fit your requirements you can update the file during your normal MR. Updating -------- The Operator generally needs to be updated for each Major and Minor version change. Updating hereby means to use whatever method we define to release Yaook to obtain a newer version and push it to the Kubernetes cluster. The user is hereby responsible to only follow supported upgrade paths (only one Minor version jump at a time). For Patch releases it might be possible in the future to enable auto-updating. This causes the Operator to regularly check for upstream image updates and apply them automatically. Changes in this category normaly include critical security updates. Alternatively the user can choose to not auto-update but follow the normal update process to get the change at a user defined time. In all cases the Operator takes care that resources are properly evacuated before updating (if necessary). When only updating the Patch release of images the Operator will do the update without evacuating services (if possible, e.g. in the case of VMs). This allows to rollout to complete a lot faster than a traditional update but is only suitable for small and precise changes.