Releases and Versioning

Note

This is our current concept and not yet fully implemented. The dedicated tests for release and stable builds are missing.

In order to ensure that we do not ship broken things we need to test all changes before releasing them.

Requirements

Guarantee stability by testing the entire system before releasing it
Support new OpenStack versions quickly after their release
Release new features and fixes in a regular and fast cycle
Be able to hotfix old stable releases (this does not mean that we support every old release forever)
Be able to fix critical bugs/security issues within a few hours while not sacrificing stability

Versioning Overview

Yaook is developed and versioned according to cycles. Each development cycle should last about one to two weeks. During each cycle contributors can merge changes to the main branch of the Operator repository.

When the cycle ends all changes on the main branch are merged to a rolling branch. The goal of the rolling branch is to have a chance to test changes in a stable and controlled way. Ideally no changes are made to the rolling branch anymore. Only necessary hotfixes are allowed.

The rolling branch is kept like that until the next cycle ends at which point it merges to stable and gets a final version number.

Images are automatically versioned with distinct version numbers for each merged MR. The connection between the version of the image and the Operator that deploys this image is handled in the Operator repository. The Operator defines for each image the tag which should be used.

Detailed branching model for the Operator

Each (pre-)release of Yaook $R$ supports $N$ consecutive releases of OpenStack denoted as $V_1 \ldots V_N$ which are effecting the tests below. The repositories are structured in 4 branches:

devel

The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. External Dependencies (like our Docker images) are pinned to specific versions. When developing on Yaook this should be your base branch. You should not run any useful deployment from here.

rolling

Roughly every one to two weeks the current state of devel is merged to the rolling branch. During this merge and after any change to the branch the Operator is built and versioned with a rolling version. To be merged to this branch the code has to pass the following tests across all repositories:

a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ must pass a full Tempest run
for each $k = 1, \ldots, N-1$ do a full deployment with OpenStack release $V_k$ and upgrade it to release $V_{k+1}$. The installation must pass a full Tempest test.
a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ from rolling version $R-1$ is upgraded to version $R$ and must then pass a full Tempest run

This version should already be stable enough that it is usable for non-prod use cases.

stable

The rolling branch is merged to the stable branch at the end of each one to two week period (directly before rolling is set to devel). To be merged to stable we run the following tests:

a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ on bare-metal must pass a full Tempest run
a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ form stable version $R-1$ is upgraded to version $R$ and must then pass a full Tempest run

This version is basically the same as the rolling one but accounts for two additional things:

its not realistic for everyone to upgrade every day. So we need a version with a little longer release cycle
it allows for some manual tests by users running a local rolling testcluster

stable-$Major.$Minor

This branch is created when a change is merged from rolling to stable. It points to the stable commit for this version and is used to build the Operator image and build hotfixes against old versions.

Detailed branching model for the images

The image repositories are using a single branch by default:

devel: The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. When developing on Yaook this should be your base branch. The Operator is refering to specific versions/commits on this branch when using an image.

If a hotfix would need to be created, an additional branch can be created based on the specific commit in devel.

Versioning specification

In combination with the above we define the following versioning scheme following the SemVer concept (https://semver.org/). stable is hereby using the plain Major.Minor.Patch versioning:

We increment Major if we have a incompatible change. This number is defined to be 0 during our current development.
We increment Minor every time we merge rolling into stable. This number is the releasedate in the format “YYYYMMDD”.
We increment Patch for every hotfix we merge (see below). It starts from 0 for each normal release.

For the following examples we define stable as having the version X.Y.Z.

As rolling will become the next version it will use X.(Y+1).0-rolling.W. Hereby W is the build number being incremented each time during the nightly builds.

As devel is always containing the current development version we just use devel as a version.

nova-compute images can get an additional “evacuation serial number” that allows the Operator to determine if a given hotfix needs to evacuate all VMs before applying it. The number is set to 0 on each new Minor release and incremented each time a hotfix is merged that requires a evacuation. The Operator can then determine if the previous serial is different from the new one and issue an evacuation if this is the case. The serial is added to the image as a build metadata, e.g. 1.0.2+evacuation1. If the serial is missing it is assumed to require evacuation.

Hotfix process for Operator

We might from time to time need to build hotfixes for the Operator. To do this we follow the following process:

Create a branch of the merge-base of stable and devel into hotfix/base/$name and create the fix.
Create a branch of devel named hotfix/devel/$name and merge hotfix/base/$name into there. Create and merge a MR to devel.
Create a branch of rolling named hotfix/rolling/$name and cherry pick hotfix/base/$name into there. Create and merge a MR to rolling. We will bump W automatically.
Create a branch of stable named hotfix/stable/$name and cherry pick hotfix/base/$name into there. Create and merge a MR to stable. We will bump Z automatically.
For each old release needing this: Create a branch of stable-<oldversion> named hotfix/stable-<oldversion>/$name and cherry pick hotfix/base/$name into there. Create and merge a MR to stable. We will bump Z automatically.

Each commit must contain a reference to the original issue using Hotfix-For: #$issueid to help with transparency.

This process ensures that each hotfix has run through the normal validation pipeline and we can consider it stable.

Note

If a hotfix is only relevant for an old version, then create a MR again stable-<oldversion> directly and skip the other steps.

Hotfix process for images

As the images are pinned in the Operator a change on the image side does not directly affect the Operators. To create a hotfix follow the following process:

Create a branch of the relevant commit at the image repository.
Update the Dockerfile of the image to refer to the existing image tag.
Develop the bugfix.
Push the changes and let the image build run. At this point the image can already be used if the user sets YAOOK_OP_VERSIONS_OVERRIDE.
Potentially also merge the changes to devel if the fix is also needed for the current version.
Build a hotfix to the Operator repo as described above and updated the version pins.

Graphical example

Below you can find a overview over the process for normal releases and for hotfixes.

../_images/release_operator_versions.drawio.svg

Practical implementation

The following describes the practical implementation of these concepts across Yaook.

Operator implementation

The release pipeline of the Operator repositories are following these steps:

create a stable-preparation-<newversion> branch based off stable
merge rolling to this branch
create a MR from this branch to stable and set it to merge when pipeline succeeds
create a rolling-preparation-<newversion> branch based off rolling
merge devel to this branch
create a MR from this branch to rolling and set it to merge when pipeline succeeds

The MR pipelines just grabs the code of the branch, builds it and runs the tests defined above.

The pipeline for the stable branch does the following:

Build the Operator, tag it with $Major.$Minor.0-base and push it to the repository
Checkout a new branch named stable-<currentversion_only_major_and_minor>
Pin the Operator dependencies and add the yaook/assets/pinned_version.yml to git
Create a stable-Dockerfile using the newly pushed version as a base and adding a layer to copy the yaook/assets/pinned_version.yml to the image
Commit this change to the new branch and push

The pipeline for stable-<version> does the following:

find the appropriate version tag by searching for previous versions
Build the Operator using the stable-Dockerfile, tag it with the appropriate version and push it

The pipeline for rolling does the following:

find the appropriate version tag by searching for previous versions
Pin the Operator dependencies
Update the Dockerfile to copy yaook/assets/pinned_version.yml to the image
build the Operator, tag it with the version tag found in the first step and push it

Docker image implementation

The image pipeline for devel does the following:

build the image with the version specified in the version file
bump the version and commit the new version to the repo

If the version change in the version file does not fit your requirements you can update the file during your normal MR.

Updating

The Operator generally needs to be updated for each Major and Minor version change. Updating hereby means to use whatever method we define to release Yaook to obtain a newer version and push it to the Kubernetes cluster. The user is hereby responsible to only follow supported upgrade paths (only one Minor version jump at a time).

For Patch releases it might be possible in the future to enable auto-updating. This causes the Operator to regularly check for upstream image updates and apply them automatically. Changes in this category normaly include critical security updates. Alternatively the user can choose to not auto-update but follow the normal update process to get the change at a user defined time. In all cases the Operator takes care that resources are properly evacuated before updating (if necessary).

When only updating the Patch release of images the Operator will do the update without evacuating services (if possible, e.g. in the case of VMs). This allows to rollout to complete a lot faster than a traditional update but is only suitable for small and precise changes.