Releases and Versioning¶

Note

This is our current concept and not yet fully implemented. The dedicated tests for release and stable builds are missing.

In order to ensure that we do not ship broken things we need to test all changes before releasing them.

Requirements¶

Guarantee stability by testing the entire system before releasing it
Support new OpenStack versions quickly after their release
Release new features and fixes in a regular and fast cycle
Be able to hotfix old stable releases (this does not mean that we support every old release forever)
Be able to fix critical bugs/security issues within a few hours while not sacrificing stability

Versioning Overview¶

Yaook is developed and versioned according to cycles. Each development cycle should last about one to two weeks. During each cycle contributors can merge changes to the main branch (devel) of the Operator repository.

When the cycle ends all changes on the main branch are merged to a release-prepare/v$Major.$Minor.$Patch branch. The goal of the release-prepare/v$Major.$Minor.$Patch branch is to have a chance to test changes in a stable and controlled way. Ideally no changes are made to the release-prepare/v$Major.$Minor.$Patch branch anymore. Only necessary hotfixes are allowed.

The release-prepare/v$Major.$Minor.$Patch branch merges to stable and finally gets released after the pipeline finished.

Images are automatically versioned with distinct version numbers for each merged MR. The connection between the version of the image and the Operator that deploys this image is handled in the Operator repository. The Operator defines for each image the tag which should be used.

Operator information¶

Detailed branching model¶

Each (pre-)release of Yaook $R$ supports $N$ consecutive releases of OpenStack denoted as $V_1 \ldots V_N$ which are effecting the tests below. The repository is structured in four branch-types:

devel

The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. External Dependencies (like our Docker images) are pinned to specific versions. When developing on Yaook this should be your base branch. You should not run any useful deployment from here.

feature

All changes are developed in feature or fix-branches (the name is free), branching off and merging back to devel and therefore passing the beforementioned tests.

release-prepare/v$Major.$Minor.$Patch

Roughly every one to two weeks the current state of devel is pushed into a new release-prepare/v$Major.$Minor.$Patch-branch. During this process and after any change to the branch the Operator is built and versioned as a pre-release. On this branch the code has to pass the following tests across all repositories:

a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ must pass the integration-test checks. This check is included into the following upgrade test.
for each $k = 1, \ldots, N-1$ do a full deployment with OpenStack release $V_k$ and upgrade it to release $V_{k+1}$. The installation must pass the integration-test checks.

This pre-release version should already be stable enough that it is usable for non-prod use cases. Also testing the yaook release upgrade from release-prepare version $R-1$ is upgraded to version $R$ should be added.

In future we plan to run a “full” tempest run for each OpenStack release.

stable

The release-prepare/v$Major.$Minor.$Patch-branch is merged to the stable branch after at most one week. To be merged to stable we run the following tests:

a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ must pass the integration-test checks. This check is included into the following upgrade test.
a full deployment of all supported OpenStack releases $V_1 \ldots V_N$ form stable version $R-1$ is upgraded to version $R$ and must then pass the integration-test checks.

In future we plan to run a “full” tempest run for each OpenStack release.

This version is basically the same as the pre-release one but accounts for two additional things:

its not realistic for everyone to upgrade every day. So we need a version with a little longer release cycle
it allows for some manual tests by users running a local pre-release testcluster

Versioning specification¶

In combination with the above we define the following versioning scheme following the SemVer concept (https://semver.org/). stable is hereby using the plain $Major.$Minor.$Patch versioning:

We increment $Major by 1 if we have a incompatible change.
We increment $Minor by 1 every time we added at least one new feature in the release. It starts from 0 for each major release.
We increment $Patch by 1 for every release not introducing a new feature. It starts from 0 for each normal release.

Hotfix process¶

Note

This section needs adjustments to our changed release process.

We might from time to time need to build hotfixes for the Operator. To do this we follow the following process:

Create a branch of the merge-base of stable and devel into hotfix/base/$name and create the fix.
Create a branch of devel named hotfix/devel/$name and merge hotfix/base/$name into there. Create and merge a MR to devel.
Create a branch of stable named hotfix/stable/$name and cherry pick hotfix/base/$name into there. Create and merge a MR to stable. We will bump the version automatically.
For each old release needing this: Create a branch of stable-<oldversion> named hotfix/stable-<oldversion>/$name and cherry pick hotfix/base/$name into there. Create and merge a MR to stable-<oldversion>. We will bump the patch-version automatically. The branch stable-<oldversion> may not exist before, so it needs to be created from a tag. Also <oldversion> should only be the major and minor version, not patch version.

Each commit must contain a reference to the original issue using Hotfix-For: #$issueid to help with transparency.

This process ensures that each hotfix has run through the normal validation pipeline and we can consider it stable.

Note

If a hotfix is only relevant for an old version, then create a MR again stable-<oldversion> directly and skip the other steps.

Practical implementation¶

A renovate-bot (in renovate-bot) updates the operator dependencies every day. A pinned-version-job (in rdm) updates yaook/assets/pinned_version.yml with the newest image tags, used by yaook. Both jobs create merge requests to devel that needs to be approved and merged by a reviewer.

The release pipeline (in rdm) of the yaook/operator repository is following these steps:

create a release-prepare/v$Major.$Minor.$Patch branch based off stable
merge devel to this branch
on the branch do the following:
- calculate the next version number based on the provided releasenote-files since the last release and write it to the version-file
- generate the changelog using towncrier and remove the old releasenote-files

The pipeline for the release-prepare/v$Major.$Minor.$Patch-branch does the following:

builds the Operator
creates a delayed job that at the moment get’s executed instantly which
- merges to stable
- triggers a MR back to devel

The pipeline for the stable-branch

builds the Operator, tag it with $Major.$Minor.$Patch and push it to the repository
push the same image with stable tag
tags the release with v$Major.$Minor.$Patch and
creates a Gitlab Release.

The MR pipelines just grab the code of the branches, build it and run the tests defined above.

Image information¶

Detailed branching model¶

The image repositories are using a single branch by default:

devel: The main working branch of the repository any change is first merged in here. Code in here can be expected to pass all linting and unittest requirements as well as very basic functionality tests. When developing on Yaook this should be your base branch. The Operator is refering to specific versions/commits on this branch when using an image.

If a hotfix would need to be created, an additional branch can be created based on the specific commit in devel.

Versioning specification¶

The versions for the images also follow (https://semver.org/) mainly using the plain $Major.$Minor.$Patch.

Some images where the installed software has as well semver versions have the tag format $Software-Version-$Yaook-Semver (like currently done for OVN, openvswitch and mariadb-galera)

nova-compute images can get an additional “evacuation serial number” that allows the Operator to determine if a given hotfix needs to evacuate all VMs before applying it. The number is set to 0 on each new Minor release and incremented each time a hotfix is merged that requires a evacuation. The Operator can then determine if the previous serial is different from the new one and issue an evacuation if this is the case. The serial is added to the image as a build metadata, e.g. 1.0.2+evacuation1. If the serial is missing it is assumed to require evacuation. This feature is currently not used/supported, so every image change will start a migration for VMs.

Hotfix process¶

As the images are pinned in the Operator a change on the image side does not directly affect the Operators. To create a hotfix follow the following process:

Create a branch of the relevant commit at the image repository.
Update the Dockerfile of the image to refer to the existing image tag.
Develop the bugfix.
Push the changes and let the image build run. At this point the image can already be used if the user sets YAOOK_OP_VERSIONS_OVERRIDE.
Potentially also merge the changes to devel if the fix is also needed for the current version.
Build a hotfix to the Operator repo as described above and updated the version pins.

Create CVE Hotfix¶

The process of building images for embargoed security vulnerabilities is described here.

Docker image implementation¶

The image pipeline for devel does the following:

build the image with the version specified in the version file
create a SBOM for each image
bump the version and commit the new version to the repo

If the version change in the version file does not fit your requirements you can update the file during your normal MR.

Updating¶

The Operator generally needs to be updated for each Major and Minor version change. Updating hereby means to use whatever method we define to release Yaook to obtain a newer version and push it to the Kubernetes cluster. The user is hereby responsible to only follow supported upgrade paths (only one Minor version jump at a time).

For Patch releases it might be possible in the future to enable auto-updating. This causes the Operator to regularly check for upstream image updates and apply them automatically. Changes in this category normaly include critical security updates. Alternatively the user can choose to not auto-update but follow the normal update process to get the change at a user defined time. In all cases the Operator takes care that resources are properly evacuated before updating (if necessary).