Connect with STACKIT-KMS

The STACKIT-KMS is basically some kind of network-HSM. User can upload and manage keys there, which can be used as key-encryption-keys (kek) by other services. In order to provide Openstack the ability to use this STACKIT-KMS as crypto-backend for creating encrypted volumes, Barbican and Cinder were enhanced to be able to work with the KMS. The primary key, which is used to encrypt and decrypt the Cinder volume, is still stored inside the database of Barbican, but within Barbican this key is encrypted against a user-defined key within the STACKIT-KMS. The key in the STACKID-KMS is identified by various IDs and linked by a service-account.

Normally the STACKIT-KMS-plugin can only store keys, which depends on a STACKIT service-account. This would break some workflows, which doesn’t have an account, like for example when Nova generates a key by Barbican for its vTPM. To prevent such breaking behavior, the key-generation and decryption of the simple-crypto plugin were merged as fallback-workflow into the STACKIT-KMS plugin. This way, when generating a key without linked service-account, the STACKIT-KMS behaves like the simple-crypto plugin and is compatible with vanilla Openstack workflows.

Requirements

In openstack there is an encrypted volume type necessary. Every new Volume, which should be an encrypted volume, requires this encrypted volume type while creation. The following example-command creates such a volume type named LUKS. The parameters like cipher, key-size and name can be replaced by other values, if necessary.

openstack volume type create \
--encryption-provider nova.volume.encryptors.luks.LuksEncryptor \
--encryption-cipher aes-xts-plain64 \
--encryption-key-size 256 \
--encryption-control-location front-end \
LUKS

Configurations

In openstack the new feature is not used per default, because it can not be used without a working STACKIT-KMS, so it has to be explicitly within the openstack configs.

Cinder

In Cinder the usage of secret-consumer has to be enabled by setting use_secret_consumer inside the key_manager-section to true. If this option is not set, it is false per default.

key_manager:
    backend: barbican
    use_secret_consumer: true

This option prevents Cinder from cloning Barbican-secrets. This is absolute necessary when using the STACKIT-KMS as backend. When using the STACKIT-KMS, there are a bunch of metadata in the Barbican-database stored, which never leaving Barbican again and so they can not be cloned by Cinder, when Cinder try to clone the Barbican-secret. If this option is not set, an error will be thrown by Cinder when an encrypted volume, while creating a backup of a volume from a snapshot of an encrypted volume. Creating a new encrypted volume would still work, even if the flag was not set in the config.

The option should work with other crypto backends like simple-crypto as well, but was not tested, because it would have to be tested with all available backends, which is not possible at the moment. So the secret-consumer were made optional.

Barbican

Barbican got a new plugin, which can communicate with the STACKIT-KMS as backend. This stackit_kms plugin has to be enabled in the Barbican-config.

crypto:
    enabled_crypto_plugins:
    - stackit_kms

Additionally the plugin requires a bunch of configurations to be able to access the STACKIT-KMS over the service-account.

The following shows an example for the configuration:

stackit_kms_plugin:
    plugin_name: "STACKIT KMS"
    # kms connection
    public_kms_url: "https://kms.api.eu01.qa.stackit.cloud"
    public_kms_endpoint_version: v1beta
    private_kms_url: "https://kms.api.eu01.private.stackit.cloud"
    private_kms_endpoint_version: v1beta
    # service-account connection
    sa_url: "https://service-account.api.stg.stackit.cloud"
    sa_endpoint_version: v2
    # idp connection
    idp_url: "https://accounts.qa.stackit.cloud/oauth"
    idp_endpoint_version: v2
    # account for impersonation
    client_id: $CLIENT_ID
    client_secret: $CLIENT_SECRET
    # simple-crypto-kek
    simple_crypto_kek: $SIMPLE_CRYPTO_KEK

  • client_id and client_secret: predefined account, which is allowed and configured to impersonate any service-account. Such an account is for example already used by the STACKIT-Metadata-API.

  • idp_url : the IDP-endpoint, where the client can login and get an access-token.

  • sa_url : endpoint to access the service-accounts and impersonate these service-accounts. Which the Access-Token from the IDP-endpoint and new impersonated token is requested here.

  • public_kms_url and public_kms_endpoint_version: the address of the public STACKIT-KMS. The impersonated token from the service-account is used here to get indirect access to the keys of the user, to encrypt and decrypt the tokens stored in Barbican.

  • private_kms_url and private_kms_endpoint_version : address of an additional private STACKIT-KMS, which is only accessable for internal services like Barbican, to store and use Keys, which are not available over the public KMS endpoint.

  • simple_crypto_kek: master-key for the simple-crypto fallback route in case of normal openstack key-generation without linked STACKIT service-account.

In case the fields are missing or invalid or the client has no impersonation permissions, trying to store secrets in Barbican will fail with an internal error-message and only print error-information in the logs of Barbican.

Migration from simple-crypto to STACKIT-KMS plugin

In case, the simple-crypto plugin of Barbican was already used in the deployment before switching to the STACKIT-KMS plugin, some additional configuration is necessary in order to still be able to use the old simple-crypto keys to not break existing keys in Barbican. Barbican provides the capability to enable multiple crypto backends at the same time. So to continue using existing keys, simple-crypto plugin and STACKIT-KMS-plugin have to be enabled at the same time with the STACKIT-KMS plugin marked as default. This way Barbican use for decryption of the old keys the simple-crypto plugin, where the keys were originally created with and for any new key, the STACKIT-KMS is used.

Use the Barbican config update for the STACKIT-KMS from above and update this config with the following:

crypto:
    namespace: barbican.crypto.plugin
    enabled_crypto_plugins:
    - simple_crypto
    - stackit_kms

secretstore:
    enable_multiple_secret_stores: True
    stores_lookup_suffix:
    - simpleCrypto
    - stackitKMS

secretstore:simpleCrypto:
    secret_store_plugin: <YOUR_OLD_SECRET_STORE_PLUGIN>
    crypto_plugin: simple_crypto

secretstore:stackitKMS:
    secret_store_plugin: <YOUR_NEW_OR_OLD_SECRET_STORE_PLUGIN>
    crypto_plugin: stackit_kms
    global_default: True

simple_crypto_plugin:
    kek: <YOUR_OLD_SIMPLE_CRYPTO_KEY>

This enables simple-crypto plugin and STACKIT-KMS plugin at the same time and defines the STACKIT-KMS as default for new keys. Of course the secret_store_plugin for simple-crypto must be the same like in the old configuration, so it can find its keys again. STACKIT-KMS plugin can depend on the same secret-store or use a different one.

Note

The simple-crypto plugin can only be removed from the Barbican-config, when no used keys depend on the simple-crypto plugin anymore.

Usage

create Barbican-secret

Basically storing a secret in Barbican requires the following information:

  • Project-ID

  • Region

  • Service-Account

  • Keyring-ID

  • Key-ID

  • Key-Version

  • Payload

Project-ID and Region are necessary to access the STACKIT-KMS endpoint. The service-account is the account linked to the key. Keyring-ID, Key-ID and Key-Version the used to identify the key within the account of the user within the STACKIT-KMS. The Payload is the key, which will be encrypted by the key in the STACKIT-KMS and stored within the database of Barbican. This is also at the end the key, which will be attached to the encrypted volume in Openstack.

Creating a secret in Barbican with STACKIT-KMS backend it not very convenient. It was the goal to provide the feature only with a plugin and without any changes at the API. Unfortunately the API of Barbican is very limited and doesn’t provide an input for additional key metadata. As workaround the secret payload is used to insert the additional information. The payload when storing a secret in barbican with the STACKIT-KMS-plugin is required to be a json-string, which contains also the metadata beside the real secret-payload. This json looks like this:

{
    "stackit_kms_protect_id": "467803a8-9925-4fcb-9225-78ef0474711f",
    "stackit_kms_region": "eu01",
    "stackit_service_account": "kms-operator-ka4i9h1@sa.stackit.cloud",
    "stackit_kms_keyring_id": "b5eab460-4aa1-4de8-821c-06c7ba12e4fc",
    "stackit_kms_key_id": "69cb35ff-ff64-4b44-8af2-0fcd133af583",
    "stackit_kms_key_version": "1",
    "data":"dGVzdC1zZWNyZXQ="
}

All keys are mandatory. The keys which start with stackit_kms_ are the connection information to link and identify the key within STACKIT-KMS. data is the secret-payload. This data has to be base64-encrypted.

To be able to give this into barbican, the whole json-string has to be converted into base64 too. For the provided example-json, this would look like this

eyJzdGFja2l0X2ttc19wcm9qZWN0X2lkIjogIjQ2NzgwM2E4LTk5MjUtNGZjYi05MjI1LTc4ZWYwNDc0NzExZiIsCiJzdGFja2l0X2ttc19yZWdpb24iOiAiZXUwMSIsCiJzdGFja2l0X3NlcnZpY2VfYWNjb3VudCI6ICJrbXMtb3BlcmF0b3Ita2E0aTloMUBzYS5zdGFja2l0LmNsb3VkIiwKInN0YWNraXRfa21zX2tleXJpbmdfaWQiOiAiYjVlYWI0NjAtNGFhMS00ZGU4LTgyMWMtMDZjN2JhMTJlNGZjIiwKInN0YWNraXRfa21zX2tleV9pZCI6ICI2OWNiMzVmZi1mZjY0LTRiNDQtOGFmMi0wZmNkMTMzYWY1ODMiLAoic3RhY2tpdF9rbXNfa2V5X3ZlcnNpb24iOiAiMSIsCiJkYXRhIjoiWVc1dmRHaGxjaTEwWlhOMExYTmxZM0psZEE9PSJ9

This can now be stored into Barbican with the following command:

openstack secret store \
--name "MySecret" \
--payload-content-type "application/octet-stream" \
--payload-content-encoding "base64" \
--payload "eyJzdGFja2l0X2ttc19wcm9qZWN0X2lkIjogIjQ2NzgwM2E4LTk5MjUtNGZjYi05MjI1LTc4ZWYwNDc0NzExZiIsCiJzdGFja2l0X2ttc19yZWdpb24iOiAiZXUwMSIsCiJzdGFja2l0X3NlcnZpY2VfYWNjb3VudCI6ICJrbXMtb3BlcmF0b3Ita2E0aTloMUBzYS5zdGFja2l0LmNsb3VkIiwKInN0YWNraXRfa21zX2tleXJpbmdfaWQiOiAiYjVlYWI0NjAtNGFhMS00ZGU4LTgyMWMtMDZjN2JhMTJlNGZjIiwKInN0YWNraXRfa21zX2tleV9pZCI6ICI2OWNiMzVmZi1mZjY0LTRiNDQtOGFmMi0wZmNkMTMzYWY1ODMiLAoic3RhY2tpdF9rbXNfa2V5X3ZlcnNpb24iOiAiMSIsCiJkYXRhIjoiWVc1dmRHaGxjaTEwWlhOMExYTmxZM0psZEE9PSJ9"

The base64 converted json becomes the payload of this command. The flags --payload-content-type "application/octet-stream" --payload-content-encoding "base64" are mandatory like this, so the secret has the correct format. When the secret is not defined as application/octet-stream, the secret will be rejected by Cinder. name can be freely chosen.

create encrypted volume

Normally when creating an encrypted volume in Openstack, Cinder would generate a new secret in Barbican and store it there afterwards. Vanilla Openstack currently doesn’t provide a the feature to use an already existing key from Barbican. The Cinder in YAOOK was modified to provide bring-your-own-key. Here the API also shouldn’t be modified, in order to keep the code-changes as minimal as possible. So the secret-id from Barbican will be provided to the Cinder-API via the properties for the new created volume.

Beginning of the volume-create command to create an encrypted volume with pre-created key would look like this:

openstack volume create --type LUKS --property encryption_key_id=b60f3d9a-018a-4e51-bd90-f6265be993ef ...

For --type the encrypted volume type has to be defined, in order to tell cinder that the volume should be an encrypted volume. As --property a key-value-pair encryption_key_id=SECRET_ID has to be added. This secret-id is the uuid of the secret in Barbican. This uuid will be filtered in the Cinder-API and stored in the private encryption_key_id-field of the volume and will not be in the public metadata of the created volume.

When creating a volume this way, barbican will register a secret-consumer for this secret. For each additional backup or from snapshot of this volume create volume, a new secret-consumer will be created. Only when the last volume/backup related to this secret is deleted, the secret itself will be deleted by Cinder from Barbican.

Note

The deletion of secret-consumer in the current implementation in cinder is not fully free from race-conditions. The chance is very small, but still exist. So volumes and backups, which belongs to the same secret should be deleted one after another to avoid the risk.