Operators

Operators are the primary tool Yaook uses for the resource management in Kubernetes.

Operator overview

Operators generally take some kind of Kubernetes native resource request (mostly using custom resource definitions). An Operator is then running a reconcile loop to implement the resource request.

If you want to know more about Operators in general please take a look here: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/

As the deployment of OpenStack services needs to follow specific steps most of the Yaook Operators are leveraging a state machine. This approach conflicts with the Cloud concept of eventual consistency where you would just spawn all resources at once and let them take care of the synchronization. But it allows us to define dependencies for certain resources, e.g. running database migrations for an upgrade, before upgrading the services.

If it becomes necessary to stop an operator from doing anything, one can annotate its customresource with the ANNOTATION_PAUSE annotation. The presence of this annotation prevents the operator from starting another reconciliation run.

Operator implementation

Operators watch the Kubernetes API for changes in their custom resources and execute the appropriate steps to implement them afterwards. The Operators might also take other, externally managed, Kubernetes resources into account when implementing or updating the deployed resources.

../../../_images/operator_implementation.drawio.svg

To ensure the resources do not drift from the desired state the operators regularly reconcile the resources even if no changes occure.

Operator behavior in degraded mode

Operators can continue operating even if some pods or resources are degraded. This is controlled via the maxDegraded field in the CRD, allowing users to define the maximum number or percentage of unavailable pods before the operator pauses dependency reconciliation.

However, there are important caveats related to Kubernetes scheduling constraints.

StatefulSets

For StatefulSets managed by Yaook, pod anti-affinity rules are typically defined to ensure proper distribution across nodes. If a StatefulSet is degraded and the missing pods cannot be scheduled due to these anti-affinity rules, Kubernetes will not create replacement pods.

In this situation, rolling updates of the StatefulSet will not progress while the resource remains in degraded mode, even though the operator continues reconciling.

Deployments

For Deployments, a similar situation can occur when topology spread constraints are defined. If the scheduler cannot place new pods while respecting the configured topology constraints, rolling updates may stall until sufficient capacity becomes available again.

Summary

  • On initial deployment, Yaook operators behave as before and wait for all dependencies.

  • With maxDegraded configured, the operator continues operating despite degraded resources.

  • Updates may still be blocked if Kubernetes scheduling constraints (anti-affinity or topology spread constraints) prevent new pods from being scheduled.

This behavior is expected and reflects Kubernetes scheduling semantics rather than operator failure.