Safe eviction ############# Motivation ========== - Avoid data loss at all cost - Avoid data plane disruption wherever possible - Recover from data plane disruption Affected Services ================= - Nova Compute - Neutron L3 Agent - Neutron DHCP Agent - Neutron BGP Agent - Neutron L2 Agent - OVN Agent - OVN BGP Agent Surroundings / Kubernetes Tools =============================== Finalizers ---------- - Finalizers DO prevent deletion of an object from the API - Finalizers DO NOT prevent termination of containers in a Pod Container Lifecycle Hooks ------------------------- .. seealso:: `Official Kubernetes documentation on Container Lifecycle Hooks `_ - `preStop` hook allows to execute code inside a container or send an HTTP request to it - execution time bounded by the termination grace period (per-Pod setting or deletion request setting) - Docs say: Users should make their hook handlers as lightweight as possible. There are cases, however, when long running commands make sense, such as when saving state prior to stopping a Container. Node Draining ------------- - Implemented by adding Evictions for matching Pods - Protection of DaemonSets is HARDCODED in kubectl against the DaemonSet resource!! -> Our CDSes are unprotected and will be evicted by Drain!! - Not useful at all for our use cases Implementation ============== Nova Compute ------------ - Finalizers on Pods are not sufficient to protect anything worth saving - We need to prevent the CDS from descheduling Pods until the node has been evicted - Approach: - Subclass :class:`ConfiguredDaemonSetState` to prevent descheduling of nodes which still have state which needs to be saved - Track nodes with state using finalizers? or annotations? - On reconcile, trigger task (in the queue? how?) which performs the eviction (via OpenStack API) - Once eviction is complete, trigger reconcile which will then allow the node to be cleared - If the node is unreachable (detect how? OpenStack API + k8s pod/node status?), do hard eviction and delete pod parallely. If a ironicNodeShutdown is specify in the eviction, the node gets shutdown. - Challenges: - Trying to mess with lifecycle management of CDS is probably not a wise idea - Need to keep track of the state the nodes are in separately (where?) To overcome those challenges, we decided to split the Nova Operator into two Operators: One for the big picture (``nova``) and one which manages the individual compute nodes (``nova_compute``). This also means that there is a new resource (``NovaComputeNode``). Neutron L2 Agent ---------------- - Like Nova Compute, Finalizers on Pods don't help us here - Components that depend on ``NeutronL2Agent`` are ``NeutronL3Agent``, ``NeutronDHCPAgent``, ``NeutronBGPDRAgent``, ``NovaComputeNode`` - Approach to safely schedule or evict them: - Use annotations and labels on the nodes running the agents/services - L2 operator sets label `maintenance.yaook.cloud/maintenance-required-l2-agent: False` after the L2 agent has been successfully created. - If L2 agent needs to be removed (e.g. for updating configuration) the operator set `maintenance.yaook.cloud/maintenance-required-l2-agent: True` - Operators (nova, neutron) creating resources needing L2 agent, won't schedule them on nodes which don't have the label `maintenance.yaook.cloud/maintenance-required-l2-agent: False` (so either, no annotation or value `True`, will lead to not scheduling resources on the node) - Operators responsible for agents/services needing L2 agent set an annotation `l2-lock.maintenance.yaook.cloud/*: ''` to the node, at the very beginning of reconcile - The annotation `l2-lock.maintenance.yaook.cloud/*: ''` will be removed, after the agent/service is deleted. This way, each agent/service can be safely evicted before. - L2 operator waits till all `l2-lock.maintenance.yaook.cloud/*: ''` annotations got removed from the node. Before that, the L2 agent won't be touched by the operator - Once all `l2-lock.maintenance.yaook.cloud/*: ''` annotations are gone, L2 operator will delete the L2 agent - After L2 agent is updated/recreated, the label `maintenance.yaook.cloud/maintenance-required-l2-agent: False` is set again We decided to retain the maintenance-required annotation on the node, even after the L2 agent has been deleted. That way, if the L2AgentResource doesn't got deleted right away by k8s and the pods are still there, other operators still see that there is a maintenance required. - Implementation details: - We added a :class:`~.L2Lock` that will be used by each operator needing L2, to set the `l2-lock.maintenance.yaook.cloud/*: ''` annotation. - Introduced subclass :class:`~.L2AwareStatefulAgentResource` from `StatefulAgentResource` that each agents resource inherits from, that needs L2. It is used to check if label `maintenance.yaook.cloud/maintenance-required-l2-agent: False` is set, so agent/service can be scheduled on the node. - L2 operator has it's own :class:`L2StateResource` instead of inheriting from `APIStateResource` so the specific behavior can be implemented there. This class adds the label `maintenance.yaook.cloud/maintenance-required-l2-agent: False` to the node after L2 agent is created and changes it to `True` on deletion. It also waits, till all the maintenance locks are gone from the node. Neutron OVN Agents ------------------ Components that depends on ``OVNAgent`` are ``NovaComputeNode`` and ``OVNBGPAgent``. For scheduling the dependent agents on the node we use the same mechanisim and the **exact same** annotation i.e `maintenance.yaook.cloud/maintenance-required-l2-agent: False` in order to make sure the meets the prerequisite of ``OVNAgent`` before they are deployed. Similar to the Neutron L2 agent, OVNAgent dependencies, ``NovaComputeNode`` and ``OVNBGPAgent`` makes use of `l2-lock.maintenance.yaook.cloud/*: ''`, in order to inform the Neutron operator to not evict the ``OVNAgent`` from the node, unless the locks are released.