Placement sur les nœuds

OpenKeel node placement

Status: design proposal, 2026-10-29, updated the same day with the user's decisions (§ 0). Implemented on 2026-11-02: the role node selectors and tolerations of every chart, set by okadm in production (§ 5.2); the rest of § 5 (spread, priority classes, platform values) is still a proposal.

0. Decisions (2026-10-29)

This document says which OpenKeel component runs on which node of the Kubernetes cluster, how nodes are given roles, how the charts turn roles into scheduling constraints, and how operators add, drain and lose nodes.

Sources:

• the skill references kubernetes.md, openkeel-chart.md, and the backends nova-kubevirt.md, neutron-ovn.md, swift-csi-block.md, masakari-kubevirt.md, ironic-baremetal.md, cinder-csi-block.md, glance-csi-block.md, designate-external-dns.md;
• the charts in openkeel-chart/ and openkeel-{keystone,placement,cinder,glance}/deploy/helm/ (they set topologySpreadConstraints only; no node selector, toleration or priority class yet);
• the reverse-specs reverse-spec-nova/ (compute host model), reverse-spec-neutron/ (ML2/OVN, gateway scheduling, metadata agent) and reverse-spec-ovn-bgp-agent/ (per-node agent, FRR, privileges);
• API types of the cloned operators: KubeVirt NodePlacement (kubevirt:staging/src/kubevirt.io/api/core/v1/componentconfig.go:12), MariaDB Operator (mariadb-operator:api/v1alpha1/base_types.go:379-474: affinity, nodeSelector, tolerations, priorityClassName), RabbitmqCluster (rabbitmq-cluster-operator:api/v1beta1/rabbitmqcluster_types.go:77-85: affinity, tolerations, override), Kube-OVN v1.16.15 (ProviderNetwork.spec.nodeSelector, Vpc.spec.bfdPort.nodeSelector).

Statements marked [verify] come from knowledge of upstream projects whose source is not cloned in the workspace.

1. Node roles

1.1 Label scheme

One boolean label per role, so that a node can carry several roles:

node-role.openkeel.io/<role>: ""

A single-valued key such as openkeel.io/role=compute was rejected: a hyperconverged node needs several roles at once, and a label selector cannot match one item of a comma-separated value.

The charts never hard-code the key. Each role's label key and taint live in global.nodeRoles (§ 5), so a deployment can switch to a hardened key set:

The hardened profile matters for the roles that attract secrets or privileged access (control plane: Galera, Barbican, Keystone keys; provisioning: BMC credentials). With the default keys, a compromised hypervisor could label itself control-plane and get a Galera replica scheduled on it. Decided (D1): hardened keys in production; the default keys are kept for labs. Examples in this document use the short default keys for readability; in production every node-role.openkeel.io/<role> reads node-restriction.kubernetes.io/openkeel-<role>, which global.nodeRoles switches in one place (§ 5.1).

Consequence for the tooling: hardened labels cannot come from the node's own registration (kubelet --node-labels, kubeadm nodeRegistration, and probably Talos machine.nodeLabels, which Talos applies with the node's credentials [verify]). They are set by Cluster API (synced from the Machine template) or by an administrator (kubectl label, § 5.4).

Other labels used by OpenKeel (not roles):

1.2 Roles

The Kubernetes control plane (etcd, kube-apiserver) is a separate concern: it keeps the upstream label node-role.kubernetes.io/control-plane. The OpenKeel control plane may share those nodes on small and medium clusters (§ 6).

1.3 Sizing hints

1.4 Combining roles

Taints are off on small clusters (every node carries every role, nothing to repel) and on at scale, where they keep generic pods off hypervisors and gateways. The charts always set the tolerations of their role, so switching taints on is a node operation, not a chart change.

2. Placement matrix

Conventions:

• Role = the node role selected (nodeSelector on the role label, through the chart helper of § 5). "all" means every node that runs VMs or volumes.
• Spread: unless stated, Deployments use the baseline of kubernetes.md: topologySpreadConstraints on topology.kubernetes.io/zone (ScheduleAnyway) and on kubernetes.io/hostname (DoNotSchedule when replicas ≤ nodes of the role, else ScheduleAnyway).
• PDB: APIs minAvailable: 2 (3 replicas); Lease-elected workers maxUnavailable: 1; quorum StatefulSets maxUnavailable: 1.
• Priority classes (new, created by openkeel-chart; openkeel-vm is created by the openkeel-nova chart when installed alone): | PriorityClass | Value | Used by | |---|---|---| | system-node-critical (built in) | 2000001000 | CNI and OVS DaemonSets, virt-handler, CSI node plugins | | openkeel-infra-critical | 1000000 | Galera, RabbitMQ, OVN central | | openkeel-control | 900000 | OpenKeel APIs, conductors, schedulers, workers, operators' controllers | | openkeel-vm | 500000, preemptionPolicy: Never | virt-launcher pods: never preempt anything, and are only preempted by infra that lost its node |
• Host / priv: hostNetwork and privileges. "no" means the kubernetes.md baseline (non-root, read-only root FS, all capabilities dropped).

2.1 Shared infrastructure

2.2 OpenKeel services

2.3 Platform operators and agents

3. Neutron / OVN specifics

3.1 What runs where

Decided (D2): mode B, a parallel OVN, for now. Mode A stays in this document as the alternative the Go Neutron's NetworkBackend interface keeps possible.

Two OVNs on one node (mode B on a Kube-OVN cluster) needs two OVS databases and integration bridges per node, which neutron-ovn.md calls fragile: both ovs-vswitchd instances share the one openvswitch kernel datapath module. Three ways out, still open (§ 7):

1. a non-OVN primary CNI (Cilium, Calico): each hypervisor and gateway has one OVS/OVN stack, OpenKeel's (recommended);
2. keep Kube-OVN as the CNI and run OpenKeel's ovn-controller against **Kube-OVN's ovs-vswitchd** with a second integration bridge (br-int-ok, ovn-bridge external ID) and its own SB remote [verify that one OVS can serve two ovn-controllers];
3. keep Kube-OVN only on the nodes that run no tenant traffic (Talos control plane, OpenKeel control plane), which needs a different CNI on hypervisors and gateways, so in practice option 1.

3.2 Gateway chassis selection from the node label

The administrator never edits OVS external IDs by hand. The chassis DaemonSet (mode B) or a small OpenKeel node agent (mode A) reads its Node object (RBAC: get on its own node, the name from the downward API spec.nodeName; node labels are not exposed by the downward API) and derives:

A change of label is reconciled live (the agent watches its Node). Removing the network label from a node that hosts router gateways: Neutron moves the gateways (higher-priority Gateway_Chassis takes over) before the agent clears the CMS option, so the change is a planned failover, not an outage.

At least two network nodes for HA chassis groups (one active, one standby); three when BFD tie-breaking or a maintenance window must keep redundancy.

3.3 Distributed vs centralised floating IPs

Decided (D3): distributed (enable_distributed_floating_ip = true; Neutron chart value gateway.floatingIpMode: distributed). Every compute rack has the external VLANs: FIP traffic leaves from the VM's own hypervisor, the network nodes only carry SNAT (VMs without FIP) and router gateway traffic, and losing a gateway does not cut FIPs. The cost: the external bridge mapping (ovn-bridge-mappings) on every hypervisor, and, when BGP is used, ovn-bgp-agent on every hypervisor to announce the FIPs of its VMs. Centralised stays a supported setting.

3.4 Metadata service (169.254.169.254) — decided (D4)

VMs keep config drive / NoCloud from Nova, and also get the HTTP metadata service, built on the ML2/OVN design so that the OVN rows stay identical to Python (reverse-spec-neutron 03 § metadata, 05 § 5.4):

Request path:

sequenceDiagram
  participant VM as VM (tenant NIC on br-int)
  participant NS as ovnmeta-<net> namespace (veth, iface-id = metadata port)
  participant AG as openkeel-neutron-metadata (Go, DaemonSet)
  participant SB as OpenKeel OVN SB
  participant NM as Nova metadata API (ClusterIP :8775)
  VM->>NS: GET http://169.254.169.254/openstack/latest/meta_data.json
  NS->>AG: listener inside the namespace adds X-OVN-Network-ID, X-Forwarded-For
  AG->>SB: Port_Binding lookup (network, source IP) → port, device_id, project
  AG->>NM: GET + X-Instance-ID, X-Tenant-ID, X-Instance-ID-Signature (HMAC-SHA256)
  NM-->>VM: metadata (through the agent)

• The agent provisions, per network that has a VIF bound to the local chassis, the namespace ovnmeta-<network id>, the veth pair, the addresses (169.254.169.254/32, the metadata port IP of each subnet) and the OVS port with external_ids:iface-id = <metadata port id>, with the same triggers as the Python agent (SB Port_Binding events, start-up sync, live-migration additional_chassis).
• No haproxy: the Go agent opens one listener inside each namespace (a goroutine locked to an OS thread does setns, binds 169.254.169.254:80, and the socket stays in that namespace) and handles the request in-process. Same headers, same HMAC as Python, so Nova sees an identical request.
• Privileges: hostNetwork, CAP_NET_ADMIN and CAP_SYS_ADMIN (namespaces, veths, addresses), hostPath to /run/netns (mount propagation Bidirectional) and to the OpenKeel OVS DB socket. No other host path.
• metadata_proxy_shared_secret: one Secret, mounted by the agent and by the Nova metadata Deployment.
• The agent reaches Nova through its ClusterIP Service from the host network namespace (kube-proxy or the CNI's service handling applies to hostNetwork pods).
• Failure handling: the agent's DaemonSet is listed among the node's critical pods, so Masakari's process monitor cordons the node when it is not ready (masakari-kubevirt.md).
• Mode A, if ever used: Kube-OVN has no equivalent, and this design does not apply as is.

4. Nova / KubeVirt specifics

4.1 Compute host = Kubernetes node

4.2 Placement resource providers

The compute agent creates one root resource provider per compute node (name = node name, uuid = node UID) and keeps its inventories current:

Dynamic reservation (D6). The agent watches the pods bound to its node and sets the inventories' reserved to the sum of the requests of every pod that is not a virt-launcher (plus systemReserved/kubeReserved, already outside allocatable). Rules that keep it safe:

• the update only moves reserved, never total, so existing allocations stay valid; when reserved + used > total after a new infra pod lands, Placement refuses new claims on the node, which is the wanted effect, and the agent logs a WARNING with the pods involved;
• changes are debounced (default 30 s) and sent only when the reservation moves by more than one VCPU or 256 MiB, so that short-lived Jobs do not flap the inventory;
• pods in Succeeded/Failed phase are ignored; DaemonSet pods of the node count as soon as they are scheduled, before they run.

Traits: COMPUTE_* capability traits from the KubeVirt node labeller (cpu-feature.node.kubevirt.io/<f> → HW_CPU_X86_<F>), plus every trait.openkeel.io/<TRAIT> label copied as is. Allocation ratios (cpu_allocation_ratio, ram_allocation_ratio) come from the agent's YAML configuration, per node through an annotation override.

4.3 Host aggregates and availability zones

• Availability zones come from topology.kubernetes.io/zone, which the infrastructure owns. Nova exposes one AZ per zone value: the agent puts its node in an aggregate az-<zone> with metadata availability_zone=<zone>, mirrored in Placement. os-aggregates calls that would move a host to another AZ are rejected with Nova's 400 for conflicting AZs, since the zone label is not Nova's to change.
• Other host aggregates (flavor isolation, tenant isolation, hardware generations) keep Nova's API and database as the source of truth. On add_host / remove_host the Nova conductor patches the label aggregate.openkeel.io/<aggregate uuid> on the node, and mirrors the aggregate into Placement (limit_tenants_to_placement_aggregate, isolated_aggregate_filtering keep working through Placement's member_of). The label lets KubeVirt-side constraints, Masakari segments and operators see the membership with kubectl get nodes -l.
• Masakari segments use their own label (openkeel.io/ha-segment), set by Masakari from segments/{id}/hosts.

4.4 Scheduling and KubeVirt node placement

1. The Go scheduler gets allocation candidates from Placement, claims one, and writes on the VM template nodeSelector: {kubernetes.io/hostname: <node>}, the tolerations of the compute role, and priorityClassName: openkeel-vm.
2. The Kubernetes scheduler only confirms (resources, taints); if it refuses (ErrorUnschedulable), the Nova conductor drops the claim and reschedules, as a RescheduledException would. The pin only lasts for the start. KubeVirt renders a migration target from the VMI and copies the VM's selector to it (vmRolloutStrategy: LiveUpdate), so a pinned guest could never leave its node on a drain. Once a server runs and has no task, the conductor's periodic pass replaces the pin with a preferredDuringScheduling node affinity on its host, for a restart KubeVirt decides by itself. Each start Nova asks for (start, reboot, resize) pins it again first, since its ports are bound to that host. Without a pin, a target goes where the Kubernetes scheduler sends it (nodes with virt-handler, tolerated taints), the conductor adopts the migration, and Placement confirms the node, or the CNI plugin holds the target until KubeVirt gives up. Availability zones and server groups do not constrain that choice: they apply at Nova's own scheduling only.
3. KubeVirt's own components are placed by the KubeVirt CR (and CDI's by the CDI CR):

apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata: {name: kubevirt, namespace: kubevirt}
spec:
  infra:                       # virt-api, virt-controller
    nodePlacement:
      nodeSelector: {node-role.openkeel.io/control-plane: ""}
      tolerations:
        - {key: node-role.openkeel.io/control-plane, operator: Exists, effect: NoSchedule}
  workloads:                   # virt-handler (and where VMIs may run)
    nodePlacement:
      nodeSelector: {node-role.openkeel.io/compute: ""}
      tolerations:
        - {key: node-role.openkeel.io/compute, operator: Exists, effect: NoSchedule}
        - {key: openkeel.io/reserved, operator: Exists, effect: NoSchedule}
  configuration:
    migrations: {parallelMigrationsPerCluster: 5, parallelOutboundMigrationsPerNode: 2}
    evictionStrategy: LiveMigrate
---
apiVersion: cdi.kubevirt.io/v1beta1
kind: CDI
metadata: {name: cdi}
spec:
  infra:
    nodeSelector: {node-role.openkeel.io/control-plane: ""}
    tolerations: [{key: node-role.openkeel.io/control-plane, operator: Exists, effect: NoSchedule}]
  workloads:                   # importer, cloner, upload server pods
    nodeSelector: {node-role.openkeel.io/storage: ""}
    tolerations: [{key: node-role.openkeel.io/storage, operator: Exists, effect: NoSchedule}]

spec.workloads.nodePlacement restricts virt-handler; a VMI that lands on a node without virt-handler never starts. Since the Nova scheduler only chooses nodes that have a resource provider, and only compute nodes have one, the two stay consistent.

5. How it is deployed

5.1 Values layout of openkeel-chart

Helm passes global to every subchart, so the role definitions live once in global.nodeRoles; each component names a role, and may override any field.

global:
  nodeRoles:
    taints: false                       # true: dedicated nodes are tainted; charts already tolerate their role
    roles:
      controlPlane:
        label: node-role.openkeel.io/control-plane     # production (D1): node-restriction.kubernetes.io/openkeel-control-plane
        taint: {key: node-role.openkeel.io/control-plane, effect: NoSchedule}
      compute:
        label: node-role.openkeel.io/compute
        taint: {key: node-role.openkeel.io/compute, effect: NoSchedule}
      network:
        label: node-role.openkeel.io/network
        taint: {key: node-role.openkeel.io/network, effect: NoSchedule}
      storage:
        label: node-role.openkeel.io/storage
        taint: {key: node-role.openkeel.io/storage, effect: NoSchedule}
      containers:
        label: node-role.openkeel.io/containers
        taint: {key: node-role.openkeel.io/containers, effect: NoSchedule}
      provisioning:
        label: node-role.openkeel.io/provisioning
        taint: {key: node-role.openkeel.io/provisioning, effect: NoSchedule}
  placementDefaults:
    zoneKey: topology.kubernetes.io/zone
    zoneSpread: ScheduleAnyway
    hostSpread: DoNotSchedule           # relaxed to ScheduleAnyway when replicas > nodes of the role
    priorityClasses:
      infra: openkeel-infra-critical
      control: openkeel-control
      vm: openkeel-vm

priorityClasses:
  create: true                          # openkeel-chart renders the three PriorityClasses

infrastructure:
  mariadb:
    placement: {role: controlPlane, antiAffinity: required}
  rabbitmq:
    placement: {role: controlPlane, antiAffinity: required}

# Per-subchart overrides: <subchart>.placement.<component>
openkeel-nova:
  placement:
    api:          {role: controlPlane}
    conductor:    {role: controlPlane}
    scheduler:    {role: controlPlane}
    consoleProxy: {role: controlPlane}
    computeAgent: {role: compute, extraTolerations: [{key: openkeel.io/reserved, operator: Exists, effect: NoSchedule}]}
openkeel-neutron:
  backend: ovn-nb                       # ovn-nb (decided, D2) | kube-ovn
  placement:
    api:     {role: controlPlane}
    worker:  {role: controlPlane}
    ovnCentral: {role: controlPlane, antiAffinity: required}       # ovn-nb only
    chassis:    {roles: [compute, network]}                         # ovn-nb only (DaemonSet)
    metadataAgent: {role: compute, enabled: true}                   # 169.254.169.254 (D4)
    bgpAgent:   {roles: [network, compute], enabled: false}         # ovn-bgp-agent, when the fabric speaks BGP
  gateway:
    floatingIpMode: distributed         # decided (D3); centralized | distributed
openkeel-chart:
  ingress:                              # public Gateway API data plane (D8)
    placement: {roles: [controlPlane, network]}
openkeel-swift:
  placement:
    proxy:   {role: controlPlane}
    storage: {role: storage, antiAffinity: required}
openkeel-ironic:
  placement:
    api:       {role: controlPlane}
    conductor: {role: provisioning, antiAffinity: required}
openkeel-keystone:
  placement:
    api:
      role: controlPlane
      # any raw field wins over what the role renders:
      nodeSelector: {node-role.openkeel.io/control-plane: "", topology.kubernetes.io/zone: zone-a}

Precedence, highest first: raw fields of the component (nodeSelector, tolerations, affinity, topologySpreadConstraints, priorityClassName) → the component's role / roles → the chart's default role (control plane for every API and worker, documented per chart).

roles: [a, b] (DaemonSets spanning roles) renders a nodeAffinity with one nodeSelectorTerm per role (OR), and the tolerations of all listed roles.

5.2 Chart template change (implemented 2026-11-02)

Every project chart has one helper, shared through the template rather than a library chart (the project charts stay standalone). As implemented, it renders nodeSelector, tolerations and priorityClassName; spread and affinity stay in each template:

{{- /* openkeel.placement: renders nodeSelector/affinity/tolerations/spread/priority for one component */ -}}
{{- define "openkeel.placement" -}}
{{- $g := .root.Values.global.nodeRoles | default dict -}}
{{- $p := .placement | default dict -}}
{{- $role := get ($g.roles | default dict) ($p.role | default .defaultRole) -}}
{{- if $p.nodeSelector }}
nodeSelector: {{- toYaml $p.nodeSelector | nindent 2 }}
{{- else if $role }}
nodeSelector:
  {{ $role.label | quote }}: ""
{{- end }}
tolerations:
{{- if and $role $role.taint }}
  - {key: {{ $role.taint.key | quote }}, operator: Exists, effect: {{ $role.taint.effect }}}
{{- end }}
{{- with $p.extraTolerations }}{{ toYaml . | nindent 2 }}{{ end }}
{{- /* topologySpreadConstraints and affinity follow the same pattern */ -}}
{{- end -}}

Tolerations are always rendered for the component's role, whether or not global.nodeRoles.taints is true, so tainting nodes later needs no redeploy. Without global.nodeRoles (a project chart installed alone), the helper renders nothing and the chart behaves as today.

openkeel-chart additionally renders the operators' CRs with the same roles: MariaDB.spec.{nodeSelector,tolerations,affinity,priorityClassName}, and RabbitmqCluster.spec.{affinity,tolerations} (+ override.statefulSet for the node selector).

What is implemented (2026-11-02):

What okadm sets. In production, okadm install and okadm upgrade pass global.nodeRoles to every release (the infrastructure release and one per component), with the role labels okadm plan checks:

global:
  nodeRoles:
    taints: false
    roles:
      controlPlane: {label: node-restriction.kubernetes.io/openkeel-control-plane, taint: {key: node-role.openkeel.io/control-plane, effect: NoSchedule}}
      compute:      {label: node-restriction.kubernetes.io/openkeel-compute,       taint: {key: node-role.openkeel.io/compute, effect: NoSchedule}}
      network:      {label: node-restriction.kubernetes.io/openkeel-network,       taint: {key: node-role.openkeel.io/network, effect: NoSchedule}}
      storage:      {label: node-restriction.kubernetes.io/openkeel-storage,       taint: {key: node-role.openkeel.io/storage, effect: NoSchedule}}
      containers:   {label: node-restriction.kubernetes.io/openkeel-containers,    taint: {key: node-role.openkeel.io/containers, effect: NoSchedule}}
      provisioning: {label: node-restriction.kubernetes.io/openkeel-provisioning,  taint: {key: node-role.openkeel.io/provisioning, effect: NoSchedule}}

The label keys are the hardened ones (D1); the taint keys stay those of § 1.2 in every mode, since the NodeRestriction admission plugin restricts labels, not taints. It also sets Nova's config.kubernetes.compute_role_label to the compute label, so the driver and the compute agent find the same hypervisors. A global.nodeRoles in the user's values is merged over these (and is the only source in lab mode, where okadm passes none: role labels are optional there and a lab node usually carries none).

What okadm plan requires in production, now that the charts enforce the roles: 3 control plane nodes (Galera, RabbitMQ's required anti-affinity, OVN central), 1 compute node with Nova, 1 network node with Neutron, storage recommended, and one provisioning node per Ironic conductor (openkeel-ironic.conductor.replicas, 2 by default: required anti-affinity). Fewer provisioning nodes fail the plan; the remediation offers to label more nodes or to run as many conductors as there are provisioning nodes.

5.3 Platform components (installed before openkeel-chart)

These are installed from their own charts (openkeel-chart.md, operators section). A companion values file per component, under openkeel-chart/platform/ [proposal], keeps their placement consistent with the roles:

# platform/longhorn-values.yaml
defaultSettings:
  createDefaultDiskLabeledNodes: true          # only nodes labelled node.longhorn.io/create-default-disk=true get disks
  systemManagedComponentsNodeSelector: "node-role.openkeel.io/control-plane:"
  taintToleration: "node-role.openkeel.io/control-plane:NoSchedule;node-role.openkeel.io/compute:NoSchedule;node-role.openkeel.io/storage:NoSchedule;node-role.openkeel.io/network:NoSchedule"
  storageNetwork: longhorn-system/storage-net  # Multus NAD, optional
# [verify every key on the deployed Longhorn version]

# platform/kube-ovn-values.yaml  (mode A, or Kube-OVN as plain CNI)
MASTER_NODES_LABEL: "kube-ovn/role=master"     # put on the control plane nodes [verify chart key]

# platform/mariadb-operator-values.yaml, rabbitmq-*-values.yaml, infra-operator-values.yaml
nodeSelector: {node-role.openkeel.io/control-plane: ""}
tolerations: [{key: node-role.openkeel.io/control-plane, operator: Exists, effect: NoSchedule}]

Plus the KubeVirt and CDI CRs of § 4.4, the Medik8s NodeHealthCheck per Masakari segment (created by Masakari) and one for the control plane (§ 5.6).

5.4 Labelling the nodes

Manually (kubeadm, bare metal, labs):

# control plane (kube-ovn/role=master: Kube-OVN's own central, while Kube-OVN is the CNI)
kubectl label node cp-1 cp-2 cp-3 node-role.openkeel.io/control-plane= kube-ovn/role=master
# hypervisors
kubectl label node cmp-01 cmp-02 cmp-03 node-role.openkeel.io/compute=
# gateways, with their external wiring
kubectl label node net-1 net-2 node-role.openkeel.io/network= network.openkeel.io/external-nic=bond1
# Zun container nodes (Kata)
kubectl label node kata-1 kata-2 node-role.openkeel.io/containers=
# storage
kubectl label node sto-1 sto-2 sto-3 node-role.openkeel.io/storage= node.longhorn.io/create-default-disk=true
# zones (if the infrastructure does not set them)
kubectl label node cmp-01 topology.kubernetes.io/zone=zone-a
# taints, on dedicated nodes only (global.nodeRoles.taints: true)
kubectl taint node cmp-01 cmp-02 cmp-03 node-role.openkeel.io/compute=:NoSchedule
kubectl taint node net-1 net-2 node-role.openkeel.io/network=:NoSchedule

**At node registration (kubeadm JoinConfiguration, also used by CAPI's KubeadmConfigTemplate):**

nodeRegistration:
  kubeletExtraArgs:
    - {name: node-labels, value: "node-role.openkeel.io/compute="}
  taints:
    - {key: node-role.openkeel.io/compute, effect: NoSchedule}

This works for the default keys only (labs); the kubelet cannot self-set the hardened node-restriction.kubernetes.io/* keys used in production (D1). Taints can still come from registration.

Talos Linux. Taints come from machine.nodeTaints and kubelet settings from machine.kubelet in each role's machine config patch. Hardened role labels cannot be put in machine.nodeLabels (applied with the node's own identity [verify on the Talos version]): set them with kubectl label after the node joins, or through Cluster API (Sidero / CAPI Talos providers) from the Machine template.

# production labels (D1), by an administrator
kubectl label node ok-cp-1 ok-cp-2 ok-cp-3 node-restriction.kubernetes.io/openkeel-control-plane=
kubectl label node ok-net-1 ok-net-2 ok-net-3 node-restriction.kubernetes.io/openkeel-network=
kubectl label node ok-prov-1 node-restriction.kubernetes.io/openkeel-provisioning=

With Cluster API, one MachineDeployment (machine pool) per role, hardened keys carried by the Machine template and synced to the Node by CAPI:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata: {name: openkeel-compute-zone-a}
spec:
  replicas: 8
  template:
    metadata:
      labels:
        node-restriction.kubernetes.io/openkeel-compute: ""   # synced to the Node by CAPI
    spec:
      failureDomain: zone-a                                     # becomes topology.kubernetes.io/zone via the provider [verify]
      bootstrap: {configRef: {kind: KubeadmConfigTemplate, name: openkeel-compute}}   # taints, hugepages kernel args, CPU manager
      infrastructureRef: {kind: Metal3MachineTemplate, name: openkeel-compute}

The hypervisor-specific kubelet settings (cpuManagerPolicy: static, topologyManagerPolicy, reservedSystemCPUs, systemReserved) belong to that role's KubeadmConfigTemplate, so that a node gets them with its role.

5.5 Day 2: adding, draining and removing a hypervisor

Add

1. Provision the node with the compute role (scale the MachineDeployment, or kubeadm join + labels), kernel arguments for hugepages and IOMMU, kubelet CPU manager settings.
2. DaemonSets start by themselves: CNI/OVS (chassis registered in OVN SB), virt-handler (node becomes kubevirt.io/schedulable=true), Longhorn, SNR, the Nova compute agent.
3. The agent registers the nova-compute service and the resource provider. With Nova's enable_new_services (default true) the host is schedulable at once; set it to false to add hosts disabled and enable them after a smoke test (openstack compute service set --enable).
4. Add the host to its aggregates (openstack aggregate add host, which labels the node) and to its Masakari segment (openstack segment host create, which labels it openkeel.io/ha-segment).

Drain (maintenance)

1. openstack segment host update --on_maintenance True so that Masakari ignores the coming events (or create a Medik8s NodeMaintenance CR, which Masakari also reads as maintenance).
2. openstack compute service set --disable --disable-reason maintenance <node>: no new VMs.
3. kubectl drain <node> --ignore-daemonsets --delete-emptydir-data: KubeVirt turns the eviction of each virt-launcher pod into a live migration (evictionStrategy: LiveMigrate; the per-VMI PDB blocks plain eviction). VMs that cannot migrate (RWO volumes, host devices) follow their strategy (LiveMigrateIfPossible: restart elsewhere; External: left to Nova, which cold-migrates them). Each move is recorded by Masakari as a vmove with payload.reason: maintenance.
4. After the work: kubectl uncordon, re-enable the service, clear on_maintenance.

Remove

1. Drain as above.
2. openstack compute service delete <id>: refused while the resource provider has allocations (Nova/Placement behaviour), then the agent deletes the provider.
3. Remove the role label (the DaemonSets leave), kubectl delete node, scale the MachineDeployment down.

5.6 Failure: Masakari and Medik8s

sequenceDiagram
  participant N as compute node cmp-07
  participant NHC as NodeHealthCheck (segment S)
  participant SNR as Self Node Remediation
  participant M as Go Masakari
  participant NV as Go Nova
  N--xNHC: Ready=Unknown for 300 s
  NHC->>SNR: SelfNodeRemediation cmp-07 (strategy OutOfServiceTaint)
  SNR->>N: watchdog reboot (fencing)
  SNR->>N: taint node.kubernetes.io/out-of-service
  M->>M: notification COMPUTE_HOST STOPPED (only once fenced)
  M->>NV: evacuate HA VMs of cmp-07 within segment S (auto / reserved_host)
  NV->>NV: new Placement claims, VMs pinned to healthy segment nodes

Placement consequences:

• **One NodeHealthCheck per Masakari segment**, selecting openkeel.io/ha-segment=<uuid> (always within the compute role). maxUnhealthy stops remediation storms (masakari-kubevirt.md).
• **A separate NodeHealthCheck for the control plane**, without Masakari, with a stricter maxUnhealthy: 1 so that a Galera or RabbitMQ quorum is never fenced away; Medik8s is control-plane aware [verify on v0.12.0]. Network and storage nodes get their own NHC too: fencing a gateway triggers an OVN gateway failover, fencing a storage node a Longhorn / Swift rebuild.
• Reserved hosts carry the taint openkeel.io/reserved=true:NoSchedule; only virt-handler, the compute agent and the DaemonSets tolerate it, so the node stays empty until Masakari removes the taint.
• The SNR agent must run on every node it may fence, including tainted ones (it tolerates everything).

6. Reference topologies

flowchart TB
  subgraph S["Small: 3 nodes, hyperconverged, no taints"]
    direction LR
    s1["node-1<br/>k8s CP + etcd<br/>control-plane, compute,<br/>network, storage"]
    s2["node-2<br/>same roles"]
    s3["node-3<br/>same roles"]
  end
  subgraph M["Medium: 3 control + N compute + 2 network"]
    direction LR
    m1["cp-1..3<br/>k8s CP + etcd<br/>control-plane<br/>(Galera, RabbitMQ, OVN central, APIs)"]
    m2["cmp-01..N<br/>compute + storage<br/>(VMs, Longhorn disks, Swift storage)<br/>tainted compute"]
    m3["net-1..2<br/>network<br/>(gateway chassis, BGP)<br/>tainted"]
  end
  subgraph L["Large: dedicated everything, 3 zones"]
    direction LR
    l0["k8s-cp-1..3<br/>etcd + kube-apiserver only"]
    l1["ok-cp-1..6 (2/zone)<br/>control-plane"]
    l2["cmp-zone-a/b/c<br/>compute (+ distributed FIP path)"]
    l3["net-1..3 (1/zone)<br/>network + ingress"]
    l4["sto-1..9 (3/zone)<br/>storage: Longhorn, Swift"]
    l5["prov-1..2<br/>provisioning (Ironic)"]
  end
  m1 --- m2
  m1 --- m3
  l1 --- l2
  l1 --- l3
  l1 --- l4
  l1 --- l5

6.1 Small: 3-node hyperconverged (lab, edge, PoC)

• Every node carries every role and kube-ovn/role=master; global.nodeRoles.taints: false.
• Galera, RabbitMQ, OVN central: 3 replicas, one per node (required anti-affinity holds with exactly 3 nodes; one node down leaves quorum, two down does not).
• Longhorn: 3 replicas for VM, image and Cinder volumes; Galera / RabbitMQ / Swift volumes with 1 replica strict-local (they replicate themselves). Swift: 3 pods, replication policy with 3 replicas, one per node.
• Floating IPs: distributed (D3); all 3 nodes are also gateway chassis for SNAT (HA chassis group of 3).
• Placement reserved on each node = everything the infra requests (§ 4.2): budget roughly 8 cores / 32 GiB per node for OpenKeel + platform before any VM.
• Masakari: one segment of 3 nodes, maxUnhealthy: 1; live migration needs Longhorn RWX (migratable) volumes.
• No Ironic, or the conductor on one node with its taint and a provisioning VLAN.

6.2 Medium: 3 control + N compute + 2 network

• Control plane nodes also host the Kubernetes control plane; label control-plane, taint optional (needed only if generic workloads run in the cluster).
• Compute nodes are also storage nodes (Longhorn data locality best-effort for VM disks); taint compute.
• 2 network nodes (3 recommended), tainted, with the external NICs; distributed floating IPs (compute racks have the external VLANs, D3), so the network nodes carry SNAT and router gateways; ovn-bgp-agent on network and compute nodes if the fabric speaks BGP.
• One dedicated provisioning node for the Ironic conductor (D7).
• Masakari segments per rack or zone; one reserved host per segment if reserved_host is used.
• Public ingress (Gateway API) on the network and control plane nodes (D8), behind a LoadBalancer VIP (MetalLB, or BGP via the network nodes).

6.3 Large: dedicated storage and network, 3 zones

• Kubernetes control plane alone on 3 nodes (etcd latency is not shared with Galera).
• OpenKeel control plane on 6+ nodes spread over 3 zones; Galera 3 (or 5) members, one per zone; RabbitMQ 3, one per zone; APIs scaled by HPA.
• Compute per zone, one MachineDeployment per zone and hardware class; host aggregates for hardware generations, GPU and SR-IOV pools.
• Network nodes per zone (gateway chassis carry availability-zones=<zone>), distributed floating IPs when every rack has the external VLANs, ovn-bgp-agent on network and compute nodes.
• Storage nodes per zone: Longhorn (block for Cinder, Glance, VM disks) and Swift capacity nodes (separate StorageClass on HDD, Swift zones = Kubernetes zones), dedicated storage network.
• Provisioning nodes, one pair per provisioning network, in their own namespace.
• Hardened label keys (§ 1.1), with labels coming only from Cluster API.

7. Open questions

Answered on 2026-10-29: label keys, network mode, distributed floating IPs, metadata service, Galera storage, capacity reservation, Ironic, ingress (§ 0). Still open:

1. Primary CNI with the parallel OVN (D2): replace Kube-OVN by a non-OVN CNI (Cilium, Calico) so that each node runs one OVS/OVN stack (recommended), or keep Kube-OVN and share its ovs-vswitchd with a second integration bridge (§ 3.1, to be validated)?
2. Aggregates and AZs: AZs read-only from topology.kubernetes.io/zone (Nova rejects AZ changes through os-aggregates), other aggregates owned by Nova and mirrored to node labels: is this split acceptable?
3. Galera / RabbitMQ recovery with strict-local volumes (D5): when a control plane node is lost for good, the member's PVC must be deleted to be recreated elsewhere. Automate it (an OpenKeel controller reacting to a deleted Node), or keep it a documented manual step?
4. Hardened labels on Talos (D1): labels applied by an administrator after join, or Talos machines managed through Cluster API so that labels come from Machine templates?
5. Upstream keys to confirm on the deployed versions (marked [verify]): Kube-OVN master-node label, Longhorn settings, CAPI label sync prefixes, Talos machine.nodeLabels identity, Medik8s control-plane handling, one OVS serving two ovn-controllers.