magnum — Async processing

Processing outside the API — magnum — branch stable/2026.2

Branch stable/2026.2 @ 1910928b7d9f — repository https://opendev.org/openstack/magnum — analysed on 2026-11-09

Most of what Magnum does happens in magnum-conductor, outside the HTTP request: the API validates, then hands the work to the conductor over RabbitMQ. The conductor runs the cluster driver and two periodic loops: one that asks the driver for the cluster status, and one that polls the Kubernetes API for health. Magnum has no notification listener. It emits CADF notifications but consumes none, and it has no OVSDB or cache events.

Since stable/2026.2 Magnum ships no driver (the magnum.drivers entry point is empty, pyproject.toml:69). Every processing step below that says "the driver" runs code from an out-of-tree package: the Cluster API driver is described in 08.

Summary

The RPC server

• Process: magnum-conductor (magnum/cmd/conductor.py:44) starts [conductor]/workers worker processes (default: CPU count, magnum/cmd/conductor.py:71), each with an oslo.messaging RPC server on **topic [conductor]/topic = magnum-conductor** and a random 12-character server name generated at start (magnum/cmd/conductor.py:57, magnum/conf/conductor.py:20).
• Endpoints: indirection API, cluster, listener, credential, CA, federation and nodegroup handlers (magnum/cmd/conductor.py:58). No Target version is declared (magnum/common/rpc_service.py:37): there is no RPC version negotiation, and calls go to any conductor through the topic queue.
• Serializer: MagnumObjectSerializer (magnum/common/rpc_service.py:41) — clusters, nodegroups and templates travel as oslo versioned objects (magnum_object.name, .version, .data, .changes).
• Executor: threading, with [DEFAULT]/rpc_response_timeout for calls [inferred from oslo.messaging defaults].
• Periodic tasks only run in the parent process, once per host, not per worker (magnum/cmd/conductor.py:76). With several conductor hosts, every host polls every cluster: there is no leader election and no lock (see each task).

cluster_create (RPC cast)

• Trigger: cast from POST /v1/clusters (magnum/conductor/api.py:39).
• Purpose: materialise the cluster requested by the API and start building it.
• Flow and effects:
  1. Status CREATE_IN_PROGRESS, INSERT cluster (magnum/conductor/handlers/cluster_conductor.py:47). The UUID comes from the API; a duplicate raises ClusterAlreadyExists (magnum/db/sqlalchemy/api.py:183), which is only logged.
  2. INSERT nodegroup default-master (size = master_count, flavor = cluster or template master flavor) and default-worker (size = node_count, flavor = cluster or template flavor, min_node_count = label min_node_count if present), both is_default = true, CREATE_IN_PROGRESS, labels copied from the cluster (magnum/conductor/utils.py:157).
  3. Generate the Kubernetes CA, etcd CA and front-proxy CA (3 self-signed RSA CAs, SHA-256, password-encrypted private keys) and the Magnum client certificate (CN=admin, O=system:masters), store them with the cert manager, and record the four references on the cluster (magnum/conductor/handlers/common/cert_manager.py:99).
  4. CADF notification magnum.cluster.create, outcome pending (magnum/conductor/handlers/cluster_conductor.py:64).
  5. Load the driver by the template's driver name (magnum/drivers/common/driver.py:193) and call create_cluster(context, cluster, create_timeout).
  6. Save the cluster (the driver typically sets stack_id), and copy stack_id to each nodegroup (magnum/conductor/handlers/cluster_conductor.py:72).
• Error behaviour: any exception from step 3 onwards sets CREATE_FAILED with status_reason = str(e), emits magnum.cluster.create failure (priority error) and re-raises; the exception is logged by oslo.messaging and lost for the client (magnum/conductor/handlers/cluster_conductor.py:77). The nodegroups stay CREATE_IN_PROGRESS.
• Concurrency and idempotency: a replayed cast fails at step 1 (duplicate UUID). Nothing protects two concurrent creates from exceeding the project quota (checked in the API before the insert).
• Cost: 3 inserts, 3 to 4 certificate stores (with Barbican: 3 or 4 secrets + 1 container per certificate, i.e. up to 20 Barbican calls), then the driver.

cluster_update (RPC cast)

• Trigger: PATCH /v1/clusters/{ident} (magnum/conductor/api.py:58).
• Flow:
  1. Refuse unless the cluster is CREATE_COMPLETE, UPDATE_COMPLETE, ROLLBACK_COMPLETE or one of the legacy *_COMPLETE states (magnum/conductor/handlers/cluster_conductor.py:94) — NotSupported, CADF update failure, nothing changes and the client is not told.
  2. Nothing changed → return (magnum/conductor/handlers/cluster_conductor.py:115).
  3. Only health changed → save the health fields, done (magnum/conductor/handlers/cluster_conductor.py:125).
  4. Else set the default worker nodegroup to the new size, call driver.update_cluster(context, cluster, scale_manager, rollback), cluster UPDATE_IN_PROGRESS (magnum/conductor/handlers/cluster_conductor.py:146).
• Error behaviour: driver exception → cluster UPDATE_FAILED with the reason, the nodegroup size is restored (magnum/conductor/handlers/cluster_conductor.py:151).
• Scale manager: scale_manager.get_scale_manager asks the driver for a manager that picks the nodes to remove; the base driver returns none (magnum/drivers/common/driver.py:301).

cluster_delete (RPC cast)

• Flow: CADF delete pending, driver.delete_cluster, cluster DELETE_IN_PROGRESS (magnum/conductor/handlers/cluster_conductor.py:174). Removal of the row is done later by sync_cluster_status when the driver reports DELETE_COMPLETE.
• Error behaviour: DELETE_FAILED with the reason, CADF failure (magnum/conductor/handlers/cluster_conductor.py:181). A DELETE_FAILED cluster can be deleted again.
• Driver.pre_delete_cluster exists in the interface (magnum/drivers/common/driver.py:228) but the conductor never calls it in this branch [verified by search: no caller outside the drivers].

cluster_resize (RPC cast)

• Flow: allowed in *_COMPLETE, **UPDATE_FAILED and UPDATE_IN_PROGRESS** (magnum/conductor/handlers/cluster_conductor.py:205), so that the autoscaler can retry and stack requests. The nodegroup gets the new size and UPDATE_IN_PROGRESS, then driver.resize_cluster(context, cluster, resize_manager, node_count, nodes_to_remove, nodegroup), cluster UPDATE_IN_PROGRESS.
• Error behaviour: cluster and nodegroup UPDATE_FAILED, size restored (magnum/conductor/handlers/cluster_conductor.py:250).
• Concurrency: two resizes of the same nodegroup can run at once on two conductors; the last write wins [inferred: no lock, nodegroup.save() without version check].

cluster_upgrade (RPC call)

• Flow: allowed in *_COMPLETE states only; driver.upgrade_cluster(context, cluster, cluster_template, max_batch_size, nodegroup, rollback); cluster and nodegroup UPDATE_IN_PROGRESS (magnum/conductor/handlers/cluster_conductor.py:301).
• Error behaviour: NotSupported from the driver is returned to the caller without changing any status (magnum/conductor/handlers/cluster_conductor.py:306); any other exception sets UPDATE_FAILED on both.
• The new template is not recorded on the cluster by the conductor: it is the driver's job to set cluster.cluster_template_id [inferred from the absence of an assignment in magnum/conductor/handlers/cluster_conductor.py:266].

nodegroup_create / nodegroup_update / nodegroup_delete (RPC cast)

• Guard: allowed_operation refuses an operation when the nodegroup status is not in an allowed list (magnum/conductor/handlers/nodegroup_conductor.py:51); a nodegroup freshly built by the API has no status, so creation is never refused. The error message prints the cluster status instead of the nodegroup's (magnum/conductor/handlers/nodegroup_conductor.py:62).
• create: cluster UPDATE_IN_PROGRESS, insert nodegroup CREATE_IN_PROGRESS, driver.create_nodegroup (magnum/conductor/handlers/nodegroup_conductor.py:75).
• update: cluster UPDATE_IN_PROGRESS, nodegroup UPDATE_IN_PROGRESS, driver.update_nodegroup, then the nodegroup (with the patched min/max/labels/taints) is saved (magnum/conductor/handlers/nodegroup_conductor.py:97).
• delete: cluster UPDATE_IN_PROGRESS, nodegroup DELETE_IN_PROGRESS (not saved by the conductor), driver.delete_nodegroup (magnum/conductor/handlers/nodegroup_conductor.py:118).
• Error behaviour: nodegroup *_FAILED with the reason, cluster UPDATE_FAILED. No CADF notification for nodegroups (magnum/conductor/handlers/nodegroup_conductor.py:33).

CA and credential handlers (RPC call)

• sign_certificate: reads the selected CA (Kubernetes by default) and signs the CSR with it (magnum/conductor/handlers/common/cert_manager.py:215). Nothing is stored.
• rotate_ca_certificate: status guard, then new CAs and client certificate are generated and the cluster references updated in memory, then driver.rotate_ca_certificate. The old certificates are not deleted. On failure the cluster goes to UPDATE_FAILED (magnum/conductor/handlers/ca_conductor.py:100).
• credential_rotate: status guard, then driver.rotate_credential; NotImplementedError becomes NotSupported (magnum/conductor/handlers/credential_conductor.py:56). No status change by the conductor.

Versioned-object indirection (RPC call)

• Emitter: magnum-api when started with the magnum-api binary (magnum/cmd/api.py:65). Every @remotable method of a Magnum object (create, save, destroy, get_by_uuid, list…) is then executed by the conductor, which returns the changed fields (magnum/conductor/handlers/indirection_api.py:52).
• Not used when the API runs under a WSGI server through magnum/wsgi/api.py, which does not set indirection_api (magnum/wsgi/api.py:37): the API then opens its own database connections. The same deployment therefore has two database access patterns depending on how magnum-api is started.
• Error behaviour: exceptions are wrapped in ExpectedException (magnum/conductor/handlers/indirection_api.py:35) and re-raised in the API as <Class>_Remote with the original message (magnum/common/exception.py:114).

Periodic tasks

All three are oslo.service periodic_tasks scheduled by a dynamic timer whose maximum sleep is [DEFAULT]/periodic_interval_max (60 s, magnum/conf/rpc.py:20), and are enabled by [DEFAULT]/periodic_enable (true, magnum/conf/rpc.py:17, magnum/common/rpc_service.py:51). They run with an **admin context with all_tenants** (magnum/service/periodic.py:39). An exception in one task is logged and the task runs again on the next cycle (magnum/service/periodic.py:144).

sync_cluster_status

• Trigger: every 10 s, also at start (magnum/service/periodic.py:159).
• Purpose: drive every cluster that is in progress towards its final state, and finish deletions.
• Data read: SELECT * FROM cluster WHERE status IN ('CREATE_IN_PROGRESS','UPDATE_IN_PROGRESS','DELETE_IN_PROGRESS','ROLLBACK_IN_PROGRESS') (magnum/service/periodic.py:166, magnum/db/sqlalchemy/api.py:135).
• Flow and effects, per cluster (magnum/service/periodic.py:66), each in its own one-shot looping call:
  1. Load the driver of the cluster's template and call driver.update_cluster_status(ctx, cluster). The driver reads its back end (for the CAPI driver: the Cluster API objects on the management cluster) and updates cluster.status, status_reason, nodegroup statuses, api_address, addresses, coe_version.
  2. If the status ends with _COMPLETE or _FAILED, emit the CADF notification magnum.cluster.<create|update|delete> with outcome success or failure (magnum/service/periodic.py:77).
  3. If DELETE_COMPLETE: delete the Kubernetes CA and the client certificate from the cert manager (not the etcd and front-proxy CAs, magnum/conductor/handlers/common/cert_manager.py:240), destroy every nodegroup, then the cluster row (magnum/service/periodic.py:86).
• Concurrency and idempotency: no lock. With N conductor hosts, each in-progress cluster is polled N times every 10 s, and the deletion sequence can run concurrently on two hosts; the second one fails on a missing row (ClusterNotFound, logged). Driver status reads must be idempotent.
• Error behaviour: an exception stops that cluster's looping call (stop_on_exception=True, magnum/service/periodic.py:179) and is retried 10 s later. A status ending in _COMPLETE that is not in status_to_event (for example RESUME_COMPLETE) raises KeyError [inferred from magnum/service/periodic.py:79]; drivers only set the standard states.
• Cost: 1 query per cycle plus the driver's calls per in-progress cluster; nothing for clusters at rest. **The base driver's update_cluster_status does nothing** (magnum/drivers/common/driver.py:199): without a driver that implements it, clusters stay in progress forever.

sync_cluster_health_status

• Trigger: every [kubernetes]/health_polling_interval s (60, magnum/conf/kubernetes.py:31), also at start (magnum/service/periodic.py:186).
• Purpose: keep health_status (HEALTHY, UNHEALTHY, UNKNOWN) and health_status_reason up to date.
• Data read: clusters in CREATE_IN_PROGRESS, CREATE_FAILED, CREATE_COMPLETE, UPDATE_*, DELETE_IN_PROGRESS, DELETE_FAILED, ROLLBACK_IN_PROGRESS (magnum/service/periodic.py:194); then, per cluster, the driver's monitor (magnum/conductor/monitors.py:59).
• Flow (with the Kubernetes monitor of magnum/drivers/common/k8s_monitor.py, when the driver returns it):
  1. Skip the cluster if labels auto_healing_enabled=true and auto_healing_controller=magnum-auto-healer: the in-cluster healer reports health through PATCH /v1/clusters instead (magnum/drivers/common/k8s_monitor.py:64).
  2. If the API is not reachable from outside (no floating IP on the API load balancer or on the masters, magnum/drivers/common/k8s_monitor.py:70), set UNKNOWN with reason {"api": "The cluster <name> is not accessible."}.
  3. Else GET <api_address>/healthz and GET <api_address>/api/v1/nodes with the Magnum client certificate (magnum/conductor/k8s_api.py:57); HEALTHY if healthz is ok and every node's Ready condition is true; the reason maps <node>.Ready to a boolean and api to the healthz answer (magnum/drivers/common/k8s_monitor.py:226).
  4. Save the cluster if a status was computed (magnum/service/periodic.py:121).
• Concurrency: same as above, no lock; every conductor host polls every cluster.
• Error behaviour: an exception in the monitor is logged as a warning and the previous health value is kept (magnum/service/periodic.py:111). The HTTP requests have no timeout (magnum/conductor/k8s_api.py:44): an unreachable API server blocks that green thread until the TCP timeout.
• Cost: per cluster and cycle, 2 HTTPS requests and, when [cluster]/temp_cache_dir does not exist, 2 cert-manager reads (with Barbican: several secret reads) to write temporary certificate files (magnum/conductor/handlers/common/cert_manager.py:157). On a large fleet this is the main load Magnum puts on Barbican.

update_magnum_service

• Trigger: every run of the dynamic timer (no spacing, so at most every periodic_interval_max s) [inferred], also at start (magnum/servicegroup/magnum_service_periodic.py:39).
• Purpose: heartbeat used by GET /v1/mservices.
• Flow: read magnum_service by (host, binary), insert it the first time, then report_state_up (sets last_seen_up, increments report_count) (magnum/servicegroup/magnum_service_periodic.py:44).
• Consumer: the API marks a service down when its last heartbeat is older than service_down_time (180 s) or forced_down is set (magnum/api/servicegroup.py:26).

Source: reverse-spec-magnum/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.