ironic — Async processing

Processing outside the API — ironic — branch stable/2026.2

Branch stable/2026.2 @ 39ce09ec3b0d — repository https://opendev.org/openstack/ironic — analysed on 2026-10-29

Almost all of Ironic's work happens in ironic-conductor, outside the HTTP request that started it. Four kinds of triggers drive it: RPC messages from the API (57 conductor methods, table in 03 § 1.3), heartbeats from the ramdisk agent (relayed by the API as an RPC call), periodic tasks (14 in the conductor manager, 9 in the shipped hardware interfaces), and process start (registration, stuck-state recovery, takeover). There are no OVSDB events, no notification listeners and no cache polling.

1. Summary

2. Common mechanics

2.1 Which conductor runs what

Periodic tasks run in every conductor, but each only touches the nodes the hash ring maps to it: iter_nodes filters on _mapped_to_this_conductor (ironic/conductor/base_manager.py:530, ironic/conductor/base_manager.py:596). node_periodic (ironic/conductor/periodics.py:108) wraps this: query nodes with filters, run predicate, take a lock per node (shared or exclusive), ignore NodeLocked/NodeNotFound, stop after limit nodes, and run up to [conductor]/periodic_max_workers (8) in parallel. There is no leader election: the ring partitions the work.

2.2 Worker pool

RPC handlers return quickly and push long work to a thread pool of [conductor]/workers_pool_size (300) threads plus a reserved pool of reserved_workers_pool_percentage (5 %) usable by user-initiated work only (ironic/conductor/base_manager.py:555). When both are full, the RPC answers NoFreeConductorWorker (503) and the node state is rolled back by provisioning_error_handler (ironic/conductor/utils.py:488). Heartbeats are not allowed into the reserved pool (_allow_reserved_pool=False, ironic/conductor/manager.py:3554).

2.3 Locks

Every operation on a node goes through task_manager.acquire (ironic/conductor/task_manager.py:158): an exclusive lock is the nodes.reservation column (see 03 § 3), retried node_locked_retry_attempts (3) times every node_locked_retry_interval (1 s); a shared lock takes nothing in the database. Long operations keep the lock across threads: the TaskManager is released only when the spawned worker ends (ironic/conductor/task_manager.py:546). A conductor that dies keeps its reservations until another conductor runs _check_orphan_nodes (it notices the dead conductor after heartbeat_timeout, 60 s) or until it restarts with the same hostname (ironic/db/sqlalchemy/api.py:1538).

3. Sheets

3.1 Heartbeat processing (deploy interface heartbeat)

• Trigger: POST /v1/heartbeat/{node} → RPC call heartbeat → shared lock, token check, worker task.driver.deploy.heartbeat (ironic/drivers/modules/agent_base.py:639).
• Emitter: IPA, every heartbeat_timeout from the lookup config with jitter ([api]/ramdisk_heartbeat_timeout, 300 s, is the timeout the agent is told).
• Purpose: record that the agent is alive and where (agent_url), then move the current operation forward.
• Flow:
  1. Ignore heartbeats in states that do not wait for an agent (HEARTBEAT_ALLOWED, or fast-track states), ironic/drivers/modules/agent_base.py:491.
  2. Upgrade to an exclusive lock without retry; if locked, skip (next heartbeat retries).
  3. Save agent_url, agent_version, agent_last_heartbeat, agent_verify_ca.
  4. Maintenance: abort the current operation unless provisioning in maintenance is allowed (ironic/drivers/modules/agent_base.py:497).
  5. Dispatch on the provision state: wait call-back → load in-band deploy steps once, then process_next_step('deploy') unless a step is polling (ironic/drivers/modules/agent_base.py:529); clean wait → first heartbeat after boot starts cleaning (resume, fetch in-band steps, set_node_cleaning_steps, continue_node_clean), later ones check the running step (ironic/drivers/modules/agent_base.py:558); rescue wait → finalize_rescue (ironic/drivers/modules/agent_base.py:595); service wait → like cleaning (ironic/drivers/modules/agent_base.py:607).
• Concurrency: shared lock first, exclusive lock without retry: two heartbeats never run steps twice.
• Errors: any exception moves the node to the corresponding *failed state through the error handlers; a missing or wrong token is refused before the worker (400).
• Cost: one DB write per heartbeat per node, plus one GET /v1/commands to the agent when a step is running.

3.2 Step framework (clean, deploy, service, verify)

Formal specification in 07 § 3.3. Summary of the runtime loop (ironic/conductor/cleaning.py:229, ironic/conductor/deployments.py:323, ironic/conductor/servicing.py:133):

1. The ordered step list is stored in driver_internal_info.<kind>_steps, the current index in <kind>_step_index, the running step in node.<kind>_step.
2. For each remaining step: save it as current; reserved names (power_on, power_off, reboot, hold, wait) are handled by the conductor (ironic/conductor/steps.py:1063); execute_on_child_nodes steps run on child nodes; others call interface.execute_<kind>_step.
3. A step returns None (done, next step), or *WAIT (asynchronous: process_event('wait'), the step continues on a heartbeat, a driver periodic or continue_node_*), or raises (failure: *failed).
4. After the last step: tear down the ramdisk environment, done (or manage for manual cleaning, or for retired nodes).
5. abort (API) on a *wait/*hold node: a non-abortable running step gets abort_after: true and the abort happens when it completes (ironic/conductor/manager.py:1553, ironic/conductor/cleaning.py:514).

3.3 _sync_power_states

• Trigger: every 120 s; nodes not in maintenance mapped to this conductor, failing nodes first.
• Data read: SELECT uuid, driver, conductor_group, … FROM nodes WHERE maintenance = 0 (then hash-ring filter in Python), then for each node a BMC call (get_power_state), in up to [conductor]/sync_power_state_workers (8) threads.
• Flow (ironic/conductor/manager.py:4574): power state unknown and power interface invalid → skip; BMC error → count, after power_state_sync_max_retries (3) put the node in maintenance with fault power failure (ironic/conductor/manager.py:4524); same state → nothing; node in a provisioning state where power changes are expected (SYNC_EXCLUDED_STATES) or in maintenance → nothing; DB state unknown → record it and send Nova power-update; otherwise, with force_power_state_during_sync (true) and a power interface that supports it, power the machine back to the DB state, else record the observed state and notify Nova.
• Concurrency: shared lock per node, upgraded only when something must change.
• Cost: one BMC request per node per cycle; IPMI calls are additionally throttled per BMC (5 s).

3.4 Callback timeouts (_check_*_timeouts)

• Query: nodes mapped here, in wait call-back / clean wait / rescue wait / service wait / inspect wait, unreserved, with provision_updated_at < now - timeout (ironic/conductor/base_manager.py:611).
• Effect: fail event with last_error "Timeout reached while waiting for callback for node …" and the error handler of the flow (tear down cleaning, power off…). Every heartbeat touches provision_updated_at (node.touch_provisioning(), ironic/drivers/modules/agent_base.py:529), so only a silent agent times out.

3.5 Conductor failure and takeover

1. A conductor stops heartbeating; after heartbeat_timeout (60 s) the others consider it offline and the hash rings, rebuilt every 15 s, map its nodes elsewhere.
2. _check_orphan_nodes on the new owner releases the dead conductor's reservations and fails what was running (ironic/conductor/manager.py:1950).
3. _sync_local_state re-prepares active nodes (PXE/iPXE configs, boot ISOs) on the new conductor (ironic/conductor/manager.py:2107): needed because these files live on the conductor's local disk.
4. _check_orphan_allocations resumes allocations.
5. When the dead conductor restarts, _correct_stuck_states fails the nodes it had left in deploying, cleaning, … (ironic/conductor/base_manager.py:327).

3.6 Inspection (built in; ironic-inspector is not used)

• Start: PUT …/states/provision {"target": "inspect"} → inspect_hardware (ironic/conductor/manager.py:3414, ironic/conductor/inspection.py:29). Out-of-band (redfish inspect) completes synchronously from BMC data; in-band (agent inspect) boots the ramdisk and waits in inspect wait (ironic/drivers/modules/inspector/interface.py:183).
• Continue: IPA posts to /v1/continue_inspection → RPC continue_inspection (ironic/conductor/manager.py:4250) → inspection.continue_inspection (ironic/conductor/inspection.py:119): run the interface's processing (hooks from [inspector]/hooks: ramdisk-error, validate-interfaces, ports, architecture by default; others such as memory, cpus, root-device, local-link-connection, parse-lldp, physical-network are opt-in, entry points ironic.inspection.hooks in pyproject.toml), then inspection rules of phase main sorted by priority (ironic/common/inspection_rules/engine.py:192; 15 operators in ironic/common/inspection_rules/operators.py:27, 18 actions in ironic/common/inspection_rules/actions.py:31), store inventory and plugin data (node_inventory), cache firmware, done → manageable.
• Auto-discovery: unknown machines posting inventory are enrolled when [auto_discovery]/enabled (ironic/api/controllers/v1/ramdisk.py:360).

3.7 Allocation (create_allocation RPC)

ironic/conductor/manager.py:3915 spawns do_allocate (ironic/conductor/allocations.py:34): candidate query by resource class, available, not maintenance, not reserved, no instance; trait filter; candidate list filter; owner filter; random shuffle; try to lock and update each candidate up to a retry limit; on success allocation.state = active, node.instance_uuid = allocation.uuid, node.allocation_id. Not deterministic by design (random order).

Source: reverse-spec-ironic/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.