nova — Async processing

Asynchronous processing — nova — branch stable/2026.2

Branch stable/2026.2 @ 8df1922926dd (34.0.0 + 7 commits) — repository https://opendev.org/openstack/nova — analysed on 2026-10-28

Related files: REST API · errors · interactions · scenarios.

Almost everything Nova does after a 202 happens here: RPC consumers in nova-conductor, nova-scheduler and nova-compute, periodic tasks that poll the cell database through the conductor, and hypervisor events. The inventory found 175 triggers (_inventory/inventory_stable-2026.2.json, triggers): 130 RPC server methods, 44 periodic tasks (19 in the compute manager, the rest in virt drivers, the image cache and the console proxy) and 1 notification subscription.

1. Summary

2. nova-conductor

nova-conductor (nova/cmd/conductor.py) runs two RPC endpoints on topic conductor (nova/conductor/rpcapi.py:30): ConductorManager (version 3.0, nova/conductor/manager.py:120) and ComputeTaskManager (namespace compute_task, version 1.26, nova/conductor/manager.py:269). It is the only service that talks to both the API database and the cell databases, and the only one that orchestrates scheduling. It has no periodic task.

2.1 Object proxy for nova-compute (object_class_action_versions, object_action, object_backport_versions)

• Trigger: RPC call from nova-compute whenever it saves or loads a versioned object (Instance.save(), Migration.save(), BlockDeviceMapping.create()…), because computes run with indirection_api set and have no database credentials (nova/conductor/rpcapi.py:237-255).
• Purpose: give compute hosts database access without exposing the database to them.
• Flow: deserialise the object, call the named method on the conductor side against the cell database (nova/conductor/manager.py:133 _object_dispatch), return the updated object and the changed fields (nova/conductor/manager.py:172-190). object_backport_versions downgrades an object to the version a compute understands during rolling upgrades.
• Concurrency: none beyond the DB layer; Instance.save(expected_task_state=…) implements compare-and-swap on task_state (UnexpectedTaskStateError).
• Error behaviour: exceptions are serialised back to the compute (registered remote exceptions) and re-raised there.
• Cost: the bulk of conductor traffic: several calls per instance operation per host, plus every periodic task of every compute.

2.2 schedule_and_build_instances

• Trigger: RPC cast from nova-api after POST /servers (nova/compute/api.py:1836, nova/conductor/rpcapi.py:395).
• Purpose: pick hosts, create the instance in its cell and start the build.
• Flow (nova/conductor/manager.py:1675-1868):
  1. select_destinations RPC call to the scheduler with return_alternates=True (nova/conductor/manager.py:967); on NoValidHost/MaxRetriesExceeded the instances are buried in cell0: created in the cell0 database with vm_state=error, an instance fault and an instance action, instance mapping pointed at cell0, build request deleted, compute_task.build_instances error notification (nova/conductor/manager.py:1578-1673).
  2. For each selected host: resolve the host's cell through host_mappings (missing mapping → bury in cell0, log "No host-to-cell mapping found for selected host", nova/conductor/manager.py:1705-1718).
  3. Create the instances row (and its instance_extra, instance_system_metadata) in that cell (nova/conductor/manager.py:1758), then update instance_mappings.cell_id (nova/conductor/manager.py:1944 _map_instance_to_cell).
  4. Re-check quotas if [quota]/recheck_quota (nova/conductor/manager.py:1762-1777); on failure the instance is set to ERROR and cleaned up.
  5. Create BDMs and tags in the cell (nova/conductor/manager.py:1528, nova/conductor/manager.py:1545); create Cyborg ARQs if the flavor requests accelerators (nova/conductor/manager.py:1870).
  6. Delete the build_requests row (nova/conductor/manager.py:2027); if it is already gone, the user deleted the server meanwhile: the new instance is destroyed and allocations removed.
  7. RPC cast build_and_run_instance to the selected host with the alternates as host_list (nova/conductor/manager.py:1857).
• Concurrency/idempotency: allocations are claimed by the scheduler, not the conductor; a replayed cast would hit InstanceExists / missing build request and clean up.
• Error behaviour: any exception after scheduling triggers _cleanup_build_artifacts (nova/conductor/manager.py:1974): instances set to ERROR in their cell, build requests and mappings cleaned.

2.3 build_instances (reschedule path)

• Trigger: RPC cast from nova-compute when a build fails with RescheduledException (nova/compute/manager.py:2573), carrying the remaining alternate hosts.
• Flow (nova/conductor/manager.py:671): take the next alternate from host_list, claim its allocation request in Placement (claim_resources), and cast build_and_run_instance to it; no new scheduler call while alternates remain. When none is left: MaxRetriesExceeded → instance set to ERROR with a fault (nova/conductor/manager.py:644 _cleanup_when_reschedule_fails). The total number of attempts is [scheduler]/max_attempts (nova/conf/scheduler.py:25).

2.4 migrate_server (cold migration / resize) and live_migrate_instance

• Trigger: resize/migrate actions (RPC call with [DEFAULT]/long_rpc_timeout when the API needs to know the scheduling result, nova/conductor/rpcapi.py:317-353); os-migrateLive (cast live_migrate_instance, nova/conductor/rpcapi.py:302).
• Cold migration / resize (nova/conductor/manager.py:408, nova/conductor/tasks/migrate.py:232):
  1. create a migrations record and move the instance's allocations to the migration consumer in Placement (nova/conductor/tasks/migrate.py:29 replace_allocation_with_migration), so that the source usage is held by the migration and the destination will be held by the instance;
  2. select_destinations (or use the requested host), claim_resources on the chosen host (nova/conductor/tasks/migrate.py:342-372);
  3. RPC cast prep_resize to the destination compute (nova/conductor/tasks/migrate.py:329); a cross-cell resize (allowed by the policy compute:servers:resize:cross_cell, nova/policies/servers.py:23) goes through nova/conductor/tasks/cross_cell_migrate.py instead (snapshot-based, instance copied to the target cell);
  4. on NoValidHost the instance keeps its vm_state with task_state=None and a compute_task.migrate_server error notification (nova/conductor/manager.py:417-431); the API sees 400 only when the call was synchronous.
• Live migration (nova/conductor/tasks/live_migrate.py:81):
  1. check the source is up (nova/conductor/tasks/live_migrate.py:301) and move allocations to the migration (nova/conductor/tasks/live_migrate.py:91);
  2. with a target host and force=False: check the host, claim allocations on it (nova/conductor/tasks/live_migrate.py:128); without a target: select_destinations in a loop over candidates (nova/conductor/tasks/live_migrate.py:533-575), each checked by _check_compatible_with_source_hypervisor (same hypervisor type, version not older) and by check_can_live_migrate_destination, an RPC call to the destination which itself calls check_can_live_migrate_source on the source (nova/conductor/tasks/live_migrate.py:373-378);
  3. RPC cast live_migration to the source compute (nova/conductor/tasks/live_migrate.py:154).
  4. Failures before step 3 set the migration to error/failed, revert allocations and leave the instance ACTIVE with task_state=None.

2.5 rebuild_instance (rebuild and evacuate), unshelve_instance

• rebuild_instance (nova/conductor/manager.py:1233): for a plain rebuild with a new image, the scheduler is asked to validate the image against the current host (request_is_rebuild, no Placement query, nova/scheduler/manager.py:217); for an evacuate without a host, select_destinations picks a new host and its allocations are claimed (nova/conductor/manager.py:1190 _allocate_for_evacuate_dest_host); then cast rebuild_instance to the (new) host with recreate=True.
• unshelve_instance (nova/conductor/manager.py:1033): a SHELVED instance is simply restarted on its host; a SHELVED_OFFLOADED one is rescheduled (select_destinations, honouring the requested AZ/host from 2.77/2.91), its allocations claimed, and unshelve_instance cast to the new host.

2.6 Other compute-task methods

3. nova-scheduler

nova-scheduler (nova/cmd/scheduler.py) serves topic scheduler, version 4.5 (nova/scheduler/manager.py:65), with several workers per host ([scheduler]/workers, nova/conf/scheduler.py:86). It reads both databases (host states come from the cell compute_nodes tables) and Placement.

3.1 select_destinations

• Trigger: RPC call from nova-conductor (nova/scheduler/rpcapi.py:132), timeout [DEFAULT]/long_rpc_timeout with a heartbeat every rpc_response_timeout.
• Purpose: return, for each instance, the chosen host plus alternates, with resources already claimed in Placement.
• Flow (nova/scheduler/manager.py:188-307):
  1. Request filters (nova/scheduler/request_filter.py:626 process_reqspec) translate the request into Placement constraints: tenant/AZ aggregates (require_tenant_aggregate, map_az_to_placement_aggregate), image type support trait, COMPUTE_STATUS_DISABLED forbidden trait, accelerators, routed networks, isolated aggregates, ephemeral encryption, virtio sound, TPM secret security, owner filter (nova/scheduler/request_filter.py:608 ALL_REQUEST_FILTERS). A failing request filter → NoValidHost.
  2. GET /allocation_candidates on Placement built from the flavor, image and port resource requests (nova/scheduler/manager.py:232-237); if CPU pinning is requested a second query with VCPU instead of PCPU is merged unless [workarounds]/disable_fallback_pcpu_query (nova/scheduler/manager.py:243-262). No candidates → log "Got no allocation candidates from the Placement API…" and NoValidHost (nova/scheduler/manager.py:276-283). The number of candidates is capped by [scheduler]/max_placement_results (default 1000, nova/conf/scheduler.py:63).
  3. Host states are built for the candidate providers only, from the cell compute_nodes tables and the in-memory instance tracking (nova/scheduler/manager.py:787).
  4. For each instance: run the enabled filters (default ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, nova/conf/scheduler.py:323-336) and weighers (all by default, nova/conf/scheduler.py:355), pick randomly among the best [filter_scheduler]/host_subset_size hosts (nova/scheduler/manager.py:740-785), then claim the allocation request for that host in Placement (PUT /allocations/{instance_uuid}, nova/scheduler/manager.py:489); a failed claim (concurrent consumer) moves on to the next host.
  5. Not enough hosts for all instances → allocations already claimed in this request are deleted (_cleanup_allocations) and NoValidHost is raised (nova/scheduler/manager.py:542-572).
  6. Alternates: up to max_attempts - 1 further hosts per instance, filtered and weighed but not claimed (nova/scheduler/manager.py:649).
• Concurrency: several schedulers run in parallel; correctness relies on Placement's consumer/provider generations, not on locks. Instance tracking in memory is eventually consistent.
• Error behaviour: NoValidHost is an expected exception of the RPC endpoint (nova/scheduler/manager.py:187), so it is re-raised in the conductor as is.
• Cost: 1–2 Placement GETs + one PUT per instance, plus host-state DB reads per cell.

3.2 Instance and aggregate tracking (fanout casts)

All are fanout casts to every scheduler worker (nova/scheduler/rpcapi.py:162-186).

3.3 _discover_hosts_in_cells (periodic)

• Trigger: oslo.service periodic task every [scheduler]/discover_hosts_in_cells_interval seconds (default -1 = disabled, nova/conf/scheduler.py:41), run at start (nova/scheduler/manager.py:108-110).
• Purpose: map newly started compute hosts to their cell (host_mappings rows), instead of running nova-manage cell_v2 discover_hosts.
• Concurrency: only the scheduler whose host name sorts first among the up schedulers runs it (nova/scheduler/manager.py:112-120); HostMappingExists is logged once as a warning when both mechanisms are used.
• Data read: compute_nodes / services of every cell not yet mapped (nova/objects/host_mapping.py discover_hosts).

4. nova-compute

nova-compute (nova/cmd/compute.py) runs one per hypervisor host (or per Ironic conductor group). It consumes topics compute and compute-alt with server=<host> (03 § 1.1), owns the virt driver selected by [DEFAULT]/compute_driver, and has no database access: every object load/save goes through the conductor (§ 2.1). For OpenKeel the relevant contract is the driver interface (§ 4.5): KubeVirt replaces the libvirt driver, so libvirt internals are out of scope and only listed.

4.1 build_and_run_instance (boot on the host)

• Trigger: RPC cast from the conductor (nova/conductor/manager.py:1857), handler nova/compute/manager.py:2396. The work runs in a background thread under a per-instance lock; the number of concurrent builds per host is bounded by [DEFAULT]/max_concurrent_builds (nova/conf/compute.py:657), a semaphore taken around _locked_do_build_and_run_instance (nova/compute/manager.py:2403-2408).
• Purpose: create the guest on this host from the scheduled request.
• Flow (nova/compute/manager.py:2644 _build_and_run_instance):
  1. read the instance's allocations from Placement (GET /allocations/{uuid}, nova/compute/manager.py:2682) and claim host resources in the local resource tracker (NUMA, PCI, ComputeResourcesUnavailable → reschedule) (nova/compute/manager.py:2688, nova/compute/resource_tracker.py:162);
  2. late server-group policy check: a hard (anti-)affinity violation raises GroupAffinityViolation → reschedule by policy (nova/compute/manager.py:2693);
  3. _build_resources (nova/compute/manager.py:2875): Cyborg accelerator requests; network allocation in Neutron started asynchronously (allocate_for_instance: create/update ports with binding:host_id, nova/compute/manager.py:1993-2015); task_state=block_device_mapping and volume attachment in Cinder (_prep_block_device, nova/compute/manager.py:2940-2944); image-backed root disk metadata;
  4. task_state=spawning, driver.spawn(...) (nova/compute/manager.py:2714): the driver fetches the image from Glance, creates disks, defines and starts the guest, and **waits for network-vif-plugged** events from Neutron for up to [DEFAULT]/vif_plugging_timeout (§ 02 § 4);
  5. on success vm_state=active, task_state=None, power_state from the driver, launched_at=now, instance.create.end notification, scheduler instance info update (nova/compute/manager.py:2858).
• Error behaviour (nova/compute/manager.py:2722-2815): ComputeResourcesUnavailable and generic spawn failures → RescheduledException (allocations on this host deleted, networks cleaned, compute_task.build_instances cast to the conductor with the remaining alternates, nova/compute/manager.py:2536-2582); invalid image/flavor/network errors → BuildAbortException → ERROR without reschedule; instance deleted meanwhile → silent cleanup.
• Cost: per boot, Placement 1 GET, Neutron ~2–4 calls per port, Cinder 2–3 calls per volume, Glance image download (cached per host by the image cache).

4.2 Other RPC endpoints

71 compute RPC methods are called by nova-api, nova-conductor or other computes. The table shows, for each: mode, the instance-action event recorded (compute_<method> via wrap_instance_event, nova/compute/utils.py:1457), the error decorators (wrap_instance_fault → instance_faults row, reverts_task_state → task_state=None on exception, errors_out_migration → migration error), and the virt driver / external calls found in the handler and its helpers up to two levels deep (calls made in threads or deeper helpers are not shown).

4.3 Periodic tasks (SQL polling through the conductor)

oslo.service periodic tasks of ComputeManager; tasks without spacing run on every tick (60 s by default), the first tick is delayed by a random [DEFAULT]/periodic_fuzzy_delay (≤ 60 s, nova/conf/service.py:66). All instance queries are InstanceList.get_by_filters(host=<this host>, …) executed by the conductor on the cell DB (often with use_slave=True).

Concurrency: periodic tasks run in the compute's own thread; each host only touches its own instances, so no cross-host lock is needed. Operations on one instance are serialised by utils.synchronized(instance.uuid) locks in the manager, and resource-tracker updates by the compute_resources semaphore (nova/compute/resource_tracker.py:58).

4.4 Driver events and the service heartbeat

• Lifecycle events: the driver emits started/stopped/paused/resumed/suspended events (libvirt domain events); handle_lifecycle_event (nova/compute/manager.py:1327) runs _sync_instance_power_state for the instance, with a delay (in the libvirt host event loop, nova/virt/libvirt/host.py:314-317) to ignore transient stops during reboots.
• Heartbeat: every [DEFAULT]/report_interval (10 s, nova/conf/service.py:25) each service updates services.report_count/updated_at through the conductor (nova/servicegroup/drivers/db.py). A service is down when its last report is older than [DEFAULT]/service_down_time (60 s, nova/conf/service.py:41). The API (check_instance_host(check_is_up=True), local delete, evacuate preconditions) and the scheduler's ComputeFilter rely on it.
• Startup (init_host, nova/compute/manager.py:1557): the compute reconciles instances left in transient states (e.g. deleting → finish delete; rebooting → reboot; migrating → recover), destroys evacuated instances still on the host, and registers its compute node and Placement provider.

4.5 Virt driver interface (the contract a KubeVirt driver implements)

nova.virt.driver.ComputeDriver (nova/virt/driver.py) is the boundary between the compute manager and the hypervisor. A KubeVirt-backed compute (OpenKeel) replaces nova/virt/libvirt/ behind this interface. The table lists every public method of the interface, whether the base class raises NotImplementedError (mandatory for the operations a deployment wants), whether the compute manager calls it, and which in-tree drivers implement it. Capabilities (ComputeDriver.capabilities, 35 keys, e.g. supports_evacuate, supports_attach_interface, supports_device_tagging, supports_multiattach, supports_trusted_certs, supports_image_type_*, supports_vtpm, supports_socket_pci_numa_affinity…) are published as Placement traits on the compute node provider (COMPUTE_*, nova/virt/driver.py:219 CAPABILITY_TRAITS_MAP, nova/virt/driver.py:1339), so the scheduler only selects hosts whose driver supports what the request needs; the API also rejects some operations based on them.

Source: reverse-spec-nova/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.