cinder — Async processing

Processing outside the API — cinder — branch stable/2026.2

Branch stable/2026.2 @ c117f9cf08b4 (29.0.0-1-gc117f9cf0) — repository https://opendev.org/openstack/cinder — analysed on 2026-10-28

Almost every Cinder write operation finishes outside the API: the API answers 202 after writing the database and casting an RPC message. This file lists what the three back-end services do on their own: RPC consumers, timers and startup processing. Vendor drivers' internal polling loops (about 40 FixedIntervalLoopingCalls in cinder/volume/drivers/*, waiting for array jobs) are out of scope.

1. Summary

There is no notification listener, no OVSDB event and no cache polling in Cinder. Its "polling" is the capability reporting from volume services to schedulers and the timer-driven cleanups above.

2. cinder-scheduler

2.1 create_volume (RPC cast)

• Trigger: cast on topic cinder-scheduler, from VolumeCastTask of the API (cinder/volume/flows/api/create_volume.py:785-868). Any scheduler instance may consume it.
• Emitter: cinder-api; also cinder-volume when rescheduling a failed creation.
• Purpose: choose the backend and pool that will create the volume.
• Data read: the in-memory state of each live backend and pool (capacity, capabilities, last update), refreshed from services (cinder/scheduler/host_manager.py:680-700); the volume type's extra specs.
• Flow:
  1. Wait, at most scheduler_driver_init_wait_time s (60) after startup, until every volume service has reported its capabilities (cinder/scheduler/manager.py:113-121, cinder/scheduler/manager.py:174-185).
  2. Run the scheduler flow: ExtractSchedulerSpecTask, then ScheduleCreateVolumeTask (cinder/scheduler/flows/create_volume.py:36-160).
  3. The filter scheduler applies the filters (default AvailabilityZoneFilter, CapacityFilter, CapabilitiesFilter, cinder/scheduler/host_manager.py:45-57), then the weighers (default CapacityWeigher), and removes backends already tried (RetryFilter, at most scheduler_max_attempts = 3) (cinder/scheduler/filter_scheduler.py:297-364, cinder/scheduler/filter_scheduler.py:539-568). When the request is bound to a backend (group, snapshot_same_host, clone), only that backend is considered.
  4. Write volumes.host = host@backend#pool and cast create_volume to that backend (cinder/scheduler/filter_scheduler.py:94-121).
• Error behaviour: NoValidBackend → volume error, user message VOLUME_VOLUME_001_003, notification scheduler.create_volume (ERROR) (cinder/scheduler/flows/create_volume.py:109-160). Other exceptions also set error and are logged.
• Concurrency: several schedulers keep independent caches. Each one consumes capacity in its own cache when it places a volume (consume_from_volume) [inferred], so two schedulers can choose the same nearly full pool; the backend then fails and the volume is rescheduled.
• Cost: no SQL per candidate; one services read per request (_update_backend_state_map).

2.2 Capacity filter and weigher

• CapacityFilter (cinder/scheduler/filters/capacity_filter.py:32-200): free = free_capacity_gb − total × reserved_percentage/100. For thin provisioning (thin_provisioning_support and not provisioning:type=thick), the virtual free capacity is total × max_over_subscription_ratio − provisioned − reserved, and the request passes only if (provisioned + size) / (total − reserved) ≤ max_over_subscription_ratio. Backends reporting infinite/unknown pass.
• CapacityWeigher (cinder/scheduler/weights/capacity.py:93-140): weight = virtual free capacity × capacity_weight_multiplier (1.0); AllocatedCapacityWeigher and VolumeNumberWeigher are optional (multipliers −1.0).

2.3 Other scheduler RPCs

• extend_volume: re-checks the capacity of the volume's own backend (backend_passes_filters) before casting extend_volume to it; failure → volume back to its previous status, quota rolled back, user message …007… [inferred for the message] (cinder/scheduler/manager.py:457-503).
• migrate_volume, retype, manage_existing*: find or validate a target backend, then cast to the volume service (cinder/scheduler/manager.py:281-430).
• create_backup: pick a live cinder-backup in the backup's AZ (get_backup_host), save backups.host, cast to it; no backup service → backup error, volume back to its previous status, user message BACKUP_SCHEDULE_ERROR (cinder/scheduler/manager.py:637-661).
• work_cleanup: for POST /workers/cleanup, select the services concerned and cast do_cleanup to each (cinder/scheduler/manager.py:577-636).

3. cinder-volume

3.1 create_volume (RPC cast)

• Trigger: cast from the scheduler to cinder-volume.<host>@<backend>.
• Purpose: create the volume on the array, optionally fill it, and make it available.
• Flow (cinder/volume/flows/manager/create_volume.py:1354-1413):
  1. ExtractVolumeRefTask, OnFailureRescheduleTask (reschedules through the scheduler unless the failure is not reschedulable, cinder/volume/flows/manager/create_volume.py:72-222), ExtractVolumeSpecTask, notification volume.create.start.
  2. CreateVolumeFromSpecTask: raw (driver.create_volume), from snapshot, clone, from image (download through Glance into the volume, or image-volume cache: clone of a cached volume of the image, lock cinder-<image_id>), from backup (cinder/volume/flows/manager/create_volume.py:384-1300).
  3. CreateVolumeOnFinishTask: status available, launched_at, notification volume.create.end (cinder/volume/flows/manager/create_volume.py:1303-1349).
• Error behaviour: after the last reschedule, the volume is error, with a user message (…_005_… for image copies, …_008_012 for driver failures). The quota stays consumed until the volume is deleted.
• Concurrency and idempotency: the workers row created for the volume lets another service of the same cluster finish or clean the job if this one dies (@cleanable, cinder/manager.py:265-330).

3.2 delete_volume (RPC cast, lock cinder-<volume_id>-delete_volume)

Unexport, delete on the array (driver.delete_volume), delete snapshots if cascade, release quotas (volumes, gigabytes, per type), soft-delete the row, notifications volume.delete.start/end (cinder/volume/manager.py:929-1150). A VolumeIsBusy from the driver puts the volume back to available; other errors to error_deleting.

3.3 Attach and detach (RPC calls)

• attachment_update (call, from POST/PUT /attachments): driver.create_export and driver.initialize_connection(connector); stores connection_info and connector in volume_attachment, status attaching (cinder/volume/manager.py:4929-5033).
• attachment_delete (call): driver.terminate_connection and, when no attachment is left on the host, remove_export; the volume becomes available or stays in-use (cinder/volume/manager.py:5035-5065).
• Legacy attach_volume/detach_volume/initialize_connection/terminate_connection work the same way for the os-* actions (cinder/volume/manager.py:1399-2120).

3.4 extend_volume (RPC cast)

driver.extend_volume; on success extend_volume_completion: quota commit, size and status (available or in-use) updated, notification volume.resize.end; then, for each attached instance, Nova POST /os-server-external-events volume-extended (2.51). A failed Nova call only records user message VOLUME_VOLUME_007_009. A driver failure → error_extending, quota rolled back, user message …007_010 (cinder/volume/manager.py:2967-3070, cinder/compute/nova.py:216-227).

3.5 publish_service_capabilities (periodic)

• Trigger: periodic task every backend_stats_polling_interval (60 s); also a fanout cast from the API's publish_service_capabilities and at startup.
• Data read: driver.get_volume_stats(refresh=True), which queries the array; plus allocated_capacity_gb kept in memory.
• Effects: fanout cast update_service_capabilities to every scheduler with the pools' capacities and capabilities (thin_provisioning_support, max_over_subscription_ratio, reserved_percentage, multiattach, QoS_support, replication_enabled, filter_function, goodness_function…), plus capacity.pool notifications (cinder/volume/manager.py:2743-2906).
• Error behaviour: a driver that is not initialised reports nothing, and the backend disappears from the scheduler's view; the heartbeat stops too (is_working).
• Cost: one array query per backend per minute, one fanout message per backend per minute, received by every scheduler.

3.6 Startup cleanup (init_host)

• Driver initialisation, retried reinit_driver_count times (3) with a 2^n s backoff (cinder/volume/manager.py:484-511).
• Re-export of in-use volumes; failure → error (cinder/volume/manager.py:597-604).
• Worker cleanup (CleanableManager.do_cleanup, cinder/manager.py:265-330): for each workers row of this service, check the resource still has the recorded status, then: downloading → driver clears the download, uploading → status back from the attachments, deleting → resume the deletion, creating/downloading → error (cinder/volume/manager.py:691-724). The same is triggered remotely by POST /workers/cleanup for a dead service.
• Finally, publish capabilities and start the RPC servers.

4. cinder-backup

4.1 create_backup (RPC cast)

• Trigger: cast from the scheduler to cinder-backup.<host>.
• Flow [inferred, summarised from cinder/backup/manager.py]: call get_backup_device on the volume's service (a temporary snapshot or clone for an in-use volume), attach it locally with os-brick, stream it to the backup driver (Swift by default, cinder.backup.drivers.swift.SwiftBackupDriver; chunked drivers compress, checksum and upload chunks, with backup.createprogress notifications), detach, remove temporary resources, set backup available and the volume back to its previous status.
• Error behaviour: backup error with fail_reason, volume back to its previous status, user messages …013_017–…013_023.

4.2 Restore and delete

restore_backup writes into the target volume (restoring-backup → available, or error_restoring); delete_backup deletes the objects then the row (quota backups, backup_gigabytes released).

5. Commands run by operators

• cinder-manage db sync | purge <days> | online_data_migrations, cinder-manage service list | remove, cinder-manage cluster list | remove | rename, cinder-manage volume delete | update_host, cinder-manage quota check | sync, cinder-manage util clean_locks (cinder/cmd/manage.py).
• cinder-status upgrade check (cinder/cmd/status.py).
• cinder-volume-usage-audit --start_time … --end_time … for billing (cinder/cmd/volume_usage_audit.py).

Source: reverse-spec-cinder/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.