swift — Async processing

Processing outside the API — swift — branch stable/2026.2

Branch stable/2026.2 @ a1bdaeba21bd — repository https://opendev.org/openstack/swift — analysed on 2026-10-29

Swift has no message bus and no RPC server. Everything that is not a client request is done by daemons that walk the local disks on a timer and talk to their peers over the internal HTTP protocol and rsync (03-interactions_stable-2026.2.md). They are what makes Swift eventually consistent: a PUT is acknowledged on a quorum, and these loops repair the rest. Every daemon is a swift.common.daemon.Daemon with run_once / run_forever, an interval (sleep between passes, measured from pass start) and per-device or per-partition work; several support workers/processes to split devices between processes.

object-replicator

• Trigger: timer interval (or run_pause), run_forever (swift/obj/replicator.py:1150); one job per local partition and policy (swift/obj/replicator.py:834).
• Purpose: make every replica of a partition hold the same files, and move handoff partitions back to their primaries.
• Flow (primary partition) (swift/obj/replicator.py:649):
  1. Compute local suffix hashes (_get_hashes, which consolidates hashes.invalid, swift/obj/diskfile.py:1317).
  2. For each other primary: REPLICATE /<dev>/<part> → remote hashes (pickle) (swift/obj/replicator.py:728); suffixes whose hashes differ are recalculated locally, compared again, then pushed with rsync (default) or ssync (swift/obj/replicator.py:434, swift/obj/replicator.py:491).
  3. After rsync, REPLICATE /<dev>/<part>/<suf>-<suf> makes the remote rehash those suffixes (swift/obj/replicator.py:480).
• Flow (handoff partition): push every suffix to all primaries; when they all succeeded (or handoff_delete of them), delete the local partition (swift/obj/replicator.py:619). handoffs_first processes handoffs before primaries (swift/obj/replicator.py:189).
• Concurrency: concurrency greenthreads, replicator_workers processes splitting devices; the partition is locked with replication_lock only for ssync; rsync relies on --ignore-existing and the renames being atomic. A replayed pass is harmless (idempotent copy of immutable files).
• Errors: 507 → remote device skipped, next handoff used; rsync exit codes logged; lockup_timeout restarts a stuck pass. Stats dumped to object.recon (swift/common/recon.py:17).
• Cost: one REPLICATE per partition per peer per pass, plus a hash of every changed suffix; a full rehash (corrupt hashes.pkl) reads every directory of the partition.

object-reconstructor (EC)

• Purpose: rebuild missing EC fragments on primaries and revert fragments found on handoffs to their primary (swift/obj/reconstructor.py:1026, swift/obj/reconstructor.py:1082).
• Flow: for each EC partition, compare its fragment-index hashes with the two neighbours in the ring (left and right primaries) through REPLICATE (swift/obj/reconstructor.py:893); for each out-of-sync suffix open an SSYNC session; when the receiver lacks a fragment, the sender rebuilds it on the fly by GETting ec_ndata other fragments from the other primaries (X-Backend-Fragment-Preferences) and decoding/re-encoding with PyECLib (swift/obj/reconstructor.py:632, swift/obj/reconstructor.py:705). Handoff fragments are pushed with SSYNC to the node with the right backend_index, then deleted (swift/obj/reconstructor.py:973).
• Configuration: concurrency, reconstructor_workers, handoffs_only, quarantine_threshold (quarantine fragments that cannot be rebuilt), rebuild_handoff_node_count.
• Dependency: liberasurecode through PyECLib, same ec_type on all nodes (fragment format in 07 § 3.4).

object-auditor

• Purpose: detect bit rot and bad metadata before a client reads it.
• Flow: walk every object in every policy (swift/obj/diskfile.py:580); open it, read the whole file (or only zero-byte files in the ZBF pass, zero_byte_files_per_second 50) checking size and MD5 against the metadata, and metadata checksum (swift/obj/auditor.py:244). A failing file's hash directory is quarantined (swift/obj/diskfile.py:350) and the suffix invalidated, so that the replicator restores a good copy.
• Rate: files_per_second 20, bytes_per_second 10 MB/s (swift/obj/auditor.py:50); progress checkpointed in auditor_status_*.json per device (swift/obj/diskfile.py:652). Optional watchers plugins (dark-data detection, swift/obj/watchers).

object-updater

• Purpose: deliver the container listing updates that failed during PUT/DELETE.
• Data read: every file of async_pending[-N]/<suffix>/ (pickle, 03 § 4.4), newest timestamp per object first (swift/obj/updater.py:679).
• Flow (swift/obj/updater.py:769, swift/obj/updater.py:837): for each pending, send the stored op and headers to every primary of the container (or of container_path for a shard) (swift/obj/updater.py:934); when all succeed delete the file, otherwise rewrite it with the list of successful nodes and retry next pass; a 301 from a sharded root rewrites container_path (swift/obj/updater.py:899). Older pendings for the same object are deleted.
• Rate limiting: objects_per_second (50), max_objects_per_container_per_second (per-container buckets, deferred updates), concurrency 8, updater_workers (swift/obj/updater.py:426).
• Idempotency: container rows merge by timestamp (03 § 5.3), so replays are safe.

object-expirer

• Queue: when an object has X-Delete-At, the object server adds a row to the hidden account .expiring_objects, container delete_at // 86400 * 86400 − (hash(a/c/o) % 100) (swift/obj/expirer.py:97), object name <delete_at>-<account>/<container>/<object> (swift/obj/expirer.py:157), content type carrying the object's own ETag/size info (swift/obj/server.py:650).
• Flow (swift/obj/expirer.py:553): list task containers whose time has passed, list their task objects in round-robin order across target containers (swift/obj/expirer.py:352), for each: DELETE the target object with X-If-Delete-At: <delete_at> and X-Timestamp: <delete_at> through the InternalClient (object servers return 412 if the object was rewritten with another expiry, swift/obj/server.py:1354), then delete the task row; empty task containers are deleted.
• Concurrency: processes/process split the queue by hash between expirer instances (swift/obj/expirer.py:322); concurrency greenthreads.
• Visible effect: an expired object is already invisible (404) before the expirer runs (swift/obj/diskfile.py:2800); the expirer only frees space and listings.

container-replicator / account-replicator

• Trigger: timer; one item per DB file found under containers/ or accounts/ (swift/common/db_replicator.py:860).
• Flow (swift/common/db_replicator.py:659): read the replication info; for every primary (and handoffs when a primary is down) run the sync RPC; then in-sync / usync (rows since the remote's sync point, 1000 per merge_items, max 100 batches) / rsync + complete_rsync or rsync_then_merge (03 § 2.8). A DB on a handoff is deleted after it reached all primaries. Deleted DBs older than reclaim_age are removed (swift/common/db_replicator.py).
• Container specifics (swift/container/replicator.py:68): replicate shard ranges first (merge_shard_ranges); enqueue rows whose storage_policy_index differs from the container's into .misplaced_objects for the reconciler (swift/container/reconciler.py:190).
• Cost: one sync per DB per peer per pass; large DBs rsync whole.

container-updater

• Purpose: push each container's object count, bytes and timestamps to its account (the account listing shows them).
• Flow: walk container DBs whose reported_* differ from current values; PUT /<dev>/<part>/<account>/<container> to each account primary with X-Put-Timestamp, X-Delete-Timestamp, X-Object-Count, X-Bytes-Used, X-Backend-Storage-Policy-Index (swift/container/updater.py:313); on quorum, store the reported values. Rate containers_per_second 50 (swift/container/updater.py:61). Accounts too slow to answer are skipped for account_suppression_time.

container-auditor / account-auditor

Open each DB, call get_info() (and for containers check the policy and shard ranges), quarantine malformed DBs (swift/common/db_auditor.py:31, swift/common/db.py:502). Rate containers_per_second / accounts_per_second 200.

container-sharder

• Purpose: split containers with too many rows (default shard_container_threshold 1 000 000, swift/container/sharder.py:783) into shard containers held in the hidden account .shards_<account>, and shrink nearly empty shards back.
• Flow (swift/container/sharder.py:2396): on the node that owns index 0 of the root, find split points (rows_per_shard, swift/container/sharder.py:817) when auto_shard = true (swift/container/sharder.py:805; otherwise operators use swift-manage-shard-ranges), store shard_range rows, set the root to sharding; on every replica, create the new <hash>_<epoch>.db, cleave rows range by range into shard containers (created through direct PUTs and replicated with quorum shard_replication_quorum) (swift/container/sharder.py:2139); when done set sharded. Objects that land in the wrong DB are moved (swift/container/sharder.py:1835); shard stats are reported to the root (swift/container/sharder.py:2352); audits check shard range consistency (swift/container/sharder.py:1550).
• Effects elsewhere: proxies switch to shard listings (01 § 3.2); object servers receive redirects (301) and update shards.
• Concurrency: one sharder per device set; the root's leader is the replica at index 0; cleaving is resumable (CleavingContext saved in the DB metadata, swift/container/sharder.py:589).

container-reconciler

• Purpose: fix objects written to the wrong policy after a split-brain on container creation (two replicas created with different policies).
• Queue: account .misplaced_objects, container = hour of the object timestamp (swift/container/reconciler.py:134), object <policy_index>:/<account>/<container>/<object> (swift/container/reconciler.py:151).
• Flow (swift/container/reconciler.py:841): for each entry, find the container's real policy (oldest status_changed_at wins), copy the object from the wrong policy to the right one (keeping its timestamp, with an offset), delete it from the wrong policy, remove the entry.

container-sync

Every interval, for each local container with X-Container-Sync-To/-Key, read rows since the stored sync points (x_container_sync_point1/2 in container_info), and for each row this node is responsible for (one third of rows on each replica, then the rest as a safety net) send PUT/DELETE of the object to the remote cluster with X-Container-Sync-Key or realm signature X-Container-Sync-Auth, through sync_proxy (swift/container/sync.py:77, swift/container/sync.py:167). container_time (60 s) bounds the time per container (swift/container/sync.py:152).

account-reaper

• Trigger: account DBs in DELETED status found under accounts/, older than delay_reaping (swift/account/reaper.py:83).
• Flow: only the node holding the first replica of the account partition works on it (swift/account/reaper.py:149); for each container, DELETE every object (direct to object servers, all policies) then the container, then the account is left for the replicator to reclaim (swift/account/reaper.py:213, swift/account/reaper.py:324).

Lifecycle of a container DB

stateDiagram-v2
  [*] --> unsharded: first PUT (broker initialize)
  unsharded --> sharding: sharder finds or is given shard ranges
  sharding --> sharded: all ranges cleaved
  sharded --> collapsed: all shards shrunk into the root
  collapsed --> sharding: new shard ranges
  unsharded --> deleted: DELETE (empty), put_timestamp < delete_timestamp
  deleted --> unsharded: PUT with newer timestamp
  deleted --> [*]: replicator reclaims after reclaim_age

Kubernetes mapping notes (for OpenKeel)

• These loops assume each process sees the local devices under /srv/node. In a StatefulSet of storage pods with one PVC per device, each daemon runs in the pod that mounts the PVCs (sidecars or one multi-process container), never in a separate Deployment.
• rsyncd must run in every storage pod while any Python replicator remains (03 § 3).
• .expiring_objects, .misplaced_objects and .shards_* are ordinary hidden accounts: the expirer and reconciler can run anywhere they can reach the storage network (they use an InternalClient pipeline).
• Recon cache files (/var/cache/swift) and auditor_status_*.json are node-local state; losing them only costs a rescan.

Source: reverse-spec-swift/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.