zun — Async processing

Asynchronous processing — zun — branch stable/2026.2

Branch stable/2026.2 @ 34135a563699 — repository https://opendev.org/openstack/zun — analysed on 2026-11-01

No notification listener, no OVSDB event, no tooz or etcd coordination. zun-compute is the only RPC server, and each RPC cast handler hands work to a green thread immediately (zun/common/utils.py:193).

Summary

Periodic tasks start in zun/common/rpc_service.py:62: heartbeat, then init_containers, then the dynamic timer. All run under an admin context with all projects (zun/common/rpc_service.py:66).

RPC server

• Target: topic [compute]/topic, default zun-compute, server CONF.host (zun/conf/compute.py:64, zun/cmd/compute.py:47). No version on the target (zun/common/rpc_service.py:51), so RPC versioning is unused: a rewrite must treat the method names and arguments as the contract.
• Executor: eventlet (zun/common/rpc_service.py:55).
• Concurrency: each cast handler takes a per-container lock (utils.synchronized(container.uuid)) in the green thread, so actions on one container run one at a time (zun/compute/manager.py:292). The lock is in-process (lockutils.synchronized_with_prefix, no external, zun/common/utils.py:52), which is enough because a container lives on one node.
• Replay: a cast is not acknowledged at the business level. A lost message leaves task_state empty and the status unchanged, with no timeout on the API side.

zun-compute: creation

container_create

• Trigger: RPC cast zun-compute.<host>, from zun/compute/rpcapi.py:58.
• Purpose: turn a Creating row into a Docker container, or a capsule with its members.
• Flow (zun/compute/manager.py:292):
  1. Wait for each requested volume to be available, or in-use when multi-attach, polling every second up to [volume]/timeout_wait_volume_available, 60 s (zun/compute/manager.py:195, zun/volume/driver.py:204). Error or timeout: created volumes removed, Error.
  2. Attach volumes: mapping row, then Cinder and a mount, or a local file (zun/compute/manager.py:419, zun/compute/manager.py:446).
  3. Check disk quota support, set disk (zun/compute/manager.py:252).
  4. Resource claim under the compute_resources lock, and set container.host (zun/compute/compute_node_tracker.py:136, zun/compute/compute_node_tracker.py:218, zun/compute/claims.py:120).
  5. Task state container_creating; pull the image (zun/compute/manager.py:330, zun/compute/manager.py:340): see § images in 03.
  6. Create: Docker driver for a container, capsule driver for a capsule (zun/compute/manager.py:369).
  7. If run: docker start, then network hook (zun/compute/manager.py:303, zun/container/docker/driver.py:791).
• Docker create (zun/container/docker/driver.py:246): network provisioned on the node, volumes bound, exposed ports turned into a security group, first network through process_networking_config, the others after create; mem_limit, cpu_shares = 1024 * cpu, storage_opt.size, cpuset_cpus when dedicated, privileged, runtime, restart_policy, healthcheck (zun/container/docker/driver.py:307, zun/container/docker/driver.py:320, zun/container/docker/driver.py:322). Name: zun-<uuid> (zun/common/consts.py:56, zun/container/docker/driver.py:1048).
• **Capsule, default cri driver** (zun/container/cri/driver.py:49, zun/conf/container_driver.py:33): RunPodSandbox over gRPC with runtime_handler, the Neutron port and VIF written to cni_metadata, which zun-cni reads, then init containers in order, waited for up to 3600 s, then containers (zun/container/cri/driver.py:72, zun/container/cri/driver.py:96, zun/container/cri/driver.py:226).
• **Capsule, docker driver**: same order with Docker, network_mode: container:<sandbox> and ipc_mode shared, pid_mode left commented out (zun/container/docker/driver.py:1199, zun/container/docker/driver.py:1255).
• Claim failure: ResourcesUnavailable, then Error, host cleared, Placement allocation removed (zun/compute/manager.py:404).
• Other failures: image errors give Error keeping the host; Docker create errors give Error with host cleared (zun/compute/manager.py:346, zun/compute/manager.py:382). Placement allocation is not removed on those branches, and the claim's __exit__ handles local usage [inferred from zun/compute/manager.py:399].
• Cost: one Docker call set, one to several Neutron calls, one to several Cinder calls.

Network configuration

Two drivers, selected by [network]/driver, default kuryr (zun/conf/network.py:20):

• **kuryr** (zun/network/kuryr_network.py:200): one Docker network per Neutron network per node, created with the driver kuryr and options giving the subnets. Docker then asks Kuryr to bind the port, which Zun created with an admin Neutron client (zun/network/neutron.py:79). Concurrency: the row in table network with its unique key on (neutron_net_id, host) serialises creators; losers retry with back-off (zun/network/kuryr_network.py:119, zun/network/kuryr_network.py:129).
• **cni** (zun/network/cni_network.py:88): network_mode: none, and the entrypoint is wrapped so that the process waits for a file /zun-init-network-ready before starting (zun/network/cni_network.py:35, zun/network/cni_network.py:112). After docker start, on_container_started runs zun-cni ADD, then creates the file inside the container (zun/network/cni_network.py:152). This requires /bin/sh and touch in the image [inferred from the command].

zun-compute: deletion

container_delete

zun/compute/manager.py:545:

1. Task state container_deleting.
2. Driver delete: networks disconnected, the exposed-port security group deleted (the first of the list, on the assumption that it is the dedicated one, zun/container/docker/driver.py:468), docker rm; a Docker 404 is treated as success (zun/container/docker/driver.py:430).
3. Volumes detached and auto-remove ones deleted, polling up to timeout_wait_volume_deleted, 60 s (zun/compute/manager.py:484, zun/compute/manager.py:226).
4. Local usage released, Placement allocation deleted (zun/compute/manager.py:569).
5. Row destroyed, only after Placement succeeded (zun/compute/manager.py:573).

Errors: without force, status Error and the row stays; with force, the exception is swallowed and cleanup continues (zun/compute/manager.py:549, zun/compute/manager.py:557).

Timers

sync_container_state

• Trigger: sync_container_state_interval, 60 s, at start-up too (zun/compute/manager.py:1234, zun/conf/services.py:29).
• Purpose: reflect what Docker knows; detect containers that vanished.
• Data read: all containers of all projects, SELECT … FROM container with no host filter (zun/compute/manager.py:1240), and all capsules, then Docker's list; rows for this host and for UUIDs found in Docker (zun/container/docker/driver.py:538).
• Flow (zun/container/docker/driver.py:545): status copied when different; host rewritten to this node when a local container points elsewhere; missing containers: auto_remove gives Deleted, others go to heal_with_rebuilding_container.
• Healing: with auto_heal and a rebuildable status it should rebuild, otherwise Error with No such container (zun/container/docker/driver.py:506). Bug: manager.container_rebuild(context, container) lacks the run argument, so the TypeError is caught and logged as a warning; automatic healing never rebuilds (zun/container/docker/driver.py:518, zun/compute/manager.py:675).
• Concurrency: task_state set means no status update (zun/container/docker/driver.py:645). Rows in Creating, Deleting or Deleted are skipped (zun/container/docker/driver.py:490).
• Cost: one full-table read per node per minute, so O(nodes × containers) on the database. A point of attention on a large fleet.

inventory_host

• Flow: resources read from Docker, /proc, disk (zun/container/driver.py:167, zun/container/docker/driver.py:1125); compute_node row created or updated (zun/compute/compute_node_tracker.py:61); usage recomputed from containers on the host (zun/compute/compute_node_tracker.py:437); inventory, traits and aggregates pushed to Placement when changed (zun/compute/compute_node_tracker.py:341).
• Provider: UUID of the compute_node row; or with [compute]/host_shared_with_nova, Nova's provider found by name (zun/compute/compute_node_tracker.py:106, zun/conf/compute.py:57).
• Failure: ResourceProviderUpdateConflict is retried up to 4 attempts (zun/compute/compute_node_tracker.py:338).

delete_unused_containers

auto_remove true, status Deleted, task state empty: calls container_delete(force=True) (zun/compute/manager.py:1205). Stops the whole cycle at the first exception. The query has no host filter, so every node tries to delete every such container, even if it does not hold it [inferred from zun/compute/manager.py:1216]. With the container absent locally, the Docker delete is a no-op and the row is removed.

update_zun_service

Creates the zun_service row if needed, then report_state_up (zun/servicegroup/zun_service_periodic.py:40). A node is down after service_down_time, 180 s (zun/api/servicegroup.py:26). A dead node does not trigger evacuation or rescheduling: Zun has no such process. Its containers stay in the database, and the API refuses actions with ContainerHostNotUp (zun/compute/rpcapi.py:25).

Recovery at start-up

init_containers (zun/compute/manager.py:80):

zun-cni-daemon

Started by zun/cni/cmd/cni_daemon.py:25: two workers and a shared multiprocessing.Manager dict registry (zun/cni/daemon/service.py:221).

• **add**: called by zun-cni from containerd or from the Zun cni driver. Reads the VIF from cni_metadata, plugs it with os-vif and pyroute2, waits up to vif_active_timeout, 60 s, for Neutron to report the port ACTIVE (zun/cni/plugins/zun_cni_registry.py:51, zun/cni/plugins/zun_cni_registry.py:72). A timeout gives 500, not the 504 the daemon intends, because the retry library raises RetryError (02 T21, zun/cni/daemon/service.py:69).
• **delete**: unplugs; ResourceNotReady is ignored with 204 (zun/cni/daemon/service.py:77).
• **poll_vif_status**: every second, when any VIF is inactive, lists Neutron ports for binding:host_id of this node (zun/cni/daemon/service.py:184). One REST call per second per node while a VIF waits.
• **sync_containers**: every 60 s, entries with no DB row are marked or removed (zun/cni/daemon/service.py:147).

zun-wsproxy

zun/websocket/websocketproxy.py:174. Reads uuid, token, optional exec_id from the URL, loads the container as admin, compares the token, checks the Origin, then connects to Docker over its TCP API (docker_remote_api_url, tcp://$my_ip:2375 by default) and copies bytes both ways (zun/websocket/websocketproxy.py:209, zun/websocket/websocketproxy.py:238, zun/websocket/websocketproxy.py:268, zun/conf/docker.py:31).

• Token: a UUID stored on the row. It does not expire and is not single-use: it stays valid until the next attach overwrites it (zun/compute/manager.py:933). exec_instance rows live as long as the container.
• Not authenticated by Keystone: the token alone gives access to the console.

Source: reverse-spec-zun/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.