heat — Async processing

Processing outside the API — heat — branch stable/2026.2

Branch stable/2026.2 @ 5872396e548c (27.0.1) — repository https://opendev.org/openstack/heat — analysed on 2026-10-29

Almost everything Heat does happens in heat-engine, after the API has answered. This file covers the engine's RPC servers, the convergence engine (the default, [DEFAULT]/convergence_engine = True, heat/common/config.py:247), the legacy engine still used for stacks created before convergence, the timers, and the processing triggered by signals and software deployments. The RPC frames themselves are in 03.

Summary

client_call_found: three EngineService methods are public but called by no client in Heat (reset_stack_status, service_manage_report, service_manage_cleanup), and two WorkerService methods (stop_traversal, stop_all_workers) — oslo.messaging's DefaultRPCAccessPolicy (heat/common/messaging.py:122) exposes every public method of an endpoint, so anything that can publish on the engine/engine_worker topics can invoke them ([_rewrite/rpc-contract_stable-2026.2.json](_rewrite/rpc-contract_stable-2026.2.json)).

1. Engine processes and identities

• heat-engine (heat/cmd/engine.py) forks [DEFAULT]/num_engine_workers processes (default max(4, CPU count)), each running one EngineService with the threading backend of oslo.service (no eventlet on this branch).
• At start() each process generates a random engine id (UUID4, heat/common/service_utils.py:80-81) and starts three RPC servers (heat/engine/service.py:445-478):
  • engine topic, server [DEFAULT]/host: the API ↔ engine contract;
  • engine_worker topic, server = engine id: the convergence workers (heat/engine/worker.py:85-97);
  • heat-engine-listener topic, server = engine id: liveness and legacy thread control (heat/engine/service.py:369-374).
• Locks (stack lock, resource lock) store the engine id. A lock is stale when the engine that holds it does not answer a listening call within [DEFAULT]/engine_life_check_timeout (2 s) (heat/rpc/listener_client.py:36-48). There is no lease or TTL: liveness is only this RPC probe.
• The database max_overflow is raised to executor_thread_pool_size so that every RPC thread can get a connection (heat/engine/service.py:480-490).

2. Convergence engine

2.1 Data model

2.2 Starting a traversal: Stack.converge_stack

heat/engine/stack.py:1445-1504, then _converge_create_or_update in a thread (heat/engine/stack.py:1507-1566):

1. For CREATE/UPDATE/ROLLBACK/DELETE/RESTORE/ADOPT the stack definition switches to the new template (prev_raw_template_id keeps the old one, except for CREATE/ADOPT); CHECK/SUSPEND/RESUME/SNAPSHOT keep the current template (heat/engine/stack.py:1449-1462).
2. action, IN_PROGRESS, reason "Stack <ACTION> started", and a new traversal id are stored with UPDATE stack … WHERE current_traversal = <previous> (heat/engine/stack.py:1477-1493, heat/db/api.py:1003-1028). If another request changed the traversal in between, nothing happens (logged warning). This compare-and-swap is what makes concurrent updates safe without a stack lock.
3. The previous traversal's sync points are deleted (heat/engine/stack.py:1496-1498): workers still running for it will find their sync point missing and stop propagating.
4. (thread) _update_or_store_resources: for each resource of the new template, pick the best existing row of that name — score −30 if FAILED, −50 if DELETE, −1 if replaced, +5 if its template is the previous one, +10 if it is the current one (rollback), ties broken by last change (heat/engine/stack.py:1587-1617); create an INIT_COMPLETE row when none exists (heat/engine/stack.py:1619-1632).
5. _compute_convg_dependencies (heat/engine/stack.py:1634-1666):
   • the template's dependency graph translated to update nodes (row id, true);
   • for every existing row: a cleanup node (id, false); edges cleanup(req) → cleanup(id) for each requires (a resource is deleted after what depends on it), cleanup(replaces) → cleanup(id), and cleanup(id) → update(id) when the row is also updated (clean up after update);
   • for DELETE, one independent snapshot node per snapshot of the stack.
6. The edges are stored in current_deps (with the traversal check), pre_converge runs (DELETE: cancel the old workers), a sync_point is created for each node and for the stack (heat/engine/stack.py:1534-1541).
7. The leaves (nodes with no requirement) get a check_resource cast each, update nodes after cleanup nodes; no leaf → mark_complete immediately (heat/engine/stack.py:1543-1566).

2.3 Processing a node: check_resource

Received by any engine (engine_worker topic, no server: round-robin), heat/engine/worker.py:202-248:

1. Deserialise data ({"input_data": {...}}, tuple keys restored, heat/engine/sync_point.py:83-117) into NodeData per predecessor (heat/engine/node_data.py:114-124): id, name, reference id, attributes, action, status, uuid of each resource this one depends on.
2. Resource.load (heat/engine/resource.py:323-383): the resource row; the stack with the latest definition; but the resource is instantiated from the definition of its own current_template_id when it differs (so updates compare old and new properties); a type change is honoured only if the new type is a valid substitute.
3. If current_traversal of the message is no longer the stack's: re-trigger (_retrigger_replaced, heat/engine/worker.py:154-166) — a replacement not needed by the new traversal is marked DELETE so it gets purged, and its predecessor is re-triggered in the new traversal.
4. Otherwise CheckResource.check (heat/engine/check_resource.py:327-369):
   • stack timed out (time_elapsed > timeout_secs, computed from updated_at or created_at) → mark_failed('Timed out') and stop (heat/engine/check_resource.py:335-337, heat/engine/stack.py:2444-2458);
   • update node: create_convergence if the resource is INIT, update_convergence otherwise, check/resume/snapshot_convergence for those stack actions, nothing for SUSPEND (heat/engine/check_resource.py:517-532);
   • cleanup node: delete_convergence (deletes only if current_template_id ≠ traversal template; an INIT row is just deleted from the database), suspend_convergence for SUSPEND, nothing for CHECK/RESUME/SNAPSHOT (heat/engine/check_resource.py:535-545, heat/engine/resource.py:1976-2001);
   • each operation holds the resource lock for its whole duration: state_set(<ACTION>, IN_PROGRESS, lock=LOCK_ACQUIRE) writes engine_id = <this engine> only if engine_id IS NULL AND atomic_key = <n>; the final state_set(COMPLETE|FAILED, lock=LOCK_RELEASE) writes engine_id = NULL only if engine_id = <this engine> (heat/engine/resource.py:861-932, heat/engine/resource.py:2159-2183). UPDATE and DELETE retry the acquisition client_retry_limit (2) times with a random wait ≤ 2 s (heat/engine/resource.py:878-898).
5. On success, _initiate_propagate_resource (heat/engine/check_resource.py:231-303):
   • a replacement impersonates the node of the resource it replaces (heat/engine/check_resource.py:238-245);
   • for each node that requires this one (deps.required_by), cleanup nodes first: the data sent is the resource's node_data().as_dict() (attributes resolved once and forwarded) for update nodes, or the id of the resource (or its replacement) for cleanup nodes;
   • propagate_check_resource → sync_point.sync on the dependant's sync point; when all its predecessors have reported, the dependant gets its own check_resource cast with the accumulated input_data (heat/engine/check_resource.py:479-500);
   • resolved attributes are stored in resource_properties_data (attr_data_id) for show_stack/outputs (heat/engine/check_resource.py:278-284, heat/engine/resource.py:2369-2386);
   • check_stack_complete: if this node is a root of the graph, it reports to the stack's sync point; when all roots have reported, mark_complete (heat/engine/check_resource.py:447-476).
6. Replacement (UpdateReplace raised by the update): make_replacement inserts the new row (replaces = old id, INIT_COMPLETE, requires) and sets replaced_by on the old row in one transaction, conditional on the old row's atomic_key and engine_id IS NULL (heat/engine/resource.py:385-437, heat/db/api.py:287-309); then a check_resource cast for the new row in the same traversal (heat/engine/check_resource.py:130-152). The old row is deleted later by its cleanup node.

2.4 Sync points

heat/engine/sync_point.py:132-207:

• update_sync_point is an optimistic loop: read the row, merge the new predecessor data into input_data (and failures / skip_propagate into extra_data), then UPDATE sync_point SET input_data=?, extra_data=?, atomic_key = n+1 WHERE entity_id=? AND traversal_id=? AND is_update=? AND atomic_key = n (heat/db/api.py:2120-2135); retry until one row is updated, with exponential random jitter (tenacity.wait_random_exponential(max=60), multiplier 10 ms per expected concurrent writer, capped at 10 s). There is no retry limit.
• A missing sync point (deleted by a newer traversal) raises EntityNotFound(entity="Sync Point") (heat/engine/sync_point.py:46-55): the sender reloads the stack and re-triggers itself into the new traversal if needed (heat/engine/check_resource.py:288-303).
• sync calls the propagation callback when predecessors − keys(input_data) is empty (heat/engine/sync_point.py:198-207). Because every predecessor writes its own key, a node becomes ready exactly once — unless two predecessors' final writes both observe "ready": the CAS on atomic_key prevents that (only one update succeeds per atomic_key value, and the second re-reads).
• Serialisation of keys: tuple keys of the graph (ConvergenceNode, 3 fields) are stored as the Python repr of the tuple, prefixed with tuple: — for example "tuple:(42, True, 'resource')" — and parsed back with ast.literal_eval (heat/engine/sync_point.py:74-102). The same encoding applies to nested dict keys (attribute paths in attrs). A non-Python engine must emit and parse exactly this repr (True/False, single quotes, ', ' separators).
• extra_data holds {"extra_data": {"resource_failures": {"<resource name or id>": "<reason>"}, "skip_propagate": true}}, used by the CHECK action to keep traversing after failures and report them all at the end (heat/engine/check_resource.py:96-117, heat/engine/check_resource.py:461-472).

2.5 End of a traversal

• mark_complete (heat/engine/stack.py:2362-2380): state_set(<ACTION>, COMPLETE) guarded by the traversal id; SNAPSHOT also writes the snapshot data; then purge_db.
• mark_failed (heat/engine/stack.py:2334-2360): state_set(<ACTION>, FAILED, reason); if rollback is enabled (disable_rollback = false) and the action is CREATE/ADOPT/UPDATE/RESTORE, start a ROLLBACK traversal to the previous template (or an empty template after a failed create) (heat/engine/stack.py:1568-1585); otherwise purge_db.
• purge_db (heat/engine/stack.py:2392-2442): delete the DELETE_COMPLETE resource rows and their attribute data, set current_traversal = '' if FAILED (so late workers stop), drop prev_raw_template_id (and the template) after success, delete the traversal's sync points, and on DELETE_COMPLETE delete the trust/credentials and soft-delete the stack row.

2.6 Concurrency and cancellation

• New update during an update: allowed; the new traversal id makes the old one's sync points disappear. A worker of the old traversal that finishes a resource re-triggers the node in the new traversal (heat/engine/check_resource.py:119-124, heat/engine/check_resource.py:207-229). A worker that cannot lock a resource (UpdateInProgress) checks whether the holder is alive: if not, it steals the lock (sets engine_id = NULL), marks the resource FAILED "Worker went down during resource <ACTION>" and re-casts itself; if no one holds it and the template is unchanged, it simply retries; otherwise it calls handle_preempt and lets the other traversal finish (heat/engine/check_resource.py:60-94, heat/engine/check_resource.py:177-187).
• Cancel (cancel_update / cancel_without_rollback): rollback = a ROLLBACK traversal; without rollback = stop_traversal: new traversal id, stack <ACTION>_FAILED "Stack <ACTION> cancelled", sync points deleted, the same on every in-progress nested stack (heat/engine/worker.py:112-133, heat/engine/worker.py:378-404). In-flight resource operations finish normally.
• Delete during an operation: stop_traversal, then stop_all_workers casts cancel_check_resource to each engine holding a resource lock of the stack (from SELECT DISTINCT engine_id FROM resource WHERE stack_id=? AND engine_id IS NOT NULL, heat/db/api.py:374-380) and waits up to 3 × 5 s for the locks to clear (heat/engine/worker.py:413-442). The receiving engine puts THREAD_CANCEL in the message queue of each running worker; the worker raises CancelOperation at its next progress check (heat/engine/check_resource.py:503-514), the resource ends FAILED "<ACTION> aborted".
• Engine death mid-traversal: nothing restarts the traversal by itself. The resources it held keep engine_id set; the stack stays IN_PROGRESS until the stack timeout is noticed by some later check_resource, or until the user issues an update/delete (the new traversal steals stale locks as above). reset_stack_status at engine start only handles legacy stacks (convergence = False, heat/engine/service.py:2568-2575); heat-manage reset_stack_status <id> forces IN_PROGRESS → FAILED, clears engine_id, hook data and the stack lock (heat/db/api.py:1054-1099).
• Idempotency: a re-delivered check_resource for a node already done finds the resource's current_template_id equal to the traversal template and does nothing but propagate again; the sync-point merge is idempotent (same key, same data). A replayed cast of an older traversal is ignored (step 3 of § 2.3).

2.7 Nested stacks

Resources of type OS::Heat::Stack-like template resources, OS::Heat::ResourceGroup, AutoScalingGroup… are StackResources: they create a child stack through the engine RPC — create_stack/update_stack with a pre-stored template_id and the parent's owner_id, nested_depth + 1, user_creds_id and stack_user_project_id (heat/engine/resources/stack_resource.py:300-336, heat/engine/resources/stack_resource.py:487-533), delete_stack or abandon_stack for deletion (heat/engine/resources/stack_resource.py:570-600) — and poll the child's state with show_stack in check_create_complete (heat/engine/resources/stack_resource.py:407-410). The RPC goes to any engine: the child's traversal is independent of the parent's. The child runs its own traversal; max_nested_stack_depth (5) bounds the depth. stop_traversal and stop_all_workers act on all descendants (stack_get_all_by_root_owner_id).

2.8 Cost

• RPC: one cast per graph node per traversal (plus re-triggers); the payload of each cast carries the resolved attributes of all predecessors.
• SQL per node: load resource, stack, template (twice when the resource uses an older template), lock write, state writes, one event insert per state change, the attribute store, and for each dependant at least two sync-point statements (SELECT, conditional UPDATE) — more under contention.
• A wide graph (hundreds of resources sharing one dependant) turns the dependant's sync point into a hot row: every predecessor retries its CAS with jitter.

3. Legacy engine (stacks with convergence = False)

Stacks created by old releases, or with convergence_engine = False, run in one thread of the engine that took the stack lock (stack_lock row, heat/engine/stack_lock.py:52-107): create/update/delete/suspend/resume/check/snapshot/restore all go through ThreadGroupManager.start_with_lock (heat/engine/service.py:194-259). The lock is released (and the final state persisted) when the thread ends (heat/engine/service.py:233-253); if the release itself fails the engine exits (sys.exit(-1)). A lock held by a dead engine is stolen when the next operation finds it stale. heat-manage migrate_convergence_1 <stack id> converts a COMPLETE legacy stack tree to convergence (heat/engine/service.py:2480-2521, heat/engine/stack.py:2460-2486).

4. Timers and startup tasks

service_manage_report (heat-engine)

• Trigger: ThreadGroup.add_timer(periodic_interval) (60 s), first run at start (heat/engine/service.py:476-477).
• Purpose: register the engine in service and keep updated_at fresh, so that GET /services reports it up (within 2 × report_interval).
• Data: first run INSERT service (host, hostname, binary='heat-engine', engine_id, topic, report_interval), then UPDATE service SET deleted_at = NULL (the updated_at column is bumped by the ORM) (heat/engine/service.py:2523-2548).
• Errors: logged, retried next period. One row per engine process; rows of stopped engines are deleted at stop() (heat/engine/service.py:523-527).

service_manage_cleanup (heat-engine, at start)

heat/engine/service.py:2550-2566: soft-deletes this host's service rows whose updated_at is older than 3 × periodic_interval.

reset_stack_status (heat-engine, at start)

heat/engine/service.py:2568-2613: for each legacy stack IN_PROGRESS (nested included), try the stack lock without retry; if acquired (holder dead), reload and run reset_stack_and_resources_in_progress("Engine went down during stack <ACTION>"): IN_PROGRESS resources → FAILED, stack → <ACTION>_FAILED, previous template restored for updates (heat/engine/stack.py:1702-1718). Stacks locked by a live engine are skipped (ActionInProgress).

5. Signals, wait conditions and software deployments

5.1 Signal sources

The URLs and credentials are generated by SignalResponder (heat/engine/resources/signal_responder.py:100-336): the stack-domain user is created on demand in the stack's project of the heat domain with role [DEFAULT]/heat_stack_user_role (heat/engine/clients/os/keystone/heat_keystoneclient.py:344-395), its password is stored encrypted in resource_data (password, redacted), EC2 keys in resource_data (access_key, secret_key).

5.2 resource_signal processing

heat/engine/service.py:1953-2004:

1. Load the stack with the stored context (use_stored_context=True: the trust of the stack owner, not the caller's token), find the resource.
2. _signal_check_action and _signal_check_hook (heat/engine/resource.py:2457-2486): a signal that only clears a hook ({"unset_hook": "pre-create"}) is handled even during an operation.
3. If the resource has handle_signal, run rsrc.signal(details) in a thread of the engine that received the call; otherwise synchronously. sync_call=True (CFN wait condition) forces synchronous handling and returns the resource metadata.
4. When the signal changes metadata other resources use (wait condition data), every created resource's metadata and node data are refreshed (heat/engine/service.py:1966-1978).
5. Errors: exceptions in the thread are logged; the HTTP caller already got 200. Validation errors (unsupported action, hidden hook) are returned synchronously (400).
6. Concurrency: signals and convergence writes on the same resource are serialised by the atomic_key CAS; make_replacement retries when a signal bumped it (heat/engine/resource.py:407-425).

5.3 Software deployment metadata push

Whenever a deployment is created, updated with a new config, or deleted, _push_metadata_software_deployments (heat/engine/service_software_config.py:109-162):

1. finds the server's resource row by physical id (nova_instance = server_id);
2. sets rsrc_metadata.deployments = the configs of all the server's deployments, conditional on the row's atomic_key (retry on conflict, ConcurrentTransaction);
3. PUTs the JSON metadata to the server's Swift TempURL (metadata_put_url, timeout [DEFAULT]/metadata_put_timeout, 60 s) and/or posts it to its Zaqar queue;
4. bumps atomic_key once more when a TempURL is used. In-instance agents read it with POLL_SERVER_CFN (CFN DescribeStackResource), POLL_SERVER_HEAT (GET …/resources/<server>/metadata), POLL_TEMP_URL or ZAQAR_MESSAGE (heat/engine/resources/server_base.py:186-200; default [DEFAULT]/default_software_config_transport = POLL_SERVER_CFN, heat/common/config.py:259).

5.4 signal_software_deployment

heat/engine/service_software_config.py:302-361: a deployment IN_PROGRESS receives {"deploy_status_code", "deploy_stdout", "deploy_stderr", <outputs>}; a non-zero status code, or an output flagged error_output, makes it FAILED with a reason listing them; otherwise COMPLETE "Outputs received". Signals to a deployment not IN_PROGRESS are ignored.

6. Hooks and lifecycle plugins

• Breakpoints (resource_registry.resources.<name>.hooks: [pre-create, pre-update, pre-delete, post-create, …] in the environment) make the resource wait, storing a resource_data row named after the hook, until a signal {"unset_hook": "<hook>"} clears it (heat/engine/resource.py:533-553, heat/engine/resource.py:2464-2486). reset_stack_status deletes those rows (heat/db/api.py:1074-1084).
• Stack lifecycle plugins (heat.stack_lifecycle_plugins entry point, empty on this branch, setup.cfg) and event sinks (heat.event_sinks: only zaqar-queue) are called on stack actions and events (heat/engine/stack.py:1071, dispatch_event).

7. Errors that never reach the client

They end up as statuses and events (see 02 § asynchronous errors): Resource CREATE failed: <exception> (heat/engine/check_resource.py:188-194), Timed out, Worker went down during resource <ACTION>, <ACTION> aborted (user triggered cancel), Stack <ACTION> cancelled, Engine went down during stack <ACTION>, Stack status manually reset, Resource CHECK failed: name: reason; ..., Snapshot deletion failed: ….

Source: reverse-spec-heat/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.