heat — Scenarios

End-to-end scenarios — heat — branch stable/2026.2

Branch stable/2026.2 @ 5872396e548c (27.0.1) — repository https://opendev.org/openstack/heat — analysed on 2026-10-29

Participants: API (heat-api), E1/E2 (two heat-engine processes), MQ (RabbitMQ), DB (MariaDB), KS (Keystone), SVC (the service a resource plugin calls). Message names are the RPC methods of 03. Real database states recorded during these flows are in [_rewrite/convergence-state_stable-2026.2.json](_rewrite/convergence-state_stable-2026.2.json).

1. Stack create (convergence)

Template: a ← b ← d and a ← c ← d (b and c depend on a, d on b and c).

sequenceDiagram
  autonumber
  actor U as User
  participant API as heat-api
  participant MQ as RabbitMQ
  participant E1 as engine E1
  participant E2 as engine E2
  participant DB as MariaDB
  participant KS as Keystone
  participant SVC as Service
  U->>API: POST /v1/{p}/stacks (token)
  API->>KS: validate token (keystonemiddleware)
  API->>MQ: call create_stack (topic engine)
  MQ->>E1: create_stack
  E1->>E1: parse + validate template, policies resource_types:*
  E1->>DB: INSERT raw_template, stack (current_traversal=T0)
  E1->>KS: POST OS-TRUST/trusts (trustor=user, trustee=heat)
  E1->>DB: INSERT user_creds (Fernet trust_id)
  E1->>KS: POST projects (heat domain, "<p>-<stack id>")
  E1->>DB: UPDATE stack SET current_traversal=T1, CREATE_IN_PROGRESS WHERE current_traversal=T0
  E1-->>MQ: reply {stack_identity}
  MQ-->>API: reply
  API-->>U: 201 Location
  Note over E1: thread _converge_create_or_update
  E1->>DB: INSERT resource a,b,c,d (INIT COMPLETE)
  E1->>DB: UPDATE stack SET current_deps=edges
  E1->>DB: INSERT sync_point x 4 nodes + 1 stack
  E1->>MQ: cast check_resource(a, T1, is_update=true) (engine_worker)
  MQ->>E2: check_resource(a)
  E2->>DB: UPDATE resource a SET engine_id=E2, CREATE IN_PROGRESS WHERE engine_id IS NULL AND atomic_key=n
  E2->>SVC: create (user token from the RPC context)
  E2->>DB: UPDATE resource a COMPLETE, engine_id=NULL WHERE engine_id=E2
  E2->>DB: sync_point b: merge {(a,True): node data} (CAS atomic_key)
  E2->>MQ: cast check_resource(b, data)
  E2->>DB: sync_point c: merge
  E2->>MQ: cast check_resource(c, data)
  par b and c in parallel on any engines
    MQ->>E1: check_resource(b)
    E1->>DB: sync_point d: merge {(b,True)} — still waiting c
  and
    MQ->>E2: check_resource(c)
    E2->>DB: sync_point d: merge {(c,True)} — ready
    E2->>MQ: cast check_resource(d)
  end
  MQ->>E1: check_resource(d)
  E1->>DB: d is a root: sync_point stack: merge — ready
  E1->>DB: UPDATE stack CREATE_COMPLETE WHERE current_traversal=T1
  E1->>DB: purge_db: DELETE sync_point WHERE traversal=T1
  E1->>MQ: notification orchestration.stack.create.end

Diagnostics: stack stuck CREATE_IN_PROGRESS with no resource IN_PROGRESS → a check_resource cast was lost or the dependant's sync point never became ready: compare sync_point.input_data keys with the predecessors in stack.current_deps. A resource IN_PROGRESS with engine_id of a dead engine → nothing will resume it until the stack times out or a new update steals the lock (05 § 2.6). 403 at step 5 → resource_types:<type> policy. 500 at step 7 → Keystone trust creation (check [trustee]).

2. Update while an update is in progress

sequenceDiagram
  autonumber
  actor U as User
  participant API as heat-api
  participant E1 as engine E1
  participant E2 as engine E2
  participant DB as MariaDB
  U->>API: PUT stack (update 1)
  API->>E1: call update_stack
  E1->>DB: UPDATE stack SET current_traversal=T2 WHERE current_traversal=T1
  E1->>E2: cast check_resource(a, T2)
  E2->>DB: lock a (engine_id=E2), update in progress
  U->>API: PUT stack (update 2)
  API->>E1: call update_stack
  E1->>DB: UPDATE stack SET current_traversal=T3 WHERE current_traversal=T2
  E1->>DB: DELETE sync_point WHERE traversal=T2
  E1->>E2: cast check_resource(a, T3)
  E2->>DB: lock a fails (engine_id=E2 already) -> UpdateInProgress
  E2->>E2: holder alive (listening) -> handle_preempt, give up
  E2->>DB: update 1 of a COMPLETE, release lock
  E2->>DB: sync_point (b, T2) missing -> EntityNotFound
  E2->>DB: reload stack: traversal is T3
  E2->>E2: retrigger_check_resource(a) in T3
  E2->>DB: lock a, update to template of T3
  Note over E2,DB: traversal T3 continues; T2 never completes

Diagnostics: Failed to store stack … with traversal ID … warnings are expected when two requests race; the final state belongs to the last accepted traversal.

3. Failure with rollback

sequenceDiagram
  autonumber
  participant W as engine (check_resource bad)
  participant SVC as Service
  participant DB as MariaDB
  participant MQ as RabbitMQ
  W->>SVC: create
  SVC-->>W: error
  W->>DB: resource bad CREATE_FAILED (lock released)
  W->>DB: UPDATE stack CREATE_FAILED WHERE traversal=T1
  alt disable_rollback = false
    W->>DB: UPDATE stack ROLLBACK_IN_PROGRESS, traversal=T2 (empty template)
    W->>MQ: cast check_resource(cleanup nodes) for every created resource
    Note over W: cleanup nodes delete resources not in the empty template
    W->>DB: ROLLBACK_COMPLETE, purge_db
  else disable_rollback = true
    W->>DB: purge_db, current_traversal = ''
  end

Diagnostics: the cause is in the CREATE_FAILED event of the failing resource, not in the final ROLLBACK_COMPLETE reason.

4. Delete during a create

sequenceDiagram
  autonumber
  actor U as User
  participant API as heat-api
  participant E1 as engine E1
  participant E2 as engine E2 (holds locks)
  participant DB as MariaDB
  U->>API: DELETE stack
  API->>E1: call delete_stack
  E1->>DB: stop_traversal: traversal=T2, CREATE_FAILED "Stack CREATE cancelled"
  E1->>DB: DELETE sync_point WHERE traversal=T1
  E1-->>API: reply
  API-->>U: 204
  Note over E1: thread converge_stack(empty, DELETE)
  E1->>DB: SELECT DISTINCT engine_id FROM resource WHERE stack_id=? AND engine_id IS NOT NULL
  E1->>E2: cast cancel_check_resource(stack_id) (server=E2)
  E2->>E2: THREAD_CANCEL -> CancelOperation in running workers
  E2->>DB: resources FAILED "CREATE aborted", locks released
  E1->>DB: poll locks (3 x 5 s)
  E1->>DB: DELETE traversal T3: cleanup node per resource
  E1->>DB: DELETE_COMPLETE; delete trust, domain project, soft-delete stack

Diagnostics: Failed to stop all workers of stack …, stack cancel not complete → an engine did not release its locks within 15 s; the delete traversal still runs and steals the locks of dead engines.

5. Wait condition signalled from an instance (CFN signal)

sequenceDiagram
  autonumber
  participant E as heat-engine
  participant KS as Keystone
  participant VM as Instance (cfn-signal / curl)
  participant CFN as heat-api-cfn
  participant MQ as RabbitMQ
  E->>KS: create stack-domain user (heat_stack_user role) + ec2 credential
  E->>E: build pre-signed URL /v1/waitcondition/<arn>?AWSAccessKeyId&Signature&Timestamp
  E->>VM: URL in user_data / metadata (via Nova)
  VM->>CFN: PUT /v1/waitcondition/<arn> {"Status":"SUCCESS","UniqueId":"1","Data":"…"}
  CFN->>KS: POST /v3/ec2tokens (signature check)
  KS-->>CFN: token of the stack-domain user
  CFN->>MQ: call resource_signal(sync_call=True)
  MQ->>E: resource_signal
  E->>E: load stack with stored context (trust), WaitConditionHandle.handle_signal
  E-->>CFN: metadata
  CFN-->>VM: 200 {"resource", "metadata"}
  Note over E: the WaitCondition resource's check_create_complete sees the count reached

Diagnostics: SignatureDoesNotMatch / InvalidClientTokenId (403) → URL modified (case, host) or EC2 credential deleted; heat_waitcondition_server_url must be reachable from instances.

6. Software deployment (POLL_SERVER_HEAT, HEAT_SIGNAL)

sequenceDiagram
  autonumber
  participant E as heat-engine
  participant DB as MariaDB
  participant VM as os-collect-config + heat-config
  participant API as heat-api
  E->>DB: INSERT software_config, software_deployment (IN_PROGRESS)
  E->>DB: UPDATE resource(server).rsrc_metadata.deployments WHERE atomic_key=n
  loop every poll
    VM->>API: GET …/resources/<server>/metadata (stack-domain user token)
    API->>E: call describe_stack_resource
    E-->>VM: {"metadata": {"deployments": [...]}}
  end
  VM->>VM: run config (script, puppet, ansible hook)
  VM->>API: POST …/resources/<deployment>/signal {"deploy_status_code":0,"deploy_stdout":…}
  API->>E: call resource_signal
  E->>DB: software_deployment COMPLETE, output_values
  Note over E: the SoftwareDeployment resource's check_create_complete reads the status

Diagnostics: deployment stays IN_PROGRESS → the agent cannot reach heat-api (or Swift/Zaqar for other transports), or its stack-domain user lacks heat_stack_user; a non-zero deploy_status_code makes the resource CREATE_FAILED with the agent's stderr in deploy_stderr.

Source: reverse-spec-heat/04-scenarios_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.