masakari — Scenarios

End-to-end scenarios — masakari — branch stable/2026.2

Branch stable/2026.2 @ ae41528e1ebc — repository https://opendev.org/openstack/masakari — analysed on 2026-10-28

Each scenario was also run in-process (scenario names in brackets, [_rewrite/exchanges_stable-2026.2.json](_rewrite/exchanges_stable-2026.2.json), Nova frames in [_rewrite/nova-calls_stable-2026.2.json](_rewrite/nova-calls_stable-2026.2.json)).

1. Compute host failure, `auto` method [host-failure-auto]

A monitor (masakari-monitors hostmonitor, or a Kubernetes node check) sees a node down and reports it. Nova chooses where each instance goes.

sequenceDiagram
  autonumber
  participant MON as masakari-monitors
  participant API as masakari-api
  participant DB as MariaDB
  participant MQ as RabbitMQ
  participant ENG as masakari-engine
  participant KS as Keystone
  participant NOVA as nova-api
  MON->>API: POST /v1/notifications {type COMPUTE_HOST, hostname, payload {event STOPPED, host_status NORMAL}}
  API->>DB: SELECT hosts WHERE name (on_maintenance?)
  alt host unknown
    API-->>MON: 400 badRequest (monitor stops)
  else host on maintenance / duplicate
    API-->>MON: 409 conflictingRequest (monitor stops)
  end
  API->>DB: SELECT notifications (duplicate window 180 s)
  API->>DB: INSERT notifications (status new)
  API->>MQ: cast process_notification (topic ha_engine, v1.0)
  API-->>MON: 202 {notification, status new}
  MQ->>ENG: process_notification
  ENG->>DB: UPDATE notifications SET status=running
  ENG->>DB: UPDATE hosts SET on_maintenance=1
  ENG->>KS: POST /v3/auth/tokens (privileged user, on every Nova call)
  ENG->>NOVA: GET /os-services?binary=nova-compute&host=H
  ENG->>NOVA: PUT /os-services/{id} {status disabled, disabled_reason}
  ENG->>ENG: sleep wait_period_after_service_update (180 s)
  ENG->>NOVA: GET /servers/detail?all_tenants=true&host=H
  ENG->>DB: INSERT vmoves (pending) x N
  loop each instance, 3 green threads
    ENG->>NOVA: GET /servers/{id}; POST action lock
    opt vm_state not active/stopped/error, or task_state set
      ENG->>NOVA: POST action os-resetState {state error}
    end
    ENG->>DB: UPDATE vmoves SET status=ongoing
    ENG->>NOVA: POST action evacuate {}
    loop every verify_interval until another host, at most wait_period_after_evacuation
      ENG->>NOVA: GET /servers/{id}
    end
    opt was stopped or error
      ENG->>NOVA: POST action os-stop, then poll; os-resetState error if it was error
    end
    ENG->>DB: UPDATE vmoves SET status=succeeded|failed, dest_host, end_time
    ENG->>NOVA: POST action unlock
  end
  alt every vmove of the notification succeeded
    ENG->>DB: UPDATE notifications SET status=finished
  else one failed or still pending
    ENG->>DB: UPDATE notifications SET status=error
    Note over ENG: retried by _process_unfinished_notifications (120 s), then failed
  end

Diagnostics:

The 202 only means "recorded". Follow GET /v1/notifications/{id} (1.1 for the task log) and GET …/vmoves.
A notification stuck in new means the cast was lost or no engine consumes ha_engine: it is retried after 60 s.
One stuck in running for a long time: an engine died mid-flow, or a non-Masakari exception escaped; it expires after 24 h.
Instances left locked in Nova: look for "Failed to unlock instance".
The host stays on_maintenance=true after the recovery: reset it once the node is repaired, otherwise its next failure is refused with 409.

2. Compute host failure, `reserved_host` method [host-failure-reserved-host]

The same as scenario 1, with these differences:

The engine lists the segment's hosts with reserved=true, on_maintenance=false. With none, the notification goes to error right away (ReservedHostsUnavailable) [host-failure-no-reserved-host-then-periodic].
For the first reserved host, under a process-local lock on its name:
1. If add_reserved_host_to_aggregate: GET /os-aggregates, then POST /os-aggregates/{id}/action {"add_host": {"host": R}} for each aggregate holding the failed host.
2. PUT /os-services/{id} {"status": "enabled"} on R.
3. UPDATE hosts SET reserved=0 for R.
4. evacuate {"host": R} for each instance.
If the main tasks fail, taskflow's ParameterizedForEach retries them with the next reserved host. Because of the vmove accounting defect (05), a retry cannot succeed once one vmove has failed.

sequenceDiagram
  autonumber
  participant ENG as masakari-engine
  participant DB as MariaDB
  participant NOVA as nova-api
  ENG->>DB: SELECT hosts WHERE segment=S AND reserved=1 AND on_maintenance=0
  alt none
    ENG->>DB: UPDATE notifications SET status=error (ReservedHostsUnavailable)
  else R found
    ENG->>NOVA: disable compute on failed host H, sleep 180 s, list servers, INSERT vmoves
    opt add_reserved_host_to_aggregate
      ENG->>NOVA: GET /os-aggregates; POST /os-aggregates/{id}/action add_host R
    end
    ENG->>NOVA: PUT /os-services/{R} {status enabled}
    ENG->>DB: UPDATE hosts SET reserved=0 WHERE name=R
    ENG->>NOVA: POST /servers/{id}/action {evacuate: {host: R}} (x N)
    ENG->>DB: UPDATE notifications SET status=finished|error
  end

3. `auto_priority` fallback [host-failure-auto-priority-fallback]

The auto evacuation times out (Nova never moves the instance), so the engine logs "Failed to evacuate all instances … retrying using 'reserved_host' workflow." and runs the RH flow from the start: it disables the service and sleeps again, lists the servers again, and inserts a second vmove for the same instance. The evacuation to the reserved host succeeds, but the notification ends **error** because the first vmove is failed. The periodic retry then replays the flow; in the run it found no instance left on the host and set finished.

Diagnostics: with *_priority methods, read the vmoves: a succeeded vmove next to a failed one for the same instance means the recovery actually worked.

4. Instance (VM) failure [instance-failure]

sequenceDiagram
  autonumber
  participant MON as instancemonitor (libvirt events)
  participant API as masakari-api
  participant ENG as masakari-engine
  participant NOVA as nova-api
  MON->>API: POST /v1/notifications {type VM, payload {event LIFECYCLE, vir_domain_event STOPPED_FAILED, instance_uuid}}
  API-->>MON: 202
  API->>ENG: cast process_notification
  alt event pair not handled
    ENG->>ENG: status ignored
  else
    ENG->>NOVA: GET /servers/{id}
    alt not HA_Enabled and not process_all_instances
      ENG->>ENG: status finished (skipped)
    else paused or rescued
      ENG->>ENG: status ignored
    else
      ENG->>NOVA: POST action os-stop (if not stopped); poll until stopped (180 s)
      ENG->>NOVA: POST action os-start; poll until active (60 s)
      ENG->>ENG: status finished (error on timeout)
    end
  end

Diagnostics: an IO_ERROR/IO_ERROR_REPORT on an instance that the user had stopped restarts it. The flow does not lock the instance.

5. Process failure (`nova-compute`) [process-failure]

sequenceDiagram
  autonumber
  participant MON as processmonitor
  participant API as masakari-api
  participant ENG as masakari-engine
  participant DB as MariaDB
  participant NOVA as nova-api
  MON->>API: POST /v1/notifications {type PROCESS, payload {event STOPPED, process_name nova-compute}}
  API-->>MON: 202
  API->>ENG: cast process_notification
  ENG->>DB: UPDATE hosts SET on_maintenance=1
  alt process_name == nova-compute
    ENG->>NOVA: GET /os-services (disabled already?)
    ENG->>NOVA: PUT /os-services/{id} {status disabled, disabled_reason "Masakari detected process failed."}
    loop until disabled, at most 180 s
      ENG->>NOVA: GET /os-services
    end
  else any other process
    ENG->>ENG: skip (finished), host stays on maintenance
  end

Diagnostics: after any PROCESS STOPPED, even for a process that Masakari does not handle, the host refuses new notifications (409) until it is taken out of maintenance. A real host failure of that node is then lost; masakari-monitors drops 409s silently.

6. Lost cast and expiry [periodic-retry-lost-new-notification, periodic-expired-notification]

sequenceDiagram
  autonumber
  participant ENG as masakari-engine (every engine)
  participant DB as MariaDB
  loop every 120 s
    ENG->>DB: SELECT notifications WHERE status IN (error, new)
    ENG->>ENG: run the workflow again for error, and for new older than 60 s
    ENG->>DB: UPDATE status=failed if still error
  end
  loop every 600 s
    ENG->>DB: SELECT notifications WHERE status IN (running, error, new)
    ENG->>DB: UPDATE status=failed WHERE generated_time older than 24 h
  end

Diagnostics: with several engines, the same retry can run in each of them at the same time. A notification with a generated_time far in the past (sender clock skew) expires at the next check without being processed.

Source: reverse-spec-masakari/04-scenarios_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.

masakari — Scenarios

1. Compute host failure, auto method [host-failure-auto]

2. Compute host failure, reserved_host method [host-failure-reserved-host]

3. auto_priority fallback [host-failure-auto-priority-fallback]