masakari — Summary

Reverse-specification of masakari — branch stable/2026.2

Branch stable/2026.2 @ ae41528e1ebc — repository https://opendev.org/openstack/masakari — analysed on 2026-10-28

• Repository: https://opendev.org/openstack/masakari (local clone) — specified branch: stable/2026.2 @ ae41528e1ebc (22.0.0-2-gae41528, commit of 2026-09-09) — analysed on: 2026-10-28
• Scope: the whole project (API, engine, taskflow driver, database, CLI), plus the rewrite pack (step 8). The client expectations include masakari-monitors stable/2026.2 @ b792a7ae040a, openstacksdk instance_ha @ fbdc40381 and python-masakariclient @ 073cd2c. The change history (step 7) was not requested.

Overview

Masakari provides instance high availability for Nova. Detectors (masakari-monitors, or any client) report failures as notifications of three types: a compute host down, an instance crashed, or a process (nova-compute) died. masakari-api records them and casts them to masakari-engine. The engine runs recovery workflows (taskflow) that drive Nova with an admin service account:

• host: disable the compute service, then evacuate the host's instances, either where Nova decides or to reserved spare hosts of the same failover segment;
• instance: stop and start the instance;
• process: disable the compute service.

Segments and hosts are configuration managed through the same API. The engine keeps per-instance evacuation records (vmoves) and per-task progress, and two periodic tasks retry failed notifications and expire old ones. Dependencies are MariaDB (two databases: Masakari and taskflow persistence), RabbitMQ, Keystone, Nova and optionally a tooz backend.

Architecture

flowchart LR
  mon[masakari-monitors / detector] -->|HTTPS POST /v1/notifications| api[masakari-api]
  admin[Operator / openstack CLI] -->|HTTPS /v1/segments, hosts| api
  api -->|SQL| db[(MariaDB masakari)]
  api -->|AMQP cast process_notification, call workflow details| mq{{RabbitMQ topic ha_engine}}
  mq --> eng[masakari-engine]
  eng -->|SQL| db
  eng -->|SQL taskflow logbooks| tf[(MariaDB taskflow)]
  api -->|REST os-services check| nova[nova-api 2.53]
  eng -->|REST services, servers, evacuate, lock, stop/start, aggregates| nova
  api -->|token| ks[Keystone]
  eng -->|token per Nova call| ks
  api -.->|lock create_host_notification| tooz[(tooz backend, optional)]
  api -.->|versioned notifications| mq
  eng -.->|versioned notifications| mq

Processes

External dependencies

Key figures

• 19 REST operations: 17 on resources under /v1, plus GET / and GET /v1. The 21 routes of _rewrite/routes_stable-2026.2.json add the /v1/ and /v1/versions/{id} aliases. The api-ref documents 17 operations. Microversions 1.0–1.3. All policies are admin-only (27 rules).
• 2 RPC methods received by the engine (process_notification cast 1.0, get_notification_recovery_workflow_details call 1.1), 2 periodic tasks, 5 polling loops on Nova inside the workflows.
• 8 taskflow tasks (plus no_op) in 3 flows, recovery methods auto, reserved_host, auto_priority, rh_priority.
• 55 exception classes, 4 tables (+3 taskflow tables), 3 alembic revisions, 11 Nova operations.
• 382 references checked by check_refs.py (0 problems) ([_verification_stable-2026.2.md](_verification_stable-2026.2.md)); 23 scenarios, 201 HTTP exchanges, 143 Nova requests recorded.

Files

Points of attention

1. Fallback and retry of host recovery cannot succeed once a vmove has failed (observed): the failed rows of earlier attempts are counted (masakari/engine/drivers/taskflow/host_failure.py:424). auto_priority/rh_priority therefore report error even when the second method evacuated the instance.
2. Non-HA instances are evacuated first with the default evacuate_all_instances = true, against the documentation (vmoves read newest first, masakari/engine/drivers/taskflow/host_failure.py:350; observed).
3. No coordination between engines: the periodic tasks run in every engine, and the per-host lock is process-local (masakari/engine/manager.py:274). With several engines, error notifications are retried in parallel.
4. **A host stays on_maintenance forever** after any processed host or process failure, including a process that Masakari does not handle (masakari/engine/manager.py:90). Its next failure is refused with 409, and masakari-monitors drops 409s silently.
5. Duplicate detection matches any status, uses the sender's clock, and is not atomic. Without tooz, every POST /notifications sleeps 0–5 s (masakari/ha/api.py:359).
6. API errors that turn into a misleading or unexpected status:
   • GET …/vmoves/{id} → 500 for an unknown notification;
   • a Nova 5xx during host create → 500;
   • a Nova 403 is passed through as 403;
   • the microversion 1.3 of vmoves is not enforced;
   • pagination markers are integer ids, not UUIDs.
7. Nova client: a new Keystone token per call, no timeout, one page of servers (≤ 1000) per host, and an empty service list in the engine → IndexError → generic error.
8. The VM flow restarts instances that were stopped on purpose on an IO error. PROCESS STOPPED for a non-nova-compute process still sets maintenance.
9. Orphan data: vmoves are never purged, and the taskflow database is never purged.
10. For OpenKeel ("Masakari with node health checks"): masakari-monitors already has a Kubernetes host check (masakari-monitors:masakarimonitors/hostmonitor/kubernetes_check/manager.py:36). It sends COMPUTE_HOST STOPPED host_status=NORMAL keyed by node name. A Medik8s-based detector must send the same contract, and the Kubernetes node name must equal the Masakari host name and the Nova (KubeVirt) compute host. host_status=NORMAL is the assertion that the node is fenced: Masakari never fences, and only refuses UNKNOWN. Self-node-remediation must therefore have completed before the notification is sent [inferred].

Blind spots / [inferred]

• No real Nova, RabbitMQ, MySQL or Keystone was used. Nova timing (service-down detection before evacuation) and multi-engine races are inferred from the code. The scenarios use fakes, with sleeps set to 0 and timeouts to 3 s.
• The functional tests (masakari/tests/functional, openstacksdk against devstack) were not run.
• keystonemiddleware was left out of the scenario pipeline: token validation errors are described from the paste pipeline, not observed.
• Third-party taskflow tasks plugged through masakari.task_flow.tasks are not covered.
• The masakari-dashboard (Horizon plugin) was not read.

Source: reverse-spec-masakari/00-summary_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.