masakari — REST API

REST API — masakari — branch stable/2026.2

Branch stable/2026.2 @ ae41528e1ebc — repository https://opendev.org/openstack/masakari — analysed on 2026-10-28

Masakari exposes one API, /v1, on port 15868 by default (masakari/conf/service.py:56), as the instance-ha service type. Routes are built at run time by the Routes library from the stevedore extensions masakari.api.v1.extensions (setup.cfg, masakari/api/openstack/__init__.py:168). The behaviour below was checked by reading the code and by the 23 in-process scenarios of _rewrite/tools/adapter_masakari.py (201 recorded exchanges, [_rewrite/exchanges_stable-2026.2.json](_rewrite/exchanges_stable-2026.2.json)); "observed" marks a fact seen in those exchanges.

Cross-cutting rules

• Pipeline (etc/masakari/api-paste.ini): cors → http_proxy_to_wsgi → request_id → faultwrap → sizelimit → authtoken → keystonecontext → APIRouterV1. With [DEFAULT]/auth_strategy = noauth2 the authtoken and keystonecontext filters are replaced by NoAuthMiddleware, which makes every request admin (masakari/api/auth.py:124). GET / uses a separate pipeline without authentication (faultwrap http_proxy_to_wsgi apiversionsapp).
• Authentication: MasakariKeystoneContext builds the request context from the keystonemiddleware headers and answers a plain-text 401 when there is no user id (masakari/api/auth.py:90). Observed: GET /v1 without identity headers → 401.
• Authorisation: every operation calls context.can('os_masakari_api:<resource>:<action>'). All rules default to rule:admin_api = is_admin:True (masakari/policies/base.py, [_rewrite/policies_stable-2026.2.json](_rewrite/policies_stable-2026.2.json)), and is_admin is computed from context_is_admin = role:admin (masakari/context.py:113). There is no project scoping: the API is admin-only, and the rules ignore system scope and reader/member. Observed: a member gets 403 forbidden "Policy doesn't allow os_masakari_api:segments:index to be performed.".
• Microversions: header OpenStack-API-Version: instance-ha X.Y (masakari/api/openstack/wsgi.py:65), range 1.0–1.3 (masakari/api/api_version_request.py:51), default 1.0, latest = 1.3 (masakari/api/openstack/wsgi.py:186). Every response echoes OpenStack-API-Version and Vary: OpenStack-API-Version. Out of range → 406 masakariFault "Version 1.4 is not supported by the API. Minimum is 1.0 and maximum is 1.3."; malformed → 400 (observed). | Version | Change | Where it is enforced | |---|---|---| | 1.0 | initial API | — | | 1.1 | GET /notifications/{id} adds recovery_workflow_details | masakari/api/openstack/ha/notifications.py:128 | | 1.2 | enabled on segments (create, update) | masakari/api/openstack/ha/segments.py:80, masakari/api/openstack/ha/segments.py:96 | | 1.3 | VM moves (/notifications/{id}/vmoves) | not enforced: masakari/api/openstack/ha/vmoves.py:38 has no version check; observed 200 at 1.0 |
• Error body: {"<faultName>": {"code": <status>, "message": "<text>"}} with badRequest, forbidden, itemNotFound, badMethod, conflictingRequest, masakariFault… (masakari/api/openstack/wsgi.py:1003). See errors.
• Unexpected exceptions: @extensions.expected_errors turns any exception that is not a webob error of a declared status, a Forbidden or a ValidationError into 500 "Unexpected API Error…" (masakari/api/openstack/extensions.py:380).
• Routes declared but not implemented: Routes' resource() maps the full CRUD set plus new, edit and action. The controllers implement only some of these, and the others answer **405 badMethod** "The request method: 'PUT' with action: 'update' is not allowed on this resource" (masakari/api/openstack/wsgi.py:707). The member action POST /segments/{id}/action and POST /notifications/{id}/action is declared (masakari/api/openstack/ha/segments.py:135, masakari/api/openstack/ha/notifications.py:147), but no action is registered, so it also answers 405 (observed).
• Path aliases: every resource is also mapped under /v1/{project_id}/… (project id regex [0-9a-f\-]+ or [osapi_v1]/project_id_regex, masakari/api/openstack/__init__.py:100), and accepts a .json/.xml suffix. The project id in the path is ignored by the handlers.
• Pagination and sorting (segments, hosts, notifications, vmoves): limit (default and maximum [DEFAULT]/osapi_max_limit = 1000, masakari/conf/api.py:69; negative or non-integer → 400) and marker = the **integer id** of the last item, not its UUID (masakari/db/sqlalchemy/api.py:200; observed: a UUID marker → 400 "Marker <uuid> could not be found."). sort_key/sort_dir may repeat. The default is created_at desc, id desc (masakari/db/sqlalchemy/api.py:104), and an unknown key or direction → 400. No links/next is returned in the bodies.
• Booleans in bodies accept true/false and the strings True TRUE true 1 ON On on YES Yes yes and their negatives (masakari/api/validation/parameter_types.py:151). In query strings, oslo_utils.strutils.bool_from_string(strict=True) accepts a different set (t, y, f, n…), observed in GET …/hosts?on_maintenance=perhaps → 400.
• Timestamps have their microseconds stripped by the models (masakari/db/sqlalchemy/models.py:34), except vmoves.start_time/end_time, which are set by the engine and keep them (observed).

Summary

(/v1/versions/{id} and /v1/ are aliases of GET /v1; the policy names are prefixed with os_masakari_api:.)

Failover segments

Functional description

• Purpose: a segment groups the compute hosts that back each other up and fixes the recovery method used when one of them fails: auto (Nova chooses the target host), reserved_host (evacuate to a spare host of the segment), auto_priority (auto, then reserved hosts on failure) or rh_priority (reserved hosts, then auto).
• Actor: cloud administrator.
• Business rules:
  • name is unique among non-deleted segments (uniq_segment0name0deleted, masakari/db/sqlalchemy/models.py:79; 409 "Failover segment with name seg-a already exists." observed). service_type is free text (≤255) and has no effect on the processing. description is optional.
  • enabled (from 1.2, default true, masakari/ha/api.py:97) switches recovery off. Notifications from hosts of a disabled segment are still accepted (202) but the engine sets them to ignored with message "Recovery aboart: the segment HA is disabled." (masakari/engine/manager.py:339; spelling as in the code).
  • A segment cannot be updated or deleted while one of its hosts has a notification in new, running or error (masakari/ha/api.py:41, masakari/ha/api.py:114, masakari/ha/api.py:135) → 409 "Failover segment <uuid> can't be updated as it is in-use to process notifications." This includes notifications still queued for the engine (observed in segment-host-in-use).
  • Deleting a segment soft-deletes its hosts in the same transaction (masakari/db/sqlalchemy/api.py:298).
  • The update schema requires at least one known field (anyOf, masakari/api/openstack/ha/schemas/segments.py:53); before 1.2 enabled is rejected as an additional property (observed 400).
• Flow: synchronous, with a database write and versioned notifications segment.create.start/end (masakari/objects/segment.py:80). On error, segment.create.error with the fault (masakari/ha/api.py:105).
• Limitations: GET accepts the filters recovery_method, service_type and enabled (compared as raw strings against a boolean column: enabled=1 and enabled=true behave differently depending on the database dialect [inferred]). An unknown or non-UUID id → 404 (masakari/ha/api.py:55).

Technical sheet

• Request (POST) {"segment": {"name", "recovery_method", "service_type", "description"?, "enabled"? (1.2)}}; additionalProperties: false at both levels (masakari/api/openstack/ha/schemas/segments.py:21).
• Response {"segment": {id, uuid, name, service_type, enabled, description, recovery_method, created_at, updated_at, deleted_at, deleted}}. enabled is returned at every microversion (observed at 1.0). The internal integer id and the deleted* fields are exposed.

Hosts

Functional description

• Purpose: register a compute node in a segment, flag it as a reserved (spare) host, or put it on maintenance (notifications from it are then refused).
• Preconditions: the segment exists. **Nova must know a nova-compute service whose host equals the name**, case-sensitive (masakari/ha/api.py:155, masakari/compute/nova.py:256), otherwise 400 "Compute service with name ghost could not be found." This is checked again on rename.
• Business rules:
  • The host name is unique across all segments (uniq_host0name0deleted, masakari/db/sqlalchemy/models.py:100): a compute node belongs to one segment only (409 observed in another segment).
  • type (^[a-zA-Z0-9]*$) and control_attributes (free text) are required on create but are not used by any recovery workflow.
  • reserved and on_maintenance default to false (masakari/ha/api.py:197).
  • Update and delete are refused with 409 while the segment is under recovery (masakari/ha/api.py:222, masakari/ha/api.py:254).
  • The engine itself writes on_maintenance=true when it processes a host or process failure, and reserved=false when it consumes a reserved host (see 05). **Nothing ever resets on_maintenance**: the operator must do it with PUT.
• Flow: synchronous; Nova GET /os-services?binary=nova-compute&host=<name> with the service account (see 03), database write, versioned notifications host.*.
• Errors from Nova: a Nova 403 is translated to Forbidden and reaches the client as 403 (masakari/compute/nova.py:72, observed); a 5xx or an untranslated novaclient exception → 500 (observed nova_503).
• Filters on GET: name, type, control_attributes, on_maintenance, reserved (masakari/api/openstack/ha/hosts.py:60).

Technical sheet

• Request {"host": {"name", "type", "control_attributes", "reserved"?, "on_maintenance"?}} (masakari/api/openstack/ha/schemas/hosts.py:20).
• Response {"host": {id, uuid, name, failover_segment_id, failover_segment: {<full segment>}, type, reserved, control_attributes, on_maintenance, created_at, updated_at, deleted_at, deleted}} (masakari/api/openstack/ha/views/hosts.py:36).
• A host looked up under the wrong segment → 404 "Host '<uuid>' under failover_segment '<uuid>' could not be found." (masakari/db/sqlalchemy/api.py:408).

Notifications

POST /v1/notifications

Functional description

• Purpose: report a failure to Masakari. This is the entry point used by masakari-monitors (or any detector, such as Medik8s in OpenKeel). Three types exist:
  • COMPUTE_HOST: a whole node is down. The response is to evacuate its instances.
  • VM: an instance crashed. The response is to stop it, start it again and confirm it is active.
  • PROCESS: a process on a node died. The response, for nova-compute only, is to disable the compute service.
• Actor: admin (the monitors use a service account with the admin role).
• Preconditions: the hostname is a registered Masakari host (400 "Host with name ghost could not be found."), and that host is not on maintenance (409 "…is ignored as the host is already under maintenance." masakari/ha/api.py:304).
• Business rules:
  • The payload is validated per type (masakari/api/openstack/ha/notifications.py:65, schemas in masakari/api/openstack/ha/schemas/payload.py):
    • PROCESS: {process_name, event};
    • VM: {instance_uuid, vir_domain_event, event};
    • COMPUTE_HOST: {event, host_status?, cluster_status?}. event is STARTED/STOPPED for PROCESS and COMPUTE_HOST but free text for VM. The outer schema also forces every payload value to be a string of at most 255 characters (masakari/api/validation/parameter_types.py:166).
  • Duplicate detection (409 "Notification received from host compute-1 of type PROCESS is duplicate."): a notification with the same type, the same source host, a generated_time at least as recent as the new generated_time minus [DEFAULT]/duplicate_notification_detection_interval (180 s), and an identical payload (masakari/ha/api.py:277). It is matched in any status, including finished (the option's help text says "new or running"). The window is computed on the sender's clock (generated_time), not on the server's.
  • For COMPUTE_HOST only, creation is serialised per host name by a tooz lock create_host_notification-<hostname> when [coordination]/backend_url is set (masakari/ha/api.py:344). Without coordination, every POST first sleeps a random 0–[DEFAULT]/notification_delay_max seconds (5 s by default, masakari/ha/api.py:359, masakari/conf/api.py:105) to make concurrent duplicates less likely.
• Flow:
  1. (synchronous) Validation, host lookup, maintenance check, duplicate check.
  2. (synchronous) INSERT notifications with status new, then notification.create.start/end versioned notifications.
  3. (synchronous) RPC cast process_notification to the topic ha_engine (masakari/ha/api.py:333), then 202 with the stored notification.
  4. (asynchronous) masakari-engine runs the recovery workflow (see 05 § process_notification).
• Outcome: status: new in the response, then running and finally finished, ignored, error or failed. The client observes it by GET /notifications/{id} (1.1 for task progress).
• Limitations: the 202 does not prove the cast reached the bus: if cast raises, the row stays new and the client gets a 500 (masakari/ha/api.py:334). The engine's periodic task picks such rows up after retry_notification_new_status_interval (observed in periodic-retry-lost-new-notification).

Technical sheet

• Request {"notification": {"type": "COMPUTE_HOST|VM|PROCESS", "hostname": "^[a-zA-Z0-9-._]*$", "generated_time": "<ISO 8601>", "payload": {…}}} (masakari/api/openstack/ha/schemas/notifications.py:18). A naive time (no zone, as sent by openstacksdk) is accepted and taken as UTC (masakari/api/validation/validators.py:92).
• Response 202 {"notification": {id, notification_uuid, generated_time, source_host_uuid, type, payload, status, failover_segment_uuid, message, created_at, updated_at, deleted_at, deleted}}.

GET /v1/notifications and GET /v1/notifications/{id}

• Filters: status, source_host_uuid, type, generated-since (ISO 8601; invalid → 400 "Invalid generated-since value", masakari/api/openstack/ha/notifications.py:103).
• show at ≥ 1.1 makes a synchronous RPC call get_notification_recovery_workflow_details to an engine. The engine reads the taskflow persistence database and returns recovery_workflow_details: a list of {name, progress, progress_details: [{timestamp, progress, message}], state} in execution order (masakari/engine/drivers/taskflow/driver.py:296). If no engine answers, the call times out after rpc_response_timeout and the client gets a 500, because MessagingTimeout is not an expected error [inferred from masakari/api/openstack/extensions.py:380]. When a task never wrote progress, it is left out (observed: ignored VM notifications return []).

VM moves (/v1/notifications/{notification_id}/vmoves)

• Purpose: the per-instance record of a host-failure evacuation. It is written by the engine: one row per instance to evacuate, with status pending → ongoing → succeeded|failed, type evacuation, source_host, dest_host, start_time, end_time and message (masakari/engine/drivers/taskflow/host_failure.py:136).
• Rules: only COMPUTE_HOST notifications have vmoves; any other type → 400 "This notification <uuid> without vm moves." (masakari/ha/api.py:419). Filters are status and type.
• Spec/code mismatches:
  • Documented as 1.3 but served at every version.
  • GET …/vmoves/{id} does not check that the vmove belongs to the notification (masakari/ha/api.py:436).
  • For an unknown notification, GET …/vmoves/{id} returns 500 (NotificationNotFound is not caught, masakari/api/openstack/ha/vmoves.py:77, observed), whereas GET …/vmoves returns 404.

Lifecycle: notification

stateDiagram-v2
  [*] --> new: POST /notifications (API)
  new --> ignored: process_notification, segment disabled (engine)
  new --> running: process_notification (RPC cast, engine)
  new --> running: _process_unfinished_notifications, new for more than retry_notification_new_status_interval (timer)
  running --> finished: workflow done or skipped
  running --> ignored: host_status != NORMAL, host_status missing, unknown event, VM event not handled, VM paused/rescued
  running --> error: workflow failed
  error --> running: _process_unfinished_notifications (timer, retry)
  error --> failed: _process_unfinished_notifications after the retry is still error
  new --> failed: _check_expired_notifications (generated_time older than notifications_expired_interval)
  running --> failed: _check_expired_notifications
  error --> failed: _check_expired_notifications
  finished --> [*]
  failed --> [*]
  ignored --> [*]

Lifecycle: host flags set by the engine

stateDiagram-v2
  [*] --> active: POST /hosts (on_maintenance=false)
  active --> on_maintenance: COMPUTE_HOST STOPPED or PROCESS STOPPED processed (engine)
  on_maintenance --> active: PUT /hosts on_maintenance=false (operator only)
  state "reserved=true" as res
  [*] --> res: POST/PUT reserved=true
  res --> active: picked as evacuation target (engine sets reserved=false)
  res --> on_maintenance: the reserved host itself fails (engine sets reserved=false, on_maintenance=true)

Lifecycle: vmove

stateDiagram-v2
  [*] --> pending: PrepareHAEnabledInstancesTask
  pending --> ongoing: evacuate call sent
  ongoing --> succeeded: instance seen on another host
  ongoing --> failed: Nova error, vm_state error, or timeout (wait_period_after_evacuation)

Source: reverse-spec-masakari/01-rest-api_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.