masakari — Rewrite pack

Rewrite pack — masakari — branch stable/2026.2

Branch stable/2026.2 @ ae41528e1ebc — repository https://opendev.org/openstack/masakari — analysed on 2026-10-28

Clients read for § 4: masakari-monitors stable/2026.2 @ b792a7ae040a (prefix masakari-monitors:), openstacksdk stable/2026.2 @ fbdc40381 (prefix openstacksdk:), python-masakariclient stable/2026.2 @ 073cd2c (prefix python-masakariclient:).

1. Contents of _rewrite/

Extraction: Python 3.12.7, upper-constraints stable/2026.2 (taskflow 6.4.0, oslo.db 18.1.1, python-novaclient 18.13.1). psycopg2 was left out of the test requirements because it does not build without pg_config; it is not needed.

2. HTTP behaviour that the contracts do not show

2.1 Middleware pipeline (outermost first)

cors → http_proxy_to_wsgi → request_id → faultwrap → sizelimit → authtoken → keystonecontext → router (etc/masakari/api-paste.ini). Consequences:

• An invalid token is a keystonemiddleware 401 (plain text), before any routing: an unknown path with a bad token is 401, not 404.
• MasakariKeystoneContext answers a 401 without a JSON body when no user id is present (masakari/api/auth.py:90).
• Microversion parsing happens in the router's Resource (masakari/api/openstack/wsgi.py:178), so a bad version header on an unknown path is 404.
• GET / is served without authentication. GET /v1 requires it.

2.2 Error body

{"<faultName>": {"code": <int>, "message": "<text>"}} with the fault names of masakari/api/openstack/wsgi.py:1006, Content-Type: application/json; charset=UTF-8, plus OpenStack-API-Version and Vary when a version was parsed. 401/404 from the middlewares are webob plain-text pages. Messages are gettext-translated per Accept-Language, so clients must not parse them; masakari-monitors only looks at the status (§ 4).

2.3 Other cross-cutting rules

• 405 badMethod for every route that Routes' resource() generates but the controller does not implement (PUT/DELETE on notifications, POST/PUT/DELETE on vmoves and extensions, new, edit, action). 404 for paths outside the mapper.
• /v1/{project_id}/… aliases and the .json/.xml suffix are accepted everywhere, and the project id is ignored.
• The id (integer primary key) is exposed in every body and is the pagination marker.
• deleted, deleted_at, created_at and updated_at are in every body. Timestamps are YYYY-MM-DDTHH:MM:SS.000000 (microseconds zeroed by the models), except vmove start_time/end_time.
• Responses never carry links/next.
• Booleans in bodies accept 22 spellings (§ 01). The response always carries JSON booleans.
• enabled is returned on segments at every microversion, but only accepted in requests from 1.2.
• vmoves are served at every microversion although documented as 1.3.

3. The recovery engine: formal specification

3.1 Input model and validation

• Notification types and payloads: masakari/api/openstack/ha/schemas/payload.py:18. All payload values are strings ≤ 255 (masakari/api/validation/parameter_types.py:166).
• The handled VM event pairs: masakari/engine/instance_events.py:22.
• Host flags: reserved, on_maintenance; segment: recovery_method ∈ {auto, reserved_host, auto_priority, rh_priority}, enabled.

3.2 Configuration that changes the algorithm

3.3 Algorithm (pseudo-code)

on POST /notifications(n):                                  # masakari/ha/api.py:348
  if not coordination: sleep(uniform(0, notification_delay_max))
  with lock("create_host_notification-"+n.hostname) if n.type == COMPUTE_HOST and coordination:
    h = host_by_name(n.hostname) or 400                     # masakari/ha/api.py:301
    if h.on_maintenance: 409
    if exists m: m.type==n.type and m.source_host==h and m.generated_time >= n.generated_time - window
                 and m.payload == n.payload (any status): 409   # masakari/ha/api.py:277
    insert n(status=new, failover_segment_uuid=h.segment); cast process_notification(n); 202

process_notification(n):                                    # masakari/engine/manager.py:335
  if not segment(n).enabled: n.status=ignored; return
  with local_lock(n.source_host_uuid):                     # masakari/engine/manager.py:274
    if n.status==new and db(n).status!=new: return
    n.status=running
    n.status = {PROCESS: process(n), VM: instance(n), COMPUTE_HOST: host(n)}[n.type]

host(n):                                                    # masakari/engine/manager.py:174
  if n.payload.host_status is None or upper != NORMAL: return ignored
  if event == STARTED: return finished
  h.on_maintenance = true; if h.reserved: h.reserved = false
  R = [x.name for x in hosts(segment, reserved, not on_maintenance) order by created_at desc, id desc] if method != auto
  try: run(method)  ->  finished ; except Skip: finished ; except Failure: error
     auto            = flow(pre, main, post of host_auto_failure_recovery_tasks)
     reserved_host   = R empty ? ReservedHostsUnavailable : flow_rh(R)    # main retried per R[i]
     auto_priority   = try auto except (not Skip) -> reserved_host
     rh_priority     = try reserved_host except (not Skip) -> auto

DisableComputeService(H): svc=GET os-services(host=H, binary=nova-compute)[0]; PUT svc {disabled, reason}; sleep(180)
PrepareHAEnabledInstances(H): S = GET servers/detail?all_tenants&host=H (1 page)
     ha = [s | bool(s.metadata[key])] minus error ones if ignore_error ; rest likewise
     L = ha + (rest if evacuate_all) ; if L empty: raise Skip
     insert vmove(pending, evacuation) for s in L                          # masakari/engine/drivers/taskflow/host_failure.py:136
Evacuate(H, R?):                                            # masakari/engine/drivers/taskflow/host_failure.py:349
  V = vmoves(n, status=pending) order by created_at desc, id desc   # => last inserted first
  if R: lock_nonblocking(R) else LockAlreadyAcquired; [aggregates]; enable R; host R.reserved=false
  pool(3).each v in V: evacuate_and_confirm(v, R)
  if any vmove(n) failed or any not in (succeeded, failed): raise HostRecoveryFailure   # counts ALL attempts

evacuate_and_confirm(v, R):                                 # masakari/engine/drivers/taskflow/host_failure.py:193
  s=GET server; locked=GET server.locked; if not locked: lock
  st=s.vm_state; stop_after=true
  if st not in (active,error,stopped): resetState(error); s=GET; if st==resized and power!=4: stop_after=false
  elif st==stopped and task_state is None: stop_after=false
  elif task_state is not None: resetState(error); s=GET; if st==active: stop_after=false
  v.ongoing; POST evacuate(host=R)
  poll(1s, 90s): s'=GET; if s'.vm_state==error != st: raise ; if host(s')!=H and (st==error and s'==active or s'.vm_state==st): done
  if st != active and stop_after: os-stop; poll until stopped (180s); if st==error: resetState(error)
  v.succeeded(dest=host(GET)) ; on timeout: v.failed("Failed reason: timeout.") ; on exception e: v.failed(str(e))
  finally: v.end_time; if not locked: unlock (errors logged)

instance(n):                                                # masakari/engine/manager.py:130
  if (event, vir_domain_event) not in INSTANCE_EVENTS: return ignored
  s=GET; if not HA and not process_all: finished
  if s.vm_state in (paused, rescued): ignored
  if not stopped: (resized -> resetState(active)); os-stop (409 ok if stopped); poll stopped (180s) else error
  if GET.vm_state != stopped: error ; os-start ; poll active (60s) else error ; finished

process(n):                                                 # masakari/engine/manager.py:67
  if event==STARTED: finished
  if event==STOPPED: h.on_maintenance=true
     if process_name != nova-compute: finished
     if not disabled(H): PUT os-services {disabled, reason}; poll disabled (180s) else error ; finished
  else ignored

3.4 Ordering and determinism

• The vmove order (last inserted first, so non-HA before HA by default) is visible in Nova's evacuation order and in the progress messages. A rewrite that "fixes" it should say so.
• The evacuations run in a green thread pool, but with the fakes the call sequence was identical across two runs (143 Nova calls, same order). With a real Nova, network latency interleaves the threads. Compare the sets of calls per instance, not the global order.
• recovery_workflow_details order comes from the configured task lists, not from execution time.

3.5 Recorded behaviour of the algorithm

[nova-calls_stable-2026.2.json](_rewrite/nova-calls_stable-2026.2.json) holds, per scenario, the exact Nova requests:

• the auto, reserved_host (with aggregates, an error VM and a resized VM) and auto_priority fallback flows;
• the VM recovery for an HA instance, a non-HA instance, a paused instance, an ignored event pair and an IO error on a stopped instance;
• the process recovery for nova-compute and for another process;
• the periodic retry.

Together with the final notification status, the vmoves and the recovery_workflow_details (in exchanges), they are the test vectors of the engine. They do not encode real-time behaviour: the sleeps were set to 0 and the timeouts to 3 s.

3.6 Defects a rewrite must decide on

4. What the clients expect

5. Database strategy

• Reuse the schema ([schema-sqlite-migrated_stable-2026.2.sql](_rewrite/schema-sqlite-migrated_stable-2026.2.sql); models = migrations). Keep the alembic table masakari_alembic_version at the head 13adff5efb9a if the Python tools must keep working on the same database.
• Integrity kept by code: vmoves → notification, and notification → host/segment, have no foreign keys; hosts → segment uses the UUID FK. Soft delete everywhere, with deleted = id and unique keys (name, deleted).
• Integer ids are client-visible (markers): keep AUTO_INCREMENT.
• No locking, no optimistic versioning. Notification status transitions should become conditional updates (UPDATE … SET status='running' WHERE notification_uuid=? AND status IN ('new','error')) to make multi-engine processing safe. That is a behaviour improvement, compatible with the API.
• The taskflow persistence database exists only to serve recovery_workflow_details. A rewrite may store the progress in its own table, as long as it returns the same {name, progress, progress_details[], state} list (state values are taskflow's: PENDING, RUNNING, SUCCESS, FAILURE, REVERTING, REVERTED).
• db purge semantics in 05; orphan vmoves are a known gap.

6. Authentication, policies, configuration and CLI

• keystonemiddleware auth_token (incoming), keystoneauth password (outgoing to Nova) with [keystone_authtoken]/auth_type reused as the plugin name (masakari/compute/nova.py:113).
• oslo.policy: 27 rules, every API rule rule:admin_api → is_admin:True ← role:admin. A rewrite can implement role:admin plus the policy file override mechanism, and needs no scope logic to be equivalent.
• oslo.config: the options of § 3.2, [database], [taskflow]/connection, [coordination]/backend_url, [oslo_messaging_*], [keystone_authtoken], [DEFAULT]/os_privileged_user_* (secret: os_privileged_user_password, [taskflow]/connection).
• CLI to keep for deployment tools: masakari-manage db sync|version|purge, masakari-status upgrade check, masakari-api (or the WSGI module masakari/wsgi/api.py), masakari-engine.

7. Conformance strategy

• Scenario suite: tools/adapter_masakari.py drives the real API (keystone pipeline without authtoken, SQLite, in-process RPC delivering to a real MasakariManager) with fake Keystone and Nova behind requests-mock and the real novaclient.
  • Proof: 23/23 scenarios as expected, and a second run gave the same 201 statuses and the same 143 Nova requests in the same order.
  • To check a rewrite: start it with a fake Nova that implements the same small state machine (FakeNova in the adapter, ~150 lines), replay the requests of exchanges_stable-2026.2.json scenario by scenario, and compare statuses, bodies (ignoring UUIDs and timestamps) and the Nova request sequence of nova-calls_stable-2026.2.json.
  • Engine scenarios: deliver the cast synchronously after the 202, or wait for the notification status.
• Project tests: masakari/tests/functional/ drive a real deployment with openstacksdk (segments, hosts, process and VM notifications). They need devstack and were not run. masakari/tests/unit/engine/drivers/taskflow/ hold the per-task unit tests (mocked Nova), useful as reading material for edge cases.
• Not covered: real Nova timing (service-down detection, evacuation duration), keystonemiddleware's token handling, tooz locking, multi-engine races, MySQL-specific behaviour (ENUM, DeleteFromSelect purge).

8. Gaps found while building the pack

• rewrite_pack.py cannot read Masakari's routes (Routes mapper built from stevedore extensions) or its lower-case schema constants. Hence tools/masakari_routes.py and --schemas-modules. Policy operations are declared without /v1, so the post-processor maps them.
• The api-ref documents 17 operations. It does not document the /v1/{project_id} aliases or /v1/extensions. It says vmoves are 1.3 and that markers are "the ID of the last-seen item" without saying it is the integer id.
• conformance_capture.py's error table does not read Nova-style fault bodies, so observed-errors_stable-2026.2.md is produced by the post-processor.
• No gabbi or Tempest-like suite runs in-process. The scenarios are the conformance base.

9. Non-functional requirements

• API latency: POST /notifications sleeps 0–5 s by default without tooz. A rewrite with a real lock (or a unique constraint) should drop the sleep.
• Recovery time of a host failure is at least wait_period_after_service_update (180 s) plus the evacuation (≤ 90 s per instance, 3 in parallel).
• One periodic loop can block for many minutes (sequential retries).
• Scale: the server list is one page (≤ 1000 per host); the periodic queries are unbounded.

10. Rewrite checklist

• [ ] /v1 routes, the /v1/{project_id} aliases, the .json suffix and 405 on unimplemented routes
• [ ] microversion header instance-ha, 1.0–1.3, latest, 406/400 messages
• [ ] fault body {"<faultName>": {code, message}}, plain-text 401 from the auth layer
• [ ] admin-only policies with file overrides
• [ ] segment/host/notification/vmove bodies including id, deleted* and nested failover_segment
• [ ] integer-id markers, sort keys and directions, filters, generated-since
• [ ] Nova nova-compute check on host create and rename (400; 403 passthrough)
• [ ] duplicate detection (any status, sender clock), maintenance 409, unknown host 400, 202 + async
• [ ] engine: the four methods, three flows, vmoves, on_maintenance/reserved side effects, ignored cases
• [ ] retry and expiry loops, single-instance execution (leader election)
• [ ] recovery_workflow_details at 1.1
• [ ] versioned notifications (optional, off by default)
• [ ] masakari-manage db sync/purge, masakari-status upgrade check
• [ ] decide on each defect of § 3.6

Source: reverse-spec-masakari/07-rewrite-pack_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.