OpenKeel: the sovereign French Cloud solution of the inherent group, with a European reach Discover inherent

Reverse-spec · Services

masakari — Errors

Errors — masakari — branch stable/2026.2

Branch stable/2026.2 @ ae41528e1ebc — repository https://opendev.org/openstack/masakari — analysed on 2026-10-28

All exception classes are in masakari/exception.py (55 classes found by the inventory, _inventory/inventory_stable-2026.2.json). The base class MasakariException carries msg_fmt (formatted with the keyword arguments) and code (HTTP status, 500 by default, masakari/exception.py:110). Messages marked observed were seen in [_rewrite/observed-errors_stable-2026.2.md](_rewrite/observed-errors_stable-2026.2.md).

Exception → HTTP translation

Four layers, innermost first:

  1. The handler catches the business exceptions and raises a webob error: except exception.FailoverSegmentExists: raise exc.HTTPConflict(explanation=e.format_message()). This is the only place where FailoverSegmentExists/HostExists (which have no code, so 500) become 409.
  2. **@extensions.expected_errors((…))** (masakari/api/openstack/extensions.py:380) lets through a webob error whose status is declared, a Forbidden or a ValidationError. Everything else becomes 500 masakariFault "Unexpected API Error. Please report this at https://bugs.launchpad.net/masakari/ and attach the Masakari API log if possible.\n<class '…'>", and is logged as "Unexpected exception in API method". The message reveals the Python class.
  3. **ResourceExceptionHandler** (masakari/api/openstack/wsgi.py:347) turns Forbidden into 403, Invalid into its code, a TypeError into 400, and wraps webob errors into a Fault.
  4. **Fault** (masakari/api/openstack/wsgi.py:1003) writes the JSON body {"<faultName>": {"code": N, "message": "…"}}. The fault name comes from the status (badRequest 400, unauthorized 401, forbidden 403, itemNotFound 404, badMethod 405, conflictingRequest 409, overLimit 413/429, badMediaType 415, notImplemented 501, serviceUnavailable 503, otherwise masakariFault), and the message is translated to the request's Accept-Language. Outside the router, **FaultWrapper** (masakari/api/openstack/__init__.py:36) maps any exception to its code (or 500). For an exception flagged safe (none in this branch) it also prefixes the class name.

Errors raised before the router (keystonemiddleware 401, MasakariKeystoneContext 401, paste 404 for an unknown path) are webob plain-text pages, not JSON (observed: "404 Not Found\n\nThe resource could not be found."). The sizelimit filter (oslo.middleware, [oslo_middleware]/max_request_body_size, 114688 bytes by default) answers 413.

Catalogue

ExceptionHTTPMessage (template)SituationRaised atSeen by the client as
ValidationError (masakari/exception.py:246)400%(detail)sthe body breaks the JSON schema of the operation, or of the payload for the notification typemasakari/api/validation/validators.py:145400 badRequest "Invalid input for field/attribute segment. Value: {…}. 'name' is a required property" (observed)
MalformedRequestBody (masakari/exception.py:196)400 via handlerMalformed message body: %(reason)sbody is not JSON, or an action body has 0 or 2+ keysmasakari/api/openstack/wsgi.py:225, masakari/api/openstack/wsgi.py:336400 "Malformed request body" (observed)
InvalidAPIVersionString (masakari/exception.py:191)400API Version String %(version)s is of invalid format. Must be of format MajorNum.MinorNum.header OpenStack-API-Version: instance-ha abc or 0.9masakari/api/api_version_request.py:111400 badRequest (observed)
InvalidGlobalAPIVersion (masakari/exception.py:236)406Version %(req_ver)s is not supported by the API. Minimum is %(min_ver)s and maximum is %(max_ver)s.requested version > 1.3masakari/api/openstack/wsgi.py:199406 masakariFault (masakari/api/openstack/wsgi.py:566, observed)
VersionNotFoundForAPIMethod (masakari/exception.py:232)404API version %(version)s is not supported on this method.no versioned method matches (no handler uses api_version ranges in this branch: never raised)masakari/api/openstack/wsgi.py:745 (caught)—
InvalidInput (masakari/exception.py:187)400Invalid input received: %(reason)slimit negative or not an integer; unknown sort direction; more sort_dir than sort_keymasakari/db/sqlalchemy/api.py:149, masakari/db/sqlalchemy/api.py:160, masakari/api/openstack/common.py:178400 via except exception.Invalid in the handler (observed)
InvalidInput400idema Nova 400 during a host checkmasakari/compute/nova.py:71500 (not declared by expected_errors) [inferred]
InvalidSortKey (masakari/exception.py:250)400Sort key supplied was not valid. but built with the oslo.db exception as messagesort_key is not a columnmasakari/db/sqlalchemy/api.py:212400 "InvalidSortKey('Sort key supplied is invalid: bogus')" (observed: the repr of the oslo.db exception leaks)
MarkerNotFound (masakari/exception.py:254)404 → 400Marker %(marker)s could not be found.the marker is not the integer id of an existing row (a UUID never matches)masakari/db/sqlalchemy/api.py:204 (segments; same in hosts, notifications, vmoves)400 (handlers convert it, observed)
FailoverSegmentNotFound (masakari/exception.py:258)404No failover segment with id %(id)s.id not UUID-like, or no non-deleted segment with itmasakari/ha/api.py:62, masakari/db/sqlalchemy/api.py:242404 itemNotFound (observed)
HostNotFound (masakari/exception.py:262)404No host with id %(id)s.host id not UUID-likemasakari/ha/api.py:168404
HostNotFoundUnderFailoverSegment (masakari/exception.py:374)404Host '%(host_uuid)s' under failover_segment '%(segment_uuid)s' could not be found.the host exists in another segment, or not at all, when a segment is givenmasakari/db/sqlalchemy/api.py:408404 (observed)
HostNotFoundByName (masakari/exception.py:275)404 → 400Host with name %(host_name)s could not be found.POST /notifications from a host that is not registeredmasakari/db/sqlalchemy/api.py:447400 (masakari/api/openstack/ha/notifications.py:79, observed)
ComputeNotFoundByName (masakari/exception.py:279)404 → 400Compute service with name %(compute_name)s could not be found.Nova lists no nova-compute service on that host (case-sensitive) on host create or renamemasakari/compute/nova.py:266400 (masakari/api/openstack/ha/hosts.py:110, observed)
NotificationNotFound (masakari/exception.py:266)404No notification with id %(id)s.id not UUID-like or unknownmasakari/ha/api.py:399, masakari/db/sqlalchemy/api.py:569404; 500 on GET …/vmoves/{id} (not caught, masakari/api/openstack/ha/vmoves.py:77, observed)
VMoveNotFound (masakari/exception.py:284)404No vm move with id %(id)s.vmove id not UUID-like or unknownmasakari/ha/api.py:444, masakari/db/sqlalchemy/api.py:692404 (observed)
NotificationWithoutVMoves (masakari/exception.py:288)400This notification %(id)s without vm moves.vmoves asked for a VM or PROCESS notificationmasakari/ha/api.py:423400 (observed)
FailoverSegmentExists (masakari/exception.py:292)500 → 409Failover segment with name %(name)s already exists.duplicate name (DB unique name, deleted) on create or renamemasakari/db/sqlalchemy/api.py:268, masakari/db/sqlalchemy/api.py:282409 conflictingRequest (observed)
HostExists (masakari/exception.py:296)500 → 409Host with name %(name)s already exists.the name is used by a host of any segmentmasakari/db/sqlalchemy/api.py:460, masakari/db/sqlalchemy/api.py:474409 (observed)
FailoverSegmentInUse (masakari/exception.py:352)409built message: Failover segment %s can't be updated as it is in-use to process notifications. / Failover segment (%s) can't be deleted as …a host of the segment has a notification new, running or errormasakari/ha/api.py:118, masakari/ha/api.py:139409 (observed)
HostInUse (masakari/exception.py:357)409built message: Host %s can't be updated as it is in-use to process notifications. / …deleted…same condition, on the host's segmentmasakari/ha/api.py:226, masakari/ha/api.py:258409 (observed)
DuplicateNotification (masakari/exception.py:313)409built message: Notification received from host %(host)s of type %(type)s is duplicate.same type, host and payload within the detection windowmasakari/ha/api.py:329409 (observed); masakari-monitors stops retrying
HostOnMaintenanceError (masakari/exception.py:318)409built message: Notification received from host %(host)s of type '%(type)s' is ignored as the host is already under maintenance.the host is on_maintenance (often because the engine set it during a previous recovery)masakari/ha/api.py:311409 (observed)
Forbidden / PolicyNotAuthorized (masakari/exception.py:220)403Policy doesn't allow %(action)s to be performed.the caller does not have the admin rolemasakari/policy.py:126403 forbidden (observed)
Forbidden403Nova's message, e.g. Policy doesn't allow (HTTP 403)the service account is refused by Nova during a host checkmasakari/compute/nova.py:75403 to the Masakari client (observed): a service-account problem looks like a client authorisation error
Conflict (masakari/exception.py:172)409Conflicta Nova 409 (in the engine: stopping an already stopped server, adding a host already in an aggregate)masakari/compute/nova.py:81engine only: handled in masakari/engine/drivers/taskflow/instance_failure.py:78 and masakari/engine/drivers/taskflow/host_failure.py:383
NotFound (masakari/exception.py:202)404Resource could not be found. (the reason kwarg is not in msg_fmt and is dropped)Nova 404 (unknown server)masakari/compute/nova.py:78engine: vmove failed, or a workflow error
MasakariException (masakari/exception.py:110)500An unknown exception occurred. (reason is dropped as above)Nova/Keystone timeout, connection error, discovery failure, novaclient CommandErrormasakari/compute/nova.py:67API: 500; engine: notification error
FailoverSegmentDisabled (masakari/exception.py:383)—the message is a tuple (format string and dict, never formatted)notification for a host of a disabled segmentmasakari/engine/manager.py:350not seen by the client: the notification becomes ignored; the exception escapes the cast handler and oslo.messaging logs it as an error (observed in _rewrite/rpc-frames_stable-2026.2.json, engine_exceptions)
InvalidConfiguration (masakari/exception.py:387)—[taskflow] connection should be configured to use this driver.[taskflow]/connection missing at engine startmasakari/engine/drivers/taskflow/driver.py:51the engine does not start
LockAlreadyAcquired (masakari/exception.py:366)—Lock is already acquired on %(resource)s.two evacuations target the same reserved host in the same engine process (non-blocking lock)masakari/utils.py:255caught as HostRecoveryFailureException (masakari/engine/drivers/taskflow/driver.py:80); taskflow then tries the next reserved host

Classes that exist but are not raised by the API or engine code in this branch: APIException, APITimeout, InvalidName, ConfigNotFound, AdminRequired, PasteAppNotFound (raised by the WSGI loader at start-up), InvalidContentType, ApiVersionsIntersect, Unauthorized, ObjectActionError (object misuse), OrphanedObjectError, DBNotAllowed, FailoverSegmentNotFoundByName.

Asynchronous errors (engine)

They never reach the POST caller: they show up in the notification's status, in the vmoves, in the recovery_workflow_details (1.1) and in the logs.

ExceptionRaised whenNotification statusRaised at
SkipHostRecoveryException (masakari/exception.py:339)no instance to evacuate (none on the host, or none HA-enabled when evacuate_all_instances=False)finishedmasakari/engine/drivers/taskflow/host_failure.py:133
HostRecoveryFailureException (masakari/exception.py:323)at least one vmove failed, or still pending/ongoing after the pool finished; the reserved-host lock is takenerrormasakari/engine/drivers/taskflow/host_failure.py:433, masakari/engine/drivers/taskflow/host_failure.py:450, masakari/engine/drivers/taskflow/driver.py:81
ReservedHostsUnavailable (masakari/exception.py:362)reserved_host/rh_priority (or the fallback of auto_priority) with no host reserved=true, on_maintenance=false in the segmenterrormasakari/engine/drivers/taskflow/driver.py:70
InstanceEvacuateFailed (masakari/exception.py:379)Nova put the instance in error while evacuatingvmove failed with that messagemasakari/engine/drivers/taskflow/host_failure.py:222
SkipInstanceRecoveryException (masakari/exception.py:331)VM not HA-enabled and [instance_failure]/process_all_instances=Falsefinished (nothing done)masakari/engine/drivers/taskflow/instance_failure.py:56
IgnoreInstanceRecoveryException (masakari/exception.py:370)VM in paused or rescuedignoredmasakari/engine/drivers/taskflow/instance_failure.py:66
InstanceRecoveryFailureException (masakari/exception.py:327)VM not stopped within wait_period_after_power_off; not stopped when starting; not active within wait_period_after_power_onerrormasakari/engine/drivers/taskflow/instance_failure.py:107, masakari/engine/drivers/taskflow/instance_failure.py:139, masakari/engine/drivers/taskflow/instance_failure.py:174
SkipProcessRecoveryException (masakari/exception.py:335)process other than nova-computefinished (but the host stays on maintenance)masakari/engine/drivers/taskflow/driver.py:223
ProcessRecoveryFailureException (masakari/exception.py:343)compute service not seen disabled within wait_period_after_service_updateerrormasakari/engine/drivers/taskflow/process_failure.py:90
MasakariException "Failed to execute host failure flow for notification '%s'."any other exception in a host flow (Nova down, Keystone down…)errormasakari/engine/drivers/taskflow/driver.py:167
MasakariException "Failed to execute instance failure flow…"any other exception in an instance or process flow (the process flow reuses the instance message, masakari/engine/drivers/taskflow/driver.py:245)errormasakari/engine/drivers/taskflow/driver.py:203
MasakariException "Failed to fetch notification recovery workflow details for %s"the taskflow database cannot be readAPI 500masakari/engine/manager.py:436

**Exceptions that are not MasakariException** (for example a KeyError in a third-party task, or a database error) escape the except clauses of masakari/engine/manager.py:99, masakari/engine/manager.py:149 and masakari/engine/manager.py:241. The notification then stays **running** until _check_expired_notifications sets it to failed after notifications_expired_interval (24 h) [inferred].

error is retried once by the periodic task and then becomes failed (05).

Key log messages

LevelMessageSituationWhere
ERROR (exception)Unexpected exception in API methodany untranslated exception in the API (500)masakari/api/openstack/extensions.py:404
ERROR (exception)Caught error: %sexception that reached FaultWrappermasakari/api/openstack/__init__.py:50
ERRORFailover segment %s can't be updated as it is in-use to process notifications.409 in usemasakari/ha/api.py:117
WARNINGProcessing of notification is skipped to avoid recovering from failure twice…a replayed new message for a notification that is no longer newmasakari/engine/manager.py:290
ERRORFailed to process notification '%(uuid)s'. Reason: %(error)sworkflow failure, notification errormasakari/engine/manager.py:245
WARNINGNotification '%(uuid)s' ignored as host_status is '%(host_status)s'host notification with host_status=UNKNOWNmasakari/engine/manager.py:189
WARNINGFailed to evacuate all instances from failed_host: '%(failed_host)s' using '%(auto)s' workflow, retrying using '%(reserved_host)s' workflow.auto_priority fallbackmasakari/engine/drivers/taskflow/driver.py:95
WARNINGTimeout for instance '%(uuid)s' evacuation.evacuation not confirmed within wait_period_after_evacuationmasakari/engine/drivers/taskflow/host_failure.py:240
WARNINGFailed to unlock instance %(uuid)s after evacuation: %(error)sthe instance may stay locked in Novamasakari/engine/drivers/taskflow/host_failure.py:344
ERRORPeriodic task 'process_unfinished_notifications': Notification %(notification_uuid)s exits with status: %(status)s.error → failedmasakari/engine/manager.py:385
ERRORPeriodic task 'check_expired_notifications': Notification %(notification_uuid)s is expired.expired notification set to failedmasakari/engine/manager.py:415
INFOSleeping %(wait)s sec before starting recovery thread until nova recognizes the node down.host flow, after disabling the compute service (default 180 s)masakari/engine/drivers/taskflow/host_failure.py:56

Source: reverse-spec-masakari/02-errors_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.