Errors — masakari — branch stable/2026.2
Branch
stable/2026.2@ae41528e1ebc— repository https://opendev.org/openstack/masakari — analysed on 2026-10-28
All exception classes are in masakari/exception.py (55 classes found by the inventory, _inventory/inventory_stable-2026.2.json). The base class MasakariException carries msg_fmt (formatted with the keyword arguments) and code (HTTP status, 500 by default, masakari/exception.py:110). Messages marked observed were seen in [_rewrite/observed-errors_stable-2026.2.md](_rewrite/observed-errors_stable-2026.2.md).
Exception → HTTP translation
Four layers, innermost first:
- The handler catches the business exceptions and raises a webob error:
except exception.FailoverSegmentExists: raise exc.HTTPConflict(explanation=e.format_message()). This is the only place whereFailoverSegmentExists/HostExists(which have nocode, so 500) become 409. - **
@extensions.expected_errors((…))** (masakari/api/openstack/extensions.py:380) lets through a webob error whose status is declared, aForbiddenor aValidationError. Everything else becomes 500masakariFault"Unexpected API Error. Please report this at https://bugs.launchpad.net/masakari/ and attach the Masakari API log if possible.\n<class '…'>", and is logged as "Unexpected exception in API method". The message reveals the Python class. - **
ResourceExceptionHandler** (masakari/api/openstack/wsgi.py:347) turnsForbiddeninto 403,Invalidinto itscode, aTypeErrorinto 400, and wraps webob errors into aFault. - **
Fault** (masakari/api/openstack/wsgi.py:1003) writes the JSON body{"<faultName>": {"code": N, "message": "…"}}. The fault name comes from the status (badRequest400,unauthorized401,forbidden403,itemNotFound404,badMethod405,conflictingRequest409,overLimit413/429,badMediaType415,notImplemented501,serviceUnavailable503, otherwisemasakariFault), and the message is translated to the request'sAccept-Language. Outside the router, **FaultWrapper** (masakari/api/openstack/__init__.py:36) maps any exception to itscode(or 500). For an exception flaggedsafe(none in this branch) it also prefixes the class name.
Errors raised before the router (keystonemiddleware 401, MasakariKeystoneContext 401, paste 404 for an unknown path) are webob plain-text pages, not JSON (observed: "404 Not Found\n\nThe resource could not be found."). The sizelimit filter (oslo.middleware, [oslo_middleware]/max_request_body_size, 114688 bytes by default) answers 413.
Catalogue
| Exception | HTTP | Message (template) | Situation | Raised at | Seen by the client as |
|---|---|---|---|---|---|
ValidationError (masakari/exception.py:246) | 400 | %(detail)s | the body breaks the JSON schema of the operation, or of the payload for the notification type | masakari/api/validation/validators.py:145 | 400 badRequest "Invalid input for field/attribute segment. Value: {…}. 'name' is a required property" (observed) |
MalformedRequestBody (masakari/exception.py:196) | 400 via handler | Malformed message body: %(reason)s | body is not JSON, or an action body has 0 or 2+ keys | masakari/api/openstack/wsgi.py:225, masakari/api/openstack/wsgi.py:336 | 400 "Malformed request body" (observed) |
InvalidAPIVersionString (masakari/exception.py:191) | 400 | API Version String %(version)s is of invalid format. Must be of format MajorNum.MinorNum. | header OpenStack-API-Version: instance-ha abc or 0.9 | masakari/api/api_version_request.py:111 | 400 badRequest (observed) |
InvalidGlobalAPIVersion (masakari/exception.py:236) | 406 | Version %(req_ver)s is not supported by the API. Minimum is %(min_ver)s and maximum is %(max_ver)s. | requested version > 1.3 | masakari/api/openstack/wsgi.py:199 | 406 masakariFault (masakari/api/openstack/wsgi.py:566, observed) |
VersionNotFoundForAPIMethod (masakari/exception.py:232) | 404 | API version %(version)s is not supported on this method. | no versioned method matches (no handler uses api_version ranges in this branch: never raised) | masakari/api/openstack/wsgi.py:745 (caught) | — |
InvalidInput (masakari/exception.py:187) | 400 | Invalid input received: %(reason)s | limit negative or not an integer; unknown sort direction; more sort_dir than sort_key | masakari/db/sqlalchemy/api.py:149, masakari/db/sqlalchemy/api.py:160, masakari/api/openstack/common.py:178 | 400 via except exception.Invalid in the handler (observed) |
InvalidInput | 400 | idem | a Nova 400 during a host check | masakari/compute/nova.py:71 | 500 (not declared by expected_errors) [inferred] |
InvalidSortKey (masakari/exception.py:250) | 400 | Sort key supplied was not valid. but built with the oslo.db exception as message | sort_key is not a column | masakari/db/sqlalchemy/api.py:212 | 400 "InvalidSortKey('Sort key supplied is invalid: bogus')" (observed: the repr of the oslo.db exception leaks) |
MarkerNotFound (masakari/exception.py:254) | 404 → 400 | Marker %(marker)s could not be found. | the marker is not the integer id of an existing row (a UUID never matches) | masakari/db/sqlalchemy/api.py:204 (segments; same in hosts, notifications, vmoves) | 400 (handlers convert it, observed) |
FailoverSegmentNotFound (masakari/exception.py:258) | 404 | No failover segment with id %(id)s. | id not UUID-like, or no non-deleted segment with it | masakari/ha/api.py:62, masakari/db/sqlalchemy/api.py:242 | 404 itemNotFound (observed) |
HostNotFound (masakari/exception.py:262) | 404 | No host with id %(id)s. | host id not UUID-like | masakari/ha/api.py:168 | 404 |
HostNotFoundUnderFailoverSegment (masakari/exception.py:374) | 404 | Host '%(host_uuid)s' under failover_segment '%(segment_uuid)s' could not be found. | the host exists in another segment, or not at all, when a segment is given | masakari/db/sqlalchemy/api.py:408 | 404 (observed) |
HostNotFoundByName (masakari/exception.py:275) | 404 → 400 | Host with name %(host_name)s could not be found. | POST /notifications from a host that is not registered | masakari/db/sqlalchemy/api.py:447 | 400 (masakari/api/openstack/ha/notifications.py:79, observed) |
ComputeNotFoundByName (masakari/exception.py:279) | 404 → 400 | Compute service with name %(compute_name)s could not be found. | Nova lists no nova-compute service on that host (case-sensitive) on host create or rename | masakari/compute/nova.py:266 | 400 (masakari/api/openstack/ha/hosts.py:110, observed) |
NotificationNotFound (masakari/exception.py:266) | 404 | No notification with id %(id)s. | id not UUID-like or unknown | masakari/ha/api.py:399, masakari/db/sqlalchemy/api.py:569 | 404; 500 on GET …/vmoves/{id} (not caught, masakari/api/openstack/ha/vmoves.py:77, observed) |
VMoveNotFound (masakari/exception.py:284) | 404 | No vm move with id %(id)s. | vmove id not UUID-like or unknown | masakari/ha/api.py:444, masakari/db/sqlalchemy/api.py:692 | 404 (observed) |
NotificationWithoutVMoves (masakari/exception.py:288) | 400 | This notification %(id)s without vm moves. | vmoves asked for a VM or PROCESS notification | masakari/ha/api.py:423 | 400 (observed) |
FailoverSegmentExists (masakari/exception.py:292) | 500 → 409 | Failover segment with name %(name)s already exists. | duplicate name (DB unique name, deleted) on create or rename | masakari/db/sqlalchemy/api.py:268, masakari/db/sqlalchemy/api.py:282 | 409 conflictingRequest (observed) |
HostExists (masakari/exception.py:296) | 500 → 409 | Host with name %(name)s already exists. | the name is used by a host of any segment | masakari/db/sqlalchemy/api.py:460, masakari/db/sqlalchemy/api.py:474 | 409 (observed) |
FailoverSegmentInUse (masakari/exception.py:352) | 409 | built message: Failover segment %s can't be updated as it is in-use to process notifications. / Failover segment (%s) can't be deleted as … | a host of the segment has a notification new, running or error | masakari/ha/api.py:118, masakari/ha/api.py:139 | 409 (observed) |
HostInUse (masakari/exception.py:357) | 409 | built message: Host %s can't be updated as it is in-use to process notifications. / …deleted… | same condition, on the host's segment | masakari/ha/api.py:226, masakari/ha/api.py:258 | 409 (observed) |
DuplicateNotification (masakari/exception.py:313) | 409 | built message: Notification received from host %(host)s of type %(type)s is duplicate. | same type, host and payload within the detection window | masakari/ha/api.py:329 | 409 (observed); masakari-monitors stops retrying |
HostOnMaintenanceError (masakari/exception.py:318) | 409 | built message: Notification received from host %(host)s of type '%(type)s' is ignored as the host is already under maintenance. | the host is on_maintenance (often because the engine set it during a previous recovery) | masakari/ha/api.py:311 | 409 (observed) |
Forbidden / PolicyNotAuthorized (masakari/exception.py:220) | 403 | Policy doesn't allow %(action)s to be performed. | the caller does not have the admin role | masakari/policy.py:126 | 403 forbidden (observed) |
Forbidden | 403 | Nova's message, e.g. Policy doesn't allow (HTTP 403) | the service account is refused by Nova during a host check | masakari/compute/nova.py:75 | 403 to the Masakari client (observed): a service-account problem looks like a client authorisation error |
Conflict (masakari/exception.py:172) | 409 | Conflict | a Nova 409 (in the engine: stopping an already stopped server, adding a host already in an aggregate) | masakari/compute/nova.py:81 | engine only: handled in masakari/engine/drivers/taskflow/instance_failure.py:78 and masakari/engine/drivers/taskflow/host_failure.py:383 |
NotFound (masakari/exception.py:202) | 404 | Resource could not be found. (the reason kwarg is not in msg_fmt and is dropped) | Nova 404 (unknown server) | masakari/compute/nova.py:78 | engine: vmove failed, or a workflow error |
MasakariException (masakari/exception.py:110) | 500 | An unknown exception occurred. (reason is dropped as above) | Nova/Keystone timeout, connection error, discovery failure, novaclient CommandError | masakari/compute/nova.py:67 | API: 500; engine: notification error |
FailoverSegmentDisabled (masakari/exception.py:383) | — | the message is a tuple (format string and dict, never formatted) | notification for a host of a disabled segment | masakari/engine/manager.py:350 | not seen by the client: the notification becomes ignored; the exception escapes the cast handler and oslo.messaging logs it as an error (observed in _rewrite/rpc-frames_stable-2026.2.json, engine_exceptions) |
InvalidConfiguration (masakari/exception.py:387) | — | [taskflow] connection should be configured to use this driver. | [taskflow]/connection missing at engine start | masakari/engine/drivers/taskflow/driver.py:51 | the engine does not start |
LockAlreadyAcquired (masakari/exception.py:366) | — | Lock is already acquired on %(resource)s. | two evacuations target the same reserved host in the same engine process (non-blocking lock) | masakari/utils.py:255 | caught as HostRecoveryFailureException (masakari/engine/drivers/taskflow/driver.py:80); taskflow then tries the next reserved host |
Classes that exist but are not raised by the API or engine code in this branch: APIException, APITimeout, InvalidName, ConfigNotFound, AdminRequired, PasteAppNotFound (raised by the WSGI loader at start-up), InvalidContentType, ApiVersionsIntersect, Unauthorized, ObjectActionError (object misuse), OrphanedObjectError, DBNotAllowed, FailoverSegmentNotFoundByName.
Asynchronous errors (engine)
They never reach the POST caller: they show up in the notification's status, in the vmoves, in the recovery_workflow_details (1.1) and in the logs.
| Exception | Raised when | Notification status | Raised at |
|---|---|---|---|
SkipHostRecoveryException (masakari/exception.py:339) | no instance to evacuate (none on the host, or none HA-enabled when evacuate_all_instances=False) | finished | masakari/engine/drivers/taskflow/host_failure.py:133 |
HostRecoveryFailureException (masakari/exception.py:323) | at least one vmove failed, or still pending/ongoing after the pool finished; the reserved-host lock is taken | error | masakari/engine/drivers/taskflow/host_failure.py:433, masakari/engine/drivers/taskflow/host_failure.py:450, masakari/engine/drivers/taskflow/driver.py:81 |
ReservedHostsUnavailable (masakari/exception.py:362) | reserved_host/rh_priority (or the fallback of auto_priority) with no host reserved=true, on_maintenance=false in the segment | error | masakari/engine/drivers/taskflow/driver.py:70 |
InstanceEvacuateFailed (masakari/exception.py:379) | Nova put the instance in error while evacuating | vmove failed with that message | masakari/engine/drivers/taskflow/host_failure.py:222 |
SkipInstanceRecoveryException (masakari/exception.py:331) | VM not HA-enabled and [instance_failure]/process_all_instances=False | finished (nothing done) | masakari/engine/drivers/taskflow/instance_failure.py:56 |
IgnoreInstanceRecoveryException (masakari/exception.py:370) | VM in paused or rescued | ignored | masakari/engine/drivers/taskflow/instance_failure.py:66 |
InstanceRecoveryFailureException (masakari/exception.py:327) | VM not stopped within wait_period_after_power_off; not stopped when starting; not active within wait_period_after_power_on | error | masakari/engine/drivers/taskflow/instance_failure.py:107, masakari/engine/drivers/taskflow/instance_failure.py:139, masakari/engine/drivers/taskflow/instance_failure.py:174 |
SkipProcessRecoveryException (masakari/exception.py:335) | process other than nova-compute | finished (but the host stays on maintenance) | masakari/engine/drivers/taskflow/driver.py:223 |
ProcessRecoveryFailureException (masakari/exception.py:343) | compute service not seen disabled within wait_period_after_service_update | error | masakari/engine/drivers/taskflow/process_failure.py:90 |
MasakariException "Failed to execute host failure flow for notification '%s'." | any other exception in a host flow (Nova down, Keystone down…) | error | masakari/engine/drivers/taskflow/driver.py:167 |
MasakariException "Failed to execute instance failure flow…" | any other exception in an instance or process flow (the process flow reuses the instance message, masakari/engine/drivers/taskflow/driver.py:245) | error | masakari/engine/drivers/taskflow/driver.py:203 |
MasakariException "Failed to fetch notification recovery workflow details for %s" | the taskflow database cannot be read | API 500 | masakari/engine/manager.py:436 |
**Exceptions that are not MasakariException** (for example a KeyError in a third-party task, or a database error) escape the except clauses of masakari/engine/manager.py:99, masakari/engine/manager.py:149 and masakari/engine/manager.py:241. The notification then stays **running** until _check_expired_notifications sets it to failed after notifications_expired_interval (24 h) [inferred].
error is retried once by the periodic task and then becomes failed (05).
Key log messages
| Level | Message | Situation | Where |
|---|---|---|---|
| ERROR (exception) | Unexpected exception in API method | any untranslated exception in the API (500) | masakari/api/openstack/extensions.py:404 |
| ERROR (exception) | Caught error: %s | exception that reached FaultWrapper | masakari/api/openstack/__init__.py:50 |
| ERROR | Failover segment %s can't be updated as it is in-use to process notifications. | 409 in use | masakari/ha/api.py:117 |
| WARNING | Processing of notification is skipped to avoid recovering from failure twice… | a replayed new message for a notification that is no longer new | masakari/engine/manager.py:290 |
| ERROR | Failed to process notification '%(uuid)s'. Reason: %(error)s | workflow failure, notification error | masakari/engine/manager.py:245 |
| WARNING | Notification '%(uuid)s' ignored as host_status is '%(host_status)s' | host notification with host_status=UNKNOWN | masakari/engine/manager.py:189 |
| WARNING | Failed to evacuate all instances from failed_host: '%(failed_host)s' using '%(auto)s' workflow, retrying using '%(reserved_host)s' workflow. | auto_priority fallback | masakari/engine/drivers/taskflow/driver.py:95 |
| WARNING | Timeout for instance '%(uuid)s' evacuation. | evacuation not confirmed within wait_period_after_evacuation | masakari/engine/drivers/taskflow/host_failure.py:240 |
| WARNING | Failed to unlock instance %(uuid)s after evacuation: %(error)s | the instance may stay locked in Nova | masakari/engine/drivers/taskflow/host_failure.py:344 |
| ERROR | Periodic task 'process_unfinished_notifications': Notification %(notification_uuid)s exits with status: %(status)s. | error → failed | masakari/engine/manager.py:385 |
| ERROR | Periodic task 'check_expired_notifications': Notification %(notification_uuid)s is expired. | expired notification set to failed | masakari/engine/manager.py:415 |
| INFO | Sleeping %(wait)s sec before starting recovery thread until nova recognizes the node down. | host flow, after disabling the compute service (default 180 s) | masakari/engine/drivers/taskflow/host_failure.py:56 |
Source: reverse-spec-masakari/02-errors_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.