ironic — Rewrite pack

Rewrite pack — ironic — branch stable/2026.2

Branch stable/2026.2 @ 39ce09ec3b0d — repository https://opendev.org/openstack/ironic — analysed on 2026-10-29 Target: openkeel-ironic (Go): an API Deployment plus a conductor StatefulSet on the provisioning network, Redfish (virtual media, HTTP boot) or IPMI; ironic-python-agent stays in Python in the ramdisk; Nova routes bare metal flavors to Ironic. Client repositories cited (prefix: commit): nova: 8df1922926dd (34.0.0+7), ipa: = ironic-python-agent daa18e2103ef (13.0.0+3), ironicclient: = python-ironicclient af2ceeb9c4f2 (6.3.0+2), sdk: = openstacksdk fbdc403819e0 (4.20.0+3), heat: 5872396e548c (27.0.1), skyline: = skyline-console 806d200392ee (9.0.0+1), backend: = ~/.claude/skills/generate-go-from-spec/references/backends/ (skill directory).

1. Contents of _rewrite/

Library versions used for the extraction (Python 3.12 venv, upper-constraints stable/2026.2): pecan 1.8.0, WebOb 1.8.11, automaton 3.5.0, tooz 9.1.0, oslo.messaging 18.3.1, oslo.policy 6.0.1, oslo.db 18.1.1, SQLAlchemy 2.0.52, alembic 1.19.1, keystonemiddleware 13.0.1, sushy 5.13.1.dev6 (local stable/2026.2 clone), futurist 3.5.0, jsonschema 4.26.0, os-traits 3.9.0.

2. HTTP behaviour that the contracts do not show

2.1 Middleware pipeline (outermost first, ironic/api/app.py:80)

1. custom [api]/middleware entry points (none by default);
2. RequestLogMiddleware;
3. JsonExtensionMiddleware (strips .json suffix);
4. IronicCORS;
5. IronicRequestId (x-openstack-request-id + openstack-request-id);
6. /healthcheck dispatcher (before auth, when [healthcheck]/enabled);
7. HTTPProxyToWSGI; 8. osprofiler (optional);
8. AuthPublicRoutes → keystonemiddleware auth_token / HTTP basic, skipped for /, /v1, /v1/lookup, /v1/heartbeat/*, /v1/continue_inspection;
9. audit (optional);
10. JsonDepthMiddleware (depth and size, before parsing);
11. ParsableErrorMiddleware (error body wrapping, default Accept);
12. Pecan with hooks ConfigHook, DBHook, ContextHook, RPCHook, NoExceptionTracebackHook, PublicUrlHook (ironic/api/hooks.py), then the v1 controller which sets the version headers and checks the version (406) before routing to resources (ironic/api/controllers/v1/__init__.py:177).

Which error wins: 401 (auth) before 400/413 (JSON size/depth) before 406 (version) before 404 (unknown resource) before policy (403/404) before validation (400).

2.2 Error body

{"error_message": "{\"faultcode\": \"Client\", \"faultstring\": \"Node … could not be found.\", \"debuginfo\": null}"} — the inner document is a JSON string. Every client in § 4 parses it this way (ironicclient decodes error_message then reads faultstring, IPA too, the SDK reads faultstring). debuginfo carries a traceback only for 5xx with [DEFAULT]/debug_tracebacks_in_api. Response Content-Type: application/json with the length recomputed.

2.3 Other cross-cutting rules

• No header = microversion 1.1 (not latest). latest is accepted. Version headers on every response including errors (needed by ironicclient's 406 negotiation, ironicclient:ironicclient/common/http.py:131).
• Field visibility by microversion: VERSIONED_FIELDS (ironic/api/controllers/v1/utils.py:850) — a field newer than the request is removed from responses and rejected in requests (406).
• Secrets: driver_info passwords masked as "******", all of driver_info redacted without baremetal:node:get:driver_info, instance_info secrets and configdrive masked (ironic/api/controllers/v1/node.py:1905).
• Foreign project → 404, never 403, for node-scoped resources (conceal_node), while a missing role on an owned node gives 403.
• Location header on 201/202 (…/states for provision changes).
• Public URL: links use [api]/public_endpoint or the request host (PublicUrlHook).

3. Core algorithms: formal specifications

3.1 Provision state machine

Model. An automaton finite machine wrapped by ironic/common/fsm.py:50 and populated in ironic/common/state_machine.py. Each state has stable (bool) and a default target (a stable state). Each TaskManager copies the machine and initialises it from node.provision_state and node.target_provision_state (ironic/conductor/task_manager.py:256).

States (30):

<!-- GEN:sm-states -->

<!-- /GEN:sm-states -->

Transitions (93, the complete relation; any other (state, event) pair raises InvalidState):

<!-- GEN:sm-transitions -->

<!-- /GEN:sm-transitions -->

State sets used by the rest of the code:

<!-- GEN:sm-sets -->

<!-- /GEN:sm-sets -->

Semantics (ironic/common/fsm.py:101, ironic/conductor/task_manager.py:553):

1. process_event(e, target_state=t): if (current, e) is not a transition → InvalidState (409). Else current := dest.
2. FSM target: if the new state is the previous target, clear it; if the new state has a default target, it becomes the target; an explicit target_state (must be a stable state) overrides it. This is how manual cleaning keeps target = manageable through cleaning/clean wait (ironic/conductor/cleaning.py:183, ironic/conductor/task_manager.py:636) while automated cleaning targets available.
3. Node update by TaskManager: provision_state := current; target_provision_state := None if the new state is stable and no background callback is attached, else the FSM target; last_error set when given; node.save(); then the provision notification (start when going stable→unstable, end unstable→stable, success otherwise, error for fail), emitted after the spawned worker is accepted (ironic/conductor/task_manager.py:508).
4. The API pre-checks with is_actionable_event on a fresh copy (ironic/api/controllers/v1/node.py:1282); the conductor re-checks under the node lock (authoritative).
5. Verbs → events: VERBS (ironic/common/states.py:24): API active and deploy → deploy, deleted and undeploy → delete, others identical; rebuild is an event, not a state.
6. Conductor-side dispatch of do_provisioning_action (ironic/conductor/manager.py:1458): provide from manageable (refused in maintenance unless allowed, refused for retired nodes) → provide + automated cleaning; manage from enroll → verify; adopt from manageable/adopt failed; abort from clean wait, rescue wait, inspect wait, clean hold, wait call-back, deploy hold, service wait, service hold, service failed → _do_abort (§ 3.3.6); unhold; anything else → plain process_event, with InvalidState turned into InvalidStateRequested (400).
7. Stuck states: on conductor start, nodes in STUCK_STATES_TREATED_AS_FAIL get fail (ironic/conductor/base_manager.py:327); for verifying the fail event goes back to enroll, for deleting to error.

Vectors. state-machine-vectors_stable-2026.2.json: 30 states × 21 events = 630 cases, 93 valid; each valid case gives new_state, fsm_target, target_provision_state_no_callback and target_provision_state_with_callback. A Go implementation is conformant when it reproduces all 630 outcomes and the 9 target_overrides cases. They are generated from the live machine, so they are the machine; they do not encode the conductor-side guards of item 6 (maintenance, retired, abortable steps), which are covered by the unit tests listed in § 7.

3.2 Node locks (TaskManager)

• acquire(ctx, node, shared=False, purpose, retry=True, patient=False) (ironic/conductor/task_manager.py:158).
• Exclusive: UPDATE nodes SET reservation=<conductor host> WHERE id=? AND reservation IS NULL (ironic/db/sqlalchemy/api.py:752); failure → NodeLocked; retried node_locked_retry_attempts (3) × node_locked_retry_interval (1 s) with retry, forever with patient, once without (ironic/conductor/task_manager.py:343).
• Shared: no DB write; upgrade_lock(retry) turns it exclusive and reloads the node.
• Release when the context exits, or when the spawned worker finishes (spawn_after, ironic/conductor/task_manager.py:411); a worker exception is written to last_error ("Async execution of %(method)s failed with error: %(error)s").
• Lock holder name = the conductor host (with :port under JSON-RPC). A Go conductor must use a stable per-pod name (StatefulSet ordinal) so that a restarted pod can clear its own reservations (ironic/db/sqlalchemy/api.py:1538).
• require_exclusive_lock guards (ironic/conductor/task_manager.py:128) → ExclusiveLockRequired (403).

3.3 Step framework (clean, deploy, service, verify)

3.3.1 Step model

A step is {"interface", "step", "priority", "args", "abortable"?, "argsinfo"?, "requires_ramdisk"?, "execute_on_child_nodes"?, "limit_child_node_execution"?}, declared on driver methods by @clean_step, @deploy_step, @service_step, @verify_step (ironic/drivers/base.py:2214, ironic/drivers/base.py:2298, ironic/drivers/base.py:2369, ironic/drivers/base.py:2410), or returned at run time by the agent (clean.get_clean_steps, deploy.get_deploy_steps, service.get_service_steps), cached in driver_internal_info.agent_cached_*_steps. Step id = "<interface>.<step>" (ironic/conductor/steps.py:571).

3.3.2 Ordering

• Interface priorities (tie-breakers, higher first): clean, deploy and service vendor 7, power 6, management 5, firmware 4, deploy 3, bios 2, raid 1 (service adds network 8); verify power 13, management 12, firmware 11, inspect 10, deploy 9, boot 8, bios 7, raid 6, vendor 5, network 4, storage 3, console 2, rescue 1 (ironic/conductor/steps.py:30).
• Sort key (priority, interface_priority), descending, stable sort (Python sorted(..., reverse=True) keeps input order for full ties), ironic/conductor/steps.py:125. Service steps are not sorted (user order).
• Priority 0 = disabled, removed when enabled=True after applying [conductor]/clean_step_priority_override / verify_step_priority_override ({"<iface>.<step>": prio}), ironic/conductor/steps.py:148.
• Operator deny lists [api]/disallow_{clean,deploy,service}_steps: user steps → StepNotAllowed (400), automated/runbook steps silently filtered (ironic/conductor/steps.py:582).

3.3.3 Composition per flow

• Automated cleaning: enabled driver clean steps sorted (ironic/conductor/steps.py:198), or a runbook's steps when provide uses one (declarative cleaning), validated as user steps. Skipped entirely when node.automated_clean is false or [conductor]/automated_clean false (ironic/conductor/utils.py:1227).
• Manual cleaning: the user's list in the user's order (no sort), each validated against the driver's steps (ironic/conductor/steps.py:925); abortable and priority copied from the driver step.
• Deploy (ironic/conductor/steps.py:457): user deploy steps (deploy_steps argument, 1.69+) ∪ steps of deploy templates whose name is in instance_info.traits (user wins on the same interface.step) — then driver enabled steps not overridden — then user/template steps with priority > 0 — sorted by the deploy key. Duplicate interface.step across templates → error. Overriding the core deploy.deploy step with priority > 0 → error ("core step").
• Service: user list, unsorted, validated against get_service_steps.
• Verify (on manage): enabled verify steps sorted by the verify key.

3.3.4 Validation of a user step (ironic/conductor/steps.py:650, ironic/conductor/steps.py:814)

1. Skip validation for execute_on_child_nodes and reserved names power_on, power_off, reboot, hold, wait.
2. Unknown interface.step → error "node does not support this <type> step", unless skip_missing (in-band steps not yet known before the agent boots; revalidated after the first heartbeat).
3. Unexpected args (not in argsinfo) → error; missing required args → error (for deploy steps only when priority > 0); disable_ramdisk with a step requiring the ramdisk → error.
4. All errors joined with "; " into one InvalidParameterValue (400) with an optional prefix.
5. Per-step RBAC: if policy rule baremetal:step:execute:<iface>.<step> exists, it must pass with node.owner/node.lessee in the target (ironic/conductor/steps.py:753).

3.3.5 Execution loop

See 05 § 3.2. State variables: node.<kind>_step (current), driver_internal_info.<kind>_steps (list), <kind>_step_index, <kind>ing_reboot, skip_current_<kind>_step, <kind>ing_polling, agent_cached_<kind>_steps. Results: None → next; *WAIT → wait event; anything else → error "step returned invalid value". Reserved steps (ironic/conductor/steps.py:87): power_on/power_off/reboot call node_power_action; hold → hold event and exit; wait → wait, sleep min(args.seconds, [conductor]/max_conductor_wait_step_seconds=30), resume. Next index after an asynchronous step: update_next_step_index (ironic/conductor/utils.py:1496) skips the current step unless skip_current_<kind>_step is false (agent was rebooting).

3.3.6 Abort (ironic/conductor/manager.py:1553)

3.4 Conductor hash ring

• Input: online conductors (updated_at newer than heartbeat_timeout) with their enabled hardware types and group; key "<group>:<hw_type>" (ironic/db/sqlalchemy/api.py:1573).
• tooz.hashring.HashRing(hosts, partitions=2**hash_partition_exponent, hash_function=hash_ring_algorithm); owner = get_nodes(node.uuid.encode('utf-8')) (ironic/conductor/rpcapi.py:247). Cached 15 s, rebuilt on miss.
• Vectors: hash-ring-vectors_stable-2026.2.json. Reproducing tooz bit-for-bit is required only if Python and Go conductors share a deployment during migration; otherwise any consistent hashing with the same inputs works, but the API and all conductors must agree.
• In Kubernetes the conductor identity should be the StatefulSet pod name (stable), not the pod IP.

3.5 Agent token

• Generated once per ramdisk session: secrets.token_urlsafe() stored in driver_internal_info.agent_secret_token (ironic/conductor/utils.py:1509); pre-generated for virtual media (embedded in the ISO kernel parameters, agent_secret_token_pregenerated).
• Returned only by the first lookup after generation; later lookups get "******" (ironic/conductor/manager.py:4013).
• Required on heartbeat (ironic/conductor/utils.py:1543, compared with ==: a rewrite should use a constant-time comparison) and sent by the conductor on every agent command (agent_token query parameter, ironic/drivers/modules/agent_client.py:200).
• Wiped with agent_url when the node powers off or the flow ends (ironic/conductor/utils.py:655).

3.6 Microversion gating

Two layers to reproduce: route availability (version_probe in routes_*.json: status 404/405/406 below the first version, faults such as "The API version does not allow allocations", "Request not acceptable.") and field/parameter availability (VERSIONED_FIELDS, allow_* functions in version-gates_*.json). The status used for "too old" is inconsistent across resources (404 for GET, 405 for writes of allocations/deploy templates/runbooks/inspection rules, 406 for verbs and fields) and must be copied per route from the probe data, not unified.

4. What the clients expect

Statuses that must not change: 202 for heartbeat; 409 for locked nodes (retried by 3 clients); 404 (not 403) for foreign nodes; 406 with version headers for unsupported versions; 400 InvalidStateRequested vs 409 NodeLocked on refused transitions.

5. Database strategy

• Reuse the migrated schema (schema-mariadb-migrated_stable-2026.2.sql); models and migrations agree except one index name (schema-diff). JSON columns (driver_info, instance_info, driver_internal_info, properties, extra, step lists) are TEXT with JSON content (JsonEncodedDict): keep them opaque JSON in Go.
• Concurrency: no optimistic versioning; the only node mutual exclusion is nodes.reservation (§ 3.2). Allocations use compare-and-swap on conductor_affinity. Writes by the API (chassis, deploy templates, runbooks, inspection rules) are plain.
• Galera: the reservation UPDATE … WHERE reservation IS NULL is a single-row write; Galera certification conflicts surface as deadlocks, retried by oslo.db retry_on_deadlock (ironic/db/sqlalchemy/api.py:780): the Go layer must retry deadlocks too.
• Secrets at rest: driver_info (BMC passwords) and driver_internal_info.agent_secret_token are stored in clear in the database; only API output is masked (ironic/api/controllers/v1/node.py:1905).
• Startup data: none (no standard rows); conductors register themselves.
• Online data migrations: ironic-dbsync online_data_migrations exists for upgrades from older releases; a new OpenKeel deployment starts at the head revision.

6. Authentication, policies, configuration and CLI

• Tokens: keystonemiddleware auth_token (validation and caching against Keystone) or HTTP basic; the API also needs the service catalog of the token (X-Service-Catalog) for some outbound calls with the user's token.
• Policies: 133 oslo.policy rules with the oslo rule language (role:, project_id:%(node.owner)s, system_scope:all, rule:), deprecated rules and enforce_new_defaults/enforce_scope; targets built per request with node.owner, node.lessee, allocation.owner, runbook.owner (ironic/api/controllers/v1/utils.py:1751). Per-field PATCH policies (ironic/api/controllers/v1/node.py:3147) and per-step policies (baremetal:step:execute:*) are dynamic and must be implemented.
• Configuration: oslo.config sample in _rewrite/config_stable-2026.2.conf. The Go service should accept the same option names in YAML for the options it implements; § 10 lists the ones OpenKeel needs.
• CLI: ironic-dbsync (upgrade, online migrations), ironic-status upgrade check, ironic-manage — replace by an init Job (migrate) in the Helm chart.

7. Conformance strategy

• RBAC matrix (replayable): exchanges-rbac + fixture-seeds-rbac = 613 cases from ironic/tests/unit/api/test_rbac_legacy.yaml, ironic/tests/unit/api/test_rbac_system_scoped.yaml, ironic/tests/unit/api/test_rbac_project_scoped.yaml and ironic/tests/unit/api/test_acl_basic.yaml, run by ironic/tests/unit/api/test_acl.py:33. Each case: database rows (load by natural key: uuid, name; re-point node_id, portgroup_id foreign keys), configuration overrides, persona headers (X-Roles, X-Project-Id, OpenStack-System-Scope, as keystonemiddleware would set them), request, expected status and body. Proof: replayed from the seeds alone against the original (adapters/ironic.py replay_seeds): 613/613 statuses identical, 590 bodies identical, 23 identical status with differing bodies (timestamps and generated links) (replay-report_stable-2026.2.json). The conductor is absent in this suite (get_topic_for raises TemporaryFailure): write operations end in 503 after the policy check, which is exactly what the matrix tests.
• API examples: exchanges-api-tests (2 215 exchanges from 2 021 tests of ironic/tests/unit/api/controllers/v1/ and ironic/tests/unit/api/test_root.py, ironic/tests/unit/api/test_hooks.py, ironic/tests/unit/api/test_middleware.py) with inferred response schemas per microversion and observed errors. The RPC layer is mocked in these tests, so they show API validation and serialisation, not conductor effects; they are differential-testing material, not replayable cases.
• State machine: 630 + 9 vectors (§ 3.1). Step ordering: step-order-vectors. Hash ring: hash-ring-vectors.
• Route availability per microversion: version_probe of every route (§ 3.6).
• IPA contract: the agent's own tests mock Ironic; the contract is in § 4 and in ironic/tests/unit/api/controllers/v1/test_ramdisk.py (exchanges recorded in exchanges-api-tests). A live conformance run needs a VM booting the real IPA ramdisk (sushy-tools Redfish emulator + libvirt), which is the recommended end-to-end gate (§ 8).
• Original fixtures: ironic/tests/unit/api/base.py:37 (BaseApiTest, Pecan test app, noauth), ironic/tests/unit/db/base.py:34 (SQLite schema from the models, cached and re-applied per test), ironic/tests/unit/api/test_acl.py:33 (keystone auth with stubbed process_request, no conductor), ironic/tests/base.py:144 (config fixture: enabled interfaces fake…, policy file with ipa_lookup/ipa_heartbeat/ipa_continue_inspection = @).

8. Gaps found while building the pack

• No route table in the code: routes were discovered (probe); a route reachable only through a code path the candidates do not name would be missed. Candidates = api-ref ∪ policy operations ∪ all verbs on those paths.
• api-ref typos and the indicator path mismatch (01 § 3.1); DELETE /v1/inspection_rules answers 500 in the probe.
• rewrite_pack.py --schemas-package ironic.api.schemas.v1 fails with a circular import (schemas are lower-case attributes and import each other): replaced by tools/ironic_api.py; --version-history does not parse Ironic's 1.x (Release) headings: same tool.
• ipmitool interfaces: get_properties() could not be introspected without the binary path (empty property lists in drivers_*.json for power:ipmitool/management:ipmitool); the property list is in ironic/drivers/modules/ipmitool.py:350.
• In-band step lists (erase_devices, write_image internals) come from IPA at run time and are not in drivers_*.json.
• No conformance data for the conductor flows (deploy/clean/inspect end-to-end): the unit tests mock drivers and the agent. Recommended: Metal3-style e2e with sushy-tools + virtual BMC + real IPA ramdisk, recording conductor ↔ IPA and conductor ↔ BMC exchanges.
• Release note/commit CVE id mismatch for the maintenance policy fix (06 § 22).

9. Non-functional requirements and Kubernetes design notes

1. Conductor identity: StatefulSet ordinal names (ironic-conductor-0…) as host; under JSON-RPC the ring entry is host:port, so the conductor must be reachable by that name (headless Service).
2. RPC transport: JSON-RPC over HTTPS with Keystone or mTLS between API and conductor removes RabbitMQ; the API → conductor addressing is by hash ring; cast = JSON-RPC notification (no id, 204).
3. Local state per conductor: /tftpboot, /httpboot (iPXE scripts, ISOs, configdrives), image cache (master copies + per-node links). It is rebuilt on takeover (_do_takeover), so a per-pod PVC is an optimisation, not a correctness requirement; but the HTTP server that IPA and the BMC download from must serve the owning conductor's files: one HTTP sidecar per conductor pod, URL in [deploy]/http_url per pod (or external_http_url per node).
4. Network reachability: conductor → BMC (HTTPS/IPMI UDP 623), conductor → IPA callback_url (port 9999 on the provisioning network), IPA → API (lookup/heartbeat) and IPA → conductor HTTP (images), BMC → conductor HTTP (virtual media ISO). With Redfish virtual media no DHCP/TFTP is needed if the ramdisk gets its IP from the ISO's network config or the provisioning network's DHCP.
5. Timers: keep the defaults of § 05 (heartbeat 10 s / timeout 60 s, callback timeouts 1800 s, power sync 120 s).
6. Workers: bounded pool (300 + 5 % reserved) with 503 on saturation; heartbeats must not consume the reserved pool.
7. Throttling: IPMI one command per 5 s per BMC; Redfish sessions cached per BMC (ironic/drivers/modules/redfish/utils.py:285).
8. Secrets: BMC credentials from Kubernetes Secrets rather than driver_info in clear (a deliberate divergence: keep driver_info.redfish_password accepted and masked in the API, but store it encrypted or as a Secret reference), agent token compared in constant time, step arguments masked in logs (change 16 of 06), BMC URL components validated (change 21).

10. Relation to backend:ironic-baremetal.md

11. Rewrite checklist

• [ ] Router from routes-list_*.json; return 405 on dispatch artifacts; keep per-route "too old" statuses from version_probe.
• [ ] Microversion middleware: default 1.1, latest, 406 + min/max headers on every response.
• [ ] Error envelope {"error_message": "<json string>"} with faultcode/faultstring/debuginfo.
• [ ] VERSIONED_FIELDS filtering on output, 406 on input.
• [ ] oslo.policy-compatible engine, 133 default rules, owner/lessee targets, per-field PATCH policies (incl. /maintenance), per-step policies, 404 concealment.
• [ ] State machine from state-machine_*.json; pass all 639 vectors.
• [ ] TaskManager lock semantics on nodes.reservation; deadlock retries; startup clearing of own reservations.
• [ ] Step framework: ordering keys, overrides, deny lists, composition per flow, validation messages, reserved steps, abort/abort_after.
• [ ] Hash ring over live conductors per group/hardware type; takeover periodic; orphan handling.
• [ ] JSON-RPC server/client (or keep oslo.messaging-compatible frames if Python conductors must coexist).
• [ ] Ramdisk API: lookup, heartbeat (202/409), continue_inspection; token generate-once, constant-time compare; config document.
• [ ] Agent client: command contract, timeouts, busy (409) handling, last-command recovery.
• [ ] Redfish (power, boot override, virtual media insert/eject, HTTP boot, secure boot, task monitors) and IPMI (ipmitool or Go IPMI with 5 s throttle).
• [ ] Image pipeline: Glance metadata/download, cache, HTTP publishing per conductor, Swift temp URLs optional; ISO build for virtual media.
• [ ] Network interfaces flat and neutron (provisioning/cleaning networks, tenant binding); DHCP provider choice.
• [ ] Periodic tasks with the same defaults; power sync with Nova power-update.
• [ ] Inspection: hooks (ramdisk-error, validate-interfaces, ports, architecture + chosen extras), inspection rules engine (15 operators, 18 actions).
• [ ] Pass the RBAC matrix (613), then an e2e with sushy-tools + real IPA, then Nova's ironic driver flows (§ 4).

Source: reverse-spec-ironic/07-rewrite-pack_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.