Processing outside the API — neutron — branch stable/2026.2
Branch
stable/2026.2@eef1a0f7ee36— repository https://opendev.org/openstack/neutron — analysed on 2026-10-28
Scope: the ML2/OVN reference profile of 01-rest-api_stable-2026.2.md. This file covers the server-side processes (API workers, RPC workers, OVN maintenance worker, periodic workers). The OVN metadata agent is in § 5. The OVS, Linux bridge, DHCP, L3 and metering agents are out of scope.
1. Processes and where the processing runs
| Process | Entry point | What it runs outside HTTP |
|---|---|---|
API workers (uWSGI neutron-api, or neutron-server) | neutron/wsgi/api.py, neutron/cmd/server/ | one OVN NB IDL with events and one OVN SB IDL with events per worker; events are dispatched by the hash ring (§ 2); the hash-ring heartbeat thread |
neutron-rpc-server | neutron.cmd.server:main_rpc | AMQP consumers of topics q-plugin, q-reports-plugin, q-server-resource-versions (neutron/plugins/ml2/plugin.py:452-469); OVN IDLs (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:351-354). Can be 0 workers with OVN: [DEFAULT]/rpc_workers = 0 disables all agent RPC (neutron/plugins/ml2/plugin.py:436-445) |
neutron-ovn-maintenance-worker | neutron.cmd.server:main_ovn_maintenance → neutron/server/ovn_maintenance.py:24 | start-up DB sync NB and SB (§ 4), then the maintenance thread (§ 3); holds the OVSDB lock ovn_db_inconsistencies_periodics |
neutron-ovn-metadata-agent (deprecated) or neutron-ovn-agent with the metadata extension, on every compute node | neutron/cmd/agents/ovn_metadata.py, neutron/agent/ovn/extensions/metadata.py:93 | SB OVSDB events, heartbeat, metadata proxy (§ 5) |
neutron-periodic-workers | neutron.cmd.server:main_periodic → neutron/service.py:317 | the plugins' other workers (quota cleanup…), grouped in one AllServicesNeutronWorker; the OVN maintenance worker is excluded (neutron/service.py:290-314) |
The OVN driver connects its IDLs in every worker class listed above after the fork (post_fork_initialize, neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:399-457). API workers also register in the hash ring (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:356-397); the maintenance worker runs the DB sync and starts the maintenance thread (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:438-474).
2. Summary of triggers
| Process | Trigger | Type | Purpose | Reads | Writes / emits | Defined at |
|---|---|---|---|---|---|---|
| API worker | LogicalSwitchPortCreateEvent | OVSDB event NB Logical_Switch_Port CREATE | report a port that is already up when created | LSP up, enabled | port status, provisioning blocks, Nova event | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:618 |
| API worker | LogicalSwitchPortUpdateUpEvent | OVSDB event NB LSP UPDATE, up or enabled → true | the VM's interface came up | LSP | ports.status ACTIVE via provisioning, network-vif-plugged to Nova, NAT external_mac | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:639 |
| API worker | LogicalSwitchPortUpdateDownEvent | OVSDB event NB LSP UPDATE, up or enabled → false | the interface went down | LSP | ports.status DOWN, network-vif-unplugged | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:673 |
| API worker | LogicalSwitchPortUpdateLogicalRouterPortEvent | OVSDB event NB LSP UPDATE of router ports | propagate up to the sibling LRP status | LSP | router port status | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:708 |
| API worker | FIPAddDeleteEvent | OVSDB event NB NAT CREATE/DELETE, or UPDATE of external_mac | flush stale ARP entries of a floating IP | NAT | SB MAC_Binding rows deleted (ovsdb-client transact) | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:835 |
| API worker | FIPAddExternalMacEvent | OVSDB event NB NAT CREATE | set external_mac of a distributed FIP when the port is already up | NAT, LSP | NAT external_mac | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:873 |
| API worker | HAChassisGroupRouterEvent | OVSDB event NB HA_Chassis_Group UPDATE | re-point networks' HA groups when a router's gateway chassis list changes | HCG | network HCGs | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:916 |
| API worker | ChassisEvent | OVSDB event SB Chassis CREATE/DELETE, UPDATE of bridge mappings, gateway flag or AZs | keep segment-host mappings and gateway scheduling right | Chassis other_config | segmenthostmappings, gateway LRP rescheduling, HA groups | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:49 |
| API worker | PortBindingChassisEvent | OVSDB event SB Port_Binding UPDATE of a chassisredirect port | record which host carries a router gateway | Port_Binding chassis | gateway port binding:host_id | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:559 |
| API worker | PortBindingChassisUpdateEvent | OVSDB event SB Port_Binding UPDATE (chassis change while up, live migration) | finish a live migration seen by OVN | Port_Binding, LSP | port binding, NB nb_cfg bump | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:274 |
| API worker | PortBindingUpdateVirtualPortsEvent | OVSDB event SB Port_Binding UPDATE/DELETE of virtual ports | track the host of a VRRP/virtual IP owner | Port_Binding | virtual port binding:host_id | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:770 |
| API worker | ChassisAgentWriteEvent, ChassisOVNAgentWriteEvent, ChassisAgentDownEvent, ChassisAgentDeleteEvent, ChassisAgentTypeChangeEvent | OVSDB events SB Chassis_Private (create, nb_cfg update, delete), SB_Global.external_ids:delete_agent, Chassis type change | maintain the in-memory agent cache that answers GET /agents | Chassis, Chassis_Private | AgentCache (memory only) | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:387-557 |
| API worker | ChassisBandwidthConfigEvent | OVSDB event SB Chassis CREATE/UPDATE | report the chassis' bandwidth inventories to Placement | Chassis other_config | Placement REST | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/extensions/placement.py:110 |
| API worker | touch_hash_ring_node | timer, every HASH_RING_TOUCH_INTERVAL = 15 s | heartbeat of this worker in the hash ring | ovn_hash_ring | UPDATE ovn_hash_ring.updated_at | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1611 |
| maintenance | check_for_inconsistencies | SQL polling every 300 s, at start-up, OVSDB lock | re-apply to OVN whatever failed after a commit | ovn_revision_numbers ⋈ standardattributes | OVN NB, ovn_revision_numbers | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:488 |
| maintenance | 27 one-shot and daily tasks | timer 5 s until done (NeverAgain), or 86 400 s | data migrations and NB settings | NB, DB | NB, DB | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:610-1534 |
| maintenance | OvnNbSynchronizer.do_sync, OvnSbSynchronizer.do_sync | once at start-up | compare the Neutron DB with NB/SB; log or repair | all | NB, segmenthostmappings | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:109, 1835 |
| any | AFTER_CREATE/AFTER_UPDATE/AFTER_DELETE callbacks of routers, router interfaces, floating IPs, port forwardings, security groups, segments | in-process registry callbacks | write the L3 and SG objects to OVN after the DB commit | DB | NB | neutron/services/ovn_l3/service_providers/ovn.py:46-252, neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:277-335 |
| any | PROVISIONING_COMPLETE | in-process callback | set the port ACTIVE once every provisioning entity is done | provisioningblocks | ports.status | neutron/plugins/ml2/plugin.py:370 |
| any | Nova notifier batch | timer [DEFAULT]/send_events_interval (2 s) | send queued port events to Nova | queue | POST /os-server-external-events | neutron/notifiers/nova.py:79-80, 277 |
| RPC worker | q-plugin, q-reports-plugin, q-server-resource-versions | AMQP consumers | serve non-OVN agents (SR-IOV, DHCP…); OVN agents do not use RPC | — | — | neutron/plugins/ml2/plugin.py:452-469 |
3. OVSDB events and the hash ring
How events arrive. Each worker keeps an in-memory replica of NB and SB through ovsdbapp's OvsdbIdl (monitor_cond, then pushed update2/update3 messages); these are **notifications pushed by ovsdb-server, not polling**. The NB IDL of API workers registers all tables (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1146-1150); the SB IDL registers Chassis_Private, Chassis, Encap, Port_Binding, Datapath_Binding and SB_Global.external_ids only (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1167-1175). Chassis, Port_Binding events are only watched after the first connection dump, to avoid replaying every row at start-up (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1177-1189).
Who handles an event. Every API worker receives every update, but only one handles it: the hash ring maps the row UUID to one live node of group mechanism_driver (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1079-1117). Nodes are rows of table ovn_hash_ring (one per API worker, keyed by a node UUID, with hostname, group_name, updated_at); a node is live when updated_at is younger than HASH_RING_NODES_TIMEOUT = 120 s (neutron/common/ovn/constants.py:327). The handling worker also touches its own node when its last heartbeat is older than 15 s. Events flagged GLOBAL (WaitForChassisNbCfgEvent) are delivered to all workers.
Concurrency and failure.
- At start-up the first API worker of a host deletes the host's older nodes and each worker adds its own (
neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:385-397); onSIGTERMthe node is removed (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:337-344). - An empty ring logs
HashRing is empty, error: …and drops the event (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1085-1087). A worker whose heartbeat stalls keeps receiving events assigned to it until the timeout, then the ring rebalances: events in that window can be lost; the maintenance task and the start-up sync compensate. - The heartbeat thread logs
CRITICAL The number of nodes in the Hash Ring (%d) is higher than the number of API workerswhen a host has more nodes than workers (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1623-1634). - Exceptions inside an event are logged and swallowed (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:1116-1117). cleanup_old_hash_ring_nodes(daily) deletes nodes untouched for 5 days (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1074-1083).
3.1 Port up / down (set_port_status_up, set_port_status_down)
- Trigger: NB
Logical_Switch_Port.up(written byovn-northdfrom SBPort_Binding.up, itself written byovn-controllerwhen it claims the interface) orenabledchanges (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:653-705). - Purpose: tell Neutron and Nova that the VM's interface is wired.
- Flow (up) (
neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:1453-1507): 1. for a distributed floating IP on this port, set the NATexternal_mac; 2. complete theL2provisioning entity of the port, which firesPROVISIONING_COMPLETEwhen no other entity is pending, and ML2 setsports.status = ACTIVE, which records the Nova eventnetwork-vif-plugged(neutron/plugins/ml2/plugin.py:370); 3. router ports are set ACTIVE directly; compute ports whose status did not change triggernotify_port_active_direct; 4. write the host to the LSPexternal_ids; 5. forexternalports, re-sync the network's HA chassis group. - Flow (down) (
neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:1509-1538): clear the NATexternal_mac, re-insert the provisioning block, setDOWN, sendnetwork-vif-unpluggedto Nova. - Idempotency: repeated events are harmless; a port deleted meanwhile is ignored (
PortNotFound,DBReferenceError). - Cost: one SQL transaction per event plus one Nova event (batched).
3.2 Agent liveness (GET /agents)
OVN controllers have no RPC heartbeat. GET /agents first calls ping_all_chassis, which bumps NB_Global.nb_cfg at most every agent_down_time / 2 seconds (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:1620-1638, 1715-1720). ovn-northd copies it to SB, every ovn-controller echoes it into its Chassis_Private.nb_cfg, and the ChassisAgentWriteEvent updates the cache. An agent is alive when its nb_cfg is at most one generation behind NB_Global.nb_cfg, or when its last update is younger than [DEFAULT]/agent_down_time (default 75 s) (neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:96-110). The cache is per worker and in memory: two API workers can answer differently right after a change [inferred].
4. OVN maintenance worker
4.1 check_for_inconsistencies — the consistency loop
- Trigger:
futuristperiodic everyDB_CONSISTENCY_CHECK_INTERVAL= 300 s, first run at start-up (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:488,neutron/common/ovn/constants.py:297); not configurable. - Purpose: OVN is written after the Neutron DB commit, so any failure (OVSDB down, timeout, bug) leaves OVN behind. Each resource carries a revision:
standardattributes.revision_numberin Neutron,external_ids:neutron:revision_numberin OVN, and the last revision written to OVN in tableovn_revision_numbers. The loop re-applies every resource whose two numbers differ. - Data read (
neutron/db/ovn_revision_numbers_db.py:224-263):sql -- create/update inconsistencies, older than 60 s, ordered by type SELECT r.* FROM ovn_revision_numbers r JOIN standardattributes s ON r.standard_attr_id = s.id WHERE s.created_at < now() - INTERVAL 60 SECOND AND r.revision_number != s.revision_number ORDER BY CASE r.resource_type WHEN 'security_groups' THEN 1 … END; -- deletions not applied to OVN: the FK was set to NULL by the delete SELECT * FROM ovn_revision_numbers WHERE standard_attr_id IS NULL ORDER BY …; - Flow (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:318-549): for each row, read the Neutron object and the OVN object (_resources_func_map,neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:240-297); if the OVN object is missing →ovn_create; if the OVN revision differs →ovn_update; if equal → only bumpovn_revision_numbers. Subnets and segments have their own fixers. Deletions callovn_delete, or drop the revision row when OVN has nothing left. Types: networks, ports, floating IPs (with port forwardings), routers, router ports, security groups, SG rules, address groups, subnets, segments. - Revision guard: OVN writes include
CheckRevisionNumberCommand, which refuses to overwrite an OVN row with an older revision (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py:1103);bump_revisionnever lowersovn_revision_numbers(neutron/db/ovn_revision_numbers_db.py:183-221). A stale update raisesRevisionConflict"OVN revision number for … is higher than the given resource. Skipping update" (neutron/common/ovn/exceptions.py:21). - Concurrency: only the process that holds the OVSDB lock
ovn_db_inconsistencies_periodicson the NB connection runs the tasks (has_lock_periodic,neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:64-89; lock nameneutron/common/ovn/constants.py:300; set atneutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:414-415). With several maintenance workers, OVSDB elects one; failover is automatic when its connection drops. - Error behaviour: each resource is fixed in its own
try; errors are logged (Maintenance task: Failed to fix resource …) and retried at the next run (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:523-549). A resource that always fails is retried forever. - Cost: two SQL queries per run plus one NB transaction per inconsistent resource; with a healthy OVN, nothing else.
4.2 One-shot and daily tasks
All use has_lock_periodic and the same thread pool of one worker (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:144-172). "One-shot" tasks run every MAINTENANCE_ONE_RUN_TASK_SPACING = 5 s while the lock is not held, up to MAINTENANCE_TASK_RETRY_LIMIT = 100 tries (neutron/common/ovn/constants.py:298-299), and stop with NeverAgain once they have done their job. They are upgrade data migrations of OVN NB content; a rewrite that creates NB rows in their final form does not need them, but must produce the same final form:
| Task | What it ensures in NB | Defined at |
|---|---|---|
update_lrouter_ports_ext_ids_name_prefix | LRP external_ids names carry the neutron- prefix | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:610 |
check_global_dhcp_opts | [ovn]/ovn_dhcp4_global_options, ovn_dhcp6_global_options applied to every DHCP_Options | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:648 |
check_for_igmp_snoop_support | mcast_snoop/mcast_flood_unregistered on every switch | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:679 |
check_localnet_port_has_learn_fdb | localnet ports learn FDB when [ovn]/localnet_learn_fdb | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:717 |
add_provnet_ext_id_to_localnet_ports | localnet ports carry the segment ID | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:749 |
check_redirect_type_router_gateway_ports | gateway LRPs use redirect-type=bridged on VLAN when [ovn]/enable_distributed_floating_ip | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:798 |
check_provider_distributed_ports | metadata/distributed ports on provider networks have the right options | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:872 |
check_fdb_aging_settings, update_mac_aging_settings | fdb_age_threshold on switches, mac_binding_age_threshold on routers | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:915, 957 |
check_baremetal_ports_dhcp_options | PXE options of baremetal ports | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:974 |
check_router_default_route_empty_dst_ip | default routes have a valid destination | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1025 |
check_fair_meter_consistency | the ACL log meter matches [network_log] rate/burst | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1058 |
configure_nb_global (daily) | [ovn_nb_global] options copied into NB_Global.options | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1088 |
update_router_distributed_flag (daily) | router_extra_attributes.distributed mirrors enable_distributed_floating_ip | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1108 |
set_network_type_and_physnet | switches carry the network type and physnet in external_ids | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1135 |
check_network_broadcast_arps_to_all_routers | external switches broadcast ARPs per [ovn]/broadcast_arps_to_all_routers | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1175 |
set_fip_distributed_flag | NAT rows record whether they are distributed | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1212 |
set_ovn_owned_dns_option, remove_stale_dns_records | DNS rows owned by OVN and free of deleted ports | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1237, 1263 |
update_ha_failover | HA chassis groups of routers have the failover options | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1288 |
check_dhcp_options_consistency | each subnet's DHCP_Options matches the DB | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1323 |
update_security_group_with_address_group | ACLs of rules using address groups reference the address sets | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1364 |
update_virtual_port_parent_hostname | virtual ports know their parent's host | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1412 |
migrate_lrp_gateway_chassis_to_ha_chassis_group | gateway LRPs use HA_Chassis_Group instead of Gateway_Chassis | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1444 |
update_neutron_pg_drop_priority | the drop ACLs of neutron_pg_drop have the expected priority | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1497 |
check_pvlan_plugin_status | PVLAN port groups exist only when the plugin is loaded | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1534 |
cleanup_old_hash_ring_nodes (daily) | hash ring rows untouched for 5 days deleted | neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1074 |
Tasks marked rerun_on_schema_updates run again when the NB schema version changes (OVNNBDBReconnectionEvent, neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:181-226). Other service plugins can add periodics through ovn_maintenance_periodics() (neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:466-472).
4.3 Start-up DB sync ([ovn]/neutron_sync_mode)
- Trigger: once, in the maintenance worker before the maintenance thread starts (
neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:438-452); also run by hand withneutron-ovn-db-sync-util(neutron/cmd/ovn/neutron_ovn_db_sync_util.py). - Modes (
neutron/conf/plugins/ml2/drivers/ovn/ovn_conf.py:103-120):off;log(default) — report differences as warnings;repair— create missing NB objects and delete NB objects unknown to Neutron;migrate— used by the OVS→OVN migration. - NB order (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:109-131): port groups, networks/ports/DHCP options (with metadata ports), DNS records, ACLs, PVLAN, routers and router ports (static routes, SNAT, FIPs, port forwardings), port QoS, FIP QoS, FIP DNAT rules, distributed NAT. - SB (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1835-1901): rebuildsegmenthostmappingsfrom each chassis' bridge mappings. - Cost: reads every network, port, router, FIP and SG of the cloud and the whole NB replica; minutes on large clouds; runs on every restart of the maintenance worker.
5. OVN metadata agent
5.1 Role and process
- Purpose: give every VM the Nova metadata service at
http://169.254.169.254(andhttp://[fe80::a9fe:a9fe]for IPv6) without a router, on every compute node. For each network that has a VIF bound to the local chassis, the agent creates a namespaceovnmeta-<network id>wired to OVN through the network's metadata port (device_owner: network:distributed, LSPtype: localport, created by the server, 01 § 4.1), runs onehaproxyin it, and proxies the requests tonova-metadata-apiafter identifying the instance. - Processes:
neutron-ovn-metadata-agent(neutron/cmd/agents/ovn_metadata.py, classMetadataAgent,neutron/agent/ovn/metadata/agent.py:335). It is deprecated: the start-up log says it "has been deprecated in 2025.2 and will be removed in 2026.2" (neutron/agent/ovn/metadata/agent.py:475-478), but it still ships in this branch. Its replacement isneutron-ovn-agentwith themetadataextension, which inherits the same class (MetadataExtension(extension_manager.OVNAgentExtension, metadata_agent.MetadataAgent),neutron/agent/ovn/extensions/metadata.py:93-96) and therefore has the same behaviour, plus the SB/OVS connections shared with the other extensions (neutron/agent/ovn/extensions/metadata.py:194-212). - Connections (no AMQP, no SQL): the local OVS database (
Open_vSwitch,Bridge,Port,Interface,neutron/agent/ovn/metadata/ovsdb.py:66-80) and the OVN SB database (Encap,Port_Binding,Datapath_Binding,SB_Global,Chassis,Chassis_Private,neutron/agent/ovn/metadata/agent.py:444-455), plus HTTP to Nova. - Identity: the chassis name is
Open_vSwitch.external_ids:system-id, the bridgeexternal_ids:ovn-bridge(defaultbr-int) (neutron/agent/ovn/metadata/agent.py:491-514); the agent ID isuuid5(chassis_id, 'metadata_agent'), written inChassis_Private.external_ids:neutron:ovn-metadata-id(neutron/agent/ovn/metadata/agent.py:482-489).
5.2 Triggers
| Trigger | Type | Purpose | Reads | Writes / emits | Defined at |
|---|---|---|---|---|---|
start-up sync(provision=False) then sync() on reconnection | once / OVSDB Chassis_Private CREATE of the own chassis after the first dump | remove namespaces of networks without local VIFs; provision the others | system namespaces, SB Port_Binding on this chassis | namespaces, veth, OVS ports, haproxy | neutron/agent/ovn/metadata/agent.py:525-557, 268-295 |
PortBindingCreateWithChassis | OVSDB SB Port_Binding CREATE already bound to this chassis | provision the port's network | Port_Binding | provision_datapath | neutron/agent/ovn/metadata/agent.py:117-124 |
PortBindingUpdatedEvent | OVSDB SB Port_Binding UPDATE: chassis set to / removed from this chassis, live-migration additional_chassis added/removed, metadata localport neutron:cidrs changed | (un)provision the network | Port_Binding | provision_datapath (which tears down when nothing is left) | neutron/agent/ovn/metadata/agent.py:127-244 |
PortBindingDeletedEvent | OVSDB SB Port_Binding DELETE of a row bound to this chassis | re-evaluate the network | Port_Binding | provision_datapath / teardown | neutron/agent/ovn/metadata/agent.py:247-265 |
SbGlobalUpdateEvent | OVSDB SB SB_Global UPDATE (nb_cfg bumped by the server's liveness ping) | heartbeat: prove the agent is alive | SB_Global.nb_cfg | Chassis_Private.external_ids:neutron:ovn-metadata-sb-cfg = <nb_cfg> after a random delay of 0 to max(min(agent_down_time // 3, 10), 3) seconds | neutron/agent/ovn/metadata/agent.py:298-332 |
| HTTP request from a VM | haproxy → Unix socket [DEFAULT]/metadata_proxy_socket ($state_path/metadata_proxy) | serve metadata | SB Port_Binding | HTTP to Nova | neutron/agent/ovn/metadata/server_socket.py:27-94 |
Every event handler runs under one global re-entrant lock shared with sync, so provisioning is serialised per agent (neutron/agent/ovn/metadata/agent.py:45, 61-67, 100-114). A ConfigException (the configured OVN bridge does not exist) triggers a full resync (neutron/agent/ovn/metadata/agent.py:107-114, 428-437).
5.3 provision_datapath — wiring a network
(neutron/agent/ovn/metadata/agent.py:690-866)
- Find the network's metadata port: the SB
Port_Bindingof typelocalportwithexternal_ids:neutron:device_owner = network:distributedon the datapath (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:1095-1106). No port, or no MAC → tear down and stop (the namespace disappears until the server creates the port). - Collect the IPv4 addresses of the VIFs of this network bound to this chassis (types
"",external,localport); none and no IPv6 → tear down (neutron/agent/ovn/metadata/agent.py:735-750). - Create the veth pair
tap<first 10 chars of the datapath name>0/…1with the inner end inovnmeta-<network id>, set the metadata port's MAC and the network MTU (neutron:mtufrom the port'sexternal_ids), bring both ends up (neutron/agent/ovn/metadata/agent.py:776-804). - Addresses inside the namespace:
169.254.169.254/32, plusfe80::a9fe:a9fe/128and the link-local address derived from the MAC when IPv6 is used, plus the metadata port's IP in each subnet that has a local VIF; addresses no longer needed are removed (neutron/agent/ovn/metadata/agent.py:645-688,806-823). - Plug the outer end into the OVN bridge with
Interface.external_ids:iface-id = <metadata port id>(removing it from any other bridge), which makesovn-controllerbind the localport (neutron/agent/ovn/metadata/agent.py:828-850); on DPDK (datapath_type = netdev) add an iptablesCHECKSUM --checksum-fillrule in the namespace (neutron/agent/ovn/metadata/agent.py:600-617). - Start (or keep) a monitored
haproxyin the namespace, listening on169.254.169.254:80(and[fe80::a9fe:a9fe]:80on the veth) and forwarding to the Unix socket with the headerX-OVN-Network-ID: <network id>(neutron/agent/ovn/metadata/agent.py:857-866,neutron/agent/ovn/metadata/driver.py:18-31,neutron/agent/metadata/driver_base.py:42-46); haproxy runs as[DEFAULT]/metadata_proxy_user/group, its configuration lives in$state_path/ovn-metadata-proxy/and the process monitor respawns it (neutron/agent/metadata/driver_base.py:259-335).
Teardown (neutron/agent/ovn/metadata/agent.py:575-598): stop haproxy, delete the OVS port and the veth, delete the namespace.
Privileges: namespace, veth, address and iptables operations go through ip_lib / iptables_manager, which use oslo.privsep (neutron/privileged/) and rootwrap; haproxy is started as root and drops to the configured user [inferred].
Cost: one SB lookup and a few netlink calls per event; sync lists every namespace and every port of the chassis.
5.4 Serving a metadata request
(neutron/agent/metadata/proxy_base.py:121-251, neutron/agent/ovn/metadata/server_socket.py:43-63)
- haproxy forwards the VM's request with
X-Forwarded-For: <VM IP>andX-OVN-Network-ID. - The handler looks for the SB
Port_Bindingof that network whosemaccolumn contains the IP (or, for an IPv6 link-local source, the MAC derived from it) and that is bound to a chassis (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:1116-1150). It readsneutron:device_id(instance UUID) andneutron:project_idfrom itsexternal_ids. **This lists everyPort_Bindingrow of the local SB replica for each request**: the cost grows with the cloud size. - Exactly one port → forward to
[DEFAULT]/nova_metadata_protocol://nova_metadata_host:nova_metadata_port(defaulthttp://127.0.0.1:8775) with the original path and query, headersX-Forwarded-For,X-Instance-ID,X-Tenant-IDandX-Instance-ID-Signature = HMAC-SHA256(metadata_proxy_shared_secret, instance_id)in hex (neutron/common/utils.py:1090-1094), optional client certificate (nova_client_cert,nova_client_priv_key) and CA (auth_ca_cert),timeout=60s. - Responses: Nova 200 → body and content type returned; 403 → 403 with WARNING "…shared secrets do not match"; 500 → 500; 400, 404, 409, 502, 503, 504 → same status without body; connection error → 503 "The remote metadata server is temporarily unavailable. Please try again later."; any other status →
RuntimeError→ 500 "An unknown error has occurred. Please try your request again.". Zero matching ports → 404 (WARNINGNo port found in network %s with IP address %s); several → 404 and ERROR asking to runneutron-ovn-db-sync-util. - The Unix server handles requests in a thread pool of
[DEFAULT]/metadata_workersthreads; with ML2/OVN the option defaults to 0, and requests are then handled one at a time in the server's own thread (socketserver.UnixStreamServer.process_request), so a slow Nova blocks all VMs of the node for up to 60 s per request [inferred from the code; setmetadata_workers> 0 in production]. Listen backlog:metadata_backlog(4096) a listen backlog ofmetadata_backlog(4096) (neutron/agent/metadata/proxy_base.py:44-79,neutron/agent/ovn/metadata/server_socket.py:75-86). The socket mode followsmetadata_proxy_socket_mode(neutron/agent/metadata/proxy_base.py:259-275). No cache (has_cache=False).
5.5 Liveness as seen by GET /agents
The server lists the agent as OVN Metadata agent from Chassis_Private.external_ids:neutron:ovn-metadata-id; it is alive when neutron:ovn-metadata-sb-cfg follows NB_Global.nb_cfg (neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:182-215, 05 § 3.2). The random delay of the heartbeat spreads the SB writes of thousands of nodes.
6. Notifications to Nova and Placement
6.1 Nova external events
- Trigger: port status changes recorded by ML2 (
record_port_status_changed,neutron/notifiers/nova.py:209-246), theBEFORE_RESPONSEcallback of port and floating IP API calls (neutron/notifiers/nova.py:124-154), andnotify_port_active_direct(neutron/notifiers/nova.py:253-269). Only ports whosedevice_ownerstarts withcompute:and only with[DEFAULT]/notify_nova_on_port_status_changes/notify_nova_on_port_data_changes(both defaultTrue). - Events:
network-vif-plugged(DOWN/unknown → ACTIVE, statuscompleted),network-vif-unplugged(ACTIVE → DOWN),network-vif-deleted(port deleted),network-changed(fixed IPs or floating IP changed) (neutron/notifiers/nova.py:43-45,112-121). - Delivery: queued in a
BatchNotifierand sent every[DEFAULT]/send_events_intervalseconds (2) as onePOST /v2.1/os-server-external-events(neutron/notifiers/nova.py:79-80,277-300), with Keystone credentials from[nova]. Errors are logged; events are not re-queued. See 03.
6.2 Placement
ChassisBandwidthConfigEvent(SBChassiscreate/update) readsovn-cms-optionsresource_provider_bandwidths,resource_provider_inventory_defaults,resource_provider_hypervisorsand pushes resource providers, inventories and traits to Placement through theplacementservice plugin's batch notifier (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/extensions/placement.py:110-140,neutron/services/placement_report/plugin.py:58-202). Used by QoS minimum bandwidth.
7. Router gateway scheduling (OVN L3)
OVN routers have no L3 agent. A router's gateway port (the LRP on the external network) must run on one "gateway chassis" at a time, with standby chassis for failover; OVN does the failover itself (BFD between chassis), Neutron only chooses the chassis and their priorities.
7.1 Choosing the chassis when a gateway is added
- Trigger: creation of the gateway LRP in
_create_lrouter_port(router create withexternal_gateway_info,PUT /routers/{id}setting the gateway,add_external_gateways) (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1996-2056). - External network without physical network (Geneve external network): no scheduling by physnet; the router is pinned through the HA chassis group of the router and
Logical_Router.options:chassis(gateway-router mode) (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:2030-2041). - External network on a physnet (flat/VLAN, the usual case):
ScheduleNewGatewayCommandruns inside the same NB transaction (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py:558-591):- Candidates (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1807-1847): SB chassis whoseother_config:ovn-cms-optionscontainsenable-chassis-as-gwand whoseovn-bridge-mappingscontain the physnet; if the router hasavailability_zone_hints, only chassis whoseovn-cms-optionsavailability-zones=list one of them. - Selection by
[ovn]/ovn_l3_scheduler(leastloadedby default, orchance) (neutron/scheduler/l3_ovn_scheduler.py:271-272), at mostMAX_GW_CHASSIS= 5 chassis (neutron/common/ovn/constants.py:333). - Priorities: the selected list is turned into priorities from 32 767 (
HA_CHASSIS_GROUP_HIGHEST_PRIORITY) downwards and written as anHA_Chassis_Groupnamed after the router, referenced by the LRP'sha_chassis_group(neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py:580-591,neutron/common/ovn/constants.py:460-461). The highest-priority live chassis carries the gateway.
- Candidates (
- **
leastloadedalgorithm** (neutron/scheduler/l3_ovn_scheduler.py:165-263), to be reproduced by a rewrite that wants the same placement:bindings = for each candidate chassis: list of (LRP name, priority) it already hosts if the target router already has LRPs on some chassis: anti_affinity = MAX_GW_CHASSIS * number of LRPs of the router # avoid putting two # gateways of one router on one chassis for priority from highest to lowest (one slot per priority, up to 5): load(c) = number of LRPs hosted by c at exactly this priority (+ anti_affinity if c already hosts an LRP of this router) keep the chassis with the minimum load, excluding chassis already chosen tie-break: prefer chassis with MORE gateways at lower priorities (they are standby elsewhere), then hash(sha512("<gateway name>_<priority>")) XOR hash(chassis name) finally reorder the list so that the first chassis of each availability zone comes firstThe final tie-break uses Python'shash()of the chassis name, which is randomised per process (PYTHONHASHSEED) (neutron/scheduler/l3_ovn_scheduler.py:244-257): two API workers can break the same tie differently. A rewrite does not need to reproduce the exact tie-break, only the load balancing. - **
chance**: random shuffle, then the same AZ reordering (neutron/scheduler/l3_ovn_scheduler.py:139-153). - No candidate: WARNING
Gateway %s was not scheduled on any chassis, no candidates are available; the router exists but has no external connectivity until a chassis appears (neutron/scheduler/l3_ovn_scheduler.py:71-74).
7.2 Rescheduling
- Trigger: SB
Chassisevents — a chassis appears, disappears, changes its bridge mappings, its gateway flag or its AZs (ChassisEvent,neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:162-223) →schedule_unhosted_gateways(neutron/services/ovn_l3/plugin.py:347-370). - What: for each gateway LRP (restricted to the HA groups that contained the removed chassis when the event is a removal), drop chassis that are no longer valid (not a gateway, physnet not mapped, AZ no longer matching), keep the remaining order, and fill up to 5 with new candidates using the same scheduler (
neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/commands.py:493-555,neutron/services/ovn_l3/plugin.py:372-392). Existing primaries are not moved when they stay valid, so adding a gateway chassis does not rebalance existing routers. - Concurrency: runs in the API worker chosen by the hash ring for the
Chassisrow; all LRPs are rescheduled in one NB transaction. - Cost: reads every gateway LRP and its HA group from the local NB replica, then one transaction.
7.3 Which host reports the gateway
PortBindingChassisEvent (05 § 2) writes the host of the active chassisredirect port binding as binding:host_id of the router gateway port, so GET /ports?device_owner=network:router_gateway and GET /routers/{id}/l3-agents show where the gateway is active (neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py:598-608, neutron/services/ovn_l3/plugin.py:315-345, 500-518).
7.4 Networks attached to a router (external ports, VLAN)
When a router gains or loses its gateway, each internal network gets the router's HA chassis group so that external ports (SR-IOV, baremetal) and VLAN tenant networks are served by the same chassis as the gateway (update_router_ha_chassis_group, neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py:1770-1795). HAChassisGroupRouterEvent keeps them aligned when the router's chassis list changes (05 § 2).
Source: reverse-spec-neutron/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.