cloudkitty — Async processing

Processing outside the API — CloudKitty — branch stable/2026.2

Branch stable/2026.2 @ 0685cc330e83 — repository https://opendev.org/openstack/cloudkitty — analysed on 2026-11-09

All background work runs in cloudkitty-processor, a cotyledon service manager that starts [orchestrator]/max_workers CloudKittyProcessor processes and [orchestrator]/max_workers_reprocessing CloudKittyReprocessor processes (both default to the CPU count; 0 disables) (cloudkitty/cli/processor.py:28-54, cloudkitty/orchestrator.py:56-69). There is no periodic task framework, no notification listener, no cache polling.

1. Rating loop (CloudKittyProcessor)

• Trigger: an endless while True: internal_run(); each pass ends with time.sleep([collect]/period) (3600 s), so the effective cadence is period + processing time — the code itself notes the drift (cloudkitty/orchestrator.py:690-737). It starts immediately at worker start.
• Purpose: bring every scope's rated data up to now − wait_periods × period.
• Flow:
  1. Load scopes from the fetcher and shuffle them (cloudkitty/orchestrator.py:760-765).
  2. For each scope, try the tooz lock without blocking; skip if taken (cloudkitty/orchestrator.py:702-732).
  3. Compute the next period: none yet → start of the current month; else last + period if last + period + wait_periods × period < now (cloudkitty/orchestrator.py:283-287, cloudkitty/utils/__init__.py:200-223).
  4. Create a Worker for the scope: load the enabled rating modules for this tenant (SQL), the metric file, a thread pool (cloudkitty/orchestrator.py:290-310).
  5. Loop over ready periods (cloudkitty/orchestrator.py:393-416): skip inactive scopes (cloudkitty/orchestrator.py:406-411); reload rules for the period start (cloudkitty/orchestrator.py:329-339); collect every metric in parallel (cloudkitty/orchestrator.py:356-391); rate (cloudkitty/orchestrator.py:509-521); filter points with skip_datapoints_expression (cloudkitty/orchestrator.py:462-507); push (cloudkitty/orchestrator.py:445-460); advance last_processed_timestamp (cloudkitty/orchestrator.py:523-524).
• Catch-up: a scope that is behind (new scope, outage, reset) is processed period after period inside the same lock until it is up to date, while other scopes of this worker wait.
• Concurrency and idempotency: the per-scope tooz lock prevents two workers from rating the same scope; with the default file:// back end this holds only on one host (03 § 6). The state is advanced after the push; a crash between push and state update re-rates that period and stores it twice [inferred: the v2 back ends append, they do not upsert].
• Error behaviour: any exception is logged and the scope is retried at the next loop (cloudkitty/orchestrator.py:716-725); a fetcher error escapes internal_run [inferred: cotyledon restarts the worker process].
• Cost per loop and per worker: fetcher (Keystone: 1 + N projects requests); per scope and ready period: about 2 + services × 5 + fields × 5 + rules SQL queries to reload rules (03 § 2.3), M collector calls (+ M resource searches with metadata), one or more storage writes. All workers fetch the full scope list and contend for the same locks.

2. RPC endpoints

The flags of the last four are read by get_reload_list / get_module_state (cloudkitty/orchestrator.py:131-143), whose only caller, process_messages, is a stub (cloudkitty/orchestrator.py:683-688). Module changes take effect because the Worker re-reads module state and rules from SQL every scope cycle, not because of these messages. reload_modules is cast with version 1.1 against a 1.0 endpoint (cloudkitty/api/v1/controllers/rating.py:51-53, cloudkitty/orchestrator.py:122-123).

3. Reprocessing loop (CloudKittyReprocessor)

• Trigger: same loop and sleep as § 1 (inherited), over schedules instead of scopes (cloudkitty/orchestrator.py:768-800).
• Data read: unfinished schedules (current_reprocess_time IS NULL OR current_reprocess_time < end_reprocess_time), default limit 100, newest first (cloudkitty/storage_state/__init__.py:310-347) — more than 100 pending schedules are served 100 at a time.
• Flow (cloudkitty/orchestrator.py:527-631):
  1. Lock named after the schedule (non-blocking).
  2. Check the scope still exists in cloudkitty_storage_states (exactly one row), take its scope_key.
  3. Next period: start_reprocess_time, then current_reprocess_time + period, while ≤ end_reprocess_time.
  4. On the first period only: delete [start, end) in storage for {scope_key: scope}.
  5. Collect, rate with the rules active at that period, push.
  6. current_reprocess_time = period start.
• Deactivated scopes: the inherited check skips a scope whose active is false, so its schedule stays pending until the scope is reactivated (cloudkitty/orchestrator.py:406-411).
• Error behaviour: as § 1; a vanished scope raises at every loop (cloudkitty/orchestrator.py:540-551).

4. Startup work

5. Configuration options that drive background behaviour

Source: reverse-spec-cloudkitty/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.