skyline-apiserver — Async processing

Processing outside the API — skyline-apiserver — branch stable/2026.2

Branch stable/2026.2 @ 4af72bfd4577 — repository https://opendev.org/openstack/skyline-apiserver — analysed on 2026-10-28

skyline-apiserver has no bus consumer, notification listener, periodic task, looping call, OVSDB event or background thread (the inventory finds 0 triggers; grep finds no asyncio.create_task, BackgroundTasks or scheduler). All its work happens inside an HTTP request or at process start. This file lists the processing that is not an endpoint's own logic but runs implicitly, because a re-implementation must reproduce it.

Start-up of a worker (lifespan)

• Trigger: gunicorn starts (1 + CPUs) // 2 uvicorn workers (etc/gunicorn.py:21); each runs the lifespan hook (skyline_apiserver/main.py:45).
• Flow: 1. configure("skyline") reads $OS_CONFIG_DIR/skyline.yaml and validates every option with pydantic (skyline_apiserver/config/base.py:113) — missing file or wrong type stops the worker; 2. logging to <log_dir>/<log_file>; 3. policies_setup() parses the embedded rules of the 16 services (skyline_apiserver/policy/__init__.py:25); 4. db_setup() creates the SQLAlchemy engine (no connection yet, skyline_apiserver/db/base.py:65); 5. CORS middleware if cors_allow_origins is not empty — which fails on Starlette 1.x (00 § 3).
• Not done at start-up: no Keystone call (the system session is created on first use), no database migration, no check that the database is reachable.
• Cost: the policy setup rebuilds an Enforcer after each rule of a service (skyline_apiserver/policy/__init__.py:33): quadratic in the number of rules per service, negligible in absolute terms.

Middleware work on every authenticated request

• Trigger: any path under /api/v1 except the public list (skyline_apiserver/main.py:81). The match is a prefix match, so /api/v1/login/totp and /api/v1/contrib/regionsXYZ are public too.
• Purpose: turn a cookie or a raw Keystone token into a validated profile.
• Data read: revoked_token by primary key; the Keystone token data.
• Flow and effects: see 01 § Authentication and 04 § 3. The purge DELETE FROM revoked_token WHERE expire < now - 1 runs on every request (skyline_apiserver/db/api.py:65), before the revocation check.
• Concurrency and idempotency: several workers and replicas purge concurrently; the delete is idempotent. A logged-out session is refused by every worker as soon as its row is committed (shared database). With SQLite on local disk (default /tmp/skyline.db), replicas do not share revocations.
• Error behaviour: any exception → 401 (02).
• Cost: 1 DELETE + 1 SELECT + 2 Keystone HTTP requests per call; the console issues one Skyline call per list page, plus its direct OpenStack calls which do not go through Skyline.

System token lifecycle

• Trigger: first call needing the system session in a worker (skyline_apiserver/client/utils.py:60); then keystoneauth re-authenticates when the cached token is about to expire (reauthenticate=True).
• Purpose: one privileged token per worker, reused by all users' requests.
• Error behaviour: if the password is wrong, every authenticated request fails with 401 and every public catalog call with 500, until the configuration is fixed and the workers restarted (the configuration is read once).
• Concurrency: the global SESSION is created without a lock [inferred: two threads of the same worker may authenticate twice on the first requests, harmless].

Policy file reload

• Trigger: each Enforcer.authorize call (skyline_apiserver/policy/base.py:128), i.e. ~1 860 times for GET /policies.
• Data read: stat of <policy_file_path>/<service>_<policy_file_suffix>; the file is re-parsed only when its modification time changes (oslo.policy _cache_handler).
• Effect: the file's rules replace the embedded rule of the same name; rules absent from the file keep the embedded default.
• Error behaviour: an unreadable file is ignored at DEBUG level (skyline_apiserver/policy/base.py:131).

nginx configuration generation

• Trigger: container start (container/start_service.sh:23).
• Data read: all Keystone endpoints of interface_type and the service list (skyline_apiserver/cmd/generate_nginx.py:124).
• Flow: for each endpoint whose service type is in service_mapping: location /<nginx_prefix>/<region lower>/<service>/, upstream = endpoint URL without the trailing version segment (v2, v2.1…) or %(tenant_id)s/%(project_id)s segment (skyline_apiserver/cmd/generate_nginx.py:149).
• Keystone unreachable at start: the script runs with set -ex (container/start_service.sh:2), so a ConnectFailure of the generator stops the container; under Kubernetes the pod crash-loops with back-off until Keystone answers, then recovers by itself. An init container that waits for Keystone avoids the back-off. Checked in kind by the skyline-console reverse-spec session (reverse-spec-skyline-console/07-rewrite-pack_stable-2026.2.md § 3.1, § 6.3).
• Readiness: nginx is started (container/start_service.sh:25) before gunicorn binds /var/lib/skyline/skyline.sock (container/start_service.sh:28), so GET / answers while /api/openstack/skyline/… still returns 502 for a few seconds. Probe GET /api/openstack/skyline/api/v1/config (public) instead.
• Error behaviour: two endpoints of the same type in one region → KeyError, the container start fails (skyline_apiserver/cmd/generate_nginx.py:137); the generator is not re-run when the catalog changes, so new services or regions need a restart.

Offline tools

Source: reverse-spec-skyline-apiserver/05-async-processing_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.