skyline-apiserver — Rewrite pack

Rewrite pack — skyline-apiserver — branch stable/2026.2

Branch stable/2026.2 @ 4af72bfd4577 — repository https://opendev.org/openstack/skyline-apiserver — analysed on 2026-10-28 — client cited: skyline-console stable/2026.2 @ 806d2003 (prefix skyline-console:), cinder stable/2026.2 @ c117f9cf0 (prefix cinder:)

This file is what a re-implementation (for OpenKeel: the Go "Skyline UI" back end) needs beyond files 01–05: the machine-readable contracts, the behaviour only the code defines, what skyline-console relies on, and how to prove conformance.

1. Contents of _rewrite/

Extraction environment: Python 3.13, stable/2026.2 upper-constraints (without the skyline pin), FastAPI 0.162.0 (not in upper-constraints), Starlette 1.10.0, pydantic 2.13.4, SQLAlchemy 2.0.52, alembic 1.19.1, keystoneauth1 5.17.0, python-keystoneclient 6.0.0, python-novaclient 18.13.1, python-cinderclient 9.10.0, python-neutronclient 14.0.0, openstacksdk 4.20.0, oslo.policy 6.0.1, python-jose 3.3.0, httpx 0.28.1. The outbound wire format depends on these client versions.

Extractor added for this project. rewrite_pack.py had no FastAPI support; a small generic --fastapi-app MOD:ATTR extractor was added: it walks app.routes (including the lazily included routers of FastAPI ≥ 0.16x), records each route's handler and def line, and writes app.openapi() as the OpenAPI document. --policies now also accepts a function returning {service: [rules]} (flattened to service:rule). Command used:

<venv>/bin/python rewrite_pack.py skyline-apiserver --branch stable/2026.2 --project skyline-apiserver \
    --fastapi-app skyline_apiserver.main:app --models skyline_apiserver.db.models:METADATA \
    --policies skyline_apiserver.policy.manager:get_service_rules --out reverse-spec-skyline-apiserver/_rewrite

2. HTTP behaviour that the contracts do not show

2.1 Middleware pipeline (outermost first)

1. Starlette ServerErrorMiddleware: uncaught exception → 500 text/plain "Internal Server Error".
2. (CORS when configured — broken on this branch, 00 § 3.)
3. validate_token HTTP middleware (skyline_apiserver/main.py:75): public-prefix bypass, then authentication. It runs before FastAPI's validation, so a request without credentials and with invalid parameters gets 401, not 422.
4. FastAPI ExceptionMiddleware: HTTPException → {"detail": …} with the status; RequestValidationError → 422.
5. Router: unknown path → 404 {"detail": "Not Found"} — but only for public prefixes or authenticated callers; an unknown /api/v1/xyz without credentials gives 401. Wrong method → 405 {"detail": "Method Not Allowed"}.

2.2 Error bodies

• Middleware: {"message": "Unauthorized: <text>"} (always 401).
• Endpoints: {"detail": "<text>"}; the only structured detail is the TOTP challenge {"totp_required": true, "receipt": "…"} (skyline_apiserver/schemas/login.py:118).
• Validation: FastAPI/pydantic v2 list format (type, loc, msg, input, ctx).
• Texts that the console matches are listed in § 4; other texts are free, but texts coming from Keystone must be passed through unchanged.

2.3 Other cross-cutting rules

• JSON only; no content negotiation; websso takes a form body.
• Every success is 200 (websso: 302).
• Response models drop nothing except where response_model_exclude_none=True (compute services, Prometheus): there null fields are omitted; elsewhere they are serialised as null.
• Query lists use repeated keys (sort_keys=a&sort_keys=b, uuid=…&uuid=…); booleans accept FastAPI's forms (true, 1, yes, on…).
• The X-Openstack-Request-Id header regex is anchored only at the start (^req-\w{8}(-\w{4}){3}-\w{12}, skyline_apiserver/types/constants.py:26): a longer value with that prefix is accepted.

3. Core behaviours: formal specification

3.1 Session token (JWT) and cookies

• Format: JWS compact, alg=HS256, key default.secret_key (UTF-8 string), payload exactly {"keystone_token": str, "region": str, "exp": int, "uuid": str(32 hex)} (skyline_apiserver/schemas/login.py:76). A rewrite that must accept sessions created by the Python server during a rolling upgrade has to keep this format and key.
• Issue: exp = now + access_token_expire (default 3 600 s), uuid = uuid4().hex (skyline_apiserver/core/security.py:81). Note the session can outlive the Keystone token: validity is the intersection, enforced by the per-request Keystone validation.
• Cookies: session=<jwt>, time_expired=<exp> (decimal string), path /, no Max-Age/Expires, SameSite=Lax, no HttpOnly, no Secure (skyline_apiserver/api/v1/login.py:171). The console reads time_expired from JavaScript (§ 4), so it must stay readable; session could become HttpOnly without breaking the console [inferred from the console code]. Seen on the wire as Path=/; SameSite=lax only. default.ssl_enabled does not add Secure or turn TLS on: it only picks the scheme of the WebSSO origin URL (skyline_apiserver/api/v1/login.py:527).
• Renewal: on an authenticated request with 0 < exp - now < access_token_renew (default 1 800 s), new JWT with exp = now + access_token_expire, same uuid and token; both cookies re-set (skyline_apiserver/main.py:160).
• Revocation: logout inserts (uuid, exp); requests with that uuid are refused until the row is purged (expire < now - 1).

3.2 Profile building

login(credential):                                               # skyline_apiserver/api/v1/login.py:395
  region  := CONF.default_region; domain := credential.domain or CONF.user_default_domain
  unscoped := keystone.password_auth(username, password, domain)  # 401 handling: 02
  projects := [p in GET /v3/auth/projects (unscoped) if p.enabled] # skyline_apiserver/api/v1/login.py:296-300
  if projects == []: fail 401 "You are not authorized for any projects or domains."
  default := GET /v3/users/{token.user.id}.default_project_id     # skyline_apiserver/api/v1/login.py:230 (system session)
  landing := default if default in ids(projects) else projects[0] # skyline_apiserver/api/v1/login.py:155 (Keystone order)
  scoped  := rescope(unscoped, landing)                            # skyline_apiserver/client/openstack/system.py:30
  profile := token_data(scoped) + {exp, uuid, version, base_domains}   # skyline_apiserver/core/security.py:54
  patch(profile)                                                   # skyline_apiserver/api/v1/login.py:330
patch(profile):
  regions   := sorted(unique(region_id of every endpoint of interface_type in the USER catalog)) or [] on error
  endpoints := {service_mapping[type]: "/" + nginx_prefix + "/" + region.lower() + "/" + service
                for each type with an endpoint in region, in the SYSTEM catalog}
             + {extension_mapping[alias]: endpoints["neutron"] for alias in Neutron /v2.0/extensions}
  projects  := GET /v3/users/{uid}/projects (system session)  (fallback: /v3/auth/projects with the user token)
  default_project_id := GET /v3/users/{uid}.default_project_id

Note that endpoints comes from the system catalog while regions comes from the user catalog.

3.3 Policy evaluation (/policies)

• Rule language: oslo.policy (role:, rule:, project_id:%(project_id)s, is_admin:True, system_scope:all, and/or/not, @, !), evaluated with oslo.policy's own _checks._check (skyline_apiserver/policy/base.py:140). A Go rewrite needs an oslo.policy-compatible evaluator.
• Rules: embedded defaults per service (skyline_apiserver/policy/manager/*.py, dumped in policies_stable-2026.2.json), overridden by <policy_file_path>/<service>_<policy_file_suffix> rule by rule (skyline_apiserver/policy/base.py:134). The evaluator uses basic_check_str when a rule defines one, else check_str.
• Credentials (skyline_apiserver/policy/base.py:32): user_id, project_id, tenant_id, domain_id, user_domain_id, project_domain_id, username, project_name, domain_name, user_domain_name, project_domain_name, role_ids, roles (names), is_admin (any of system_admin_roles), is_reader_admin (any of system_reader_roles), system_scope (all if the user can obtain a system-scoped token, § 3.2 of 01).
• Target: _generate_target (skyline_apiserver/api/v1/policy.py:40), every value being the user's own ID, project or domain; /policies/check merges the request's target over it.
• Output order: services in the order of SUPPORTED_SERVICE_EPS (skyline_apiserver/types/constants.py:49), rules in definition order of each module. A service whose evaluation raises is skipped with a warning.

3.4 Aggregations

The joins of /extension/* are specified in 01 (business rules) and 03 (calls, sessions, batch size 100). Points a rewrite gets wrong easily:

• servers.image*: root-volume image metadata has priority over the server's image (boot from volume) (skyline_apiserver/api/v1/extension.py:244).
• servers.host is null unless all_projects, even for admins.
• volumes: attachment server_name searched among normal and soft-deleted servers.
• volume_snapshots.child_volumes: volumes whose snapshot_id is the snapshot, [{volume_id, volume_name}].
• ports: one page only, no count; network_name filters by name then intersects with network_id.
• recycle_servers.reclaim_timestamp = epoch(updated_at) + reclaim_instance_interval (float seconds).
• Prometheus filtering for non-admins: keep a series only if metric.project_id == current project (skyline_apiserver/api/v1/prometheus.py:50).

3.5 Determinism

Nothing is random except the session uuid. List order is the service's order; /contrib/regions is unordered (Python set), /login's regions is sorted.

4. What the clients expect (skyline-console)

On the OpenStack side (calls Skyline makes, § 03): Cinder must accept list-literal filters (cinder:cinder/volume/api.py:2369); Nova must accept microversion 2.79 and status=soft_deleted&deleted=True from an admin; Glance must accept id=in:; the skyline account must be allowed to list all projects' servers, volumes and images, to read users and user projects, and to validate tokens.

5. Database strategy

• Two tables, no foreign key, primary keys on uuid and key; reuse the migrated schema (schema-sqlite-migrated_stable-2026.2.sql, same as the models apart from constraint names). Keep the Alembic version table at 001 if the Python server may run against the same database.
• settings.value is JSON: SQLite stores text, MariaDB a JSON (LONGTEXT + check) column, PostgreSQL JSON.
• Use real transactions: a single-statement upsert (INSERT … ON DUPLICATE KEY UPDATE / ON CONFLICT) fixes the race of update_setting (00 § 9).
• The revocation list only needs uuid lookups and a purge by expire; an index on expire is absent and would help large purges [inferred]. Purging on every request is a choice, not a requirement: a periodic purge gives the same observable behaviour, since expired JWTs are refused by their signature check anyway.
• In OpenKeel (MariaDB Galera), replicas must share the database for logout to be global.

6. Authentication, policies, configuration and CLI

• Authentication: no keystonemiddleware; the rewrite must validate the Keystone token itself (GET /v3/auth/tokens with a service token) on each request, or cache the validation for a short time (the Python server does not cache) and accept the delayed revocation that implies.
• Configuration: one YAML file, $OS_CONFIG_DIR/skyline.yaml (default /etc/skyline/skyline.yaml), three sections default, openstack, setting, validated with strict types (StrictStr, StrictInt, HttpUrl…) (skyline_apiserver/config/base.py:49); unknown keys are ignored; not oslo.config. Options in use: default.{debug, log_dir, log_file, secret_key, access_token_expire, access_token_renew, cors_allow_origins, session_name, ssl_enabled, cafile, secure_proxy_addr_header, database_url, prometheus_endpoint, prometheus_enable_basic_auth, prometheus_basic_auth_user, prometheus_basic_auth_password, policy_file_suffix, policy_file_path}, openstack.{keystone_url, system_*, default_region, user_default_domain, interface_type, nginx_prefix, base_domains, system_admin_roles, system_reader_roles, service_mapping, extension_mapping, reclaim_instance_interval, enforce_new_defaults, sso_enabled, sso_protocols, sso_region, sso_identity_providers}, setting.*. access_log_file and error_log_file are only used by the nginx generator. Keeping the same file format lets deployment tools (kolla-ansible, the devstack plugin) configure the rewrite unchanged.
• Policies: the 16 embedded rule sets are data; ship policies_stable-2026.2.json and keep the per-service override files. enforce_new_defaults is only passed as a target attribute (skyline_apiserver/api/v1/policy.py:45).
• Dependencies at login: a network (Neutron) endpoint is required in every region a user logs into, otherwise login answers 401 Endpoint not found (03 § outbound). A rewrite should treat missing extensions as an empty list rather than failing the login, and say so as a deliberate difference.
• Start-up and probes: the container exits while Keystone is unreachable (nginx generation, 05) and serves / before the API socket exists. For OpenKeel: wait for Keystone in an init container (or retry inside the process), and use GET /api/openstack/skyline/api/v1/config as readiness probe. Checked in kind by the skyline-console reverse-spec session (reverse-spec-skyline-console/07-rewrite-pack_stable-2026.2.md § 3.1, § 6.3).
• CLI: skyline-nginx-generator is part of the deployment contract (container start). In OpenKeel the routing to the services is better done by a Kubernetes Ingress or Gateway built from the same catalog [inferred design choice], but the path scheme /<nginx_prefix>/<region lower>/<service>/ must be kept because the profile announces it to the console.

7. Conformance strategy

• No gabbi, no Tempest suite exists for Skyline. The project's 115 unit tests (all pass) mock the client modules and test handler logic in isolation; they cannot be replayed against another implementation.
• Outbound contract as the conformance suite: _rewrite/tools/outbound_capture.py drives the inbound API and records the outbound calls against a fake OpenStack. Pointed at a rewrite (replace the in-process TestClient by an HTTP base URL and the fake's catalog URL in the rewrite's configuration), compare per scenario: inbound status, error body text, cookies set, and the ordered list of (method, path, query, session label) — where fewer Keystone calls are an acceptable difference, the rest is not. Two known divergences are deliberate fixes and should be listed as such: recycle_servers non-empty (500 → 200) and non-UUID-v4 IDs.
• Policy evaluation: compute allowed for the 1 860 rules and a set of personas (member, reader, admin, system admin) with the Python server (the policies (admin) and policies check (demo) scenarios are a starting point) and compare.
• End-to-end: skyline-console's own E2E tests (Cypress, skyline-console:test/e2e/) run against a real deployment are the final check [inferred: not run here].

8. Gaps found while building the pack

• rewrite_pack.py had no FastAPI extractor (added, § 1). FastAPI ≥ 0.16x stores included routers as _IncludedRouter objects without path, which the extractor now walks.
• No api-ref, no microversion history, no oslo config or policy generator namespaces to run: those extractors do not apply.
• The OpenAPI document declares 400/403/500 responses that some handlers never return and omits the middleware 401 and the 422 on the request-ID header.
• Unit tests mock RecycleServersResponseBase and schemas, which hides the recycle-bin bug.

9. Non-functional requirements

• Latency is dominated by Keystone: ~3 Keystone requests per call, 16 for a login; a rewrite should cache the system token, the catalog and (briefly) token validations.
• Workers: (1 + CPUs) // 2 processes, gunicorn timeout 300 s (etc/gunicorn.py:23); OpenStack timeouts 30 s.
• Security: change secret_key (no default in a rewrite), Secure+HttpOnly on session, verify Prometheus TLS, do not return the Keystone token in the body once the console no longer needs it (it currently does).
• Logs: one line per request at DEBUG only; consider structured access logs including the 401 causes.

10. Rewrite checklist

• [ ] 26 routes under /api/v1 with the same paths, methods, query parameters, enums and response fields (openapi_stable-2026.2.json).
• [ ] Middleware: public-prefix list, cookie or X-Auth-Token, revocation check, live Keystone validation, renewal window, {"message": …} 401 bodies.
• [ ] JWT format and cookies (session, time_expired) compatible with § 3.1; time_expired readable by JavaScript.
• [ ] Login, TOTP (receipt header), WebSSO (form post, 302 to /base/overview), landing-project rule, profile building (§ 3.2) including endpoints paths and Neutron extension aliases.
• [ ] Keystone error texts passed through in detail; exact strings receipt_expired, invalid_totp, authentication_failed, Region not accessible, Not a valid setting key., Not allowed to … for all projects..
• [ ] Role checks by role name against system_admin_roles / system_reader_roles.
• [ ] Aggregations with the same session choice (user vs system), batch size 100, filters and enrichment rules (§ 3.4); Cinder list-literal filters; Nova 2.79; Cinder 3.59.
• [ ] Policy evaluation with an oslo.policy-compatible engine, the embedded rules and the override files.
• [ ] Prometheus proxy with project filtering and status pass-through.
• [ ] Settings store (JSON values, admin-only writes, defaults from configuration).
• [ ] skyline.yaml compatibility, or a documented mapping to the OpenKeel configuration.
• [ ] nginx/Ingress routes /<nginx_prefix>/<region>/<service>/ matching the profile's endpoints.
• [ ] Decide and document the fixes: recycle-bin 500, UUID-v4 strictness, CORS start-up failure, dead switch_project check, %s logs, non-transactional setting upsert.
• [ ] Replay _rewrite/tools/outbound_capture.py scenarios against the rewrite and review every difference.

Source: reverse-spec-skyline-apiserver/07-rewrite-pack_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.