keystone — Rewrite pack

Rewrite pack — keystone — branch stable/2026.2

Branch stable/2026.2 @ e6dc5ae079e6 — repository https://opendev.org/openstack/keystone — analysed on 2026-10-28 Client references: keystonemiddleware stable/2026.2 @ 3a8a835bb573 (keystonemiddleware:), keystoneauth stable/2026.2 @ 1cd5400202c6 (keystoneauth:), tempest @ 9831863ccfea (tempest:)

This file is the entry point for re-implementing Keystone in another language (OpenKeel, in Go) as a drop-in replacement:

• every OpenStack service (keystonemiddleware), every client (keystoneauth, openstacksdk, Horizon) and the deployment tools must keep working unchanged;
• ideally, the Python and Go versions can run side by side on the same database and the same keys during the migration.

1. Contents of _rewrite/

Production: rewrite_pack.py (Flask extraction, --flask-app, --schemas-modules, --import-packages), then conformance_capture.py with the Keystone adapter (http, calls, replay) of the generate-reverse-spec skill, in a venv with Keystone, its test requirements and python-ldap installed.

2. HTTP behaviour that the contracts do not show

• Pipeline order (keystone/server/flask/core.py:39-142): ProxyFix → CORS → body size limit → HTTPProxyToWSGI → osprofiler → RequestId → URL normalisation (trailing / removed) → AuthContextMiddleware → Flask (before_request: logging, JSON body check) → handler. X-Auth-Token is therefore decoded before routing: an invalid token sent to a non-existent route gives 401, not 404 [inferred: AuthContextMiddleware order].
• No microversions: version discovery is GET / (300, the version list) and GET /v3, in JSON or JSON Home depending on Accept (keystone/api/discovery.py:44-65). keystoneauth reads them to find the v3 endpoint.
• Error body: {"error": {"code", "title", "message"}}, plus error.identity for MFA. WWW-Authenticate: Keystone uri="…" on every 401. Messages are masked unless insecure_debug (see errors). OAuth2 uses the RFC 6749 format.
• Fixed headers: X-Subject-Token on the response of POST/GET /v3/auth/tokens; Vary: X-Auth-Token on every response; no Content-Type on a 204.
• Lists: attribute filters with comparators (__contains…); limit/marker pagination bounded by max_db_limit; truncated and links.next; links.self on every entity (see REST API § 1).
• Member methods on a collection (DELETE /v3/policies…): 400, coming from a TypeError. A rewrite should answer 405 and list this difference among its known deviations.

3. Algorithms and formats a rewrite must reproduce exactly

3.1 Fernet token format (inter-version and inter-node contract)

A Go rewrite that shares the key repository with the Python service must produce and read the same bytes. The specification:

1. Fernet (standard specification): 0x80 | timestamp (8 bytes, big-endian, seconds) | IV (16) | AES-128-CBC(PKCS7(plaintext)) | HMAC-SHA256 (32), then URL-safe base64. Keystone **removes the trailing =** and restores it before decoding (keystone/token/token_formatters.py:39-120).
2. Keys: each file of the repository is a 32-byte URL-safe base64 key (16 bytes for signing, 16 for encryption).
   • Encryption: with the file of the highest number.
   • Decryption: by trying every key (MultiFernet).
   • 0 is the staged key (keystone/common/fernet_utils.py:202-266).
3. Plaintext = msgpack.packb((version, *fields)) (keystone/token/token_formatters.py:138-191), with the field order of each version in interactions § 2 (keystone/token/token_formatters.py:470-1160).
4. Field encodings:
   • IDs: msgpack pair [true, bin(16)] when the ID is a UUID (32 hex characters, without dashes, written back without dashes), otherwise [false, str];
   • methods: the sum of 2^i over the indexes i of the methods in [auth]/methods, in the order of that option;
   • expires_at: a float of seconds since the epoch;
   • audit_ids: a list of bin(16). An audit ID is base64url(uuid4.bytes) without padding, 22 characters (keystone/token/provider.py:69-75). The last element of the list is the token's own audit_id, and the one before it, if any, is the parent's (audit_chain_id);
   • system: the string "all".
5. **issued_at** is the Fernet timestamp, not a payload field. Both dates are rendered as ISO 8601 with microseconds and Z.
6. Old payloads: msgpack < 1.0 encoded some strings as raw, so decoding must accept bin where str is expected (keystone/token/token_formatters.py:193-210).

Test vectors (cases-fernet-tokens_stable-2026.2.json): 227 tokens produced by the unit and API tests, up to 25 per version. Versions covered: all 11 (0 to 10): 25 vectors per version, except version 6 (20), 8 (12), 9 (19) and 10 (1). Each case gives the key repository, the [auth]/methods order, the version, the token and the decoded fields. Replay: 227/227 decoded to the same fields, on two runs. A rewrite must:

• decode every vector to the same fields;
• produce, with the same keys, tokens that Python decodes to the same fields. To test this, encrypt the expected fields, then decode with keystone-manage or the replay.

JWS ([token]/provider = jws): ES256-signed JWT whose claims carry the same information (keystone/token/providers/jws/core.py). No vectors were captured; Fernet is the provider to support first.

3.2 Token validation (what keystonemiddleware relies on)

Steps of validate_token (keystone/token/provider.py:139-275):

1. decrypt (else 404);
2. expiry, plus allow_expired_window when ?allow_expired=1;
3. revocation (§ 3.3);
4. user still enabled (else revocation of the user, side effect);
5. access rules of the application credential, if any, which require the OpenStack-Identity-Access-Rules header ≥ 1.0;
6. rendering: roles recomputed at validation time, from the current assignments and implied roles, so a role removed after issuance disappears from the next validation; catalog filtered (OS-EP-FILTER, endpoint URL substitutions $(project_id)s… through keystone/common/utils.py:418-451, else 500 MalformedEndpoint).

Any failure → 404.

3.3 Revocation

• Event: a row of revocation_event with the fields user_id, project_id, domain_id, role_id, trust_id, consumer_id, access_token_id, audit_id, audit_chain_id, expires_at, issued_before and revoked_at (keystone/models/revoke_model.py:24-66).
• A token is revoked if an event exists with issued_before >= token.issued_at, for which every non-null field matches the token. The rules are those of matches() (keystone/models/revoke_model.py:160-222): the user is compared with the user, the trustor or the trustee; the domain with the identity domain or the scope domain.
• SQL pre-filter: see interactions § 3.
• Purge: on every write, of the events older than the lifetime of a token plus the expiry tolerance.

3.4 Federation mapping engine

• Input: the mapping rules (a JSON list of {"local": [...], "remote": [...]}) and the assertion. The assertion is the set of attributes the web server put in the WSGI environment, filtered by [federation]/assertion_prefix.
• Algorithm: RuleProcessor.process (keystone/federation/utils.py:495-737), helpers in keystone/federation/utils.py:738-977. For each rule, the remote conditions must all hold, with the operators type, any_one_of, not_any_of, whitelist, blacklist and regex. The values are then injected into local ({0}, {1}… in the order of remote). The results of the matching rules are merged into:
  • user: id, name, domain, type (ephemeral or local);
  • group_ids, group_names (with their domain);
  • projects, with roles, for automatic project creation (mapping schema 2.0).
• Errors: ValidationError (assertion without the required attributes, user absent from the mapping); DirectMappingError (500) when {n} refers to a remote that does not exist.
• Variants: schema_version 1.0, 2.0 and 3.0 choose the processor class (RuleProcessor, RuleProcessorToHonorDomainOption, RuleProcessorForProjectJsonAttribute, keystone/federation/utils.py:996-1130).
• Cases (cases-federation-mapping_stable-2026.2.json): 705 distinct cases recorded from 370 tests (test_v3_federation, contrib/federation/test_utils). Replay: 705/705. 13 of them are matching errors. Of the 692 others, about 520 to 530 are identical and about 165 to 172 are identical up to list order: the split changes from one run to the next.
  • Lesson from the replay: group_ids and group_names come out in an order that changes between runs, because they are built from sets. Compare them as sets.
  • These cases only exercise the base RuleProcessor class: no recorded test goes through the 2.0 or 3.0 variants. Add cases for them.

3.5 Passwords and accounts

• Hashing: the algorithm is recognised from the hash prefix, so a rewrite must verify every format: scrypt, bcrypt, bcrypt_sha256, pbkdf2_sha512 (keystone/common/password_hashing.py:33-60). It must also produce the format of [identity]/password_hash_algorithm. A password longer than [identity]/max_password_length gives 403 PasswordVerificationError (keystone/common/password_hashing.py:112).
• Account rules:
  • lockout after lockout_failure_attempts failures, for lockout_duration;
  • expiry after password_expires_days;
  • history of unique_last_password_count passwords;
  • minimum_password_age;
  • change_password_upon_first_use;
  • inactivity is computed on read (keystone/identity/backends/sql_model.py:217-238);
  • user options ignore_lockout_failure_attempts, ignore_user_inactivity… (keystone/identity/backends/resource_options.py:73-90).

3.6 Effective roles

list_role_assignments(effective=True) (keystone/assignment/core.py:1068, keystone/assignment/core.py:772-1066) expands:

1. groups (user memberships);
2. inheritance (OS-INHERIT: from a domain to its projects, from a project to its subtree, keystone/assignment/core.py:445);
3. implied roles (add_implied_roles, keystone/assignment/core.py:692, transitively);
4. domain-specific roles: they are removed from the final result when they only serve as relays (strip_domain_roles).

The token roles and GET /v3/role_assignments?effective depend on it: compare the rewrite with the exchanges of § 7 for the effective-list endpoints.

4. What the clients expect

5. Database strategy

Recommendation: share the existing schema at the Alembic expand and contract heads of this branch, and leave the migrations to keystone-manage db_sync (Python) until the end of the migration.

• The model DDL is not a faithful contract here. The real migrations (schema-sqlite-migrated_stable-2026.2.sql) differ from it (schema-diff_stable-2026.2.md):
  • a **token** table is still present (legacy, unused since Fernet: Keystone writes nothing to it);
  • assignment.role_id → role.id is a foreign key in the migrations but not in the models;
  • limit.registered_limit_id, registered_limit.service_id and registered_limit.region_id are foreign keys in the models but not in the migrations;
  • project_tag (project_id, name) is unique in the migrations;
  • indexes differ (access_token.consumer_id, endpoint.service_id, user_group_membership.group_id in the migrations; system_assignment.actor_id in the models). Write the rewrite against the migrated schema.
• IDs: uuid4().hex (32 characters, no dashes) for most entities. They are fixed by the client when allowed (domains, regions…). EC2 credentials are sha256(access). LDAP and federated users go through id_mapping (sha256).
• No token table: tokens are stateless. Revocation is the only state, and it must be shared.
• Cascades: Keystone's code does them (05 § 1), not foreign keys. Keep them in the same transaction as the triggering request.
• Cache: optional, but if the Go and Python versions run together they must share the invalidation scheme (<<<region>>>: keys in memcached, 03 § 4), or neither must cache tokens.

6. Authentication, policies, configuration and CLI

• Keystone is the authority: it does not use keystonemiddleware. Its AuthContextMiddleware decodes X-Auth-Token locally (same validation as § 3.2), and it also accepts tokenless authentication (mTLS) and REMOTE_USER (external).
• Policies: 204 rules in the oslo.policy language, with scope_types and deprecated rules. Operators override them through policy.yaml: the rewrite must parse that language (same need as Placement).
• Configuration: the oslo.config INI format and option names of config_stable-2026.2.conf. These are critical: [token], [fernet_tokens], [credential], [auth]/methods (encoding order!), [security_compliance], [identity], [federation], [cache].
• Key files: the same repository formats (Fernet tokens, credentials, receipts; JWS PEM). Rotation must stay compatible with keystone-manage fernet_rotate if both versions coexist.
• CLI: keystone-manage bootstrap, db_sync, *_setup/*_rotate, credential_migrate, trust_flush, mapping_purge… (05 § 4). They can stay in Python during the transition, as long as the rewrite reads the same files and the same tables.

7. Conformance strategy

1. Black box, language-independent: Tempest. tempest:tempest/api/identity contains 138 tests (admin and v3), run against the rewrite's URL in a devstack or kolla deployment. Add keystone-tempest-plugin (cloned next to this repository) for federation and RBAC [inferred: not counted].
2. Format vectors: the 227 Fernet tokens (§ 3.1) and the 705 mapping cases (§ 3.4). These are pure-function tests, without a database, to run in the rewrite's unit tests.
3. Differential testing on HTTP: exchanges_stable-2026.2.json holds 8,908 exchanges recorded from 1,679 Python API tests (16 modules test_v3_*, 0 failures, 16 skipped), each with its test ID.
   • Seeds: the capture was re-run with --seed-per-test (conformance_capture.py http): [fixture-seeds_stable-2026.2.json](_rewrite/fixture-seeds_stable-2026.2.json) gives, for each of the 1,489 tests that send requests, the database rows at its first request, the configuration options that differ from their defaults ([auth]/methods order included) and the policy overrides. The tests build their data through the Python API, so 276 exchanges (in 231 tests) also carry a **db_delta**: the rows inserted, updated or deleted outside HTTP since the previous request of the same test. The exchanges are grouped by test id (file = test id): one fresh database per test.
   • Replay: load the seed by natural key (as for Placement), apply each exchange's db_delta before sending it, and bind the IDs and tokens the service generates (UUIDs in bodies, X-Subject-Token). Tests that freeze or move the clock (time_override) or depend on token expiry are expected to differ.
4. Response schemas: the rewrite's responses must validate against the schemas declared in the code (request-schemas_stable-2026.2.json, *_response_body keys, attached to 81 operations of the OpenAPI document). For the others, use the observed schemas (response-schemas_stable-2026.2.json, 472 operation × status pairs, over 284 method + route keys (HEAD included) operations × status).
5. Errors: the 301 observed combinations (IDs templated, with or without dashes) (observed-errors_stable-2026.2.md), especially the masked messages of 401 and 403.

8. Gaps found while building the pack

9. Non-functional requirements

• Validation latency: GET /v3/auth/tokens is called by every service on every request whose token is not in the middleware cache. It is the hot path, together with the revocation query and the role computation.
• Stateless and horizontal: any number of instances, provided they share the database, the keys and (if enabled) the cache.
• Auditability: CADF notifications for the same events and with the same event_types (03 § 5) if Ceilometer, Designate or a SIEM consume them.
• Security: messages masked; lockout and password rules; strip the EC2 delegation fields (06, bug 2165281); constant-time comparison for secrets [inferred: to check in Python].

10. Rewrite checklist

• [ ] Routes and statuses from routes_stable-2026.2.json; discovery / and /v3 with JSON Home
• [ ] Middleware order and error body (masking, WWW-Authenticate, X-Subject-Token, Vary)
• [ ] Fernet: decode the 227 vectors; encode tokens that Python decodes; MultiFernet keys and rotation
• [ ] Validation: expiry, allow_expired, revocation, user enabled, access rules, catalog
• [ ] Revocation events: same table, same matching, same purge
• [ ] The 705 mapping cases (as sets); schemas 2.0 and 3.0
• [ ] Passwords: verify every hash format; account rules
• [ ] Effective roles: groups, inheritance, implied roles, domain-specific roles
• [ ] Cascades of 05 § 1 in the same transaction
• [ ] 204 oslo.policy rules and the policy language; oslo.config options
• [ ] Migrated schema (not the model one); no new foreign key
• [ ] CADF notifications
• [ ] Tempest identity (138 tests) green; differential tests on the exchanges; responses valid against the declared schemas
• [ ] EC2/S3 security fix (06)

Source: reverse-spec-keystone/07-rewrite-pack_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.