swift — Rewrite pack

Rewrite pack — swift — branch stable/2026.2

Branch stable/2026.2 @ a1bdaeba21bd — repository https://opendev.org/openstack/swift — analysed on 2026-10-29 Client repositories cited: python-swiftclient stable/2026.2 @ dae63b4621d3, python-openstackclient @ 8bb083b83829, glance_store @ 3bced266129d, cinder @ c117f9cf08b4, skyline-console @ 806d200392ee, skyline-apiserver @ 4af72bfd4577, liberasurecode tag 1.8.0 @ d971d8554e05.

Target: OpenKeel Swift in Go, keeping Swift's own model (rings, replication or EC, SQLite listings), devices on block-CSI PVCs (Longhorn by default) in one StatefulSet of storage pods. The design must allow a rolling migration: Go proxies and Go storage nodes join a running Python cluster, share its rings, disks, peers and clients, and replace the Python processes one by one. Sections 3 and 4 are therefore the core of this file: they state every wire and disk contract that a Go node must honour byte for byte, and how to read Swift's pickles without executing them.

1. Contents of _rewrite/

Library versions used for extraction: CPython 3.13.0, eventlet/greenlet/lxml/boto3 per requirements/upper-constraints.txt (stable/2026.2); PyECLib not installable on the build host (no liberasurecode): a placement-only stub was used, so EC encode/decode was not exercised (§ 8).

2. HTTP behaviour the contracts do not show

2.1 Middleware pipeline (outermost first) and which error wins

catch_errors → gatekeeper → healthcheck → proxy-logging → cache → listing_formats → container_sync → bulk → tempurl → ratelimit → (authtoken → s3api → s3token →) tempauth/keystoneauth → copy → container-quotas → account-quotas → slo → dlo → versioned_writes → symlink → proxy-logging → proxy-server (etc/proxy-server.conf-sample:160). Required filters are auto-inserted (swift/proxy/server.py:75, swift/proxy/server.py:785).

Consequences to keep:

• catch_errors turns any exception below into 500 and sets X-Trans-Id / X-Openstack-Request-Id on every response, errors included (swift/common/middleware/catch_errors.py:150). Trans-id format tx<21 hex>-<10 hex epoch>[<suffix>] (swift/common/utils/__init__.py:453).
• gatekeeper strips X-Backend-* and sysmeta before anything else sees them (swift/common/middleware/gatekeeper.py:87).
• tempurl and formpost run before auth: they set swift.authorize themselves; bulk and slo subrequests go through the rest of the pipeline again with the caller's identity.
• Authorization is not a middleware response but a hook (swift.authorize) called by the proxy before the handler (swift/proxy/server.py:601): a request for a missing container returns 401/403 before 404 for an unauthorised user, and 404 for an authorised one. With delay_denial, ACL reads (container info) happen first.
• Quotas (413) are checked before SLO/DLO/versioning see the request; copy turns COPY into GET+PUT above them.

2.2 Error body

No JSON error body anywhere in the Swift API: HTML <html><h1>…</h1><p>…</p></html> from RESPONSE_REASONS (swift/common/swob.py:59, swift/common/swob.py:1469) or short plain text. HEAD has none. Clients parse the status only (§ 5). S3: XML <Error> with Code, Message, RequestId (swift/common/middleware/s3api/s3response.py). Storage servers put a Python traceback in 500 bodies (swift/obj/server.py:1452): a Go node may return any text there.

2.3 Other cross-cutting rules

• Metadata limits and name limits of 01 § 1.3 (swift/common/constraints.py:48), overridable in swift.conf [swift-constraints] and published in /info.
• Header names are case-insensitive, returned title-cased (X-Object-Meta-Color); values are WSGI latin-1 strings carrying UTF-8 bytes.
• ETags are returned unquoted for plain objects, quoted for SLO/DLO manifests; etag_quoter can quote all (swift/common/middleware/etag_quoter.py:55). Cinder compares the PUT ETag with its own MD5 after swiftclient strips quotes (§ 5).
• Last-Modified is the X-Timestamp rounded up to the second; listings use ISO 8601 with microseconds.
• Unknown query parameters are ignored; limit > 10000 → 412.
• Expect: 100-continue is honoured; Transfer-Encoding: chunked beats Content-Length (swift/proxy/server.py:526).

3. Core algorithms: formal specification

3.1 Ring placement and handoffs

Input: ring (§ 3.2), swift.conf [swift-hash] prefix/suffix (latin-1 bytes), names (UTF-8).

hash      = md5(prefix + "/" + "/".join(names) + suffix)                 # swift/common/utils/__init__.py:1193
part      = be_uint32(hash[0:4]) >> part_shift                           # swift/common/ring/ring.py:498
primaries = dedup([devs[row[part]] for row in replica2part2dev if part < len(row)]), index = position
                                                                          # swift/common/ring/ring.py:487
handoffs:  start = be_uint32(md5(decimal(part))[0:4]) >> part_shift; inc = max(1, parts // 65536)
           seq   = range(start, parts, inc) ++ range(inc - ((parts - start) % inc), start, inc)
           for tier in (region, zone, ip, device):                        # swift/common/ring/ring.py:561
               if every value of that tier among assigned devices is used: next tier
               for hp in seq: for row in rows (in order) where hp < len(row):
                   d = devs[row[hp]]; if d unused and d's tier value unused: yield d (handoff_index++),
                   mark d's region, zone, ip and id used; stop the tier when all values are used

Deterministic, no randomness; only devices with assigned partitions count (swift/common/ring/ring.py:364). Proof: spec-reference reproduces all 128 lookups of vectors-ring (4 rings × 32 names). The ring is reloaded when its mtime changes, at most every 15 s (swift/common/ring/ring.py:33).

Not to reimplement as-is: the builder's rebalance. RingBuilder.rebalance(seed) (swift/common/ring/builder.py:541) uses Python's random (MT19937 with Python's seeding) and dict ordering; a Go builder cannot produce the same assignment from the same seed. It does not need to: any valid assignment works, as long as the file is in Swift's format and partition moves respect min_part_hours (data moves are what cost). Keep swift-ring-builder usable on the produced rings (v2 is readable by Swift ≥ 2.36.0, the version that introduced it, commit ae062f8b0; use v1 while any Python node older than 2.36 remains).

3.2 Ring file bytes

See 03 § 1.3 for the full layout. Summary to implement:

• gzip; first 6 decompressed bytes R1NG + uint16 BE version (swift/common/ring/io.py:270).
• v1: uint32 BE JSON length, JSON (devs, part_shift, replica_count = row count, byteorder, version, next_part_power), rows of uint16 in byteorder (swift/common/ring/ring.py:117).
• v2: Z_FULL_FLUSH-separated sections, each uint64 BE length + data: swift/ring/metadata (JSON with dev_id_bytes, float replica_count), swift/ring/devices, swift/ring/assignments (rows in network order, 2 or 4 bytes, last row may be short), then the JSON index {name: [cstart, ustart, cend, uend, "sha256", hex]}, then two uncompressed uint64 (index offsets) so that seek(-31, END) finds the compressed index offset (swift/common/ring/io.py:301, swift/common/ring/io.py:627). Digest covers length prefix + data (swift/common/ring/io.py:365).
• Writers: fixed gzip mtime 1300507380, JSON sort_keys, ensure_ascii, atomic rename (swift/common/ring/ring.py:282, swift/common/ring/io.py:400). Device id 0xFFFF (or 0xFFFFFFFF) = none (swift/common/ring/utils.py:41).
• Go readers must accept both versions and both byte orders in v1; Go writers should produce v2 only when every reader is ≥ 2.36.

3.3 On-disk object state

Rules of 03 § 4.2 (swift/obj/diskfile.py:971): file-name grammar (<ts>.data, <ts>#<f>[#d].data, <ts>.meta, <ts>[+-]<hexdelta>.meta, <ts>.ts, legacy <ts>.durable), selection of the newest tombstone/data/meta/ctype-meta, obsolete and reclaimable files, reclaim_age (7 days) and commit_window (60 s) (swift/obj/diskfile.py:82). Timestamps: 10 µs units + 64-bit offset; text %016.05f[_%016x] (swift/common/utils/timestamp.py:24). Vectors: vectors-ondisk (19 sets), vectors-timestamp.

Write protocol (crash consistency, swift/obj/diskfile.py:1958): temp file in tmp[-N] on the same filesystem → write → xattrs → fsync → rename into <hash>/ → fsync dir → append suffix to hashes.invalid under the partition lock (swift/obj/diskfile.py:477) → cleanup. A Go writer must never leave a .data without its xattrs visible under the final name, and must invalidate the suffix, otherwise the replicators of other nodes never notice the change.

3.4 Suffix hashes (hashes.pkl, REPLICATE)

• Replication: md5 over every hash directory of the suffix in sorted order, feeding <ts>.meta, <ts>.ts, <ts>.data of the chosen files then <ctype ts>_ctype (swift/obj/diskfile.py:1190, swift/obj/diskfile.py:3210).
• EC: per fragment index md5 of the timestamps of its fragment files; key None gets meta/ts, <ts>.durable of the newest durable set and ctype (swift/obj/diskfile.py:3898).
• Cleanup runs during hashing (obsolete files deleted, empty dirs removed → suffix dropped, swift/obj/diskfile.py:1282).
• hashes.pkl concurrency: read, compute, re-read under lock, write only if unchanged else restart (swift/obj/diskfile.py:1393).
• Proof: spec-reference reproduces the 7 suffix hashes (4 replication, 3 EC).

3.5 Quorum and best response

best_response (swift/proxy/controllers/base.py:2225): group statuses by 2xx/3xx/4xx, first class with ≥ quorum statuses wins, highest status of that class returned, else 503 (swift/proxy/controllers/base.py:2272). Quorums: n//2+1 (accounts, containers, replication, swift/common/utils/__init__.py:2550); EC PUT ec_ndata + min_parity_fragments_needed() (swift/common/storage_policy.py:637). Container updates spread by num_container_updates (swift/proxy/controllers/obj.py:90). Error limiting: 10 errors / 60 s (swift/proxy/server.py:229). Read path: try nodes in sorting_method order, newest X-Backend-Timestamp wins between 404s and data.

3.6 Pickled metadata: format and safe reading

Every pickle Swift writes, where, and the opcodes it uses are listed in 03 § 4.7–§ 4.8 and in vectors-pickle. Requirements for the Go codec:

1. Never run a pickle VM. Implement a decoder for the closed opcode set: PROTO, EMPTY_DICT, EMPTY_LIST, EMPTY_TUPLE, MARK, STOP, BINUNICODE, SHORT_BINUNICODE, SHORT_BINSTRING, BINSTRING, SHORT_BINBYTES, BINBYTES, BININT1, BININT2, BININT, LONG1, BINFLOAT, NONE, NEWTRUE, NEWFALSE, TUPLE, TUPLE1, TUPLE2, TUPLE3, SETITEM, SETITEMS, APPEND, APPENDS, BINPUT, LONG_BINPUT, BINGET, LONG_BINGET, GLOBAL, REDUCE, BUILD; refuse everything else (spec-reference_stable-2026.2.py unpickle is a 120-line model).
2. GLOBAL only for the allow-list of swift/common/utils/pickle.py:49; REDUCE only for _codecs.encode(str, "latin1") (→ bytes) and dict/bytes/copyreg._reconstructor (→ map). A forbidden global (vector os.system) must be an error, not a skip.
3. Bounds: input ≤ 1 MiB (metadata is at most a few KB plus system keys; split xattrs reassembled first), depth ≤ 32, memo indices checked.
4. Metadata semantics: reassemble user.swift.metadata, …1, …2… until ENODATA; verify user.swift.metadata_checksum (MD5 hex) when present, else optionally add it (swift/obj/diskfile.py:218); decode keys/values as UTF-8 keeping invalid bytes (surrogateescape); Python-2 pickles (no _codecs\nencode in the first 32 bytes) need the latin-1 round trip except for name (swift/obj/diskfile.py:164).
5. Writing: emit what CPython 3 pickle.dumps(dict_of_bytes, protocol=2) emits (each bytes value as _codecs.encode(<latin-1 text>, 'latin1') with memoised GLOBAL), split into 64 KiB xattrs, then the checksum xattr (swift/obj/diskfile.py:272). The vectors-pickle blobs are byte-exact targets. hashes.pkl, .pending tuples, async pendings and REPLICATE bodies use plain protocol-2 types (no globals).
6. Error mapping: decode failure on xattrs → quarantine the hash dir (as DiskFileQuarantined); on hashes.pkl → treat as invalid and rehash (swift/obj/diskfile.py:404); on a .pending record → log and drop that record (swift/common/db.py:849).
7. Ring builder files are unrestricted pickles of rich objects: do not read them in Go servers (§ 3.1).

3.7 Listings: row merge and DB hash

• chexor(old, name, created_at) XOR-accumulates md5("<name>-<created_at>") (swift/common/db.py:176); must be registered as SQL function chexor/3 on every connection because the triggers call it (swift/common/db.py:224). Vectors: 3 steps in vectors-replication.
• Row merge update_new_item_from_existing (swift/container/backend.py:180): data, content-type and metadata timestamps merged independently, stored in created_at as encode_timestamps (swift/common/utils/timestamp.py:490); swift_bytes follows data. Rows are DELETE+INSERT, never UPDATE (trigger). Listing order: by name (binary UTF-8), deleted = 0 filter via ix_object_deleted_name.
• Metadata column JSON {key: [value, ts]} merge by timestamp (swift/common/db.py:1015).

3.8 Erasure coding

• Policy options: ec_type, ec_num_data_fragments, ec_num_parity_fragments, ec_object_segment_size (default 1 MiB, swift/common/storage_policy.py:35), ec_duplication_factor (swift/common/storage_policy.py:463); ring replicas = (k+m)·duplication; node index → fragment index index % (k+m).
• PUT: the client body is cut into segments of segment_size; each segment is encoded by PyECLib into k+m fragments; fragment i of each segment is appended to fragment archive i (swift/proxy/controllers/obj.py:2131, encode at swift/proxy/controllers/obj.py:2165, last short segment swift/proxy/controllers/obj.py:2189). The archive stored as <ts>#<i>[#d].data is the concatenation of those fragments.
• Each fragment = 80-byte liberasurecode header + payload: fragment_metadata (packed: idx u32, size u32, frag_backend_metadata_size u32, orig_data_size u64, chksum_type u8, chksum 8×u32, chksum_mismatch u8, backend_id u8, backend_version u32 = 59 bytes) + magic u32 0xb0c5ecc + libec_version u32 + metadata_chksum u32 (crc32 of the 59 bytes) + 9 padding bytes (liberasurecode:include/erasurecode/erasurecode.h:264, liberasurecode:include/erasurecode/erasurecode.h:326). Fields are host-endian (little-endian in practice); readers byte-swap when the magic is swapped (liberasurecode:src/erasurecode.c:1135); the metadata checksum has a legacy alternative crc (liberasurecode:src/erasurecode.c:1162). Backend ids: liberasurecode:include/erasurecode/erasurecode.h:43.
• fragment_size (full fragment incl. header) comes from get_segment_info(segment_size, segment_size) (swift/common/storage_policy.py:586); ranged GETs map client ranges to segment-aligned fragment ranges (swift/proxy/controllers/obj.py:1688, swift/proxy/controllers/obj.py:1746) and decode with any k fragments of the same timestamp (swift/proxy/controllers/obj.py:1667).
• Recommendation: do not reimplement Reed-Solomon in pure Go for compatibility. Python nodes, the reconstructor and the proxy must be able to decode fragments a Go proxy wrote and vice versa; the matrices and padding of each ec_type (Jerasure, ISA-L, liberasurecode_rs_vand) are defined by those C libraries. Link liberasurecode via cgo (same version family as the cluster's PyECLib 1.8.0 / liberasurecode 1.8.x) and treat fragment archives as opaque on storage nodes (they never decode: only the proxy and reconstructor do). A pure-Go codec is acceptable only for a Go-only cluster and a new policy index [inferred].
• Not verified here: no EC encode/decode ran (no liberasurecode on the build host); EC placement, file naming, suffix hashing and the MIME/commit protocol were exercised.

4. Mixed-cluster compatibility contract (Go nodes beside Python nodes)

A migration step is "replace one pod's processes by Go processes on the same PVCs". It is safe only if all of the following hold. Each item is testable with the pack.

Recommended order: (1) Go proxies (stateless; verify C1–C4, C12 with differential replay of exchanges-client against a mixed cluster); (2) Go object servers + Python daemons on the same pods (C3–C6); (3) Go object daemons (C6–C8, C11); (4) container/account servers and daemons (C9–C10); (5) remove rsync once no Python replicator remains; (6) EC policies last (C13).

5. What the clients expect

Statuses that must not change: 404 (not 403) for a missing object to an authorised user; 202 for container PUT of an existing container; 204 for POST/DELETE; 201 for object PUT; 409 for non-empty container DELETE (Glance, Cinder and S3 BucketNotEmpty rely on it); 412 for If-None-Match: *.

6. Database strategy (SQLite listings)

• Reuse, do not migrate. Python and Go nodes must read each other's DB files through replication (rsync of whole DB files). Use the exact DDL of schema-sqlite-*, the same PRAGMAs, rollback journal mode (swift/common/db.py:220), and register chexor. Accept old schemas (no storage_policy_index, no reported/tombstones) and migrate them lazily with the same ALTER TABLEs (swift/container/backend.py:1561–swift/container/backend.py:1662).
• Integrity is enforced by triggers (counts, hash, no UPDATE of rows), not foreign keys.
• Concurrency: one writer per DB through the .pending file and lock_parent_directory (flock on the DB directory), BEGIN IMMEDIATE in merges (swift/container/backend.py:1331); readers commit pendings first (swift/common/db.py:822).
• Quarantine malformed DBs (swift/common/db.py:502).
• No startup data; hidden accounts are created on demand (auto_create_account_prefix .).

7. Authentication, configuration and CLI

• Auth: re-implement keystoneauth semantics (operator roles, reseller admin, service tokens, ACL v1/v2, system reader, swift_owner) on top of the OpenKeel Keystone token validation (instead of keystonemiddleware), and tempauth only for tests. Keep the account naming AUTH_<project_id> (swift/common/middleware/keystoneauth.py:277).
• Middleware-provided authorisation (tempurl HMACs, formpost, container-sync realms, s3 signatures) must be bit-exact: they are verified against keys stored in metadata by users.
• Encryption: if used, keep the key derivation (HMAC-SHA256 of path under the root secret) and crypto-meta header formats; otherwise existing encrypted objects become unreadable (swift/common/middleware/crypto/keymaster.py:307, swift/common/middleware/crypto/crypto_utils.py:205).
• Configuration: config_stable-2026.2.conf feeds oslo_conf_to_yaml.py; keep swift.conf itself as a file shared with Python nodes during migration (it is a cluster contract, not a per-service option file).
• CLI tools operators use against the cluster and that must keep working on Go-written data: swift-ring-builder, swift-get-nodes, swift-object-info, swift-account-info/-container-info, swift-recon, swift-manage-shard-ranges, swift-dispersion-* (setup.cfg:55). swift-recon needs /recon/* on storage servers with the recon cache JSON shapes (swift/common/middleware/recon.py:424).

8. Conformance strategy

1. Unit vectors (no Python needed): port spec-reference_stable-2026.2.py and run it on the vectors (153 checks pass on the reference). Add EC fragment vectors once liberasurecode is available.
2. Client-level differential tests: replay exchanges-client test by test (each test builds its own data through the API, starting from an empty account AUTH_test), compare status, key headers (etag, content-length, x-*-count, content-type) and JSON bodies; normalise timestamps, trans ids and S3 signatures. 1080 tests ran (225 skipped by configuration); 5 failures/errors in the recorded run are recorder artefacts: the recorder buffers bodies, which breaks SLO heartbeating, upload timeouts and the file-size-limit test. All 15 distinct tests failing under any of the four recordings pass when re-run without the recorder.
3. Backend wire tests: replay exchanges-object-server/-container-server/-account-server requests against a Go storage server on an empty device in order (chunked PUT bodies were not captured: replay with a synthetic body of the recorded Content-Length/ETag), compare statuses and X-Backend-* response headers.
4. Run Swift's functional tests against the Go proxy: test/functional accepts an external cluster through SWIFT_TEST_CONFIG_FILE (test/__init__.py:46); in-process mode is how this pack was recorded (test/functional/__init__.py:476).
5. Probe tests (test/probe/) need a multi-node SAIO with daemons; run them in the mixed-cluster stage (C5–C11). Not run here.
6. Adapter used: ~/.claude/skills/generate-reverse-spec/scripts/adapters/swift.py with conformance_capture.py http --wrap test.functional:loadapp (proxy) and --wrap swift.obj.server:ObjectController etc. (class wrap patches __call__ in place).

9. Gaps found while building the pack

• EC encode/decode, fragment archive bytes and the reconstructor's rebuild were not exercised (no liberasurecode on the host). Fragment format is documented from liberasurecode 1.8.0 headers only.
• Chunked request bodies (object data to object servers, EC MIME documents, SSYNC streams) are not in the backend exchanges; their framing is specified from code (03 § 2.3, § 2.7).
• Daemons (replicators, auditors, updaters, sharder, reconciler, expirer, reaper) produce no recorded exchanges: probe tests need a real multi-node setup.
• keystoneauth and s3token were not exercised (the functional cluster uses tempauth).
• The api-ref does not cover the internal API, UPDATE, or private methods.
• Pickle writer byte-exactness is checked against CPython 3.13 output only [inferred stable across 3.x for these simple types].

10. Non-functional requirements

• Streaming everywhere: objects up to 5 GiB per PUT (SLO beyond), constant memory; 64 KiB chunks (object_chunk_size, client_chunk_size).
• Timeouts: conn_timeout 0.5 s, node_timeout 10 s, client_timeout 60 s, container_update_timeout 1 s.
• fsync discipline of § 3.3; fallocate_reserve to keep space for metadata.
• Each device is an independent failure domain (mount_check → 507); in Kubernetes a PVC not mounted must answer 507, not write to the container's root filesystem.
• Background daemons rate-limited (files/s, bytes/s, objects/s) as in 05.

11. Rewrite checklist

• [ ] Ring reader v1/v2, placement and handoffs pass vectors-ring; reload on mtime.
• [ ] Ring writer produces files swift-ring-builder and Swift ≥ 2.36 load (v1 while older nodes exist).
• [ ] Timestamp type passes vectors-timestamp.
• [ ] Restricted pickle codec passes vectors-pickle (decode all, refuse the forbidden global, re-encode byte-exact).
• [ ] Diskfile selection and suffix hashes pass vectors-ondisk; write protocol with fsync and invalidation.
• [ ] Object server verbs and headers (C3, C4) pass the object-server replay; REPLICATE pickle; SSYNC sender/receiver.
• [ ] rsyncd sidecar with the three modules until all replicators are Go/ssync.
• [ ] SQLite brokers with the exact DDL, chexor, .pending, merge rules, sharding states; DB REPLICATE RPC.
• [ ] Proxy: pipeline order, quorum/best_response, error limiting, account autocreate, listings formats, sharded listings.
• [ ] Middlewares: gatekeeper, tempurl, formpost, slo, dlo, symlink, versioning (both), copy, bulk, quotas, staticweb, container_sync, ratelimit, encryption (if used), s3api (+ Keystone s3tokens).
• [ ] /info sections and limits as in info_stable-2026.2.json.
• [ ] Client expectations of § 5, statuses that must not change.
• [ ] Daemons of 05 with the same queues (.expiring_objects, .misplaced_objects, .shards_*) and recon cache files.
• [ ] EC via liberasurecode (cgo), bit-compatible with PyECLib 1.8.0; add EC vectors before enabling EC policies.
• [ ] Mixed-cluster stages of § 4 run with Swift's functional and probe tests.

Source: reverse-spec-swift/07-rewrite-pack_stable-2026.2.md. The _rewrite/ and _inventory/ files are not published on the site.