Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Verstack LocalandClaude Opus 5
|
daeed7ac12 | ||
|
Verstack LocalandClaude Opus 5
|
473ee47c46 | ||
|
Verstack LocalandClaude Opus 5
|
2cb25d9018 |
@@ -1,4 +1,5 @@
|
||||
/target/
|
||||
/crates/*/target/
|
||||
/.cargo-home/
|
||||
/data/
|
||||
/config.json
|
||||
@@ -46,6 +47,11 @@ __pycache__/
|
||||
*.pyc
|
||||
/tools/
|
||||
|
||||
# Reconstructed emulator profiles kept only for already-running legacy consumers.
|
||||
# New code consumes emulator/upstream + emulator/patches through source_profiles.py.
|
||||
/emulator/vendor/
|
||||
/emulator/bundles/vendor/
|
||||
|
||||
node_modules/
|
||||
/workbench/*/lib/
|
||||
/workbench/browser-app/src-gen/
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
# Working in this repo
|
||||
|
||||
Implementation continuation is tracked in `docs/plan-implementation-status.md`.
|
||||
It distinguishes tested code, deployed outcomes, and still-open PLAN requirements;
|
||||
do not treat the large historical "done" sections as proof that the full plan is complete.
|
||||
|
||||
## Current implementation contracts
|
||||
|
||||
* `/api/compare` returns a canonical release-diff object, not the old array of
|
||||
filesystem changes. Workbench `counts` has the same denominator as spine deltas;
|
||||
`view_counts` / `view_total` describe the filtered decoder-profile view. CLI
|
||||
filesystem comparison still answers preservation questions separately.
|
||||
* A full Ghidra mask is **not** the FLIRT export string. Schema 13 preserves it as
|
||||
`function_signatures.full_mask` before slimming new `symbols.payload` rows.
|
||||
Matching reads immutable facts; tests that reconstruct facts from SQL must join
|
||||
the full mask, not the lossy prefix/CRC `pattern` column.
|
||||
* Schema 13 builds three FTS indices and backfills masks. On the real catalog the
|
||||
debug migration rehearsal took 291 seconds. Check live process/WAL progress;
|
||||
the old six-second readiness assumption does not apply to the first migration.
|
||||
* Native `/api/decode/inspect` is bounded metadata inspection over `SourceReader`.
|
||||
It does not replace the Python extraction chain or persist recovered names.
|
||||
* `verstack mcp` talks to the running backend and does not acquire archive ownership.
|
||||
Its proposal tools verify actual cited bytes and store only proposed claims.
|
||||
`search_strings` currently scans an explicit bounded file range, not a corpus index.
|
||||
* Large import stages can borrow idle scratch within the configured pool. Each stage
|
||||
reacquires its own budget; never silently reuse an undersized outer reservation.
|
||||
Uploads reserve before receiving bytes but retain their existing upload-size cap.
|
||||
|
||||
Short version of what will otherwise cost you hours. The plan itself is in **`PLAN.md`** — read that
|
||||
next. What has already been done, with evidence, is in `docs/phase0-verification.md`.
|
||||
|
||||
## Traps
|
||||
|
||||
**`plugins/*.py` is not what runs.** `config.json` points several tools at a *frozen bundle* under
|
||||
`data/archive/tool-sources/<sha256>/`, pinned by `source_digest`. The working tree's digest already
|
||||
differs. Editing `plugins/ghidra/analyze.py` or `plugins/spike_probe.py` changes nothing until the
|
||||
pipeline is reconfigured (`scripts/configure_pipeline.py`). Check before you debug by editing.
|
||||
|
||||
**POST to the Rust API needs a header.** `X-Verstack-Client: 1`, or you get a bare `403` with no
|
||||
body (`src/http.rs:145`, `browser_boundary`). A second middleware allows only one in-flight POST and
|
||||
returns `409` on contention.
|
||||
|
||||
```sh
|
||||
curl -X POST http://127.0.0.1:8080/api/process \
|
||||
-H 'Content-Type: application/json' -H 'X-Verstack-Client: 1' \
|
||||
-d '{"snapshot":"<id>","plugin":"media-preview"}'
|
||||
```
|
||||
|
||||
**`processing_enabled` is `false`.** The background queue consumes nothing. Explicit `/api/process`
|
||||
and `/api/import` still run. Flip it only deliberately.
|
||||
|
||||
**Use the right Python.** System `python3` lacks `numpy`/`PIL` and reports 3 collection errors.
|
||||
`tools/decoder-env/bin/python` is the interpreter the pipeline uses and runs all 97 tests green.
|
||||
|
||||
**The archive holds an instance lock.** Mutating CLI commands need the service stopped. Read-only
|
||||
queries against the catalog are fine — use `file:...?mode=ro`.
|
||||
|
||||
**Do NOT add `immutable=1` to a read-only catalog URI while the service is running.** It tells
|
||||
SQLite the file can never change, so it skips the WAL — and recent writes live in the WAL until a
|
||||
checkpoint. `verify_phase0.py` carried that flag and reported 1 of 15 releases had a platform
|
||||
immediately after all 15 were written; earlier runs were correct only because a service restart had
|
||||
checkpointed first. It is safe only against a genuinely static copy.
|
||||
|
||||
**`logical_path` exists twice** — `src/identity.rs` and mirrored in `scripts/verify_phase0.py`. If
|
||||
you change the rule, change both.
|
||||
|
||||
**There are THREE comparison paths, and they were fixed one at a time, months apart.**
|
||||
`archive::compare` (Phase 0/1), `workspace::browse_workspace` (what the workbench "Compare
|
||||
releases" screen calls — found still broken in Phase 6, reporting 8,151 added / 8,150 removed for
|
||||
two builds differing by 208 assets), and `analysis::pair_programs` (code). Before adding a fourth,
|
||||
check whether it joins two releases on anything path-shaped.
|
||||
|
||||
**The version-in-the-path defect is not only in asset paths.** `code_programs.source_path` embeds
|
||||
it too, so `pair_programs` joined on raw equality and every packaged release paired with nothing —
|
||||
the API answered "these snapshots share no analysed program" for Pokémon 0.85→0.86. Anywhere you
|
||||
join two releases on a path, join on `identity::logical_path`. Assume it is wrong until checked.
|
||||
|
||||
**Resolving an analysis snapshot to its release means walking to the *root* of the snapshot chain.**
|
||||
Analysis outputs are siblings branching off a common ancestor, not ancestors of `versions.snapshot`.
|
||||
A head-or-immediate-parent join resolves 8 of 47 programs, finds zero shared programs, and looks
|
||||
entirely plausible while doing it.
|
||||
|
||||
**Version strings sort numerically, not lexically.** Stern zero-pads the minor field (`0.81.0`,
|
||||
`1.01.0`) so SQL TEXT ordering happens to be right on the current 11 versions, but one unpadded
|
||||
version would silently pair the wrong releases in `compute_all_deltas`. Use
|
||||
`spine::version_order`.
|
||||
|
||||
**`data/` is gitignored** and holds key material, including `data/deployment/workbench-auth.json`
|
||||
(mode 0600). Never commit anything from it, never print key values.
|
||||
|
||||
**Never GC 6 of the 15 archived originals.** Only 9 have an exact match still in `/srv/firmware`;
|
||||
the other 6 (26.2 GB packed, including a 63 GB Pokémon 0.83 SD image) exist nowhere else. Verify by
|
||||
hash, not filename.
|
||||
|
||||
## The review application
|
||||
|
||||
`/review` is the version-diff review screen: `web/review.html`, `web/review.js`, `web/review.css`,
|
||||
compiled into the binary with `include_str!` and served by axum. **There is no second build step** —
|
||||
`cargo build` is the whole pipeline. That is deliberate, given the Theia trap above.
|
||||
|
||||
Reachable at `http://<host>:3000/review` behind the password gate. The gateway routes `/review*`
|
||||
and `/api/*` to the Rust binary on 8080 and everything else to Theia on 3001.
|
||||
|
||||
The layout is five stacked bands — rail, HUD, stage, verdict bar, filmstrip — with no list pane.
|
||||
That is a deliberate response to the first version being rejected: a sidebar of 208 rows whose
|
||||
names share their first 60 characters is unreadable, and a text row says nothing about a sprite.
|
||||
|
||||
Three rules it must keep:
|
||||
|
||||
**Set overlay styles through the CSSOM, never an inline `style=` attribute.** The API sends
|
||||
`style-src 'self'` with no `unsafe-inline`, so the browser parses an inline style attribute and
|
||||
then *refuses to apply it*. The onion-skin and wipe sliders moved their value and changed nothing,
|
||||
with a console warning as the only evidence. `element.style.opacity = …` is not an inline style
|
||||
under CSP and works.
|
||||
|
||||
**Both sides of any comparison must use the SAME representation.** `comparable_pair` picks the
|
||||
best representation present on *both* sides. Resolving each side independently paired a decompiled
|
||||
`.gd` against `.gdc` bytecode and reported a confident 166-line deletion for a one-line edit. An
|
||||
asset is stored several times over (`original`, `decoded`, `derived`, `preview`); two of them are
|
||||
not interchangeable just because they belong to the same asset.
|
||||
|
||||
**Give any full-width grid an explicit `minmax(0, 1fr)` column.** A grid track sizes to its widest
|
||||
child, so the 2,000-frame filmstrip stretched the whole page past the viewport and took the rail,
|
||||
HUD and stage with it.
|
||||
|
||||
**A missing severity is `null`, never 0.** An added or removed asset has nothing to compare
|
||||
against. Rendering that as 0.000 would tell the operator the change is trivial when nothing has
|
||||
looked at it. `severity IS NULL` sorts last and displays as "unscored".
|
||||
|
||||
`window.__review` exposes state so a browser test can stage a known pair of artifacts. It exists
|
||||
because the archive contains no *modified* image to drive the comparison modes with — see PLAN.md
|
||||
§11 for why that is structural, not incidental.
|
||||
|
||||
## Services
|
||||
|
||||
All are `systemctl --user` units. Only port 3000 is reachable off-host.
|
||||
|
||||
```
|
||||
0.0.0.0:3000 verstack-gateway password gate (workbench/auth-gateway.mjs)
|
||||
/review*, /api/* -> 8080
|
||||
everything else -> 3001
|
||||
127.0.0.1:3001 verstack-workbench Theia
|
||||
127.0.0.1:8080 verstack-backend Rust archive API + /review (no auth of its own)
|
||||
127.0.0.1:8096 verstack-exports VM export worker
|
||||
```
|
||||
|
||||
The gateway is the only control point: the Rust API has no auth, and Theia's JSON-RPC websocket
|
||||
carries `@theia/filesystem` and `@theia/process`. That websocket bypasses Express middleware, which
|
||||
is why the gate is a front proxy rather than a Theia contribution. Sign-in sets an HttpOnly cookie
|
||||
valid for a year. Rotate with:
|
||||
|
||||
```sh
|
||||
node workbench/auth-gateway.mjs --set-password && systemctl --user restart verstack-gateway
|
||||
```
|
||||
|
||||
Transport is plain HTTP — password and cookie cross the LAN in the clear. Fine on a trusted network,
|
||||
not fine anywhere else. TLS on the gateway is the fix if that changes.
|
||||
|
||||
## Build and deploy
|
||||
|
||||
```sh
|
||||
export PATH="$HOME/.cargo/bin:$PATH"
|
||||
cargo build --release --locked && systemctl --user restart verstack-backend
|
||||
```
|
||||
|
||||
The backend takes ~6 s to become ready after restart (it reconciles a 5.9 GB catalog). Poll
|
||||
`/api/info` rather than assuming.
|
||||
|
||||
The workbench is a **separate build** and is not rebuilt by cargo. It needs **two** steps, and
|
||||
`build:browser` alone is not enough:
|
||||
|
||||
```sh
|
||||
cd workbench
|
||||
npm run prepare # tsc: stern-catalog/src -> stern-catalog/lib <-- do not skip
|
||||
npm run build:browser # bundles browser-app FROM stern-catalog/lib
|
||||
systemctl --user restart verstack-workbench
|
||||
```
|
||||
|
||||
**`npm run build:browser` only runs the bundler, and the bundler reads the extension's prebuilt
|
||||
`stern-catalog/lib/`.** Editing `stern-catalog/src/**` and running just `build:browser` prints
|
||||
`Finished with 0 errors` and ships the *previous* build — the bundler never sees your source. The
|
||||
symptom is a successful build and an unchanged UI, which reads exactly like "the change did not
|
||||
apply". Verified: `lib/browser/workspace-views.js` stayed at its old timestamp across a clean
|
||||
`build:browser`, and the new UI only appeared after `npm run prepare`.
|
||||
|
||||
`npm run prepare` is also the only step that **typechecks** — the bundler does not. A TypeScript
|
||||
error will not fail `build:browser`.
|
||||
|
||||
## The logical asset spine (Phases 1-2)
|
||||
|
||||
`src/identity.rs` defines `LogicalAssetKey`; `src/spine.rs` derives one per artifact and populates
|
||||
`assets` / `asset_observations` / `asset_names` / `edges` / `asset_deltas` (schema 10). Everything is
|
||||
recomputable from the existing catalog — no container is re-read — so after new imports:
|
||||
|
||||
```sh
|
||||
H='X-Verstack-Client: 1'
|
||||
curl -X POST http://127.0.0.1:8080/api/spine/backfill -H "$H" # ~50 s, derives keys
|
||||
curl -X POST http://127.0.0.1:8080/api/spine/classify -H "$H" # ~7 s, semantic types + animation edges
|
||||
python3 scripts/import_recovered_names.py --apply # wiki + Godot names
|
||||
tools/decoder-env/bin/python scripts/recover_radium_names.py --apply
|
||||
curl -X POST http://127.0.0.1:8080/api/spine/deltas -H "$H" -H 'Content-Type: application/json' -d '{}'
|
||||
curl -X POST http://127.0.0.1:8080/api/spine/siblings -H "$H" # release pairs worth comparing
|
||||
```
|
||||
|
||||
`GET /api/spine/siblings?version_id=<id>` answers "what should I compare this against, and why":
|
||||
`successor`, `edition`, `platform_port` and `shared_program` (the Rosetta-stone set — both releases
|
||||
have Ghidra output for the same analysed program). The table is rebuilt from scratch each run;
|
||||
nothing in it is hand-entered.
|
||||
|
||||
Order matters: keys must exist before names can attach, and names before deltas if you want the
|
||||
diff to read back named. Changing key derivation invalidates `asset_names`, so re-run the importers.
|
||||
|
||||
**Four rules the implementation had to learn the hard way.** Each was a silent wrong answer, not a
|
||||
crash, and each was only caught by dumping real rows:
|
||||
|
||||
1. Structural keys are **scoped by game and by container path**. Radium record 216 exists in every
|
||||
title, and a release ships both `image.bin` and `spike_menu/image.bin`. `version_id` includes the
|
||||
platform generation, because one release can ship as both SPIKE 2 and SPIKE 3.
|
||||
2. One asset has **several representations per release** (container original, decompressed copy,
|
||||
decoded rendering, side-car index, generated preview) and one path can appear in more than one
|
||||
snapshot of a release. `asset_observations` keys on `(lak, version_id, snapshot, path)`, and
|
||||
`representation()` falls back to `derived` — never `original`, or a decoded rendering ties with
|
||||
the raw record it came from and the diff picks between them by row order.
|
||||
3. Only **firmware content** belongs in the spine. Ghidra facts and generated previews are artifacts
|
||||
of this tool and are excluded by producing operation.
|
||||
4. Join on `entry.path` (the archived path), never on a browse row's `path` — that is a display
|
||||
label re-rooted under `Assets/`.
|
||||
|
||||
## Verification gate
|
||||
|
||||
Run all of these before claiming anything works:
|
||||
|
||||
```sh
|
||||
cargo test --locked # 12 binaries, all green
|
||||
cargo clippy --locked --all-targets -- -D warnings # currently 0 — keep it that way
|
||||
tools/decoder-env/bin/python -m unittest discover -s tests -p 'test_*.py' # 104 tests
|
||||
python3 scripts/verify_phase0.py # 73 checks, recomputes from the live archive
|
||||
```
|
||||
|
||||
`scripts/verify_phase0.py` is the useful one: it re-derives the canary diff numbers from the real
|
||||
catalog and exercises the live gateway, so it fails if a regression is real rather than cosmetic.
|
||||
`--skip-build` skips cargo for a fast pass.
|
||||
|
||||
## Function matching: uniqueness, not size
|
||||
|
||||
`compare_functions` pairs two functions when their exact body is unique on **both** sides. There is
|
||||
deliberately no minimum size — a blanket `size >= 32` floor used to exclude 22% of a real program,
|
||||
and uniqueness is the property that makes a pairing safe, not length. Verified against every
|
||||
sub-32-byte pair whose two sides carry real symbol names: 65 of 65 agree on the name, 0 disagree.
|
||||
|
||||
Do not "improve" this by accepting a non-unique body match. On the canary pair 21,486 short
|
||||
functions have *some* exact body match and only 6,056 are unique; accepting existence alone would
|
||||
invent ~15,000 arbitrary pairings that are indistinguishable from real ones in the UI.
|
||||
|
||||
Thunks stay excluded: a thunk's identity is its call target, not its body.
|
||||
|
||||
**Measure before building a comparison feature.** Three times now the plan called for a diff and
|
||||
the corpus had nothing to diff: images, sounds and fonts are all keyed by content hash for the bulk
|
||||
of their population, so a change appears as add+remove and *never* as a modification. Check
|
||||
`asset_deltas` for `change_kind='modified'` rows of that semantic type before writing the code.
|
||||
|
||||
## Two habits worth keeping
|
||||
|
||||
**A check that verifies a rule does not verify the code that ships it.** The canary check
|
||||
re-implements `logical_path` in Python and confirms it produces a point-release delta from real
|
||||
catalog rows. It passed continuously while the UI's own comparison ignored the rule entirely and
|
||||
reported 8,151 added against 1 unchanged. Drive the endpoint, not a reimplementation of it.
|
||||
|
||||
**Verify against the live system, not just offline.** Deploying the identity fix and calling the
|
||||
real endpoint revealed 20,095 rows of `metadata_only` noise that offline hash comparison never
|
||||
showed. The offline measurement was correct and still misleading.
|
||||
|
||||
**Say plainly what is a capability versus an outcome.** `plugins/godot_names.py` recovered 5,716 real
|
||||
asset names while nothing called it, and the names sat in a JSON report because no table could hold
|
||||
them. Shipping the parser was not the same as naming the assets, and describing it as the latter
|
||||
wasted the operator's time checking a UI that could not have changed. Both are wired now — the point
|
||||
stands: a recovery pass that nothing invokes has changed nothing.
|
||||
|
||||
**A backend change is not a visible change.** Phases 1 and 2 landed entirely behind new
|
||||
`/api/spine/*` endpoints the workbench never called, and the operator twice reported seeing no
|
||||
difference before the UI was joined to the data. If work is meant to be seen, wire it and drive a
|
||||
real browser to confirm — `tests/ui-*.mjs` and the Playwright setup at
|
||||
`/tmp/verstack-browser-check` do this.
|
||||
|
||||
## State of the tree
|
||||
|
||||
Branch `overhaul/phase-0`, **committed, not pushed**. There is no remote for this branch yet;
|
||||
`git log master..overhaul/phase-0` is the whole body of work. `docs/phase0-verification.md` lists
|
||||
what changed and what was deliberately left undone, and `scripts/verify_phase0.py` re-proves it
|
||||
against the live archive in about a minute.
|
||||
|
||||
Nothing here has been merged to `master`. Before merging, re-run the verification gate — several
|
||||
checks talk to the running services, so they only mean anything on this host.
|
||||
Generated
+690
-8
File diff suppressed because it is too large
Load Diff
+18
@@ -8,6 +8,7 @@ publish = false
|
||||
anyhow = "1"
|
||||
axum = { version = "0.8", features = ["multipart"] }
|
||||
blake3 = "1"
|
||||
crc32fast = "1"
|
||||
sha2 = "0.11"
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
fs2 = "0.4"
|
||||
@@ -24,6 +25,23 @@ uuid = { version = "1", features = ["v4", "serde"] }
|
||||
walkdir = "2"
|
||||
rusqlite = { version = "0.38", features = ["bundled"] }
|
||||
fast-flirt = "=0.2.2"
|
||||
image = { version = "0.25.10", default-features = false, features = ["png", "webp", "jpeg", "gif", "bmp"] }
|
||||
rustfft = "6.4.1"
|
||||
ttf-parser = "0.25"
|
||||
linkme = "0.3"
|
||||
rmcp = { version = "3.4.0", features = ["server", "transport-io"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json"] }
|
||||
base64 = "0.22"
|
||||
goblin = "0.10"
|
||||
ext4-view = { version = "=1.0.0", features = ["std"] }
|
||||
flate2 = "1.1.10"
|
||||
md-5 = "0.10"
|
||||
verstack-spk = { path = "crates/verstack-spk" }
|
||||
verstack-radium = { path = "crates/verstack-radium" }
|
||||
verstack-luks = { path = "crates/verstack-luks" }
|
||||
zeroize = "1.9"
|
||||
verstack-code = { path = "crates/verstack-code" }
|
||||
|
||||
[dev-dependencies]
|
||||
image = { version = "0.25.10", default-features = false, features = ["png"] }
|
||||
tower = { version = "0.5", features = ["util"] }
|
||||
|
||||
@@ -21,7 +21,7 @@ cargo build --locked --release
|
||||
./target/release/verstack --config config.json serve
|
||||
```
|
||||
|
||||
Use the optimized build for real firmware; debug builds make archive restoration substantially slower. For the native Explorer, asset editors and release comparisons, start the [Theia workbench](workbench/README.md) on port 3000.
|
||||
Use the optimized build for real firmware; debug builds make archive restoration substantially slower. For the native Explorer, asset editors and release comparisons, start the [Theia workbench](workbench/README.md), reached on port 3000 through the password gateway.
|
||||
|
||||
The compatibility client remains at **http://127.0.0.1:8080**. It includes filesystem or streamed single-file upload imports, snapshot/asset browsing, safe media previews, text/hex inspection, file comparisons, plugin execution, function comparisons, downloads, and polling run status. It uses a Bootstrap-era utility layout with local CSS/JavaScript and no frontend build requirement.
|
||||
|
||||
@@ -33,7 +33,7 @@ Edit `config.json` before using real data:
|
||||
- `require_ram_workspace`: defaults to true. Startup and processing reject disk-backed scratch or insufficient free space; there is no disk fallback. Child tools inherit RAM scratch and cache locations.
|
||||
- `workspace_bytes`: the scratch allowance must fit available RAM filesystem space (live configuration: 64 GiB). Usage is polled, not a kernel quota.
|
||||
- `processing_enabled`: pauses automatic submission and queue consumption when false, preserving job history. The live service and example configuration are paused. Explicit CLI/API processing and user-selected import workflows remain available.
|
||||
- `bind`: the Rust listener defaults to `127.0.0.1:8080`. For the LAN workbench, keep this on loopback and start Theia with `--hostname 0.0.0.0 --port 3000`; its same-origin proxy connects to Rust.
|
||||
- `bind`: the Rust listener defaults to `127.0.0.1:8080` and must stay on loopback: it has no authentication. LAN access goes through the password gateway (`workbench/auth-gateway.mjs`) on port 3000, which proxies to Theia on loopback 3001, which proxies `/api` to Rust. See [workbench access](workbench/README.md).
|
||||
- `plugins`: local command argument arrays and versioned settings. Use absolute paths for plugin scripts. The example uses this workspace's path; change it if you move the project.
|
||||
|
||||
The Theia **Import release** view browses server files/folders or URL sources. Paste an Internet Archive item link, click **Browse URL**, choose a file, and supply the game, edition and version. Extraction and asset previews are selected by default; code analysis is optional. **Jobs** shows live progress, cancellation, retry and an **Open release** action. Supported containers include ZIP, SPK, partitioned SD-card images, ext/FAT filesystems and configured LUKS volumes; other files remain cataloged. Downloads and scratch work use RAM, including sparse zero regions in disk images.
|
||||
@@ -69,7 +69,8 @@ loginctl enable-linger "$USER"
|
||||
|
||||
The units assume the checkout is at `~/verstack` and use `config.json`. Stop any
|
||||
manually launched instances first, allowing active processing to finish. Theia
|
||||
listens on LAN port 3000; configure Rust on loopback port 8080. Lingering starts
|
||||
is reached on LAN port 3000 through the password gateway; Theia itself and Rust stay on
|
||||
loopback (3001 and 8080). Lingering starts
|
||||
the services at boot and keeps them running after logout. Inspect status and logs
|
||||
with `systemctl --user status verstack-backend verstack-workbench` and
|
||||
`journalctl --user -u verstack-backend -u verstack-workbench`.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
/target/
|
||||
Generated
+289
@@ -0,0 +1,289 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "anyhow"
|
||||
version = "1.0.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
||||
|
||||
[[package]]
|
||||
name = "block-buffer"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "capstone"
|
||||
version = "0.14.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f442ae0f2f3f1b923334b4a5386c95c69c1cfa072bafa23d6fae6d9682eb1dd4"
|
||||
dependencies = [
|
||||
"capstone-sys",
|
||||
"static_assertions",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "capstone-sys"
|
||||
version = "0.18.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4e8087cab6731295f5a2a2bd82989ba4f41d3a428aab2e7c98d8f4db38aac05"
|
||||
dependencies = [
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.4.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"shlex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "const-oid"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||
dependencies = [
|
||||
"block-buffer",
|
||||
"const-oid",
|
||||
"crypto-common",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
|
||||
|
||||
[[package]]
|
||||
name = "goblin"
|
||||
version = "0.10.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "17582616a7718cca54cec18e534a76c7c4aec11a8b9a85695712f262fd15a4c8"
|
||||
dependencies = [
|
||||
"log",
|
||||
"plain",
|
||||
"scroll",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hybrid-array"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "libc"
|
||||
version = "0.2.189"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
|
||||
|
||||
[[package]]
|
||||
name = "log"
|
||||
version = "0.4.34"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
|
||||
|
||||
[[package]]
|
||||
name = "plain"
|
||||
version = "0.2.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.47"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "scroll"
|
||||
version = "0.13.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c1257cd4248b4132760d6524d6dda4e053bc648c9070b960929bf50cfb1e7add"
|
||||
dependencies = [
|
||||
"scroll_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "scroll_derive"
|
||||
version = "0.13.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e1a36a382ed65dbcc0ab47fd5e9a94112417ccd34560a392ef3b7b0f0ec39148"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.151"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "shlex"
|
||||
version = "2.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
|
||||
|
||||
[[package]]
|
||||
name = "static_assertions"
|
||||
version = "1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.25"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ab72a15cf68d77cb0987d3684aa8a45c5ef827e8cb49ee2f30bfd7ba2feb519f"
|
||||
|
||||
[[package]]
|
||||
name = "verstack-code"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"capstone",
|
||||
"goblin",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
@@ -0,0 +1,17 @@
|
||||
[package]
|
||||
name = "verstack-code"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
license = "MIT"
|
||||
description = "Verified ARM function normalization and conservative CFG evidence"
|
||||
|
||||
[workspace]
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
capstone = { version = "=0.14.0", default-features = false, features = ["std", "full", "arch_arm", "arch_arm64"] }
|
||||
goblin = "=0.10.7"
|
||||
sha2 = "0.11"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
@@ -0,0 +1,55 @@
|
||||
# verstack-code
|
||||
|
||||
A standalone native A32/AArch64 normalization and control-flow evidence crate. It is intentionally outside the application workspace: its pinned decoder, lockfile, tests and CLI can be reviewed without changing the deployed application. It does not mutate an archive, start Ghidra, recover arbitrary stripped-function boundaries, match functions automatically, or propagate names.
|
||||
|
||||
## Inputs and trust boundary
|
||||
|
||||
`analyze(&Input)` requires a nonempty, bounded function body, its exact SHA-256, address, explicit instruction mode, and boundary provenance. Its caller establishes that the bytes are a complete contiguous function; a SHA-256 alone cannot establish a boundary. `ElfImage` additionally verifies the full ELF fingerprint, architecture, little-endian EXEC/DYN format, and a unique executable file-backed mapping for the whole body. Sized ELF symbols supply native extents; conflicting sizes/modes at one start are omitted. Saved Ghidra extents are accepted only with the pinned exporter's contiguous-body signature marker and a matching full on-disk body hash. ARM mode must come from a sized symbol or `$a`/`$t` mapping symbol; the ARM language string does not prove TMode.
|
||||
|
||||
Thumb is explicitly unsupported and returns no normalized/CFG hashes. Big-endian normalization, mode changes inside one extent, inferred ends from adjacent symbols, rebased/relocated Ghidra memory, unwind boundary recovery, and jump tables remain future work. No code inventory is presented as complete. Known data/padding or undecoded/unreachable bytes commonly prevent complete CFG evidence; a successful byte decode does not itself prove that all bytes are instructions.
|
||||
|
||||
Limits: 1 MiB/function, 16,384 basic blocks, 256 MiB ELF and aggregate mapped bytes, 4,096 load headers, 500,000 ELF symbols/functions, and 1,024 bytes/symbol name. The offline saved-facts input cap is 512 MiB. These are input limits, not promises about peak process memory. Each function is decoded with Capstone 0.14.0 / capstone-sys 0.18.0 (vendored Capstone 5.0.6), pinned in Cargo.lock, compiled only for ARM and ARM64 with detailed register/operand information. Normalization coverage is deliberately narrower than Capstone's instruction coverage.
|
||||
|
||||
## Evidence
|
||||
|
||||
- BL/BLX immediate targets and external direct branch targets use **bit masks** retaining instruction class, condition, registers and widths. Indirect calls retain their register and an explicitly unresolved target. Internal direct targets remain in the body fingerprint.
|
||||
- A32 immediate PC-relative LDR/LDRB and A64 literal loads mask displacement bits only when the target maps to available original bytes. The exact loaded bytes have a separate SHA-256; they are never wildcarded as arbitrary constants.
|
||||
- Adjacent unconditional A32 MOVW/MOVT and A64 ADRP/ADD pairs mask address fields only when their register flow and mapped target agree and either an explicit instruction relocation proof exists or the next instruction actually uses that register as a memory base. Merely landing numerically in a mapped region is insufficient. Arithmetic constants, mismatched registers, conditional pairs and unproven address uses stay unchanged. The ELF adapter currently supplies **no instruction relocation proofs**; architecture-specific relocation-type mapping is still needed. Nonadjacent address construction is not analyzed.
|
||||
- Basic blocks have instruction hashes and directed, labelled taken/fallthrough/jump/return/trap edges. The instruction-labelled CFG hash uses three Weisfeiler-Lehman rounds, incoming/outgoing edges, entry color and node multiplicity. Internal branch displacements are removed from block labels, while their actual edges preserve topology. The shape-only hash deliberately discards instruction semantics and must never establish a match by itself.
|
||||
- Full hashes require complete decoding, resolved intraprocedural control flow and full reachability of the supplied extent. Indirect branches, undecoded tails, unknown fallthrough past the extent and unclassified/unreachable bytes leave hashes absent. Calls are syntactic: exception edges and nonreturning-callee analysis are not inferred.
|
||||
|
||||
`masked_body_sha256` and both WL hashes are **candidate evidence**, not equivalence claims. `reference_bound_sha256` additionally binds each external reference to its supplied symbol/import identity or the actual loaded literal bytes; an unresolved indirect call or address reference leaves it absent. Literal bytes take precedence over symbolic labels. Symbol equality does not prove unchanged callee implementation. This reference-bound fingerprint is still only evidence for a future matcher that enforces bidirectional uniqueness, verifies anchors, and grades changes. Neither an empty graph nor size alone adds confidence.
|
||||
|
||||
## Commands
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
cargo test --manifest-path crates/verstack-code/Cargo.toml --locked --offline
|
||||
cargo clippy --manifest-path crates/verstack-code/Cargo.toml --locked --offline --all-targets -- -D warnings
|
||||
cargo run --manifest-path crates/verstack-code/Cargo.toml --locked --offline -- /path/to/game --symbols output.json
|
||||
cargo run --manifest-path crates/verstack-code/Cargo.toml --locked --offline -- /path/to/game /path/to/functions.json output.json
|
||||
python3 crates/verstack-code/scripts/compare_evidence.py baseline.json target.json > comparison.json
|
||||
python3 crates/verstack-code/scripts/check_ghidra_oracle.py output.json /path/to/functions.json > oracle.json
|
||||
```
|
||||
|
||||
The latter audit opens only existing saved detail JSON. It never runs Ghidra. The CLI checks actual input/body fingerprints and emits coverage, masks, references and function-level hashes. It does not write the application catalog. Reports from incomplete saved analyses are rejected. The `--symbols` path remains useful even when older Ghidra exports lack contiguous-body proof.
|
||||
|
||||
## Corpus rehearsal, 2026-09-16
|
||||
|
||||
Firmware binaries and full generated reports remain in ignored validation storage. [corpus-evidence.json](corpus-evidence.json) records exact input fingerprints, decoder metadata, denominators, residual results and the oracle audit.
|
||||
|
||||
| Pair / native sized-symbol cohort | Verified bodies | Complete CFGs | Unique exact pairs | Additional reference-bound pairs |
|
||||
| --- | --- | --- | ---: | ---: |
|
||||
| Game of Thrones Pro/LE ARM32 | 11,519 / 11,694 | 4,594 / 4,661 | 3,308 | 612 |
|
||||
| Pokémon LE 0.85/0.86 AArch64 | 262 / 263 | 241 / 241 | 139 | 0 |
|
||||
|
||||
Every exact and reference-bound pair in these measured cohorts had agreeing ELF names: 0 disagreements. These are cohorts with known native boundaries, not a whole-corpus recovery rate. The richer reference-bound key also disambiguates some otherwise repeated masked bodies, so its count can exceed the unique unbound count.
|
||||
|
||||
The ARM32 unbound masked/WL residual produced 577 unique candidates, **one with disagreeing names**: `_ZL21should_post_be_raisedv` versus `_Z26should_send_ball_to_thronev`. Their direct calls agree but both also invoke an unresolved register target. The generic unresolved-reference rule excludes both from reference-bound matching; no symbol-name special case is used. The two 76-byte verified bodies and source identities form a reproducible regression fixture. On AArch64, 15 unbound masked/WL candidates had agreeing names, but none had sufficient resolved reference evidence for the stricter fingerprint. ADRP/ADD argument-address sequences without local memory-use proof remain unchanged rather than overstating coverage.
|
||||
|
||||
Against saved Ghidra 12.1.3 facts from the identical ARM32 input, 4,769 functions had the same complete body hash and a complete native decode. All 8,238 Ghidra direct/tail-call targets were accounted for, with 0 missing or conflicting targets. Of these, 1,152 were encoded as tail branches and are explicitly distinguished by the native pass. Another 209 external branches are reported as branches, without inventing Ghidra call semantics.
|
||||
|
||||
The existing spike-symbol-bridge results were inspected as a semantic oracle/design constraint: its 263 high-confidence ARM32→ARM64 pairs depend on distinctive strings and named call-graph context, not opcode hashes or function size. This crate does not reproduce or claim those cross-ISA matches. Future integration needs string/import anchors, nonadjacent address data flow, Thumb, stripped-binary boundary recovery, independently resolved callees, fixed-point propagation and graded code deltas. The application codesig endpoint and schema are unchanged.
|
||||
|
||||
Decoder API references: [Capstone Rust architecture details](https://docs.rs/capstone/0.14.0/capstone/arch/index.html) and [Capstone detailed instruction API](https://www.capstone-engine.org/lang_c.html). No whole-instruction wildcard fallback from Ghidra's older FLIRT export is copied into this implementation.
|
||||
@@ -0,0 +1,197 @@
|
||||
{
|
||||
"arm32": {
|
||||
"method": "native-code-candidate-audit/1",
|
||||
"left": {
|
||||
"boundary_provenance": "sized_elf_symbols",
|
||||
"caveat": "Candidate signatures only. No automatic function matching, name propagation or complete boundary recovery. Thumb and cross-ISA matching unsupported.",
|
||||
"complete_cfg": 4594,
|
||||
"decoder": "capstone-rs 0.14.0 / capstone-sys 0.18.0",
|
||||
"failures": {},
|
||||
"input_records": 11519,
|
||||
"input_sha256": "637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"method": "verstack-code/1",
|
||||
"mode": "arm",
|
||||
"verified_bodies": 11519,
|
||||
"mask_reasons": {
|
||||
"bl_target": 46803,
|
||||
"pc_literal_displacement": 26829,
|
||||
"external_branch_target": 3408
|
||||
},
|
||||
"coverage_diagnostics": {
|
||||
"elf_data_mapping_in_function_extent": 6571,
|
||||
"incomplete_control_flow": 6806,
|
||||
"undecoded_tail": 1151,
|
||||
"unreachable_or_unclassified_bytes": 6685
|
||||
}
|
||||
},
|
||||
"right": {
|
||||
"boundary_provenance": "sized_elf_symbols",
|
||||
"caveat": "Candidate signatures only. No automatic function matching, name propagation or complete boundary recovery. Thumb and cross-ISA matching unsupported.",
|
||||
"complete_cfg": 4661,
|
||||
"decoder": "capstone-rs 0.14.0 / capstone-sys 0.18.0",
|
||||
"failures": {},
|
||||
"input_records": 11694,
|
||||
"input_sha256": "240db7bed730b8958c8d1d34e160e296d2306fda22533e5f1461df389e65d410",
|
||||
"method": "verstack-code/1",
|
||||
"mode": "arm",
|
||||
"verified_bodies": 11694,
|
||||
"mask_reasons": {
|
||||
"bl_target": 47518,
|
||||
"pc_literal_displacement": 27293,
|
||||
"external_branch_target": 3480
|
||||
},
|
||||
"coverage_diagnostics": {
|
||||
"elf_data_mapping_in_function_extent": 6678,
|
||||
"incomplete_control_flow": 6911,
|
||||
"undecoded_tail": 1182,
|
||||
"unreachable_or_unclassified_bytes": 6788
|
||||
}
|
||||
},
|
||||
"exact": {
|
||||
"unique_pairs": 3308,
|
||||
"both_named": 3308,
|
||||
"names_agree": 3308,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"residual": {
|
||||
"masked_body_sha256": {
|
||||
"unique_pairs": 577,
|
||||
"both_named": 577,
|
||||
"names_agree": 576,
|
||||
"names_disagree": 1,
|
||||
"disagreement_examples": [
|
||||
{
|
||||
"left": "_ZL21should_post_be_raisedv",
|
||||
"right": "_Z26should_send_ball_to_thronev",
|
||||
"left_address": "4f3ec",
|
||||
"right_address": "207a4"
|
||||
}
|
||||
]
|
||||
},
|
||||
"reference_bound_sha256": {
|
||||
"unique_pairs": 612,
|
||||
"both_named": 612,
|
||||
"names_agree": 612,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"cfg_wl_sha256": {
|
||||
"unique_pairs": 577,
|
||||
"both_named": 577,
|
||||
"names_agree": 576,
|
||||
"names_disagree": 1,
|
||||
"disagreement_examples": [
|
||||
{
|
||||
"left": "_ZL21should_post_be_raisedv",
|
||||
"right": "_Z26should_send_ball_to_thronev",
|
||||
"left_address": "4f3ec",
|
||||
"right_address": "207a4"
|
||||
}
|
||||
]
|
||||
},
|
||||
"cfg_shape_wl_sha256": {
|
||||
"unique_pairs": 159,
|
||||
"both_named": 159,
|
||||
"names_agree": 159,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
}
|
||||
},
|
||||
"caveat": "Residual methods are independently measured over the exact residual, not additive. Unique normalized/CFG candidates are not automatic semantic matches or cross-ISA evidence."
|
||||
},
|
||||
"aarch64": {
|
||||
"method": "native-code-candidate-audit/1",
|
||||
"left": {
|
||||
"boundary_provenance": "sized_elf_symbols",
|
||||
"caveat": "Candidate signatures only. No automatic function matching, name propagation or complete boundary recovery. Thumb and cross-ISA matching unsupported.",
|
||||
"complete_cfg": 241,
|
||||
"decoder": "capstone-rs 0.14.0 / capstone-sys 0.18.0",
|
||||
"failures": {},
|
||||
"input_records": 262,
|
||||
"input_sha256": "5f8bd90b23c35d15252315b2003fb99d0c5950bca2017a085db49f7b52862293",
|
||||
"method": "verstack-code/1",
|
||||
"mode": "aarch64",
|
||||
"verified_bodies": 262,
|
||||
"mask_reasons": {
|
||||
"bl_target": 574,
|
||||
"external_branch_target": 78
|
||||
},
|
||||
"coverage_diagnostics": {
|
||||
"adrp_add_unproven_address_retained": 171,
|
||||
"incomplete_control_flow": 21
|
||||
}
|
||||
},
|
||||
"right": {
|
||||
"boundary_provenance": "sized_elf_symbols",
|
||||
"caveat": "Candidate signatures only. No automatic function matching, name propagation or complete boundary recovery. Thumb and cross-ISA matching unsupported.",
|
||||
"complete_cfg": 241,
|
||||
"decoder": "capstone-rs 0.14.0 / capstone-sys 0.18.0",
|
||||
"failures": {},
|
||||
"input_records": 263,
|
||||
"input_sha256": "b1ef1b5585141a24f9997babff0738851807680439954a61b6e491ca7e76d87d",
|
||||
"method": "verstack-code/1",
|
||||
"mode": "aarch64",
|
||||
"verified_bodies": 263,
|
||||
"mask_reasons": {
|
||||
"bl_target": 581,
|
||||
"external_branch_target": 78
|
||||
},
|
||||
"coverage_diagnostics": {
|
||||
"adrp_add_unproven_address_retained": 172,
|
||||
"incomplete_control_flow": 22
|
||||
}
|
||||
},
|
||||
"exact": {
|
||||
"unique_pairs": 139,
|
||||
"both_named": 139,
|
||||
"names_agree": 139,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"residual": {
|
||||
"masked_body_sha256": {
|
||||
"unique_pairs": 15,
|
||||
"both_named": 15,
|
||||
"names_agree": 15,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"reference_bound_sha256": {
|
||||
"unique_pairs": 0,
|
||||
"both_named": 0,
|
||||
"names_agree": 0,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"cfg_wl_sha256": {
|
||||
"unique_pairs": 15,
|
||||
"both_named": 15,
|
||||
"names_agree": 15,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
},
|
||||
"cfg_shape_wl_sha256": {
|
||||
"unique_pairs": 66,
|
||||
"both_named": 66,
|
||||
"names_agree": 66,
|
||||
"names_disagree": 0,
|
||||
"disagreement_examples": []
|
||||
}
|
||||
},
|
||||
"caveat": "Residual methods are independently measured over the exact residual, not additive. Unique normalized/CFG candidates are not automatic semantic matches or cross-ISA evidence."
|
||||
},
|
||||
"ghidra_call_oracle": {
|
||||
"input_sha256": "637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"ghidra_version": "12.1.3",
|
||||
"verified_equal_bodies": 4769,
|
||||
"complete_decode_functions_checked": 4769,
|
||||
"saved_detail_missing": 0,
|
||||
"direct_calls_checked": 8238,
|
||||
"ghidra_calls_recovered_as_explicit_native_tail_branches": 1152,
|
||||
"additional_native_tail_branches": 209,
|
||||
"disagreements": 0,
|
||||
"examples": []
|
||||
},
|
||||
"decoder_engine": "Capstone5.0.6 (vendored in capstone-sys0.18.0)"
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Independently check report addresses/bytes and import slots against ELF + readelf.
|
||||
Usage: check_anchor_bytes.py ELF REPORT.json SUMMARY.json
|
||||
Only the reference patterns encountered in the recorded GOT/Pokemon cohorts are
|
||||
accepted; a new pattern needs its own independent verifier rather than a pass.
|
||||
"""
|
||||
import collections
|
||||
import hashlib
|
||||
import json
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
elf_path, report_path, output = sys.argv[1:]
|
||||
raw = open(elf_path, 'rb').read()
|
||||
report = json.load(open(report_path))
|
||||
assert hashlib.sha256(raw).hexdigest() == report['input_sha256']
|
||||
assert raw[:4] == b'\x7fELF' and raw[5] == 1
|
||||
is64 = raw[4] == 2
|
||||
phoff = struct.unpack_from('<Q' if is64 else '<I', raw, 32 if is64 else 28)[0]
|
||||
entsize, count = struct.unpack_from('<HH', raw, 54 if is64 else 42)
|
||||
segments = []
|
||||
for n in range(count):
|
||||
p = phoff + n * entsize
|
||||
if struct.unpack_from('<I', raw, p)[0] != 1:
|
||||
continue
|
||||
if is64:
|
||||
offset, address, _, size = struct.unpack_from('<QQQQ', raw, p + 8)
|
||||
else:
|
||||
offset, address, _, size = struct.unpack_from('<IIII', raw, p + 4)
|
||||
segments.append((address, offset, size))
|
||||
def read(address, size):
|
||||
choices = [raw[o + address-a:o + address-a + size] for a,o,n in segments if a <= address and address+size <= a+n]
|
||||
assert choices and all(c == choices[0] for c in choices)
|
||||
return choices[0]
|
||||
def word(address):
|
||||
return int.from_bytes(read(address, 4), 'little')
|
||||
def sign(v,b):
|
||||
return v-(1<<b) if v & (1<<(b-1)) else v
|
||||
relocations = {}
|
||||
for line in subprocess.check_output(['readelf','-rW',elf_path], text=True).splitlines():
|
||||
fields = line.split()
|
||||
if len(fields)>=5 and fields[2] in ('R_AARCH64_GLOB_DAT','R_AARCH64_JUMP_SLOT','R_ARM_GLOB_DAT','R_ARM_JUMP_SLOT'):
|
||||
relocations[int(fields[0],16)] = fields[4].split('@')[0]
|
||||
counts = collections.Counter()
|
||||
examples = []
|
||||
for function in report['functions']:
|
||||
base = int(function['address'],16)
|
||||
assert hashlib.sha256(read(base,function['size'])).hexdigest() == function['body_sha256']
|
||||
for anchor in function['anchor_evidence']['anchors']:
|
||||
counts[anchor['kind']] += 1
|
||||
pc = base + anchor['instruction_offset']
|
||||
w = word(pc)
|
||||
proof = anchor['proof']
|
||||
address = anchor['address']
|
||||
if proof.startswith('aarch64_adrp_'):
|
||||
assert w & 0x9f000000 == 0x90000000
|
||||
page = (pc & ~4095) + sign((((w>>5)&0x7ffff)<<2)|((w>>29)&3),21)*4096
|
||||
n = word(pc+4)
|
||||
assert (n>>5)&31 == w&31
|
||||
if proof == 'aarch64_adrp_add_address':
|
||||
assert n&0xff800000 == 0x91000000 and n&31 == w&31
|
||||
target = page + (((n>>10)&0xfff) << (12 if n&(1<<22) else 0))
|
||||
else:
|
||||
assert n&0xffc00000 == 0xf9400000
|
||||
target = page + ((n>>10)&0xfff)*8
|
||||
elif proof.startswith('literal_'):
|
||||
assert not is64 and w & 0x0f3f0000 == 0x051f0000
|
||||
target = pc+8 + (w&0xfff)*(1 if w&(1<<23) else -1)
|
||||
if proof == 'literal_pointer_used_as_memory_base':
|
||||
target = word(target)
|
||||
n = word(pc+4)
|
||||
assert w>>28 == 14 and n>>28 == 14
|
||||
assert (n>>26)&3 == 1 and (n>>16)&15 == (w>>12)&15
|
||||
else:
|
||||
raise AssertionError('unverified reference pattern: '+proof)
|
||||
assert address == target
|
||||
if anchor['kind'] == 'string':
|
||||
value = anchor['value'].encode()
|
||||
assert read(address,len(value)+1) == value+b'\0'
|
||||
elif anchor['kind'] == 'literal_constant':
|
||||
value = bytes.fromhex(anchor['value'])
|
||||
assert read(address,len(value)) == value
|
||||
else:
|
||||
assert anchor['kind'] == 'import_slot'
|
||||
assert relocations[address] == anchor['value']
|
||||
value = None
|
||||
if value is not None:
|
||||
assert hashlib.sha256(value).hexdigest() == anchor['bytes_sha256']
|
||||
if anchor['kind'] != 'literal_constant' and len(examples)<6:
|
||||
examples.append({'function':function['name'],'function_address':function['address'],**anchor})
|
||||
summary = {'input_sha256':report['input_sha256'],'verified_functions':len(report['functions']),'anchors':dict(counts),'independent_byte_address_and_relocation_verification':True,'truncated_functions':sum(f['anchor_evidence']['truncated'] for f in report['functions']),'examples':examples}
|
||||
open(output,'w').write(json.dumps(summary,indent=2)+'\n')
|
||||
print(json.dumps(summary,indent=2))
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only call-target audit against already saved Ghidra detail files; no Ghidra run."""
|
||||
import json,pathlib,sys
|
||||
native=json.load(open(sys.argv[1]));path=pathlib.Path(sys.argv[2]);facts=json.load(open(path));assert facts['input_sha256']==native['input_sha256']
|
||||
lookup={int(f['address'],16):f for f in facts['functions']};checked=0;direct_calls=0;different=[];same_bytes=0;missing=0;tail_calls=0;additional_tails=0
|
||||
for f in native['functions']:
|
||||
address=int(f['address'],16);g=lookup.get(address)
|
||||
if not g or g['body_sha256']!=f['body_sha256'] or g['size']!=f['size']:continue
|
||||
same_bytes+=1;code=path.parent/g['code_path']
|
||||
if not code.exists():missing+=1;continue
|
||||
detail=json.load(open(code));expected={(int(r['from'],16)-address,int(r['to'],16)) for r in detail['references'] if r['type'] in ['UNCONDITIONAL_CALL','CONDITIONAL_CALL']}
|
||||
actual={(r['offset'],r['target']) for r in f['references'] if r['kind']=='call' and r['target'] is not None}
|
||||
tails={(r['offset'],r['target']) for r in f['references'] if r['kind']=='tail_branch' and r['target'] is not None}
|
||||
# A prefix decoder must not imply complete call coverage.
|
||||
if not f['complete_decode']:continue
|
||||
checked+=1;direct_calls+=len(expected)
|
||||
tail_calls+=len((expected-actual)&tails);additional_tails+=len(tails-expected)
|
||||
uncovered=expected-(actual|tails)
|
||||
if (actual-expected) or uncovered:different.append({'address':f['address'],'name':f['name'],'only_native':sorted(actual-expected),'only_ghidra':sorted(uncovered)})
|
||||
print(json.dumps({'input_sha256':native['input_sha256'],'ghidra_version':facts['ghidra_version'],'verified_equal_bodies':same_bytes,'complete_decode_functions_checked':checked,'saved_detail_missing':missing,'direct_calls_checked':direct_calls,'ghidra_calls_recovered_as_explicit_native_tail_branches':tail_calls,'additional_native_tail_branches':additional_tails,'disagreements':len(different),'examples':different[:30]},indent=2))
|
||||
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare independently verified reports; candidates are never applied as labels."""
|
||||
import collections,json,sys
|
||||
left,right=[json.load(open(p)) for p in sys.argv[1:3]]
|
||||
a,b=left['functions'],right['functions']
|
||||
def unique(field,used_a=set(),used_b=set()):
|
||||
def index(rows,used):
|
||||
d=collections.defaultdict(list)
|
||||
for n,row in enumerate(rows):
|
||||
if n not in used and row.get(field): d[row[field]].append(n)
|
||||
return d
|
||||
x,y=index(a,used_a),index(b,used_b)
|
||||
return [(v[0],y[k][0]) for k,v in x.items() if len(v)==1 and k in y and len(y[k])==1]
|
||||
def report(pairs):
|
||||
named=[(a[x],b[y]) for x,y in pairs if a[x].get('symbol_source') not in ('DEFAULT','',None) and b[y].get('symbol_source') not in ('DEFAULT','',None)]
|
||||
bad=[{'left':x['name'],'right':y['name'],'left_address':x['address'],'right_address':y['address']} for x,y in named if x['name']!=y['name']]
|
||||
return {'unique_pairs':len(pairs),'both_named':len(named),'names_agree':len(named)-len(bad),'names_disagree':len(bad),'disagreement_examples':bad[:20]}
|
||||
exact=unique('body_sha256');used_a={x for x,y in exact};used_b={y for x,y in exact}
|
||||
out={'method':'native-code-candidate-audit/1','left':{k:v for k,v in left.items() if k!='functions'},'right':{k:v for k,v in right.items() if k!='functions'},'exact':report(exact),'residual':{}}
|
||||
for field in ['masked_body_sha256','reference_bound_sha256','cfg_wl_sha256','cfg_shape_wl_sha256']:
|
||||
out['residual'][field]=report(unique(field,used_a,used_b))
|
||||
# Independent semantic changes: same named function changing must remain observable.
|
||||
for label,data in [('left',left),('right',right)]:
|
||||
out[label]['mask_reasons']=dict(collections.Counter(m['reason'] for f in data['functions'] for m in f['masks']))
|
||||
out[label]['coverage_diagnostics']=dict(sum((collections.Counter(f['diagnostics']) for f in data['functions']),collections.Counter()))
|
||||
out['caveat']='Residual methods are independently measured over the exact residual, not additive. Unique normalized/CFG candidates are not automatic semantic matches or cross-ISA evidence.'
|
||||
print(json.dumps(out,indent=2))
|
||||
@@ -0,0 +1,409 @@
|
||||
//! Bounded evidence anchors. These never change normalization or confer a function name.
|
||||
use crate::*;
|
||||
pub const METHOD: &str = "verstack-code-anchors/1";
|
||||
pub const MAX_ANCHORS: usize = 512;
|
||||
pub const MAX_STRING_BYTES: usize = 1024;
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Anchor {
|
||||
pub instruction_offset: u64,
|
||||
pub address: u64,
|
||||
pub kind: String,
|
||||
pub value: String,
|
||||
pub bytes_sha256: Option<String>,
|
||||
pub proof: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Report {
|
||||
pub method: String,
|
||||
pub input_sha256: String,
|
||||
pub body_sha256: String,
|
||||
pub anchors: Vec<Anchor>,
|
||||
pub truncated: bool,
|
||||
pub complete_decode: bool,
|
||||
pub coverage: String,
|
||||
}
|
||||
fn signed(v: u64, bits: u32) -> i64 {
|
||||
((v << (64 - bits)) as i64) >> (64 - bits)
|
||||
}
|
||||
fn text_at(input: &Input, address: u64) -> Option<String> {
|
||||
let first = input.memory.iter().find_map(|m| {
|
||||
let offset = usize::try_from(address.checked_sub(m.start)?).ok()?;
|
||||
m.bytes.get(offset..).filter(|b| !b.is_empty())
|
||||
})?;
|
||||
let stop = first
|
||||
.iter()
|
||||
.take(MAX_STRING_BYTES + 1)
|
||||
.position(|b| *b == 0)?;
|
||||
if stop < 4 {
|
||||
return None;
|
||||
}
|
||||
let end = address.checked_add(stop as u64 + 1)?;
|
||||
// Verify every overlapping file mapping, including partial conflicting aliases.
|
||||
for m in &input.memory {
|
||||
let start = address.max(m.start);
|
||||
let stop = end.min(m.start.checked_add(m.bytes.len() as u64)?);
|
||||
if start < stop
|
||||
&& first[(start - address) as usize..(stop - address) as usize]
|
||||
!= m.bytes[(start - m.start) as usize..(stop - m.start) as usize]
|
||||
{
|
||||
return None;
|
||||
}
|
||||
}
|
||||
let text = std::str::from_utf8(&first[..stop]).ok()?;
|
||||
text.chars()
|
||||
.all(|c| !c.is_control() || matches!(c, '\n' | '\r' | '\t'))
|
||||
.then(|| text.to_owned())
|
||||
}
|
||||
pub(crate) fn extract(
|
||||
input: &Input,
|
||||
e: &Evidence,
|
||||
input_sha256: &str,
|
||||
imports: &BTreeMap<u64, String>,
|
||||
data: &[(u64, u64)],
|
||||
) -> Report {
|
||||
let mut report = Report { method: METHOD.into(), input_sha256:input_sha256.into(), body_sha256:e.body_sha256.clone(), anchors:vec![], truncated:false,complete_decode:e.complete_decode,coverage:"Partial: proven direct address references and literal bytes only; no arbitrary immediate pointer guesses, multi-hop pointer chasing, PLT guesses, general register dataflow, Thumb, or complete import/constant inventory. Strings are referenced printable NUL-terminated UTF-8 bytes, not established semantic identities.".into() };
|
||||
report.truncated = e.references.len() > 4096 || e.instructions.len() > 4096;
|
||||
let mut targets = BTreeSet::new();
|
||||
for r in e.references.iter().take(4096) {
|
||||
let Some(target) = r.target else { continue };
|
||||
if r.kind == "address_materialization" {
|
||||
targets.insert((r.offset, target, "proven_address_materialization"));
|
||||
}
|
||||
if r.kind == "literal_load" {
|
||||
let i = &e.instructions[(r.offset / 4) as usize];
|
||||
let width = if input.mode == Mode::Arm {
|
||||
if i.word & (1 << 22) != 0 { 1 } else { 4 }
|
||||
} else {
|
||||
match (i.word >> 26 & 1, i.word >> 30) {
|
||||
(0, 0) | (0, 2) | (1, 0) => 4,
|
||||
(0, 1) | (1, 1) => 8,
|
||||
(1, 2) => 16,
|
||||
_ => 0,
|
||||
}
|
||||
};
|
||||
if width > 0
|
||||
&& let Some(bytes) = memory(input, target, width)
|
||||
{
|
||||
report.anchors.push(Anchor {
|
||||
instruction_offset: r.offset,
|
||||
address: target,
|
||||
kind: "literal_constant".into(),
|
||||
value: bytes.iter().map(|b| format!("{b:02x}")).collect(),
|
||||
bytes_sha256: Some(sha256(bytes)),
|
||||
proof: format!("literal_load_{width}_bytes_little_endian"),
|
||||
});
|
||||
if imports.contains_key(&target) {
|
||||
targets.insert((r.offset, target, "literal_import_slot_load"));
|
||||
}
|
||||
// A printable literal pool word is often a floating-point constant, not
|
||||
// a string. Follow a loaded pointer only when the next instruction uses
|
||||
// the same register as a memory base, without conditional execution.
|
||||
if !i.conditional
|
||||
&& i.flow == Flow::Next
|
||||
&& width == if input.mode == Mode::Arm { 4 } else { 8 }
|
||||
&& let Some(next) = e.instructions.get((r.offset / 4) as usize + 1)
|
||||
&& !next.conditional
|
||||
&& i.first_reg
|
||||
.is_some_and(|reg| next.memory_bases.contains(®))
|
||||
&& !data.iter().any(|(start, end)| {
|
||||
input.address + next.offset >= *start && input.address + next.offset < *end
|
||||
})
|
||||
{
|
||||
let mut raw = [0u8; 8];
|
||||
raw[..width].copy_from_slice(bytes);
|
||||
targets.insert((
|
||||
r.offset,
|
||||
u64::from_le_bytes(raw),
|
||||
"literal_pointer_used_as_memory_base",
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// ADR is an address by ISA semantics, even when it is passed to a callee rather
|
||||
// than dereferenced locally. This evidence does NOT weaken normalization masks.
|
||||
for (n, i) in e.instructions.iter().take(4096).enumerate() {
|
||||
if data.iter().any(|(start, end)| {
|
||||
input.address + i.offset >= *start && input.address + i.offset < *end
|
||||
}) {
|
||||
continue;
|
||||
}
|
||||
let w = i.word;
|
||||
let pc = input.address + i.offset;
|
||||
if input.mode == Mode::Aarch64 {
|
||||
let imm = (u64::from((w >> 5) & 0x7ffff) << 2) | u64::from((w >> 29) & 3);
|
||||
if w & 0x9f000000 == 0x10000000 && w & 31 != 31 {
|
||||
if let Some(target) = pc.checked_add_signed(signed(imm, 21)) {
|
||||
targets.insert((i.offset, target, "aarch64_adr_address"));
|
||||
}
|
||||
} else if w & 0x9f000000 == 0x90000000
|
||||
&& w & 31 != 31
|
||||
&& let Some(next) = e.instructions.get(n + 1)
|
||||
{
|
||||
if data.iter().any(|(start, end)| {
|
||||
input.address + next.offset >= *start && input.address + next.offset < *end
|
||||
}) {
|
||||
continue;
|
||||
}
|
||||
let v = next.word;
|
||||
if (v >> 5) & 31 == w & 31
|
||||
&& let Some(page) = (pc & !4095).checked_add_signed(signed(imm, 21) * 4096)
|
||||
{
|
||||
let offset = if v & 0xff800000 == 0x91000000 && v & 31 == w & 31 {
|
||||
Some((
|
||||
u64::from((v >> 10) & 0xfff) << if v & (1 << 22) != 0 { 12 } else { 0 },
|
||||
"aarch64_adrp_add_address",
|
||||
))
|
||||
} else if v & 0xffc00000 == 0xf9400000 {
|
||||
Some((u64::from((v >> 10) & 0xfff) * 8, "aarch64_adrp_ldr_slot"))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
if let Some((offset, proof)) = offset
|
||||
&& let Some(target) = page.checked_add(offset)
|
||||
{
|
||||
targets.insert((i.offset, target, proof));
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if input.mode == Mode::Arm
|
||||
&& w >> 28 != 15
|
||||
&& w & 0x0e1f0000 == 0x020f0000
|
||||
&& matches!((w >> 21) & 15, 2 | 4)
|
||||
&& w & 0xf000 != 0xf000
|
||||
{
|
||||
let imm = (w & 255).rotate_right(((w >> 8) & 15) * 2);
|
||||
let delta = if (w >> 21) & 15 == 2 {
|
||||
-i64::from(imm)
|
||||
} else {
|
||||
i64::from(imm)
|
||||
};
|
||||
if let Some(target) = pc
|
||||
.checked_add(8)
|
||||
.and_then(|pc| pc.checked_add_signed(delta))
|
||||
{
|
||||
targets.insert((i.offset, target, "arm_adr_address"));
|
||||
}
|
||||
}
|
||||
}
|
||||
report.truncated |= targets.len() > MAX_ANCHORS;
|
||||
for (offset, address, proof) in targets.into_iter().take(MAX_ANCHORS) {
|
||||
if report.anchors.len() >= MAX_ANCHORS {
|
||||
report.truncated = true;
|
||||
break;
|
||||
}
|
||||
if let Some(name) = imports.get(&address).filter(|_| {
|
||||
memory(
|
||||
input,
|
||||
address,
|
||||
if input.mode == Mode::Aarch64 { 8 } else { 4 },
|
||||
)
|
||||
.is_some()
|
||||
}) {
|
||||
report.anchors.push(Anchor {
|
||||
instruction_offset: offset,
|
||||
address,
|
||||
kind: "import_slot".into(),
|
||||
value: name.clone(),
|
||||
bytes_sha256: None,
|
||||
proof: format!("{proof}+undefined_dynamic_symbol_relocation"),
|
||||
});
|
||||
} else if let Some(value) = text_at(input, address) {
|
||||
report.anchors.push(Anchor {
|
||||
instruction_offset: offset,
|
||||
address,
|
||||
kind: "string".into(),
|
||||
bytes_sha256: Some(sha256(value.as_bytes())),
|
||||
value,
|
||||
proof: proof.into(),
|
||||
});
|
||||
}
|
||||
}
|
||||
if report.anchors.len() > MAX_ANCHORS {
|
||||
report.truncated = true;
|
||||
report.anchors.truncate(MAX_ANCHORS);
|
||||
}
|
||||
report
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
fn input(mode: Mode, words: &[u32], address: u64, text: &[u8]) -> Input {
|
||||
let bytes: Vec<_> = words.iter().flat_map(|w| w.to_le_bytes()).collect();
|
||||
Input {
|
||||
mode,
|
||||
address: 0x1000,
|
||||
expected_body_sha256: sha256(&bytes),
|
||||
bytes,
|
||||
boundary_provenance: "unit".into(),
|
||||
memory: vec![MemoryRange {
|
||||
start: address,
|
||||
bytes: text.to_vec(),
|
||||
}],
|
||||
relocations: BTreeSet::new(),
|
||||
target_identities: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn addresses_have_provenance_but_arbitrary_constants_do_not_become_strings() {
|
||||
// ADR x0,+32; RET versus MOV x0,#0x1020; RET.
|
||||
let a = input(
|
||||
Mode::Aarch64,
|
||||
&[0x10000100, 0xd65f03c0],
|
||||
0x1020,
|
||||
b"ball locked\0",
|
||||
);
|
||||
let e = analyze(&a).unwrap();
|
||||
let r = extract(&a, &e, "elf", &BTreeMap::new(), &[]);
|
||||
assert_eq!(r.anchors.len(), 1);
|
||||
assert_eq!(r.anchors[0].value, "ball locked");
|
||||
assert_eq!(r.anchors[0].proof, "aarch64_adr_address");
|
||||
assert!(e.masks.is_empty()); // Anchor extraction does not weaken candidate hashes.
|
||||
let b = input(
|
||||
Mode::Aarch64,
|
||||
&[0xd2820400, 0xd65f03c0],
|
||||
0x1020,
|
||||
b"ball locked\0",
|
||||
);
|
||||
assert!(
|
||||
extract(&b, &analyze(&b).unwrap(), "elf", &BTreeMap::new(), &[])
|
||||
.anchors
|
||||
.is_empty()
|
||||
);
|
||||
assert!(
|
||||
extract(&a, &e, "elf", &BTreeMap::new(), &[(0x1000, 0x1004)])
|
||||
.anchors
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn changed_literal_values_and_import_slots_remain_distinct() {
|
||||
let a = input(
|
||||
Mode::Arm,
|
||||
&[0xe59f0010, 0xe12fff1e],
|
||||
0x1018,
|
||||
&42u32.to_le_bytes(),
|
||||
);
|
||||
let b = input(
|
||||
Mode::Arm,
|
||||
&[0xe59f0010, 0xe12fff1e],
|
||||
0x1018,
|
||||
&43u32.to_le_bytes(),
|
||||
);
|
||||
let x = extract(&a, &analyze(&a).unwrap(), "a", &BTreeMap::new(), &[]);
|
||||
let y = extract(&b, &analyze(&b).unwrap(), "b", &BTreeMap::new(), &[]);
|
||||
assert_ne!(x.anchors[0].bytes_sha256, y.anchors[0].bytes_sha256);
|
||||
assert_eq!(x.anchors[0].kind, "literal_constant");
|
||||
let a = input(
|
||||
Mode::Aarch64,
|
||||
&[0x90000000, 0xf9401000, 0xd65f03c0],
|
||||
0x1020,
|
||||
&[0; 8],
|
||||
);
|
||||
let imports = BTreeMap::from([(0x1020, "puts".into())]);
|
||||
let r = extract(&a, &analyze(&a).unwrap(), "elf", &imports, &[]);
|
||||
assert_eq!(r.anchors[0].kind, "import_slot");
|
||||
assert_eq!(r.anchors[0].value, "puts");
|
||||
assert!(
|
||||
extract(&a, &analyze(&a).unwrap(), "elf", &BTreeMap::new(), &[])
|
||||
.anchors
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn strings_require_termination_printable_utf8_and_consistent_mappings() {
|
||||
for bytes in [
|
||||
b"abcd".to_vec(),
|
||||
b"a\x01bc\0".to_vec(),
|
||||
vec![b'a'; MAX_STRING_BYTES + 2],
|
||||
b"abc\0".to_vec(),
|
||||
] {
|
||||
let a = input(Mode::Aarch64, &[0x10000100, 0xd65f03c0], 0x1020, &bytes);
|
||||
assert!(
|
||||
extract(&a, &analyze(&a).unwrap(), "elf", &BTreeMap::new(), &[])
|
||||
.anchors
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
let mut a = input(Mode::Aarch64, &[0x10000100, 0xd65f03c0], 0x1020, b"abcd\0");
|
||||
a.memory.push(MemoryRange {
|
||||
start: 0x1020,
|
||||
bytes: b"efgh\0".to_vec(),
|
||||
});
|
||||
assert!(
|
||||
extract(&a, &analyze(&a).unwrap(), "elf", &BTreeMap::new(), &[])
|
||||
.anchors
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod pointer_tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn printable_pool_constants_are_not_strings_and_pointer_use_is_required() {
|
||||
let bytes: Vec<_> = [0xe59f0010u32, 0xe12fff1e]
|
||||
.iter()
|
||||
.flat_map(|w| w.to_le_bytes())
|
||||
.collect();
|
||||
let mut input = Input {
|
||||
mode: Mode::Arm,
|
||||
address: 0x1000,
|
||||
expected_body_sha256: sha256(&bytes),
|
||||
bytes,
|
||||
boundary_provenance: "unit".into(),
|
||||
memory: vec![MemoryRange {
|
||||
start: 0x1018,
|
||||
bytes: b"gfffhW(\0".to_vec(),
|
||||
}],
|
||||
relocations: BTreeSet::new(),
|
||||
target_identities: BTreeMap::new(),
|
||||
};
|
||||
let report = extract(
|
||||
&input,
|
||||
&analyze(&input).unwrap(),
|
||||
"elf",
|
||||
&BTreeMap::new(),
|
||||
&[],
|
||||
);
|
||||
assert_eq!(report.anchors.len(), 1);
|
||||
assert_eq!(report.anchors[0].kind, "literal_constant");
|
||||
input.memory[0].bytes = 0x8000u32.to_le_bytes().to_vec();
|
||||
input.memory.push(MemoryRange {
|
||||
start: 0x8000,
|
||||
bytes: b"ball locked\0".to_vec(),
|
||||
});
|
||||
assert_eq!(
|
||||
extract(
|
||||
&input,
|
||||
&analyze(&input).unwrap(),
|
||||
"elf",
|
||||
&BTreeMap::new(),
|
||||
&[]
|
||||
)
|
||||
.anchors
|
||||
.len(),
|
||||
1
|
||||
);
|
||||
input.bytes = [0xe59f0010u32, 0xe5d01000, 0xe12fff1e]
|
||||
.iter()
|
||||
.flat_map(|w| w.to_le_bytes())
|
||||
.collect();
|
||||
input.expected_body_sha256 = sha256(&input.bytes);
|
||||
let report = extract(
|
||||
&input,
|
||||
&analyze(&input).unwrap(),
|
||||
"elf",
|
||||
&BTreeMap::new(),
|
||||
&[],
|
||||
);
|
||||
assert_eq!(report.anchors.len(), 2);
|
||||
assert_eq!(report.anchors[1].value, "ball locked");
|
||||
assert_eq!(
|
||||
report.anchors[1].proof,
|
||||
"literal_pointer_used_as_memory_base"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,156 @@
|
||||
use crate::*;
|
||||
use std::collections::VecDeque;
|
||||
pub(crate) fn build(input: &Input, e: &mut Evidence) -> Result<()> {
|
||||
if e.instructions.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
let end = input.address + e.decoded_bytes as u64;
|
||||
let mut leaders = BTreeSet::from([input.address]);
|
||||
let mut complete = e.complete_decode;
|
||||
for i in &e.instructions {
|
||||
if let Flow::Branch(Some(t)) = i.flow {
|
||||
if t >= input.address && t < end {
|
||||
if t % 4 == 0 {
|
||||
leaders.insert(t);
|
||||
} else {
|
||||
complete = false;
|
||||
}
|
||||
} else if t >= end && t < input.address + input.bytes.len() as u64 {
|
||||
complete = false;
|
||||
}
|
||||
}
|
||||
if matches!(i.flow, Flow::Branch(_) | Flow::Return | Flow::Trap)
|
||||
&& input.address + i.offset + 4 < end
|
||||
{
|
||||
leaders.insert(input.address + i.offset + 4);
|
||||
}
|
||||
if matches!(i.flow, Flow::Branch(None)) {
|
||||
complete = false;
|
||||
}
|
||||
}
|
||||
ensure!(leaders.len() <= MAX_BLOCKS, "basic block limit");
|
||||
let starts: Vec<_> = leaders.into_iter().collect();
|
||||
let lookup: BTreeMap<_, _> = starts.iter().enumerate().map(|(i, s)| (*s, i)).collect();
|
||||
for (id, start) in starts.iter().enumerate() {
|
||||
let stop = starts.get(id + 1).copied().unwrap_or(end);
|
||||
let begin = ((start - input.address) / 4) as usize;
|
||||
let finish = ((stop - input.address) / 4) as usize;
|
||||
let mut labels = format!("{METHOD}:{:?}:block", input.mode).into_bytes();
|
||||
for i in &e.instructions[begin..finish] {
|
||||
let internal = matches!(i.flow,Flow::Branch(Some(t)) if t>=input.address && t<end);
|
||||
labels.extend((i.normalized & !if internal { i.branch_mask } else { 0 }).to_le_bytes());
|
||||
}
|
||||
let last = &e.instructions[finish - 1];
|
||||
let mut edges = vec![];
|
||||
let mut push = |kind: &str, target: Option<usize>| {
|
||||
edges.push(Edge {
|
||||
kind: kind.into(),
|
||||
target,
|
||||
})
|
||||
};
|
||||
match last.flow {
|
||||
Flow::Branch(target) => {
|
||||
push(
|
||||
if last.conditional { "taken" } else { "jump" },
|
||||
target.and_then(|t| lookup.get(&t).copied()),
|
||||
);
|
||||
}
|
||||
Flow::Return => push("return", None),
|
||||
Flow::Trap => push("trap", None),
|
||||
_ => {}
|
||||
}
|
||||
if matches!(last.flow, Flow::Next | Flow::Call(_)) || last.conditional {
|
||||
if id + 1 < starts.len() {
|
||||
push("fallthrough", Some(id + 1));
|
||||
} else {
|
||||
push("extent_fallthrough", None);
|
||||
complete = false;
|
||||
}
|
||||
}
|
||||
e.blocks.push(Block {
|
||||
start: start - input.address,
|
||||
end: stop - input.address,
|
||||
instruction_hash: sha256(&labels),
|
||||
edges,
|
||||
reachable: false,
|
||||
});
|
||||
}
|
||||
let mut queue = VecDeque::from([0]);
|
||||
while let Some(id) = queue.pop_front() {
|
||||
if e.blocks[id].reachable {
|
||||
continue;
|
||||
}
|
||||
e.blocks[id].reachable = true;
|
||||
for target in e.blocks[id].edges.iter().filter_map(|x| x.target) {
|
||||
queue.push_back(target)
|
||||
}
|
||||
}
|
||||
e.reachable_bytes = e
|
||||
.blocks
|
||||
.iter()
|
||||
.filter(|b| b.reachable)
|
||||
.map(|b| (b.end - b.start) as usize)
|
||||
.sum();
|
||||
if !complete {
|
||||
diag(e, "incomplete_control_flow");
|
||||
}
|
||||
if e.reachable_bytes != e.size {
|
||||
diag(e, "unreachable_or_unclassified_bytes");
|
||||
}
|
||||
e.complete_cfg = complete && e.reachable_bytes == e.size;
|
||||
if e.complete_cfg {
|
||||
e.cfg_wl_sha256 = Some(wl(&e.blocks, false, input.mode));
|
||||
e.cfg_shape_wl_sha256 = Some(wl(&e.blocks, true, input.mode));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
/// Directed, edge-labelled 3-round WL; includes entry color, node multiplicities and exits.
|
||||
/// Shape-only deliberately ignores instructions and is never a semantic match criterion.
|
||||
fn wl(blocks: &[Block], shape: bool, mode: Mode) -> String {
|
||||
let mut colors: Vec<String> = blocks
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(id, b)| {
|
||||
sha256(
|
||||
format!(
|
||||
"{}:{}",
|
||||
if id == 0 { "entry" } else { "node" },
|
||||
if shape { "shape" } else { &b.instruction_hash }
|
||||
)
|
||||
.as_bytes(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
let mut incoming = vec![vec![]; blocks.len()];
|
||||
for (from, b) in blocks.iter().enumerate() {
|
||||
for edge in &b.edges {
|
||||
if let Some(to) = edge.target {
|
||||
incoming[to].push((from, edge.kind.as_str()));
|
||||
}
|
||||
}
|
||||
}
|
||||
for _ in 0..3 {
|
||||
colors = blocks
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(id, b)| {
|
||||
let mut neighbors = vec![];
|
||||
for edge in &b.edges {
|
||||
neighbors.push(format!(
|
||||
"out:{}:{}",
|
||||
edge.kind,
|
||||
edge.target.map(|n| colors[n].as_str()).unwrap_or("exit")
|
||||
));
|
||||
}
|
||||
for (from, kind) in &incoming[id] {
|
||||
neighbors.push(format!("in:{kind}:{}", colors[*from]));
|
||||
}
|
||||
neighbors.sort();
|
||||
sha256(format!("{}:{}", colors[id], neighbors.join("|")).as_bytes())
|
||||
})
|
||||
.collect();
|
||||
}
|
||||
let entry = colors[0].clone();
|
||||
colors.sort();
|
||||
sha256(format!("{METHOD}:{mode:?}:wl3:{shape}:{entry}:{}", colors.join("|")).as_bytes())
|
||||
}
|
||||
@@ -0,0 +1,433 @@
|
||||
//! ELF adapter verifies both the executable fingerprint and exact complete function bytes.
|
||||
use crate::*;
|
||||
use anyhow::Context;
|
||||
use goblin::elf::{Elf, header, program_header};
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct SymbolFunction {
|
||||
pub address: u64,
|
||||
pub size: usize,
|
||||
pub name: String,
|
||||
pub mode: Mode,
|
||||
pub body_sha256: String,
|
||||
}
|
||||
pub struct ElfImage {
|
||||
input: Input,
|
||||
ranges: Vec<(u64, u64, usize)>,
|
||||
pub sha256: String,
|
||||
pub mode: Mode,
|
||||
pub symbols: Vec<SymbolFunction>,
|
||||
mapping_modes: Vec<(u64, u64, Option<Mode>)>,
|
||||
imports: BTreeMap<u64, String>,
|
||||
}
|
||||
impl ElfImage {
|
||||
pub fn parse(bytes: &[u8], expected_sha256: &str) -> Result<Self> {
|
||||
ensure!(bytes.len() <= 256 * 1024 * 1024, "ELF byte limit");
|
||||
let actual = sha256(bytes);
|
||||
ensure!(actual == expected_sha256, "ELF input hash mismatch");
|
||||
let elf = Elf::parse(bytes)?;
|
||||
ensure!(elf.little_endian, "big-endian normalization unsupported");
|
||||
let mode = match elf.header.e_machine {
|
||||
header::EM_ARM if !elf.is_64 => Mode::Arm,
|
||||
header::EM_AARCH64 if elf.is_64 => Mode::Aarch64,
|
||||
_ => anyhow::bail!("unsupported ELF architecture"),
|
||||
};
|
||||
ensure!(
|
||||
[header::ET_EXEC, header::ET_DYN].contains(&elf.header.e_type),
|
||||
"ELF must be EXEC/DYN"
|
||||
);
|
||||
ensure!(
|
||||
elf.program_headers.len() <= 4096
|
||||
&& elf.syms.len().saturating_add(elf.dynsyms.len()) <= 500_000,
|
||||
"ELF record limit"
|
||||
);
|
||||
let mut memory = vec![];
|
||||
let mut ranges = vec![];
|
||||
let mut mapped_bytes = 0usize;
|
||||
for p in elf
|
||||
.program_headers
|
||||
.iter()
|
||||
.filter(|p| p.p_type == program_header::PT_LOAD && p.p_filesz > 0)
|
||||
{
|
||||
let start = usize::try_from(p.p_offset)?;
|
||||
let length = usize::try_from(p.p_filesz)?;
|
||||
mapped_bytes = mapped_bytes
|
||||
.checked_add(length)
|
||||
.context("mapping size overflow")?;
|
||||
ensure!(
|
||||
mapped_bytes <= 256 * 1024 * 1024,
|
||||
"aggregate ELF mapping byte limit"
|
||||
);
|
||||
let end = start
|
||||
.checked_add(length)
|
||||
.ok_or_else(|| anyhow::anyhow!("segment overflow"))?;
|
||||
let segment = bytes
|
||||
.get(start..end)
|
||||
.ok_or_else(|| anyhow::anyhow!("truncated segment"))?;
|
||||
p.p_vaddr
|
||||
.checked_add(p.p_filesz)
|
||||
.ok_or_else(|| anyhow::anyhow!("segment address overflow"))?;
|
||||
if p.is_executable() {
|
||||
ranges.push((p.p_vaddr, p.p_filesz, memory.len()));
|
||||
}
|
||||
memory.push(MemoryRange {
|
||||
start: p.p_vaddr,
|
||||
bytes: segment.to_vec(),
|
||||
});
|
||||
}
|
||||
let mut target_identities = BTreeMap::new();
|
||||
for (symbols, strings) in [(&elf.syms, &elf.strtab), (&elf.dynsyms, &elf.dynstrtab)] {
|
||||
for s in symbols
|
||||
.iter()
|
||||
.filter(|s| s.is_function() && s.st_value != 0)
|
||||
{
|
||||
if let Some(name) = strings
|
||||
.get_at(s.st_name)
|
||||
.filter(|n| !n.is_empty() && n.len() <= 1024)
|
||||
{
|
||||
target_identities
|
||||
.entry(s.st_value & !if mode == Mode::Arm { 1 } else { 0 })
|
||||
.or_insert_with(|| name.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut imports = BTreeMap::new();
|
||||
let mut conflicts = BTreeSet::new();
|
||||
ensure!(
|
||||
elf.dynrelas
|
||||
.len()
|
||||
.saturating_add(elf.dynrels.len())
|
||||
.saturating_add(elf.pltrelocs.len())
|
||||
<= 500_000,
|
||||
"ELF relocation limit"
|
||||
);
|
||||
for r in elf
|
||||
.dynrelas
|
||||
.iter()
|
||||
.chain(elf.dynrels.iter())
|
||||
.chain(elf.pltrelocs.iter())
|
||||
{
|
||||
let supported = match mode {
|
||||
Mode::Arm => matches!(r.r_type, 21 | 22),
|
||||
Mode::Aarch64 => matches!(r.r_type, 1025 | 1026),
|
||||
_ => false,
|
||||
};
|
||||
if !supported {
|
||||
continue;
|
||||
}
|
||||
let Some(sym) = elf.dynsyms.get(r.r_sym) else {
|
||||
continue;
|
||||
};
|
||||
if sym.st_shndx != 0 {
|
||||
continue;
|
||||
}
|
||||
let Some(name) = elf
|
||||
.dynstrtab
|
||||
.get_at(sym.st_name)
|
||||
.filter(|n| !n.is_empty() && n.len() <= 1024)
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
if imports
|
||||
.insert(r.r_offset, name.to_owned())
|
||||
.is_some_and(|old| old != name)
|
||||
{
|
||||
conflicts.insert(r.r_offset);
|
||||
}
|
||||
}
|
||||
for address in conflicts {
|
||||
imports.remove(&address);
|
||||
}
|
||||
let mut mapping_modes = vec![];
|
||||
if mode == Mode::Arm {
|
||||
let mut by_section = BTreeMap::<usize, BTreeMap<u64, Option<Mode>>>::new();
|
||||
for s in elf.syms.iter() {
|
||||
let Some(name) = elf.strtab.get_at(s.st_name) else {
|
||||
continue;
|
||||
};
|
||||
let kind = match name.split('.').next().unwrap_or("") {
|
||||
"$a" => Some(Mode::Arm),
|
||||
"$t" => Some(Mode::Thumb),
|
||||
"$d" => None,
|
||||
_ => continue,
|
||||
};
|
||||
by_section
|
||||
.entry(s.st_shndx)
|
||||
.or_default()
|
||||
.insert(s.st_value, kind);
|
||||
}
|
||||
for (section, entries) in by_section {
|
||||
let Some(section) = elf.section_headers.get(section) else {
|
||||
continue;
|
||||
};
|
||||
if section.sh_flags
|
||||
& u64::from(
|
||||
goblin::elf::section_header::SHF_ALLOC
|
||||
| goblin::elf::section_header::SHF_EXECINSTR,
|
||||
)
|
||||
!= u64::from(
|
||||
goblin::elf::section_header::SHF_ALLOC
|
||||
| goblin::elf::section_header::SHF_EXECINSTR,
|
||||
)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let Some(section_end) = section.sh_addr.checked_add(section.sh_size) else {
|
||||
continue;
|
||||
};
|
||||
let entries: Vec<_> = entries.into_iter().collect();
|
||||
for (i, (start, mode)) in entries.iter().enumerate() {
|
||||
let end = entries.get(i + 1).map(|e| e.0).unwrap_or(section_end);
|
||||
if *start >= section.sh_addr && end <= section_end && end > *start {
|
||||
mapping_modes.push((*start, end, *mode));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let mut symbols = vec![];
|
||||
let mut extents = BTreeMap::<u64, BTreeSet<(usize, Mode)>>::new();
|
||||
for table in [&elf.syms, &elf.dynsyms] {
|
||||
for s in table
|
||||
.iter()
|
||||
.filter(|s| s.is_function() && s.st_value != 0 && s.st_size > 0)
|
||||
{
|
||||
let instruction_mode = if mode == Mode::Arm && s.st_value & 1 != 0 {
|
||||
Mode::Thumb
|
||||
} else {
|
||||
mode
|
||||
};
|
||||
let address = s.st_value & !if mode == Mode::Arm { 1 } else { 0 };
|
||||
extents
|
||||
.entry(address)
|
||||
.or_default()
|
||||
.insert((usize::try_from(s.st_size)?, instruction_mode));
|
||||
}
|
||||
}
|
||||
for (address, choices) in extents {
|
||||
if choices.len() != 1 {
|
||||
continue;
|
||||
}
|
||||
let (size, instruction_mode) = *choices.first().unwrap();
|
||||
if size > MAX_FUNCTION_BYTES {
|
||||
continue;
|
||||
}
|
||||
let matches: Vec<_> = ranges
|
||||
.iter()
|
||||
.filter(|(start, len, _)| {
|
||||
address >= *start
|
||||
&& address
|
||||
.checked_add(size as u64)
|
||||
.is_some_and(|end| end <= start + len)
|
||||
})
|
||||
.collect();
|
||||
if matches.len() != 1 {
|
||||
continue;
|
||||
}
|
||||
let (start, _, index) = *matches[0];
|
||||
let offset = (address - start) as usize;
|
||||
symbols.push(SymbolFunction {
|
||||
address,
|
||||
size,
|
||||
name: target_identities.get(&address).cloned().unwrap_or_default(),
|
||||
mode: instruction_mode,
|
||||
body_sha256: sha256(&memory[index].bytes[offset..offset + size]),
|
||||
});
|
||||
}
|
||||
// Dynamic data relocations are not instruction-immediate relocation proofs. Only
|
||||
// instruction-use proofs are enabled here until architecture relocation types are mapped.
|
||||
Ok(Self {
|
||||
input: Input {
|
||||
mode,
|
||||
address: 0,
|
||||
bytes: vec![],
|
||||
expected_body_sha256: String::new(),
|
||||
boundary_provenance: String::new(),
|
||||
memory,
|
||||
relocations: BTreeSet::new(),
|
||||
target_identities,
|
||||
},
|
||||
ranges,
|
||||
sha256: actual,
|
||||
mode,
|
||||
symbols,
|
||||
mapping_modes,
|
||||
imports,
|
||||
})
|
||||
}
|
||||
pub fn function_with_anchors(
|
||||
&mut self,
|
||||
address: u64,
|
||||
size: usize,
|
||||
expected_body_sha256: &str,
|
||||
mode: Mode,
|
||||
boundary_provenance: &str,
|
||||
) -> Result<(Evidence, crate::anchors::Report)> {
|
||||
let evidence = self.function(
|
||||
address,
|
||||
size,
|
||||
expected_body_sha256,
|
||||
mode,
|
||||
boundary_provenance,
|
||||
)?;
|
||||
let data: Vec<_> = self
|
||||
.mapping_modes
|
||||
.iter()
|
||||
.filter(|(_, _, mode)| mode.is_none())
|
||||
.map(|(start, end, _)| (*start, *end))
|
||||
.collect();
|
||||
let anchors =
|
||||
crate::anchors::extract(&self.input, &evidence, &self.sha256, &self.imports, &data);
|
||||
Ok((evidence, anchors))
|
||||
}
|
||||
pub fn known_mode(&self, address: u64) -> Option<Mode> {
|
||||
if self.mode == Mode::Aarch64 {
|
||||
return Some(Mode::Aarch64);
|
||||
}
|
||||
if let Some((_, _, mode)) = self
|
||||
.mapping_modes
|
||||
.iter()
|
||||
.find(|(start, end, _)| address >= *start && address < *end)
|
||||
{
|
||||
return *mode;
|
||||
}
|
||||
self.symbols
|
||||
.iter()
|
||||
.find(|s| s.address == address)
|
||||
.map(|s| s.mode)
|
||||
}
|
||||
pub fn function(
|
||||
&mut self,
|
||||
address: u64,
|
||||
size: usize,
|
||||
expected_body_sha256: &str,
|
||||
mode: Mode,
|
||||
boundary_provenance: &str,
|
||||
) -> Result<Evidence> {
|
||||
ensure!(
|
||||
size > 0 && size <= MAX_FUNCTION_BYTES,
|
||||
"function byte limit"
|
||||
);
|
||||
ensure!(
|
||||
mode == self.mode || (self.mode == Mode::Arm && mode == Mode::Thumb),
|
||||
"mode conflicts with ELF architecture"
|
||||
);
|
||||
if let Some(known) = self.known_mode(address & !if self.mode == Mode::Arm { 1 } else { 0 })
|
||||
{
|
||||
ensure!(
|
||||
known == mode,
|
||||
"mode contradicts ELF symbol or mapping symbol"
|
||||
);
|
||||
}
|
||||
let address = if mode == Mode::Thumb {
|
||||
address & !1
|
||||
} else {
|
||||
address
|
||||
};
|
||||
let end = address
|
||||
.checked_add(size as u64)
|
||||
.ok_or_else(|| anyhow::anyhow!("function overflow"))?;
|
||||
let candidates: Vec<_> = self
|
||||
.ranges
|
||||
.iter()
|
||||
.filter(|(start, length, _)| address >= *start && end <= start + length)
|
||||
.collect();
|
||||
ensure!(
|
||||
candidates.len() == 1,
|
||||
"function must have one complete file-backed executable mapping"
|
||||
);
|
||||
let (start, _, index) = *candidates[0];
|
||||
let offset = (address - start) as usize;
|
||||
self.input.mode = mode;
|
||||
self.input.address = address;
|
||||
self.input.bytes = self.input.memory[index].bytes[offset..offset + size].to_vec();
|
||||
self.input.expected_body_sha256 = expected_body_sha256.into();
|
||||
self.input.boundary_provenance = boundary_provenance.into();
|
||||
let overlapping: Vec<_> = self
|
||||
.mapping_modes
|
||||
.iter()
|
||||
.filter(|(start, stop, _)| *start < end && *stop > address)
|
||||
.collect();
|
||||
ensure!(
|
||||
!overlapping
|
||||
.iter()
|
||||
.any(|(_, _, mapped)| mapped.is_some_and(|m| m != mode)),
|
||||
"mixed instruction modes within function extent are unsupported"
|
||||
);
|
||||
let mut evidence = analyze(&self.input)?;
|
||||
let data: Vec<_> = overlapping
|
||||
.iter()
|
||||
.filter(|(_, _, mapped)| mapped.is_none())
|
||||
.map(|(start, stop, _)| (*start, *stop))
|
||||
.collect();
|
||||
if !data.is_empty() {
|
||||
diag(&mut evidence, "elf_data_mapping_in_function_extent");
|
||||
evidence.complete_decode = false;
|
||||
evidence.complete_cfg = false;
|
||||
evidence.masked_body_sha256 = None;
|
||||
evidence.reference_bound_sha256 = None;
|
||||
evidence.cfg_wl_sha256 = None;
|
||||
evidence.cfg_shape_wl_sha256 = None;
|
||||
evidence.references.retain(|r| {
|
||||
!data
|
||||
.iter()
|
||||
.any(|(start, stop)| address + r.offset >= *start && address + r.offset < *stop)
|
||||
});
|
||||
evidence.masks.retain(|r| {
|
||||
!data
|
||||
.iter()
|
||||
.any(|(start, stop)| address + r.offset >= *start && address + r.offset < *stop)
|
||||
});
|
||||
}
|
||||
Ok(evidence)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod mapping_tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn data_and_mixed_mode_mapping_symbols_do_not_become_complete_code() {
|
||||
let bytes: Vec<_> = [0xe1a00000u32, 0xe12fff1e]
|
||||
.iter()
|
||||
.flat_map(|w| w.to_le_bytes())
|
||||
.collect();
|
||||
let body = sha256(&bytes);
|
||||
let mut elf = ElfImage {
|
||||
input: Input {
|
||||
mode: Mode::Arm,
|
||||
address: 0,
|
||||
bytes: vec![],
|
||||
expected_body_sha256: String::new(),
|
||||
boundary_provenance: String::new(),
|
||||
memory: vec![MemoryRange {
|
||||
start: 0x1000,
|
||||
bytes,
|
||||
}],
|
||||
relocations: BTreeSet::new(),
|
||||
target_identities: BTreeMap::new(),
|
||||
},
|
||||
ranges: vec![(0x1000, 8, 0)],
|
||||
sha256: "fixture".into(),
|
||||
mode: Mode::Arm,
|
||||
symbols: vec![],
|
||||
mapping_modes: vec![(0x1000, 0x1004, None), (0x1004, 0x1008, Some(Mode::Arm))],
|
||||
imports: BTreeMap::new(),
|
||||
};
|
||||
let e = elf
|
||||
.function(0x1000, 8, &body, Mode::Arm, "mapping test")
|
||||
.unwrap();
|
||||
assert!(!e.complete_decode);
|
||||
assert!(e.masked_body_sha256.is_none());
|
||||
assert!(
|
||||
e.diagnostics
|
||||
.contains_key("elf_data_mapping_in_function_extent")
|
||||
);
|
||||
elf.mapping_modes = vec![
|
||||
(0x1000, 0x1004, Some(Mode::Arm)),
|
||||
(0x1004, 0x1008, Some(Mode::Thumb)),
|
||||
];
|
||||
assert!(
|
||||
elf.function(0x1000, 8, &body, Mode::Arm, "mapping test")
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
//! Native evidence, not a semantic-equivalence or name-propagation oracle.
|
||||
//! Inputs must carry verified complete contiguous function bytes and exact ISA mode.
|
||||
pub mod anchors;
|
||||
mod cfg;
|
||||
pub mod elf;
|
||||
mod normalize;
|
||||
use anyhow::{Result, ensure};
|
||||
use capstone::{
|
||||
InsnGroupType,
|
||||
arch::{arm, arm64},
|
||||
prelude::*,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
pub const METHOD: &str = "verstack-code/1";
|
||||
pub const DECODER: &str = "capstone-rs 0.14.0 / capstone-sys 0.18.0";
|
||||
pub const MAX_FUNCTION_BYTES: usize = 1024 * 1024;
|
||||
pub const MAX_BLOCKS: usize = 16_384;
|
||||
#[derive(Clone, Copy, Debug, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Mode {
|
||||
Arm,
|
||||
Thumb,
|
||||
Aarch64,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct MemoryRange {
|
||||
pub start: u64,
|
||||
pub bytes: Vec<u8>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Input {
|
||||
pub mode: Mode,
|
||||
pub address: u64,
|
||||
pub bytes: Vec<u8>,
|
||||
/// Mandatory complete-body oracle; a prefix digest is never acceptable.
|
||||
pub expected_body_sha256: String,
|
||||
pub boundary_provenance: String,
|
||||
/// File-backed ELF mappings used to prove address operands. Never inferred from numerics alone.
|
||||
#[serde(default)]
|
||||
pub memory: Vec<MemoryRange>,
|
||||
/// Relocation instruction addresses supplied by verified ELF metadata.
|
||||
#[serde(default)]
|
||||
pub relocations: BTreeSet<u64>,
|
||||
/// Resolved symbol/import identities are optional evidence, not inferred from an address.
|
||||
#[serde(default)]
|
||||
pub target_identities: BTreeMap<u64, String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Mask {
|
||||
pub offset: u64,
|
||||
pub bits: u32,
|
||||
pub reason: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Reference {
|
||||
pub offset: u64,
|
||||
pub kind: String,
|
||||
pub target: Option<u64>,
|
||||
pub target_mode: Option<Mode>,
|
||||
pub identity: Option<String>,
|
||||
pub literal_sha256: Option<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Instruction {
|
||||
pub offset: u64,
|
||||
pub word: u32,
|
||||
pub normalized: u32,
|
||||
pub mnemonic: String,
|
||||
pub operands: String,
|
||||
pub flow: Flow,
|
||||
pub conditional: bool,
|
||||
pub branch_mask: u32,
|
||||
#[serde(skip)]
|
||||
pub memory_bases: Vec<u16>,
|
||||
#[serde(skip)]
|
||||
pub first_reg: Option<u16>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(tag = "kind", content = "target", rename_all = "snake_case")]
|
||||
pub enum Flow {
|
||||
Next,
|
||||
Call(Option<u64>),
|
||||
Branch(Option<u64>),
|
||||
Return,
|
||||
Trap,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Edge {
|
||||
pub kind: String,
|
||||
pub target: Option<usize>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Block {
|
||||
pub start: u64,
|
||||
pub end: u64,
|
||||
pub instruction_hash: String,
|
||||
pub edges: Vec<Edge>,
|
||||
pub reachable: bool,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Evidence {
|
||||
pub method: String,
|
||||
pub decoder: String,
|
||||
pub mode: Mode,
|
||||
pub address: u64,
|
||||
pub size: usize,
|
||||
pub body_sha256: String,
|
||||
pub boundary_provenance: String,
|
||||
pub decoded_bytes: usize,
|
||||
pub reachable_bytes: usize,
|
||||
pub complete_decode: bool,
|
||||
pub complete_cfg: bool,
|
||||
pub masked_body_sha256: Option<String>,
|
||||
pub reference_bound_sha256: Option<String>,
|
||||
pub cfg_wl_sha256: Option<String>,
|
||||
pub cfg_shape_wl_sha256: Option<String>,
|
||||
pub masks: Vec<Mask>,
|
||||
pub references: Vec<Reference>,
|
||||
pub instructions: Vec<Instruction>,
|
||||
pub blocks: Vec<Block>,
|
||||
pub diagnostics: BTreeMap<String, usize>,
|
||||
pub caveat: String,
|
||||
}
|
||||
pub fn sha256(bytes: &[u8]) -> String {
|
||||
Sha256::digest(bytes)
|
||||
.iter()
|
||||
.map(|b| format!("{b:02x}"))
|
||||
.collect()
|
||||
}
|
||||
pub(crate) fn diag(e: &mut Evidence, s: &str) {
|
||||
*e.diagnostics.entry(s.into()).or_default() += 1;
|
||||
}
|
||||
pub(crate) fn memory(input: &Input, address: u64, size: usize) -> Option<&[u8]> {
|
||||
let end = address.checked_add(size as u64)?;
|
||||
let mut matches = input.memory.iter().filter_map(|m| {
|
||||
let m_end = m.start.checked_add(m.bytes.len() as u64)?;
|
||||
(address >= m.start && end <= m_end)
|
||||
.then(|| &m.bytes[(address - m.start) as usize..(end - m.start) as usize])
|
||||
});
|
||||
let first = matches.next()?;
|
||||
// Overlapping contradictory mappings are not evidence.
|
||||
matches.all(|v| v == first).then_some(first)
|
||||
}
|
||||
pub(crate) fn reference(
|
||||
input: &Input,
|
||||
offset: u64,
|
||||
kind: &str,
|
||||
target: Option<u64>,
|
||||
target_mode: Option<Mode>,
|
||||
literal: usize,
|
||||
) -> Reference {
|
||||
Reference {
|
||||
offset,
|
||||
kind: kind.into(),
|
||||
target,
|
||||
target_mode,
|
||||
identity: target.and_then(|a| input.target_identities.get(&a).cloned()),
|
||||
literal_sha256: target.and_then(|a| {
|
||||
(literal > 0)
|
||||
.then(|| memory(input, a, literal))
|
||||
.flatten()
|
||||
.map(sha256)
|
||||
}),
|
||||
}
|
||||
}
|
||||
pub fn analyze(input: &Input) -> Result<Evidence> {
|
||||
ensure!(
|
||||
!input.bytes.is_empty() && input.bytes.len() <= MAX_FUNCTION_BYTES,
|
||||
"function byte limit"
|
||||
);
|
||||
ensure!(
|
||||
!input.boundary_provenance.is_empty(),
|
||||
"verified boundary provenance required"
|
||||
);
|
||||
let body_sha256 = sha256(&input.bytes);
|
||||
ensure!(
|
||||
body_sha256 == input.expected_body_sha256,
|
||||
"complete body hash mismatch"
|
||||
);
|
||||
ensure!(
|
||||
input
|
||||
.address
|
||||
.checked_add(input.bytes.len() as u64)
|
||||
.is_some(),
|
||||
"address overflow"
|
||||
);
|
||||
ensure!(
|
||||
input.mode == Mode::Aarch64
|
||||
|| input.address + input.bytes.len() as u64 <= u64::from(u32::MAX) + 1,
|
||||
"A32/Thumb address width exceeded"
|
||||
);
|
||||
ensure!(
|
||||
input.memory.len() <= 4096
|
||||
&& input.memory.iter().map(|m| m.bytes.len()).sum::<usize>() <= 256 * 1024 * 1024,
|
||||
"mapping limit"
|
||||
);
|
||||
let mut e=Evidence{method:METHOD.into(),decoder:DECODER.into(),mode:input.mode,address:input.address,size:input.bytes.len(),body_sha256,boundary_provenance:input.boundary_provenance.clone(),decoded_bytes:0,reachable_bytes:0,complete_decode:false,complete_cfg:false,masked_body_sha256:None,reference_bound_sha256:None,cfg_wl_sha256:None,cfg_shape_wl_sha256:None,masks:vec![],references:vec![],instructions:vec![],blocks:vec![],diagnostics:BTreeMap::new(),caveat:"Masks and WL hashes are candidate evidence, not proof of equal behavior. Resolve callees and referenced data independently; require bidirectional uniqueness before considering a pairing. This does not recover boundaries, indirect jump tables, exception edges, or cross-ISA semantic equivalence.".into()};
|
||||
if input.mode == Mode::Thumb {
|
||||
diag(&mut e, "unsupported_thumb_normalization");
|
||||
return Ok(e);
|
||||
}
|
||||
ensure!(
|
||||
input.address.is_multiple_of(4) && input.bytes.len().is_multiple_of(4),
|
||||
"unaligned A32/A64 function extent"
|
||||
);
|
||||
let cs = match input.mode {
|
||||
Mode::Arm => Capstone::new()
|
||||
.arm()
|
||||
.mode(arm::ArchMode::Arm)
|
||||
.endian(capstone::Endian::Little)
|
||||
.detail(true)
|
||||
.build()?,
|
||||
Mode::Aarch64 => Capstone::new()
|
||||
.arm64()
|
||||
.mode(arm64::ArchMode::Arm)
|
||||
.endian(capstone::Endian::Little)
|
||||
.detail(true)
|
||||
.build()?,
|
||||
Mode::Thumb => unreachable!(),
|
||||
};
|
||||
let decoded = cs.disasm_all(&input.bytes, input.address)?;
|
||||
for insn in decoded.iter() {
|
||||
if insn.bytes().len() != 4 {
|
||||
diag(&mut e, "unsupported_instruction_width");
|
||||
break;
|
||||
}
|
||||
let detail = cs.insn_detail(insn)?;
|
||||
let arch = detail.arch_detail();
|
||||
let mut first_reg = None;
|
||||
let mut memory_bases = vec![];
|
||||
let mut immediates = vec![];
|
||||
if let Some(a) = arch.arm() {
|
||||
for (i, o) in a.operands().enumerate() {
|
||||
match o.op_type {
|
||||
arm::ArmOperandType::Reg(r) if i == 0 => first_reg = Some(r.0),
|
||||
arm::ArmOperandType::Mem(m) => memory_bases.push(m.base().0),
|
||||
arm::ArmOperandType::Imm(n) => immediates.push(u64::from(n as u32)),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
if let Some(a) = arch.arm64() {
|
||||
for (i, o) in a.operands().enumerate() {
|
||||
match o.op_type {
|
||||
arm64::Arm64OperandType::Reg(r) if i == 0 => first_reg = Some(r.0),
|
||||
arm64::Arm64OperandType::Mem(m) => memory_bases.push(m.base().0),
|
||||
arm64::Arm64OperandType::Imm(n) => immediates.push(n as u64),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
let word = u32::from_le_bytes(insn.bytes().try_into()?);
|
||||
let in_group = |g: u32| detail.groups().iter().any(|v| u32::from(v.0) == g);
|
||||
let mnemonic = insn.mnemonic().unwrap_or("").to_owned();
|
||||
let mut conditional = input.mode == Mode::Arm && (word >> 28) < 14;
|
||||
let pc_write = input.mode == Mode::Arm
|
||||
&& detail
|
||||
.regs_write()
|
||||
.iter()
|
||||
.any(|r| u32::from(r.0) == arm::ArmReg::ARM_REG_PC);
|
||||
let mut flow = Flow::Next;
|
||||
let mut branch_mask = 0;
|
||||
if in_group(InsnGroupType::CS_GRP_CALL) {
|
||||
flow = Flow::Call(immediates.last().copied());
|
||||
} else if in_group(InsnGroupType::CS_GRP_RET)
|
||||
|| (input.mode == Mode::Arm
|
||||
&& ((word & 0x0fffffff) == 0x012fff1e || (mnemonic.starts_with("pop") && pc_write)))
|
||||
{
|
||||
flow = Flow::Return;
|
||||
} else if in_group(InsnGroupType::CS_GRP_JUMP) || pc_write {
|
||||
flow = Flow::Branch(immediates.last().copied());
|
||||
if input.mode == Mode::Aarch64 {
|
||||
conditional = (word & 0xff000010) == 0x54000000
|
||||
|| (word & 0x7e000000) == 0x34000000
|
||||
|| (word & 0x7e000000) == 0x36000000;
|
||||
branch_mask = if word & 0x7c000000 == 0x14000000 {
|
||||
0x03ffffff
|
||||
} else if word & 0x7e000000 == 0x36000000 {
|
||||
0x0007ffe0
|
||||
} else if conditional {
|
||||
0x00ffffe0
|
||||
} else {
|
||||
0
|
||||
};
|
||||
} else if word & 0x0e000000 == 0x0a000000 {
|
||||
branch_mask = 0x00ffffff;
|
||||
}
|
||||
if branch_mask == 0 {
|
||||
flow = Flow::Branch(None);
|
||||
}
|
||||
} else if in_group(InsnGroupType::CS_GRP_INT)
|
||||
|| in_group(InsnGroupType::CS_GRP_IRET)
|
||||
|| mnemonic == "udf"
|
||||
{
|
||||
flow = Flow::Trap;
|
||||
}
|
||||
e.instructions.push(Instruction {
|
||||
offset: insn.address() - input.address,
|
||||
word,
|
||||
normalized: word,
|
||||
mnemonic,
|
||||
operands: insn.op_str().unwrap_or("").into(),
|
||||
flow,
|
||||
conditional,
|
||||
branch_mask,
|
||||
memory_bases,
|
||||
first_reg,
|
||||
});
|
||||
e.decoded_bytes += 4;
|
||||
}
|
||||
e.complete_decode = e.decoded_bytes == input.bytes.len();
|
||||
if !e.complete_decode {
|
||||
diag(&mut e, "undecoded_tail");
|
||||
}
|
||||
normalize::apply(input, &mut e);
|
||||
cfg::build(input, &mut e)?;
|
||||
if e.complete_decode && e.complete_cfg && e.reachable_bytes == e.size {
|
||||
let mut normalized = format!("{METHOD}:{:?}:masked", input.mode).into_bytes();
|
||||
for i in &e.instructions {
|
||||
normalized.extend(i.normalized.to_le_bytes());
|
||||
}
|
||||
let fingerprint = sha256(&normalized);
|
||||
let mut bound = vec![fingerprint.clone()];
|
||||
let mut resolved = true;
|
||||
for r in &e.references {
|
||||
let internal = r
|
||||
.target
|
||||
.filter(|t| *t >= input.address && *t < input.address + input.bytes.len() as u64);
|
||||
let identity = r
|
||||
.literal_sha256
|
||||
.clone()
|
||||
.or_else(|| r.identity.clone())
|
||||
.or_else(|| internal.map(|t| format!("local:{}", t - input.address)));
|
||||
if let Some(identity) = identity {
|
||||
bound.push(format!("{}:{}:{identity}", r.offset, r.kind));
|
||||
} else {
|
||||
resolved = false;
|
||||
}
|
||||
}
|
||||
if resolved {
|
||||
e.reference_bound_sha256 = Some(sha256(bound.join("|").as_bytes()));
|
||||
}
|
||||
e.masked_body_sha256 = Some(fingerprint);
|
||||
}
|
||||
Ok(e)
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
//! Offline CLI: verstack-code ELF {saved-functions.json|--symbols} OUTPUT.json
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use serde_json::{Value, json};
|
||||
use std::{collections::BTreeMap, io::Read};
|
||||
use verstack_code::{Mode, elf::ElfImage, sha256};
|
||||
fn bounded(path: &str, cap: usize) -> Result<Vec<u8>> {
|
||||
let mut bytes = vec![];
|
||||
std::fs::File::open(path)?
|
||||
.take(cap as u64 + 1)
|
||||
.read_to_end(&mut bytes)?;
|
||||
ensure!(bytes.len() <= cap, "input limit");
|
||||
Ok(bytes)
|
||||
}
|
||||
fn main() -> Result<()> {
|
||||
let args: Vec<_> = std::env::args().collect();
|
||||
ensure!(
|
||||
args.len() == 4,
|
||||
"usage: verstack-code ELF {{facts.json|--symbols}} OUTPUT.json"
|
||||
);
|
||||
let bytes = bounded(&args[1], 256 * 1024 * 1024)?;
|
||||
let facts = if args[2] == "--symbols" {
|
||||
None
|
||||
} else {
|
||||
Some(serde_json::from_slice::<Value>(&bounded(
|
||||
&args[2],
|
||||
512 * 1024 * 1024,
|
||||
)?)?)
|
||||
};
|
||||
let digest = sha256(&bytes);
|
||||
let expected = facts
|
||||
.as_ref()
|
||||
.map(|f| {
|
||||
f["input_sha256"]
|
||||
.as_str()
|
||||
.context("missing input fingerprint")
|
||||
})
|
||||
.transpose()?
|
||||
.unwrap_or(&digest);
|
||||
let mut image = ElfImage::parse(&bytes, expected)?;
|
||||
let mut rows = vec![];
|
||||
let mut failures = BTreeMap::<String, usize>::new();
|
||||
let functions = if let Some(f) = &facts {
|
||||
ensure!(
|
||||
f["schema"] == 1 && f["analysis_timed_out"] == false,
|
||||
"incomplete/unsupported saved facts"
|
||||
);
|
||||
let language = f["language"].as_str().context("missing ISA language")?;
|
||||
ensure!(
|
||||
(image.mode == Mode::Arm && language.starts_with("ARM:LE:32:"))
|
||||
|| (image.mode == Mode::Aarch64 && language.starts_with("AARCH64:LE:64:")),
|
||||
"facts architecture/mode conflicts with ELF"
|
||||
);
|
||||
// ARM TMode is not exported; skip Thumb rather than inferring its mode from bytes.
|
||||
f["functions"]
|
||||
.as_array()
|
||||
.context("missing functions")?
|
||||
.clone()
|
||||
} else {
|
||||
image.symbols.iter().map(|f|json!({"address":format!("{:x}",f.address),"size":f.size,"body_sha256":f.body_sha256,"name":f.name,"symbol_source":"ELF","thunk":false,"mode":f.mode})).collect()
|
||||
};
|
||||
ensure!(functions.len() <= 500_000, "function count limit");
|
||||
for f in &functions {
|
||||
let address = u64::from_str_radix(
|
||||
f["address"]
|
||||
.as_str()
|
||||
.context("missing address")?
|
||||
.trim_start_matches("0x"),
|
||||
16,
|
||||
)?;
|
||||
let size = usize::try_from(f["size"].as_u64().context("missing size")?)?;
|
||||
let body = f["body_sha256"].as_str().unwrap_or("");
|
||||
let mode = if facts.is_none() {
|
||||
serde_json::from_value(f["mode"].clone())?
|
||||
} else if let Some(mode) = image.known_mode(address) {
|
||||
mode
|
||||
} else {
|
||||
*failures.entry("unproven_arm_mode".into()).or_default() += 1;
|
||||
continue;
|
||||
};
|
||||
if f["thunk"] == true {
|
||||
*failures.entry("thunk_excluded".into()).or_default() += 1;
|
||||
continue;
|
||||
}
|
||||
if facts.is_some()
|
||||
&& !(f["signature_method"] == "flirt-operand-mask/1"
|
||||
&& f["signature_bytes"].as_str().is_some_and(|s| !s.is_empty()))
|
||||
{
|
||||
*failures
|
||||
.entry("unproven_contiguous_extent_or_mode".into())
|
||||
.or_default() += 1;
|
||||
continue;
|
||||
}
|
||||
let provenance = if facts.is_some() {
|
||||
"verified_saved_ghidra_contiguous_body"
|
||||
} else {
|
||||
"elf_sized_function_symbol"
|
||||
};
|
||||
match image.function_with_anchors(address,size,body,mode,provenance) {
|
||||
Ok((e,anchors))=>rows.push(json!({"anchor_evidence":anchors,"address":f["address"],"name":f["name"],"symbol_source":f["symbol_source"],"size":size,"body_sha256":e.body_sha256,"masked_body_sha256":e.masked_body_sha256,"reference_bound_sha256":e.reference_bound_sha256,"cfg_wl_sha256":e.cfg_wl_sha256,"cfg_shape_wl_sha256":e.cfg_shape_wl_sha256,"complete_decode":e.complete_decode,"complete_cfg":e.complete_cfg,"decoded_bytes":e.decoded_bytes,"reachable_bytes":e.reachable_bytes,"blocks":e.blocks.len(),"references":e.references,"masks":e.masks,"diagnostics":e.diagnostics})),
|
||||
Err(error)=>{*failures.entry(error.to_string()).or_default()+=1;}
|
||||
}
|
||||
}
|
||||
let report = json!({"method":verstack_code::METHOD,"decoder":verstack_code::DECODER,"input_sha256":digest,"mode":image.mode,"boundary_provenance":if facts.is_some(){"verified_saved_ghidra"}else{"sized_elf_symbols"},"input_records":functions.len(),"verified_bodies":rows.len(),"complete_cfg":rows.iter().filter(|r|r["complete_cfg"]==true).count(),"failures":failures,"functions":rows,"caveat":"Candidate signatures only. No automatic function matching, name propagation or complete boundary recovery. Thumb and cross-ISA matching unsupported."});
|
||||
std::fs::write(&args[3], serde_json::to_vec(&report)?)?;
|
||||
eprintln!(
|
||||
"{} verified, {} complete CFGs, failures={}",
|
||||
report["verified_bodies"], report["complete_cfg"], report["failures"]
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
use crate::*;
|
||||
fn signed(value: u64, bits: u32) -> i64 {
|
||||
((value << (64 - bits)) as i64) >> (64 - bits)
|
||||
}
|
||||
fn add(address: u64, delta: i64) -> Option<u64> {
|
||||
address.checked_add_signed(delta)
|
||||
}
|
||||
fn mask(e: &mut Evidence, index: usize, bits: u32, reason: &str) {
|
||||
e.instructions[index].normalized &= !bits;
|
||||
e.masks.push(Mask {
|
||||
offset: e.instructions[index].offset,
|
||||
bits,
|
||||
reason: reason.into(),
|
||||
});
|
||||
}
|
||||
fn in_body(input: &Input, target: u64) -> bool {
|
||||
target >= input.address && target < input.address + input.bytes.len() as u64
|
||||
}
|
||||
fn address_use(
|
||||
input: &Input,
|
||||
e: &Evidence,
|
||||
index: usize,
|
||||
register: Option<u16>,
|
||||
target: u64,
|
||||
) -> bool {
|
||||
memory(input, target, 1).is_some()
|
||||
&& (input
|
||||
.relocations
|
||||
.contains(&(input.address + e.instructions[index].offset))
|
||||
|| (index + 2 < e.instructions.len()
|
||||
&& register.is_some_and(|r| e.instructions[index + 2].memory_bases.contains(&r))
|
||||
&& !e.instructions[index + 2].conditional))
|
||||
}
|
||||
pub(crate) fn apply(input: &Input, e: &mut Evidence) {
|
||||
for index in 0..e.instructions.len() {
|
||||
let w = e.instructions[index].word;
|
||||
let offset = e.instructions[index].offset;
|
||||
let pc = input.address + offset;
|
||||
let flow = e.instructions[index].flow.clone();
|
||||
match flow {
|
||||
Flow::Call(target) => {
|
||||
let mode = if target.is_none() {
|
||||
None
|
||||
} else if input.mode == Mode::Arm && w >> 28 == 15 {
|
||||
Some(Mode::Thumb)
|
||||
} else {
|
||||
Some(input.mode)
|
||||
};
|
||||
e.references
|
||||
.push(reference(input, offset, "call", target, mode, 0));
|
||||
// Preserve internal targets: local calls may encode different recursion semantics.
|
||||
if target.is_some_and(|t| !in_body(input, t)) {
|
||||
if input.mode == Mode::Aarch64 && w & 0xfc000000 == 0x94000000 {
|
||||
mask(e, index, 0x03ffffff, "bl_target");
|
||||
} else if input.mode == Mode::Arm && w & 0xfe000000 == 0xfa000000 {
|
||||
mask(e, index, 0x01ffffff, "blx_target");
|
||||
} else if input.mode == Mode::Arm && w & 0x0f000000 == 0x0b000000 {
|
||||
mask(e, index, 0x00ffffff, "bl_target");
|
||||
}
|
||||
}
|
||||
}
|
||||
Flow::Branch(Some(target)) if !in_body(input, target) => {
|
||||
e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"tail_branch",
|
||||
Some(target),
|
||||
Some(input.mode),
|
||||
0,
|
||||
));
|
||||
let bits = e.instructions[index].branch_mask;
|
||||
if bits != 0 {
|
||||
mask(e, index, bits, "external_branch_target");
|
||||
}
|
||||
}
|
||||
Flow::Branch(None) => e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"indirect_branch",
|
||||
None,
|
||||
Some(input.mode),
|
||||
0,
|
||||
)),
|
||||
_ => {}
|
||||
}
|
||||
if input.mode == Mode::Arm {
|
||||
// A32 LDR immediate literal: P=1,W=0,Rn=PC,L=1,I=0. Offset sign is an address bit.
|
||||
if w >> 28 != 15 && w & 0x0f3f0000 == 0x051f0000 {
|
||||
let delta = i64::from(w & 0xfff) * if w & (1 << 23) != 0 { 1 } else { -1 };
|
||||
if let Some(target) = pc.checked_add(8).and_then(|p| add(p, delta)) {
|
||||
let width = if w & (1 << 22) != 0 { 1 } else { 4 };
|
||||
e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"literal_load",
|
||||
Some(target),
|
||||
None,
|
||||
width,
|
||||
));
|
||||
if memory(input, target, width).is_some() {
|
||||
mask(e, index, 0x00800fff, "pc_literal_displacement");
|
||||
} else {
|
||||
diag(e, "unmapped_literal_target");
|
||||
}
|
||||
}
|
||||
}
|
||||
// MOVW/MOVT are constants unless relocation or an actual subsequent memory base proves an address.
|
||||
if index + 1 < e.instructions.len() && w & 0x0ff00000 == 0x03000000 {
|
||||
let next = e.instructions[index + 1].word;
|
||||
if next & 0x0ff00000 == 0x03400000
|
||||
&& w >> 28 == 14
|
||||
&& next >> 28 == 14
|
||||
&& w & 0xf000 == next & 0xf000
|
||||
&& w & 0xf000 != 0xf000
|
||||
{
|
||||
let low = ((w >> 4) & 0xf000) | (w & 0xfff);
|
||||
let high = ((next >> 4) & 0xf000) | (next & 0xfff);
|
||||
let target = u64::from(low | (high << 16));
|
||||
if address_use(input, e, index, e.instructions[index].first_reg, target) {
|
||||
mask(e, index, 0x000f0fff, "movw_movt_address");
|
||||
mask(e, index + 1, 0x000f0fff, "movw_movt_address");
|
||||
e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"address_materialization",
|
||||
Some(target),
|
||||
None,
|
||||
0,
|
||||
));
|
||||
} else {
|
||||
diag(e, "movw_movt_constant_retained");
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// A64 literal loads (integer/SIMD); PRFM is an address hint, not a loaded literal.
|
||||
if w & 0x3b000000 == 0x18000000 {
|
||||
let width = match (w >> 26 & 1, w >> 30) {
|
||||
(0, 0) => 4,
|
||||
(0, 1) => 8,
|
||||
(0, 2) => 4,
|
||||
(1, 0) => 4,
|
||||
(1, 1) => 8,
|
||||
(1, 2) => 16,
|
||||
_ => 0,
|
||||
};
|
||||
if width > 0
|
||||
&& let Some(target) = add(pc, signed(u64::from((w >> 5) & 0x7ffff), 19) * 4)
|
||||
{
|
||||
e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"literal_load",
|
||||
Some(target),
|
||||
None,
|
||||
width,
|
||||
));
|
||||
if memory(input, target, width).is_some() {
|
||||
mask(e, index, 0x00ffffe0, "pc_literal_displacement");
|
||||
} else {
|
||||
diag(e, "unmapped_literal_target");
|
||||
}
|
||||
}
|
||||
}
|
||||
// Adjacent ADRP Xn; ADD Xn,Xn,#imm, with matching register/64-bit/non-flags.
|
||||
if index + 1 < e.instructions.len() && w & 0x9f000000 == 0x90000000 {
|
||||
let n = e.instructions[index + 1].word;
|
||||
let rd = w & 31;
|
||||
if n & 0xff800000 == 0x91000000 && n & 31 == rd && (n >> 5) & 31 == rd && rd != 31 {
|
||||
let imm = (u64::from((w >> 5) & 0x7ffff) << 2) | u64::from((w >> 29) & 3);
|
||||
let low =
|
||||
u64::from((n >> 10) & 0xfff) << if n & (1 << 22) != 0 { 12 } else { 0 };
|
||||
if let Some(target) =
|
||||
add(pc & !4095, signed(imm, 21) * 4096).and_then(|p| p.checked_add(low))
|
||||
{
|
||||
if address_use(input, e, index, e.instructions[index + 1].first_reg, target)
|
||||
{
|
||||
mask(e, index, 0x60ffffe0, "adrp_add_address");
|
||||
mask(e, index + 1, 0x003ffc00, "adrp_add_address");
|
||||
e.references.push(reference(
|
||||
input,
|
||||
offset,
|
||||
"address_materialization",
|
||||
Some(target),
|
||||
None,
|
||||
0,
|
||||
));
|
||||
} else {
|
||||
diag(e, "adrp_add_unproven_address_retained");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
use verstack_code::{elf::ElfImage, *};
|
||||
fn u16at(b: &mut [u8], p: usize, v: u16) {
|
||||
b[p..p + 2].copy_from_slice(&v.to_le_bytes())
|
||||
}
|
||||
fn u32at(b: &mut [u8], p: usize, v: u32) {
|
||||
b[p..p + 4].copy_from_slice(&v.to_le_bytes())
|
||||
}
|
||||
fn u64at(b: &mut [u8], p: usize, v: u64) {
|
||||
b[p..p + 8].copy_from_slice(&v.to_le_bytes())
|
||||
}
|
||||
fn fixture(a64: bool) -> (Vec<u8>, u64, Vec<u8>) {
|
||||
let mut b = vec![0; 512];
|
||||
b[..4].copy_from_slice(b"\x7fELF");
|
||||
b[4] = if a64 { 2 } else { 1 };
|
||||
b[5] = 1;
|
||||
b[6] = 1;
|
||||
u16at(&mut b, 16, 2);
|
||||
u16at(&mut b, 18, if a64 { 183 } else { 40 });
|
||||
u32at(&mut b, 20, 1);
|
||||
let base = if a64 { 0x400000 } else { 0x1000 };
|
||||
if a64 {
|
||||
u64at(&mut b, 24, base + 256);
|
||||
u64at(&mut b, 32, 64);
|
||||
u16at(&mut b, 52, 64);
|
||||
u16at(&mut b, 54, 56);
|
||||
u16at(&mut b, 56, 1);
|
||||
u16at(&mut b, 58, 64);
|
||||
u32at(&mut b, 64, 1);
|
||||
u32at(&mut b, 68, 5);
|
||||
u64at(&mut b, 80, base);
|
||||
u64at(&mut b, 96, 512);
|
||||
u64at(&mut b, 104, 512);
|
||||
u64at(&mut b, 112, 4096);
|
||||
} else {
|
||||
u32at(&mut b, 24, (base + 256) as u32);
|
||||
u32at(&mut b, 28, 52);
|
||||
u16at(&mut b, 40, 52);
|
||||
u16at(&mut b, 42, 32);
|
||||
u16at(&mut b, 44, 1);
|
||||
u16at(&mut b, 46, 40);
|
||||
u32at(&mut b, 52, 1);
|
||||
u32at(&mut b, 60, base as u32);
|
||||
u32at(&mut b, 68, 512);
|
||||
u32at(&mut b, 72, 512);
|
||||
u32at(&mut b, 76, 5);
|
||||
u32at(&mut b, 80, 4096);
|
||||
}
|
||||
let code = if a64 { 0xd65f03c0u32 } else { 0xe12fff1e }
|
||||
.to_le_bytes()
|
||||
.to_vec();
|
||||
b[256..260].copy_from_slice(&code);
|
||||
(b, base + 256, code)
|
||||
}
|
||||
#[test]
|
||||
fn verified_elf_extent_and_hash_are_mandatory() {
|
||||
for a64 in [false, true] {
|
||||
let (bytes, address, code) = fixture(a64);
|
||||
let hash = sha256(&bytes);
|
||||
let mut elf = ElfImage::parse(&bytes, &hash).unwrap();
|
||||
let mode = if a64 { Mode::Aarch64 } else { Mode::Arm };
|
||||
let evidence = elf
|
||||
.function(address, 4, &sha256(&code), mode, "test:ELF extent")
|
||||
.unwrap();
|
||||
assert!(evidence.complete_cfg);
|
||||
assert!(evidence.masked_body_sha256.is_some());
|
||||
assert!(ElfImage::parse(&bytes, "wrong").is_err());
|
||||
assert!(elf.function(address, 4, "wrong", mode, "fixture").is_err());
|
||||
assert!(
|
||||
elf.function(address + 1024, 4, &sha256(&code), mode, "fixture")
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
elf.function(address, usize::MAX, "wrong", mode, "fixture")
|
||||
.is_err()
|
||||
);
|
||||
if a64 {
|
||||
assert!(
|
||||
elf.function(address, 4, &sha256(&code), Mode::Arm, "fixture")
|
||||
.is_err()
|
||||
);
|
||||
} else {
|
||||
assert!(elf.known_mode(address).is_none());
|
||||
}
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn malformed_segments_and_unsupported_machine_fail_explicitly() {
|
||||
let (mut b, _, _) = fixture(true);
|
||||
u64at(&mut b, 96, u64::MAX);
|
||||
assert!(ElfImage::parse(&b, &sha256(&b)).is_err());
|
||||
let (mut b, _, _) = fixture(true);
|
||||
u16at(&mut b, 18, 62);
|
||||
assert!(ElfImage::parse(&b, &sha256(&b)).is_err());
|
||||
let (mut b, _, _) = fixture(true);
|
||||
u32at(&mut b, 68, 4);
|
||||
let mut elf = ElfImage::parse(&b, &sha256(&b)).unwrap();
|
||||
assert!(
|
||||
elf.function(0x400100, 4, "wrong", Mode::Aarch64, "fixture")
|
||||
.is_err()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nonallocated_debug_mapping_symbols_cannot_override_executable_arm_mode() {
|
||||
let (mut b, address, code) = fixture(false);
|
||||
b.resize(0x4000, 0);
|
||||
u32at(&mut b, 32, 512);
|
||||
u16at(&mut b, 48, 5);
|
||||
// .text, nonallocated debug bytes, .symtab, .strtab; no section-name table needed.
|
||||
let text = 512 + 40;
|
||||
u32at(&mut b, text + 4, 1);
|
||||
u32at(&mut b, text + 8, 6);
|
||||
u32at(&mut b, text + 12, address as u32);
|
||||
u32at(&mut b, text + 16, 256);
|
||||
u32at(&mut b, text + 20, 4);
|
||||
let debug = 512 + 80;
|
||||
u32at(&mut b, debug + 4, 1);
|
||||
u32at(&mut b, debug + 16, 0x1000);
|
||||
u32at(&mut b, debug + 20, 0x2000);
|
||||
let sym = 512 + 120;
|
||||
u32at(&mut b, sym + 4, 2);
|
||||
u32at(&mut b, sym + 16, 720);
|
||||
u32at(&mut b, sym + 20, 48);
|
||||
u32at(&mut b, sym + 24, 4);
|
||||
u32at(&mut b, sym + 28, 3);
|
||||
u32at(&mut b, sym + 36, 16);
|
||||
let strings = 512 + 160;
|
||||
u32at(&mut b, strings + 4, 3);
|
||||
u32at(&mut b, strings + 16, 768);
|
||||
u32at(&mut b, strings + 20, 7);
|
||||
b[768..775].copy_from_slice(b"\0$a\0$d\0");
|
||||
u32at(&mut b, 736, 1);
|
||||
u32at(&mut b, 740, address as u32);
|
||||
u16at(&mut b, 750, 1);
|
||||
u32at(&mut b, 752, 4);
|
||||
u16at(&mut b, 766, 2);
|
||||
let mut image = ElfImage::parse(&b, &sha256(&b)).unwrap();
|
||||
assert_eq!(image.known_mode(address), Some(Mode::Arm));
|
||||
assert!(
|
||||
image
|
||||
.function(address, 4, &sha256(&code), Mode::Arm, "ELF mapping fixture")
|
||||
.unwrap()
|
||||
.complete_cfg
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
[
|
||||
{
|
||||
"name": "_ZL21should_post_be_raisedv",
|
||||
"elf_sha256": "637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"input": {
|
||||
"mode": "arm",
|
||||
"address": 324588,
|
||||
"bytes": [
|
||||
16,
|
||||
64,
|
||||
45,
|
||||
233,
|
||||
5,
|
||||
0,
|
||||
160,
|
||||
227,
|
||||
23,
|
||||
128,
|
||||
255,
|
||||
235,
|
||||
0,
|
||||
64,
|
||||
160,
|
||||
225,
|
||||
237,
|
||||
206,
|
||||
255,
|
||||
235,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
10,
|
||||
1,
|
||||
0,
|
||||
160,
|
||||
227,
|
||||
16,
|
||||
128,
|
||||
189,
|
||||
232,
|
||||
230,
|
||||
171,
|
||||
0,
|
||||
235,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
250,
|
||||
255,
|
||||
255,
|
||||
26,
|
||||
0,
|
||||
48,
|
||||
148,
|
||||
229,
|
||||
4,
|
||||
0,
|
||||
160,
|
||||
225,
|
||||
72,
|
||||
48,
|
||||
147,
|
||||
229,
|
||||
51,
|
||||
255,
|
||||
47,
|
||||
225,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
16,
|
||||
128,
|
||||
189,
|
||||
8,
|
||||
243,
|
||||
255,
|
||||
255,
|
||||
234
|
||||
],
|
||||
"expected_body_sha256": "ad048f4c782ea858bcd499d66cda43abd896795bac4a83424b3105a4601c6613",
|
||||
"boundary_provenance": "sized ELF symbol verified in 637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"memory": [],
|
||||
"relocations": [],
|
||||
"target_identities": {
|
||||
"193624": "_Z13get_multiballj",
|
||||
"274360": "_Z24can_we_start_mystery_nowv",
|
||||
"500656": "_Z32sys_extra_ball_is_extra_ball_litv"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "_Z26should_send_ball_to_thronev",
|
||||
"elf_sha256": "240db7bed730b8958c8d1d34e160e296d2306fda22533e5f1461df389e65d410",
|
||||
"input": {
|
||||
"mode": "arm",
|
||||
"address": 133028,
|
||||
"bytes": [
|
||||
16,
|
||||
64,
|
||||
45,
|
||||
233,
|
||||
5,
|
||||
0,
|
||||
160,
|
||||
227,
|
||||
27,
|
||||
68,
|
||||
0,
|
||||
235,
|
||||
0,
|
||||
64,
|
||||
160,
|
||||
225,
|
||||
192,
|
||||
150,
|
||||
0,
|
||||
235,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
10,
|
||||
1,
|
||||
0,
|
||||
160,
|
||||
227,
|
||||
16,
|
||||
128,
|
||||
189,
|
||||
232,
|
||||
248,
|
||||
126,
|
||||
1,
|
||||
235,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
250,
|
||||
255,
|
||||
255,
|
||||
26,
|
||||
0,
|
||||
48,
|
||||
148,
|
||||
229,
|
||||
4,
|
||||
0,
|
||||
160,
|
||||
225,
|
||||
72,
|
||||
48,
|
||||
147,
|
||||
229,
|
||||
51,
|
||||
255,
|
||||
47,
|
||||
225,
|
||||
0,
|
||||
0,
|
||||
80,
|
||||
227,
|
||||
16,
|
||||
128,
|
||||
189,
|
||||
8,
|
||||
243,
|
||||
255,
|
||||
255,
|
||||
234
|
||||
],
|
||||
"expected_body_sha256": "3d4068cc9b2376e122a1a3efec28ef43d3555e4c5d77d53c621c857807a0befb",
|
||||
"boundary_provenance": "sized ELF symbol verified in 240db7bed730b8958c8d1d34e160e296d2306fda22533e5f1461df389e65d410",
|
||||
"memory": [],
|
||||
"relocations": [],
|
||||
"target_identities": {
|
||||
"202784": "_Z13get_multiballj",
|
||||
"287420": "_Z24can_we_start_mystery_nowv",
|
||||
"525232": "_Z32sys_extra_ball_is_extra_ball_litv"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,315 @@
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
use verstack_code::*;
|
||||
fn input(mode: Mode, address: u64, words: &[u32]) -> Input {
|
||||
let bytes: Vec<u8> = words.iter().flat_map(|w| w.to_le_bytes()).collect();
|
||||
Input {
|
||||
mode,
|
||||
address,
|
||||
expected_body_sha256: sha256(&bytes),
|
||||
bytes,
|
||||
boundary_provenance: "test:verified-contiguous-extent".into(),
|
||||
memory: vec![],
|
||||
relocations: BTreeSet::new(),
|
||||
target_identities: BTreeMap::new(),
|
||||
}
|
||||
}
|
||||
fn a64_bl(pc: u64, target: u64) -> u32 {
|
||||
0x94000000 | (((target as i64 - pc as i64) / 4) as u32 & 0x03ffffff)
|
||||
}
|
||||
fn movw(reg: u32, n: u16) -> u32 {
|
||||
0xe3000000 | ((u32::from(n) & 0xf000) << 4) | (reg << 12) | (u32::from(n) & 0xfff)
|
||||
}
|
||||
fn movt(reg: u32, n: u16) -> u32 {
|
||||
movw(reg, n) | 0x00400000
|
||||
}
|
||||
#[test]
|
||||
fn relocated_calls_are_candidates_but_changed_callees_are_not_reference_bound_matches() {
|
||||
let mut a = input(Mode::Aarch64, 0x1000, &[a64_bl(0x1000, 0x8000), 0xd65f03c0]);
|
||||
let mut b = input(Mode::Aarch64, 0x2000, &[a64_bl(0x2000, 0xa000), 0xd65f03c0]);
|
||||
let x = analyze(&a).unwrap();
|
||||
let y = analyze(&b).unwrap();
|
||||
assert_eq!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
assert!(x.masked_body_sha256.is_some());
|
||||
assert!(x.reference_bound_sha256.is_none());
|
||||
assert_eq!(x.references[0].target, Some(0x8000));
|
||||
a.target_identities.insert(0x8000, "malloc".into());
|
||||
b.target_identities.insert(0xa000, "malloc".into());
|
||||
assert_eq!(
|
||||
analyze(&a).unwrap().reference_bound_sha256,
|
||||
analyze(&b).unwrap().reference_bound_sha256
|
||||
);
|
||||
b.target_identities.insert(0xa000, "free".into());
|
||||
assert_ne!(
|
||||
analyze(&a).unwrap().reference_bound_sha256,
|
||||
analyze(&b).unwrap().reference_bound_sha256
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn a32_bl_blx_mask_only_immediate_and_preserve_mode_condition_and_register_calls() {
|
||||
for word in [0xeb000010, 0xfa000010] {
|
||||
let a = analyze(&input(Mode::Arm, 0x1000, &[word, 0xe12fff1e])).unwrap();
|
||||
let b = analyze(&input(Mode::Arm, 0x1000, &[word + 3, 0xe12fff1e])).unwrap();
|
||||
assert_eq!(a.masked_body_sha256, b.masked_body_sha256);
|
||||
assert!(a.masked_body_sha256.is_some());
|
||||
assert_eq!(
|
||||
a.references[0].target_mode,
|
||||
Some(if word == 0xfa000010 {
|
||||
Mode::Thumb
|
||||
} else {
|
||||
Mode::Arm
|
||||
})
|
||||
);
|
||||
}
|
||||
let unconditional = analyze(&input(Mode::Arm, 0x1000, &[0xeb000010, 0xe12fff1e])).unwrap();
|
||||
let conditional = analyze(&input(Mode::Arm, 0x1000, &[0x0b000010, 0xe12fff1e])).unwrap();
|
||||
assert_ne!(
|
||||
unconditional.masked_body_sha256,
|
||||
conditional.masked_body_sha256
|
||||
);
|
||||
let x = analyze(&input(Mode::Arm, 0x1000, &[0xe12fff33, 0xe12fff1e])).unwrap();
|
||||
let y = analyze(&input(Mode::Arm, 0x1000, &[0xe12fff34, 0xe12fff1e])).unwrap();
|
||||
assert_ne!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
assert!(x.references[0].target.is_none());
|
||||
}
|
||||
#[test]
|
||||
fn literal_displacements_move_but_loaded_values_remain_independent_evidence() {
|
||||
for (mode, words, target) in [
|
||||
(Mode::Arm, vec![0xe59f0010, 0xe12fff1e], 0x1018),
|
||||
(Mode::Aarch64, vec![0x18000100, 0xd65f03c0], 0x1020),
|
||||
] {
|
||||
let mut a = input(mode, 0x1000, &words);
|
||||
a.memory.push(MemoryRange {
|
||||
start: target,
|
||||
bytes: 42u32.to_le_bytes().to_vec(),
|
||||
});
|
||||
let mut b = a.clone();
|
||||
b.memory[0].bytes = 43u32.to_le_bytes().to_vec();
|
||||
let x = analyze(&a).unwrap();
|
||||
let y = analyze(&b).unwrap();
|
||||
assert_eq!(x.masks.len(), 1);
|
||||
assert_eq!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
assert_ne!(x.reference_bound_sha256, y.reference_bound_sha256);
|
||||
assert!(x.reference_bound_sha256.is_some());
|
||||
let unknown = analyze(&input(mode, 0x1000, &words)).unwrap();
|
||||
assert!(unknown.masks.is_empty());
|
||||
assert!(unknown.reference_bound_sha256.is_none());
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn movw_movt_constants_are_not_addresses_just_because_they_fall_in_a_mapping() {
|
||||
let mut a = input(
|
||||
Mode::Arm,
|
||||
0x1000,
|
||||
&[movw(0, 0x8000), movt(0, 0), 0xe12fff1e],
|
||||
);
|
||||
a.memory.push(MemoryRange {
|
||||
start: 0x8000,
|
||||
bytes: vec![1; 32],
|
||||
});
|
||||
let mut b = input(
|
||||
Mode::Arm,
|
||||
0x1000,
|
||||
&[movw(0, 0x8004), movt(0, 0), 0xe12fff1e],
|
||||
);
|
||||
b.memory = a.memory.clone();
|
||||
let x = analyze(&a).unwrap();
|
||||
let y = analyze(&b).unwrap();
|
||||
assert!(x.masks.is_empty());
|
||||
assert_ne!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
let mut address = input(
|
||||
Mode::Arm,
|
||||
0x1000,
|
||||
&[movw(0, 0x8000), movt(0, 0), 0xe5901000, 0xe12fff1e],
|
||||
);
|
||||
address.memory = a.memory;
|
||||
let out = analyze(&address).unwrap();
|
||||
assert_eq!(out.masks.len(), 2);
|
||||
assert_eq!(out.references[0].target, Some(0x8000));
|
||||
// An unrelated memory register cannot prove r0 is an address.
|
||||
let mut wrong = input(
|
||||
Mode::Arm,
|
||||
0x1000,
|
||||
&[movw(0, 0x8000), movt(0, 0), 0xe5921000, 0xe12fff1e],
|
||||
);
|
||||
wrong.memory = address.memory;
|
||||
assert!(analyze(&wrong).unwrap().masks.is_empty());
|
||||
}
|
||||
#[test]
|
||||
fn adrp_add_pair_requires_address_use_and_preserves_ordinary_add_constants() {
|
||||
// ADRP x0,0x8000 from0x1000; ADD x0,x0,#16; LDR x1,[x0]; RET.
|
||||
let mut a = input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[0xf0000020, 0x91004000, 0xf9400001, 0xd65f03c0],
|
||||
);
|
||||
a.memory.push(MemoryRange {
|
||||
start: 0x8000,
|
||||
bytes: vec![0; 64],
|
||||
});
|
||||
let x = analyze(&a).unwrap();
|
||||
assert_eq!(x.masks.len(), 2, "{:?}", x);
|
||||
assert_eq!(x.references[0].target, Some(0x8010));
|
||||
let mut b = a.clone();
|
||||
b.bytes[4..8].copy_from_slice(&0x91008000u32.to_le_bytes());
|
||||
b.expected_body_sha256 = sha256(&b.bytes);
|
||||
let y = analyze(&b).unwrap();
|
||||
assert_eq!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
let mut constant = input(Mode::Aarch64, 0x1000, &[0xf0000020, 0x91004000, 0xd65f03c0]);
|
||||
constant.memory = a.memory;
|
||||
assert!(analyze(&constant).unwrap().masks.is_empty());
|
||||
assert_ne!(
|
||||
analyze(&input(Mode::Aarch64, 0x1000, &[0x91000400, 0xd65f03c0]))
|
||||
.unwrap()
|
||||
.masked_body_sha256,
|
||||
analyze(&input(Mode::Aarch64, 0x1000, &[0x91000800, 0xd65f03c0]))
|
||||
.unwrap()
|
||||
.masked_body_sha256
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn cfg_preserves_conditions_topology_and_semantic_instruction_labels() {
|
||||
// CBZ w0,+12; ADD x1,x1,#1; B +8; SUB x1,x1,#1; RET
|
||||
let a = input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[0x34000060, 0x91000421, 0x14000002, 0xd1000421, 0xd65f03c0],
|
||||
);
|
||||
let mut moved = a.clone();
|
||||
moved.address = 0x5000;
|
||||
let x = analyze(&a).unwrap();
|
||||
let y = analyze(&moved).unwrap();
|
||||
assert_eq!(x.blocks.len(), 4);
|
||||
assert_eq!(x.cfg_wl_sha256, y.cfg_wl_sha256);
|
||||
assert!(x.cfg_wl_sha256.is_some());
|
||||
let z = analyze(&input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[0x35000060, 0x91000421, 0x14000002, 0xd1000421, 0xd65f03c0],
|
||||
))
|
||||
.unwrap();
|
||||
assert_ne!(x.cfg_wl_sha256, z.cfg_wl_sha256);
|
||||
let semantic = analyze(&input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[0x34000060, 0x91000821, 0x14000002, 0xd1000421, 0xd65f03c0],
|
||||
))
|
||||
.unwrap();
|
||||
assert_ne!(x.cfg_wl_sha256, semantic.cfg_wl_sha256);
|
||||
assert_eq!(x.cfg_shape_wl_sha256, semantic.cfg_shape_wl_sha256);
|
||||
}
|
||||
#[test]
|
||||
fn unsupported_incomplete_unreachable_and_changed_bodies_never_get_complete_hashes() {
|
||||
let thumb = analyze(&input(Mode::Thumb, 0x1000, &[0x47702000])).unwrap();
|
||||
assert!(!thumb.complete_decode);
|
||||
assert!(thumb.masked_body_sha256.is_none());
|
||||
let truncated = analyze(&input(Mode::Aarch64, 0x1000, &[0xffffffff, 0xd65f03c0])).unwrap();
|
||||
assert!(!truncated.complete_decode);
|
||||
assert!(truncated.masked_body_sha256.is_none());
|
||||
let jump = analyze(&input(Mode::Aarch64, 0x1000, &[0xd61f0000])).unwrap();
|
||||
assert!(!jump.complete_cfg);
|
||||
assert!(jump.cfg_wl_sha256.is_none());
|
||||
let tail = analyze(&input(Mode::Aarch64, 0x1000, &[0xd65f03c0, 0xd503201f])).unwrap();
|
||||
assert_eq!(tail.reachable_bytes, 4);
|
||||
assert!(tail.masked_body_sha256.is_none());
|
||||
let mut wrong = input(Mode::Aarch64, 0x1000, &[0xd65f03c0]);
|
||||
wrong.bytes[0] ^= 1;
|
||||
assert!(analyze(&wrong).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn wl_detects_rewired_backedge_with_identical_block_instruction_labels() {
|
||||
let diamond = analyze(&input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[
|
||||
0x34000060, 0x91000421, 0x14000003, 0xd1000421, 0x14000001, 0xd65f03c0,
|
||||
],
|
||||
))
|
||||
.unwrap();
|
||||
let looped = analyze(&input(
|
||||
Mode::Aarch64,
|
||||
0x1000,
|
||||
&[
|
||||
0x34000060, 0x91000421, 0x17fffffe, 0xd1000421, 0x14000001, 0xd65f03c0,
|
||||
],
|
||||
))
|
||||
.unwrap();
|
||||
assert!(diamond.complete_cfg && looped.complete_cfg);
|
||||
assert_eq!(
|
||||
diamond
|
||||
.blocks
|
||||
.iter()
|
||||
.map(|b| &b.instruction_hash)
|
||||
.collect::<Vec<_>>(),
|
||||
looped
|
||||
.blocks
|
||||
.iter()
|
||||
.map(|b| &b.instruction_hash)
|
||||
.collect::<Vec<_>>()
|
||||
);
|
||||
assert_ne!(diamond.cfg_wl_sha256, looped.cfg_wl_sha256);
|
||||
assert_ne!(diamond.cfg_shape_wl_sha256, looped.cfg_shape_wl_sha256);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn literal_masks_keep_register_width_and_condition_and_relocation_proof_is_explicit() {
|
||||
let make = |word| {
|
||||
let mut f = input(Mode::Arm, 0x1000, &[word, 0xe12fff1e]);
|
||||
f.memory.push(MemoryRange {
|
||||
start: 0x1018,
|
||||
bytes: vec![42; 8],
|
||||
});
|
||||
analyze(&f).unwrap()
|
||||
};
|
||||
let base = make(0xe59f0010);
|
||||
for word in [0xe59f1010, 0xe5df0010, 0x059f0010] {
|
||||
let changed = make(word);
|
||||
assert!(changed.masked_body_sha256.is_some());
|
||||
assert_ne!(base.masked_body_sha256, changed.masked_body_sha256);
|
||||
}
|
||||
let mut f = input(
|
||||
Mode::Arm,
|
||||
0x1000,
|
||||
&[movw(0, 0x8000), movt(0, 0), 0xe12fff1e],
|
||||
);
|
||||
f.memory.push(MemoryRange {
|
||||
start: 0x8000,
|
||||
bytes: vec![0; 16],
|
||||
});
|
||||
assert!(analyze(&f).unwrap().masks.is_empty());
|
||||
f.relocations.insert(0x1000);
|
||||
assert_eq!(analyze(&f).unwrap().masks.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn real_named_mask_collision_is_not_reference_bound_without_indirect_call_identity() {
|
||||
let fixtures: serde_json::Value =
|
||||
serde_json::from_str(include_str!("fixtures/real_named_ambiguity.json")).unwrap();
|
||||
let a: Input = serde_json::from_value(fixtures[0]["input"].clone()).unwrap();
|
||||
let b: Input = serde_json::from_value(fixtures[1]["input"].clone()).unwrap();
|
||||
let x = analyze(&a).unwrap();
|
||||
let y = analyze(&b).unwrap();
|
||||
assert_ne!(fixtures[0]["name"], fixtures[1]["name"]);
|
||||
assert_eq!(x.masked_body_sha256, y.masked_body_sha256);
|
||||
assert!(x.masked_body_sha256.is_some());
|
||||
assert!(x.reference_bound_sha256.is_none() && y.reference_bound_sha256.is_none());
|
||||
assert!(
|
||||
x.references
|
||||
.iter()
|
||||
.any(|r| r.kind == "call" && r.target.is_none())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prefetch_is_not_a_literal_load_and_arm_addresses_cannot_exceed_the_isa_width() {
|
||||
let mut f = input(Mode::Arm, 0x1000, &[0xf5dff010, 0xe12fff1e]);
|
||||
f.memory.push(MemoryRange {
|
||||
start: 0x1018,
|
||||
bytes: vec![42; 8],
|
||||
});
|
||||
let e = analyze(&f).unwrap();
|
||||
assert!(e.complete_decode);
|
||||
assert!(e.masks.is_empty());
|
||||
assert!(!e.references.iter().any(|r| r.kind == "literal_load"));
|
||||
assert!(analyze(&input(Mode::Arm, 0x1_0000_1000, &[0xe12fff1e])).is_err());
|
||||
}
|
||||
Generated
+365
@@ -0,0 +1,365 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "aes"
|
||||
version = "0.8.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cipher",
|
||||
"cpufeatures 0.2.17",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anyhow"
|
||||
version = "1.0.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
||||
|
||||
[[package]]
|
||||
name = "bitflags"
|
||||
version = "2.13.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
|
||||
|
||||
[[package]]
|
||||
name = "block-buffer"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "cipher"
|
||||
version = "0.4.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
|
||||
dependencies = [
|
||||
"crypto-common 0.1.7",
|
||||
"inout",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "const-oid"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.1.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||
dependencies = [
|
||||
"block-buffer",
|
||||
"const-oid",
|
||||
"crypto-common 0.2.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "errno"
|
||||
version = "0.3.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastrand"
|
||||
version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
|
||||
|
||||
[[package]]
|
||||
name = "generic-array"
|
||||
version = "0.14.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
"version_check",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.4.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"r-efi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hybrid-array"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "inout"
|
||||
version = "0.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "libc"
|
||||
version = "0.2.189"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
|
||||
|
||||
[[package]]
|
||||
name = "linux-raw-sys"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.47"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "r-efi"
|
||||
version = "6.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
|
||||
|
||||
[[package]]
|
||||
name = "rustix"
|
||||
version = "1.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"errno",
|
||||
"libc",
|
||||
"linux-raw-sys",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.151"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.3.1",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tempfile"
|
||||
version = "3.27.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
|
||||
dependencies = [
|
||||
"fastrand",
|
||||
"getrandom",
|
||||
"once_cell",
|
||||
"rustix",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.25"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ab72a15cf68d77cb0987d3684aa8a45c5ef827e8cb49ee2f30bfd7ba2feb519f"
|
||||
|
||||
[[package]]
|
||||
name = "version_check"
|
||||
version = "0.9.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
|
||||
|
||||
[[package]]
|
||||
name = "verstack-luks"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"aes",
|
||||
"anyhow",
|
||||
"libc",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"tempfile",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-link"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.61.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zeroize"
|
||||
version = "1.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e"
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
@@ -0,0 +1,15 @@
|
||||
[package]
|
||||
name = "verstack-luks"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
aes = { version = "=0.8.4", features = ["zeroize"] }
|
||||
zeroize = "=1.9.0"
|
||||
libc = "0.2"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
sha2 = "0.11"
|
||||
tempfile = "3"
|
||||
@@ -0,0 +1,65 @@
|
||||
# verstack-luks
|
||||
|
||||
A bounded `Read + Seek` plaintext view over a single LUKS2 `aes-xts-plain64`
|
||||
segment. `Slice<R>` confines archived partition coordinates, and `LuksReader<R>`
|
||||
reuses expanded AES keys with a bounded, zeroized plaintext cache. Every XTS data
|
||||
unit gets the little-endian plain64 tweak measured in 512-byte sectors, including
|
||||
4096-byte data units. It does not activate a kernel mapping or restore an image.
|
||||
|
||||
`Capsule::read` copies only bounded encrypted metadata/keyslot bytes to a private
|
||||
sparse temporary file. Its logical length preserves device geometry. The loaded
|
||||
library is SHA-256 pinned and opened through its verified file descriptor;
|
||||
libcryptsetup validates metadata and retrieves a volume key in process. Only
|
||||
public loading, inspection and key-retrieval APIs are bound; no activation or
|
||||
mutation API is exposed. The verified metadata must match the bounded preflight.
|
||||
The [upstream public API declarations](https://raw.githubusercontent.com/mbroz/cryptsetup/master/lib/libcryptsetup.h)
|
||||
define these bindings and the sector units used for geometry checks.
|
||||
|
||||
The capsule reports KDF CPU/memory/work demand before `unlock`. Callers must
|
||||
acquire those resources and pass the granted limits. PBKDF2 and Argon2 are
|
||||
bounded, unsupported profiles fail explicitly, and key material uses zeroizing
|
||||
buffers. Cancellation is checked between source operations and keyslots; an
|
||||
in-progress libcryptsetup KDF call is not preemptible. Library logs are suppressed
|
||||
and fixed errors never include credentials or volume keys. The library pin is
|
||||
for libcryptsetup itself, not its complete dynamic dependency closure.
|
||||
|
||||
Supported geometry is 512/4096-byte sectors and 32/64-byte XTS keys. Multiple or
|
||||
reencryption segments, detached/unbound keyslots, integrity profiles and required
|
||||
features are rejected. The encrypted capsule is capped at 32 MiB; directory JSON
|
||||
at 4 MiB. Sparse logical length is not a promise to allocate the whole partition.
|
||||
No LUKS1, crash-journal repair, or general container pipeline completion is claimed.
|
||||
|
||||
## Evidence
|
||||
|
||||
Four portable host tests cover independent OpenSSL ciphertext fixtures, random
|
||||
seeks and cross-sector reads, both key/data-unit sizes, 32-bit IV crossing,
|
||||
invalid geometry, cancellation and failed-refill cache safety. A cryptsetup-built
|
||||
synthetic fixture checks library pinning, actual unlock/wrong credentials and
|
||||
KDF admission, including volume-key digest work and Argon2 geometry. Test prerequisites are cryptsetup and the pinned library; fixture
|
||||
credentials are explicitly public test values.
|
||||
|
||||
The retained 63,281,562,112-byte Pokémon 0.83 source was probed through exact HTTP
|
||||
ranges, without restoration. All four encrypted partitions (2, 3, 5, 6) unlocked
|
||||
and produced an ext4 superblock using 163,840 encrypted metadata/keyslot bytes
|
||||
plus 4,096 ciphertext payload bytes per partition. All use PBKDF2/250,000,
|
||||
32-byte volume keys, 512-byte sectors and a 16 MiB payload offset. This first
|
||||
probe verifies headers, not full filesystems. Its receipt is
|
||||
`data/validation/pokemon083-native-luks-probe.json` at the repository root.
|
||||
|
||||
```
|
||||
cargo test --manifest-path crates/verstack-luks/Cargo.toml --offline --locked
|
||||
cargo clippy --manifest-path crates/verstack-luks/Cargo.toml --offline --locked --all-targets -- -D warnings
|
||||
```
|
||||
|
||||
`examples/inspect_archive.rs` is an explicit local read-only corpus probe. It reads
|
||||
the configured disk credential privately and requires an explicit
|
||||
`VERSTACK_LIBCRYPTSETUP_SHA256` pin; its range helper never receives credentials.
|
||||
|
||||
A subsequent root integration probe fully extracted the real small partition 3:
|
||||
92 regular files (280,825 bytes), all read back identically through the native
|
||||
plaintext filesystem view. It used an 8 MiB bounded cache, 8,388,608 ciphertext
|
||||
payload bytes, and the same small encrypted capsule; disposable output was
|
||||
removed. See `data/validation/pokemon083-native-luks-ext4-part3.json`. This is
|
||||
not an independent legacy-file oracle and does not cover full extraction of
|
||||
large partitions 5/6. Root tests independently encrypt an ext4 fixture with
|
||||
OpenSSL and verify guarded archive publication and original preservation.
|
||||
@@ -0,0 +1,161 @@
|
||||
//! Explicit local read-only probe: bounded encrypted ranges, credentials only in memory.
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use std::{
|
||||
io::{Read, Seek, SeekFrom, Write},
|
||||
path::PathBuf,
|
||||
process::{Child, ChildStdin, ChildStdout, Command, Stdio},
|
||||
};
|
||||
use verstack_luks::{Capsule, LibraryPin, LuksReader, Slice, UnlockLimits};
|
||||
use zeroize::Zeroizing;
|
||||
struct Remote {
|
||||
child: Child,
|
||||
input: ChildStdin,
|
||||
output: ChildStdout,
|
||||
position: u64,
|
||||
size: u64,
|
||||
bytes: u64,
|
||||
}
|
||||
impl Read for Remote {
|
||||
fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
|
||||
let n = out
|
||||
.len()
|
||||
.min(self.size.saturating_sub(self.position) as usize);
|
||||
if n == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
writeln!(self.input, "[{},{}]", self.position, n)?;
|
||||
self.input.flush()?;
|
||||
self.output.read_exact(&mut out[..n])?;
|
||||
self.position += n as u64;
|
||||
self.bytes += n as u64;
|
||||
Ok(n)
|
||||
}
|
||||
}
|
||||
impl Seek for Remote {
|
||||
fn seek(&mut self, from: SeekFrom) -> std::io::Result<u64> {
|
||||
let p = match from {
|
||||
SeekFrom::Start(n) => n as i128,
|
||||
SeekFrom::Current(n) => self.position as i128 + n as i128,
|
||||
SeekFrom::End(n) => self.size as i128 + n as i128,
|
||||
};
|
||||
self.position = u64::try_from(p).map_err(std::io::Error::other)?;
|
||||
Ok(self.position)
|
||||
}
|
||||
}
|
||||
impl Drop for Remote {
|
||||
fn drop(&mut self) {
|
||||
let _ = self.child.kill();
|
||||
let _ = self.child.wait();
|
||||
}
|
||||
}
|
||||
fn main() -> Result<()> {
|
||||
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||
let map: serde_json::Value = serde_json::from_slice(&std::fs::read(
|
||||
root.join("data/validation/pokemon083-disk-map.json"),
|
||||
)?)?;
|
||||
let config: serde_json::Value =
|
||||
serde_json::from_slice(&std::fs::read(root.join("config.json"))?)?;
|
||||
let settings = &config["plugins"]["import-extract"]["settings"];
|
||||
ensure!(
|
||||
settings["disk_key_encoding"] == "vcmailbox",
|
||||
"explicit corpus probe requires configured vcmailbox credential encoding"
|
||||
);
|
||||
let key_path = settings["disk_key_file"]
|
||||
.as_str()
|
||||
.context("missing configured disk credential reference")?;
|
||||
let mut raw = Zeroizing::new(Vec::new());
|
||||
std::fs::File::open(key_path)?
|
||||
.take(65537)
|
||||
.read_to_end(&mut raw)?;
|
||||
ensure!(raw.len() <= 65536, "credential too large");
|
||||
let words = Zeroizing::new(
|
||||
std::str::from_utf8(&raw)?
|
||||
.split_whitespace()
|
||||
.map(|s| {
|
||||
u32::from_str_radix(
|
||||
s.strip_prefix("0x")
|
||||
.context("invalid credential encoding")?,
|
||||
16,
|
||||
)
|
||||
.map_err(Into::into)
|
||||
})
|
||||
.collect::<Result<Vec<u32>>>()?,
|
||||
);
|
||||
ensure!(
|
||||
words.len() == 16
|
||||
&& words[..7] == [64, 0x80000000, 0x00030021, 40, 0x80000028, 0, 8]
|
||||
&& words[15] == 0,
|
||||
"invalid mailbox credential framing"
|
||||
);
|
||||
let credential = Zeroizing::new(
|
||||
words[7..15]
|
||||
.iter()
|
||||
.flat_map(|w| w.to_be_bytes())
|
||||
.collect::<Vec<_>>(),
|
||||
);
|
||||
let expected = std::env::var("VERSTACK_LIBCRYPTSETUP_SHA256")
|
||||
.context("explicit library SHA256 pin required")?;
|
||||
let pin = LibraryPin {
|
||||
path: "/usr/lib/x86_64-linux-gnu/libcryptsetup.so.12".into(),
|
||||
sha256: expected,
|
||||
};
|
||||
let work = tempfile::tempdir()?;
|
||||
let mut reports = Vec::new();
|
||||
for part in map["partitions"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.filter(|p| p["format"] == "luks")
|
||||
{
|
||||
let mut child = Command::new("python3")
|
||||
.arg(root.join("crates/verstack-luks/examples/range_server.py"))
|
||||
.arg(map["snapshot"].as_str().unwrap())
|
||||
.arg(map["path"].as_str().unwrap())
|
||||
.arg(map["size"].as_u64().unwrap().to_string())
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::piped())
|
||||
.stderr(Stdio::null())
|
||||
.spawn()?;
|
||||
let input = child.stdin.take().unwrap();
|
||||
let output = child.stdout.take().unwrap();
|
||||
let remote = Remote {
|
||||
child,
|
||||
input,
|
||||
output,
|
||||
position: 0,
|
||||
size: map["size"].as_u64().unwrap(),
|
||||
bytes: 0,
|
||||
};
|
||||
let mut slice = Slice::new(
|
||||
remote,
|
||||
part["offset"].as_u64().unwrap(),
|
||||
part["length"].as_u64().unwrap(),
|
||||
)?;
|
||||
let capsule = Capsule::read(&mut slice, &pin, work.path(), Default::default())?;
|
||||
// This explicit serial probe grants the measured bounded PBKDF2 profile only.
|
||||
let granted = UnlockLimits {
|
||||
cpu: 1,
|
||||
memory_bytes: 64 << 20,
|
||||
max_iterations: 250000,
|
||||
};
|
||||
let key = capsule.unlock(&credential, &granted)?;
|
||||
let mut plain = LuksReader::new(
|
||||
slice,
|
||||
capsule.profile.clone(),
|
||||
&key,
|
||||
4096,
|
||||
1 << 20,
|
||||
Default::default(),
|
||||
)?;
|
||||
plain.seek(SeekFrom::Start(1024))?;
|
||||
let mut sb = [0; 1024];
|
||||
plain.read_exact(&mut sb)?;
|
||||
ensure!(
|
||||
sb[56..58] == [0x53, 0xef],
|
||||
"decrypted partition is not ext4"
|
||||
);
|
||||
reports.push(serde_json::json!({"partition":part["index"],"profile":capsule.profile,"unlock":true,"ext_magic_verified":true,"capsule_bytes":capsule.demand.capsule_bytes,"ciphertext_read":plain.ciphertext_bytes_read(),"filesystem_block_size":1024u64<<u32::from_le_bytes(sb[24..28].try_into().unwrap())}));
|
||||
}
|
||||
println!("{}", serde_json::to_string_pretty(&reports)?);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Read-only bounded archive bridge for the explicit local-corpus example (no credentials)."""
|
||||
import io,json,sys,urllib.parse,urllib.request
|
||||
snapshot,path,size=sys.argv[1],sys.argv[2],int(sys.argv[3]);used=0
|
||||
url='http://127.0.0.1:8080/api/file/'+urllib.parse.quote(snapshot,safe='')+'?'+urllib.parse.urlencode({'path':path})
|
||||
for line in sys.stdin.buffer:
|
||||
offset,length=json.loads(line)
|
||||
if offset<0 or length<0 or length>4<<20 or offset+length>size or used+length>16<<20:raise ValueError('bounded archive inspection limit')
|
||||
req=urllib.request.Request(url,headers={'Range':f'bytes={offset}-{offset+length-1}'})
|
||||
with urllib.request.urlopen(req,timeout=180) as r:
|
||||
if r.status!=206 or r.headers.get('Content-Range')!=f'bytes {offset}-{offset+length-1}/{size}':raise ValueError('incorrect archive range response')
|
||||
data=r.read(length+1)
|
||||
if len(data)!=length:raise ValueError('truncated archive range')
|
||||
used+=length;sys.stdout.buffer.write(data);sys.stdout.buffer.flush()
|
||||
@@ -0,0 +1,272 @@
|
||||
//! Bounded, seekable LUKS2 AES-XTS plaintext. No kernel mapping or plaintext staging.
|
||||
mod unlock;
|
||||
use aes::{
|
||||
Aes128, Aes256,
|
||||
cipher::{BlockDecrypt, BlockEncrypt, KeyInit, generic_array::GenericArray},
|
||||
};
|
||||
use anyhow::{Result, ensure};
|
||||
use serde::Serialize;
|
||||
use std::io::{self, Read, Seek, SeekFrom};
|
||||
use std::sync::{
|
||||
Arc,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
};
|
||||
pub use unlock::{Capsule, LibraryPin, UnlockDemand, UnlockLimits};
|
||||
use zeroize::{Zeroize, Zeroizing};
|
||||
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Profile {
|
||||
pub offset: u64,
|
||||
pub length: u64,
|
||||
pub sector_size: u64,
|
||||
pub iv_tweak: u64,
|
||||
pub key_bytes: usize,
|
||||
}
|
||||
impl Profile {
|
||||
pub fn validate(&self, source_length: u64) -> Result<()> {
|
||||
ensure!(
|
||||
[512, 4096].contains(&self.sector_size),
|
||||
"unsupported XTS sector size"
|
||||
);
|
||||
ensure!(
|
||||
[32, 64].contains(&self.key_bytes),
|
||||
"unsupported XTS key size"
|
||||
);
|
||||
ensure!(
|
||||
self.length > 0
|
||||
&& self.length.is_multiple_of(self.sector_size)
|
||||
&& self.offset.is_multiple_of(self.sector_size),
|
||||
"misaligned XTS geometry"
|
||||
);
|
||||
ensure!(
|
||||
self.offset
|
||||
.checked_add(self.length)
|
||||
.is_some_and(|n| n <= source_length),
|
||||
"XTS range outside source"
|
||||
);
|
||||
ensure!(
|
||||
self.iv_tweak
|
||||
.checked_add((self.length - self.sector_size) / 512)
|
||||
.is_some(),
|
||||
"plain64 sector overflow"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
/// Slice positions and lengths are relative to a pinned archived artifact.
|
||||
pub struct Slice<R> {
|
||||
source: R,
|
||||
start: u64,
|
||||
length: u64,
|
||||
position: u64,
|
||||
}
|
||||
impl<R: Read + Seek> Slice<R> {
|
||||
pub fn new(mut source: R, start: u64, length: u64) -> Result<Self> {
|
||||
let total = source.seek(SeekFrom::End(0))?;
|
||||
ensure!(
|
||||
start.checked_add(length).is_some_and(|end| end <= total),
|
||||
"slice outside source"
|
||||
);
|
||||
Ok(Self {
|
||||
source,
|
||||
start,
|
||||
length,
|
||||
position: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
impl<R: Read + Seek> Read for Slice<R> {
|
||||
fn read(&mut self, out: &mut [u8]) -> io::Result<usize> {
|
||||
let count = (self.length.saturating_sub(self.position)).min(out.len() as u64) as usize;
|
||||
if count == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
self.source
|
||||
.seek(SeekFrom::Start(self.start + self.position))?;
|
||||
let n = self.source.read(&mut out[..count])?;
|
||||
self.position += n as u64;
|
||||
Ok(n)
|
||||
}
|
||||
}
|
||||
fn position(current: u64, length: u64, from: SeekFrom) -> io::Result<u64> {
|
||||
let n = match from {
|
||||
SeekFrom::Start(n) => n as i128,
|
||||
SeekFrom::Current(n) => current as i128 + n as i128,
|
||||
SeekFrom::End(n) => length as i128 + n as i128,
|
||||
};
|
||||
u64::try_from(n).map_err(|_| io::Error::new(io::ErrorKind::InvalidInput, "invalid seek"))
|
||||
}
|
||||
impl<R> Seek for Slice<R> {
|
||||
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
|
||||
self.position = position(self.position, self.length, from)?;
|
||||
Ok(self.position)
|
||||
}
|
||||
}
|
||||
|
||||
enum Cipher {
|
||||
A128(Box<Aes128>, Box<Aes128>),
|
||||
A256(Box<Aes256>, Box<Aes256>),
|
||||
}
|
||||
impl Cipher {
|
||||
fn new(key: &[u8]) -> Result<Self> {
|
||||
ensure!(
|
||||
[32, 64].contains(&key.len()) && key[..key.len() / 2] != key[key.len() / 2..],
|
||||
"invalid XTS key geometry"
|
||||
);
|
||||
Ok(if key.len() == 32 {
|
||||
Self::A128(
|
||||
Box::new(Aes128::new(GenericArray::from_slice(&key[..16]))),
|
||||
Box::new(Aes128::new(GenericArray::from_slice(&key[16..]))),
|
||||
)
|
||||
} else {
|
||||
Self::A256(
|
||||
Box::new(Aes256::new(GenericArray::from_slice(&key[..32]))),
|
||||
Box::new(Aes256::new(GenericArray::from_slice(&key[32..]))),
|
||||
)
|
||||
})
|
||||
}
|
||||
fn decrypt(&self, sector: &mut [u8], number: u64) {
|
||||
let mut tweak = [0u8; 16];
|
||||
tweak[..8].copy_from_slice(&number.to_le_bytes());
|
||||
match self {
|
||||
Self::A128(_, k) => k.encrypt_block(GenericArray::from_mut_slice(&mut tweak)),
|
||||
Self::A256(_, k) => k.encrypt_block(GenericArray::from_mut_slice(&mut tweak)),
|
||||
}
|
||||
for block in sector.as_chunks_mut::<16>().0 {
|
||||
for (b, t) in block.iter_mut().zip(tweak) {
|
||||
*b ^= t;
|
||||
}
|
||||
match self {
|
||||
Self::A128(k, _) => k.decrypt_block(GenericArray::from_mut_slice(block)),
|
||||
Self::A256(k, _) => k.decrypt_block(GenericArray::from_mut_slice(block)),
|
||||
}
|
||||
for (b, t) in block.iter_mut().zip(tweak) {
|
||||
*b ^= t;
|
||||
}
|
||||
let mut carry = 0;
|
||||
for b in &mut tweak {
|
||||
let next = *b >> 7;
|
||||
*b = (*b << 1) | carry;
|
||||
carry = next;
|
||||
}
|
||||
if carry != 0 {
|
||||
tweak[0] ^= 0x87;
|
||||
}
|
||||
}
|
||||
tweak.zeroize();
|
||||
}
|
||||
}
|
||||
/// One bounded plaintext window; seeking decrypts only requested ranges. Expanded AES keys
|
||||
/// are reused, with a fresh little-endian plain64 tweak for every data unit.
|
||||
pub struct LuksReader<R> {
|
||||
source: R,
|
||||
profile: Profile,
|
||||
cipher: Cipher,
|
||||
position: u64,
|
||||
start: u64,
|
||||
cache: Zeroizing<Vec<u8>>,
|
||||
window: usize,
|
||||
cancelled: Arc<AtomicBool>,
|
||||
read_bytes: u64,
|
||||
read_limit: u64,
|
||||
}
|
||||
impl<R: Read + Seek> LuksReader<R> {
|
||||
pub fn new(
|
||||
mut source: R,
|
||||
profile: Profile,
|
||||
key: &[u8],
|
||||
window: usize,
|
||||
read_limit: u64,
|
||||
cancelled: Arc<AtomicBool>,
|
||||
) -> Result<Self> {
|
||||
profile.validate(source.seek(SeekFrom::End(0))?)?;
|
||||
ensure!(
|
||||
key.len() == profile.key_bytes,
|
||||
"volume key size differs from profile"
|
||||
);
|
||||
ensure!(
|
||||
window >= profile.sector_size as usize
|
||||
&& window <= 8 << 20
|
||||
&& (window as u64).is_multiple_of(profile.sector_size),
|
||||
"invalid plaintext cache bound"
|
||||
);
|
||||
Ok(Self {
|
||||
source,
|
||||
profile,
|
||||
cipher: Cipher::new(key)?,
|
||||
position: 0,
|
||||
start: 0,
|
||||
cache: Zeroizing::new(Vec::new()),
|
||||
window,
|
||||
cancelled,
|
||||
read_bytes: 0,
|
||||
read_limit,
|
||||
})
|
||||
}
|
||||
pub fn profile(&self) -> &Profile {
|
||||
&self.profile
|
||||
}
|
||||
pub fn ciphertext_bytes_read(&self) -> u64 {
|
||||
self.read_bytes
|
||||
}
|
||||
}
|
||||
impl<R: Read + Seek> Read for LuksReader<R> {
|
||||
fn read(&mut self, out: &mut [u8]) -> io::Result<usize> {
|
||||
if self.cancelled.load(Ordering::Relaxed) {
|
||||
return Err(io::Error::new(
|
||||
io::ErrorKind::Interrupted,
|
||||
"LUKS read cancelled",
|
||||
));
|
||||
}
|
||||
if out.is_empty() || self.position >= self.profile.length {
|
||||
return Ok(0);
|
||||
}
|
||||
if self.cache.is_empty()
|
||||
|| self.position < self.start
|
||||
|| self.position - self.start >= self.cache.len() as u64
|
||||
{
|
||||
self.cache.zeroize();
|
||||
self.cache.clear();
|
||||
self.start = self.position / self.window as u64 * self.window as u64;
|
||||
let n = (self.profile.length - self.start).min(self.window as u64) as usize;
|
||||
self.read_bytes = self
|
||||
.read_bytes
|
||||
.checked_add(n as u64)
|
||||
.ok_or_else(|| io::Error::other("LUKS read accounting overflow"))?;
|
||||
if self.read_bytes > self.read_limit {
|
||||
return Err(io::Error::other("LUKS read budget exceeded"));
|
||||
}
|
||||
self.cache.zeroize();
|
||||
self.cache.resize(n, 0);
|
||||
self.source
|
||||
.seek(SeekFrom::Start(self.profile.offset + self.start))?;
|
||||
if let Err(e) = self.source.read_exact(&mut self.cache) {
|
||||
self.cache.zeroize();
|
||||
self.cache.clear();
|
||||
return Err(e);
|
||||
}
|
||||
for (i, sector) in self
|
||||
.cache
|
||||
.chunks_exact_mut(self.profile.sector_size as usize)
|
||||
.enumerate()
|
||||
{
|
||||
self.cipher.decrypt(
|
||||
sector,
|
||||
self.profile.iv_tweak
|
||||
+ (self.start + i as u64 * self.profile.sector_size) / 512,
|
||||
);
|
||||
}
|
||||
}
|
||||
let offset = (self.position - self.start) as usize;
|
||||
let n = out.len().min(self.cache.len() - offset);
|
||||
out[..n].copy_from_slice(&self.cache[offset..offset + n]);
|
||||
self.position += n as u64;
|
||||
Ok(n)
|
||||
}
|
||||
}
|
||||
impl<R> Seek for LuksReader<R> {
|
||||
fn seek(&mut self, from: SeekFrom) -> io::Result<u64> {
|
||||
self.position = position(self.position, self.profile.length, from)?;
|
||||
Ok(self.position)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,521 @@
|
||||
//! Narrow bindings to libcryptsetup's public API. Only metadata loading and key retrieval;
|
||||
//! no activation, keyring, device mapper, or write operations are exposed.
|
||||
use crate::Profile;
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use serde::Serialize;
|
||||
use serde_json::Value;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
ffi::{CStr, CString, c_char, c_int, c_void},
|
||||
fs::File,
|
||||
io::{Read, Seek, SeekFrom, Write},
|
||||
os::fd::AsRawFd,
|
||||
path::{Path, PathBuf},
|
||||
ptr,
|
||||
sync::{
|
||||
Arc, Mutex,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
},
|
||||
};
|
||||
use zeroize::Zeroizing;
|
||||
const CAPSULE_LIMIT: u64 = 32 << 20;
|
||||
const METADATA_LIMIT: u64 = 4 << 20;
|
||||
static LIBRARY_LOCK: Mutex<()> = Mutex::new(());
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct LibraryPin {
|
||||
pub path: PathBuf,
|
||||
pub sha256: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct UnlockDemand {
|
||||
pub cpu: usize,
|
||||
pub memory_bytes: u64,
|
||||
pub capsule_bytes: u64,
|
||||
pub max_iterations: u64,
|
||||
pub slots: usize,
|
||||
}
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct UnlockLimits {
|
||||
pub cpu: usize,
|
||||
pub memory_bytes: u64,
|
||||
pub max_iterations: u64,
|
||||
}
|
||||
|
||||
struct Library {
|
||||
handle: *mut c_void,
|
||||
_file: File,
|
||||
}
|
||||
impl Library {
|
||||
fn open(pin: &LibraryPin) -> Result<Self> {
|
||||
ensure!(
|
||||
pin.path.is_absolute()
|
||||
&& pin.sha256.len() == 64
|
||||
&& pin.sha256.bytes().all(|b| b.is_ascii_hexdigit()),
|
||||
"libcryptsetup needs absolute path and SHA256 pin"
|
||||
);
|
||||
let mut file = File::open(&pin.path).context("open pinned libcryptsetup")?;
|
||||
ensure!(
|
||||
file.metadata()?.len() <= 32 << 20,
|
||||
"libcryptsetup file exceeds bound"
|
||||
);
|
||||
let mut hash = Sha256::new();
|
||||
let mut buf = [0u8; 65536];
|
||||
loop {
|
||||
let n = file.read(&mut buf)?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
hash.update(&buf[..n]);
|
||||
}
|
||||
let actual = hash
|
||||
.finalize()
|
||||
.iter()
|
||||
.map(|b| format!("{b:02x}"))
|
||||
.collect::<String>();
|
||||
ensure!(
|
||||
actual == pin.sha256.to_ascii_lowercase(),
|
||||
"libcryptsetup SHA256 pin mismatch"
|
||||
);
|
||||
let path = CString::new(format!("/proc/self/fd/{}", file.as_raw_fd()))?;
|
||||
// The open descriptor binds the verified inode throughout dlopen, avoiding pathname replacement.
|
||||
let handle = unsafe { libc::dlopen(path.as_ptr(), libc::RTLD_NOW | libc::RTLD_LOCAL) };
|
||||
ensure!(!handle.is_null(), "cannot load pinned libcryptsetup");
|
||||
Ok(Self {
|
||||
handle,
|
||||
_file: file,
|
||||
})
|
||||
}
|
||||
unsafe fn symbol<T: Copy>(&self, name: &CStr) -> Result<T> {
|
||||
let symbol = unsafe { libc::dlsym(self.handle, name.as_ptr()) };
|
||||
ensure!(
|
||||
!symbol.is_null(),
|
||||
"required libcryptsetup symbol unavailable"
|
||||
);
|
||||
ensure!(
|
||||
std::mem::size_of::<T>() == std::mem::size_of::<*mut c_void>(),
|
||||
"invalid native function pointer size"
|
||||
);
|
||||
// Call sites specify the exact public C function-pointer prototype.
|
||||
Ok(unsafe { std::mem::transmute_copy(&symbol) })
|
||||
}
|
||||
}
|
||||
impl Drop for Library {
|
||||
fn drop(&mut self) {
|
||||
unsafe {
|
||||
libc::dlclose(self.handle);
|
||||
}
|
||||
}
|
||||
}
|
||||
type Free = unsafe extern "C" fn(*mut c_void);
|
||||
type KeyGet = unsafe extern "C" fn(
|
||||
*mut c_void,
|
||||
c_int,
|
||||
*mut c_char,
|
||||
*mut usize,
|
||||
*const c_char,
|
||||
usize,
|
||||
) -> c_int;
|
||||
struct Device {
|
||||
pointer: *mut c_void,
|
||||
free: Free,
|
||||
key_get: KeyGet,
|
||||
_library: Library,
|
||||
}
|
||||
impl Drop for Device {
|
||||
fn drop(&mut self) {
|
||||
unsafe { (self.free)(self.pointer) }
|
||||
}
|
||||
}
|
||||
unsafe extern "C" fn quiet_log(_level: c_int, _message: *const c_char, _user: *mut c_void) {}
|
||||
impl Device {
|
||||
fn load(path: &Path, pin: &LibraryPin) -> Result<(Self, Value, Profile)> {
|
||||
let _lock = LIBRARY_LOCK
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("libcryptsetup initialization lock poisoned"))?;
|
||||
let library = Library::open(pin)?;
|
||||
type Init = unsafe extern "C" fn(*mut *mut c_void, *const c_char) -> c_int;
|
||||
type Load = unsafe extern "C" fn(*mut c_void, *const c_char, *mut c_void) -> c_int;
|
||||
type Log = unsafe extern "C" fn(
|
||||
*mut c_void,
|
||||
unsafe extern "C" fn(c_int, *const c_char, *mut c_void),
|
||||
*mut c_void,
|
||||
);
|
||||
type Dump = unsafe extern "C" fn(*mut c_void, *mut *const c_char, u32) -> c_int;
|
||||
type GetInt = unsafe extern "C" fn(*mut c_void) -> c_int;
|
||||
type GetU64 = unsafe extern "C" fn(*mut c_void) -> u64;
|
||||
let init: Init = unsafe { library.symbol(c"crypt_init")? };
|
||||
let free: Free = unsafe { library.symbol(c"crypt_free")? };
|
||||
let load: Load = unsafe { library.symbol(c"crypt_load")? };
|
||||
let log: Log = unsafe { library.symbol(c"crypt_set_log_callback")? };
|
||||
let dump: Dump = unsafe { library.symbol(c"crypt_dump_json")? };
|
||||
let key_get: KeyGet = unsafe { library.symbol(c"crypt_volume_key_get")? };
|
||||
let get_sector: GetInt = unsafe { library.symbol(c"crypt_get_sector_size")? };
|
||||
let get_key: GetInt = unsafe { library.symbol(c"crypt_get_volume_key_size")? };
|
||||
let get_offset: GetU64 = unsafe { library.symbol(c"crypt_get_data_offset")? };
|
||||
let get_iv: GetU64 = unsafe { library.symbol(c"crypt_get_iv_offset")? };
|
||||
let path = CString::new(path.as_os_str().as_encoded_bytes())?;
|
||||
let mut pointer = ptr::null_mut();
|
||||
// Suppress library diagnostics: callers receive fixed errors, never keyslot/passphrase material.
|
||||
unsafe {
|
||||
log(ptr::null_mut(), quiet_log, ptr::null_mut());
|
||||
}
|
||||
let result = unsafe { init(&mut pointer, path.as_ptr()) };
|
||||
if result < 0 {
|
||||
if !pointer.is_null() {
|
||||
unsafe { free(pointer) }
|
||||
};
|
||||
bail!("cannot initialize LUKS metadata capsule");
|
||||
}
|
||||
ensure!(!pointer.is_null(), "empty cryptsetup device");
|
||||
let device = Self {
|
||||
pointer,
|
||||
free,
|
||||
key_get,
|
||||
_library: library,
|
||||
};
|
||||
unsafe {
|
||||
log(pointer, quiet_log, ptr::null_mut());
|
||||
}
|
||||
ensure!(
|
||||
unsafe { load(pointer, c"LUKS2".as_ptr(), ptr::null_mut()) } >= 0,
|
||||
"libcryptsetup rejected LUKS2 metadata"
|
||||
);
|
||||
let mut json = ptr::null();
|
||||
ensure!(
|
||||
unsafe { dump(pointer, &mut json, 0) } >= 0 && !json.is_null(),
|
||||
"cannot inspect verified LUKS2 metadata"
|
||||
);
|
||||
let bytes = unsafe { CStr::from_ptr(json) }.to_bytes();
|
||||
ensure!(
|
||||
bytes.len() <= METADATA_LIMIT as usize,
|
||||
"verified LUKS2 metadata exceeds bound"
|
||||
);
|
||||
let metadata: Value = serde_json::from_slice(bytes)?;
|
||||
let sector = unsafe { get_sector(pointer) };
|
||||
let key = unsafe { get_key(pointer) };
|
||||
ensure!(sector > 0 && key > 0, "invalid cryptsetup geometry");
|
||||
let profile = Profile {
|
||||
offset: unsafe { get_offset(pointer) }
|
||||
.checked_mul(512)
|
||||
.context("data offset overflow")?,
|
||||
iv_tweak: unsafe { get_iv(pointer) },
|
||||
sector_size: sector as u64,
|
||||
key_bytes: key as usize,
|
||||
length: 0,
|
||||
};
|
||||
Ok((device, metadata, profile))
|
||||
}
|
||||
}
|
||||
fn number(v: &Value) -> Result<u64> {
|
||||
v.as_u64()
|
||||
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
|
||||
.context("invalid LUKS2 numeric field")
|
||||
}
|
||||
fn checked_metadata(
|
||||
metadata: &Value,
|
||||
length: u64,
|
||||
) -> Result<(Profile, UnlockDemand, u64, Vec<i32>)> {
|
||||
let segments = metadata["segments"]
|
||||
.as_object()
|
||||
.context("missing LUKS2 segments")?;
|
||||
ensure!(
|
||||
segments.len() == 1 && segments.contains_key("0"),
|
||||
"multiple or reencryption segments unsupported"
|
||||
);
|
||||
let s = &segments["0"];
|
||||
ensure!(
|
||||
s["type"] == "crypt"
|
||||
&& s["encryption"] == "aes-xts-plain64"
|
||||
&& s.get("flags")
|
||||
.is_none_or(|v| v.as_array().is_some_and(|a| a.is_empty())),
|
||||
"unsupported LUKS2 cipher profile"
|
||||
);
|
||||
ensure!(
|
||||
s.get("integrity").is_none() && metadata["config"].get("requirements").is_none(),
|
||||
"LUKS2 integrity or required feature unsupported"
|
||||
);
|
||||
let offset = number(&s["offset"])?;
|
||||
let payload = length
|
||||
.checked_sub(offset)
|
||||
.context("LUKS data outside source")?;
|
||||
ensure!(
|
||||
s["size"] == "dynamic" || number(&s["size"])? == payload,
|
||||
"unsupported fixed LUKS2 segment length"
|
||||
);
|
||||
let mut profile = Profile {
|
||||
offset,
|
||||
length: payload,
|
||||
sector_size: number(&s["sector_size"])?,
|
||||
iv_tweak: number(&s["iv_tweak"])?,
|
||||
key_bytes: 0,
|
||||
};
|
||||
let keys = metadata["keyslots"]
|
||||
.as_object()
|
||||
.context("missing LUKS2 keyslots")?;
|
||||
ensure!(
|
||||
!keys.is_empty() && keys.len() <= 32,
|
||||
"LUKS2 keyslot count exceeds bound"
|
||||
);
|
||||
let mut demand = UnlockDemand {
|
||||
cpu: 1,
|
||||
memory_bytes: 64 << 20,
|
||||
capsule_bytes: 0,
|
||||
max_iterations: 0,
|
||||
slots: keys.len(),
|
||||
};
|
||||
// Volume-key digest verification is another PBKDF2 operation, not just metadata.
|
||||
// Bound it before passing any credential to the library.
|
||||
let digests = metadata["digests"]
|
||||
.as_object()
|
||||
.context("missing LUKS2 digests")?;
|
||||
ensure!(
|
||||
!digests.is_empty() && digests.len() <= 32,
|
||||
"LUKS2 digest count exceeds bound"
|
||||
);
|
||||
for digest in digests.values() {
|
||||
ensure!(digest["type"] == "pbkdf2", "unsupported volume-key digest");
|
||||
let iterations = number(&digest["iterations"])?;
|
||||
ensure!(
|
||||
iterations > 0 && iterations <= 10_000_000,
|
||||
"volume-key digest iteration bound exceeded"
|
||||
);
|
||||
demand.max_iterations = demand.max_iterations.max(iterations);
|
||||
}
|
||||
let mut end = 0;
|
||||
let mut slots = Vec::new();
|
||||
for (id, k) in keys {
|
||||
ensure!(
|
||||
k["type"] == "luks2"
|
||||
&& k["area"]["type"] == "raw"
|
||||
&& k["area"]["encryption"] == "aes-xts-plain64"
|
||||
&& matches!(number(&k["area"]["key_size"]), Ok(32 | 64)),
|
||||
"unsupported keyslot profile"
|
||||
);
|
||||
let key_bytes = number(&k["key_size"])? as usize;
|
||||
ensure!(
|
||||
[32, 64].contains(&key_bytes)
|
||||
&& (profile.key_bytes == 0 || profile.key_bytes == key_bytes),
|
||||
"inconsistent LUKS2 key sizes"
|
||||
);
|
||||
profile.key_bytes = key_bytes;
|
||||
let start = number(&k["area"]["offset"])?;
|
||||
let size = number(&k["area"]["size"])?;
|
||||
let last = start.checked_add(size).context("keyslot range overflow")?;
|
||||
ensure!(
|
||||
start >= 8192 && size > 0 && last <= offset && last <= CAPSULE_LIMIT,
|
||||
"keyslot range exceeds metadata capsule bound"
|
||||
);
|
||||
end = end.max(last);
|
||||
let slot: i32 = id.parse().context("invalid keyslot id")?;
|
||||
ensure!((0..32).contains(&slot), "invalid keyslot id");
|
||||
let bound = metadata["digests"]
|
||||
.as_object()
|
||||
.context("missing LUKS2 digests")?
|
||||
.values()
|
||||
.any(|d| {
|
||||
d["segments"]
|
||||
.as_array()
|
||||
.is_some_and(|a| a.iter().any(|v| v == "0"))
|
||||
&& d["keyslots"]
|
||||
.as_array()
|
||||
.is_some_and(|a| a.iter().any(|v| v == id))
|
||||
});
|
||||
ensure!(bound, "unbound LUKS2 keyslot unsupported");
|
||||
slots.push(slot);
|
||||
let kdf = &k["kdf"];
|
||||
match kdf["type"].as_str() {
|
||||
Some("pbkdf2") => {
|
||||
let iterations = number(&kdf["iterations"])?;
|
||||
ensure!(
|
||||
iterations > 0 && iterations <= 10_000_000,
|
||||
"PBKDF2 iteration bound exceeded"
|
||||
);
|
||||
demand.max_iterations = demand.max_iterations.max(iterations);
|
||||
}
|
||||
Some("argon2i" | "argon2id") => {
|
||||
let memory = number(&kdf["memory"])?;
|
||||
let cpu = number(&kdf["cpus"])?;
|
||||
let time = number(&kdf["time"])?;
|
||||
ensure!(
|
||||
memory > 0
|
||||
&& memory <= 4 * 1024 * 1024
|
||||
&& cpu > 0
|
||||
&& cpu <= 64
|
||||
&& time > 0
|
||||
&& time <= 100,
|
||||
"Argon2 demand exceeds supported bound"
|
||||
);
|
||||
demand.memory_bytes = demand.memory_bytes.max(
|
||||
memory
|
||||
.checked_mul(1024)
|
||||
.and_then(|n| n.checked_add(64 << 20))
|
||||
.context("Argon2 memory overflow")?,
|
||||
);
|
||||
demand.cpu = demand.cpu.max(cpu as usize);
|
||||
demand.max_iterations = demand.max_iterations.max(time);
|
||||
}
|
||||
_ => bail!("unsupported LUKS2 KDF"),
|
||||
}
|
||||
}
|
||||
profile.validate(length)?;
|
||||
Ok((profile, demand, end, slots))
|
||||
}
|
||||
/// Encrypted metadata/keyslots only. The sparse logical size preserves device geometry;
|
||||
/// the private temporary file contains no passphrase, volume key or plaintext payload.
|
||||
pub struct Capsule {
|
||||
device: Device,
|
||||
_file: tempfile::NamedTempFile,
|
||||
pub profile: Profile,
|
||||
pub demand: UnlockDemand,
|
||||
slots: Vec<i32>,
|
||||
cancelled: Arc<AtomicBool>,
|
||||
}
|
||||
impl Capsule {
|
||||
pub fn read<R: Read + Seek>(
|
||||
source: &mut R,
|
||||
pin: &LibraryPin,
|
||||
work: &Path,
|
||||
cancelled: Arc<AtomicBool>,
|
||||
) -> Result<Self> {
|
||||
ensure!(
|
||||
!cancelled.load(Ordering::Relaxed),
|
||||
"LUKS operation cancelled"
|
||||
);
|
||||
let length = source.seek(SeekFrom::End(0))?;
|
||||
source.seek(SeekFrom::Start(0))?;
|
||||
let mut header = [0u8; 4096];
|
||||
source.read_exact(&mut header)?;
|
||||
ensure!(
|
||||
&header[..8] == b"LUKS\xba\xbe\0\x02",
|
||||
"unsupported LUKS header version"
|
||||
);
|
||||
let header_size = u64::from_be_bytes(header[8..16].try_into().unwrap());
|
||||
ensure!(
|
||||
(16384..=METADATA_LIMIT).contains(&header_size) && header_size.is_power_of_two(),
|
||||
"invalid LUKS2 metadata length"
|
||||
);
|
||||
let mut json = vec![0; (header_size - 4096) as usize];
|
||||
source.read_exact(&mut json)?;
|
||||
let end = json.iter().position(|b| *b == 0).unwrap_or(json.len());
|
||||
let metadata: Value =
|
||||
serde_json::from_slice(&json[..end]).context("parse bounded LUKS2 metadata")?;
|
||||
let (profile, mut demand, key_end, slots) = checked_metadata(&metadata, length)?;
|
||||
let capsule_bytes = key_end.max(header_size * 2);
|
||||
ensure!(
|
||||
capsule_bytes <= CAPSULE_LIMIT && capsule_bytes <= profile.offset,
|
||||
"metadata capsule overlaps payload"
|
||||
);
|
||||
let mut file = tempfile::NamedTempFile::new_in(work)?;
|
||||
file.as_file().set_len(length)?;
|
||||
source.seek(SeekFrom::Start(0))?;
|
||||
let mut left = capsule_bytes;
|
||||
let mut buffer = [0u8; 65536];
|
||||
while left > 0 {
|
||||
ensure!(
|
||||
!cancelled.load(Ordering::Relaxed),
|
||||
"LUKS operation cancelled"
|
||||
);
|
||||
let n = left.min(buffer.len() as u64) as usize;
|
||||
source.read_exact(&mut buffer[..n])?;
|
||||
file.write_all(&buffer[..n])?;
|
||||
left -= n as u64;
|
||||
}
|
||||
file.flush()?;
|
||||
let (device, verified, reported) = Device::load(file.path(), pin)?;
|
||||
ensure!(
|
||||
verified == metadata,
|
||||
"verified LUKS2 copy differs from bounded preflight"
|
||||
);
|
||||
ensure!(
|
||||
profile.offset == reported.offset
|
||||
&& profile.sector_size == reported.sector_size
|
||||
&& profile.iv_tweak == reported.iv_tweak
|
||||
&& profile.key_bytes == reported.key_bytes,
|
||||
"libcryptsetup profile disagrees with metadata"
|
||||
);
|
||||
demand.capsule_bytes = capsule_bytes;
|
||||
Ok(Self {
|
||||
device,
|
||||
_file: file,
|
||||
profile,
|
||||
demand,
|
||||
slots,
|
||||
cancelled,
|
||||
})
|
||||
}
|
||||
pub fn unlock(&self, credential: &[u8], granted: &UnlockLimits) -> Result<Zeroizing<Vec<u8>>> {
|
||||
ensure!(
|
||||
!credential.is_empty() && credential.len() <= 65536,
|
||||
"invalid credential length"
|
||||
);
|
||||
ensure!(
|
||||
granted.cpu >= self.demand.cpu
|
||||
&& granted.memory_bytes >= self.demand.memory_bytes
|
||||
&& granted.max_iterations >= self.demand.max_iterations,
|
||||
"KDF admission below declared demand"
|
||||
);
|
||||
for slot in &self.slots {
|
||||
ensure!(
|
||||
!self.cancelled.load(Ordering::Relaxed),
|
||||
"LUKS operation cancelled"
|
||||
);
|
||||
let mut key = Zeroizing::new(vec![0u8; 64]);
|
||||
let mut length = key.len();
|
||||
let result = unsafe {
|
||||
(self.device.key_get)(
|
||||
self.device.pointer,
|
||||
*slot,
|
||||
key.as_mut_ptr().cast(),
|
||||
&mut length,
|
||||
credential.as_ptr().cast(),
|
||||
credential.len(),
|
||||
)
|
||||
};
|
||||
if result >= 0 {
|
||||
ensure!(
|
||||
length == self.profile.key_bytes,
|
||||
"unlocked key differs from profile"
|
||||
);
|
||||
key.truncate(length);
|
||||
return Ok(key);
|
||||
}
|
||||
ensure!(
|
||||
result == -libc::EPERM || result == -libc::ENOENT,
|
||||
"libcryptsetup key retrieval failed"
|
||||
);
|
||||
}
|
||||
bail!("credential did not unlock this container")
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
fn metadata() -> Value {
|
||||
serde_json::json!({"segments":{"0":{"type":"crypt","offset":"16777216","size":"dynamic","iv_tweak":"0","encryption":"aes-xts-plain64","sector_size":512}},"keyslots":{"0":{"type":"luks2","key_size":32,"area":{"type":"raw","offset":"32768","size":"131072","encryption":"aes-xts-plain64","key_size":32},"kdf":{"type":"argon2id","memory":131072,"cpus":4,"time":3}}},"digests":{"0":{"type":"pbkdf2","iterations":250000,"segments":["0"],"keyslots":["0"]}},"config":{}})
|
||||
}
|
||||
#[test]
|
||||
fn kdf_digest_geometry_and_capsule_bounds_are_checked_before_unlock() {
|
||||
let good = metadata();
|
||||
let (_, demand, end, _) = checked_metadata(&good, 32 << 20).unwrap();
|
||||
assert_eq!(demand.cpu, 4);
|
||||
assert_eq!(demand.memory_bytes, 192 << 20);
|
||||
assert_eq!(demand.max_iterations, 250000);
|
||||
assert_eq!(end, 163840);
|
||||
let mut bad = good.clone();
|
||||
bad["digests"]["0"]["iterations"] = u64::MAX.into();
|
||||
assert!(checked_metadata(&bad, 32 << 20).is_err());
|
||||
let mut bad = good.clone();
|
||||
bad["keyslots"]["0"]["area"]["offset"] = u64::MAX.into();
|
||||
assert!(checked_metadata(&bad, 32 << 20).is_err());
|
||||
let mut bad = good.clone();
|
||||
bad["keyslots"]["0"]["kdf"]["memory"] = u64::MAX.into();
|
||||
assert!(checked_metadata(&bad, 32 << 20).is_err());
|
||||
let mut bad = good.clone();
|
||||
bad["segments"]["0"]["flags"] = serde_json::json!(["iv_large_sectors"]);
|
||||
assert!(checked_metadata(&bad, 32 << 20).is_err());
|
||||
let mut bad = good;
|
||||
bad["segments"]["1"] = bad["segments"]["0"].clone();
|
||||
assert!(checked_metadata(&bad, 32 << 20).is_err());
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,220 @@
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
io::{Cursor, Read, Seek, SeekFrom, Write},
|
||||
process::{Command, Stdio},
|
||||
sync::{
|
||||
Arc,
|
||||
atomic::{AtomicBool, Ordering},
|
||||
},
|
||||
};
|
||||
use verstack_luks::{Capsule, LibraryPin, LuksReader, Profile, Slice, UnlockLimits};
|
||||
fn profile(key: usize, sector: u64) -> Profile {
|
||||
Profile {
|
||||
offset: sector,
|
||||
length: 5 * sector,
|
||||
sector_size: sector,
|
||||
iv_tweak: (1 << 32) - 1 + 3 * sector / 512,
|
||||
key_bytes: key,
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn independent_openssl_fixtures_cover_random_reads_keys_sectors_and_iv_wrap() {
|
||||
for (key, sector, cipher) in [
|
||||
(
|
||||
32,
|
||||
512,
|
||||
include_bytes!("fixtures/xts-32-512.bin").as_slice(),
|
||||
),
|
||||
(
|
||||
64,
|
||||
512,
|
||||
include_bytes!("fixtures/xts-64-512.bin").as_slice(),
|
||||
),
|
||||
(
|
||||
32,
|
||||
4096,
|
||||
include_bytes!("fixtures/xts-32-4096.bin").as_slice(),
|
||||
),
|
||||
(
|
||||
64,
|
||||
4096,
|
||||
include_bytes!("fixtures/xts-64-4096.bin").as_slice(),
|
||||
),
|
||||
] {
|
||||
let mut source = vec![0; sector as usize];
|
||||
source.extend(cipher);
|
||||
let plain = (0..5 * sector)
|
||||
.map(|i| ((i * 17 + i / sector) % 256) as u8)
|
||||
.collect::<Vec<_>>();
|
||||
let mut reader = LuksReader::new(
|
||||
Cursor::new(source),
|
||||
profile(key, sector),
|
||||
&(0..key as u8).collect::<Vec<_>>(),
|
||||
sector as usize * 2,
|
||||
1 << 20,
|
||||
Arc::default(),
|
||||
)
|
||||
.unwrap();
|
||||
for (at, n) in [
|
||||
(sector - 3, 20),
|
||||
(0, 71),
|
||||
(4 * sector + 5, 51),
|
||||
(sector * 2 - 1, 2),
|
||||
] {
|
||||
reader.seek(SeekFrom::Start(at)).unwrap();
|
||||
let mut bytes = vec![0; n];
|
||||
reader.read_exact(&mut bytes).unwrap();
|
||||
assert_eq!(bytes, plain[at as usize..at as usize + n]);
|
||||
}
|
||||
reader.seek(SeekFrom::Start(0)).unwrap();
|
||||
let mut all = Vec::new();
|
||||
reader.read_to_end(&mut all).unwrap();
|
||||
assert_eq!(all, plain);
|
||||
assert!(reader.seek(SeekFrom::End(-100000)).is_err());
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn bounds_cancellation_and_failed_refill_never_expose_stale_plaintext() {
|
||||
let profile = profile(32, 512);
|
||||
let key = (0..32).collect::<Vec<u8>>();
|
||||
let mut source = vec![0; 512];
|
||||
source.extend(include_bytes!("fixtures/xts-32-512.bin"));
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
let mut reader = LuksReader::new(
|
||||
Cursor::new(source.clone()),
|
||||
profile.clone(),
|
||||
&key,
|
||||
512,
|
||||
512,
|
||||
cancel.clone(),
|
||||
)
|
||||
.unwrap();
|
||||
let mut byte = [0];
|
||||
reader.read_exact(&mut byte).unwrap();
|
||||
reader.seek(SeekFrom::Start(512)).unwrap();
|
||||
assert!(reader.read(&mut byte).is_err());
|
||||
assert!(reader.read(&mut byte).is_err());
|
||||
cancel.store(true, Ordering::Relaxed);
|
||||
assert_eq!(
|
||||
reader.read(&mut byte).unwrap_err().kind(),
|
||||
std::io::ErrorKind::Interrupted
|
||||
);
|
||||
let mut bad = profile.clone();
|
||||
bad.iv_tweak = u64::MAX;
|
||||
assert!(
|
||||
LuksReader::new(
|
||||
Cursor::new(source.clone()),
|
||||
bad,
|
||||
&key,
|
||||
512,
|
||||
10000,
|
||||
Arc::default()
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
LuksReader::new(
|
||||
Cursor::new(source.clone()),
|
||||
profile,
|
||||
&[0; 32],
|
||||
512,
|
||||
10000,
|
||||
Arc::default()
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
assert!(Slice::new(Cursor::new(source), u64::MAX, 100).is_err());
|
||||
let mut slice = Slice::new(Cursor::new(vec![1, 2, 3, 4]), 1, 2).unwrap();
|
||||
let mut bytes = Vec::new();
|
||||
slice.read_to_end(&mut bytes).unwrap();
|
||||
assert_eq!(bytes, [2, 3]);
|
||||
}
|
||||
fn pin() -> LibraryPin {
|
||||
let path = std::path::PathBuf::from("/usr/lib/x86_64-linux-gnu/libcryptsetup.so.12");
|
||||
let bytes = std::fs::read(&path).unwrap();
|
||||
let sha256 = Sha256::digest(bytes)
|
||||
.iter()
|
||||
.map(|b| format!("{b:02x}"))
|
||||
.collect();
|
||||
LibraryPin { path, sha256 }
|
||||
}
|
||||
#[test]
|
||||
fn actual_libcryptsetup_capsule_unlock_checks_pin_kdf_and_credentials() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let image = temp.path().join("fixture.luks");
|
||||
let mut file = std::fs::File::create(&image).unwrap();
|
||||
file.set_len(32 << 20).unwrap();
|
||||
let mut command = Command::new("/usr/sbin/cryptsetup")
|
||||
.args([
|
||||
"luksFormat",
|
||||
"--type",
|
||||
"luks2",
|
||||
"--pbkdf",
|
||||
"pbkdf2",
|
||||
"--pbkdf-force-iterations",
|
||||
"1000",
|
||||
"--batch-mode",
|
||||
"--key-file",
|
||||
"-",
|
||||
])
|
||||
.arg(&image)
|
||||
.stdin(Stdio::piped())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::piped())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
command
|
||||
.stdin
|
||||
.take()
|
||||
.unwrap()
|
||||
.write_all(b"public-test-fixture-passphrase")
|
||||
.unwrap();
|
||||
assert!(command.wait_with_output().unwrap().status.success());
|
||||
file = std::fs::File::open(&image).unwrap();
|
||||
let mut wrong = pin();
|
||||
wrong.sha256 = "00".repeat(32);
|
||||
assert!(Capsule::read(&mut file, &wrong, temp.path(), Arc::default()).is_err());
|
||||
let capsule = Capsule::read(&mut file, &pin(), temp.path(), Arc::default()).unwrap();
|
||||
assert!(capsule.demand.capsule_bytes < 1 << 20);
|
||||
assert_eq!(capsule.demand.cpu, 1);
|
||||
assert!(capsule.demand.max_iterations >= 1000);
|
||||
let mut limits = UnlockLimits {
|
||||
cpu: 1,
|
||||
memory_bytes: 64 << 20,
|
||||
max_iterations: capsule.demand.max_iterations,
|
||||
};
|
||||
limits.memory_bytes -= 1;
|
||||
assert!(
|
||||
capsule
|
||||
.unlock(b"public-test-fixture-passphrase", &limits)
|
||||
.is_err()
|
||||
);
|
||||
limits.memory_bytes += 1;
|
||||
assert!(
|
||||
capsule
|
||||
.unlock(b"incorrect-public-fixture-passphrase", &limits)
|
||||
.is_err()
|
||||
);
|
||||
let key = capsule
|
||||
.unlock(b"public-test-fixture-passphrase", &limits)
|
||||
.unwrap();
|
||||
assert_eq!(key.len(), capsule.profile.key_bytes);
|
||||
drop(capsule);
|
||||
let mut corrupt = std::fs::OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.open(&image)
|
||||
.unwrap();
|
||||
let mut header = [0; 16];
|
||||
corrupt.read_exact(&mut header).unwrap();
|
||||
let header_size = u64::from_be_bytes(header[8..16].try_into().unwrap());
|
||||
for offset in [4095, header_size + 4095] {
|
||||
corrupt.seek(SeekFrom::Start(offset)).unwrap();
|
||||
let mut byte = [0];
|
||||
corrupt.read_exact(&mut byte).unwrap();
|
||||
byte[0] ^= 0xff;
|
||||
corrupt.seek(SeekFrom::Start(offset)).unwrap();
|
||||
corrupt.write_all(&byte).unwrap();
|
||||
}
|
||||
assert!(Capsule::read(&mut corrupt, &pin(), temp.path(), Arc::default()).is_err());
|
||||
}
|
||||
Generated
+315
@@ -0,0 +1,315 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "anyhow"
|
||||
version = "1.0.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
||||
|
||||
[[package]]
|
||||
name = "array-init"
|
||||
version = "2.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3d62b7694a562cdf5a74227903507c56ab2cc8bdd1f781ed5cb4cf9c9f810bfc"
|
||||
|
||||
[[package]]
|
||||
name = "arrayvec"
|
||||
version = "0.7.8"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56"
|
||||
|
||||
[[package]]
|
||||
name = "binrw"
|
||||
version = "0.15.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6ad120d555272286c1017d25165ab8bd74806f13fc85b258484ec7e4ce75458f"
|
||||
dependencies = [
|
||||
"array-init",
|
||||
"binrw_derive",
|
||||
"bytemuck",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "binrw_derive"
|
||||
version = "0.15.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6df92e0e9baae4dc82c7bad7715ca40c0a5c71539057bf2ea04a5c29c980410b"
|
||||
dependencies = [
|
||||
"either",
|
||||
"owo-colors",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 2.0.119",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "blake3"
|
||||
version = "1.8.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6d9e454fc11f76977dc803893aff6304ed33d6a26efae8696573bea74baa27ae"
|
||||
dependencies = [
|
||||
"arrayvec",
|
||||
"cc",
|
||||
"cfg-if",
|
||||
"constant_time_eq",
|
||||
"cpufeatures",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "block-buffer"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bytemuck"
|
||||
version = "1.25.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "95832e849adfb21180ccb6826a99da14e5d266ae5c2e668e1602cf234f153797"
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.4.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"shlex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "const-oid"
|
||||
version = "0.10.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c"
|
||||
|
||||
[[package]]
|
||||
name = "constant_time_eq"
|
||||
version = "0.4.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5ca28b0ae3115b884660db4118d803791fd6756b6e88f39c0f3f7859060d7566"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crc32fast"
|
||||
version = "1.5.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.2.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
|
||||
dependencies = [
|
||||
"hybrid-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
|
||||
dependencies = [
|
||||
"block-buffer",
|
||||
"const-oid",
|
||||
"crypto-common",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "either"
|
||||
version = "1.18.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
|
||||
|
||||
[[package]]
|
||||
name = "hybrid-array"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27f864f10dfb56725ce5ce5472bc52252c8f93a4ab86327122cebf62c5f59a17"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "libc"
|
||||
version = "0.2.189"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
|
||||
|
||||
[[package]]
|
||||
name = "owo-colors"
|
||||
version = "4.4.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13c45bb4a6ae1280ec0803b1ef9d3455eb50f01efbbe1447ab020f1d54fba9d8"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.47"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.151"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha2"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "shlex"
|
||||
version = "2.0.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.119"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.25"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ab72a15cf68d77cb0987d3684aa8a45c5ef827e8cb49ee2f30bfd7ba2feb519f"
|
||||
|
||||
[[package]]
|
||||
name = "verstack-radium"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"binrw",
|
||||
"blake3",
|
||||
"crc32fast",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
@@ -0,0 +1,15 @@
|
||||
[package]
|
||||
name = "verstack-radium"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
binrw = "=0.15.2"
|
||||
crc32fast = "=1.5.2"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = "1"
|
||||
blake3 = "1"
|
||||
sha2 = "0.11"
|
||||
@@ -0,0 +1,61 @@
|
||||
# Native Radium directories and DMD indices
|
||||
|
||||
The crate reads `Read + Seek` sources without requiring `Send`, a path or a staged
|
||||
input copy. `binrw` decodes the 13-qword header and section-8 records. The sound
|
||||
directory's CRC covers aligned table bytes; sound extents must remain before the
|
||||
directory and agree with the following record. Original PCM streams to a writer
|
||||
in at most 1 MiB chunks. No sample rate is inferred from a filename or title.
|
||||
|
||||
`BitmapIndex` validates section 3 independently. Real SPIKE 2 images use sentinel
|
||||
values in the sound-section words while retaining a valid bitmap table; rejecting
|
||||
those entire images would lose valid DMD coverage. Duplicate bitmap pointers retain
|
||||
separate directory ordinals. Encoded consumed length is distinct from pointer extent.
|
||||
|
||||
DMD decoding supports raw indexed8, row/column whitespace, row/column deltas and
|
||||
low-nibble-first indexed4. It returns exact indices, not a claimed palette or final
|
||||
screen rendering. Delta frames retain both the actual base directory ordinal and
|
||||
base identity, including wrapped identities and same-ID fallback. This allows the
|
||||
caller to retain frame occurrences without guessing edges from names. Cache changes
|
||||
are committed only after a whole frame validates; failed frames do not poison the
|
||||
next frame. A cache limit causes an explicit failure rather than silent eviction.
|
||||
|
||||
Defaults bound directories to one million records / 32 MiB serialized metadata,
|
||||
individual encoded reads to 32 MiB, frame dimensions to 4096×4096 / 16,777,216 pixels,
|
||||
and cached keyframes to 64 MiB. Checks cover cancellation, checked source ranges,
|
||||
CRC, complete packet coverage, unsupported modes, matching base dimensions and
|
||||
allocation bounds. Raw and packed pixels can include transparent index 255;
|
||||
color/palette semantics remain separate from this decoder.
|
||||
|
||||
## Verification
|
||||
|
||||
`cargo test --locked --offline --manifest-path crates/verstack-radium/Cargo.toml`
|
||||
checks all six modes, column ordering, nibble ordering/odd dimensions, exact consumed
|
||||
length, wrapped and same-ID delta references, truncation, invalid packets, CRC,
|
||||
count limits, cancellation, duplicate pointers, SPIKE 2 sentinel handling, original
|
||||
PCM streaming and cache failure atomicity. Clippy passes with warnings denied.
|
||||
|
||||
`scripts/collect_radium_reference.py` records the existing Python decoder's exact
|
||||
pixel hashes, dimensions, consumption and sound-range hashes against complete,
|
||||
SHA-256-verified local images. The ignored corpus test verifies every recorded
|
||||
frame and streams every PCM chunk through a SHA-256 writer. Its environment path
|
||||
must be absolute because Cargo runs tests from the crate directory:
|
||||
|
||||
```
|
||||
VERSTACK_RADIUM_REFERENCE=/home/jordan/verstack/data/validation/native-radium-20260917 \
|
||||
cargo test --locked --offline --manifest-path crates/verstack-radium/Cargo.toml \
|
||||
--test corpus -- --include-ignored --nocapture
|
||||
```
|
||||
|
||||
The GOT input hash `b52720c75d5650b61005694cc61fd941d4ab5c054166f9a4eff2c71faf454d92`
|
||||
is the same image whose Python output was checked against the named ARM firmware
|
||||
routines under Unicorn (see the repository's `docs/decoder-evidence.md`). All
|
||||
17,159 native frames match that Python reference. Pokémon verification uses
|
||||
`2f1a958c1f859c339c716e53cc9ee752412373c486ea2a1bfd116eefce4617a3`, 1,477 frames
|
||||
and 2,491 PCM records. Every frame and PCM chunk matched; both final caches
|
||||
were 4,096 bytes. The full debug-mode verification completed in 766.47 seconds
|
||||
(`/tmp/verstack-radium-corpus.log`). This includes hashing both entire originals
|
||||
and every decoded pixel/PCM payload, not only matching decoder counts.
|
||||
|
||||
Root archive publication, Sequence/edge persistence and pipeline selection are
|
||||
integration work, not implied by a standalone parser test. No palette claim,
|
||||
SPK/wrapper streaming completion or sample-rate recovery is made by this crate.
|
||||
@@ -0,0 +1,208 @@
|
||||
//! Stateful exact index decoding; keyframe references retain directory ordinals.
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use binrw::BinRead;
|
||||
use serde::Serialize;
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
io::{Cursor, Read},
|
||||
sync::Arc,
|
||||
};
|
||||
|
||||
#[derive(BinRead)]
|
||||
#[br(little)]
|
||||
struct Header {
|
||||
identity: u32,
|
||||
flags: u32,
|
||||
width: u16,
|
||||
height: u16,
|
||||
mode: u8,
|
||||
}
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Frame {
|
||||
pub index: usize,
|
||||
pub identity: u32,
|
||||
pub flags: u32,
|
||||
pub width: u16,
|
||||
pub height: u16,
|
||||
pub mode: u8,
|
||||
pub consumed: usize,
|
||||
pub base_index: Option<usize>,
|
||||
pub base_identity: Option<u32>,
|
||||
#[serde(skip)]
|
||||
pub pixels: Arc<Vec<u8>>,
|
||||
}
|
||||
pub struct Decoder {
|
||||
frames: BTreeMap<u16, Frame>,
|
||||
cached_bytes: usize,
|
||||
max_cached_bytes: usize,
|
||||
max_pixels: usize,
|
||||
}
|
||||
impl Default for Decoder {
|
||||
fn default() -> Self {
|
||||
Self::new(16_777_216, 64 << 20)
|
||||
}
|
||||
}
|
||||
impl Decoder {
|
||||
pub fn new(max_pixels: usize, max_cached_bytes: usize) -> Self {
|
||||
Self {
|
||||
frames: BTreeMap::new(),
|
||||
cached_bytes: 0,
|
||||
max_cached_bytes,
|
||||
max_pixels,
|
||||
}
|
||||
}
|
||||
pub fn cached_bytes(&self) -> usize {
|
||||
self.cached_bytes
|
||||
}
|
||||
pub fn decode(
|
||||
&mut self,
|
||||
index: usize,
|
||||
record: &[u8],
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Frame> {
|
||||
ensure!(!cancel(), "DMD operation cancelled");
|
||||
let mut reader = Cursor::new(record);
|
||||
let h = Header::read(&mut reader).context("truncated DMD header")?;
|
||||
let size = usize::from(h.width) * usize::from(h.height);
|
||||
ensure!(
|
||||
h.width > 0
|
||||
&& h.height > 0
|
||||
&& h.width <= 4096
|
||||
&& h.height <= 4096
|
||||
&& size <= self.max_pixels,
|
||||
"DMD dimensions exceed bound"
|
||||
);
|
||||
ensure!(
|
||||
matches!(h.mode, 0 | 1 | 3 | 7 | 9 | 12),
|
||||
"unsupported DMD encoding {}",
|
||||
h.mode
|
||||
);
|
||||
let delta = matches!(h.mode, 3 | 9);
|
||||
let key = h.identity as u16;
|
||||
let base = if delta {
|
||||
Some(
|
||||
[key.wrapping_sub(1), key]
|
||||
.into_iter()
|
||||
.filter_map(|k| self.frames.get(&k))
|
||||
.find(|f| (f.width, f.height) == (h.width, h.height))
|
||||
.context("DMD delta lacks matching preceding keyframe")?,
|
||||
)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let mut pixels = base.map_or_else(|| vec![0; size], |f| f.pixels.as_ref().clone());
|
||||
let (base_index, base_identity) = (base.map(|f| f.index), base.map(|f| f.identity));
|
||||
let mut cursor = 0usize;
|
||||
let column = matches!(h.mode, 1 | 3);
|
||||
match h.mode {
|
||||
0 => {
|
||||
reader.read_exact(&mut pixels)?;
|
||||
}
|
||||
12 => {
|
||||
for pair in pixels.chunks_mut(2) {
|
||||
let byte = u8::read(&mut reader)?;
|
||||
pair[0] = byte & 15;
|
||||
if pair.len() == 2 {
|
||||
pair[1] = byte >> 4;
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
loop {
|
||||
ensure!(!cancel(), "DMD operation cancelled");
|
||||
let token = u8::read(&mut reader)?;
|
||||
if token == 0 {
|
||||
break;
|
||||
}
|
||||
if delta && token >= 128 {
|
||||
let skip = 256 - usize::from(token);
|
||||
ensure!(skip <= size - cursor, "DMD delta skip exceeds bounds");
|
||||
cursor += skip;
|
||||
continue;
|
||||
}
|
||||
let count = if delta {
|
||||
usize::from(token)
|
||||
} else {
|
||||
usize::from(token >> 2)
|
||||
};
|
||||
ensure!(count != 0, "empty DMD whitespace packet");
|
||||
let mut values = vec![0; count];
|
||||
match if delta { 0 } else { token & 3 } {
|
||||
0 => reader.read_exact(&mut values)?,
|
||||
1 => {}
|
||||
2 => values.fill(15),
|
||||
3 => values.fill(u8::read(&mut reader)?),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
ensure!(
|
||||
values.len() <= size - cursor,
|
||||
"DMD packet exceeds pixel bounds"
|
||||
);
|
||||
for value in values {
|
||||
let at = if column {
|
||||
(cursor % usize::from(h.height)) * usize::from(h.width)
|
||||
+ cursor / usize::from(h.height)
|
||||
} else {
|
||||
cursor
|
||||
};
|
||||
pixels[at] = value;
|
||||
cursor += 1;
|
||||
}
|
||||
}
|
||||
ensure!(cursor == size, "DMD packet did not cover complete frame");
|
||||
}
|
||||
}
|
||||
self.finish(
|
||||
index,
|
||||
h,
|
||||
pixels,
|
||||
reader.position() as usize,
|
||||
base_index,
|
||||
base_identity,
|
||||
)
|
||||
}
|
||||
fn finish(
|
||||
&mut self,
|
||||
index: usize,
|
||||
h: Header,
|
||||
pixels: Vec<u8>,
|
||||
consumed: usize,
|
||||
base_index: Option<usize>,
|
||||
base_identity: Option<u32>,
|
||||
) -> Result<Frame> {
|
||||
let result = Frame {
|
||||
index,
|
||||
identity: h.identity,
|
||||
flags: h.flags,
|
||||
width: h.width,
|
||||
height: h.height,
|
||||
mode: h.mode,
|
||||
consumed,
|
||||
base_index,
|
||||
base_identity,
|
||||
pixels: Arc::new(pixels),
|
||||
};
|
||||
let delta = matches!(h.mode, 3 | 9);
|
||||
if delta || h.flags & 4 != 0 {
|
||||
let key = h.identity as u16;
|
||||
let remove = if delta {
|
||||
self.frames
|
||||
.get(&key.wrapping_sub(1))
|
||||
.map_or(0, |f| f.pixels.len())
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let replace = self.frames.get(&key).map_or(0, |f| f.pixels.len());
|
||||
let needed = self.cached_bytes - remove - replace + result.pixels.len();
|
||||
if needed > self.max_cached_bytes {
|
||||
bail!("DMD keyframe cache exceeds bound");
|
||||
}
|
||||
if delta {
|
||||
self.frames.remove(&key.wrapping_sub(1));
|
||||
}
|
||||
self.frames.insert(key, result.clone());
|
||||
self.cached_bytes = needed;
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,284 @@
|
||||
//! Bounded Radium image directories and exact DMD indices, without palette guesses.
|
||||
//! Grammar: plugins/media_extract.py and plugins/dmd_bitmap.py in Verstack,
|
||||
//! backed by named GOT 1.37 loader/decompressor functions (docs/decoder-evidence.md).
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use binrw::BinRead;
|
||||
use serde::Serialize;
|
||||
use std::io::{Read, Seek, SeekFrom};
|
||||
|
||||
pub mod dmd;
|
||||
const CHUNK: usize = 1024 * 1024;
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct Limits {
|
||||
pub records: usize,
|
||||
pub metadata_bytes: u64,
|
||||
pub record_bytes: u64,
|
||||
}
|
||||
impl Default for Limits {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
records: 1_000_000,
|
||||
metadata_bytes: 32 << 20,
|
||||
record_bytes: 32 << 20,
|
||||
}
|
||||
}
|
||||
}
|
||||
#[derive(BinRead, Debug, Clone, Serialize)]
|
||||
#[br(little)]
|
||||
pub struct Header {
|
||||
pub sections: [u64; 10],
|
||||
pub unknown: [u64; 2],
|
||||
pub sound_count: u64,
|
||||
}
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Bitmap {
|
||||
pub index: usize,
|
||||
pub offset: u64,
|
||||
/// Container boundary, not the byte length consumed by a decoded bitmap.
|
||||
pub extent: u64,
|
||||
}
|
||||
#[derive(BinRead, Debug, Clone, Serialize)]
|
||||
#[br(little)]
|
||||
pub struct Sound {
|
||||
pub offset: u64,
|
||||
pub flags: u64,
|
||||
pub samples_word: u64,
|
||||
}
|
||||
impl Sound {
|
||||
pub fn channels(&self) -> u8 {
|
||||
((((self.flags >> 25) & 1) << 3)
|
||||
| (((self.flags >> 30) & 1) << 4)
|
||||
| (((self.flags >> 17) & 1) << 1)
|
||||
| (((self.flags >> 31) & 1) << 2)
|
||||
| ((self.flags >> 7) & 1)) as u8
|
||||
}
|
||||
pub fn samples(&self) -> u64 {
|
||||
self.samples_word & 0xffff_ffff
|
||||
}
|
||||
pub fn byte_len(&self) -> u64 {
|
||||
u64::from(self.channels()) * self.samples() * 2
|
||||
}
|
||||
}
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Index {
|
||||
pub size: u64,
|
||||
pub header: Header,
|
||||
pub bitmaps: Vec<Bitmap>,
|
||||
pub sounds: Vec<Sound>,
|
||||
pub sound_table_crc32: u32,
|
||||
}
|
||||
/// SPIKE 2 images have sentinel sound-section words; their bitmap directory is
|
||||
/// independently validated instead of rejecting all DMD coverage with the PCM table.
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct BitmapIndex {
|
||||
pub size: u64,
|
||||
pub bitmaps: Vec<Bitmap>,
|
||||
}
|
||||
impl BitmapIndex {
|
||||
pub fn read<R: Read + Seek>(
|
||||
source: &mut R,
|
||||
limits: Limits,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Self> {
|
||||
check(&cancel)?;
|
||||
let size = source.seek(SeekFrom::End(0))?;
|
||||
bounds(size, 0, 56)?;
|
||||
source.rewind()?;
|
||||
let header = <[u64; 7]>::read_le(source)?;
|
||||
let bitmaps = bitmap_directory(source, size, header[3], header[2], limits, &cancel)?;
|
||||
Ok(Self { size, bitmaps })
|
||||
}
|
||||
pub fn bitmap_bytes<R: Read + Seek>(
|
||||
&self,
|
||||
source: &mut R,
|
||||
index: usize,
|
||||
limits: Limits,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Vec<u8>> {
|
||||
read_bitmap(source, &self.bitmaps, index, limits, cancel)
|
||||
}
|
||||
}
|
||||
fn bitmap_directory<R: Read + Seek>(
|
||||
source: &mut R,
|
||||
size: u64,
|
||||
start: u64,
|
||||
end: u64,
|
||||
limits: Limits,
|
||||
cancel: &impl Fn() -> bool,
|
||||
) -> Result<Vec<Bitmap>> {
|
||||
ensure!(
|
||||
56 <= start && start < end && end < size && (end - start).is_multiple_of(8),
|
||||
"invalid Radium bitmap table"
|
||||
);
|
||||
let count = (end - start) / 8;
|
||||
ensure!(
|
||||
count <= limits.records as u64 && count * 8 <= limits.metadata_bytes,
|
||||
"Radium bitmap table exceeds bound"
|
||||
);
|
||||
source.seek(SeekFrom::Start(start))?;
|
||||
let mut pointers = Vec::with_capacity(count as usize);
|
||||
for _ in 0..count {
|
||||
check(&cancel)?;
|
||||
let offset = u64::read_le(source)?;
|
||||
bounds(size, offset, 13)?;
|
||||
ensure!(
|
||||
offset >= end && offset.is_multiple_of(8),
|
||||
"bitmap pointer outside asset area"
|
||||
);
|
||||
pointers.push(offset);
|
||||
}
|
||||
let mut ordered = pointers.clone();
|
||||
ordered.sort_unstable();
|
||||
ordered.dedup();
|
||||
let bitmaps = pointers
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.map(|(index, offset)| {
|
||||
let at = ordered.binary_search(&offset).expect("known bitmap offset");
|
||||
let next = ordered.get(at + 1).copied().unwrap_or(size);
|
||||
Bitmap {
|
||||
index,
|
||||
offset,
|
||||
extent: next - offset,
|
||||
}
|
||||
})
|
||||
.collect();
|
||||
Ok(bitmaps)
|
||||
}
|
||||
fn read_bitmap<R: Read + Seek>(
|
||||
source: &mut R,
|
||||
records: &[Bitmap],
|
||||
index: usize,
|
||||
limits: Limits,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Vec<u8>> {
|
||||
let record = records.get(index).context("unknown bitmap index")?;
|
||||
let length = record.extent.min(limits.record_bytes);
|
||||
let mut bytes = vec![0; usize::try_from(length)?];
|
||||
source.seek(SeekFrom::Start(record.offset))?;
|
||||
for chunk in bytes.chunks_mut(CHUNK) {
|
||||
check(&cancel)?;
|
||||
source.read_exact(chunk)?;
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
fn bounds(size: u64, offset: u64, length: u64) -> Result<()> {
|
||||
ensure!(
|
||||
offset <= size && length <= size - offset,
|
||||
"Radium range exceeds source"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
fn check(cancel: &impl Fn() -> bool) -> Result<()> {
|
||||
ensure!(!cancel(), "Radium operation cancelled");
|
||||
Ok(())
|
||||
}
|
||||
impl Index {
|
||||
pub fn read<R: Read + Seek>(
|
||||
source: &mut R,
|
||||
limits: Limits,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Self> {
|
||||
check(&cancel)?;
|
||||
let size = source.seek(SeekFrom::End(0))?;
|
||||
bounds(size, 0, 104)?;
|
||||
source.rewind()?;
|
||||
let header = Header::read(source).context("Radium header")?;
|
||||
ensure!(
|
||||
header.sections[0] >= 104 && header.sections.iter().all(|p| *p < size),
|
||||
"unrecognized Radium section layout"
|
||||
);
|
||||
ensure!(
|
||||
header.sound_count >> 32 == 0 && header.sound_count <= limits.records as u64,
|
||||
"Radium sound count exceeds bound"
|
||||
);
|
||||
let start = header.sections[3];
|
||||
let end = header.sections[2];
|
||||
ensure!(
|
||||
start >= 56 && start < end && (end - start).is_multiple_of(8),
|
||||
"invalid Radium bitmap table"
|
||||
);
|
||||
let count = (end - start) / 8;
|
||||
ensure!(
|
||||
count <= limits.records as u64,
|
||||
"Radium bitmap count exceeds bound"
|
||||
);
|
||||
let sound_bytes = (header.sound_count * 24 + 15) & !15;
|
||||
ensure!(
|
||||
sound_bytes + count * 8 + 108 <= limits.metadata_bytes,
|
||||
"Radium metadata exceeds bound"
|
||||
);
|
||||
bounds(size, header.sections[8], sound_bytes + 4)?;
|
||||
// CRC includes alignment bytes. Stream it instead of materializing the table.
|
||||
source.seek(SeekFrom::Start(header.sections[8]))?;
|
||||
let mut hasher = crc32fast::Hasher::new();
|
||||
let mut buffer = vec![0u8; CHUNK];
|
||||
let mut remaining = sound_bytes;
|
||||
while remaining > 0 {
|
||||
check(&cancel)?;
|
||||
let n = remaining.min(CHUNK as u64) as usize;
|
||||
source.read_exact(&mut buffer[..n])?;
|
||||
hasher.update(&buffer[..n]);
|
||||
remaining -= n as u64;
|
||||
}
|
||||
let crc = u32::read_le(source)?;
|
||||
ensure!(hasher.finalize() == crc, "Radium sound table CRC mismatch");
|
||||
source.seek(SeekFrom::Start(header.sections[8]))?;
|
||||
let mut sounds: Vec<Sound> = Vec::with_capacity(header.sound_count as usize);
|
||||
for _ in 0..header.sound_count {
|
||||
check(&cancel)?;
|
||||
let sound = Sound::read(source)?;
|
||||
bounds(size, sound.offset, sound.byte_len())?;
|
||||
ensure!(
|
||||
sound.offset + sound.byte_len() <= header.sections[8],
|
||||
"PCM overlaps sound directory"
|
||||
);
|
||||
if let Some(previous) = sounds.last() {
|
||||
ensure!(
|
||||
previous.offset + previous.byte_len() == sound.offset,
|
||||
"PCM range does not match next record"
|
||||
);
|
||||
}
|
||||
sounds.push(sound);
|
||||
}
|
||||
let bitmaps = bitmap_directory(source, size, start, end, limits, &cancel)?;
|
||||
Ok(Self {
|
||||
size,
|
||||
header,
|
||||
bitmaps,
|
||||
sounds,
|
||||
sound_table_crc32: crc,
|
||||
})
|
||||
}
|
||||
/// Read at most the decoding budget, not an arbitrarily large last-record extent.
|
||||
pub fn bitmap_bytes<R: Read + Seek>(
|
||||
&self,
|
||||
source: &mut R,
|
||||
index: usize,
|
||||
limits: Limits,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<Vec<u8>> {
|
||||
read_bitmap(source, &self.bitmaps, index, limits, cancel)
|
||||
}
|
||||
/// Original PCM bytes only: sample rate is not encoded by this directory.
|
||||
pub fn copy_sound<R: Read + Seek, W: std::io::Write>(
|
||||
&self,
|
||||
source: &mut R,
|
||||
index: usize,
|
||||
writer: &mut W,
|
||||
cancel: impl Fn() -> bool,
|
||||
) -> Result<u64> {
|
||||
let sound = self.sounds.get(index).context("unknown sound index")?;
|
||||
source.seek(SeekFrom::Start(sound.offset))?;
|
||||
let mut remaining = sound.byte_len();
|
||||
let mut buffer = vec![0; CHUNK];
|
||||
while remaining > 0 {
|
||||
check(&cancel)?;
|
||||
let n = remaining.min(CHUNK as u64) as usize;
|
||||
source.read_exact(&mut buffer[..n])?;
|
||||
writer.write_all(&buffer[..n])?;
|
||||
remaining -= n as u64;
|
||||
}
|
||||
Ok(sound.byte_len())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
use anyhow::Result;
|
||||
use serde_json::Value;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
fs::File,
|
||||
io::{Read, Write},
|
||||
path::PathBuf,
|
||||
};
|
||||
use verstack_radium::{BitmapIndex, Index, Limits, dmd::Decoder};
|
||||
struct DigestWriter(Sha256);
|
||||
impl Write for DigestWriter {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.0.update(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
fn hex(bytes: impl AsRef<[u8]>) -> String {
|
||||
bytes.as_ref().iter().map(|b| format!("{b:02x}")).collect()
|
||||
}
|
||||
#[test]
|
||||
#[ignore = "requires VERSTACK_RADIUM_REFERENCE with hashed real images and Python receipts"]
|
||||
fn native_outputs_match_every_reference_record() -> Result<()> {
|
||||
let directory = PathBuf::from(std::env::var("VERSTACK_RADIUM_REFERENCE")?);
|
||||
let summary: Value = serde_json::from_slice(&std::fs::read(directory.join("summary.json"))?)?;
|
||||
for item in 0..summary.as_array().unwrap().len() {
|
||||
let reference: Value =
|
||||
serde_json::from_slice(&std::fs::read(directory.join(format!("{item}.json")))?)?;
|
||||
assert!(reference["failures"].as_array().unwrap().is_empty());
|
||||
let mut source = File::open(reference["source"].as_str().unwrap())?;
|
||||
let mut hash = Sha256::new();
|
||||
let mut buffer = vec![0u8; 1024 * 1024];
|
||||
loop {
|
||||
let n = source.read(&mut buffer)?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
hash.update(&buffer[..n]);
|
||||
}
|
||||
assert_eq!(hex(hash.finalize()), reference["sha256"]);
|
||||
let index = BitmapIndex::read(&mut source, Limits::default(), || false)?;
|
||||
assert_eq!(
|
||||
index.bitmaps.len(),
|
||||
reference["frames"].as_array().unwrap().len()
|
||||
);
|
||||
let mut decoder = Decoder::default();
|
||||
for row in reference["frames"].as_array().unwrap() {
|
||||
let ordinal = row["index"].as_u64().unwrap() as usize;
|
||||
assert_eq!(
|
||||
index.bitmaps[ordinal].offset,
|
||||
row["offset"].as_u64().unwrap()
|
||||
);
|
||||
assert_eq!(
|
||||
index.bitmaps[ordinal].extent,
|
||||
row["extent"].as_u64().unwrap()
|
||||
);
|
||||
let record = index.bitmap_bytes(&mut source, ordinal, Limits::default(), || false)?;
|
||||
let frame = decoder.decode(ordinal, &record, || false)?;
|
||||
assert_eq!(
|
||||
hex(Sha256::digest(frame.pixels.as_slice())),
|
||||
row["pixel_sha256"],
|
||||
"frame {ordinal}"
|
||||
);
|
||||
assert_eq!(frame.identity as u64, row["record_id"].as_u64().unwrap());
|
||||
assert_eq!(frame.consumed as u64, row["consumed"].as_u64().unwrap());
|
||||
assert_eq!(frame.width as u64, row["width"].as_u64().unwrap());
|
||||
assert_eq!(frame.height as u64, row["height"].as_u64().unwrap());
|
||||
assert_eq!(frame.mode as u64, row["mode"].as_u64().unwrap());
|
||||
}
|
||||
if reference["sound_error"].is_null() {
|
||||
let index = Index::read(&mut source, Limits::default(), || false)?;
|
||||
assert_eq!(
|
||||
index.sounds.len(),
|
||||
reference["sounds"].as_array().unwrap().len()
|
||||
);
|
||||
for row in reference["sounds"].as_array().unwrap() {
|
||||
let ordinal = row["index"].as_u64().unwrap() as usize;
|
||||
let sound = &index.sounds[ordinal];
|
||||
assert_eq!(sound.offset, row["offset"].as_u64().unwrap());
|
||||
assert_eq!(sound.flags, row["flags"].as_u64().unwrap());
|
||||
assert_eq!(sound.samples_word, row["samples_word"].as_u64().unwrap());
|
||||
assert_eq!(sound.channels() as u64, row["channels"].as_u64().unwrap());
|
||||
let mut output = DigestWriter(Sha256::new());
|
||||
assert_eq!(
|
||||
index.copy_sound(&mut source, ordinal, &mut output, || false)?,
|
||||
row["length"].as_u64().unwrap()
|
||||
);
|
||||
assert_eq!(hex(output.0.finalize()), row["sha256"]);
|
||||
}
|
||||
} else {
|
||||
assert!(Index::read(&mut source, Limits::default(), || false).is_err());
|
||||
}
|
||||
eprintln!(
|
||||
"{}: {} exact frames, {} PCM chunks; cache {} bytes",
|
||||
reference["sha256"],
|
||||
index.bitmaps.len(),
|
||||
reference["sounds"].as_array().unwrap().len(),
|
||||
decoder.cached_bytes()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,165 @@
|
||||
use std::io::Cursor;
|
||||
use verstack_radium::{BitmapIndex, Index, Limits, dmd::Decoder};
|
||||
fn frame(mode: u8, payload: &[u8], id: u32, flags: u32, w: u16, h: u16) -> Vec<u8> {
|
||||
let mut b = Vec::new();
|
||||
b.extend(id.to_le_bytes());
|
||||
b.extend(flags.to_le_bytes());
|
||||
b.extend(w.to_le_bytes());
|
||||
b.extend(h.to_le_bytes());
|
||||
b.push(mode);
|
||||
b.extend(payload);
|
||||
b
|
||||
}
|
||||
#[test]
|
||||
fn all_encodings_and_exact_consumption() {
|
||||
let cases = [
|
||||
(0, vec![1, 2, 3, 4], vec![1, 2, 3, 4]),
|
||||
(7, vec![9, 7, 6, 4, 9, 0], vec![0, 0, 6, 9]),
|
||||
(1, vec![9, 7, 6, 4, 9, 0], vec![0, 6, 0, 9]),
|
||||
(12, vec![0x21, 0x43], vec![1, 2, 3, 4]),
|
||||
];
|
||||
for (mode, payload, expected) in cases {
|
||||
let mut raw = frame(mode, &payload, 1, 4, 2, 2);
|
||||
let consumed = raw.len();
|
||||
raw.extend([99; 8]);
|
||||
let got = Decoder::default().decode(0, &raw, || false).unwrap();
|
||||
assert_eq!(*got.pixels, expected);
|
||||
assert_eq!(got.consumed, consumed);
|
||||
}
|
||||
let got = Decoder::default()
|
||||
.decode(0, &frame(12, &[0x21, 0xf3], 1, 0, 3, 1), || false)
|
||||
.unwrap();
|
||||
assert_eq!(*got.pixels, [1, 2, 3]);
|
||||
for mode in [3, 9] {
|
||||
let mut d = Decoder::default();
|
||||
d.decode(10, &frame(0, &[1, 2, 3, 4], 65535, 4, 2, 2), || false)
|
||||
.unwrap();
|
||||
let got = d
|
||||
.decode(11, &frame(mode, &[254, 2, 8, 9, 0], 0, 0, 2, 2), || false)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
*got.pixels,
|
||||
if mode == 3 {
|
||||
vec![1, 8, 3, 9]
|
||||
} else {
|
||||
vec![1, 2, 8, 9]
|
||||
}
|
||||
);
|
||||
assert_eq!(got.base_index, Some(10));
|
||||
assert_eq!(got.base_identity, Some(65535));
|
||||
assert_eq!(d.cached_bytes(), 4);
|
||||
// Same-ID fallback records the actual base, not the hypothetical id-1.
|
||||
let got = d
|
||||
.decode(12, &frame(mode, &[252, 0], 0, 0, 2, 2), || false)
|
||||
.unwrap();
|
||||
assert_eq!(got.base_index, Some(11));
|
||||
assert_eq!(got.base_identity, Some(0));
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn malformed_packets_cancellation_and_failed_cache_writes_are_atomic() {
|
||||
let samples = [
|
||||
frame(0, &[1, 2, 3, 4], 1, 4, 2, 2),
|
||||
frame(12, &[0x21, 0x43], 1, 4, 2, 2),
|
||||
frame(1, &[16, 1, 2, 3, 4, 0], 1, 4, 2, 2),
|
||||
frame(7, &[16, 1, 2, 3, 4, 0], 1, 4, 2, 2),
|
||||
];
|
||||
for raw in samples {
|
||||
for len in 0..raw.len() {
|
||||
let mut d = Decoder::default();
|
||||
assert!(d.decode(0, &raw[..len], || false).is_err());
|
||||
assert_eq!(d.cached_bytes(), 0);
|
||||
}
|
||||
}
|
||||
let mut d = Decoder::new(4, 4);
|
||||
d.decode(1, &frame(0, &[1, 2, 3, 4], 1, 4, 2, 2), || false)
|
||||
.unwrap();
|
||||
for raw in [
|
||||
frame(0, &[5, 6, 7, 8], 4, 4, 2, 2),
|
||||
frame(9, &[251, 0], 2, 0, 2, 2),
|
||||
frame(9, &[0], 2, 0, 2, 2),
|
||||
frame(7, &[3, 0], 2, 0, 2, 2),
|
||||
frame(2, &[0], 2, 0, 2, 2),
|
||||
] {
|
||||
assert!(d.decode(2, &raw, || false).is_err());
|
||||
assert_eq!(d.cached_bytes(), 4);
|
||||
}
|
||||
assert!(
|
||||
d.decode(2, &frame(9, &[252, 0], 2, 0, 2, 2), || true)
|
||||
.is_err()
|
||||
);
|
||||
let got = d
|
||||
.decode(2, &frame(9, &[252, 0], 2, 0, 2, 2), || false)
|
||||
.unwrap();
|
||||
assert_eq!(*got.pixels, [1, 2, 3, 4]);
|
||||
}
|
||||
fn fixture() -> Vec<u8> {
|
||||
let mut b = vec![0u8; 256];
|
||||
let words = [104u64, 144, 136, 120, 104, 104, 104, 252, 208, 104, 0, 0, 1];
|
||||
for (i, w) in words.iter().enumerate() {
|
||||
b[i * 8..i * 8 + 8].copy_from_slice(&w.to_le_bytes());
|
||||
}
|
||||
// Two directory ordinals legitimately share one record offset.
|
||||
b[120..128].copy_from_slice(&144u64.to_le_bytes());
|
||||
b[128..136].copy_from_slice(&144u64.to_le_bytes());
|
||||
let raw = frame(0, &[1, 2, 3, 4], 1, 4, 2, 2);
|
||||
b[144..144 + raw.len()].copy_from_slice(&raw);
|
||||
b[200..204].copy_from_slice(&[1, 2, 3, 4]);
|
||||
b[208..216].copy_from_slice(&200u64.to_le_bytes());
|
||||
b[216..224].copy_from_slice(&128u64.to_le_bytes());
|
||||
b[224..232].copy_from_slice(&2u64.to_le_bytes());
|
||||
let crc = crc32fast::hash(&b[208..240]);
|
||||
b[240..244].copy_from_slice(&crc.to_le_bytes());
|
||||
b
|
||||
}
|
||||
#[test]
|
||||
fn directories_crc_ranges_and_sound_copy() {
|
||||
let raw = fixture();
|
||||
let mut c = Cursor::new(&raw);
|
||||
let i = Index::read(&mut c, Limits::default(), || false).unwrap();
|
||||
assert_eq!(i.bitmaps.len(), 2);
|
||||
assert_eq!(i.bitmaps[0].offset, i.bitmaps[1].offset);
|
||||
assert_eq!(i.sounds[0].channels(), 1);
|
||||
assert_eq!(i.sounds[0].samples(), 2);
|
||||
let mut out = Vec::new();
|
||||
assert_eq!(i.copy_sound(&mut c, 0, &mut out, || false).unwrap(), 4);
|
||||
assert_eq!(out, [1, 2, 3, 4]);
|
||||
let rec = i
|
||||
.bitmap_bytes(&mut c, 0, Limits::default(), || false)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
Decoder::default()
|
||||
.decode(0, &rec, || false)
|
||||
.unwrap()
|
||||
.consumed,
|
||||
17
|
||||
);
|
||||
for len in 0..244 {
|
||||
assert!(Index::read(&mut Cursor::new(&raw[..len]), Limits::default(), || false).is_err());
|
||||
}
|
||||
let mut corrupt = raw.clone();
|
||||
corrupt[239] ^= 1;
|
||||
assert!(Index::read(&mut Cursor::new(corrupt), Limits::default(), || false).is_err());
|
||||
assert!(
|
||||
Index::read(
|
||||
&mut Cursor::new(&raw),
|
||||
Limits {
|
||||
records: 0,
|
||||
..Limits::default()
|
||||
},
|
||||
|| false
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
assert!(Index::read(&mut Cursor::new(&raw), Limits::default(), || true).is_err());
|
||||
let mut legacy = raw;
|
||||
legacy[56..104].fill(255);
|
||||
assert!(Index::read(&mut Cursor::new(&legacy), Limits::default(), || false).is_err());
|
||||
assert_eq!(
|
||||
BitmapIndex::read(&mut Cursor::new(&legacy), Limits::default(), || false)
|
||||
.unwrap()
|
||||
.bitmaps
|
||||
.len(),
|
||||
2
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
/target/
|
||||
Generated
+319
@@ -0,0 +1,319 @@
|
||||
# This file is automatically @generated by Cargo.
|
||||
# It is not intended for manual editing.
|
||||
version = 4
|
||||
|
||||
[[package]]
|
||||
name = "anyhow"
|
||||
version = "1.0.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
|
||||
|
||||
[[package]]
|
||||
name = "bitflags"
|
||||
version = "2.13.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3ded4057c258ba199e2d26386d3af3780957ecaee6c4ef4041c6b4b8b97c0b06"
|
||||
|
||||
[[package]]
|
||||
name = "block-buffer"
|
||||
version = "0.10.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.2.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
|
||||
dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crypto-common"
|
||||
version = "0.1.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
|
||||
dependencies = [
|
||||
"generic-array",
|
||||
"typenum",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "digest"
|
||||
version = "0.10.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
|
||||
dependencies = [
|
||||
"block-buffer",
|
||||
"crypto-common",
|
||||
"subtle",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "errno"
|
||||
version = "0.3.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastrand"
|
||||
version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
|
||||
|
||||
[[package]]
|
||||
name = "generic-array"
|
||||
version = "0.14.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
|
||||
dependencies = [
|
||||
"typenum",
|
||||
"version_check",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "getrandom"
|
||||
version = "0.4.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"libc",
|
||||
"r-efi",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hmac"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e"
|
||||
dependencies = [
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "itoa"
|
||||
version = "1.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
|
||||
|
||||
[[package]]
|
||||
name = "libc"
|
||||
version = "0.2.189"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
|
||||
|
||||
[[package]]
|
||||
name = "linux-raw-sys"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
|
||||
|
||||
[[package]]
|
||||
name = "md-5"
|
||||
version = "0.10.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.107"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.47"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "r-efi"
|
||||
version = "6.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
|
||||
|
||||
[[package]]
|
||||
name = "rustix"
|
||||
version = "1.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"errno",
|
||||
"libc",
|
||||
"linux-raw-sys",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_core"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
|
||||
dependencies = [
|
||||
"serde_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_derive"
|
||||
version = "1.0.229"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "serde_json"
|
||||
version = "1.0.151"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
|
||||
dependencies = [
|
||||
"itoa",
|
||||
"memchr",
|
||||
"serde",
|
||||
"serde_core",
|
||||
"zmij",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sha1"
|
||||
version = "0.10.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures",
|
||||
"digest",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "subtle"
|
||||
version = "2.6.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tempfile"
|
||||
version = "3.27.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
|
||||
dependencies = [
|
||||
"fastrand",
|
||||
"getrandom",
|
||||
"once_cell",
|
||||
"rustix",
|
||||
"windows-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "typenum"
|
||||
version = "1.20.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.25"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ab72a15cf68d77cb0987d3684aa8a45c5ef827e8cb49ee2f30bfd7ba2feb519f"
|
||||
|
||||
[[package]]
|
||||
name = "version_check"
|
||||
version = "0.9.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
|
||||
|
||||
[[package]]
|
||||
name = "verstack-spk"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"hmac",
|
||||
"md-5",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha1",
|
||||
"tempfile",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-link"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.61.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
|
||||
dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
|
||||
@@ -0,0 +1,21 @@
|
||||
[package]
|
||||
name = "verstack-spk"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
autotests = false
|
||||
license = "MIT"
|
||||
description = "Bounded streaming reader and verifier for raw SPIKE SPKS packages"
|
||||
|
||||
[workspace]
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
hmac = "0.12"
|
||||
sha1 = "0.10"
|
||||
md-5 = "0.10"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025 Mark Rowe
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,5 @@
|
||||
SPK binary grammar and verification constants are derived from spike-spk,
|
||||
Copyright (c) 2025 Mark Rowe, MIT License; see LICENSE-upstream.
|
||||
Reference: tools/src/bdash-spike-spk-63c5d9a-type4, based on upstream commit
|
||||
63c5d9a with the local numeric package-type4 compatibility patch.
|
||||
The bounded parser, streaming I/O and transactional extraction are new code.
|
||||
@@ -0,0 +1,102 @@
|
||||
# verstack-spk
|
||||
|
||||
An isolated MIT-licensed Rust library for bounded raw `SPKS` indexing,
|
||||
streaming MD5/HMAC-SHA1 verification, and regular-file extraction. Binary grammar
|
||||
and the public format verification constant come from Mark Rowe's MIT `spike-spk`
|
||||
implementation; attribution and license text are in `NOTICE` and `LICENSE-upstream`.
|
||||
The root archive, native decode registry, CLI, guarded HTTP extraction, and opt-in
|
||||
Python helper now consume this crate; see `../../docs/native-spk.md`.
|
||||
|
||||
```rust,no_run
|
||||
use std::{fs::File, sync::atomic::AtomicBool};
|
||||
use verstack_spk::{Limits, Spk};
|
||||
# fn example() -> anyhow::Result<()> {
|
||||
let cancelled = AtomicBool::new(false);
|
||||
let mut package = Spk::parse(File::open("input.spk")?, Limits::new(8 << 30), &cancelled)?;
|
||||
let index = package.index(); // numeric package types, names, versions, byte ranges, digests
|
||||
println!("{} packages", index.packages.len());
|
||||
let verified = package.verify(&cancelled)?;
|
||||
println!("{} bytes verified", verified.bytes);
|
||||
package.extract(std::path::Path::new("new-output-directory"), &cancelled)?;
|
||||
# Ok(()) }
|
||||
```
|
||||
|
||||
`Spk<R>` accepts any `Read + Seek`, including borrowed readers with no `Send`,
|
||||
`Sync`, or `'static` requirement. `copy_verified` streams one selected member to a
|
||||
writer; callers must supply an unpublished, rollback-capable sink because digest
|
||||
failure is discovered after bytes have been written. `extract` creates an
|
||||
exclusive destination and removes it on error, cancellation, or unwinding. It
|
||||
never deletes an existing destination. It emits only regular files, strips
|
||||
setuid/setgid/sticky bits, and records skipped paths. Every member is verified,
|
||||
including skipped special files. The index retains full original mode/digest
|
||||
metadata. No input content is executed.
|
||||
|
||||
Index parsing is limited to 256 packages, 200,000 files, 32 MiB of metadata reads,
|
||||
16 MiB per string table, 16 MiB cumulative output-path bytes, and 4 KiB filenames.
|
||||
Path accounting also bounds amplification from shared/overlapping string offsets.
|
||||
All chunk boundaries, seeks, and data ranges are checked against the source and
|
||||
enclosing package. Payload output has an explicit caller budget and streams with
|
||||
a 1 MiB buffer. Cancellation is checked between metadata operations and payload
|
||||
chunks; an already blocked source read cannot be interrupted by this flag.
|
||||
|
||||
Observed format contracts are deliberately explicit:
|
||||
|
||||
* `SPKS` length counts its `SPK0` children, excluding the count word and `SEND`.
|
||||
* `SIDX` length includes its fixed metadata and following STRS/file-index/FEND.
|
||||
* Real `SDAT` length is commonly zero. The enclosing SPK0 supplies the bounds.
|
||||
* FINF and FI64 records have 60-byte and 80-byte bodies respectively.
|
||||
* Extended 64-bit SPKS/SPK0/SIDX/SDAT lengths and optional SZ64 are supported;
|
||||
unknown SZ64 bytes are retained rather than given invented meaning.
|
||||
* A 12-byte SEND trailer is required; its four payload bytes are preserved.
|
||||
* Types 1, 2, 3 and 4 are retained numerically; hardware generation is not inferred.
|
||||
* Declared `file_size` and stored `data_size` remain separate. Verification and
|
||||
extraction consume `data_size`, matching the oracle; no padding, truncation,
|
||||
decompression, or equality assumption is introduced.
|
||||
|
||||
Wrappers (ZIP, gzip, LUKS, split SquashFS) are not implemented here. Empty directory
|
||||
records and other nonregular objects are not recreated on the host. Ownership,
|
||||
xattrs and timestamps are not restored. A complete streaming unwrap tower and
|
||||
remaining wrapper integration remain unfinished. The HMAC uses a publicly known format constant; it checks format integrity,
|
||||
not trust or publisher authentication.
|
||||
|
||||
## Validation
|
||||
|
||||
```sh
|
||||
cargo test --offline --locked --manifest-path crates/verstack-spk/Cargo.toml
|
||||
cargo clippy --offline --locked --manifest-path crates/verstack-spk/Cargo.toml --all-targets -- -D warnings
|
||||
# Host-specific proprietary fixtures and existing pinned tool required:
|
||||
cargo test --offline --locked --manifest-path crates/verstack-spk/Cargo.toml -- --include-ignored
|
||||
```
|
||||
|
||||
The example executable accepts `inspect`, `verify`, or `extract`:
|
||||
|
||||
```sh
|
||||
cargo run --offline --locked --manifest-path crates/verstack-spk/Cargo.toml --example spk -- inspect input.spk
|
||||
```
|
||||
|
||||
Actual evidence from this host is kept outside Git in `data/validation/native-spk`:
|
||||
|
||||
* Twelve real raw packages: 15,694,521,658 source bytes, 24 packages, 6,956 files;
|
||||
native indexing read 1,276,180 metadata bytes. All declared/stored sizes agree.
|
||||
These were indexed, not fully payload-verified.
|
||||
* Pokémon 0.86 archived source: four exact HTTP 206 ranges totaling 233,644 bytes
|
||||
indexed 708 FINF files, including numeric type4. All sizes agree. This source
|
||||
does not provide real FI64 coverage.
|
||||
* Two derived fixtures preserve complete original system SPK0 bytes while
|
||||
rewriting the SPKS count/length to one package: GOT (10,602,323 original bytes)
|
||||
and Pokémon 0.86 (24,384,253 original bytes). All 36 members pass native and
|
||||
pinned upstream MD5/HMAC checks, and every extracted file matches the upstream
|
||||
output. Original files/catalog entries are unchanged. Receipts include source
|
||||
offsets, snapshot/path, SHA-256, and derivation details.
|
||||
* Synthetic FI64/extended-header/SZ64 fixture passes the upstream oracle too.
|
||||
This is grammar coverage, not a claim of real FI64 corpus verification.
|
||||
* Debug-only Pokémon system verification read 24,382,441 payload bytes in 5.28 s,
|
||||
with 3,648 KiB maximum RSS from `/usr/bin/time`. This is a single local fixture
|
||||
measurement, not a release performance or production workload claim.
|
||||
|
||||
Host-specific read-only fixture preparation scripts are under `scripts/` and
|
||||
write only to ignored validation storage. Default tests cover malformed lengths
|
||||
and paths, independent MD5/HMAC failures, bounds, cancellation, I/O failure,
|
||||
panic cleanup, permissions, and maximum payload-read size. Two oracle tests are
|
||||
ignored by default because their local tool and proprietary fixtures are not
|
||||
shipped with this crate.
|
||||
@@ -0,0 +1,25 @@
|
||||
use anyhow::{Context, Result};
|
||||
use std::{fs::File, path::Path, sync::atomic::AtomicBool};
|
||||
use verstack_spk::{Limits, Spk};
|
||||
fn main() -> Result<()> {
|
||||
let args: Vec<_> = std::env::args().collect();
|
||||
let command = args
|
||||
.get(1)
|
||||
.context("usage: spk inspect|verify|extract SOURCE [DEST]")?;
|
||||
let source = args.get(2).context("missing source")?;
|
||||
let cancel = AtomicBool::new(false);
|
||||
let mut spk = Spk::parse(File::open(source)?, Limits::new(128 << 30), &cancel)?;
|
||||
match command.as_str() {
|
||||
"inspect" => println!("{}", serde_json::to_string_pretty(spk.index())?),
|
||||
"verify" => println!("{}", serde_json::to_string_pretty(&spk.verify(&cancel)?)?),
|
||||
"extract" => println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(&spk.extract(
|
||||
Path::new(args.get(3).context("missing destination")?),
|
||||
&cancel
|
||||
)?)?
|
||||
),
|
||||
_ => anyhow::bail!("unknown command"),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
# Host-specific, read-only fixture preparation; firmware bytes remain in ignored data/.
|
||||
import urllib.request,urllib.parse,struct,json,pathlib
|
||||
SNAPSHOT='90d7833a-d381-41b0-8567-b8b8e5a43b1c'
|
||||
PATH='wrapper-payload/filesystem/pokemon_le-0_86_0_spike3.spk'
|
||||
SIZE=2319683756
|
||||
url='http://127.0.0.1:8080/api/file/'+SNAPSHOT+'?'+urllib.parse.urlencode({'path':PATH})
|
||||
cache={}; fetched=0; pos=0
|
||||
root=pathlib.Path(__file__).resolve().parents[3]/'data/validation/native-spk';root.mkdir(parents=True,exist_ok=True)
|
||||
def read(n):
|
||||
global pos,fetched
|
||||
assert 0<=n<=8*1024**2
|
||||
start=pos//65536*65536
|
||||
if start not in cache:
|
||||
end=min(SIZE,start+max(65536,n));assert fetched+end-start<=8*1024**2
|
||||
r=urllib.request.urlopen(urllib.request.Request(url,headers={'Range':f'bytes={start}-{end-1}'}),timeout=60)
|
||||
assert r.status==206 and r.headers['Content-Range']==f'bytes {start}-{end-1}/{SIZE}'
|
||||
b=r.read(end-start+1);assert len(b)==end-start;cache[start]=b;fetched+=len(b);assert fetched<=8*1024**2
|
||||
b=cache[start];off=pos-start
|
||||
if off+n>len(b):
|
||||
first=b[off:];pos=start+len(b);return first+read(n-len(first))
|
||||
pos+=n;return b[off:off+n]
|
||||
def head():
|
||||
tag,n=struct.unpack('<4sI',read(8));h=8
|
||||
if n==0xffffffff:n=struct.unpack('<Q',read(8))[0];h=16
|
||||
return tag,n,h
|
||||
roothead=head();count=struct.unpack('<I',read(4))[0];packages=[]
|
||||
for i in range(count):
|
||||
base=pos;tag,n,h=head();end=base+n+h;ih=head();md=read(48);sh=head()
|
||||
if sh[0]==b'SZ64':read(sh[1]);sh=head()
|
||||
assert sh[0]==b'STRS';strings=read(sh[1]);files=[]
|
||||
while True:
|
||||
rh=head()
|
||||
if rh[0]==b'FEND':break
|
||||
b=read(rh[1]);w=8 if rh[0]==b'FI64' else 4;v=struct.unpack_from('<QQQQ' if w==8 else '<IIII',b)
|
||||
name=strings[v[0]:].split(b'\0',1)[0].decode();files.append({'name':name,'file_size':v[1],'data_offset':v[2],'data_size':v[3],'mode':struct.unpack_from('<H',b,4*w)[0],'record':rh[0].decode()})
|
||||
dh=head();data=pos
|
||||
packages.append({'base':base,'end':end,'name':md[:29].split(b'\0')[0].decode(),'type':md[35],'version':list(md[32:35]),'index_header':list(ih[1:]),'data_start':data,'sdat':list(dh[1:]),'files':files})
|
||||
pos=end
|
||||
trailer=read(SIZE-pos)
|
||||
for start,b in cache.items():(root/f'pokemon086-range-{start}.bin').write_bytes(b)
|
||||
result={'snapshot':SNAPSHOT,'path':PATH,'size':SIZE,'fetched':fetched,'root_header':list(roothead[1:]),'packages':packages,'trailer_hex':trailer.hex(),'ranges':[{'start':start,'bytes':len(b),'file':f'pokemon086-range-{start}.bin'} for start,b in cache.items()]}
|
||||
(root/'pokemon086-index.json').write_text(json.dumps(result,indent=2))
|
||||
print(json.dumps({'fetched':fetched,'ranges':len(cache),'packages':[{'name':p['name'],'type':p['type'],'files':len(p['files']),'records':sorted(set(f['record'] for f in p['files'])),'size_mismatches':sum(f['file_size']!=f['data_size'] for f in p['files']),'sdat':p['sdat'],'package_bytes':p['end']-p['base']} for p in packages],'trailer':trailer.hex()},indent=2))
|
||||
@@ -0,0 +1,28 @@
|
||||
# Host-specific, read-only fixture preparation; firmware bytes remain in ignored data/.
|
||||
import pathlib,struct,json,urllib.request,urllib.parse,hashlib
|
||||
root=pathlib.Path(__file__).resolve().parents[3]/'data/validation/native-spk'
|
||||
records=[]
|
||||
local=pathlib.Path('/srv/firmware/images/stern_game_code/GOT-1_37_0.spk')
|
||||
with local.open('rb') as source:
|
||||
source.seek(12);h=source.read(8);assert h[:4]==b'SPK0';length=8+struct.unpack_from('<I',h,4)[0];assert length<32*1024**2;source.seek(12)
|
||||
dest=root/'got-system.spk'
|
||||
with dest.open('wb') as output:
|
||||
output.write(b'SPKS'+struct.pack('<II',length,1));remaining=length
|
||||
while remaining:
|
||||
b=source.read(min(1024**2,remaining));assert b;output.write(b);remaining-=len(b)
|
||||
output.write(b'SEND'+struct.pack('<II',4,0))
|
||||
records.append({'fixture':dest.name,'source':str(local),'source_offset':12,'source_bytes':length,'derivation':'complete original SPK0 with SPKS count/length rewritten to one package; unchanged SEND'})
|
||||
x=json.loads((root/'pokemon086-index.json').read_text());p=x['packages'][0];length=p['end']-p['base'];assert length<32*1024**2
|
||||
url='http://127.0.0.1:8080/api/file/'+x['snapshot']+'?'+urllib.parse.urlencode({'path':x['path']})
|
||||
dest=root/'pokemon086-system.spk';requests=0
|
||||
with dest.open('wb') as output:
|
||||
output.write(b'SPKS'+struct.pack('<II',length,1))
|
||||
for start in range(p['base'],p['end'],1024**2):
|
||||
end=min(start+1024**2,p['end']);response=urllib.request.urlopen(urllib.request.Request(url,headers={'Range':f'bytes={start}-{end-1}'}),timeout=60)
|
||||
assert response.status==206 and response.headers['Content-Range']==f'bytes {start}-{end-1}/{x["size"]}'
|
||||
data=response.read(end-start+1);assert len(data)==end-start;output.write(data);requests+=1
|
||||
output.write(bytes.fromhex(x['trailer_hex']))
|
||||
records.append({'fixture':dest.name,'source_snapshot':x['snapshot'],'source_path':x['path'],'source_offset':p['base'],'source_bytes':length,'http_requests':requests,'derivation':'complete original SPK0 with SPKS count/length rewritten to one package; unchanged SEND'})
|
||||
for record in records:
|
||||
with (root/record['fixture']).open('rb') as f:record['sha256']=hashlib.file_digest(f,'sha256').hexdigest()
|
||||
(root/'system-fixtures.json').write_text(json.dumps(records,indent=2));print(json.dumps(records,indent=2))
|
||||
@@ -0,0 +1,570 @@
|
||||
//! Streaming raw SPKS parser. Grammar attribution: NOTICE and LICENSE-upstream.
|
||||
//! No whole payload is buffered. Wrappers (gzip/ZIP/LUKS/SquashFS) are separate.
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use hmac::{Hmac, Mac};
|
||||
use md5::{Digest, Md5};
|
||||
use serde::Serialize;
|
||||
use sha1::Sha1;
|
||||
use std::{
|
||||
collections::BTreeSet,
|
||||
fs,
|
||||
io::{Read, Seek, SeekFrom, Write},
|
||||
os::unix::fs::PermissionsExt,
|
||||
path::Path,
|
||||
sync::atomic::{AtomicBool, Ordering},
|
||||
};
|
||||
|
||||
const BUFFER: usize = 1 << 20;
|
||||
const MAX_METADATA: u64 = 32 << 20;
|
||||
const MAX_STRINGS: u64 = 16 << 20;
|
||||
// Public format verification constant from the MIT reference implementation.
|
||||
const HMAC_KEY: &[u8] = &[
|
||||
0x8e, 0x1f, 0x55, 0x43, 0xc2, 0xf5, 0x4a, 0x11, 0x67, 0x3a, 0x28, 0x2a, 0x2f, 0x87, 0xc0, 0x06,
|
||||
];
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Limits {
|
||||
pub output_bytes: u64,
|
||||
pub files: usize,
|
||||
pub metadata_bytes: u64,
|
||||
}
|
||||
impl Limits {
|
||||
pub fn new(output_bytes: u64) -> Self {
|
||||
Self {
|
||||
output_bytes,
|
||||
files: 200_000,
|
||||
metadata_bytes: MAX_METADATA,
|
||||
}
|
||||
}
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct FileInfo {
|
||||
pub name: String,
|
||||
/// Logical size declared in FINF/FI64, retained independently from stored_size.
|
||||
pub size: u64,
|
||||
pub stored_size: u64,
|
||||
pub offset: u64,
|
||||
pub mode: u16,
|
||||
pub md5: [u8; 16],
|
||||
pub hmac_sha1: [u8; 20],
|
||||
pub record: String,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Package {
|
||||
pub name: String,
|
||||
pub package_id: [u8; 3],
|
||||
pub version: [u8; 3],
|
||||
/// Preserve numeric values; do not infer hardware generation.
|
||||
pub package_type: u8,
|
||||
pub unknown_metadata: [u8; 12],
|
||||
pub sz64: Option<Vec<u8>>,
|
||||
pub declared_sdat_bytes: u64,
|
||||
pub files: Vec<FileInfo>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize)]
|
||||
pub struct Index {
|
||||
pub source_bytes: u64,
|
||||
pub output_bytes: u64,
|
||||
pub metadata_bytes: u64,
|
||||
pub packages: Vec<Package>,
|
||||
pub trailer: [u8; 4],
|
||||
}
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Verification {
|
||||
pub files: usize,
|
||||
pub bytes: u64,
|
||||
}
|
||||
#[derive(Debug, Serialize)]
|
||||
pub struct Extraction {
|
||||
pub verified: Verification,
|
||||
pub regular_files: usize,
|
||||
pub skipped: Vec<String>,
|
||||
}
|
||||
pub struct Spk<R> {
|
||||
source: R,
|
||||
index: Index,
|
||||
}
|
||||
|
||||
/// A bounded view of one verified SPK member. The view retains the SPK source
|
||||
/// and streams payload bytes directly, allowing another seekable decoder to be
|
||||
/// composed without extracting the member to a filesystem path. A complete
|
||||
/// read from offset zero verifies both member digests at EOF.
|
||||
pub struct MemberReader<'a, R> {
|
||||
source: &'a mut R,
|
||||
file: FileInfo,
|
||||
position: u64,
|
||||
md5: Md5,
|
||||
hmac: Hmac<Sha1>,
|
||||
verify: bool,
|
||||
}
|
||||
fn cancelled(flag: &AtomicBool) -> Result<()> {
|
||||
ensure!(!flag.load(Ordering::Relaxed), "SPK operation cancelled");
|
||||
Ok(())
|
||||
}
|
||||
fn safe_path(path: &str) -> Result<()> {
|
||||
ensure!(
|
||||
!path.is_empty()
|
||||
&& path.len() <= 4096
|
||||
&& !path.contains(['\\', '\0', '\r', '\n'])
|
||||
&& path
|
||||
.split('/')
|
||||
.all(|p| !p.is_empty() && p != "." && p != ".."),
|
||||
"unsafe SPK path"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
struct Header {
|
||||
tag: [u8; 4],
|
||||
length: u64,
|
||||
data: u64,
|
||||
}
|
||||
impl Header {
|
||||
fn end(&self) -> Result<u64> {
|
||||
self.data
|
||||
.checked_add(self.length)
|
||||
.context("SPK chunk size overflow")
|
||||
}
|
||||
}
|
||||
struct Parser<'a, R> {
|
||||
source: &'a mut R,
|
||||
size: u64,
|
||||
metadata: u64,
|
||||
limits: &'a Limits,
|
||||
cancel: &'a AtomicBool,
|
||||
}
|
||||
impl<R: Read + Seek> Parser<'_, R> {
|
||||
fn bytes(&mut self, count: usize, boundary: u64) -> Result<Vec<u8>> {
|
||||
cancelled(self.cancel)?;
|
||||
self.metadata = self
|
||||
.metadata
|
||||
.checked_add(count as u64)
|
||||
.context("SPK metadata overflow")?;
|
||||
ensure!(
|
||||
self.metadata <= self.limits.metadata_bytes,
|
||||
"SPK metadata budget exceeded"
|
||||
);
|
||||
let position = self.source.stream_position()?;
|
||||
ensure!(
|
||||
position <= boundary && count as u64 <= boundary - position && boundary <= self.size,
|
||||
"truncated SPK metadata"
|
||||
);
|
||||
let mut bytes = vec![0; count];
|
||||
self.source.read_exact(&mut bytes)?;
|
||||
Ok(bytes)
|
||||
}
|
||||
fn header(&mut self, boundary: u64) -> Result<Header> {
|
||||
let bytes = self.bytes(8, boundary)?;
|
||||
let tag = bytes[..4].try_into().unwrap();
|
||||
let mut length = u64::from(u32::from_le_bytes(bytes[4..].try_into().unwrap()));
|
||||
if length == u64::from(u32::MAX) {
|
||||
length = u64::from_le_bytes(self.bytes(8, boundary)?.try_into().unwrap());
|
||||
}
|
||||
Ok(Header {
|
||||
tag,
|
||||
length,
|
||||
data: self.source.stream_position()?,
|
||||
})
|
||||
}
|
||||
fn index(&mut self) -> Result<Index> {
|
||||
let root = self.header(self.size)?;
|
||||
ensure!(root.tag == *b"SPKS", "not a raw SPKS package");
|
||||
let count = u32::from_le_bytes(self.bytes(4, self.size)?.try_into().unwrap());
|
||||
ensure!(
|
||||
count > 0 && count <= 256,
|
||||
"SPK package count exceeds bounds"
|
||||
);
|
||||
// Observed SPKS length excludes its count word and trailing SEND.
|
||||
let children_end = root.end()?.checked_add(4).context("SPKS size overflow")?;
|
||||
ensure!(children_end <= self.size, "SPKS children outside source");
|
||||
let mut packages = Vec::new();
|
||||
let mut total_files = 0usize;
|
||||
let mut path_bytes = 0usize;
|
||||
let mut output_bytes = 0u64;
|
||||
let mut output_paths = BTreeSet::new();
|
||||
let mut package_names = BTreeSet::new();
|
||||
for _ in 0..count {
|
||||
let container = self.header(children_end)?;
|
||||
ensure!(container.tag == *b"SPK0", "expected SPK0");
|
||||
let package_end = container.end()?;
|
||||
ensure!(package_end <= children_end, "SPK0 exceeds SPKS bounds");
|
||||
let index = self.header(package_end)?;
|
||||
ensure!(index.tag == *b"SIDX", "expected SIDX");
|
||||
let index_end = index.end()?;
|
||||
ensure!(index_end <= package_end, "SIDX exceeds SPK0 bounds");
|
||||
let metadata = self.bytes(48, index_end)?;
|
||||
let name_end = metadata[..29]
|
||||
.iter()
|
||||
.position(|b| *b == 0)
|
||||
.context("unterminated package name")?;
|
||||
let name = std::str::from_utf8(&metadata[..name_end])?.to_owned();
|
||||
safe_path(&name)?;
|
||||
ensure!(
|
||||
!name.contains('/') && package_names.insert(name.clone()),
|
||||
"duplicate or invalid package directory"
|
||||
);
|
||||
let package_type = metadata[35];
|
||||
ensure!(
|
||||
(1..=4).contains(&package_type),
|
||||
"unsupported numeric SPK package type {package_type}"
|
||||
);
|
||||
let mut strings_header = self.header(index_end)?;
|
||||
let sz64 = if strings_header.tag == *b"SZ64" {
|
||||
ensure!(strings_header.length == 8, "unsupported SZ64 size");
|
||||
let bytes = self.bytes(8, index_end)?;
|
||||
strings_header = self.header(index_end)?;
|
||||
Some(bytes)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
ensure!(
|
||||
strings_header.tag == *b"STRS" && strings_header.length <= MAX_STRINGS,
|
||||
"invalid or oversized STRS"
|
||||
);
|
||||
let strings = self.bytes(strings_header.length as usize, index_end)?;
|
||||
let mut files = Vec::new();
|
||||
loop {
|
||||
let header = self.header(index_end)?;
|
||||
if header.tag == *b"FEND" {
|
||||
ensure!(header.length == 0, "nonempty FEND");
|
||||
break;
|
||||
}
|
||||
let wide = match &header.tag {
|
||||
b"FINF" => false,
|
||||
b"FI64" => true,
|
||||
_ => bail!("unknown SPK file record"),
|
||||
};
|
||||
ensure!(
|
||||
header.length == if wide { 80 } else { 60 },
|
||||
"unexpected file record length"
|
||||
);
|
||||
total_files += 1;
|
||||
ensure!(
|
||||
total_files <= self.limits.files,
|
||||
"SPK file count exceeds bounds"
|
||||
);
|
||||
let bytes = self.bytes(header.length as usize, index_end)?;
|
||||
let width = if wide { 8 } else { 4 };
|
||||
let value = |n: usize| -> u64 {
|
||||
let start = n * width;
|
||||
if wide {
|
||||
u64::from_le_bytes(bytes[start..start + 8].try_into().unwrap())
|
||||
} else {
|
||||
u64::from(u32::from_le_bytes(
|
||||
bytes[start..start + 4].try_into().unwrap(),
|
||||
))
|
||||
}
|
||||
};
|
||||
let string_offset = usize::try_from(value(0)).context("STRS offset overflow")?;
|
||||
let tail = strings
|
||||
.get(string_offset..)
|
||||
.context("filename outside STRS")?;
|
||||
let end = tail
|
||||
.iter()
|
||||
.take(4097)
|
||||
.position(|b| *b == 0)
|
||||
.context("unterminated or oversized filename")?;
|
||||
path_bytes = path_bytes
|
||||
.checked_add(end + name.len() + 1)
|
||||
.context("SPK path accounting overflow")?;
|
||||
ensure!(
|
||||
path_bytes <= 16 << 20,
|
||||
"SPK cumulative path bytes exceed bounds"
|
||||
);
|
||||
let file_name = std::str::from_utf8(&tail[..end])?.to_owned();
|
||||
safe_path(&file_name)?;
|
||||
let output_path = format!("{name}/{file_name}");
|
||||
ensure!(
|
||||
output_paths.insert(output_path),
|
||||
"duplicate SPK output path"
|
||||
);
|
||||
output_bytes = output_bytes
|
||||
.checked_add(value(3))
|
||||
.context("SPK output size overflow")?;
|
||||
ensure!(
|
||||
output_bytes <= self.limits.output_bytes,
|
||||
"SPK payload exceeds output budget"
|
||||
);
|
||||
let mode_at = width * 4;
|
||||
files.push(FileInfo {
|
||||
name: file_name,
|
||||
size: value(1),
|
||||
offset: value(2),
|
||||
stored_size: value(3),
|
||||
mode: u16::from_le_bytes(bytes[mode_at..mode_at + 2].try_into().unwrap()),
|
||||
hmac_sha1: bytes[mode_at + 5..mode_at + 25].try_into().unwrap(),
|
||||
md5: bytes[mode_at + 25..mode_at + 41].try_into().unwrap(),
|
||||
record: if wide { "FI64" } else { "FINF" }.into(),
|
||||
});
|
||||
}
|
||||
ensure!(
|
||||
self.source.stream_position()? == index_end,
|
||||
"SIDX length disagrees with records"
|
||||
);
|
||||
let data = self.header(package_end)?;
|
||||
ensure!(data.tag == *b"SDAT", "expected SDAT");
|
||||
// Real SPKS files commonly declare SDAT size zero; SPK0 supplies its bounds.
|
||||
ensure!(
|
||||
data.length == 0 || data.end()? == package_end,
|
||||
"SDAT length disagrees with enclosing SPK0"
|
||||
);
|
||||
for file in &mut files {
|
||||
file.offset = file
|
||||
.offset
|
||||
.checked_add(data.data)
|
||||
.context("SDAT offset overflow")?;
|
||||
ensure!(
|
||||
file.offset <= package_end && file.stored_size <= package_end - file.offset,
|
||||
"file payload outside SPK0 data"
|
||||
);
|
||||
}
|
||||
packages.push(Package {
|
||||
name,
|
||||
package_id: metadata[29..32].try_into().unwrap(),
|
||||
version: metadata[32..35].try_into().unwrap(),
|
||||
package_type,
|
||||
unknown_metadata: metadata[36..48].try_into().unwrap(),
|
||||
sz64,
|
||||
declared_sdat_bytes: data.length,
|
||||
files,
|
||||
});
|
||||
self.source.seek(SeekFrom::Start(package_end))?;
|
||||
}
|
||||
ensure!(
|
||||
self.source.stream_position()? == children_end,
|
||||
"SPKS length disagrees with children"
|
||||
);
|
||||
let trailer = self.header(self.size)?;
|
||||
ensure!(
|
||||
trailer.tag == *b"SEND" && trailer.length == 4 && trailer.end()? == self.size,
|
||||
"unsupported SPK trailer or trailing bytes"
|
||||
);
|
||||
let trailer = self.bytes(4, self.size)?.try_into().unwrap();
|
||||
Ok(Index {
|
||||
source_bytes: self.size,
|
||||
output_bytes,
|
||||
metadata_bytes: self.metadata,
|
||||
packages,
|
||||
trailer,
|
||||
})
|
||||
}
|
||||
}
|
||||
impl<R: Read + Seek> Spk<R> {
|
||||
pub fn parse(mut source: R, limits: Limits, cancel: &AtomicBool) -> Result<Self> {
|
||||
ensure!(
|
||||
limits.files <= 200_000 && limits.metadata_bytes <= MAX_METADATA,
|
||||
"SPK configured limits exceed hard caps"
|
||||
);
|
||||
let size = source.seek(SeekFrom::End(0))?;
|
||||
source.seek(SeekFrom::Start(0))?;
|
||||
let index = Parser {
|
||||
source: &mut source,
|
||||
size,
|
||||
metadata: 0,
|
||||
limits: &limits,
|
||||
cancel,
|
||||
}
|
||||
.index()?;
|
||||
Ok(Self { source, index })
|
||||
}
|
||||
pub fn into_index(self) -> Index {
|
||||
self.index
|
||||
}
|
||||
pub fn index(&self) -> &Index {
|
||||
&self.index
|
||||
}
|
||||
|
||||
/// Open one regular SPK member as a bounded `Read + Seek` view. The
|
||||
/// member metadata is cloned, while payload bytes remain in the retained
|
||||
/// source. Callers must read from the beginning through EOF to obtain
|
||||
/// checksum verification; partial/random views are explicitly unverified.
|
||||
pub fn member_reader(&mut self, package: usize, file: usize) -> Result<MemberReader<'_, R>> {
|
||||
let file = self
|
||||
.index
|
||||
.packages
|
||||
.get(package)
|
||||
.and_then(|p| p.files.get(file))
|
||||
.context("unknown SPK member")?
|
||||
.clone();
|
||||
self.source.seek(SeekFrom::Start(file.offset))?;
|
||||
Ok(MemberReader {
|
||||
source: &mut self.source,
|
||||
file,
|
||||
position: 0,
|
||||
md5: Md5::new(),
|
||||
hmac: Hmac::<Sha1>::new_from_slice(HMAC_KEY).expect("HMAC accepts any key size"),
|
||||
verify: true,
|
||||
})
|
||||
}
|
||||
/// The writer receives bytes before final digest verification. Callers must use
|
||||
/// an unpublished/rollback-capable sink. `extract` provides that contract.
|
||||
pub fn copy_verified<W: Write>(
|
||||
&mut self,
|
||||
package: usize,
|
||||
file: usize,
|
||||
output: &mut W,
|
||||
cancel: &AtomicBool,
|
||||
) -> Result<u64> {
|
||||
let file = self
|
||||
.index
|
||||
.packages
|
||||
.get(package)
|
||||
.and_then(|p| p.files.get(file))
|
||||
.context("unknown SPK member")?;
|
||||
stream(&mut self.source, file, output, cancel)
|
||||
}
|
||||
pub fn verify(&mut self, cancel: &AtomicBool) -> Result<Verification> {
|
||||
let mut result = Verification { files: 0, bytes: 0 };
|
||||
for package in &self.index.packages {
|
||||
for file in &package.files {
|
||||
result.bytes += stream(&mut self.source, file, &mut std::io::sink(), cancel)?;
|
||||
result.files += 1;
|
||||
}
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
/// Creates an exclusive destination and removes it on any failure or unwind.
|
||||
/// Checksums cover every member, including members not emitted to the host.
|
||||
pub fn extract(&mut self, destination: &Path, cancel: &AtomicBool) -> Result<Extraction> {
|
||||
cancelled(cancel)?;
|
||||
fs::create_dir(destination).context("create exclusive SPK output directory")?;
|
||||
let mut guard = OutputGuard {
|
||||
path: destination,
|
||||
committed: false,
|
||||
};
|
||||
let mut result = Extraction {
|
||||
verified: Verification { files: 0, bytes: 0 },
|
||||
regular_files: 0,
|
||||
skipped: Vec::new(),
|
||||
};
|
||||
for package in &self.index.packages {
|
||||
for file in &package.files {
|
||||
cancelled(cancel)?;
|
||||
let relative = format!("{}/{}", package.name, file.name);
|
||||
let kind = file.mode & 0o170000;
|
||||
// Some producers store permission bits only; both encodings mean regular file.
|
||||
if kind == 0 || kind == 0o100000 {
|
||||
let target = destination.join(&relative);
|
||||
fs::create_dir_all(target.parent().context("missing SPK parent")?)?;
|
||||
let mut output = fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.create_new(true)
|
||||
.open(&target)?;
|
||||
result.verified.bytes += stream(&mut self.source, file, &mut output, cancel)?;
|
||||
output.set_permissions(fs::Permissions::from_mode(u32::from(
|
||||
file.mode & 0o777,
|
||||
)))?;
|
||||
result.regular_files += 1;
|
||||
} else {
|
||||
result.verified.bytes +=
|
||||
stream(&mut self.source, file, &mut std::io::sink(), cancel)?;
|
||||
result.skipped.push(relative);
|
||||
}
|
||||
result.verified.files += 1;
|
||||
}
|
||||
}
|
||||
cancelled(cancel)?;
|
||||
guard.committed = true;
|
||||
Ok(result)
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: Read + Seek> Read for MemberReader<'_, R> {
|
||||
fn read(&mut self, output: &mut [u8]) -> std::io::Result<usize> {
|
||||
if output.is_empty() || self.position == self.file.stored_size {
|
||||
if self.position == self.file.stored_size && self.verify {
|
||||
if self.md5.clone().finalize().as_slice() != self.file.md5 {
|
||||
return Err(std::io::Error::other(format!(
|
||||
"SPK MD5 mismatch: {}",
|
||||
self.file.name
|
||||
)));
|
||||
}
|
||||
self.hmac
|
||||
.clone()
|
||||
.verify_slice(&self.file.hmac_sha1)
|
||||
.map_err(std::io::Error::other)?;
|
||||
self.verify = false;
|
||||
}
|
||||
return Ok(0);
|
||||
}
|
||||
let count = output
|
||||
.len()
|
||||
.min((self.file.stored_size - self.position) as usize);
|
||||
self.source.read_exact(&mut output[..count])?;
|
||||
self.position += count as u64;
|
||||
if self.verify {
|
||||
self.md5.update(&output[..count]);
|
||||
self.hmac.update(&output[..count]);
|
||||
}
|
||||
Ok(count)
|
||||
}
|
||||
}
|
||||
|
||||
impl<R: Read + Seek> Seek for MemberReader<'_, R> {
|
||||
fn seek(&mut self, from: SeekFrom) -> std::io::Result<u64> {
|
||||
let target = match from {
|
||||
SeekFrom::Start(n) => i128::from(n),
|
||||
SeekFrom::Current(n) => i128::from(self.position) + i128::from(n),
|
||||
SeekFrom::End(n) => i128::from(self.file.stored_size) + i128::from(n),
|
||||
};
|
||||
let target =
|
||||
u64::try_from(target).map_err(|_| std::io::Error::other("invalid SPK member seek"))?;
|
||||
if target > self.file.stored_size {
|
||||
return Err(std::io::Error::other("SPK member seek outside bounds"));
|
||||
}
|
||||
self.source
|
||||
.seek(SeekFrom::Start(self.file.offset + target))?;
|
||||
self.position = target;
|
||||
self.md5 = Md5::new();
|
||||
self.hmac = Hmac::<Sha1>::new_from_slice(HMAC_KEY).expect("HMAC accepts any key size");
|
||||
self.verify = target == 0;
|
||||
Ok(target)
|
||||
}
|
||||
}
|
||||
fn stream<R: Read + Seek, W: Write>(
|
||||
source: &mut R,
|
||||
file: &FileInfo,
|
||||
output: &mut W,
|
||||
cancel: &AtomicBool,
|
||||
) -> Result<u64> {
|
||||
cancelled(cancel)?;
|
||||
source.seek(SeekFrom::Start(file.offset))?;
|
||||
let mut md5 = Md5::new();
|
||||
let mut hmac = Hmac::<Sha1>::new_from_slice(HMAC_KEY).expect("HMAC accepts any key size");
|
||||
let mut buffer = vec![0; BUFFER];
|
||||
let mut left = file.stored_size;
|
||||
while left > 0 {
|
||||
cancelled(cancel)?;
|
||||
let n = left.min(BUFFER as u64) as usize;
|
||||
source
|
||||
.read_exact(&mut buffer[..n])
|
||||
.context("truncated SPK payload")?;
|
||||
md5.update(&buffer[..n]);
|
||||
hmac.update(&buffer[..n]);
|
||||
output.write_all(&buffer[..n])?;
|
||||
left -= n as u64;
|
||||
}
|
||||
ensure!(
|
||||
md5.finalize().as_slice() == file.md5,
|
||||
"SPK MD5 mismatch: {}",
|
||||
file.name
|
||||
);
|
||||
hmac.verify_slice(&file.hmac_sha1)
|
||||
.with_context(|| format!("SPK HMAC-SHA1 mismatch: {}", file.name))?;
|
||||
cancelled(cancel)?;
|
||||
Ok(file.stored_size)
|
||||
}
|
||||
struct OutputGuard<'a> {
|
||||
path: &'a Path,
|
||||
committed: bool,
|
||||
}
|
||||
impl Drop for OutputGuard<'_> {
|
||||
fn drop(&mut self) {
|
||||
if !self.committed {
|
||||
let _ = fs::remove_dir_all(self.path);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "../tests/reader.rs"]
|
||||
mod tests;
|
||||
@@ -0,0 +1,420 @@
|
||||
use super::*;
|
||||
use std::io::Cursor;
|
||||
|
||||
fn header(tag: &[u8; 4], size: usize, wide: bool) -> Vec<u8> {
|
||||
let mut b = tag.to_vec();
|
||||
if wide {
|
||||
b.extend(u32::MAX.to_le_bytes());
|
||||
b.extend((size as u64).to_le_bytes());
|
||||
} else {
|
||||
b.extend((size as u32).to_le_bytes());
|
||||
}
|
||||
b
|
||||
}
|
||||
fn fixture(wide: bool, name: &str, mode: u16, payload: &[u8]) -> Vec<u8> {
|
||||
let mut strings = name.as_bytes().to_vec();
|
||||
strings.push(0);
|
||||
let mut metadata = vec![0; 48];
|
||||
metadata[..4].copy_from_slice(b"game");
|
||||
metadata[32..36].copy_from_slice(&[1, 2, 3, 4]);
|
||||
if wide {
|
||||
metadata.extend(header(b"SZ64", 8, false));
|
||||
metadata.extend([1, 2, 3, 4, 5, 6, 7, 8]);
|
||||
}
|
||||
metadata.extend(header(b"STRS", strings.len(), false));
|
||||
metadata.extend(strings);
|
||||
metadata.extend(header(
|
||||
if wide { b"FI64" } else { b"FINF" },
|
||||
if wide { 80 } else { 60 },
|
||||
false,
|
||||
));
|
||||
for value in [0, payload.len() as u64, 0, payload.len() as u64] {
|
||||
if wide {
|
||||
metadata.extend(value.to_le_bytes());
|
||||
} else {
|
||||
metadata.extend((value as u32).to_le_bytes());
|
||||
}
|
||||
}
|
||||
metadata.extend(mode.to_le_bytes());
|
||||
metadata.extend([0; 3]);
|
||||
let mut hmac = Hmac::<Sha1>::new_from_slice(HMAC_KEY).unwrap();
|
||||
hmac.update(payload);
|
||||
metadata.extend(hmac.finalize().into_bytes());
|
||||
metadata.extend(Md5::digest(payload));
|
||||
metadata.extend(vec![0; if wide { 7 } else { 3 }]);
|
||||
metadata.extend(header(b"FEND", 0, false));
|
||||
let mut package = header(b"SIDX", metadata.len(), wide);
|
||||
package.extend(metadata);
|
||||
package.extend(header(b"SDAT", 0, wide));
|
||||
package.extend(payload);
|
||||
let mut container = header(b"SPK0", package.len(), wide);
|
||||
container.extend(package);
|
||||
let mut all = header(b"SPKS", container.len(), wide);
|
||||
all.extend(1u32.to_le_bytes());
|
||||
all.extend(container);
|
||||
all.extend(header(b"SEND", 4, false));
|
||||
all.extend([0; 4]);
|
||||
all
|
||||
}
|
||||
fn parse(bytes: Vec<u8>) -> Result<Spk<Cursor<Vec<u8>>>> {
|
||||
Spk::parse(
|
||||
Cursor::new(bytes),
|
||||
Limits::new(8 << 20),
|
||||
&AtomicBool::new(false),
|
||||
)
|
||||
}
|
||||
#[test]
|
||||
fn old_and_wide_headers_records_stream_identical_payloads() {
|
||||
let data: Vec<_> = (0..3_000_019).map(|i| (i % 251) as u8).collect();
|
||||
for wide in [false, true] {
|
||||
let mut spk = parse(fixture(wide, "assets/game.bin", 0o106755, &data)).unwrap();
|
||||
assert_eq!(spk.index.packages[0].package_type, 4);
|
||||
assert_eq!(
|
||||
spk.index.packages[0].files[0].record,
|
||||
if wide { "FI64" } else { "FINF" }
|
||||
);
|
||||
assert!(spk.index.metadata_bytes < 1024);
|
||||
let mut output = Vec::new();
|
||||
spk.copy_verified(0, 0, &mut output, &AtomicBool::new(false))
|
||||
.unwrap();
|
||||
assert_eq!(output, data);
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let destination = temp.path().join("out");
|
||||
let report = spk.extract(&destination, &AtomicBool::new(false)).unwrap();
|
||||
assert_eq!(report.regular_files, 1);
|
||||
assert_eq!(
|
||||
fs::metadata(destination.join("game/assets/game.bin"))
|
||||
.unwrap()
|
||||
.permissions()
|
||||
.mode()
|
||||
& 0o7777,
|
||||
0o755
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(destination.join("game/assets/game.bin")).unwrap(),
|
||||
data
|
||||
);
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn invalid_paths_lengths_checksums_and_output_rollback() {
|
||||
for name in ["../escape", "/absolute", "a//b", "a/../b", "a\\b", "a\nb"] {
|
||||
assert!(parse(fixture(false, name, 0o100644, b"payload")).is_err());
|
||||
}
|
||||
let original = fixture(false, "file", 0o100644, b"payload");
|
||||
for offset in [4, 16, 24] {
|
||||
let mut bytes = original.clone();
|
||||
bytes[offset..offset + 4].copy_from_slice(&0xfffffff0u32.to_le_bytes());
|
||||
assert!(parse(bytes).is_err());
|
||||
}
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let destination = temp.path().join("out");
|
||||
let mut bytes = original.clone();
|
||||
let at = bytes.windows(7).position(|b| b == b"payload").unwrap();
|
||||
bytes[at] ^= 1;
|
||||
let mut spk = parse(bytes).unwrap();
|
||||
assert!(
|
||||
spk.extract(&destination, &AtomicBool::new(false))
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("MD5 mismatch")
|
||||
);
|
||||
assert!(!destination.exists());
|
||||
let mut spk = parse(original).unwrap();
|
||||
fs::create_dir(&destination).unwrap();
|
||||
fs::write(destination.join("keep"), b"safe").unwrap();
|
||||
assert!(spk.extract(&destination, &AtomicBool::new(false)).is_err());
|
||||
assert_eq!(fs::read(destination.join("keep")).unwrap(), b"safe");
|
||||
}
|
||||
#[test]
|
||||
fn hmac_is_checked_independently_and_declared_size_is_retained() {
|
||||
let mut bytes = fixture(false, "file", 0o100644, b"data");
|
||||
let record = bytes.windows(4).position(|b| b == b"FINF").unwrap();
|
||||
bytes[record + 8 + 4..record + 8 + 8].copy_from_slice(&999u32.to_le_bytes());
|
||||
let mut spk = parse(bytes.clone()).unwrap();
|
||||
assert_eq!(spk.index.packages[0].files[0].size, 999);
|
||||
assert_eq!(spk.verify(&AtomicBool::new(false)).unwrap().bytes, 4);
|
||||
bytes[record + 8 + 21] ^= 1;
|
||||
let mut spk = parse(bytes).unwrap();
|
||||
assert!(
|
||||
format!("{:#}", spk.verify(&AtomicBool::new(false)).unwrap_err())
|
||||
.contains("HMAC-SHA1 mismatch")
|
||||
);
|
||||
}
|
||||
#[test]
|
||||
fn budgets_cancellation_and_special_files() {
|
||||
let bytes = fixture(false, "link", 0o120777, b"/etc/passwd");
|
||||
assert!(
|
||||
Spk::parse(
|
||||
Cursor::new(bytes.clone()),
|
||||
Limits::new(1),
|
||||
&AtomicBool::new(false)
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
Spk::parse(
|
||||
Cursor::new(bytes.clone()),
|
||||
Limits::new(100),
|
||||
&AtomicBool::new(true)
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
let mut spk = parse(bytes).unwrap();
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let destination = temp.path().join("out");
|
||||
let report = spk.extract(&destination, &AtomicBool::new(false)).unwrap();
|
||||
assert_eq!(report.regular_files, 0);
|
||||
assert_eq!(report.skipped, ["game/link"]);
|
||||
assert!(!destination.join("game/link").exists());
|
||||
assert!(spk.verify(&AtomicBool::new(true)).is_err());
|
||||
}
|
||||
#[test]
|
||||
fn no_payload_read_exceeds_buffer_and_failure_or_panic_removes_output() {
|
||||
use std::sync::Arc;
|
||||
struct Reader {
|
||||
inner: Cursor<Vec<u8>>,
|
||||
max: usize,
|
||||
fail: bool,
|
||||
panic: bool,
|
||||
cancel: Option<Arc<AtomicBool>>,
|
||||
armed: bool,
|
||||
}
|
||||
impl Read for Reader {
|
||||
fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
|
||||
assert!(out.len() <= self.max);
|
||||
if self.armed {
|
||||
if self.panic {
|
||||
panic!("fixture read panic")
|
||||
};
|
||||
if self.fail {
|
||||
return Err(std::io::Error::other("fixture failure"));
|
||||
}
|
||||
if let Some(flag) = &self.cancel {
|
||||
flag.store(true, Ordering::Relaxed)
|
||||
}
|
||||
}
|
||||
self.inner.read(out)
|
||||
}
|
||||
}
|
||||
impl Seek for Reader {
|
||||
fn seek(&mut self, p: SeekFrom) -> std::io::Result<u64> {
|
||||
self.inner.seek(p)
|
||||
}
|
||||
}
|
||||
let bytes = fixture(false, "large", 0o100644, &vec![0x55; 3 << 20]);
|
||||
for action in 0..4 {
|
||||
let cancel = Arc::new(AtomicBool::new(false));
|
||||
let reader = Reader {
|
||||
inner: Cursor::new(bytes.clone()),
|
||||
max: BUFFER,
|
||||
fail: action == 1,
|
||||
panic: action == 2,
|
||||
cancel: (action == 3).then(|| cancel.clone()),
|
||||
armed: false,
|
||||
};
|
||||
let mut spk = Spk::parse(reader, Limits::new(8 << 20), &cancel).unwrap();
|
||||
spk.source.armed = true;
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let out = temp.path().join("out");
|
||||
let result =
|
||||
std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| spk.extract(&out, &cancel)));
|
||||
if action == 0 {
|
||||
assert!(result.unwrap().is_ok());
|
||||
} else {
|
||||
assert!(result.is_err() || result.unwrap().is_err());
|
||||
assert!(!out.exists());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "requires local proprietary derived system fixtures and pinned upstream oracle"]
|
||||
fn real_system_packages_match_upstream_verification_and_extracted_bytes() {
|
||||
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||
let fixtures = root.join("data/validation/native-spk");
|
||||
let oracle = root.join("tools/bin/spike-spk-type4");
|
||||
for (name, kind) in [("got-system", 1), ("pokemon086-system", 4)] {
|
||||
let input = fixtures.join(format!("{name}.spk"));
|
||||
let cancel = AtomicBool::new(false);
|
||||
let mut spk = Spk::parse(
|
||||
fs::File::open(&input).unwrap(),
|
||||
Limits::new(64 << 20),
|
||||
&cancel,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(spk.index.packages.len(), 1);
|
||||
assert_eq!(spk.index.packages[0].package_type, kind);
|
||||
let verified = spk.verify(&cancel).unwrap();
|
||||
assert_eq!(verified.files, 18);
|
||||
let reference = std::process::Command::new(&oracle)
|
||||
.arg("verify")
|
||||
.arg(&input)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(reference.status.success());
|
||||
let text = String::from_utf8(reference.stdout).unwrap();
|
||||
assert!(!text.contains('✗'));
|
||||
assert_eq!(text.matches('✔').count(), 36);
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let native = temp.path().join("native");
|
||||
spk.extract(&native, &cancel).unwrap();
|
||||
let reference = temp.path().join("reference");
|
||||
fs::create_dir(&reference).unwrap();
|
||||
let output = std::process::Command::new(&oracle)
|
||||
.arg("extract")
|
||||
.arg(&input)
|
||||
.arg("--output")
|
||||
.arg(&reference)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
fn digest(path: &Path) -> [u8; 16] {
|
||||
let mut reader = fs::File::open(path).unwrap();
|
||||
let mut hash = Md5::new();
|
||||
let mut buffer = vec![0; BUFFER];
|
||||
loop {
|
||||
let n = reader.read(&mut buffer).unwrap();
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
hash.update(&buffer[..n]);
|
||||
}
|
||||
hash.finalize().into()
|
||||
}
|
||||
for package in &spk.index.packages {
|
||||
for file in &package.files {
|
||||
let relative = format!("{}/{}", package.name, file.name);
|
||||
assert_eq!(
|
||||
digest(&native.join(&relative)),
|
||||
digest(&reference.join(name).join(&relative)),
|
||||
"{relative}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "requires the pinned local upstream oracle"]
|
||||
fn synthetic_fi64_and_extended_headers_match_upstream_oracle() {
|
||||
let root = Path::new(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let input = temp.path().join("wide.spk");
|
||||
let data = vec![0x52; 200_003];
|
||||
fs::write(&input, fixture(true, "dir/data", 0o100755, &data)).unwrap();
|
||||
let result = std::process::Command::new(root.join("tools/bin/spike-spk-type4"))
|
||||
.arg("verify")
|
||||
.arg(&input)
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
result.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&result.stderr)
|
||||
);
|
||||
let text = String::from_utf8(result.stdout).unwrap();
|
||||
assert!(!text.contains('✗'));
|
||||
assert_eq!(text.matches('✔').count(), 2);
|
||||
let mut native = Spk::parse(
|
||||
fs::File::open(input).unwrap(),
|
||||
Limits::new(1 << 20),
|
||||
&AtomicBool::new(false),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
native.verify(&AtomicBool::new(false)).unwrap().bytes,
|
||||
data.len() as u64
|
||||
);
|
||||
assert_eq!(
|
||||
native.index.packages[0].sz64.as_deref(),
|
||||
Some([1, 2, 3, 4, 5, 6, 7, 8].as_slice())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn metadata_file_count_and_member_ranges_are_bounded() {
|
||||
let original = fixture(false, "file", 0o100644, b"data");
|
||||
let mut limits = Limits::new(10);
|
||||
limits.metadata_bytes = 20;
|
||||
assert!(
|
||||
Spk::parse(
|
||||
Cursor::new(original.clone()),
|
||||
limits,
|
||||
&AtomicBool::new(false)
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
let mut limits = Limits::new(10);
|
||||
limits.files = 0;
|
||||
assert!(
|
||||
Spk::parse(
|
||||
Cursor::new(original.clone()),
|
||||
limits,
|
||||
&AtomicBool::new(false)
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
let mut outside = original.clone();
|
||||
let record = outside.windows(4).position(|b| b == b"FINF").unwrap();
|
||||
outside[record + 16..record + 20].copy_from_slice(&u32::MAX.to_le_bytes());
|
||||
assert!(
|
||||
parse(outside)
|
||||
.err()
|
||||
.unwrap()
|
||||
.to_string()
|
||||
.contains("outside SPK0")
|
||||
);
|
||||
let mut trailing = original;
|
||||
trailing.push(0);
|
||||
assert!(parse(trailing).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn shared_string_offsets_cannot_amplify_names_beyond_path_budget() {
|
||||
let empty_hmac = {
|
||||
let h = Hmac::<Sha1>::new_from_slice(HMAC_KEY).unwrap();
|
||||
h.finalize().into_bytes()
|
||||
};
|
||||
let mut containers = Vec::new();
|
||||
for package in *b"12" {
|
||||
let mut metadata = vec![0; 48];
|
||||
metadata[..2].copy_from_slice(&[b'p', package]);
|
||||
metadata[35] = 2;
|
||||
metadata.extend(header(b"STRS", 4097, false));
|
||||
metadata.extend(vec![b'a'; 4096]);
|
||||
metadata.push(0);
|
||||
for offset in 0u32..4096 {
|
||||
metadata.extend(header(b"FINF", 60, false));
|
||||
metadata.extend(offset.to_le_bytes());
|
||||
metadata.extend([0; 12]);
|
||||
metadata.extend(0o100644u16.to_le_bytes());
|
||||
metadata.extend([0; 3]);
|
||||
metadata.extend(empty_hmac);
|
||||
metadata.extend(Md5::digest([]));
|
||||
metadata.extend([0; 3]);
|
||||
}
|
||||
metadata.extend(header(b"FEND", 0, false));
|
||||
let mut contents = header(b"SIDX", metadata.len(), false);
|
||||
contents.extend(metadata);
|
||||
contents.extend(header(b"SDAT", 0, false));
|
||||
containers.extend(header(b"SPK0", contents.len(), false));
|
||||
containers.extend(contents);
|
||||
}
|
||||
let mut bytes = header(b"SPKS", containers.len(), false);
|
||||
bytes.extend(2u32.to_le_bytes());
|
||||
bytes.extend(containers);
|
||||
bytes.extend(header(b"SEND", 4, false));
|
||||
bytes.extend([0; 4]);
|
||||
assert!(
|
||||
parse(bytes)
|
||||
.err()
|
||||
.unwrap()
|
||||
.to_string()
|
||||
.contains("cumulative path bytes")
|
||||
);
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
[Unit]
|
||||
Description=Stern ROM archive and processing service
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
[Unit]
|
||||
Description=Verstack workbench LAN gateway (password gate in front of Theia)
|
||||
After=network.target
|
||||
Requires=verstack-workbench.service
|
||||
After=verstack-workbench.service
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/verstack
|
||||
# Theia's JSON-RPC websocket carries @theia/filesystem and @theia/process. It bypasses Express
|
||||
# middleware, so the gate must sit in front of the whole app, not inside it.
|
||||
ExecStart=/usr/bin/node %h/verstack/workbench/auth-gateway.mjs \
|
||||
--listen 0.0.0.0:3000 \
|
||||
--upstream 127.0.0.1:3001 \
|
||||
--secrets %h/verstack/data/deployment/workbench-auth.json
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
TimeoutStopSec=15
|
||||
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
@@ -1,11 +1,13 @@
|
||||
[Unit]
|
||||
Description=Stern ROM game workbench
|
||||
Description=Stern ROM game workbench (loopback only; reached through verstack-gateway)
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=%h/verstack/workbench/browser-app
|
||||
Environment=VERSTACK_API_PORT=8080
|
||||
ExecStart=/usr/bin/node %h/verstack/workbench/browser-app/lib/backend/main.js --hostname 0.0.0.0 --port 3000
|
||||
ExecStart=/usr/bin/node %h/verstack/workbench/browser-app/lib/backend/main.js --hostname 127.0.0.1 --port 3001
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
TimeoutStopSec=30
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# AES-XTS context reuse (2026-09-16)
|
||||
|
||||
`plugins/luks_extract.py` now allocates one OpenSSL EVP context per input chunk,
|
||||
loads the AES key once, and resets only the tweak for each sector. The public
|
||||
`cryptography` context does not expose XTS tweak reset. The implementation uses
|
||||
libcrypto's public EVP API through ctypes, as the adapter already does for
|
||||
libcryptsetup. Each update supplies exactly one complete XTS data unit. See the
|
||||
[OpenSSL EVP documentation](https://docs.openssl.org/3.0/man3/EVP_EncryptInit/).
|
||||
|
||||
The plain64 tweak still counts 512-byte sectors for both supported data-unit
|
||||
sizes. Invalid keys, sector alignment and 64-bit sector-number overflow fail
|
||||
before decryption. The EVP context is freed and the temporary key buffer cleared
|
||||
on success and failure. Worker scheduling and bounded 4 MiB chunks are unchanged.
|
||||
|
||||
Verification:
|
||||
|
||||
```
|
||||
tools/decoder-env/bin/python -m unittest discover -s tests -p test_luks.py
|
||||
# 9 tests passed
|
||||
|
||||
tools/decoder-env/bin/python scripts/benchmark_luks_xts.py
|
||||
# 512-byte sectors, fresh context: 16.9 MB/s
|
||||
# 512-byte sectors, reused context: 49.6 MB/s
|
||||
# Speedup: 2.93x
|
||||
# 4096-byte sectors, fresh context: 117.4 MB/s
|
||||
# 4096-byte sectors, reused context: 322.6 MB/s
|
||||
# Speedup: 2.75x
|
||||
```
|
||||
|
||||
The benchmark uses synthetic ciphertext, five repetitions of 16 MiB per mode,
|
||||
reports median elapsed time and checks every output against the original
|
||||
per-sector implementation. Tests independently encrypt using cryptography,
|
||||
cover both 256- and 512-bit XTS keys, both sector sizes, nonzero chunk positions,
|
||||
a tweak crossing 2^32, malformed inputs, and serial/two-worker decryption across
|
||||
three chunks including a short final chunk. Existing real LUKS-header tests pass.
|
||||
|
||||
This is a measured approximately 3x improvement, **not** the plan's proposed 82x
|
||||
win. Reusing the context removes setup cost, but Python/ctypes calls per sector
|
||||
remain; reaching the quoted 1,379 MB/s native ceiling requires moving the entire
|
||||
sector loop into compiled code. These checks do not establish end-to-end import
|
||||
throughput. No live config or frozen tool bundle was updated by this change;
|
||||
pipeline reconfiguration is necessary for deployed imports to use it.
|
||||
@@ -0,0 +1,387 @@
|
||||
{
|
||||
"method": "verstack-code-anchors/1",
|
||||
"saved_row_samples": {
|
||||
"unfiltered": {
|
||||
"method": "verstack-code-anchors/1",
|
||||
"programs": [
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 0,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 200,
|
||||
"snapshot": "09c07c28-3a04-4438-9a06-ef7f0370c49e",
|
||||
"source_path": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 0
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"entrypoint_extent_unknown": 1,
|
||||
"saved_contiguous_extent_unproven": 200,
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 632
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:18f8f4e198b880330fdb1817397294f06a403a66f5edde1e163cb087b0c895b6",
|
||||
"elf": "data/validation/anchor-programs/deadpool.elf",
|
||||
"facts": "data/validation/anchor-programs/deadpool.facts.json",
|
||||
"facts_path": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19/functions.json",
|
||||
"facts_snapshot": "09c07c28-3a04-4438-9a06-ef7f0370c49e",
|
||||
"input_sha256": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19",
|
||||
"label": "deadpool",
|
||||
"original_snapshot": "21a5f51e-afe0-47f7-ab53-3fd3551a9474",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 saved rows in address order, size1..65536; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_verified_contiguous_extent": 200
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 0,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 200,
|
||||
"snapshot": "07ae6fcc-7f72-47d8-b0ba-ae55f24ac776",
|
||||
"source_path": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 0
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"entrypoint_extent_unknown": 1,
|
||||
"saved_contiguous_extent_unproven": 200,
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 633
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:2ca2e5d5302748599c18b4850996a2207631ea4b5be53bfab50280be8db4d31c",
|
||||
"elf": "data/validation/anchor-programs/godzilla.elf",
|
||||
"facts": "data/validation/anchor-programs/godzilla.facts.json",
|
||||
"facts_path": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2/functions.json",
|
||||
"facts_snapshot": "07ae6fcc-7f72-47d8-b0ba-ae55f24ac776",
|
||||
"input_sha256": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2",
|
||||
"label": "godzilla",
|
||||
"original_snapshot": "7f60ebe7-ec34-4ac9-9ba7-f7ae561c909f",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 saved rows in address order, size1..65536; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_verified_contiguous_extent": 200
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 0,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 200,
|
||||
"snapshot": "4bc5dd42-959b-47b8-aa88-2d1facfcf382",
|
||||
"source_path": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 0
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"entrypoint_extent_unknown": 1,
|
||||
"saved_contiguous_extent_unproven": 200,
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 629
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:a55a2cdb789097ff0e502f5498d9decdfe9ab998ae91067bd3bc554626cfa0d8",
|
||||
"elf": "data/validation/anchor-programs/starwars.elf",
|
||||
"facts": "data/validation/anchor-programs/starwars.facts.json",
|
||||
"facts_path": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6/functions.json",
|
||||
"facts_snapshot": "4bc5dd42-959b-47b8-aa88-2d1facfcf382",
|
||||
"input_sha256": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6",
|
||||
"label": "starwars",
|
||||
"original_snapshot": "6c0b96f3-7e21-4bee-b101-33ec572af7e5",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 saved rows in address order, size1..65536; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_verified_contiguous_extent": 200
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 0,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "AARCH64:LE:64:v8A",
|
||||
"metadata_observations": 200,
|
||||
"path": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 200,
|
||||
"snapshot": "7c474aaf-9ce0-4f3a-ab48-63d6063d9dbf",
|
||||
"source_path": "partition-02/lib/libc.so.6",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 0
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"entrypoint_extent_unknown": 1,
|
||||
"saved_body_hash_mismatch": 100,
|
||||
"saved_contiguous_extent_unproven": 100,
|
||||
"undefined_function_symbol": 14
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:844d4a5bd7dafd6a7aaaee53450cdc08c05b0bb6a61b78a464f7653f39801404",
|
||||
"elf": "data/validation/anchor-programs/libc.elf",
|
||||
"facts": "data/validation/anchor-programs/libc.facts.json",
|
||||
"facts_path": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944/functions.json",
|
||||
"facts_snapshot": "7c474aaf-9ce0-4f3a-ab48-63d6063d9dbf",
|
||||
"input_sha256": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944",
|
||||
"label": "libc",
|
||||
"original_snapshot": "830d0450-50ee-4159-a0d5-3fafb1410b74",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 saved rows in address order, size1..65536; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "partition-02/lib/libc.so.6"
|
||||
},
|
||||
"skips": {
|
||||
"no_verified_contiguous_extent": 200
|
||||
}
|
||||
}
|
||||
],
|
||||
"selection": "Bounded saved-row selection specified per program; independent original SHA/body/extent/mode verification; fresh offline catalogs."
|
||||
},
|
||||
"explicit_contiguous_method": {
|
||||
"method": "verstack-code-anchors/1",
|
||||
"programs": [
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 1,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 199,
|
||||
"snapshot": "09c07c28-3a04-4438-9a06-ef7f0370c49e",
|
||||
"source_path": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 1
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 632
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:18f8f4e198b880330fdb1817397294f06a403a66f5edde1e163cb087b0c895b6",
|
||||
"elf": "data/validation/anchor-programs/deadpool.elf",
|
||||
"facts": "data/validation/anchor-programs/deadpool.proof.facts.json",
|
||||
"facts_path": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19/functions.json",
|
||||
"facts_snapshot": "09c07c28-3a04-4438-9a06-ef7f0370c49e",
|
||||
"input_sha256": "9ef129ff1586ca962ee5fb33966deeacf32644d431a5b62d38a4c8d487160b19",
|
||||
"label": "deadpool",
|
||||
"original_snapshot": "21a5f51e-afe0-47f7-ab53-3fd3551a9474",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 explicit contiguous-method rows, size32..65536 in address order; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_independent_instruction_mode": 199
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 1,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 199,
|
||||
"snapshot": "07ae6fcc-7f72-47d8-b0ba-ae55f24ac776",
|
||||
"source_path": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 1
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"conflicting_function_boundaries": 6,
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 633
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:2ca2e5d5302748599c18b4850996a2207631ea4b5be53bfab50280be8db4d31c",
|
||||
"elf": "data/validation/anchor-programs/godzilla.elf",
|
||||
"facts": "data/validation/anchor-programs/godzilla.proof.facts.json",
|
||||
"facts_path": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2/functions.json",
|
||||
"facts_snapshot": "07ae6fcc-7f72-47d8-b0ba-ae55f24ac776",
|
||||
"input_sha256": "a5725b6711914a6795d761b1d90a7408bfa999d79e1475f438dc2309f72347e2",
|
||||
"label": "godzilla",
|
||||
"original_snapshot": "7f60ebe7-ec34-4ac9-9ba7-f7ae561c909f",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 explicit contiguous-method rows, size32..65536 in address order; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_independent_instruction_mode": 196,
|
||||
"no_verified_contiguous_extent": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 1,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "ARM:LE:32:v8",
|
||||
"metadata_observations": 200,
|
||||
"path": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 199,
|
||||
"snapshot": "4bc5dd42-959b-47b8-aa88-2d1facfcf382",
|
||||
"source_path": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro/game",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 1
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"symbol_extent_unknown": 2,
|
||||
"undefined_function_symbol": 629
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:a55a2cdb789097ff0e502f5498d9decdfe9ab998ae91067bd3bc554626cfa0d8",
|
||||
"elf": "data/validation/anchor-programs/starwars.elf",
|
||||
"facts": "data/validation/anchor-programs/starwars.proof.facts.json",
|
||||
"facts_path": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6/functions.json",
|
||||
"facts_snapshot": "4bc5dd42-959b-47b8-aa88-2d1facfcf382",
|
||||
"input_sha256": "fee7abfa8569fef905dbfba8146867b7a4efda672fca2614eb2f1ad708fbd7a6",
|
||||
"label": "starwars",
|
||||
"original_snapshot": "6c0b96f3-7e21-4bee-b101-33ec572af7e5",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 explicit contiguous-method rows, size32..65536 in address order; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro/game"
|
||||
},
|
||||
"skips": {
|
||||
"no_independent_instruction_mode": 199
|
||||
}
|
||||
},
|
||||
{
|
||||
"anchors": {},
|
||||
"coverage": {
|
||||
"anchor_method": "verstack-code-anchors/1",
|
||||
"caveat": "Denominator is saved analysis function rows, not all executable code. Counts require an exact input/architecture/address/size/body instance with independently verified native anchors. Missing observations require metadata backfill; unproven extents/modes and unsupported instructions remain unsupported. No accepted names or correspondence.",
|
||||
"complete_decode_functions": 0,
|
||||
"coverage_complete": false,
|
||||
"input_sha256": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944",
|
||||
"invalid_identity_claims": 0,
|
||||
"language": "AARCH64:LE:64:v8A",
|
||||
"metadata_observations": 200,
|
||||
"path": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944/functions.json",
|
||||
"saved_functions": 200,
|
||||
"saved_functions_without_verified_anchors": 200,
|
||||
"snapshot": "7c474aaf-9ce0-4f3a-ab48-63d6063d9dbf",
|
||||
"source_path": "partition-02/lib/libc.so.6",
|
||||
"truncated_anchor_functions": 0,
|
||||
"verified_anchor_functions": 0
|
||||
},
|
||||
"examples": [],
|
||||
"inventory_diagnostics": {
|
||||
"entrypoint_extent_unknown": 1,
|
||||
"saved_body_hash_mismatch": 200,
|
||||
"undefined_function_symbol": 14
|
||||
},
|
||||
"program": {
|
||||
"artifact": "blake3:844d4a5bd7dafd6a7aaaee53450cdc08c05b0bb6a61b78a464f7653f39801404",
|
||||
"elf": "data/validation/anchor-programs/libc.elf",
|
||||
"facts": "data/validation/anchor-programs/libc.proof.facts.json",
|
||||
"facts_path": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944/functions.json",
|
||||
"facts_snapshot": "7c474aaf-9ce0-4f3a-ab48-63d6063d9dbf",
|
||||
"input_sha256": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944",
|
||||
"label": "libc",
|
||||
"original_snapshot": "830d0450-50ee-4159-a0d5-3fafb1410b74",
|
||||
"selected_rows": 200,
|
||||
"selection": "first200 explicit contiguous-method rows, size32..65536 in address order; authoritative full_mask joined, no reconstructed method/ISA assumptions",
|
||||
"source_path": "partition-02/lib/libc.so.6"
|
||||
},
|
||||
"skips": {
|
||||
"no_verified_contiguous_extent": 200
|
||||
}
|
||||
}
|
||||
],
|
||||
"selection": "Bounded saved-row selection specified per program; independent original SHA/body/extent/mode verification; fresh offline catalogs."
|
||||
}
|
||||
},
|
||||
"independent_native_symbol_sample": {
|
||||
"anchors": {
|
||||
"import_slot": 13,
|
||||
"string": 37
|
||||
},
|
||||
"asset_names": 0,
|
||||
"failures": 0,
|
||||
"input_sha256": "055d5acbba277f86b4760cebbdde0a142797fd0dc200f591dfd4246a45a55944",
|
||||
"method": "verstack-code-anchors/1",
|
||||
"native_sized_symbols": 2249,
|
||||
"search_probe": {
|
||||
"address": 1351632,
|
||||
"bytes_sha256": "c1e7fc6c85d03d7aacbb41a2f308bec79a2ec709440645750eca44d84eb893ae",
|
||||
"instruction_offset": 296,
|
||||
"kind": "string",
|
||||
"proof": "aarch64_adrp_add_address",
|
||||
"value": "\ntransferring control: %s\n\n"
|
||||
},
|
||||
"search_results": 1,
|
||||
"selected_symbol_limit": 400,
|
||||
"truncated_functions": 0,
|
||||
"verified": 400
|
||||
},
|
||||
"coordinate_diagnostic": {
|
||||
"diagnostic_only": true,
|
||||
"tested_delta": "-0x100000",
|
||||
"verified_full_body_hashes": 200,
|
||||
"selected_rows": 200,
|
||||
"application_behavior": "no automatic address translation; missing exported coordinate provenance remains unsupported"
|
||||
},
|
||||
"limitations": "Native-symbol evidence is not silently joined to saved rows with incompatible address coordinates. No blanket ARM mode inference, name propagation or complete-binary coverage claim."
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
# Scene-backed bitmap font inspection
|
||||
|
||||
Implemented September 17, 2026; isolated verification only, not a deployment receipt.
|
||||
|
||||
`GET /api/scene/fonts?snapshot=…&path=…/scene.radium` lists serialized Font objects and their standard/custom instances. Adding `instance=<object ID>` returns every glyph's codepoint, width/height, origins, advances, padding, UV rectangle, rotation and kerning table, plus the exact Image objects referenced by those glyphs. Adding `atlas=<Image ID>` decodes that referenced BC1/BC3 atlas to PNG. An arbitrary Image ID or unreferenced FontInstance cannot be requested through this endpoint.
|
||||
|
||||
Inline atlases come from the parser's recorded byte slice. External atlases resolve only to the exact `scene.assets/<recorded filename>` sibling in the same snapshot; traversal and mismatched lengths fail. No basename search, unrelated image candidate, or system font substitution is used. Existing scene-parser limits apply; each atlas is at most 4096×4096, with 32 atlases/4096 glyphs per instance. Unsupported scene or texture formats return explicit errors.
|
||||
|
||||
The workbench's raw scene cards expose **Bitmap fonts from scene atlases**. `/review` shows baseline/target specimen panels when the selected artifact is `scene.radium`. Both use `web/review-bitmap-font.js`, with editable text, glyph crops and complete metrics, original atlas links and JSON download. Whitespace/null-image records advance without fabricated pixels. Missing codepoints are listed explicitly.
|
||||
|
||||
The renderer follows the recorded UV rectangle, scales atlas pixels to the glyph's metrics plus padding, and applies recorded rotation. Rotation is independently established by `tools/src/ske-radium/radium-player.py:668` at `f3a3dcaad59c558910b56c4cc6497899f5faba9b`: destination corners TL/TR/BR/BL sample source TR/BR/BL/TL. The font draw call passes `TextureRotated` at line 1517. This matters because scaled font instances reuse larger atlas crops; crop size must not be mistaken for layout dimensions.
|
||||
|
||||
## Evidence
|
||||
|
||||
The 42 existing, archived scene fixtures contain 22 Font objects, 92 FontInstances and 10,908 FontGlyph objects across 20 scenes. 6,492 glyphs reference BC1/BC3 images; 4,416 have no image. All image glyphs have integral, in-bounds UV crops and nonnegative dimensions/padding. 1,129 image glyphs are rotated. No fixture uses nonzero vertical advance. 321 glyphs contain kerning tables.
|
||||
|
||||
Deadpool fixture `data/validation/scene-fixtures/004.radium` contains `Stern_GovernmentAgentBB`, custom `Stern_GovtAgentBB_Outline3`, 113 glyphs and three 512×512 BC3 atlases (Image IDs 7, 47, 89). All 169 distinct referenced atlases across the 20 scenes (33 BC1, 136 BC3) match the separate Python decoder byte for byte. The corpus browser check renders all 6,492 image-backed glyphs from all 92 instances and compares every output pixel against the reference UV mapping, including all 1,129 rotated glyphs; it passes. Reports are under `data/validation/bitmap-font-corpus/`.
|
||||
|
||||
For the Deadpool example, native RGBA output matches the separate Python `plugins/vendor/bcn.py` decoder byte for byte. RGBA SHA256:
|
||||
|
||||
- 7: `7ec115adbf6067ab3d40b5e8b7b7297cb565d7b3e010c6f1392c40e31dc350a3`
|
||||
- 47: `67117043853ce182f06b8b16ede718655f31c5570014a94c4e33b8fa9e82f66b`
|
||||
- 89: `bc052e802f41ee6e96bf842bc6d56eda244d4b25efbeeff30ecc903ea52930a9`
|
||||
|
||||
`tests/bitmap_font.rs` covers BC1 transparent indices, BC3 both alpha modes/four-color rules, partial blocks, invalid dimensions/formats/lengths, typed graph references, null-image glyphs, and archived inline/external readback through Archive and HTTP, including traversal refusal. Five tests pass. Global `cargo clippy --offline --locked --all-targets -- -D warnings` and the workbench TypeScript check pass.
|
||||
|
||||
`tests/ui-bitmap-font.mjs` drives the shipped `/review` and shared viewer under CSP with the actual exported font and atlas PNGs. It checks editable visible pixels (the AB sample has 7,629 nontransparent pixels), absent-codepoint reporting, exact atlas links, recorded metrics, actual rotated glyph rendering, and the asymmetric six-pixel rotation permutation. The isolated screenshot is `data/validation/bitmap-font-004/review.png`.
|
||||
|
||||
Reproduce local export/browser checks:
|
||||
|
||||
```sh
|
||||
cargo run --offline --locked --example inspect_bitmap_font -- \
|
||||
data/validation/scene-fixtures/004.radium data/validation/bitmap-font-004
|
||||
node tests/ui-bitmap-font.mjs
|
||||
|
||||
# All available, SHA256-verified local scene fixtures:
|
||||
tools/decoder-env/bin/python scripts/verify_bitmap_font_corpus.py
|
||||
node tests/ui-bitmap-font-corpus.mjs
|
||||
```
|
||||
|
||||
## Limits still open
|
||||
|
||||
The specimen uses serialized origins and advances; it does not claim to reconstruct a game's text layout, shader/material, centering, wrapping or other scene transforms. Kerning tables are shown but their pair direction is not yet established, so the specimen explicitly uses unkerned advances. Standalone DMD glyph banks and older scene/texture formats have not been established by these scene-backed fixtures. This therefore does not mark the entire PLAN font requirement complete. Live endpoint/workbench verification still requires the coordinated deployment.
|
||||
@@ -0,0 +1,502 @@
|
||||
{
|
||||
"copy": "data/validation/identity-dryrun-20260916-1.sqlite3",
|
||||
"baseline": {
|
||||
"names": {
|
||||
"count": 8366,
|
||||
"sha256": "759aaf6d8b6ea3d411dba2345bdbea3b891f2368739d8c57b6e523d83d21edad"
|
||||
},
|
||||
"reviews": {
|
||||
"count": 4,
|
||||
"sha256": "2f9a309e7ed29e04e6c508664e81ba8c75f55f14c66956d728fb5afcc51c4202"
|
||||
},
|
||||
"signatures": {
|
||||
"count": 1170298,
|
||||
"sha256": "cd8fd67903a2111a605e2dcd31d91289f405c46afb6ec7b0a5a45df02c23a5aa"
|
||||
},
|
||||
"observations": {
|
||||
"count": 296763,
|
||||
"sha256": "4899a932a9a126dfaf1343d115a5b22373bf619ac362014bbeb4a6b284befc23"
|
||||
},
|
||||
"artifacts": {
|
||||
"count": 1521210,
|
||||
"sha256": "829e3b6f7feddba985089972cfa20e30a1d6de35a013de51990562b0ce1e9050"
|
||||
},
|
||||
"edge_occurrences": {
|
||||
"count": 0,
|
||||
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
|
||||
},
|
||||
"versions": [
|
||||
[
|
||||
"Deadpool|1.16.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Dungeons and Dragons|1.01.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Godzilla|1.16.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Jaws|1.02.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Jurassic Park (2019)|1.16.0|Pro|"
|
||||
],
|
||||
[
|
||||
"King Kong|0.97.0|LE|SPIKE 2"
|
||||
],
|
||||
[
|
||||
"King Kong|0.97.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.81.0|LE|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.81.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.82.0|LE|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.83.0|LE|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.85.0|LE|"
|
||||
],
|
||||
[
|
||||
"Pokémon|0.86.0|LE|"
|
||||
],
|
||||
[
|
||||
"Star Wars (2017)|1.31.0|Pro|"
|
||||
],
|
||||
[
|
||||
"Transformers (2026)|0.90.0|LE|"
|
||||
]
|
||||
],
|
||||
"counts": {}
|
||||
},
|
||||
"apply": {
|
||||
"aliases": 50361,
|
||||
"observations": 296763,
|
||||
"preserved_name_claims": 8366,
|
||||
"preserved_reviews": 4,
|
||||
"recomputed_pairs": 6,
|
||||
"conflicts": []
|
||||
},
|
||||
"recheck": {
|
||||
"aliases": 0,
|
||||
"observations": 296763,
|
||||
"preserved_name_claims": 8366,
|
||||
"preserved_reviews": 4,
|
||||
"recomputed_pairs": 6,
|
||||
"conflicts": []
|
||||
},
|
||||
"validation": {
|
||||
"schema": 15,
|
||||
"frame_edges": 280,
|
||||
"cross_container_frame_edges": 0,
|
||||
"edge_history_rows": 720,
|
||||
"obsolete_assets": 0,
|
||||
"source_lak_observations": 139604,
|
||||
"preservation": {
|
||||
"names": true,
|
||||
"reviews": true,
|
||||
"signatures": true,
|
||||
"observations": true,
|
||||
"artifacts": true,
|
||||
"edge_occurrences": true
|
||||
}
|
||||
},
|
||||
"before": [
|
||||
{
|
||||
"from": "King Kong|0.97.0|LE|SPIKE 2",
|
||||
"to": "King Kong|0.97.0|Pro|",
|
||||
"counts": {
|
||||
"removed": 9537,
|
||||
"added": 8815,
|
||||
"unchanged": 1358,
|
||||
"modified": 2
|
||||
},
|
||||
"left": 10897,
|
||||
"right": 10175
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.81.0|LE|",
|
||||
"to": "Pokémon|0.81.0|Pro|",
|
||||
"counts": {
|
||||
"unchanged": 12466,
|
||||
"added": 4108,
|
||||
"removed": 4110,
|
||||
"modified": 3
|
||||
},
|
||||
"left": 16579,
|
||||
"right": 16577
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.85.0|LE|",
|
||||
"to": "Pokémon|0.86.0|LE|",
|
||||
"counts": {
|
||||
"unchanged": 12173,
|
||||
"added": 57,
|
||||
"removed": 95,
|
||||
"modified": 56
|
||||
},
|
||||
"left": 12324,
|
||||
"right": 12286
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.83.0|LE|",
|
||||
"to": "Pokémon|0.85.0|LE|",
|
||||
"counts": {
|
||||
"removed": 10304,
|
||||
"added": 4408,
|
||||
"unchanged": 7893,
|
||||
"modified": 23
|
||||
},
|
||||
"left": 18220,
|
||||
"right": 12324
|
||||
}
|
||||
],
|
||||
"after": [
|
||||
{
|
||||
"from": "King Kong|0.97.0|LE|SPIKE 2",
|
||||
"to": "King Kong|0.97.0|Pro|",
|
||||
"counts": {
|
||||
"removed": 3487,
|
||||
"unchanged": 7398,
|
||||
"added": 2765,
|
||||
"modified": 12
|
||||
},
|
||||
"left": 10897,
|
||||
"right": 10175
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.81.0|LE|",
|
||||
"to": "Pokémon|0.81.0|Pro|",
|
||||
"counts": {
|
||||
"unchanged": 16541,
|
||||
"added": 26,
|
||||
"removed": 28,
|
||||
"modified": 10
|
||||
},
|
||||
"left": 16579,
|
||||
"right": 16577
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.85.0|LE|",
|
||||
"to": "Pokémon|0.86.0|LE|",
|
||||
"counts": {
|
||||
"unchanged": 12173,
|
||||
"removed": 95,
|
||||
"added": 57,
|
||||
"modified": 56
|
||||
},
|
||||
"left": 12324,
|
||||
"right": 12286
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.83.0|LE|",
|
||||
"to": "Pokémon|0.85.0|LE|",
|
||||
"counts": {
|
||||
"removed": 6099,
|
||||
"unchanged": 12071,
|
||||
"added": 203,
|
||||
"modified": 50
|
||||
},
|
||||
"left": 18220,
|
||||
"right": 12324
|
||||
}
|
||||
],
|
||||
"anchors": [
|
||||
{
|
||||
"snapshot": "1314f21f-4f8f-4ed8-a1b3-cdc6a0505ea1",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "partition-06/pokemon_pro/game",
|
||||
"artifact": "blake3:78458719cb111b7aa9d023e5e7e3bdb283ab6a846218038154b6c270a52044f3",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "1314f21f-4f8f-4ed8-a1b3-cdc6a0505ea1",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_pro/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "partition-06/pokemon_pro/spike3/spike_menu/game",
|
||||
"artifact": "blake3:9203838793545ebf6b4c1f5d222dbd38aba51b30d67a463604c321803d1b3c37",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "1a218766-98ee-4501-8acd-cb902016fc82",
|
||||
"scope": "King Kong",
|
||||
"root": "package-0000/king_kong_pro-0_97_0/king_kong_pro/king_kong_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/king_kong_pro-0_97_0/king_kong_pro/king_kong_pro/game",
|
||||
"artifact": "blake3:dfca3139b3c260b76e4e8eaf6064782708d2f4fbf4ad7a3f220bdb4dfab8ed99",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "1a218766-98ee-4501-8acd-cb902016fc82",
|
||||
"scope": "King Kong",
|
||||
"root": "package-0000/king_kong_pro-0_97_0/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/king_kong_pro-0_97_0/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:7045f67c7b4ad1c068e0ec27efd2b578e49a8985e110647dd84a7040082ac761",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "21a5f51e-afe0-47f7-ab53-3fd3551a9474",
|
||||
"scope": "Deadpool",
|
||||
"root": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/deadpool_pro-1_16_0/deadpool_pro/deadpool_pro/game",
|
||||
"artifact": "blake3:18f8f4e198b880330fdb1817397294f06a403a66f5edde1e163cb087b0c895b6",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "21a5f51e-afe0-47f7-ab53-3fd3551a9474",
|
||||
"scope": "Deadpool",
|
||||
"root": "package-0000/deadpool_pro-1_16_0/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/deadpool_pro-1_16_0/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:8f5727699be00f55e6a9c9d227850881f33fecd68c5340b428571da6447f96f4",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "2a2a4737-cf72-4eaf-b0aa-4e848bd3092e",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_le",
|
||||
"canonical_root": "@game",
|
||||
"program": "partition-06/pokemon_le/game",
|
||||
"artifact": "blake3:d658e12a2059a07c36cc0210a92c0e9a86c84958fd4c4835ef129d3217a2aeab",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "2a2a4737-cf72-4eaf-b0aa-4e848bd3092e",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_le/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "partition-06/pokemon_le/spike3/spike_menu/game",
|
||||
"artifact": "blake3:d896acd14aa504da6be1a6688ab1913771b1cb844fa80e7137eaa5462f93a762",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "5acc64d9-d332-420d-9223-2ec98135dc9f",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "partition-06/pokemon_pro/game",
|
||||
"artifact": "blake3:276a88581c69b9988afd4834394b7e62b560fba12c7aeff54332c1c9aee21fa1",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "5acc64d9-d332-420d-9223-2ec98135dc9f",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_pro/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "partition-06/pokemon_pro/spike3/spike_menu/game",
|
||||
"artifact": "blake3:72179a1c665ed140b86bbf8c70b6ca81d91ef8c77d7565d2066abd39629034cd",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "6b57050e-bac7-4190-8935-e5aaabf96f88",
|
||||
"scope": "Jurassic Park (2019)",
|
||||
"root": "package-0000/jurassic_park_pro-1_16_0/jurassic_park_pro/jurassic_park_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/jurassic_park_pro-1_16_0/jurassic_park_pro/jurassic_park_pro/game",
|
||||
"artifact": "blake3:0408f39a254f267a2c37eb17a778418dfc18e6a8fcd2aa4a4fe949c9434f751b",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "6b57050e-bac7-4190-8935-e5aaabf96f88",
|
||||
"scope": "Jurassic Park (2019)",
|
||||
"root": "package-0000/jurassic_park_pro-1_16_0/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/jurassic_park_pro-1_16_0/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:8f5727699be00f55e6a9c9d227850881f33fecd68c5340b428571da6447f96f4",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "6c0b96f3-7e21-4bee-b101-33ec572af7e5",
|
||||
"scope": "Star Wars (2017)",
|
||||
"root": "package-0000/star_wars_pro-1_31_0/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/star_wars_pro-1_31_0/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:8f5727699be00f55e6a9c9d227850881f33fecd68c5340b428571da6447f96f4",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "6c0b96f3-7e21-4bee-b101-33ec572af7e5",
|
||||
"scope": "Star Wars (2017)",
|
||||
"root": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/star_wars_pro-1_31_0/star_wars_pro/star_wars_pro/game",
|
||||
"artifact": "blake3:a55a2cdb789097ff0e502f5498d9decdfe9ab998ae91067bd3bc554626cfa0d8",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "77a23356-a8ee-4e9a-a7ef-4aff578ef9b5",
|
||||
"scope": "Star Wars",
|
||||
"root": "package-0000/star_wars_elg-1_10_0_spike2/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/star_wars_elg-1_10_0_spike2/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:8f5727699be00f55e6a9c9d227850881f33fecd68c5340b428571da6447f96f4",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "77a23356-a8ee-4e9a-a7ef-4aff578ef9b5",
|
||||
"scope": "Star Wars",
|
||||
"root": "package-0000/star_wars_elg-1_10_0_spike2/star_wars_elg/star_wars_elg",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/star_wars_elg-1_10_0_spike2/star_wars_elg/star_wars_elg/game",
|
||||
"artifact": "blake3:47ae639d64772c637d0fd2c81eb3e55dd040cff37ca38bfae829fee55e6ef5f8",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "7f60ebe7-ec34-4ac9-9ba7-f7ae561c909f",
|
||||
"scope": "Godzilla",
|
||||
"root": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/godzilla_pro-1_16_0_spike2/godzilla_pro/godzilla_pro/game",
|
||||
"artifact": "blake3:2ca2e5d5302748599c18b4850996a2207631ea4b5be53bfab50280be8db4d31c",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "7f60ebe7-ec34-4ac9-9ba7-f7ae561c909f",
|
||||
"scope": "Godzilla",
|
||||
"root": "package-0000/godzilla_pro-1_16_0_spike2/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/godzilla_pro-1_16_0_spike2/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:8f5727699be00f55e6a9c9d227850881f33fecd68c5340b428571da6447f96f4",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "830d0450-50ee-4159-a0d5-3fafb1410b74",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_le",
|
||||
"canonical_root": "@game",
|
||||
"program": "partition-06/pokemon_le/game",
|
||||
"artifact": "blake3:64981c61ee0da86e466a87baada8ee01e940267aa5ab43b4fe2df8a7c4be8b84",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "830d0450-50ee-4159-a0d5-3fafb1410b74",
|
||||
"scope": "Pokémon",
|
||||
"root": "partition-06/pokemon_le/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "partition-06/pokemon_le/spike3/spike_menu/game",
|
||||
"artifact": "blake3:72179a1c665ed140b86bbf8c70b6ca81d91ef8c77d7565d2066abd39629034cd",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "86eac228-770b-44e8-83ea-1278ec262689",
|
||||
"scope": "King Kong",
|
||||
"root": "partition-02/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "partition-02/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:7045f67c7b4ad1c068e0ec27efd2b578e49a8985e110647dd84a7040082ac761",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "86eac228-770b-44e8-83ea-1278ec262689",
|
||||
"scope": "King Kong",
|
||||
"root": "partition-03/king_kong_le",
|
||||
"canonical_root": "@game",
|
||||
"program": "partition-03/king_kong_le/game",
|
||||
"artifact": "blake3:cebc6521ada13e1cb9cb617889a9f5a26eab5a75dbf9f1fd22e225e42052af89",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "90d7833a-d381-41b0-8567-b8b8e5a43b1c",
|
||||
"scope": "Pokémon",
|
||||
"root": "package-0000/pokemon_le-0_86_0_spike3/pokemon_le/pokemon_le",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/pokemon_le-0_86_0_spike3/pokemon_le/pokemon_le/game",
|
||||
"artifact": "blake3:8bab9abf40d145d0fe9e9073eb1560e1aa0f0ce6d3ab876dc45d8f8203dfa97d",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "90d7833a-d381-41b0-8567-b8b8e5a43b1c",
|
||||
"scope": "Pokémon",
|
||||
"root": "package-0000/pokemon_le-0_86_0_spike3/pokemon_le/pokemon_le/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/pokemon_le-0_86_0_spike3/pokemon_le/pokemon_le/spike3/spike_menu/game",
|
||||
"artifact": "blake3:d25d679ff69e015f17fbc30582ba6e2841ca95fbfcb2175cd8f30fd96549ee89",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "9105cf5f-52c6-42eb-bee6-f971f85cbfc2",
|
||||
"scope": "Dungeons and Dragons",
|
||||
"root": "package-0000/dungeons_and_dragons_pro-1_01_0/dungeons_and_dragons_pro/dungeons_and_dragons_pro",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/dungeons_and_dragons_pro-1_01_0/dungeons_and_dragons_pro/dungeons_and_dragons_pro/game",
|
||||
"artifact": "blake3:40959f3f89f5da45222dfd3923e8880121e6e06b59fd58c30dadb5b70134e629",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "9105cf5f-52c6-42eb-bee6-f971f85cbfc2",
|
||||
"scope": "Dungeons and Dragons",
|
||||
"root": "package-0000/dungeons_and_dragons_pro-1_01_0/spike/usr/local/spike/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/dungeons_and_dragons_pro-1_01_0/spike/usr/local/spike/spike_menu/game",
|
||||
"artifact": "blake3:7045f67c7b4ad1c068e0ec27efd2b578e49a8985e110647dd84a7040082ac761",
|
||||
"elf_header": "7f454c4601010100000000000000000002002800"
|
||||
},
|
||||
{
|
||||
"snapshot": "e560c764-7318-493b-acdb-0a9aeb3d2e42",
|
||||
"scope": "Pokémon",
|
||||
"root": "package-0000/pokemon_le-0_85_0_spike3/pokemon_le/pokemon_le",
|
||||
"canonical_root": "@game",
|
||||
"program": "package-0000/pokemon_le-0_85_0_spike3/pokemon_le/pokemon_le/game",
|
||||
"artifact": "blake3:1c2b46b3de53abeb2fbca421f580770e87602471cefd8d3ef4d1d3e30ad0dfa0",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
},
|
||||
{
|
||||
"snapshot": "e560c764-7318-493b-acdb-0a9aeb3d2e42",
|
||||
"scope": "Pokémon",
|
||||
"root": "package-0000/pokemon_le-0_85_0_spike3/pokemon_le/pokemon_le/spike3/spike_menu",
|
||||
"canonical_root": "@system/spike_menu",
|
||||
"program": "package-0000/pokemon_le-0_85_0_spike3/pokemon_le/pokemon_le/spike3/spike_menu/game",
|
||||
"artifact": "blake3:2d7e091f4196817cf151e4bb728c4e9d67fc0701ed7f92552f787cb63341e487",
|
||||
"elf_header": "7f454c460201010300000000000000000300b700"
|
||||
}
|
||||
],
|
||||
"radium_edition_evidence": [
|
||||
{
|
||||
"from": "King Kong|0.97.0|LE|SPIKE 2",
|
||||
"to": "King Kong|0.97.0|Pro|",
|
||||
"role": "/@game/image.bin|",
|
||||
"left": 1493,
|
||||
"right": 1493,
|
||||
"shared": 1493,
|
||||
"identical_originals": 1493
|
||||
},
|
||||
{
|
||||
"from": "King Kong|0.97.0|LE|SPIKE 2",
|
||||
"to": "King Kong|0.97.0|Pro|",
|
||||
"role": "/@system/spike_menu/image.bin|",
|
||||
"left": 1469,
|
||||
"right": 1469,
|
||||
"shared": 1469,
|
||||
"identical_originals": 1469
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.81.0|LE|",
|
||||
"to": "Pokémon|0.81.0|Pro|",
|
||||
"role": "/@game/image.bin|",
|
||||
"left": 1477,
|
||||
"right": 1477,
|
||||
"shared": 1477,
|
||||
"identical_originals": 1477
|
||||
},
|
||||
{
|
||||
"from": "Pokémon|0.81.0|LE|",
|
||||
"to": "Pokémon|0.81.0|Pro|",
|
||||
"role": "/@system/spike_menu/image.bin|",
|
||||
"left": 1469,
|
||||
"right": 1469,
|
||||
"shared": 1469,
|
||||
"identical_originals": 1469
|
||||
}
|
||||
],
|
||||
"cross_generation_canary": "Pending actual Star Wars ELG 1.10 SPIKE2/SPIKE3 media outputs; not replaced by a synthetic pair."
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
# Verified container identity
|
||||
|
||||
Schema 15 adds an explicit container-root identity migration. Opening an archive adds the schema only; it does not rewrite asset identities. The migration is intended to join the same game and system-menu containers across SD images, update packages, and editions while preserving game scopes and platform-specific release IDs.
|
||||
|
||||
`GET /api/spine/identity/plan` returns the exact proposed aliases, source anchors, observation stamp, and conflicts. `POST /api/spine/identity/apply` accepts that reviewed JSON (maximum 32 MiB, `X-Verstack-Client: 1`). Apply reads each original program header again, then reconstructs the plan under an immediate SQLite transaction using those bounded source reads. Stale observations, altered proofs, new candidates, contradictory contexts, and modified plans reject the transaction. No archive payload is changed.
|
||||
|
||||
An anchor needs a recognized layout ending in `game`, an indexed artifact identity, and a 20-byte original ELF header identifying ARM32 or AArch64 EXEC/DYN. Supported layouts are:
|
||||
|
||||
- SD `partition-N/TITLE_EDITION/game` and packaged `package-N/PACKAGE/TITLE_EDITION/TITLE_EDITION/game`, mapped to `@game`.
|
||||
- SD `partition-N/usr/local/spike/spike_menu/game` and packaged `package-N/PACKAGE/spike/usr/local/spike/spike_menu/game`, mapped to `@system/spike_menu`.
|
||||
- SPIKE3 menu beneath the recognized title root at `spike3/spike_menu/game`, mapped to the same menu role.
|
||||
|
||||
Recognized edition suffixes are `_pro`, `_le`, `_prem`, and `_elg`. This is a structural grammar, not a basename heuristic. Duplicate or unverified candidate roots are not canonicalized. Nested package/partition hierarchies remain distinct. Each canonical path retains its repository scope, path relative to the container, Radium section/record ID, or scene instance name. Conflicting original records or File bytes within one release block the affected role instead of discarding observations. Edition metadata is not inferred from a directory suffix: real archived builds sometimes contain a different edition's directory spelling.
|
||||
|
||||
Apply moves derived observations and edges onto canonical keys, removes obsolete derived asset rows, and recomputes existing stored comparison pairs atomically. `source_lak`, immutable alias mappings, original asset metadata, aggregated edge history, and complete edge-occurrence history retain provenance. Names, their IDs and evidence, review rows, agent findings, immutable artifacts, payload bytes, and full signature masks remain unchanged. Views resolve historical name/review/finding keys; old asset and review URLs resolve aliases. Conflicting historical verdicts appear as undecided conflicts. A new canonical verdict resolves them; clearing it leaves a tombstone so old decisions do not reappear.
|
||||
|
||||
Once explicitly enabled, backfill verifies new source anchors and uses canonical identities at indexing time. Existing accepted aliases remain authoritative. Native scene indexing uses the same proven roots; unknown inputs retain their original keys until a verified backfill. Frame-chain links resolve record IDs within their container role and section, preventing game/menu collisions.
|
||||
|
||||
`tests/identity_migration.rs` covers original-key compatibility, immutable claims, occurrence multiplicity, review conflicts/clears, nested and ambiguous roots, conflicting File originals, stale/forged/changed proofs, a candidate added during planning, and HTTP plan/apply. Its ignored `rehearse_live_catalog_copy` test requires an explicit `VERSTACK_IDENTITY_COPY` path under `data/validation/identity-dryrun-*`; only this copy is writable. It reads original headers using local HTTP GET Range and writes the plan/report beside the copy. Never point a rehearsal Archive at the live Rustic store.
|
||||
|
||||
Real-corpus rehearsal results are recorded separately. A canonical role does not prove complete decoder coverage or semantic equivalence of every asset; unmatched sounds and unknown structures remain visible differences. Native codesig and complete Phase 5 function recovery are separate requirements.
|
||||
|
||||
## Copy rehearsal, 2026-09-16
|
||||
|
||||
A fresh WAL-aware SQLite backup of the deployed schema-13 catalog was upgraded and migrated at `data/validation/identity-dryrun-20260916-1.sqlite3`. No live writes were performed. The plan verified 28 ARM ELF anchors and applied 50,361 aliases with zero conflicts. It preserved all 296,763 observations, 8,366 name claims, and four review claims and recomputed six stored comparison pairs. A fresh second plan found zero further aliases. The final derived index contains no obsolete alias asset rows.
|
||||
|
||||
| Real pair | Unchanged before → after | Modified before → after | Added before → after | Removed before → after |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| King Kong 0.97 LE → Pro | 1,358 → 7,398 | 2 → 12 | 8,815 → 2,765 | 9,537 → 3,487 |
|
||||
| Pokémon 0.81 LE → Pro | 12,466 → 16,541 | 3 → 10 | 4,108 → 26 | 4,110 → 28 |
|
||||
| Pokémon 0.85 → 0.86 | 12,173 → 12,173 | 56 → 56 | 57 → 57 | 95 → 95 |
|
||||
| Pokémon 0.83 SD → 0.85 package | 7,893 → 12,071 | 23 → 50 | 4,408 → 203 | 10,304 → 6,099 |
|
||||
|
||||
In King Kong, all 1,493 game records and 1,469 menu records join across editions with byte-identical originals. File paths are normalized by the same proven root rule, explaining the larger full-inventory improvement. Modified counts can increase when two previously unmatched files become comparable. Remaining differences stay visible.
|
||||
|
||||
Full-row SHA-256 fingerprints and cardinalities were identical before/after for 1,521,210 artifact records, 1,170,298 function signatures including masks, immutable names, review rows, and observation payload/provenance fields. The initial source had no native edge-occurrence rows; synthetic native-scene regressions verify occurrence preservation and reindexing. Of 400 old Radium frame links, 200 crossed container roles incorrectly. The final rebuild contains 280 proven links and zero cross-container links; missing bases remain unlinked. The copy retains both original and intermediate edge history because the corrected rebuild was exercised in the fresh-plan recheck.
|
||||
|
||||
[The machine-readable report](container-identity-canary.json) contains source anchors, artifact identities, ELF proof bytes, preservation digests, and numeric results. The third required axis—actual Star Wars ELG 1.10 SPIKE2 versus SPIKE3—is still pending complete media outputs. This rehearsal does not claim that canary or the entire identity plan finished.
|
||||
|
||||
## Live three-axis gate, 2026-09-17
|
||||
|
||||
The retained Star Wars SPIKE 3 original retried successfully after release2's
|
||||
ZIP expansion admission fix. Both complete imports now have extraction, media,
|
||||
Godot script, audio and preview outputs. Backfill indexed 331,270 observations
|
||||
across 78 source snapshots (analysis/preview artifacts excluded). The deployed
|
||||
identity policy verified both architectures from original ELF headers and kept
|
||||
the two platform release IDs distinct.
|
||||
|
||||
`scripts/verify_identity_canaries.py` passed against live, paginated canonical
|
||||
comparison endpoints and original-byte identities for all three axes. Pokémon
|
||||
0.85→0.86 retains 57 added / 95 removed / 56 modified / 12,173 unchanged; King Kong
|
||||
LE→Pro retains 2,765 / 3,487 / 12 / 7,398. Star Wars SPIKE2→SPIKE3 reports
|
||||
58,007 added / 1,254 removed / 69 modified / 8,508 unchanged. These full-inventory
|
||||
counts include differing extraction/representation coverage; they do not imply
|
||||
58,007 changed gameplay assets.
|
||||
|
||||
Every shared original Radium record is byte-identical: Pokémon 1,477 game +
|
||||
1,469 menu; King Kong 1,493 + 1,469; Star Wars 1,481 + 1,469. No shared key has
|
||||
ambiguous original bytes. Sibling derivation now reports one platform-port pair.
|
||||
The machine-readable live report is retained at
|
||||
`data/validation/identity-three-axis-live-20260917.json`. The portable regression
|
||||
in `tests/identity_migration.rs` uses the actual four Star Wars roots and ELF
|
||||
headers with explicitly synthetic payloads, checking cross-architecture joins,
|
||||
separate menu/game records and executable differences. It passes independently
|
||||
of the full real-corpus gate.
|
||||
|
||||
Bulk import remains gated by streaming/storage capacity, sustained scheduling
|
||||
verification and the remaining PLAN requirements; no destructive cleanup occurred.
|
||||
+6
-2
@@ -37,8 +37,12 @@ still unavailable in the donor broker.
|
||||
|
||||
## Runtime and donor adaptations
|
||||
|
||||
The source came from the user-supplied `~/pokemon_emulator`. Vendored source and
|
||||
its local changes are recorded in [provenance](../emulator/vendor/PROVENANCE.md).
|
||||
The source came from the user-supplied `~/pokemon_emulator`. The pinned upstream
|
||||
export and its local changes are recorded in `emulator/upstream/SOURCE.json`, the
|
||||
`emulator/patches/{worker,bundle}` series and the
|
||||
[de-fork audit](emulator-defork-audit.md). The former `emulator/vendor` and
|
||||
`emulator/bundles/vendor` trees are untracked compatibility copies for
|
||||
already-running services.
|
||||
Only process, device-shim, machine and netbridge source plus table assets are
|
||||
included; game binaries and system libraries are not committed.
|
||||
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# Emulator source and de-fork audit — 2026-09-16
|
||||
|
||||
The audit below records the pre-refactor state. The source integration described here is now implemented and tested locally. No running emulator, configuration, image, service or canonical source was changed. It does **not** establish a working SPIKE 2 bridge or gameplay acceptance.
|
||||
|
||||
Subsequent isolated real-runtime evidence, the NVRAM permission fix and actual ARM32/ARM64 container ABI gates are recorded in [emulator-runtime-canary.md](emulator-runtime-canary.md). Disposable canary processes were launched separately from the live worker; the historical host-toolchain limitation below is superseded by those container checks.
|
||||
|
||||
## Implemented source integration
|
||||
|
||||
`emulator/upstream/pokemon_emulator` contains the union of 129 allowlisted files from the pinned canonical Git objects. `emulator/upstream/SOURCE.json` records each original hash/mode and a filtered source-only Git export. The self-contained `source-export.bundle` resolves to commit `bcb80ef2bda6d5735489798d61d210bb6862f77a`, tree `44b0482814423c3e41088f039bb7d739bdc64bd8`. This is verifiable Git provenance, not a claim that shared repository subtree merge history has been committed. That history integration remains a coordinated commit step; no commits were made here.
|
||||
|
||||
`emulator/patches/{worker,bundle}` declares ordered unified patch series, preimage hashes and exact output manifests. `source_profiles.py` applies those patches without Git, external patch tools, network access or fuzzy matching. It verifies all output hashes before writing. Worker sessions, bundle staging, standalone SPIKE 2 workers and quick export tools now consume materialized profiles. Source metadata travels in bundle records, and export cache identity includes the declared source and patches. The image builder mounts these source inputs explicitly.
|
||||
|
||||
Both old vendor trees were removed **from the Git index only**, after byte-and-mode equivalence was verified. Their physical directories remain untouched and ignored: already running services may have imported the old paths and still need them. Do not delete these compatibility directories until those processes have stopped/reloaded using the new consumers. New Docker build contexts exclude them. No live restart or deployment was performed.
|
||||
|
||||
Verification on 2026-09-16:
|
||||
|
||||
```sh
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 scripts/verify_emulator_source.py --canonical /home/jordan/pokemon_emulator --legacy
|
||||
cd emulator
|
||||
PYTHONDONTWRITEBYTECODE=1 /tmp/verstack-emulator-test-env/bin/python -m unittest test_source_profiles test_game_startup test_runtimes test_bundle_builder test_bundle_selection test_bundle_storage test_bundle_universal test_bundle_api test_server test_uart_wakeup
|
||||
```
|
||||
|
||||
The verifier matched all 129 upstream files against canonical Git objects and the isolated export, and reproduced all 20 worker and 125 bundle files byte-for-byte with identical modes. All 41 tests passed, including tamper rejection, standalone source packaging/imports, isolated HTTP export/session flows, cache invalidation, and native UART probes. The temporary test environment supplies aiohttp/Pillow; it does not change runtime environments. Cross-architecture shim ABI tests were not rerun: host ARM cross-compilers and QEMU binaries are absent. No release image or gameplay acceptance was attempted. The canonical working tree still has only its unrelated dirty VPX file.
|
||||
|
||||
This first gate removed duplicated maintained source trees while preserving the two existing behaviors. The subsequent canonical bridge adapter gate is recorded below; captured SPIKE 2 wire protocol implementation remains outstanding.
|
||||
|
||||
## Canonical Linux bridge adapter gate
|
||||
|
||||
The worker now materializes **47 files** and omits its shorter `emulation/netbridge-emulator` implementation. Its `analysis/netbridge-emulator/netbridge_emu.py` is the same unmodified canonical blob as the Mac bundle (`02062ca4367e23cf12d7f7bfd6961d775381dcd32909bd1b066315399000e085`). Analysis support modules, tools and fixtures are staged beside it. Worker-specific behavior is declared in `canonical_bridge.py`, machine/service patches, and the worker manifest. `scripts/update_emulator_profile.py worker <edited-materialized-directory>` regenerates reviewed patches and verifies replay before replacing the series. The upstream pin/export is unchanged.
|
||||
|
||||
`RuntimeBridge` retains the existing runtime-derived power-board version and display-ready replies. Canonical `CPUSPIControl` owns the worker SPI input bank, using the worker's configured byte/bit layout rather than silently substituting Mac GPIO positions. Unnamed worker-local inputs use internal negative identifiers, never invented nodebus addresses. Canonical `PlayfieldAPIServer` is started by `MachineService` on a per-session ephemeral loopback port (or explicit `netbridge.playfield_port`); machine state reports the port and implementation path. Its CPU facade writes the same in-process bank read by the actual SPI request handler. API Start/Action contacts also update canonical power-distribution inputs. The facade acknowledges local memory delivery, not QEMU delivery. This is service integration, not a new browser/remote VPX connection flow.
|
||||
|
||||
Generic runtime node overlays remain explicit. `magikarp_topper` defaults false; enabling it uses canonical `TopperNodeDevice` objects and rejects generic overlays of their addresses. No topper is inferred from a filename or game title. Reset/restore replace and stop prior API/control threads; service close releases switches, cancels pulse timers and stops bridge/API threads. Explicit hold after pulse cancels the pending release. Synthetic flipper end-of-stroke release is retained.
|
||||
|
||||
The shim's UART transport remains synchronous RPC. Canonical unsolicited power-distribution events are drained on the **next UART feed**, including an explicit empty poll; the worker does not yet push spontaneous UART events to an idle guest. SPIKE 2 raw packets remain unsupported by this SPIKE 3 parser. No live restart, real game launch, cross-architecture build or frame/audio/input acceptance is claimed.
|
||||
|
||||
Verification: the 45-test emulator suite passed after the adapter port; a subsequent targeted 11-test gate covers the final UART drain addition. All 31 upstream `emulation/machine/test_machine.py` contracts passed against the materialized worker module (isolated temporary state/sockets). New tests exercise actual API messages through loopback sockets, SPI bits, power-board event frames, disconnect cleanup, pulse/hold behavior, reset, source identity and explicit topper selection. `scripts/verify_emulator_source.py --canonical /home/jordan/pokemon_emulator` verifies current worker47/bundle125 replay and upstream129 provenance. The earlier `--legacy` comparison is intentionally a historical pre-port gate: worker behavior now differs from the preserved live compatibility directory. Formal shared-repository subtree history integration and deployment cleanup are still pending.
|
||||
|
||||
## Verified source and copies
|
||||
|
||||
Canonical repository: `/home/jordan/pokemon_emulator`, commit **`37d70e64286460938772e7de952dc009d889c198`** (`Volume is adjustable on the VM now.`). Both Verstack copies name this same revision. The only canonical working-tree change is the unrelated `emulation/pokemon-vpx/dist/Pokemon-Pro.vpx`; do not include or overwrite that working file. Read source bytes from the pinned Git object, not the working directory. No additional `AGENTS.md` files were found under the canonical repository or `verstack/emulator`; Verstack's root rules apply.
|
||||
|
||||
| Source | Current tracked footprint | Evidence |
|
||||
|---|---:|---|
|
||||
| `emulator/vendor` | 20 files, approximately 836 KiB on disk | `PROVENANCE.md` names the canonical revision; 19 upstream files and one local provenance document |
|
||||
| `emulator/bundles/vendor/pokemon_emulator` | 125 files, approximately 2.3 MiB on disk | `SOURCE.json` lists 123 original files and SHA-256 hashes; all 123 match the pinned canonical Git objects exactly; 11 current copies differ from their original bytes |
|
||||
|
||||
The bundle has two files outside the original 123-file manifest: `SOURCE.json` itself and the local `emulation/conagent-machine-emulator/prepare_export.py`. The worker and bundle share 13 paths; **three shared files disagree**: `emulation/stubs/spike3emu_stub.c`, `emulation/stubs/spike3machine_client.c`, and `emulation/netbridge-emulator/netbridge_emu.py`. Replacing either directory with the other loses behavior.
|
||||
|
||||
Both vendor directories arrived in Verstack commit `f73167d` (`Add downloadable VMs and improve catalog processing`). Current provenance is a pinned copy plus prose, not an independently replayable patch series.
|
||||
|
||||
Netbridge implementations:
|
||||
|
||||
| File | Lines | SHA-256 |
|
||||
|---|---:|---|
|
||||
| canonical `analysis/netbridge-emulator/netbridge_emu.py` | 4,116 | `02062ca4367e23cf12d7f7bfd6961d775381dcd32909bd1b066315399000e085` |
|
||||
| canonical `emulation/netbridge-emulator/netbridge_emu.py` | 863 | `fd93c55b55d6f68ccfae9b194fe6ec1e7bd24020f6ee8633227c7e2b624e9b8a` |
|
||||
| worker `vendor/emulation/netbridge-emulator/netbridge_emu.py` | 918 | `1fdec355a3ba7c219d4344ec747d624b2098bd1a6543930232b3ce0dabd9ebfb` |
|
||||
|
||||
The bundle's 4,116-line analysis implementation is byte-identical to canonical. Its 863-line emulation implementation is also byte-identical to canonical. The **macOS bundle already launches the analysis implementation**: `emulator/bundles/vendor/pokemon_emulator/emulation/scripts/run-macos.sh` uses `$workspace/analysis/netbridge-emulator/netbridge_emu.py`. The Linux worker is the path that uses the shorter implementation: `server.py::Session.start` copies only `vendor/emulation`, and `machine.py::load_netbridge_module` imports its sibling `emulation/netbridge-emulator/netbridge_emu.py`.
|
||||
|
||||
## Why changing one loader path is insufficient
|
||||
|
||||
A read-only import confirmed that the analysis module exports `CPUSPIControl`, `PlayfieldAPIServer`, `TopperNodeDevice`, `NetbridgeModel`, `FrameDecoder` and `SWITCH_MAP`. However, the worker adapter constructs:
|
||||
|
||||
```python
|
||||
NetbridgeModel(image_crc=image_crc, powerdist_version=config.get("powerdist_version"))
|
||||
```
|
||||
|
||||
Calling that constructor on the canonical analysis module reproduced **`TypeError: NetbridgeModel.__init__() got an unexpected keyword argument 'powerdist_version'`**. The worker's startup-reply extension must be ported or implemented by an adapter, with its existing tests preserved.
|
||||
|
||||
The canonical module also imports `topper_protocol`, which imports `tools.extract_magikarp_topology`, and reads topper fixture JSON plus `emulation/vpx/layout.json`. Merely copying `netbridge_emu.py` leaves an incomplete dependency tree. Its complete 26-file analysis directory is already represented in the bundle manifest; preserve that directory's relative layout and the vpx/cpu-spi profiles.
|
||||
|
||||
Loading a class is not wiring its behavior. The Linux machine service currently has its own cabinet/SPI state and node overlay, and does not instantiate `PlayfieldAPIServer` or join that state to `CPUSPIControl`. The canonical model's `magikarp_topper` flag selects `TopperNodeDevice` behavior; the worker presently overlays generic `NodeDevice` objects. Default discovery also differs: the worker includes node `0x0A`, while canonical's base discovery is `0x01,0x04,0x08,0x09` and expands when the topper profile is enabled. Those choices require explicit profile mapping and behavioral tests, not unconditional global defaults.
|
||||
|
||||
## Local patches that must survive
|
||||
|
||||
Worker modifications relative to the pinned commit are confined to five source files:
|
||||
|
||||
- `emulation/machine/machine.py`: runtime power-board metadata, raw UART diagnostics, expanded cabinet switch vocabulary, and pulse-release cancellation behavior.
|
||||
- `emulation/netbridge-emulator/netbridge_emu.py`: dashboard contact map and runtime-derived power-board version/ready replies.
|
||||
- `emulation/process/spike3_emu.py`: ARM32/ARM64 tool and library selection, generic game/runtime namespace, runtime closures, process/shim setup and Verstack launcher adaptations.
|
||||
- `emulation/stubs/spike3emu_stub.c`: real software framebuffer capture/headless rendering, private NVRAM access and display/ABI adaptations; synthetic output remains diagnostic-only.
|
||||
- `emulation/stubs/spike3machine_client.c`: UART eventfd wakeups, SPI pacing, optional legacy UART alias, real-time thread fallback and absent-ALSA-mixer handling.
|
||||
|
||||
The bundle modifies these 11 upstream paths:
|
||||
|
||||
- `emulation/scripts/prepare-rootfs-image.sh`: module dependency indexes.
|
||||
- `emulation/scripts/run-macos.sh`: pre-provisioned local IC handling.
|
||||
- `emulation/guest/emu-init`: virtio networking module load.
|
||||
- `emulation/dashboard/server.py` and `test_server.py`: service-menu controls.
|
||||
- `emulation/conagent-machine-emulator/prepare_guest.py`, `provisioning.py`, `test_provisioning.py`, `web_ui.py`: export-specific local provisioning and associated UI/tests.
|
||||
- `emulation/docker/Dockerfile` and `fetch-debian-arm64-kernel.sh`: architecture-correct guest kernel/initramfs construction.
|
||||
|
||||
The local `prepare_export.py` must also be retained. Provisioning uses exact game hashes; retain those guards. Do not apply these export-specific transformations to arbitrary firmware or to the Linux worker merely because sources are unified.
|
||||
|
||||
## Exact migration sequence
|
||||
|
||||
1. **Create one reproducible, allowlisted upstream subtree** at `emulator/upstream/pokemon_emulator`, pinned to the full canonical commit above. Start with the union of the existing manifest's 123 paths and the worker's 19 upstream paths (129 distinct upstream files). Add relevant upstream machine/process/shim tests explicitly. Include the complete canonical netbridge support directory and its referenced profiles. Verify every blob against `git show <commit>:<path>`. Do not import the whole repository: it also contains large VPX material and unrelated artifacts.
|
||||
2. **Preserve Git provenance without importing unrelated history/content.** Generate a filtered source-only export commit in a temporary repository from the allowlist and pinned Git objects, recording upstream commit, original modes and SHA-256 per path. Add that export as a squashed subtree. Record both the canonical commit and filtered export commit; never call the filtered commit the canonical upstream revision. Generation must reject unlisted files, missing source blobs or hash mismatches.
|
||||
3. **Extract patch series before switching consumers.** Store `emulator/patches/common/series`, `worker/series`, and `bundle/series`, with base commit and expected pre/post hashes. Initially preserve the five worker changes and eleven bundle changes exactly, plus the local export helper. Shared source lives once; platform-specific patches apply to disposable build/session staging, not the source subtree. Check that rebuilding each profile is byte-identical to today's corresponding code before introducing behavior changes.
|
||||
4. **Rewire source staging atomically.** Replace `server.py`'s hardcoded worker-copy source and `bundles/builder.py`'s `VENDOR` source with the same materializer. Update `bundles/spike2.py::stage_worker`, which currently copies `source/'vendor'`, too. Docker and export builds must include the upstream tree and patch manifests. Generated runtime trees remain disposable and are not committed as a second vendor copy. Bundle metadata should report canonical revision, patch-set digest and profile; the existing single `emulator_source` commit alone does not identify local adaptations.
|
||||
5. **Port the Linux bridge behind a tested adapter.** Stage the analysis module with all its dependencies. Preserve power-board startup contracts; map cabinet/SPI state, switch lifecycle and cleanup to the canonical model explicitly. Select topper and node inventory from runtime/profile evidence. Assert that Linux worker and macOS bundle now resolve the same canonical netbridge blob and differ only by declared profile patches. Keep this change separable from the byte-equivalent de-fork so regressions are attributable.
|
||||
6. **Only after verification, retire the two tracked copies** and their hardcoded consumers together. No live service restart or active-session filesystem replacement is part of source migration. Deployment remains a separate action, and existing exports remain usable artifacts.
|
||||
|
||||
Acceptance for the source refactor: clean reconstruction from pinned objects and patches; no unmanifested edits; expected module paths at runtime; both architecture toolchains build; existing worker and bundle suites pass; real frames/input/audio acceptance is rerun before claiming behavioral improvement.
|
||||
|
||||
## SPIKE 2: path mapping and wire protocol are separate gaps
|
||||
|
||||
### Implemented platform device table (2026-09-16)
|
||||
|
||||
The worker shim now selects a declared C table through `SPIKE_MACHINE_PLATFORM`:
|
||||
|
||||
| Platform | SPI | I2C | UART |
|
||||
|---|---|---|---|
|
||||
| `spike2` | `/dev/spidev1.0` | `/dev/i2c-1` | `/dev/ttymxc1` |
|
||||
| `spike3` | `/dev/spidev4.0` | `/dev/i2c-10` | `/dev/ttyAMA5` |
|
||||
|
||||
The runner requires explicit runtime/release generation and checks it against both runtime architecture and the selected executable's actual ELF header **before launch setup**. Missing, unknown, conflicting or mismatched metadata raises a prelaunch error. It does not guess a board generation from arbitrary ARM32 code. Materialization verifies declared release generation against remote ELF discovery and the downloaded executable, then records `generation`; SPIKE 2 appliance exports similarly verify archive release metadata and game ELF before recording their runtime manifest. Old runtimes missing generation must be rematerialized or supplied correct provenance before a future launch; existing running processes are unchanged.
|
||||
|
||||
For standalone historical SPIKE 3 launchers only, an unset shim selector keeps SPIKE 3 paths. An explicitly unknown selector intercepts no platform devices. Opposite-platform and unknown paths pass through. Absolute `open`, `open64`, `openat`, and `/dev`-relative `openat` are covered. Unknown-descriptor ioctl now returns libc's real result/errno; the old shim incorrectly converted `ENOTTY` to success even for untracked ordinary files. Simulated-device compatibility behavior remains scoped to tracked descriptors.
|
||||
|
||||
All **52 affected emulator tests passed**, including compiled native preload probes for both tables, unset/unknown selectors, wrong-platform paths, relative-directory handling, ordinary-file passthrough, SPI mode ioctl, I2C address ioctl, UART pending-byte ioctl, and delayed `poll`/`select` wakeups for both UART paths. Tests refuse to probe opposite-platform real device nodes if present. Source provenance/replay remains verified (upstream129, worker47, bundle125). ARM cross-architecture ABI execution and live acceptance remain pending. These path aliases do not implement the SPIKE 2 transport.
|
||||
|
||||
### Bounded next-step transport investigation
|
||||
|
||||
The current worker shim's `write(FD_UART)` calls `rpc_netbridge_feed`, appends replies to its 4096-byte queue and signals the eventfd. `read(FD_UART)` only consumes that queue. `machine_rpc` serializes request/reply traffic on one connection under `rpc_lock`. This proves why no guest write means no new unsolicited event: there is no receiver path independent of `write`. The canonical model already provides `drain_unsolicited_host_frames`; the adapter drains it on each feed, not on a separate receiver.
|
||||
|
||||
An asynchronous implementation should add a distinct bounded event subscription/drain operation and a dedicated connection; a blocking event wait must not hold the existing `rpc_lock`, which would starve SPI/I2C calls. A shim receiver must wake the same eventfd, preserve whole-frame ordering, report overflow instead of truncating frames, and use a descriptor generation token so close/reopen cannot send stale bytes to a reused fd. Shutdown must cancel/join the receiver before releasing its queue. Acceptance requires a blocked `poll`/`select` waking after an API Start event **without any guest UART write**, simultaneous SPI traffic, partial reads, queue overflow, disconnect/reconnect, fd reuse and ARM32/ARM64 probes. None of this asynchronous path is implemented yet.
|
||||
|
||||
For `Spike2Bridge`, the recorded request fragments (`0a00`, `070101`, `8002f18d00`, `8003f0226b00`) establish incompatibility, not field semantics. The next evidence artifact should pair timestamped CPU request/reply traces with the same release's bridge driver/disassembly and plaintext LPC firmware hash. Recover boundaries, opcode/length/checksum rules and startup exchanges before writing an incremental parser. Keep unknown commands explicit rather than manufacturing successful SPIKE 3 replies. A future parser must have split/coalesced packet, invalid-length/checksum and captured startup replay fixtures. Existing node-level behavior can be shared only after CPU-side framing/command meaning is proven. No hardware commands, live session capture or speculative packet responses were introduced here.
|
||||
|
||||
At the original audit baseline, canonical `emulation/stubs/spike3machine_client.c::classify_path` recognized `/dev/spidev4.0`, `/dev/i2c-10` and `/dev/ttyAMA5`. The worker additionally aliased `/dev/ttymxc1` under `SPIKE3_EXPERIMENTAL_SPIKE2_UART=1`; `/dev/spidev1.0` and `/dev/i2c-1` were absent. The worker implementation above replaces that partial alias with an explicit platform table; pinned upstream source remains untouched.
|
||||
|
||||
A data-driven platform table should select SPI/I2C/UART paths from verified generation metadata, with unknown paths delegated rather than assigned guessed semantics. Preserve existing SPIKE 3 handling and test every positive/negative mapping, including `open`, `open64`, `openat`, ioctl and poll/select wakeup behavior for both architectures.
|
||||
|
||||
**Adding aliases cannot complete SPIKE 2.** `docs/emulation.md` records a real King Kong LE 0.97 run opening the aliased UART and receiving empty responses. Captured requests include `0a00`, `070101`, `8002f18d00`, and `8003f0226b00`, which are not the STX/ETX protocol accepted by the SPIKE 3 decoder. A separate `Spike2Bridge` parser/encoder must translate that wire protocol to the shared model. Identical node firmware is not proof of identical CPU-side transport. The plaintext LPC firmware identified by PLAN.md is an investigation input, not an implemented adapter.
|
||||
|
||||
## Verification executed and remaining evidence
|
||||
|
||||
Executed without starting or modifying emulator services, with `PYTHONDONTWRITEBYTECODE=1`:
|
||||
|
||||
- Canonical `python3 -m unittest test_topper_protocol test_topper_topology test_magikarp_topology`: **17 cases, 16 passed, 1 skipped** with the explicit reason `ignored exact game image not present`. This verifies fixture-backed protocol/topology behavior, not the skipped extraction gate.
|
||||
- Verstack `python3 -m unittest test_game_startup test_runtimes test_bundle_builder test_bundle_selection test_bundle_storage`: **19 passed**.
|
||||
- Read-only canonical import/export check and the constructor incompatibility reproduction described above.
|
||||
- Compared all 123 manifest hashes to pinned Git objects, current canonical files, and vendored copies; canonical source files matched, with the 11 bundle adaptations listed above.
|
||||
|
||||
Additional existing gates are `emulator/test_uart_wakeup.py`, `test_shim_abi.py`, `test_server.py`, `tests/ui-emulator.mjs`, canonical `emulation/machine/test_machine.py`, and canonical netbridge/playfield suites. They include native compilation, temporary sockets/processes, or diagnostic sessions and were not run in this audit.
|
||||
|
||||
Historical runtime evidence remains bounded: `docs/emulation.md` records 1,848 King Kong frames over 210 seconds and 463 actual Pokémon frames with approximately 3.9 MiB PCM, while Pokémon still displayed “Startup In Progress.” The bundle README reports user-observed Mac Pokémon operation at the pinned revision, without identifying an exact acceptance-tested export. None proves this future unified source tree, SPIKE 2 bridge, or complete game behavior. After the source refactor, re-run fixed-input captures and the same-version self-consistency canary before trusting cross-version behavioral diffs.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Frame boundary and deterministic replay contract
|
||||
|
||||
Initial source inspection (2026-09-16), before the presentation repair, showed that
|
||||
`drmModePageFlip` calls `copy_gl_pixels_to_fbdev`, reading the current GL framebuffer
|
||||
and writing `/dev/fb0` in place. The callback in `drmHandleEvent` sleeps for
|
||||
16,666,667 host nanoseconds, reads host `CLOCK_MONOTONIC`, then increments
|
||||
`fake_fb_counter`. That counter is also used to allocate GBM handles. The worker
|
||||
samples the mutable framebuffer independently. None of these is a deterministic
|
||||
guest frame boundary, and the current canary's per-second SHA256 samples must not
|
||||
be described as PLAN's `frames.jsonl(seq, virtualtime, blake3)` result.
|
||||
|
||||
## Producer-side frame journal
|
||||
|
||||
The worker now resolves submitted DRM framebuffer IDs to retained BO storage and
|
||||
matching imported textures in the current EGL context. It captures through a
|
||||
private read FBO and restores GL pack/read bindings; CPU-backed buffers use their
|
||||
retained storage. Texture deletion/context destruction clear associations. The
|
||||
red/blue selected-buffer regression fails before repair and passes on ARM32 and
|
||||
ARM64, alongside stale-ID/pending-event tests. Native DRM remains passthrough.
|
||||
Flip callbacks now have a sequence separate from GBM handles, but still use host
|
||||
time. The framebuffer file is still mutable and lacks a frame journal.
|
||||
|
||||
Remaining work (step 1 is implemented for the tested worker path):
|
||||
|
||||
1. **Implemented and tested:** Track framebuffer ID to its actual GBM buffer at DRM add/remove operations.
|
||||
Resolve the `fb_id` submitted to page flip, rather than assuming the currently
|
||||
bound GL read framebuffer is the buffer being presented. Test alternating
|
||||
framebuffer IDs while another FBO remains bound, and reject unknown IDs.
|
||||
2. Copy the selected completed pixels into a stable, canonical top-down XRGB8888
|
||||
buffer (record width, height and stride in the run manifest). Give presentations
|
||||
their own monotonic sequence, unrelated to GBM object allocation.
|
||||
3. Hash that stable buffer with BLAKE3 before publishing it. Append exactly one
|
||||
journal record per accepted presentation: `seq`, scheduler-supplied
|
||||
`virtualtime` in integer nanoseconds, and `blake3`. Publish the matching image
|
||||
and sequence atomically, or through a sequence-checked shared-memory protocol;
|
||||
the browser may drop frames but the journal must not hash torn samples.
|
||||
4. Preserve multiple queued presentations and invoke each callback once. Tests
|
||||
should cover duplicate-looking images, pending flip rejection/queue policy,
|
||||
stale IDs, short writes and capture consumers reading during publication.
|
||||
|
||||
This alone provides reliable observation, not deterministic execution.
|
||||
|
||||
## Clock and input boundary
|
||||
|
||||
A replay coordinator must own guest virtual time and the cabinet model's event
|
||||
queue. A script specifies input changes at `(frame sequence, virtual offset)`;
|
||||
changes due at a boundary are applied to the canonical CPU switch bank and UART
|
||||
queue before releasing the guest to the next interval. The coordinator records
|
||||
an acknowledgement identifying the first permitted guest observation. Replace
|
||||
wall-clock switch pulse timers, EOS timers and peripheral periodic replies with
|
||||
that same virtual event queue in replay mode. Keep ordinary interactive mode
|
||||
separate.
|
||||
|
||||
Guest time must agree across clock/time/gettimeofday calls, vDSO paths, timerfd,
|
||||
poll/select, condition waits, sleeps and device completion timestamps. Setting
|
||||
page-flip callback time to `seq / 60` while the rest of the guest uses host time
|
||||
would invent a clock and does not meet this requirement. QEMU user-mode plus a
|
||||
preload hook does not by itself control direct syscalls, scheduling or vDSO reads.
|
||||
The implementation therefore needs either an explicitly validated restricted
|
||||
execution profile with every used timing path intercepted, or a QEMU system-mode
|
||||
record/replay configuration with controlled devices and instruction-based time.
|
||||
The latter is a separate runtime integration, not a flag available on the current
|
||||
user-mode worker.
|
||||
|
||||
Freeze the executable/assets, NVRAM initial state, canonical bridge profile,
|
||||
random sources, decoder thread configuration and virtual peripheral schedule.
|
||||
Start acceptance with two runs of the same version and the same fixed input
|
||||
script that advance beyond startup. Require matching producer frame journals,
|
||||
input acknowledgements and deterministic audio sample boundaries. Only after that
|
||||
self-check passes should a cross-version divergence be attributed to the game.
|
||||
The current static splash/Startup In Progress captures and silent PCM do not pass.
|
||||
@@ -0,0 +1,290 @@
|
||||
# Isolated canonical worker canary — 2026-09-16
|
||||
|
||||
This records actual ARM64 Pokémon LE 0.83.0 execution through the updated Linux worker, not the synthetic stream diagnostic. It does not establish gameplay acceptance or deterministic cross-version behavior comparison.
|
||||
|
||||
Latest outcome: HEVC corruption was traced to reused software-GBM file descriptors closing the archive, and fixed. Software video pixel storage/import was separately repaired and verified on both ABIs. Submitted-framebuffer capture is now verified, and the latest isolated run shows the logo / Startup In Progress with clean video packets. PCM is already silent at the game’s write boundary; startup remains open. See the follow-up sections below.
|
||||
|
||||
The runtime is archive snapshot `1314f21f-4f8f-4ed8-a1b3-cdc6a0505ea1`, mounted read-only. Its original generation field is blank. Only the private canary manifest was explicitly annotated `spike3`, supported by the game's ARM64 ELF and archived `spike3/netbridge/netbridge.elf`, RP2040 configuration and power-distribution firmware layout. The original manifest, runtime files, canonical source checkout, live worker, services and live sessions were not changed.
|
||||
|
||||
Each pair ran sequentially in a disposable offline container using existing image `sha256:bd7e14143ca08775898327cff0632d59ceb92169548346fabb22b6384fcf28b5`, frozen source, 3 CPUs, 6 GiB RAM and private 2 GiB session tmpfs. It reused the approved worker AppArmor/seccomp/capability configuration, published no ports and made no archive/API calls. `scripts/emulator_canary.py` starts the actual `server.Session`/runner and saves JPEGs, raw-frame SHA-256 samples, PCM, input replies, machine state and logs before stopping the private process group.
|
||||
|
||||
Evidence is retained untracked under `data/emulator/canaries/20260916-canonical/`; `environment.json` records image, source and generation-evidence hashes. Game media remains outside the tracked source tree.
|
||||
|
||||
## Fixed-input method and baseline
|
||||
|
||||
For each fresh private state, the sequence is relative to the first observed nonblack frame: service Enter down/up at 10/10.5 seconds, service Up at 20/20.5, service Back at 30/30.5, Start at 40/40.5. Raw framebuffer hashes are sampled once per second. These are repeatable **wall-clock input instructions**, not a deterministic guest virtual clock or seeded game. The self-consistency comparison measures equality; it does not assume it.
|
||||
|
||||
The 60-second baseline pair produced:
|
||||
|
||||
| Observation | Run 1 | Run 2 |
|
||||
|---|---:|---:|
|
||||
| First actual frame | 37.89 s | 37.21 s |
|
||||
| Captured JPEG updates | 641 | 631 |
|
||||
| Raw-frame samples | 60 | 60 |
|
||||
| Unique raw frames | 2 | 2 |
|
||||
| Captured PCM bytes | 6,574,080 | 6,435,840 |
|
||||
| Nonzero PCM bytes | 0 | 0 |
|
||||
| Input RPC errors | 0 | 0 |
|
||||
|
||||
Captured-update counts are server sampling counts, **not unique guest frames or measured game FPS**. Both last images showed “Startup In Progress.” Only 21/60 aligned raw-frame hashes matched; PCM lengths/hashes differed. This fails the same-version deterministic self-consistency gate. All captured PCM was silence, so these bytes prove transport activity, not audible game sound.
|
||||
|
||||
Broker input changes were real: service Enter toggled SPI bank `3ffcffffffffffff` → `37fcffffffffffff`, service Up → `2ffcffffffffffff`, service Back → `3fbcffffffffffff`, Start → `3efcffffffffffff`, with releases restoring the prior bank. Guest SPI transfer counts continued increasing. This proves delivery to the simulated bank; no visible gameplay response was established.
|
||||
|
||||
Both baseline game logs report fatal error 256: `NVMigration: create_current_map_file created an invalid or mismatched file?!?`. Private filesystem observation found `.crc32` modes such as `0210` and data modes `0300`/`0500`, missing owner read or write permission. The prior shim only supplied owner permissions when **both** bits were absent. The source fix now ensures both owner read/write bits on creation beneath `/data/nv/`; ordinary paths retain their requested modes. A separate status fix prevents an already-rendered splash frame from hiding a logged fatal error.
|
||||
|
||||
## Cross-architecture gates
|
||||
|
||||
The existing container does contain ARM32/ARM64 GCC, QEMU and Mesa even though the host lacks these tools. In disposable offline containers:
|
||||
|
||||
- `test_shim_abi`: passed on ARM32 and ARM64, including `dlsym`, `mmap64` and real software-rendered pixels.
|
||||
- `VERSTACK_TEST_TRIPLET=arm-linux-gnueabihf python3 -m unittest -v test_device_profiles test_uart_wakeup`: 6 passed.
|
||||
- The same six tests with `aarch64-linux-gnu`: 6 passed.
|
||||
- UART probes explicitly create/reopen private NVRAM modes `0000`, `0200`, `0400`, `0210`, `0300`, `0410`, `0500`, checking actual owner read/write bits. Device probes cover both platform tables, four open variants, SPI/I2C/UART ioctls and unknown-path passthrough; UART replies wake blocked poll/select.
|
||||
|
||||
QEMU's unknown ioctl returns `ENOSYS` where the native host returns `ENOTTY`. The passthrough test compares against the actual raw syscall result/errno, not an architecture-specific hardcoded errno. A native ordinary-file test additionally verifies that the NVRAM fix does not change mode `0200` outside `/data/nv/`.
|
||||
|
||||
## Post-fix verification
|
||||
|
||||
The first post-fix attempt removed the observed NVRAM fatal and produced owner-readable/writable modes (`0610`/`0700`), but its 60-second observation still showed the logo with silent PCM. A harness cleanup race closed the PCM file before cancelling its producer, aborting between runs. That partial attempt is preserved separately as `nvfix-harness-failure`; it is not counted as a completed pair. The harness now cancels capture before closing its output.
|
||||
|
||||
The corrected 120-second pair completed:
|
||||
|
||||
| Observation | Run 1 | Run 2 |
|
||||
|---|---:|---:|
|
||||
| First actual frame | 40.97 s | 37.75 s |
|
||||
| Captured JPEG updates | 1,083 | 1,024 |
|
||||
| Raw-frame samples | 120 | 120 |
|
||||
| Unique raw frames | 1 | 2 |
|
||||
| Captured PCM bytes | 13,190,400 | 12,349,440 |
|
||||
| Nonzero PCM bytes | 0 | 0 |
|
||||
| Input RPC errors | 0 | 0 |
|
||||
| Guest SPI transfers at final state | 42,858 | 41,755 |
|
||||
| Observed private NVRAM files | 18 | 26 |
|
||||
| NVRAM files missing owner read/write | 0 | 0 |
|
||||
|
||||
Neither run produced the prior fatal game log. Run 1 retained the Pokémon logo; run 2 showed “Startup In Progress..”. Both console logs contain HEVC invalid-NAL-size/input-splitting errors and decoded-frame notices. No audible audio, visible gameplay or working game-level input response was demonstrated. Only **26/120** aligned raw-frame samples matched and PCM lengths/hashes differed, so deterministic same-version replay remains unproven. Fixing the observed NVRAM failure did not complete startup acceptance.
|
||||
|
||||
See `nvfix-repeat/summary.json`, each run's `record.json`, `last.jpg`, `console.log`, `machine-service-0.log` and PCM capture beneath the evidence directory. `baseline/` and `nvfix-harness-failure/` remain separate; none was overwritten. The source/status fix and harness shutdown fix have their own targeted tests; the completed runtime pair used frozen source recorded in `inputs.json` and `environment.json`.
|
||||
|
||||
Next investigation should distinguish the video packet/decoder/capture failure from cabinet-startup waiting, then determine why the PCM producer is silent. The observed logs do not prove asynchronous UART delivery is the sole startup blocker. Before any cross-version behavioral conclusion, use controlled guest timing/state or otherwise establish a nontrivial same-version self-consistency baseline. Static identical splash frames would not satisfy gameplay acceptance either.
|
||||
|
||||
## Reproduction
|
||||
|
||||
`scripts/run_emulator_canary.sh` stages frozen source and launches the same bounded offline container policy. It requires a **new** output directory and explicit generation evidence; it never changes the source runtime manifest or invokes a live worker. The wrapper passed shell syntax checking; the recorded runs used the equivalent explicit Docker invocation with the Python harness.
|
||||
|
||||
```sh
|
||||
bash scripts/run_emulator_canary.sh \
|
||||
"$PWD/data/emulator/runtime/1314f21f-4f8f-4ed8-a1b3-cdc6a0505ea1" \
|
||||
"$PWD/data/emulator/canaries/new-verification" spike3 \
|
||||
'Archive snapshot with ARM64 game and verified SPIKE3 RP2040/netbridge/powerdist firmware layout' 120
|
||||
```
|
||||
|
||||
The output includes game-derived media and is intentionally not committed. This canary does not start a production session, change service configuration, rebuild a release image or modify the canonical repository's unrelated dirty VPX file.
|
||||
|
||||
## Video diagnosis: exact embedded stream and pre-decoder corruption
|
||||
|
||||
Additional isolated runs enabled a bounded packet trace in the worker's video shim.
|
||||
The trace is opt-in (`VERSTACK_EMULATOR_VIDEO_TRACE=1` on the worker process), writes
|
||||
only private run artifacts, and records packets *before* `avcodec_send_packet`.
|
||||
It caps packet data at 16 MiB, 32 decoder contexts and 256 packets per context;
|
||||
extradata is capped at 64 KiB per context. The source profile contains the patch;
|
||||
upstream and bundle profiles remain unchanged.
|
||||
|
||||
The first 120-second diagnostic reached decoding but the harness looked in the
|
||||
wrong artifact directory. That run supplies logs, not packet evidence. Correcting
|
||||
the copy path to `emulation/work/runs/*/video-*` was necessary because that run
|
||||
directory is bound to `/run/spike3-emu/artifacts`. Separate 60/45-second diagnostics
|
||||
never reached codec initialization; absence of their packets does not establish
|
||||
an interception failure. The final 120-second diagnostic did capture decoder input.
|
||||
All attempts are retained separately beneath `video-diagnosis/` in the evidence
|
||||
folder. No live service or existing runtime was modified.
|
||||
|
||||
A first pass checked the 248 directly recognizable HEVC MP4 assets (41,418 packets):
|
||||
all NAL length boundaries were valid. The other 15 recognizable MP4s use H.264.
|
||||
That was a subset of game video, not proof about the active failing stream. The
|
||||
runtime's 1940×1100 dimensions are also documented by the donor and are not an
|
||||
ABI-corruption indicator.
|
||||
|
||||
Exact packet matching identified the active video in `assets/godot/main.pck`:
|
||||
|
||||
- Entry: `scenes/shared_items/gfx/bknd/Forest/Forest_01_22_26_h265_2.mp4`.
|
||||
- Absolute PCK offset: 83,819,312; size: 6,853,951 bytes.
|
||||
- PCK directory MD5 verifies against the extracted bytes.
|
||||
- SHA256: `296bee9e380b793bad7efa5e1a9fd7281046d9f42ff4892a6981adaf062f8445`.
|
||||
- Native FFmpeg decoded all 450 HEVC frames (1940×1100, 15 seconds) with exit 0
|
||||
and no diagnostic output.
|
||||
|
||||
The final trace captured 648 packets across four decoder contexts. Context 1's
|
||||
114 packets match the exact archived stream. Context 2 matches through packet 16;
|
||||
packet 17 diverges after byte 46,897. Packets 18–25 match complete archived byte
|
||||
ranges **131,000 bytes earlier** than their packet positions require. Further
|
||||
packets show repeated 131,000-byte displacement increments. Contexts 3 and 4 also
|
||||
contain shifted data. Overall, 145 packets match their expected archived ranges;
|
||||
500 already have invalid first-NAL lengths at the shim's send-packet entry.
|
||||
For example, context 2 packet 18 expects a NAL length of 906, but receives
|
||||
1,192,700,206, exactly the value subsequently reported by FFmpeg.
|
||||
|
||||
This rules out malformed original bytes for the identified clip and establishes
|
||||
corruption before decoding; post-decoder rendering or stale browser capture cannot
|
||||
cause these NAL errors. It points to the custom stream read/seek/buffer path, not
|
||||
to decryption of that unencrypted PCK entry. It does **not** identify which callback,
|
||||
ABI assumption or concurrency behavior causes the displacement. No speculative
|
||||
runtime I/O or codec fix was applied. The next bounded probe is to wrap each
|
||||
`avio_alloc_context` custom read/seek callback and record requested count, returned
|
||||
count, logical position and a bounded byte fingerprint, with decoder/thread IDs.
|
||||
Compare those results against the verified PCK entry and against single-decoder
|
||||
versus concurrent-decoder execution before changing behavior.
|
||||
|
||||
The final diagnostic produced 1,125 sampled frame updates and 13,762,560 PCM bytes;
|
||||
this is not gameplay or deterministic replay acceptance. Raw packets, logs, source
|
||||
hashes and comparison reports are in `video-diagnosis/trace-long/` and
|
||||
`video-diagnosis/analysis/`. The 16 focused source-profile, canonical-bridge and
|
||||
server tests passed; the actual ARM64 canary compiled and exercised the diagnostic
|
||||
shim. [The frame-boundary proposal](emulator-frame-boundary.md) separately explains
|
||||
why the current wall-clock capture cannot meet the deterministic journal contract.
|
||||
|
||||
After the final diagnostic counter-lock adjustment, `test_shim_abi` also passed in
|
||||
an offline worker-image container (one test covering both ARM32 and ARM64 build,
|
||||
QEMU ABI and graphics probes). Source replay verification passed again. The
|
||||
canonical checkout still has only its pre-existing dirty VPX file.
|
||||
|
||||
## AVIO follow-up: descriptor ownership caused the read failure
|
||||
|
||||
The subsequent custom-AVIO probe resolved the earlier open cause. Each custom
|
||||
reader requests a 131,000-byte buffer. The failing callback starts returning the
|
||||
same buffer repeatedly while claiming a full read. Stdio tracing shows why:
|
||||
|
||||
1. A decoded frame is released.
|
||||
2. The preload shim calls `close(34)` while that descriptor points to
|
||||
`assets/godot/main.pck`.
|
||||
3. `fread` first returns only 3,532 bytes, then zero, with `EBADF` and `ferror=1`.
|
||||
Its last position is 84,373,504, exactly a 4 KiB boundary.
|
||||
4. The game's custom reader ignores the short/error result and reports 131,000
|
||||
bytes. FFmpeg therefore consumes stale buffer contents as new NAL units.
|
||||
|
||||
The software GBM implementation returned its **cached backing descriptor** from
|
||||
`gbm_bo_get_fd`, although the caller owns an exported descriptor and closes it.
|
||||
`free_drm_frame` correctly closed that export; a pooled BO later reused the cached
|
||||
integer after the OS had assigned it to the PCK. Another frame release then
|
||||
closed the unrelated archive descriptor. The repair exports an independent
|
||||
`F_DUPFD_CLOEXEC` descriptor. BO destruction closes only the backing descriptor.
|
||||
Native GBM passthrough is unchanged.
|
||||
|
||||
The ownership regression fails on both ARM32 and ARM64 against the frozen old
|
||||
profile and passes after repair. It exercises multiple exports, closing one,
|
||||
reusing the closed number for an unrelated file, exporting again, and destroying
|
||||
the BO while exports remain open. The actual descriptor-fixed 120-second canary
|
||||
captured **164/164 packets matching the exact archived clip**, all with valid NAL
|
||||
boundaries, and recorded **zero invalid-NAL errors and zero stdio error flags**.
|
||||
It still had one unique sampled framebuffer and 13,562,880 all-zero PCM bytes;
|
||||
removing the corruption did not establish gameplay or audible output.
|
||||
|
||||
Evidence: `video-diagnosis/avio-trace/`, `stdio-before-fix/`, `descriptor-fixed/`
|
||||
and `analysis/`. An earlier AVIO build attempt lacked an unnecessary libavformat
|
||||
header and produced no guest frames; it is retained as `avio-compile-failure` and
|
||||
excluded from runtime conclusions. The final probe uses the stable opaque API
|
||||
signature. Its cross-ABI forwarding fixture checks original opaque pointers,
|
||||
callback bytes, errno, and unwrapped fallback after 32 slots; the real archived
|
||||
libavformat is exercised by the canaries.
|
||||
|
||||
## Software video pixel transfer
|
||||
|
||||
An independent regression found that the old software GBM map allocated a new
|
||||
zeroed buffer, and unmap freed the converted video pixels. EGL image import then
|
||||
ignored the supplied descriptor and allocated an empty texture. This was a second
|
||||
worker graphics defect, separate from the archive read failure.
|
||||
|
||||
The software path now stores BO pixels in a sized memfd, maps shared storage,
|
||||
exports independently owned descriptors, and keeps an EGL-owned reference for
|
||||
RGBA/BGRA video import. Texture binding uploads the actual rows while preserving
|
||||
GLES unpack state; RGBX variants receive opaque alpha. Invalid descriptors,
|
||||
unsupported formats, insufficient pitch/backing length and out-of-bounds mappings
|
||||
are rejected. Native graphics passthrough is unchanged.
|
||||
|
||||
The ARM32/ARM64 regression fails before this repair when it remaps written pixels.
|
||||
After repair, actual Mesa/GLES readback verifies the written color through
|
||||
GBM→exported descriptor→EGL image→texture, including closing the original export
|
||||
and destroying the BO before binding the EGL image. Descriptor, pitch, format and
|
||||
mapping-bound negative tests also pass. Runtime outcome is recorded separately
|
||||
below; these pixel tests alone are not game acceptance.
|
||||
|
||||
The pixel-fixed 120-second isolated canary completed with 199/199 captured packets
|
||||
matching the exact archived video, all valid NAL boundaries, zero invalid-NAL
|
||||
messages and zero stdio error flags. First capture arrived after 37.73 seconds;
|
||||
1,082 captured updates still contained only one unique raw framebuffer (the
|
||||
Pokémon logo), and all 13,128,960 PCM bytes were zero. The final JPEG was inspected.
|
||||
Thus both source defects are repaired and their targeted runtime/graphics contracts
|
||||
are verified, but game startup, game-level input response and audible audio remain
|
||||
unproven. No claim of gameplay or deterministic replay follows from these results.
|
||||
|
||||
`video-diagnosis/pixels-fixed/` preserves this run; `analysis/pixels-fixed-summary.json`
|
||||
records the comparison. Final cross-ABI graphics tests pass, including negative
|
||||
imports and mapping bounds, and six ARM64 device/UART regressions pass after the
|
||||
bounded diagnostic close logger. The worker profile replays all 47 files, the
|
||||
bundle profile replays 125, and all 129 upstream files still match the canonical
|
||||
pin. No live session, service, runtime manifest or canonical checkout was changed.
|
||||
|
||||
Next diagnosis should distinguish the actual presented framebuffer from the current
|
||||
GL read target, using the accepted DRM framebuffer ID and corresponding GBM/EGL
|
||||
object, then inspect remaining game startup waits and the PCM producer. The current
|
||||
page-flip capture still reads the current GL framebuffer rather than resolving
|
||||
`fb_id`. Runtime logs also show Godot resource/node errors; they have not yet been
|
||||
proved causal. A producer frame journal and controlled clock remain separate open
|
||||
requirements, as described in `emulator-frame-boundary.md`.
|
||||
|
||||
## Submitted framebuffer and application PCM boundary
|
||||
|
||||
A new cross-ABI regression reproduced incorrect presentation: red was submitted
|
||||
through its DRM framebuffer ID while a blue GL framebuffer remained bound, and
|
||||
both architectures captured blue. The repaired software path registers each DRM
|
||||
framebuffer's retained BO storage, associates imported textures with that storage
|
||||
and EGL context, and captures the submitted texture through a temporary read FBO.
|
||||
It restores read/pack bindings and does not change the application's draw FBO.
|
||||
CPU-backed buffers use their retained storage. Removing a framebuffer releases
|
||||
its reference; deleted textures and destroyed contexts clear associations.
|
||||
Unknown handles/stale IDs fail, and a second flip with an outstanding event returns
|
||||
`EBUSY`. Flip callback sequence is now separate from GBM handle allocation.
|
||||
Native DRM and graphics calls retain their passthrough path.
|
||||
|
||||
The regression now passes on ARM32 and ARM64, including alternating red/blue IDs,
|
||||
restored GL state, one callback per accepted event, independent sequence, pending
|
||||
flip rejection and stale-ID rejection. PCM interposition is separately opt-in and
|
||||
bounded: at most 16 stream records, at most 64 KiB inspected per accepted write,
|
||||
and a fixed-size periodic summary. Forwarding tests verify partial-write results,
|
||||
errno and counters with tracing off/on; production samples are never changed.
|
||||
|
||||
The subsequent isolated 120-second canary recorded:
|
||||
|
||||
- First captured frame after 39.34 seconds; 1,054 captured updates and two unique
|
||||
raw samples (logo and “Startup In Progress..”). Images were inspected.
|
||||
- All 193 captured video packets matched the exact archived stream; zero NAL errors.
|
||||
- At the application boundary, 15,040 successful `snd_pcm_writei` calls accepted
|
||||
3,008,000 S16 stereo frames at 44.1 kHz. All 12,032,000 inspected producer bytes
|
||||
were zero; there were no PCM write errors.
|
||||
- The 48 kHz capture likewise contained 13,025,280 all-zero bytes. Silence therefore
|
||||
originates before the capture/resampling path in this observed run.
|
||||
- Machine state reported no model faults, 42,044 SPI transfers, and ball lifecycle
|
||||
sequence 0. No gameplay or game-level response to the input script was demonstrated.
|
||||
|
||||
The remaining logged `SpiVideoStreamDecoderBase::Play while loop timeout` was
|
||||
located in the exact archived executable without modifying it. Its loop at
|
||||
`0x958e64–0x958e80` polls an atomic readiness byte at object offset `0xc`, up to
|
||||
250 iterations with `usleep(1000)`. The logging path returns success (sets `w19=1`
|
||||
at `0x9590d4` and returns through `0x958dd8`). It is a readiness warning, not by
|
||||
itself proof of fatal startup failure. No timeout extension, readiness bypass or
|
||||
binary patch was applied. The writer/consumer of that readiness state and the
|
||||
outer game startup state remain the next focused diagnostic boundary.
|
||||
|
||||
Evidence is retained in `data/emulator/canaries/20260916-canonical/presentation-diagnosis/`:
|
||||
`canary/` contains immutable source hashes, frames, PCM, producer counters and logs;
|
||||
`analysis/` contains before/after tests, exact summary, string-reference finder and
|
||||
bounded disassembly. This is still not a virtual-clock frame journal or same-seed
|
||||
self-consistency pass. The worker source profile now replays 49 files; the bundle
|
||||
and pinned upstream remain unchanged. No live service, session or archived source
|
||||
runtime was modified.
|
||||
|
||||
Final verification for this pass: the combined ARM32/ARM64 ABI, ownership,
|
||||
GBM→EGL pixels, selected-framebuffer and PCM counter suite passes; all four
|
||||
source-profile/tamper/standalone-packaging tests pass after updating the expected
|
||||
worker file count for the two new helper headers. Replay verification confirms
|
||||
49 worker files, 125 bundle files and 129 unchanged pinned upstream files.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Image similarity review
|
||||
|
||||
In a game's working tree, choose **Browse assets → Images → Group similar images
|
||||
→ Process version images**. Processing scans all current images in that version,
|
||||
independent of gallery pagination, folder/search filters and comparisons.
|
||||
|
||||
The browser decodes up to three images at a time and caches small fingerprints in
|
||||
IndexedDB. The first pass downloads each preview (or original if no preview
|
||||
exists); later passes reuse fingerprints keyed by immutable artifact and preview
|
||||
identity. No server analysis jobs are added. Cancel stops further downloads;
|
||||
changing versions or closing the review cancels processing. A browser worker
|
||||
performs grouping so the controls stay responsive. Very large collections with
|
||||
many distinct images can still take time: representative comparisons are
|
||||
quadratic in the worst case. Unreadable assets are listed, not silently omitted.
|
||||
|
||||
Fingerprints compare an 8×8 RGB layout after compositing transparency on neutral
|
||||
gray, with an aspect-ratio penalty. The displayed percentage is a visual score,
|
||||
not a probability or semantic classification. Small text, crops, translations,
|
||||
and subtle details can be missed. The default minimum is 92%; adjust it to split
|
||||
or combine groups without decoding again. Every member must meet that minimum
|
||||
against its representative; there are no transitive chains. Path ordering makes
|
||||
representative selection repeatable for a fixed collection.
|
||||
|
||||
Groups appear as a thumbnail grid, with one representative and a frame count per
|
||||
card. Up to 120 groups appear per page. Click a card to open its frames; **Back to
|
||||
groups** restores the grid page and scroll position. Smallest groups appear first
|
||||
to help inspect unusual screens. Within a group,
|
||||
weaker matches appear first. Open an image for detailed inspection, or choose
|
||||
**Find similar to this image** to rank the full processed version against it.
|
||||
Group frames and reference matches are paginated at 32 images per page. An
|
||||
isolated image receives 100% against itself, not an easter-egg confidence score.
|
||||
|
||||
Validation: `node tests/image-similarity.cjs` after the extension build covers
|
||||
score behavior, spatial/aspect differences, threshold membership and worker
|
||||
serialization. `node tests/ui-image-review.mjs` covers multiple index pages,
|
||||
unreadable files, reference matching, caching, threshold changes and cancellation
|
||||
in an isolated browser context. A live King Kong LE 0.97.0 check processed all
|
||||
157 indexed images into 71 groups at 92%, with zero unreadable images. This does
|
||||
not claim automatic identification of the cat photos or hidden D&D screens.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Import scratch admission (2026-09-16)
|
||||
|
||||
## Compressed wrapper correction (working tree, not deployed)
|
||||
|
||||
Real Star Wars SPIKE 3 import task `a526486a-d669-47bf-bde1-2bd1ac7ba652`
|
||||
retained its original, then extraction run `eb68c8af-22c8-4417-8dfd-352892aa485d`
|
||||
failed with `insufficient declared workspace for extraction`. The ZIP is
|
||||
3,569,550,802 bytes; its two uncompressed members total 3,984,588,800 bytes.
|
||||
The adapter requires four uncompressed footprints after subtracting the staged
|
||||
source. The deployed 17,179,869,184-byte allowance cannot satisfy that demand.
|
||||
|
||||
`src/scratch_plan.rs` now uses native bounded ZIP central-directory reads over
|
||||
the archived source. Wrapper admission requests staged source plus four expanded
|
||||
footprints plus 64 MiB, capped by the configured pool. Raw LUKS uses its source
|
||||
size; unsupported ZIP metadata conservatively requests the pool. Other plugins
|
||||
retain their ordinary allowance. This is admission accounting, not streaming
|
||||
extraction or a guarantee that every format fits the configured capacity.
|
||||
|
||||
`tests/zip_directory.rs` passed all four tests, including an explicit read-only
|
||||
check against both real Star Wars ZIPs. `tests/wrapper_admission.rs` passed a
|
||||
compressed 20 MiB fixture through an actual archive, direct plugin execution and
|
||||
durable queued execution. The plugin checked its remaining allowance against
|
||||
Python's independent ZIP expansion total; both runs completed and returned all
|
||||
scratch reservations. The real SPIKE 3 retry awaits deployment of this fix.
|
||||
|
||||
## Earlier input-sized admission change
|
||||
|
||||
The live catalog contains failed `import-extract` run
|
||||
`f5018937-2069-4f03-986d-ecc528f05fc6`, with the error `plugin exceeded its time or
|
||||
workspace budget`. Its input artifact is the archived `Stern Pokemon Pinball
|
||||
0.83.bin`; original snapshot `1406ea86-1fe1-4517-8f9b-ee620e7d0c62` has a logical
|
||||
size of 63,281,562,112 bytes. Evidence was queried using SQLite `mode=ro`, including
|
||||
the live WAL, without reading credential values. The recorded error combines timeout and workspace
|
||||
exhaustion, so it does not by itself distinguish which check killed that run. The
|
||||
independent admission-arithmetic defect is reproduced by the bounded fixture below.
|
||||
|
||||
The current 72 GiB pool, four analysis slots capped at 6 GiB and three preparation
|
||||
slots previously assigned each import 16 GiB, regardless of source size or idle
|
||||
capacity. Input materialization could consume at most 8 GiB. The extraction
|
||||
adapter further partitions the remaining allowance. Running an offline adapter
|
||||
with a larger allowance therefore exercised materially different limits.
|
||||
|
||||
Admission now reserves a source-sized allowance: four logical input footprints,
|
||||
with the old share as minimum and configured pool as maximum. The known 63 GB
|
||||
sparse image therefore requests 72 GiB and waits for idle capacity. Logical size
|
||||
is an estimate, not a hard rejection: the stage/materialization loops continue
|
||||
counting nonzero allocated writes, and plugin monitoring continues checking total
|
||||
workspace allocation. No live capacity setting was raised.
|
||||
|
||||
The reservation change also closes two safety holes:
|
||||
|
||||
- The former free-space check happened outside the reservation mutex. Two callers
|
||||
could both pass before either incremented it. Check and increment are now one
|
||||
atomic operation against both physical free space and configured pool capacity.
|
||||
- Uploads previously held only a worker permit during body transfer, entering the
|
||||
scratch ledger after receipt. Upload body transfer now holds a movable scratch
|
||||
reservation from the start; a large import cannot borrow those promised bytes.
|
||||
|
||||
Nested processing uses the actual admitted allowance, and all error/drop paths
|
||||
return reservations. Physical admission conservatively subtracts all outstanding
|
||||
promises even when some have already been written. This trades some concurrency
|
||||
for safety; it does not claim a perfect spent-versus-unspent scratch model.
|
||||
|
||||
Tests cover concurrent admission, configured and physical capacity boundaries,
|
||||
integer overflow, pending upload exclusion, larger-job exclusion, nested reuse,
|
||||
release on drop, direct/queued import and direct/legacy-queued plugin processing
|
||||
of a 6 MiB nonzero fixture that
|
||||
exceeded the old 5.33 MiB staging allowance with three preparation slots.
|
||||
|
||||
No full-size SD import was launched for this change. The real failing input now
|
||||
receives a larger feasible reservation when the pool is idle; successful complete
|
||||
extraction of that image remains unverified. Highly expanding compressed inputs
|
||||
may still exceed the four-footprint estimate and fail safely. Streaming unwrap
|
||||
and per-format precise peak-scratch estimates are separate remaining work.
|
||||
|
||||
Verified commands on the changed tree:
|
||||
|
||||
```
|
||||
cargo test --locked --offline --lib execution::tests
|
||||
# 4 passed
|
||||
cargo test --locked --offline --test imports --test catalog_jobs --test archive
|
||||
# imports: 9 passed; catalog_jobs: 12 passed; archive: 13 passed
|
||||
```
|
||||
|
||||
The new regression exercised original retention in direct and durable-import
|
||||
paths, then plugin processing in direct and legacy-job paths. It verified the
|
||||
plugin received more than 16 MiB of remaining allowance after restoring the
|
||||
6 MiB input, both queued paths completed, and no scratch files remained.
|
||||
@@ -0,0 +1,75 @@
|
||||
# Import decoder graph and shared input
|
||||
|
||||
An import now persists its decoder `TaskGraph` in `imports.payload.decoder_graph`.
|
||||
The activity API includes the graph. It contains concrete operations and source
|
||||
references:
|
||||
|
||||
```
|
||||
retained/extracted filesystem ─┬─ godot-scripts
|
||||
├─ media-extract ─ media-preview
|
||||
└─ spike2-audio
|
||||
```
|
||||
|
||||
Configured optional operations determine the nodes. Original preservation and
|
||||
extraction precede this graph; per-program Ghidra analysis uses its separate
|
||||
durable job group. The graph is not a claim that all archive work has been moved
|
||||
into one general scheduler.
|
||||
|
||||
`Task` records operation, snapshot/dependency input, state, attempt count,
|
||||
durable output and error. Validation rejects duplicate nodes, missing inputs,
|
||||
cycles, unsupported graph versions, invalid output/state combinations and wrong
|
||||
media-preview dependencies. Only ready tasks start. Successful outputs are saved
|
||||
individually, so a failed script decoder does not discard completed audio, media
|
||||
or preview results. Retry restores those output references and runs only missing
|
||||
branches. Completed graph outputs survive recovery; unfinished attempts may run
|
||||
again after a crash. Publication and parent graph updates are not an exactly-once
|
||||
transaction across the archive backend and catalog.
|
||||
|
||||
When two or more incomplete independent decoders use the filesystem snapshot,
|
||||
the backend stages it once and shares that tree. Files/directories have write bits
|
||||
removed; trusted plugin code receives the common `input_dir`, with separate
|
||||
output/tmp directories. This is a filesystem protocol, not streaming decoder I/O
|
||||
or OS sandbox isolation. Media preview has a different source and stages its
|
||||
media output normally. Shared input therefore removes two filesystem-tree
|
||||
materializations when all three independent consumers run; it does not eliminate
|
||||
original capture, extraction staging, or preview input staging.
|
||||
|
||||
Staging first promises the largest child CPU/process allowance and full scratch
|
||||
estimate. This prevents several retained inputs from filling the headroom needed
|
||||
to launch any of their children. Once materialization is verified, its
|
||||
reservation transfers to a shared-input lease charged for allocated scratch/RAM,
|
||||
with no CPU promise or runnable task slot. Child budgets subtract those already
|
||||
charged input bytes. Impossible combined requests fail rather than wait while
|
||||
retaining their own source. Maintenance acquires every runnable permit and checks
|
||||
shared leases; transfer keeps its original permit until the shared count is
|
||||
visible. The final lease deletes the staged tree before returning its bytes.
|
||||
This also works with one task slot and one CPU: branches run sequentially over the
|
||||
same retained input. Temporary shared allocation counts are conservative.
|
||||
|
||||
Each child inherits parent import cancellation attribution. Finishing one child
|
||||
cannot clear cancellation for its siblings. The parent joins all children before
|
||||
releasing graph state and shared input. A claimed graph node can wait for resource
|
||||
admission; the governor's actual reservation counters distinguish that from an
|
||||
executing process.
|
||||
|
||||
## Verification
|
||||
|
||||
`cargo test --offline --locked --test import_dag` covers four integration tests:
|
||||
|
||||
- Three barrier-held processes simultaneously observe the same source pathname
|
||||
and inode, with write bits absent. Preview starts after media publication while
|
||||
scripts and audio are still held. All five stage outputs (including extraction)
|
||||
are retained and the graph completes.
|
||||
- Cancellation stops every active child and releases shared files/counters.
|
||||
Reopening with one CPU and one task slot and retrying completes without a lease
|
||||
deadlock.
|
||||
- A deliberately failed branch can be retried while successful sibling output
|
||||
snapshot IDs and invocation counts remain unchanged.
|
||||
- Two concurrent imports under a tight memory ceiling wait to stage while a
|
||||
competing process occupies their launch headroom, then both complete after it
|
||||
releases. They cannot accumulate source leases that leave neither child runnable.
|
||||
|
||||
Graph unit tests cover dependency ordering, serialized recovery, cancellation,
|
||||
retry, missing dependencies, cycles and output invariants. A resource unit test
|
||||
checks that retained input owns no CPU/task slot while still excluding maintenance.
|
||||
Existing import and queue concurrency regressions remain part of the gate.
|
||||
+31
-48
@@ -1,59 +1,42 @@
|
||||
# Import and analysis scheduling
|
||||
|
||||
The service runs one import/extraction worker and two Ghidra analysis workers by
|
||||
default. Set `analysis_workers` in the service configuration to an integer from
|
||||
1 through 8 to change the analysis limit. An omitted setting means 2.
|
||||
`preparation_workers` controls concurrent imports, extraction, and media decoding
|
||||
(1 through 8; defaults to 1 for compatibility with small workspaces). This host
|
||||
is configured for 3 preparation workers. Settings take effect on service restart.
|
||||
The working tree uses one shared resource pool, defaulting to 40 task permits.
|
||||
`analysis_workers` remains the explicit Ghidra concurrency cap (1–8, default 2).
|
||||
`resource_limits` governs total tasks, CPU, memory and publication headroom;
|
||||
`preparation_workers` remains only for legacy upload budget arithmetic. These new
|
||||
settings require a verified service deployment; the first deployed batch still
|
||||
uses the older fixed-pool scheduler.
|
||||
|
||||
An import retains its original, extracts files, and prepares requested assets on
|
||||
the import worker. It then saves an `analysis_input` snapshot reference and enters
|
||||
`Waiting for code analysis`. This durable handoff frees the import worker before
|
||||
Ghidra starts. Analysis workers claim these tasks independently. A queued handoff
|
||||
survives restart; retrying an interrupted task reuses its saved original and stage
|
||||
outputs. Active Ghidra computation itself must restart after interruption.
|
||||
Workers alternate import and job dispatch across both task classes. An input/tool
|
||||
pair may run independent Ghidra programs concurrently, with one durable leaf job
|
||||
per program. Coordinators report the complete output list, never the last child
|
||||
as if it represented all programs. See [program-analysis-jobs.md](program-analysis-jobs.md)
|
||||
for cancellation, retry, old-queue conversion, and per-program publication.
|
||||
`processing_enabled: false` pauses automatic jobs; explicit import-owned analyses
|
||||
remain authorized by their import request. Direct processing reserves resources
|
||||
too. Multi-program synchronous Ghidra calls must use the durable job endpoint.
|
||||
|
||||
Legacy jobs use the same execution slots. Each worker alternates between imports
|
||||
and legacy jobs when both have work, with job priority applied within its lane.
|
||||
`processing_enabled: false` pauses legacy jobs and automatic submission; explicitly
|
||||
requested imports and their analyses still run. Jobs with the same input and tool
|
||||
do not execute concurrently. Direct API/CLI processing also reserves a slot.
|
||||
|
||||
External analysis and file preparation run outside the archive writer lock.
|
||||
Archive commit, readback verification, manifest publication, and indexing remain
|
||||
serialized. Deletion and garbage collection reserve all execution slots and reject
|
||||
active or pending work. Progress and cancellation are attributed to each import's
|
||||
worker thread, and shutdown interrupts every active plugin process group.
|
||||
Original retention, archive commit, readback, and publication remain serialized.
|
||||
Plugin computation runs outside the writer lock. Maintenance excludes active work;
|
||||
shutdown/cancellation interrupts the relevant plugin process group. Completed
|
||||
originals and outputs survive retry and restart.
|
||||
|
||||
## Resource allowances
|
||||
|
||||
Scratch capacity is partitioned, rather than giving each concurrent operation the
|
||||
entire configured `workspace_bytes` allowance:
|
||||
See [shared-resource-admission.md](shared-resource-admission.md) for the exact
|
||||
configuration and measured admission rules. Scratch is reserved atomically;
|
||||
small tasks estimate from input size instead of claiming a fixed lane share.
|
||||
Large imports can borrow configured capacity. Ghidra heap/CPU reservations use
|
||||
its frozen job settings. Upload reservations precede receiving bytes, with the
|
||||
existing upload ceiling preserved. Reservations are planning bounds, not hard
|
||||
OS isolation; runtime scratch checks remain in force.
|
||||
|
||||
- Each analysis gets `min(6 GiB, workspace_bytes / (analysis_workers + 2))`.
|
||||
- Each import/extraction slot gets the remainder after reserving every analysis
|
||||
slot, divided by `preparation_workers`. Increasing this count reduces the
|
||||
maximum scratch allowance for an individual large image.
|
||||
- Input materialization uses at most half a slot's allowance. Original import
|
||||
staging also uses half, leaving room for downloaded or uploaded source bytes.
|
||||
- Plugin execution and output capture are checked against that slot's allowance.
|
||||
|
||||
For a 64 GiB workspace and two analysis workers, this gives 6 GiB per analysis and
|
||||
52 GiB for import/extraction. These are scratch allowances, not JVM heap limits or
|
||||
OS memory isolation. Ghidra's configured `max_heap` and `max_cpu` apply separately
|
||||
to each analysis process. Trusted plugins are monitored for excess scratch use;
|
||||
the monitor is not a hard filesystem quota. Reserve host RAM for JVMs and other
|
||||
processes in addition to scratch, and limit individual extraction tools' threads
|
||||
when CPU contention warrants it.
|
||||
|
||||
`GET /api/info` exposes `analysis_workers`, `preparation_workers`, `analysis_workspace_bytes`, and
|
||||
`import_workspace_bytes`. Existing activity endpoints show running imports and
|
||||
the `Waiting for code analysis` handoff without a frontend upgrade.
|
||||
|
||||
The synchronous `work_import_one()` helper remains available for callers that
|
||||
want to prepare an import and attempt its analysis in one call. The service uses
|
||||
separate preparation and analysis lanes so a long analysis cannot hold up imports.
|
||||
The sampler reports CPU tick deltas, effective cgroup-v2 RAM/headroom, pressure,
|
||||
and free scratch. Admission accounts for pending publications and uses pressure
|
||||
hysteresis. Startup checks filesystem type, not availability of the entire pool.
|
||||
`GET /api/governor`, `/api/info`, and the Jobs toolbar expose shared capacity and
|
||||
its most recent admission constraint. The import TaskGraph/shared-input work is
|
||||
still under implementation and is not implied by removing static pools.
|
||||
|
||||
## Verification
|
||||
|
||||
|
||||
+70
@@ -0,0 +1,70 @@
|
||||
# Archive MCP surface
|
||||
|
||||
Run `target/release/verstack --config config.json mcp` with the backend already
|
||||
running. The official `rmcp` SDK serves newline-delimited JSON-RPC over stdio;
|
||||
stdout is reserved for the protocol. The process calls the configured local HTTP
|
||||
API and does not open the archive, so it can coexist with the backend's instance
|
||||
lock. This uses the SDK's [stdio and tool router APIs](https://github.com/modelcontextprotocol/rust-sdk).
|
||||
|
||||
Example client configuration:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"verstack": {
|
||||
"command": "/home/jordan/verstack/target/release/verstack",
|
||||
"args": ["--config", "/home/jordan/verstack/config.json", "mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The tools are `search_assets`, `get_asset`, `fetch_bytes`,
|
||||
`fetch_decoded_asset`, `diff_versions`, `search_strings`, `search_symbols`,
|
||||
`fetch_disassembly`, `fetch_xrefs`, `propose_name`, and `propose_finding`.
|
||||
Searches treat query text literally. Asset and symbol results are paginated;
|
||||
`diff_versions` accepts `page` and `kind`, returning 100 changes per page plus
|
||||
`changes_total` and full release counts. `get_asset` includes paginated name
|
||||
provenance, proposed findings and outgoing edges for readback.
|
||||
byte reads are capped at 1 MiB and return base64 plus SHA-256 evidence. Decoded
|
||||
fetches require an existing decoded observation and never substitute original
|
||||
bytes. Code tools read saved analysis and do not start Ghidra.
|
||||
|
||||
`search_strings` currently scans a caller-selected file range for printable ASCII
|
||||
strings. It reports its scanned bounds and possible boundary truncation. It is
|
||||
not yet a corpus-wide normalized code-string index; the Phase 5 `code_strings`
|
||||
schema/backfill remains required. This limitation is also in tool discovery.
|
||||
|
||||
The HTTP endpoint `GET /api/code/strings/coverage` reports durable denominators
|
||||
for the native reference-string projection: saved evidence, reference rows,
|
||||
indexed identity/literal-hash rows, and a bounded per-program breakdown. Opening
|
||||
the catalog replays legacy `code_references` rows into this projection
|
||||
idempotently. This is accounting evidence only; it does not claim executable
|
||||
decode completeness, semantic string validity, accepted names, or correspondence.
|
||||
|
||||
Proposals require an exact logical asset, observation snapshot/path, archived
|
||||
artifact identity, and nonempty byte range of at most 64 KiB with its SHA-256.
|
||||
The backend verifies membership and reads/hashes the cited bytes before storing
|
||||
anything. Names are appended as `T4_ai`/`proposed`; findings are `proposed`.
|
||||
Neither accepts a name, changes `best_name`, nor validates the claim's meaning.
|
||||
Repeating the same proposal is idempotent. The HTTP write routes retain the
|
||||
existing client-header and write-serialization boundary.
|
||||
|
||||
Verification: `cargo test --locked --offline --test mcp` tests evidence rejection,
|
||||
append-only proposal idempotence, non-promotion, MCP initialization/discovery,
|
||||
tool calls against a temporary running backend, bounded bytes, literal asset and
|
||||
symbol search, string scanning, and saved disassembly/references. Socket fixtures
|
||||
need the same local access as the repository's other integration tests. Deployment
|
||||
and a real archive protocol smoke test are separate gates.
|
||||
|
||||
## Explicit headless-agent accounting
|
||||
|
||||
`verstack agent prompt.txt --estimate-usd 1.25` produces a dry-run launch plan
|
||||
without contacting a provider or writing a file. The optional `--execute` flag is
|
||||
the explicit operator gate: it runs `claude -p <prompt>` (or the executable named
|
||||
by `--executable`) and appends a synced JSONL reservation/result record to
|
||||
`--ledger`. Before either mode, the launcher rejects non-finite estimates and
|
||||
checks both `--per-run-ceiling-usd` and `--daily-ceiling-usd` against prior
|
||||
reservations for the current UTC day. A malformed ledger fails closed. No API
|
||||
key, provider SDK, or network request is handled by this wrapper; credentials
|
||||
remain the provider's concern.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Append-only name claims
|
||||
|
||||
Schema 14 replaces the uniqueness rule on `(lak,name,source)` with uniqueness over
|
||||
the entire claim: LAK, name, tier, source, evidence and status. NULL and empty
|
||||
evidence remain distinct. Existing IDs, timestamps and claims are preserved.
|
||||
SQLite triggers reject UPDATE and DELETE on `asset_names`.
|
||||
|
||||
`POST /api/spine/names` defaults omitted status to `proposed`. A verified importer
|
||||
must explicitly send `"status":"accepted"`; only `proposed` and `accepted` are
|
||||
accepted by this endpoint. Structural/unknown tiers remain proposed regardless of
|
||||
the requested status. The existing recovered-name import scripts now request
|
||||
acceptance explicitly; their default dry-run behavior is unchanged.
|
||||
|
||||
Repeating the same claim is idempotent. Revising evidence or tier appends a new
|
||||
claim. Accepting a proposal appends its accepted form and retains the original
|
||||
proposed row. Best names are recomputed for touched assets from accepted claims,
|
||||
with strongest tier first and insertion ID as a stable tiebreaker. MCP proposals
|
||||
still always append as proposed and never promote names. This does not implement
|
||||
claim revocation or supersession; those need explicit historical relationships,
|
||||
not mutation of an old row.
|
||||
|
||||
Verification: two name-provenance tests cover default proposal, explicit
|
||||
acceptance, evidence/tier history, exact retry, NULL/empty evidence, structural
|
||||
rejection, SQL mutation rejection and schema-13 migration/reopen. Existing MCP
|
||||
and signature-storage tests also passed (five tests total).
|
||||
|
||||
A disposable copy of the rehearsed schema-13 catalog migrated in **2.918 s**.
|
||||
All **8,366** original name rows retained their complete contents, IDs and
|
||||
timestamps. Ordered JSON claim SHA-256 before/after:
|
||||
`028f6a19e04e19568b49f5224b2b6fd4ed4bcd67d26be941e833aae4298b16c9`.
|
||||
Name-table integrity check passed. Rehearsal helper:
|
||||
`examples/validate_name_migration.rs`; disposable copy:
|
||||
`data/validation/names-dryrun-20260916-1.sqlite3`.
|
||||
|
||||
This migration has **not** been deployed. The running first-batch service remains
|
||||
on schema 13 until the next verified release.
|
||||
@@ -0,0 +1,133 @@
|
||||
{
|
||||
"method": "verstack-code-anchors/1",
|
||||
"cohorts": {
|
||||
"got": {
|
||||
"input_sha256": "637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"verified_functions": 11519,
|
||||
"anchors": {
|
||||
"literal_constant": 26829,
|
||||
"string": 2
|
||||
},
|
||||
"independent_byte_address_and_relocation_verification": true,
|
||||
"truncated_functions": 0,
|
||||
"examples": [
|
||||
{
|
||||
"function": "_Z17def_award_martellv",
|
||||
"function_address": "39f10",
|
||||
"address": 3115368,
|
||||
"bytes_sha256": "1d7830363b6cc97f12a3ea1d9536174152e68734fc3e5e2ec98f16ba3ff90b69",
|
||||
"instruction_offset": 692,
|
||||
"kind": "string",
|
||||
"proof": "literal_pointer_used_as_memory_base",
|
||||
"value": "WXYZ[\\]^_`abcdefghi"
|
||||
},
|
||||
{
|
||||
"function": "tzset_internal",
|
||||
"function_address": "2246bc",
|
||||
"address": 2898444,
|
||||
"bytes_sha256": "b21df4cc4e54c6ce3c254c02f439fe4fc15e0cba3e23de366b06f0d332b589fb",
|
||||
"instruction_offset": 232,
|
||||
"kind": "string",
|
||||
"proof": "literal_pointer_used_as_memory_base",
|
||||
"value": "/etc/localtime"
|
||||
}
|
||||
]
|
||||
},
|
||||
"pokemon": {
|
||||
"input_sha256": "b1ef1b5585141a24f9997babff0738851807680439954a61b6e491ca7e76d87d",
|
||||
"verified_functions": 263,
|
||||
"anchors": {
|
||||
"import_slot": 40,
|
||||
"string": 155
|
||||
},
|
||||
"independent_byte_address_and_relocation_verification": true,
|
||||
"truncated_functions": 0,
|
||||
"examples": [
|
||||
{
|
||||
"function": "vsprintf",
|
||||
"function_address": "4aeb40",
|
||||
"address": 52448328,
|
||||
"bytes_sha256": null,
|
||||
"instruction_offset": 0,
|
||||
"kind": "import_slot",
|
||||
"proof": "aarch64_adrp_ldr_slot+undefined_dynamic_symbol_relocation",
|
||||
"value": "__stack_chk_guard"
|
||||
},
|
||||
{
|
||||
"function": "vsprintf",
|
||||
"function_address": "4aeb40",
|
||||
"address": 52448328,
|
||||
"bytes_sha256": null,
|
||||
"instruction_offset": 128,
|
||||
"kind": "import_slot",
|
||||
"proof": "aarch64_adrp_ldr_slot+undefined_dynamic_symbol_relocation",
|
||||
"value": "__stack_chk_guard"
|
||||
},
|
||||
{
|
||||
"function": "vsprintf",
|
||||
"function_address": "4aeb40",
|
||||
"address": 40110848,
|
||||
"bytes_sha256": "fb329000228cc5a24c264c57139de8bf854fc86fc18bf1c04ab61a2b5cb4b921",
|
||||
"instruction_offset": 636,
|
||||
"kind": "string",
|
||||
"proof": "aarch64_adrp_add_address",
|
||||
"value": "NULL"
|
||||
},
|
||||
{
|
||||
"function": "sprintf",
|
||||
"function_address": "4af990",
|
||||
"address": 52448328,
|
||||
"bytes_sha256": null,
|
||||
"instruction_offset": 0,
|
||||
"kind": "import_slot",
|
||||
"proof": "aarch64_adrp_ldr_slot+undefined_dynamic_symbol_relocation",
|
||||
"value": "__stack_chk_guard"
|
||||
},
|
||||
{
|
||||
"function": "sprintf",
|
||||
"function_address": "4af990",
|
||||
"address": 52448328,
|
||||
"bytes_sha256": null,
|
||||
"instruction_offset": 124,
|
||||
"kind": "import_slot",
|
||||
"proof": "aarch64_adrp_ldr_slot+undefined_dynamic_symbol_relocation",
|
||||
"value": "__stack_chk_guard"
|
||||
},
|
||||
{
|
||||
"function": "_ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEED1Ev",
|
||||
"function_address": "4baf20",
|
||||
"address": 52428656,
|
||||
"bytes_sha256": null,
|
||||
"instruction_offset": 0,
|
||||
"kind": "import_slot",
|
||||
"proof": "aarch64_adrp_ldr_slot+undefined_dynamic_symbol_relocation",
|
||||
"value": "_ZTVNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEEE"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"persistence": {
|
||||
"anchors": {
|
||||
"import_slot": 40,
|
||||
"string": 155
|
||||
},
|
||||
"asset_names": 0,
|
||||
"failures": 0,
|
||||
"input_sha256": "b1ef1b5585141a24f9997babff0738851807680439954a61b6e491ca7e76d87d",
|
||||
"method": "verstack-code-anchors/1",
|
||||
"native_sized_symbols": 263,
|
||||
"search_probe": {
|
||||
"address": 40110848,
|
||||
"bytes_sha256": "fb329000228cc5a24c264c57139de8bf854fc86fc18bf1c04ab61a2b5cb4b921",
|
||||
"instruction_offset": 636,
|
||||
"kind": "string",
|
||||
"proof": "aarch64_adrp_add_address",
|
||||
"value": "NULL"
|
||||
},
|
||||
"search_results": 1,
|
||||
"selected_symbol_limit": 400,
|
||||
"truncated_functions": 0,
|
||||
"verified": 263
|
||||
},
|
||||
"coverage": "Native sized ELF symbols only, not all stripped functions; no accepted correspondence or names. GOT printable literal-pool coincidences rejected. These are reference occurrences, not unique strings."
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
# Cross-architecture code bridge evidence
|
||||
|
||||
`POST /api/code/bridges` accepts a bounded batch of externally produced
|
||||
ARM32/AArch64 correspondence records. Each record names the two immutable ELF
|
||||
input hashes, instruction modes, byte addresses, a score in `0..=1`, and a JSON
|
||||
evidence object. Addresses are canonicalized and the two architecture families
|
||||
must differ. Replaying the same normalized row is idempotent.
|
||||
|
||||
Rows are stored in the append-only `code_architecture_bridges` table and can be
|
||||
read with `GET /api/code/bridges?input_sha256=...&limit=...`. Every response
|
||||
states `candidate_only: true`, `accepted_matches: 0`, and `names_propagated: 0`.
|
||||
The bridge index is intentionally separate from `code_functions`,
|
||||
`code_observations`, and `asset_names`: a semantic bridge can rank a review
|
||||
candidate but cannot establish an architecture-specific function identity or
|
||||
transfer a symbol name. Invalid hashes, unsupported architectures, same-ISA
|
||||
pairs, non-finite/out-of-range scores, malformed evidence, and non-hexadecimal
|
||||
addresses fail closed.
|
||||
|
||||
The schema migration is additive (schema 20). The focused regression
|
||||
`tests/native_bridges.rs` verifies normalization, idempotent replay, candidate
|
||||
only persistence, and rejection of unsafe claims. This is an ingestion and
|
||||
review-index slice of the Phase 5 symbol bridge; it does not claim corpus-wide
|
||||
semantic matching or propagation.
|
||||
@@ -0,0 +1,226 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"method": "native-instruction-similarity/1",
|
||||
"acceptance_policy": "similarity is not match confidence; no accepted matches or names published",
|
||||
"cohorts": {
|
||||
"got_arm32": {
|
||||
"before_input_sha256": "637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828",
|
||||
"after_input_sha256": "240db7bed730b8958c8d1d34e160e296d2306fda22533e5f1461df389e65d410",
|
||||
"selection": "first128 alphabetical unique same-name changed-body symbol extents,32\u20134096B, complete A32/A64 CFG; at most2000 attempts",
|
||||
"attempted": 1943,
|
||||
"persisted_pair_checks": 8,
|
||||
"caveat": "Names select this diagnostic cohort; these results are not precision/recall calibration, semantic equivalence or a propagation threshold. Different names can have similar or identical code.",
|
||||
"named_pair_count": 128,
|
||||
"named_score_min": 0.625,
|
||||
"named_score_max": 1.0,
|
||||
"named_score_mean": 0.9750974114388694,
|
||||
"different_name_control_count": 127,
|
||||
"different_name_max_score": 0.96,
|
||||
"different_name_at_least_09": 4,
|
||||
"different_name_top_controls": [
|
||||
{
|
||||
"after_name": "_Z20Light_GREYJOY_GoNoGov",
|
||||
"before_name": "_Z20Light_MARTELL_GoNoGov",
|
||||
"scores": {
|
||||
"block_similarity": 0.8333333333333334,
|
||||
"changed_after_count": 1,
|
||||
"changed_before_count": 1,
|
||||
"instruction_counts": {
|
||||
"after": 25,
|
||||
"before": 25
|
||||
},
|
||||
"instruction_similarity": 0.96,
|
||||
"literal_instruction_similarity": 0.92
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_name": "_Z22Light_LANNISTER_GoNoGov",
|
||||
"before_name": "_Z22Light_TARGARYEN_GoNoGov",
|
||||
"scores": {
|
||||
"block_similarity": 0.8333333333333334,
|
||||
"changed_after_count": 1,
|
||||
"changed_before_count": 1,
|
||||
"instruction_counts": {
|
||||
"after": 25,
|
||||
"before": 25
|
||||
},
|
||||
"instruction_similarity": 0.96,
|
||||
"literal_instruction_similarity": 0.92
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_name": "_Z22dgef_targaryen_2_awardv",
|
||||
"before_name": "_Z22dgef_targaryen_3_awardv",
|
||||
"scores": {
|
||||
"block_similarity": 0.0,
|
||||
"changed_after_count": 1,
|
||||
"changed_before_count": 1,
|
||||
"instruction_counts": {
|
||||
"after": 19,
|
||||
"before": 19
|
||||
},
|
||||
"instruction_similarity": 0.9473684210526315,
|
||||
"literal_instruction_similarity": 0.6842105263157895
|
||||
}
|
||||
}
|
||||
],
|
||||
"near_0994_examples": [
|
||||
{
|
||||
"after_address": 148140,
|
||||
"after_body_sha256": "a1e788ee0df2331ebf0b6afd148e031fcd5bcfb840343cf77d7c23a342729022",
|
||||
"before_address": 144692,
|
||||
"before_body_sha256": "f393822430daa3f7dc528ef47415640ca003e1e1b0491a9c18eef28b12c88a43",
|
||||
"name": "_Z15lef_ball_lockedv",
|
||||
"scores": {
|
||||
"block_similarity": 0.0,
|
||||
"changed_after_count": 1,
|
||||
"changed_before_count": 1,
|
||||
"instruction_counts": {
|
||||
"after": 119,
|
||||
"before": 119
|
||||
},
|
||||
"instruction_similarity": 0.9915966386554622,
|
||||
"literal_instruction_similarity": 0.8067226890756303
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_address": 820836,
|
||||
"after_body_sha256": "136d4b65ab32016a675a7dd8bdff11e08d11332e98a8e0198bab0a8920536a81",
|
||||
"before_address": 796072,
|
||||
"before_body_sha256": "91332fa52ef7a73e59a33c397da0c6fff89d8638396178ebf7ed8ffd37737ae0",
|
||||
"name": "SHA1_Final",
|
||||
"scores": {
|
||||
"block_similarity": 1.0,
|
||||
"changed_after_count": 0,
|
||||
"changed_before_count": 0,
|
||||
"instruction_counts": {
|
||||
"after": 91,
|
||||
"before": 91
|
||||
},
|
||||
"instruction_similarity": 1.0,
|
||||
"literal_instruction_similarity": 0.945054945054945
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"pokemon_aarch64": {
|
||||
"before_input_sha256": "5f8bd90b23c35d15252315b2003fb99d0c5950bca2017a085db49f7b52862293",
|
||||
"after_input_sha256": "b1ef1b5585141a24f9997babff0738851807680439954a61b6e491ca7e76d87d",
|
||||
"selection": "first128 alphabetical unique same-name changed-body symbol extents,32\u20134096B, complete A32/A64 CFG; at most2000 attempts",
|
||||
"attempted": 114,
|
||||
"persisted_pair_checks": 8,
|
||||
"caveat": "Names select this diagnostic cohort; these results are not precision/recall calibration, semantic equivalence or a propagation threshold. Different names can have similar or identical code.",
|
||||
"named_pair_count": 97,
|
||||
"named_score_min": 0.5,
|
||||
"named_score_max": 0.9954441913439636,
|
||||
"named_score_mean": 0.864683494586441,
|
||||
"different_name_control_count": 96,
|
||||
"different_name_max_score": 0.7555555555555555,
|
||||
"different_name_at_least_09": 0,
|
||||
"different_name_top_controls": [
|
||||
{
|
||||
"after_name": "png_set_cHRM_XYZ_fixed",
|
||||
"before_name": "png_set_cHRM_fixed",
|
||||
"scores": {
|
||||
"block_similarity": 0.2857142857142857,
|
||||
"changed_after_count": 12,
|
||||
"changed_before_count": 10,
|
||||
"instruction_counts": {
|
||||
"after": 46,
|
||||
"before": 44
|
||||
},
|
||||
"instruction_similarity": 0.7555555555555555,
|
||||
"literal_instruction_similarity": 0.8
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_name": "_ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEED0Ev",
|
||||
"before_name": "_ZNSt7__cxx1115basic_stringbufIcSt11char_traitsIcESaIcEED1Ev",
|
||||
"scores": {
|
||||
"block_similarity": 0.0,
|
||||
"changed_after_count": 8,
|
||||
"changed_before_count": 5,
|
||||
"instruction_counts": {
|
||||
"after": 24,
|
||||
"before": 21
|
||||
},
|
||||
"instruction_similarity": 0.7111111111111111,
|
||||
"literal_instruction_similarity": 0.7111111111111111
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_name": "png_get_x_offset_inches",
|
||||
"before_name": "png_get_y_offset_inches",
|
||||
"scores": {
|
||||
"block_similarity": 0.0,
|
||||
"changed_after_count": 3,
|
||||
"changed_before_count": 3,
|
||||
"instruction_counts": {
|
||||
"after": 10,
|
||||
"before": 10
|
||||
},
|
||||
"instruction_similarity": 0.7,
|
||||
"literal_instruction_similarity": 0.7
|
||||
}
|
||||
}
|
||||
],
|
||||
"near_0994_examples": [
|
||||
{
|
||||
"after_address": 15879392,
|
||||
"after_body_sha256": "11c3f80d9e52f2cc6eade4713ecbbb2d8204dbc2b26279b98551bb08824f2871",
|
||||
"before_address": 15683216,
|
||||
"before_body_sha256": "d2a735cfe5e8c3f52d08b96e29bcc2a03de41c3d4d072a82bf803aa8d1314767",
|
||||
"name": "png_set_quantize",
|
||||
"scores": {
|
||||
"block_similarity": 0.9805825242718447,
|
||||
"changed_after_count": 2,
|
||||
"changed_before_count": 2,
|
||||
"instruction_counts": {
|
||||
"after": 439,
|
||||
"before": 439
|
||||
},
|
||||
"instruction_similarity": 0.9954441913439636,
|
||||
"literal_instruction_similarity": 0.9977220956719818
|
||||
}
|
||||
},
|
||||
{
|
||||
"after_address": 8366816,
|
||||
"after_body_sha256": "a732978e61497dded2425c840f5ab1d1729439291c2c9337a36f90c7bb1b07f8",
|
||||
"before_address": 7495936,
|
||||
"before_body_sha256": "85c983d08eba5efdf42fdeddcb1a66a84dfbf299a5965edb9d71d062336691e5",
|
||||
"name": "_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEaSEOS4_",
|
||||
"scores": {
|
||||
"block_similarity": 0.9375,
|
||||
"changed_after_count": 1,
|
||||
"changed_before_count": 1,
|
||||
"instruction_counts": {
|
||||
"after": 66,
|
||||
"before": 66
|
||||
},
|
||||
"instruction_similarity": 0.9848484848484849,
|
||||
"literal_instruction_similarity": 0.9848484848484849
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"schema16_copy_rehearsal": {
|
||||
"functions": 1018403,
|
||||
"invalid_claims": 19456,
|
||||
"observations": 1170298,
|
||||
"processed": 1170298,
|
||||
"provenance_records": 48,
|
||||
"schema": 16,
|
||||
"seconds": 949.451260829,
|
||||
"signature_fingerprint_before_and_after": "cad8cad20720dfba5ff1c315872a77e21f90802a51c05b754007dedffef98020",
|
||||
"signature_fingerprint_method": "blake3-framed-columns-rowid-order/1",
|
||||
"signature_rows": 1170298,
|
||||
"source_rows": 1170298,
|
||||
"valid_claims": 1150842,
|
||||
"native_evidence_rows": 0,
|
||||
"orphan_observations": 0,
|
||||
"missing_provenance": 0,
|
||||
"backfill_replay_processed": 0,
|
||||
"interpretation": "Saved analysis claims accounted for without rereading original ELFs or running Ghidra; not a corpus native-normalization pass."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,303 @@
|
||||
# Native code evidence persistence (schema 16)
|
||||
|
||||
This layer records saved analysis claims and independently verified native evidence.
|
||||
It does not recover names, establish semantic equivalence, or make a stripped
|
||||
binary's function inventory complete.
|
||||
|
||||
`code_functions` identifies an input SHA-256, architecture/endianness, address,
|
||||
complete extent and claimed body SHA-256. `code_observations` retains every
|
||||
indexed Ghidra function row, its original symbol/name/thunk status and a reference
|
||||
to deduplicated immutable analysis provenance. Invalid identity claims remain
|
||||
observations with a null function ID. Merely backfilling a reported body hash does
|
||||
not verify it against original bytes. Dialect suffixes in Ghidra language IDs do
|
||||
not create duplicate instances of the same input/address/body.
|
||||
|
||||
`native_code_evidence` is append-only and includes method, pinned decoder, original
|
||||
artifact identity, ELF input SHA-256, saved-facts artifact (if used), boundary
|
||||
provenance, decode/reachability coverage, diagnostics and candidate fingerprints.
|
||||
`code_basic_blocks`, `code_instructions` and `code_references` preserve the corresponding
|
||||
instruction/graph/reference evidence. `code_normalization_pages` retains bounded
|
||||
attempt reports, including skips and incomplete boundary coverage. Repeating an
|
||||
identical page is idempotent. Different evidence or methods append records.
|
||||
Existing signature rows, including full masks, are never modified by this layer.
|
||||
|
||||
New Ghidra indexing records observations automatically. Upgrading the schema does
|
||||
not decode executables or scan the old symbol corpus during startup. Explicitly
|
||||
resume old metadata with `POST /api/code/backfill`, body `{"limit":10000}`; repeat
|
||||
until `processed` is zero. This replays saved SQL facts without Ghidra or original
|
||||
ELF reads. It indexes only symbol inventories with a `code_programs` provenance
|
||||
record; unrelated media inventories remain outside code observations.
|
||||
|
||||
`POST /api/code/normalize` takes `snapshot`, `path`, optional `facts_snapshot` and
|
||||
`facts_path`, `offset` and `limit`. Both saved-facts fields must be provided together.
|
||||
All reads use SourceReader and recheck the archived artifact's BLAKE3 identity.
|
||||
ELF reads cap at 256 MiB and saved facts at 512 MiB. The native boundary inventory
|
||||
uses sized ELF symbols and hash-verified contiguous saved extents, without running
|
||||
Ghidra. Each page processes at most 200 functions, at most 2 MiB of function bodies,
|
||||
and at most 64 KiB per function. Use `report.next_offset` until it reaches `total`;
|
||||
this denominator is recovered extents, not every function in the binary. ARM mode
|
||||
must have ELF evidence; Thumb currently returns explicit unsupported coverage.
|
||||
|
||||
Source reads and disassembly occur outside the catalog write transaction. A page
|
||||
commits its evidence and report atomically. The response includes the persisted
|
||||
`page_id`; `GET /api/code/normalization/{id}` retrieves that report. Mutating routes
|
||||
require `X-Verstack-Client: 1` and the existing browser-boundary protection.
|
||||
|
||||
`GET /api/code/evidence?snapshot=...&path=...&offset=0&limit=100` returns summary
|
||||
rows with verified provenance and coverage. `GET /api/code/evidence/{id}` pages
|
||||
instructions, blocks and references independently using the same offset and limit.
|
||||
Both endpoints cap pages at 200 records. Candidate fingerprints remain distinct
|
||||
from accepted comparisons and all name claims.
|
||||
|
||||
Reproduce focused contracts with:
|
||||
|
||||
```
|
||||
cargo test --offline --locked --test native_code --test catalog_storage --test name_provenance
|
||||
cargo test --offline --locked --lib native_code
|
||||
```
|
||||
|
||||
The ignored `rehearse_native_code_catalog_copy` test accepts only a path containing
|
||||
`/data/validation/native-code-dryrun-`. Create that file with SQLite's online backup
|
||||
from a read-only live connection (never `immutable=1`), then run it with
|
||||
`VERSTACK_NATIVE_CODE_COPY` set. It migrates and backfills only that copy, accounts
|
||||
for every source row, proves restart/idempotence, and hashes all original signature
|
||||
columns before and after. Never point Archive::open at a copied database whose
|
||||
store is a symlink to live storage.
|
||||
|
||||
Immediate remaining Phase 5 requirements include corpus-wide native normalization
|
||||
and calibrated candidate retrieval, broader referenced-string/import/constant
|
||||
coverage, Thumb decoding, stripped boundary recovery, cross-ISA anchors,
|
||||
call-graph propagation and durable graded `code_function_deltas`. The bounded
|
||||
instruction/block comparison and visible UI described below are implemented;
|
||||
they do not complete these corpus-scale or accepted-correspondence requirements.
|
||||
The standalone implementation's controls and real-corpus measurements are in
|
||||
`crates/verstack-code/README.md` and `corpus-evidence.json`.
|
||||
|
||||
## Graded comparisons
|
||||
|
||||
`GET /api/code/similarity?before=<evidence-id>&after=<evidence-id>` compares
|
||||
persisted, independently byte-verified evidence. Complete decoding, complete CFG,
|
||||
full reachable extent, same ISA mode and same normalization method are required.
|
||||
No partial score is returned when those conditions fail or alignment exceeds its
|
||||
budget. Each of two sequence-alignment passes is capped at 4 million cells, with
|
||||
at most 4096 instructions per function; changed excerpts cap at 100 per side.
|
||||
|
||||
The instruction score is `2 × equal aligned instructions / (before + after
|
||||
instruction counts)`. The literal score compares encoded words. The normalized
|
||||
score masks address operands only when their referenced symbol/content/internal
|
||||
location supplies comparable proof; unresolved references retain literal operands.
|
||||
Changed constants, register choices, conditions and referenced literal contents
|
||||
remain visible. Exact block overlap counts equal proof-aware blocks. Per-block
|
||||
instruction scores use equal aligned instruction pairs within candidate block
|
||||
pairs; a split can produce multiple pairs. Both expose their denominators.
|
||||
|
||||
`GET /api/code/candidates?id=<seed>&snapshot=<optional target>&path=<optional target>&limit=20`
|
||||
retrieves weak block-hash candidates only. It examines at most 64 seed anchors,
|
||||
10,001 index hits (10,000 admitted) and returns at most 50 candidates. The scope
|
||||
filter follows this bounded global hit pool, so common blocks can exhaust the
|
||||
pool before the requested program contributes candidates. Truncation is explicit;
|
||||
this is not an exhaustive nearest-neighbor search or a calibrated acceptance rule.
|
||||
The explicit pair endpoint computes the score after candidate selection.
|
||||
|
||||
The workbench Code comparison view contains **Binary function similarity**.
|
||||
It resolves originals along the saved program's ancestor chain, indexes bounded
|
||||
pages or one explicit hexadecimal function address, reads cached evidence,
|
||||
retrieves target-program candidates and shows scores, denominators, changed
|
||||
instructions, block-pair scores and limits. No code match or name is published.
|
||||
The component browser regression uses an isolated temporary server and the actual
|
||||
React component: `node tests/ui-native-code.mjs`. It does not deploy the workbench.
|
||||
|
||||
A real GOT Pro/LE diagnostic cohort compared 128 alphabetical unique same-name
|
||||
changed-body ELF symbol extents (32–4096 bytes, complete CFG, at most 2000 extent
|
||||
attempts). Mean instruction similarity was 0.975097, range 0.625–1.0.
|
||||
`_Z15lef_ball_lockedv` measured 118/119 equal instructions = 0.991597, while literal
|
||||
instruction similarity was 0.806723. Eight real pairs were persisted and their
|
||||
readback scores checked against direct computation. Among 127 adjacent
|
||||
**different-name** controls, four scored at least 0.9 and the maximum was 0.96
|
||||
(`Light_GREYJOY_GoNoGo` versus `Light_MARTELL_GoNoGo`). This explicitly disproves
|
||||
using a high similarity score alone as accepted correspondence or name confidence.
|
||||
This name-selected, bounded diagnostic sample is not a corpus precision/recall
|
||||
calibration. Reproduce it without Ghidra:
|
||||
|
||||
```
|
||||
VERSTACK_CODE_ELF_A=data/validation/ghidra-got-revision2/input/game \
|
||||
VERSTACK_CODE_ELF_B=data/validation/ghidra-got-le-revision2/input/game \
|
||||
cargo test --offline --locked --lib named_real_function_scores_and_different_name_controls -- --ignored --nocapture
|
||||
```
|
||||
|
||||
The report is written to `data/validation/native-code/real-similarity.json` with
|
||||
both input/body hashes, the explicit selection rule, all scores and controls.
|
||||
|
||||
Native normalization, similarity, candidate lookup and metadata backfill use the
|
||||
shared atomic task admission pool. They request one CPU and zero scratch; busy
|
||||
capacity is reported immediately for retry. Normalization reserves a conservative
|
||||
heap allowance of six times actual ELF/facts bytes plus 2 GiB for parsed inventories
|
||||
and decoded-page structures, and a 512 MiB archive allowance. Comparison reserves
|
||||
128 MiB, candidate retrieval 1 GiB, and a metadata-backfill request 256 MiB. These
|
||||
are admission estimates, while independent byte/function/page/work limits remain
|
||||
enforced. Readback endpoints remain ordinary bounded catalog reads.
|
||||
|
||||
Explicit catalog deletion removes native evidence and reports when either the
|
||||
original snapshot or the cited saved-facts snapshot is deleted. Instruction,
|
||||
block, mask and reference rows are removed in the same catalog transaction.
|
||||
Observations are removed with their facts snapshot; shared function instances and
|
||||
analysis provenance remain while any surviving observation/evidence references
|
||||
them. This explicit deletion lifecycle is the exception to append-only indexing.
|
||||
Regression tests cover cross-release facts dependencies, shared-instance survival,
|
||||
transaction rollback and removal of stale symbol FTS hits.
|
||||
|
||||
## Full metadata rehearsal
|
||||
|
||||
A fresh SQLite online backup of the live schema 15 catalog was migrated to schema 16
|
||||
and explicitly backfilled on the copy. All 1,170,298 source rows became observations:
|
||||
1,150,842 valid saved-analysis claims and 19,456 explicit invalid-identity claims.
|
||||
They identify 1,018,403 distinct input/address/body instances and share 48 provenance
|
||||
records. Replay processed zero additional rows. Every original signature column,
|
||||
including all full masks, retained its before/after BLAKE3 framed-column fingerprint
|
||||
`cad8cad20720dfba5ff1c315872a77e21f90802a51c05b754007dedffef98020`.
|
||||
Backfill plus final verification took 949.45 seconds; the complete test took
|
||||
983.68 seconds. No original executable was reread and no Ghidra process ran.
|
||||
|
||||
The copy contains zero native evidence rows: metadata backfill is not a corpus
|
||||
normalization outcome. The authoritative rehearsal file is
|
||||
`data/validation/native-code-dryrun-20260916-2.sqlite3`; its adjacent report and
|
||||
`docs/native-code-canary.json` record the counts and measurement semantics.
|
||||
The earlier interrupted -1 copy was removed after retaining its log and row-count
|
||||
record. No live catalog was mutated for this rehearsal.
|
||||
|
||||
A second native score cohort over archived Pokémon 0.85/0.86 originals covered 97
|
||||
complete unique same-name changed-body symbol pairs and 96 different-name controls.
|
||||
`png_set_quantize` at 0xef4e90 /0xf24ce0 measured 437/439 = 0.995444 instruction
|
||||
similarity, literal 438/439 = 0.997722, and exact block overlap 0.980583. Referent-aware
|
||||
scores can be lower than literal scores when identical encodings refer to changed
|
||||
or unresolved targets. The separate 84-byte stringbuf destructor pair at 0x4bb590 /
|
||||
0x4baf20 measured 16/21 = 0.761905. These explicit original/extent/body facts support
|
||||
live endpoint canaries; they are not acceptance thresholds. Both cohort summaries
|
||||
are retained in `docs/native-code-canary.json`.
|
||||
|
||||
The current workbench program selectors come from saved code exports. The native
|
||||
API can normalize an ELF without saved facts; exposing native-only program discovery
|
||||
in those selectors remains part of the remaining Tier 0 integration work.
|
||||
|
||||
Schema 17 adds separately versioned native anchors (`verstack-code-anchors/1`).
|
||||
Normalizing a verified original function now appends its anchor report, occurrence
|
||||
records and searchable `code_strings` rows in the same transaction. The original
|
||||
`verstack-code/1` evidence, hashes, saved masks and naming claims are unchanged.
|
||||
Existing native evidence explicitly reports `not_indexed` until its original
|
||||
function is re-indexed; the metadata-only 1.17M-row backfill cannot invent strings
|
||||
that saved function metadata did not contain. No Ghidra rerun is required.
|
||||
|
||||
Supported anchors are directly referenced, printable, NUL-terminated UTF-8 bytes
|
||||
(4–1024 bytes), raw literal-load values with their exact widths and SHA-256, and
|
||||
undefined dynamic-symbol GLOB_DAT/JUMP_SLOT relocation slots. Direct address
|
||||
proofs include ARM ADR, proven MOVW/MOVT address uses, AArch64 ADR/ADRP+ADD, and
|
||||
ADRP+LDR import-slot loads. A literal-pool pointer is followed only when the next
|
||||
unconditional instruction uses its destination register as a memory base. There
|
||||
is no arbitrary immediate-to-pointer inference. Printable literal constants are
|
||||
**not** indexed as strings: real ARM floating-point words produced misleading
|
||||
`gfff...` text in an initial experiment and are retained only as raw constants.
|
||||
A referenced byte table can still be printable; the string row describes bytes,
|
||||
not an established semantic label or function identity.
|
||||
|
||||
Extraction inspects at most 4096 instructions/references, 512 distinct reference
|
||||
targets, and emits at most 512 anchors per function. It reports truncation and
|
||||
incomplete decode. Import rows identify slots, including imported objects such as
|
||||
`__stack_chk_guard`; they do not assert that an indirect call was resolved. ELF
|
||||
symbol versions/libraries, general dataflow, multi-hop pointers, PLT resolution,
|
||||
Thumb and comprehensive immediate-constant inventories remain open. No anchor
|
||||
changes similarity scores, publishes names, or accepts correspondence.
|
||||
|
||||
`GET /api/code/evidence/{id}` now includes paged `anchor_evidence` with the ELF
|
||||
SHA-256, body SHA-256, instruction offset, target address, method and proof.
|
||||
`GET /api/code/strings?q=phrase&snapshot=optional&path=optional&limit=50` searches
|
||||
literal FTS phrases, with at most 100 returned rows and a 10,000-hit candidate
|
||||
pool. A separate bounded 10,001-hit probe detects truncation even if scope
|
||||
filtering returns zero rows. Scope filtering follows that pool, so a frequent
|
||||
phrase can omit scoped hits; this is disclosed. Search covers explicitly indexed
|
||||
functions only. The Binary function similarity panel exposes selected-function
|
||||
anchors and scoped/corpus string search. Deleting an original or facts dependency
|
||||
removes derived anchors and FTS rows transactionally while retaining other
|
||||
observations; rollback and stale-search controls are tested.
|
||||
|
||||
Real corpus verification is recorded in [native-code-anchors.json](native-code-anchors.json).
|
||||
GOT ARM32's 11,519 verified sized bodies yielded 26,829 literal occurrences and two
|
||||
proven loaded-pointer text references, including `/etc/localtime`. Pokémon 0.86
|
||||
AArch64's 263 native sized bodies yielded 155 string occurrences and 40 import-slot
|
||||
references. These deliberately partial inventories are not full stripped-program
|
||||
coverage. All bytes, literal addresses, pointer/address instruction encodings and
|
||||
import slot names in these cohorts were independently checked by
|
||||
`crates/verstack-code/scripts/check_anchor_bytes.py` (Python ELF mappings plus
|
||||
`readelf -rW`, without Capstone). Pokémon's 263 bodies were also persisted into a
|
||||
fresh catalog, queried through the real FTS API method, and verified to create
|
||||
zero asset names. Schema 16→17/reopen, original-mask preservation, HTTP indexing
|
||||
and search, bounded-pool truncation, deletion/rollback and the real browser
|
||||
component have regression coverage. No live catalog migration is part of this
|
||||
verification.
|
||||
|
||||
Reproduce with the standalone CLI (ELF + `--symbols` + report path), then:
|
||||
|
||||
```sh
|
||||
python3 crates/verstack-code/scripts/check_anchor_bytes.py ELF REPORT.json SUMMARY.json
|
||||
VERSTACK_CODE_ANCHOR_ELF=ELF VERSTACK_CODE_ANCHOR_REPORT=SUMMARY.json \
|
||||
cargo test --offline --locked --lib code_anchors::corpus_tests -- --ignored --nocapture
|
||||
cargo test --offline --locked --lib code_anchors
|
||||
cargo test --offline --locked --test native_code
|
||||
node tests/ui-native-code.mjs
|
||||
```
|
||||
|
||||
Archived-program anchor paging is available at
|
||||
`POST /api/code/anchors/backfill` with `{snapshot,path,offset,limit}`. The snapshot
|
||||
and path must identify an exact existing `code_programs` facts row. The route
|
||||
resolves the original through recorded ancestry and reuses full original/facts
|
||||
artifact verification and native extent/body/mode checks. A page remains bounded
|
||||
to 1–200 recovered extents, 64 KiB per function and 2 MiB selected body bytes.
|
||||
Offsets refer to the recovered extent inventory, **not** the saved function rows.
|
||||
Exhausting the page inventory never claims complete binary coverage. The response
|
||||
includes the persisted normalization page, next offset and before/after coverage.
|
||||
The workbench's “Index next100 extents” action now uses this path and displays
|
||||
its saved-row denominator separately from verified/complete/truncated counts.
|
||||
|
||||
`GET /api/code/anchors/coverage?snapshot=FACTS&path=FUNCTIONS_JSON` reports saved
|
||||
function count, indexed metadata observations, invalid claims, independently
|
||||
verified anchor-report instances, complete decodes, truncated anchor inventories
|
||||
and saved rows without verified anchors. The join requires the same executable
|
||||
SHA, architecture, address, size and complete body digest. An independently
|
||||
verified ELF function without a matching saved instance is not silently counted
|
||||
as covering a saved row. A verified report can contain no usable anchors; complete
|
||||
decode is reported separately. No name or candidate similarity is promoted.
|
||||
The descendant source-resolution helper now returns the actual selected facts
|
||||
snapshot and searches originals through its ancestry, with a regression test.
|
||||
|
||||
Broader coverage measurements are in
|
||||
[archived-anchor-coverage.json](archived-anchor-coverage.json). They deliberately
|
||||
record rejected input claims as well as positive native evidence. The first 200
|
||||
size-bounded saved rows of each of Deadpool, Godzilla, Star Wars and AArch64 libc
|
||||
were predominantly tiny/default entries and all 800 lacked verified contiguous
|
||||
extent proof. A separate explicitly proof-bearing sample of 200 rows per program
|
||||
produced three verified ARM functions (no usable anchors), 594 missing-mode
|
||||
rejections, three conflicting extents, and 200 libc body mismatches. No blanket
|
||||
ARM mode assumption was added. A diagnostic translation of libc addresses by
|
||||
`-0x100000` independently matched all 200 full bodies, demonstrating why raw ELF
|
||||
and Ghidra address coordinates must not be conflated. Image-base provenance is
|
||||
not exported by these saved facts, so the application still refuses that join.
|
||||
The independent native-symbol libc pass verified 400 functions (of 2,249 sized
|
||||
symbols), persisted 37 string and 13 import-slot occurrences, and exercised FTS
|
||||
readback with zero published names. It is reported separately and does not
|
||||
inflate saved-row coverage.
|
||||
|
||||
Reproduce without live writes:
|
||||
|
||||
```sh
|
||||
python3 scripts/collect_anchor_programs.py
|
||||
python3 scripts/collect_anchor_programs.py --proof-bearing
|
||||
VERSTACK_ANCHOR_PROGRAMS=data/validation/anchor-programs/manifest-proof.json \
|
||||
VERSTACK_ANCHOR_COVERAGE_REPORT=data/validation/anchor-programs/coverage-proof.json \
|
||||
cargo test --offline --locked --lib code_anchors::archived_corpus_tests -- --ignored --nocapture
|
||||
```
|
||||
|
||||
The collector reads the live catalog using `mode=ro`, downloads at most four
|
||||
original ELFs (256 MiB/file ceiling), verifies their SHA-256, and selects bounded
|
||||
saved rows with the authoritative `full_mask` joined. It does not reconstruct
|
||||
missing method/mode claims, modify the archive, or rerun Ghidra. Tests use fresh
|
||||
offline catalogs. Broader ARM mode recovery, verified address-coordinate handling,
|
||||
and corpus-wide scheduled coverage remain required work.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Native ELF code evidence
|
||||
|
||||
`codesig::analyze_elf` and `GET /api/code/signatures?snapshot=ID&path=PATH&offset=0&limit=200`
|
||||
produce read-only ARM32/AArch64 evidence directly from archived ELF bytes. They do not
|
||||
launch Ghidra, materialize scratch trees, write the catalog, or propagate names.
|
||||
Linked executables and shared/PIE objects in either byte order are supported.
|
||||
|
||||
Sized, defined `STT_FUNC` symbols establish candidate boundaries, with explicit
|
||||
`elf_symtab` or `elf_dynsym` provenance. ARM's low Thumb bit is removed from the
|
||||
byte address and retained as instruction mode. Each accepted body must map uniquely
|
||||
to executable file-backed bytes. SHA-256 covers its entire body. Conflicting extents
|
||||
or instruction modes at one start remain unresolved. Aliases merge; overlapping
|
||||
bodies count only once toward byte coverage. Zero-sized symbols and entry points
|
||||
are hints, never an excuse to guess an end from the next symbol.
|
||||
|
||||
Optional `facts_snapshot=ID&facts_path=PATH` accepts archived ExportFacts JSON.
|
||||
Its schema, completed analysis status, architecture, byte order and input SHA-256
|
||||
must agree. A boundary additionally requires the pinned exporter's contiguous-body
|
||||
signature metadata and an exact SHA-256 match of the entire proposed on-disk body.
|
||||
The exporter otherwise reports a sum of possibly discontiguous ranges: address plus
|
||||
size alone is insufficient. Small functions without that metadata, rebased functions,
|
||||
and relocated memory bytes may therefore remain unresolved. Saved analysis has
|
||||
`verified_saved_ghidra` provenance; it is not counted as native boundary discovery.
|
||||
|
||||
The response always declares `function_inventory_complete: false`. Executable
|
||||
segments include padding and data; byte coverage is not a recovery accuracy score.
|
||||
Stripped binaries may return no accepted functions. Identical bytes do not establish
|
||||
semantic equivalence or cross-architecture identity. The symbol-bridge semantic
|
||||
matching work remains separate.
|
||||
|
||||
Limits: ELF input 256 MiB; saved facts 512 MiB; ELF symbol records and saved function
|
||||
records each 500,000; aggregate distinct-body hashing 2 GiB. Exhausted hashing
|
||||
produces unresolved hints. Names are capped at 1,024 bytes, 64 aliases per function
|
||||
(with a truncation flag). HTTP pages contain 1–1,000 functions and the same window
|
||||
of hints; totals and coverage describe the full analyzed input. Each request analyzes
|
||||
the input afresh; pagination bounds response size, not parsing cost. SourceReader
|
||||
reads bounded archived bytes. Malformed or unsupported input fails explicitly.
|
||||
|
||||
Next boundary work for stripped binaries needs ARM `.ARM.exidx` and AArch64
|
||||
`.eh_frame` evidence with careful distinction between unwind ranges and actual
|
||||
function extents. Disassembly, relocation-aware masking, call graphs, cached native
|
||||
indexes and semantic cross-ISA matching are not implemented by this foundation.
|
||||
|
||||
Run `cargo test --locked --offline --test codesig` for generated ELF and actual
|
||||
Archive/HTTP contract regressions. Fixtures require no cross compiler.
|
||||
|
||||
Read-only corpus check (2026-09-16): King Kong Pro 0.97 ARM32, input SHA-256
|
||||
`81133a08e65fae1070f02a3db50c867f064ffef21e6f5c6737c40f99980af6cb`,
|
||||
provided 64 native functions (13,708 of 8,330,960 executable bytes). Verified saved
|
||||
facts raised this to 10,018 functions and 3,561,176 bytes; 3,575 saved records lacked
|
||||
contiguous-body proof and 68 conflicting extents remained unresolved.
|
||||
Pokémon LE 0.85 AArch64, input SHA-256
|
||||
`5f8bd90b23c35d15252315b2003fb99d0c5950bca2017a085db49f7b52862293`,
|
||||
provided 262 native functions (38,816 of 50,978,991 executable bytes). Its saved
|
||||
facts added no boundaries: 96,484 body hashes disagreed with on-disk ranges and
|
||||
33,325 lacked contiguous-body proof. Loader rebasing/relocation provenance must be
|
||||
investigated before attempting to reuse these boundaries. The reader does not
|
||||
silently adjust addresses to improve coverage. These measurements do not establish
|
||||
complete function recovery.
|
||||
|
||||
The opt-in test `live_native_boundary_coverage` fetches those immutable artifacts
|
||||
from the local backend and opens the catalog read-only. It requires a running local
|
||||
service; run the test with `--ignored --nocapture` separately from ordinary gates.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Native decoder inspection
|
||||
|
||||
`GET /api/decode/inspect?snapshot=<id>&path=<archived-path>` inspects one existing
|
||||
artifact through `Archive::source_reader`. URL-encode the path. This is read-only;
|
||||
it does not restore a source to scratch, write catalog names, or extract payloads.
|
||||
The `Archive::inspect_decode` method exposes the same operation to Rust callers.
|
||||
|
||||
`src/decode/mod.rs` defines `FormatDecoder`, a linkme distributed registry and
|
||||
deterministic dispatch (priority descending, then decoder ID). Implement a decoder
|
||||
in a module and register a static `&dyn FormatDecoder` with
|
||||
`#[distributed_slice(DECODERS)]`; no central match statement must change. Readers
|
||||
support `Read + Seek`. Dispatch reads at most 4 KiB to choose a decoder and resets
|
||||
the reader before each attempt. `SourceReader` coalesces backing-store requests
|
||||
into 8 MiB windows without whole-file materialization.
|
||||
|
||||
Every report retains the input's `ContentOnly` identity. Unknown formats and
|
||||
rejected parser inputs produce an explicit opaque report with warnings; no input
|
||||
is silently omitted. Reports use `coverage=metadata` for recognized inputs, not
|
||||
complete extraction coverage. A storage failure during initial probing is an error.
|
||||
|
||||
Included implementations:
|
||||
|
||||
- `zip-directory`: classic and ZIP64 central-directory metadata through bounded
|
||||
seeks, including member sizes and expansion totals. Single-disk, UTF-8/ASCII
|
||||
names only; at most 64 MiB of directory metadata and 200,000 entries. This does
|
||||
not verify local headers, decompress members, authenticate packages or check
|
||||
payload CRCs. Extraction admission uses these expansion totals.
|
||||
|
||||
- `radium-scene`: bounded sequential modern scene object-graph parsing with
|
||||
interned class IDs, named Elements, and ordered frame references. See
|
||||
[native scene parser](native-scene-parser.md) for corpus verification and limits.
|
||||
The semantic classification pass also retains the serialized class as evidence:
|
||||
`Bitmap`, `Sprite`, `Font`, `Text`, `Video`, `Shape`, and `StreamingFlipbook`
|
||||
become typed assets when an observation contains one unambiguous class. Mixed
|
||||
or auxiliary objects remain untyped until a stronger relationship is available;
|
||||
the path suffix is never used to guess a media class for `scene.radium`.
|
||||
|
||||
- `spike-probe`: LUKS2, SquashFS, SPKS, gzip and ZIP signatures. LUKS2/SquashFS
|
||||
generation labels match the existing corpus convention. Bare SPKS/gzip/ZIP
|
||||
remains undetermined. LUKS UUID and credential reference are reported when
|
||||
present. Magic recognition does not verify payload integrity.
|
||||
- `godot-sidecar`: `.import` and `.remap` section parsing, explicit source paths,
|
||||
platform-specific `path.*` targets, deduplicated `dest_files`, importer/type/UID,
|
||||
and exact target-name evidence. JSON-compatible Godot quoted strings and string
|
||||
arrays are parsed, including commas and escapes. Unsupported literals fail
|
||||
explicitly to opaque coverage. Parsing is bounded to 1 MiB. Resource types
|
||||
describe mapping targets; the sidecar itself is metadata, never a font/texture.
|
||||
When only an archived source path is known, the key is `File`, not an invented
|
||||
`res://` identity.
|
||||
The separate [native name pass](native-godot-names.md) resolves exact pack-scoped
|
||||
targets, verifies bounded orphan path hashes and persists append-only claims
|
||||
after spine indexing.
|
||||
|
||||
Verification command:
|
||||
|
||||
```
|
||||
cargo test --locked --offline --test decode
|
||||
```
|
||||
|
||||
Tests cover signature classification, generation uncertainty, malformed/unknown
|
||||
fallback, name deduplication and escaped paths, remap inference, oversized inputs,
|
||||
truncated storage and an HTTP call against an actual imported archive after the
|
||||
original input is removed. The endpoint test checks the scratch workspace stays
|
||||
empty.
|
||||
|
||||
Remaining Phase 2 work is explicit: native ZIP payload extraction, `spike_package`
|
||||
assembly/authentication/extraction, legacy scene layouts and Radium/DMD native
|
||||
decoding remain open. Native Godot name persistence and the
|
||||
[ext4 reader](native-ext4.md) are implemented separately in the working tree.
|
||||
The existing pinned Python/native
|
||||
adapters still perform those imports. This inspection endpoint does not eliminate
|
||||
materialization in the legacy extraction pipeline or demonstrate an end-to-end
|
||||
17.1x import amplification improvement.
|
||||
@@ -0,0 +1,44 @@
|
||||
# Native ext4 streaming integration
|
||||
|
||||
The working tree adds two entry points that avoid staging an entire filesystem
|
||||
image before extracting its regular files. These changes are not yet deployed.
|
||||
|
||||
`import-extract/8` recognizes an ext4 partition inside a validated disk partition
|
||||
map and passes the original image, absolute offset, and exact partition length
|
||||
to the existing native helper. It does not create a partition image. The helper
|
||||
still enforces slice bounds, ext4 geometry, journal state, output size, and safe
|
||||
paths; symlinks and special files remain in the retained original. A configured
|
||||
native failure is fatal rather than falling back to an unbounded alternative.
|
||||
|
||||
`POST /api/decode/ext4` accepts `snapshot`, `path`, `offset`, and `length`.
|
||||
`verstack decode-ext4 SNAPSHOT PATH --offset N --length N` uses the same archive
|
||||
operation. It reads the source through `SourceReader`, reserves metadata and
|
||||
output/capture resources, publishes an extracted child, and records the source
|
||||
artifact and byte slice in its receipt. The original image is retained. Neither
|
||||
entry point establishes filesystem authenticity: ext4 parsing and available
|
||||
filesystem checksums are not a cryptographic signature on the source content.
|
||||
|
||||
The import test creates a real ext4 filesystem inside an imageUSB/MBR wrapper,
|
||||
checks exact extracted bytes, proves no partition-sized sparse copy occurred,
|
||||
and verifies source bytes remain unchanged. All 20 import-source tests passed
|
||||
with `VERSTACK_NATIVE_EXT4=target/release/verstack`; HTTP fixtures require local
|
||||
socket access. The archive integration test additionally uses a scratch budget
|
||||
smaller than the archived source, removes the external source before extraction,
|
||||
and checks parentage, receipt, original preservation, skipped links, malformed
|
||||
slice rejection, HTTP authorization, and scratch cleanup. Staging temporarily
|
||||
adds owner read/traverse permission to the disposable tree when the source mode
|
||||
would make copying impossible, then restores the native mode in published
|
||||
snapshot metadata. All eight ext4 tests
|
||||
passed; the Godot pack-isolation/append-only recovery integration also passed
|
||||
(one separate corpus fixture remains explicitly ignored in that invocation).
|
||||
Log: `/tmp/verstack-root-streaming-tests.log`.
|
||||
|
||||
The 63,281,562,112-byte Pokémon 0.83 source was inspected read-only with
|
||||
`scripts/inspect_archived_disk.py`. Exactly 30,208 bytes were requested. Evidence
|
||||
in `data/validation/pokemon083-disk-map.json` identifies an imageUSB wrapper, one
|
||||
FAT partition and four LUKS partitions, including 25,769,803,776- and
|
||||
35,433,480,192-byte encrypted partitions. This proves that direct ext4 partition
|
||||
handling alone does not solve that full-size retry. The next required stage is
|
||||
seekable LUKS plaintext over archived ranges, followed by complete extraction and
|
||||
verification. ZIP staging, plugin input materialization, and encrypted/plaintext
|
||||
LUKS staging remain in the legacy adapter; the full streaming chain is open.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Native ext4 view and extraction
|
||||
|
||||
`src/ext4.rs` uses pinned `ext4-view` 1.0.0 for native ext filesystem reads.
|
||||
`Archive::ext4_list` and `Archive::ext4_read_range` use the existing seekable
|
||||
`SourceReader`, so stored filesystem images can be inspected without restoring
|
||||
whole images to scratch. A scoped request-channel bridge keeps the borrowed,
|
||||
non-`Send` source on its owning thread; no unsafe lifetime or thread-trait changes
|
||||
are used. The parser and its `Ext4` object stay on the worker thread.
|
||||
|
||||
The local helper is runnable without an archive config or archive instance lock:
|
||||
|
||||
```sh
|
||||
verstack ext4-extract filesystem.ext4 new-output-directory --budget 8589934592
|
||||
verstack ext4-extract disk.img new-output-directory --budget 8589934592 --offset 1048576 --length 4294967296
|
||||
```
|
||||
|
||||
It emits a JSON report containing file count, logical output bytes, and skipped
|
||||
symlink/special-file metadata. `plugins/import_extract.py` accepts an opt-in
|
||||
`settings.native_ext4` binary path. When configured, it calls this helper once
|
||||
instead of spawning `debugfs` for every directory and file. Helper failure is
|
||||
fatal; it does not silently fall back. When unset, the existing debugfs path
|
||||
remains. This change does not alter any frozen plugin bundle or live config.
|
||||
|
||||
Extraction preflights the entire bounded tree and logical output budget, then
|
||||
creates an exclusive destination. It emits regular files and directories,
|
||||
including empty directories; hard links become independent regular files.
|
||||
Ordinary permission bits are preserved, but setuid/setgid/sticky bits are never
|
||||
applied to host paths. Full source permission bits remain in listing evidence.
|
||||
Symlinks and special files are retained only as report entries; symlink targets
|
||||
are recorded, never followed. All-zero output chunks become holes. A guard
|
||||
removes a newly created output tree on error, cancellation, or worker panic;
|
||||
existing destinations are rejected and never removed.
|
||||
|
||||
Bounds include 8 MiB per source request, 4 million requests, a caller-controlled
|
||||
aggregate read budget (default four times output allowance plus 64 MiB), 16,384
|
||||
block groups, at most 64 KiB filesystem blocks, 200,000 entries, depth 64, 4 KiB
|
||||
paths/link targets, and 16 MiB aggregate entry-path bytes. Range reads return at
|
||||
most 16 MiB. SourceReader retains its existing 8 MiB coalescing cache: these read
|
||||
limits count bytes requested by the ext4 parser, not backend compressed traffic.
|
||||
Cancellation is checked between reads and entries/chunks; it cannot interrupt an
|
||||
already blocked backend read. Parser panics close the channel and become errors.
|
||||
|
||||
Dirty filesystems requiring journal recovery are rejected, rather than replayed.
|
||||
Unsupported filesystem features are reported by ext4-view. Non-UTF-8 names or
|
||||
link targets are rejected explicitly. Backslash/CR/LF filename characters are
|
||||
rejected to preserve the archive importer's path-safety contract. Non-64-bit images with nonzero reserved
|
||||
high block counts are rejected (the dependency combines those fields, so masking
|
||||
only in the adapter would disagree with the parser). Directory hard-link loops
|
||||
terminate at the traversal bounds. This is not a writable mount, does not preserve
|
||||
ownership/xattrs/ACLs/timestamps, and does not replace the LUKS plugin's separate
|
||||
`debugfs rdump` path. Decryption and outer package staging remain separate; the
|
||||
complete streaming Zip/LUKS/ext4/SPK tower is still unfinished.
|
||||
|
||||
Tests construct real filesystems with `mkfs.ext4` at 1 KiB and 4 KiB block sizes,
|
||||
check native bytes against source content and a debugfs reference, exercise
|
||||
filesystem offsets, sparse files, hard links, permission bits, symlinks, malformed
|
||||
bounds, read failures, cancellation, panic cleanup, an archived SourceReader,
|
||||
the standalone CLI, and the actual Python/native handoff. No production filesystem
|
||||
or catalog is mutated by these tests. This fixture evidence is not a live large
|
||||
SD-card import outcome.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Native Godot name recovery
|
||||
|
||||
`Archive::recover_godot_names(snapshot)` reads archived `.import`/`.remap`
|
||||
sidecars and appends accepted claims through the existing name-provenance API.
|
||||
`POST /api/decode/godot-names` accepts `{"snapshot":"..."}` and requires the
|
||||
normal `X-Verstack-Client: 1` header. Spine backfill automatically invokes the
|
||||
pass after committing observations; its response includes inserted-name counts
|
||||
and warnings. No artifact paths, bytes, logical identities or old claims change.
|
||||
|
||||
Targets must match a full `res://` path within the same archived
|
||||
`.pck/original/` or `.pck/decompressed/` tree as the sidecar. Different packs and
|
||||
representations never join by basename. Exact sidecar declarations produce T1
|
||||
claims, with source/target paths, snapshot and both content identities as
|
||||
evidence. Conflicting declarations for one target are reported and skipped.
|
||||
Missing or ambiguous spine observations are counted as unresolved.
|
||||
|
||||
Orphan cache entries under `.godot/imported/` or `.import/` retain the original
|
||||
basename and an MD5 suffix. Candidate directories come only from that same pack
|
||||
tree and its sidecar declarations. A unique candidate whose UTF-8 full `res://`
|
||||
path reproduces the suffix produces a T2 claim. This is a bounded forward hash
|
||||
check, not arbitrary preimage recovery. Unknown directories remain unresolved.
|
||||
|
||||
The pass accepts canonical relative resource components, rejects traversal and
|
||||
control characters, and bounds sidecars to 20,000 / 64 MiB total / 1 MiB each,
|
||||
candidate directories to 20,000 per tree and hash attempts to two million per
|
||||
snapshot. Work limits and source reads are checked before any claims are written.
|
||||
Existing accepted claims remain append-only, including older imported claims;
|
||||
this pass does not silently retract or overwrite them.
|
||||
|
||||
Verification covers two packs with identical target basenames, a distinct
|
||||
decompressed representation, conflicting declarations, invalid resource paths,
|
||||
cross-pack orphan candidates, automatic backfill, HTTP boundary checks and
|
||||
idempotent exact claim preservation. The golden hash is the real archived
|
||||
Stern_Aztech font path.
|
||||
|
||||
The real-firmware canary copies three Pokémon 0.86 sidecars and their three exact
|
||||
font targets (37,233 bytes) using read-only HTTP. Its isolated archive verifies
|
||||
each downloaded SHA-256 and archived BLAKE3, checks all names against the Python
|
||||
reference parser, then removes one sidecar from a second input fixture to test
|
||||
native orphan recovery. The production archive is never opened for writes.
|
||||
|
||||
```sh
|
||||
python3 scripts/collect_godot_canary.py data/validation/native-godot-20260916-1
|
||||
VERSTACK_GODOT_FIXTURE=data/validation/native-godot-20260916-1 \
|
||||
cargo test --locked --offline --test godot_names -- --include-ignored
|
||||
```
|
||||
|
||||
Release2 deployed the native recovery path. Its first full Pokémon 0.86 pass
|
||||
read 1,179 sidecars and appended 1,179 claims, but 1,062 texture sidecars
|
||||
failed parsing and their cache paths instead resolved through the T2 hash route.
|
||||
The cause was multiline importer metadata dictionaries. The subsequent parser
|
||||
fix tracks nested containers, quoted strings and comments with a 64-level bound;
|
||||
embedded section/assignment-looking data cannot become mapping evidence.
|
||||
Malformed, unterminated and conflicting assignments remain rejected.
|
||||
|
||||
The corrected working-tree parser matches all 1,179 real sidecars and 1,179
|
||||
mappings against the Python reference, with archived BLAKE3 verification and
|
||||
zero decoder fallbacks. Fixture: `data/validation/godot-sidecars-20260917/`;
|
||||
collector: `scripts/collect_godot_sidecars.py`; opt-in regression uses
|
||||
`VERSTACK_GODOT_SIDECARS` with `tests/decode.rs`. Corrected parser deployment
|
||||
and a full-corpus name backfill remain separate verification steps. Non-PCK layouts and arbitrary directory
|
||||
inference are not claimed as recovered.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Archive-backed LUKS streaming
|
||||
|
||||
The working tree now includes `verstack-luks`, a bounded native reader for one
|
||||
validated LUKS2 `aes-xts-plain64` segment. It reads encrypted ranges through
|
||||
`SourceReader`, checks the pinned `libcryptsetup` library hash, unlocks using a
|
||||
credential referenced by configuration, and keeps the key in zeroizing memory.
|
||||
The reader validates sector geometry, IV arithmetic, metadata/keyslot bounds,
|
||||
KDF demand, cancellation and cache invalidation. Neither keys nor credentials
|
||||
are accepted in HTTP payloads or command-line arguments.
|
||||
|
||||
`POST /api/decode/luks-ext4` composes that reader with the bounded native ext4
|
||||
walker and publishes a child snapshot. `verstack decode-luks-ext4` exposes the
|
||||
same operation. Resource admission charges cryptsetup KDF CPU/heap, encrypted
|
||||
read ranges, ext4 output and publication capture separately. The encrypted
|
||||
original remains retained, and receipts state that neither the ciphertext image
|
||||
nor a plaintext filesystem image was materialized.
|
||||
|
||||
The guarded Rust integration fixture creates an independent encrypted ext4
|
||||
source, verifies output and source preservation, rejects a wrong credential,
|
||||
checks the pinned library and confirms no credential text enters run receipts.
|
||||
It passed in 99.29 seconds. The standalone reader has four tests covering both
|
||||
XTS key sizes, 512/4096-byte sectors, random seeks, cancellation, bounds,
|
||||
stale-cache errors, library pinning and KDF admission.
|
||||
|
||||
Read-only probing of the real Pokémon 0.83 archive unlocked all four encrypted
|
||||
partitions. The probe requested 737,280 bytes in total (184,320 per partition,
|
||||
plus a separate 64 KiB metadata-only profile probe). Each uses one
|
||||
`aes-xts-plain64` segment,
|
||||
512-byte sectors, a 16 MiB data offset, a 32-byte key and PBKDF2 with 250,000
|
||||
iterations. A bounded partition-3 verifier has now passed: 92 files, 280,825
|
||||
logical bytes, every file read back equal to the native plaintext ranges, and
|
||||
exactly 8,388,608 ciphertext payload bytes plus bounded capsule/preflight reads.
|
||||
This did not initially establish full partitions 5/6 publication or an independent
|
||||
legacy-file oracle. Evidence is `data/validation/pokemon083-native-luks-probe.json`
|
||||
and `data/validation/pokemon083-native-luks-ext4-part3.json`.
|
||||
|
||||
The backend capability is now deployed with a fresh rollback checkpoint and a
|
||||
live partition-5/partition-6 publication canary. Frontend deployment checks and
|
||||
the broader whole-import campaign remain. Native LUKS stays gated for broad
|
||||
automatic imports until those checks and a controlled default-decoder campaign
|
||||
are complete.
|
||||
@@ -0,0 +1,79 @@
|
||||
# Archived LUKS2 range decoding
|
||||
|
||||
`POST /api/decode/luks-ext4` accepts `snapshot`, `path`, `offset`, `length`, and
|
||||
optional `package_key` (default false, selecting the disk credential profile).
|
||||
`verstack decode-luks-ext4 SNAPSHOT PATH --offset OFFSET --length LENGTH`
|
||||
performs the same operation with archive ownership. The existing mutation guard
|
||||
applies. Requests never contain credential bytes or arbitrary credential paths.
|
||||
|
||||
The configured `import-extract` settings supply `disk_key_file` and
|
||||
`disk_key_encoding`, or `key_file`/`key_encoding` for package-key requests.
|
||||
Explicit raw, text-stripped and vcmailbox encodings are supported; auto-guessing
|
||||
is not. `native_cryptsetup_library` must be an absolute library path and
|
||||
`native_cryptsetup_sha256` its SHA-256 pin. This implementation does not install
|
||||
those settings or change the live pipeline.
|
||||
|
||||
The source chain is `SourceReader → bounded partition Slice → LuksReader →
|
||||
native ext4`, preserving the archived encrypted source. Only encrypted header
|
||||
and keyslot ranges enter a private temporary capsule. Metadata admission occurs
|
||||
before capsule loading; declared KDF CPU/heap demand is admitted before unlock.
|
||||
Filesystem listing computes output allocation, then publication reserves both
|
||||
output/capture scratch and archive capacity. No full ciphertext or plaintext
|
||||
image is staged. The original snapshot stays the parent of the extracted child,
|
||||
and a receipt records profile, source artifact, library pin and read accounting.
|
||||
Credentials and volume keys never enter receipts, CLI arguments or logs.
|
||||
|
||||
The publication path supports the same ext4 regular-file/mode preservation and
|
||||
symlink/special-file exclusion as direct native ext4. Unsupported cipher or
|
||||
filesystem profiles remain explicit failures. A KDF already executing in the
|
||||
library cannot be interrupted midway; cancellation is checked before and after
|
||||
bounded source work. Existing ZIP/gzip/split-wrapper and other container paths
|
||||
remain separate work; this is not a claim that the full import chain is native.
|
||||
|
||||
See `crates/verstack-luks/README.md` for bounds, library API provenance and actual
|
||||
63 GB source header evidence. Root tests build an independently OpenSSL-encrypted
|
||||
ext4 fixture, publish it through guarded HTTP, compare extracted bytes, retain
|
||||
the encrypted original, and reject an incorrect configured credential.
|
||||
|
||||
The actual retained Pokémon 0.83 partition 3 now passed complete regular-file
|
||||
extraction and readback: 92 files, 280,825 logical bytes, with all 92 compared
|
||||
against decrypted filesystem ranges. The explicit read-only example used the
|
||||
reader's bounded 8 MiB cache and read exactly 8,388,608 ciphertext payload bytes,
|
||||
plus 163,840 encrypted capsule bytes and 16,384 preflight bytes. Its HTTP bridge
|
||||
kept a 16 MiB aggregate cap and 4 MiB per-request cap. No plaintext image file
|
||||
was staged; the exclusively owned output directory was removed automatically.
|
||||
The receipt is `data/validation/pokemon083-native-luks-ext4-part3.json`.
|
||||
|
||||
This fixture exposed a write-only source mode (ordinary 0310). Publication now
|
||||
adds owner access only to its disposable capture tree, then projects the source
|
||||
ordinary modes back into published Entry metadata; privileged permission bits
|
||||
are not applied to host output. The example similarly grants temporary owner-read
|
||||
access solely for verification. An earlier probe failed at that readback step;
|
||||
a subsequent small-cache attempt ended with a bridge EOF before a success
|
||||
receipt. The final bounded-cache attempt completed successfully. Production
|
||||
reader budgets were not raised to bypass either failure.
|
||||
|
||||
Actual file readback uses the same native filesystem view; it is not an
|
||||
independent legacy-file oracle. Crypto correctness additionally has independent
|
||||
OpenSSL ciphertext fixtures and an independently encrypted root publication test.
|
||||
The live backend has since published full partitions 5 and 6 through the same
|
||||
native reader. Partition 5 produced 18 regular files and 1,064,022,428 logical
|
||||
bytes; partition 6 produced 689 regular files and 2,276,741,239 logical bytes.
|
||||
Receipts are `data/archive/receipts/` and the compact verification records are
|
||||
`data/validation/pokemon083-native-luks-ext4-part5.json` and
|
||||
`data/validation/pokemon083-native-luks-ext4-part6.json`.
|
||||
|
||||
Local import helper (opt-in): `luks-extract IMAGE NEW_DIRECTORY --request
|
||||
REQUEST.json --budget BYTES [--offset N --length N] [--package-key]` does not
|
||||
open the archive. The bounded request supplies `settings` and `cpu_threads`;
|
||||
credentials are read only from configured file references. `native_luks` and
|
||||
`native_luks_sha256` select and pin the helper in import-extract; the existing
|
||||
libcryptsetup path/hash settings remain mandatory. `native_luks_memory_bytes`
|
||||
defaults to 256 MiB and is charged in addition to the Python plugin heap.
|
||||
KDF demand exceeding that allowance fails before unlock. Disk partitions pass
|
||||
ranges of the staged original directly, without staging encrypted partitions or
|
||||
plaintext filesystem images. Selected helper errors are fatal, never a trigger
|
||||
for legacy fallback. Evidence retains the source modes and skipped symlinks.
|
||||
This local plugin path currently requires source files/directories readable by
|
||||
the invoking user; the archive publication API separately supports unreadable
|
||||
source modes through temporary staging permissions. The helper stays opt-in.
|
||||
@@ -0,0 +1,178 @@
|
||||
# Native Radium archive integration
|
||||
|
||||
`POST /api/decode/radium` accepts `{"snapshot":"…","path":"…/image.bin"}` under
|
||||
the existing mutation guard. `verstack decode-radium SNAPSHOT PATH` does the same
|
||||
with archive ownership. It reads archived ranges through SourceReader without
|
||||
restoring the input, publishes a child snapshot, and preserves the original.
|
||||
`verstack radium-extract IMAGE NEW_DIRECTORY --budget BYTES --source-path PATH`
|
||||
is the config-free helper. Inspection registry selection for image*.bin validates
|
||||
directories only; it never claims to have decoded the frames.
|
||||
|
||||
The outputs preserve each consumed encoded bitmap record, exact indexed pixels
|
||||
(`.idx`), and a grayscale/transparent-255 PNG labelled `approximate_palette`.
|
||||
The PNG is a preview representation. Exact pixel identity hashes the versioned
|
||||
`radium-indexed8/1` domain, little-endian dimensions, and pixel indices; original
|
||||
record, index-file and PNG byte hashes remain separate. No original palette or
|
||||
final screen composition is claimed. Section 8 streams original PCM and preserves
|
||||
flags, channel count and samples. The direct native decoder leaves sample rate and duration null. The optional Python
|
||||
adapter adds WAV outputs only using the existing exact source-SHA256 rate profiles,
|
||||
configured hash-scoped profiles, or verified ELF ALSA-call evidence. WAV samples
|
||||
stream from retained native PCM in 1 MiB chunks; unknown rates stay unavailable.
|
||||
The adapter retains exact source SHA256 and sound index for cue-bank joins.
|
||||
|
||||
`media-evidence.json` feeds the existing publication ingestion path immediately.
|
||||
Native observations and relationships therefore exist in the catalog without a
|
||||
whole-catalog backfill. DMD header identities and directory ordinals are recorded
|
||||
as provenance, not asserted to be stable cross-release keys. Frame assets use
|
||||
ContentOnly keys for exact decoded pixels. Every output occurrence retains its
|
||||
source path, source artifact, parent snapshot, directory index and encoded range.
|
||||
Repeated pixels and repeated header identities do not erase occurrences.
|
||||
|
||||
Each connected decode-dependency component gets a canonical JSON Sequence asset,
|
||||
including singleton components. Its ordered frame list stores pixel identities
|
||||
and base references normalized to positions within that component. The canonical
|
||||
bytes omit source filenames and absolute directory ordinals; these remain in
|
||||
provenance and `edge_observations`. `frame_of` points to the Sequence and
|
||||
`depends_on` to the actual base occurrence, preserving both `base_index` and
|
||||
`base_identity`. These are structural, directory-ordered dependency sequences,
|
||||
not verified playback animations: timing, scene membership and audio association
|
||||
are separate evidence. ContentOnly Sequence keys make no unsupported structural
|
||||
identity claim. Backfill and classification retain the native keys and edges.
|
||||
|
||||
The index is limited to 50,000 records and 4 MiB serialized directory metadata;
|
||||
encoded record reads are capped at 32 MiB, exact frames at 4096×4096, and the DMD
|
||||
cache at 64 MiB. PCM copies use 1 MiB chunks. Evidence has a 32 MiB bound. Direct decoding reserves 512 MiB heap and one CPU;
|
||||
the Python helper estimate adds that heap allowance to its parent-process allowance. Preflight
|
||||
sums encoded extents, dimensions-based PNG/index bounds, PCM lengths, sequence
|
||||
records and filesystem-node allowance. Archive admission reserves extraction plus
|
||||
capture copies before creating output. Output writes also enforce a cumulative
|
||||
budget. Cancellation, decode failure and panic remove only the newly created
|
||||
output directory; an existing destination is never removed. Unsupported section-8
|
||||
layouts (including real SPIKE 2 sentinels) leave explicit section-3-only coverage.
|
||||
Invalid DMD frames fail extraction and retain the source rather than silently
|
||||
publishing a complete-looking set.
|
||||
|
||||
The optional media plugin selection uses `settings.native_radium` with a required
|
||||
`native_radium_sha256`. Configuration regeneration preserves both. A configured
|
||||
helper failure is fatal, with no silent Python fallback. This hook handles
|
||||
image*.bin with native section-8 layout. A bounded, validated pre-launch header2
|
||||
selector keeps the existing full Python handler for the older plaintext sound
|
||||
format (including the retained GOT fixture); this describes a sound format, not a
|
||||
hardware generation. Admission mirrors that dispatch. Other containers retain
|
||||
their existing handlers. It preserves
|
||||
all occurrence metadata while prefixing paths into the plugin output. The Python
|
||||
plugin still consumes the pipeline's staged/shared input; only direct native
|
||||
archive decoding currently avoids input materialization. No live configuration,
|
||||
frozen bundle, release binary or deployment was changed by this integration.
|
||||
|
||||
## Verification
|
||||
|
||||
The standalone crate's full real-corpus gate verified all 18,636 frames (17,159
|
||||
GOT and 1,477 Pokémon), all 2,491 Pokémon PCM chunks, and both source SHA-256s.
|
||||
See `crates/verstack-radium/README.md` for source hashes and reproduction commands.
|
||||
That proves the decoder independently of the root archive publication gate below.
|
||||
|
||||
The root integration tests exercise guarded HTTP publication, exact bytes and
|
||||
separate representations, repeated same-ID/wrapped-ID dependency occurrences,
|
||||
immediate catalog Sequence/edge ingestion, original preservation, backfill,
|
||||
classification, SPIKE 2 sentinel coverage, config-free CLI, actual pinned Python
|
||||
helper execution, inadequate budgets, invalid frames and existing destinations.
|
||||
A unit test injects cancellation and reader panic after output creation and checks
|
||||
rollback. The ignored integration test packages the first 64 unchanged encoded
|
||||
records from each real source in explicitly synthetic directory framing, publishes
|
||||
them through the archive, and compares all 128 resulting pixel hashes against the
|
||||
full-source reference receipts. It is not a claim that both whole real images
|
||||
have been published through this new path.
|
||||
|
||||
```
|
||||
cargo test --offline --locked --test native_radium -- --include-ignored
|
||||
cargo test --offline --locked --lib radium::tests
|
||||
tools/decoder-env/bin/python -m unittest discover -s tests -p test_native_radium.py
|
||||
```
|
||||
|
||||
Native media plugin admission uses the exact frozen config. Its bounded directory
|
||||
and frame-header preflight is cached by immutable source identity and helper hash
|
||||
(up to 256 entries), charged one CPU/64 MiB heap, and runs before a queue claim's
|
||||
catalog writer transaction. Eligibility and the job fingerprint are rechecked
|
||||
under that transaction. The demand includes staged source allocation, both output
|
||||
copies and aggregate evidence; non-native plugin estimates are unchanged. Potential verified WAV copies and headers
|
||||
are reserved in addition to native PCM, and aggregate evidence is bounded. For the
|
||||
1,154,884,359-byte Pokémon image, native output plus potential WAV allowance is
|
||||
2,567,004,644 bytes and the complete plugin reservation is 6,356,043,471 bytes,
|
||||
within an 8 GiB pool. The 784,411,100-byte GOT image has a direct native DMD
|
||||
output bound of 2,241,457,054 bytes; configured plugin dispatch keeps its complete
|
||||
legacy sound-format handler and existing admission estimate. Shared input
|
||||
staging considers both maximum child CPU and maximum child heap.
|
||||
|
||||
The serial decoder's CPU request does not establish a one-core bound for the
|
||||
archive backend: existing rustic import/publication uses parallel workers. The
|
||||
full-corpus publication measurement includes the preceding import and records
|
||||
that baseline separately from native decoding; it is not a sustained workload
|
||||
or a claim that archive publication worker accounting has been solved.
|
||||
|
||||
The optimized config-free helper was also run on the unchanged complete GOT image:
|
||||
all 17,159 index hashes matched the reference, 17,139 dependency Sequence records
|
||||
were emitted, and the evidence file was 18,582,808 bytes. Elapsed extraction plus
|
||||
readback was 15.82 seconds; `/usr/bin/time` measured 42,204 KiB peak child RSS.
|
||||
This supports the helper's 512 MiB allowance for that corpus, not a claim about
|
||||
whole-archive publication memory. A bounded real Pokémon audio test verified its
|
||||
complete source SHA256, then converted one 278,896-byte retained PCM range to WAV
|
||||
at the existing hash-proven 44.1 kHz rate; the WAV samples matched the reference
|
||||
SHA256 exactly. Its source hash and bank index remained attached.
|
||||
|
||||
Audio canonical identities use `verstack-pcm-s16le/1` framing with channel count,
|
||||
explicit unknown or verified sample rate, and the retained raw PCM BLAKE3 hash,
|
||||
then SHA-256. The raw artifact identity is unchanged. Unknown-rate PCM keeps this typed identity.
|
||||
For verified-rate PCM/WAV pairs, catalog identity now uses the exact WAV artifact
|
||||
BLAKE3 as a transitional compatibility key matching legacy WAV assets; both
|
||||
representations retain the typed PCM identity as their payload ID. Different rates
|
||||
or channel counts produce different WAV headers and keys. This is compatibility
|
||||
with existing content identities, not completion of PLAN-wide canonical PCM
|
||||
migration. Native Radium remains opt-in; this implementation does not change the
|
||||
live configuration. The full GOT publication measurement passed as recorded below.
|
||||
|
||||
The complete retained GOT source also passed archive publication and byte-hash
|
||||
readback in the optimized build: 17,159 frames, 17,139 dependency Sequence
|
||||
occurrences, 68,622 files, and 202,314,277 logical output bytes. Import completed
|
||||
at 16.78 seconds, publication at 129.31 seconds, and all readbacks at 132.45
|
||||
seconds from test start. After-import resident memory was 446,980 KiB; whole
|
||||
process peak after publication was 872,384 KiB. The observed increase of
|
||||
425,404 KiB (about 415 MiB) fits the 512 MiB incremental native allowance for
|
||||
this fixture; the whole-process peak includes the already resident archive
|
||||
backend and does not represent native decoder memory alone. These measurements
|
||||
are a single-corpus validation, not a sustained workload guarantee.
|
||||
|
||||
The earlier debug run also completed successfully before the requested cleanup:
|
||||
all the same output counts passed, elapsed 2,284.40 seconds, peak RSS 811,404 KiB.
|
||||
Its exclusively owned temporary fixture was automatically removed on completion;
|
||||
no original or reference source was removed. The optimized phase measurements
|
||||
supersede debug timing for deployment evidence; neither run failed decoding.
|
||||
Both receipts remain in `data/validation/native-radium-20260917/` as
|
||||
`root-got-publication-release.log` and `root-got-publication.log`.
|
||||
|
||||
Legacy/native audio compatibility is verified before indexing, outside the catalog
|
||||
writer transaction. The archive strips any supplied `verified_wave_identity`,
|
||||
checks the canonical 44-byte signed-16-bit PCM WAV header against rate/channels,
|
||||
streams WAV samples against retained PCM, verifies both actual BLAKE3 artifact
|
||||
hashes and the typed PCM interpretation, then records the verified WAV identity
|
||||
in artifact provenance. Spine replay uses this server-established identity.
|
||||
Unknown rates never get a WAV compatibility key; malformed or changed samples
|
||||
fail verification. Buffers are bounded to two 1 MiB blocks plus archive readers.
|
||||
|
||||
The hash-verified retained Pokémon 0.85 bank passed byte-for-byte WAV parity for
|
||||
all 2,491 sounds between the actual legacy extractor and native streamed writer.
|
||||
A disposable archive regression checks legacy/native identical WAV identity,
|
||||
shared PCM/WAV identity, existing accepted names and review claims, and two
|
||||
idempotent spine replays. Additional archive fixtures distinguish 44.1/48 kHz and
|
||||
mono/stereo, reject changed WAV samples despite forged receipt identity, and keep
|
||||
unknown-rate PCM separate. No historical names, reviews, aliases or live catalog
|
||||
rows are rewritten. Compatibility is exact-WAV-byte based: alternate headers,
|
||||
other audio encodings, resampling, and old ambiguous raw-PCM identities remain
|
||||
outside this transition.
|
||||
|
||||
Cross-release delta selection now chooses the highest-priority representation
|
||||
shared by both versions before comparing bytes. A newly retained original PCM
|
||||
therefore does not displace the decoded WAV when comparing against a legacy
|
||||
WAV-only asset. The legacy/native archive fixture spans two releases and verifies
|
||||
that identical WAV audio produces no delta while names and review history remain
|
||||
attached after replay.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Native scene.radium graph reader
|
||||
|
||||
`src/decode/scene.rs` reads the modern little-endian cereal portable-binary
|
||||
Composition sequentially through `SourceReader`. It is registered as
|
||||
`radium-scene`, and requires complete consumption before publishing a graph.
|
||||
It does not scan for plausible names, textures, or frame headers.
|
||||
|
||||
The grammar reference is `tools/src/ske-radium/radium.py` at commit
|
||||
`f3a3dcaad59c558910b56c4cc6497899f5faba9b`, a reader derived from reverse
|
||||
engineering and game binary decompilation. The native reader preserves the
|
||||
polymorphic class registry, shared object references, typed object fields,
|
||||
named Elements, ordered character keyframes, and StreamingFlipbook frame lists.
|
||||
Class names are interned by polymorphic IDs. Ordinary strings in this grammar
|
||||
are inline length-prefixed UTF-8; a general interned string table is not established.
|
||||
|
||||
A null polymorphic pointer consumes only the zero type word. The reference
|
||||
reader rejected this case; the native reader follows cereal's
|
||||
[polymorphic serialization implementation](https://uscilab.github.io/cereal/assets/doxygen/polymorphic_8hpp_source.html).
|
||||
A Jurassic Park scene exercised this case in Shape.bitmap at offset 62.
|
||||
|
||||
BinaryFile inline data, Spine JSON, and atlas payloads are bounded source slices;
|
||||
they are not loaded or rendered. External BinaryFile filenames are retained as
|
||||
references. The reader caps file size at 128 MiB, aggregate collection entries
|
||||
and objects at 100,000 each, nesting at 64, individual strings at 16 MiB and total
|
||||
string bytes at 32 MiB. It rejects unknown classes, incompatible reference types,
|
||||
duplicate IDs/map keys, dangling references, invalid strings/scalars, truncation,
|
||||
and trailing data. Failed decoding falls through to mandatory ContentOnly.
|
||||
|
||||
## Verification
|
||||
|
||||
On 2026-09-16, read-only archived file requests supplied 42 distinct scenes from
|
||||
seven titles (the four smallest, median, and largest at most 16 MiB per title).
|
||||
The manifest, content hashes, native summaries, and independent reference report
|
||||
are under ignored `data/validation/scene-fixtures/`; proprietary scenes are not
|
||||
committed. Total input: 105,224,714 bytes.
|
||||
|
||||
All 42 parsed to EOF, yielding 21,664 objects, 3,542 Elements, and 3,618
|
||||
`instantiates` occurrences. Object counts and every Element name plus its full
|
||||
ordered `(frame, character-reference)` list matched the independent Python reader.
|
||||
The reference harness disabled image/sound output and handled the verified null
|
||||
polymorphic case; it otherwise used the upstream grammar.
|
||||
|
||||
Reproduce the independent comparison with
|
||||
`tools/decoder-env/bin/python scripts/check_scene_reference.py data/validation/scene-fixtures/manifest.json data/validation/scene-fixtures/native-report.jsonl`.
|
||||
|
||||
Observed types were BinaryFile, Element, Video, Bitmap, Image, Sprite, Font,
|
||||
FontGlyph, FontInstance, Text, and Shape. StreamingFlipbook/frame_of is covered by
|
||||
constructed grammar fixtures, not by this archived sample. Sound, Spine, and
|
||||
callback variants likewise have no real-sample verification here.
|
||||
|
||||
`cargo test --offline --locked --test scene --test decode` passes 16 tests.
|
||||
Scene tests cover shared references, ordered keyframes, external binary references,
|
||||
frame_of ordering, null pointers, every truncated prefix of a complete graph,
|
||||
unsupported classes, duplicate/dangling IDs, trailing bytes, and allocation bounds.
|
||||
Catalog tests exercise actual imports and the guarded POST endpoint, idempotency,
|
||||
repeated edges across releases, game scoping, duplicate names, malformed inputs,
|
||||
and metadata-only observations. A cyclic shared-reference fixture remains finite.
|
||||
`examples/inspect_scene.rs` produces compact summaries for local fixture comparison.
|
||||
|
||||
## Remaining limits
|
||||
|
||||
The parser supports the modern Text tail. Older Batman 0.65 Text layout and
|
||||
big-endian archives are unsupported. It does not claim full support for every
|
||||
Radium generation, scene rendering, native archive extraction, or generic string
|
||||
interning. Empty or duplicate Element names do not produce SceneInstance keys.
|
||||
Read-only inspection emits source offsets and relationships. Catalog integration
|
||||
is described below; it does not make these slices decoded images or canonical graphs.
|
||||
|
||||
## Catalog integration
|
||||
|
||||
Newly committed snapshots automatically index supported `scene.radium` entries
|
||||
after original content registration. Parse failures keep the original archived file
|
||||
and emit a diagnostic. Existing scenes can be indexed explicitly with
|
||||
`POST /api/decode/scene`, JSON `{"snapshot":"...","path":".../scene.radium"}`
|
||||
and the normal `X-Verstack-Client: 1` mutation header. The endpoint uses the shared
|
||||
mutation guard. `Archive::decode_scene` exposes the same transaction to callers.
|
||||
|
||||
Unique nonempty Element names become SceneInstance keys scoped by repository and
|
||||
version-normalized source path. Accepted T1 names retain snapshot, artifact and
|
||||
object-ID provenance. Empty/duplicate names remain ContentOnly. Anonymous objects
|
||||
use BLAKE3 of their actual serialized source slice, including serialization IDs;
|
||||
this is intentionally not semantic graph equivalence. Cycles remain references,
|
||||
and no recursive graph hashing occurs. Cumulative overlapping-slice hash work is
|
||||
capped at 512 MiB per scene.
|
||||
|
||||
The original source ContentOnly observation is retained. Object observations use
|
||||
`representation=metadata`, with source offset/length, typed fields and source
|
||||
artifact in `decode_metadata`; `payload_id` is NULL. Observation paths refer to the
|
||||
stored source file. They do not represent separately extracted byte streams, and
|
||||
cannot be fetched through the decoded-asset API. Named-instance comparisons may
|
||||
report serialization changes; they do not yet compare canonical graph behavior.
|
||||
|
||||
`edge_observations` preserves each ordered instantiates/frame_of occurrence per
|
||||
version, snapshot, path and source/target serialization object IDs. `edges` is a
|
||||
deduplicated summary with NULL ordinal, not the authoritative ordered sequence.
|
||||
Reindexing is idempotent. No catalog rows are published until the entire supported
|
||||
graph and all bounded source-slice hashes have succeeded.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Native SPK archive integration
|
||||
|
||||
The root crate now depends on `crates/verstack-spk` and uses its bounded raw-SPKS
|
||||
index parser and streaming verifier. The native decode registry selects
|
||||
`native-spk` for a complete supported index and retains ContentOnly identity.
|
||||
Inspection explicitly reports `payload_verified: false`; parsing metadata is not
|
||||
payload verification. Bad structures retain the existing probe/content fallback.
|
||||
|
||||
The runnable paths are:
|
||||
|
||||
```sh
|
||||
verstack spk-inspect input.spk
|
||||
verstack spk-verify input.spk
|
||||
verstack spk-extract input.spk new-output-directory --budget 8589934592
|
||||
verstack --config config.json decode-spk SNAPSHOT_ID path/to/stored.spk
|
||||
```
|
||||
|
||||
The first three commands need no archive config or instance lock. The last uses
|
||||
an exclusively owned archive, like other mutating CLI operations. Against a
|
||||
running server, use `POST /api/decode/spk` with the normal mutation header and
|
||||
JSON `{ "snapshot": "...", "path": "..." }` instead. The existing shared
|
||||
mutation reservation remains held until publication finishes.
|
||||
|
||||
`Archive::spk_index`, `verify_spk`, and `decode_spk` read the original using
|
||||
`SourceReader`; no whole input image is restored to scratch. Decode reserves
|
||||
parsing CPU/heap first and atomically grows the scratch allowance from the
|
||||
summed indexed stored payload bytes, two output copies, and filesystem node
|
||||
allocation overhead. A pressured nested upgrade fails immediately instead of
|
||||
waiting with a partial allocation. Cancellation/shutdown checks run between
|
||||
source reads/seeks and during normal capture/publication.
|
||||
|
||||
Decode creates a real Extracted child snapshot only after every member passes
|
||||
MD5 and HMAC-SHA1 verification. Regular files retain ordinary permission bits;
|
||||
symlinks/special files are not created on the host. The original snapshot and
|
||||
container stay intact. A receipt records the original content identity, every
|
||||
indexed source range/digest/mode, extraction results, parent/output snapshots,
|
||||
and run ID. Failure before publication cleans temporary output and preserves
|
||||
originals. As in the existing plugin publication path, a post-publication receipt
|
||||
I/O failure can leave a published snapshot and failed run; publication is not
|
||||
claimed to be a transaction across storage, catalog, and receipts.
|
||||
|
||||
For existing imports, set `settings.native_spk` to a helper binary and
|
||||
`settings.native_spk_sha256` to its exact SHA-256. The Python SPK adapter uses this
|
||||
helper only for raw SPKS inputs and preserves the legacy output directory layout.
|
||||
A bad pin or native extraction error is fatal and includes the helper diagnostic;
|
||||
it never silently falls back. Split SquashFS and other wrappers still use the
|
||||
explicitly configured legacy tool and its existing hash pin. Output budgets include conservative filesystem node overhead and subtract earlier
|
||||
candidate outputs using allocated blocks, logical bytes, and node overhead. Pipeline regeneration preserves each
|
||||
plugin's opt-in path/hash separately and does not enable the helper elsewhere.
|
||||
No live settings, frozen bundle, or release binary were changed by this work.
|
||||
|
||||
`POST /api/decode/spk-member` accepts `{ "snapshot": "...", "path": "archive.zip",
|
||||
"member": "firmware/payload.spk" }` and performs verified publication through a
|
||||
bounded `ZipMemberReader`. The member is never staged as a separate file:
|
||||
indexing, MD5/HMAC verification, and extraction read the view directly from the
|
||||
retained ZIP. The child receipt records both wrapper paths and the ZIP parent
|
||||
remains retained. This establishes the ZIP → SPK publication link; ZIP →
|
||||
LUKS/ext4 and SPK → Radium remain separate stages.
|
||||
|
||||
The standalone crate README documents the grammar, source attribution, limits,
|
||||
and corpus verification. Full Zip/LUKS/SquashFS/SPK/Radium streaming composition
|
||||
remains unfinished: native SPK archive reads and extraction are one implemented
|
||||
stage. The Python wrapper pipeline still stages/decrypts wrappers and retains
|
||||
inner wrappers as before.
|
||||
|
||||
Tests exercise registry metadata claims, guarded HTTP publication, original
|
||||
retention, source-range receipts, checksum-failure rollback, config-free CLI,
|
||||
the actual pinned Python/native helper call, pin/diagnostic handling, cumulative
|
||||
output allowance, and pipeline preference preservation. A separate explicit
|
||||
real-fixture regression publishes the retained Pokémon system package through
|
||||
an archive with a 64 MiB scratch pool and streams every output back to check its
|
||||
original MD5. That derived fixture is the complete original system SPK0 with
|
||||
its outer count/length rewritten; it is not a full-game import or live outcome.
|
||||
|
||||
Validation completed for this integration: all five Rust native-SPK tests passed,
|
||||
including the explicitly enabled retained-fixture test (103.58 seconds for the
|
||||
combined suite); ten existing decoder tests passed. Five native-helper Python
|
||||
regressions and nineteen import-source regressions passed. The real archive
|
||||
fixture emitted 24,382,441 payload bytes across eighteen files with 64 MiB of
|
||||
scratch admission. This is disposable-fixture evidence, not a deployed outcome.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Offline ASR provenance
|
||||
|
||||
`scripts/offline_asr.py` provides a bounded local pass for archived WAV audio.
|
||||
It reads a JSONL manifest (`audio_path`, with an optional `source_ref` such as
|
||||
`radium:8:42`) and emits JSON containing the source SHA-256, PCM geometry,
|
||||
transcript status, and provenance. The report always records
|
||||
`network_access: false` and `credentials_required: false`.
|
||||
|
||||
The default `auto` mode is a readiness pass. `--backend sidecar` imports a
|
||||
reviewed transcript from a local JSON object keyed by `source_ref` or absolute
|
||||
audio path; these rows are explicitly marked `reviewed_local_sidecar`, never
|
||||
machine confidence. `--backend whisper --model /path/to/model` runs the locally
|
||||
installed `whisper` package against an existing model file and marks output as
|
||||
`machine_draft`. A missing model is an error and no model download is attempted.
|
||||
Remote URLs and symlinked audio are refused.
|
||||
|
||||
Example:
|
||||
|
||||
```sh
|
||||
python3 scripts/offline_asr.py manifest.jsonl \
|
||||
--backend sidecar --transcripts reviewed-transcripts.json
|
||||
```
|
||||
|
||||
This is a local/report-only capability. It does not promote names, modify the
|
||||
catalog, or assert that a transcript identifies a Radium sound. Native/script
|
||||
references and human review remain required for semantic naming.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Serve-time PCM WAV synthesis
|
||||
|
||||
`GET /api/media/pcm-wav` wraps one retained raw PCM artifact in a canonical
|
||||
44-byte RIFF/WAVE header while streaming the original bytes. The caller must
|
||||
declare `format=pcm_s16le`, a sample rate from 8 kHz through 192 kHz, and one
|
||||
or two channels. The endpoint rejects oversized or non-frame-aligned payloads
|
||||
before opening the reader.
|
||||
|
||||
Example:
|
||||
|
||||
```
|
||||
/api/media/pcm-wav?snapshot=<id>&path=raw.pcm&format=pcm_s16le&sample_rate=44100&channels=2
|
||||
```
|
||||
|
||||
The response carries immutable caching, source-artifact, format, rate, and
|
||||
channel headers. This is a playback/containerization capability; it does not
|
||||
promote an unverified sample rate into native audio identity or enable native
|
||||
decoding by default. `tests/video_audio.rs` verifies the header, streamed
|
||||
payload, provenance headers, and fail-closed format validation.
|
||||
@@ -0,0 +1,261 @@
|
||||
{
|
||||
"cohort": "existing decoded GOT DMD PNG fixtures; synthetically doubled with nearest-neighbor, not actual release-change correspondence",
|
||||
"frames": 28,
|
||||
"low_information": 0,
|
||||
"method": "image-perceptual/1",
|
||||
"resized_candidates": 28,
|
||||
"rows": [
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "da37847478bcd8bb3707a835ab0d1c9956dfe2f8f2cc0b095f10e131f281aeb4",
|
||||
"file": "frame-215.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "ed154ba4f673e507a52f7984df19cfb71fb66027e96eaa1021ab303222baa0f5",
|
||||
"file": "frame-216.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "8fcd864916e9f0f4aa2ed463ee78a39be3b442f72553fb292057f8723255dfe1",
|
||||
"file": "frame-217.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "94e833460ab038d601fdb925f963f15b43b7bdb0814e30a7cd643d4209ebf95e",
|
||||
"file": "frame-218.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "ad5e3412a635148178d8cc069ab750b2cfd18c2d60594e25c99645bd73ca6e0c",
|
||||
"file": "frame-219.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "781ed7b7750608a27f136640d1e583afeac970318bd1ba8d21663b0cb93383fc",
|
||||
"file": "frame-220.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "5c2fd00779d70db399eb916b57922ec0822d397073839fa68db088896ecfd452",
|
||||
"file": "frame-221.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "1051adc31bf73e9536523e4a6cf65f8c7dbf6729530ea8e967790d066faab8f6",
|
||||
"file": "frame-222.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "91aa3c41a0af64e4fb016260bdf5f71dbd86f4b5b2a0b4ae2494d270a032dcae",
|
||||
"file": "frame-223.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "3aa833ceee7f8c28ce92ff249d89a20c728f8d0ff92a278d869b4b5d33ae3bd4",
|
||||
"file": "frame-224.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "d3bd824b89aa33192f83ea2ee36d055027c1b8626fe7de7d295995259ffc07ad",
|
||||
"file": "frame-225.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "e92f67abf40d03c8992bc622da1ea7aa15d7ef576cac49274e3e0b42e8cf1337",
|
||||
"file": "frame-226.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "03f36a008db01f40cb1880ad4371917bb5b54458fae3b1907f67d58f46b03c10",
|
||||
"file": "frame-227.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "2db80431b10bd5e7b51586168ce69e2bb016f042825c578d4094d60743f3a2e0",
|
||||
"file": "frame-228.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "5b50f955b04b52b15a9fc7ca46cdbf97e8af400b900c29018215d12a5aa87597",
|
||||
"file": "frame-229.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "0119c6fda35d7156fd79cbcf6ea92d5d175ccffa398c5ba409c7acc70902b9fd",
|
||||
"file": "frame-230.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "a11aa662ec15f0bb7cb54708756bc898a4a4ad9b3b0a0a8129bfa3cc2da24834",
|
||||
"file": "frame-231.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "9d4dcf4a95ee98992fce16510dbb8d81e4d2d05973e157ed9f7a91f0eff0941a",
|
||||
"file": "frame-232.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "8823aceaa1243f31914eb72efe72b43dc83a6c3d7e6a47f2277ebb33f64c3918",
|
||||
"file": "frame-233.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "e39c47511cdf9848ad768be20302472a5d2d9218969f65f419d8121dce0b3010",
|
||||
"file": "frame-234.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "3d41f9bd81c566bd1c7ba2e4f8cb44d1f314da111353a49a86df5a7d4376b442",
|
||||
"file": "frame-235.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "49c0d173e695a0c6297860b6cf9d5ed11c284966ff7f0b1c316ba6e02c677e35",
|
||||
"file": "frame-236.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "bca1f60ea73923be386c2d560b4d6cb036e34a02a72be95e444392934896a413",
|
||||
"file": "frame-246.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "e1007169f0b1055171a6bbc3b0805c634e7e079f085c8bafdc655e17b8678c5f",
|
||||
"file": "frame-247.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "77ae3eecd548a5df6a93d04bc9b79ffebc38995dea20418e539dfcee00169368",
|
||||
"file": "frame-248.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "a3eb014d158afa105d64a867d541ef8f39e7fd13ccd528f92c7f29a5d08fab86",
|
||||
"file": "frame-249.png",
|
||||
"height": 32,
|
||||
"phash_distance": 2,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "3f85f5f57b43e1f91f373e0dcdc822d071f890ad372545c5e7068f83bf43aea4",
|
||||
"file": "frame-250.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
},
|
||||
{
|
||||
"dhash_distance": 0,
|
||||
"encoded_sha256": "171c86ff02a8f6e2b72cb8de1fcf5a263a0568f429ec1213200f8fdda46d98c4",
|
||||
"file": "frame-251.png",
|
||||
"height": 32,
|
||||
"phash_distance": 0,
|
||||
"subclassification": "resized_candidate",
|
||||
"width": 128
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
# Perceptual image subclassification
|
||||
|
||||
Schema 18 adds `image_perceptual_hashes` and `image_perceptual_pairs`, keyed by
|
||||
exact archived artifact identity and `image-perceptual/1`. Migration creates empty
|
||||
cache tables; it does not decode the corpus or rewrite identities, names, severity,
|
||||
or delta counts. Cache rows are removed transactionally when no archived artifact
|
||||
references either side. A cached request reuses fingerprints without reading or
|
||||
decoding original image bytes.
|
||||
|
||||
`POST /api/review/perceptual` accepts `before`, `before_path`, `after`, and
|
||||
`after_path` (snapshot IDs and archived paths), with `X-Verstack-Client: 1`.
|
||||
It computes and persists one explicit pair. Readback includes exact artifacts,
|
||||
encoded SHA-256, decoded RGBA SHA-256, both fingerprints, cache-hit flags and the
|
||||
classification evidence. The existing `/api/review/compare` also derives the same
|
||||
perceptual evidence during its already-requested SSIM decode and displays the
|
||||
classification and hash distances in the review screen. That GET does not fill
|
||||
the persistent cache. There is no automatic bulk pass.
|
||||
|
||||
The algorithm composites RGBA over white, uses rounded Rec.601 integer luma,
|
||||
and computes:
|
||||
|
||||
* dHash: 9×8 triangle-filtered luma, 64 horizontal comparisons;
|
||||
* pHash: 32×32 triangle-filtered luma, unnormalized cosine DCT, first 8×8
|
||||
frequencies excluding DC, coefficients rounded to 1e-6, median threshold,
|
||||
63 bits. This explicitly versioned variant is not asserted to be bit-compatible
|
||||
with another library's pHash;
|
||||
* mean visible RGB, mean alpha, dimensions and luma standard deviation.
|
||||
|
||||
Exact decoded RGBA equality is reported separately. Otherwise standard deviation
|
||||
below 2 refuses perceptual subclassification. Candidate structural similarity
|
||||
requires dHash distance ≤4, pHash distance ≤6 and equal aspect ratio. Dimension
|
||||
changes suggest `resized_candidate`; mean visible RGB distance ≥12 with mean
|
||||
alpha difference <1 suggests `recolored_candidate`. Both may occur together.
|
||||
These thresholds are uncalibrated heuristics, not probabilities, proof of palette
|
||||
changes, or accepted cross-release correspondence. Lossy hashes can collide.
|
||||
They never change a logical key, content/payload hash, name, or diff denominator.
|
||||
|
||||
Bounds: 16 MiB encoded bytes, 4 Mi pixels, maximum side8192, decoder/header
|
||||
allocation limit128 MiB. The explicit pair task reserves CPU1/memory256 MiB and
|
||||
zero scratch. Animated WebP/APNG and all GIF inputs require prior explicit frame
|
||||
extraction, so first-frame equality cannot masquerade as animation equality.
|
||||
Unsupported/corrupt/oversized inputs return errors and do not populate caches.
|
||||
|
||||
Validation covers actual archive/API cache hits, same-representation review
|
||||
readback, unchanged assets/observations/deltas/names, deletion cleanup, schema
|
||||
17→18/reopen, synthetic recolor/resize/structural-change controls, transparency,
|
||||
flat images, malformed bytes, GIF refusal and dimension/byte limits.
|
||||
The existing five review integration tests also pass.
|
||||
|
||||
A bounded real fixture pass used 28 previously decoded GOT DMD frames. All 28
|
||||
synthetic nearest-neighbor 2× versions classified as resize candidates, none was
|
||||
low-information. These are transformation controls, **not** 28 actual release
|
||||
changes or a calibrated false-match estimate. Full measurements and encoded
|
||||
fingerprints are in [perceptual-dmd-evidence.json](perceptual-dmd-evidence.json).
|
||||
The original fixture receipt binds the source game SHA-256
|
||||
`637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828`
|
||||
and image-container SHA-256
|
||||
`2f1a958c1f859c339c716e53cc9ee752412373c486ea2a1bfd116eefce4617a3`.
|
||||
|
||||
```sh
|
||||
cargo test --offline --locked --lib perceptual
|
||||
cargo test --offline --locked --test perceptual --test review
|
||||
VERSTACK_PERCEPTUAL_CORPUS=data/validation/dmd-real \
|
||||
VERSTACK_PERCEPTUAL_REPORT=data/validation/perceptual-dmd.json \
|
||||
cargo test --offline --locked --test perceptual -- --include-ignored --nocapture
|
||||
```
|
||||
|
||||
No live deployment/migration was performed for these measurements. Remaining
|
||||
work includes release-change calibration and explicitly scheduled corpus coverage;
|
||||
perceptual scores must not become a substitute for structural identity.
|
||||
@@ -0,0 +1,470 @@
|
||||
# Phase 0 + Phase 1 (identity) — verification record
|
||||
|
||||
Branch: `overhaul/phase-0`. Implemented 2026-09-16 against the plan at
|
||||
<https://claude.ai/artifact/MsfVZmmiyAgdGTEEhtTSLG>.
|
||||
|
||||
**This document is written to be audited, not trusted.** Every claim below names the file and the
|
||||
command that proves it. The executable half is `scripts/verify_phase0.py`, which re-checks all 36
|
||||
assertions and exits non-zero on any failure:
|
||||
|
||||
```sh
|
||||
python3 scripts/verify_phase0.py # human readable
|
||||
python3 scripts/verify_phase0.py --json # machine readable
|
||||
python3 scripts/verify_phase0.py --skip-build # skip cargo (fast)
|
||||
```
|
||||
|
||||
Status at time of writing: **36 checks, 36 pass, 0 fail, 0 skip.**
|
||||
|
||||
## Full-suite results
|
||||
|
||||
| Suite | Command | Result |
|
||||
|---|---|---|
|
||||
| Rust tests | `cargo test --locked` | **79 passed, 0 failed** (12 binaries), exit 0 |
|
||||
| Rust lint | `cargo clippy --locked --all-targets -- -D warnings` | **exit 0** — was 9 errors on `master` |
|
||||
| Rust build | `cargo check --locked --all-targets` | **clean** — failed on `master` |
|
||||
| Python tests | `tools/decoder-env/bin/python -m unittest discover -s tests -p 'test_*.py'` | **97 passed**, exit 0 |
|
||||
|
||||
Note on the Python interpreter: the system `python3` reports 3 collection errors
|
||||
(`test_parallel_media`, `test_preview`, `test_spike2_audio`) because it lacks `numpy`/`PIL`. Those
|
||||
are environmental and pre-existing; `tools/decoder-env/bin/python` is the interpreter the pipeline
|
||||
actually uses and it runs all 97 green. An auditor should use the venv interpreter.
|
||||
|
||||
---
|
||||
|
||||
## 1. LAN access, behind a password gateway
|
||||
|
||||
**Was:** Theia started with `--hostname 0.0.0.0`. Its `/api` handler
|
||||
(`workbench/stern-catalog/src/node/backend-module.ts:43`) proxies **any** method to the Rust API on
|
||||
`127.0.0.1:8080`, which has no authentication, with no method allowlist (unlike the `/emulator`
|
||||
handler above it). The only guard was `if (origin && origin !== ...)`, which a request with no
|
||||
`Origin` header — every non-browser client — skips. `POST /api/catalog/delete` exists at
|
||||
`src/http.rs:86`.
|
||||
|
||||
**Now:** LAN access is restored, gated by a password.
|
||||
|
||||
```
|
||||
LAN ──▶ 0.0.0.0:3000 verstack-gateway (password gate, workbench/auth-gateway.mjs)
|
||||
└──▶ 127.0.0.1:3001 Theia workbench
|
||||
└──▶ 127.0.0.1:8080 Rust archive API
|
||||
```
|
||||
|
||||
**Why a gateway rather than middleware inside Theia.** Theia's JSON-RPC channel is a websocket
|
||||
upgrade handled on the raw HTTP server; Express middleware registered by a
|
||||
`BackendApplicationContribution` never sees it. That channel carries `@theia/filesystem` and
|
||||
`@theia/process` — file access and process spawning. Gating `/api` alone would have protected the
|
||||
archive API and left the more powerful surface open.
|
||||
|
||||
**Why a cookie rather than localStorage.** The requirement was to type the password once. The browser
|
||||
must authenticate requests this code does not issue: Theia's static assets and its websocket upgrade.
|
||||
Only a cookie is attached to those automatically; a localStorage token can only be added by script to
|
||||
fetch/XHR calls. The cookie is `HttpOnly`, so unlike localStorage it is also not readable by injected
|
||||
script. Max-Age is one year, and the session token is an HMAC of a persisted secret, so it survives
|
||||
service restarts. `GET /__auth/logout` clears it.
|
||||
|
||||
**Construction:**
|
||||
* Password verified with `scrypt` (N=2^15, r=8, 64-byte output) against a per-install random salt;
|
||||
compared with `crypto.timingSafeEqual`. Only the hash and a session secret are stored, in
|
||||
`data/deployment/workbench-auth.json`, mode `0600`, inside the gitignored `data/` tree.
|
||||
* Failed logins throttle per client address: 5 attempts, then exponential backoff to 15 minutes.
|
||||
* Clients outside RFC1918/loopback are refused before the password is considered
|
||||
(`--allow-any-client` lifts this). "Local network" is the stated scope.
|
||||
* Non-HTML clients get a plain-text 401 rather than an HTML page.
|
||||
|
||||
**Proof — real browser over the LAN address** (Playwright against `http://172.16.0.87:3000`):
|
||||
|
||||
| | Result |
|
||||
|---|---|
|
||||
| Unauthenticated landing page | login form, title `Verstack workbench` |
|
||||
| After submitting the password | `Assets - Stern ROM Catalog` — the real app |
|
||||
| Websocket | `ws://172.16.0.87:3000/socket.io/?EIO=4&transport=websocket&sid=…` **connected through the gateway** |
|
||||
| Theia shell widgets rendered | yes |
|
||||
| Page errors | none |
|
||||
| Second visit in the same browser | no login prompt (cookie persists) |
|
||||
| Fresh browser profile | gated again |
|
||||
|
||||
**Proof — the gate itself:**
|
||||
```
|
||||
GET /api/info no cookie -> 401
|
||||
GET / (Accept: html) no cookie -> 401 + login form
|
||||
POST /__auth/login wrong pw -> 401
|
||||
POST /__auth/login correct -> 303 + Set-Cookie
|
||||
GET /api/info with cookie-> 200 {"analysis_workers":4,...}
|
||||
websocket upgrade no cookie -> 401
|
||||
websocket upgrade bad cookie -> 401
|
||||
```
|
||||
|
||||
**Proof — nothing else is off-host:**
|
||||
```
|
||||
$ ss -ltn
|
||||
127.0.0.1:3001 Theia 127.0.0.1:8080 Rust API
|
||||
127.0.0.1:8096 VM exports 0.0.0.0:3000 gateway <- the only LAN listener
|
||||
```
|
||||
Direct connections from the LAN address to `:8080`, `:3001`, `:8095` and `:8096` are all refused.
|
||||
|
||||
**What this does and does not protect.** It stops anyone on the network reaching the archive without
|
||||
the password. It does **not** authenticate the Rust API itself — anything running on this host can
|
||||
still call `/api/catalog/delete` directly on loopback. And the transport is plain HTTP: the password
|
||||
and cookie cross the LAN in the clear, so anyone able to observe your local traffic can replay the
|
||||
cookie. That is a reasonable trade on a trusted home network and a bad one anywhere else. If you want
|
||||
it closed, the next step is TLS on the gateway.
|
||||
|
||||
**Your password is in this session's transcript, not in the repo.** Rotate it any time with
|
||||
`node workbench/auth-gateway.mjs --set-password && systemctl --user restart verstack-gateway`.
|
||||
|
||||
## 1b. The `verstack-exports` crashloop — resolved
|
||||
|
||||
Root cause was not the missing restart limit: the unit could never bind, because a manually started
|
||||
instance (pid 2723, from Sep 14) already held port 8096. The restart counter had reached **36,648**.
|
||||
|
||||
Stopped the manual instance and started the unit, after confirming no export job had touched
|
||||
`data/emulator/exports/jobs` in the preceding two hours. The unit now owns the port with
|
||||
`NRestarts=0`, and the start limit added earlier means any future bind conflict fails visibly instead
|
||||
of looping silently.
|
||||
|
||||
## 2. Cross-version asset identity (the headline correctness bug)
|
||||
|
||||
**Was:** `Archive::compare` (`src/archive.rs:777`) joined snapshots on `Entry.path`. Extracted asset
|
||||
paths embed the input package filename, which embeds the release version, so paths differ between any
|
||||
two releases by construction. `Entry.artifact` — a BLAKE3 content hash — sat unused beside the join.
|
||||
|
||||
**Change:**
|
||||
* New `src/identity.rs` with `logical_path()`, which removes the version-bearing package-root
|
||||
component **structurally** (by its position under `package-NNNN/`), not by matching a list of
|
||||
release names. A hardcoded alias table would silently reintroduce the same bug for any future
|
||||
naming, which is why it is not used.
|
||||
* `Archive::compare` now joins on that key, then runs a second **content-identity pass** over the
|
||||
leftovers so an asset that moved within the container is reported as `moved` rather than as a
|
||||
delete plus an unrelated add. `Change` gains an optional `from_path` (serialised only for moves,
|
||||
so existing consumers are unaffected).
|
||||
|
||||
**Proof against the live archive** (`scripts/verify_phase0.py`, read-only, `mode=ro&immutable=1`):
|
||||
|
||||
| | Pokémon LE 0.85.0 → 0.86.0, `media-extract` |
|
||||
|---|---|
|
||||
| Raw path overlap (old behaviour) | **3 of 20,214** |
|
||||
| After canonicalisation | **78 modified / 41 removed / 5 added**, out of 20,173 shared |
|
||||
|
||||
That is a 0.4% delta — what a point release should look like — against roughly 38,375 changes
|
||||
reported before. The verification script recomputes this from the real catalog on every run.
|
||||
|
||||
### Follow-up found by running it live
|
||||
|
||||
Deploying the fix and calling the real endpoint exposed a second layer of noise the offline
|
||||
measurement had not: the corrected delta was present but buried.
|
||||
|
||||
```
|
||||
before: 20,219 changes, 6.1 MB (added 5, changed 78, removed 41, metadata_only 20,095)
|
||||
after: 124 changes, 39.6 KB (added 5, changed 78, removed 41)
|
||||
```
|
||||
|
||||
All 20,095 `metadata_only` rows were directories (1,008) or files with **byte-identical content
|
||||
hashes** (19,087); none had differing hashes. They came from `mode`/`modified_ns`, which in an
|
||||
Extracted or Derived layer record when the decoder happened to run, not anything about the release.
|
||||
`Archive::compare` now reports `metadata_only` only for the `Original` layer, where those fields are
|
||||
genuinely preserved source filesystem metadata. `tests/asset_identity.rs` asserts the derived-layer
|
||||
case emits none.
|
||||
|
||||
Live confirmation against the running service:
|
||||
|
||||
```
|
||||
$ curl 'http://127.0.0.1:8080/api/spine/diff?from=<Pokémon|0.85.0|LE|3>&to=<Pokémon|0.86.0|LE|3>'
|
||||
http 200 39591 bytes 0.38s
|
||||
added 5 · changed 78 · removed 41
|
||||
e.g. .../main.pck/original/.godot/exported/133200997/export-34dd…
|
||||
```
|
||||
|
||||
**Regression tests:** `tests/asset_identity.rs` (4 tests) and `src/identity.rs` unit tests (7 tests).
|
||||
They cover the version-stamped root, extraction-ordinal moves, the `pokemon_le`→`pokemon_pro` vendor
|
||||
rename, and all eight real package-root namings taken from `/srv/firmware/images/stern_game_code`.
|
||||
`tests/asset_identity.rs` pins fixture timestamps so the assertions measure content change, not mtime.
|
||||
|
||||
**Honest limits:**
|
||||
* This normalises the **package-root** component only. The `pokemon_le`→`pokemon_pro` vendor
|
||||
directory rename deeper in the tree is deliberately *not* collapsed — it is real structure, and the
|
||||
content-identity pass is what pairs those files up. The test asserts this explicitly.
|
||||
* The catalog has **not** been rewritten. Stored `artifacts.path` rows still contain version-bearing
|
||||
paths; canonicalisation happens at comparison time. A schema-level migration is Phase 1 proper.
|
||||
* `logical_path` is mirrored in Python inside `scripts/verify_phase0.py`. Two implementations of one
|
||||
rule can drift; if the rule changes, change both.
|
||||
|
||||
---
|
||||
|
||||
## 3. Function comparison
|
||||
|
||||
**Was, two separate defects:**
|
||||
1. `stern-catalog-widget.tsx:160` picked comparison inputs with
|
||||
`entries.find(/functions\.json$/)` — the *first* match on each side. A snapshot can hold one
|
||||
analysis per executable under content-addressed directories, so this compared unrelated programs;
|
||||
for Pokémon 0.81 Pro vs LE both sides resolved to the same `spike_menu` output and it reported a
|
||||
perfect match.
|
||||
2. `comparable_settings` (`src/analysis.rs:107`) included `max_cpu`/`max_heap`, refusing legitimate
|
||||
Pro-vs-LE comparisons because an operational knob differed.
|
||||
|
||||
**Change:**
|
||||
* `analysis::pair_programs()` pairs the two snapshots by **analysed source path**, preferring the game
|
||||
binary, and errors clearly when the snapshots share no analysed program.
|
||||
`Archive::function_program_pair` wires it to the catalog via the new `Catalog::code_programs`.
|
||||
* `/api/functions/compare` now takes `before_path`/`after_path` as **optional**. Omitted, the server
|
||||
pairs correctly. Existing callers that pass them still work.
|
||||
* The workbench stopped guessing and now omits them.
|
||||
* `compare_function_files` refuses selecting *one* analysis twice (same snapshot **and** same path),
|
||||
which can only be a selection error.
|
||||
* `comparable_settings` now also drops `max_cpu` and `max_heap`, justified by measurement: the pinned
|
||||
adapter run three times over `conagent` produced byte-identical `functions.json` and all 754
|
||||
`code/*.json` files at `max_cpu` 2 and 8.
|
||||
|
||||
**A deliberate narrowing you should know about.** My first version rejected *any* comparison where
|
||||
both sides had the same artifact hash. That was wrong and two existing tests caught it: two
|
||||
**different** snapshots with byte-identical analysis output is a legitimate question with the
|
||||
legitimate answer "this edition's code did not change". It is now reported with a caveat rather than
|
||||
refused. Only same-snapshot-same-path is an error. `tests/archive.rs` asserts both halves.
|
||||
|
||||
**Not verified:** the workbench TypeScript change is **not** compile-checked. The Theia build was
|
||||
already broken before this work and I did not rebuild it. See §8.
|
||||
|
||||
---
|
||||
|
||||
## 4. Ghidra bounding
|
||||
|
||||
**Was:** neither `subprocess.run` in `plugins/ghidra/analyze.py` passed `timeout=`. The second
|
||||
invocation — the `-noanalysis` reopen/verify pass — had no bound of any kind. `-analysisTimeoutPerFile`
|
||||
is cooperative: it is checked between analyzers, so an analyzer wedged in the native decompiler ignores
|
||||
it.
|
||||
|
||||
**Change:** a `run_bounded()` helper that spawns with `start_new_session=True` and, on timeout, signals
|
||||
the **whole process group** (SIGTERM then SIGKILL, each with a bounded wait). Ghidra spawns a JVM and a
|
||||
separate native `decompile` process, so killing only the direct child leaves those running. Both
|
||||
invocations now use it:
|
||||
* analysis: `analysis_wallclock_seconds`, defaulting to `analysis_timeout_seconds * 2 + 600`
|
||||
* reopen/verify: `reopen_timeout_seconds`, defaulting to 1800
|
||||
|
||||
New `plugins/ghidra/TuneAnalyzers.java` runs as `-preScript` and disables
|
||||
`Non-Returning Functions - Discovered`, the pass reported in Ghidra issue #4296 to spin without
|
||||
yielding. It logs every option it sets so a run log shows which passes produced a given
|
||||
`functions.json`.
|
||||
|
||||
**Not verified:** no Ghidra run was executed. The timeouts and the pre-script are **untested against
|
||||
real Ghidra**. Before trusting them, run one analysis and confirm (a) `TuneAnalyzers` appears in
|
||||
`analysis.log`, (b) a deliberately tiny `reopen_timeout_seconds` actually kills the process group.
|
||||
This is the highest-residual-risk change in this batch.
|
||||
|
||||
---
|
||||
|
||||
## 5. Generation inference
|
||||
|
||||
**Was:** `spike_probe.classify()` branched on the right magic values and then hardcoded
|
||||
`"generation": "unknown"` on every branch. 119 of 122 snapshots carry an empty generation.
|
||||
|
||||
**Change:** inference from container structure, derived from all 73 packages in
|
||||
`docs/corpus-inventory.json` and cross-checked against the vendor's own `_spike2`/`_spike3` filename
|
||||
labels:
|
||||
|
||||
| Structure | Generation | Packages |
|
||||
|---|---|---|
|
||||
| inner LUKS2 | SPIKE 3 | 8 |
|
||||
| inner squashfs | SPIKE 2 | 48 |
|
||||
| bare SPKS / gzip wrapper | *undetermined* | 17 |
|
||||
|
||||
`aggregate_generation()` returns one generation per multi-part package, or `""` when parts disagree.
|
||||
|
||||
**Result: 56 of 73 determined, 0 contradictions** with the vendor labels (re-checked by the
|
||||
verification script).
|
||||
|
||||
**This corrects the plan.** The audit claimed container magic separates *all 73* with "zero
|
||||
ambiguity". It does not. The 17 bare-SPKS and gzip packages carry no structural discriminator and no
|
||||
filename label, so they are left undetermined rather than guessed. Determining those needs evidence
|
||||
from inside the package.
|
||||
|
||||
**Not done:** existing snapshots are **not** backfilled. This affects future probes only. Backfilling
|
||||
the 122 existing rows is a small follow-up.
|
||||
|
||||
---
|
||||
|
||||
## 6. Media, caching, logging
|
||||
|
||||
Verified against the running service after `cargo build --release && systemctl --user restart
|
||||
verstack-backend`:
|
||||
|
||||
```
|
||||
$ curl -D- 'http://127.0.0.1:8080/api/file/<snapshot>?path=<asset>'
|
||||
HTTP/1.1 200 OK
|
||||
cache-control: public, max-age=31536000, immutable
|
||||
etag: "blake3:2553125db57ff1b31199912eeac9da887bd6b8b61d0b17bbc07142a689f483d1"
|
||||
|
||||
$ curl -H 'If-None-Match: "blake3:2553…"' ...
|
||||
HTTP/1.1 304 Not Modified bytes transferred: 0
|
||||
```
|
||||
|
||||
The catalog WAL also truncated on that restart, **491 MB -> 0 bytes**, confirming §7's
|
||||
`wal_checkpoint(TRUNCATE)`.
|
||||
|
||||
| Change | File | Note |
|
||||
|---|---|---|
|
||||
| Image thumbnails enabled | `config.json` | `media-preview.include_extensions` was `[".mp4",".webm"]`; now includes `.png`, `.jpg`, `.webp`. `processing_revision` 3 → 4. |
|
||||
| Immutable caching + ETag | `src/http.rs` | `Cache-Control: public, max-age=31536000, immutable` plus an ETag from the BLAKE3 artifact hash; `If-None-Match` returns 304 without touching the store. |
|
||||
| Live plugin logs | `src/plugins.rs` | `capture_log` wrote the whole run from a RAM buffer at exit — a long job had no file to tail, and a budget-killed job left nothing. Now writes and flushes per chunk, and keeps draining past the 256 KiB cap so the child never blocks on a full pipe. |
|
||||
|
||||
**Important:** I listed only extensions `media_extract.sniff()` can actually return. It content-sniffs;
|
||||
it does not read filenames, and it can only emit `.png .jpg .webp .wav .ogg .flac .mp4 .webm`. Listing
|
||||
`.gif`/`.bmp`/`.tga` would have been inert.
|
||||
|
||||
**Not done:** the `processing_revision` bump means existing releases must be **re-run** through
|
||||
`media-preview` to gain thumbnails. Nothing is backfilled automatically. Also note `config.json` points
|
||||
`media-preview` at the frozen bundle under `data/archive/tool-sources/`, not `plugins/` — see §8.
|
||||
|
||||
---
|
||||
|
||||
## 7. Build, lint, ops, disk
|
||||
|
||||
| Item | Evidence |
|
||||
|---|---|
|
||||
| `cargo check --all-targets` fixed | `examples/storage_spike.rs` was missing `analysis_workers`/`preparation_workers` (E0063). |
|
||||
| Clippy green | 9 errors on `master` → 0. Fixes in `src/signatures.rs` and `src/workspace.rs` are mechanical (`is_multiple_of`, `as_chunks`, collapsed `if let` chains, `contains_key`). |
|
||||
| Dangling frontend sources staged | `image-review.tsx`, `image-similarity.ts`, plus `docs/image-similarity.md` and two test files, were untracked; commit `2d844e3` imported them. **They are staged, not committed** — committing is yours to do. |
|
||||
| Restart storms bounded | `StartLimitIntervalSec=300` / `StartLimitBurst=5` on backend, workbench, and exports units. |
|
||||
| `data/workspace` removed | 9.2 GB of dead scratch dated Sep 13. Verified unreferenced in `src/ plugins/ config.json deploy/ scripts/ emulator/ tests/`, configured workspace is `/tmp/verstack-workspace`, and `lsof +D` showed no open handles. Inventory recorded before deletion. Disk went 344 G → 335 G used. |
|
||||
| WAL truncation | `Catalog::open` now runs `PRAGMA wal_checkpoint(TRUNCATE)` at startup, when no reader is attached. The live 491 MB WAL was **not** touched; it truncates on next service restart. |
|
||||
|
||||
### `verstack-exports` root cause
|
||||
|
||||
Not just a missing restart limit. The unit cannot bind:
|
||||
`OSError: [Errno 98] Address already in use` on port 8096, because a **manually started** instance
|
||||
(pid 2723, from Sep 14) already owns it. The restart counter had reached **36,648**. The unit is
|
||||
version-controlled at `emulator/bundles/verstack-exports.service` and symlinked into
|
||||
`~/.config/systemd/user/`.
|
||||
|
||||
**Action for you:** decide which instance should own port 8096, then stop the other. The start limit
|
||||
now makes the unit fail visibly instead of looping silently, but the conflict itself is unresolved.
|
||||
|
||||
---
|
||||
|
||||
## 8. Godot naming oracle
|
||||
|
||||
New `plugins/godot_names.py` + `tests/test_godot_names.py` (12 tests, all pass).
|
||||
|
||||
Recovers original `res://` names for Godot import-cache artifacts. The derivation is confirmed against
|
||||
real archive data: `md5("res://fonts/Stern_Aztech.ttf")` = `3064810bd5908d7a5012fbe63bbbf40b`, exactly
|
||||
matching the archived `Stern_Aztech.ttf-3064810bd5908d7a5012fbe63bbbf40b.fontdata`. Source:
|
||||
`core/io/resource_importer.cpp::get_import_base_path` (Godot 4.4.1).
|
||||
|
||||
Two routes: exact mapping from the plaintext `.import`/`.remap` sidecars (no hashing needed — the
|
||||
mapping is stored directly), and forward-hash confirmation for cache files orphaned from their
|
||||
sidecar.
|
||||
|
||||
**Deliberately NOT done, and why.** The plan proposed dropping `--scripts-only` from the GDRE
|
||||
invocation in `plugins/godot_scripts.py:115`. I did not, because on inspection that flag is load-bearing
|
||||
for the validated script flow: the function builds an inventory, runs a compile/decompile round-trip
|
||||
validation, and publishes against a fixed output budget. Full `--recover` writes many more files, which
|
||||
would consume that budget without anything publishing them — so the flag change alone gains nothing
|
||||
and risks a working decoder. Wiring recovered names through extraction is a Phase 2 change that needs
|
||||
the semantic-typing schema to land first. The parser above is the reusable half, delivered and tested
|
||||
now; **nothing in the pipeline calls it yet.**
|
||||
|
||||
---
|
||||
|
||||
## What was NOT implemented
|
||||
|
||||
The plan runs to eight phases. This branch is Phase 0 plus the Phase 1 correctness slice. Untouched:
|
||||
|
||||
* **Phase 1 proper** — the `LogicalAssetKey` enum, `assets`/`asset_observations`/`edges`/`asset_names`/
|
||||
`asset_deltas` schema, persisted diffs, and the bulk import of the wiki's 2,545 sound names.
|
||||
* **Phase 2** — the `FormatDecoder` registry, the `scene.radium` parser, semantic asset typing,
|
||||
animation grouping, the AES-XTS 82× fix.
|
||||
* **Phase 3** — the resource governor. The machine still runs at ~1.65 mean concurrent plugin
|
||||
processes; nothing here changes that.
|
||||
* **Phase 4** — importing the other 63 corpus packages.
|
||||
* **Phase 5** — Tier 0 signatures, CFG hashing, anchored propagation, the symbol bridge.
|
||||
* **Phase 6** — the review application. Theia remains.
|
||||
* **Phase 7** — storage GC. Still 127 GB store + 113 GB outside it.
|
||||
* **Phase 8** — MCP surface, AI, emulation.
|
||||
|
||||
## Known gaps in this batch
|
||||
|
||||
1. **The workbench is not rebuilt.** `stern-catalog-widget.tsx` was edited but not compiled; the
|
||||
browser check above loaded the **existing** built bundle, so it proves the gateway works, not the
|
||||
widget change. The Theia build was already broken before this work (the dangling imports) and
|
||||
rebuilding 875 npm packages was out of scope. **Verify before relying on the function-comparison UI.**
|
||||
2. **No Ghidra run was executed.** §4 is untested against real Ghidra.
|
||||
2b. **The gateway is plain HTTP.** Password and cookie cross the LAN unencrypted. See §1.
|
||||
3. **`plugins/` is not what runs.** `config.json` points several tools at a frozen bundle under
|
||||
`data/archive/tool-sources/2d4af8bb…/`, whose digest differs from the working tree. **The changes to
|
||||
`plugins/ghidra/analyze.py` and `plugins/spike_probe.py` are not live** until the pipeline is
|
||||
reconfigured (`scripts/configure_pipeline.py`). This was true before this work and is unchanged.
|
||||
4. **Nothing is backfilled.** Generation on existing snapshots, thumbnails for existing releases, and
|
||||
`artifacts.path` rows are all untouched.
|
||||
5. **Nothing is committed.** All changes are in the working tree on `overhaul/phase-0`, with new files
|
||||
staged. Review the diff before committing.
|
||||
6. **`logical_path` exists twice** — Rust and Python. Keep them in step.
|
||||
|
||||
---
|
||||
|
||||
## 10. Post-deployment work (2026-09-16, after the first live review)
|
||||
|
||||
Three things were reported as "nothing changed in the UI". All three were correct.
|
||||
|
||||
### The workbench was never rebuilt
|
||||
|
||||
The build did not fail. `image-review.tsx` and `image-similarity.ts` were on disk the whole time —
|
||||
only untracked in git — so a build from the working tree always worked; only a fresh clone would
|
||||
fail. The real reason nothing changed is that **the build was never run**. It has now been:
|
||||
`tsc` clean, `theia build` 0 errors, workbench restarted, and the app re-verified through the
|
||||
gateway (loads, shell renders, no page errors). Bundle timestamp moved Sep 15 10:21 -> Sep 16 10:32.
|
||||
|
||||
While rebuilding, a **second** function-comparison call site surfaced at
|
||||
`workspace-views.tsx:195` (the native workbench). It lets the operator pick programs from dropdowns,
|
||||
so it was not blind-guessing, but both defaulted to index 0 and could open on unrelated executables.
|
||||
`pairSources()` now defaults both pickers to the same analysed source program, preferring the game
|
||||
binary.
|
||||
|
||||
### What the comparison actually shows now
|
||||
|
||||
| View | Before | Now |
|
||||
|---|---|---|
|
||||
| `extracted` layer (the UI default) | 1,687 rows (840 added + 840 removed + 3 changed + 4 metadata) | **57** (21 changed, 18 removed, 18 added) |
|
||||
| `media-extract` layer | ~20,219 rows | **124** (78 changed, 41 removed, 5 added) |
|
||||
|
||||
The 18 removed / 18 added in the extracted view are **correct, not residual noise**: node board
|
||||
firmware carries its version in the filename, so `coil4node-LPC1313-1_35_0.hex` becomes
|
||||
`coil4node-LPC1313-1_37_0.hex`, with zero content-hash overlap. The node firmware was upgraded
|
||||
between those releases — a real finding the 1,687-row wall had buried.
|
||||
|
||||
### Image thumbnails, generated
|
||||
|
||||
`POST /api/process` with `media-preview` on Pokémon LE 0.86.0 (note: POST requires
|
||||
`X-Verstack-Client: 1`; the Rust API rejects it with 403 otherwise).
|
||||
|
||||
```
|
||||
20m 14s, coverage complete, new snapshot b0586cbd
|
||||
previews 256 -> 5,173 (4,917 WebP thumbnails + 255 mp4)
|
||||
4,917 original images 231.2 MB -> 17.2 MB of thumbnails
|
||||
~46 KB/image -> ~3 KB/image
|
||||
5,191 parent artifacts now carry a preview pointer
|
||||
GET /api/artifacts?...&kind=image -> 24/24 items on page 1 have a thumbnail
|
||||
```
|
||||
|
||||
Incidental confirmation of §6's streaming-log fix: the run's `.stdout.txt`/`.stderr.txt` existed in
|
||||
`data/archive/logs/` **while the job was still running**. Before the fix, no file appeared until the
|
||||
process exited.
|
||||
|
||||
**Only Pokémon 0.86.0 was re-run.** The other 12 releases still carry revision-3 previews (video
|
||||
only) and need the same call, roughly 20 minutes each.
|
||||
|
||||
### Godot name recovery — report only, nothing written
|
||||
|
||||
`scripts/recover_godot_names.py` reads the archived `.import`/`.remap` sidecars through the local API
|
||||
and resolves them to original `res://` paths. Across all 13 releases: **5,716 artifacts named.**
|
||||
The catalog was not modified.
|
||||
|
||||
```
|
||||
Pokémon 0.86.0 LE sidecars=1179 godot-cache 1068/2133 (50.1%) radium-ordinal 15124 (not covered)
|
||||
export-34ddcc78… -> res://scenes/pokedex_cine_home/pokedex_cine_home.tscn
|
||||
Stern_Aztech.ttf… -> res://fonts/Stern_Aztech.ttf
|
||||
```
|
||||
|
||||
Two limits worth stating plainly: it resolves **half** the Godot cache artifacts (the rest lost their
|
||||
sidecar), and **none** of the ~15,000 radium ordinal assets per release, which `.import` sidecars do
|
||||
not describe. Only the SPIKE 3 Pokémon titles have Godot content; the SPIKE 2 games have none.
|
||||
|
||||
These names have nowhere durable to live until Phase 1's `asset_names` table exists, which is why
|
||||
this is a report rather than an applied change. Report:
|
||||
`data/validation/godot-name-recovery.json`.
|
||||
@@ -0,0 +1,198 @@
|
||||
{
|
||||
"active_tasks": {
|
||||
"max": 0,
|
||||
"samples": [
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
]
|
||||
},
|
||||
"elapsed_seconds": 4.034,
|
||||
"endpoint": "http://127.0.0.1:8080/api/governor",
|
||||
"errors": [],
|
||||
"note": "Read-only governor telemetry; no task, plugin, archive, or catalog work was submitted.",
|
||||
"readings": [
|
||||
{
|
||||
"execution": {
|
||||
"active_budgets": [],
|
||||
"analysis_limit": 4,
|
||||
"analysis_used": 0,
|
||||
"archive_reserved_bytes": 0,
|
||||
"blocked_by": null,
|
||||
"cpu_threads_reserved": 0,
|
||||
"memory_reserved_bytes": 0,
|
||||
"preparation_used": 0,
|
||||
"scratch_reserved_bytes": 0,
|
||||
"scratch_unreserved_free_bytes": 92003209216,
|
||||
"shared_inputs": 0,
|
||||
"task_limit": 40
|
||||
},
|
||||
"host": {
|
||||
"cpu_pressure": 0.03,
|
||||
"cpu_utilisation": 0.20561442647332218,
|
||||
"cpus": 54,
|
||||
"io_pressure": 0.0,
|
||||
"load": [
|
||||
4.76,
|
||||
4.19,
|
||||
4.34
|
||||
],
|
||||
"memory_available_bytes": 177325674496,
|
||||
"memory_pressure": 0.0,
|
||||
"memory_total_bytes": 185682075648,
|
||||
"sampled_at": 1789643642,
|
||||
"scratch_free_bytes": 92048588800,
|
||||
"scratch_total_bytes": 92841037824
|
||||
},
|
||||
"index": 0
|
||||
},
|
||||
{
|
||||
"execution": {
|
||||
"active_budgets": [],
|
||||
"analysis_limit": 4,
|
||||
"analysis_used": 0,
|
||||
"archive_reserved_bytes": 0,
|
||||
"blocked_by": null,
|
||||
"cpu_threads_reserved": 0,
|
||||
"memory_reserved_bytes": 0,
|
||||
"preparation_used": 0,
|
||||
"scratch_reserved_bytes": 0,
|
||||
"scratch_unreserved_free_bytes": 92073943040,
|
||||
"shared_inputs": 0,
|
||||
"task_limit": 40
|
||||
},
|
||||
"host": {
|
||||
"cpu_pressure": 0.03,
|
||||
"cpu_utilisation": 0.046153846153846156,
|
||||
"cpus": 54,
|
||||
"io_pressure": 0.0,
|
||||
"load": [
|
||||
4.76,
|
||||
4.19,
|
||||
4.34
|
||||
],
|
||||
"memory_available_bytes": 177352974336,
|
||||
"memory_pressure": 0.0,
|
||||
"memory_total_bytes": 185682075648,
|
||||
"sampled_at": 1789643643,
|
||||
"scratch_free_bytes": 92001030144,
|
||||
"scratch_total_bytes": 92841037824
|
||||
},
|
||||
"index": 1
|
||||
},
|
||||
{
|
||||
"execution": {
|
||||
"active_budgets": [],
|
||||
"analysis_limit": 4,
|
||||
"analysis_used": 0,
|
||||
"archive_reserved_bytes": 0,
|
||||
"blocked_by": null,
|
||||
"cpu_threads_reserved": 0,
|
||||
"memory_reserved_bytes": 0,
|
||||
"preparation_used": 0,
|
||||
"scratch_reserved_bytes": 0,
|
||||
"scratch_unreserved_free_bytes": 92073943040,
|
||||
"shared_inputs": 0,
|
||||
"task_limit": 40
|
||||
},
|
||||
"host": {
|
||||
"cpu_pressure": 0.02,
|
||||
"cpu_utilisation": 0.0638100537933593,
|
||||
"cpus": 54,
|
||||
"io_pressure": 0.0,
|
||||
"load": [
|
||||
4.76,
|
||||
4.19,
|
||||
4.34
|
||||
],
|
||||
"memory_available_bytes": 177295302656,
|
||||
"memory_pressure": 0.0,
|
||||
"memory_total_bytes": 185682075648,
|
||||
"sampled_at": 1789643644,
|
||||
"scratch_free_bytes": 92073943040,
|
||||
"scratch_total_bytes": 92841037824
|
||||
},
|
||||
"index": 2
|
||||
},
|
||||
{
|
||||
"execution": {
|
||||
"active_budgets": [],
|
||||
"analysis_limit": 4,
|
||||
"analysis_used": 0,
|
||||
"archive_reserved_bytes": 0,
|
||||
"blocked_by": null,
|
||||
"cpu_threads_reserved": 0,
|
||||
"memory_reserved_bytes": 0,
|
||||
"preparation_used": 0,
|
||||
"scratch_reserved_bytes": 0,
|
||||
"scratch_unreserved_free_bytes": 92073943040,
|
||||
"shared_inputs": 0,
|
||||
"task_limit": 40
|
||||
},
|
||||
"host": {
|
||||
"cpu_pressure": 0.02,
|
||||
"cpu_utilisation": 0.053236876275273605,
|
||||
"cpus": 54,
|
||||
"io_pressure": 0.0,
|
||||
"load": [
|
||||
4.76,
|
||||
4.19,
|
||||
4.34
|
||||
],
|
||||
"memory_available_bytes": 177018966016,
|
||||
"memory_pressure": 0.0,
|
||||
"memory_total_bytes": 185682075648,
|
||||
"sampled_at": 1789643645,
|
||||
"scratch_free_bytes": 92073943040,
|
||||
"scratch_total_bytes": 92841037824
|
||||
},
|
||||
"index": 3
|
||||
},
|
||||
{
|
||||
"execution": {
|
||||
"active_budgets": [],
|
||||
"analysis_limit": 4,
|
||||
"analysis_used": 0,
|
||||
"archive_reserved_bytes": 0,
|
||||
"blocked_by": null,
|
||||
"cpu_threads_reserved": 0,
|
||||
"memory_reserved_bytes": 0,
|
||||
"preparation_used": 0,
|
||||
"scratch_reserved_bytes": 0,
|
||||
"scratch_unreserved_free_bytes": 92065542144,
|
||||
"shared_inputs": 0,
|
||||
"task_limit": 40
|
||||
},
|
||||
"host": {
|
||||
"cpu_pressure": 0.02,
|
||||
"cpu_utilisation": 0.05938021896455743,
|
||||
"cpus": 54,
|
||||
"io_pressure": 0.0,
|
||||
"load": [
|
||||
4.46,
|
||||
4.13,
|
||||
4.32
|
||||
],
|
||||
"memory_available_bytes": 176992980992,
|
||||
"memory_pressure": 0.0,
|
||||
"memory_total_bytes": 185682075648,
|
||||
"sampled_at": 1789643646,
|
||||
"scratch_free_bytes": 92073943040,
|
||||
"scratch_total_bytes": 92841037824
|
||||
},
|
||||
"index": 4
|
||||
}
|
||||
],
|
||||
"report_only": true,
|
||||
"requested_samples": 5,
|
||||
"schema": 1,
|
||||
"successful_samples": 5,
|
||||
"verification": {
|
||||
"all_samples_succeeded": true,
|
||||
"endpoint_reachable": true,
|
||||
"work_submitted": false,
|
||||
"writes_performed": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
# Phase 4 import accounting
|
||||
|
||||
`scripts/corpus_import_queue.py` builds a read-only, resumable queue from the
|
||||
corpus inventory and validation receipts. The queue now reports a stable
|
||||
`failure_class` for every non-ready package and checks the extraction and media
|
||||
stage payloads even when an older receipt incorrectly says `status: complete`.
|
||||
|
||||
The classes distinguish work that has not run (`unprocessed`), a changed source
|
||||
whose receipt must be discarded (`stale_receipt`), an extraction decoder error
|
||||
(`extraction_failed`), an extraction stage that returned partial coverage
|
||||
(`extraction_incomplete`), a missing staged input (`missing_input`), and a
|
||||
derived media-only shortfall (`derived_coverage_incomplete`). A missing receipt
|
||||
is never counted as an extraction failure.
|
||||
|
||||
This is accounting evidence, not a claim that the corpus is imported. For
|
||||
example, the current report-only run can be reproduced without opening a source
|
||||
package:
|
||||
|
||||
```sh
|
||||
python3 scripts/corpus_import_queue.py \
|
||||
docs/corpus-inventory.json \
|
||||
data/validation/corpus-extraction-revision1 \
|
||||
/tmp/phase4-queue.json
|
||||
```
|
||||
|
||||
The 2026-09-17 receipt set reports 73 packages: 42 unprocessed, 26 with
|
||||
incomplete extraction/derived coverage, and 5 failed receipts. Those five split
|
||||
into four decoder extraction failures and one missing staged input. The report
|
||||
therefore does not support the older claim that only two packages are unresolved;
|
||||
the receipts need a fresh bounded validation pass before that number can be
|
||||
re-established.
|
||||
|
||||
The classification is covered by
|
||||
`tests/test_corpus_import_queue.py`, including a fixture where `complete` hides
|
||||
partial extraction and fixtures separating decoder failure from missing input.
|
||||
|
||||
The queue can also perform a bounded, read-only workspace admission check for
|
||||
the selected page. `--workspace-budget` supplies the configured byte cap and
|
||||
`--workspace` additionally measures free bytes at a workspace path; the
|
||||
effective cap is the smaller value. The report records the selected input-byte
|
||||
sum, largest source, unknown sizes, and separate largest-source and page-sum
|
||||
fit results:
|
||||
|
||||
```sh
|
||||
python3 scripts/corpus_import_queue.py \
|
||||
docs/corpus-inventory.json data/validation/corpus-extraction-revision1 \
|
||||
/tmp/phase4-queue.json --limit 4 --workspace-budget $((16*1024**3)) \
|
||||
--workspace /tmp
|
||||
```
|
||||
|
||||
This check is intentionally `report_only` and scoped to source bytes. Compressed
|
||||
packages can expand substantially, so a passing result is staging evidence and
|
||||
does not replace the importer's format-aware admission or prove that a package
|
||||
fits the complete extraction chain.
|
||||
|
||||
## Bounded extraction revalidation
|
||||
|
||||
On 2026-09-17, Jaws Pro 1.02.0 was revalidated in a disposable workspace using
|
||||
`scripts/validate_corpus.py` with a one-package source directory. The source was
|
||||
not copied into the archive and the receipt was written under `/tmp`. The run
|
||||
reproduced the blocker, so the older receipt was not merely stale:
|
||||
|
||||
| Evidence | Result |
|
||||
| --- | --- |
|
||||
| source | `jaws_pro-1_02_0.spk.zip`, 7,399,891,406 bytes |
|
||||
| outer split wrapper | complete inventory; four members, no retained wrapper payload |
|
||||
| extraction | `partial`, 1,195 files |
|
||||
| media | `partial`, 3,767 decoded assets and 226 coverage failures |
|
||||
| blocker | nested asset containers remain opaque; SPIKE 3 LUKS wrappers need a separate decoder |
|
||||
|
||||
The validator's `decode_before_store` policy remained active and reported
|
||||
`wrapper_retained: false`. This is direct evidence for the remaining ZIP → LUKS
|
||||
chain work and does not count the package as imported.
|
||||
|
||||
## Recursive wrapper handoff
|
||||
|
||||
The corpus-facing `spike-package` adapter now hands the files produced by SPK
|
||||
decoding to the pinned `import-extract` recursion when native LUKS or ext4
|
||||
helpers are configured. This closes the prior adapter boundary: nested
|
||||
SPIKE 3 LUKS members are sent through the verified bounded LUKS → ext4 → SPK
|
||||
path instead of being reported as opaque files. The handoff preserves
|
||||
`decode_before_store`; wrapper payloads remain temporary unless explicit
|
||||
retention is configured. A focused fixture proves that the handoff replaces
|
||||
the partial wrapper result with the recursive result and does not invoke a
|
||||
fallback decoder. Real Jaws coverage still requires a fresh bounded run with
|
||||
the configured credentials and available workspace.
|
||||
@@ -0,0 +1,493 @@
|
||||
# Approved deployment, 2026-09-16
|
||||
|
||||
The user explicitly approved the prepared release deployment. The backend was
|
||||
stopped for a consistent catalog backup, the existing configuration was preserved
|
||||
except for the seven prepared plugin source pins, and backend and workbench were
|
||||
restarted. The initial schema migration completed without a restart; the API
|
||||
became available within the several-minute rehearsal window.
|
||||
|
||||
Verified running executable SHA-256:
|
||||
`cc9891d44287068dd50983591cba9ed4f92ce7cf5952661ad7adff25b6a41293`.
|
||||
Frozen plugin source bundle:
|
||||
`9170fea4b04f2ff7dc85a12d8e30cf711ea78937ba36406cadc671d10a521f88`.
|
||||
Automatic processing remains disabled.
|
||||
|
||||
Rollback artifacts are under `data/deployment/plan-rollback/`: the stopped
|
||||
7,380,164,608-byte `catalog-before-deploy.sqlite3`,
|
||||
`config-before-deploy.json`, and `verstack-before-native`. Restoring an old catalog
|
||||
would discard subsequent writes, so these are recovery artifacts, not a command
|
||||
to overwrite a live database.
|
||||
|
||||
Live verification:
|
||||
|
||||
* `scripts/verify_phase0.py --skip-build`: **74 pass, 0 fail, 0 skip**.
|
||||
* Catalog schema is **13**, with 1,170,298 signature rows and 836,618 nonempty
|
||||
full masks, matching the migration rehearsal.
|
||||
* `tests/ui-plan-live.mjs`: authoritative Pokémon 0.85→0.86 counts of 57 added,
|
||||
95 removed, 56 modified and 12,173 unchanged; **19,470 function matches**;
|
||||
evidence displayed and Monaco difference opened; no API writes or page errors.
|
||||
* `tests/ui-review-fonts-live.mjs`: real archived Stern_Aztech font, 104 glyphs,
|
||||
three specimen sizes and supported repertoire rendered without browser errors.
|
||||
* `scripts/verify_mcp_live.py`: real deployed executable initialized over stdio,
|
||||
discovered **11 tools**, searched/read an archived font and verified the SHA-256
|
||||
of 64 fetched bytes. No proposals or other archive writes were made.
|
||||
* `/api/governor`: timestamped CPU/pressure/memory/scratch sample, zero active
|
||||
reservations. `/proc/<backend-pid>/exe` matches the tested release hash above.
|
||||
|
||||
Screenshots are in `data/validation/review/`: `canonical-workbench.png`,
|
||||
`tiered-workbench.png`, and `font-specimen.png`. Gate output is currently in
|
||||
`/tmp/verstack-deployed-gate.log`.
|
||||
|
||||
This deploys the first verified implementation batch. Subsequent scene-parser and
|
||||
native codesig work in the working tree is not part of this binary. The remaining
|
||||
requirements in `plan-implementation-status.md` remain open; this deployment does
|
||||
not establish completion of the entire plan or any destructive storage cleanup.
|
||||
|
||||
## Second approved batch, 2026-09-17 UTC
|
||||
|
||||
The user explicitly approved this deployment. Backend and workbench are active
|
||||
with zero automatic restarts. The backend was stopped with no running or queued
|
||||
imports and no reservations. Automatic processing remains disabled.
|
||||
|
||||
Running executable SHA-256:
|
||||
`4c514237fabb7d0984cd2766a019a63da4c2ab093a152b41406b37f704f8b265`.
|
||||
All 13 Python plugins are frozen at source bundle
|
||||
`73b6d89498a99b6f465b095aae2bee295c9de9b450d38db0f859cf8fd8fe4489`.
|
||||
Import extraction revision 6 selects the verified, hash-named native ext4 helper.
|
||||
|
||||
Rollback artifacts are under `data/deployment/release2-20260917/`: the previous
|
||||
running binary, exact previous configuration, workbench bundle, and a stopped
|
||||
7,884,779,520-byte SQLite backup. Its quick check returned `ok`; SHA-256 is
|
||||
`2d01ad64d6b4a98f1c55ea773ba47535fbb3fafa6996702300d1e1b240c33f43`.
|
||||
A catalog restore would discard later writes and requires preserving that state.
|
||||
|
||||
Release gate: 233 Rust tests passed (six opt-in tests ignored by default),
|
||||
109 Python tests passed, all-target Clippy passed with warnings denied, and the
|
||||
optimized build passed. Real native Godot/ZIP and host admission fixtures were
|
||||
also explicitly exercised before deployment. Workbench preparation/typecheck
|
||||
and browser bundle both passed.
|
||||
|
||||
Startup upgraded the catalog to schema 15. The explicit live identity plan
|
||||
matched the rehearsal: 28 original ELF proofs, 50,361 aliases, 296,763 observations,
|
||||
and zero conflicts. Applying the saved plan preserved 8,366 name claims and four
|
||||
reviews and recomputed six comparison pairs. Exact row comparison against the
|
||||
backup confirmed unchanged name and review records. The signature inventory
|
||||
remains 1,170,298 rows, including 836,618 nonempty full masks. A fresh plan reports
|
||||
zero remaining aliases and zero conflicts.
|
||||
|
||||
Live verification after migration:
|
||||
|
||||
* Archive/gateway verification: **74 pass, zero failures or skips**.
|
||||
* Workbench: 57 added, 95 removed, 56 modified, 12,173 unchanged; 19,470 code
|
||||
matches, evidence visible, Monaco difference opened, no API writes/page errors.
|
||||
* Font browser check: Stern_Aztech, 104 glyphs, three specimen sizes, no errors.
|
||||
* Deployed executable MCP: 11 tools, archived byte readback with verified SHA-256.
|
||||
* Governor: shared 40-task capacity, 52 available CPU threads, no idle reservations;
|
||||
configured memory, scratch and archive-space admission limits reported.
|
||||
|
||||
The first font request overlapped the migration transaction and timed out; the
|
||||
post-migration rerun passed. This was within the approved interruption window.
|
||||
Detailed manifests and identity JSON are retained alongside the backup. Live
|
||||
check logs use `/tmp/verstack-release2-*-live.log` and screenshots remain under
|
||||
`data/validation/review/`.
|
||||
|
||||
Native SPK and instruction-normalization standalone crates and isolated emulator
|
||||
changes are not included in this release. Actual Star Wars SPIKE 3 retry,
|
||||
cross-generation comparison and the broader PLAN remain open. No GC was done.
|
||||
|
||||
Deployed native Godot recovery on Pokémon 0.86 media snapshot
|
||||
`101b56cd-39c4-48a7-93b6-e2ab47baf88f` read 1179 sidecars across 2 pack trees,
|
||||
named 1179 targets and recovered 1062 orphan names,
|
||||
appending 1179 claims. It reported 3 unresolved targets
|
||||
and 1062 warnings; detailed evidence is `godot-names.json` beside
|
||||
the deployment manifest. This is one real snapshot, not corpus-wide coverage.
|
||||
|
||||
## Third batch — deployed 2026-09-17
|
||||
|
||||
The approved deployment is running with schema 16. Backend PID 2350827 and
|
||||
workbench PID 2350828 were active with zero restarts at the health check.
|
||||
The running executable hash was verified against the prepared artifact:
|
||||
|
||||
* Backend: `f176ff27c0ac0cc05062725a0ee07d8d9bc75f537a17e5911beb9e4836f7af6a`.
|
||||
* All 13 Python plugins: bundle `7eef7c9346e5f055930cc7ee5b41cda20cb5972684f78f9ff4133548b26edff2`.
|
||||
* Configuration: `6394005a9a46120864ac1244b8c5bd672e739563dc14e7ef12a9948bb16890bd`.
|
||||
* Automatic processing remains false. `import-extract/7` uses the hash-pinned
|
||||
native SPK helper and updated native ext4 helper. `media-extract/5` retains
|
||||
the existing default decoder; native Radium extraction is explicitly opt-in
|
||||
pending legacy/native audio identity compatibility.
|
||||
|
||||
Rollback artifacts and manifests are under `data/deployment/release3-20260917/`.
|
||||
The stopped schema-15 catalog backup is 8,351,817,728 bytes, passed `quick_check`,
|
||||
and has SHA-256 `d420a78f0c0c02ad74a6991999d6287ae5a6f30ed6992fcc8db9ffbf3c81cc46`.
|
||||
It contains 1,170,298 signatures, 14,642 name claims and four reviews. The exact
|
||||
previous backend/configuration are retained. Frontend fallback uses the earlier
|
||||
verified `release2-20260917/workbench-before.tar`; it is not an exact capture of
|
||||
the immediately preceding frontend, whose files had already been rebuilt.
|
||||
|
||||
The final gate passed 264 Rust tests (13 explicit corpus/host tests ignored by
|
||||
default), 121 Python tests (one corpus test skipped by default), all-target
|
||||
Clippy with warnings denied, and the optimized build. The native Python corpus
|
||||
gate was separately run: eight tests passed, including real PCM/WAV samples and
|
||||
GOT/Pokémon format selection. Both Theia build steps passed. The schema-16 copy
|
||||
rehearsal accounted for all 1,170,298 legacy rows, preserved every original
|
||||
signature/mask, and proved idempotence. See `native-code-canary.json`.
|
||||
|
||||
Live feature checks passed:
|
||||
|
||||
* The workbench compared real Pokémon `png_set_quantize` functions: 437 aligned
|
||||
instructions out of 439 on each side, score **0.9954441913439636**, with no
|
||||
browser errors. Only the two explicit native evidence records were added.
|
||||
* The Videos view resolved Pokédex on/off cues to records **2063/1857**, with
|
||||
source lines **59/62**. Audio played with nonzero samples, the 309,288-byte WAV
|
||||
downloaded successfully, and there were no browser errors or API writes.
|
||||
* The optimized isolated full GOT publication test passed all **17,159** pixel
|
||||
hashes and frame occurrences, **17,139** structural sequence occurrences,
|
||||
**68,622** files and **202,314,277** output bytes. Import took 16.78 seconds;
|
||||
publication finished at 129.31 seconds and full readback at 132.45 seconds.
|
||||
Post-import RSS was 446,980 KiB; publication peak was 872,384 KiB, an observed
|
||||
additional 425,404 KiB. This is one corpus measurement, not a universal heap
|
||||
guarantee. The slower debug run also passed and cleaned up automatically.
|
||||
|
||||
Evidence includes `health.json`, `prepared.json`, `gates.json`, the live browser
|
||||
JSON/screenshots under `data/validation/{native-code,video-audio}/`, and
|
||||
`data/validation/native-radium-20260917/root-got-publication-release.log`.
|
||||
Post-deployment recovery and regression evidence is recorded below. The broader PLAN remains
|
||||
open; no destructive GC was performed.
|
||||
|
||||
Post-deployment native Godot recovery completed on the same snapshot: all 1,179
|
||||
sidecar targets were named, with zero parser warnings and zero orphan-name
|
||||
fallbacks. It appended 1,062 direct claims; three unresolved targets remain.
|
||||
A live SQL comparison against the stopped backup found no missing or changed
|
||||
rows among all 14,642 previous name claims, all four reviews, or all 1,170,298
|
||||
function signatures (including their full masks). There are now 15,704 name
|
||||
claims. Evidence: `godot-names.json` and `preservation.json` in the deployment
|
||||
directory.
|
||||
|
||||
The live saved-code backfill completed all 1,170,298 observations in 392.23 seconds,
|
||||
ending with a zero-row replay. The live catalog has 1,018,405 code functions,
|
||||
including the two separately verified native browser-test functions. Saved SQL
|
||||
claims are not implied to have native byte verification. Evidence:
|
||||
`code-backfill.json` and `code-backfill-counts.json`.
|
||||
|
||||
The first final regression pass found 72/74 passing checks: the LAN gateway had
|
||||
remained stopped after deployment. The existing password-protected gateway was
|
||||
started. The repeated full live regression passed **74/74 checks**, including
|
||||
the LAN password gate, canonical comparisons, thumbnails, and audio delivery.
|
||||
Backend, workbench and gateway are all active with zero restarts. Evidence:
|
||||
`live-regression.log` in the deployment directory.
|
||||
|
||||
## Fourth batch — deployed 2026-09-17
|
||||
|
||||
The approved storage reporting batch is running with schema 18. The optimized
|
||||
backend artifact is `06a56b45f64c5f814e7fdaa1f3d0d0bd1151e586c558276357d2614b10036dae`;
|
||||
configuration remains `40b4ddad06e14839731a769e0540ee9e233c29d1cc96b32a2af48100a7c018fd`.
|
||||
Rollback copies for the prior executable and configuration are
|
||||
`release4-20260917/verstack-before-storage-roots` and
|
||||
`release4-20260917/config-before-storage-roots.json`.
|
||||
|
||||
The report-only storage-root inventory is deployed. It reads catalog, active-output,
|
||||
and deletion-journal references and reports bounded unreferenced backend snapshots;
|
||||
it performs no deletion or garbage collection. Focused catalog-management tests
|
||||
passed 4/4, catalog storage tests passed 2/2, formatting, locked offline Clippy,
|
||||
and the optimized build passed. The full live regression passed **74/74** after
|
||||
restart, and the native browser canary remains **0.9954441913439636** with 437 of
|
||||
439 aligned instructions and no page errors. A full measured catalog overview was
|
||||
not used as a gate because the 9.6 GB catalog scan exceeded the smoke timeout;
|
||||
that path is covered by the focused tests and the unmeasured live catalog endpoint.
|
||||
The broader PLAN remains open and no destructive GC was performed.
|
||||
|
||||
## Fifth batch — deployed 2026-09-17
|
||||
|
||||
The approved schema-19 durability and streaming-composition batch is live. The
|
||||
optimized backend hash is `759a7d3f959507d049e6cde3d673ea8a4f19fb29d1c85f8ad72a9284ab98ae00`;
|
||||
configuration remains `40b4ddad06e14839731a769e0540ee9e233c29d1cc96b32a2af48100a7c018fd`.
|
||||
Rollback copies are `release4-20260917/verstack-before-schema19` and
|
||||
`config-before-schema19.json`.
|
||||
|
||||
Startup migrated the catalog to schema 19 and published 138 immutable snapshot
|
||||
recipe rows. The new report-only endpoint `/api/catalog/storage-roots` returned
|
||||
138 catalog roots, zero pinned roots, and one unreferenced backend snapshot;
|
||||
`/api/catalog/pins` returned an empty pin set. Recipe and pin behavior is covered
|
||||
by the catalog-management suite, including deletion protection until unpinning.
|
||||
The ZIP member reader and nested decoder dispatch are deployed as a bounded
|
||||
inspection boundary, and native call-graph context remains candidate-only.
|
||||
|
||||
The live regression passed **74/74** after migration. The native browser canary
|
||||
returned **0.9954441913439636**, with 437 of 439 aligned instructions and no page
|
||||
errors. Focused catalog, native-code, import-DAG, and ZIP tests passed; formatting,
|
||||
locked offline Clippy, and the optimized build passed. Recipe execution,
|
||||
reacquisition, decode-before-store, full unwrap-chain publication, and destructive
|
||||
GC remain open; no GC was performed.
|
||||
|
||||
## Sixth batch — deployed 2026-09-17
|
||||
|
||||
The approved plugin-streaming, agent-accounting, reacquisition-report, and legacy
|
||||
comparison-retirement batch is live. The backend artifact is
|
||||
`247a989428cb512e1a0cfd11ceb87f0d709409aa378436745fa6d0ef1ceeb871`; rollback
|
||||
copies are `release4-20260917/verstack-before-stream-agent-legacy` and
|
||||
`config-before-stream-agent-legacy.json`.
|
||||
|
||||
The legacy `/api/compare` route now returns 404; the workbench and verifier use
|
||||
canonical `/api/spine/diff` release IDs. The opt-in `stream_input` protocol serves
|
||||
bounded read-only catalog ranges over a local Unix socket, and `verstack agent`
|
||||
is dry-run by default with explicit execution and append-only daily/per-run cost
|
||||
ceilings. `verify-reacquirable` reports exact source BLAKE3 evidence and cannot
|
||||
authorize GC.
|
||||
|
||||
The full live regression passed **74/74** and the native canary remained
|
||||
**0.9954441913439636** with 437/439 aligned instructions and no page errors.
|
||||
Focused Rust/Python tests, formatting, denied-warning Clippy, and the optimized
|
||||
build passed. Existing plugins have not all migrated to streaming, and the full
|
||||
unwrap chain, recipe execution, source reacquisition at corpus scale, and
|
||||
human-confirmed GC remain open.
|
||||
|
||||
## Seventh batch — deployed 2026-09-17
|
||||
|
||||
The approved schema-20 bridge and unwrap batch is live. The running backend hash is
|
||||
`a8bd7874662a66c38e8f91678bdeb9bfd0e1ff8ad04188eee695379720466e80`; rollback
|
||||
copies are `release4-20260917/verstack-before-schema20` and
|
||||
`config-before-schema20.json`.
|
||||
|
||||
Schema 20 migrated successfully. `/api/code/bridges` is live and explicitly
|
||||
candidate-only with zero accepted matches or propagated names; the storage-root
|
||||
report remains report-only. ZIP→SPK publication now consumes a retained ZIP member
|
||||
through the bounded reader and records parent/provenance evidence without an
|
||||
intermediate member file. The corpus queue reports 73 packages, 6 complete and
|
||||
67 coverage-blocked, with bounded resumable selection and no import side effects.
|
||||
|
||||
The full live regression passed **74/74**. The native comparison canary returned
|
||||
**0.9954441913439636**, with 437/439 aligned instructions and no page errors.
|
||||
Focused Rust tests, denied-warning Clippy, formatting, and the optimized build
|
||||
passed. ZIP→LUKS/ext4, SPK→Radium, complete corpus imports, and destructive GC
|
||||
remain open.
|
||||
|
||||
## Eighth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release5-20260917` deployed with backend SHA-256 `7b514774afed53b2ea088cea719e2cb35ad6a14a294e434ba7d1427ffd31d7da`; rollback binary/config are retained in the release directory.
|
||||
- Live regression: **74/74 pass**; governor idle after restart.
|
||||
- Phase 5: candidate-only function similarity reports now persist bounded graded deltas and expose paged readback at `GET /api/code/deltas`.
|
||||
- Phase 4: bounded ZIP-member → LUKS2 → ext4 composition is covered by a stored-member fixture; source wrappers and plaintext images remain unmaterialized.
|
||||
- Phase 7: wrapper imports default to decoded-only storage; explicit forensic retention is opt-in and recorded in `wrapper-evidence.json`.
|
||||
- Live storage-root inventory remains report-only: 138 catalog roots, one unreferenced backend snapshot; no deletion performed.
|
||||
- Indexed search latency smoke: nine live asset-search requests returned HTTP 200; maximum observed latency was 32.3 ms across `font`, `pokemon`, and `message` queries. See `docs/search-latency-verification.md`.
|
||||
|
||||
## Ninth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release6-20260917` deployed with backend SHA-256 `8b11bd61422a0f91c4ae53fbc60b6bcad68ab6ac1bdae57dc7aedd20723444fd`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 5 bridge readback now resolves endpoint function IDs/status by exact input hash, language, and canonical address; unresolved candidates remain explicit and candidate-only.
|
||||
- Phase 4 adds a bounded verified SPK member reader that streams one member without extraction staging.
|
||||
|
||||
## Tenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release7-20260917` deployed with backend SHA-256 `8b7608b41551a41c410e66370a481dc8b0764539525bfce8717ff66558298939`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 5 call-graph context now ranks corroborated direct-anchor candidates and exposes bounded reference evidence, remaining candidate-only.
|
||||
- Phase 4 now hands verified SPK members directly into the Radium reader/publication path without member materialization; provenance records source coordinates and retention.
|
||||
|
||||
## Eleventh batch — deployed 2026-09-17
|
||||
|
||||
- Release `release8-20260917` deployed with backend SHA-256 `8089a1edb4b6e1279ce4f70e76cf7b9c44987b7e3b788ad165df69f8c8bdf45d`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 2 adds deterministic decoder-registry metadata at `GET /api/decode/registry`, covering native Radium, SPK, ZIP, and scene decoders.
|
||||
- Phase 7 reacquisition verification now supports bounded resumable paging while remaining report-only and `gc_eligible=false`.
|
||||
- Phase 5 exposes candidate-only Thumb/stripped boundary calibration evidence without guessing extents or propagating names.
|
||||
|
||||
## Twelfth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release9-20260917` deployed with backend SHA-256 `8c3d40b72323d14a069ac7d717e8737ee094fc9b80b15a43906933373de70d0b`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 5/8 now indexes verified native reference strings (identity and literal hashes) in a separate bounded FTS index, preserving candidate-only provenance and lifecycle cleanup.
|
||||
- Phase 3 adds `measure-concurrency`, a report-only bounded TaskGraph admission harness; an 8-graph/3-slot run completed 32 tasks with max active 3 and no archive/plugin writes.
|
||||
|
||||
## Thirteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release10-20260917` deployed with backend SHA-256 `bf26e1780052582956f461d81672741a5cd49763c490c2819200fb772cd2c0f9`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 5 adds bounded native calibration readback for ARM mode/decode/reference coverage, with candidate-only semantics.
|
||||
- Phase 7 adds `recipe-dry-run`, validating snapshot recipes, parents, runs, pins, layers, and output bounds without execution or GC.
|
||||
- Phase 8 emulator self-consistency canary validates bounded frame/audio hash traces without starting gameplay.
|
||||
|
||||
## Fourteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release11-20260917` deployed with backend SHA-256 `4bddd8993a4c28796879037a9e38a7da129b0afe9eb36fba6417107235547740`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 2 audio compatibility evidence now distinguishes native-default-eligible geometry/identity matches from opt-in or unrated records without catalog mutation.
|
||||
- Phase 5 exposes bounded corpus-native calibration aggregation with explicit truncation and candidate-only semantics.
|
||||
- Phase 7 provides report-only Star Wars ELG SPIKE2/SPIKE3 storage usage accounting; canonicalization remains false and no storage mutation occurred.
|
||||
|
||||
## Fifteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release12-20260917` deployed with backend SHA-256 `5dbe8f37b85b647625de9f1985fbf576e6a66016e8134f59bdceac4312d08643`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Phase 2 scene graphs now preserve serialized frame order, declared frame rate, and sound-event ordinals while explicitly marking runtime timing unverified.
|
||||
- Native/legacy audio compatibility reports default-eligible matches only when source, section, ordinal, channel geometry, raw PCM identity, and verified rate agree.
|
||||
- The bounded audio gate also requires exact identity-set coverage and equal verified rates; subset, rate-divergence, and payload-divergence reports remain `opt-in` with `report_only:true`.
|
||||
|
||||
## Sixteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release13-20260917` deployed with backend SHA-256 `33ed7a7e07c169467fc25c8c9362a64d2ddfcff4c848184a52ab64602b045f53`; rollback binary/config are retained.
|
||||
- Live regression: **74/74 pass** after restart.
|
||||
- Star Wars storage reports now include a report-only chunk-overlap canonicalization estimate; `canonicalization.performed` remains false.
|
||||
- Native normalization-group smoke verifies bounded groups remain candidate-only and every comparison has `comparison_allowed=false`.
|
||||
|
||||
## Seventeenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release14-20260917` deployed with backend SHA-256 `6d222bff7d19aaa6c612c3317637471653f33ec34b56c26b6e6d802684a1319a`; rollback binary/config are retained.
|
||||
- Backend, workbench, and gateway services are active after restart. The release artifact hash matches the running backend executable.
|
||||
- Phase 5 boundary calibration now persists bounded Thumb/stripped candidate metrics in native normalization reports without guessing extents or promoting matches.
|
||||
- Phase 7 audio storage audit now reports WAV/PCM migration readiness, retained-file checks, and FLAC tool availability; migration remains report-only and unperformed.
|
||||
- Scene catalog readback now exposes serialized timing semantics and sound-event edges while keeping runtime playback timing explicitly unverified.
|
||||
|
||||
## Eighteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release15-20260917` deployed with backend SHA-256 `0b45eb7d6b2b0fb2bf8046dbd7b22e243c67ac61299c4d982e28ece610398722`; rollback binary/config/catalog are retained.
|
||||
- Backend, workbench, and gateway are active; the running backend hash matches the release artifact after restart and schema startup.
|
||||
- Binary function comparison now presents the bounded overlap score in decimal and percentage form, for example `0.994 (99.4%)`, with the existing denominator and candidate-only caveats.
|
||||
- Video/audio scene reports now expose contextual timeline sound events, embedded-track/probe requirements, and an explicit `direct_binding_verified` flag. No sound is assigned to a clip without independent evidence.
|
||||
- Browser regressions passed for the native comparison and video/audio evidence components; focused Rust tests passed for both report paths.
|
||||
|
||||
## Nineteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release17-20260917` deployed with backend SHA-256 `19b74c7e13c63e769332ace5f43478b4983d0f750a60d5e305fca46b9537788b`; rollback binary/config/catalog are retained.
|
||||
- The additive native reference-string projection is now installed and replayed for existing schema-17+ catalogs. Live `GET /api/code/strings/coverage?limit=2` reports 527 evidence rows, 1,355 references, 521 indexed identity/literal rows, and 203 evidence rows with indexed strings.
|
||||
- Coverage remains candidate-only and does not infer names, semantic validity, or function correspondence. All three services are active and the running backend hash matches the release artifact.
|
||||
|
||||
## Nineteenth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release17-20260917` deployed with backend SHA-256 `19b74c7e13c63e769332ace5f43478b4983d0f750a60d5e305fca46b9537788b`; rollback binary/config/catalog are retained.
|
||||
- The additive native reference-string projection is now installed and replayed for existing schema-17+ catalogs. Live `GET /api/code/strings/coverage?limit=2` reports 527 evidence rows, 1,355 references, 521 indexed identity/literal rows, and 203 evidence rows with indexed strings.
|
||||
- Coverage remains candidate-only and does not infer names, semantic validity, or function correspondence. All three services are active and the running backend hash matches the release artifact.
|
||||
|
||||
## Phase 3 live governor telemetry
|
||||
|
||||
The bounded `scripts/verify_live_governor.py` probe reads `GET /api/governor`
|
||||
only. It records the host and execution snapshots and reports the peak active
|
||||
analysis plus preparation count. Its evidence explicitly records that no task
|
||||
was submitted and no archive or catalog write was performed. This verifies the
|
||||
live admission telemetry endpoint; it does not close the separate requirement
|
||||
for sustained production workload throughput.
|
||||
|
||||
## Twentieth batch — deployed 2026-09-17
|
||||
|
||||
- Release `release18-20260917` deployed with backend SHA-256 `083817f37f882a541299c86dd20f3df7ff8bb955c0de9489d8ced7bbc97edd04`; rollback binary/config/catalog are retained.
|
||||
- Scene observations now receive conservative semantic types from serialized object classes (`Bitmap`, `Sprite`, `Font`, `Text`, `Video`, `Shape`, `StreamingFlipbook`); mixed or auxiliary classes remain untyped.
|
||||
- A bounded live governor probe completed 5/5 samples with zero active tasks, no reservations, no submitted work, and no writes. All services are active and the running backend hash matches the release artifact.
|
||||
|
||||
## Twenty-first batch — deployed 2026-09-17
|
||||
|
||||
- Release `release19-20260917` deployed with backend SHA-256 `d0c469f121416e44946956401d5eca59140d56767e735d773aff4363955d1058`; rollback binary/config/catalog are retained.
|
||||
- Schema 21 persists bounded code-propagation candidates with hop, shared-block, context-priority, and provenance fields. Live readback is candidate-only with zero accepted matches and zero propagated names.
|
||||
- Existing code-string coverage remains live after migration, and all three services are active with the running backend hash matching the release artifact.
|
||||
|
||||
## Twenty-second batch — deployed 2026-09-17
|
||||
|
||||
- Release `release20-20260917` deployed with backend SHA-256 `7208e7cb885493d104a3520d8f463dd02170cc6d749269d49b51ff10a93f65ed`; rollback binary/config/catalog are retained.
|
||||
- The ext4 reader now exposes a bounded seekable regular-file view to nested decoders with filesystem/file-size budgets, avoiding plaintext image and intermediate payload materialization.
|
||||
- Fixture verification passed for a 128 KiB nested payload; live decoder registry and candidate-only propagation readback remain healthy, with all three services active.
|
||||
|
||||
## Twenty-third batch — deployed 2026-09-17
|
||||
|
||||
- Release `release21-20260917` deployed with backend SHA-256 `75c9dd2459f1f455d54373428094558d1b79becd370a5d2715c7ddb1005fb096`; rollback binary/config/catalog are retained.
|
||||
- The composed bounded decoder route `/api/decode/luks-ext4-spk-radium` is live. It wires ZIP-member, LUKS2, ext4 regular-file, SPK-member, and Radium readers while retaining explicit no-materialization provenance.
|
||||
- Route dispatch was verified live (malformed payload correctly returned HTTP 422), the decoder registry remains healthy, and all three services are active.
|
||||
|
||||
## Live gate refresh — 2026-09-17
|
||||
|
||||
- `python3 scripts/verify_phase0.py --skip-build` completed against the live gateway/backend with **73/73 pass, 0 fail, 0 skip**. This includes the canonical release comparison, binary function pairing, review application, thumbnail pyramid, text diff, sound playback, and spine invariants.
|
||||
## Release 22 compatibility fix — deployed 2026-09-17
|
||||
|
||||
- Release `release22-20260917` deployed with backend SHA-256 `e0faeb3a012183d63b0f04af966acca501697007e18f99805d535a28681ae079`.
|
||||
- The schema-21 propagation cleanup now checks for the additive table before deleting dependent rows, preserving rollback and rehearsal compatibility with older catalogs.
|
||||
- The full Rust library gate passed: **105 passed, 0 failed, 4 ignored**; Clippy with `-D warnings` also passed.
|
||||
- The approved live gate completed **74/74 checks passed, 0 failed, 0 skipped**. Backend, workbench, and gateway are active and the decoder registry responds.
|
||||
- Rollback artifacts include the pre-change binary/config and a hard-linked catalog backup shared with release 21; no catalog data was removed.
|
||||
## Release 23 scene-name propagation — deployed 2026-09-17
|
||||
|
||||
- Release `release23-20260917` deployed with backend SHA-256 `caad98b823ecefc9bb9b7558c0d078da9368ad4099018567ed55e7f78fb2fba1`.
|
||||
- Native scene indexing now propagates an exact serialized Element name to a uniquely instantiated media object as an evidence-backed `T3_propagated` claim; ambiguous bindings remain unnamed.
|
||||
- Focused scene/decoder tests passed (20 passed, 1 ignored), Clippy passed, and the post-deploy live gate completed **73/73 checks passed, 0 failed, 0 skipped**.
|
||||
- Backend, workbench, and gateway are active; rollback binary/config and the hard-linked catalog backup are retained.
|
||||
## Release 24 PCM-WAV synthesis — deployed 2026-09-17
|
||||
|
||||
- Release `release24-20260917` deployed with backend SHA-256 `950c9f6ff37844dfb5398161ba1b187065bbdeedbe0715206d4aa59411672886`.
|
||||
- `GET /api/media/pcm-wav` now streams explicitly typed `pcm_s16le` artifacts with a bounded canonical WAV header and source/geometry provenance.
|
||||
- The video/audio integration suite passed 2/2, Clippy passed, and the live gate completed **73/73 checks passed, 0 failed, 0 skipped**.
|
||||
- All services are active; rollback binary/config and the hard-linked catalog backup are retained.
|
||||
|
||||
## Working-tree verification and deployment drift — 2026-09-17
|
||||
|
||||
This section records a re-verification of the uncommitted working tree. It is a
|
||||
measurement, not a deployment or an approval; nothing was restarted.
|
||||
|
||||
Repository gates, all green against the tree as it stands:
|
||||
|
||||
* `cargo build --locked --all-targets` and `cargo build --release --locked`: clean
|
||||
and already up to date. The optimized artifact is SHA-256
|
||||
`0c2ec778cb2357f30b4eef4306c7be825d56be473e4665259e308b6eec8177ad`.
|
||||
* `cargo fmt --check`: clean. `cargo clippy --locked --all-targets -- -D warnings`:
|
||||
zero warnings.
|
||||
* `cargo test --locked`: **322 passed, 0 failed, 16 ignored** across 42 binaries.
|
||||
Every ignore carries an explicit corpus/host reason.
|
||||
* Standalone crates, which the root workspace does not cover: `verstack-spk`
|
||||
7 passed / 2 ignored, `verstack-code` 19 passed, `verstack-luks` 4 passed,
|
||||
`verstack-radium` 3 passed / 1 ignored.
|
||||
* `tools/decoder-env/bin/python -m unittest discover -s tests -p 'test_*.py'`:
|
||||
**172 passed, 3 skipped**.
|
||||
* `workbench: npm run prepare` (the only step that typechecks): clean.
|
||||
* `python3 scripts/verify_phase0.py --skip-build` against the live gateway and
|
||||
backend: **74 checks, 74 pass, 0 fail, 0 skip**.
|
||||
|
||||
Two deployment gaps were measured and are **not** closed by the gates above.
|
||||
|
||||
**The running backend is not this tree.** `verstack-backend` started 10:22:56 on a
|
||||
build whose SHA-256 is `fe71789fa2b8295fbeaf14907429cb2a07de006871b36f4a8a4d8f99ccca6d5e`;
|
||||
`src/plugins.rs` was written at 10:23:38, after that build. The tree's optimized
|
||||
artifact differs from the running one by 23,288 bytes. A restart deploys the tree.
|
||||
`data/deployment/release25-20260917/` also holds only `config-before-ext4-dispatch.json`:
|
||||
unlike release 22 through 24 it has no `verstack-before-*` binary, no hard-linked
|
||||
catalog backup and no `after.json`, so the currently running release has no
|
||||
rollback binary recorded.
|
||||
|
||||
**The served Theia bundle was behind the backend, and is now rebuilt.**
|
||||
`stern-catalog/lib` was last compiled at 02:39 and `browser-app/lib/frontend/bundle.js`
|
||||
was bundled at 05:52 — from that 02:39 `lib`, which is the documented trap.
|
||||
`native-code-comparison.tsx` was edited at 05:49 and its output was absent from the
|
||||
served bundle, so the decimal-plus-percentage score recorded as deployed in the
|
||||
eighteenth batch was **not** what the workbench served. Both steps have now been
|
||||
re-run: the score helper
|
||||
`` `${n.toFixed(3)} (${(n * 100).toFixed(1)}%)` `` is present in the 11:03 bundle.
|
||||
|
||||
**`stern-catalog-widget.tsx` is dead code and its edits never reach the UI.** This
|
||||
is a separate finding, not bundle staleness. `SternCatalogWidget` is referenced
|
||||
nowhere outside its own file; the live views are `workspace-views.tsx` and the
|
||||
components it imports. Strings that exist only in `stern-catalog-widget.js`
|
||||
(`unique masked signature`, `Download verified function facts`,
|
||||
`Absent from this release`) are correctly absent from the bundle, while strings it
|
||||
shares with `workspace-views.js` are present. The file is 34 KB of maintained
|
||||
source — 17 insertions in this pass — that nothing loads. Either wire it in or
|
||||
delete it; do not read its contents as evidence of what the workbench shows.
|
||||
|
||||
### Tree reload — deployed 2026-09-17
|
||||
|
||||
The working tree was committed as `473ee47` and both drift gaps above were closed.
|
||||
|
||||
* Rollback artifacts are `data/deployment/release26-20260917/`: the previous running
|
||||
binary `verstack-before-tree-reload` (SHA-256 `fe71789f…`, recovered from the live
|
||||
process before it was unlinked), `config-before-tree-reload.json`, and a hard-linked
|
||||
9,596,272,640-byte `catalog-before.sqlite3` taken with the backend stopped, which
|
||||
passes `quick_check` at schema 21.
|
||||
* No migration ran: the live catalog and `SCHEMA_VERSION` are both 21. The backend was
|
||||
stopped with zero active budgets, zero reservations and an empty job list.
|
||||
* The running backend is now SHA-256
|
||||
`0c2ec778cb2357f30b4eef4306c7be825d56be473e4665259e308b6eec8177ad`, matching the
|
||||
committed tree's optimized artifact.
|
||||
* The workbench was rebuilt with both steps and restarted; the served
|
||||
`bundle.js` is 24,010,838 bytes and contains the score helper that the previous
|
||||
bundle lacked.
|
||||
* Backend, workbench, gateway and exports are active with **zero** restarts, and
|
||||
`scripts/verify_phase0.py --skip-build` passed **74 checks, 74 pass, 0 fail, 0 skip**
|
||||
after the reload.
|
||||
|
||||
The corpus campaign, the code-match acceptance policy and destructive GC all remain
|
||||
open; no GC was performed.
|
||||
@@ -0,0 +1,965 @@
|
||||
# PLAN.md implementation and verification ledger
|
||||
|
||||
The objective remains completion of the entire current plan. This ledger does not
|
||||
replace or narrow PLAN.md. Historical "original plan" passages are interpreted
|
||||
with the later explicit corrections and deferrals in that document. A capability
|
||||
implemented but not deployed or exercised against the archive is not an outcome.
|
||||
|
||||
## Baseline, 2026-09-16
|
||||
|
||||
The unchanged live service passed `python3 scripts/verify_phase0.py --skip-build`:
|
||||
73 pass, 0 fail, 0 skip. Local socket access requires running this command outside
|
||||
the sandbox. The first sandbox attempt failed on blocked sockets, not a product
|
||||
regression. This baseline verifies existing work, not the new changes below.
|
||||
|
||||
## Work in this implementation pass
|
||||
|
||||
The second verified batch is deployed with schema 15 and all 13 configured Python
|
||||
plugins pinned to one verified source bundle. Live checks passed: 74/74 archive
|
||||
checks, real font rendering, canonical workbench totals, 19,470 code matches, and
|
||||
executable stdio MCP readback. The explicit identity migration applied 50,361
|
||||
aliases with no conflicts while preserving every existing name and review row.
|
||||
See `plan-deployment-verification.md` for binary, bundle, backup and evidence.
|
||||
Deployment does not establish completion of the remaining plan requirements.
|
||||
|
||||
The third batch is now deployed with schema 16, native SPK in the import adapter,
|
||||
graded function/block similarity, and source-backed video/animation sound links.
|
||||
Both new features passed actual live browser checks: a real function pair scored
|
||||
0.995444 (437/439 aligned instructions), and Pokédex on/off cues played their
|
||||
verified audio and supplied a working WAV download. Native Radium publication is
|
||||
implemented and corpus-verified but remains opt-in pending legacy sound identity
|
||||
compatibility. Post-deployment Godot recovery named all 1,179 sidecar targets with
|
||||
zero parser warnings and appended 1,062 direct claims; three unresolved targets
|
||||
remain. The live saved-code backfill completed all 1,170,298 observations in
|
||||
392.23 seconds, with an explicit zero-row replay confirming completion. The final
|
||||
live regression passed 74/74 checks after restoring the LAN gateway.
|
||||
|
||||
The next working-tree batch is now deployed to the backend with a fresh
|
||||
rollback checkpoint at `data/deployment/release4-20260917/`. It adds scene-backed bitmap
|
||||
font atlas rendering, schema-17 native code anchors with searchable `code_strings`,
|
||||
direct archived ext4 publication, and verified legacy/native Radium audio identity.
|
||||
Bounded checks are green: 5 bitmap-font Rust tests; 5 review tests; 8 ext4 archive
|
||||
tests; 124 decoder Python tests (3 explicit skips); focused native-code, Radium,
|
||||
identity, review and asset tests; and all-target Clippy with denied warnings.
|
||||
Real evidence covers 20 font scenes (22 fonts, 92 instances, 6,492 image glyphs,
|
||||
all 1,129 rotated glyphs, 169 atlases), 11,782 verified function bodies with
|
||||
27,026 independently checked anchor occurrences, and 2,491/2,491 Pokémon WAV
|
||||
parity records. This batch still needs the live migration, browser deployment
|
||||
checks. The integrated local `luks-extract` helper and import dispatch are
|
||||
verified against an independently encrypted ext4 fixture and the live backend
|
||||
has now published full real partitions 5 and 6 through the native reader.
|
||||
Frontend deployment checks and the broader corpus campaign remain open.
|
||||
|
||||
The following bounded batch is now verified in the working tree and awaits
|
||||
deployment: schema-18 perceptual image evidence, admission-before-spawn for
|
||||
import-DAG tasks, and a seekable ZIP-member reader. These additions preserve
|
||||
logical identities and delta counts; they do not claim complete streaming
|
||||
unwrap or corpus-wide import.
|
||||
|
||||
| Requirement | Implementation | Evidence / outstanding verification |
|
||||
|---|---|---|
|
||||
| Phase 2: AES-XTS context reuse | `plugins/luks_extract.py` retains EVP context and key schedule across data units | Nine LUKS tests; reproducible benchmark and limits in `aes-xts-verification.md`. Frozen bundle deployed in the first batch |
|
||||
| Phase 2: buffered Read+Seek SourceReader | `src/source.rs`, `Archive::source_reader` | `tests/source_reader.rs`: bounded 8 MiB coalescing, boundary crossing, seeking, overflow, truncated-source failure |
|
||||
| Phase 2: registry and first native adapters | `src/decode/`, read-only `/api/decode/inspect` | Registry/probe/sidecar tests passed and first-batch inspection is deployed. Native ZIP directory tests also pass in the working tree; metadata inspection is not streaming extraction or persisted name recovery |
|
||||
| Phase 5: residual masked matching | `src/analysis.rs`, both workbench views | Real corpus: 17,232 exact + 2,238 full-body masked matches; 12/12 meaningful named masked pairs agree. Sixteen fixture tests plus explicit live canary passed. No prefix-only equivalence claims |
|
||||
| Phase 6: font specimens | `src/review.rs`, `/api/review/font`, `/api/scene/fonts`, review/workbench bitmap-font viewers | Scene-backed BC1/BC3 atlas decoding and rotation now cover 6,492 image glyphs across 20 real scenes; independent atlas and browser pixel checks pass. Layout transforms, kerning direction, and standalone DMD banks remain open |
|
||||
| Phases 6/8: indexed search | schema 13 FTS5 trigram indices with content triggers; catalog and review queue queries | Literal syntax, short-query fallback, mutation/integrity, filtering-before-limit, warm interactive latency, and a bounded 4-worker/80-request live probe passed (63.7 requests/s; p95 112.6 ms). Cold-cache and sustained production-capacity claims remain open |
|
||||
| Phases 1/6: authoritative comparisons | Legacy HTTP compare and workspace read persisted release deltas; separate filtered view counts | Reverse/same-release/missing-pair contracts, LAK joins and filtered-count invariance tested. CLI filesystem comparison retains preservation semantics. Live API and browser counts passed |
|
||||
| Phases 3/4: scratch admission | Atomic movable reservations before uploads, variable input-based budgets, fitting-candidate queue selection, per-stage reacquisition | Regression covers 6→8→11 MiB expanding import chain, blocked-head queues, nested growth/rejection and reservation release. Existing browser-upload size cap remains; actual 63 GB extraction not yet run |
|
||||
| Phase 3: 1 Hz resource sampling | `ResourceLedger` samples host pressure, memory, scratch and /proc/stat | CPU fraction uses measured tick deltas instead of load-average ratio. Background sampling and prompt shutdown tested. Shared admission and import DAG are implemented in the second batch; sustained production verification remains |
|
||||
| Phase 7: additive archival savings | New GZF stays in scratch/checkpoints; verified facts remain downloadable; full masks preserved in dedicated signature column | Storage/export and Python Ghidra tests passed; `storage-additive-verification.md` distinguishes future avoided duplication from actual freed bytes. No GC performed |
|
||||
| Phase 8: stdio MCP and code anchors | Eleven tools over running HTTP service; schema-17 append-only native anchors/code-string FTS | GOT 11,519 and Pokémon 263 verified functions yielded independently checked literal/import evidence; FTS scope, 10k truncation, migration/reopen, rollback, deletion, and browser search checks pass. Search covers explicitly anchor-indexed verified functions; it does not imply names or correspondence |
|
||||
| Phase 2/4: direct archived filesystem extraction | `/api/decode/ext4`, `verstack decode-ext4`, `import-extract/8` partition handoff | Eight Rust archive tests and 20 Python source tests pass; original image is retained and the partition-sized copy is absent. The real 63 GB image has four LUKS partitions, so encrypted range extraction remains open |
|
||||
| Phase 2: native/legacy audio identity | Verified WAV artifact identity plus typed PCM payload and geometry-aware shared delta selection | 2,491/2,491 real Pokémon WAV bytes match; six native Radium integration tests and nine Python tests pass. Native Radium remains opt-in until encrypted range extraction and live compatibility deployment are complete |
|
||||
| Phase 2/4: encrypted range extraction | `verstack-luks`, `/api/decode/luks-ext4`, `decode-luks-ext4` CLI, local `luks-extract` import helper | Native reader and guarded publication fixture pass; all four real Pokémon 0.83 LUKS partitions unlock from bounded capsules. Live full partition-5 publication read 18 files/1,064,022,428 logical bytes and 634,454,016 ciphertext bytes; partition 6 read 689 files/2,276,741,239 logical bytes and 2,679,111,680 ciphertext bytes. Both receipts prove no input or plaintext-image materialization. The pinned helper passes direct-slice dispatch, pin rejection, tiny-budget rejection, and KDF heap-admission tests. `spike-package/4` now hands nested files to the same recursive native importer when pinned helpers are configured; a focused fixture verifies the handoff. Whole-corpus validation and broader coverage remain open |
|
||||
|
||||
## Additional user requirements, 2026-09-17
|
||||
|
||||
* Binary/function comparisons must expose graded similarity (for example 0.994),
|
||||
changed instructions/blocks and the scoring denominator. Similarity and evidence
|
||||
that two functions correspond are separate; neither automatically transfers names.
|
||||
* Associate on-screen videos with their sound: distinguish embedded audio tracks
|
||||
from separately triggered Radium/Godot sounds, preserve source references and
|
||||
extraction provenance, and expose evidenced associations in the product. Filename
|
||||
resemblance or co-occurrence alone is insufficient to claim an exact association.
|
||||
|
||||
Both capabilities are deployed and live browser verification passed. Function and
|
||||
basic-block similarity expose the exact denominator,
|
||||
changed instructions, coverage and bounded candidate searches. The real GOT Pro/LE
|
||||
study covered 128 changed same-name functions and 127 different-name controls;
|
||||
different-name controls reached 0.96, so scores never authorize name transfer.
|
||||
The sound view separates embedded-track extraction from source-backed animation
|
||||
calls. Actual Pokédex calls resolve to Radium records 2063 and 1857; all 264
|
||||
inspected Pokémon MP4s have no embedded audio. See `native-code-evidence.md` and
|
||||
`video-audio-associations.md` for evidence and limits.
|
||||
|
||||
The final combined gate passed 121 Python tests (one corpus test skipped by
|
||||
default, separately exercised) and 264 Rust tests (13 explicit corpus/host checks
|
||||
ignored by default), Clippy and both builds. Its first Rust run exposed
|
||||
two admission-test fixtures depending on live host pressure; controlled healthy
|
||||
samples fixed the fixtures without changing production pressure admission.
|
||||
Native plugin budgets, code-evidence deletion, derived video provenance and raw
|
||||
PCM playback guards passed their required gates before deployment.
|
||||
|
||||
## Remaining scope to audit and implement
|
||||
|
||||
Second-batch deployed capabilities include native scene parsing and catalog
|
||||
persistence, bounded ARM32/AArch64 codesig, native Godot name recovery, optional
|
||||
native ext4 extraction, shared resource admission, per-program Ghidra fan-out,
|
||||
import DAG/input sharing, ZIP expansion budgeting, append-only name claims, and
|
||||
verified container identity. Production coverage and throughput requirements
|
||||
remain where listed below. Emulator changes are isolated and are not deployed.
|
||||
|
||||
The final release gate passed 233 Rust tests (six explicit corpus/host tests
|
||||
ignored by default), 109 Python tests, all-target Clippy with warnings denied,
|
||||
and the optimized build. Required real fixtures and host exercises were also
|
||||
run explicitly. Historical migration-fixture and signature-sync assertions were
|
||||
updated to reflect schema 15 and durable per-program jobs; the final gate passes.
|
||||
|
||||
King Kong and Pokémon edition/container identity canaries passed on the copied
|
||||
catalog and the exact live migration. Both Star Wars imports are complete after retrying SPIKE 3 with the deployed
|
||||
ZIP expansion budget fix. The live three-axis gate passes: Star Wars has 1,481
|
||||
shared original game Radium records and 1,469 menu records, all byte-identical;
|
||||
full canonical comparison counts and remaining coverage limits are recorded in
|
||||
`container-identity-migration.md`. The live native Godot check appended 1179 claims, including 1062
|
||||
MD5 path-preimage claims, but reported 1062 sidecar-parser warnings and
|
||||
3 unresolved targets. A bounded multiline ConfigFile fix now passes all 1,179 real sidecars against
|
||||
the Python reference. The fix is now deployed: the live recovery appended 1,062
|
||||
direct claims with zero parser warnings, superseding the need for orphan-name
|
||||
recovery on those sidecars. Three unresolved targets remain.
|
||||
Native SPK and instruction normalization are now integrated into the deployed
|
||||
binary; their standalone crates remain independently testable.
|
||||
|
||||
|
||||
Native Radium work now lives in `crates/verstack-radium`: binrw header/section-8
|
||||
records, sound-directory CRC and streaming PCM, an independent section-3 index
|
||||
for SPIKE 2 sentinel headers, and all six known DMD encodings with exact cached
|
||||
base occurrences. Three grouped regression tests and Clippy pass. The real GOT
|
||||
and Pokémon images match all 18,636 Python-reference frames and all 2,491 PCM
|
||||
chunks by SHA-256 after verifying both complete source hashes. Root publication
|
||||
and occurrence edges now pass the full GOT archive test: 17,159 frames, 17,139
|
||||
structural sequence occurrences, 68,622 files. Optimized publication/readback
|
||||
took 132.45 seconds with an observed additional peak RSS of about 415 MiB over
|
||||
the post-import baseline. This does not imply verified animation timing or
|
||||
universal resource bounds. Native PCM identities now include channels and an
|
||||
explicit unknown/verified sample rate; legacy/native audio identity continuity
|
||||
must still be measured before enabling this decoder by default.
|
||||
|
||||
These are open requirements, not a list of optional future enhancements.
|
||||
|
||||
* Phase 1: perceptual hash subclassification;
|
||||
the three real identity canaries now pass, including cross-platform Star Wars
|
||||
and Pro/LE; broader corpus identity coverage remains to verify.
|
||||
Preserve dual byte/payload identities and append-only name provenance.
|
||||
Schema 14 now fixes `import_names` overwriting tier/evidence/status on repeat
|
||||
keys, defaults claims to proposed and preserves accepted mechanical imports
|
||||
explicitly. Tests and an 8,366-row real-catalog migration rehearsal passed;
|
||||
schema 14/15 is deployed. See `name-provenance.md`.
|
||||
Schema 18 adds bounded cached dHash64/pHash63 evidence and an explicit
|
||||
`/api/review/perceptual` endpoint. Results are candidate-only and refuse
|
||||
low-information or animated inputs; no identity, delta, or name changes.
|
||||
The legacy path-keyed `/api/compare` route is retired. The workbench and live
|
||||
gate now read canonical release identities through `/api/spine/diff`; filesystem
|
||||
preservation comparison remains an internal/CLI concern.
|
||||
* Phase 2: finish streaming decoder dispatch and native/legacy audio identity
|
||||
compatibility; retain representation-scoped Godot recovery (the three unresolved native
|
||||
attempts are decompressed fonts already named by existing evidence);
|
||||
retroactive semantic coverage; evidence-backed font/sprite classification and
|
||||
remaining non-Pokémon names. Native SPK, ext4, scene graphs and Radium
|
||||
header/sections 3 and 8 plus dependency sequences now have real integration
|
||||
evidence. Dependency order alone does not prove animation timing.
|
||||
* Phase 3: verify shared Task/TaskGraph admission, per-program Ghidra
|
||||
fan-out and import DAG against production work. Shared staged inputs are implemented;
|
||||
bounded streaming plugin IO and CatalogWriter batching are implemented; existing plugins
|
||||
still opt in individually. Verify sustained realized
|
||||
concurrency under workload, not only the passing 32-process fixture.
|
||||
The import DAG now reserves movable resources before marking a task Running;
|
||||
busy peers remain Pending and cancellation does not launch them. Production
|
||||
sustained-concurrency measurement remains open; CatalogWriter batching is
|
||||
deployed and covered by the large-inventory regression.
|
||||
* Phase 4: verify workspace-budget changes against full-size corpus inputs;
|
||||
streaming unwrap chain; import all remaining
|
||||
packages after identity gates; resolve or explicitly establish the two extraction
|
||||
failures. King Kong LE media extraction now completed with 14,414 entries and
|
||||
was backfilled; its canonical edition comparison passed the deployed identity migration.
|
||||
Measure actual completed
|
||||
imports, structural platform evidence and sibling relations.
|
||||
A bounded `ZipMemberReader` streams stored/raw-deflate members with local
|
||||
header, size, seek, and CRC checks without member-sized staging. ZIP→SPK
|
||||
publication is deployed through that reader; ZIP→LUKS/ext4 and SPK→Radium
|
||||
remain open chain links.
|
||||
* Phase 5: call-graph disambiguation; Thumb and stripped boundary recovery;
|
||||
corpus-wide native normalization and calibration; broader referenced
|
||||
strings/imports/constants beyond the verified GOT/Pokémon anchor cohorts;
|
||||
corpus anchor
|
||||
propagation with provenance/hop decay; durable graded function deltas;
|
||||
watchdog progress/cgroup controls; opt-in cached BSim;
|
||||
The existing cross-architecture bridge is now validated and persisted as
|
||||
candidate-only schema-20 evidence; corpus-wide bridge absorption and the
|
||||
remaining matching work stay open.
|
||||
* Phase 6: bitmap font atlases; complete live UI
|
||||
verification of new fonts and search. Retain Theia per the explicit decision.
|
||||
ContentOnly image reassignment and audio diff remain explicitly deferred by the
|
||||
plan until their stated corpus conditions change; do not silently relabel that
|
||||
deferral as implemented.
|
||||
* Phase 7 (last): additive GZF/signature changes are deployed; report-only live
|
||||
roots now include catalog, active-output, and deletion-journal roots with a
|
||||
bounded unreferenced-snapshot sample. Durability/recipes/pins and report-only
|
||||
reacquisition verification are deployed; decode-before-store;
|
||||
Star Wars storage measurement; hash-verified reacquisition and human-confirmed
|
||||
GC; canonical PCM/FLAC and WAV synthesis; zstd dictionaries; qcow2 chains.
|
||||
Protect all irreplaceable originals. No destructive GC is authorized by a green
|
||||
test alone; the plan explicitly calls for human confirmation after verification.
|
||||
* Phase 8: MCP deployed readback passed; corpus-wide code-string index remains;
|
||||
the bounded headless-agent launcher and cost ceilings are now implemented
|
||||
locally (`verstack agent`), with dry-run default, explicit `--execute`, and
|
||||
append-only UTC JSONL accounting; batch captions with
|
||||
provenance; offline ASR readiness and a local, provenance-recording ASR
|
||||
fixture/Whisper pass (`scripts/offline_asr.py`, see
|
||||
`docs/offline-asr.md`); pinned emulator source plus patch series; canonical
|
||||
netbridge module; platform device map; time-boxed SPIKE 2 bridge research;
|
||||
deterministic frame capture and same-version self-consistency before behavior
|
||||
comparisons. External credentials and costs must be resolved with concrete
|
||||
prepared work, not assumed from installed tooling.
|
||||
|
||||
## Next batch in progress after release 3
|
||||
|
||||
* Direct ext4 partition extraction now passes a real mkfs-generated filesystem
|
||||
inside an imageUSB/MBR wrapper through the configured native helper's bounded
|
||||
offset/length interface. The test verifies exact output bytes, original
|
||||
preservation, and that no partition-sized sparse copy occurs. All 20 import
|
||||
source tests pass with local HTTP fixtures enabled. `import-extract/8` is
|
||||
prepared in the configuration generator, not deployed. ZIP member staging,
|
||||
LUKS plaintext staging, and the original plugin input materialization remain;
|
||||
this is one removed copy in the full streaming chain, not its completion.
|
||||
* Plugins now have an opt-in bounded input stream. A plugin setting
|
||||
`stream_input=true` leaves the retained input tree empty and receives a local
|
||||
read-only Unix socket plus a path/size manifest in the existing protocol-1
|
||||
request. Requests are capped at 1 MiB, checked against catalog size and
|
||||
manifest paths, and served through `ArtifactStorage::read_range`; the
|
||||
default materialized-input path remains compatible. `plugins/stream_input.py`
|
||||
is the reference client and `tests/test_stream_input.py` covers manifest and
|
||||
range admission. This is a plugin-IO slice, not completion of the ZIP →
|
||||
LUKS/ext4 → SPK unwrap chain or a claim that existing plugins have migrated.
|
||||
* Archived ext4 extraction now publishes a child directly from `SourceReader`
|
||||
through `/api/decode/ext4` and the matching CLI. All eight integration tests
|
||||
pass, including a source larger than the scratch allowance, original
|
||||
preservation, slice rejection, authorization and cleanup. See
|
||||
`native-ext4-streaming.md`. The real 63 GB disk map was verified with 30,208
|
||||
bytes of reads: its large partitions are LUKS, so full extraction still
|
||||
depends on the pending encrypted range reader.
|
||||
* Godot recovery now emits bounded path/reason diagnostics for unresolved
|
||||
attempts. Pack isolation and append-only recovery integration tests pass.
|
||||
Read-only live evidence identifies all three remaining attempts as decompressed
|
||||
font copies with no same-representation preimage directories; all three
|
||||
already have best names. See
|
||||
`data/validation/godot-sidecars-20260917/unresolved-diagnosis.json`.
|
||||
|
||||
### ZIP member streaming composition (bounded slice)
|
||||
|
||||
`decode::zip::inspect_member` now indexes a retained ZIP and passes a
|
||||
`ZipMemberReader` directly to the native decoder registry. Stored and raw
|
||||
DEFLATE members remain seekable and bounded by their central-directory sizes;
|
||||
the nested report uses a `parent.zip!member` path while retaining the parent
|
||||
content identity. The integration test exercises ZIP-to-SPK structural
|
||||
dispatch without creating a member file. This is a composable inspection
|
||||
boundary, not completion of the full import unwrap chain: ZIP-to-LUKS/ext4,
|
||||
SPK payload verification, publication, and the Python plugin's input
|
||||
materialization still require separate work.
|
||||
|
||||
## Required final gates
|
||||
|
||||
Run the repository gates (Rust tests, clippy with denied warnings, decoder Python
|
||||
tests, live verifier), plus new feature integration/browser/corpus checks. Rebuild
|
||||
Theia with both prepare and build:browser whenever its source changes. Deploy and
|
||||
inspect actual endpoints and visible behavior. Freeze the updated plugin source
|
||||
bundle before claiming imports use changes. Audit every named plan item against
|
||||
current files, runtime results, corpus rows and storage evidence before marking
|
||||
the overall goal complete.
|
||||
|
||||
### Import DAG admission tick (deployed and verified)
|
||||
|
||||
The import decoder graph now atomically reserves its frozen configuration's CPU,
|
||||
heap and scratch before spawning a worker and persisting `Running`. Ready nodes
|
||||
that cannot currently fit stay `Pending` with unchanged attempt counts; each tick
|
||||
continues considering other ready nodes. A movable reservation enters the child
|
||||
thread and is reused by plugin execution. Temporary pressure retries without
|
||||
creating waiting worker threads, while impossible demands produce durable node
|
||||
failures. Cancelling an import cancels unadmitted nodes without launching them.
|
||||
This advances the existing shared ledger/DAG implementation; it is not evidence
|
||||
of sustained production concurrency or a replacement for the remaining catalog
|
||||
writer work.
|
||||
|
||||
### Native function context candidates (deployed and verified)
|
||||
|
||||
The existing native candidate lookup now includes `context_evidence` when an
|
||||
explicit target snapshot/program is selected. It resolves duplicate-body
|
||||
candidate ambiguity only when independently unique exact-body neighbors bracket
|
||||
both functions in the same address order, direct calls reach matching unique
|
||||
anchors, and the combined context is unique on both sides. The pass is single
|
||||
round: its own candidates never become anchors. Results are graded
|
||||
`corroborated_duplicate_candidate`, with `accepted_match: false`; they neither
|
||||
propagate names nor claim semantic equivalence. Uniqueness is explicitly scoped
|
||||
to retained native evidence, not an assertion of complete executable coverage.
|
||||
Incomplete/overlapping evidence or limits (20,000 functions and 200,000 direct
|
||||
call records per program) leave context unavailable. Programs larger than this
|
||||
bounded pass remain unresolved. Tests cover shifted addresses, repeated contexts,
|
||||
missing call support, reversed anchors, architecture mismatch, and catalog
|
||||
readback with incomplete coverage. Production corpus match-rate gains remain
|
||||
unmeasured.
|
||||
|
||||
### Archived native anchor paging and catalog batching
|
||||
|
||||
Archived-program anchor paging now resolves descendant facts snapshots and
|
||||
reports saved-row, verified, incomplete, and truncated coverage explicitly in
|
||||
the API and workbench. Four archived ELF identities and bounded 200-row cohorts
|
||||
were read back without changing names or accepted correspondences. Large code
|
||||
inventories now publish symbols and signatures through bounded catalog-writer
|
||||
transactions; the completion marker remains a final transaction. These changes
|
||||
are deployed with the same release and do not claim complete corpus coverage.
|
||||
|
||||
### Phase 7 durability metadata slice (deployed and verified)
|
||||
|
||||
The catalog now records an immutable `snapshot_recipes` row for every published snapshot. The
|
||||
recipe identifies the release, layer, parent, producing run, entry count, logical byte count and
|
||||
warnings, and classifies original snapshots as `sacred` while extracted/derived snapshots are
|
||||
`derived` and marked rebuildable. Schema 19 creates the table additively and backfills recipes for
|
||||
newly reconciled manifests without rewriting archive objects.
|
||||
|
||||
Operator pins are stored in `pinned_snapshots` and count as storage roots. The report-only storage
|
||||
root endpoint and measured catalog overview include pinned snapshot/backend-root counts. Deletion
|
||||
of a pinned snapshot is rejected until the operator explicitly unpins it; deleting a snapshot also
|
||||
removes its recipe metadata. `tests/catalog_management.rs::snapshot_recipe_and_pin_are_durable_roots`
|
||||
proves recipe publication, pin root accounting, deletion protection, unpinning and reopen behavior.
|
||||
This slice does not run recipes or perform garbage collection; recipe execution, source
|
||||
reacquisition coverage beyond the report, and destructive collection remain open Phase 7
|
||||
requirements, and no destructive collection was performed.
|
||||
|
||||
### Phase 7 decode-before-store policy (implemented and verified)
|
||||
|
||||
The `spike_package` adapter now treats decoded extraction output as the durable
|
||||
publication boundary. Temporary inner wrapper files are measured and discarded
|
||||
after decoding by default, so new imports no longer create `wrapper-payload/`
|
||||
trees. `wrapper-evidence.json` records the measured temporary bytes,
|
||||
`decode_before_store: true`, and the selected `storage_policy`; it does not
|
||||
claim that an outer package can be reacquired without its original input.
|
||||
|
||||
Forensic retention remains available only through the explicit
|
||||
`retain_wrapper_payload: true` setting and is reported as
|
||||
`decoded-plus-wrapper`. The focused Python tests prove the default has no
|
||||
wrapper-payload output and that opt-in retention copies only the bounded
|
||||
temporary files. Existing archive objects are untouched and no garbage
|
||||
collection is performed by this policy.
|
||||
|
||||
### Phase 7 audio storage audit (report-only)
|
||||
|
||||
`scripts/audit_audio_storage.py` scans retained WAV previews and groups only
|
||||
verified PCM s16le mono/stereo payloads with matching sample geometry. It
|
||||
reports duplicate WAV bytes, unique canonical PCM bytes, malformed/skipped
|
||||
files, and the fixed 44-byte header that can be synthesized at serve time.
|
||||
The audit does not write PCM or FLAC objects, delete WAV files, alter catalog
|
||||
identities, or claim that an encoder migration has run. Its report explicitly
|
||||
marks FLAC re-encoding as unperformed and uses SHA-256 only as an audit-group
|
||||
key. Three focused tests cover duplicate grouping, malformed input rejection,
|
||||
and exact canonical header construction. A future human-reviewed migration can
|
||||
use these groups to select a pinned FLAC encoder and preserve the original
|
||||
WAV/PCM evidence before any deletion.
|
||||
|
||||
### Phase 7 zstd dictionary feasibility measurement (report-only)
|
||||
|
||||
`scripts/measure_zstd_dictionaries.py` provides a bounded measurement pass for
|
||||
the planned per-asset-class dictionaries. It groups small files by extension,
|
||||
caps the selected corpus and training bytes, trains dictionaries only inside a
|
||||
temporary directory, and probes baseline versus dictionary-compressed sizes on
|
||||
a bounded subset. The report records the dictionary SHA-256, training limits,
|
||||
probe savings, insufficient-sample/tool-error states, and a truncation marker.
|
||||
No dictionary artifact is retained and no archive/catalog object is changed.
|
||||
The focused tests cover successful measurement and deterministic file-count
|
||||
truncation. This is feasibility evidence; it does not authorize a compression
|
||||
migration or claim corpus-wide savings.
|
||||
|
||||
### Phase 7 resumable reacquisition pages (report-only)
|
||||
|
||||
`verify-reacquirable` now supports bounded deterministic pages with `--limit`
|
||||
and the returned `next_cursor` (`snapshot-id<TAB>path`) passed back through
|
||||
`--after`. Entries are sorted by immutable snapshot/path keys, so a corpus
|
||||
check can resume after an interrupted hash pass without storing mutable job
|
||||
state. Each page reports processed and total entry counts, `has_more`, and its
|
||||
cursor while retaining exact BLAKE3, basename/size, ambiguity, and mismatch
|
||||
evidence. The CLI and library remain report-only with `gc_eligible: false`;
|
||||
they never delete, demote, or rewrite archive objects. Focused tests cover a
|
||||
two-page verification and cursor continuation. This is resumable verification
|
||||
plumbing, not proof that the production source corpus has been fully checked.
|
||||
|
||||
### Phase 7 recipe execution dry-run (report-only)
|
||||
|
||||
`verstack recipe-dry-run` validates bounded pages of durable
|
||||
`snapshot_recipes` against retained snapshots, parent snapshots, producing
|
||||
runs, layer/rebuildability, entry counts, logical bytes, and operator pins. It
|
||||
classifies derived recipes as `ready`, sacred originals as `preserved-only`,
|
||||
and missing or over-bound inputs as `blocked`; it returns per-check evidence
|
||||
and a truncation marker. `--limit` is capped at 256 snapshots and
|
||||
`--max-logical-bytes` bounds planned output. No plugin is invoked, no recipe is
|
||||
executed, no catalog/storage state is written, and `gc_eligible` remains false.
|
||||
Focused tests cover pin visibility, sacred classification, output-bound
|
||||
blocking, and request bounds. This is execution planning and validation, not
|
||||
reacquisition or garbage collection.
|
||||
|
||||
### Phase 7 Star Wars SPIKE2/SPIKE3 storage gate (report-only)
|
||||
|
||||
`verstack star-wars-storage` and `GET /api/catalog/star-wars-storage` select
|
||||
the Star Wars ELG 1.10.0 SPIKE2/SPIKE3 snapshots and measure their selected
|
||||
Rustic chunks, raw bytes, packed bytes, backend packed bytes, and combined
|
||||
reuse. The report keeps generations separate, records an explicit
|
||||
`canonicalization.performed: false` state, and never rewrites an object or
|
||||
catalog row. The selector excludes unrelated releases, and focused tests cover
|
||||
both generation grouping and exclusion. This is the required before gate for a
|
||||
future canonicalization rehearsal; it does not claim that an after
|
||||
canonicalization measurement or storage reduction has occurred.
|
||||
|
||||
The gate now also reports a non-mutating overlap estimate: the sum of the
|
||||
generation packed-byte totals minus the combined unique packed-byte total.
|
||||
This identifies the upper bound of cross-generation chunk overlap that a
|
||||
future canonicalization rehearsal could investigate, while retaining
|
||||
`canonicalization.performed: false` and an explicit caveat that it is not a
|
||||
projected reclaim or a completed after-measurement.
|
||||
|
||||
The report-only `verify-reacquirable` CLI command now hashes source-tree candidates by exact
|
||||
basename, size, and BLAKE3 against sacred original snapshots. It reports verified, missing,
|
||||
mismatched, and ambiguous candidates and always returns `gc_eligible: false`; the fixture test
|
||||
covers both a verified source and a changed source. It is evidence for a future human-confirmed
|
||||
GC review, not a deletion mechanism.
|
||||
|
||||
The command now accepts bounded `--limit` and resumable `--after` cursors. The catalog API
|
||||
continues to return `report_only: true` and `gc_eligible: false`; a two-page continuation test
|
||||
proves deterministic snapshot/path ordering without changing archive objects.
|
||||
|
||||
The reacquisition report now hashes every same-basename, same-size candidate before classifying
|
||||
it. A stale copy beside the current original therefore cannot mask a unique exact BLAKE3 match;
|
||||
the verified row records the number of candidates considered. A single non-matching candidate is
|
||||
still reported as `mismatched`, while multiple exact matches or multiple non-matching candidates
|
||||
remain `ambiguous` for human review. The change is report-only and keeps `gc_eligible: false`.
|
||||
`cargo test --locked --offline --test reacquisition -q` passes 3/3, including the collision fixture.
|
||||
|
||||
### Phase 4 bounded corpus admission report (deployed and verified)
|
||||
|
||||
`scripts/corpus_import_queue.py` turns the immutable corpus inventory and
|
||||
disposable validation receipts into a deterministic, resumable queue. It does
|
||||
not start imports or modify source and receipt trees. Missing receipts are
|
||||
reported as `unprocessed`; failed extraction and incomplete decoder stages
|
||||
retain their error text; a receipt whose recorded source SHA-256 or byte count
|
||||
no longer matches the inventory is `stale`. Operators can select a bounded
|
||||
batch and resume after a package name, for example:
|
||||
|
||||
```sh
|
||||
python3 scripts/corpus_import_queue.py \
|
||||
docs/corpus-inventory.json data/validation/corpus-extraction \
|
||||
/tmp/verstack-corpus-queue.json --limit 4
|
||||
```
|
||||
|
||||
The report includes status counts, explicit failure reasons, source hashes,
|
||||
receipt paths, and a `truncated` marker so a worker cannot mistake a bounded
|
||||
selection for full-corpus completion. `validate_corpus.py` now records the
|
||||
source SHA-256 and mtime in each new receipt, allowing later queue runs to
|
||||
detect source replacement. This is admission and evidence plumbing only; it
|
||||
does not claim that the 63 untouched packages have been imported.
|
||||
|
||||
The queue now reclassifies a receipt that says `complete` but contains a
|
||||
family-stage error or non-complete coverage result as `coverage_blocked`.
|
||||
This prevents a successful outer validation from admitting a package whose
|
||||
audio/media/native stage remains partial. The focused queue fixture covers
|
||||
that mismatch; the command remains report-only and does not start imports.
|
||||
|
||||
The same bounded queue now emits a report-only `workspace_admission` section.
|
||||
With `--workspace-budget` and optionally `--workspace`, it measures the
|
||||
selected page's known source-byte total and largest source against the
|
||||
configured/free effective cap, while marking unknown sizes and compressed
|
||||
expansion as caveats. Focused tests cover both a blocked page and a free-space
|
||||
cap. This is staging evidence only; format-aware importer admission and the
|
||||
63 untouched package imports remain open.
|
||||
|
||||
### Phase 5 cross-architecture bridge ingestion (deployed and verified)
|
||||
|
||||
`POST /api/code/bridges` now imports bounded ARM32/AArch64 bridge records from
|
||||
an external semantic matcher into the append-only schema-20
|
||||
`code_architecture_bridges` index. Input hashes, architecture families,
|
||||
canonical addresses, score range, and evidence shape are validated; replay is
|
||||
idempotent. `GET /api/code/bridges` exposes the records for candidate review.
|
||||
Every response is explicitly candidate-only with zero accepted matches and
|
||||
zero propagated names. The focused `native_bridges` test covers normalization,
|
||||
replay, persistence, and fail-closed validation. This is bridge ingestion and
|
||||
review plumbing, not corpus-wide semantic matching or name propagation.
|
||||
|
||||
### Phase 4 ZIP-to-LUKS/ext4 composition (bounded fixture verified)
|
||||
|
||||
`Archive::decode_luks_ext4_member` and `POST /api/decode/luks-ext4-member` now
|
||||
compose the bounded `ZipMemberReader` directly with the native LUKS range
|
||||
reader and ext4 reader. Stored ZIP members are checked for local/central
|
||||
metadata, bounds, and CRC while the encrypted range is consumed; neither the
|
||||
ZIP member nor a plaintext disk image is materialized. The native LUKS fixture
|
||||
test publishes `disk/encrypted.bin` from a stored ZIP member and verifies the
|
||||
same ext4 file output and no-materialization receipt. Compressed ZIP and full
|
||||
corpus admission remain separate requirements; SPK/Radium now has its own
|
||||
bounded reader-backed publication fixture below.
|
||||
|
||||
### Phase 5 bridge endpoint resolution (bounded candidate evidence)
|
||||
|
||||
`GET /api/code/bridges` now resolves each bridge endpoint against the
|
||||
independently indexed `code_functions` rows using the exact input SHA-256,
|
||||
language, and canonical address. Readback reports each endpoint as
|
||||
`verified_exact_function` or `unresolved_function`, includes resolved
|
||||
function IDs when present, and exposes `both_endpoints_verified`. This is
|
||||
calibration evidence only: bridge rows remain candidate-only, accepted
|
||||
matches remain zero, and no names or observations are changed. The focused
|
||||
`native_bridges` test covers both-endpoint resolution and the existing replay
|
||||
and fail-closed checks; unresolved endpoints remain visible rather than being
|
||||
silently guessed.
|
||||
|
||||
### Phase 5 call-graph candidate ranking (bounded, candidate-only)
|
||||
|
||||
Native candidate retrieval now joins the existing unique-anchor context pass
|
||||
to rank corroborated direct-call candidates ahead of address-order-only
|
||||
candidates, then shared block-hash support. Each candidate carries its
|
||||
context grade and bounded context evidence; the context report also exposes
|
||||
the seed's raw `code_references` call records with ordinal and payload. These
|
||||
records are evidence for review only: no candidate is accepted, no names are
|
||||
propagated, and unresolved targets remain unresolved. `native_code` tests
|
||||
cover the candidate-only response and context-ranked readback; the focused
|
||||
test passes 3/3 (one live integration case remains ignored).
|
||||
|
||||
### Phase 5/8 native reference-string index (bounded, candidate-only)
|
||||
|
||||
Verified native `code_references` now have a separate `code_reference_strings`
|
||||
table and FTS5 index. Only bounded identity and literal-hash values are indexed,
|
||||
with the original payload and evidence ID retained for readback. Search combines
|
||||
anchor and reference hits, preserves the 10,000-hit scan budget and scope filters,
|
||||
and labels every result `candidate_only`; cleanup removes derived rows and FTS
|
||||
entries transactionally. Four focused catalog tests pass.
|
||||
|
||||
### Phase 3 synthetic concurrency report
|
||||
|
||||
`verstack measure-concurrency` drives the same bounded `TaskGraph` transitions
|
||||
used by import admission without opening the archive, launching plugins, or
|
||||
writing state. It caps graphs, slots, and task duration and reports max active
|
||||
tasks, pending peak, elapsed time, throughput, and start samples. This provides
|
||||
reproducible admission evidence; it does not replace sustained production
|
||||
throughput measurement.
|
||||
|
||||
### Phase 4 SPK member streaming handoff (bounded and verified)
|
||||
|
||||
`verstack-spk` now exposes a bounded `MemberReader` for one indexed SPK file.
|
||||
It keeps the retained SPK source behind `Read + Seek`, streams the selected
|
||||
member without extracting it, and verifies MD5/HMAC when a complete read starts
|
||||
at offset zero and reaches EOF. The focused native SPK fixture confirms the
|
||||
payload bytes and digest checks without creating an extraction directory. This
|
||||
is the source handoff needed for a future SPK-to-Radium decoder composition;
|
||||
the Radium publisher still stages its final decoded assets and the full chain
|
||||
is not claimed complete.
|
||||
|
||||
### Phase 2 decoder registry metadata (bounded and verified)
|
||||
|
||||
`GET /api/decode/registry` now exposes the deterministic native decoder order
|
||||
and priorities used by `decode::inspect`. The report is read-only metadata and
|
||||
contains no input-derived claims or executable paths. A focused registry test
|
||||
checks stable priority ordering and the Radium, SPK, ZIP, and scene boundaries;
|
||||
individual parsers retain their existing fail-closed limits.
|
||||
|
||||
Scene graph evidence now exposes explicit `timing_semantics`: serialized frame
|
||||
order and declared frame rate are retained, while runtime playback timing is
|
||||
marked unverified. Sprite sound-event relationships retain frame ordinal and
|
||||
sync flags with `timing_verified: false`, preventing structural scene parsing
|
||||
from implying runtime audio synchronization. The bounded scene fixture covers
|
||||
the timing caveat alongside the existing object/reference checks.
|
||||
|
||||
### Phase 8 emulator self-consistency canary (bounded)
|
||||
|
||||
`scripts/emulator_self_consistency.py` compares two bounded JSONL capture
|
||||
traces containing frame and audio sequence hashes. It validates event kinds,
|
||||
non-negative sequence numbers, finite non-negative virtual times, lowercase
|
||||
64-hex digests, and a 100,000-event cap, then reports the first divergent event
|
||||
without launching an emulator or opening a live gameplay session. The fixture
|
||||
tests identical frame/audio traces, hash and virtual-time divergence, and
|
||||
missing timing. When `--first-inputs` and
|
||||
`--second-inputs` are supplied, it also requires matching upstream commit,
|
||||
filtered export commit, profile, and source digest from each canary's
|
||||
`inputs.json`; mismatched source provenance is refused before hashes are
|
||||
compared. This is the same-seed canary gate needed before trusting
|
||||
cross-version behavioral diffs; it does not claim live capture or gameplay
|
||||
equivalence.
|
||||
|
||||
### Phase 8 SPIKE 2 bridge research boundary
|
||||
|
||||
`scripts/audit_spike2_bridge.py` now provides a reproducible, non-executing
|
||||
audit for the three archived `netbridge-0_5_0.hex` copies. All three contain
|
||||
392 valid Intel HEX data records plus one EOF record, cover the contiguous
|
||||
6,268-byte address range, and have SHA-256
|
||||
`2f2cc96a66fb61cd912cbd8cc18d92327b3e26362837c3e3b18edd402448d654`.
|
||||
The audit explicitly reports `protocol_adapter_ready=false`: identity and
|
||||
integrity are proven, but command semantics still require ARM disassembly or
|
||||
captured SPIKE 2 traffic. It never executes the firmware or invents a bridge
|
||||
protocol. Two fixture tests cover checksum rejection and identical-copy
|
||||
accounting.
|
||||
|
||||
`scripts/verify_audio_compatibility.py` now provides the bounded native/legacy
|
||||
audio default decision. It joins evidence by source, section, ordinal, and
|
||||
channel geometry, requires matching raw PCM payload identities and verified
|
||||
sample rates, and reports mismatches without changing catalog identities. The
|
||||
focused tests prove identical evidence is default-safe while changed or
|
||||
unrated payloads remain opt-in.
|
||||
|
||||
`Archive::decode_spk_radium` now uses that handoff for a bounded publication
|
||||
entry point: it parses a retained SPK index, opens one member reader, and feeds
|
||||
it directly to the Radium decoder. Run provenance records the SPK path,
|
||||
package/file coordinates, member name, retained source, and
|
||||
`member_materialized: false`. The native SPK test wraps the real Radium fixture
|
||||
in a synthetic verified-layout SPK and confirms the member reader reaches the
|
||||
Radium decoder; this remains a bounded fixture gate, not corpus completion.
|
||||
|
||||
### Follow-up bounded evidence after release 6
|
||||
|
||||
- The live FTS-backed asset-search smoke measured 5.7–32.3 ms for nine requests across `font`, `pokemon`, and `message`; this closes the previously missing interactive latency measurement for warm endpoint reads while leaving cold-cache and sustained-concurrency measurement open. See `docs/search-latency-verification.md`.
|
||||
- Release 6 adds exact endpoint calibration to candidate-only architecture bridges. Each bridge row now reports endpoint function IDs and verification status only when input hash, canonical language, and address all match; live readback remains empty and candidate-only.
|
||||
- Release 6 adds a verified bounded SPK member reader; release 7 wires it into a reader-backed Radium publication fixture. Full-corpus chain coverage remains open.
|
||||
- Phase 7 now has a report-only canonical audio storage audit that groups duplicate PCM payloads, validates WAV geometry, and specifies serve-time headers. FLAC encoding and any deletion remain unperformed.
|
||||
- Report-only Phase 7 measurements are executable: the canonical audio audit ran on the bitmap-font validation tree (0 WAVs, no writes), and the bounded zstd feasibility probe measured 4,286 bytes JSON savings and 18,759 bytes PNG savings over a 200-file sample. Temporary dictionaries were discarded; no archive/catalog mutation occurred.
|
||||
- Release 7 deployed the call-graph candidate ranking and reader-backed SPK/Radium handoff. The live 74-check gate passed after restart; the deployed backend remains candidate-only and retains all source wrappers.
|
||||
|
||||
The native HTTP integration gate now exercises `/api/code/calibration/groups`
|
||||
after normalizing two identical fixture programs. It verifies a nonempty
|
||||
language/mode group report, `candidate_only:true`, zero accepted matches, and
|
||||
that every emitted pair has `comparison_allowed:false`. This is a runtime
|
||||
smoke of the refusal boundary; it does not claim corpus-wide live coverage.
|
||||
|
||||
Native normalization responses also return the ELF
|
||||
`inventory_boundary_calibration` object alongside coverage and diagnostics.
|
||||
The integration fixture checks that Thumb/stripped calibration metrics are
|
||||
present on persisted normalization output and remain candidate-only; the
|
||||
focused native-code gate passes 3/3 (one ignored live integration case).
|
||||
|
||||
### Phase 5 normalization comparison by language/mode (report-only)
|
||||
|
||||
`GET /api/code/calibration/groups?limit=...` now compares persisted native
|
||||
evidence distributions by language and decoded instruction mode. It reports
|
||||
declared/decoded/reachable bytes, complete decode/CFG counts, reference
|
||||
counts, and bounded pairwise group metadata. Pair rows explicitly set
|
||||
`comparison_allowed:false`; no cross-architecture or cross-mode function
|
||||
correspondence is accepted and no names are propagated. The native
|
||||
calibration fixture verifies two language/mode groups and the refusal to
|
||||
accept comparisons; it passes 1/1.
|
||||
|
||||
### Phase 5 corpus-wide native calibration (bounded)
|
||||
|
||||
`GET /api/code/calibration/corpus?limit=...` aggregates the persisted
|
||||
per-program calibration reports across a bounded program selection. It keeps
|
||||
the per-program reports, aggregates ARM32/AArch64 modes, complete decode/CFG
|
||||
counts, reference kinds, identity/hash availability, and reports explicit
|
||||
function and selection truncation. It remains candidate-only with zero
|
||||
accepted matches and zero propagated names. The native calibration fixture
|
||||
verifies the corpus aggregation and passes 1/1; this report does not claim
|
||||
complete corpus normalization or semantic matching.
|
||||
|
||||
### Phase 5 corpus native calibration (bounded report-only evidence)
|
||||
|
||||
`GET /api/code/calibration?snapshot=...&path=...` now summarizes persisted
|
||||
native evidence for one explicitly selected program: ARM32/AArch64 mode
|
||||
counts, complete decode/CFG counts, reference totals, and bounded reference
|
||||
kinds with identity and literal-hash counts. Function and reference limits
|
||||
are explicit in the report, and it returns zero accepted matches and zero
|
||||
propagated names. The focused `native_calibration` fixture verifies scoped
|
||||
aggregation and candidate-only semantics; it passes 1/1.
|
||||
|
||||
### Phase 5/8 native reference-string index expansion (bounded candidate evidence)
|
||||
|
||||
Native normalization now maintains a separate `code_reference_strings` FTS5
|
||||
index for bounded `identity` and `literal_sha256` values already present in
|
||||
verified `code_references`. The code-string search endpoint reads anchor and
|
||||
reference hits with explicit `source` values, provenance, and
|
||||
`candidate_only:true`; it keeps the existing 10,000-hit scan budget and does
|
||||
not promote reference identities to names or function correspondence. The
|
||||
focused code-anchor tests cover indexing, scoped search, replay, and cleanup;
|
||||
the library slice passes 4 tests (2 real-corpus cases remain ignored).
|
||||
The projection now replays pre-existing `code_references` rows during additive
|
||||
schema opening, so catalogs created before the FTS table do not silently lose
|
||||
searchable identities or literal hashes. `GET /api/code/strings/coverage` provides
|
||||
durable evidence and bounded per-program denominators. Replay and coverage have
|
||||
focused tests; this remains report-only evidence and does not establish complete
|
||||
executable or semantic corpus coverage.
|
||||
- Release 8 deployed the deterministic decoder registry, resumable report-only reacquisition paging, and Thumb/stripped boundary calibration. The live decoder registry readback contains `godot-sidecar`, `scene-radium`, native SPK/Radium, ZIP, and probe entries; the live 74-check gate passed after restart.
|
||||
- Release 9 deployed the bounded native reference-string FTS index and the report-only synthetic concurrency measurement command. The live 74-check gate passed after restart; the concurrency run completed 32 synthetic tasks at three admitted slots without opening the archive or launching plugins.
|
||||
- Release 10 deployed bounded native calibration readback, report-only recipe dry-run validation, and the emulator frame/audio self-consistency harness. The live 74-check gate passed after restart; no recipe execution, GC, or live gameplay was performed.
|
||||
- Release 11 deployed corpus-native calibration aggregation and report-only Star Wars storage accounting. A bounded live calibration request returned two programs with 527 functions scanned, 484 complete CFGs, and zero accepted matches or propagated names; storage accounting remains report-only.
|
||||
- Release 12 deployed explicit scene timing caveats and native/legacy audio compatibility eligibility. The live 74-check gate passed after restart; runtime playback timing and unrated audio remain explicitly unverified rather than inferred.
|
||||
|
||||
Scene publication responses now read back the bounded timing semantics and
|
||||
count sound-event edges. The response explicitly distinguishes serialized
|
||||
frame order from runtime playback timing, so API consumers can verify the
|
||||
caveat without inspecting the full graph or inferring synchronization.
|
||||
|
||||
### Phase 5 Thumb and stripped-boundary calibration (candidate evidence)
|
||||
|
||||
The ELF code-signature report now includes bounded `boundary_calibration`
|
||||
counts for recovered Thumb and ARM functions, unknown ARM32 mode functions,
|
||||
zero-sized symbol extent hints, stripped entrypoint boundary hints, and
|
||||
uncovered executable bytes. The counts are explicitly candidate-only and do
|
||||
not turn an entrypoint or a zero-sized symbol into a guessed function extent.
|
||||
The `codesig` fixture suite verifies Thumb-bit address normalization and
|
||||
stripped ELF reporting (8 passing tests, one ignored live corpus case).
|
||||
|
||||
Audio compatibility eligibility now rejects rates outside the verified PCM
|
||||
profile set, even when native and legacy raw payload hashes match. Unsupported
|
||||
rate evidence stays opt-in and is reported as `missing_verified_rate`.
|
||||
|
||||
The compatibility command is now an exact bounded default gate: it refuses a
|
||||
native default when either side has an unmatched identity, when verified rates
|
||||
differ, or when raw PCM hashes differ. It emits `default_eligible`, the
|
||||
`default`/`opt-in` decision, bounded mismatch keys, and `report_only:true`;
|
||||
it never changes decoder selection or catalog rows. This makes a legacy
|
||||
subset appear as opt-in rather than silently treating overlap as full
|
||||
compatibility.
|
||||
|
||||
### Phase 7 canonical audio migration planning (report-only)
|
||||
|
||||
The audio storage audit now emits an explicit migration plan. It checks that
|
||||
every verified WAV source remains a regular retained file, reports skipped or
|
||||
malformed sources, and records callable `flac --version` availability and
|
||||
version text. The plan is marked ready only when those checks pass; it still
|
||||
performs no FLAC encoding, writes no PCM/FLAC objects, and requires human
|
||||
approval plus a pinned encoder recipe before migration.
|
||||
|
||||
`scripts/rehearse_flac_migration.py` now exercises that recipe without touching
|
||||
the archive. A bounded real sample of 16 SPIKE 2 WAV files (6,350,526 source
|
||||
bytes) encoded with FFmpeg/libFLAC at compression level 8, one thread, and
|
||||
decoded back to s16le with exact PCM SHA-256 equality. The temporary FLAC
|
||||
outputs totaled 3,586,928 bytes and were discarded. The retained report is
|
||||
`data/validation/audio/flac-rehearsal-spike2-16.json` (SHA-256
|
||||
`57d01d24341e2d4c2dfc1b68a67921247eecead1eafd16abd12533828b3edd0c`). This is
|
||||
encoder/round-trip evidence only; no catalog migration or source deletion was
|
||||
performed.
|
||||
|
||||
- Release 14 deployed the persisted Thumb/stripped boundary calibration evidence, report-only WAV/PCM-to-FLAC migration readiness audit, and scene timing/sound-edge readback. Services are active and the running backend hash matches the release artifact. Runtime playback timing, FLAC migration, and the remaining corpus and production-scale phases remain open.
|
||||
|
||||
- Release 15 deployed the user-facing binary similarity score (`0.994 (99.4%)`) and report-only video/audio association map with explicit evidence and verification flags. The workbench bundle was rebuilt and all three live services are active. Runtime media probing, direct clip binding, FLAC migration, and broader corpus/production phases remain open.
|
||||
|
||||
- Release 17 deployed the schema-open replay fix for the native reference-string FTS projection. Live coverage now reads 527 evidence rows, 1,355 references, and 521 indexed identity/literal rows across the saved corpus; the report remains candidate-only and does not claim semantic matching.
|
||||
|
||||
- Release 17 deployed the schema-open replay fix for the native reference-string FTS projection. Live coverage now reads 527 evidence rows, 1,355 references, and 521 indexed identity/literal rows across the saved corpus; the report remains candidate-only and does not claim semantic matching.
|
||||
|
||||
### Phase 3 live governor telemetry (read-only, bounded)
|
||||
|
||||
`python3 scripts/verify_live_governor.py` now samples the deployed
|
||||
`GET /api/governor` endpoint for a bounded number of readings (default: five,
|
||||
one second apart). It reports host snapshots, admitted analysis/preparation
|
||||
counts, peak observed active tasks, probe errors, and explicit
|
||||
`writes_performed=false` / `work_submitted=false` evidence. The script submits
|
||||
no task and mutates no catalog or archive state, so it verifies the production
|
||||
admission telemetry surface without pretending to measure sustained workload
|
||||
throughput. Unit tests cover successful sampling, timing, and failed probes.
|
||||
|
||||
Live evidence captured in `docs/phase3-live-governor-20260917.json`: five
|
||||
successful one-second samples from the deployed backend, zero active tasks in
|
||||
every sample, zero reserved CPU/memory/scratch bytes, and no probe errors. The
|
||||
host reported 54 CPUs, approximately 175--177 GiB available memory, and
|
||||
approximately 92.04 GB free scratch throughout. The report records
|
||||
`work_submitted=false` and `writes_performed=false`.
|
||||
|
||||
- Release 18 deployed conservative scene object-class semantic typing and a report-only live governor evidence probe. Five live samples succeeded with zero active work and no writes; sustained production throughput remains open.
|
||||
|
||||
### Phase 5 bounded propagation frontier (candidate-only)
|
||||
|
||||
The existing call-graph candidate ranker can now persist its bounded frontier
|
||||
in `code_propagation_candidates` and read it back through
|
||||
`GET /api/code/propagation-candidates?seed=...`. `POST /api/code/propagation-candidates`
|
||||
materializes only the selected candidate rows, retaining shared-block support,
|
||||
context priority, hop count, and the ranking provenance. Rows are immutable,
|
||||
candidate-only, and carry zero accepted matches and zero propagated names;
|
||||
they do not mutate `asset_names`, observations, or function identity. The
|
||||
schema migration is additive and snapshot cleanup removes dependent frontier
|
||||
rows before evidence deletion. The focused persistence fixture passes 1/1.
|
||||
This is durable review evidence for a future fixed-point propagator, not the
|
||||
corpus-wide Phase 5 propagation gate.
|
||||
|
||||
- Release 19 deployed schema-21 candidate propagation persistence and bounded readback. Candidates retain provenance and hop/context evidence while accepted matches and name propagation remain zero by policy; the live endpoint returned the expected candidate-only empty readback for an unknown seed.
|
||||
|
||||
### Phase 4 bounded unwrap handoff
|
||||
|
||||
`ext4::with_file` now exposes one bounded regular-file view to nested decoders.
|
||||
The ext4 parser continues to read blocks from the retained `Read+Seek` source,
|
||||
while the consumer receives a seekable file handle with explicit file-size and
|
||||
filesystem read budgets. It does not create a plaintext image, extracted file,
|
||||
or intermediate byte buffer. A fixture-backed test creates an ext4 image,
|
||||
writes a 128 KiB payload, and consumes it through this handoff; `cargo test
|
||||
--locked --offline --lib ext4::tests::bounded_file_consumer_streams_nested_payload_without_staging`
|
||||
passes. This closes the ext4-to-nested-reader boundary as a verified bounded
|
||||
slice; full ZIP/LUKS/squashfs/SPK/Radium corpus coverage and bulk import remain
|
||||
open.
|
||||
|
||||
- Release 20 deployed the bounded seekable ext4 nested-reader view. A fixture streamed a 128 KiB payload without staging a plaintext image or intermediate file; full ZIP/LUKS/squashfs/SPK/Radium chain coverage and the 63-package campaign remain open.
|
||||
|
||||
- Release 21 deployed the composed bounded ZIP→LUKS2→ext4→SPK→Radium reader route. Live dispatch is present and fail-closed on malformed input; full-corpus chain coverage, squashfs handling, and the 63-package campaign remain open.
|
||||
|
||||
### Phase 7 qcow2 export audit (report-only)
|
||||
|
||||
`scripts/audit_qcow2_exports.py` classifies bounded VM-export scans as raw standalone images, standalone qcow2 files, or resolved qcow2 backing chains. It refuses symlinks, outside-root parents, cycles, malformed headers, and depth overrun, and performs no mounts, rewrites, flattening, deletion, or GC. A complete read-only scan of `data/emulator` covered 42,575 files with no truncation and found only `raw_standalone` images. The retained report is `data/validation/qcow2/emulator-all.json` (SHA-256 `ba4783eaddeb1fe1f749e794ad01eebf50eb80c69580afa041227e03b300ee8b`). Fixture tests cover chain resolution and mutation boundaries.
|
||||
|
||||
### Phase 7 zstd dictionary measurement (report-only)
|
||||
|
||||
`scripts/measure_zstd_dictionaries.py` was run against the retained emulator
|
||||
runtime corpus with a 500-file bound. It considered 33,248 eligible files and
|
||||
selected 500 across four extension classes; the report is retained at
|
||||
`data/validation/zstd/emulator-runtime-max500.json` (SHA-256
|
||||
`8fd728809d6a2b9730991b5170118fecf081d9ea6fb77c01762d0e9cbc9f0477`). The
|
||||
largest measured classes reported 12,220 and 18,211 probe-savings bytes. Two
|
||||
small classes returned explicit zstd tool errors rather than being treated as
|
||||
successful measurements. Temporary dictionaries were discarded; no archive or
|
||||
catalog object was written. The report pins Zstandard CLI v1.5.7, single-thread
|
||||
execution, and a recipe hash for repeatability. This is measurement evidence,
|
||||
not authorization to train or deploy dictionaries.
|
||||
|
||||
### Phase 2 audio default eligibility gate
|
||||
|
||||
`verify_audio_compatibility.py` requires complete native coverage, matching source/section/ordinal and channel geometry, identical raw PCM identity, and matching verified sample rates before reporting `default_eligible=true`. Partial, unrated, unsupported-rate, or changed-payload records remain opt-in and report-only. The focused suite passes 5/5; no decoder default was changed.
|
||||
|
||||
- Live verification gate was rerun against release 21 with local network access: **73/73 checks passed, 0 failed, 0 skipped**. This proves the deployed baseline and current review/comparison behavior; it does not close the explicitly open corpus-scale PLAN requirements.
|
||||
## Release 22 compatibility fix — deployed 2026-09-17
|
||||
|
||||
- Release `release22-20260917` deployed with backend SHA-256 `e0faeb3a012183d63b0f04af966acca501697007e18f99805d535a28681ae079`.
|
||||
- The schema-21 propagation cleanup now checks for the additive table before deleting dependent rows, preserving rollback and rehearsal compatibility with older catalogs.
|
||||
- The full Rust library gate passed: **105 passed, 0 failed, 4 ignored**; Clippy with `-D warnings` also passed.
|
||||
- The approved live gate completed **74/74 checks passed, 0 failed, 0 skipped**. Backend, workbench, and gateway are active and the decoder registry responds.
|
||||
- Rollback artifacts include the pre-change binary/config and a hard-linked catalog backup shared with release 21; no catalog data was removed.
|
||||
|
||||
### Phase 8 caption batch preparation (local, report-only)
|
||||
|
||||
`scripts/prepare_caption_batch.py` prepares deterministic JSONL request records
|
||||
for a later reviewed captioning submission. Each record hashes the exact local
|
||||
image, carries its archive/source reference and taxonomy-prompt hash, and uses a
|
||||
content-addressed `custom_id`. The command refuses remote URLs, symlinks,
|
||||
unsupported media types, and oversized inputs. It embeds no image bytes and
|
||||
makes no network call or credential lookup; `submission` remains
|
||||
`pending_external_review`. `tests/test_prepare_caption_batch.py` covers
|
||||
provenance and fail-closed source handling (2 tests). This is preparation and
|
||||
cost-control evidence, not completed captions.
|
||||
|
||||
### Phase 2 scene-name propagation (bounded, candidate-safe)
|
||||
|
||||
The native scene catalog now follows a unique serialized `Element` binding to
|
||||
its instantiated media object and persists the exact name as an accepted
|
||||
`T3_propagated` claim with decoder, snapshot, path, artifact, and binding
|
||||
evidence. Ambiguous/conflicting bindings remain unnamed, and the media object
|
||||
keeps its `ContentOnly` identity. `cargo test --locked --offline --test scene
|
||||
--test decode` passed (20 passed, 1 ignored). This improves non-Pokémon Radium
|
||||
ordinal naming without treating ordinal position as cross-version identity.
|
||||
## Release 23 scene-name propagation — deployed 2026-09-17
|
||||
|
||||
- Release `release23-20260917` deployed with backend SHA-256 `caad98b823ecefc9bb9b7558c0d078da9368ad4099018567ed55e7f78fb2fba1`.
|
||||
- Native scene indexing now propagates an exact serialized Element name to a uniquely instantiated media object as an evidence-backed `T3_propagated` claim; ambiguous bindings remain unnamed.
|
||||
- Focused scene/decoder tests passed (20 passed, 1 ignored), Clippy passed, and the post-deploy live gate completed **73/73 checks passed, 0 failed, 0 skipped**.
|
||||
- Backend, workbench, and gateway are active; rollback binary/config and the hard-linked catalog backup are retained.
|
||||
|
||||
### Phase 4 Jaws extraction revalidation (2026-09-17)
|
||||
|
||||
A disposable validation of `jaws_pro-1_02_0.spk.zip` reproduced the nested
|
||||
SPIKE 3 LUKS boundary: 7,399,891,406 source bytes, 1,195 extracted files,
|
||||
3,767 decoded media assets, and 226 media failures. The wrapper inventory was
|
||||
complete and no wrapper payload was retained. This is direct blocker evidence,
|
||||
not an import completion claim; full corpus admission remains open.
|
||||
|
||||
### Verification refresh — 2026-09-17
|
||||
|
||||
After release 23, the complete host-permissioned Rust gate passed: **all test
|
||||
binaries green, 105 library tests passed, 4 ignored**, including the expensive
|
||||
native LUKS fixture. The gate also caught and corrected two stale test fixtures:
|
||||
the perceptual migration now tracks schema 21, and the workspace HTTP fixture
|
||||
asserts the two responses it actually creates. Clippy with warnings denied and
|
||||
the full decoder Python suite (171 tests, 3 skips) remains green.
|
||||
|
||||
### Phase 7 canonical PCM playback wrapper
|
||||
|
||||
`GET /api/media/pcm-wav` now streams an explicitly typed `pcm_s16le` artifact
|
||||
with a canonical WAV header. Rate, channel geometry, RIFF size, and frame
|
||||
alignment are bounded before reading; response headers preserve source
|
||||
artifact and geometry provenance. The endpoint does not change identity or
|
||||
decoder-default policy. The video/audio integration suite passes 2/2.
|
||||
## Release 24 PCM-WAV synthesis — deployed 2026-09-17
|
||||
|
||||
- Release `release24-20260917` deployed with backend SHA-256 `950c9f6ff37844dfb5398161ba1b187065bbdeedbe0715206d4aa59411672886`.
|
||||
- `GET /api/media/pcm-wav` now streams explicitly typed `pcm_s16le` artifacts with a bounded canonical WAV header and source/geometry provenance.
|
||||
- The video/audio integration suite passed 2/2, Clippy passed, and the live gate completed **73/73 checks passed, 0 failed, 0 skipped**.
|
||||
- All services are active; rollback binary/config and the hard-linked catalog backup are retained.
|
||||
|
||||
## Reacquisition walk hardening — deployed 2026-09-17
|
||||
|
||||
The report-only original reacquisition verifier now skips permission-denied
|
||||
source entries and records their paths under `inaccessible`, instead of
|
||||
discarding the entire report when a mixed-permission source root contains a
|
||||
protected subtree. It still hashes every readable basename/size candidate and
|
||||
always returns `gc_eligible=false`. The focused reacquisition suite passed 3/3,
|
||||
the optimized build passed, and the post-restart live gate completed **73/73
|
||||
checks passed, 0 failed, 0 skipped**. The deployed backend SHA-256 is
|
||||
`4d12c624ea435b08c73e172507840a925a0b02d33e7e4b7d72604554aea42d22`.
|
||||
|
||||
The deployed verifier was then run against the actual `/srv/firmware` root
|
||||
using a disposable catalog copy and a bounded page. It processed all 17 sacred
|
||||
original records: **11 exact matches, 6 missing, 0 mismatched, 0 ambiguous**,
|
||||
with 587 protected source paths recorded as `inaccessible`; `gc_eligible`
|
||||
remained false. The report is retained at `/tmp/verstack-reacq/report-root.json`
|
||||
(SHA-256 `c7623a7234984e2a1a83a2657cd4e059418e4e2db8a0be35171394619a77713c`).
|
||||
This is evidence for later human review, not authorization to collect anything.
|
||||
|
||||
### Phase 2 legacy Radium Text layout
|
||||
|
||||
The native scene parser now retries the archived Batman '66 0.65 Text layout,
|
||||
which omits the two fields introduced by the modern format. Both layouts still
|
||||
require complete EOF consumption, and absent legacy fields are represented as
|
||||
JSON `null` rather than guessed values. The focused scene/decode suite passes
|
||||
21 tests with one explicit ignore. The parser and reacquisition hardening are
|
||||
deployed in the backend whose SHA-256 is
|
||||
`cbcdfcabc9587ddb540f546b44a404e9c6c30c74efac5dfd3921279986e34277`; the
|
||||
post-restart live gate passed **74/74 checks**.
|
||||
|
||||
### Phase 4 gzip expansion admission
|
||||
|
||||
Gzip-wrapped SPIKE packages now reserve the configured workspace ceiling during
|
||||
extraction because gzip has no trusted expanded-size index; the previous
|
||||
four-times estimate under-admitted large images. The pure budget regression and
|
||||
the 11-test import suite pass. A controlled 292 MB `WN-1_55_0.spk` retry
|
||||
confirmed the admission fix but then entered a runaway decoder path, reading
|
||||
about 69 GB without leaving `Unpacking and detecting formats`; it was cancelled
|
||||
and the retained original snapshot remains intact. No output snapshot was
|
||||
published. This is explicit blocker evidence for the still-open gzip/native
|
||||
handoff, not an import completion claim. The recovery restart passed the live
|
||||
gate (74/74); deployed backend SHA-256 is
|
||||
`05d47671db23b80102490fcc9d3465042a13a768ab0484f809854828bbbe369b`.
|
||||
|
||||
### Phase 4 exact-half workspace admission fix
|
||||
|
||||
Import staging and plugin materialization now use the same inclusive half-budget
|
||||
boundary. Previously an input exactly half of its reservation passed the staging
|
||||
check but was rejected when the decoder materialized it. The regression uses a
|
||||
32 MiB input with a 64 MiB reservation and verifies that the plugin receives the
|
||||
remaining 32 MiB. The full import integration suite passes **11/11** tests.
|
||||
The fix is deployed in backend SHA-256
|
||||
`a20919a6b3e48a67f6acb058f07f8572813a6821d790e41365221bc1268e212e`; the
|
||||
post-restart live gate passed **74/74 checks**.
|
||||
|
||||
### Real streaming-chain gates
|
||||
|
||||
The retained 24 MiB Pokémon system SPK publication gate passed: 18 files were
|
||||
read through the archive reader, every recorded MD5 payload matched, the source
|
||||
remained readable as `SPKS`, and the temporary workspace was empty afterward.
|
||||
The real Star Wars split-package metadata gate also passed without payload
|
||||
extraction: SPIKE2 expanded 3,088,605,184 bytes with a bounded
|
||||
15,509,920,717-byte budget, and SPIKE3 expanded 3,984,588,800 bytes with a
|
||||
19,575,014,866-byte budget. These gates validate retained-wrapper streaming
|
||||
boundaries; the full 63-package import campaign remains open.
|
||||
|
||||
Binary function comparison is available at `/api/code/similarity`. It reports
|
||||
normalized instruction, literal-instruction, and block scores; the regression
|
||||
covers a 166/167 instruction match (about 0.994). Public ratio output is
|
||||
defensively clamped to the documented 0..1 range.
|
||||
|
||||
The canceled WN path was also reproduced in a disposable workspace with the
|
||||
pinned native SPK helper and full nested handoff: the 291 MB payload completed
|
||||
in about 13 seconds and produced 28 verified files. This narrows the live-only
|
||||
runaway to service task execution or surrounding reservation state; the source
|
||||
snapshot was not retried after recovery.
|
||||
+1
-1
@@ -81,7 +81,7 @@ Run `python3 scripts/install_local_tools.py` from the project root to install th
|
||||
}
|
||||
```
|
||||
|
||||
An optional `paths` array selects logical executable paths. Otherwise ELF/PE magic selects candidates and Ghidra chooses the loader/processor. Generation is not used to guess architecture. Each unique selected binary produces `<sha256>/program.gzf`, `functions.json`, `reopened.json`, `analysis.log`, and `reopen.log`. Download the `.gzf` through the UI for local review.
|
||||
An optional `paths` array selects logical executable paths. Otherwise ELF/PE magic selects candidates and Ghidra chooses the loader/processor. Generation is not used to guess architecture. Each unique selected binary archives `<sha256>/functions.json`, decompiled `code/` files, `reopened.json`, `analysis.log`, and `reopen.log`. The adapter still generates and reopens `program.gzf`, then saves any configured recovery checkpoint and moves the nondeterministic database into disposable job scratch before archiving. The export UI downloads verified function facts for new runs and still offers existing archived GZF files for older runs. Activate adapter changes by rebuilding the pinned tool bundle with `scripts/configure_pipeline.py`; editing the working copy alone does not change the running pipeline.
|
||||
|
||||
Java user/cache/temp directories stay in the disposable workspace. External or uninitialized function bodies receive no exact-body hash and cannot establish an exact match.
|
||||
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# Durable program analysis
|
||||
|
||||
Ghidra selections submitted through `/api/jobs` or release import workflows now
|
||||
create one durable leaf job per validated ELF/PE path. Each leaf has a frozen
|
||||
one-path configuration, original input snapshot, distinct execution/run, output,
|
||||
state, cancellation flag and retry history through its saved runs. Programs sharing
|
||||
an input can run concurrently when resource admission permits it.
|
||||
|
||||
A multi-program submission returns a coordinator with `children` and `outputs`.
|
||||
Its `output` and `run` remain null: no final child represents the whole group.
|
||||
Import tasks expose `analysis_job` plus an `analysis_outputs` map keyed by source
|
||||
path. A one-program import retains its existing `outputs.ghidra` field. Coordinators
|
||||
consume no worker reservation. The queue claims and executes individual leaves;
|
||||
there is no snapshot-wide Ghidra invocation hidden behind a loop.
|
||||
|
||||
`POST /api/jobs/{id}/control` with `{"action":"cancel"}` or `{"action":"retry"}`
|
||||
controls either a leaf or all unfinished children of a coordinator. Completed
|
||||
siblings remain available. Retrying an import-owned failed leaf explicitly resumes
|
||||
its owner; automatic jobs remain paused when automatic processing is disabled.
|
||||
The Jobs view shows program paths, saved-output counts, per-program logs and controls.
|
||||
|
||||
Import preparation dispatches analysis only after its prerequisites are saved.
|
||||
Its analysis workers use the same durable leaf queue and frozen-config resource
|
||||
admission as ordinary jobs. Explicit import leaves are eligible only while their
|
||||
owning import is active. Pending selections normalize to leaves on archive reopen;
|
||||
interrupted work still requires explicit retry. Successful program outputs replace
|
||||
only their matching program selection. Failed or cancelled replacements do not
|
||||
become active code exports. Source-parent provenance and archived GZF downloads
|
||||
remain intact.
|
||||
|
||||
The synchronous `Archive::process` and `/api/process` Snapshot contract accepts
|
||||
exactly one Ghidra program. Multi-program direct calls fail before creating a run,
|
||||
with guidance to submit durable jobs. The low-level trusted Python analyzer remains
|
||||
compatible with its existing protocol, but Rust's execution path supplies one path.
|
||||
|
||||
Tests in `tests/catalog_jobs.rs` exercise overlapping actual workers with a barrier,
|
||||
restart, independent failure/retry and cancellation, paused automatic work with
|
||||
explicit import leaves, old queued-job normalization, archived export access,
|
||||
workspace visibility and the actual control endpoint. Existing import reuse tests
|
||||
verify that matching saved program outputs are reused without rerunning Ghidra.
|
||||
No live multi-program Ghidra run has been performed for this change yet.
|
||||
@@ -0,0 +1,33 @@
|
||||
# VM export backing-chain audit
|
||||
|
||||
`scripts/audit_qcow2_exports.py` measures the current VM export shape for the
|
||||
Phase 7 qcow2 item. It walks a selected directory and classifies regular files
|
||||
as `raw_standalone`, `qcow2_standalone`, or `qcow2_backing_chain`. For qcow2
|
||||
files it reads the format header, records the virtual size and declared backing
|
||||
filename, and follows relative backing names while they remain inside the
|
||||
selected root. Missing files, symlinks, cycles, path escapes, malformed headers,
|
||||
and depth limits are reported as incomplete chains.
|
||||
|
||||
The audit is report-only. It reads a small header, metadata, and (when needed)
|
||||
the backing filename. It does not mount or open an image as a block device, and
|
||||
it never creates a delta, flattens a chain, rewrites an image, deletes an image,
|
||||
or performs garbage collection. A complete chain means that declared backing
|
||||
paths resolve under the selected root; it does not prove that the guest
|
||||
filesystem, qcow2 allocation tables, or image contents are consistent.
|
||||
|
||||
Run it against an export directory with:
|
||||
|
||||
```sh
|
||||
python3 scripts/audit_qcow2_exports.py /path/to/exports \
|
||||
--output /tmp/qcow2-export-audit.json
|
||||
```
|
||||
|
||||
The default scan is bounded at 20,000 files and 32 backing levels. These can be
|
||||
lowered or raised with `--max-files` and `--max-depth`. The command does not
|
||||
hash complete image payloads, so it is safe to run against large sparse raw
|
||||
exports. Hash verification remains a separate prerequisite before any future
|
||||
human-approved migration or collection.
|
||||
|
||||
Fixture coverage in `tests/test_audit_qcow2_exports.py` verifies raw versus
|
||||
qcow2 classification, a qcow2-to-raw chain, missing and outside-root backing
|
||||
paths, and the report-only/no-image-mutation boundary.
|
||||
@@ -0,0 +1,429 @@
|
||||
# Radium asset identity — specification and measured stability
|
||||
|
||||
Read-only investigation, 2026-09-16. All numbers below were measured against the real archive
|
||||
(`data/archive/catalog.sqlite3`, accessed read-only, plus `http://127.0.0.1:8080/api/file/...`
|
||||
against real Pokémon LE/Pro `image.bin` and `scene.radium` bytes). Every measurement command is
|
||||
given so it can be re-run. Where I could not measure something, I say so explicitly rather than
|
||||
asserting it.
|
||||
|
||||
Corpus used: Pokémon LE/Pro releases 0.81.0, 0.82.0, 0.83.0, 0.85.0, 0.86.0. Snapshot IDs:
|
||||
`830d0450` (0.81.0, extracted), `2a2a4737` (0.82.0, extracted), `1314f21f` (0.83.0, extracted),
|
||||
`a4dc88d1` (0.82.0, `media-extract` derived), `4f4aa552` (0.83.0, `media-extract` derived),
|
||||
`6b7f69ef` (0.85.0, `media-extract` derived), `101b56cd`/`90d7833a` (0.86.0, `media-extract`
|
||||
derived / extracted), `2a5c3f6e` (0.81.0, `media-extract` derived).
|
||||
|
||||
Relevant source: `plugins/sound_names.py`, `plugins/media_extract.py`, `plugins/dmd_bitmap.py`,
|
||||
`plugins/scene_media.py`, `plugins/pcm_profile.py`, `src/identity.rs`,
|
||||
`~/pokemon-triage-wiki/radium-image-format.md`, `PLAN.md`.
|
||||
|
||||
---
|
||||
|
||||
## 0. What already exists in this codebase
|
||||
|
||||
`src/identity.rs` already defines the target type, `LogicalAssetKey`, with variants
|
||||
`Radium{role,section,record_id}`, `RadiumSound{masked_key}`, `GodotAsset`, `SceneInstance{scene,
|
||||
instance}`, `CodeCanon`, `File`, `ContentOnly`. Per `PLAN.md` §3, this is Phase 1, ~30% done:
|
||||
"`compare()` fixed and live; **schema not built**" — i.e. the enum and its round-trip tests exist,
|
||||
but nothing in `plugins/media_extract.py` (the actual extraction/provenance code) constructs a
|
||||
`LogicalAssetKey` yet. This document evaluates each variant's real-world stability so Phase 1 can
|
||||
decide which are worth wiring up as-is, which need a different derivation, and which need new
|
||||
decoder work before they can be computed at all.
|
||||
|
||||
`src/archive.rs` already ships a coarser, orthogonal fix (`identity::logical_path`): it strips the
|
||||
version-stamped `package-NNNN/<pkg-root>` path component structurally, which is enough to make
|
||||
`compare()` not explode on every release, but it does nothing for ordinal renumbering *inside* a
|
||||
container (a bitmap or sound inserted mid-table still renumbers everything after it even after
|
||||
`logical_path` normalization). The four keys below are the finer-grained fix.
|
||||
|
||||
---
|
||||
|
||||
## 1. Sounds (section 8 PCM table)
|
||||
|
||||
### Byte layout (confirmed, CRC-validated against the wiki's documented value)
|
||||
|
||||
`image.bin` begins with a 13×`u64le` header. Two fields matter here: `header[8]` = byte offset of
|
||||
the section-8 table, `header[12] & 0xffffffff` = row count. The table is `row_count × 24` bytes,
|
||||
CRC32-validated (aligned to 16, +4 trailing bytes for the stored CRC32). Each 24-byte row is three
|
||||
`u64le` fields:
|
||||
|
||||
```
|
||||
offset 0: q0 — PCM byte offset (absolute, into image.bin)
|
||||
offset 8: q1 — "masked lookup key" + packed channel/format bits (see below)
|
||||
offset 16: q2 — low32 = sample count
|
||||
```
|
||||
|
||||
Computed in `plugins/media_extract.py:167` (`extract_radium`) and `plugins/media_extract.py:157`
|
||||
(`radium_header`, the CRC check):
|
||||
|
||||
```python
|
||||
start, flags, samples = struct.unpack_from('<3Q', data, h[8] + i*24) # q0, q1, q2
|
||||
channels = (((flags>>25)&1)<<3)|(((flags>>30)&1)<<4)|(((flags>>17)&1)<<1)|(((flags>>31)&1)<<2)|((flags>>7)&1)
|
||||
length = channels * (samples & 0xffffffff) * 2
|
||||
```
|
||||
|
||||
I re-derived this from raw bytes and it matches the wiki exactly: for 0.83.0 the parsed
|
||||
`h[8]=0x44d5396f`, row count `2491`, and CRC32 `0x3f401e79` — identical to
|
||||
`~/pokemon-triage-wiki/radium-image-format.md`'s documented values. This cross-check gives high
|
||||
confidence the parsing here is correct.
|
||||
|
||||
### The masked key
|
||||
|
||||
`KEY_MASK = 0xFFFC0003FFFFFFFF` is defined once, in `plugins/sound_names.py:13`, and used at two
|
||||
call sites:
|
||||
|
||||
* `plugins/sound_names.py:118` — `key = struct.unpack_from('<Q', data, header[8]+index*24+8)[0] & KEY_MASK` (masks **q1** of every section-8 row, building an in-memory `key -> section8_index` map for one image).
|
||||
* `plugins/sound_names.py:134` — masks the same way on a section-5 opcode-`0x0b` command payload (`cursor+4`), to look up which section-8 row a Godot-declared sound name's command stream is pointing at.
|
||||
|
||||
This exactly matches the native evidence in the wiki (`FUN_007a46e0`): the runtime builds a lookup
|
||||
map keyed by `image_id<<0x22 | (q1 & 0xfffc0003ffffffff)`, and opcode `0x0b` command records carry
|
||||
the same masked value at their `+4` offset to resolve which PCM chunk to play. **This is a
|
||||
same-binary, same-build lookup mechanism** — it lets `sound_names.py`'s `resolve()` (line 108)
|
||||
correlate a Godot script's declared cue name to a section-8 row *within one firmware image*. It is
|
||||
invoked from `plugins/godot_scripts.py:89` (`from sound_names import enrich`), which is the
|
||||
name-recovery ladder documented in `PLAN.md`/`radio-image-format.md` — never from
|
||||
`plugins/media_extract.py`. **`media_extract.py`'s `extract_radium` does not compute or store
|
||||
`masked_key` anywhere** — I grepped for it; the only occurrences of `KEY_MASK`/`masked_key` outside
|
||||
`sound_names.py` are in `src/identity.rs`'s (unwired) `RadiumSound` variant and its tests.
|
||||
|
||||
Bit layout of the mask: `0xFFFC0003FFFFFFFF` clears bits 34–49 inclusive (16 bits) and keeps bits
|
||||
0–33 and 50–63. The cleared 16-bit window is presumably where some of the packed
|
||||
width/channel/format bits documented in the wiki live; the retained ~48 bits are what the task
|
||||
description calls "the loader's own stable identity."
|
||||
|
||||
### Measured cross-version stability — **the masked key does not hold up**
|
||||
|
||||
I downloaded the raw `image.bin` for three adjacent releases (0.81.0, 0.82.0, 0.83.0 — note 0.82→0.83
|
||||
also renames the vendor directory `pokemon_le`→`pokemon_pro`, per `PLAN.md`), parsed every
|
||||
section-8 row locally with the exact formula above, and computed `sha256` of each row's raw PCM
|
||||
byte range `[q0, q0+length)` as ground truth for "is this the same sound."
|
||||
|
||||
```
|
||||
python3 sound_stability.py image_081.bin image_082.bin
|
||||
python3 sound_stability.py image_082.bin image_083.bin
|
||||
```
|
||||
|
||||
| Pair | Rows (A→B) | Content reappears somewhere in B (ignoring key/index) | Masked-key join: unambiguous shared keys | …of those, also content-match | Ordinal-index join content-match |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 0.81.0 → 0.82.0 | 2452 → 2498 | not separately re-run for this pair | 2375 | **53 (2.23%)** | 0/2452 (0.00%) |
|
||||
| 0.82.0 → 0.83.0 | 2498 → 2491 | 2465/2498 rows, i.e. **98.68%** | 2451 | **11 (0.45%)** | 0/2491 (0.00%) |
|
||||
|
||||
Within one build the masked key is a clean, collision-free identifier (0 collisions among
|
||||
2452/2498/2491 rows in every version tested — it is a valid *intra-build* lookup key, consistent
|
||||
with its only confirmed use). But across versions, for the *same* content (verified by matching
|
||||
raw PCM `sha256`, which itself shows 98.68% of sound content survives 0.82→0.83 byte-for-byte),
|
||||
the masked key is essentially uncorrelated: only 0.45–2.23% of masked-key-linked pairs across two
|
||||
independent version pairs also share content. The **53** figure for 0.81.0→0.82.0 is suspicious in
|
||||
a useful way — it is exactly the number the task prompt quotes as "53 of 2,452 sound slots kept
|
||||
their index," which suggests this masked-key/content-match pathway is the same measurement that
|
||||
number originally came from (I could not find a "53" in `scripts/verify_phase0.py` or
|
||||
`docs/phase0-verification.md` to confirm the exact original methodology, so treat this as a
|
||||
strong coincidence rather than a confirmed identical derivation).
|
||||
|
||||
**Working hypothesis for why q1 fails cross-version** (unconfirmed — would need a link-map or two
|
||||
independently-built binaries with source access to verify): the retained ~48 bits of q1 behave
|
||||
like a build-time-assigned value (address/handle/table-slot pointer baked in at compile/link time
|
||||
on the vendor's build machine) rather than a content-derived value. It is deterministic *within*
|
||||
one compiled binary — which is all its one confirmed use case (`sound_names.py`'s intra-image
|
||||
name resolution) requires — but has no reason to reproduce across two separate compilations of
|
||||
the same logical sound.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Do **not** wire `LogicalAssetKey::RadiumSound{masked_key}` as the cross-version key for sounds
|
||||
without a further fix — measured stability is 0.45–2.23%, far below useful. The metric that
|
||||
*does* measure stable is straightforward content-addressing: `sha256` of the raw PCM byte range
|
||||
(`data[q0:q0+length]`), i.e. effectively `LogicalAssetKey::ContentOnly`, which is already
|
||||
available and measured **98.68%** stable 0.82.0→0.83.0. This is not "computable from catalog
|
||||
already": `media-evidence.json`'s section-8 rows currently store `index, offset, length, channels,
|
||||
sample_rate` but no per-row content hash and no `masked_key` — either would require re-reading the
|
||||
24-byte row (for masked_key) or the full PCM range (for a content hash, already read once during
|
||||
extraction, just not hashed and persisted) from raw `image.bin` bytes. Recording a per-sound
|
||||
`sha256` in provenance at extraction time (cheap: it's already read into memory to write the
|
||||
`.wav`) would be sufficient to realize the 98.68% figure as the actual stable key, no raw
|
||||
re-reads needed on future runs.
|
||||
|
||||
---
|
||||
|
||||
## 2. Bitmaps (section 3)
|
||||
|
||||
### Byte layout (confirmed against real provenance)
|
||||
|
||||
Section 3 is a pointer table at `header[3]..header[2]` (the first 7 header qwords, read separately
|
||||
in `plugins/media_extract.py:192`, `extract_bitmaps`), one `u64le` pointer per record. Each pointed
|
||||
record starts with (`plugins/dmd_bitmap.py:12`, `decode()`):
|
||||
|
||||
```
|
||||
u32le identity (record_id)
|
||||
u32le flags
|
||||
u16le width
|
||||
u16le height
|
||||
u8 mode (0,1,3,7,9,12 — see FORMATS in dmd_bitmap.py)
|
||||
...payload...
|
||||
```
|
||||
|
||||
`extract_bitmaps` writes one provenance row per pointer-table slot with `section=3, index=<pointer
|
||||
position>, record_id=<identity>, flags, base_record, width, height, mode, offset, length`. This
|
||||
**is already in `media-evidence.json`** — verified directly, e.g.:
|
||||
|
||||
```json
|
||||
{"section": 3, "index": 0, "offset": 122448, "length": 404, "record_id": 1, "flags": 0,
|
||||
"base_record": null, "width": 17, "height": 23, "mode": 0, ...}
|
||||
```
|
||||
|
||||
### `record_id` is a genuine, independent field — not a restatement of pointer position
|
||||
|
||||
Checked directly on 0.83.0's main `image.bin` (1477 records): `record_id != index+1` for 1222 of
|
||||
1477 records. The relationship is not simple — the first 254 slots have `record_id == index+1`,
|
||||
then there's an offset-by-11 stretch (`index=255 → record_id=266`, continuing linearly to
|
||||
`index=1466 → record_id=1477`), then the last 10 slots wrap back to `record_id=256..265`
|
||||
(`index=1467..1476`). This is exactly the kind of independent, container-assigned identity a good
|
||||
key needs — it is *not* simply "index+1" and evidently survives some reordering the pointer table
|
||||
itself has already undergone by the time these bytes were captured. `record_id` was also verified
|
||||
unique within every (source, version) pair tested — zero collisions, and the per-source count
|
||||
(1477 for main `image.bin`, 1469 for `spike_menu/image.bin`) matches the wiki's documented
|
||||
pointer-table sizes exactly.
|
||||
|
||||
### Measured cross-version stability
|
||||
|
||||
```
|
||||
python3 bitmap_stability.py me_a4dc88d1-*.json me_4f4aa552-*.json image_082.bin image_083.bin \
|
||||
'partition-06/pokemon_le/image.bin' 'partition-06/pokemon_pro/image.bin'
|
||||
```
|
||||
|
||||
Content hash = `sha256` of `image.bin[offset:offset+length]`, sliced directly from the raw bytes
|
||||
of each version (same downloaded files used for §1).
|
||||
|
||||
| Join | 0.82.0 → 0.83.0 (1477 records each) |
|
||||
|---|---|
|
||||
| Ordinal (`index`) join | 1477/1477 present in both, **100.00%** content-match |
|
||||
| `record_id` join | 1477/1477 present in both, **100.00%** content-match, 0 index changes |
|
||||
|
||||
**Honest caveat, stated plainly:** on this specific adjacent pair, section 3 did not change or
|
||||
reorder *at all* — every record kept both its ordinal index and its content. This means I could
|
||||
**not**, on the available corpus, produce a version pair where `record_id` and ordinal diverge for
|
||||
bitmaps, so I cannot empirically demonstrate `record_id`'s superiority over plain ordinal for this
|
||||
asset class the way I could for sounds (where insertions constantly reshuffle order) or for scene
|
||||
textures (below). What I *can* say from measurement: section 3 (DMD bitmap/frame table) is far
|
||||
more static across point releases than section 8 (sounds) — plausible, since it holds display
|
||||
graphics rather than a constantly-growing Pokémon voice/roster table — and `record_id` is
|
||||
structurally a real, independent, collision-free, container-assigned identifier (unlike sounds'
|
||||
`q1`), which is a necessary property a good key must have even though I could not force a
|
||||
divergence to prove sufficiency on this corpus. I'd recommend re-running this same script on a
|
||||
wider-separated pair (e.g. 0.81.0 vs 0.86.0) or a release known to add/remove DMD art before
|
||||
trusting this beyond "not disproven."
|
||||
|
||||
### Recommendation
|
||||
|
||||
`LogicalAssetKey::Radium{role, section: 3, record_id}` is **already computable directly from
|
||||
existing catalog provenance** — no raw container re-read needed, `record_id` is already a field
|
||||
in every section-3 `media-evidence.json` row. Wire it as-is; it is structurally sound and not
|
||||
contradicted by measurement, even though the measurement available couldn't stress it.
|
||||
|
||||
---
|
||||
|
||||
## 3. DMD animation chains
|
||||
|
||||
Same provenance rows as §2; this section is about the semantics of `base_record` and `flags`,
|
||||
which live in the same section-3 records.
|
||||
|
||||
### `base_record` semantics
|
||||
|
||||
In `plugins/dmd_bitmap.py:12`, for delta-encoded modes (`3`="column-delta", `9`="row-delta"), the
|
||||
decoder looks up a `keyframes` dict for key `(identity-1)&0xffff` first, falling back to
|
||||
`identity&0xffff`, requiring a `(width,height)` match:
|
||||
|
||||
```python
|
||||
base=next((keyframes[k] for k in ((identity-1)&0xffff,identity&0xffff) if k in keyframes and keyframes[k][0:2]==(width,height)),None)
|
||||
...
|
||||
'base_record': (identity-1)&0xffff if base else None
|
||||
```
|
||||
|
||||
**Note this imprecision in the current decoder**: the reported `base_record` is *always*
|
||||
`identity-1` whenever *any* base was found — even in the (unobserved-here, but code-permitted)
|
||||
case where the actual dictionary hit was on `identity` itself, not `identity-1`. On the one real
|
||||
chain I could inspect, every link was in fact consecutive (`base_record == record_id - 1`
|
||||
throughout), so this ambiguity never manifested in this corpus, but it's a latent correctness gap
|
||||
worth flagging for anyone hardening this into a schema.
|
||||
|
||||
### `flags` semantics (partially confirmed)
|
||||
|
||||
`flags` is a full `u32`. Only one bit has confirmed decoder semantics — bit 2 (`0x4`):
|
||||
"cache this frame's decoded pixels as the keyframe available to future deltas of this identity"
|
||||
(`elif flags&4: keyframes[identity&0xffff]=(width,height,result)`). Cross-tabulating real
|
||||
`(mode, flags, has_base)` triples from 0.83.0's 1477 main-image records:
|
||||
|
||||
| mode | flags | has base_record | count | interpretation |
|
||||
|---|---|---|---:|---|
|
||||
| 0 | 0 | no | 1308 | static frame, not chained |
|
||||
| 0 | 2 | no | 141 | static frame, bit1 set — **unconfirmed meaning** |
|
||||
| 1 | 0/2 | no | 2 | column-whitespace, not chained |
|
||||
| 7 | 0/2/4 | no | 6 | row-whitespace; the `flags=4` one is the **chain's seed keyframe** |
|
||||
| 3 | 5 | yes | 2 | column-delta, mid-chain (bit0 delta + bit2 re-cache) |
|
||||
| 9 | 1 | yes | 1 | row-delta, **terminal** frame of the chain (bit0 only — not re-cached) |
|
||||
| 9 | 5 | yes | 17 | row-delta, mid-chain |
|
||||
|
||||
So: bit0 (`0x1`) is set on every record that has a `base_record` and clear on every record that
|
||||
doesn't — a clean, perfectly-correlated "this is a delta continuation" marker. Bit2 (`0x4`) marks
|
||||
"this frame becomes available as a future base." Bit1 (`0x2`, seen on 145/1477 records across
|
||||
modes 0/1/7) is **not read anywhere in the current decoder**; I found no evidence of its meaning
|
||||
and am not asserting one.
|
||||
|
||||
### Reconstructing an ordered sequence
|
||||
|
||||
Walk backward from any `mode∈{3,9}` record via `base_record = record_id - 1` until reaching a
|
||||
record with `base_record = None` (the seed keyframe, `flags&4` set, arbitrary mode). Forward
|
||||
order is simply increasing `record_id` while each successive record's `base_record` equals the
|
||||
previous record's `record_id` and `flags&1` holds; the terminal frame has `flags&4` clear. Note
|
||||
the code computes indices mod `0xffff` (16-bit wraparound), so a chain could in principle cross
|
||||
the `65536` boundary — not observed in this corpus.
|
||||
|
||||
### Measured stability
|
||||
|
||||
```
|
||||
python3 analyze_dmd2.py me_a4dc88d1-*.json # 0.82.0
|
||||
python3 analyze_dmd2.py me_4f4aa552-*.json # 0.83.0 (re-run filtered to main image.bin per source)
|
||||
python3 analyze_dmd2.py me_086.json # 0.86.0
|
||||
```
|
||||
|
||||
Independently, for each of 0.82.0, 0.83.0 and 0.86.0's main `image.bin` (1477 section-3 records
|
||||
each): exactly **20** records carry a non-null `base_record`, forming exactly **one** 21-member
|
||||
connected chain (`record_id` 215→235, all `128×32` — the physical DMD panel resolution). The
|
||||
chain's `record_id`s, `flags` values, and `mode` sequence are byte-for-byte identical across all
|
||||
three releases tested (spanning at least 0.82.0 through 0.86.0). This is the strongest, cleanest
|
||||
measured-stable result of the four asset classes.
|
||||
|
||||
**Caveat on generality**: this game has exactly one such chain (n=1). I can say this one chain is
|
||||
stable across three releases; I cannot say anything from measurement about what happens when a
|
||||
*new* chain is added or an existing one is extended/shortened, because that never happened in the
|
||||
tested range.
|
||||
|
||||
### Recommendation
|
||||
|
||||
`base_record`/`flags` are already present in the same section-3 `media-evidence.json` rows as
|
||||
§2 — **computable directly from existing catalog provenance**, no raw re-read needed. Chain
|
||||
reconstruction (walking `base_record` links) can be done entirely from stored provenance at query
|
||||
time; no new extraction work required, only the graph-walk logic, which should also record the
|
||||
resolved chain's ordinal position (0..N) as separate metadata since `record_id` order already
|
||||
gives frame order for free once linked.
|
||||
|
||||
---
|
||||
|
||||
## 4. Scene assets (`scene.radium` / `texture-NNNN.bcn`)
|
||||
|
||||
### How textures are currently found — no container-internal ID exists in the decoder today
|
||||
|
||||
`plugins/scene_media.py:24` (`inline_images`) finds BC1/BC3 texture blobs by **linear byte-pattern
|
||||
scanning** for a `(format, 0, 0)` `u32×3` signature, reading `width,height` from the 8 bytes
|
||||
immediately before the match and `length` from the 4 bytes immediately after:
|
||||
|
||||
```python
|
||||
sig = struct.pack('<3I', fmt, 0, 0)
|
||||
... i = data.find(sig, start) ...
|
||||
w, h = struct.unpack_from('<2I', data, i-8)
|
||||
length = struct.unpack_from('<I', data, i+12)[0]
|
||||
```
|
||||
|
||||
`texture-{i:04}.bcn`'s `i` is purely "the i-th match found scanning low-to-high offset" — there is
|
||||
**no id field read from the record at all**, unlike section 3's `record_id`. `offset` (already in
|
||||
provenance) is exactly as fragile as the ordinal for the same reason every other ordinal in this
|
||||
project is fragile: any upstream insertion into the `scene.radium` blob shifts every subsequent
|
||||
match's offset and hence its `i`.
|
||||
|
||||
One exception: the `path.suffix == '.asset'` branch (`extract_scene`, line 38) pairs a single
|
||||
texture file with a companion `scene.radium` descriptor via `texture_descriptor()`, which matches
|
||||
the **filename itself** (`path.name`) as a length-prefixed string embedded in the descriptor. That
|
||||
gives *that* texture a genuine human-readable, filesystem-derived name — structurally much
|
||||
stronger than ordinal — but this only covers the single-file-per-asset layout, not the common case
|
||||
of many inline textures packed into one `scene.radium`.
|
||||
|
||||
`PLAN.md` records that `scene.radium` is actually "a self-describing typed object graph whose
|
||||
class vocabulary is literally `Bitmap, Sprite, Font, Text, Video, Shape, StreamingFlipbook`, with
|
||||
named instances (`Attack_Instance`, `Kill_Instance`) and explicit ordered frame lists." If that
|
||||
graph were parsed, named instances would very plausibly give inline textures a stable
|
||||
name-based key. **This is not implemented**: `scene_media.py`'s non-`.asset` branch explicitly
|
||||
records `status='scene_metadata_preserved', reason='Exact scene descriptor remains in parent;
|
||||
script semantics are not decoded'`. I did not measure this path because the decoder does not
|
||||
produce it — flagging as a concrete, promising, but unverified opportunity, not a finding.
|
||||
|
||||
### The sha1-named scene directory
|
||||
|
||||
Each `scene.radium` lives under a 40-hex-character directory, e.g.
|
||||
`assets/lcd/auto_loaded/18dad27e209b169855855422f6578e527adbb5cf/scene.radium` (this exact hash
|
||||
also appears as `identity.rs`'s own worked example for `SceneInstance{scene: "18dad27e...", ...}`).
|
||||
|
||||
**Measured directly** by scanning the full archived file tree (`/api/snapshots`, all entries) for
|
||||
every `.../<40-hex>/scene.radium` path, across four adjacent Pokémon LE version pairs:
|
||||
|
||||
| Transition | dirs in A | dirs in B | common | only-A | only-B |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| 0.81.0 → 0.82.0 | 36 | 37 | 36 (**100%**) | 0 | 1 |
|
||||
| 0.82.0 → 0.83.0 | 37 | 38 | 37 (**100%**) | 0 | 1 |
|
||||
| 0.83.0 → 0.85.0 | 38 | 39 | 38 (**100%**) | 0 | 1 |
|
||||
| 0.85.0 → 0.86.0 | 39 | 39 | 39 (**100%**) | 0 | 0 |
|
||||
|
||||
Every scene directory present in the earlier release of every tested pair is still present, with
|
||||
the identical sha1 name, in the later release; new scenes are only ever added, never renamed or
|
||||
removed, across this entire span (0.81.0→0.86.0). This is a strong, clean, measured-stable result
|
||||
for the directory-level identity — as good as §3's DMD chain.
|
||||
|
||||
**Caveat on interpretation**: this looks like a content/UID hash from the upstream Godot import
|
||||
pipeline, not something Radium itself guarantees. Its stability plausibly follows from "the source
|
||||
scene asset's bytes/UID didn't change between builds," which is a property of the studio's asset
|
||||
pipeline, not a Radium format guarantee — an artist re-saving/re-importing an otherwise-unchanged
|
||||
scene could in principle change this hash, and that cannot be tested from archived firmware
|
||||
alone; I found no case of it in this corpus, but I can't rule it out in general.
|
||||
|
||||
### Texture-within-scene ordinal stability, measured directly
|
||||
|
||||
For the 34 scene directories shared between 0.82.0 and 0.83.0 (texture count unchanged in every
|
||||
one of them), I fetched and `sha256`-hashed the actual extracted bytes of all 483 texture
|
||||
positions in both versions:
|
||||
|
||||
```
|
||||
python3 scene_texture_stability.py
|
||||
```
|
||||
|
||||
| Join | Result |
|
||||
|---|---|
|
||||
| Position-in-scene (ordinal) | 288/483 match = **59.63%** |
|
||||
| Content-only reappearance (ignoring position) | 275/311 distinct A textures reappear in B = **88.42%** |
|
||||
|
||||
So even in scene directories whose *container identity* (the sha1) is rock-solid, the ordinal
|
||||
position of an individual texture inside that container is materially weaker than content
|
||||
addressing, and — per above — no stronger structural field is available from the current decoder
|
||||
to close that gap without new work parsing the object graph.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Use the sha1 directory as the `scene` half of `LogicalAssetKey::SceneInstance` — it is **already
|
||||
present directly in the archived path**, no decoder work needed, and is measured 100% stable
|
||||
across every tested transition. For the `instance` half (per-texture identity), there is currently
|
||||
no structural field better than ordinal; either (a) accept content-hash-based texture identity
|
||||
(measured 88.42% stable, still meaningfully better than ordinal's 59.63%) as an interim floor, or
|
||||
(b) invest in parsing the named-instance object graph PLAN.md already documents, which is
|
||||
unimplemented and therefore unmeasured, but structurally the most promising lead of all four asset
|
||||
classes for closing this gap.
|
||||
|
||||
---
|
||||
|
||||
## Summary table
|
||||
|
||||
| Asset class | Proposed key | Derivation | Computable from existing catalog provenance? | Measured cross-version stability |
|
||||
|---|---|---|---|---|
|
||||
| Sounds (section 8) | ~~`RadiumSound{masked_key}`~~ → `ContentOnly{sha256(pcm bytes)}` | `q1 & 0xFFFC0003FFFFFFFF` computed but **falsified**; use `sha256(data[q0:q0+length])` instead | No — neither is in `media-evidence.json` today; both require a raw re-read (or persisting a hash already computed in memory during extraction) | masked_key: **0.45–2.23%** (falsified); content hash: **98.68%** (0.82.0→0.83.0) |
|
||||
| Bitmaps (section 3) | `Radium{role, section:3, record_id}` | `record_id` = `identity` field read from the pointed record's own bytes, independent of pointer-table position | **Yes** — already a field in every section-3 provenance row | **100%** on the one pair I could test (0.82.0→0.83.0), but that pair had zero reordering/removal, so this doesn't prove superiority over ordinal on this corpus — only that it isn't contradicted, and that `record_id` is structurally independent of position (verified: `record_id != index+1` for 1222/1477 records within one version) |
|
||||
| DMD chains | walk `base_record` links within section-3 rows | `base_record = record_id-1` when a delta frame's decode found a matching prior keyframe; `flags&1`=is-delta, `flags&4`=becomes-future-keyframe, `flags&2` unconfirmed | **Yes** — `base_record`/`flags` already in provenance | **100%** — identical single 21-frame chain (same record_ids, flags, modes) across 0.82.0, 0.83.0, 0.86.0. n=1 chain in this corpus; no data on chain add/remove/resize |
|
||||
| Scene assets | `SceneInstance{scene: <sha1 dir>, instance: ordinal or content-hash}` | sha1 = path component under the container; per-texture instance has no structural ID today | Directory: **yes**, it's a path component. Per-texture ID: **no** — current decoder finds textures by byte-pattern scan, no id field, no object-graph parsing | Directory: **100%** across 4 tested transitions (0.81.0→0.86.0), purely additive. Per-texture ordinal: **59.63%**; per-texture content-only reappearance: **88.42%** (0.82.0→0.83.0) |
|
||||
|
||||
## Key overall finding
|
||||
|
||||
The one identity the task explicitly asked me to verify — the sounds' masked `q1` key — is the one
|
||||
that **fails** measurement (0.45–2.23% stable), despite being real, deterministic, and
|
||||
collision-free *within* a build; it is a legitimate intra-build lookup key (confirmed use:
|
||||
`plugins/sound_names.py`'s Godot-name resolution) but not a version-independent identity in
|
||||
practice. The three keys that already have a real container-assigned field independent of
|
||||
extraction order — bitmap `record_id`, DMD `base_record` chains, and the scene sha1 directory —
|
||||
all either measured 100% stable or were at least not contradicted by measurement. Raw content
|
||||
hashing (`sha256` of the exact decoded byte range) was the single most reliable signal across
|
||||
every asset class tested and is a reasonable universal floor (`LogicalAssetKey::ContentOnly`)
|
||||
wherever a better structural field doesn't exist or hasn't been wired up yet.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Resource admission verification
|
||||
|
||||
On 2026-09-16, `tests/resource_admission.rs` exercised the shared admission pool
|
||||
through public Archive APIs using temporary archives and local Python fixture
|
||||
plugins. No live configuration or catalog was changed.
|
||||
|
||||
Command:
|
||||
|
||||
```
|
||||
cargo test --offline --locked --test resource_admission -- --include-ignored --test-threads=1
|
||||
```
|
||||
|
||||
All three tests passed; test execution took 3.08 seconds on this host. The
|
||||
32-process case is explicitly ignored by portable default test runs because it
|
||||
requires at least 32 available CPUs. It was run explicitly here.
|
||||
|
||||
The process test holds 32 actual child processes at file barriers, verifies each
|
||||
PID exists under `/proc`, and checks the resource ledger while all are blocked.
|
||||
Each task requests one CPU, 256 MiB process allowance, and 64 MiB scratch. The
|
||||
ledger reported 32 CPUs, 2 GiB scratch, and 10 GiB combined process/scratch
|
||||
promises. These are reservations, not measured allocations: the fixture processes
|
||||
are small and mostly waiting. A 33rd queued task cannot start until one running
|
||||
job is cancelled. Its replacement starts while the other 31 remain blocked.
|
||||
Releasing the barrier produces 32 completed jobs and one cancelled job. CPU,
|
||||
memory, scratch and active-task counters return to zero; maintenance is rejected
|
||||
while work is active and succeeds afterward. Temporary scratch directories are
|
||||
removed.
|
||||
|
||||
The analysis fixture checks a three-CPU request and `max_heap=384M` reach the
|
||||
plugin request, and that admission reserves 384 MiB heap plus 512 MiB native
|
||||
allowance plus its 1 GiB scratch estimate. It does not launch a JVM or claim to
|
||||
measure actual heap enforcement. Impossible CPU and heap requests fail before
|
||||
process startup without leaking permits.
|
||||
|
||||
A restart regression queues a frozen three-CPU/384M job, reopens the archive with
|
||||
current settings reduced to one CPU/64M, and verifies the original frozen demand
|
||||
is both reserved and passed to the child. Review found that the first admission
|
||||
implementation used current lane-wide settings, which could under-reserve pinned
|
||||
jobs. Exact frozen-config admission fixes that defect.
|
||||
|
||||
Follow-up regressions cover a frozen numeric-string CPU request (`"8"`), explicit
|
||||
zero/negative/fractional/malformed CPU values, and permanent admission failures in
|
||||
queues. An impossible head job becomes failed while a fitting peer completes;
|
||||
invalid analysis children also settle their group and owning import as failed.
|
||||
Transient pressure/busy outcomes remain queued. Five portable resource tests pass;
|
||||
the separate 32-process host exercise remains explicitly opt-in.
|
||||
|
||||
Limits remain explicit: this is a short, deterministic overlap/cancellation test,
|
||||
not sustained production throughput, an out-of-memory stress test, or proof of
|
||||
hard OS isolation. Configured allowances and measured headroom control admission;
|
||||
they do not independently restrict arbitrary plugin process allocation. The
|
||||
review also identified fixed per-plugin memory allowances underestimating parallel
|
||||
workers and missing cgroup memory headroom; root changes scale the allowance by
|
||||
requested CPU and separately test cgroup sampling.
|
||||
@@ -0,0 +1,63 @@
|
||||
# Schema 13 migration rehearsal — 2026-09-16
|
||||
|
||||
The current `Catalog::open` migrated a disposable copy of the consistent schema-12
|
||||
backup successfully. Production was not written, and the original backup remains
|
||||
schema 12 with unchanged size and modification time. A read-only query of the live
|
||||
catalog after the rehearsal still reported schema 12 and the same artifact,
|
||||
symbol, asset, signature and snapshot counts as the backup.
|
||||
|
||||
Artifacts (gitignored):
|
||||
|
||||
- Source: `data/validation/schema12-before-native.sqlite3` (7,380,164,608 bytes).
|
||||
- Migrated copy: `data/validation/schema13-dryrun-20260916-1.sqlite3`.
|
||||
- Complete machine-readable evidence: matching `.report.json` and `.log` files.
|
||||
- Independent full-mask comparison against the original backup:
|
||||
`data/validation/schema13-dryrun-20260916-1-old-masks.json`.
|
||||
|
||||
The reusable `examples/validate_schema13.rs` refuses identical source/target paths,
|
||||
requires a target named `schema13-dryrun-*` under a `validation` directory, opens
|
||||
its source read-only, and requires both files to start at schema 12. It invokes
|
||||
the shipping migration rather than reproducing its SQL in a script.
|
||||
|
||||
## Results
|
||||
|
||||
- `Catalog::open`, including its final WAL checkpoint, took **291.249 seconds**.
|
||||
This was the **debug helper**, including debug bundled SQLite; it is an observed
|
||||
conservative rehearsal time, not an optimized-release timing claim. Subsequent
|
||||
verification time is excluded. Deployment should allow minutes for migration,
|
||||
rather than relying on the usual six-second startup expectation.
|
||||
- All **23 original table row counts stayed identical**. These include 1,484,269
|
||||
artifacts, 1,170,298 symbols and function signatures, 125,372 assets, 283,936
|
||||
observations, 8,366 names, 51,703 deltas, 123 snapshots and 15 versions.
|
||||
- FTS5 external-content integrity checks (`rank=1`) passed for all three indexes:
|
||||
artifacts, symbols and assets. This verifies index agreement with table content.
|
||||
- **836,618 full masks** were populated. Every signature matched its expected old
|
||||
`symbols.payload.signature_pattern`, with **zero mismatches**. A separate
|
||||
read-only join directly to the untouched backup independently confirmed this
|
||||
across all 1,170,298 signatures in 14.47 seconds.
|
||||
- All **30 complete result-set comparisons** matched literal baseline searches,
|
||||
including punctuation (`_`, `%`, `::`), short terms (`0x`), an embedded quote,
|
||||
operator-like text (`NOT attack`) and the empty query. Ten further checks
|
||||
exercised the actual public `Catalog::symbols` and `Catalog::artifacts` methods.
|
||||
- After checkpoint, size was **7,716,790,272 bytes**, a growth of **336,625,664
|
||||
bytes (321 MiB, 4.56%)**. An approximately 1.7 GiB transient WAL was observed
|
||||
during the transaction; it was truncated afterward.
|
||||
|
||||
## Observed search times
|
||||
|
||||
Each row compares equivalent full result-set retrieval, including row IDs, on the
|
||||
backup and migrated copy. These are single local debug-build measurements, not a
|
||||
release HTTP latency benchmark. Positive-hit examples avoid overstating the
|
||||
benefit from extremely fast zero-result queries.
|
||||
|
||||
| Table / literal | Matches | Baseline | Indexed | Speedup |
|
||||
|---|---:|---:|---:|---:|
|
||||
| artifacts / `sound` | 61,793 | 3,223.75 ms | 132.28 ms | 24.37× |
|
||||
| artifacts / `pokemon` | 130,484 | 4,475.65 ms | 413.41 ms | 10.83× |
|
||||
| artifacts / `image.bin` | 170,499 | 3,274.98 ms | 612.50 ms | 5.35× |
|
||||
| assets / `sound` | 35,792 | 856.58 ms | 75.30 ms | 11.38× |
|
||||
| assets / `pokemon` | 13,022 | 798.11 ms | 42.84 ms | 18.63× |
|
||||
|
||||
One- and two-character searches intentionally retain scans. Their results stayed
|
||||
exact and their timings remained at scan speed; the migration does not claim to
|
||||
accelerate those queries.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user