A local, library-first Rust framework for immutable game snapshots and independent extraction/analysis revisions. The CLI and static HTTP client use the same core. One physical Rustic/restic store shares chunks across every logical game repository.

This is a development slice with a real Game of Thrones 1.37.0 extraction and native Ghidra pilot. It is not yet a complete SPIKE asset decoder. See local installation, corpus coverage, and required keys. Keep your source collection while validating it.

The new Theia game workbench is available separately while full-corpus validation continues. See redesign status and acceptance evidence. The compatible static client remains the Rust service default.

The Emulator view provides isolated client sessions, a working screen/audio and switch diagnostic, and experimental Pokémon LE SD-runtime launch targets. Open Emulator in the workbench activity bar. See architecture and browser execution options.

For repository contents, cloning, commit identity, and pushing with a forwarded SSH agent, see the Git workflow. Local firmware, archive data, tools, credentials, and generated builds stay outside Git.

Run

Requirements: Linux x86-64, a current stable Rust toolchain, and Python 3 for the example plugins. Browsing saved releases stays local; URL imports contact the selected source when requested. Building initially downloads Cargo dependencies; the lockfile is checked in.

export PATH="$HOME/.cargo/bin:$PATH"
test -f config.json || cp config.example.json config.json
cargo build --locked --release
./target/release/verstack --config config.json serve

Use the optimized build for real firmware; debug builds make archive restoration substantially slower. For the native Explorer, asset editors and release comparisons, start the Theia workbench on port 3000.

The compatibility client remains at http://127.0.0.1:8080. It includes filesystem or streamed single-file upload imports, snapshot/asset browsing, safe media previews, text/hex inspection, file comparisons, plugin execution, function comparisons, downloads, and polling run status. It uses a Bootstrap-era utility layout with local CSS/JavaScript and no frontend build requirement.

Edit config.json before using real data:

archive: authoritative local archive directory, shared by all games.
workspace: a disposable directory on tmpfs/ramfs. The example uses /tmp/verstack-workspace; verify that /tmp is mounted in RAM on your machine.
import_roots: existing server directories the HTTP API may import from.
require_ram_workspace: defaults to true. Startup and processing reject disk-backed scratch or insufficient free space; there is no disk fallback. Child tools inherit RAM scratch and cache locations.
workspace_bytes: the scratch allowance must fit available RAM filesystem space (live configuration: 64 GiB). Usage is polled, not a kernel quota.
processing_enabled: pauses automatic submission and queue consumption when false, preserving job history. The live service and example configuration are paused. Explicit CLI/API processing and user-selected import workflows remain available.
bind: the Rust listener defaults to 127.0.0.1:8080. For the LAN workbench, keep this on loopback and start Theia with --hostname 0.0.0.0 --port 3000; its same-origin proxy connects to Rust.
plugins: local command argument arrays and versioned settings. Use absolute paths for plugin scripts. The example uses this workspace's path; change it if you move the project.

The Theia Import release view browses server files/folders or URL sources. Paste an Internet Archive item link, click Browse URL, choose a file, and supply the game, edition and version. Extraction and asset previews are selected by default; code analysis is optional. Jobs shows live progress, cancellation, retry and an Open release action. Supported containers include ZIP, SPK, partitioned SD-card images, ext/FAT filesystems and configured LUKS volumes; other files remain cataloged. Downloads and scratch work use RAM, including sparse zero regions in disk images.

Godot packages can also recover their scripts through the optional godot-scripts stage. Install the pinned tool with python3 scripts/install_godot_tools.py, then regenerate the pipeline configuration. Browse assets → Scripts provides folders, syntax-highlighted source, filtering, and downloads. See asset correlation findings for links between scripts, scenes, and media.

The example config allows samples/. Import samples/demo-v2 as version 2, followed by samples/demo-v1 as version 1, to exercise out-of-order history. These are synthetic files, not game assets.

SD-card imports support primary and extended MBR partitions (logical partitions appear as partition-05 onward), as well as GPT. When the pinned spike2-audio plugin is configured, asset imports also decode SPIKE 2 sounds using each image's companion executable. Sound decoding uses up to 16 independent workers, bounded by CPU availability and the longest sound's estimated memory use; sound-directory derivation remains sequential. scripts/configure_pipeline.py configures that local decoder. Retrying an older failed import adds the audio stage while reusing its retained original, extracted files, and completed media outputs.

LUKS SD partitions use up to 16 CPU- and RAM-bounded decryption workers. The parent writes decrypted blocks in order, with a bounded queue of 4 MiB blocks; the serial decoder remains available for smaller resource allowances.

For large imports, build with cargo build --release --locked and run ./target/release/verstack --config config.json serve. The default development build is useful for testing but substantially slower at chunking.

The CLI works without the service. Stop the service before using the mutating CLI: an instance ownership lock prevents another process from accessing the same archive concurrently.

Persistent user services for this installation are in deploy/. After building the release binary and workbench, install them with:

systemctl --user enable --now "$PWD/deploy/verstack-backend.service" "$PWD/deploy/verstack-workbench.service"
loginctl enable-linger "$USER"

The units assume the checkout is at ~/verstack and use config.json. Stop any manually launched instances first, allowing active processing to finish. Theia listens on LAN port 3000; configure Rust on loopback port 8080. Lingering starts the services at boot and keeps them running after logout. Inspect status and logs with systemctl --user status verstack-backend verstack-workbench and journalctl --user -u verstack-backend -u verstack-workbench.

cargo run --locked -- import samples/demo-v2 --repository Demo --version 2
cargo run --locked -- list
cargo run --locked -- verify SNAPSHOT_ID
cargo run --locked -- restore SNAPSHOT_ID /tmp/restored-demo
cargo run --locked -- compare EARLIER_SNAPSHOT LATER_SNAPSHOT
cargo run --locked -- process INPUT_SNAPSHOT zip-extract
cargo run --locked -- functions BEFORE_SNAPSHOT BEFORE_FUNCTIONS_PATH AFTER_SNAPSHOT AFTER_FUNCTIONS_PATH

Restore requires a destination that does not already exist. Exact file bytes, directories, and symlinks are restored. File permission bits are restored without setuid/setgid. Timestamps and original modes are cataloged; timestamp, directory permission, owner, xattr, ACL, hardlink, and non-UTF8 filename restoration are not implemented. Unsupported filenames and special files cause an explicit failed import, not silent omission.

Implemented preservation model

An import first captures its inputs into a bounded workspace, then archives them through Rustic, reads back and hashes retained files, and finally publishes a durable immutable manifest. An extractor creates another snapshot with parent and run links; the earlier snapshot is never rewritten. Run states include running, complete, failed, and interrupted. Reopening reconciles an interrupted final run update against already-published manifests.

Artifact identity is blake3:<hex> over uncompressed bytes. This is a framework identity, not a change to restic's storage algorithms. Rustic supplies its compatible content-defined chunks, compression, authenticated encryption, and packs. Backend indexes are cached in memory and refreshed after commits. Artifact range reads avoid restoring a full build.

A versioned SQLite catalog indexes immutable JSON snapshots and run history on startup. JSON remains recovery evidence; library, artifact and symbol queries use SQLite. Durable jobs record queued/running/completed/failed/interrupted states and explicit retries. Release metadata includes repository, version, edition, generation, and an optional explicit release date; import time is separate. Generation is unknown unless supplied as metadata. Tool processing revisions describe semantic changes separately from executable versions and source digests. Fingerprints include effective settings, schema, semantic revision and input identities. Successful current results are reused; failed replacements preserve the prior active output.

All original inputs currently remain archived. Filesystem originals are never deleted. Temporary uploaded inputs are removed after processing; their captured original bytes remain in the archive on success. Cleanup based on extracted-content guarantees is deferred until real extractor coverage is validated.

Ghidra and SPIKE adapters

See plugin setup and protocol. Included adapters:

ZIP: exact regular-entry extraction with duplicate/path validation, opaque-parent retention, and expansion budgeting.
SPIKE: wrapper for a SHA-256-pinned local bdash/spike-spk executable, including first split-package parts. A separate spike-probe identifies wrapper signatures and required credential references. A separate spike3-unpack plugin decrypts supported LUKS2/ext4 update wrappers; spike3-extract uses the pinned type-4 parser extension. Generation inference and decoding proprietary inner asset formats remain incomplete.
Ghidra: pinned local headless installation, ELF/PE selection, function inventories and exact body hashes, .gzf export, and a second headless invocation requiring a successful reopen receipt. Export and reopen have been tested with Ghidra 12.1.3 on synthetic x86-64 and a real ARM game executable.

Function comparison is available in the CLI/API and through Compare functions on an analysis snapshot's functions.json asset. It accepts only unique exact bodies of at least 32 bytes, excluding thunks, with compatible profiles. Executable selection paths can differ; all other recorded analysis settings must agree. It explains unmatched functions without labeling them publisher additions. The WebUI offers filtering, pages of 100 results, and saved immutable comparison reports; the CLI saves reports with functions ... --save. BSim, Version Tracking, structural matching, global symbol propagation, strings/import indexing, and decompilation-text indexing remain next steps.

Verification and recovery

cargo test --locked
cargo clippy --locked --all-targets -- -D warnings
cargo run --locked --example storage_spike
python3 -m unittest discover -s tests -p 'test_*.py'

An optional Playwright browser smoke test is in tests/ui.mjs. Start a server with disposable data on port 18765, install Playwright and its browser dependencies separately, and run node tests/ui.mjs. VERSTACK_TEST_URL changes the target; VERSTACK_PLAYWRIGHT can point to a separate Playwright installation. This test imports synthetic files into the target archive.

The synthetic storage spike imports a 16 MiB binary, an identical copy under another game, and a version with a 4 KiB insertion. It verifies restored bytes and reports introduced chunks, packed bytes, and timings. It is not a prediction of savings for the 2.4 TB game collection.

Back up the entire archive directory, especially archive-key.json, store/, snapshots/, and runs/. The archive key is generated locally with restrictive file permissions and is required to read encrypted storage. It is separate from package-decryption keys. Domain metadata is not encrypted. Missing chunks are detectable but cannot be recreated without another copy.

Interrupted imports may leave unreferenced backend snapshots/packs and temporary directories. They are not published as complete framework snapshots. Use cleanup-orphans for framework-aware reclamation of unpublished backend snapshots. It checks references across every game and records actual reclaimed packed bytes. Published snapshots, including original wrappers, remain retained; do not independently prune them. Failed-plugin workspaces normally disappear; after an abrupt host termination, remove abandoned workspace directories only while the instance and its child tools are stopped.

Current scope and next work

See architecture and remaining milestones. The next acceptance tests are inner asset-pack extraction and a related release comparison. The real pilot establishes one package/executable path; it does not establish compatibility with every SPIKE generation, power-loss recovery, or 2.4 TB scale.

No project license has been selected yet; GPL is the stated preference under consideration. Dependency licenses remain their authors' licenses.

Game library and media browser

The main screen groups archive snapshots by game. Select a game, then a version (edition and generation remain separate), to browse artifacts from every import and processing output for that release. Version links survive reload and browser back/forward. The output filter narrows the browser to one job's snapshot; analysis tools have a separate input selector. Processing history in a version is scoped to its outputs and input snapshot IDs, including failed jobs.

The default media gallery serves 24 artifacts per page, with image enlargement, native audio/video controls, path search, media filters, and original downloads. Use All files or Other files for reports, text/hex inspection, and function comparison. Preview failures show a download fallback. Browser-supported image, audio, and video formats are previewed directly; proprietary .asset/.radium containers and unsupported codecs still need a decoder/transcoding stage.

GET /api/library omits entry manifests. GET /api/artifacts filters by repository, version, edition, generation, optional source, search, kind (media, image, audio, video, file, all), and zero-based page. Responses contain items, total, page, and page_size (24). Pagination is server-side through the SQLite artifact index.

Media uses the existing bounded HTTP streaming and byte-range endpoint, allowing native seeking and cancellation. Images load lazily and audio/video use preload="none". Optional media-preview processing creates verified browser previews and thumbnails. Without that processing, images are original-resolution assets, so individual large images can be expensive. Job progress continues to poll every two seconds.

Browser regressions: tests/ui-functions.mjs and tests/ui-media.mjs use isolated fixtures (including actual WAV/WebM playback). tests/ui.mjs requires a disposable server and imports synthetic files. tests/ui-gallery.mjs is a read-only pilot check against the current Pokémon/Game of Thrones archive on port 8080, covering 311 images, navigation, paging, previews, empty search, and mobile width.