Baseline existing archive, SPIKE adapters and web client
This commit is contained in:
Verstack Local
@@ -0,0 +1,7 @@
|
||||
/target/
|
||||
/.cargo-home/
|
||||
/data/
|
||||
/config.json
|
||||
__pycache__/
|
||||
*.pyc
|
||||
/tools/
|
||||
Generated
+2952
File diff suppressed because it is too large
Load Diff
+26
@@ -0,0 +1,26 @@
|
||||
[package]
|
||||
name = "verstack"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
publish = false
|
||||
|
||||
[dependencies]
|
||||
anyhow = "1"
|
||||
axum = { version = "0.8", features = ["multipart"] }
|
||||
blake3 = "1"
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
fs2 = "0.4"
|
||||
nix = { version = "0.31", features = ["signal", "process"] }
|
||||
rustic_core = "0.13"
|
||||
rustic_backend = { version = "0.7", default-features = false }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
tempfile = "3"
|
||||
tokio = { version = "1", features = ["full"] }
|
||||
tokio-util = { version = "0.7", features = ["io", "io-util"] }
|
||||
http-body-util = "0.1"
|
||||
uuid = { version = "1", features = ["v4", "serde"] }
|
||||
walkdir = "2"
|
||||
|
||||
[dev-dependencies]
|
||||
tower = { version = "0.5", features = ["util"] }
|
||||
@@ -0,0 +1,122 @@
|
||||
# Artifact archive (working title: verstack)
|
||||
|
||||
A local, library-first Rust framework for immutable game snapshots and independent extraction/analysis revisions. The CLI and static HTTP client use the same core. One physical Rustic/restic store shares chunks across every logical game repository.
|
||||
|
||||
This is a development slice with a real Game of Thrones 1.37.0 extraction and native Ghidra pilot. It is not yet a complete SPIKE asset decoder. See [local installation, corpus coverage, and required keys](docs/local-pilot.md). Keep your source collection while validating it.
|
||||
|
||||
## Run
|
||||
|
||||
Requirements: Linux x86-64, a current stable Rust toolchain, and Python 3 for the example plugins. Normal operation does not contact external services. Building initially downloads Cargo dependencies; the lockfile is checked in.
|
||||
|
||||
```sh
|
||||
export PATH="$HOME/.cargo/bin:$PATH"
|
||||
test -f config.json || cp config.example.json config.json
|
||||
cargo run --locked -- --config config.json serve
|
||||
```
|
||||
|
||||
Open **http://127.0.0.1:8080**. The UI includes filesystem or streamed single-file upload imports, snapshot/asset browsing, safe media previews, text/hex inspection, file comparisons, plugin execution, function comparisons, downloads, and polling run status. It uses a Bootstrap-era utility layout with local CSS/JavaScript and no frontend build requirement.
|
||||
|
||||
Edit `config.json` before using real data:
|
||||
|
||||
- `archive`: authoritative local archive directory, shared by all games.
|
||||
- `workspace`: a separate disposable directory; an existing tmpfs path can be selected manually.
|
||||
- `import_roots`: existing server directories the HTTP API may import from.
|
||||
- `workspace_bytes`: default 100 GiB; capture is bounded, and plugins receive a workspace allowance. Plugin usage is polled, so this is not a kernel-enforced quota.
|
||||
- `bind`: use your server's LAN address or `0.0.0.0:8080` for LAN access. There is no authentication in this slice.
|
||||
- `plugins`: local command argument arrays and versioned settings. Use absolute paths for plugin scripts. The example uses this workspace's path; change it if you move the project.
|
||||
|
||||
The example config allows `samples/`. Import `samples/demo-v2` as version 2, followed by `samples/demo-v1` as version 1, to exercise out-of-order history. These are synthetic files, not game assets.
|
||||
|
||||
For large imports, build with `cargo build --release --locked` and run `./target/release/verstack --config config.json serve`. The default development build is useful for testing but substantially slower at chunking.
|
||||
|
||||
The CLI works without the service. Stop the service before using the mutating CLI: an instance ownership lock prevents another process from accessing the same archive concurrently.
|
||||
|
||||
```sh
|
||||
cargo run --locked -- import samples/demo-v2 --repository Demo --version 2
|
||||
cargo run --locked -- list
|
||||
cargo run --locked -- verify SNAPSHOT_ID
|
||||
cargo run --locked -- restore SNAPSHOT_ID /tmp/restored-demo
|
||||
cargo run --locked -- compare EARLIER_SNAPSHOT LATER_SNAPSHOT
|
||||
cargo run --locked -- process INPUT_SNAPSHOT zip-extract
|
||||
cargo run --locked -- functions BEFORE_SNAPSHOT BEFORE_FUNCTIONS_PATH AFTER_SNAPSHOT AFTER_FUNCTIONS_PATH
|
||||
```
|
||||
|
||||
Restore requires a destination that does not already exist. Exact file bytes, directories, and symlinks are restored. File permission bits are restored without setuid/setgid. Timestamps and original modes are cataloged; timestamp, directory permission, owner, xattr, ACL, hardlink, and non-UTF8 filename restoration are not implemented. Unsupported filenames and special files cause an explicit failed import, not silent omission.
|
||||
|
||||
## Implemented preservation model
|
||||
|
||||
An import first captures its inputs into a bounded workspace, then archives them through Rustic, reads back and hashes retained files, and finally publishes a durable immutable manifest. An extractor creates another snapshot with `parent` and `run` links; the earlier snapshot is never rewritten. Run states include running, complete, failed, and interrupted. Reopening reconciles an interrupted final run update against already-published manifests.
|
||||
|
||||
Artifact identity is `blake3:<hex>` over uncompressed bytes. This is a framework identity, **not a change to restic's storage algorithms**. Rustic supplies its compatible content-defined chunks, compression, authenticated encryption, and packs. Backend indexes are cached in memory and refreshed after commits. Artifact range reads avoid restoring a full build.
|
||||
|
||||
The first catalog reads durable JSON records directly. There is no SQLite index yet. Release metadata includes repository, version, edition, generation, and an optional explicit release date; import time is separate. Generation is unknown unless supplied as metadata. Updating a plugin produces a new revision.
|
||||
|
||||
All original inputs currently remain archived. Filesystem originals are never deleted. Temporary uploaded inputs are removed after processing; their captured original bytes remain in the archive on success. Cleanup based on extracted-content guarantees is deferred until real extractor coverage is validated.
|
||||
|
||||
## Ghidra and SPIKE adapters
|
||||
|
||||
See [plugin setup and protocol](docs/plugins.md). Included adapters:
|
||||
|
||||
- ZIP: exact regular-entry extraction with duplicate/path validation, opaque-parent retention, and expansion budgeting.
|
||||
- SPIKE: wrapper for a SHA-256-pinned local `bdash/spike-spk` executable, including first split-package parts. A separate `spike-probe` identifies wrapper signatures and required credential references. A separate `spike3-unpack` plugin decrypts supported LUKS2/ext4 update wrappers; `spike3-extract` uses the pinned type-4 parser extension. Generation inference and decoding proprietary inner asset formats remain incomplete.
|
||||
- Ghidra: pinned local headless installation, ELF/PE selection, function inventories and exact body hashes, `.gzf` export, and a second headless invocation requiring a successful reopen receipt. Export and reopen have been tested with Ghidra 12.1.3 on synthetic x86-64 and a real ARM game executable.
|
||||
|
||||
Function comparison is available in the CLI/API and through **Compare functions** on an analysis snapshot's `functions.json` asset. It accepts only unique exact bodies of at least 32 bytes, excluding thunks, with compatible profiles. Executable selection paths can differ; all other recorded analysis settings must agree. It explains unmatched functions without labeling them publisher additions. The WebUI offers filtering, pages of 100 results, and saved immutable comparison reports; the CLI saves reports with `functions ... --save`. BSim, Version Tracking, structural matching, global symbol propagation, strings/import indexing, and decompilation-text indexing remain next steps.
|
||||
|
||||
## Verification and recovery
|
||||
|
||||
```sh
|
||||
cargo test --locked
|
||||
cargo clippy --locked --all-targets -- -D warnings
|
||||
cargo run --locked --example storage_spike
|
||||
python3 -m unittest discover -s tests -p 'test_*.py'
|
||||
```
|
||||
|
||||
An optional Playwright browser smoke test is in `tests/ui.mjs`. Start a server with disposable data on port 18765, install Playwright and its browser dependencies separately, and run `node tests/ui.mjs`. `VERSTACK_TEST_URL` changes the target; `VERSTACK_PLAYWRIGHT` can point to a separate Playwright installation. This test imports synthetic files into the target archive.
|
||||
|
||||
The synthetic storage spike imports a 16 MiB binary, an identical copy under another game, and a version with a 4 KiB insertion. It verifies restored bytes and reports introduced chunks, packed bytes, and timings. It is not a prediction of savings for the 2.4 TB game collection.
|
||||
|
||||
Back up **the entire archive directory**, especially `archive-key.json`, `store/`, `snapshots/`, and `runs/`. The archive key is generated locally with restrictive file permissions and is required to read encrypted storage. It is separate from package-decryption keys. Domain metadata is not encrypted. Missing chunks are detectable but cannot be recreated without another copy.
|
||||
|
||||
Interrupted imports may leave unreferenced backend snapshots/packs and temporary directories. They are not published as complete framework snapshots. Do not run independent Rustic pruning against this archive: framework-aware retention/GC is not implemented. Failed-plugin workspaces normally disappear; after an abrupt host termination, remove abandoned workspace directories only while the instance and its child tools are stopped.
|
||||
|
||||
## Current scope and next work
|
||||
|
||||
See [architecture and remaining milestones](docs/architecture.md). The next acceptance tests are inner asset-pack extraction and a related release comparison. The real pilot establishes one package/executable path; it does not establish compatibility with every SPIKE generation, power-loss recovery, or 2.4 TB scale.
|
||||
|
||||
No project license has been selected yet; GPL is the stated preference under consideration. Dependency licenses remain their authors' licenses.
|
||||
|
||||
### Game library and media browser
|
||||
|
||||
The main screen groups archive snapshots by game. Select a game, then a version
|
||||
(edition and generation remain separate), to browse artifacts from every import
|
||||
and processing output for that release. Version links survive reload and browser
|
||||
back/forward. The output filter narrows the browser to one job's snapshot; analysis
|
||||
tools have a separate input selector. Processing history in a version is scoped
|
||||
to its outputs and input snapshot IDs, including failed jobs.
|
||||
|
||||
The default media gallery serves 24 artifacts per page, with image enlargement,
|
||||
native audio/video controls, path search, media filters, and original downloads.
|
||||
Use All files or Other files for reports, text/hex inspection, and function
|
||||
comparison. Preview failures show a download fallback. Browser-supported image,
|
||||
audio, and video formats are previewed directly; proprietary `.asset`/`.radium`
|
||||
containers and unsupported codecs still need a decoder/transcoding stage.
|
||||
|
||||
`GET /api/library` omits entry manifests. `GET /api/artifacts` filters by
|
||||
`repository`, `version`, `edition`, `generation`, optional `source`, `search`,
|
||||
`kind` (`media`, `image`, `audio`, `video`, `file`, `all`), and zero-based `page`.
|
||||
Responses contain `items`, `total`, `page`, and `page_size` (24). Pagination is
|
||||
server-side, though the server currently scans the archive manifests per query;
|
||||
a persistent artifact index is a future scaling improvement.
|
||||
|
||||
Media uses the existing bounded HTTP streaming and byte-range endpoint, allowing
|
||||
native seeking and cancellation. Images load lazily and audio/video use
|
||||
`preload="none"`. No socket/SSE media transport or transcoded thumbnail cache is
|
||||
introduced: images are still original-resolution assets, so individual large
|
||||
images can be expensive. Job progress continues to poll every two seconds.
|
||||
|
||||
Browser regressions: `tests/ui-functions.mjs` and `tests/ui-media.mjs` use isolated
|
||||
fixtures (including actual WAV/WebM playback). `tests/ui.mjs` requires a disposable
|
||||
server and imports synthetic files. `tests/ui-gallery.mjs` is a read-only pilot
|
||||
check against the current Pokémon/Game of Thrones archive on port 8080, covering
|
||||
311 images, navigation, paging, previews, empty search, and mobile width.
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"archive": "./data/archive",
|
||||
"workspace": "./data/workspace",
|
||||
"import_roots": ["./samples"],
|
||||
"workspace_bytes": 107374182400,
|
||||
"bind": "127.0.0.1:8080",
|
||||
"plugins": {
|
||||
"zip-extract": {
|
||||
"command": ["python3", "/home/jordan/verstack/plugins/zip_extract.py"],
|
||||
"version": "zip-extract/1",
|
||||
"settings": {},
|
||||
"timeout_seconds": 7200
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
# Architecture and implementation status
|
||||
|
||||
## Decisions carried forward
|
||||
|
||||
- One authoritative Linux instance with shared physical storage across logical game repositories; no clustering or mandatory cloud services.
|
||||
- Immutable original snapshots and separate extraction/analysis revisions; out-of-order releases work without predecessor reconstruction dependencies.
|
||||
- Storage reduction first, native/function research second, asset-history browsing third.
|
||||
- Exact executable and asset bytes are authoritative; decoded representations and research exports are separate artifacts.
|
||||
- Trusted executable plugins in any suitable language. Core operations do not require HTTP; the UI consumes the same Rust library through Axum.
|
||||
- Stern SPIKE packages are the pilot, with structure-based identification needed for all three generations. Game version, Pro/Premium/LE edition, and generation are separate metadata.
|
||||
- Ghidra is a local optional tool installation, with retained per-program GZF exports. AI assistance is optional and not present in the initial implementation.
|
||||
- Retain releases indefinitely until an explicit deletion facility is implemented. No source deletion or GC in this development slice.
|
||||
- Local disk persistence, configurable filesystem workspaces, 100 GiB starting budget, no authentication requirement for initial trusted-LAN use.
|
||||
|
||||
## Module boundaries
|
||||
|
||||
`model` defines versioned records. `archive` owns import, publication, verification, restoration, and file comparison. `storage::ArtifactStorage` isolates backend details. `plugins` manages external steps and validates their output. `analysis` interprets retained function facts. `http` and the CLI are adapters. `web/` is a locally served static client.
|
||||
|
||||
The current logical repository is the `Release.repository` field; standalone repository/collection records, annotations, and explicit release relationships need a later schema. JSON manifests and runs are authoritative durable records. A future SQLite search index must be rebuildable from those records.
|
||||
|
||||
Publication order: captured input → backend snapshot → read-back file verification → immutable domain manifest → terminal run update. Reopening reconciles a crash between the last two steps. A failure before domain publication leaves an incomplete run and may leave backend content for later reclamation. Source mutation checks catch ordinary concurrent file changes; imports are not atomic filesystem snapshots of a live-changing directory tree.
|
||||
|
||||
The backend remains restic-compatible. Framework BLAKE3 IDs do not replace Rustic's internal content IDs or chunker. Function matches are evidence relationships and never storage equality. The current exact-body function comparison is intentionally a baseline for later Version Tracking/BSim adapters, not a replacement for them.
|
||||
|
||||
## What the storage feasibility spike establishes
|
||||
|
||||
The runnable synthetic spike exercises independent snapshot reads, cross-game exact reuse, content-defined reuse after an insertion, whole-file identities, compressed backing storage, and verified restoration. Integration tests also cover damaged packs, bounded import failures, process ownership, out-of-order versions, and plugin revisions.
|
||||
|
||||
It does not establish performance at 2.4 TB, startup/index RSS, media-heavy compressibility, concurrency capacity, recovery under actual power loss, network filesystem suitability, or safe retention. Keep Rustic behind the interface while testing those requirements. A custom store has not been justified by current evidence.
|
||||
|
||||
## Remaining acceptance milestones
|
||||
|
||||
1. **Real package support:** the local corpus, SPKS extraction, and structural wrapper probe have been exercised. LUKS2/ext4 update extraction has passed the Pokémon LE pilot. Implement generation evidence and remaining proprietary asset decoders. Keep README evidence distinct from inferred metadata and component versions.
|
||||
2. **Real Ghidra validation:** x86/ARM analysis and GZF reopening pass. Exact comparison reports now retain both source inventories and their release metadata. Check known structural function changes. Add selected strings/imports and decompiled text, then Version Tracking and global BSim feasibility measurements.
|
||||
3. **Workflow engine:** versioned DAG definitions, resolved dependency identities, idempotent cache keys, durable queued jobs, retry policy, resource reservations, progress events, and a UI workflow editor. Current plugins are explicit single steps; HTTP requests wait for their completion while run status is separately readable.
|
||||
4. **Scale and browsing:** resumable/bundled uploads, server-path browser, paginated catalog queries, reusable SQLite indexes, richer byte/script/config comparisons, timeline-neighbor selection, preview cache, and accurate instance-wide physical/workspace statistics. Current listing loads JSON records directly.
|
||||
5. **Retention and recovery:** coverage-based source cleanup, pins, cross-project reachability, safe GC, backup/restore drills, format migrations, annotations, and project-only exports. No pruning/reset endpoint exists yet.
|
||||
|
||||
Before source cleanup can be enabled, prove that the retained outputs cover the selected preservation policy. A plugin's success status and byte verification alone do not establish container coverage.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,15 @@
|
||||
# Function comparison reports v2
|
||||
|
||||
The core compares retained function inventories through the Rust API, `functions BASELINE BASELINE_PATH TARGET TARGET_PATH`, or `GET /api/functions/compare` with the same four parameter names (`before`, `before_path`, `after`, `after_path`). Baseline and target identify selection order; they do not establish publisher chronology.
|
||||
|
||||
Schema 2 / method `exact-functions/2` matches unique exact SHA-256 body hashes of equal size (at least 32 bytes), excluding thunks and unavailable hashes. Names and addresses are never matching evidence. Duplicate addresses invalidate an inventory. Ambiguous body hashes remain unmatched. Address sets avoid a quadratic scan when assembling unmatched results.
|
||||
|
||||
The report includes both unmatched inventories, target-side reasons (`no_exact_body_match`, `ambiguous_exact_body`, `thunk_excluded`, `below_minimum_size`, `body_hash_unavailable`), and both source snapshots, function-facts artifact identities, paths, analysis run IDs, and release metadata. Edition and repository differences are disclosed. A function match does not establish equal behavior, data references, or the provenance of a feature. No names are transferred.
|
||||
|
||||
For schema-1 function facts, compatibility requires equal language, compiler, Ghidra version, and settings except the documented executable-selection field `paths`. Policy `schema1_settings_except_paths_v1` is recorded in each report. Every unknown future setting remains significant. This changes comparison eligibility, never retained analysis facts. Tool-version changes still require compatible new analyses.
|
||||
|
||||
Use `--save` on the CLI command, or `POST /api/functions/compare` with the four fields in a JSON body, to retain `function-comparison.json` through the normal verified publication path. POST requires `X-Verstack-Client: 1`. A new processing run records both input artifacts and snapshot/path selectors. The derived snapshot uses the target as its parent; the report retains the baseline relationship too. Saved reports can be restored or downloaded independently and do not replace previous reports. Future garbage collection must follow both input relationships.
|
||||
|
||||
The WebUI initially displays unmatched target functions and supports baseline unmatched functions, exact matches, filtering, and pages of 100 rows. **Save report to archive** preserves a computed report; **View report** reopens a retained report without recomputing it. `tests/ui-functions.mjs` tests this flow using synthetic intercepted responses and no listener.
|
||||
|
||||
BSim, structural matching, global function search, and automatic symbol propagation are still pending.
|
||||
@@ -0,0 +1,118 @@
|
||||
# Local SPIKE / Ghidra pilot
|
||||
|
||||
Validated on 2026-09-11 in `/home/jordan/verstack`. Firmware sources are under `/srv/firmware/images/stern_game_code`; no source files were changed or removed.
|
||||
|
||||
## Installed tools and configuration
|
||||
|
||||
- Ghidra 12.1.3: `tools/ghidra_12.1.3_PUBLIC`.
|
||||
- Temurin JDK 21.0.12.1+1: `tools/jdk-21.0.12.1+1`.
|
||||
- Launch GUI: `./tools/bin/ghidra` (requires a graphical display).
|
||||
- Launch headless: `./tools/bin/ghidra-headless`.
|
||||
- SPK extraction: `tools/bin/spike-spk`, built from upstream commit `63c5d9a527a4431086cfa828d72fe71fb343b9f7`.
|
||||
|
||||
Official download URLs and checked SHA-256 digests are in [toolchain.lock.json](toolchain.lock.json). `python3 scripts/install_local_tools.py` installs Ghidra and Java without modifying system packages. The SPIKE executable was separately built with Cargo from the pinned source; its built digest is also recorded. Tool binaries/downloads and local `config.json` are ignored by source control.
|
||||
|
||||
`config.json` points at the firmware source root, local tools, and the shared archive in `data/archive`. It registers `zip-extract`, `spike-probe`, `spike-extract`, and `ghidra`. Start the UI with `./target/release/verstack serve` and open `http://127.0.0.1:8080`. No service is installed or started automatically.
|
||||
|
||||
This execution environment reports 16 GiB RAM. The pilot uses disk workspaces, two Ghidra analysis CPUs, a 2 GiB JVM heap, and a 100 GiB workspace limit. The configuration can be raised on a larger machine. The limit is polled, not a filesystem quota.
|
||||
|
||||
## Corpus inventory and keys to supply
|
||||
|
||||
The supplied directory contains 252 files totaling 304,821,524,946 bytes (about 305 GB). Its 73 SPK downloads comprise 17 direct packages (16 SPKS, one gzip wrapper) and 56 ZIPs (48 containing SquashFS-first split packages, eight containing LUKS2-first split packages). Legacy SAM/ROM inputs and supporting documents are also present. This is a header inventory, not proof of complete extraction compatibility. [Full inventory](corpus-inventory.json).
|
||||
|
||||
The following eight encrypted containers need a LUKS unlocking keyfile or passphrase. Headers expose keyslot 0 and AES-XTS-plain64 with 4096-byte sectors. They do not reveal the credential, or prove that one key unlocks every title.
|
||||
|
||||
| Download | LUKS UUID |
|
||||
| --- | --- |
|
||||
| `pokemon_le-0_85_0_spike3.spk.zip` | `5b22533c-7ee6-4e7c-8e9b-abb2392be418` |
|
||||
| `pokemon_pro-0_85_0_spike3.spk.zip` | `6beaec1c-b400-423e-aa75-143a01ee3772` |
|
||||
| `star_wars_2025_le-0_97_0.spk.zip` | `0d86aaf6-3d98-42e3-afdf-5f74ddafd83d` |
|
||||
| `star_wars_2025_pro-0_97_0.spk.zip` | `9536b5fa-6e29-4ba9-85d2-14314fce911f` |
|
||||
| `star_wars_elg-1_10_0_spike3.spk.zip` | `3a85d189-c46c-4908-9eaf-98999b276347` |
|
||||
| `transformers_mtmte_le-0_90_0_spike3.spk.zip` | `80fd253e-bb24-4d7e-8818-4656cebad5eb` |
|
||||
| `transformers_mtmte_pro-0_90_0_spike3.spk.zip` | `e35e5c9e-6626-441b-8ace-81ddedaa2af8` |
|
||||
| `walking_dead_remastered_le-0_93_0.spk.zip` | `6b4aa8a3-d5d9-4d20-bbd1-96156de0c2a3` |
|
||||
|
||||
Provide **private keyfile paths, not key values in chat**. The five title families are Pokémon, Star Wars 2025, Star Wars ELG, Transformers MTMTE, and Walking Dead Remastered. Keep secret contents outside plugin settings and manifests: settings are durable public provenance. [Machine-readable key requirements](spike3-key-requirements.json).
|
||||
|
||||
`spike-probe` emits the non-secret requirement identifier `stern.spike3.luks`. It detects these wrappers; an adapter to unlock and decode them is still pending. Supplying keys alone does not yet enable LUKS extraction. No additional credential was needed for the older SPKS pilot. Firmware signing private keys are not needed for read-only extraction. No additional encryption requirement has been established for the preserved inner `image.bin` format.
|
||||
|
||||
## Real package and storage results
|
||||
|
||||
Input: `GOT-1_37_0.spk`, 801,626,655 bytes. The pilot labels it Game of Thrones / 1.37.0 / Pro / generation 1; edition and generation are supplied catalog labels, not independently verified automatic detection.
|
||||
|
||||
| Stage | Snapshot | Logical bytes | New chunk bytes | New packed bytes |
|
||||
| --- | --- | ---: | ---: | ---: |
|
||||
| Original import | `a9dc4bef-4bd9-4ffd-a8aa-56d55889c07e` | 801,626,655 | 797,357,229 | 735,748,176 |
|
||||
| Package extraction | `e246f1eb-8cbb-4281-9d3c-8b1b47fad5a3` | 801,623,588 | 12,640,199 | 4,929,676 |
|
||||
|
||||
Extraction produced 49 manifest entries, including 32 regular files. The core verified read-back artifact hashes before publication and restored the extracted snapshot to `data/validation/got-restored`. The original and extracted snapshots remain independently addressable. New packed bytes are backend per-commit metrics; they exclude some catalog/index overhead and are not total directory usage. These results show reuse between one package and its extraction, not cross-version corpus savings.
|
||||
|
||||
The main executable is `package-0000/GOT-1_37_0/GOT/GOT/game` (6,187,348 bytes), a symbol-bearing, statically linked 32-bit little-endian ARM ELF. Its SHA-256 is `637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828`. Ghidra selected `ARM:LE:32:v8`; that is its analysis language, not proof of the physical CPU generation. Imported executables were not run.
|
||||
|
||||
The main `image.bin` (784,411,100 bytes) remains opaque. Package extraction does **not** yet deliver individually browsable sounds/images/scripts from that container. Binaries, firmware, and extracted files can be browsed/downloaded now.
|
||||
|
||||
## Native analysis validation
|
||||
|
||||
A generated x86-64 ELF and the real ARM game both completed headless analysis, exported a packed program database, and reopened that database in a second headless invocation with matching input identity. Direct ARM output contains 11,549 non-external function inventory records; the reopened database reports 11,550 total functions including external functions. Analysis did not time out. Loader warnings, including unsupported thread-local symbols, remain relevant limitations; successful export is not proof of perfect code recovery.
|
||||
|
||||
The pilot fixed external/uninitialized function-body handling, GZF export transaction handling, local Java runtime/cache selection, and Ghidra's rejection of dot-prefixed workspace path elements. Failed jobs remain recorded and leave their parent snapshots intact. Structured bounded failure details and successful Ghidra logs are now retained.
|
||||
|
||||
Function matching currently covers unique exact bodies of at least 32 bytes, excluding thunks and unavailable hashes. BSim/Version Tracking and broader structural matching remain unimplemented. No names are transferred automatically.
|
||||
|
||||
## Completed archive analysis
|
||||
|
||||
The core `ghidra` operation completed as run `289ad050-5ce1-4ddc-8024-9285c6cccbaa`, publishing derived snapshot `db17ab15-cfaf-40cb-8ac1-dbb19e29b497` with extraction snapshot `e246f1eb-8cbb-4281-9d3c-8b1b47fad5a3` as parent. It retains five files: function facts, `.gzf`, reopen receipt, and both headless logs. Logical output is 28,233,847 bytes and newly packed storage is 25,546,814 bytes. A separate CLI verification passed, followed by restoration to `data/exports/game-of-thrones-1.37.0/`.
|
||||
|
||||
The program database is `data/exports/game-of-thrones-1.37.0/637acf6d6ff171def2c6e75437534ece8ea4017959eccd616829ccd621f97828/program.gzf`. It is also downloadable from the analysis snapshot in the WebUI. Restored files are disposable copies of the retained archive artifacts.
|
||||
|
||||
## Related-build comparison: Game of Thrones Pro / LE 1.37.0
|
||||
|
||||
The second real input was `GOT_LE-1_37_0.spk`, imported as snapshot `8e210c4c-d2d5-4bb8-967b-9aeacf021e01`, then extracted as `c9adc8f4-5505-4f71-a5f2-63f863e06e80`. Both publisher packages are version 1.37.0. This is an edition comparison, not an earlier/later release experiment.
|
||||
|
||||
| Stage | Logical bytes | New chunk bytes | New packed bytes |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| LE original | 801,706,834 | 7,677,126 | 2,889,302 |
|
||||
| LE extraction | 801,703,734 | 3,508,037 | 1,545,623 |
|
||||
|
||||
Of the LE extraction's 32 regular files, 30 have artifact identities already present in the Pro extraction, totaling 795,436,059 bytes. The two commits introduced 4,434,925 packed bytes, excluding Ghidra output and catalog/index overhead. These are incremental measurements against the populated archive; they do not predict savings on other games. Firmware source files remain untouched.
|
||||
|
||||
LE Ghidra analysis snapshot `19e0e553-0d2c-4d7e-a674-2ade38d87798` completed and reopened its `.gzf`. Its outputs were restored to `data/exports/game-of-thrones-le-1.37.0/`.
|
||||
|
||||
[Comparison report format](function-comparison.md) `exact-functions/2` found 5,526 unique exact body matches between 11,549 Pro and 11,724 LE inventory entries. There are 6,023 unmatched baseline and 6,198 unmatched target entries. Target reasons: 3,310 below the 32-byte minimum, 48 excluded thunks, 187 ambiguous exact bodies, and 2,653 with no exact body match. These counts are not counts of developer-added functions. Relocation and compilation changes can defeat exact matching; structural matching remains future work.
|
||||
|
||||
The comparison took approximately 2.25 seconds including opening the archive and reading inventories. It was saved as immutable derived snapshot `045e7e69-84fa-4af2-ac5a-993b8b807e70`, then separately verified and restored to `data/exports/got-pro-vs-le-1.37.0/function-comparison.json`. Both input snapshot/artifact/run identities and release labels are retained in the report. Existing analysis outputs were not rewritten.
|
||||
|
||||
## LAN and Pokémon SPIKE 3 test
|
||||
|
||||
The user approved full unauthenticated LAN access. The service listens on `0.0.0.0:8080`, reachable on this host at `http://172.16.0.87:8080`. It is a running process, not an installed boot service.
|
||||
|
||||
Pokémon LE 0.85.0 original ZIP is archived as snapshot `bf7e6f96-8022-4c97-be30-172078ec5455`. Both LE and Pro LUKS2 headers were tested with the credential referenced by `/srv/firmware/spike3_key.txt`. The supplied file is a sixteen-word hexadecimal representation; raw text, stripped text, packed little/big-endian words, and concatenated hexadecimal text all failed unlocking. The same checking code correctly accepted a generated fixture's known credential and rejected a wrong one. Key contents were not logged, archived, or modified.
|
||||
|
||||
The local `luks-check` plugin records a diagnostic derived snapshot for LE. No Pokémon inner SPK, assets, executable, or Ghidra output has been extracted yet. Correct credential/derivation information is required before decryption can proceed; this result does not establish that a particular untested key derivation would fail.
|
||||
|
||||
## Pokémon SD-card image key check
|
||||
|
||||
The user supplied the Pokémon LE 0.85.0 secure SD-card ZIP at the Backblaze gamecode download URL recorded in `data/validation/pokemon-sd/report.json`. HTTP range requests confirmed a 3,430,070,310-byte ZIP containing one 61,924,705,792-byte raw image. Streaming reads avoided materializing the whole raw image; headers were retained for reproducibility. The complete ZIP CRC/raw image was not verified.
|
||||
|
||||
The partition table contains a FAT boot partition, two primary LUKS2 partitions, and an extended partition with two logical LUKS2 partitions. All four encrypted partition headers were tested, at byte offsets 67,109,376; 696,254,976; 721,421,312; and 26,491,225,600. The supplied key failed raw, stripped-text, little/big-endian word, and concatenated-hex keyslot tests on each. Direct volume-key digest tests also found no match in the tested byte representations; that verification method passed a generated known-volume-key fixture.
|
||||
|
||||
This does not establish whether a separate key derivation or different historical key would work. The next useful input is the exact command/tool and key conversion previously used to unlock an SD image. No decrypted Pokémon content was extracted, no key contents were logged, and the user's keyfile was unchanged. The LAN service remains available on port 8080.
|
||||
|
||||
## Correct key and successful SPIKE 3 extraction
|
||||
|
||||
The corrected credential reference `/srv/firmware/luks.key` was accepted by all eight local encrypted update-container headers (Pokémon LE/Pro, Star Wars 2025 LE/Pro, Star Wars ELG, Transformers MTMTE LE/Pro, Walking Dead Remastered). [Header validation results](spike3-validation.json). This is not full extraction validation for all eight titles. The same credential did not unlock any of the four tested SD-card partitions; update and SD-image results remain separate.
|
||||
|
||||
Pokémon LE 0.85.0 completed the HTTP pipeline: original ZIP → LUKS2/ext4 wrapper extraction → inner SPK extraction. Every published snapshot passed read-back artifact verification. The original parser rejected numeric package type 4; the separately pinned `spike3-extract` tool adds only that variant and retains all upstream MD5/HMAC checks. Those checks passed for all 708 package files. The previous failed run remains recorded; its input snapshot was retained intact.
|
||||
|
||||
| Stage | Snapshot | Logical bytes | Newly packed bytes |
|
||||
| --- | --- | ---: | ---: |
|
||||
| Original ZIP | `bf7e6f96-8022-4c97-be30-172078ec5455` | 2,405,286,702 | 2,405,088,162 |
|
||||
| Decrypted wrapper output | `35015e3b-f151-451a-a329-97e1ea49219e` | 2,319,247,828 | 1,879,769,642 |
|
||||
| SPK extraction | `e632b195-3a5c-419b-9e5e-45d17ce3cdde` | 2,319,134,644 | 339,687,643 |
|
||||
|
||||
The final snapshot includes 708 package files plus `package-evidence.json`: 311 PNGs, 278 `.asset` files, 51 `.radium` files, 11 TTF fonts, and other content. Proprietary asset decoding is still incomplete; native PNGs can be previewed now. The main game is a 53,037,544-byte stripped, dynamically linked AArch64 ELF. A Pokémon Ghidra analysis has not yet been run.
|
||||
|
||||
The live browser successfully previewed an extracted Pokémon PNG. An HTTP-downloaded PNG also matched the independently extracted file exactly. The service remains at `http://172.16.0.87:8080`. New plugin configurations use the corrected key reference; no key contents were logged or archived. The earlier rejection reports remain historical records of the earlier key.
|
||||
|
||||
The local parser source is `tools/src/bdash-spike-spk-63c5d9a-type4`, built with the upstream Cargo lockfile plus `patches/spike-spk-type4.patch`. Source commit, patch digest, and executable digest are recorded in `docs/toolchain.lock.json`; the original parser binary remains available. New wrapper dependencies are libcryptsetup, Python cryptography, and read-only debugfs, without root or kernel mounts.
|
||||
+106
@@ -0,0 +1,106 @@
|
||||
# Local plugin protocol v1
|
||||
|
||||
Plugins are trusted local programs, not sandboxed extensions. They receive ordinary filesystem paths and inherit the host environment. The host owns catalog and archive writes. Do not launch imported game programs from the supplied adapters.
|
||||
|
||||
The configured command receives one additional argument: an absolute request JSON filename. Its working directory is a disposable workspace. Input/output bytes are never embedded in JSON.
|
||||
|
||||
```json
|
||||
{
|
||||
"protocol": 1,
|
||||
"input_dir": "/workspace/input",
|
||||
"output_dir": "/workspace/output",
|
||||
"result_file": "/workspace/result.json",
|
||||
"settings": {},
|
||||
"workspace_bytes": 107374182400
|
||||
}
|
||||
```
|
||||
|
||||
Write retained output files under `output_dir` and finish with a result file:
|
||||
|
||||
```json
|
||||
{
|
||||
"protocol": 1,
|
||||
"layer": "extracted",
|
||||
"coverage": "partial",
|
||||
"warnings": ["Nested custom archive retained as opaque content"],
|
||||
"files": ["game/main.elf", "assets/opaque.pack"]
|
||||
}
|
||||
```
|
||||
|
||||
`layer` is `extracted` or `derived`. Coverage is `complete`, `partial`, or `unknown`. Every regular output file must occur exactly once in `files`. Paths must be relative and free of traversal. Live output symlinks are rejected; represent them as metadata until a portable output-entry schema is added. An unknown field or unsupported protocol fails validation.
|
||||
|
||||
A successful exit is necessary but insufficient. The host validates the inventory, captures output bytes, archives them, and verifies read-back hashes before publishing an immutable child snapshot. Parent inputs remain retained even when coverage is declared complete. `coverage` describes the plugin operation, not permission to discard inputs.
|
||||
|
||||
The host records configured tool version, settings, input artifact identities, stage, output snapshot, and failure state. Version/build identities must be updated when plugin behavior changes. Automatic cache reuse and full dependency-build hashing are not yet implemented. A rerun always creates a new record; identical bytes still deduplicate.
|
||||
|
||||
**Settings are durable public provenance, not a secret store.** Put decryption keys in environment variables or private files referenced by non-secret identifiers in settings. No environment values are copied into manifests. The bundled SPIKE wrapper does not introduce new decryption support beyond the configured upstream tool. Plugin stdout/stderr are discarded by the host. On failure, a plugin may write `{"protocol":1,"error":"brief diagnostic"}` to its result file before exiting nonzero; the host records a bounded diagnostic. Successful Ghidra runs retain analysis/reopen logs. Plugins must keep secret values out of all diagnostics and retained outputs.
|
||||
|
||||
Input materialization uses less than half the configured workspace limit, reserving capacity for output and capture. The host polls workspace size and kills the process group on timeout/budget violation. CPU/RAM quotas, cgroups, automatic tmpfs placement, resumable steps, and shared worker scheduling are not implemented. Do not use this polling limit as a hard disk quota.
|
||||
|
||||
## ZIP
|
||||
|
||||
The example config registers `plugins/zip_extract.py`; Python's standard library is sufficient. It handles ZIP containers from any publisher, and reports non-ZIP supporting inputs as opaque in the retained parent. Paths are prefixed with `<input-name>.entries/` to avoid cross-package collisions. Nested extraction can be requested by running another plugin on the output snapshot.
|
||||
|
||||
## SPIKE
|
||||
|
||||
Build/install the [upstream spike-spk tool](https://github.com/bdash/spike-spk) separately and calculate its executable SHA-256. Its source supports `verify PATH` and `extract PATH --output DIR`, including first split-package parts; the wrapper delegates structure validation to it. The framework does not claim support for all three SPIKE generations.
|
||||
|
||||
Add a plugin config (replace paths and digest):
|
||||
|
||||
```json
|
||||
"spike-extract": {
|
||||
"command": ["python3", "/absolute/verstack/plugins/spike_extract.py"],
|
||||
"version": "spike-adapter/2+PINNED-UPSTREAM-COMMIT",
|
||||
"settings": {
|
||||
"tool": "/opt/spike-spk/bin/spike-spk",
|
||||
"expected_sha256": "SHA256-OF-LOCAL-EXECUTABLE"
|
||||
},
|
||||
"timeout_seconds": 7200
|
||||
}
|
||||
```
|
||||
|
||||
Import all split parts together as a directory. ZIP-wrapped packages can first use `zip-extract`, then `spike-extract`. Supporting files stay in parent snapshots. Symlinks emitted by the upstream tool are recorded in `package-evidence.json` rather than followed. This adapter has passed the [Game of Thrones pilot](local-pilot.md); other wrappers and titles still need testing. Upstream extraction validates checksums. `max_threads` defaults to 2; the upstream split reader may materialize a large inner package in RAM.
|
||||
|
||||
## Ghidra
|
||||
|
||||
Run `python3 scripts/install_local_tools.py` from the project root to install the pinned official Ghidra and JDK archives under `tools/`, with SHA-256 verification from `docs/toolchain.lock.json`. Downloading requires network access; normal analysis is local. The script is intended for Linux x86-64 and Python 3.11 or newer. Launchers are `tools/bin/ghidra` and `tools/bin/ghidra-headless`. Alternatively, configure an existing local distribution. The adapter uses [headless analysis](https://github.com/NationalSecurityAgency/ghidra/blob/master/Ghidra/RuntimeScripts/support/analyzeHeadlessREADME.md) and the documented [GZF exporter](https://ghidra.re/ghidra_docs/api/ghidra/app/util/exporter/GzfExporter.html). A `.gzf` represents one analyzed program, not an entire multi-program project.
|
||||
|
||||
```json
|
||||
"ghidra": {
|
||||
"command": ["python3", "/absolute/verstack/plugins/ghidra/analyze.py"],
|
||||
"version": "ghidra-adapter/2+LOCAL-GHIDRA-VERSION",
|
||||
"settings": {
|
||||
"ghidra_home": "/opt/ghidra",
|
||||
"java_home": "/opt/jdk-21",
|
||||
"max_heap": "2G",
|
||||
"expected_version": "EXACT-application.version-FROM-application.properties",
|
||||
"max_cpu": 2,
|
||||
"analysis_timeout_seconds": 3600
|
||||
},
|
||||
"timeout_seconds": 14400
|
||||
}
|
||||
```
|
||||
|
||||
An optional `paths` array selects logical executable paths. Otherwise ELF/PE magic selects candidates and Ghidra chooses the loader/processor. Generation is not used to guess architecture. Each unique selected binary produces `<sha256>/program.gzf`, `functions.json`, `reopened.json`, `analysis.log`, and `reopen.log`. Download the `.gzf` through the UI for local review.
|
||||
|
||||
Java user/cache/temp directories stay in the disposable workspace. External or uninitialized function bodies receive no exact-body hash and cannot establish an exact match.
|
||||
|
||||
The adapter fails on analysis timeout, input identity mismatch, exporter failure, or absence of the reopen receipt. It retains baseline outputs only; manual GZF reimport, enrichment, and automatic symbol propagation are future operations.
|
||||
|
||||
## LUKS credential validation
|
||||
|
||||
`plugins/luks_check.py` checks LUKS2 headers in files and ZIP-wrapped split packages through local `cryptsetup open --test-passphrase`. It does not mount filesystems or extract decrypted content. Configure exactly one `key_file` or `key_env` reference in settings, plus optional `cryptsetup` executable path. `key_encoding` defaults to `raw`; `auto` tests raw bytes, stripped text, and (for 0x-prefixed 32-bit word lists) little-endian bytes, big-endian bytes, and concatenated hexadecimal text.
|
||||
|
||||
Credentials are passed through stdin, never argv, logs, manifests, or temporary keyfiles. The retained `luks-key-check.json` records only the reference, header UUID, attempted encoding names, fixed status values, and `unlock_verified`. A completed diagnostic run can report failed unlocking; it does not imply successful extraction. `tool_error` is distinct from a rejected credential. The header workspace is limited to the first 16 MiB per candidate; layouts requiring more keyslot data need an extended adapter.
|
||||
|
||||
Validated using a generated LUKS2 fixture with correct/incorrect keys and the supplied Pokémon LE/Pro headers. This machine has cryptsetup 2.8.4. No additional system packages were needed.
|
||||
|
||||
## SPIKE 3 encrypted update extraction
|
||||
|
||||
`plugins/luks_extract.py` accepts one `input_path` within a snapshot: a ZIP containing a complete, consistently numbered split SPK set, or an assembled LUKS2 file. Configure exactly one raw `key_file` or `key_env` reference. It assembles parts in numeric order, obtains the validated volume key in memory through libcryptsetup, decrypts AES-XTS sectors into a disposable filesystem image, and uses read-only `debugfs` to recover the inner SPK. There are no kernel mounts, device-mapper changes, or keyfiles written to the workspace.
|
||||
|
||||
The first implementation supports a single LUKS2 AES-XTS-plain64 segment with 512- or 4096-byte sectors and an ext4 filesystem containing an SPK. Unsupported layouts fail explicitly. It checks a conservative four-times-container workspace allowance before assembly; the host continues polling total usage. Original inputs remain retained and filesystem metadata is not promised byte-identical restoration. XTS is not authenticated encryption: the following SPK stage must still verify its payload checksums/HMACs.
|
||||
|
||||
Local dependencies used in the pilot: libcryptsetup/cryptsetup 2.8.4, Python cryptography 46.0.5, and debugfs/e2fsprogs 1.47.2. These are optional plugin dependencies; generic core operations do not require them. The installed local configuration records these dependency versions and selects Pokémon LE. `input_path` must be changed (or another plugin configuration registered) for another package.
|
||||
|
||||
The standard SPIKE parser rejected observed system package type 4. A minimal local patch, `patches/spike-spk-type4.patch`, adds that numeric variant without disabling MD5 or HMAC verification. It is built as a separate executable and plugin registration so the existing pinned SPIKE 1/2 tool remains available.
|
||||
@@ -0,0 +1,122 @@
|
||||
[
|
||||
{
|
||||
"download": "pokemon_le-0_85_0_spike3.spk.zip",
|
||||
"entry": "pokemon_le-0_85_0_spike3.spk.002.000",
|
||||
"uuid": "5b22533c-7ee6-4e7c-8e9b-abb2392be418",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "pokemon_pro-0_85_0_spike3.spk.zip",
|
||||
"entry": "pokemon_pro-0_85_0_spike3.spk.002.000",
|
||||
"uuid": "6beaec1c-b400-423e-aa75-143a01ee3772",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "star_wars_2025_le-0_97_0.spk.zip",
|
||||
"entry": "star_wars_2025_le-0_97_0.spk.007.000",
|
||||
"uuid": "0d86aaf6-3d98-42e3-afdf-5f74ddafd83d",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "star_wars_2025_pro-0_97_0.spk.zip",
|
||||
"entry": "star_wars_2025_pro-0_97_0.spk.007.000",
|
||||
"uuid": "9536b5fa-6e29-4ba9-85d2-14314fce911f",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "star_wars_elg-1_10_0_spike3.spk.zip",
|
||||
"entry": "star_wars_elg-1_10_0_spike3.spk.002.000",
|
||||
"uuid": "3a85d189-c46c-4908-9eaf-98999b276347",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "transformers_mtmte_le-0_90_0_spike3.spk.zip",
|
||||
"entry": "transformers_mtmte_le-0_90_0_spike3.spk.004.000",
|
||||
"uuid": "80fd253e-bb24-4d7e-8818-4656cebad5eb",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "transformers_mtmte_pro-0_90_0_spike3.spk.zip",
|
||||
"entry": "transformers_mtmte_pro-0_90_0_spike3.spk.004.000",
|
||||
"uuid": "e35e5c9e-6626-441b-8ace-81ddedaa2af8",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"download": "walking_dead_remastered_le-0_93_0.spk.zip",
|
||||
"entry": "walking_dead_remastered_le-0_93_0.spk.002.000",
|
||||
"uuid": "6b4aa8a3-d5d9-4d20-bbd1-96156de0c2a3",
|
||||
"keyslots": [
|
||||
"0"
|
||||
],
|
||||
"segments": {
|
||||
"0": {
|
||||
"type": "crypt",
|
||||
"encryption": "aes-xts-plain64",
|
||||
"sector_size": 4096
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"credential_reference": "/srv/firmware/luks.key",
|
||||
"containers": [
|
||||
{
|
||||
"download": "pokemon_le-0_85_0_spike3.spk.zip",
|
||||
"uuid": "5b22533c-7ee6-4e7c-8e9b-abb2392be418",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "pokemon_pro-0_85_0_spike3.spk.zip",
|
||||
"uuid": "6beaec1c-b400-423e-aa75-143a01ee3772",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "star_wars_2025_le-0_97_0.spk.zip",
|
||||
"uuid": "0d86aaf6-3d98-42e3-afdf-5f74ddafd83d",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "star_wars_2025_pro-0_97_0.spk.zip",
|
||||
"uuid": "9536b5fa-6e29-4ba9-85d2-14314fce911f",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "star_wars_elg-1_10_0_spike3.spk.zip",
|
||||
"uuid": "3a85d189-c46c-4908-9eaf-98999b276347",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "transformers_mtmte_le-0_90_0_spike3.spk.zip",
|
||||
"uuid": "80fd253e-bb24-4d7e-8818-4656cebad5eb",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "transformers_mtmte_pro-0_90_0_spike3.spk.zip",
|
||||
"uuid": "e35e5c9e-6626-441b-8ace-81ddedaa2af8",
|
||||
"unlock_verified": true
|
||||
},
|
||||
{
|
||||
"download": "walking_dead_remastered_le-0_93_0.spk.zip",
|
||||
"uuid": "6b4aa8a3-d5d9-4d20-bbd1-96156de0c2a3",
|
||||
"unlock_verified": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
# Initial storage feasibility result
|
||||
|
||||
Measured with `cargo run --offline --example storage_spike` in the development profile on 2026-09-11. Temporary data was discarded after measurement. This synthetic corpus is intentionally incompressible except for its insertion and contains no game content.
|
||||
|
||||
| Import | Logical bytes | Newly introduced file-chunk bytes | New packed file-chunk bytes | Import + verification | Restore + verification |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Game A, version 2 | 16,777,216 | 16,777,216 | 16,777,998 | 33,971 ms | 49 ms |
|
||||
| Game B, identical binary | 16,777,216 | 0 | 0 | 23,789 ms | 51 ms |
|
||||
| Game A, older version, 4 KiB insertion | 16,781,312 | 4,954,745 | 4,950,871 | 31,098 ms | 47 ms |
|
||||
|
||||
Total archive file lengths, including backend metadata and domain records: **21,739,248 bytes**. All three reconstructed binaries compared equal to their respective inputs. The repeated binary introduced no file chunks; the insertion reused unaffected content without requiring an earlier snapshot for restoration.
|
||||
|
||||
These unoptimized-build timings are not deployment performance numbers. Measure an optimized build before setting throughput expectations. This spike does not measure peak RSS, temporary disk usage, filesystem allocation, index scaling, realistic compressed containers, extractor output, or Ghidra storage. Chunk boundaries are repository-dependent, so the exact introduced-byte count may differ on another run.
|
||||
|
||||
The result supports keeping the existing backend for the next real-corpus feasibility test. It does not justify claims about compression ratios or savings on the 2.4 TB SPIKE collection.
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"ghidra": {
|
||||
"version": "12.1.3",
|
||||
"url": "https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_12.1.3_build/ghidra_12.1.3_PUBLIC_20260817.zip",
|
||||
"sha256": "93a5d11a9ad510622acaaf908c556a7b9b764d338e78a7567f3689bf5081fd54"
|
||||
},
|
||||
"jdk": {
|
||||
"version": "jdk-21.0.12.1+1",
|
||||
"url": "https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.12.1%2B1/OpenJDK21U-jdk_x64_linux_hotspot_21.0.12.1_1.tar.gz",
|
||||
"sha256": "ce79869e1307ed8ee1e2baa86a412b1eb5b75d10a01006d788a6f968bcfaee94"
|
||||
},
|
||||
"spike_spk": {
|
||||
"commit": "63c5d9a527a4431086cfa828d72fe71fb343b9f7",
|
||||
"url": "https://github.com/bdash/spike-spk",
|
||||
"executable_sha256": "1e0647f170c566918b5a847eb5a473cb39b6c455a28acb1305a8c26a9ef8c831"
|
||||
},
|
||||
"spike_spk_type4": {
|
||||
"upstream_commit": "63c5d9a527a4431086cfa828d72fe71fb343b9f7",
|
||||
"patch": "patches/spike-spk-type4.patch",
|
||||
"patch_sha256": "2566dc17db0f27a05f16498056e6eea3ddf0abf03e637ed3e9817b0b5dbbd540",
|
||||
"executable_sha256": "887f8e86d458094809213e6d7fa2ecf26305ab2de0a66cf20c8bb2aeead940ba"
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 203 KiB |
@@ -0,0 +1,42 @@
|
||||
# Validation record
|
||||
|
||||
Development environment: Linux x86-64, Rust/Cargo 1.98.1, Python 3.14.4. Ghidra 12.1.3 and Temurin JDK 21 were installed locally; see the [real SPIKE pilot](local-pilot.md).
|
||||
|
||||
- Rust integration tests: 15 passed, including exact restoration, cross-game reuse, out-of-order imports, input retention on failed import, corrupt packs, missing backend configuration, interrupted-run reconciliation, ownership/path boundaries, plugin revisions, range reads, and conservative function comparison.
|
||||
- Python adapter tests: 12 passed, covering exact ZIP entries, opaque supporting content, unsafe paths, expansion limits, missing Ghidra, rejected Ghidra version mismatch, structural wrapper probing, and credential references.
|
||||
- `cargo clippy --all-targets -- -D warnings`: passed.
|
||||
- `cargo build --locked --offline`: passed using the downloaded dependency cache.
|
||||
- Browser smoke: Chromium/Playwright exercised upload, release-date retention, browsing, text inspection, download, verification, and comparison; no JavaScript errors. [UI screenshot](ui-preview.png).
|
||||
- [Synthetic storage feasibility measurement](storage-spike.md): all restored binaries matched their respective inputs; exact cross-game content introduced no new file chunks.
|
||||
|
||||
Browser tooling and missing browser shared libraries were downloaded into `/tmp`; no system packages or frontend runtime dependencies were added to the application. Test servers and archives were disposable.
|
||||
|
||||
Real-tool validation: the pinned SPIKE extractor processed Game of Thrones 1.37.0, and Ghidra exported/reopened synthetic x86-64 and real ARM program databases. The real ARM run also completed through the core plugin host, published an immutable derived snapshot, passed archive verification, and restored its five output files. Still unvalidated: other package families/wrappers, SD-card filesystem decryption, proprietary inner asset formats, real power-loss behavior, RAM workspaces, and corpus-scale performance/memory. Retention/GC is not implemented.
|
||||
|
||||
Comparison milestone: real Pro/LE ARM inventories produced a verified/restored immutable report. New tests cover differing executable selection paths, incompatible analysis options, unmatched reasons, duplicate addresses, both-source provenance, persistence after reopening, and the HTTP save route. `tests/ui-functions.mjs` passed in Chromium using synthetic intercepted responses (no archive listener): source labels, pagination, filtering, matches, saving, and reopening a report. Clippy passed after implementation.
|
||||
|
||||
LUKS checks: generated LUKS2 fixture accepted its known key and rejected a wrong key. Both Pokémon LE/Pro headers rejected five representations of the user-supplied key. Pokémon LE original import and a retained credential-check report were exercised through HTTP. That earlier credential was incorrect for these updates; see the corrected-key results below.
|
||||
|
||||
Corrected-key validation: `/srv/firmware/luks.key` unlocks all eight local SPIKE 3 update-container headers. It does not unlock the four tested Pokémon SD-card partitions. A new userspace LUKS2/ext4 wrapper plugin recovered Pokémon LE's SPK; a separately pinned numeric-type-4 extension to the upstream SPK parser verified MD5/HMAC and extracted 708 files. Python tests now also cover split-part ordering/completeness/budgets and retrieval of a known volume key through libcryptsetup.
|
||||
|
||||
Pokémon LE full pipeline passed through HTTP, with verified immutable wrapper and extracted snapshots. All 708 SPK payloads passed upstream MD5/HMAC verification. A live Chromium preview and byte-identical HTTP PNG download passed. Pokémon Ghidra and full extraction of other SPIKE 3 titles remain pending.
|
||||
|
||||
### 2026-09-11 grouped library and gallery
|
||||
|
||||
- `cargo test --locked --offline`: 16 integration tests passed, including new
|
||||
library response size, version/source isolation, media type, search, page bounds,
|
||||
and empty-result checks. Existing streaming byte-range tests remain green.
|
||||
- `tests/ui-functions.mjs`: source selection, comparison paging/filtering, saving,
|
||||
and reopening reports passed with the grouped navigation.
|
||||
- `tests/ui-media.mjs`: real browser decoding/playback of synthetic WAV and VP8 WebM,
|
||||
deferred preload, broken-image fallback, and media filtering passed.
|
||||
- `tests/ui.mjs`: import/upload, text preview, download, release date, verification,
|
||||
and file comparison passed against a disposable archive.
|
||||
- `tests/ui-gallery.mjs`: read-only current corpus checks cover two games, three
|
||||
edition/version groups, Pokémon's 311 PNGs, pagination, image decoding and
|
||||
enlargement, reload, search, all-files navigation, and mobile page width.
|
||||
|
||||
The original archive was used only for browsing. Native format support is not a
|
||||
claim that proprietary game media containers or every codec can play in browsers.
|
||||
Gallery images retain original bytes; thumbnail generation and a persistent
|
||||
artifact query index remain future performance work.
|
||||
@@ -0,0 +1,72 @@
|
||||
//! Reproducible synthetic feasibility measurement; not a Stern-corpus savings estimate.
|
||||
use anyhow::Result;
|
||||
use std::{collections::BTreeMap, fs, time::Instant};
|
||||
use verstack::{Archive, Config, Release};
|
||||
fn main() -> Result<()> {
|
||||
let temp = tempfile::tempdir()?;
|
||||
let input = temp.path().join("inputs");
|
||||
fs::create_dir(&input)?;
|
||||
let config = Config {
|
||||
archive: temp.path().join("archive"),
|
||||
workspace: temp.path().join("work"),
|
||||
import_roots: vec![input.clone()],
|
||||
workspace_bytes: 128 * 1024 * 1024,
|
||||
bind: "127.0.0.1:0".into(),
|
||||
plugins: BTreeMap::new(),
|
||||
};
|
||||
let mut state = 42u64;
|
||||
let bytes: Vec<u8> = (0..16 * 1024 * 1024)
|
||||
.map(|_| {
|
||||
state ^= state << 13;
|
||||
state ^= state >> 7;
|
||||
state ^= state << 17;
|
||||
state as u8
|
||||
})
|
||||
.collect();
|
||||
fs::write(input.join("binary"), &bytes)?;
|
||||
let a = Archive::open(config.clone())?;
|
||||
let mut measurements = vec![];
|
||||
for (game, version, changed) in [("A", "2", false), ("B", "1", false), ("A", "1", true)] {
|
||||
if changed {
|
||||
let mut changed = bytes[..2 * 1024 * 1024].to_vec();
|
||||
changed.extend_from_slice(&[0x7f; 4096]);
|
||||
changed.extend_from_slice(&bytes[2 * 1024 * 1024..]);
|
||||
fs::write(input.join("binary"), changed)?;
|
||||
}
|
||||
let started = Instant::now();
|
||||
let s = a.import(
|
||||
&input,
|
||||
Release {
|
||||
repository: game.into(),
|
||||
version: version.into(),
|
||||
edition: String::new(),
|
||||
generation: String::new(),
|
||||
released_at: None,
|
||||
},
|
||||
)?;
|
||||
let import_ms = started.elapsed().as_millis();
|
||||
let started = Instant::now();
|
||||
let dest = temp.path().join(format!("restore-{game}-{version}"));
|
||||
a.restore(&s.id, &dest)?;
|
||||
let restore_ms = started.elapsed().as_millis();
|
||||
anyhow::ensure!(
|
||||
fs::read(dest.join("binary"))? == fs::read(input.join("binary"))?,
|
||||
"restoration differs"
|
||||
);
|
||||
measurements.push(serde_json::json!({"game":game,"version":version,"logical_bytes":s.logical_bytes,"new_chunk_bytes":s.new_chunk_bytes,"new_packed_bytes":s.new_packed_bytes,"import_and_verify_ms":import_ms,"restore_and_verify_ms":restore_ms}));
|
||||
}
|
||||
let mut physical = 0u64;
|
||||
for e in walkdir::WalkDir::new(&config.archive) {
|
||||
let e = e?;
|
||||
if e.file_type().is_file() {
|
||||
physical += e.metadata()?.len();
|
||||
}
|
||||
}
|
||||
println!(
|
||||
"{}",
|
||||
serde_json::to_string_pretty(
|
||||
&serde_json::json!({"corpus":"synthetic 16 MiB high-entropy binary, identical cross-game copy, 4 KiB insertion at 2 MiB","backend":"rustic_core 0.13 / rustic_backend 0.7","measurements":measurements,"archive_file_bytes":physical,"limitations":"Does not measure RSS, peak temporary disk, block allocation, real SPIKE extraction, Ghidra, or corpus-scale indexes."})
|
||||
)?
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
--- a/src/chunks.rs
|
||||
+++ b/src/chunks.rs
|
||||
@@ -6,6 +6,8 @@
|
||||
pub enum PackageType {
|
||||
Spike1 = 1,
|
||||
Spike2 = 3,
|
||||
+ // Observed in the SPIKE 3 Pokemon system component; retain its numeric type.
|
||||
+ SystemType4 = 4,
|
||||
Game = 2,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
// @category Verstack
|
||||
import ghidra.app.util.headless.HeadlessScript;
|
||||
import ghidra.app.util.exporter.GzfExporter;
|
||||
import ghidra.program.model.listing.Function;
|
||||
import ghidra.program.model.address.AddressRange;
|
||||
import ghidra.framework.Application;
|
||||
import com.google.gson.*;
|
||||
import java.io.*;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.security.MessageDigest;
|
||||
import java.util.HexFormat;
|
||||
|
||||
public class ExportFacts extends HeadlessScript {
|
||||
@Override public void run() throws Exception {
|
||||
String[] args = getScriptArgs();
|
||||
File output = new File(args[0]);
|
||||
if (!args[1].equalsIgnoreCase(currentProgram.getExecutableSHA256()))
|
||||
throw new IOException("Input identity mismatch");
|
||||
JsonObject facts = new JsonObject();
|
||||
facts.addProperty("schema", 1);
|
||||
facts.addProperty("input_sha256", currentProgram.getExecutableSHA256());
|
||||
facts.addProperty("ghidra_version", Application.getApplicationVersion());
|
||||
facts.addProperty("language", currentProgram.getLanguageID().toString());
|
||||
facts.addProperty("compiler", currentProgram.getCompilerSpec().getCompilerSpecID().toString());
|
||||
facts.addProperty("analysis_timed_out", analysisTimeoutOccurred());
|
||||
JsonArray functions = new JsonArray();
|
||||
for (Function function : currentProgram.getFunctionManager().getFunctions(true)) {
|
||||
monitor.checkCancelled();
|
||||
JsonObject item = new JsonObject();
|
||||
item.addProperty("address", function.getEntryPoint().toString());
|
||||
item.addProperty("name", function.getName());
|
||||
item.addProperty("symbol_source", function.getSymbol().getSource().toString());
|
||||
item.addProperty("size", function.getBody().getNumAddresses());
|
||||
item.addProperty("thunk", function.isThunk());
|
||||
boolean hashable = !function.isExternal() && currentProgram.getMemory()
|
||||
.getAllInitializedAddressSet().contains(function.getBody());
|
||||
item.addProperty("external", function.isExternal());
|
||||
item.addProperty("body_status", hashable ? "exact" : "unavailable");
|
||||
// Hash exact function body ranges. This is evidence only, never storage identity.
|
||||
MessageDigest hash = MessageDigest.getInstance("SHA-256");
|
||||
if (hashable) for (AddressRange range : function.getBody().getAddressRanges(true)) {
|
||||
long left = range.getLength();
|
||||
var address = range.getMinAddress();
|
||||
while (left > 0) {
|
||||
byte[] buffer = new byte[(int)Math.min(left, 65536)];
|
||||
int got = currentProgram.getMemory().getBytes(address, buffer);
|
||||
if (got != buffer.length) throw new IOException("Incomplete function bytes");
|
||||
hash.update(buffer);
|
||||
left -= got;
|
||||
if (left > 0) address = address.add(got);
|
||||
}
|
||||
}
|
||||
item.addProperty("body_sha256", hashable ? HexFormat.of().formatHex(hash.digest()) : "");
|
||||
functions.add(item);
|
||||
}
|
||||
facts.add("functions", functions);
|
||||
GzfExporter exporter = new GzfExporter();
|
||||
// GhidraScript wraps run() in a transaction; packed export needs its own
|
||||
// program lock. Close that transaction, then restore the script lifecycle.
|
||||
end(true);
|
||||
try {
|
||||
if (!exporter.export(new File(output, "program.gzf"), currentProgram, null, monitor))
|
||||
throw new IOException("GZF export failed: " + exporter.getMessageLog());
|
||||
} finally {
|
||||
start();
|
||||
}
|
||||
try (Writer writer = new OutputStreamWriter(new FileOutputStream(new File(output, "functions.json")), StandardCharsets.UTF_8)) {
|
||||
new Gson().toJson(facts, writer);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
// @category Verstack
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import com.google.gson.JsonObject;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
|
||||
public class VerifyExport extends GhidraScript {
|
||||
@Override public void run() throws Exception {
|
||||
String[] args = getScriptArgs();
|
||||
String identity = currentProgram.getExecutableSHA256();
|
||||
if (!args[1].equalsIgnoreCase(identity)) throw new IllegalStateException("GZF input identity mismatch");
|
||||
JsonObject receipt = new JsonObject();
|
||||
receipt.addProperty("schema", 1);
|
||||
receipt.addProperty("input_sha256", identity);
|
||||
receipt.addProperty("function_count", currentProgram.getFunctionManager().getFunctionCount());
|
||||
Files.writeString(Path.of(args[0]), receipt.toString());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Trusted, locally installed Ghidra adapter. Does not execute imported programs."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
def run(request):
|
||||
if request["protocol"] != 1:
|
||||
raise ValueError("unsupported protocol")
|
||||
settings = request["settings"]
|
||||
home = pathlib.Path(settings["ghidra_home"]).resolve()
|
||||
launcher = home / "support" / "analyzeHeadless"
|
||||
properties = (home / "Ghidra" / "application.properties").read_text()
|
||||
version = next(line.split("=", 1)[1].strip() for line in properties.splitlines() if line.startswith("application.version="))
|
||||
if version != settings["expected_version"]:
|
||||
raise ValueError("Ghidra version differs from pinned expected_version")
|
||||
source = pathlib.Path(request["input_dir"])
|
||||
output = pathlib.Path(request["output_dir"])
|
||||
scripts = pathlib.Path(__file__).resolve().parent
|
||||
project = output.parent / "ghidra-work"
|
||||
project.mkdir()
|
||||
env = os.environ.copy()
|
||||
if settings.get("java_home"):
|
||||
env["JAVA_HOME"] = str(pathlib.Path(settings["java_home"]).resolve())
|
||||
env["GHIDRA_HEADLESS_MAXMEM"] = settings.get("max_heap", "2G")
|
||||
for directory in (project / "user", project / "tmp"):
|
||||
directory.mkdir()
|
||||
env["XDG_CONFIG_HOME"] = str(project / "user" / "config")
|
||||
env["XDG_CACHE_HOME"] = str(project / "user" / "cache")
|
||||
env["JAVA_TOOL_OPTIONS"] = (
|
||||
env.get("JAVA_TOOL_OPTIONS", "") +
|
||||
f' -Duser.home="{project / "user"}" -Djava.io.tmpdir="{project / "tmp"}"'
|
||||
)
|
||||
files, warnings = [], []
|
||||
selected = settings.get("paths")
|
||||
count = 0
|
||||
for path in sorted(source.rglob("*")):
|
||||
if not path.is_file() or path.is_symlink():
|
||||
continue
|
||||
rel = path.relative_to(source).as_posix()
|
||||
if selected and rel not in selected:
|
||||
continue
|
||||
with path.open("rb") as stream:
|
||||
magic = stream.read(4)
|
||||
if magic != b"\x7fELF" and magic[:2] != b"MZ":
|
||||
continue
|
||||
count += 1
|
||||
with path.open("rb") as stream:
|
||||
identity = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
dest = output / identity
|
||||
if dest.exists():
|
||||
continue
|
||||
dest.mkdir()
|
||||
command = [str(launcher), str(project), identity, "-import", str(path), "-scriptPath", str(scripts),
|
||||
"-postScript", "ExportFacts.java", str(dest), identity, "-deleteProject",
|
||||
"-max-cpu", str(settings.get("max_cpu", 2)),
|
||||
"-analysisTimeoutPerFile", str(settings.get("analysis_timeout_seconds", 3600))]
|
||||
with (dest / "analysis.log").open("w") as log:
|
||||
subprocess.run(command, check=True, env=env, stdout=log, stderr=subprocess.STDOUT)
|
||||
facts = json.loads((dest / "functions.json").read_text())
|
||||
if facts["input_sha256"].lower() != identity:
|
||||
raise ValueError("analysis input identity mismatch")
|
||||
if facts["analysis_timed_out"]:
|
||||
raise ValueError("Ghidra analysis timed out; result is incomplete")
|
||||
# A successful process exit is insufficient: re-import the packed database,
|
||||
# and require a post-script receipt with the same input identity.
|
||||
receipt = dest / "reopened.json"
|
||||
with (dest / "reopen.log").open("w") as log:
|
||||
subprocess.run([str(launcher), str(project), identity + "-check", "-import", str(dest / "program.gzf"),
|
||||
"-noanalysis", "-scriptPath", str(scripts), "-postScript", "VerifyExport.java", str(receipt), identity,
|
||||
"-deleteProject"], check=True, env=env, stdout=log, stderr=subprocess.STDOUT)
|
||||
if json.loads(receipt.read_text())["input_sha256"].lower() != identity:
|
||||
raise ValueError("export failed reopen validation")
|
||||
facts.update({"source_path": rel, "ghidra_version": version, "settings": settings})
|
||||
(dest / "functions.json").write_text(json.dumps(facts))
|
||||
files.extend(p.relative_to(output).as_posix() for p in dest.iterdir() if p.is_file())
|
||||
if not count:
|
||||
raise ValueError("no selected ELF or PE executable found")
|
||||
return {"protocol": 1, "layer": "derived", "coverage": "partial", "warnings": ["Only ELF/PE analysis selected. Function absence is not proof of newly added code."], "files": files}
|
||||
|
||||
if __name__ == "__main__":
|
||||
request = json.loads(pathlib.Path(sys.argv[1]).read_text())
|
||||
try:
|
||||
result = run(request)
|
||||
except Exception as error:
|
||||
diagnostic = []
|
||||
for log in pathlib.Path(request["output_dir"]).rglob("*.log"):
|
||||
with log.open("rb") as stream:
|
||||
stream.seek(max(0, log.stat().st_size - 65536))
|
||||
lines = stream.read().decode(errors="replace").splitlines()
|
||||
diagnostic.extend(line for line in lines if "ERROR" in line or "Exception" in line)
|
||||
result = {"protocol": 1, "error": (str(error) + "\n" + "\n".join(diagnostic[-6:]))[:4000]}
|
||||
pathlib.Path(request["result_file"]).write_text(json.dumps(result))
|
||||
raise
|
||||
pathlib.Path(request["result_file"]).write_text(json.dumps(result))
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check supplied LUKS credentials without mounting, decrypting, or logging secrets."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
HEADER_LIMIT = 16 * 1024 * 1024
|
||||
|
||||
def credentials(settings):
|
||||
if bool(settings.get('key_file')) == bool(settings.get('key_env')):
|
||||
raise ValueError('configure exactly one key_file or key_env reference')
|
||||
if settings.get('key_file'):
|
||||
with Path(settings['key_file']).open('rb') as stream:
|
||||
raw = stream.read(65537)
|
||||
else:
|
||||
raw = os.environ[settings['key_env']].encode()
|
||||
if not raw or len(raw) > 65536:
|
||||
raise ValueError('credential must contain 1 to 65536 bytes')
|
||||
variants = {'raw': raw, 'text-stripped': raw.strip()}
|
||||
tokens = raw.split()
|
||||
if tokens and all(re.fullmatch(rb'0x[0-9a-fA-F]{8}', t) for t in tokens):
|
||||
words = [int(t, 16) for t in tokens]
|
||||
variants.update({'u32le': struct.pack('<' + 'I' * len(words), *words),
|
||||
'u32be': struct.pack('>' + 'I' * len(words), *words),
|
||||
'hex-text': ''.join(f'{w:08x}' for w in words).encode()})
|
||||
encoding = settings.get('key_encoding', 'raw')
|
||||
if encoding == 'auto':
|
||||
return variants
|
||||
if encoding not in variants:
|
||||
raise ValueError('unsupported credential encoding for supplied file')
|
||||
return {encoding: variants[encoding]}
|
||||
|
||||
def check_stream(stream, settings, keys, work):
|
||||
header = stream.read(HEADER_LIMIT)
|
||||
if not header.startswith(b'LUKS\xba\xbe\x00\x02'):
|
||||
return None
|
||||
uuid = header[168:208].split(b'\0')[0].decode('ascii', errors='replace')
|
||||
path = work / 'header.luks'
|
||||
path.write_bytes(header)
|
||||
attempts = {}
|
||||
for encoding, secret in keys.items():
|
||||
# Credentials use stdin, never argv or a workspace file. Tool output is
|
||||
# captured and reduced to fixed statuses; it cannot become provenance.
|
||||
result = subprocess.run([settings.get('cryptsetup', '/usr/sbin/cryptsetup'),
|
||||
'open', '--test-passphrase', '--key-file', '-', str(path)],
|
||||
input=secret, capture_output=True, timeout=120)
|
||||
attempts[encoding] = ('accepted' if result.returncode == 0 else
|
||||
'rejected' if b'No key available with this passphrase' in result.stderr else 'tool_error')
|
||||
if result.returncode == 0:
|
||||
break
|
||||
return {'uuid': uuid, 'unlock_verified': 'accepted' in attempts.values(), 'attempts': attempts}
|
||||
|
||||
def run(request):
|
||||
if request['protocol'] != 1:
|
||||
raise ValueError('unsupported protocol')
|
||||
settings = request['settings']
|
||||
keys = credentials(settings)
|
||||
rows = []
|
||||
with tempfile.TemporaryDirectory(prefix='luks-header-', dir=Path(request['output_dir']).parent) as directory:
|
||||
work = Path(directory)
|
||||
for path in sorted(Path(request['input_dir']).rglob('*')):
|
||||
if not path.is_file() or path.is_symlink():
|
||||
continue
|
||||
rel = path.relative_to(request['input_dir']).as_posix()
|
||||
if zipfile.is_zipfile(path):
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
for entry in archive.infolist():
|
||||
if entry.is_dir() or not (entry.filename.endswith('.000') or entry.filename.lower().endswith('.spk')):
|
||||
continue
|
||||
with archive.open(entry) as stream:
|
||||
report = check_stream(stream, settings, keys, work)
|
||||
if report:
|
||||
rows.append({'path': rel, 'entry': entry.filename, **report})
|
||||
else:
|
||||
with path.open('rb') as stream:
|
||||
report = check_stream(stream, settings, keys, work)
|
||||
if report:
|
||||
rows.append({'path': rel, **report})
|
||||
if not rows:
|
||||
raise ValueError('no supported LUKS2 header found')
|
||||
record = {'schema': 1, 'operation': 'credential-validation', 'inputs': rows,
|
||||
'content_extracted': False, 'credential_reference': settings.get('key_file') or settings.get('key_env')}
|
||||
Path(request['output_dir'], 'luks-key-check.json').write_text(json.dumps(record, indent=2))
|
||||
warnings = ['Credential validation only. No decrypted content has been extracted.']
|
||||
if any(not row['unlock_verified'] for row in rows):
|
||||
warnings.append('One or more containers did not unlock with the supplied credential. Inspect attempts; tool_error is distinct from key rejection.')
|
||||
return {'protocol': 1, 'layer': 'derived', 'coverage': 'partial', 'files': ['luks-key-check.json'], 'warnings': warnings}
|
||||
|
||||
if __name__ == '__main__':
|
||||
request = json.loads(Path(sys.argv[1]).read_text())
|
||||
try:
|
||||
result = run(request)
|
||||
except Exception:
|
||||
# Avoid serializing exceptions that could contain credentials or tool output.
|
||||
Path(request['result_file']).write_text(json.dumps({'protocol': 1, 'error': 'LUKS credential check failed; check key reference, dependencies, and supported encoding.'}))
|
||||
sys.exit(1)
|
||||
Path(request['result_file']).write_text(json.dumps(result))
|
||||
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only LUKS2/AES-XTS/ext4 wrapper extraction, without kernel mounts."""
|
||||
import ctypes as C
|
||||
import ctypes.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import zipfile
|
||||
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
|
||||
|
||||
|
||||
def volume_key(image, credential):
|
||||
lib = C.CDLL(ctypes.util.find_library('cryptsetup') or 'libcryptsetup.so.12')
|
||||
lib.crypt_init.argtypes = [C.POINTER(C.c_void_p), C.c_char_p]
|
||||
lib.crypt_load.argtypes = [C.c_void_p, C.c_char_p, C.c_void_p]
|
||||
lib.crypt_volume_key_get.argtypes = [C.c_void_p, C.c_int, C.c_void_p, C.POINTER(C.c_size_t), C.c_char_p, C.c_size_t]
|
||||
lib.crypt_free.argtypes = [C.c_void_p]
|
||||
lib.crypt_free.restype = None
|
||||
device = C.c_void_p()
|
||||
buffer = C.create_string_buffer(64)
|
||||
length = C.c_size_t(64)
|
||||
try:
|
||||
if lib.crypt_init(C.byref(device), os.fsencode(image)) < 0 or lib.crypt_load(device, b'LUKS2', None) < 0:
|
||||
raise ValueError('cannot load LUKS2 metadata')
|
||||
if lib.crypt_volume_key_get(device, -1, buffer, C.byref(length), credential, len(credential)) < 0:
|
||||
raise ValueError('credential did not unlock this container')
|
||||
if length.value not in (32, 64):
|
||||
raise ValueError('unsupported volume key size')
|
||||
return buffer.raw[:length.value]
|
||||
finally:
|
||||
C.memset(buffer, 0, len(buffer))
|
||||
if device:
|
||||
lib.crypt_free(device)
|
||||
|
||||
|
||||
def decrypt(image, output, key, metadata):
|
||||
if list(metadata['segments']) != ['0']:
|
||||
raise ValueError('multiple/re-encrypting segments are unsupported')
|
||||
segment = metadata['segments']['0']
|
||||
if segment['type'] != 'crypt' or segment['encryption'] != 'aes-xts-plain64' or segment.get('flags'):
|
||||
raise ValueError('unsupported encryption segment')
|
||||
sector = segment['sector_size']
|
||||
if sector not in (512, 4096):
|
||||
raise ValueError('unsupported sector size')
|
||||
offset = int(segment['offset'])
|
||||
length = image.stat().st_size - offset
|
||||
if length <= 0 or length % sector or offset % sector:
|
||||
raise ValueError('truncated or misaligned container')
|
||||
if segment['size'] != 'dynamic' and int(segment['size']) != length:
|
||||
raise ValueError('unsupported fixed segment length')
|
||||
iv = int(segment['iv_tweak'])
|
||||
with image.open('rb') as source, output.open('wb') as dest:
|
||||
source.seek(offset)
|
||||
position = 0
|
||||
while block := source.read(4 * 1024 * 1024):
|
||||
plain = bytearray()
|
||||
for start in range(0, len(block), sector):
|
||||
# plain64 uses a 512-byte sector number, even for 4K data units.
|
||||
tweak = (iv + (position + start) // 512).to_bytes(16, 'little')
|
||||
context = Cipher(algorithms.AES(key), modes.XTS(tweak)).decryptor()
|
||||
plain.extend(context.update(block[start:start + sector]) + context.finalize())
|
||||
dest.write(plain)
|
||||
position += len(block)
|
||||
with output.open('rb') as stream:
|
||||
stream.seek(1024 + 56)
|
||||
if stream.read(2) != b'\x53\xef':
|
||||
raise ValueError('decryption did not produce an ext filesystem')
|
||||
|
||||
|
||||
def assemble(path, destination, budget):
|
||||
"""Strict complete split ordering; ZIP entry names never become output paths."""
|
||||
if zipfile.is_zipfile(path):
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
entries = [e for e in archive.infolist() if not e.is_dir()]
|
||||
names = [e.filename for e in entries]
|
||||
if len(names) != len(set(names)):
|
||||
raise ValueError('duplicate ZIP entry')
|
||||
matches = [re.fullmatch(r'(.+\.spk)\.(\d{3})\.(\d{3})', name, re.I) for name in names]
|
||||
if not entries or not all(matches):
|
||||
raise ValueError('expected only split SPK entries')
|
||||
count = int(matches[0][2])
|
||||
if len(entries) != count or len({(m[1],m[2]) for m in matches}) != 1 or sorted(int(m[3]) for m in matches) != list(range(count)):
|
||||
raise ValueError('missing or inconsistent split parts')
|
||||
total = sum(e.file_size for e in entries)
|
||||
if total * 4 > budget:
|
||||
raise ValueError('insufficient declared workspace for extraction')
|
||||
with destination.open('wb') as out:
|
||||
for entry in sorted(entries, key=lambda e: int(e.filename.rsplit('.',1)[1])):
|
||||
with archive.open(entry) as stream:
|
||||
shutil.copyfileobj(stream, out, 1024 * 1024)
|
||||
return total
|
||||
size = path.stat().st_size
|
||||
if size * 4 > budget:
|
||||
raise ValueError('insufficient declared workspace for extraction')
|
||||
shutil.copyfile(path, destination)
|
||||
return size
|
||||
|
||||
|
||||
def run(request):
|
||||
if request['protocol'] != 1:
|
||||
raise ValueError('unsupported protocol')
|
||||
settings = request['settings']
|
||||
source = Path(request['input_dir'])
|
||||
path = source / settings['input_path']
|
||||
if not path.resolve().is_relative_to(source.resolve()) or not path.is_file():
|
||||
raise ValueError('input_path must select a file within the input snapshot')
|
||||
if bool(settings.get('key_file')) == bool(settings.get('key_env')):
|
||||
raise ValueError('configure exactly one credential reference')
|
||||
if settings.get('key_file'):
|
||||
with Path(settings['key_file']).open('rb') as f:
|
||||
credential = f.read(65537)
|
||||
else:
|
||||
credential = os.environ[settings['key_env']].encode()
|
||||
if not credential or len(credential) > 65536:
|
||||
raise ValueError('credential size unsupported')
|
||||
output = Path(request['output_dir'])
|
||||
with tempfile.TemporaryDirectory(prefix='luks-', dir=output.parent) as tmp:
|
||||
work = Path(tmp)
|
||||
encrypted, plain = work / 'container.luks', work / 'filesystem.ext4'
|
||||
size = assemble(path, encrypted, request['workspace_bytes'])
|
||||
metadata_process = subprocess.run([settings.get('cryptsetup','/usr/sbin/cryptsetup'),'luksDump','--dump-json-metadata',str(encrypted)],capture_output=True,check=True)
|
||||
metadata = json.loads(metadata_process.stdout)
|
||||
key = volume_key(encrypted, credential)
|
||||
del credential
|
||||
decrypt(encrypted, plain, key, metadata)
|
||||
del key
|
||||
destination = output / 'filesystem'
|
||||
destination.mkdir()
|
||||
if any(c in str(destination) for c in ['"','\\','\n','\r']):
|
||||
raise ValueError('workspace path cannot be represented to debugfs')
|
||||
process = subprocess.run([settings.get('debugfs','/usr/sbin/debugfs'),'-R',f'rdump / "{destination}"',str(plain)],capture_output=True)
|
||||
if process.returncode != 0 or any(s in process.stderr.lower() for s in [b'error',b'failed',b'cannot',b'not found',b'short read']):
|
||||
raise ValueError('ext4 extraction reported an error')
|
||||
files = [p for p in destination.rglob('*') if p.is_file() and not p.is_symlink()]
|
||||
if not any(p.suffix.lower()=='.spk' for p in files):
|
||||
raise ValueError('no inner SPK was recovered')
|
||||
links=[]
|
||||
for p in destination.rglob('*'):
|
||||
if p.is_symlink():
|
||||
links.append({'path':p.relative_to(output).as_posix(),'target':str(p.readlink())});p.unlink()
|
||||
evidence={'schema':1,'input_path':settings['input_path'],'container_bytes':size,'unlock_verified':True,
|
||||
'credential_reference':settings.get('key_file') or settings.get('key_env'),
|
||||
'encryption':metadata['segments']['0']['encryption'],'sector_size':metadata['segments']['0']['sector_size'],
|
||||
'symlinks':links,'filesystem':'ext4','method':'libcryptsetup + userspace AES-XTS + read-only debugfs'}
|
||||
(output/'wrapper-evidence.json').write_text(json.dumps(evidence,indent=2))
|
||||
return {'protocol':1,'layer':'extracted','coverage':'partial','warnings':['Encrypted original retained. Inner SPK still requires extraction; filesystem metadata is not promised byte-for-byte restoration.'],
|
||||
'files':[p.relative_to(output).as_posix() for p in sorted(output.rglob('*')) if p.is_file()]}
|
||||
|
||||
if __name__ == '__main__':
|
||||
request=json.loads(Path(sys.argv[1]).read_text())
|
||||
try:
|
||||
result=run(request)
|
||||
except Exception as error:
|
||||
# Fixed messages from this adapter only; never serialize raw tool output.
|
||||
message=str(error) if isinstance(error,ValueError) else 'LUKS extraction failed; check dependencies, source completeness, and workspace capacity.'
|
||||
Path(request['result_file']).write_text(json.dumps({'protocol':1,'error':message}))
|
||||
sys.exit(1)
|
||||
Path(request['result_file']).write_text(json.dumps(result))
|
||||
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Adapter for a pinned local bdash/spike-spk executable; wrapper coverage is partial."""
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
def run(request):
|
||||
if request["protocol"] != 1:
|
||||
raise ValueError("unsupported protocol")
|
||||
settings = request["settings"]
|
||||
tool = pathlib.Path(settings["tool"]).resolve()
|
||||
with tool.open("rb") as stream:
|
||||
tool_hash = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
if tool_hash != settings["expected_sha256"]:
|
||||
raise ValueError("SPK tool build differs from pinned expected_sha256")
|
||||
source = pathlib.Path(request["input_dir"])
|
||||
output = pathlib.Path(request["output_dir"])
|
||||
env = os.environ.copy()
|
||||
env["RAYON_NUM_THREADS"] = str(settings.get("max_threads", 2))
|
||||
candidates = [p for p in sorted(source.rglob("*")) if p.is_file() and not p.is_symlink()
|
||||
and (p.name.lower().endswith(".spk") or re.search(r"\.spk\.\d{3}\.000$", p.name, re.I))]
|
||||
if not candidates:
|
||||
raise ValueError("no SPK or first split-package part found")
|
||||
packages = []
|
||||
for index, path in enumerate(candidates):
|
||||
# Let the upstream parser validate actual structure; never assume generation
|
||||
# or architecture from extension, package name, or release metadata.
|
||||
destination = output / f"package-{index:04d}"
|
||||
destination.mkdir()
|
||||
# Extraction verifies all checksums itself; avoid reading every payload twice.
|
||||
subprocess.run([str(tool), "extract", str(path), "--output", str(destination)], check=True, env=env)
|
||||
packages.append({"source": path.relative_to(source).as_posix(), "generation": "unknown", "tool_sha256": tool_hash})
|
||||
# Symlinks need portable metadata rather than links the host could traverse.
|
||||
links = []
|
||||
for path in sorted(output.rglob("*")):
|
||||
if path.is_symlink():
|
||||
links.append({"path": path.relative_to(output).as_posix(), "target": str(path.readlink())})
|
||||
path.unlink()
|
||||
(output / "package-evidence.json").write_text(json.dumps({"schema": 1, "packages": packages, "symlinks": links}))
|
||||
files = [p.relative_to(output).as_posix() for p in sorted(output.rglob("*")) if p.is_file()]
|
||||
return {"protocol": 1, "layer": "extracted", "coverage": "partial", "files": files,
|
||||
"warnings": ["Generation not inferred. Nested asset containers remain opaque; SPIKE 3 LUKS wrappers need a separate decoder. Original inputs retained."]}
|
||||
|
||||
if __name__ == "__main__":
|
||||
request = json.loads(pathlib.Path(sys.argv[1]).read_text())
|
||||
pathlib.Path(request["result_file"]).write_text(json.dumps(run(request)))
|
||||
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bounded structural probe; records missing credentials without reading key values."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
def classify(header):
|
||||
if header.startswith(b"LUKS\xba\xbe\x00\x02"):
|
||||
uuid = header[168:208].split(b"\0")[0].decode("ascii", errors="replace")
|
||||
return {"format": "luks2", "generation": "unknown", "uuid": uuid,
|
||||
"required_secret": "stern.spike3.luks", "next_step": "Provide a LUKS unlocking keyfile; encrypted-container decoding is a separate pending adapter."}
|
||||
if header.startswith(b"hsqs"):
|
||||
return {"format": "squashfs", "generation": "unknown", "required_secret": None, "next_step": "Assemble all split parts and extract the inner SPK."}
|
||||
if header.startswith(b"SPKS"):
|
||||
return {"format": "spk", "generation": "unknown", "required_secret": None, "next_step": "Run spike-extract."}
|
||||
if header.startswith(b"\x1f\x8b"):
|
||||
return {"format": "gzip", "generation": "unknown", "required_secret": None, "next_step": "Decode the legacy wrapper; do not pass it directly to the SPKS parser."}
|
||||
return {"format": "unknown", "generation": "unknown", "required_secret": None, "next_step": "Preserve as opaque content."}
|
||||
|
||||
def probe(path):
|
||||
if zipfile.is_zipfile(path):
|
||||
rows = []
|
||||
with zipfile.ZipFile(path) as z:
|
||||
for entry in z.infolist():
|
||||
if entry.is_dir() or not (entry.filename.lower().endswith(".spk") or entry.filename.endswith(".000")):
|
||||
continue
|
||||
if entry.flag_bits & 1:
|
||||
rows.append({"entry": entry.filename, "format": "encrypted_zip", "required_secret": "zip.password"})
|
||||
else:
|
||||
with z.open(entry) as stream:
|
||||
rows.append({"entry": entry.filename, **classify(stream.read(4096))})
|
||||
return {"format": "zip", "entries": rows}
|
||||
with path.open("rb") as stream:
|
||||
return classify(stream.read(4096))
|
||||
|
||||
def run(request):
|
||||
if request["protocol"] != 1:
|
||||
raise ValueError("unsupported protocol")
|
||||
source, output = Path(request["input_dir"]), Path(request["output_dir"])
|
||||
rows = [{"path": p.relative_to(source).as_posix(), **probe(p)} for p in sorted(source.rglob("*")) if p.is_file() and not p.is_symlink()]
|
||||
(output / "format-report.json").write_text(json.dumps({"schema": 1, "inputs": rows}, indent=2))
|
||||
missing = [r["path"] for r in rows if r.get("required_secret") or any(e.get("required_secret") for e in r.get("entries", []))]
|
||||
return {"protocol": 1, "layer": "derived", "coverage": "partial", "files": ["format-report.json"],
|
||||
"warnings": ["Required credentials: " + ", ".join(missing)] if missing else ["Structural probe only; generation is not inferred from container format."]}
|
||||
|
||||
if __name__ == "__main__":
|
||||
request = json.loads(Path(sys.argv[1]).read_text())
|
||||
Path(request["result_file"]).write_text(json.dumps(run(request)))
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Protocol v1 ZIP extractor. Exact entries; unknown files remain in the parent."""
|
||||
import json
|
||||
import pathlib
|
||||
import stat
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
def run(request):
|
||||
if request["protocol"] != 1:
|
||||
raise ValueError("unsupported protocol")
|
||||
source = pathlib.Path(request["input_dir"])
|
||||
output = pathlib.Path(request["output_dir"])
|
||||
files, warnings = [], []
|
||||
budget = request["workspace_bytes"] // 2
|
||||
used = 0
|
||||
archives = 0
|
||||
for path in sorted(source.rglob("*")):
|
||||
if not path.is_file() or path.is_symlink():
|
||||
continue
|
||||
if not zipfile.is_zipfile(path):
|
||||
warnings.append(f"Opaque input retained in parent: {path.relative_to(source)}")
|
||||
continue
|
||||
archives += 1
|
||||
prefix = pathlib.PurePosixPath(str(path.relative_to(source)) + ".entries")
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
for entry in archive.infolist():
|
||||
name = pathlib.PurePosixPath(entry.filename)
|
||||
if name.is_absolute() or ".." in name.parts or "\\" in entry.filename or not name.parts:
|
||||
raise ValueError("unsafe ZIP entry path")
|
||||
if stat.S_ISLNK(entry.external_attr >> 16):
|
||||
warnings.append(f"ZIP symlink left opaque in parent: {entry.filename}")
|
||||
continue
|
||||
dest = output.joinpath(prefix, name)
|
||||
if entry.is_dir():
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
continue
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
with archive.open(entry) as reader, dest.open("xb") as writer:
|
||||
while chunk := reader.read(1024 * 1024):
|
||||
used += len(chunk)
|
||||
if used > budget:
|
||||
raise ValueError("ZIP expansion exceeds workspace allowance")
|
||||
writer.write(chunk)
|
||||
files.append(dest.relative_to(output).as_posix())
|
||||
if not archives:
|
||||
raise ValueError("no supported ZIP inputs found")
|
||||
return {"protocol": 1, "layer": "extracted", "coverage": "partial" if warnings else "complete", "files": files, "warnings": warnings}
|
||||
|
||||
if __name__ == "__main__":
|
||||
request = json.loads(pathlib.Path(sys.argv[1]).read_text())
|
||||
result = run(request)
|
||||
pathlib.Path(request["result_file"]).write_text(json.dumps(result))
|
||||
@@ -0,0 +1,2 @@
|
||||
Synthetic archive fixture — contains no game content.
|
||||
An older version imported after a newer version.
|
||||
@@ -0,0 +1 @@
|
||||
Shared publisher asset: identical bytes across demo versions and games.
|
||||
@@ -0,0 +1,2 @@
|
||||
volume = 70
|
||||
attract_mode = true
|
||||
@@ -0,0 +1,2 @@
|
||||
Synthetic archive fixture — contains no game content.
|
||||
Import this directory as version 2.0, then demo-v1 as version 1.0.
|
||||
@@ -0,0 +1 @@
|
||||
Shared publisher asset: identical bytes across demo versions and games.
|
||||
@@ -0,0 +1,3 @@
|
||||
volume = 80
|
||||
attract_mode = true
|
||||
feature = enabled
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Install pinned official archives from docs/toolchain.lock.json. No system writes."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tarfile
|
||||
import urllib.request
|
||||
import zipfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
TOOLS = ROOT / "tools"
|
||||
DOWNLOADS = TOOLS / "downloads"
|
||||
|
||||
def download(url, name, digest=None):
|
||||
dest = DOWNLOADS / name
|
||||
if not dest.exists():
|
||||
part = dest.with_suffix(dest.suffix + ".part")
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "verstack-local-setup"}), timeout=120) as src, part.open("wb") as out:
|
||||
while chunk := src.read(1024 * 1024):
|
||||
out.write(chunk)
|
||||
part.rename(dest)
|
||||
with dest.open("rb") as stream:
|
||||
actual = hashlib.file_digest(stream, "sha256").hexdigest()
|
||||
if digest and actual != digest.removeprefix("sha256:"):
|
||||
raise ValueError(f"Checksum mismatch: {name}")
|
||||
print(f"Verified {name}: sha256:{actual}", flush=True)
|
||||
return dest, actual
|
||||
|
||||
def main():
|
||||
DOWNLOADS.mkdir(parents=True, exist_ok=True)
|
||||
lock = json.loads((ROOT / "docs/toolchain.lock.json").read_text())
|
||||
ghidra = lock["ghidra"]
|
||||
archive, digest = download(ghidra["url"], ghidra["url"].rsplit('/', 1)[1], ghidra["sha256"])
|
||||
with zipfile.ZipFile(archive) as z:
|
||||
folder = z.namelist()[0].split('/')[0]
|
||||
if not (TOOLS / folder).exists():
|
||||
z.extractall(TOOLS)
|
||||
for e in z.infolist():
|
||||
mode = e.external_attr >> 16
|
||||
if mode:
|
||||
(TOOLS / e.filename).chmod(mode & 0o777)
|
||||
ghidra_home = TOOLS / folder
|
||||
jdk = lock["jdk"]
|
||||
java_archive, java_digest = download(jdk["url"], jdk["url"].rsplit('/', 1)[1], jdk["sha256"])
|
||||
with tarfile.open(java_archive) as t:
|
||||
java_folder = t.getnames()[0].split('/')[0]
|
||||
if not (TOOLS / java_folder).exists():
|
||||
t.extractall(TOOLS, filter="data")
|
||||
java_home = TOOLS / java_folder
|
||||
record = {"schema": 1, "ghidra_home": str(ghidra_home), "ghidra_version": ghidra["version"], "ghidra_archive_sha256": digest,
|
||||
"java_home": str(java_home), "java_version": jdk["version"], "java_archive_sha256": java_digest}
|
||||
(TOOLS / "installed.json").write_text(json.dumps(record, indent=2))
|
||||
(TOOLS / "bin").mkdir(exist_ok=True)
|
||||
for name, script in [("ghidra", "ghidraRun"), ("ghidra-headless", "support/analyzeHeadless")]:
|
||||
launcher = TOOLS / "bin" / name
|
||||
launcher.write_text('''#!/usr/bin/env python3
|
||||
import json,os,pathlib,sys
|
||||
root=pathlib.Path(__file__).resolve().parents[2]
|
||||
record=json.loads((root/'tools/installed.json').read_text())
|
||||
env=os.environ.copy()
|
||||
env['JAVA_HOME']=record['java_home']
|
||||
env['XDG_CONFIG_HOME']=str(root/'data/ghidra-user/config')
|
||||
env['XDG_CACHE_HOME']=str(root/'data/ghidra-user/cache')
|
||||
launcher=str(pathlib.Path(record['ghidra_home'])/SCRIPT)
|
||||
os.execve(launcher,[launcher,*sys.argv[1:]],env)
|
||||
'''.replace("SCRIPT", repr(script)))
|
||||
launcher.chmod(0o755)
|
||||
print(json.dumps(record, indent=2))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+256
@@ -0,0 +1,256 @@
|
||||
//! Conservative first comparison: unique, sufficiently large exact byte bodies.
|
||||
//! Structural/BSim candidates can later supply additional evidence, never replace bytes.
|
||||
use crate::{Archive, Layer, Release, Snapshot};
|
||||
use anyhow::{Result, ensure};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{
|
||||
collections::{BTreeMap, BTreeSet},
|
||||
fs::File,
|
||||
};
|
||||
|
||||
pub const COMPARISON_VERSION: &str = "exact-functions/2";
|
||||
|
||||
#[derive(Clone, Deserialize, Serialize)]
|
||||
pub struct Function {
|
||||
pub address: String,
|
||||
pub name: String,
|
||||
pub symbol_source: String,
|
||||
pub size: u64,
|
||||
pub thunk: bool,
|
||||
pub body_sha256: String,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct FunctionFacts {
|
||||
pub schema: u32,
|
||||
pub input_sha256: String,
|
||||
pub language: String,
|
||||
pub compiler: String,
|
||||
pub ghidra_version: String,
|
||||
pub analysis_timed_out: bool,
|
||||
pub settings: serde_json::Value,
|
||||
pub functions: Vec<Function>,
|
||||
}
|
||||
#[derive(Serialize)]
|
||||
pub struct FunctionComparison {
|
||||
pub schema: u32,
|
||||
pub method: String,
|
||||
pub before_input: String,
|
||||
pub after_input: String,
|
||||
pub matches: Vec<FunctionMatch>,
|
||||
pub unmatched_after: Vec<Function>,
|
||||
pub unmatched_before: Vec<Function>,
|
||||
pub unmatched_after_reasons: BTreeMap<String, String>,
|
||||
pub profile_policy: String,
|
||||
pub sources: Vec<AnalysisSource>,
|
||||
pub caveat: String,
|
||||
}
|
||||
#[derive(Serialize)]
|
||||
pub struct AnalysisSource {
|
||||
pub snapshot: String,
|
||||
pub path: String,
|
||||
pub artifact: String,
|
||||
pub run: String,
|
||||
pub release: Release,
|
||||
}
|
||||
// Selection determines which executable is analyzed, not how it is analyzed.
|
||||
// Keep all other settings (including unknown future options) conservative.
|
||||
fn comparable_settings(settings: &serde_json::Value) -> serde_json::Value {
|
||||
let mut value = settings.clone();
|
||||
if let Some(object) = value.as_object_mut() {
|
||||
object.remove("paths");
|
||||
}
|
||||
value
|
||||
}
|
||||
#[derive(Serialize)]
|
||||
pub struct FunctionMatch {
|
||||
pub before: Function,
|
||||
pub after: Function,
|
||||
}
|
||||
pub fn compare_functions(a: &FunctionFacts, b: &FunctionFacts) -> Result<FunctionComparison> {
|
||||
ensure!(
|
||||
a.schema == 1 && b.schema == 1,
|
||||
"unsupported function facts schema"
|
||||
);
|
||||
ensure!(
|
||||
!a.analysis_timed_out && !b.analysis_timed_out,
|
||||
"incomplete analysis cannot be compared"
|
||||
);
|
||||
ensure!(
|
||||
a.language == b.language
|
||||
&& a.compiler == b.compiler
|
||||
&& a.ghidra_version == b.ghidra_version
|
||||
&& comparable_settings(&a.settings) == comparable_settings(&b.settings),
|
||||
"function comparison requires compatible languages and analysis profiles"
|
||||
);
|
||||
for facts in [a, b] {
|
||||
let mut addresses = BTreeSet::new();
|
||||
ensure!(
|
||||
facts.functions.iter().all(|f| addresses.insert(&f.address)),
|
||||
"duplicate function addresses in inventory"
|
||||
);
|
||||
}
|
||||
let index = |facts: &FunctionFacts| {
|
||||
let mut out = BTreeMap::<String, Vec<Function>>::new();
|
||||
for f in &facts.functions {
|
||||
if f.size >= 32
|
||||
&& !f.thunk
|
||||
&& f.body_sha256.len() == 64
|
||||
&& f.body_sha256.bytes().all(|b| b.is_ascii_hexdigit())
|
||||
{
|
||||
out.entry(format!("{}:{}", f.size, f.body_sha256))
|
||||
.or_default()
|
||||
.push(f.clone());
|
||||
}
|
||||
}
|
||||
out
|
||||
};
|
||||
let a_index = index(a);
|
||||
let b_index = index(b);
|
||||
let mut matches = vec![];
|
||||
for (hash, after) in &b_index {
|
||||
if let Some(before) = a_index.get(hash)
|
||||
&& before.len() == 1
|
||||
&& after.len() == 1
|
||||
{
|
||||
matches.push(FunctionMatch {
|
||||
before: before[0].clone(),
|
||||
after: after[0].clone(),
|
||||
});
|
||||
}
|
||||
}
|
||||
let matched_before: BTreeSet<_> = matches.iter().map(|m| &m.before.address).collect();
|
||||
let matched_after: BTreeSet<_> = matches.iter().map(|m| &m.after.address).collect();
|
||||
let unmatched_before = a
|
||||
.functions
|
||||
.iter()
|
||||
.filter(|f| !matched_before.contains(&f.address))
|
||||
.cloned()
|
||||
.collect();
|
||||
let unmatched_after: Vec<Function> = b
|
||||
.functions
|
||||
.iter()
|
||||
.filter(|f| !matched_after.contains(&f.address))
|
||||
.cloned()
|
||||
.collect();
|
||||
let unmatched_after_reasons = unmatched_after
|
||||
.iter()
|
||||
.map(|f| {
|
||||
let key = format!("{}:{}", f.size, f.body_sha256);
|
||||
let reason = if f.thunk {
|
||||
"thunk_excluded"
|
||||
} else if f.size < 32 {
|
||||
"below_minimum_size"
|
||||
} else if f.body_sha256.len() != 64
|
||||
|| !f.body_sha256.bytes().all(|b| b.is_ascii_hexdigit())
|
||||
{
|
||||
"body_hash_unavailable"
|
||||
} else if b_index.get(&key).is_some_and(|v| v.len() > 1)
|
||||
|| a_index.get(&key).is_some_and(|v| v.len() > 1)
|
||||
{
|
||||
"ambiguous_exact_body"
|
||||
} else {
|
||||
"no_exact_body_match"
|
||||
};
|
||||
(f.address.clone(), reason.into())
|
||||
})
|
||||
.collect();
|
||||
Ok(FunctionComparison {
|
||||
schema: 2, method: COMPARISON_VERSION.into(),
|
||||
before_input: a.input_sha256.clone(), after_input: b.input_sha256.clone(),
|
||||
matches, unmatched_after, unmatched_before, unmatched_after_reasons,
|
||||
profile_policy: "schema1_settings_except_paths_v1".into(), sources: vec![],
|
||||
caveat: "Unmatched means no match by this method, not newly added code. Only unique exact bodies of at least 32 bytes qualify; thunks are excluded. Exact bytes do not establish equal behavior or referenced data. No names are automatically transferred. Baseline and target describe selection order, not release chronology.".into(),
|
||||
})
|
||||
}
|
||||
impl Archive {
|
||||
pub fn compare_function_files(
|
||||
&self,
|
||||
before: &str,
|
||||
before_path: &str,
|
||||
after: &str,
|
||||
after_path: &str,
|
||||
) -> Result<FunctionComparison> {
|
||||
fn facts(a: &Archive, snapshot: &str, path: &str) -> Result<FunctionFacts> {
|
||||
let s = a.snapshot(snapshot)?;
|
||||
let entry = a.file_entry(&s, path)?;
|
||||
ensure!(
|
||||
entry.size <= 64 * 1024 * 1024,
|
||||
"function facts exceed 64 MiB limit"
|
||||
);
|
||||
Ok(serde_json::from_reader(a.reader(snapshot, path)?)?)
|
||||
}
|
||||
let mut report = compare_functions(
|
||||
&facts(self, before, before_path)?,
|
||||
&facts(self, after, after_path)?,
|
||||
)?;
|
||||
for (id, path) in [(before, before_path), (after, after_path)] {
|
||||
let s = self.snapshot(id)?;
|
||||
let artifact = self
|
||||
.file_entry(&s, path)?
|
||||
.artifact
|
||||
.clone()
|
||||
.ok_or_else(|| anyhow::anyhow!("missing artifact identity"))?;
|
||||
report.sources.push(AnalysisSource {
|
||||
snapshot: s.id,
|
||||
path: path.into(),
|
||||
artifact,
|
||||
run: s.run,
|
||||
release: s.release,
|
||||
});
|
||||
}
|
||||
if report.sources[0].release.edition != report.sources[1].release.edition {
|
||||
report
|
||||
.caveat
|
||||
.push_str(" Different editions selected; differences may be edition-specific.");
|
||||
}
|
||||
if report.sources[0].release.repository != report.sources[1].release.repository {
|
||||
report
|
||||
.caveat
|
||||
.push_str(" Different repositories selected; this is a cross-project comparison.");
|
||||
}
|
||||
Ok(report)
|
||||
}
|
||||
}
|
||||
|
||||
impl Archive {
|
||||
/// Retain comparison evidence through the same verified publication path as imports.
|
||||
pub fn save_function_comparison(
|
||||
&self,
|
||||
before: &str,
|
||||
before_path: &str,
|
||||
after: &str,
|
||||
after_path: &str,
|
||||
) -> Result<Snapshot> {
|
||||
let _guard = self
|
||||
.writer
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("writer lock poisoned"))?;
|
||||
let report = self.compare_function_files(before, before_path, after, after_path)?;
|
||||
let target = self.snapshot(after)?;
|
||||
let mut run = self.start_run("function-comparison", COMPARISON_VERSION,
|
||||
serde_json::json!({"before": before, "before_path": before_path, "after": after, "after_path": after_path, "profile_policy": report.profile_policy}),
|
||||
report.sources.iter().map(|s| s.artifact.clone()).collect())?;
|
||||
let result = (|| {
|
||||
let output = tempfile::tempdir_in(&self.config.workspace)?;
|
||||
let path = output.path().join("function-comparison.json");
|
||||
serde_json::to_writer(File::create(&path)?, &report)?;
|
||||
ensure!(
|
||||
std::fs::metadata(&path)?.len().saturating_mul(2) <= self.config.workspace_bytes,
|
||||
"comparison exceeds workspace budget"
|
||||
);
|
||||
let capture = tempfile::tempdir_in(&self.config.workspace)?;
|
||||
let entries = self.stage(&path, capture.path(), &mut run)?;
|
||||
self.publish(
|
||||
capture.path(),
|
||||
entries,
|
||||
target.release,
|
||||
Layer::Derived,
|
||||
Some(target.id),
|
||||
&mut run,
|
||||
vec![report.caveat.clone()],
|
||||
)
|
||||
})();
|
||||
self.finish_run(&mut run, &result)?;
|
||||
result
|
||||
}
|
||||
}
|
||||
+516
@@ -0,0 +1,516 @@
|
||||
use crate::{
|
||||
model::*,
|
||||
storage::{ArtifactStorage, RusticStore, atomic_json},
|
||||
};
|
||||
use anyhow::{Context, Result, bail, ensure};
|
||||
use fs2::FileExt;
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
fs::{self, File, OpenOptions},
|
||||
io::{Read, Write},
|
||||
os::unix::fs::{MetadataExt, PermissionsExt},
|
||||
path::{Component, Path},
|
||||
sync::Mutex,
|
||||
};
|
||||
use uuid::Uuid;
|
||||
use walkdir::WalkDir;
|
||||
|
||||
pub struct Archive {
|
||||
pub config: Config,
|
||||
pub(crate) store: RusticStore,
|
||||
pub(crate) writer: Mutex<()>,
|
||||
_lock: File,
|
||||
}
|
||||
impl Archive {
|
||||
pub fn open(mut config: Config) -> Result<Self> {
|
||||
ensure!(
|
||||
config.workspace_bytes > 0,
|
||||
"workspace budget must be positive"
|
||||
);
|
||||
fs::create_dir_all(&config.archive)?;
|
||||
fs::create_dir_all(&config.workspace)?;
|
||||
config.archive = config.archive.canonicalize()?;
|
||||
config.workspace = config.workspace.canonicalize()?;
|
||||
ensure!(
|
||||
!config.workspace.starts_with(&config.archive),
|
||||
"workspace must be outside archive"
|
||||
);
|
||||
config.import_roots = config
|
||||
.import_roots
|
||||
.iter()
|
||||
.map(|p| p.canonicalize())
|
||||
.collect::<std::io::Result<_>>()?;
|
||||
let lock = OpenOptions::new()
|
||||
.read(true)
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(false)
|
||||
.open(config.archive.join("instance.lock"))?;
|
||||
lock.try_lock_exclusive()
|
||||
.context("instance is already owned by another process; use its HTTP API")?;
|
||||
for dir in ["snapshots", "runs"] {
|
||||
fs::create_dir_all(config.archive.join(dir))?;
|
||||
}
|
||||
let format = config.archive.join("format.json");
|
||||
if format.exists() {
|
||||
let v: serde_json::Value = serde_json::from_reader(File::open(&format)?)?;
|
||||
ensure!(
|
||||
v["schema"] == SCHEMA && v["storage"] == "rustic-restic",
|
||||
"unsupported archive format"
|
||||
);
|
||||
} else {
|
||||
atomic_json(
|
||||
&format,
|
||||
&serde_json::json!({"schema": SCHEMA, "storage": "rustic-restic", "artifact_hash": "blake3"}),
|
||||
false,
|
||||
)?;
|
||||
}
|
||||
let store = RusticStore::open(&config.archive)?;
|
||||
let archive = Self {
|
||||
config,
|
||||
store,
|
||||
writer: Mutex::new(()),
|
||||
_lock: lock,
|
||||
};
|
||||
for mut run in archive.runs()? {
|
||||
if matches!(run.state, RunState::Running) {
|
||||
// Publication is the commit point, even if the final run update was interrupted.
|
||||
let output = archive.snapshots()?.into_iter().find(|s| s.run == run.id);
|
||||
if let Some(s) = output {
|
||||
run.output = Some(s.id);
|
||||
run.state = RunState::Complete;
|
||||
run.stage = "complete".into();
|
||||
} else {
|
||||
run.state = RunState::Interrupted;
|
||||
run.stage = "interrupted".into();
|
||||
}
|
||||
archive.save_run(&run)?;
|
||||
}
|
||||
}
|
||||
Ok(archive)
|
||||
}
|
||||
pub fn snapshots(&self) -> Result<Vec<Snapshot>> {
|
||||
self.records("snapshots")
|
||||
}
|
||||
pub fn runs(&self) -> Result<Vec<Run>> {
|
||||
self.records("runs")
|
||||
}
|
||||
fn records<T: serde::de::DeserializeOwned>(&self, directory: &str) -> Result<Vec<T>> {
|
||||
let mut paths = fs::read_dir(self.config.archive.join(directory))?
|
||||
.map(|e| e.map(|e| e.path()))
|
||||
.collect::<std::io::Result<Vec<_>>>()?;
|
||||
paths.sort();
|
||||
paths
|
||||
.into_iter()
|
||||
.filter(|p| p.extension().is_some_and(|e| e == "json"))
|
||||
.map(|p| Ok(serde_json::from_reader(File::open(p)?)?))
|
||||
.collect()
|
||||
}
|
||||
pub fn snapshot(&self, id: &str) -> Result<Snapshot> {
|
||||
Uuid::parse_str(id).context("invalid snapshot ID")?;
|
||||
let s: Snapshot = serde_json::from_reader(File::open(
|
||||
self.config
|
||||
.archive
|
||||
.join("snapshots")
|
||||
.join(format!("{id}.json")),
|
||||
)?)?;
|
||||
ensure!(s.schema == SCHEMA, "unsupported snapshot schema");
|
||||
Ok(s)
|
||||
}
|
||||
pub(crate) fn save_run(&self, run: &Run) -> Result<()> {
|
||||
atomic_json(
|
||||
&self
|
||||
.config
|
||||
.archive
|
||||
.join("runs")
|
||||
.join(format!("{}.json", run.id)),
|
||||
run,
|
||||
true,
|
||||
)
|
||||
}
|
||||
pub(crate) fn start_run(
|
||||
&self,
|
||||
operation: &str,
|
||||
version: &str,
|
||||
settings: serde_json::Value,
|
||||
inputs: Vec<String>,
|
||||
) -> Result<Run> {
|
||||
let run = Run {
|
||||
schema: SCHEMA,
|
||||
id: Uuid::new_v4().to_string(),
|
||||
operation: operation.into(),
|
||||
tool_version: version.into(),
|
||||
settings,
|
||||
inputs,
|
||||
started_at: now(),
|
||||
state: RunState::Running,
|
||||
stage: "staging".into(),
|
||||
bytes_processed: 0,
|
||||
output: None,
|
||||
error: None,
|
||||
};
|
||||
self.save_run(&run)?;
|
||||
Ok(run)
|
||||
}
|
||||
pub(crate) fn finish_run(&self, run: &mut Run, result: &Result<Snapshot>) -> Result<()> {
|
||||
match result {
|
||||
Ok(s) => {
|
||||
run.state = RunState::Complete;
|
||||
run.stage = "complete".into();
|
||||
run.output = Some(s.id.clone());
|
||||
}
|
||||
Err(e) => {
|
||||
run.state = RunState::Failed;
|
||||
run.stage = "failed".into();
|
||||
run.error = Some(format!("{e:#}"));
|
||||
}
|
||||
}
|
||||
self.save_run(run)
|
||||
}
|
||||
pub fn allowed_source(&self, source: &Path) -> Result<std::path::PathBuf> {
|
||||
let path = source.canonicalize()?;
|
||||
ensure!(
|
||||
self.config.import_roots.iter().any(|r| path.starts_with(r)),
|
||||
"source is outside configured import roots"
|
||||
);
|
||||
ensure!(
|
||||
!path.starts_with(&self.config.archive) && !self.config.archive.starts_with(&path),
|
||||
"cannot import the archive itself"
|
||||
);
|
||||
ensure!(
|
||||
!path.starts_with(&self.config.workspace) && !self.config.workspace.starts_with(&path),
|
||||
"cannot import active workspaces"
|
||||
);
|
||||
Ok(path)
|
||||
}
|
||||
pub fn import(&self, source: &Path, release: Release) -> Result<Snapshot> {
|
||||
let source = self.allowed_source(source)?;
|
||||
self.import_managed(&source, release)
|
||||
}
|
||||
pub(crate) fn import_managed(&self, source: &Path, release: Release) -> Result<Snapshot> {
|
||||
let _guard = self
|
||||
.writer
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("writer lock poisoned"))?;
|
||||
validate_release(&release)?;
|
||||
let mut run = self.start_run(
|
||||
"import",
|
||||
env!("CARGO_PKG_VERSION"),
|
||||
serde_json::json!({"source": source}),
|
||||
vec![],
|
||||
)?;
|
||||
let result = (|| {
|
||||
let stage = tempfile::tempdir_in(&self.config.workspace)?;
|
||||
let entries = self.stage(source, stage.path(), &mut run)?;
|
||||
self.publish(
|
||||
stage.path(),
|
||||
entries,
|
||||
release,
|
||||
Layer::Original,
|
||||
None,
|
||||
&mut run,
|
||||
vec![
|
||||
"Original bytes retained. Extraction coverage has not been established.".into(),
|
||||
],
|
||||
)
|
||||
})();
|
||||
self.finish_run(&mut run, &result)?;
|
||||
result
|
||||
}
|
||||
pub(crate) fn stage(&self, source: &Path, dest: &Path, run: &mut Run) -> Result<Vec<Entry>> {
|
||||
let base = if source.is_dir() {
|
||||
source
|
||||
} else {
|
||||
source.parent().context("missing source parent")?
|
||||
};
|
||||
let mut entries = Vec::new();
|
||||
let mut used = 0u64;
|
||||
for item in WalkDir::new(source).follow_links(false).sort_by_file_name() {
|
||||
let item = item?;
|
||||
if item.path() == base {
|
||||
continue;
|
||||
}
|
||||
let rel = item.path().strip_prefix(base)?;
|
||||
let logical = rel
|
||||
.to_str()
|
||||
.context("non-UTF8 filename is unsupported; input has not been discarded")?
|
||||
.to_owned();
|
||||
safe_path(&logical)?;
|
||||
let meta = fs::symlink_metadata(item.path())?;
|
||||
let out = dest.join(rel);
|
||||
fs::create_dir_all(out.parent().context("missing parent")?)?;
|
||||
let mut entry = Entry {
|
||||
path: logical,
|
||||
kind: EntryKind::File,
|
||||
artifact: None,
|
||||
size: meta.len(),
|
||||
mode: meta.mode(),
|
||||
modified_ns: meta.mtime() as i128 * 1_000_000_000 + meta.mtime_nsec() as i128,
|
||||
link_target: None,
|
||||
};
|
||||
if meta.is_dir() {
|
||||
fs::create_dir(&out)?;
|
||||
entry.kind = EntryKind::Directory;
|
||||
entry.size = 0;
|
||||
} else if meta.file_type().is_symlink() {
|
||||
let target = fs::read_link(item.path())?;
|
||||
entry.kind = EntryKind::Symlink;
|
||||
entry.link_target = Some(
|
||||
target
|
||||
.to_str()
|
||||
.context("non-UTF8 symlink target unsupported")?
|
||||
.into(),
|
||||
);
|
||||
std::os::unix::fs::symlink(target, &out)?;
|
||||
entry.size = 0;
|
||||
} else if meta.is_file() {
|
||||
let mut input = File::open(item.path())?;
|
||||
let mut output = File::create(&out)?;
|
||||
let mut hash = blake3::Hasher::new();
|
||||
let mut buf = vec![0; 1024 * 1024];
|
||||
let mut size = 0;
|
||||
loop {
|
||||
let count = input.read(&mut buf)?;
|
||||
if count == 0 {
|
||||
break;
|
||||
}
|
||||
used += count as u64;
|
||||
size += count as u64;
|
||||
ensure!(
|
||||
used <= self.config.workspace_bytes,
|
||||
"workspace budget exceeded; import is incomplete"
|
||||
);
|
||||
hash.update(&buf[..count]);
|
||||
output.write_all(&buf[..count])?;
|
||||
}
|
||||
let after = input.metadata()?;
|
||||
ensure!(
|
||||
size == meta.len()
|
||||
&& after.mtime() == meta.mtime()
|
||||
&& after.mtime_nsec() == meta.mtime_nsec()
|
||||
&& after.ctime() == meta.ctime()
|
||||
&& after.ctime_nsec() == meta.ctime_nsec(),
|
||||
"source changed while importing {}",
|
||||
entry.path
|
||||
);
|
||||
entry.size = size;
|
||||
entry.artifact = Some(format!("blake3:{}", hash.finalize().to_hex()));
|
||||
run.bytes_processed = used;
|
||||
self.save_run(run)?;
|
||||
} else {
|
||||
bail!("unsupported special file: {}", entry.path);
|
||||
}
|
||||
entries.push(entry);
|
||||
}
|
||||
Ok(entries)
|
||||
}
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub(crate) fn publish(
|
||||
&self,
|
||||
stage: &Path,
|
||||
entries: Vec<Entry>,
|
||||
release: Release,
|
||||
layer: Layer,
|
||||
parent: Option<String>,
|
||||
run: &mut Run,
|
||||
warnings: Vec<String>,
|
||||
) -> Result<Snapshot> {
|
||||
run.stage = "archiving".into();
|
||||
self.save_run(run)?;
|
||||
let commit = self.store.commit_directory(stage)?;
|
||||
let snapshot = Snapshot {
|
||||
schema: SCHEMA,
|
||||
id: Uuid::new_v4().to_string(),
|
||||
release,
|
||||
imported_at: now(),
|
||||
layer,
|
||||
parent,
|
||||
run: run.id.clone(),
|
||||
backend_snapshot: commit.snapshot,
|
||||
logical_bytes: entries.iter().map(|e| e.size).sum(),
|
||||
entries,
|
||||
new_chunk_bytes: commit.unique,
|
||||
new_packed_bytes: commit.packed,
|
||||
warnings,
|
||||
};
|
||||
run.stage = "verifying".into();
|
||||
self.save_run(run)?;
|
||||
self.verify_snapshot(&snapshot)?;
|
||||
atomic_json(
|
||||
&self
|
||||
.config
|
||||
.archive
|
||||
.join("snapshots")
|
||||
.join(format!("{}.json", snapshot.id)),
|
||||
&snapshot,
|
||||
false,
|
||||
)?;
|
||||
Ok(snapshot)
|
||||
}
|
||||
pub fn verify(&self, id: &str) -> Result<()> {
|
||||
self.verify_snapshot(&self.snapshot(id)?)
|
||||
}
|
||||
fn verify_snapshot(&self, snapshot: &Snapshot) -> Result<()> {
|
||||
for e in snapshot
|
||||
.entries
|
||||
.iter()
|
||||
.filter(|e| e.kind == EntryKind::File)
|
||||
{
|
||||
let mut reader = self.store.reader(&snapshot.backend_snapshot, &e.path)?;
|
||||
let mut hash = blake3::Hasher::new();
|
||||
let count = std::io::copy(&mut reader, &mut hash)?;
|
||||
ensure!(
|
||||
count == e.size
|
||||
&& e.artifact.as_deref()
|
||||
== Some(&format!("blake3:{}", hash.finalize().to_hex())),
|
||||
"verification failed: {}",
|
||||
e.path
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub fn read_range(
|
||||
&self,
|
||||
snapshot: &str,
|
||||
path: &str,
|
||||
offset: u64,
|
||||
length: usize,
|
||||
) -> Result<Vec<u8>> {
|
||||
let s = self.snapshot(snapshot)?;
|
||||
self.file_entry(&s, path)?;
|
||||
self.store
|
||||
.read_range(&s.backend_snapshot, path, offset, length)
|
||||
}
|
||||
pub fn reader(&self, snapshot: &str, path: &str) -> Result<Box<dyn Read + Send>> {
|
||||
let s = self.snapshot(snapshot)?;
|
||||
self.file_entry(&s, path)?;
|
||||
self.store.reader(&s.backend_snapshot, path)
|
||||
}
|
||||
pub fn file_entry<'a>(&self, s: &'a Snapshot, path: &str) -> Result<&'a Entry> {
|
||||
s.entries
|
||||
.iter()
|
||||
.find(|e| e.path == path && e.kind == EntryKind::File)
|
||||
.context("file not found in snapshot")
|
||||
}
|
||||
pub fn restore(&self, id: &str, dest: &Path) -> Result<()> {
|
||||
let s = self.snapshot(id)?;
|
||||
ensure!(!dest.exists(), "restore destination must not already exist");
|
||||
let parent = dest
|
||||
.parent()
|
||||
.filter(|p| !p.as_os_str().is_empty())
|
||||
.unwrap_or(Path::new("."));
|
||||
fs::create_dir_all(parent)?;
|
||||
let temp = tempfile::tempdir_in(parent)?;
|
||||
self.materialize(&s, temp.path())?;
|
||||
fs::rename(temp.path(), dest)?;
|
||||
File::open(parent)?.sync_all()?;
|
||||
Ok(())
|
||||
}
|
||||
pub(crate) fn materialize(&self, s: &Snapshot, dest: &Path) -> Result<()> {
|
||||
// Symlinks are created last so no retained link is traversed during writes.
|
||||
for e in s.entries.iter().filter(|e| e.kind != EntryKind::Symlink) {
|
||||
safe_path(&e.path)?;
|
||||
let out = dest.join(&e.path);
|
||||
fs::create_dir_all(out.parent().context("missing output parent")?)?;
|
||||
if e.kind == EntryKind::Directory {
|
||||
fs::create_dir_all(&out)?;
|
||||
continue;
|
||||
}
|
||||
let mut file = OpenOptions::new().write(true).create_new(true).open(&out)?;
|
||||
let mut reader = self.store.reader(&s.backend_snapshot, &e.path)?;
|
||||
let mut hash = blake3::Hasher::new();
|
||||
let mut count = 0u64;
|
||||
let mut buf = vec![0; 1024 * 1024];
|
||||
loop {
|
||||
let n = reader.read(&mut buf)?;
|
||||
if n == 0 {
|
||||
break;
|
||||
}
|
||||
file.write_all(&buf[..n])?;
|
||||
hash.update(&buf[..n]);
|
||||
count += n as u64;
|
||||
}
|
||||
ensure!(
|
||||
count == e.size
|
||||
&& e.artifact.as_deref()
|
||||
== Some(&format!("blake3:{}", hash.finalize().to_hex())),
|
||||
"restoration verification failed"
|
||||
);
|
||||
// Do not restore setuid/setgid bits from imported content.
|
||||
file.set_permissions(fs::Permissions::from_mode(e.mode & 0o777))?;
|
||||
file.sync_all()?;
|
||||
}
|
||||
for e in s.entries.iter().filter(|e| e.kind == EntryKind::Symlink) {
|
||||
safe_path(&e.path)?;
|
||||
let out = dest.join(&e.path);
|
||||
fs::create_dir_all(out.parent().context("missing output parent")?)?;
|
||||
std::os::unix::fs::symlink(
|
||||
e.link_target.as_ref().context("missing symlink target")?,
|
||||
out,
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
pub fn compare(&self, before: &str, after: &str) -> Result<Vec<Change>> {
|
||||
let a = self.snapshot(before)?;
|
||||
let b = self.snapshot(after)?;
|
||||
ensure!(
|
||||
a.layer == b.layer,
|
||||
"select snapshots from the same preservation layer"
|
||||
);
|
||||
let mut paths = BTreeMap::<String, (Option<Entry>, Option<Entry>)>::new();
|
||||
for e in a.entries {
|
||||
let path = e.path.clone();
|
||||
paths.entry(path).or_default().0 = Some(e);
|
||||
}
|
||||
for e in b.entries {
|
||||
let path = e.path.clone();
|
||||
paths.entry(path).or_default().1 = Some(e);
|
||||
}
|
||||
Ok(paths
|
||||
.into_iter()
|
||||
.filter_map(|(path, (a, b))| {
|
||||
let kind = match (&a, &b) {
|
||||
(None, Some(_)) => "added",
|
||||
(Some(_), None) => "removed",
|
||||
(Some(a), Some(b))
|
||||
if a.kind != b.kind
|
||||
|| a.artifact != b.artifact
|
||||
|| a.link_target != b.link_target =>
|
||||
{
|
||||
"changed"
|
||||
}
|
||||
(Some(a), Some(b)) if a.mode != b.mode || a.modified_ns != b.modified_ns => {
|
||||
"metadata_only"
|
||||
}
|
||||
_ => return None,
|
||||
};
|
||||
Some(Change {
|
||||
path,
|
||||
kind: kind.into(),
|
||||
before: a.and_then(|e| e.artifact),
|
||||
after: b.and_then(|e| e.artifact),
|
||||
})
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
}
|
||||
pub fn safe_path(path: &str) -> Result<()> {
|
||||
ensure!(
|
||||
!path.is_empty()
|
||||
&& !path.contains('\\')
|
||||
&& !path.contains('\0')
|
||||
&& Path::new(path)
|
||||
.components()
|
||||
.all(|c| matches!(c, Component::Normal(_))),
|
||||
"unsafe logical path: {path}"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
pub fn validate_release(release: &Release) -> Result<()> {
|
||||
ensure!(
|
||||
!release.repository.trim().is_empty() && !release.version.trim().is_empty(),
|
||||
"repository and version are required"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
+518
@@ -0,0 +1,518 @@
|
||||
use crate::{Archive, Config, Release};
|
||||
use axum::{
|
||||
Json, Router,
|
||||
body::Body,
|
||||
extract::{DefaultBodyLimit, Path, Query, State},
|
||||
http::{HeaderMap, StatusCode, header},
|
||||
middleware::{self, Next},
|
||||
response::{Html, IntoResponse, Response},
|
||||
routing::{get, post},
|
||||
};
|
||||
use serde::Deserialize;
|
||||
use std::sync::Arc;
|
||||
use tokio::io::AsyncWriteExt;
|
||||
|
||||
type App = Arc<Archive>;
|
||||
type Reservation = Arc<tokio::sync::OwnedSemaphorePermit>;
|
||||
struct ApiError(anyhow::Error);
|
||||
impl<E: Into<anyhow::Error>> From<E> for ApiError {
|
||||
fn from(e: E) -> Self {
|
||||
Self(e.into())
|
||||
}
|
||||
}
|
||||
impl IntoResponse for ApiError {
|
||||
fn into_response(self) -> Response {
|
||||
(
|
||||
StatusCode::BAD_REQUEST,
|
||||
Json(serde_json::json!({"error": format!("{:#}", self.0)})),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
}
|
||||
pub fn router(archive: App) -> Router {
|
||||
Router::new()
|
||||
.route(
|
||||
"/",
|
||||
get(|| async { Html(include_str!("../web/index.html")) }),
|
||||
)
|
||||
.route(
|
||||
"/app.js",
|
||||
get(|| async {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "text/javascript")],
|
||||
include_str!("../web/app.js"),
|
||||
)
|
||||
}),
|
||||
)
|
||||
.route(
|
||||
"/style.css",
|
||||
get(|| async {
|
||||
(
|
||||
[(header::CONTENT_TYPE, "text/css")],
|
||||
include_str!("../web/style.css"),
|
||||
)
|
||||
}),
|
||||
)
|
||||
.route("/api/info", get(info))
|
||||
.route("/api/snapshots", get(snapshots))
|
||||
.route("/api/library", get(library))
|
||||
.route("/api/artifacts", get(artifacts))
|
||||
.route("/api/snapshots/{id}", get(snapshot))
|
||||
.route("/api/runs", get(runs))
|
||||
.route("/api/import", post(import))
|
||||
.route(
|
||||
"/api/upload",
|
||||
post(upload).layer(DefaultBodyLimit::disable()),
|
||||
)
|
||||
.route("/api/process", post(process))
|
||||
.route("/api/compare", get(compare))
|
||||
.route(
|
||||
"/api/functions/compare",
|
||||
get(functions).post(save_functions),
|
||||
)
|
||||
.route("/api/file/{id}", get(download))
|
||||
.route("/api/verify/{id}", post(verify))
|
||||
.layer(middleware::from_fn(browser_boundary))
|
||||
.layer(middleware::from_fn_with_state(
|
||||
Arc::new(tokio::sync::Semaphore::new(1)),
|
||||
mutation_budget,
|
||||
))
|
||||
.with_state(archive)
|
||||
}
|
||||
async fn mutation_budget(
|
||||
State(gate): State<Arc<tokio::sync::Semaphore>>,
|
||||
mut request: axum::extract::Request,
|
||||
next: Next,
|
||||
) -> Response {
|
||||
if request.method() != axum::http::Method::POST {
|
||||
return next.run(request).await;
|
||||
}
|
||||
let Ok(_permit) = gate.try_acquire_owned() else {
|
||||
return (
|
||||
StatusCode::CONFLICT,
|
||||
"another mutation is running; retry after it completes",
|
||||
)
|
||||
.into_response();
|
||||
};
|
||||
let reservation = Arc::new(_permit);
|
||||
request.extensions_mut().insert(reservation.clone());
|
||||
next.run(request).await
|
||||
}
|
||||
async fn browser_boundary(request: axum::extract::Request, next: Next) -> Response {
|
||||
if request.method() == axum::http::Method::POST
|
||||
&& request
|
||||
.headers()
|
||||
.get("x-verstack-client")
|
||||
.is_none_or(|v| v != "1")
|
||||
{
|
||||
return (StatusCode::FORBIDDEN, "missing X-Verstack-Client: 1").into_response();
|
||||
}
|
||||
if let Some(origin) = request.headers().get(header::ORIGIN) {
|
||||
let host = request
|
||||
.headers()
|
||||
.get(header::HOST)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or("");
|
||||
if ![format!("http://{host}"), format!("https://{host}")]
|
||||
.iter()
|
||||
.any(|v| origin == v.as_str())
|
||||
{
|
||||
return (StatusCode::FORBIDDEN, "cross-origin request rejected").into_response();
|
||||
}
|
||||
}
|
||||
let mut response = next.run(request).await;
|
||||
let headers = response.headers_mut();
|
||||
headers.insert("x-content-type-options", "nosniff".parse().unwrap());
|
||||
headers.insert("content-security-policy", "default-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' blob:; media-src 'self' blob:; object-src 'none'; frame-ancestors 'none'; base-uri 'none'".parse().unwrap());
|
||||
response
|
||||
}
|
||||
async fn blocking<T: Send + 'static>(
|
||||
f: impl FnOnce() -> anyhow::Result<T> + Send + 'static,
|
||||
) -> Result<T, ApiError> {
|
||||
Ok(tokio::task::spawn_blocking(f).await??)
|
||||
}
|
||||
async fn info(State(a): State<App>) -> Json<serde_json::Value> {
|
||||
Json(
|
||||
serde_json::json!({"schema":1,"plugins":a.config.plugins.keys().collect::<Vec<_>>(),"import_roots":a.config.import_roots,"workspace_bytes":a.config.workspace_bytes,"storage":"rustic-restic"}),
|
||||
)
|
||||
}
|
||||
async fn snapshots(State(a): State<App>) -> Result<Json<Vec<crate::Snapshot>>, ApiError> {
|
||||
Ok(Json(blocking(move || a.snapshots()).await?))
|
||||
}
|
||||
// Keep entry manifests out of the library response; fetch only the visible page.
|
||||
async fn library(State(a): State<App>) -> Result<Json<Vec<serde_json::Value>>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || {
|
||||
a.snapshots()?
|
||||
.into_iter()
|
||||
.map(|s| {
|
||||
let count = s.entries.len();
|
||||
let mut value = serde_json::to_value(s)?;
|
||||
value.as_object_mut().unwrap().remove("entries");
|
||||
value["entry_count"] = count.into();
|
||||
Ok(value)
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct ArtifactQuery {
|
||||
repository: String,
|
||||
version: String,
|
||||
#[serde(default)]
|
||||
edition: String,
|
||||
#[serde(default)]
|
||||
generation: String,
|
||||
#[serde(default)]
|
||||
source: String,
|
||||
#[serde(default)]
|
||||
search: String,
|
||||
#[serde(default)]
|
||||
kind: String,
|
||||
#[serde(default)]
|
||||
page: usize,
|
||||
}
|
||||
async fn artifacts(
|
||||
State(a): State<App>,
|
||||
Query(q): Query<ArtifactQuery>,
|
||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||
Ok(Json(blocking(move || {
|
||||
let mut rows = Vec::new();
|
||||
let search = q.search.to_lowercase();
|
||||
for s in a.snapshots()? {
|
||||
if s.release.repository != q.repository || s.release.version != q.version
|
||||
|| s.release.edition != q.edition || s.release.generation != q.generation
|
||||
|| (!q.source.is_empty() && s.id != q.source) { continue; }
|
||||
for e in s.entries {
|
||||
let media = preview_type(&e.path).filter(|_| e.kind == crate::EntryKind::File);
|
||||
let kind = media.map(|m| m.split('/').next().unwrap()).unwrap_or("file");
|
||||
if !e.path.to_lowercase().contains(&search) { continue; }
|
||||
if !q.kind.is_empty() && q.kind != "all" &&
|
||||
!(q.kind == "media" && media.is_some()) && q.kind != kind { continue; }
|
||||
rows.push(serde_json::json!({"entry":e,"snapshot":s.id,"layer":s.layer,"run":s.run,"media_type":media}));
|
||||
}
|
||||
}
|
||||
rows.sort_by(|a,b| a["entry"]["path"].as_str().cmp(&b["entry"]["path"].as_str())
|
||||
.then(a["snapshot"].as_str().cmp(&b["snapshot"].as_str())));
|
||||
let total = rows.len();
|
||||
let page = q.page.min(total.saturating_sub(1) / 24);
|
||||
let items: Vec<_> = rows.into_iter().skip(page * 24).take(24).collect();
|
||||
Ok(serde_json::json!({"items":items,"total":total,"page":page,"page_size":24}))
|
||||
}).await?))
|
||||
}
|
||||
async fn snapshot(
|
||||
State(a): State<App>,
|
||||
Path(id): Path<String>,
|
||||
) -> Result<Json<crate::Snapshot>, ApiError> {
|
||||
Ok(Json(blocking(move || a.snapshot(&id)).await?))
|
||||
}
|
||||
async fn runs(State(a): State<App>) -> Result<Json<Vec<crate::Run>>, ApiError> {
|
||||
Ok(Json(blocking(move || a.runs()).await?))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct Import {
|
||||
path: std::path::PathBuf,
|
||||
release: Release,
|
||||
}
|
||||
async fn import(
|
||||
State(a): State<App>,
|
||||
axum::Extension(reservation): axum::Extension<Reservation>,
|
||||
Json(input): Json<Import>,
|
||||
) -> Result<Json<crate::Snapshot>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || {
|
||||
let _reservation = reservation;
|
||||
a.import(&input.path, input.release)
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct Process {
|
||||
snapshot: String,
|
||||
plugin: String,
|
||||
}
|
||||
async fn process(
|
||||
State(a): State<App>,
|
||||
axum::Extension(reservation): axum::Extension<Reservation>,
|
||||
Json(input): Json<Process>,
|
||||
) -> Result<Json<crate::Snapshot>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || {
|
||||
let _reservation = reservation;
|
||||
a.process(&input.snapshot, &input.plugin)
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct Compare {
|
||||
before: String,
|
||||
after: String,
|
||||
}
|
||||
async fn compare(
|
||||
State(a): State<App>,
|
||||
Query(q): Query<Compare>,
|
||||
) -> Result<Json<Vec<crate::Change>>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || a.compare(&q.before, &q.after)).await?,
|
||||
))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct Functions {
|
||||
before: String,
|
||||
before_path: String,
|
||||
after: String,
|
||||
after_path: String,
|
||||
}
|
||||
async fn functions(
|
||||
State(a): State<App>,
|
||||
Query(q): Query<Functions>,
|
||||
) -> Result<Json<crate::analysis::FunctionComparison>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || {
|
||||
a.compare_function_files(&q.before, &q.before_path, &q.after, &q.after_path)
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
async fn save_functions(
|
||||
State(a): State<App>,
|
||||
axum::Extension(reservation): axum::Extension<Reservation>,
|
||||
Json(q): Json<Functions>,
|
||||
) -> Result<Json<crate::Snapshot>, ApiError> {
|
||||
Ok(Json(
|
||||
blocking(move || {
|
||||
let _reservation = reservation;
|
||||
a.save_function_comparison(&q.before, &q.before_path, &q.after, &q.after_path)
|
||||
})
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
async fn verify(
|
||||
State(a): State<App>,
|
||||
axum::Extension(reservation): axum::Extension<Reservation>,
|
||||
Path(id): Path<String>,
|
||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||
blocking(move || {
|
||||
let _reservation = reservation;
|
||||
a.verify(&id)
|
||||
})
|
||||
.await?;
|
||||
Ok(Json(serde_json::json!({"verified":true})))
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct FileQuery {
|
||||
path: String,
|
||||
#[serde(default)]
|
||||
inline: bool,
|
||||
}
|
||||
pub fn byte_range(value: Option<&str>, size: u64) -> anyhow::Result<(u64, u64, bool)> {
|
||||
let Some(value) = value else {
|
||||
return Ok((0, size, false));
|
||||
};
|
||||
anyhow::ensure!(size > 0, "empty file has no satisfiable range");
|
||||
let spec = value
|
||||
.strip_prefix("bytes=")
|
||||
.ok_or_else(|| anyhow::anyhow!("unsupported range unit"))?;
|
||||
let (start, end) = spec
|
||||
.split_once('-')
|
||||
.ok_or_else(|| anyhow::anyhow!("invalid byte range"))?;
|
||||
if start.is_empty() {
|
||||
let suffix: u64 = end.parse()?;
|
||||
anyhow::ensure!(suffix > 0, "empty suffix range");
|
||||
return Ok((size.saturating_sub(suffix), size.min(suffix), true));
|
||||
}
|
||||
let start: u64 = start.parse()?;
|
||||
let end = if end.is_empty() {
|
||||
size - 1
|
||||
} else {
|
||||
end.parse::<u64>()?.min(size - 1)
|
||||
};
|
||||
anyhow::ensure!(start <= end && start < size, "unsatisfiable byte range");
|
||||
Ok((start, end - start + 1, true))
|
||||
}
|
||||
fn preview_type(path: &str) -> Option<&'static str> {
|
||||
match path.rsplit('.').next()?.to_ascii_lowercase().as_str() {
|
||||
"png" => Some("image/png"),
|
||||
"jpg" | "jpeg" => Some("image/jpeg"),
|
||||
"gif" => Some("image/gif"),
|
||||
"webp" => Some("image/webp"),
|
||||
"avif" => Some("image/avif"),
|
||||
"bmp" => Some("image/bmp"),
|
||||
"ico" => Some("image/x-icon"),
|
||||
"wav" => Some("audio/wav"),
|
||||
"mp3" => Some("audio/mpeg"),
|
||||
"ogg" | "oga" => Some("audio/ogg"),
|
||||
"flac" => Some("audio/flac"),
|
||||
"m4a" => Some("audio/mp4"),
|
||||
"opus" => Some("audio/ogg"),
|
||||
"ogv" => Some("video/ogg"),
|
||||
"mov" => Some("video/quicktime"),
|
||||
"mp4" => Some("video/mp4"),
|
||||
"webm" => Some("video/webm"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
async fn download(
|
||||
State(a): State<App>,
|
||||
Path(id): Path<String>,
|
||||
Query(q): Query<FileQuery>,
|
||||
headers: HeaderMap,
|
||||
) -> Result<Response, ApiError> {
|
||||
let s = a.snapshot(&id)?;
|
||||
let entry = a.file_entry(&s, &q.path)?;
|
||||
let media_type = if q.inline {
|
||||
preview_type(&q.path)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let range = byte_range(
|
||||
headers.get(header::RANGE).map(|v| v.to_str()).transpose()?,
|
||||
entry.size,
|
||||
);
|
||||
let (offset, length, partial) = match range {
|
||||
Ok(v) => v,
|
||||
Err(_) => {
|
||||
return Ok((
|
||||
StatusCode::RANGE_NOT_SATISFIABLE,
|
||||
[(header::CONTENT_RANGE, format!("bytes */{}", entry.size))],
|
||||
)
|
||||
.into_response());
|
||||
}
|
||||
};
|
||||
let (mut tx, rx) = tokio::io::duplex(128 * 1024);
|
||||
if partial {
|
||||
let backend = s.backend_snapshot.clone();
|
||||
let path = q.path.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut cursor = offset;
|
||||
let end = offset + length;
|
||||
while cursor < end {
|
||||
let a = a.clone();
|
||||
let backend = backend.clone();
|
||||
let path = path.clone();
|
||||
let len = (end - cursor).min(1024 * 1024) as usize;
|
||||
let result = blocking(move || {
|
||||
use crate::storage::ArtifactStorage;
|
||||
a.store.read_range(&backend, &path, cursor, len)
|
||||
})
|
||||
.await;
|
||||
let Ok(bytes) = result else {
|
||||
break;
|
||||
};
|
||||
if bytes.is_empty() {
|
||||
break;
|
||||
}
|
||||
cursor += bytes.len() as u64;
|
||||
if tx.write_all(&bytes).await.is_err() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
});
|
||||
} else {
|
||||
let mut reader = blocking(move || a.reader(&id, &q.path)).await?;
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let mut tx = tokio_util::io::SyncIoBridge::new(tx);
|
||||
let _ = std::io::copy(&mut reader, &mut tx);
|
||||
});
|
||||
}
|
||||
let mut response = Response::new(Body::from_stream(tokio_util::io::ReaderStream::new(rx)));
|
||||
*response.status_mut() = if partial {
|
||||
StatusCode::PARTIAL_CONTENT
|
||||
} else {
|
||||
StatusCode::OK
|
||||
};
|
||||
let h = response.headers_mut();
|
||||
h.insert(header::CONTENT_LENGTH, length.into());
|
||||
h.insert(header::ACCEPT_RANGES, "bytes".parse()?);
|
||||
h.insert(
|
||||
header::CONTENT_TYPE,
|
||||
media_type.unwrap_or("application/octet-stream").parse()?,
|
||||
);
|
||||
h.insert(
|
||||
header::CONTENT_DISPOSITION,
|
||||
if media_type.is_some() {
|
||||
"inline"
|
||||
} else {
|
||||
"attachment"
|
||||
}
|
||||
.parse()?,
|
||||
);
|
||||
if partial {
|
||||
h.insert(
|
||||
header::CONTENT_RANGE,
|
||||
format!("bytes {}-{}/{}", offset, offset + length - 1, entry.size).parse()?,
|
||||
);
|
||||
}
|
||||
Ok(response)
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct UploadQuery {
|
||||
repository: String,
|
||||
version: String,
|
||||
filename: String,
|
||||
#[serde(default)]
|
||||
edition: String,
|
||||
#[serde(default)]
|
||||
generation: String,
|
||||
#[serde(default)]
|
||||
released_at: Option<String>,
|
||||
}
|
||||
async fn upload(
|
||||
State(a): State<App>,
|
||||
axum::Extension(reservation): axum::Extension<Reservation>,
|
||||
Query(q): Query<UploadQuery>,
|
||||
body: Body,
|
||||
) -> Result<Json<crate::Snapshot>, ApiError> {
|
||||
crate::archive::safe_path(&q.filename)?;
|
||||
if q.filename.contains('/') {
|
||||
return Err(anyhow::anyhow!("upload filename must be a basename").into());
|
||||
}
|
||||
let temp = tempfile::tempdir_in(&a.config.workspace)?;
|
||||
let path = temp.path().join(&q.filename);
|
||||
let mut file = tokio::fs::File::create(&path).await?;
|
||||
use http_body_util::BodyExt;
|
||||
let mut body = body;
|
||||
let mut size = 0u64;
|
||||
while let Some(frame) = body.frame().await {
|
||||
let frame = frame?;
|
||||
if let Ok(data) = frame.into_data() {
|
||||
size += data.len() as u64;
|
||||
if size > a.config.workspace_bytes / 2 {
|
||||
return Err(anyhow::anyhow!("upload exceeds half the workspace budget").into());
|
||||
}
|
||||
file.write_all(&data).await?;
|
||||
}
|
||||
}
|
||||
file.sync_all().await?;
|
||||
drop(file);
|
||||
let release = Release {
|
||||
repository: q.repository,
|
||||
version: q.version,
|
||||
edition: q.edition,
|
||||
generation: q.generation,
|
||||
released_at: q.released_at.filter(|s| !s.is_empty()),
|
||||
};
|
||||
let result = blocking(move || {
|
||||
let _reservation = reservation;
|
||||
let _temp = temp;
|
||||
a.import_managed(&path, release)
|
||||
})
|
||||
.await?;
|
||||
Ok(Json(result))
|
||||
}
|
||||
pub async fn serve(config: Config) -> anyhow::Result<()> {
|
||||
let bind = config.bind.clone();
|
||||
let a = Arc::new(Archive::open(config)?);
|
||||
let listener = tokio::net::TcpListener::bind(&bind).await?;
|
||||
eprintln!("Archive UI: http://{}", listener.local_addr()?);
|
||||
axum::serve(listener, router(a))
|
||||
.with_graceful_shutdown(async {
|
||||
let _ = tokio::signal::ctrl_c().await;
|
||||
})
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
//! Library-first artifact archive and analysis framework.
|
||||
pub mod analysis;
|
||||
pub mod archive;
|
||||
pub mod http;
|
||||
pub mod model;
|
||||
pub mod plugins;
|
||||
pub mod storage;
|
||||
pub use archive::Archive;
|
||||
pub use model::*;
|
||||
+128
@@ -0,0 +1,128 @@
|
||||
use anyhow::Result;
|
||||
use clap::{Parser, Subcommand};
|
||||
use std::{fs::File, path::PathBuf};
|
||||
use verstack::{Archive, Config, Release};
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(
|
||||
version,
|
||||
about = "Local artifact archive and analysis framework (working name)"
|
||||
)]
|
||||
struct Args {
|
||||
#[arg(long, default_value = "config.json")]
|
||||
config: PathBuf,
|
||||
#[command(subcommand)]
|
||||
command: Commands,
|
||||
}
|
||||
#[derive(Subcommand)]
|
||||
enum Commands {
|
||||
Serve,
|
||||
Init,
|
||||
List,
|
||||
Runs,
|
||||
Import {
|
||||
path: PathBuf,
|
||||
#[arg(long)]
|
||||
repository: String,
|
||||
#[arg(long)]
|
||||
version: String,
|
||||
#[arg(long, default_value = "")]
|
||||
edition: String,
|
||||
#[arg(long, default_value = "")]
|
||||
generation: String,
|
||||
#[arg(long)]
|
||||
released_at: Option<String>,
|
||||
},
|
||||
Restore {
|
||||
snapshot: String,
|
||||
destination: PathBuf,
|
||||
},
|
||||
Verify {
|
||||
snapshot: String,
|
||||
},
|
||||
Compare {
|
||||
before: String,
|
||||
after: String,
|
||||
},
|
||||
Process {
|
||||
snapshot: String,
|
||||
plugin: String,
|
||||
},
|
||||
Functions {
|
||||
before: String,
|
||||
before_path: String,
|
||||
after: String,
|
||||
after_path: String,
|
||||
#[arg(long)]
|
||||
save: bool,
|
||||
},
|
||||
}
|
||||
fn print(value: &impl serde::Serialize) -> Result<()> {
|
||||
println!("{}", serde_json::to_string_pretty(value)?);
|
||||
Ok(())
|
||||
}
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
let args = Args::parse();
|
||||
let config: Config = serde_json::from_reader(File::open(args.config)?)?;
|
||||
if matches!(args.command, Commands::Serve) {
|
||||
return verstack::http::serve(config).await;
|
||||
}
|
||||
let archive = Archive::open(config)?;
|
||||
match args.command {
|
||||
Commands::Init => print(&serde_json::json!({"initialized": true})),
|
||||
Commands::List => print(&archive.snapshots()?),
|
||||
Commands::Runs => print(&archive.runs()?),
|
||||
Commands::Import {
|
||||
path,
|
||||
repository,
|
||||
version,
|
||||
edition,
|
||||
generation,
|
||||
released_at,
|
||||
} => print(&archive.import(
|
||||
&path,
|
||||
Release {
|
||||
repository,
|
||||
version,
|
||||
edition,
|
||||
generation,
|
||||
released_at,
|
||||
},
|
||||
)?),
|
||||
Commands::Restore {
|
||||
snapshot,
|
||||
destination,
|
||||
} => archive.restore(&snapshot, &destination),
|
||||
Commands::Verify { snapshot } => {
|
||||
archive.verify(&snapshot)?;
|
||||
print(&serde_json::json!({"verified":true}))
|
||||
}
|
||||
Commands::Compare { before, after } => print(&archive.compare(&before, &after)?),
|
||||
Commands::Process { snapshot, plugin } => print(&archive.process(&snapshot, &plugin)?),
|
||||
Commands::Functions {
|
||||
before,
|
||||
before_path,
|
||||
after,
|
||||
after_path,
|
||||
save,
|
||||
} => {
|
||||
if save {
|
||||
print(&archive.save_function_comparison(
|
||||
&before,
|
||||
&before_path,
|
||||
&after,
|
||||
&after_path,
|
||||
)?)
|
||||
} else {
|
||||
print(&archive.compare_function_files(
|
||||
&before,
|
||||
&before_path,
|
||||
&after,
|
||||
&after_path,
|
||||
)?)
|
||||
}
|
||||
}
|
||||
Commands::Serve => unreachable!(),
|
||||
}
|
||||
}
|
||||
+127
@@ -0,0 +1,127 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{collections::BTreeMap, path::PathBuf};
|
||||
|
||||
pub const SCHEMA: u32 = 1;
|
||||
pub fn now() -> u64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs()
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Config {
|
||||
pub archive: PathBuf,
|
||||
pub workspace: PathBuf,
|
||||
pub import_roots: Vec<PathBuf>,
|
||||
#[serde(default = "default_budget")]
|
||||
pub workspace_bytes: u64,
|
||||
#[serde(default = "default_bind")]
|
||||
pub bind: String,
|
||||
#[serde(default)]
|
||||
pub plugins: BTreeMap<String, PluginConfig>,
|
||||
}
|
||||
fn default_budget() -> u64 {
|
||||
100 * 1024 * 1024 * 1024
|
||||
}
|
||||
fn default_bind() -> String {
|
||||
"127.0.0.1:8080".into()
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct PluginConfig {
|
||||
pub command: Vec<String>,
|
||||
pub version: String,
|
||||
#[serde(default)]
|
||||
pub settings: serde_json::Value,
|
||||
#[serde(default = "default_timeout")]
|
||||
pub timeout_seconds: u64,
|
||||
}
|
||||
fn default_timeout() -> u64 {
|
||||
7200
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Release {
|
||||
pub repository: String,
|
||||
pub version: String,
|
||||
#[serde(default)]
|
||||
pub edition: String,
|
||||
#[serde(default)]
|
||||
pub generation: String,
|
||||
/// Explicit publisher chronology; never inferred from import order.
|
||||
#[serde(default)]
|
||||
pub released_at: Option<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum Layer {
|
||||
Original,
|
||||
Extracted,
|
||||
Derived,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum EntryKind {
|
||||
File,
|
||||
Directory,
|
||||
Symlink,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Entry {
|
||||
pub path: String,
|
||||
pub kind: EntryKind,
|
||||
pub artifact: Option<String>,
|
||||
pub size: u64,
|
||||
pub mode: u32,
|
||||
pub modified_ns: i128,
|
||||
pub link_target: Option<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Snapshot {
|
||||
pub schema: u32,
|
||||
pub id: String,
|
||||
pub release: Release,
|
||||
pub imported_at: u64,
|
||||
pub layer: Layer,
|
||||
pub parent: Option<String>,
|
||||
pub run: String,
|
||||
pub backend_snapshot: String,
|
||||
pub entries: Vec<Entry>,
|
||||
pub logical_bytes: u64,
|
||||
pub new_chunk_bytes: u64,
|
||||
pub new_packed_bytes: u64,
|
||||
pub warnings: Vec<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RunState {
|
||||
Running,
|
||||
Complete,
|
||||
Failed,
|
||||
Interrupted,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Run {
|
||||
pub schema: u32,
|
||||
pub id: String,
|
||||
pub operation: String,
|
||||
pub tool_version: String,
|
||||
pub settings: serde_json::Value,
|
||||
pub inputs: Vec<String>,
|
||||
pub started_at: u64,
|
||||
pub state: RunState,
|
||||
pub stage: String,
|
||||
pub bytes_processed: u64,
|
||||
pub output: Option<String>,
|
||||
pub error: Option<String>,
|
||||
}
|
||||
#[derive(Clone, Debug, Serialize, Deserialize)]
|
||||
pub struct Change {
|
||||
pub path: String,
|
||||
pub kind: String,
|
||||
pub before: Option<String>,
|
||||
pub after: Option<String>,
|
||||
}
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
//! Trusted executable plugins: JSON control files and ordinary workspace files.
|
||||
use crate::{Archive, archive::safe_path, model::*};
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{
|
||||
fs::{self, File},
|
||||
path::Path,
|
||||
process::{Command, Stdio},
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct PluginRequest<'a> {
|
||||
pub protocol: u32,
|
||||
pub input_dir: &'a Path,
|
||||
pub output_dir: &'a Path,
|
||||
pub result_file: &'a Path,
|
||||
pub settings: &'a serde_json::Value,
|
||||
pub workspace_bytes: u64,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct PluginResult {
|
||||
pub protocol: u32,
|
||||
pub layer: Layer,
|
||||
pub coverage: String,
|
||||
pub warnings: Vec<String>,
|
||||
/// Every output file must be declared; directories are implicit.
|
||||
pub files: Vec<String>,
|
||||
}
|
||||
impl Archive {
|
||||
pub fn process(&self, snapshot: &str, plugin: &str) -> Result<Snapshot> {
|
||||
let _guard = self
|
||||
.writer
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("writer lock poisoned"))?;
|
||||
let input = self.snapshot(snapshot)?;
|
||||
let cfg = self
|
||||
.config
|
||||
.plugins
|
||||
.get(plugin)
|
||||
.context("plugin is not configured")?;
|
||||
ensure!(
|
||||
!cfg.command.is_empty() && !cfg.version.is_empty(),
|
||||
"plugin command and version are required"
|
||||
);
|
||||
let identities = input
|
||||
.entries
|
||||
.iter()
|
||||
.filter_map(|e| e.artifact.clone())
|
||||
.collect();
|
||||
let mut run = self.start_run(plugin, &cfg.version, cfg.settings.clone(), identities)?;
|
||||
let result = (|| {
|
||||
ensure!(
|
||||
input.logical_bytes < self.config.workspace_bytes / 2,
|
||||
"input exceeds plugin workspace allowance (half of total budget)"
|
||||
);
|
||||
// Ghidra rejects dot-prefixed project path components. Keep plugin
|
||||
// workspaces filesystem-shaped without tempfile's default `.tmp` prefix.
|
||||
let work = tempfile::Builder::new()
|
||||
.prefix("run-")
|
||||
.tempdir_in(&self.config.workspace)?;
|
||||
let inputs = work.path().join("input");
|
||||
let outputs = work.path().join("output");
|
||||
fs::create_dir(&inputs)?;
|
||||
fs::create_dir(&outputs)?;
|
||||
self.materialize(&input, &inputs)?;
|
||||
let result_file = work.path().join("result.json");
|
||||
let request_file = work.path().join("request.json");
|
||||
let request = PluginRequest {
|
||||
protocol: 1,
|
||||
input_dir: &inputs,
|
||||
output_dir: &outputs,
|
||||
result_file: &result_file,
|
||||
settings: &cfg.settings,
|
||||
workspace_bytes: self.config.workspace_bytes - input.logical_bytes,
|
||||
};
|
||||
serde_json::to_writer(File::create(&request_file)?, &request)?;
|
||||
run.stage = "processing".into();
|
||||
self.save_run(&run)?;
|
||||
let mut command = Command::new(&cfg.command[0]);
|
||||
command
|
||||
.args(&cfg.command[1..])
|
||||
.arg(&request_file)
|
||||
.current_dir(work.path())
|
||||
.stdin(Stdio::null())
|
||||
.stdout(Stdio::null())
|
||||
.stderr(Stdio::null());
|
||||
// A process group lets timeout handling stop the tool's descendants too.
|
||||
use std::os::unix::process::CommandExt;
|
||||
command.process_group(0);
|
||||
let mut child = command.spawn().context("could not start plugin")?;
|
||||
let _process_group = ProcessGroup(child.id() as i32);
|
||||
let started = Instant::now();
|
||||
let status = loop {
|
||||
if let Some(status) = child.try_wait()? {
|
||||
break status;
|
||||
}
|
||||
if started.elapsed() > Duration::from_secs(cfg.timeout_seconds)
|
||||
|| workspace_size(work.path())? > self.config.workspace_bytes
|
||||
{
|
||||
// kill(2) via nix: no shell command or interpolated arguments.
|
||||
let _ = nix::sys::signal::killpg(
|
||||
nix::unistd::Pid::from_raw(child.id() as i32),
|
||||
nix::sys::signal::Signal::SIGKILL,
|
||||
);
|
||||
let _ = child.wait();
|
||||
anyhow::bail!("plugin exceeded its time or workspace budget");
|
||||
}
|
||||
std::thread::sleep(Duration::from_millis(200));
|
||||
};
|
||||
drop(_process_group);
|
||||
if !status.success()
|
||||
&& fs::metadata(&result_file).is_ok_and(|m| m.len() <= 65536)
|
||||
&& let Ok(file) = File::open(&result_file)
|
||||
&& let Ok(value) = serde_json::from_reader::<_, serde_json::Value>(file)
|
||||
&& value["protocol"] == 1
|
||||
&& let Some(error) = value["error"].as_str()
|
||||
{
|
||||
anyhow::bail!(
|
||||
"plugin failed: {}",
|
||||
error.chars().take(4000).collect::<String>()
|
||||
);
|
||||
}
|
||||
ensure!(
|
||||
status.success(),
|
||||
"plugin failed ({status}); inputs remain retained"
|
||||
);
|
||||
ensure!(
|
||||
fs::metadata(&result_file)?.len() <= 8 * 1024 * 1024,
|
||||
"plugin result exceeds 8 MiB"
|
||||
);
|
||||
let result: PluginResult = serde_json::from_reader(File::open(result_file)?)?;
|
||||
ensure!(result.protocol == 1, "unsupported plugin protocol");
|
||||
ensure!(
|
||||
matches!(result.layer, Layer::Extracted | Layer::Derived),
|
||||
"plugin must emit extracted or derived content"
|
||||
);
|
||||
ensure!(
|
||||
["complete", "partial", "unknown"].contains(&result.coverage.as_str()),
|
||||
"invalid coverage"
|
||||
);
|
||||
let mut expected = std::collections::BTreeSet::new();
|
||||
for path in &result.files {
|
||||
safe_path(path)?;
|
||||
ensure!(expected.insert(path.clone()), "duplicate output path");
|
||||
}
|
||||
let mut actual = std::collections::BTreeSet::new();
|
||||
for entry in walkdir::WalkDir::new(&outputs).follow_links(false) {
|
||||
let entry = entry?;
|
||||
ensure!(
|
||||
!entry.file_type().is_symlink(),
|
||||
"plugin output symlinks must be represented as metadata, not live links"
|
||||
);
|
||||
if entry.file_type().is_file() {
|
||||
actual.insert(
|
||||
entry
|
||||
.path()
|
||||
.strip_prefix(&outputs)?
|
||||
.to_str()
|
||||
.context("non-UTF8 plugin path")?
|
||||
.to_owned(),
|
||||
);
|
||||
}
|
||||
}
|
||||
ensure!(
|
||||
actual == expected,
|
||||
"declared output inventory differs from files on disk"
|
||||
);
|
||||
let output_size = workspace_size(&outputs)?;
|
||||
ensure!(
|
||||
workspace_size(work.path())?.saturating_add(output_size)
|
||||
<= self.config.workspace_bytes,
|
||||
"insufficient workspace to capture plugin outputs"
|
||||
);
|
||||
let captured = tempfile::tempdir_in(&self.config.workspace)?;
|
||||
let entries = self.stage(&outputs, captured.path(), &mut run)?;
|
||||
let mut warnings = result.warnings;
|
||||
warnings.push(format!(
|
||||
"Plugin coverage: {}. Parent snapshot retained for reprocessing.",
|
||||
result.coverage
|
||||
));
|
||||
self.publish(
|
||||
captured.path(),
|
||||
entries,
|
||||
input.release.clone(),
|
||||
result.layer,
|
||||
Some(input.id.clone()),
|
||||
&mut run,
|
||||
warnings,
|
||||
)
|
||||
})();
|
||||
self.finish_run(&mut run, &result)?;
|
||||
result
|
||||
}
|
||||
}
|
||||
struct ProcessGroup(i32);
|
||||
impl Drop for ProcessGroup {
|
||||
fn drop(&mut self) {
|
||||
let _ = nix::sys::signal::killpg(
|
||||
nix::unistd::Pid::from_raw(self.0),
|
||||
nix::sys::signal::Signal::SIGKILL,
|
||||
);
|
||||
}
|
||||
}
|
||||
fn workspace_size(path: &Path) -> Result<u64> {
|
||||
let mut size = 0u64;
|
||||
for e in walkdir::WalkDir::new(path).follow_links(false) {
|
||||
let e = e?;
|
||||
if e.file_type().is_file() {
|
||||
size = size.saturating_add(e.metadata()?.len());
|
||||
}
|
||||
}
|
||||
Ok(size)
|
||||
}
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
//! Restic-compatible storage. Artifact hashes are application identities only;
|
||||
//! Rustic owns chunking, compression, encryption, packs, and backend identities.
|
||||
use anyhow::{Context, Result, ensure};
|
||||
use rustic_backend::BackendOptions;
|
||||
use rustic_core::{
|
||||
BackupOptions, ConfigOptions, Credentials, IndexedFullStatus, KeyOptions, PathList, Repository,
|
||||
RepositoryOptions, SnapshotOptions, repofile::MasterKey,
|
||||
};
|
||||
use serde::Serialize;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::{
|
||||
fs::{self, File},
|
||||
io::Write,
|
||||
path::{Path, PathBuf},
|
||||
};
|
||||
|
||||
pub trait ArtifactStorage {
|
||||
fn commit_directory(&self, path: &Path) -> Result<Commit>;
|
||||
fn reader(&self, snapshot: &str, path: &str) -> Result<Box<dyn std::io::Read + Send>>;
|
||||
fn read_range(&self, snapshot: &str, path: &str, offset: u64, length: usize)
|
||||
-> Result<Vec<u8>>;
|
||||
}
|
||||
pub struct Commit {
|
||||
pub snapshot: String,
|
||||
pub unique: u64,
|
||||
pub packed: u64,
|
||||
}
|
||||
pub struct RusticStore {
|
||||
root: PathBuf,
|
||||
indexed: Mutex<Option<Arc<Repository<IndexedFullStatus>>>>,
|
||||
}
|
||||
impl RusticStore {
|
||||
pub fn open(root: &Path) -> Result<Self> {
|
||||
let store = Self {
|
||||
root: root.to_owned(),
|
||||
indexed: Mutex::new(None),
|
||||
};
|
||||
let key_path = root.join("archive-key.json");
|
||||
let backend = store.backends()?;
|
||||
if !root.join("store/config").exists() {
|
||||
ensure!(
|
||||
fs::read_dir(root.join("snapshots"))?
|
||||
.all(|e| e.is_ok_and(|e| e.path().extension().is_none_or(|ext| ext != "json"))),
|
||||
"backend config is missing from an existing archive; restore it from backup instead of reinitializing"
|
||||
);
|
||||
// Preserve a key left by an interrupted initialization.
|
||||
if !key_path.exists() {
|
||||
atomic_json(&key_path, &MasterKey::new(), false)?;
|
||||
}
|
||||
Repository::new(&RepositoryOptions::default().no_cache(true), &backend)?.init(
|
||||
&store.credentials()?,
|
||||
&KeyOptions::default(),
|
||||
&ConfigOptions::default(),
|
||||
)?;
|
||||
}
|
||||
store.repo()?;
|
||||
Ok(store)
|
||||
}
|
||||
fn credentials(&self) -> Result<Credentials> {
|
||||
Ok(Credentials::Masterkey(serde_json::from_reader(
|
||||
File::open(self.root.join("archive-key.json"))
|
||||
.context("archive key missing; restore it from your backup")?,
|
||||
)?))
|
||||
}
|
||||
fn backends(&self) -> Result<rustic_core::RepositoryBackends> {
|
||||
Ok(BackendOptions::default()
|
||||
.repository(
|
||||
self.root
|
||||
.join("store")
|
||||
.to_str()
|
||||
.context("non-UTF8 archive path")?,
|
||||
)
|
||||
.to_backends()?)
|
||||
}
|
||||
fn repo(&self) -> Result<Arc<Repository<IndexedFullStatus>>> {
|
||||
let mut indexed = self
|
||||
.indexed
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("storage index lock poisoned"))?;
|
||||
if let Some(repo) = indexed.as_ref() {
|
||||
return Ok(repo.clone());
|
||||
}
|
||||
let repo = Arc::new(
|
||||
Repository::new(
|
||||
&RepositoryOptions::default().no_cache(true),
|
||||
&self.backends()?,
|
||||
)?
|
||||
.open(&self.credentials()?)?
|
||||
.to_indexed()?,
|
||||
);
|
||||
*indexed = Some(repo.clone());
|
||||
Ok(repo)
|
||||
}
|
||||
}
|
||||
struct RusticReader {
|
||||
repo: Arc<Repository<IndexedFullStatus>>,
|
||||
file: rustic_core::vfs::OpenFile,
|
||||
offset: usize,
|
||||
size: usize,
|
||||
}
|
||||
impl std::io::Read for RusticReader {
|
||||
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
|
||||
let count = buf.len().min(self.size.saturating_sub(self.offset));
|
||||
if count == 0 {
|
||||
return Ok(0);
|
||||
}
|
||||
let data = self
|
||||
.repo
|
||||
.read_file_at(&self.file, self.offset, count)
|
||||
.map_err(std::io::Error::other)?;
|
||||
if data.is_empty() {
|
||||
return Err(std::io::Error::other("truncated artifact"));
|
||||
}
|
||||
buf[..data.len()].copy_from_slice(&data);
|
||||
self.offset += data.len();
|
||||
Ok(data.len())
|
||||
}
|
||||
}
|
||||
impl ArtifactStorage for RusticStore {
|
||||
fn commit_directory(&self, path: &Path) -> Result<Commit> {
|
||||
let repo = self.repo()?;
|
||||
let opts = BackupOptions::default().as_path(PathBuf::from("/content"));
|
||||
let source = PathList::from_iter([path.to_owned()]).sanitize()?;
|
||||
let snap = repo.backup(&opts, &source, SnapshotOptions::default().to_snapshot()?)?;
|
||||
*self
|
||||
.indexed
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("storage index lock poisoned"))? = None;
|
||||
// The local backend flushes files before rename. Persist renamed directory
|
||||
// entries before the domain manifest can become visible.
|
||||
for entry in walkdir::WalkDir::new(self.root.join("store")).min_depth(0) {
|
||||
let entry = entry?;
|
||||
if entry.file_type().is_dir() {
|
||||
File::open(entry.path())?.sync_all()?;
|
||||
}
|
||||
}
|
||||
let summary = snap.summary.context("backend omitted import summary")?;
|
||||
Ok(Commit {
|
||||
snapshot: snap.id.to_hex().to_string(),
|
||||
unique: summary.data_added_files,
|
||||
packed: summary.data_added_files_packed,
|
||||
})
|
||||
}
|
||||
fn reader(&self, snapshot: &str, path: &str) -> Result<Box<dyn std::io::Read + Send>> {
|
||||
let repo = self.repo()?;
|
||||
let node = repo.node_from_snapshot_path(&format!("{snapshot}:content/{path}"), |_| true)?;
|
||||
let file = repo.open_file(&node)?;
|
||||
Ok(Box::new(RusticReader {
|
||||
repo,
|
||||
file,
|
||||
offset: 0,
|
||||
size: usize::try_from(node.meta.size)?,
|
||||
}))
|
||||
}
|
||||
fn read_range(
|
||||
&self,
|
||||
snapshot: &str,
|
||||
path: &str,
|
||||
offset: u64,
|
||||
length: usize,
|
||||
) -> Result<Vec<u8>> {
|
||||
ensure!(length <= 8 * 1024 * 1024, "range reads limited to 8 MiB");
|
||||
let repo = self.repo()?;
|
||||
let node = repo.node_from_snapshot_path(&format!("{snapshot}:content/{path}"), |_| true)?;
|
||||
ensure!(offset <= node.meta.size, "range starts past EOF");
|
||||
let length = length.min(usize::try_from(node.meta.size - offset)?);
|
||||
let file = repo.open_file(&node)?;
|
||||
Ok(repo
|
||||
.read_file_at(&file, usize::try_from(offset)?, length)?
|
||||
.to_vec())
|
||||
}
|
||||
}
|
||||
|
||||
/// Flush data before publishing, then flush the containing directory. Temp files
|
||||
/// reside on the same filesystem; immutable records use no-clobber publication.
|
||||
pub fn atomic_json(path: &Path, value: &impl Serialize, replace: bool) -> Result<()> {
|
||||
let parent = path.parent().context("missing parent directory")?;
|
||||
fs::create_dir_all(parent)?;
|
||||
let mut file = tempfile::NamedTempFile::new_in(parent)?;
|
||||
serde_json::to_writer_pretty(&mut file, value)?;
|
||||
file.write_all(b"\n")?;
|
||||
file.as_file().sync_all()?;
|
||||
if replace {
|
||||
file.persist(path)?;
|
||||
} else {
|
||||
file.persist_noclobber(path)?;
|
||||
}
|
||||
File::open(parent)?.sync_all()?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
use verstack::analysis::*;
|
||||
fn function(address: &str, hash: char) -> Function {
|
||||
Function {
|
||||
address: address.into(),
|
||||
name: "unknown".into(),
|
||||
symbol_source: "DEFAULT".into(),
|
||||
size: 64,
|
||||
thunk: false,
|
||||
body_sha256: hash.to_string().repeat(64),
|
||||
}
|
||||
}
|
||||
fn facts(functions: Vec<Function>) -> FunctionFacts {
|
||||
FunctionFacts {
|
||||
schema: 1,
|
||||
input_sha256: "fixture".into(),
|
||||
language: "x86:LE:64:default".into(),
|
||||
compiler: "gcc".into(),
|
||||
ghidra_version: "fixture".into(),
|
||||
analysis_timed_out: false,
|
||||
settings: serde_json::json!({}),
|
||||
functions,
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn unique_exact_matches_keep_ambiguous_and_small_functions_unmatched() {
|
||||
let mut small = function("5", 'f');
|
||||
small.size = 4;
|
||||
let a = facts(vec![
|
||||
function("1", 'a'),
|
||||
function("2", 'b'),
|
||||
function("3", 'b'),
|
||||
small.clone(),
|
||||
]);
|
||||
let b = facts(vec![
|
||||
function("10", 'a'),
|
||||
function("20", 'b'),
|
||||
function("30", 'c'),
|
||||
small,
|
||||
]);
|
||||
let result = compare_functions(&a, &b).unwrap();
|
||||
assert_eq!(result.matches.len(), 1);
|
||||
assert_eq!(result.matches[0].before.address, "1");
|
||||
assert_eq!(result.matches[0].after.address, "10");
|
||||
assert_eq!(result.unmatched_after.len(), 3);
|
||||
}
|
||||
#[test]
|
||||
fn incompatible_or_incomplete_analysis_is_rejected() {
|
||||
let a = facts(vec![]);
|
||||
let mut b = facts(vec![]);
|
||||
b.language = "ARM".into();
|
||||
assert!(compare_functions(&a, &b).is_err());
|
||||
b.language = a.language.clone();
|
||||
b.analysis_timed_out = true;
|
||||
assert!(compare_functions(&a, &b).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn selection_paths_are_not_analysis_settings_but_other_options_are() {
|
||||
let mut a = facts(vec![function("1", 'a')]);
|
||||
let mut b = facts(vec![function("2", 'a')]);
|
||||
a.settings = serde_json::json!({"paths":["pro/game"],"max_cpu":2,"future_option":true});
|
||||
b.settings = serde_json::json!({"paths":["le/game"],"max_cpu":2,"future_option":true});
|
||||
assert_eq!(compare_functions(&a, &b).unwrap().matches.len(), 1);
|
||||
b.settings["future_option"] = false.into();
|
||||
assert!(compare_functions(&a, &b).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unmatched_evidence_distinguishes_exclusions_and_ambiguity() {
|
||||
let a = facts(vec![function("1", 'a'), function("2", 'a')]);
|
||||
let mut unavailable = function("30", 'g');
|
||||
unavailable.body_sha256.clear();
|
||||
let mut thunk = function("40", 'c');
|
||||
thunk.thunk = true;
|
||||
let b = facts(vec![
|
||||
function("10", 'a'),
|
||||
function("20", 'b'),
|
||||
unavailable,
|
||||
thunk,
|
||||
]);
|
||||
let r = compare_functions(&a, &b).unwrap();
|
||||
assert_eq!(r.unmatched_before.len(), 2);
|
||||
assert_eq!(r.unmatched_after_reasons["10"], "ambiguous_exact_body");
|
||||
assert_eq!(r.unmatched_after_reasons["20"], "no_exact_body_match");
|
||||
assert_eq!(r.unmatched_after_reasons["30"], "body_hash_unavailable");
|
||||
assert_eq!(r.unmatched_after_reasons["40"], "thunk_excluded");
|
||||
let duplicate = facts(vec![function("1", 'a'), function("1", 'b')]);
|
||||
assert!(compare_functions(&duplicate, &b).is_err());
|
||||
}
|
||||
@@ -0,0 +1,426 @@
|
||||
use std::{collections::BTreeMap, fs, io::Read, path::Path};
|
||||
use tempfile::TempDir;
|
||||
use verstack::{Archive, Config, PluginConfig, Release, RunState};
|
||||
|
||||
fn setup() -> (TempDir, Config) {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
fs::create_dir(tmp.path().join("inputs")).unwrap();
|
||||
let config = Config {
|
||||
archive: tmp.path().join("archive"),
|
||||
workspace: tmp.path().join("work"),
|
||||
import_roots: vec![tmp.path().join("inputs")],
|
||||
workspace_bytes: 64 * 1024 * 1024,
|
||||
bind: "127.0.0.1:0".into(),
|
||||
plugins: BTreeMap::new(),
|
||||
};
|
||||
(tmp, config)
|
||||
}
|
||||
fn release(game: &str, version: &str) -> Release {
|
||||
Release {
|
||||
repository: game.into(),
|
||||
version: version.into(),
|
||||
edition: "LE".into(),
|
||||
generation: "unknown".into(),
|
||||
released_at: None,
|
||||
}
|
||||
}
|
||||
fn put(root: &Path, name: &str, bytes: &[u8]) {
|
||||
let path = root.join(name);
|
||||
fs::create_dir_all(path.parent().unwrap()).unwrap();
|
||||
fs::write(path, bytes).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cross_game_dedup_restore_out_of_order_and_comparison() -> anyhow::Result<()> {
|
||||
let (_tmp, cfg) = setup();
|
||||
let input = &cfg.import_roots[0];
|
||||
put(input, "build/logo", b"shared publisher logo");
|
||||
put(input, "build/script", b"new code");
|
||||
put(input, "build/empty", b"");
|
||||
fs::create_dir(input.join("build/empty-dir"))?;
|
||||
std::os::unix::fs::symlink("logo", input.join("build/logo-link"))?;
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
let newer = archive.import(&input.join("build"), release("A", "2.0"))?;
|
||||
let other = archive.import(&input.join("build"), release("B", "1.0"))?;
|
||||
assert_eq!(
|
||||
other.new_chunk_bytes, 0,
|
||||
"identical files must reuse shared chunks"
|
||||
);
|
||||
put(input, "build/script", b"old code");
|
||||
let older = archive.import(&input.join("build"), release("A", "1.0"))?;
|
||||
let diff = archive.compare(&older.id, &newer.id)?;
|
||||
assert!(
|
||||
diff.iter()
|
||||
.any(|c| c.path == "script" && c.kind == "changed")
|
||||
);
|
||||
assert_eq!(
|
||||
archive.read_range(&newer.id, "script", 0, 100)?,
|
||||
b"new code"
|
||||
);
|
||||
assert_eq!(archive.read_range(&older.id, "script", 4, 4)?, b"code");
|
||||
let dest = cfg.workspace.parent().unwrap().join("restored");
|
||||
archive.restore(&newer.id, &dest)?;
|
||||
assert_eq!(fs::read(dest.join("script"))?, b"new code");
|
||||
assert_eq!(fs::read_link(dest.join("logo-link"))?, Path::new("logo"));
|
||||
assert!(dest.join("empty-dir").is_dir());
|
||||
assert!(archive.restore(&older.id, &dest).is_err());
|
||||
archive.verify(&newer.id)?;
|
||||
drop(archive);
|
||||
let archive = Archive::open(cfg)?;
|
||||
assert_eq!(archive.snapshots()?.len(), 3);
|
||||
archive.verify(&other.id)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn failed_import_never_publishes_and_keeps_sources() -> anyhow::Result<()> {
|
||||
let (_tmp, mut cfg) = setup();
|
||||
cfg.workspace_bytes = 3;
|
||||
put(&cfg.import_roots[0], "big", b"too large");
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
assert!(
|
||||
archive
|
||||
.import(&cfg.import_roots[0].join("big"), release("A", "1"))
|
||||
.is_err()
|
||||
);
|
||||
assert!(archive.snapshots()?.is_empty());
|
||||
assert!(matches!(archive.runs()?[0].state, RunState::Failed));
|
||||
assert_eq!(fs::read(cfg.import_roots[0].join("big"))?, b"too large");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn exclusive_ownership_and_source_boundaries() -> anyhow::Result<()> {
|
||||
let (tmp, cfg) = setup();
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
assert!(Archive::open(cfg.clone()).is_err());
|
||||
put(tmp.path(), "outside", b"outside");
|
||||
std::os::unix::fs::symlink(
|
||||
tmp.path().join("outside"),
|
||||
cfg.import_roots[0].join("escape"),
|
||||
)?;
|
||||
assert!(
|
||||
archive
|
||||
.import(&cfg.import_roots[0].join("escape"), release("A", "1"))
|
||||
.is_err()
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn zip_plugin_creates_revision_without_rewriting_input() -> anyhow::Result<()> {
|
||||
let (_tmp, mut cfg) = setup();
|
||||
let script = Path::new(env!("CARGO_MANIFEST_DIR")).join("plugins/zip_extract.py");
|
||||
cfg.plugins.insert(
|
||||
"zip".into(),
|
||||
PluginConfig {
|
||||
command: vec!["python3".into(), script.display().to_string()],
|
||||
version: "1".into(),
|
||||
settings: serde_json::json!({}),
|
||||
timeout_seconds: 30,
|
||||
},
|
||||
);
|
||||
let zip = cfg.import_roots[0].join("game.zip");
|
||||
let status=std::process::Command::new("python3").args(["-c","import zipfile,sys; z=zipfile.ZipFile(sys.argv[1],'w'); z.writestr('assets/config.txt','hello'); z.close()"] ).arg(&zip).status()?;
|
||||
assert!(status.success());
|
||||
let archive = Archive::open(cfg)?;
|
||||
let input = archive.import(&zip, release("A", "1"))?;
|
||||
let output = archive.process(&input.id, "zip")?;
|
||||
assert_eq!(output.parent.as_deref(), Some(input.id.as_str()));
|
||||
assert_eq!(output.release, input.release);
|
||||
assert_eq!(
|
||||
archive.read_range(&output.id, "game.zip.entries/assets/config.txt", 0, 100)?,
|
||||
b"hello"
|
||||
);
|
||||
archive.verify(&input.id)?;
|
||||
let second = archive.process(&input.id, "zip")?;
|
||||
assert_ne!(second.id, output.id);
|
||||
assert_eq!(second.new_chunk_bytes, 0);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn interruption_is_visible_after_reopen() -> anyhow::Result<()> {
|
||||
let (_tmp, cfg) = setup();
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
let run = verstack::Run {
|
||||
schema: 1,
|
||||
id: uuid::Uuid::new_v4().to_string(),
|
||||
operation: "fixture".into(),
|
||||
tool_version: "1".into(),
|
||||
settings: serde_json::json!({}),
|
||||
inputs: vec![],
|
||||
started_at: 0,
|
||||
state: RunState::Running,
|
||||
stage: "processing".into(),
|
||||
bytes_processed: 0,
|
||||
output: None,
|
||||
error: None,
|
||||
};
|
||||
fs::write(
|
||||
cfg.archive.join("runs").join(format!("{}.json", run.id)),
|
||||
serde_json::to_vec(&run)?,
|
||||
)?;
|
||||
drop(archive);
|
||||
let archive = Archive::open(cfg)?;
|
||||
assert!(matches!(archive.runs()?[0].state, RunState::Interrupted));
|
||||
assert!(archive.snapshots()?.is_empty());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plugin_failure_reports_structured_diagnostic() -> anyhow::Result<()> {
|
||||
let (tmp, mut cfg) = setup();
|
||||
put(&cfg.import_roots[0], "file", b"test input");
|
||||
let script = tmp.path().join("fail.py");
|
||||
fs::write(
|
||||
&script,
|
||||
"import json,pathlib,sys\nr=json.loads(pathlib.Path(sys.argv[1]).read_text())\nassert not pathlib.Path(r['input_dir']).parent.name.startswith('.')\npathlib.Path(r['result_file']).write_text(json.dumps({'protocol':1,'error':'required tool is unavailable'}))\nsys.exit(1)\n",
|
||||
)?;
|
||||
cfg.plugins.insert(
|
||||
"fail".into(),
|
||||
PluginConfig {
|
||||
command: vec!["python3".into(), script.display().to_string()],
|
||||
version: "fixture/1".into(),
|
||||
settings: serde_json::json!({}),
|
||||
timeout_seconds: 10,
|
||||
},
|
||||
);
|
||||
let a = Archive::open(cfg.clone())?;
|
||||
let s = a.import(&cfg.import_roots[0].join("file"), release("A", "1"))?;
|
||||
assert!(
|
||||
a.process(&s.id, "fail")
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("required tool is unavailable")
|
||||
);
|
||||
assert_eq!(a.snapshots()?.len(), 1);
|
||||
assert!(a.runs()?.iter().any(|r| {
|
||||
r.error
|
||||
.as_deref()
|
||||
.is_some_and(|e| e.contains("required tool is unavailable"))
|
||||
}));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn corrupt_pack_is_detected() -> anyhow::Result<()> {
|
||||
let (_tmp, cfg) = setup();
|
||||
put(&cfg.import_roots[0], "file", b"retained bytes");
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
let snapshot = archive.import(&cfg.import_roots[0].join("file"), release("A", "1"))?;
|
||||
for entry in walkdir::WalkDir::new(cfg.archive.join("store/data")) {
|
||||
let entry = entry?;
|
||||
if entry.file_type().is_file() {
|
||||
let len = entry.metadata()?.len();
|
||||
fs::write(entry.path(), vec![0; len as usize])?;
|
||||
}
|
||||
}
|
||||
assert!(archive.verify(&snapshot.id).is_err());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_backend_configuration_does_not_reinitialize_history() -> anyhow::Result<()> {
|
||||
let (_tmp, cfg) = setup();
|
||||
put(&cfg.import_roots[0], "file", b"retained bytes");
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
archive.import(&cfg.import_roots[0].join("file"), release("A", "1"))?;
|
||||
drop(archive);
|
||||
fs::remove_file(cfg.archive.join("store/config"))?;
|
||||
assert!(Archive::open(cfg.clone()).is_err());
|
||||
assert!(!cfg.archive.join("store/config").exists());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ranges_are_bounded_and_exact() -> anyhow::Result<()> {
|
||||
use verstack::http::byte_range;
|
||||
assert_eq!(byte_range(Some("bytes=3-7"), 10)?, (3, 5, true));
|
||||
assert_eq!(byte_range(Some("bytes=-3"), 10)?, (7, 3, true));
|
||||
assert_eq!(byte_range(Some("bytes=8-"), 10)?, (8, 2, true));
|
||||
for bad in [
|
||||
"bytes=10-",
|
||||
"bytes=-0",
|
||||
"bytes=8-2",
|
||||
"bytes=0-1,4-5",
|
||||
"items=0-1",
|
||||
] {
|
||||
assert!(byte_range(Some(bad), 10).is_err());
|
||||
}
|
||||
assert!(byte_range(Some("bytes=0-"), 0).is_err());
|
||||
let (_tmp, cfg) = setup();
|
||||
put(&cfg.import_roots[0], "file", b"0123456789");
|
||||
let archive = Archive::open(cfg.clone())?;
|
||||
let s = archive.import(&cfg.import_roots[0].join("file"), release("A", "1"))?;
|
||||
assert_eq!(archive.read_range(&s.id, "file", 3, 5)?, b"34567");
|
||||
let mut bytes = Vec::new();
|
||||
archive.reader(&s.id, "file")?.read_to_end(&mut bytes)?;
|
||||
assert_eq!(bytes, b"0123456789");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn api_import_download_and_origin_boundary() -> anyhow::Result<()> {
|
||||
use axum::{body::Body, http::Request};
|
||||
use http_body_util::BodyExt;
|
||||
use tower::ServiceExt;
|
||||
let (_tmp, cfg) = setup();
|
||||
put(&cfg.import_roots[0], "file", b"abcdef");
|
||||
let path = cfg.import_roots[0].join("file");
|
||||
let app = verstack::http::router(std::sync::Arc::new(Archive::open(cfg)?));
|
||||
let payload = serde_json::json!({"path":path,"release":release("A","1")});
|
||||
let rejected = app
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::post("/api/import")
|
||||
.header("content-type", "application/json")
|
||||
.body(Body::from(payload.to_string()))?,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(rejected.status(), 403);
|
||||
let imported = app
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::post("/api/import")
|
||||
.header("content-type", "application/json")
|
||||
.header("x-verstack-client", "1")
|
||||
.body(Body::from(payload.to_string()))?,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(imported.status(), 200);
|
||||
let s: verstack::Snapshot =
|
||||
serde_json::from_slice(&imported.into_body().collect().await?.to_bytes())?;
|
||||
let response = app
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::get(format!("/api/file/{}?path=file", s.id))
|
||||
.header("range", "bytes=1-3")
|
||||
.body(Body::empty())?,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(response.status(), 206);
|
||||
assert_eq!(response.headers()["content-range"], "bytes 1-3/6");
|
||||
assert_eq!(response.into_body().collect().await?.to_bytes(), "bcd");
|
||||
let rejected = app
|
||||
.oneshot(
|
||||
Request::get("/api/info")
|
||||
.header("host", "localhost")
|
||||
.header("origin", "https://other.example")
|
||||
.body(Body::empty())?,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(rejected.status(), 403);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn saved_function_report_retains_both_sources_and_survives_reopen() -> anyhow::Result<()> {
|
||||
let (_tmp, cfg) = setup();
|
||||
let facts = |path: &str| serde_json::json!({"schema":1,"input_sha256":"fixture","language":"ARM","compiler":"gcc","ghidra_version":"fixture","analysis_timed_out":false,"settings":{"paths":[path]},"functions":[{"address":"100","name":"test","symbol_source":"IMPORTED","size":64,"thunk":false,"body_sha256":"a".repeat(64)}]});
|
||||
put(
|
||||
&cfg.import_roots[0],
|
||||
"pro/functions.json",
|
||||
&serde_json::to_vec(&facts("pro/game"))?,
|
||||
);
|
||||
put(
|
||||
&cfg.import_roots[0],
|
||||
"le/functions.json",
|
||||
&serde_json::to_vec(&facts("le/game"))?,
|
||||
);
|
||||
let a = Archive::open(cfg.clone())?;
|
||||
let mut pro = release("game", "1");
|
||||
pro.edition = "Pro".into();
|
||||
let left = a.import(&cfg.import_roots[0].join("pro"), pro)?;
|
||||
let right = a.import(&cfg.import_roots[0].join("le"), release("game", "1"))?;
|
||||
let saved =
|
||||
a.save_function_comparison(&left.id, "functions.json", &right.id, "functions.json")?;
|
||||
assert_eq!(saved.parent, Some(right.id.clone()));
|
||||
assert!(saved.warnings[0].contains("Different editions"));
|
||||
drop(a);
|
||||
let a = Archive::open(cfg.clone())?;
|
||||
a.verify(&saved.id)?;
|
||||
let report: serde_json::Value =
|
||||
serde_json::from_reader(a.reader(&saved.id, "function-comparison.json")?)?;
|
||||
assert_eq!(report["sources"][0]["snapshot"], left.id);
|
||||
assert_eq!(report["sources"][1]["snapshot"], right.id);
|
||||
assert_eq!(report["matches"].as_array().unwrap().len(), 1);
|
||||
assert_eq!(a.snapshot(&left.id)?.entries.len(), left.entries.len());
|
||||
let run = a.runs()?.into_iter().find(|r| r.id == saved.run).unwrap();
|
||||
assert_eq!(run.inputs.len(), 2);
|
||||
assert!(matches!(run.state, RunState::Complete));
|
||||
use tower::ServiceExt;
|
||||
let app = verstack::http::router(std::sync::Arc::new(a));
|
||||
let request = axum::http::Request::builder()
|
||||
.method("POST").uri("/api/functions/compare")
|
||||
.header("X-Verstack-Client", "1").header("Content-Type", "application/json")
|
||||
.body(axum::body::Body::from(serde_json::to_vec(&serde_json::json!({
|
||||
"before":left.id,"before_path":"functions.json","after":right.id,"after_path":"functions.json"
|
||||
}))?))?;
|
||||
tokio::runtime::Runtime::new()?.block_on(async {
|
||||
let response = app.oneshot(request).await.unwrap();
|
||||
assert_eq!(response.status(), axum::http::StatusCode::OK);
|
||||
let body = axum::body::to_bytes(response.into_body(), 1024 * 1024)
|
||||
.await
|
||||
.unwrap();
|
||||
let output: verstack::Snapshot = serde_json::from_slice(&body).unwrap();
|
||||
assert_eq!(output.layer, verstack::Layer::Derived);
|
||||
assert_eq!(output.entries[0].path, "function-comparison.json");
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn library_pages_group_versions_and_filter_media() -> anyhow::Result<()> {
|
||||
use axum::{body::Body, http::Request};
|
||||
use http_body_util::BodyExt;
|
||||
use tower::ServiceExt;
|
||||
let (_tmp, cfg) = setup();
|
||||
for i in 0..30 {
|
||||
put(
|
||||
&cfg.import_roots[0],
|
||||
&format!("build/image-{i:02}.PNG"),
|
||||
b"image",
|
||||
);
|
||||
}
|
||||
put(&cfg.import_roots[0], "build/music.flac", b"audio");
|
||||
put(&cfg.import_roots[0], "build/report.json", b"{}");
|
||||
let a = Archive::open(cfg.clone())?;
|
||||
let first = a.import(&cfg.import_roots[0].join("build"), release("Game", "1"))?;
|
||||
a.import(&cfg.import_roots[0].join("build"), release("Game", "1"))?;
|
||||
a.import(&cfg.import_roots[0].join("build"), release("Other", "1"))?;
|
||||
let app = verstack::http::router(std::sync::Arc::new(a));
|
||||
let read = |url: String| {
|
||||
let app = app.clone();
|
||||
async move {
|
||||
let response = app
|
||||
.oneshot(Request::get(url).body(Body::empty()).unwrap())
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(response.status(), 200);
|
||||
serde_json::from_slice::<serde_json::Value>(
|
||||
&response.into_body().collect().await.unwrap().to_bytes(),
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
};
|
||||
let library = read("/api/library".into()).await;
|
||||
assert_eq!(library.as_array().unwrap().len(), 3);
|
||||
assert!(library[0].get("entries").is_none());
|
||||
assert_eq!(library[0]["entry_count"], 32);
|
||||
let base = "/api/artifacts?repository=Game&version=1&edition=LE&generation=unknown";
|
||||
let page = read(format!("{base}&kind=media")).await;
|
||||
assert_eq!(page["total"], 62);
|
||||
assert_eq!(page["items"].as_array().unwrap().len(), 24);
|
||||
let page = read(format!("{base}&kind=media&page=2")).await;
|
||||
assert_eq!(page["items"].as_array().unwrap().len(), 14);
|
||||
let page = read(format!("{base}&kind=audio&source={}", first.id)).await;
|
||||
assert_eq!(page["total"], 1);
|
||||
assert_eq!(page["items"][0]["media_type"], "audio/flac");
|
||||
let page = read(format!("{base}&kind=all&search=REPORT&page=999")).await;
|
||||
assert_eq!(page["total"], 2);
|
||||
assert_eq!(page["page"], 0);
|
||||
let page = read(format!("{base}&kind=video")).await;
|
||||
assert_eq!(page["total"], 0);
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
spec = importlib.util.spec_from_file_location('luks_check', Path(__file__).resolve().parents[1] / 'plugins/luks_check.py')
|
||||
luks = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(luks)
|
||||
|
||||
class LuksTests(unittest.TestCase):
|
||||
def test_encoding_and_secret_references(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
key = Path(temp) / 'key'
|
||||
key.write_text('0x01020304 0x05060708\n')
|
||||
variants = luks.credentials({'key_file': str(key), 'key_encoding': 'auto'})
|
||||
self.assertEqual(variants['u32le'], bytes([4,3,2,1,8,7,6,5]))
|
||||
self.assertEqual(variants['u32be'], bytes(range(1,9)))
|
||||
with self.assertRaises(ValueError):
|
||||
luks.credentials({'key_file':str(key),'key_env':'SECRET'})
|
||||
|
||||
@unittest.skipUnless(shutil.which('cryptsetup'), 'cryptsetup is not installed')
|
||||
def test_real_luks_header_accepts_correct_key_and_rejects_wrong_key(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root = Path(temp)
|
||||
device = root / 'fixture.luks'
|
||||
with device.open('wb') as f:
|
||||
f.truncate(32 * 1024 * 1024)
|
||||
secret = b'fixture-only-luks-credential'
|
||||
subprocess.run(['cryptsetup','luksFormat','--type','luks2','--pbkdf','pbkdf2','--pbkdf-force-iterations','1000','--batch-mode','--key-file','-',str(device)],input=secret,capture_output=True,check=True)
|
||||
with device.open('rb') as f:
|
||||
header = f.read(luks.HEADER_LIMIT)
|
||||
accepted = luks.check_stream(io.BytesIO(header), {}, {'raw':secret}, root)
|
||||
rejected = luks.check_stream(io.BytesIO(header), {}, {'raw':b'wrong'}, root)
|
||||
self.assertTrue(accepted['unlock_verified'])
|
||||
self.assertEqual(rejected['attempts'], {'raw':'rejected'})
|
||||
self.assertNotIn(secret.decode(), json.dumps(accepted))
|
||||
self.assertIsNone(luks.check_stream(io.BytesIO(b'not luks'),{}, {'raw':secret},root))
|
||||
|
||||
|
||||
class ExtractionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
spec = importlib.util.spec_from_file_location('luks_extract', Path(__file__).resolve().parents[1] / 'plugins/luks_extract.py')
|
||||
self.module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(self.module)
|
||||
|
||||
def test_split_validation_rejects_missing_parts_and_insufficient_budget(self):
|
||||
import zipfile
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root=Path(temp)
|
||||
archive=root/'input.zip'
|
||||
with zipfile.ZipFile(archive,'w') as z: z.writestr('game.spk.002.000',b'test')
|
||||
with self.assertRaisesRegex(ValueError,'missing'):
|
||||
self.module.assemble(archive,root/'joined',1024)
|
||||
with zipfile.ZipFile(archive,'w') as z:
|
||||
z.writestr('game.spk.002.001',b'second')
|
||||
z.writestr('game.spk.002.000',b'first')
|
||||
with self.assertRaisesRegex(ValueError,'workspace'):
|
||||
self.module.assemble(archive,root/'joined',1)
|
||||
self.module.assemble(archive,root/'joined',1024)
|
||||
self.assertEqual((root/'joined').read_bytes(),b'firstsecond')
|
||||
|
||||
@unittest.skipUnless(shutil.which('cryptsetup'), 'cryptsetup is not installed')
|
||||
def test_library_unlock_returns_known_volume_key(self):
|
||||
with tempfile.TemporaryDirectory() as temp:
|
||||
root=Path(temp);volume=bytes(range(32));(root/'volume').write_bytes(volume)
|
||||
with (root/'image').open('wb') as f:f.truncate(32*1024*1024)
|
||||
subprocess.run(['cryptsetup','luksFormat','--type','luks2','--key-size','256','--volume-key-file',str(root/'volume'),'--pbkdf','pbkdf2','--pbkdf-force-iterations','1000','--batch-mode','--key-file','-',str(root/'image')],input=b'fixture',capture_output=True,check=True)
|
||||
self.assertEqual(self.module.volume_key(root/'image',b'fixture'),volume)
|
||||
with self.assertRaisesRegex(ValueError,'credential'):
|
||||
self.module.volume_key(root/'image',b'incorrect')
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,87 @@
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
|
||||
def load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, ROOT / path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
ZIP = load("zip_plugin", "plugins/zip_extract.py")
|
||||
GHIDRA = load("ghidra_plugin", "plugins/ghidra/analyze.py")
|
||||
PROBE = load("spike_probe", "plugins/spike_probe.py")
|
||||
|
||||
class PluginTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = pathlib.Path(self.temp.name)
|
||||
self.input = self.root / "input"
|
||||
self.output = self.root / "output"
|
||||
self.input.mkdir()
|
||||
self.output.mkdir()
|
||||
self.request = {"protocol": 1, "input_dir": str(self.input), "output_dir": str(self.output), "workspace_bytes": 1024, "settings": {}}
|
||||
|
||||
def make_zip(self, name, content):
|
||||
with zipfile.ZipFile(self.input / "test.zip", "w") as archive:
|
||||
archive.writestr(name, content)
|
||||
|
||||
def test_exact_entries_and_opaque_supporting_files(self):
|
||||
self.make_zip("config.txt", b"exact\x00bytes")
|
||||
(self.input / "notes.txt").write_text("readme")
|
||||
result = ZIP.run(self.request)
|
||||
self.assertEqual(result["coverage"], "partial")
|
||||
self.assertEqual((self.output / result["files"][0]).read_bytes(), b"exact\x00bytes")
|
||||
|
||||
def test_traversal_rejected(self):
|
||||
self.make_zip("../../escaped", "content")
|
||||
with self.assertRaises(ValueError):
|
||||
ZIP.run(self.request)
|
||||
self.assertFalse((self.root / "escaped").exists())
|
||||
|
||||
def test_expansion_budget(self):
|
||||
self.make_zip("oversized", b"x" * 1024)
|
||||
with self.assertRaisesRegex(ValueError, "allowance"):
|
||||
ZIP.run(self.request)
|
||||
|
||||
def test_ghidra_missing_tool_is_explicit_failure(self):
|
||||
self.request["settings"] = {"ghidra_home": str(self.root / "missing"), "expected_version": "fixture"}
|
||||
with self.assertRaises(FileNotFoundError):
|
||||
GHIDRA.run(self.request)
|
||||
|
||||
def test_ghidra_rejects_unpinned_version(self):
|
||||
home = self.root / "ghidra"
|
||||
(home / "Ghidra").mkdir(parents=True)
|
||||
(home / "Ghidra" / "application.properties").write_text("application.version=fixture-v2\n")
|
||||
self.request["settings"] = {"ghidra_home": str(home), "expected_version": "fixture-v1"}
|
||||
with self.assertRaisesRegex(ValueError, "version"):
|
||||
GHIDRA.run(self.request)
|
||||
|
||||
def test_probe_distinguishes_wrappers_without_guessing_generation(self):
|
||||
for header, kind in [(b"SPKS", "spk"), (b"hsqs", "squashfs"), (b"\x1f\x8b", "gzip")]:
|
||||
result = PROBE.classify(header)
|
||||
self.assertEqual(result["format"], kind)
|
||||
self.assertEqual(result["generation"], "unknown")
|
||||
self.assertIsNone(result["required_secret"])
|
||||
|
||||
def test_luks_probe_reports_a_reference_not_a_key(self):
|
||||
header = bytearray(4096)
|
||||
header[:8] = b"LUKS\xba\xbe\x00\x02"
|
||||
header[168:204] = b"5b22533c-7ee6-4e7c-8e9b-abb2392be418"
|
||||
result = PROBE.classify(header)
|
||||
self.assertEqual(result["required_secret"], "stern.spike3.luks")
|
||||
self.assertEqual(result["uuid"], "5b22533c-7ee6-4e7c-8e9b-abb2392be418")
|
||||
|
||||
def test_probe_reads_first_split_header_inside_zip(self):
|
||||
self.make_zip("game.spk.002.000", b"hsqs" + bytes(100))
|
||||
report = PROBE.probe(self.input / "test.zip")
|
||||
self.assertEqual(report["entries"][0]["format"], "squashfs")
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,61 @@
|
||||
// Browser-only fixture test; no listener and no real archive writes.
|
||||
import { createRequire } from 'node:module';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import assert from 'node:assert/strict';
|
||||
const require = createRequire(import.meta.url);
|
||||
const { chromium } = require(process.env.VERSTACK_PLAYWRIGHT || 'playwright');
|
||||
const sources = ['Pro', 'LE'].map((edition, i) => ({snapshot:`snapshot-${i}`,path:'binary/functions.json',artifact:`hash-${i}`,run:`run-${i}`,release:{repository:'Fixture',version:'1',edition,generation:'1'}}));
|
||||
const f = i => ({name:`function_${i}`,address:String(i),size:64,thunk:false});
|
||||
const report = {schema:2, sources, matches:[{before:f(1),after:f(2)}],unmatched_after:Array.from({length:201},(_,i)=>f(i+100)),unmatched_before:[f(3)],unmatched_after_reasons:Object.fromEntries(Array.from({length:201},(_,i)=>[String(i+100),'no_exact_body_match'])),caveat:'Different editions selected; not newly added code.'};
|
||||
const snapshots = sources.map((s,i)=>({id:s.snapshot,release:s.release,imported_at:i,layer:'derived',logical_bytes:100,new_packed_bytes:50,warnings:[],entries:[{path:s.path,kind:'file',size:100,artifact:s.artifact}]}));
|
||||
const saved = { ...snapshots[1], id:'saved-report',entries:[{path:'function-comparison.json',kind:'file',size:100,artifact:'saved-hash'}]};
|
||||
let stored = false;
|
||||
const browser = await chromium.launch({headless:true});
|
||||
const page = await browser.newPage({viewport:{width:1440,height:1000}});
|
||||
const errors=[];page.on('pageerror',e=>errors.push(e.message));
|
||||
await page.route('http://verstack.test/**',async route=>{
|
||||
const url=new URL(route.request().url());
|
||||
let payload;
|
||||
if(url.pathname==='/') return route.fulfill({contentType:'text/html',body:await readFile('web/index.html','utf8')});
|
||||
if(url.pathname==='/app.js') return route.fulfill({contentType:'application/javascript',body:await readFile('web/app.js','utf8')});
|
||||
if(url.pathname==='/style.css') return route.fulfill({contentType:'text/css',body:await readFile('web/style.css','utf8')});
|
||||
if(url.pathname==='/api/info') payload={plugins:[],import_roots:[]};
|
||||
else if(url.pathname==='/api/runs') payload=[];
|
||||
else if(url.pathname==='/api/library') payload=stored?[...snapshots,saved]:snapshots;
|
||||
else if(url.pathname==='/api/artifacts') {
|
||||
const items=(stored?[...snapshots,saved]:snapshots).filter(s=>s.release.edition===url.searchParams.get('edition')).flatMap(s=>s.entries.map(entry=>({entry,snapshot:s.id,layer:s.layer,media_type:null})));
|
||||
payload={items,total:items.length,page:0};
|
||||
} else if(url.pathname==='/api/functions/compare') {
|
||||
if(route.request().method()==='POST') {assert.deepEqual(route.request().postDataJSON(),{before:'snapshot-0',before_path:'binary/functions.json',after:'snapshot-1',after_path:'binary/functions.json'});stored=true;payload=saved;}
|
||||
else payload=report;
|
||||
} else if(url.pathname==='/api/file/saved-report') payload=report;
|
||||
else return route.fulfill({status:404,body:'unexpected request'});
|
||||
return route.fulfill({contentType:'application/json',body:JSON.stringify(payload)});
|
||||
});
|
||||
try {
|
||||
await page.goto('http://verstack.test/');
|
||||
await page.waitForFunction(()=>document.getElementById('notice').textContent.includes('ready'));
|
||||
await page.getByRole('button',{name:'Fixture',exact:true}).click();
|
||||
await page.locator('#snapshots tr').filter({hasText:'Pro / 1'}).getByRole('button',{name:'1',exact:true}).click();
|
||||
await page.locator('#media-kind').selectOption('all');
|
||||
await page.getByRole('button',{name:'Compare functions',exact:true}).click();
|
||||
await page.locator('#breadcrumbs').getByRole('button',{name:'Fixture',exact:true}).click();
|
||||
await page.locator('#snapshots tr').filter({hasText:'LE / 1'}).getByRole('button',{name:'1',exact:true}).click();
|
||||
await page.getByRole('button',{name:'Compare functions',exact:true}).click();
|
||||
await page.waitForFunction(()=>document.getElementById('function-rows').children.length===100);
|
||||
assert.match(await page.locator('#comparison-note').innerText(),/Different editions/);
|
||||
await page.locator('#function-next').click();await page.locator('#function-next').click();
|
||||
assert.equal(await page.locator('#function-rows tr').count(),1);
|
||||
await page.locator('#function-filter').fill('function_150');
|
||||
assert.equal(await page.locator('#function-rows tr').count(),1);
|
||||
await page.locator('#function-filter').fill('');
|
||||
await page.locator('#function-view').selectOption('matches');
|
||||
assert.equal(await page.locator('#function-rows tr').count(),1);
|
||||
await page.locator('#save-comparison').click();
|
||||
await page.waitForFunction(()=>document.getElementById('notice').textContent.includes('report committed'));
|
||||
assert.ok(stored);
|
||||
await page.getByRole('button',{name:'View report'}).click();
|
||||
assert.ok(await page.locator('#save-comparison').isDisabled());
|
||||
assert.equal(errors.length,0,errors.join('\n'));
|
||||
console.log('Function UI passed: source labels, pagination, filter, exact matches, save, reopen.');
|
||||
} finally {await browser.close();}
|
||||
@@ -0,0 +1,48 @@
|
||||
// Read-only browser regression against a populated archive.
|
||||
import {createRequire} from 'node:module';
|
||||
import assert from 'node:assert/strict';
|
||||
const require=createRequire(import.meta.url);
|
||||
const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT || 'playwright');
|
||||
const browser=await chromium.launch({headless:true});
|
||||
const page=await browser.newPage({viewport:{width:1440,height:1000}});
|
||||
const errors=[]; page.on('pageerror',e=>errors.push(e.message));
|
||||
try {
|
||||
await page.goto(process.env.VERSTACK_TEST_URL || 'http://127.0.0.1:8080');
|
||||
await page.waitForFunction(()=>document.getElementById('notice').textContent.includes('ready'));
|
||||
assert.equal(await page.locator('#games .game-card').count(),2);
|
||||
await page.screenshot({path:'/tmp/verstack-library.png',fullPage:true});
|
||||
await page.getByRole('button',{name:'Pokémon',exact:true}).click();
|
||||
await page.waitForFunction(()=>document.querySelectorAll('#snapshots tr').length===1);
|
||||
assert.equal(await page.locator('#snapshots tr').count(),1);
|
||||
await page.getByRole('button',{name:'0.85.0',exact:true}).click();
|
||||
await page.waitForFunction(()=>document.querySelectorAll('#gallery .media-card').length===24);
|
||||
await page.waitForFunction(()=>[...document.querySelectorAll('#gallery img')].slice(0,4).every(i=>i.complete && i.naturalWidth>0));
|
||||
assert.match(await page.locator('#asset-status').innerText(),/311 artifacts/);
|
||||
await page.screenshot({path:'/tmp/verstack-gallery.png',fullPage:true});
|
||||
await page.getByRole('button',{name:'Enlarge',exact:true}).first().click();
|
||||
await page.waitForFunction(()=>document.querySelector('#preview img')?.naturalWidth>0);
|
||||
await page.getByRole('button',{name:'Close preview'}).click();
|
||||
await page.locator('#asset-next').click();
|
||||
await page.waitForFunction(()=>document.getElementById('asset-status').textContent.includes('Page 2'));
|
||||
const current=page.url(); await page.reload();
|
||||
await page.waitForFunction(()=>document.querySelectorAll('#gallery .media-card').length===24);
|
||||
assert.equal(page.url(),current);
|
||||
await page.locator('#asset-filter').fill('no-such-media-123');
|
||||
await page.waitForFunction(()=>document.getElementById('asset-status').textContent.includes('No artifacts'));
|
||||
assert.ok(await page.locator('#asset-next').isDisabled());
|
||||
await page.locator('#asset-filter').fill('');
|
||||
await page.locator('#media-kind').selectOption('all');
|
||||
await page.waitForFunction(()=>document.getElementById('asset-status').textContent.includes('Page 1'));
|
||||
await page.setViewportSize({width:390,height:844});
|
||||
await page.screenshot({path:'/tmp/verstack-mobile.png',fullPage:true});
|
||||
assert.ok(await page.evaluate(()=>document.documentElement.scrollWidth<=window.innerWidth));
|
||||
await page.locator('#breadcrumbs').getByRole('button',{name:'All games',exact:true}).click();
|
||||
await page.getByRole('button',{name:'Game of Thrones',exact:true}).click();
|
||||
await page.waitForFunction(()=>document.querySelectorAll('#snapshots tr').length===2);
|
||||
assert.equal(await page.locator('#snapshots tr').count(),2);
|
||||
await page.locator('#snapshots tr').filter({hasText:'Pro /'}).getByRole('button',{name:'1.37.0',exact:true}).click();
|
||||
await page.locator('#media-kind').selectOption('all');
|
||||
await page.waitForFunction(()=>document.querySelectorAll('#assets tr').length>0);
|
||||
assert.equal(errors.length,0,errors.join('\n'));
|
||||
console.log('Real archive gallery passed: grouping, 311 images, decoding, pagination, enlarge, reload, empty search, mobile overflow, all files.');
|
||||
} finally {await browser.close();}
|
||||
@@ -0,0 +1,56 @@
|
||||
// Browser fixtures cover playback and unavailable previews without archive writes.
|
||||
import {createRequire} from 'node:module';
|
||||
import {readFile} from 'node:fs/promises';
|
||||
import assert from 'node:assert/strict';
|
||||
const require=createRequire(import.meta.url);
|
||||
const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT || 'playwright');
|
||||
const browser=await chromium.launch({headless:true});
|
||||
const page=await browser.newPage();
|
||||
const errors=[];page.on('pageerror',e=>errors.push(e.message));
|
||||
const wav=Buffer.alloc(44+16000); wav.write('RIFF');wav.writeUInt32LE(wav.length-8,4);wav.write('WAVEfmt ',8);wav.writeUInt32LE(16,16);wav.writeUInt16LE(1,20);wav.writeUInt16LE(1,22);wav.writeUInt32LE(8000,24);wav.writeUInt32LE(16000,28);wav.writeUInt16LE(2,32);wav.writeUInt16LE(16,34);wav.write('data',36);wav.writeUInt32LE(16000,40);
|
||||
let video;
|
||||
const release={repository:'Media fixtures',version:'1',edition:'',generation:''};
|
||||
const snapshot={id:'fixture',release,layer:'extracted',run:'run',imported_at:1,logical_bytes:1,new_packed_bytes:1,warnings:[],entry_count:3};
|
||||
const items=[['sound.wav','audio/wav'],['clip.webm','video/webm'],['broken.png','image/png']].map(([path,media_type])=>({snapshot:'fixture',layer:'extracted',run:'run',entry:{path,kind:'file',size:16000},media_type}));
|
||||
await page.route('http://verstack.test/**',async route=>{
|
||||
const u=new URL(route.request().url());
|
||||
if(['/', '/app.js','/style.css'].includes(u.pathname)) return route.fulfill({contentType:u.pathname==='/'?'text/html':u.pathname.endsWith('.js')?'text/javascript':'text/css',body:await readFile('web/'+(u.pathname==='/'?'index.html':u.pathname.slice(1)))});
|
||||
let payload;
|
||||
if(u.pathname==='/api/info') payload={plugins:[],import_roots:[]};
|
||||
else if(u.pathname==='/api/library') payload=[snapshot];
|
||||
else if(u.pathname==='/api/runs') payload=[];
|
||||
else if(u.pathname==='/api/artifacts') {const kind=u.searchParams.get('kind');const filtered=items.filter(i=>kind==='media'||kind==='all'||i.media_type.startsWith(kind+'/'));payload={items:filtered,total:filtered.length,page:0};}
|
||||
else if(u.pathname==='/api/file/fixture') {
|
||||
const path=u.searchParams.get('path');
|
||||
return route.fulfill({contentType:path==='sound.wav'?'audio/wav':path==='clip.webm'?'video/webm':'image/png',body:path==='sound.wav'?wav:path==='clip.webm'?video:Buffer.from('invalid')});
|
||||
} else return route.fulfill({status:404,body:'unexpected'});
|
||||
return route.fulfill({contentType:'application/json',body:JSON.stringify(payload)});
|
||||
});
|
||||
try {
|
||||
await page.goto('http://verstack.test/');
|
||||
// Generate a browser-native VP8 fixture using a canvas stream.
|
||||
video=Buffer.from(await page.evaluate(async()=>{
|
||||
const canvas=document.createElement('canvas');canvas.width=64;canvas.height=64;
|
||||
const context=canvas.getContext('2d'); const stream=canvas.captureStream(10);
|
||||
const recorder=new MediaRecorder(stream,{mimeType:'video/webm;codecs=vp8'});const chunks=[];
|
||||
recorder.ondataavailable=e=>chunks.push(e.data);
|
||||
const done=new Promise(resolve=>recorder.onstop=resolve);recorder.start();
|
||||
for(let i=0;i<6;i++){context.fillStyle=i%2?'red':'blue';context.fillRect(0,0,64,64);await new Promise(r=>setTimeout(r,100));}
|
||||
recorder.stop();await done;stream.getTracks().forEach(t=>t.stop());return [...new Uint8Array(await new Blob(chunks).arrayBuffer())];
|
||||
}));
|
||||
await page.getByRole('button',{name:'Media fixtures',exact:true}).click();
|
||||
await page.locator('#snapshots').getByRole('button',{name:'1',exact:true}).click();
|
||||
await page.waitForFunction(()=>document.querySelectorAll('.media-card').length===3);
|
||||
assert.equal(await page.locator('audio').getAttribute('preload'),'none');
|
||||
assert.equal(await page.locator('video').getAttribute('preload'),'none');
|
||||
await page.locator('audio').evaluate(async m=>{await m.play();});
|
||||
await page.waitForFunction(()=>document.querySelector('audio').currentTime>0);
|
||||
await page.locator('video').evaluate(async m=>{await m.play();});
|
||||
await page.waitForFunction(()=>document.querySelector('video').videoWidth===64);
|
||||
await page.waitForFunction(()=>document.querySelector('.preview-error'));
|
||||
await page.locator('#media-kind').selectOption('audio');
|
||||
await page.waitForFunction(()=>document.querySelectorAll('.media-card').length===1);
|
||||
assert.equal(await page.locator('video').count(),0);
|
||||
assert.equal(errors.length,0,errors.join('\n'));
|
||||
console.log('Media UI passed: actual WAV and WebM playback, deferred preload, failed image fallback, type filtering.');
|
||||
} finally {await browser.close();}
|
||||
@@ -0,0 +1,46 @@
|
||||
// Optional browser smoke test. Install Playwright separately; the application
|
||||
// itself has no Node dependency. Start a server with disposable data first.
|
||||
import { createRequire } from "node:module";
|
||||
import assert from "node:assert/strict";
|
||||
const require = createRequire(import.meta.url);
|
||||
const { chromium } = require(process.env.VERSTACK_PLAYWRIGHT || "playwright");
|
||||
const browser = await chromium.launch({ headless: true });
|
||||
const page = await browser.newPage({ viewport: { width: 1440, height: 1000 } });
|
||||
const errors = [];
|
||||
const repository = `Browser smoke test ${Date.now()}`;
|
||||
page.on("pageerror", (error) => errors.push(error.message));
|
||||
try {
|
||||
await page.goto(process.env.VERSTACK_TEST_URL || "http://127.0.0.1:18765");
|
||||
await page.waitForFunction(() => document.getElementById("notice").textContent.includes("ready"));
|
||||
await page.getByRole('link', {name:'Import',exact:true}).click();
|
||||
await page.locator('[name="repository"]').fill(repository);
|
||||
await page.locator('[name="version"]').fill("2");
|
||||
await page.locator('[name="released_at"]').fill("2025-05-01");
|
||||
await page.locator('[name="file"]').setInputFiles({ name: "config.txt", mimeType: "text/plain", buffer: Buffer.from("mode=enabled\nvolume=80\n") });
|
||||
await page.getByRole("button", { name: "Import & verify" }).click();
|
||||
await page.waitForFunction(() => document.getElementById("notice").textContent.includes("committed"));
|
||||
await page.locator("#media-kind").selectOption("all");
|
||||
await page.getByRole("button", { name: "Inspect", exact: true }).click();
|
||||
await page.waitForFunction(() => document.getElementById("preview").textContent.includes("mode=enabled"));
|
||||
assert.match(await page.locator("#preview").innerText(), /mode=enabled/);
|
||||
assert.match(await page.locator("#snapshots").innerText(), /2025-05-01/);
|
||||
const [download] = await Promise.all([page.waitForEvent("download"), page.getByRole("link", { name: "Download", exact: true }).click()]);
|
||||
assert.equal(download.suggestedFilename(), "config.txt");
|
||||
await page.locator(".tools summary").click();
|
||||
await page.getByRole("button", { name: "Verify retained bytes" }).click();
|
||||
await page.waitForFunction(() => document.getElementById("notice").textContent.includes("hashes verified"));
|
||||
await page.getByRole("link", {name:"Import",exact:true}).click();
|
||||
await page.locator('[name="version"]').fill("1");
|
||||
await page.locator('[name="file"]').setInputFiles({ name: "config.txt", mimeType: "text/plain", buffer: Buffer.from("mode=disabled\nvolume=70\n") });
|
||||
await page.getByRole("button", { name: "Import & verify" }).click();
|
||||
await page.waitForFunction(() => document.getElementById("notice").textContent.includes("committed"));
|
||||
await page.locator("#breadcrumbs").getByRole("button", {name:repository,exact:true}).click();
|
||||
await page.getByRole("button", { name: "Compare", exact: true }).first().click();
|
||||
await page.getByRole("button", { name: "Compare", exact: true }).first().click();
|
||||
await page.waitForFunction(() => document.getElementById("diff").textContent.includes("changed"));
|
||||
assert.equal(errors.length, 0, errors.join("\n"));
|
||||
await page.screenshot({ path: process.env.VERSTACK_SCREENSHOT || "/tmp/verstack-ui.png", fullPage: true });
|
||||
console.log("Browser smoke passed: upload, release date, browse, preview, download, verify, compare; no JavaScript errors.");
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
+468
@@ -0,0 +1,468 @@
|
||||
"use strict";
|
||||
const $ = (id) => document.getElementById(id);
|
||||
let snapshots = [],
|
||||
selected = null,
|
||||
compareFrom = null,
|
||||
functionFrom = null,
|
||||
info = { plugins: [] };
|
||||
let activeGame = null, activeVersion = null, assetPage = 0, assetRequest = 0, allRuns = [];
|
||||
const versionKey = r => JSON.stringify([r.repository, r.version, r.edition || "", r.generation || ""]);
|
||||
let functionReport = null, functionParams = null, functionPage = 0;
|
||||
const size = (n) => {
|
||||
let u = ["B", "KiB", "MiB", "GiB", "TiB"],
|
||||
i = 0;
|
||||
while (n >= 1024 && i < 4) {
|
||||
n /= 1024;
|
||||
i++;
|
||||
}
|
||||
return `${n.toFixed(i ? 1 : 0)} ${u[i]}`;
|
||||
};
|
||||
const notice = (text, error = false) => {
|
||||
$("notice").textContent = text;
|
||||
$("notice").className = error ? "error" : "";
|
||||
};
|
||||
async function api(path, options = {}) {
|
||||
const response = await fetch("/api/" + path, {
|
||||
...options,
|
||||
headers: { "X-Verstack-Client": "1", ...options.headers },
|
||||
});
|
||||
if (!response.ok) {
|
||||
const text = await response.text();
|
||||
throw Error(text);
|
||||
}
|
||||
return response.json();
|
||||
}
|
||||
const post = (path, data) =>
|
||||
api(path, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(data),
|
||||
});
|
||||
function cell(row, text, className) {
|
||||
const td = document.createElement("td");
|
||||
td.textContent = text ?? "";
|
||||
if (className) td.className = className;
|
||||
row.append(td);
|
||||
return td;
|
||||
}
|
||||
function button(parent, label, action) {
|
||||
const b = document.createElement("button");
|
||||
b.textContent = label;
|
||||
b.setAttribute("aria-label", label);
|
||||
b.onclick = () => Promise.resolve().then(action).catch((e) => notice(e.message, true));
|
||||
parent.append(b);
|
||||
return b;
|
||||
}
|
||||
function render() {
|
||||
const filter = $("filter").value.toLowerCase();
|
||||
$("games").replaceChildren();
|
||||
const games = [...new Set(snapshots.map(s => s.release.repository))].sort();
|
||||
for (const game of games.filter(g => g.toLowerCase().includes(filter))) {
|
||||
const group = snapshots.filter(s => s.release.repository === game);
|
||||
const card = document.createElement("article"); card.className = "game-card";
|
||||
const icon = document.createElement("div"); icon.className = "game-icon"; icon.textContent = game.slice(0,2).toUpperCase();
|
||||
card.append(icon);
|
||||
button(card, game, () => { location.hash = "game=" + encodeURIComponent(game); });
|
||||
const meta = document.createElement("p"); meta.textContent = `${new Set(group.map(s => versionKey(s.release))).size} versions · ${group.length} outputs`;
|
||||
card.append(meta); $("games").append(card);
|
||||
}
|
||||
$("empty").hidden = $("games").children.length > 0;
|
||||
$("game-home").hidden = !!activeGame;
|
||||
$("version-list").hidden = !activeGame || !!activeVersion;
|
||||
$("details").hidden = !activeVersion;
|
||||
$("library-title").textContent = activeVersion ? activeVersion.version : activeGame || "Game library";
|
||||
$("library-subtitle").textContent = activeVersion ? [activeGame, activeVersion.edition, activeVersion.generation && `Generation ${activeVersion.generation}`].filter(Boolean).join(" / ") : activeGame ? "Select a version to explore its media, files, and processing history." : "Select a game to browse imported versions and their analysis artifacts.";
|
||||
$("breadcrumbs").replaceChildren();
|
||||
button($("breadcrumbs"), "All games", () => { location.hash = "library"; });
|
||||
if (activeGame) button($("breadcrumbs"), activeGame, () => { location.hash = "game=" + encodeURIComponent(activeGame); });
|
||||
if (activeVersion) { const span = document.createElement("span"); span.textContent = activeVersion.version; $("breadcrumbs").append(span); }
|
||||
$("snapshots").replaceChildren();
|
||||
const versions = new Map();
|
||||
for (const s of snapshots.filter(s => s.release.repository === activeGame)) {
|
||||
const key = versionKey(s.release);
|
||||
if (!versions.has(key)) versions.set(key, []);
|
||||
versions.get(key).push(s);
|
||||
}
|
||||
for (const group of versions.values()) {
|
||||
const s = group.find(s => s.layer === "original") || group[0];
|
||||
const row = document.createElement("tr");
|
||||
button(cell(row, ""), s.release.version, () => show(s));
|
||||
cell(row, `${s.release.edition || "Standard"} / ${s.release.generation || "Unknown"}`);
|
||||
cell(row, s.release.released_at || "Unknown");
|
||||
cell(row, group.length + " outputs");
|
||||
cell(row, group.reduce((n,s) => n + (s.entry_count || 0),0) + " artifacts");
|
||||
button(
|
||||
cell(row, ""),
|
||||
compareFrom === s.id ? "Selected baseline" : "Compare",
|
||||
async () => {
|
||||
if (!compareFrom) {
|
||||
compareFrom = s.id;
|
||||
notice("Select the second snapshot to compare.");
|
||||
render();
|
||||
return;
|
||||
}
|
||||
const before = compareFrom;
|
||||
compareFrom = null;
|
||||
const changes = await api(
|
||||
"compare?" + new URLSearchParams({ before, after: s.id }),
|
||||
);
|
||||
$("comparison").hidden = false;
|
||||
$("comparison-title").textContent = "Snapshot comparison";
|
||||
$("function-results").hidden = true;
|
||||
$("diff").hidden = false;
|
||||
$("diff").textContent = changes.length
|
||||
? changes.map((c) => `${c.kind.padEnd(15)} ${c.path}`).join("\n")
|
||||
: "No file or metadata differences.";
|
||||
$("comparison-note").textContent =
|
||||
"File identity comparison. Extraction revisions and layers can affect results; unmatched files are not proof of publisher additions.";
|
||||
notice(`Comparison complete: ${changes.length} file or metadata differences.`);
|
||||
render();
|
||||
},
|
||||
);
|
||||
$("snapshots").append(row);
|
||||
}
|
||||
$("metrics").replaceChildren();
|
||||
const scoped = snapshots.filter(s => activeVersion ? versionKey(s.release) === versionKey(activeVersion) : !activeGame || s.release.repository === activeGame);
|
||||
const metrics = activeVersion ? [["Job outputs",scoped.length],["Artifacts",scoped.reduce((n,s)=>n+(s.entry_count||0),0)],["Retained",size(scoped.reduce((n,s)=>n+s.logical_bytes,0))]] : [["Games",activeGame ? 1 : games.length],["Versions",new Set(scoped.map(s => versionKey(s.release))).size],["Retained",size(scoped.reduce((n,s) => n+s.logical_bytes,0))]];
|
||||
for (const [label,value] of metrics) {
|
||||
const div = document.createElement("div"); div.className = "metric";
|
||||
const strong = document.createElement("strong"); strong.textContent = value;
|
||||
const span = document.createElement("span"); span.textContent = label; div.append(strong,span); $("metrics").append(div);
|
||||
}
|
||||
}
|
||||
function navigate() {
|
||||
assetRequest++;
|
||||
const params = new URLSearchParams(location.hash.slice(1));
|
||||
activeGame = params.get("game"); activeVersion = null;
|
||||
const id = params.get("version");
|
||||
if (id) {
|
||||
const s = snapshots.find(s => s.id === id && s.release.repository === activeGame);
|
||||
if (s) { activeVersion = s.release; selected = s; }
|
||||
}
|
||||
render();
|
||||
$("import").hidden = location.hash !== "#import";
|
||||
$("library").hidden = ["#import", "#jobs"].includes(location.hash);
|
||||
$("preview").replaceChildren(); $("preview-panel").hidden = true;
|
||||
if (activeVersion) {
|
||||
assetPage = 0;
|
||||
const group = snapshots.filter(s => versionKey(s.release) === versionKey(activeVersion));
|
||||
$("source-filter").replaceChildren(new Option("All job outputs", ""));
|
||||
$("process-source").replaceChildren();
|
||||
for (const s of group) {
|
||||
const run = allRuns.find(r => r.id === s.run);
|
||||
const label = `${s.layer} · ${run?.operation || "Import / analysis"} · ${s.id.slice(0,8)}`;
|
||||
$("source-filter").append(new Option(label,s.id));
|
||||
$("process-source").append(new Option(label,s.id));
|
||||
}
|
||||
$("process-source").value = selected.id;
|
||||
$("warnings").textContent = [...new Set(group.flatMap(s => s.warnings))].join(" ");
|
||||
renderAssets();
|
||||
}
|
||||
renderRuns();
|
||||
}
|
||||
window.addEventListener("hashchange", navigate);
|
||||
async function refresh() {
|
||||
snapshots = await api("library");
|
||||
snapshots.sort((a, b) => b.imported_at - a.imported_at);
|
||||
render();
|
||||
await refreshRuns();
|
||||
navigate();
|
||||
}
|
||||
async function refreshRuns() {
|
||||
allRuns = await api("runs");
|
||||
renderRuns();
|
||||
}
|
||||
function renderRuns() {
|
||||
$("runs").replaceChildren();
|
||||
const group = snapshots.filter(s => activeVersion && versionKey(s.release) === versionKey(activeVersion));
|
||||
const ids = new Set(group.map(s => s.id)), runs = new Set(group.map(s => s.run));
|
||||
for (const r of allRuns.filter(r => !activeVersion || runs.has(r.id) || r.inputs.some(id => ids.has(id))).sort((a,b) => b.started_at-a.started_at)) {
|
||||
const row = document.createElement("tr");
|
||||
cell(row,new Date(r.started_at*1000).toLocaleString()); cell(row,`${r.operation} (${r.tool_version})`);
|
||||
cell(row,`${r.state} / ${r.stage}`); cell(row,size(r.bytes_processed)); cell(row,r.error || r.output || "In progress"); $("runs").append(row);
|
||||
}
|
||||
$("jobs").hidden = !activeVersion && location.hash !== "#jobs";
|
||||
}
|
||||
function show(s) {
|
||||
const hash = "game=" + encodeURIComponent(s.release.repository) + "&version=" + encodeURIComponent(s.id);
|
||||
if (location.hash.slice(1) === hash) navigate(); else location.hash = hash;
|
||||
}
|
||||
async function renderAssets() {
|
||||
if (!activeVersion) return;
|
||||
const request = ++assetRequest;
|
||||
$("assets").replaceChildren(); $("gallery").replaceChildren();
|
||||
$("asset-status").textContent = "Loading artifacts…";
|
||||
$("asset-previous").disabled = $("asset-next").disabled = true;
|
||||
try {
|
||||
const result = await api("artifacts?" + new URLSearchParams({...activeVersion, source:$("source-filter").value, search:$("asset-filter").value, kind:$("media-kind").value, page:assetPage}));
|
||||
if (request !== assetRequest) return;
|
||||
assetPage = result.page;
|
||||
$("asset-status").textContent = result.total ? `${result.total} artifacts · Page ${assetPage+1} of ${Math.ceil(result.total/24)}` : "No artifacts match this view. Try All files or another filter.";
|
||||
$("asset-previous").disabled = !assetPage;
|
||||
$("asset-next").disabled = (assetPage+1)*24 >= result.total;
|
||||
$("file-table").hidden = !result.items.some(item => !item.media_type);
|
||||
for (const item of result.items) {
|
||||
const e = item.entry, s = snapshots.find(s => s.id === item.snapshot);
|
||||
if (item.media_type) { renderMedia(item); continue; }
|
||||
const row = document.createElement("tr");
|
||||
cell(row, e.path);
|
||||
cell(row, `${e.kind} · ${s.layer} · ${s.id.slice(0,8)}`);
|
||||
cell(row, size(e.size));
|
||||
cell(
|
||||
row,
|
||||
e.artifact ? e.artifact.slice(0, 23) + "…" : e.link_target || "",
|
||||
"hash",
|
||||
);
|
||||
const actions = cell(row, "");
|
||||
if (e.kind === "file") {
|
||||
const a = document.createElement("a");
|
||||
a.textContent = "Download";
|
||||
a.href =
|
||||
"/api/file/" + s.id + "?" + new URLSearchParams({ path: e.path });
|
||||
a.download = e.path.split("/").pop();
|
||||
actions.append(a, document.createTextNode(" "));
|
||||
button(actions, "Inspect", async () => {
|
||||
const extension = e.path.split(".").pop().toLowerCase();
|
||||
const tag = ["png", "jpg", "jpeg", "gif", "webp"].includes(extension)
|
||||
? "img"
|
||||
: ["mp4", "webm"].includes(extension)
|
||||
? "video"
|
||||
: ["wav", "ogg", "mp3"].includes(extension)
|
||||
? "audio"
|
||||
: null;
|
||||
if (tag) {
|
||||
const media = document.createElement(tag);
|
||||
media.src = a.href + "&inline=true";
|
||||
if (tag === "img") media.alt = e.path;
|
||||
else media.controls = true;
|
||||
$("preview").replaceChildren(media);
|
||||
$("preview-title").textContent = e.path;
|
||||
$("preview-panel").hidden = false;
|
||||
return;
|
||||
}
|
||||
if (!e.size) {
|
||||
$("preview").textContent = "Empty file.";
|
||||
} else {
|
||||
const response = await fetch(a.href, {
|
||||
headers: { Range: "bytes=0-65535" },
|
||||
});
|
||||
if (!response.ok) throw Error("Preview failed");
|
||||
const bytes = new Uint8Array(await response.arrayBuffer());
|
||||
const text = new TextDecoder().decode(bytes);
|
||||
const printable = bytes.every(
|
||||
(b) => b === 9 || b === 10 || b === 13 || (b >= 32 && b < 127),
|
||||
);
|
||||
$("preview").textContent = printable
|
||||
? text
|
||||
: Array.from(
|
||||
{ length: Math.ceil(bytes.length / 16) },
|
||||
(_, i) =>
|
||||
`${(i * 16).toString(16).padStart(8, "0")} ${Array.from(bytes.slice(i * 16, i * 16 + 16), (b) => b.toString(16).padStart(2, "0")).join(" ")}`,
|
||||
).join("\n");
|
||||
}
|
||||
$("preview-title").textContent = e.path + " — first 64 KiB";
|
||||
$("preview-panel").hidden = false;
|
||||
});
|
||||
if (e.path === "function-comparison.json")
|
||||
button(actions, "View report", async () => {
|
||||
const response = await fetch(a.href);
|
||||
if (!response.ok) throw Error("Report download failed");
|
||||
showFunctionReport(await response.json(), null);
|
||||
});
|
||||
if (e.path.endsWith("/functions.json"))
|
||||
button(actions, "Compare functions", async () => {
|
||||
if (!functionFrom) {
|
||||
functionFrom = { before: s.id, before_path: e.path };
|
||||
notice(
|
||||
"Function baseline selected. Open another analysis snapshot and select its function inventory.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
const params = { ...functionFrom, after: s.id, after_path: e.path };
|
||||
functionFrom = null;
|
||||
const report = await api(
|
||||
"functions/compare?" + new URLSearchParams(params),
|
||||
);
|
||||
showFunctionReport(report, params);
|
||||
notice(`Function comparison complete: ${report.matches.length} exact matches.`);
|
||||
});
|
||||
}
|
||||
$("assets").append(row);
|
||||
}
|
||||
} catch(e) { if (request === assetRequest) { $("asset-status").textContent = "Could not load artifacts. Use Refresh to retry."; notice(e.message,true); } }
|
||||
}
|
||||
function renderMedia(item) {
|
||||
const e = item.entry, kind = item.media_type.split("/")[0];
|
||||
const card = document.createElement("article"); card.className = "media-card";
|
||||
const frame = document.createElement("div"); frame.className = "media-frame " + kind;
|
||||
const media = document.createElement(kind === "image" ? "img" : kind);
|
||||
const url = "/api/file/" + item.snapshot + "?" + new URLSearchParams({path:e.path});
|
||||
if (kind === "image") { media.alt = e.path; media.loading = "lazy"; media.decoding = "async"; }
|
||||
else { media.controls = true; media.preload = "none"; }
|
||||
media.src = url + "&inline=true";
|
||||
media.onerror = () => { const error = document.createElement("p"); error.className = "preview-error"; error.textContent = "Preview unavailable in this browser. Download the original to open it."; media.replaceWith(error); };
|
||||
frame.append(media); card.append(frame);
|
||||
const title = document.createElement("h3"); title.textContent = e.path.split("/").pop(); title.title = e.path;
|
||||
const path = document.createElement("p"); path.className = "media-path"; path.textContent = e.path;
|
||||
const meta = document.createElement("p"); meta.className = "muted"; meta.textContent = `${size(e.size)} · ${item.layer} · ${item.snapshot.slice(0,8)}`;
|
||||
card.append(title,path,meta);
|
||||
if (kind === "image") button(card,"Enlarge", () => {
|
||||
const full = document.createElement("img"); full.src = url + "&inline=true"; full.alt = e.path;
|
||||
$("preview").replaceChildren(full); $("preview-title").textContent = e.path; $("preview-panel").hidden = false;
|
||||
$("preview-panel").scrollIntoView({behavior:"smooth"});
|
||||
});
|
||||
const link = document.createElement("a"); link.href = url; link.download = e.path.split("/").pop(); link.textContent = "Download"; card.append(link);
|
||||
$("gallery").append(card);
|
||||
}
|
||||
|
||||
function showFunctionReport(report, params) {
|
||||
functionReport = report;
|
||||
functionParams = params;
|
||||
functionPage = 0;
|
||||
$("comparison").hidden = false;
|
||||
$("comparison-title").textContent = "Function comparison";
|
||||
$("comparison-note").textContent = report.caveat;
|
||||
$("function-results").hidden = false;
|
||||
$("diff").hidden = true;
|
||||
$("save-comparison").disabled = !params;
|
||||
const labels = (report.sources || []).map(s => `${s.release.repository} ${s.release.version} ${s.release.edition} (snapshot ${s.snapshot})`);
|
||||
$("function-summary").textContent = `${labels.join(" → ")}. ${report.matches.length} exact matches; ${report.unmatched_after.length} unmatched target functions; ${report.unmatched_before.length} unmatched baseline functions.`;
|
||||
renderFunctions();
|
||||
}
|
||||
function renderFunctions() {
|
||||
if (!functionReport) return;
|
||||
const view = $("function-view").value;
|
||||
const query = $("function-filter").value.toLowerCase();
|
||||
const label = f => f ? `${f.name} @ ${f.address}` : "—";
|
||||
const rows = functionReport[view].map(f => view === "matches"
|
||||
? [label(f.before), label(f.after), "Unique exact body ≥32 bytes"]
|
||||
: view === "unmatched_after"
|
||||
? ["—", label(f), functionReport.unmatched_after_reasons[f.address]]
|
||||
: [label(f), "—", "No match by this method"])
|
||||
.filter(row => row.join(" ").toLowerCase().includes(query));
|
||||
functionPage = Math.min(functionPage, Math.max(0, Math.ceil(rows.length / 100) - 1));
|
||||
$("function-rows").replaceChildren();
|
||||
for (const values of rows.slice(functionPage * 100, (functionPage + 1) * 100)) {
|
||||
const tr = document.createElement("tr");
|
||||
values.forEach(v => cell(tr, v));
|
||||
$("function-rows").append(tr);
|
||||
}
|
||||
$("function-page").textContent = `${rows.length} results · Page ${functionPage + 1} of ${Math.max(1, Math.ceil(rows.length / 100))}`;
|
||||
$("function-previous").disabled = !functionPage;
|
||||
$("function-next").disabled = (functionPage + 1) * 100 >= rows.length;
|
||||
}
|
||||
$("function-view").onchange = $("function-filter").oninput = () => { functionPage = 0; renderFunctions(); };
|
||||
$("function-previous").onclick = () => { functionPage--; renderFunctions(); };
|
||||
$("function-next").onclick = () => { functionPage++; renderFunctions(); };
|
||||
$("save-comparison").onclick = async () => {
|
||||
if (!functionParams) return;
|
||||
$("save-comparison").disabled = true;
|
||||
try {
|
||||
const saved = await post("functions/compare", functionParams);
|
||||
functionParams = null;
|
||||
await refresh();
|
||||
show(saved);
|
||||
notice("Comparison report committed and verified.");
|
||||
} catch (e) { notice(e.message, true); $("save-comparison").disabled = false; }
|
||||
};
|
||||
$("import-form").onsubmit = async (event) => {
|
||||
event.preventDefault();
|
||||
const form = event.target;
|
||||
const data = new FormData(form);
|
||||
const submit = form.querySelector("button[type=submit]");
|
||||
submit.disabled = true;
|
||||
notice(
|
||||
"Importing. Progress appears in Processing runs; the snapshot publishes after verification.",
|
||||
);
|
||||
try {
|
||||
const file = data.get("file");
|
||||
let s;
|
||||
if (file && file.name) {
|
||||
const q = new URLSearchParams({
|
||||
repository: data.get("repository"),
|
||||
version: data.get("version"),
|
||||
edition: data.get("edition"),
|
||||
generation: data.get("generation"),
|
||||
filename: file.name,
|
||||
released_at: data.get("released_at") || "",
|
||||
});
|
||||
s = await api("upload?" + q, { method: "POST", body: file });
|
||||
} else {
|
||||
s = await post("import", {
|
||||
path: data.get("path"),
|
||||
release: {
|
||||
repository: data.get("repository"),
|
||||
version: data.get("version"),
|
||||
edition: data.get("edition"),
|
||||
generation: data.get("generation"),
|
||||
released_at: data.get("released_at") || null,
|
||||
},
|
||||
});
|
||||
}
|
||||
await refresh();
|
||||
show(s);
|
||||
notice("Snapshot committed and retained file bytes verified.");
|
||||
} catch (e) {
|
||||
notice(e.message, true);
|
||||
} finally {
|
||||
submit.disabled = false;
|
||||
}
|
||||
};
|
||||
$("refresh-artifacts").onclick = () => refresh().catch(e => notice(e.message,true));
|
||||
$("refresh").onclick = () => refresh().catch((e) => notice(e.message, true));
|
||||
$("filter").oninput = render;
|
||||
let filterTimer;
|
||||
$("asset-filter").oninput = () => { clearTimeout(filterTimer); filterTimer = setTimeout(() => { assetPage=0; renderAssets(); },200); };
|
||||
$("source-filter").onchange = $("media-kind").onchange = () => { assetPage=0; renderAssets(); };
|
||||
$("asset-previous").onclick = () => { assetPage--; renderAssets(); };
|
||||
$("asset-next").onclick = () => { assetPage++; renderAssets(); };
|
||||
$("close-preview").onclick = () => { $("preview-panel").hidden=true; $("preview").replaceChildren(); };
|
||||
$("process-source").onchange = () => { selected = snapshots.find(s => s.id === $("process-source").value); };
|
||||
$("process").onclick = async () => {
|
||||
if (!selected || !$("plugin").value) return;
|
||||
const b = $("process");
|
||||
b.disabled = true;
|
||||
notice("Plugin running. The input snapshot remains available.");
|
||||
try {
|
||||
const s = await post("process", {
|
||||
snapshot: selected.id,
|
||||
plugin: $("plugin").value,
|
||||
});
|
||||
await refresh();
|
||||
show(s);
|
||||
notice("Processing output committed and verified.");
|
||||
} catch (e) {
|
||||
notice(e.message, true);
|
||||
} finally {
|
||||
b.disabled = false;
|
||||
}
|
||||
};
|
||||
$("verify").onclick = async () => {
|
||||
if (!selected) return;
|
||||
try {
|
||||
notice("Reading and verifying retained bytes…");
|
||||
await post("verify/" + selected.id, {});
|
||||
notice("All retained file hashes verified.");
|
||||
} catch (e) {
|
||||
notice(e.message, true);
|
||||
}
|
||||
};
|
||||
(async () => {
|
||||
try {
|
||||
info = await api("info");
|
||||
$("roots").textContent =
|
||||
"Allowed server roots: " + info.import_roots.join(", ");
|
||||
for (const name of info.plugins) {
|
||||
const o = document.createElement("option");
|
||||
o.value = name;
|
||||
o.textContent = name;
|
||||
$("plugin").append(o);
|
||||
}
|
||||
$("process").disabled = !info.plugins.length;
|
||||
await refresh();
|
||||
notice("Connected. Shared archive ready.");
|
||||
} catch (e) {
|
||||
notice(e.message, true);
|
||||
}
|
||||
})();
|
||||
setInterval(() => refreshRuns().catch(() => {}), 2000);
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Verstack · Game archive</title>
|
||||
<link rel="stylesheet" href="/style.css" />
|
||||
<script src="/app.js" defer></script>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div class="brand">
|
||||
VERSTACK <small>GAME ANALYSIS LAB</small>
|
||||
</div>
|
||||
<nav>
|
||||
<a href="#library">Library</a><a href="#import">Import</a
|
||||
><a href="#jobs">Processing runs</a>
|
||||
</nav>
|
||||
</header>
|
||||
<main>
|
||||
<div id="notice" role="status" aria-live="polite">
|
||||
Connecting to local archive…
|
||||
</div>
|
||||
<section id="library">
|
||||
<div id="breadcrumbs" class="breadcrumbs" aria-label="Breadcrumb"></div><h1 id="library-title">Game library</h1>
|
||||
<p id="library-subtitle" class="muted"></p>
|
||||
<div class="metrics" id="metrics"></div>
|
||||
<div id="game-home">
|
||||
<div class="toolbar"><label>Search games <input id="filter" placeholder="Game name…" /></label><button id="refresh">Refresh</button></div>
|
||||
<div id="games" class="game-grid"></div>
|
||||
<p id="empty" class="empty">No games found. Import a release to start your library.</p>
|
||||
</div>
|
||||
<div id="version-list" class="panel" hidden>
|
||||
<div class="panel-title">Imported versions</div>
|
||||
<table><thead><tr><th>Version</th><th>Edition / generation</th><th>Release date</th><th>Job outputs</th><th>Artifacts</th><th>Compare</th></tr></thead><tbody id="snapshots"></tbody></table>
|
||||
</div>
|
||||
</section>
|
||||
<section id="details" hidden>
|
||||
<h2 id="detail-title">Version artifacts</h2>
|
||||
<details class="provenance"><summary>Extraction notes and coverage</summary><p id="warnings" class="muted"></p></details>
|
||||
<div class="panel">
|
||||
<div class="panel-title">Artifact browser <button id="refresh-artifacts">Refresh</button></div>
|
||||
<div class="toolbar artifact-filters">
|
||||
<label>View <select id="media-kind"><option value="media">Media gallery</option><option value="image">Images</option><option value="video">Videos</option><option value="audio">Sounds</option><option value="all">All files</option><option value="file">Other files</option></select></label>
|
||||
<label>Output <select id="source-filter"></select></label>
|
||||
<label>Search <input id="asset-filter" placeholder="Filter paths…" /></label>
|
||||
</div>
|
||||
<div id="gallery" class="gallery"></div>
|
||||
<div id="file-table" class="table-wrap" hidden><table><thead><tr><th>Path</th><th>Source</th><th>Size</th><th>Artifact identity</th><th>Actions</th></tr></thead><tbody id="assets"></tbody></table></div>
|
||||
<div class="toolbar pagination"><span id="asset-status" role="status" aria-live="polite"></span><button id="asset-previous">Previous</button><button id="asset-next">Next</button></div>
|
||||
</div>
|
||||
<div id="preview-panel" class="panel" hidden>
|
||||
<div class="panel-title"><span id="preview-title">Preview</span><button id="close-preview">Close preview</button></div><pre id="preview"></pre>
|
||||
</div>
|
||||
<details class="panel tools"><summary>Analysis tools · process, verify, and compare</summary>
|
||||
<div class="toolbar"><label>Input <select id="process-source"></select></label><label>Plugin <select id="plugin"></select></label><button id="process" class="primary">Run plugin</button><button id="verify">Verify retained bytes</button></div>
|
||||
</details>
|
||||
</section>
|
||||
<section id="comparison" hidden>
|
||||
<h2 id="comparison-title">Snapshot comparison</h2>
|
||||
<p id="comparison-note" class="muted"></p>
|
||||
<div id="function-results" hidden>
|
||||
<div class="toolbar">
|
||||
<label>View <select id="function-view"><option value="unmatched_after">Unmatched target</option><option value="matches">Exact matches</option><option value="unmatched_before">Unmatched baseline</option></select></label>
|
||||
<label>Filter <input id="function-filter" placeholder="Name or address" /></label>
|
||||
<button id="save-comparison">Save report to archive</button>
|
||||
</div>
|
||||
<p id="function-summary"></p>
|
||||
<div class="panel"><table><thead><tr><th>Baseline</th><th>Target</th><th>Evidence</th></tr></thead><tbody id="function-rows"></tbody></table></div>
|
||||
<div class="toolbar"><button id="function-previous">Previous</button><span id="function-page"></span><button id="function-next">Next</button></div>
|
||||
</div>
|
||||
<pre id="diff"></pre>
|
||||
</section>
|
||||
<section id="import" hidden>
|
||||
<h2>Import a release</h2>
|
||||
<form id="import-form" class="panel">
|
||||
<div class="panel-title">Original inputs</div>
|
||||
<div class="form-grid">
|
||||
<label
|
||||
>Game / repository<input
|
||||
name="repository"
|
||||
required
|
||||
placeholder="Jurassic Park" /></label
|
||||
><label
|
||||
>Release version<input
|
||||
name="version"
|
||||
required
|
||||
placeholder="1.15.0" /></label
|
||||
><label
|
||||
>Edition<input name="edition" placeholder="Premium / LE" /></label
|
||||
><label
|
||||
>SPIKE generation<input
|
||||
name="generation"
|
||||
placeholder="Unknown" /></label
|
||||
><label>Release date<input name="released_at" type="date" /></label
|
||||
><label
|
||||
>Server path<input
|
||||
name="path"
|
||||
placeholder="/srv/games/package.spk" /></label
|
||||
><label>Or upload one file<input name="file" type="file" /></label>
|
||||
</div>
|
||||
<p id="roots" class="muted"></p>
|
||||
<p class="muted">
|
||||
Packages remain intact until a configured plugin extracts them.
|
||||
Original filesystem inputs are retained.
|
||||
</p>
|
||||
<button type="submit" class="primary">Import & verify</button>
|
||||
</form>
|
||||
</section>
|
||||
<section id="jobs" hidden>
|
||||
<h2>Processing runs</h2>
|
||||
<div class="panel">
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Started</th>
|
||||
<th>Operation</th>
|
||||
<th>State / stage</th>
|
||||
<th>Processed</th>
|
||||
<th>Result</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody id="runs"></tbody>
|
||||
</table>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<footer>
|
||||
Local first · No external services · Working title: verstack
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
+308
@@ -0,0 +1,308 @@
|
||||
:root {
|
||||
font-family: Arial, "Helvetica Neue", sans-serif;
|
||||
font-size: 14px;
|
||||
color: #333;
|
||||
background: #f7f7f7;
|
||||
}
|
||||
* {
|
||||
box-sizing: border-box;
|
||||
}
|
||||
body {
|
||||
margin: 0;
|
||||
}
|
||||
header {
|
||||
background: linear-gradient(#3b3b3b, #222);
|
||||
color: #ddd;
|
||||
border-bottom: 1px solid #080808;
|
||||
padding: 16px 32px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 50px;
|
||||
}
|
||||
.brand {
|
||||
font-size: 21px;
|
||||
}
|
||||
.brand small {
|
||||
font-size: 11px;
|
||||
color: #aaa;
|
||||
margin-left: 12px;
|
||||
}
|
||||
nav {
|
||||
display: flex;
|
||||
gap: 24px;
|
||||
}
|
||||
nav a {
|
||||
color: #ddd;
|
||||
text-decoration: none;
|
||||
}
|
||||
a {
|
||||
color: #337ab7;
|
||||
}
|
||||
main {
|
||||
max-width: 1480px;
|
||||
margin: 24px auto;
|
||||
padding: 0 28px;
|
||||
}
|
||||
h1 {
|
||||
font-size: 28px;
|
||||
font-weight: 400;
|
||||
}
|
||||
h2 {
|
||||
font-size: 23px;
|
||||
font-weight: 400;
|
||||
margin-top: 30px;
|
||||
}
|
||||
.muted {
|
||||
color: #666;
|
||||
line-height: 1.5;
|
||||
}
|
||||
.panel {
|
||||
background: white;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 4px;
|
||||
margin: 15px 0;
|
||||
overflow: auto;
|
||||
box-shadow: 0 1px 1px #0000000d;
|
||||
}
|
||||
.panel-title {
|
||||
padding: 12px 15px;
|
||||
background: linear-gradient(#fafafa, #ededed);
|
||||
border-bottom: 1px solid #ddd;
|
||||
font-weight: bold;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
}
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
}
|
||||
th {
|
||||
text-align: left;
|
||||
font-weight: bold;
|
||||
border-bottom: 2px solid #ddd;
|
||||
}
|
||||
td,
|
||||
th {
|
||||
padding: 10px 12px;
|
||||
vertical-align: top;
|
||||
overflow-wrap: anywhere;
|
||||
}
|
||||
tbody tr:nth-child(even) {
|
||||
background: #f9f9f9;
|
||||
}
|
||||
tbody tr:hover {
|
||||
background: #edf5fb;
|
||||
}
|
||||
td {
|
||||
border-top: 1px solid #ddd;
|
||||
}
|
||||
button {
|
||||
padding: 7px 12px;
|
||||
border: 1px solid #ccc;
|
||||
border-radius: 4px;
|
||||
background: linear-gradient(white, #eee);
|
||||
color: #333;
|
||||
cursor: pointer;
|
||||
font-size: 14px;
|
||||
}
|
||||
button:hover {
|
||||
background: #e6e6e6;
|
||||
}
|
||||
button:disabled {
|
||||
opacity: 0.5;
|
||||
cursor: wait;
|
||||
}
|
||||
.primary {
|
||||
color: #fff;
|
||||
background: linear-gradient(#428bca, #3071a9);
|
||||
border-color: #285e8e;
|
||||
}
|
||||
input,
|
||||
select {
|
||||
font: inherit;
|
||||
border: 1px solid #ccc;
|
||||
border-radius: 4px;
|
||||
padding: 8px;
|
||||
box-shadow: inset 0 1px 1px #00000013;
|
||||
max-width: 100%;
|
||||
}
|
||||
input:focus,
|
||||
button:focus-visible,
|
||||
select:focus {
|
||||
outline: 2px solid #66afe9;
|
||||
outline-offset: 2px;
|
||||
}
|
||||
label {
|
||||
font-weight: bold;
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
align-items: center;
|
||||
}
|
||||
.form-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(3, 1fr);
|
||||
gap: 18px;
|
||||
padding: 20px;
|
||||
}
|
||||
.form-grid label {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: stretch;
|
||||
}
|
||||
.toolbar {
|
||||
padding: 12px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
form p,
|
||||
form > button {
|
||||
margin: 15px 20px;
|
||||
}
|
||||
.metrics {
|
||||
display: flex;
|
||||
gap: 18px;
|
||||
}
|
||||
.metric {
|
||||
background: #fff;
|
||||
border: 1px solid #ddd;
|
||||
border-left: 4px solid #337ab7;
|
||||
border-radius: 3px;
|
||||
padding: 15px 20px;
|
||||
min-width: 160px;
|
||||
}
|
||||
.metric strong {
|
||||
display: block;
|
||||
font-size: 24px;
|
||||
font-weight: 400;
|
||||
margin-bottom: 5px;
|
||||
}
|
||||
.metric span {
|
||||
color: #777;
|
||||
font-size: 12px;
|
||||
}
|
||||
#notice {
|
||||
padding: 12px 16px;
|
||||
background: #d9edf7;
|
||||
border: 1px solid #bce8f1;
|
||||
border-radius: 4px;
|
||||
color: #31708f;
|
||||
}
|
||||
.error {
|
||||
background: #f2dede !important;
|
||||
color: #a94442 !important;
|
||||
border-color: #ebccd1 !important;
|
||||
}
|
||||
.empty {
|
||||
padding: 30px;
|
||||
text-align: center;
|
||||
color: #777;
|
||||
}
|
||||
pre {
|
||||
white-space: pre-wrap;
|
||||
overflow-wrap: anywhere;
|
||||
font: 12px/1.6 monospace;
|
||||
background: #fafafa;
|
||||
border: 1px solid #ddd;
|
||||
padding: 15px;
|
||||
max-height: 500px;
|
||||
overflow: auto;
|
||||
}
|
||||
.hash {
|
||||
font-family: monospace;
|
||||
font-size: 11px;
|
||||
color: #777;
|
||||
}
|
||||
footer {
|
||||
padding: 30px;
|
||||
text-align: center;
|
||||
color: #888;
|
||||
}
|
||||
#assets td:first-child {
|
||||
max-width: 500px;
|
||||
}
|
||||
[hidden] {
|
||||
display: none !important;
|
||||
}
|
||||
@media (max-width: 800px) {
|
||||
.form-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
.metrics {
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
header {
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.brand small {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
#preview img,
|
||||
#preview video {
|
||||
max-width: 100%;
|
||||
max-height: 450px;
|
||||
}
|
||||
#preview audio {
|
||||
width: 100%;
|
||||
}
|
||||
/* Archive dashboard */
|
||||
:root { font-family: Inter, ui-sans-serif, system-ui, sans-serif; color:#263748; background:#eef2f6; }
|
||||
header { background:#192b3b; border-bottom:3px solid #29a3a3; padding:20px 32px; }
|
||||
.brand {font-weight:750; letter-spacing:2px; color:white;}
|
||||
.brand small {letter-spacing:1px; color:#9eb4c7;}
|
||||
nav a {color:#d5e3ed;}
|
||||
main {max-width:1500px;}
|
||||
h1 {font-size:32px; font-weight:650; margin:18px 0 8px;}
|
||||
h2 {font-weight:600;}
|
||||
button {background:#fff; border-color:#bdcbd6; color:#245376;}
|
||||
button:hover {background:#e9f3fa;}
|
||||
button:disabled {cursor:default;}
|
||||
.primary {background:#237eaa; color:white;}
|
||||
.toolbar {flex-wrap:wrap;}
|
||||
.panel {border-color:#d4dfe7; border-radius:7px;}
|
||||
.panel-title {background:#f8fafc; color:#354e62; padding:16px;}
|
||||
.metrics {margin:22px 0;}
|
||||
.metric {flex:1; border:1px solid #d4dfe7; border-top:3px solid #27999e; border-radius:6px;}
|
||||
.metric strong {font-weight:600;}
|
||||
.breadcrumbs {display:flex; align-items:center; gap:10px; margin-top:20px;}
|
||||
.breadcrumbs button {border:0; background:none; padding:3px 0;}
|
||||
.breadcrumbs > * + *::before {content:"/"; color:#9babb8; margin-right:10px;}
|
||||
.game-grid {display:grid; grid-template-columns:repeat(auto-fill,minmax(265px,1fr)); gap:20px; margin:18px 0 40px;}
|
||||
.game-card {background:white; border:1px solid #d4dfe7; border-radius:8px; padding:24px; box-shadow:0 3px 8px #22334406;}
|
||||
.game-icon {background:#e4f0f2; color:#267c86; border:1px solid #c3e0e2; width:58px; height:58px; border-radius:12px; display:grid; place-items:center; font-size:22px; font-weight:700; margin-bottom:22px;}
|
||||
.game-card button {display:block; border:0; padding:0; font-size:20px; font-weight:650; text-align:left; background:none;}
|
||||
.game-card p {color:#677b8c; margin-bottom:0;}
|
||||
.gallery {display:grid; grid-template-columns:repeat(4,minmax(0,1fr)); gap:16px; padding:16px;}
|
||||
.gallery:empty {display:none;}
|
||||
.media-card {border:1px solid #dae2e9; border-radius:6px; overflow:hidden; padding-bottom:16px; min-width:0;}
|
||||
.media-frame {height:190px; background:#e9eef2; display:flex; align-items:center; justify-content:center; overflow:hidden;}
|
||||
.media-frame img {width:100%; height:100%; object-fit:contain;}
|
||||
.media-frame video {width:100%; height:100%; background:#17232e;}
|
||||
.media-frame.audio {background:linear-gradient(135deg,#d5e7ec,#e7e4f1); padding:16px;}
|
||||
.media-frame audio {width:100%;}
|
||||
.media-card h3 {font-size:14px; margin:14px 14px 6px; overflow-wrap:anywhere;}
|
||||
.media-card p {margin:6px 14px; font-size:12px;}
|
||||
.media-path {overflow:hidden; text-overflow:ellipsis; white-space:nowrap; color:#718494;}
|
||||
.media-card button,.media-card a {margin:8px 0 0 14px; font-size:12px;}
|
||||
.preview-error {line-height:1.6; color:#735321;}
|
||||
.pagination {border-top:1px solid #dde5ec;}
|
||||
.pagination span {margin-right:auto; color:#617687;}
|
||||
.artifact-filters {background:#fbfcfd; border-bottom:1px solid #e2e8ee;}
|
||||
.tools summary {cursor:pointer; padding:16px; font-weight:600;}
|
||||
.table-wrap {overflow:auto;}
|
||||
#notice {font-size:13px; padding:9px 14px;}
|
||||
#preview {text-align:left;}
|
||||
#preview img {display:block; margin:auto;}
|
||||
@media(max-width:1100px) {.gallery {grid-template-columns:repeat(3,minmax(0,1fr));}}
|
||||
@media(max-width:800px) {.gallery {grid-template-columns:repeat(2,minmax(0,1fr));} main {padding:0 14px;} header {padding:18px; gap:20px;} .metric {min-width:100px;padding:12px;} .toolbar label {flex-wrap:wrap;} }
|
||||
@media(max-width:480px) {.gallery {grid-template-columns:1fr;} .metrics {gap:8px;} .metric strong {font-size:20px;} }
|
||||
.media-frame img {width:auto; height:auto; max-width:100%; max-height:100%;}
|
||||
.provenance {margin:12px 0; color:#617687; font-size:13px;}
|
||||
.provenance summary {cursor:pointer;}
|
||||
@media(max-width:800px) {
|
||||
.panel table {min-width:760px;}
|
||||
.panel th {white-space:nowrap;}
|
||||
}
|
||||
Reference in New Issue
Block a user