jordan/verstack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Complete native workbench, imports, signatures, and catalog management

Browse Source 
Save the tested release workflows, RAM media processing, program discovery, FLIRT library, and shared-object maintenance. Add Git exclusions, file attributes, and instructions for a later push using a forwarded SSH agent.
This commit is contained in:
Verstack Local
2026-09-13 16:38:57 -05:00
parent 00b59ae87d
commit 841a95c56a
67 changed files with 5624 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitattributes
+9
View File
@@ -0,0 +1,9 @@
* text=auto eol=lf
*.png binary
*.jpg binary
*.jpeg binary
*.webp binary
*.gif binary
*.ico binary
*.sig binary
.gitignore
+40
View File
@@ -2,6 +2,46 @@
/.cargo-home/
/data/
/config.json
/config.*.json
!/config.example.json
/pipeline*.json
/.env
/.env.*
!/.env.example
/.ssh/
/keys/
/secrets/
*.key
*.pem
# Firmware, retained analysis, and disposable local exports belong outside Git.
/firmware/
/imports/
/exports/
/scratch/
*.spk
*.spk.zip
*.img
*.img.*
*.rom
*.gzf
*.sqlite3
*.sqlite3-*
*.db-wal
*.db-shm
*.log
# Local environments, test reports, and editor caches.
/.venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
playwright-report/
test-results/
.DS_Store
*.swp
*.swo
__pycache__/
*.pyc
/tools/
Cargo.lock
Generated
+27
View File
@@ -2,6 +2,12 @@
# It is not intended for manual editing.
version = 4
[[package]]
name = "adler2"
version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "aead"
version = "0.5.2"
@@ -922,6 +928,17 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
[[package]]
name = "fast-flirt"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "deba46f4c332e4ec6035be74f86f6066ac3a2833444e6e900cdda6948b746871"
dependencies = [
"miniz_oxide",
"smallvec",
"thiserror",
]
[[package]]
name = "fastrand"
version = "2.5.0"
@@ -1592,6 +1609,15 @@ version = "0.3.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
[[package]]
name = "miniz_oxide"
version = "0.9.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
dependencies = [
"adler2",
]
[[package]]
name = "mio"
version = "1.2.3"
@@ -2711,6 +2737,7 @@ dependencies = [
"axum",
"blake3",
"clap",
"fast-flirt",
"fs2",
"http-body-util",
"nix",
Cargo.toml
+1
View File
@@ -23,6 +23,7 @@ http-body-util = "0.1"
uuid = { version = "1", features = ["v4", "serde"] }
walkdir = "2"
rusqlite = { version = "0.38", features = ["bundled"] }
fast-flirt = "=0.2.2"
[dev-dependencies]
tower = { version = "0.5", features = ["util"] }
README.md
+30 -6
View File
@@ -6,33 +6,57 @@ This is a development slice with a real Game of Thrones 1.37.0 extraction and na
The new [Theia game workbench](workbench/README.md) is available separately while full-corpus validation continues. See [redesign status and acceptance evidence](docs/redesign-status.md). The compatible static client remains the Rust service default.
For repository contents, cloning, commit identity, and pushing with a forwarded SSH agent, see the [Git workflow](docs/git-workflow.md). Local firmware, archive data, tools, credentials, and generated builds stay outside Git.
## Run
Requirements: Linux x86-64, a current stable Rust toolchain, and Python 3 for the example plugins. Normal operation does not contact external services. Building initially downloads Cargo dependencies; the lockfile is checked in.
Requirements: Linux x86-64, a current stable Rust toolchain, and Python 3 for the example plugins. Browsing saved releases stays local; URL imports contact the selected source when requested. Building initially downloads Cargo dependencies; the lockfile is checked in.
```sh
export PATH="$HOME/.cargo/bin:$PATH"
test -f config.json || cp config.example.json config.json
cargo run --locked -- --config config.json serve
cargo build --locked --release
./target/release/verstack --config config.json serve
```
Open **http://127.0.0.1:8080**. The UI includes filesystem or streamed single-file upload imports, snapshot/asset browsing, safe media previews, text/hex inspection, file comparisons, plugin execution, function comparisons, downloads, and polling run status. It uses a Bootstrap-era utility layout with local CSS/JavaScript and no frontend build requirement.
Use the optimized build for real firmware; debug builds make archive restoration substantially slower. For the native Explorer, asset editors and release comparisons, start the [Theia workbench](workbench/README.md) on port 3000.
The compatibility client remains at **http://127.0.0.1:8080**. It includes filesystem or streamed single-file upload imports, snapshot/asset browsing, safe media previews, text/hex inspection, file comparisons, plugin execution, function comparisons, downloads, and polling run status. It uses a Bootstrap-era utility layout with local CSS/JavaScript and no frontend build requirement.
Edit `config.json` before using real data:
- `archive`: authoritative local archive directory, shared by all games.
- `workspace`: a separate disposable directory; an existing tmpfs path can be selected manually.
- `workspace`: a disposable directory on tmpfs/ramfs. The example uses `/tmp/verstack-workspace`; verify that `/tmp` is mounted in RAM on your machine.
- `import_roots`: existing server directories the HTTP API may import from.
- `workspace_bytes`: default 100 GiB; capture is bounded, and plugins receive a workspace allowance. Plugin usage is polled, so this is not a kernel-enforced quota.
- `bind`: use your server's LAN address or `0.0.0.0:8080` for LAN access. There is no authentication in this slice.
- `require_ram_workspace`: defaults to true. Startup and processing reject disk-backed scratch or insufficient free space; there is no disk fallback. Child tools inherit RAM scratch and cache locations.
- `workspace_bytes`: the scratch allowance must fit available RAM filesystem space (live configuration: 64 GiB). Usage is polled, not a kernel quota.
- `processing_enabled`: pauses automatic submission and queue consumption when false, preserving job history. The live service and example configuration are paused. Explicit CLI/API processing and user-selected import workflows remain available.
- `bind`: the Rust listener defaults to `127.0.0.1:8080`. For the LAN workbench, keep this on loopback and start Theia with `--hostname 0.0.0.0 --port 3000`; its same-origin proxy connects to Rust.
- `plugins`: local command argument arrays and versioned settings. Use absolute paths for plugin scripts. The example uses this workspace's path; change it if you move the project.
The Theia **Import release** view browses server files/folders or URL sources. Paste an Internet Archive item link, click **Browse URL**, choose a file, and supply the game, edition and version. Extraction and asset previews are selected by default; code analysis is optional. **Jobs** shows live progress, cancellation, retry and an **Open release** action. Supported containers include ZIP, SPK, partitioned SD-card images, ext/FAT filesystems and configured LUKS volumes; other files remain cataloged. Downloads and scratch work use RAM, including sparse zero regions in disk images.
The example config allows `samples/`. Import `samples/demo-v2` as version 2, followed by `samples/demo-v1` as version 1, to exercise out-of-order history. These are synthetic files, not game assets.
For large imports, build with `cargo build --release --locked` and run `./target/release/verstack --config config.json serve`. The default development build is useful for testing but substantially slower at chunking.
The CLI works without the service. Stop the service before using the mutating CLI: an instance ownership lock prevents another process from accessing the same archive concurrently.
Persistent user services for this installation are in `deploy/`. After building
the release binary and workbench, install them with:
```sh
systemctl --user enable --now "$PWD/deploy/verstack-backend.service" "$PWD/deploy/verstack-workbench.service"
loginctl enable-linger "$USER"
```
The units assume the checkout is at `~/verstack` and use `config.json`. Stop any
manually launched instances first, allowing active processing to finish. Theia
listens on LAN port 3000; configure Rust on loopback port 8080. Lingering starts
the services at boot and keeps them running after logout. Inspect status and logs
with `systemctl --user status verstack-backend verstack-workbench` and
`journalctl --user -u verstack-backend -u verstack-workbench`.
```sh
cargo run --locked -- import samples/demo-v2 --repository Demo --version 2
cargo run --locked -- list
config.example.json
+44 -5
View File
@@ -1,15 +1,54 @@
{
"archive": "./data/archive",
"workspace": "./data/workspace",
"import_roots": ["./samples"],
"workspace_bytes": 107374182400,
"workspace": "/tmp/verstack-workspace",
"import_roots": [
"./samples"
],
"workspace_bytes": 8589934592,
"bind": "127.0.0.1:8080",
"plugins": {
"zip-extract": {
"command": ["python3", "/home/jordan/verstack/plugins/zip_extract.py"],
"command": [
"python3",
"/home/jordan/verstack/plugins/zip_extract.py"
],
"version": "zip-extract/1",
"settings": {},
"timeout_seconds": 7200
},
"import-extract": {
"command": [
"python3",
"/home/jordan/verstack/plugins/import_extract.py"
],
"version": "import-extract/1",
"settings": {},
"source_root": "/home/jordan/verstack/plugins",
"timeout_seconds": 14400
},
"media-extract": {
"command": [
"python3",
"/home/jordan/verstack/plugins/media_extract.py"
],
"version": "media-extract/4",
"settings": {},
"source_root": "/home/jordan/verstack/plugins",
"processing_revision": 4,
"timeout_seconds": 14400
},
"media-preview": {
"command": [
"python3",
"/home/jordan/verstack/plugins/media_preview.py"
],
"version": "media-preview/3",
"settings": {},
"source_root": "/home/jordan/verstack/plugins",
"processing_revision": 3,
"timeout_seconds": 14400
}
}
},
"require_ram_workspace": true,
"processing_enabled": false
}
deploy/verstack-backend.service
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=Stern ROM archive and processing service
[Service]
Type=simple
WorkingDirectory=%h/verstack
ExecStart=%h/verstack/target/release/verstack --config %h/verstack/config.json serve
Restart=always
RestartSec=3
MemorySwapMax=0
TimeoutStopSec=120
[Install]
WantedBy=default.target
deploy/verstack-workbench.service
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=Stern ROM game workbench
[Service]
Type=simple
WorkingDirectory=%h/verstack/workbench/browser-app
Environment=VERSTACK_API_PORT=8080
ExecStart=/usr/bin/node %h/verstack/workbench/browser-app/lib/backend/main.js --hostname 0.0.0.0 --port 3000
Restart=always
RestartSec=3
TimeoutStopSec=30
[Install]
WantedBy=default.target
docs/git-workflow.md
+68
View File
@@ -0,0 +1,68 @@
# Git workflow
The source repository keeps the existing `master` branch and history. Its remote is:
```text
ssh://git@git.sx.gy/jordan/verstack.git
```
The repository includes application and plugin source, dependency lockfiles, tests,
synthetic demo files, service templates, and documentation. Firmware, archive data,
downloads and installed tools, generated browser bundles, local configuration,
credentials, and test output are excluded by `.gitignore`. The firmware inventory
documents contain metadata, not the firmware payloads. Git is not an archive-data
backup: preserve the live archive and its keys separately as described in
[operation and recovery](redesign-status.md#operation-and-recovery).
## Push with your forwarded SSH agent
Once your SSH agent is available in the server session, run from the repository:
```sh
git status --short --branch
git remote -v
git push -u origin master
```
SSH uses the credentials available to that session. Repository preparation does
not create a key, copy credentials, change SSH configuration, or contact the
remote. If authentication is unavailable, wait until your forwarded agent is
available before pushing.
Existing commits and the local repository identity use
`Verstack Local <verstack@localhost>`. To use your own identity for future commits,
set it only for this repository:
```sh
git config --local user.name "Your name"
git config --local user.email "Your commit email"
```
## Working from a fresh clone
```sh
git clone ssh://git@git.sx.gy/jordan/verstack.git
cd verstack
```
Follow the [backend setup](../README.md#run) and
[Theia build instructions](../workbench/README.md#build-and-run). Copy
`config.example.json` to the ignored `config.json`, then adjust its absolute plugin
paths for the clone location, Python environment, import roots, and RAM workspace.
Installed tools and the live archive are not part of the clone. Service templates
under `deploy/` assume a checkout at `~/verstack`.
For changes, stage the intended source paths and review them before committing:
```sh
git diff --check
git diff --stat
git add PATHS_TO_CHANGED_SOURCE_FILES
git diff --cached --stat
git diff --cached
git commit
git push
```
The lockfiles are tracked. Build output stays local. `.gitattributes` preserves LF
line endings for text files and binary handling for images and FLIRT SIG files.
docs/redesign-status.md
+163 -2
View File
@@ -30,13 +30,13 @@ Pokémon LE 0.85 validation emitted 2,545 WAV outputs across game and menu, with
- Aerosmith main-image audio decoded all 1,522 sounds, but its menu codec remains unsupported. Broaden real node, audio and preview evidence across the full corpus; preserve the explicit upstream node4 limitation.
- Wrapper retirement is deliberately disabled: receipts currently set `wrapper_removal_authorized=false`. Full payload coverage, retirement/reimport state and crash-safe reclamation of published wrappers are not implemented. Orphan cleanup does not reclaim retained wrappers.
- Galleries default to current active outputs and hide exact source copies and derivative thumbnails; retained history remains selectable. Validate database browsing/resource limits against the full corpus before replacing the default UI.
- Workspace limits are polled, processing is serial, logs are capped and decoder budgets/FFmpeg threads are bounded. These are not kernel-enforced memory/disk quotas.
- Workspace limits are polled, the import pipeline runs stages in order, logs are capped, and decoder workers/FFmpeg threads are bounded. DMD compression and image thumbnails run concurrently within a stage. These are not kernel-enforced memory/disk quotas.
## Operation and recovery
Back up the entire archive while the service is idle/stopped, including its key, store, SQLite files, manifests, runs and logs. Do not remove SQLite casually: manifests rebuild browsing, but queued jobs and active-output scheduling state live in the database. `Archive::open` transactionally reconciles immutable manifests, resolves interrupted runs/jobs and repairs automatic submission gaps. Failed or interrupted processing requires explicit retry.
Generate a reviewable pipeline config using `python3 scripts/configure_pipeline.py config.json pipeline.local.json`. It reuses existing tool/key-file references; it does not embed credential contents. Inspect paths and resource settings before starting the service with it. Ghidra remains manually submitted.
Generate a reviewable pipeline config using `python3 scripts/configure_pipeline.py config.json pipeline.local.json`. It reuses existing tool/key-file references; it does not embed credential contents. Inspect paths and resource settings before starting the service with it. An explicit import with code analysis enabled selects game programs automatically; opening existing releases does not submit analysis.
Use `jobs`, `queue INPUT TOOL [--retry] [--priority -10..10]`, and `work` for CLI processing while the service is stopped. `cleanup-orphans` preserves all published manifests and backend snapshots they reference. A cleanup interrupted after its journal is written resumes on next open. Preserve original filesystem source collections.
@@ -83,3 +83,164 @@ lists. Exact matches open paired Monaco decompilation/disassembly/reference view
unmatched functions open individually. Changing either immutable output selection
clears previous comparison evidence. The read-only browser regression verifies
5,526 matches using complete real GoT Pro/LE exports in a disposable archive.
The live Pokémon preview revision 3 completed with 274 verified conversion records
and 30 verified browser-compatible originals, with zero failed records. Identical
inputs share 255 converted files. All 291 visible movie assets have either their
verified preview attached (261 HEVC assets) or a compatible H.264 original (30).
The live Chromium check passes images, audio, gallery video and enlarged video;
local evidence is under `data/validation/preview-v3/` and the workbench screenshots.
Both live GoT edition exports are now published and verified: 11,549 Pro symbols
and 11,724 LE symbols, including reopened GZF exports. The read-only comparison
regression also passes against the live archive with 5,526 exact matches. Live
Code checks verify symbol search, Monaco views, and the selected GZF download.
Pokémon's current Ghidra job is still analyzing; its Code export is pending.
On September 13, the user authorized persistent LAN services. Both deployment
units are enabled and user lingering is enabled. The workbench is running under
systemd on port 3000, with the reported Pokémon processing URL returning HTTP 200.
A one-time systemd handover task waits for the existing backend queue to become
idle before gracefully stopping that process and starting the persistent backend.
This preserves the active Pokémon analysis. Handover status is available with
`systemctl --user status verstack-backend-handover`.
The 73-package revision-4 media refresh remains in progress in a disposable
archive. Unresolved media families remain acceptance blockers; these live fixes
do not establish full-corpus completion.
## Native working-tree redesign — September 13 follow-up
The earlier seven-tab React application is no longer mounted in Theia. The live
workbench now uses a native virtualized Explorer, separate Assets/Compare/Code/
Properties/Processing views, dockable media editor tabs, and Theia resource-backed
text, hex, C, assembly and difference editors. The release selector and native
status bar show game + edition/version (for example Pokémon · LE / 0.85.0).
Intermediate extraction snapshots appear only in provenance and history.
Release APIs resolve the retained extraction chain and current successful outputs.
Added/Removed/Changed/Unchanged filters apply to both Explorer and the asset view;
removed items read from the older release. Common package wrapper paths are
removed. Media comparisons use original asset identities and mark incompatible
decoder revision/schema/settings as unknown coverage. Unmatched code remains
unmatched rather than being mislabelled as publisher additions/removals.
Validation: all 34 Rust tests pass; three Python RAM-policy checks pass; TypeScript
and Theia bundle builds pass. The read-only native browser regression verifies
Pokémon's 4,916 images, native media editors, audio/video playback, C/assembly
editors, branch navigation, and GoT's 5,526 exact function matches in native diff
editors, with zero API mutations. A separate tiny RAM-only fixture archive verifies
Added-only, Removed-only image readback from 0.81.0, and changed-text differences
against 0.85.0. These fixture versions are synthetic, not Pokémon release evidence.
Scripts: `tests/ui-native-workbench.mjs`, `tests/ui-release-comparison.mjs`,
`tests/workspace.rs`, `tests/test_ram_workspace.py`. Screenshots are retained locally
under `data/validation/native-workbench/`.
The live configuration now uses `/tmp/verstack-workspace` on the host's tmpfs with
a 64 GiB allowance. RAM scratch is required and verified at startup and processing;
there is no automatic disk fallback. Child temporary/cache paths stay inside the
run; Python bytecode writes are disabled; bounded logs buffer in memory before
saving. The backend service sets `MemorySwapMax=0` so its scratch pages cannot spill
to swap. Archive publication, manifests, job history and completed logs remain
durable. Standalone corpus validation also checks RAM storage and uses it for all
extraction/preview work, persisting only receipts. For standalone validation, use a
scope with `MemorySwapMax=0` if swap must be prohibited as well.
`processing_enabled=false` pauses automatic submission and worker execution while
preserving history. No new live jobs were submitted during this redesign. Pokémon's
previous Ghidra analysis is interrupted; it has no completed live code export.
Pokémon LE 0.81 source links were located in `/srv/firmware/MORE_FIRMWARE.txt`; that
release was not downloaded or extracted. Its real comparison with 0.85 remains
unverified until it is imported. The Code browser displays retained static analysis;
a debugger, control-flow graph, strings/import indexing and semantic matching are
not implemented. Existing decoder/full-corpus acceptance limits above remain.
Video thumbnails now ask the browser decoder for the first frame when they enter
the visible area. They remain paused and retain playback controls; offscreen
videos defer loading. This uses the existing compatible video preview/original
and creates no new extraction job or saved thumbnail file. The targeted
`tests/ui-video-preview.mjs` regression covers paused frame readiness, deferred
loading and playback.
## Import workflow (September 13, 2026)
The native Import release view now browses configured server roots, supports
multiple files or a whole folder, and inspects direct HTTP(S) URLs and Internet
Archive item links. Archive items show original downloadable filenames and sizes;
inspection does not start a download. Release names remain game, edition, version.
Explicit imports use a durable queue independent of the paused automatic pipeline.
The selected workflow retains originals, extracts recognized containers, decodes
assets/prepares previews, and optionally analyzes supported ELF/PE programs.
Jobs polls live progress, keeps history across navigation/reloads, supports cancel
and retry of unfinished steps, and opens the resulting release. Retry preserves
completed original/extraction steps. A restarted running import is interrupted
until explicitly retried. Existing automatic jobs remain paused.
Progress is measured for download bytes/speed, archive bytes, file iteration,
LUKS decryption, and exported Ghidra functions. Steps with no measurable total show
an indeterminate bar and elapsed time. Live progress control files and logs stay
in RAM; durable task state is written at transitions, not on every progress tick.
Input staging also avoids per-file/per-tick persistent progress writes.
The new import-extract plugin handles nested ZIPs, recognized SPK wrappers,
MBR/GPT disk partitions, ext filesystems, FAT12/16/32, and configured LUKS2 volumes.
It does not mount images or execute imported programs. Other files, including
unsupported console ROMs, remain cataloged; Ghidra import automation currently
selects ELF/PE only. Extended MBR partition chains, unsupported encryption and
unknown filesystems are not decoded. The original is retained on failure, with
an explicit error and retry action. Extraction limits include eight nested layers,
200,000 files, 128 GiB per logical file, and the configured RAM allocation budget.
Sparse zero regions avoid allocating empty SD-card space in tmpfs. Archive roots
and source links are bounded to configured import roots; URL redirects resolve
only public addresses. Network credentials are not accepted in URLs.
Verification uses `tests/imports.rs`, `tests/test_import_sources.py`, and
`tests/ui-import-workflow.mjs`. The browser fixture imports a synthetic ZIP with
an ELF, image and video, checks live progress/reload persistence, and opens its
assets/code. Internet Archive metadata for Pokémon 0.81 was inspected, but no
real Pokémon ROM was downloaded or imported during this work. Full extraction
of that release remains to be tested through the new UI.
Activation and final checks: the live backend now runs the import endpoints and
configured import-extract adapter. Theia serves the rebuilt frontend on refresh.
The broader filesystem-root change was rejected by automatic approval review;
the approved activation preserved `/srv/firmware/images/stern_game_code` and
`/home/jordan/verstack/samples`, all existing service/network/RAM settings, and
`processing_enabled=false`, restarting only the idle backend. There were still
13 historical jobs, zero queued/running jobs, and zero import tasks after the
live smoke tests. RAM scratch is 64 GiB with `MemorySwapMax=0`.
All 38 Rust tests and 47 Python plugin tests passed, with targeted rechecks after
final changes. The disposable browser import completed extraction, previews and
Ghidra, survived a browser reload, and opened the resulting assets/code. Live
read-only native-workbench and video-first-frame regressions also passed. The
live import smoke check browsed the existing server root and the real Internet
Archive file list without submitting an import. Activity polling returned in
17 ms in the live check; it reads release summaries, not full file manifests.
## Bitmap display, bounded parallelism, and program discovery
- Native Theia **View** menus control layout and DMD tint. The orange filter applies only to assets marked **Approximate bitmap**, in thumbnails and image editors; originals and ordinary images are unchanged. The preferences persist in the browser.
- Asset, symbol, comparison, server-file, and signature pagination accept a page number (Enter or Go).
- DMD index decoding retains keyframe order, uses bulk byte operations, and compresses PNGs concurrently. Thumbnails run concurrently with ordered, budget-checked publication. Workers respect CPU affinity, cgroup CPU/memory limits, available memory, and the RAM workspace budget. On the current 54-CPU / 64-GiB-workspace setup the upper bounds are 52 DMD/SPK verification workers, 32 thumbnail workers, and 8 FFmpeg codec threads. SPK file writes, dependent DMD frames, and the overall import pipeline remain ordered.
- Synthetic RAM benchmark: 1,600 128×32 DMD frames, 14.536s before vs 1.783s automatic; 384 480×240 grayscale thumbnails, 10.705s before vs 1.114s automatic. All decoded indices, PNGs, WebPs, and evidence records matched the baseline byte-for-byte. These are stage timings, not a promised whole-import speedup.
- Imports with code analysis select every supported executable named `game` (including `.exe`, `.elf`, and `.bin`), largest first. With no game program, the largest supported executable is selected. The code browser discovers all supported program candidates, shows analysis availability and sizes, and offers explicit single-program or game-program analysis. Unrelated automatic jobs remain paused.
- Completed Ghidra exports contribute exact body hashes and bounded, operand-masked FLIRT patterns to a persistent function library. Reuse counts use distinct executable SHA-256 identities, so repeated imports and re-analysis do not increase counts. Older exports supply their already-saved exact hashes without analysis jobs; collecting FLIRT patterns for them requires an explicit re-analysis.
- Function signatures supports importing `.pat` and `.sig`, retaining original library bytes, and exporting generated `.pat` or FLIRT v9 `.sig` files (SIG processor headers currently cover x86 and ARM/AArch64). The pinned `fast-flirt` 0.2.2 parser checks imports and generated exports. Pattern matches are candidates; referenced names and behavior are not assumed equivalent and symbols are not automatically renamed. Saved match windows are bounded to 512 bytes; patterns needing more bytes cannot match those observations.
## Catalog storage and removal
**Catalog & storage** is a native Theia view available from Explorer and the command menu. It lists every release and a searchable, paged inventory of original files and saved processing results. Images and other files open in the existing native asset editors.
**Measure storage** reads stored file/chunk metadata without extracting payloads. It separates exact-file deduplication, additional reuse of byte ranges, and compression. Release rows show packed data, bytes shared with other releases, and exclusive bytes. Shared row totals are not additive. Visual similarity and function recognition do not merge storage, so those matches are not counted as disk savings.
**Delete version** removes every retained extraction and analysis belonging to that edition/version/generation. **Delete all versions** removes the selected game across editions. The confirmation previews file counts, dependent comparison reports, protected shared data, and data eligible for collection. A review token rejects a deletion if the catalog changes before confirmation. Queued/running work prevents maintenance. Durable deletion journals are replayed before manifest reconciliation, including after an index rebuild; save them with archive backups.
Deletion removes catalog references, function observations, and related activity records, while retaining chunks until explicit **Garbage collect**. The collector preserves every remaining snapshot reference and refuses collection while files are being streamed. It removes orphan backend snapshots and prunes unreferenced chunks, reporting actual packed bytes reclaimed. Original source files and separately imported FLIRT libraries remain available.
Validation: the Rust suite has 45 passing tests, including shared data across games/versions, deleting the last release, dependent comparisons, stale reviews, active work/readers, and interrupted-deletion recovery. An additional history test covers imports older than the 200-entry activity page. All 51 Python tests pass. `tests/ui-catalog-management.mjs` creates its own disposable archive and redirects all browser API traffic to it for storage measurement, file paging/opening, cancellation and confirmation, version/game deletion, garbage collection, and byte-for-byte retained-file verification. It never mutates the live archive.
The disposable catalog browser workflow passed, including switching between file inventory and releases during loading. The live page was then checked with GET requests only: four releases, 168,519 cataloged files, no browser errors, automatic processing still paused, and zero queued/running tasks. Initial storage measurement took 11.42 seconds and is cached until archive contents change. No live versions were deleted and no live garbage collection was performed. The rebuilt backend and Theia bundle are active; refresh existing browser tabs.
Program/signature validation used an isolated import with 26 executable candidates and two compiled `game` programs. Actual Ghidra analysis selected those two, exposed all candidates in the native picker, and populated exact and FLIRT observations. Repeated analysis retained two distinct executable identities. Generated PAT/SIG files round-tripped through the independent FLIRT parser and matched imported libraries; the installed IDA application itself was not used for verification. The live DMD/page-entry browser check passed with zero API writes and confirmed ordinary Pokémon images keep their original colors.
examples/storage_spike.rs
+2
View File
@@ -9,6 +9,8 @@ fn main() -> Result<()> {
let config = Config {
archive: temp.path().join("archive"),
workspace: temp.path().join("work"),
require_ram_workspace: true,
processing_enabled: false,
import_roots: vec![input.clone()],
workspace_bytes: 128 * 1024 * 1024,
bind: "127.0.0.1:0".into(),
plugins/dmd_bitmap.py
+13 -6
View File
@@ -29,15 +29,22 @@ def decode(record, keyframes):
def write(values):
nonlocal cursor
if cursor+len(values)>size:raise ValueError('DMD packet exceeds pixel bounds')
for value in values:
offset=(cursor%height)*width+cursor//height if mode in (1,3) else cursor
pixels[offset]=value;cursor+=1
if mode not in (1,3):
pixels[cursor:cursor+len(values)]=values;cursor+=len(values)
else:
start=0
while start<len(values):
count=min(height-cursor%height,len(values)-start)
offset=(cursor%height)*width+cursor//height
pixels[offset:offset+count*width:width]=values[start:start+count]
cursor+=count;start+=count
if mode==0:
pixels[:]=take(size);cursor=size
elif mode==12:
for value in take((size+1)//2):
write([value&15])
if cursor<size:write([value>>4])
packed=bytes(take((size+1)//2))
pixels[0::2]=packed.translate(bytes(value&15 for value in range(256)))
pixels[1::2]=packed[:size//2].translate(bytes(value>>4 for value in range(256)))
cursor=size
else:
while True:
token=take(1)[0]
plugins/fat_files.py
+94
View File
@@ -0,0 +1,94 @@
"""Read FAT12/16/32 volumes without mounting them or following host links."""
import pathlib
import struct
from progress import report
def recognized(head):
return len(head)>=512 and head[510:512]==b'\x55\xaa' and (head[54:62] in (b'FAT12 ',b'FAT16 ') or head[82:90]==b'FAT32 ')
def extract(image, destination, budget):
destination.mkdir(parents=True,exist_ok=True)
with image.open('rb') as stream:
head=stream.read(512)
sector=struct.unpack_from('<H',head,11)[0];cluster_sectors=head[13]
reserved=struct.unpack_from('<H',head,14)[0];copies=head[16];root_count=struct.unpack_from('<H',head,17)[0]
sectors=struct.unpack_from('<H',head,19)[0] or struct.unpack_from('<I',head,32)[0]
fat_sectors=struct.unpack_from('<H',head,22)[0] or struct.unpack_from('<I',head,36)[0]
if sector not in (512,1024,2048,4096) or cluster_sectors not in (1,2,4,8,16,32,64,128) or copies not in (1,2) or reserved<1:
raise ValueError('Invalid FAT geometry')
root_sectors=(root_count*32+sector-1)//sector
data_sector=reserved+copies*fat_sectors+root_sectors
if not 0<data_sector<sectors or sectors*sector>image.stat().st_size:raise ValueError('FAT volume exceeds image bounds')
clusters=(sectors-data_sector)//cluster_sectors;bits=12 if clusters<4085 else 16 if clusters<65525 else 32
fat_size=fat_sectors*sector;cluster_size=cluster_sectors*sector
if fat_size>128*1024**2 or fat_size*8//bits<clusters+2:raise ValueError('Invalid FAT table size')
stream.seek(reserved*sector);fat=stream.read(fat_size)
def chain(first):
current=first;seen=set();end={12:0xff8,16:0xfff8,32:0xffffff8}[bits]
while current<end:
if not 2<=current<clusters+2 or current in seen:raise ValueError('Invalid or cyclic FAT cluster chain')
seen.add(current);yield current
if bits==12:
offset=current*3//2;value=int.from_bytes(fat[offset:offset+2],'little');current=(value>>4 if current&1 else value)&0xfff
elif bits==16:current=struct.unpack_from('<H',fat,current*2)[0]
else:current=struct.unpack_from('<I',fat,current*4)[0]&0xfffffff
def read_cluster(index):
stream.seek((data_sector+(index-2)*cluster_sectors)*sector);data=stream.read(cluster_size)
if len(data)!=cluster_size:raise ValueError('Truncated FAT cluster')
return data
if bits==32:pending=[(struct.unpack_from('<I',head,44)[0],pathlib.Path())]
else:pending=[(None,pathlib.Path())]
seen_dirs=set();names=set();used=files=0
while pending:
first,relative=pending.pop()
if first in seen_dirs:raise ValueError('Repeated FAT directory cluster')
seen_dirs.add(first)
if len(seen_dirs)>200000:raise ValueError('FAT directory count exceeds limit')
if first is None:
stream.seek((reserved+copies*fat_sectors)*sector);data=stream.read(root_count*32)
else:
chunks=[];size=0
for cluster in chain(first):
size+=cluster_size
if size>16*1024**2:raise ValueError('FAT directory exceeds size limit')
chunks.append(read_cluster(cluster))
data=b''.join(chunks)
long=[]
for pos in range(0,len(data),32):
entry=data[pos:pos+32]
if not entry or entry[0]==0:break
if entry[0]==0xe5:long=[];continue
attr=entry[11]
if attr==0x0f:
if entry[0]&0x40:long=[]
long.append((entry[0]&0x1f,entry[1:11]+entry[14:26]+entry[28:32],entry[13]));continue
raw=entry[:11];checksum=0
for c in raw:checksum=(((checksum&1)<<7)+(checksum>>1)+c)&255
if long and len(long)<=20 and sorted(x[0] for x in long)==list(range(1,len(long)+1)) and all(x[2]==checksum for x in long):
name=b''.join(x[1] for x in sorted(long)).decode('utf-16le').split('\0')[0].rstrip('\uffff')
else:
base=raw[:8].decode('cp437').rstrip();ext=raw[8:].decode('cp437').rstrip();name=base+('.'+ext if ext else '')
long=[]
if attr&8 or name in ('.','..'):continue
if not name or any(c in name for c in '/\\\0\r\n'):raise ValueError('Unsafe FAT filename')
path=relative/name
if path in names:raise ValueError('Duplicate FAT path')
names.add(path)
if len(names)>200000:raise ValueError('FAT file count exceeds limit')
start=struct.unpack_from('<H',entry,26)[0]
if bits==32:start|=struct.unpack_from('<H',entry,20)[0]<<16
target=destination/path
if attr&16:
target.mkdir(parents=True,exist_ok=False);pending.append((start,path));continue
size=struct.unpack_from('<I',entry,28)[0];used+=size
if used>budget:raise ValueError('FAT files exceed RAM extraction allowance')
target.parent.mkdir(parents=True,exist_ok=True)
with target.open('xb') as output:
remaining=size
if remaining:
for cluster in chain(start):
block=read_cluster(cluster)[:remaining];output.write(block);remaining-=len(block)
if not remaining:break
if remaining:raise ValueError('FAT file chain ended early')
files+=1;report('Reading FAT filesystem',files,None,'files',path.as_posix())
return files
plugins/ghidra/ExportFacts.java
+50
View File
@@ -11,6 +11,23 @@ import java.security.MessageDigest;
import java.util.HexFormat;
public class ExportFacts extends HeadlessScript {
private long progressAt;
private void progress(int completed, int total, String detail) throws IOException {
String target = System.getenv("VERSTACK_PROGRESS_FILE");
long now = System.nanoTime();
if (target == null || now - progressAt < 250_000_000L) return;
progressAt = now;
JsonObject value = new JsonObject();
value.addProperty("stage", "Exporting functions");
value.addProperty("completed", completed);
value.addProperty("total", total);
value.addProperty("unit", "functions");
value.addProperty("detail", detail);
java.nio.file.Path path = java.nio.file.Path.of(target);
java.nio.file.Path temporary = path.resolveSibling(path.getFileName() + ".next");
java.nio.file.Files.writeString(temporary, new Gson().toJson(value));
java.nio.file.Files.move(temporary, path, java.nio.file.StandardCopyOption.REPLACE_EXISTING, java.nio.file.StandardCopyOption.ATOMIC_MOVE);
}
@Override public void run() throws Exception {
String[] args = getScriptArgs();
File output = new File(args[0]);
@@ -30,6 +47,7 @@ public class ExportFacts extends HeadlessScript {
if (!code.mkdirs() && !code.isDirectory()) throw new IOException("Cannot create code output");
JsonArray functions = new JsonArray();
for (Function function : currentProgram.getFunctionManager().getFunctions(true)) {
progress(functions.size(), currentProgram.getFunctionManager().getFunctionCount(), function.getName());
monitor.checkCancelled();
JsonObject item = new JsonObject();
item.addProperty("address", function.getEntryPoint().toString());
@@ -56,6 +74,18 @@ public class ExportFacts extends HeadlessScript {
}
}
item.addProperty("body_sha256", hashable ? HexFormat.of().formatHex(hash.digest()) : "");
// FLIRT patterns require a contiguous module starting at the entry.
// Keep the full body hash even when this narrower representation is unavailable.
byte[] signature = null;
boolean[] variant = null;
if (hashable && function.getBody().getNumAddressRanges() == 1 &&
function.getBody().getMinAddress().equals(function.getEntryPoint()) &&
function.getBody().getNumAddresses() >= 32) {
signature = new byte[(int)Math.min(512, function.getBody().getNumAddresses())];
if (currentProgram.getMemory().getBytes(function.getEntryPoint(), signature) != signature.length)
throw new IOException("Incomplete signature bytes");
variant = new boolean[signature.length];
}
String stem = function.getEntryPoint().toString().replace(':', '_');
JsonObject detail = new JsonObject();
detail.addProperty("address", function.getEntryPoint().toString());
@@ -72,8 +102,28 @@ public class ExportFacts extends HeadlessScript {
reference.addProperty("type", ref.getReferenceType().toString());
reference.addProperty("source", ref.getSource().toString());
references.add(reference);
if (signature != null && ref.getToAddress().isMemoryAddress() &&
!function.getBody().contains(ref.getToAddress()) && !ref.getReferenceType().isFallthrough()) {
long offset = instruction.getAddress().subtract(function.getEntryPoint());
int operand = ref.getOperandIndex();
if (offset >= 0 && offset < signature.length) {
var mask = operand >= 0 ? instruction.getPrototype().getOperandValueMask(operand) : null;
byte[] bits = mask == null ? null : mask.getBytes();
// Unknown operand masks conservatively wildcard the instruction.
for (int b = 0; b < instruction.getLength() && offset+b < variant.length; b++)
if (bits == null || b >= bits.length || bits[b] != 0) variant[(int)offset+b] = true;
}
}
}
}
if (signature != null) {
StringBuilder pattern = new StringBuilder();
for (int b = 0; b < signature.length; b++)
pattern.append(variant[b] ? ".." : String.format("%02X", signature[b] & 255));
item.addProperty("signature_bytes", HexFormat.of().formatHex(signature));
item.addProperty("signature_pattern", pattern.toString());
item.addProperty("signature_method", "flirt-operand-mask/1");
} else item.addProperty("signature_unavailable", "FLIRT requires a contiguous initialized body of at least 32 bytes");
detail.addProperty("disassembly", disassembly.toString());
var result = decompiler.decompileFunction(function, 30, monitor);
detail.addProperty("decompile_status", result.decompileCompleted() ? "complete" : "unavailable");
plugins/ghidra/analyze.py
+9 -2
View File
@@ -6,6 +6,8 @@ import os
import pathlib
import subprocess
import sys
sys.path.insert(0,str(pathlib.Path(__file__).resolve().parent.parent))
from progress import report
def run(request):
if request["protocol"] != 1:
@@ -37,7 +39,10 @@ def run(request):
files, warnings = [], []
selected = settings.get("paths")
count = 0
for path in sorted(source.rglob("*")):
# Analyze the largest game first; successful exports retain their real source names.
candidates=[p for p in source.rglob('*') if p.is_file() and not p.is_symlink()]
candidates.sort(key=lambda p:(p.stem.lower()!='game',-p.stat().st_size,p.as_posix()))
for path in candidates:
if not path.is_file() or path.is_symlink():
continue
rel = path.relative_to(source).as_posix()
@@ -58,6 +63,7 @@ def run(request):
"-postScript", "ExportFacts.java", str(dest), identity, "-deleteProject",
"-max-cpu", str(settings.get("max_cpu", 2)),
"-analysisTimeoutPerFile", str(settings.get("analysis_timeout_seconds", 3600))]
report('Analyzing program',None,None,'functions',rel,force=True)
with (dest / "analysis.log").open("w") as log:
subprocess.run(command, check=True, env=env, stdout=log, stderr=subprocess.STDOUT)
facts = json.loads((dest / "functions.json").read_text())
@@ -68,6 +74,7 @@ def run(request):
# A successful process exit is insufficient: re-import the packed database,
# and require a post-script receipt with the same input identity.
receipt = dest / "reopened.json"
report('Verifying saved code',None,None,'functions',rel,force=True)
with (dest / "reopen.log").open("w") as log:
subprocess.run([str(launcher), str(project), identity + "-check", "-import", str(dest / "program.gzf"),
"-noanalysis", "-scriptPath", str(scripts), "-postScript", "VerifyExport.java", str(receipt), identity, str(dest / "functions.json"),
@@ -76,7 +83,7 @@ def run(request):
raise ValueError("export failed reopen validation")
if json.loads(receipt.read_text()).get("symbols_verified") != len(facts["functions"]):
raise ValueError("exported symbols failed reopen validation")
facts.update({"source_path": rel, "ghidra_version": version, "settings": settings})
facts.update({"source_path": rel, "source_size":path.stat().st_size, "ghidra_version": version, "settings": settings})
(dest / "functions.json").write_text(json.dumps(facts))
files.extend(p.relative_to(output).as_posix() for p in dest.rglob("*") if p.is_file())
if not count:
plugins/import_extract.py
+225
View File
@@ -0,0 +1,225 @@
#!/usr/bin/env python3
"""Detect nested archives and SD-card filesystems in userspace, entirely in RAM."""
import hashlib
import json
import os
import pathlib
import re
import shutil
import stat
import struct
import subprocess
import sys
import tempfile
import zipfile
import zlib
from progress import report
import fat_files
def safe_name(name):
p=pathlib.PurePosixPath(name)
if p.is_absolute() or '..' in p.parts or '\\' in name or any(c in name for c in '\0\r\n') or not p.parts:
raise ValueError('Unsafe archive member path')
return p
def container(path):
with path.open('rb') as f:head=f.read(4096)
return zipfile.is_zipfile(path) or head[:4] in (b'SPKS',b'hsqs') or head[:6]==b'LUKS\xba\xbe' or head[1080:1082]==b'\x53\xef' or head[510:512]==b'\x55\xaa' or head[:2]==b'\x1f\x8b' and path.suffix.lower()=='.spk'
def sparse_copy(reader, target, length, budget, label):
"""Zero-filled SD-card padding becomes holes, not allocated tmpfs pages."""
if length < 0 or length > 128*1024**3:
raise ValueError('Image exceeds the 128 GiB logical image limit')
target.parent.mkdir(parents=True,exist_ok=True)
completed=allocated=0
with target.open('xb') as output:
while completed < length:
data=reader.read(min(1024*1024,length-completed))
if not data:raise ValueError('Truncated image or archive entry')
completed+=len(data)
if data.count(0)==len(data):output.seek(len(data),1)
else:
allocated+=len(data)
if allocated>budget:raise ValueError('Unpacked content exceeds available RAM')
output.write(data)
report('Unpacking',completed,length,'bytes',label)
output.truncate(length)
return allocated
def partitions(image):
size=image.stat().st_size
with image.open('rb') as f:
head=f.read(4096)
if len(head)<512 or head[510:512]!=b'\x55\xaa':return []
f.seek(512);gpt=f.read(512)
result=[]
if gpt[:8]==b'EFI PART':
header_size=struct.unpack_from('<I',gpt,12)[0]
if not 92<=header_size<=512:raise ValueError('Invalid GPT header size')
crc=struct.unpack_from('<I',gpt,16)[0];check=bytearray(gpt[:header_size]);check[16:20]=b'\0'*4
if zlib.crc32(check)!=crc:raise ValueError('GPT header checksum mismatch')
table,count,entry_size,table_crc=struct.unpack_from('<QIII',gpt,72)
if count>1024 or not 128<=entry_size<=4096 or table*512+count*entry_size>size:raise ValueError('Invalid GPT partition table bounds')
f.seek(table*512);data=f.read(count*entry_size)
if zlib.crc32(data)!=table_crc:raise ValueError('GPT partition table checksum mismatch')
for index in range(count):
entry=data[index*entry_size:(index+1)*entry_size]
if entry[:16]==bytes(16):continue
first,last=struct.unpack_from('<QQ',entry,32)
if last<first:raise ValueError('Invalid GPT partition range')
result.append((index+1,first*512,(last-first+1)*512))
else:
for index in range(4):
entry=head[446+index*16:462+index*16]
kind=entry[4];first,count=struct.unpack_from('<II',entry,8)
if not kind or not count:continue
if kind in (5,15,0x85):raise ValueError('Extended MBR partitions are not supported yet; original image is retained')
if kind==0xee:raise ValueError('Protective MBR has no valid GPT header')
result.append((index+1,first*512,count*512))
ordered=sorted(result,key=lambda p:p[1]);end=512
for _,offset,length in ordered:
if offset<end or offset+length>size:raise ValueError('Overlapping or out-of-bounds disk partitions')
end=offset+length
return result
def ext_files(image,destination,settings,budget):
"""Walk inode numbers and dump only regular files; never follow filesystem symlinks."""
tool=settings.get('debugfs','/usr/sbin/debugfs')
pending=[(2,pathlib.Path())];seen=set();used=0;files=0;links=[]
destination.mkdir(parents=True,exist_ok=True)
while pending:
inode,relative=pending.pop()
if inode in seen:continue
seen.add(inode)
process=subprocess.run([tool,'-R',f'ls -p <{inode}>',str(image)],capture_output=True,check=True,text=True)
rows=process.stdout.splitlines()
if not rows:raise ValueError('Filesystem directory could not be listed')
for line in rows:
if not line.startswith('/'):continue
fields=line.split('/')
if len(fields)<7:raise ValueError('Unsupported filesystem filename')
child,mode,name=fields[1],fields[2],fields[5]
if name in ('.','..') or child=='0':continue
safe_name(name)
if '/' in name or '"' in name:raise ValueError('Unsupported filesystem filename')
path=relative/name;kind=int(mode,8)
if stat.S_ISDIR(kind):
if len(seen)+len(pending)>200000:raise ValueError('Filesystem directory limit exceeded')
pending.append((int(child),path));continue
if not stat.S_ISREG(kind):links.append({'path':path.as_posix(),'mode':mode});continue
size=int(fields[6] or 0);used+=size
if used>budget:raise ValueError('Filesystem files exceed the RAM extraction allowance')
dest=destination/path;dest.parent.mkdir(parents=True,exist_ok=True)
dump=subprocess.run([tool,'-R',f'dump <{int(child)}> "{dest}"',str(image)],capture_output=True)
if dump.returncode or not dest.is_file() or dest.stat().st_size!=size:raise ValueError(f'Could not read complete filesystem file: {path}')
files+=1
report('Reading filesystem',files,None,'files',path.as_posix())
return links
class Extractor:
def __init__(self,request):
self.request=request;self.output=pathlib.Path(request['output_dir']);self.settings=request.get('settings') or {};self.budget=request['workspace_bytes']//2
self.records=[];self.warnings=[];self.count=0
def run_file(self,path,destination,depth=0):
self.count+=1
if depth>8 or self.count>200000:raise ValueError('Nested archive or file count limit exceeded')
with path.open('rb') as stream:head=stream.read(4096)
if zipfile.is_zipfile(path):
with zipfile.ZipFile(path) as archive:
entries=[e for e in archive.infolist() if not e.is_dir()]
if len(entries)>200000:raise ValueError('ZIP file count limit exceeded')
split=entries and all(re.fullmatch(r'.+\.spk\.\d{3}\.\d{3}',e.filename,re.I) for e in entries)
if split:return self.spk(path,destination,True)
names=set()
for entry in entries:
name=safe_name(entry.filename)
if str(name) in names:raise ValueError('Duplicate ZIP member path')
names.add(str(name))
if stat.S_ISLNK(entry.external_attr>>16):self.warnings.append('ZIP symlink retained only in original: '+entry.filename);continue
with tempfile.TemporaryDirectory(prefix='member-',dir=self.output.parent) as tmp:
staged=pathlib.Path(tmp)/name.name
with archive.open(entry) as reader:sparse_copy(reader,staged,entry.file_size,self.budget,entry.filename)
# A single archive-wrapped image/package is a transport wrapper,
# not a version-named directory in the working tree.
target=destination if len(entries)==1 and container(staged) else destination/name
self.run_file(staged,target,depth+1)
self.records.append({'source':path.name,'format':'zip','members':len(entries)})
elif head[:4] in (b'SPKS',b'hsqs') or (head[:2]==b'\x1f\x8b' and path.name.lower().endswith('.spk')):
self.spk(path,destination)
elif head[:6]==b'LUKS\xba\xbe':
self.luks(path,destination,depth)
elif len(head)>1082 and head[1080:1082]==b'\x53\xef':
with tempfile.TemporaryDirectory(prefix='filesystem-',dir=self.output.parent) as tmp:
tree=pathlib.Path(tmp)/'files'
links=ext_files(path,tree,self.settings,self.budget)
for member in sorted(tree.rglob('*')):
if member.is_file():self.run_file(member,destination/member.relative_to(tree),depth+1)
self.records.append({'source':path.name,'format':'ext','links_preserved_in_original':links})
if links:self.warnings.append('Filesystem links and special files remain in the original image')
elif fat_files.recognized(head):
with tempfile.TemporaryDirectory(prefix='fat-',dir=self.output.parent) as tmp:
tree=pathlib.Path(tmp)/'files'
fat_files.extract(path,tree,self.budget)
for member in sorted(tree.rglob('*')):
if member.is_file():self.run_file(member,destination/member.relative_to(tree),depth+1)
self.records.append({'source':path.name,'format':'fat'})
elif head[510:512]==b'\x55\xaa' and partitions(path):
parts=partitions(path)
with path.open('rb') as source:
for index,offset,length in parts:
with tempfile.TemporaryDirectory(prefix='partition-',dir=self.output.parent) as tmp:
part=pathlib.Path(tmp)/f'partition-{index:02d}.img';source.seek(offset)
sparse_copy(source,part,length,self.budget,part.name)
self.run_file(part,destination/f'partition-{index:02d}',depth+1)
self.records.append({'source':path.name,'format':'disk-image','partitions':[{'index':i,'offset':o,'bytes':n} for i,o,n in parts]})
else:
destination.parent.mkdir(parents=True,exist_ok=True)
if destination.exists():raise ValueError('Multiple inputs resolve to the same output path')
with path.open('rb') as stream:sparse_copy(stream,destination,path.stat().st_size,self.budget,path.name)
self.records.append({'source':path.name,'format':'file','bytes':path.stat().st_size})
if path.suffix.lower() in ('.raw','.img','.spk'):self.warnings.append('Unrecognized container retained as a file: '+path.name)
def spk(self,path,destination,split=False):
import spike_package
with tempfile.TemporaryDirectory(prefix='spk-',dir=self.output.parent) as tmp:
inputs=pathlib.Path(tmp)/'input';inputs.mkdir();alias=inputs/('package.spk.zip' if split else 'package.spk');os.link(path,alias)
destination.mkdir(parents=True,exist_ok=True)
result=spike_package.run({**self.request,'input_dir':str(inputs),'output_dir':str(destination),'settings':self.settings})
self.warnings.extend(result['warnings']);self.records.append({'source':path.name,'format':'spk'})
def luks(self,path,destination,depth):
import luks_extract
key_file=self.settings.get('key_file')
if not key_file:raise ValueError('Encrypted image detected, but no decryption key is configured')
metadata=json.loads(subprocess.check_output([self.settings.get('cryptsetup','/usr/sbin/cryptsetup'),'luksDump','--dump-json-metadata',str(path)]))
credential=pathlib.Path(key_file).read_bytes();key=luks_extract.volume_key(path,credential);del credential
with tempfile.TemporaryDirectory(prefix='decrypt-',dir=self.output.parent) as tmp:
plain=pathlib.Path(tmp)/'filesystem.ext4'
report('Decrypting filesystem',None,None,'bytes',path.name,force=True)
luks_extract.decrypt(path,plain,key,metadata);del key
self.run_file(plain,destination,depth+1)
self.records.append({'source':path.name,'format':'luks-ext'})
def run(request):
if request['protocol']!=1:raise ValueError('Unsupported protocol')
worker=Extractor(request);source=pathlib.Path(request['input_dir'])
paths=[p for p in sorted(source.rglob('*')) if p.is_file() and not p.is_symlink()]
for index,path in enumerate(paths):
report('Detecting files',index,len(paths),'files',path.name,force=True)
target=worker.output if len(paths)==1 and container(path) else worker.output/path.relative_to(source)
worker.run_file(path,target)
report('Cataloging extracted files',len(paths),len(paths),'files',force=True)
with (worker.output/'import-evidence.json').open('x') as evidence:
json.dump({'schema':1,'detected':worker.records,'warnings':worker.warnings},evidence)
return {'protocol':1,'layer':'extracted','coverage':'partial' if worker.warnings else 'complete','warnings':worker.warnings,'files':[p.relative_to(worker.output).as_posix() for p in sorted(worker.output.rglob('*')) if p.is_file()]}
if __name__=='__main__':
request=json.loads(pathlib.Path(sys.argv[1]).read_text())
try:result=run(request)
except Exception as error:
pathlib.Path(request['result_file']).write_text(json.dumps({'protocol':1,'error':str(error)}));raise
pathlib.Path(request['result_file']).write_text(json.dumps(result))
plugins/import_source.py
+160
View File
@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""HTTP(S) source inspection and bounded RAM downloads; never runs downloaded code."""
import email.message
import hashlib
import http.client
import ipaddress
import json
import os
import pathlib
import socket
import ssl
import sys
import time
import urllib.parse
def progress(path, **value):
if not path:
return
target = pathlib.Path(path)
temporary = target.with_suffix('.next')
temporary.write_text(json.dumps(value))
temporary.replace(target)
def connect(url, method='GET', allow_private=False):
"""Validate every redirect and pin the validated address through TLS connection."""
for _ in range(6):
parsed = urllib.parse.urlsplit(url)
if parsed.scheme not in ('https', 'http') or not parsed.hostname or parsed.username or parsed.password:
raise ValueError('Use an HTTP or HTTPS URL without embedded credentials')
port = parsed.port or (443 if parsed.scheme == 'https' else 80)
addresses = socket.getaddrinfo(parsed.hostname, port, type=socket.SOCK_STREAM)
if not addresses:
raise ValueError('Source hostname could not be resolved')
ips = [address[4][0] for address in addresses]
if not allow_private and any(not ipaddress.ip_address(ip).is_global for ip in ips):
raise ValueError('URL imports require a public internet address')
cls = http.client.HTTPSConnection if parsed.scheme == 'https' else http.client.HTTPConnection
connection = cls(parsed.hostname, port, timeout=30)
def pinned_connection(address, timeout=30, source_address=None):
last=None
for ip in ips:
try:return socket.create_connection((ip,port),timeout,source_address)
except OSError as error:last=error
raise last
connection._create_connection = pinned_connection
connection.request(method, urllib.parse.urlunsplit(('', '', parsed.path or '/', parsed.query, '')),
headers={'User-Agent':'Verstack/0.1 (user-requested archive import)', 'Accept-Encoding':'identity'})
response = connection.getresponse()
if response.status in (301, 302, 303, 307, 308):
location = response.getheader('Location')
connection.close()
if not location:
raise ValueError('Redirect is missing a destination')
url = urllib.parse.urljoin(url, location)
continue
if not 200 <= response.status < 300:
connection.close()
raise ValueError(f'Source returned HTTP {response.status} {response.reason}')
return connection, response, url
raise ValueError('Too many source redirects')
def filename(url, response):
header = email.message.Message()
header['Content-Disposition'] = response.getheader('Content-Disposition', '')
name = header.get_filename() or urllib.parse.unquote(urllib.parse.urlsplit(url).path.rsplit('/',1)[-1]) or 'download.bin'
if name in ('.', '..') or any(c in name for c in '/\\\x00\r\n') or len(name.encode()) > 240:
raise ValueError('Source filename is invalid; use a direct file URL')
return name
def inspect(url, allow_private=False):
parsed = urllib.parse.urlsplit(url)
parts = [urllib.parse.unquote(p) for p in parsed.path.split('/') if p]
if parsed.hostname in ('archive.org', 'www.archive.org') and len(parts) == 2 and parts[0] in ('details','download'):
identifier = parts[1]
endpoint = 'https://archive.org/metadata/' + urllib.parse.quote(identifier, safe='')
connection, response, _ = connect(endpoint, allow_private=allow_private)
try:
raw = response.read(8*1024*1024+1)
if len(raw) > 8*1024*1024:
raise ValueError('Archive file list exceeds the supported size; use a direct file URL')
metadata = json.loads(raw)
finally:
connection.close()
files = []
for entry in metadata.get('files', []):
name = entry.get('name', '')
if entry.get('source') != 'original' or name.endswith(('_files.xml','_meta.xml','_meta.sqlite','.torrent')):
continue
size = int(entry['size']) if entry.get('size') else None
files.append({'name':name,'size':size,'url':'https://archive.org/download/'+urllib.parse.quote(identifier,safe='')+'/'+urllib.parse.quote(name,safe='/')})
if not files:
raise ValueError('No downloadable original files were found in this archive item')
return {'kind':'collection','title':metadata.get('metadata',{}).get('title',identifier),'files':files}
try:
connection, response, resolved = connect(url, method='HEAD', allow_private=allow_private)
except ValueError as error:
if not str(error).startswith(('Source returned HTTP 405 ','Source returned HTTP 501 ')):raise
# Some file hosts reject HEAD. Read GET headers only, then close the body.
connection, response, resolved = connect(url, allow_private=allow_private)
try:
if 'text/html' in response.getheader('Content-Type','').lower():
raise ValueError('This URL is a web page. Use a direct file URL or an Internet Archive item link')
return {'kind':'file','files':[{'name':filename(resolved,response),'url':url,'size':int(response.getheader('Content-Length')) if response.getheader('Content-Length') else None}]}
finally:
connection.close()
def download(request):
connection, response, resolved = connect(request['url'], allow_private=request.get('allow_private',False))
try:
if 'text/html' in response.getheader('Content-Type','').lower():
raise ValueError('The download returned a web page, not a file')
total = int(response.getheader('Content-Length')) if response.getheader('Content-Length') else None
limit = int(request['limit'])
logical_limit=int(request.get('logical_limit',limit))
if total is not None and total > logical_limit:
raise ValueError('Download exceeds the supported logical file size')
name = filename(resolved, response)
destination = pathlib.Path(request['directory']) / name
received = allocated = 0
progress(request.get('progress_file'),stage='Downloading',completed=0,total=total,unit='bytes',detail=name)
digest = hashlib.sha256()
started = last = time.monotonic()
with destination.open('xb') as output:
while data := response.read(1024*1024):
received += len(data)
if received > logical_limit:
raise ValueError('Download exceeded the supported logical file size')
if data.count(0)==len(data):output.seek(len(data),1)
else:
allocated+=len(data)
if allocated>limit:raise ValueError('Downloaded data exceeded the RAM import allowance; choose a smaller or compressed source')
output.write(data)
digest.update(data)
now = time.monotonic()
if now-last >= .25:
progress(request.get('progress_file'),stage='Downloading',completed=received,total=total,unit='bytes',bytes_per_second=received/max(.001,now-started),detail=name)
last=now
output.truncate(received)
if total is not None and received != total:
raise ValueError('Download ended before the advertised file size was received')
if not received:
raise ValueError('Source returned an empty file')
progress(request.get('progress_file'),stage='Download complete',completed=received,total=received,unit='bytes',detail=name)
return {'path':str(destination),'name':name,'bytes':received,'sha256':digest.hexdigest(),'url':request['url']}
finally:
connection.close()
if __name__ == '__main__':
request=json.loads(pathlib.Path(sys.argv[1]).read_text())
try:
result=inspect(request['url'], request.get('allow_private',False)) if request['mode']=='inspect' else download(request)
pathlib.Path(request['result_file']).write_text(json.dumps(result))
except Exception as error:
pathlib.Path(request['result_file']).write_text(json.dumps({'error':str(error)}))
sys.exit(1)
plugins/luks_extract.py
+5 -1
View File
@@ -11,6 +11,7 @@ import subprocess
import sys
import tempfile
import zipfile
from progress import report
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
@@ -64,8 +65,11 @@ def decrypt(image, output, key, metadata):
tweak = (iv + (position + start) // 512).to_bytes(16, 'little')
context = Cipher(algorithms.AES(key), modes.XTS(tweak)).decryptor()
plain.extend(context.update(block[start:start + sector]) + context.finalize())
dest.write(plain)
if plain.count(0)==len(plain):dest.seek(len(plain),1)
else:dest.write(plain)
position += len(block)
report('Decrypting filesystem',position,length,'bytes',image.name)
dest.truncate(position)
with output.open('rb') as stream:
stream.seek(1024 + 56)
if stream.read(2) != b'\x53\xef':
plugins/media_extract.py
+34 -13
View File
@@ -5,6 +5,9 @@ Original containers stay in the parent snapshot; exact entry bytes and ranges ac
all derivatives. Unsupported layouts are recorded, never interpreted as empty media.
"""
import hashlib, json, mmap, pathlib, shutil, struct, sys, wave, zlib
from collections import deque
from concurrent.futures import ThreadPoolExecutor
from parallelism import worker_count
REVISION = 4
KNOWN_PCM_RATES = {
@@ -142,12 +145,13 @@ def extract_pck(data,out,prefix,source):
def png(width,height,pixels):
def chunk(tag,payload):return struct.pack('>I',len(payload))+tag+payload+struct.pack('>I',zlib.crc32(tag+payload)&0xffffffff)
raw=bytearray()
gray=pixels.translate(bytes(v*17 if v<=15 else v for v in range(256)))
alpha=pixels.translate(bytes(0 if v==255 else 255 for v in range(256)))
stride=width*4+1;raw=bytearray(stride*height)
for y in range(height):
raw.append(0)
for v in pixels[y*width:(y+1)*width]:
gray=v*17 if v<=15 else v
raw.extend((gray,gray,gray,0 if v==255 else 255))
row=bytearray(width*4);values=gray[y*width:(y+1)*width]
row[0::4]=values;row[1::4]=values;row[2::4]=values;row[3::4]=alpha[y*width:(y+1)*width]
raw[y*stride+1:(y+1)*stride]=row
return b'\x89PNG\r\n\x1a\n'+chunk(b'IHDR',struct.pack('>IIBBBBB',width,height,8,6,0,0,0))+chunk(b'IDAT',zlib.compress(raw))+chunk(b'IEND',b'')
def radium_header(data):
@@ -185,7 +189,7 @@ def extract_radium(data,out,prefix,source,source_hash,settings):
preview=out.write(f'{prefix}/sounds/sound-{i:05}.wav',wav.getvalue());status='decoded'
out.record(source=source,section=8,index=i,offset=start,length=length,channels=channels,sample_rate=rate,original=original,preview=preview,status=status)
def extract_bitmaps(data,out,prefix,source):
def extract_bitmaps(data,out,prefix,source,request=None):
from dmd_bitmap import decode
bounds(data,0,56);header=struct.unpack_from('<7Q',data)
start,end=header[3],header[2]
@@ -195,16 +199,29 @@ def extract_bitmaps(data,out,prefix,source):
pointers=struct.unpack_from(f'<{count}Q',data,start)
if any(p<end or p+13>len(data) or p%8 for p in pointers):raise ValueError('DMD pointer outside asset area')
ordered=sorted(set(pointers));nexts=dict(zip(ordered,ordered[1:]+[len(data)]));frames={}
for i,offset in enumerate(pointers):
workers=worker_count(request or {'cpu_threads':1},max(1,count//16));pending=deque()
def save_next():
i,offset,decoded,future=pending.popleft()
try:
decoded=decode(memoryview(data)[offset:nexts[offset]],frames)
original=out.write(f'{prefix}/original/bitmap-{i:05}.bin',data[offset:offset+decoded['consumed']])
indices=out.write(f'{prefix}/indices/bitmap-{i:05}.idx',decoded['pixels'])
preview=out.write(f'{prefix}/images/bitmap-{i:05}.png',png(decoded['width'],decoded['height'],decoded['pixels']))
preview=out.write(f'{prefix}/images/bitmap-{i:05}.png',future.result())
metadata={k:v for k,v in decoded.items() if k not in ('pixels','consumed')}
out.record(source=source,section=3,index=i,offset=offset,length=decoded['consumed'],original=original,indices=indices,preview=preview,status='decoded',profile='dmd-indexed/1',preview_accuracy='approximate_palette',**metadata)
except (ValueError,struct.error) as error:
out.record(source=source,section=3,index=i,offset=offset,status='coverage_failure',error=str(error))
except (ValueError,struct.error) as error:out.record(source=source,section=3,index=i,offset=offset,status='coverage_failure',error=str(error))
report('Decoding DMD bitmaps',i+1,count,'frames',f'{workers} image workers · {source}')
with ThreadPoolExecutor(max_workers=workers) as pool:
for i,offset in enumerate(pointers):
try:
# Delta frames depend on the preceding keyframe. Keep index decoding
# ordered; compress the independent PNG previews concurrently.
decoded=decode(memoryview(data)[offset:nexts[offset]],frames)
pending.append((i,offset,decoded,pool.submit(png,decoded['width'],decoded['height'],decoded['pixels'])))
if len(pending)>=workers:save_next()
except (ValueError,struct.error) as error:
while pending:save_next()
out.record(source=source,section=3,index=i,offset=offset,status='coverage_failure',error=str(error))
while pending:save_next()
def extract_spike1(data,out,prefix,source):
from vendor.spike1_parser import parse_master
@@ -224,11 +241,15 @@ def extract_spike1(data,out,prefix,source):
source_header_offset=offset,channels=channels,sample_rate=44100//divisor,original=original,preview=preview,status='decoded',profile='spike1-plaintext-pcm')
out.record(source=source,status='partial_container',profile='spike1-plaintext-pcm',reason='Audio master records decoded; non-audio sections remain unverified')
from progress import report
def run(request):
if request['protocol']!=1: raise ValueError('unsupported protocol')
source=pathlib.Path(request['input_dir']);output=pathlib.Path(request['output_dir']);settings=request.get('settings') or {}
out=Output(output,request['workspace_bytes']//2)
for path in sorted(source.rglob('*')):
paths=[p for p in sorted(source.rglob('*')) if p.is_file() and not p.is_symlink()]
for index,path in enumerate(paths):
report('Decoding assets',index,len(paths),'files',path.name)
if not path.is_file() or path.is_symlink(): continue
rel=path.relative_to(source).as_posix()
with path.open('rb') as f: head=f.read(104)
@@ -244,7 +265,7 @@ def run(request):
original=out.write(f'{prefix}/original{ext}',data)
out.record(source=rel,source_sha256=source_hash,offset=0,length=len(data),original=original,preview=original,status='decoded',name_confidence='confirmed_container_path')
elif path.name.startswith('image') and path.suffix=='.bin':
try:extract_bitmaps(data,out,prefix,rel)
try:extract_bitmaps(data,out,prefix,rel,request)
except (ValueError,struct.error) as error:out.record(source=rel,section=3,status='coverage_failure',error=str(error))
from vendor.spike1_parser import find_header2, Spike1Error
try:find_header2(data);spike1=True
plugins/media_preview.py
+42 -18
View File
@@ -1,10 +1,13 @@
#!/usr/bin/env python3
"""FFmpeg/ffprobe previews with retained source identities and visible codec failures."""
import hashlib,json,pathlib,subprocess,sys
import hashlib,io,itertools,json,pathlib,subprocess,sys
from collections import deque
from concurrent.futures import ThreadPoolExecutor
from parallelism import worker_count
from media_extract import sniff
REVISION=3
def image_preview(path,dest,remaining):
def image_preview(path):
from PIL import Image,ImageOps
with Image.open(path) as source:
if source.width*source.height>16777216:raise ValueError('media dimensions exceed preview limit')
@@ -12,28 +15,47 @@ def image_preview(path,dest,remaining):
source.load()
thumb=ImageOps.exif_transpose(source).convert('RGBA')
thumb.thumbnail((320,240))
thumb.save(dest,format='WEBP',quality=80)
if dest.stat().st_size>=remaining:raise ValueError('preview exceeded byte budget')
with Image.open(dest) as readback:
encoded=io.BytesIO();thumb.save(encoded,format='WEBP',quality=80)
encoded.seek(0)
with Image.open(encoded) as readback:
readback.load()
if readback.size!=thumb.size:raise ValueError('thumbnail readback dimensions differ')
metadata={'width':readback.width,'height':readback.height,'format':readback.format}
return info,metadata
return info,metadata,encoded.getvalue()
def probe(tool,path):
p=subprocess.run([tool,'-v','error','-protocol_whitelist','file,pipe','-show_streams','-show_format','-of','json',str(path)],capture_output=True,timeout=30,check=True)
if len(p.stdout)>1024*1024:raise ValueError('media metadata exceeds bound')
return json.loads(p.stdout)
from progress import report
def run(request):
source=pathlib.Path(request['input_dir']);output=pathlib.Path(request['output_dir']);settings=request.get('settings') or {}
ffmpeg=settings.get('ffmpeg','ffmpeg');ffprobe=settings.get('ffprobe','ffprobe');budget=request['workspace_bytes']//2;used=0;records=[];verified={}
for path in sorted(source.rglob('*')):
if not path.is_file() or path.is_symlink():continue
with path.open('rb') as f:ext=sniff(f.read(32))
if ext is None:continue
included=settings.get('include_extensions')
if included is not None and ext not in included:continue
paths=[p for p in sorted(source.rglob('*')) if p.is_file() and not p.is_symlink()]
def candidates():
for path in paths:
with path.open('rb') as f:ext=sniff(f.read(32))
included=settings.get('include_extensions')
if ext is not None and (included is None or ext in included):yield path,ext
workers=worker_count(request,len(paths),512*1024**2)
# Image codecs release the GIL. Keep only one worker's decoded image per
# future, then commit in source order under the shared output byte budget.
# Drain each image group before starting FFmpeg so their CPU budgets do not overlap.
def prepared():
with ThreadPoolExecutor(max_workers=workers) as pool:
for images,group in itertools.groupby(candidates(),key=lambda item:item[1] in ('.png','.jpg','.webp')):
pending=deque()
for path,ext in group:
if images:
pending.append((path,ext,pool.submit(image_preview,path)))
if len(pending)>=workers:yield pending.popleft()
else:yield path,ext,None
while pending:yield pending.popleft()
threads=str(min(8,worker_count(request,8,128*1024**2)))
for index,(path,ext,future) in enumerate(prepared()):
report('Preparing previews',index,len(paths),'files',f'{workers} image workers · {path.name}')
rel=path.relative_to(source).as_posix()
with path.open('rb') as f:digest=hashlib.file_digest(f,'sha256').hexdigest()
row={'source':rel,'source_sha256':digest,'processing_revision':REVISION};dest=None
@@ -42,8 +64,9 @@ def run(request):
row.update(verified[digest]);records.append(row);continue
if ext in ('.png','.jpg','.webp'):
dest=output/'previews'/(rel+'.webp');dest.parent.mkdir(parents=True,exist_ok=True)
info,preview=image_preview(path,dest,budget-used)
used+=dest.stat().st_size
info,preview,encoded=future.result()
if len(encoded)>=budget-used:raise ValueError('preview exceeded byte budget')
dest.write_bytes(encoded);used+=len(encoded)
row.update(status='verified',inspection=info,preview=dest.relative_to(output).as_posix(),preview_metadata=preview)
records.append(row);continue
info=probe(ffprobe,path);row['inspection']=info
@@ -51,21 +74,21 @@ def run(request):
audio=next((s for s in streams if s['codec_type']=='audio'),None)
if video and video.get('width',0)*video.get('height',0)>16777216:raise ValueError('media dimensions exceed preview limit')
if ext=='.mp4' and video and video.get('codec_name')=='h264' and video.get('pix_fmt') in ('yuv420p','yuvj420p') and (audio is None or audio.get('codec_name')=='aac'):
subprocess.run([ffmpeg,'-nostdin','-v','error','-xerror','-threads','2','-protocol_whitelist','file,pipe','-i',str(path),'-f','null','-'],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,check=True,timeout=300)
subprocess.run([ffmpeg,'-nostdin','-v','error','-xerror','-threads',threads,'-filter_threads','1','-protocol_whitelist','file,pipe','-i',str(path),'-f','null','-'],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,check=True,timeout=300)
row.update(status='verified_source',reason='Browser-compatible H.264/AAC source passed complete decode; conversion is unnecessary')
verified[digest]={k:v for k,v in row.items() if k not in ('source','source_sha256','processing_revision')}
records.append(row);continue
base=output/'previews'/rel;base.parent.mkdir(parents=True,exist_ok=True)
image=ext in ('.png','.jpg','.webp')
dest=base.with_name(base.name+('.webp' if image else '.mp4' if video else '.wav'))
args=[ffmpeg,'-nostdin','-v','error','-max_alloc','268435456','-threads','2','-protocol_whitelist','file,pipe','-i',str(path)]
args=[ffmpeg,'-nostdin','-v','error','-max_alloc','268435456','-threads',threads,'-filter_threads','1','-protocol_whitelist','file,pipe','-i',str(path)]
if image:args+=['-frames:v','1','-vf','scale=320:240:force_original_aspect_ratio=decrease','-c:v','libwebp']
elif video:args+=['-map','0:v:0','-map','0:a:0?','-vf','scale=1280:720:force_original_aspect_ratio=decrease:force_divisible_by=2','-c:v','libx264','-preset','fast','-pix_fmt','yuv420p','-c:a','aac','-movflags','+faststart']
elif audio:args+=['-map','0:a:0','-c:a','pcm_s16le']
else:raise ValueError('no supported audio/video stream')
remaining=budget-used
if remaining<1024:raise ValueError('preview budget exhausted')
args+=['-threads','2','-fs',str(remaining),str(dest)]
args+=['-threads',threads,'-fs',str(remaining),str(dest)]
subprocess.run(args,stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,check=True,timeout=300)
size=dest.stat().st_size;used+=size
if size>=remaining:raise ValueError('preview exceeded byte budget')
@@ -73,13 +96,14 @@ def run(request):
if not image:
duration=float(info.get('format',{}).get('duration',0));converted=float(preview.get('format',{}).get('duration',0))
if duration and abs(converted-duration)>max(.25,duration*.01):raise ValueError('preview duration differs from source')
subprocess.run([ffmpeg,'-nostdin','-v','error','-xerror','-threads','2','-protocol_whitelist','file,pipe','-i',str(dest),'-f','null','-'],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,check=True,timeout=300)
subprocess.run([ffmpeg,'-nostdin','-v','error','-xerror','-threads',threads,'-filter_threads','1','-protocol_whitelist','file,pipe','-i',str(dest),'-f','null','-'],stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,check=True,timeout=300)
row.update(status='verified',preview=dest.relative_to(output).as_posix(),preview_metadata=preview)
verified[digest]={k:v for k,v in row.items() if k not in ('source','source_sha256','processing_revision')}
except (ValueError,OSError,subprocess.SubprocessError) as error:
if dest is not None:dest.unlink(missing_ok=True)
row.update(status='failed',error=str(error))
records.append(row)
report('Preparing previews',len(paths),len(paths),'files','Previews checked',force=True)
failed=any(r['status']=='failed' for r in records)
(output/'preview-evidence.json').write_text(json.dumps({'schema':1,'previews':records},indent=2))
return {'protocol':1,'layer':'derived','coverage':'partial' if failed else 'complete','files':[p.relative_to(output).as_posix() for p in sorted(output.rglob('*')) if p.is_file()],'warnings':['Some media previews failed; original assets remain retained.'] if failed else []}
plugins/parallelism.py
+47
View File
@@ -0,0 +1,47 @@
"""CPU and RAM bounds for independent work inside one extraction job."""
import math
import os
import pathlib
MIB=1024**2
def cgroup_dirs():
root=pathlib.Path('/sys/fs/cgroup')
try:
group=next(line.split(':',2)[2] for line in pathlib.Path('/proc/self/cgroup').read_text().splitlines() if line.startswith('0::'))
path=(root/group.lstrip('/')).resolve()
while path.is_relative_to(root) and path!=root:
yield path
path=path.parent
except (OSError,ValueError,StopIteration):pass
yield root
def available_cpus():
count=len(os.sched_getaffinity(0)) if hasattr(os,'sched_getaffinity') else os.cpu_count() or 1
# Respect cgroup v2 quotas as well as the process affinity mask.
for path in cgroup_dirs():
try:
quota,period=(path/'cpu.max').read_text().split()
if quota!='max':count=min(count,max(1,math.ceil(int(quota)/int(period))))
except (OSError,ValueError,ZeroDivisionError):pass
return max(1,count)
def cpu_threads(request):
available=available_cpus()
requested=request.get('cpu_threads',0)
if not isinstance(requested,int) or requested<0:raise ValueError('cpu_threads must be a nonnegative integer')
# Leave two logical CPUs for the workbench and archive service on larger hosts.
return min(available,requested) if requested else max(1,available-2)
def worker_count(request,jobs,per_worker_bytes=256*MIB):
memory=int(request.get('workspace_bytes',per_worker_bytes*4))//4
try:
available=next(int(line.split()[1])*1024 for line in pathlib.Path('/proc/meminfo').read_text().splitlines() if line.startswith('MemAvailable:'))
memory=min(memory,available//4)
except (OSError,ValueError,StopIteration):pass
for path in cgroup_dirs():
try:
limit=(path/'memory.max').read_text().strip()
if limit!='max':memory=min(memory,max(0,int(limit)-int((path/'memory.current').read_text()))//4)
except (OSError,ValueError):pass
return max(1,min(cpu_threads(request),max(1,jobs),max(1,memory//per_worker_bytes)))
plugins/progress.py
+17
View File
@@ -0,0 +1,17 @@
"""Optional measured progress, atomically published to the run's RAM workspace."""
import json
import os
import pathlib
import time
_last = 0.0
def report(stage, completed=None, total=None, unit='items', detail='', force=False):
global _last
path=os.environ.get('VERSTACK_PROGRESS_FILE')
now=time.monotonic()
if not path or (not force and now-_last<.2):return
_last=now
target=pathlib.Path(path)
temporary=target.with_suffix('.next')
temporary.write_text(json.dumps({'stage':stage,'completed':completed,'total':total,'unit':unit,'detail':detail}))
temporary.replace(target)
plugins/spike_extract.py
+7 -1
View File
@@ -7,6 +7,8 @@ import pathlib
import re
import subprocess
import sys
from parallelism import worker_count
from progress import report
def run(request):
if request["protocol"] != 1:
@@ -20,7 +22,10 @@ def run(request):
source = pathlib.Path(request["input_dir"])
output = pathlib.Path(request["output_dir"])
env = os.environ.copy()
env["RAYON_NUM_THREADS"] = str(settings.get("max_threads", 2))
limit=settings.get('max_threads',0)
if not isinstance(limit,int) or limit<0:raise ValueError('max_threads must be a nonnegative integer (0 selects automatic sizing)')
threads=worker_count(request,limit or 1_000_000)
env["RAYON_NUM_THREADS"] = str(threads)
candidates = [p for p in sorted(source.rglob("*")) if p.is_file() and not p.is_symlink()
and (p.name.lower().endswith(".spk") or re.search(r"\.spk\.\d{3}\.000$", p.name, re.I))]
if not candidates:
@@ -31,6 +36,7 @@ def run(request):
# or architecture from extension, package name, or release metadata.
destination = output / f"package-{index:04d}"
destination.mkdir()
report('Extracting SPK package',None,None,'files',f'{path.name} · up to {threads} threads',force=True)
# Extraction verifies all checksums itself; avoid reading every payload twice.
subprocess.run([str(tool), "extract", str(path), "--output", str(destination)], check=True, env=env)
packages.append({"source": path.relative_to(source).as_posix(), "generation": "unknown", "tool_sha256": tool_hash})
scripts/configure_pipeline.py
+2 -1
View File
@@ -3,7 +3,7 @@
import argparse,hashlib,json,pathlib
def configure(config,root):
plugins=config['plugins'];base=plugins['spike3-extract']['settings'];keys=plugins['spike3-unpack']['settings']
plugins=config['plugins'];base={**plugins['spike3-extract']['settings'],'max_threads':0};keys=plugins['spike3-unpack']['settings']
def plugin(script,version,settings,extensions,timeout=7200,python='python3'):
path=root/'plugins'/script
digest=hashlib.sha256()
@@ -12,6 +12,7 @@ def configure(config,root):
digest.update(source.relative_to(root/'plugins').as_posix().encode()+b'\0');digest.update(hashlib.sha256(source.read_bytes()).digest())
return {'command':[python,str(path)],'version':version,'processing_revision':int(version.rsplit('/',1)[1]),'source_root':str(root/'plugins'),'source_digest':'sha256:'+digest.hexdigest(),'output_schema':1,'settings':settings,'timeout_seconds':timeout,'auto_extensions':extensions}
plugins['spike-package']=plugin('spike_package.py','spike-package/2',{**base,**{k:v for k,v in keys.items() if k in ('key_file','cryptsetup','debugfs')}},['.spk','.spk.zip'])
plugins['import-extract']=plugin('import_extract.py','import-extract/1',{**base,**{k:v for k,v in keys.items() if k in ('key_file','cryptsetup','debugfs')}},[],timeout=14400,python=str(root/'tools/decoder-env/bin/python'))
plugins['media-extract']=plugin('media_extract.py','media-extract/4',{},['.bin','.pck','.asset','.radium'],python=str(root/'tools/decoder-env/bin/python'))
plugins['media-preview']=plugin('media_preview.py','media-preview/3',{},['.png','.jpg','.webp','.wav','.mp4','.ogg','.flac','.webm'],python=str(root/'tools/decoder-env/bin/python'))
node=root/'tools/src/spike-ihex/target/release/spike-ihex'
scripts/ram_workspace.py
+31
View File
@@ -0,0 +1,31 @@
"""RAM-only scratch policy shared by standalone validation tools."""
import os
import pathlib
import re
import shutil
def ram_workspace(config):
path = pathlib.Path(config.get('workspace', '/tmp/verstack-workspace')).resolve()
path.mkdir(parents=True, exist_ok=True)
mounts = []
for line in pathlib.Path('/proc/self/mountinfo').read_text().splitlines():
fields, filesystem = line.split(' - ', 1)
point = pathlib.Path(re.sub(r'\\([0-7]{3})', lambda m: chr(int(m[1], 8)), fields.split()[4]))
if path.is_relative_to(point):
mounts.append((len(point.parts), filesystem.split()[0]))
if not mounts or max(mounts)[1] not in ('tmpfs', 'ramfs'):
raise ValueError('validation workspace must be on tmpfs/ramfs; no disk fallback')
budget = int(config.get('workspace_bytes', 8 * 1024**3))
if budget <= 0 or budget > shutil.disk_usage(path).free:
raise ValueError('insufficient free RAM workspace for configured workspace_bytes')
return path, budget
def scratch_environment(root):
root = pathlib.Path(root).resolve()
scratch = root / 'tmp'
scratch.mkdir(exist_ok=True)
return {**os.environ, 'TMPDIR': str(scratch), 'TMP': str(scratch),
'TEMP': str(scratch), 'XDG_CACHE_HOME': str(root / 'cache'),
'PYTHONDONTWRITEBYTECODE': '1'}
scripts/validate_corpus.py
+7 -4
View File
@@ -4,12 +4,13 @@ import argparse,hashlib,json,pathlib,shutil,subprocess,sys,tempfile,time,os
sys.path.insert(0,str(pathlib.Path(__file__).resolve().parents[1]/'plugins'))
import spike_package,media_extract
from inventory_corpus import atomic
from ram_workspace import ram_workspace, scratch_environment
def invoke(script,request,path):
path.write_text(json.dumps(request))
# Each decoder gets a fresh process and a hard wall timeout. Core archive
# execution adds its own process-group cleanup and workspace monitoring.
child=subprocess.Popen([sys.executable,str(pathlib.Path(__file__).resolve().parents[1]/'plugins'/script),str(path)],start_new_session=True)
child=subprocess.Popen([sys.executable,str(pathlib.Path(__file__).resolve().parents[1]/'plugins'/script),str(path)],start_new_session=True,env=scratch_environment(path.parent),cwd=path.parent)
try:
if child.wait(timeout=7200):raise ValueError(f'{script} failed; inspect validation log')
except BaseException:
@@ -65,6 +66,8 @@ def validate(source,report,config,refresh_stale=False,all_families=False):
settings={**config['plugins']['spike3-extract']['settings'],**{k:v for k,v in config['plugins']['spike3-unpack']['settings'].items() if k in ('key_file','cryptsetup','debugfs')}}
packages=sorted(p for p in source.iterdir() if p.name.lower().endswith(('.spk','.spk.zip')))
report.mkdir(parents=True,exist_ok=True)
scratch,budget=ram_workspace(config)
print(f'RAM workspace: {scratch}; allowance {budget//1024**3} GiB; only receipts are persisted',flush=True)
for i,path in enumerate(packages):
receipt=report/(path.name+'.json')
processing={'package':2,'media':media_extract.REVISION}
@@ -75,11 +78,11 @@ def validate(source,report,config,refresh_stale=False,all_families=False):
row={'schema':2,'processing':processing,'package':path.name,'bytes':path.stat().st_size,'scope':'disposable validation; not archived','started':time.time()}
print(f'BEGIN {i+1}/{len(packages)} {path.name}',flush=True)
try:
if shutil.disk_usage(report).free<path.stat().st_size*7+10*1024**3:raise ValueError('insufficient disk headroom for bounded validation')
with tempfile.TemporaryDirectory(prefix='corpus-',dir=report) as tmp:
if min(shutil.disk_usage(scratch).free,budget)<path.stat().st_size*7+1024**3:raise ValueError('insufficient RAM headroom for bounded validation')
with tempfile.TemporaryDirectory(prefix='corpus-',dir=scratch) as tmp:
root=pathlib.Path(tmp);inputs=root/'input';output=root/'output';media=root/'media';inputs.mkdir();output.mkdir();media.mkdir()
shutil.copyfile(path,inputs/path.name)
request={'protocol':1,'input_dir':str(inputs),'output_dir':str(output),'result_file':str(root/'result.json'),'settings':settings,'workspace_bytes':96*1024**3}
request={'protocol':1,'input_dir':str(inputs),'output_dir':str(output),'result_file':str(root/'result.json'),'settings':settings,'workspace_bytes':budget}
result=invoke('spike_package.py',request,root/'extract-request.json');row['extraction']=result
entries=[]
for p in sorted(output.rglob('*')):
src/analysis.rs
+36
View File
@@ -10,6 +10,42 @@ use std::{
pub const COMPARISON_VERSION: &str = "exact-functions/3";
pub fn is_game_program(path: &str) -> bool {
std::path::Path::new(path)
.file_stem()
.and_then(|s| s.to_str())
.is_some_and(|s| s.eq_ignore_ascii_case("game"))
}
impl Archive {
pub fn default_analysis_paths(&self, id: &str) -> Result<Vec<String>> {
let snapshot = self.snapshot(id)?;
let mut paths = self.executable_paths(id)?;
paths.sort_by_key(|path| {
(
std::cmp::Reverse(
snapshot
.entries
.iter()
.find(|e| &e.path == path)
.map_or(0, |e| e.size),
),
path.clone(),
)
});
let games: Vec<_> = paths
.iter()
.filter(|p| is_game_program(p))
.cloned()
.collect();
Ok(if games.is_empty() {
paths.into_iter().take(1).collect()
} else {
games
})
}
}
#[derive(Clone, Deserialize, Serialize)]
pub struct Function {
pub address: String,
src/archive.rs
+102 -24
View File
@@ -7,7 +7,7 @@ use fs2::FileExt;
use std::{
collections::BTreeMap,
fs::{self, File, OpenOptions},
io::{BufReader, Read, Write},
io::{BufReader, Read, Seek, SeekFrom, Write},
os::unix::fs::{MetadataExt, PermissionsExt},
path::{Component, Path},
sync::Mutex,
@@ -21,6 +21,11 @@ pub struct Archive {
pub(crate) store: RusticStore,
pub(crate) writer: Mutex<()>,
pub(crate) stopping: std::sync::atomic::AtomicBool,
pub(crate) activity: Mutex<crate::imports::LiveActivity>,
pub(crate) workspace_cache:
Mutex<BTreeMap<String, (String, std::sync::Arc<crate::workspace::WorkspaceIndex>)>>,
pub(crate) storage_usage_cache:
Mutex<Option<(String, std::sync::Arc<crate::storage::StorageUsage>)>>,
_lock: InstanceLock,
}
// Explicitly unlock after the archive's other fields are dropped. Another thread
@@ -50,6 +55,7 @@ impl Archive {
fs::create_dir_all(&config.workspace)?;
config.archive = config.archive.canonicalize()?;
config.workspace = config.workspace.canonicalize()?;
crate::workspace::check_scratch(&config)?;
ensure!(
!config.workspace.starts_with(&config.archive),
"workspace must be outside archive"
@@ -92,8 +98,12 @@ impl Archive {
store,
writer: Mutex::new(()),
stopping: std::sync::atomic::AtomicBool::new(false),
activity: Mutex::new(crate::imports::LiveActivity::default()),
workspace_cache: Mutex::new(BTreeMap::new()),
storage_usage_cache: Mutex::new(None),
_lock: InstanceLock(lock),
};
archive.recover_deletions()?;
for mut run in archive.runs()? {
if matches!(run.state, RunState::Running) {
// Publication is the commit point, even if the final run update was interrupted.
@@ -113,6 +123,7 @@ impl Archive {
.catalog
.reconcile(&archive.snapshots()?, &archive.runs()?)?;
archive.recover_jobs()?;
archive.recover_imports()?;
for snapshot in archive.snapshots()? {
archive.index_code(&snapshot)?;
if archive.output_is_current(&snapshot)? {
@@ -162,6 +173,7 @@ impl Archive {
.to_ascii_lowercase();
matches!(extension.as_str(), "exe" | "elf" | "so")
|| (extension.is_empty() && e.size >= 1024)
|| crate::analysis::is_game_program(&e.path)
})
.map(|e| e.path.clone())
.collect()
@@ -195,6 +207,7 @@ impl Archive {
Ok(s)
}
pub(crate) fn save_run(&self, run: &Run) -> Result<()> {
self.track_run(run, false)?;
atomic_json(
&self
.config
@@ -229,6 +242,7 @@ impl Archive {
error: None,
};
self.save_run(&run)?;
self.track_run(&run, true)?;
Ok(run)
}
pub(crate) fn finish_run(&self, run: &mut Run, result: &Result<Snapshot>) -> Result<()> {
@@ -276,6 +290,13 @@ impl Archive {
self.import_managed(&source, release)
}
pub(crate) fn import_managed(&self, source: &Path, release: Release) -> Result<Snapshot> {
self.import_managed_sources(&[source.to_owned()], release)
}
pub(crate) fn import_managed_sources(
&self,
sources: &[std::path::PathBuf],
release: Release,
) -> Result<Snapshot> {
let _guard = self
.writer
.lock()
@@ -284,12 +305,19 @@ impl Archive {
let mut run = self.start_run(
"import",
env!("CARGO_PKG_VERSION"),
serde_json::json!({"source": source}),
serde_json::json!({"sources": sources}),
vec![],
)?;
let result = (|| {
let stage = tempfile::tempdir_in(&self.config.workspace)?;
let entries = self.stage(source, stage.path(), &mut run)?;
let mut entries = Vec::new();
for source in sources {
entries.extend(self.stage(source, stage.path(), &mut run)?);
ensure!(
crate::plugins::workspace_size(stage.path())? <= self.config.workspace_bytes,
"Selected sources exceed the RAM import allowance"
);
}
self.publish(
stage.path(),
entries,
@@ -316,12 +344,24 @@ impl Archive {
};
let mut entries = Vec::new();
let mut used = 0u64;
let mut allocated = 0u64;
let mut progress_saved = std::time::Instant::now();
let total = WalkDir::new(source)
.follow_links(false)
.into_iter()
.map(|item| {
let item = item?;
Ok(if item.file_type().is_file() {
item.metadata()?.len()
} else {
0
})
})
.collect::<Result<Vec<u64>>>()?
.iter()
.sum::<u64>();
for item in WalkDir::new(source).follow_links(false).sort_by_file_name() {
ensure!(
!self.is_stopping(),
"processing interrupted by server shutdown"
);
self.check_interruption()?;
let item = item?;
if item.path() == base {
continue;
@@ -360,25 +400,40 @@ impl Archive {
std::os::unix::fs::symlink(target, &out)?;
entry.size = 0;
} else if meta.is_file() {
ensure!(
meta.len() <= 128u64 * 1024 * 1024 * 1024,
"File exceeds the 128 GiB logical file limit"
);
let mut input = File::open(item.path())?;
let mut output = File::create(&out)?;
let mut output = OpenOptions::new().write(true).create_new(true).open(&out)?;
let mut hash = blake3::Hasher::new();
let mut buf = vec![0; 1024 * 1024];
let mut size = 0;
loop {
self.check_interruption()?;
let count = input.read(&mut buf)?;
if count == 0 {
break;
}
used += count as u64;
size += count as u64;
ensure!(
used <= self.config.workspace_bytes,
"workspace budget exceeded; import is incomplete"
);
hash.update(&buf[..count]);
output.write_all(&buf[..count])?;
if buf[..count].iter().all(|b| *b == 0) {
output.seek(SeekFrom::Current(count as i64))?;
} else {
allocated += count as u64;
ensure!(
allocated <= self.config.workspace_bytes,
"workspace budget exceeded; import is incomplete"
);
output.write_all(&buf[..count])?;
}
if progress_saved.elapsed() >= std::time::Duration::from_millis(250) {
self.set_progress(&run.id, serde_json::json!({"stage":run.stage,"completed":used,"total":total,"unit":"bytes","detail":entry.path}));
progress_saved = std::time::Instant::now();
}
}
output.set_len(size)?;
let after = input.metadata()?;
ensure!(
size == meta.len()
@@ -392,12 +447,6 @@ impl Archive {
entry.size = size;
entry.artifact = Some(format!("blake3:{}", hash.finalize().to_hex()));
run.bytes_processed = used;
// A frame gallery can contain tens of thousands of tiny files.
// Progress is advisory; persist checkpoints, not two fsyncs per asset.
if progress_saved.elapsed() >= std::time::Duration::from_millis(500) {
self.save_run(run)?;
progress_saved = std::time::Instant::now();
}
} else {
bail!("unsupported special file: {}", entry.path);
}
@@ -490,10 +539,10 @@ impl Archive {
}
/// Offline collection of backend commits without published framework manifests.
pub fn cleanup_orphans(&self) -> Result<serde_json::Value> {
let _writer = self
.writer
.lock()
.map_err(|_| anyhow::anyhow!("writer lock poisoned"))?;
let _writer = self.writer.try_lock().map_err(|_| {
anyhow::anyhow!("Archive is busy; finish active work before garbage collection")
})?;
self.require_maintenance_idle()?;
ensure!(
!self.jobs()?.iter().any(|j| j.state == "running"),
"cannot collect during processing"
@@ -510,6 +559,7 @@ impl Archive {
true,
)?;
let result = self.store.cleanup_orphans(&retained)?;
*self.storage_usage_cache.lock().unwrap() = None;
atomic_json(
&journal,
&serde_json::json!({"schema":1,"state":"completed","result":result,"finished":now()}),
@@ -576,6 +626,17 @@ impl Archive {
Ok(())
}
pub(crate) fn materialize(&self, s: &Snapshot, dest: &Path) -> Result<()> {
self.materialize_progress(s, dest, "materialize")
}
pub(crate) fn materialize_progress(
&self,
s: &Snapshot,
dest: &Path,
progress_key: &str,
) -> Result<()> {
let mut restored = 0u64;
let mut allocated = 0u64;
let mut last_progress = std::time::Instant::now();
// Symlinks are created last so no retained link is traversed during writes.
for e in s.entries.iter().filter(|e| e.kind != EntryKind::Symlink) {
ensure!(
@@ -595,14 +656,31 @@ impl Archive {
let mut count = 0u64;
let mut buf = vec![0; 1024 * 1024];
loop {
self.check_interruption()?;
let n = reader.read(&mut buf)?;
if n == 0 {
break;
}
file.write_all(&buf[..n])?;
if buf[..n].iter().all(|b| *b == 0) {
file.seek(SeekFrom::Current(n as i64))?;
} else {
allocated += n as u64;
ensure!(
!dest.starts_with(&self.config.workspace)
|| allocated <= self.config.workspace_bytes / 2,
"Restored input exceeds its RAM allowance; use a smaller source"
);
file.write_all(&buf[..n])?;
}
hash.update(&buf[..n]);
count += n as u64;
restored += n as u64;
if last_progress.elapsed() >= std::time::Duration::from_millis(250) {
self.set_progress(progress_key,serde_json::json!({"stage":"Reading archived files into RAM","completed":restored,"total":s.logical_bytes,"unit":"bytes","detail":e.path}));
last_progress = std::time::Instant::now();
}
}
file.set_len(count)?;
ensure!(
count == e.size
&& e.artifact.as_deref()
src/catalog.rs
+14 -2
View File
@@ -26,7 +26,7 @@ impl Catalog {
)?;
let version: u32 = db.query_row("PRAGMA user_version", [], |r| r.get(0))?;
ensure!(
version <= 5,
version <= 7,
"catalog schema is newer than this application"
);
let tx = db.transaction()?;
@@ -57,11 +57,22 @@ impl Catalog {
CREATE TABLE IF NOT EXISTS code_sources(snapshot TEXT NOT NULL REFERENCES snapshots(id),path TEXT NOT NULL,PRIMARY KEY(snapshot,path));
CREATE TABLE IF NOT EXISTS symbols(snapshot TEXT NOT NULL,path TEXT NOT NULL,address TEXT NOT NULL,name TEXT NOT NULL,search TEXT NOT NULL,payload TEXT NOT NULL,PRIMARY KEY(snapshot,path,address));
CREATE INDEX IF NOT EXISTS symbol_search ON symbols(snapshot,path,name);
PRAGMA user_version=5;",
CREATE TABLE IF NOT EXISTS imports(id TEXT PRIMARY KEY,state TEXT NOT NULL,created INTEGER NOT NULL,payload TEXT NOT NULL);
CREATE INDEX IF NOT EXISTS import_queue ON imports(state,created);
CREATE TABLE IF NOT EXISTS code_programs(snapshot TEXT NOT NULL,path TEXT NOT NULL,input TEXT NOT NULL,source_path TEXT NOT NULL,language TEXT NOT NULL,metadata TEXT NOT NULL,PRIMARY KEY(snapshot,path));
CREATE INDEX IF NOT EXISTS program_identity ON code_programs(input);
CREATE TABLE IF NOT EXISTS function_signatures(snapshot TEXT NOT NULL,path TEXT NOT NULL,address TEXT NOT NULL,input TEXT NOT NULL,language TEXT NOT NULL,body_hash TEXT NOT NULL,size INTEGER NOT NULL,eligible INTEGER NOT NULL,pattern TEXT NOT NULL,bytes BLOB NOT NULL,PRIMARY KEY(snapshot,path,address));
CREATE INDEX IF NOT EXISTS function_body_identity ON function_signatures(language,body_hash,size,input);
CREATE TABLE IF NOT EXISTS flirt_libraries(id TEXT PRIMARY KEY,name TEXT NOT NULL,language TEXT NOT NULL,format TEXT NOT NULL,patterns INTEGER NOT NULL,created INTEGER NOT NULL,data BLOB NOT NULL);
PRAGMA user_version=7;",
)?;
if version < 5 {
tx.execute("DELETE FROM code_sources WHERE path IN ('media-evidence.json','preview-evidence.json')", [])?;
}
if version < 7 {
// Rebuild signature observations from saved facts; no decompilation jobs.
tx.execute("DELETE FROM code_sources WHERE path='functions.json' OR path LIKE '%/functions.json'", [])?;
}
tx.commit()?;
Ok(Self(Mutex::new(db)))
}
@@ -174,6 +185,7 @@ impl Catalog {
let name = function["name"].as_str().unwrap_or("");
tx.execute("INSERT INTO symbols VALUES(?1,?2,?3,?4,?5,?6) ON CONFLICT(snapshot,path,address) DO NOTHING",params![snapshot,path,address,name,format!("{name} {address}").to_lowercase(),function.to_string()])?;
}
crate::signatures::index_program(&tx, snapshot, path, facts)?;
tx.execute(
"INSERT OR IGNORE INTO code_sources VALUES(?1,?2)",
params![snapshot, path],
src/catalog_management.rs
+422
View File
@@ -0,0 +1,422 @@
//! Release-oriented storage accounting and recoverable removal of catalog references.
use crate::{Archive, EntryKind, Snapshot, storage::atomic_json};
use anyhow::{Result, ensure};
use rusqlite::params;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use std::{
collections::{BTreeMap, BTreeSet},
fs,
sync::Arc,
};
#[derive(Clone, Default, Deserialize, Serialize)]
pub struct DeleteScope {
pub repository: String,
#[serde(default)]
pub version: Option<String>,
#[serde(default)]
pub edition: Option<String>,
#[serde(default)]
pub generation: Option<String>,
}
impl DeleteScope {
fn matches(&self, r: &crate::Release) -> bool {
r.repository == self.repository
&& self.version.as_ref().is_none_or(|v| v == &r.version)
&& self.edition.as_ref().is_none_or(|v| v == &r.edition)
&& self.generation.as_ref().is_none_or(|v| v == &r.generation)
}
}
#[derive(Deserialize, Serialize)]
struct Deletion {
schema: u32,
id: String,
state: String,
scope: DeleteScope,
snapshots: BTreeSet<String>,
runs: BTreeSet<String>,
jobs: BTreeSet<String>,
imports: BTreeSet<String>,
created: u64,
}
fn group_key(r: &crate::Release) -> String {
serde_json::to_string(&(
r.repository.as_str(),
r.version.as_str(),
r.edition.as_str(),
r.generation.as_str(),
))
.unwrap()
}
fn sum_chunks(set: &BTreeSet<String>, usage: &crate::storage::StorageUsage) -> (u64, u64) {
set.iter()
.filter_map(|id| usage.chunks.get(id))
.fold((0, 0), |(a, b), (raw, packed)| (a + raw, b + packed))
}
impl Archive {
pub(crate) fn require_maintenance_idle(&self) -> Result<()> {
let pending:i64=self.catalog.db()?.query_row("SELECT (SELECT count(*) FROM imports WHERE state IN ('queued','running'))+(SELECT count(*) FROM jobs WHERE state IN ('queued','running'))",[],|r|r.get(0))?;
ensure!(
pending == 0,
"Finish or cancel queued/running imports and analyses before catalog maintenance"
);
Ok(())
}
fn measured_usage(&self, snapshots: &[Snapshot]) -> Result<Arc<crate::storage::StorageUsage>> {
let key = snapshots
.iter()
.map(|s| s.id.as_str())
.collect::<Vec<_>>()
.join(":");
let mut cache = self.storage_usage_cache.lock().unwrap();
if let Some((saved, value)) = &*cache
&& saved == &key
{
return Ok(value.clone());
}
let value = Arc::new(self.store.usage(snapshots)?);
*cache = Some((key, value.clone()));
Ok(value)
}
pub fn catalog_overview(&self, measure: bool) -> Result<Value> {
let snapshots = self.snapshots()?;
let mut groups: BTreeMap<String, Vec<&Snapshot>> = BTreeMap::new();
for s in &snapshots {
groups.entry(group_key(&s.release)).or_default().push(s);
}
let usage = if measure {
Some(self.measured_usage(&snapshots)?)
} else {
None
};
let mut group_chunks = BTreeMap::<String, BTreeSet<String>>::new();
let mut owners = BTreeMap::<String, usize>::new();
let mut live = BTreeSet::new();
if let Some(usage) = &usage {
for (key, group) in &groups {
let chunks = group
.iter()
.flat_map(|s| usage.snapshots.get(&s.id).into_iter().flatten().cloned())
.collect::<BTreeSet<_>>();
for chunk in &chunks {
*owners.entry(chunk.clone()).or_default() += 1;
live.insert(chunk.clone());
}
group_chunks.insert(key.clone(), chunks);
}
}
let mut all_files = BTreeMap::new();
let mut rows = Vec::new();
let mut total_logical = 0u64;
let mut total_files = 0usize;
for (key, group) in groups {
let release = &group[0].release;
let mut files = BTreeMap::new();
let mut logical = 0u64;
let mut count = 0usize;
for s in &group {
for e in s.entries.iter().filter(|e| e.kind == EntryKind::File) {
logical += e.size;
count += 1;
if let Some(id) = &e.artifact {
files.insert(id.clone(), e.size);
all_files.insert(id.clone(), e.size);
}
}
}
let unique = files.values().sum::<u64>();
total_logical += logical;
total_files += count;
let mut row = json!({"key":key,"release":release,"roots":group.iter().filter(|s|s.parent.is_none()).map(|s|s.id.clone()).collect::<Vec<_>>(),"files":count,"snapshots":group.len(),"logical_bytes":logical,"unique_file_bytes":unique,"exact_duplicate_bytes":logical.saturating_sub(unique)});
if let Some(usage) = &usage {
let chunks = &group_chunks[&key];
let (raw, packed) = sum_chunks(chunks, usage);
let exclusive = chunks
.iter()
.filter(|c| owners.get(*c) == Some(&1))
.cloned()
.collect();
let exclusive = sum_chunks(&exclusive, usage).1;
row["storage"] = json!({"chunk_bytes":raw,"packed_data_bytes":packed,"shared_data_bytes":packed.saturating_sub(exclusive),"exclusive_data_bytes":exclusive,"chunk_reuse_bytes":unique.saturating_sub(raw),"compression_bytes":raw.saturating_sub(packed)});
}
rows.push(row);
}
let mut totals = json!({"releases":rows.len(),"files":total_files,"logical_bytes":total_logical,"unique_file_bytes":all_files.values().sum::<u64>(),"exact_duplicate_bytes":total_logical.saturating_sub(all_files.values().sum())});
if let Some(usage) = &usage {
let (raw, packed) = sum_chunks(&live, usage);
let unused = usage
.chunks
.keys()
.filter(|id| !live.contains(*id))
.cloned()
.collect();
let retained = snapshots
.iter()
.map(|s| s.backend_snapshot.clone())
.collect::<BTreeSet<_>>();
totals["storage"] = json!({"packed_bytes_on_disk":usage.packed_bytes,"active_packed_data_bytes":packed,"chunk_bytes":raw,"chunk_reuse_bytes":all_files.values().sum::<u64>().saturating_sub(raw),"compression_bytes":raw.saturating_sub(packed),"unreferenced_data_bytes":sum_chunks(&unused,usage).1,"unreferenced_snapshots":usage.backend_snapshots.difference(&retained).count()});
}
Ok(
json!({"items":rows,"totals":totals,"measured":measure,"note":"File totals include originals and saved processing history. Exact duplicate savings count identical file contents. Chunk reuse measures repeated byte ranges, including changed files; visual similarity and function matches are not used to merge storage. Packed data excludes tree metadata and pack overhead. Shared bytes appear in each referencing release and cannot be added across rows."}),
)
}
fn deletion_plan(&self, scope: &DeleteScope) -> Result<Deletion> {
ensure!(
!scope.repository.trim().is_empty(),
"Choose a game or collection"
);
ensure!(
scope.version.is_some() || (scope.edition.is_none() && scope.generation.is_none()),
"Whole-game deletion must include all editions"
);
let snapshots = self.snapshots()?;
let runs = self.runs()?;
let mut selected = snapshots
.iter()
.filter(|s| scope.matches(&s.release))
.map(|s| s.id.clone())
.collect::<BTreeSet<_>>();
ensure!(
!selected.is_empty(),
"No cataloged release matches this selection"
);
loop {
let count = selected.len();
for s in &snapshots {
let comparison = runs
.iter()
.find(|r| r.id == s.run && r.operation == "function-comparison")
.is_some_and(|r| {
["before", "after"].iter().any(|key| {
r.settings[*key]
.as_str()
.is_some_and(|id| selected.contains(id))
})
});
if s.parent.as_ref().is_some_and(|p| selected.contains(p)) || comparison {
selected.insert(s.id.clone());
}
}
if selected.len() == count {
break;
}
}
let jobs = self
.jobs()?
.into_iter()
.filter(|j| {
selected.contains(&j.input)
|| j.output.as_ref().is_some_and(|id| selected.contains(id))
})
.collect::<Vec<_>>();
// Maintenance must include older tasks beyond the recent activity page.
let import_history = {
let db = self.catalog.db()?;
let mut stmt = db.prepare("SELECT payload FROM imports")?;
stmt.query_map([], |r| r.get::<_, String>(0))?
.map(|r| Ok(serde_json::from_str::<crate::imports::ImportTask>(&r?)?))
.collect::<Result<Vec<_>>>()?
};
let imports = import_history
.into_iter()
.filter(|t| {
scope.matches(&t.request.release)
|| t.root.as_ref().is_some_and(|id| selected.contains(id))
|| t.outputs.values().any(|id| selected.contains(id))
})
.collect::<Vec<_>>();
let mut deleted_runs = snapshots
.iter()
.filter(|s| selected.contains(&s.id))
.map(|s| s.run.clone())
.collect::<BTreeSet<_>>();
deleted_runs.extend(jobs.iter().filter_map(|j| j.run.clone()));
deleted_runs.extend(imports.iter().filter_map(|j| j.run.clone()));
Ok(Deletion {
schema: 1,
id: String::new(),
state: "committed".into(),
scope: scope.clone(),
snapshots: selected,
runs: deleted_runs,
jobs: jobs.iter().map(|j| j.id.clone()).collect(),
imports: imports.iter().map(|j| j.id.clone()).collect(),
created: 0,
})
}
fn plan_token(&self, plan: &Deletion) -> Result<String> {
Ok(blake3::hash(&serde_json::to_vec(&(
plan,
self.catalog
.library()?
.iter()
.map(|s| s["id"].clone())
.collect::<Vec<_>>(),
))?)
.to_hex()
.to_string())
}
pub fn preview_delete(&self, scope: &DeleteScope) -> Result<Value> {
self.require_maintenance_idle()?;
let plan = self.deletion_plan(scope)?;
let snapshots = self.snapshots()?;
let usage = self.measured_usage(&snapshots)?;
let selected = plan
.snapshots
.iter()
.flat_map(|s| usage.snapshots.get(s).into_iter().flatten().cloned())
.collect::<BTreeSet<_>>();
let retained = snapshots
.iter()
.filter(|s| !plan.snapshots.contains(&s.id))
.flat_map(|s| usage.snapshots.get(&s.id).into_iter().flatten().cloned())
.collect::<BTreeSet<_>>();
let exclusive = selected.difference(&retained).cloned().collect();
let shared = selected.intersection(&retained).cloned().collect();
let affected = snapshots
.iter()
.filter(|s| plan.snapshots.contains(&s.id))
.collect::<Vec<_>>();
Ok(
json!({"token":self.plan_token(&plan)?,"scope":scope,"snapshots":plan.snapshots.len(),"files":affected.iter().map(|s|s.entries.iter().filter(|e|e.kind==EntryKind::File).count()).sum::<usize>(),"logical_bytes":affected.iter().map(|s|s.logical_bytes).sum::<u64>(),"dependent_comparisons":affected.iter().filter(|s|!scope.matches(&s.release)).count(),"eligible_for_gc_bytes":sum_chunks(&exclusive,&usage).1,"shared_data_preserved_bytes":sum_chunks(&shared,&usage).1,"note":"This removes the selected release, its extracted assets, saved analysis, signature observations, and dependent comparison reports. Original files on the server are unchanged. Shared archive chunks are preserved; run garbage collection separately to reclaim unreferenced data."}),
)
}
pub fn delete_cataloged(&self, scope: &DeleteScope, token: &str) -> Result<Value> {
let _writer = self
.writer
.try_lock()
.map_err(|_| anyhow::anyhow!("Archive is busy; finish active work before deletion"))?;
self.require_maintenance_idle()?;
let mut plan = self.deletion_plan(scope)?;
ensure!(
self.plan_token(&plan)? == token,
"The catalog changed since review. Review the deletion again."
);
plan.id = uuid::Uuid::new_v4().to_string();
plan.created = crate::model::now();
let directory = self.config.archive.join("deletions");
fs::create_dir_all(&directory)?;
fs::File::open(&self.config.archive)?.sync_all()?;
let path = directory.join(format!("{}.json", plan.id));
atomic_json(&path, &plan, false)?;
self.apply_deletion(&plan)?;
plan.state = "completed".into();
atomic_json(&path, &plan, true)?;
Ok(json!({"deleted_snapshots":plan.snapshots.len(),"garbage_collection_required":true}))
}
pub(crate) fn recover_deletions(&self) -> Result<()> {
let directory = self.config.archive.join("deletions");
if !directory.exists() {
return Ok(());
}
for entry in fs::read_dir(directory)? {
let path = entry?.path();
if path.extension().is_none_or(|e| e != "json") {
continue;
}
let mut plan: Deletion = serde_json::from_reader(fs::File::open(&path)?)?;
ensure!(plan.schema == 1, "Unsupported deletion journal");
self.apply_deletion(&plan)?;
if plan.state != "completed" {
plan.state = "completed".into();
atomic_json(&path, &plan, true)?;
}
}
Ok(())
}
fn apply_deletion(&self, plan: &Deletion) -> Result<()> {
for (kind, ids) in [("snapshots", &plan.snapshots), ("runs", &plan.runs)] {
for id in ids {
uuid::Uuid::parse_str(id)?;
let path = self.config.archive.join(kind).join(format!("{id}.json"));
match fs::remove_file(path) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => return Err(e.into()),
}
}
fs::File::open(self.config.archive.join(kind))?.sync_all()?;
}
let mut db = self.catalog.db()?;
let tx = db.transaction()?;
for id in &plan.snapshots {
tx.execute("DELETE FROM imports WHERE json_extract(payload,'$.root')=?1 OR EXISTS(SELECT 1 FROM json_each(json_extract(imports.payload,'$.outputs')) WHERE value=?1)",[id])?;
tx.execute(
"DELETE FROM active_outputs WHERE input=?1 OR output=?1",
[id],
)?;
tx.execute("DELETE FROM active_outputs WHERE job IN (SELECT id FROM jobs WHERE input=?1 OR output=?1)",[id])?;
tx.execute("DELETE FROM jobs WHERE input=?1 OR output=?1", [id])?;
for table in [
"function_signatures",
"code_programs",
"symbols",
"code_sources",
"executable_sources",
"artifacts",
] {
tx.execute(&format!("DELETE FROM {table} WHERE snapshot=?1"), [id])?;
}
}
for id in &plan.jobs {
tx.execute("DELETE FROM active_outputs WHERE job=?1", [id])?;
tx.execute("DELETE FROM jobs WHERE id=?1", [id])?;
}
for id in &plan.imports {
tx.execute("DELETE FROM imports WHERE id=?1", [id])?;
}
for id in &plan.snapshots {
tx.execute("DELETE FROM snapshots WHERE id=?1", [id])?;
}
for id in &plan.runs {
tx.execute("DELETE FROM runs WHERE id=?1", [id])?;
}
tx.commit()?;
self.workspace_cache.lock().unwrap().clear();
*self.storage_usage_cache.lock().unwrap() = None;
Ok(())
}
pub fn catalog_files(&self, search: &str, repository: &str, page: usize) -> Result<Value> {
let db = self.catalog.db()?;
let filter = "FROM artifacts a JOIN snapshots s ON s.id=a.snapshot WHERE a.identity IS NOT NULL AND (?1='' OR s.repository=?1) AND instr(a.search,?2)>0";
let search = search.to_lowercase();
let total: i64 = db.query_row(
&format!("SELECT count(*) {filter}"),
params![repository, search],
|r| r.get(0),
)?;
let page = page.min((total.max(1) - 1) as usize / 100);
let mut stmt=db.prepare(&format!("SELECT a.payload,s.repository,s.version,s.edition,json_extract(s.summary,'$.layer'),(SELECT count(*) FROM artifacts copies WHERE copies.identity=a.identity) {filter} ORDER BY s.repository,s.version,a.path,a.snapshot LIMIT 100 OFFSET ?3"))?;
let items = stmt
.query_map(params![repository, search, (page * 100) as i64], |r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, String>(3)?,
r.get::<_, Option<String>>(4)?,
r.get::<_, i64>(5)?,
))
})?
.map(|row| {
let (raw, repository, version, edition, layer, copies) = row?;
let mut item: Value = serde_json::from_str(&raw)?;
item["release"] =
json!({"repository":repository,"version":version,"edition":edition});
item["layer"] = json!(layer);
item["identical_copies"] = json!(copies);
item["path"] = item["entry"]["path"].clone();
item["kind"] = json!(
item["media_type"]
.as_str()
.and_then(|m| m.split('/').next())
.filter(|k| ["image", "audio", "video"].contains(k))
.unwrap_or("file")
);
Ok(item)
})
.collect::<Result<Vec<_>>>()?;
Ok(json!({"items":items,"total":total,"page":page,"page_size":100}))
}
}
src/http.rs
+289 -4
View File
@@ -56,6 +56,26 @@ pub fn router(archive: App) -> Router {
.route("/api/info", get(info))
.route("/api/snapshots", get(snapshots))
.route("/api/library", get(library))
.route("/api/workspaces", get(workspaces))
.route("/api/workspaces/{id}", get(workspace))
.route("/api/workspaces/{id}/files", get(workspace_files))
.route(
"/api/workspaces/{id}/programs",
get(workspace_programs).post(analyze_programs),
)
.route("/api/signatures", get(signatures))
.route("/api/signatures/function", get(function_reuse))
.route("/api/signatures/export", get(export_signatures))
.route(
"/api/signatures/libraries",
post(import_signatures).layer(DefaultBodyLimit::max(16 * 1024 * 1024)),
)
.route("/api/signatures/libraries/{id}", get(download_signatures))
.route("/api/catalog", get(catalog_overview))
.route("/api/catalog/files", get(catalog_files))
.route("/api/catalog/delete-preview", get(preview_catalog_delete))
.route("/api/catalog/delete", post(delete_catalog))
.route("/api/catalog/collect", post(collect_catalog))
.route("/api/artifacts", get(artifacts))
.route("/api/snapshots/{id}", get(snapshot))
.route("/api/runs", get(runs))
@@ -67,6 +87,12 @@ pub fn router(archive: App) -> Router {
.route("/api/processing", get(processing))
.route("/api/exports", get(exports))
.route("/api/import", post(import))
.route("/api/imports", post(submit_import_task))
.route("/api/imports/browse", get(browse_imports))
.route("/api/imports/inspect", post(inspect_import_url))
.route("/api/imports/{id}/{action}", post(control_import))
.route("/api/activity", get(activity))
.route("/api/runs/{id}/logs", get(run_logs))
.route(
"/api/upload",
post(upload).layer(DefaultBodyLimit::disable()),
@@ -140,9 +166,74 @@ async fn blocking<T: Send + 'static>(
}
async fn info(State(a): State<App>) -> Json<serde_json::Value> {
Json(
serde_json::json!({"schema":1,"tools":a.config.plugins,"plugins":a.config.plugins.keys().collect::<Vec<_>>(),"import_roots":a.config.import_roots,"workspace_bytes":a.config.workspace_bytes,"storage":"rustic-restic"}),
serde_json::json!({"schema":1,"tools":a.config.plugins,"plugins":a.config.plugins.keys().collect::<Vec<_>>(),"import_roots":a.config.import_roots,"workspace_bytes":a.config.workspace_bytes,"workspace":a.config.workspace,"ram_workspace_required":a.config.require_ram_workspace,"processing_enabled":a.config.processing_enabled,"storage":"rustic-restic"}),
)
}
async fn submit_import_task(
State(a): State<App>,
Json(input): Json<crate::imports::ImportRequest>,
) -> Result<Json<crate::imports::ImportTask>, ApiError> {
Ok(Json(blocking(move || a.submit_import(input)).await?))
}
async fn control_import(
State(a): State<App>,
Path((id, action)): Path<(String, String)>,
) -> Result<Json<crate::imports::ImportTask>, ApiError> {
Ok(Json(
blocking(move || a.control_import(&id, &action)).await?,
))
}
#[derive(Deserialize)]
struct ImportBrowse {
path: Option<std::path::PathBuf>,
#[serde(default)]
search: String,
#[serde(default)]
page: usize,
}
async fn browse_imports(
State(a): State<App>,
Query(q): Query<ImportBrowse>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || a.browse_import_source(q.path.as_deref(), &q.search, q.page)).await?,
))
}
#[derive(Deserialize)]
struct ImportUrl {
url: String,
}
async fn inspect_import_url(
State(a): State<App>,
Json(q): Json<ImportUrl>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.inspect_import_url(&q.url)).await?))
}
async fn activity(State(a): State<App>) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.activity_view()).await?))
}
async fn run_logs(
State(a): State<App>,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || {
uuid::Uuid::parse_str(&id)?;
anyhow::ensure!(a.catalog.get_run(&id)?.is_some(), "Run not found");
let read = |kind: &str| {
std::fs::read_to_string(
a.config
.archive
.join("logs")
.join(format!("{id}.{kind}.txt")),
)
.unwrap_or_default()
};
Ok(serde_json::json!({"stdout":read("stdout"),"stderr":read("stderr")}))
})
.await?,
))
}
async fn snapshots(State(a): State<App>) -> Result<Json<Vec<crate::Snapshot>>, ApiError> {
Ok(Json(blocking(move || a.snapshots()).await?))
}
@@ -150,6 +241,24 @@ async fn snapshots(State(a): State<App>) -> Result<Json<Vec<crate::Snapshot>>, A
async fn library(State(a): State<App>) -> Result<Json<Vec<serde_json::Value>>, ApiError> {
Ok(Json(blocking(move || a.catalog.library()).await?))
}
async fn workspaces(State(a): State<App>) -> Result<Json<Vec<serde_json::Value>>, ApiError> {
Ok(Json(blocking(move || a.workspaces()).await?))
}
async fn workspace(
State(a): State<App>,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || Ok(serde_json::to_value(a.workspace(&id)?.as_ref())?)).await?,
))
}
async fn workspace_files(
State(a): State<App>,
Path(id): Path<String>,
Query(q): Query<crate::workspace::BrowseQuery>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.browse_workspace(&id, &q)).await?))
}
async fn artifacts(
State(a): State<App>,
Query(q): Query<crate::catalog::ArtifactQuery>,
@@ -626,9 +735,16 @@ pub async fn serve(config: Config) -> anyhow::Result<()> {
let worker = tokio::spawn(async move {
while !worker_archive.is_stopping() {
let archive = worker_archive.clone();
match tokio::task::spawn_blocking(move || archive.work_one()).await {
Ok(Ok(Some(_))) => continue,
Ok(Ok(None)) => {}
match tokio::task::spawn_blocking(move || -> anyhow::Result<bool> {
if archive.work_import_one()? {
return Ok(true);
}
Ok(archive.work_one()?.is_some())
})
.await
{
Ok(Ok(true)) => continue,
Ok(Ok(false)) => {}
error => eprintln!("Processing worker: {error:?}"),
}
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
@@ -647,3 +763,172 @@ pub async fn serve(config: Config) -> anyhow::Result<()> {
worker.await?;
Ok(())
}
async fn workspace_programs(
State(a): State<App>,
Path(id): Path<String>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.workspace_programs(&id)).await?))
}
#[derive(Deserialize)]
struct AnalysisSelection {
paths: Vec<String>,
}
async fn analyze_programs(
State(a): State<App>,
Path(id): Path<String>,
Json(q): Json<AnalysisSelection>,
) -> Result<Json<crate::imports::ImportTask>, ApiError> {
Ok(Json(
blocking(move || {
let tree = a.workspace(&id)?;
a.submit_import(crate::imports::ImportRequest {
release: tree.release.clone(),
source: crate::imports::ImportSource::Archived {
workspace: id,
snapshot: tree.file_snapshot.clone(),
paths: q.paths,
},
options: crate::imports::ImportOptions {
extract: false,
media: false,
analyze: true,
},
})
})
.await?,
))
}
async fn signatures(
State(a): State<App>,
Query(q): Query<crate::signatures::SignatureQuery>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || a.catalog.signature_library(&q)).await?,
))
}
#[derive(Deserialize)]
struct FunctionLocation {
snapshot: String,
path: String,
address: String,
}
async fn function_reuse(
State(a): State<App>,
Query(q): Query<FunctionLocation>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || a.catalog.function_reuse(&q.snapshot, &q.path, &q.address)).await?,
))
}
#[derive(Deserialize)]
struct SignatureExport {
snapshot: String,
path: String,
format: String,
}
fn signature_download(format: &str, data: Vec<u8>) -> Result<Response, ApiError> {
let mut response = Response::new(Body::from(data));
response
.headers_mut()
.insert(header::CONTENT_TYPE, "application/octet-stream".parse()?);
response.headers_mut().insert(
header::CONTENT_DISPOSITION,
format!("attachment; filename=functions.{format}").parse()?,
);
Ok(response)
}
async fn export_signatures(
State(a): State<App>,
Query(q): Query<SignatureExport>,
) -> Result<Response, ApiError> {
let format = q.format.clone();
let data = blocking(move || a.catalog.export_flirt(&q.snapshot, &q.path, &q.format)).await?;
signature_download(&format, data)
}
#[derive(Deserialize)]
struct SignatureImport {
name: String,
language: String,
}
async fn import_signatures(
State(a): State<App>,
Query(q): Query<SignatureImport>,
body: axum::body::Bytes,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || a.catalog.import_flirt(&q.name, &q.language, &body)).await?,
))
}
async fn download_signatures(
State(a): State<App>,
Path(id): Path<String>,
) -> Result<Response, ApiError> {
let (format, data) = blocking(move || a.catalog.downloaded_flirt(&id)).await?;
signature_download(&format, data)
}
#[derive(Deserialize)]
struct CatalogQuery {
#[serde(default)]
measure: bool,
}
async fn catalog_overview(
State(a): State<App>,
Query(q): Query<CatalogQuery>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.catalog_overview(q.measure)).await?))
}
#[derive(Deserialize)]
struct CatalogFilesQuery {
#[serde(default)]
search: String,
#[serde(default)]
repository: String,
#[serde(default)]
page: usize,
}
async fn catalog_files(
State(a): State<App>,
Query(q): Query<CatalogFilesQuery>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || a.catalog_files(&q.search, &q.repository, q.page)).await?,
))
}
async fn preview_catalog_delete(
State(a): State<App>,
Query(q): Query<crate::catalog_management::DeleteScope>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(blocking(move || a.preview_delete(&q)).await?))
}
#[derive(Deserialize)]
struct CatalogDelete {
scope: crate::catalog_management::DeleteScope,
token: String,
}
async fn delete_catalog(
State(a): State<App>,
axum::Extension(reservation): axum::Extension<Reservation>,
Json(q): Json<CatalogDelete>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || {
let _reservation = reservation;
a.delete_cataloged(&q.scope, &q.token)
})
.await?,
))
}
async fn collect_catalog(
State(a): State<App>,
axum::Extension(reservation): axum::Extension<Reservation>,
) -> Result<Json<serde_json::Value>, ApiError> {
Ok(Json(
blocking(move || {
let _reservation = reservation;
a.cleanup_orphans()
})
.await?,
))
}
src/imports.rs
+603
View File
@@ -0,0 +1,603 @@
//! Explicit, durable import workflows. Transfer and progress files live only in scratch RAM.
use crate::{Archive, PluginConfig, Release, Run, archive::validate_release, model::now};
use anyhow::{Context, Result, ensure};
use rusqlite::{OptionalExtension, params};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use std::{
collections::{BTreeMap, BTreeSet},
fs,
path::{Path, PathBuf},
process::{Command, Stdio},
time::{Duration, Instant},
};
#[derive(Default)]
pub(crate) struct LiveActivity {
context: Option<(std::thread::ThreadId, String)>,
cancelled: BTreeSet<String>,
progress: BTreeMap<String, Value>,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case", deny_unknown_fields)]
pub enum ImportSource {
Local {
paths: Vec<PathBuf>,
},
Url {
url: String,
},
Archived {
workspace: String,
snapshot: String,
paths: Vec<String>,
},
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ImportOptions {
pub extract: bool,
pub media: bool,
pub analyze: bool,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ImportRequest {
pub source: ImportSource,
pub release: Release,
pub options: ImportOptions,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ImportTask {
pub id: String,
pub request: ImportRequest,
pub state: String,
pub stage: String,
pub created: u64,
pub started: Option<u64>,
pub finished: Option<u64>,
pub root: Option<String>,
pub run: Option<String>,
pub error: Option<String>,
pub outputs: BTreeMap<String, String>,
pub configs: BTreeMap<String, PluginConfig>,
pub source_receipt: Option<Value>,
pub warnings: Vec<String>,
}
fn write_task(db: &rusqlite::Connection, task: &ImportTask) -> Result<()> {
db.execute("INSERT INTO imports(id,state,created,payload) VALUES(?1,?2,?3,?4) ON CONFLICT(id) DO UPDATE SET state=excluded.state,payload=excluded.payload",
params![task.id,task.state,task.created as i64,serde_json::to_string(task)?])?;
Ok(())
}
fn read_task(db: &rusqlite::Connection, id: &str) -> Result<ImportTask> {
let raw: String = db.query_row("SELECT payload FROM imports WHERE id=?1", [id], |r| {
r.get(0)
})?;
Ok(serde_json::from_str(&raw)?)
}
impl Archive {
pub fn import_tasks(&self) -> Result<Vec<ImportTask>> {
let db = self.catalog.db()?;
let mut stmt =
db.prepare("SELECT payload FROM imports ORDER BY created DESC,rowid DESC LIMIT 200")?;
stmt.query_map([], |r| r.get::<_, String>(0))?
.map(|r| Ok(serde_json::from_str(&r?)?))
.collect()
}
pub(crate) fn recover_imports(&self) -> Result<()> {
for mut task in self
.import_tasks()?
.into_iter()
.filter(|t| t.state == "running")
{
// A published original survives interruption between publication and task update.
if let Some(run) = task
.run
.as_ref()
.map(|id| self.catalog.get_run(id))
.transpose()?
.flatten()
{
if run.operation == "import" && task.root.is_none() {
task.root = run.output;
} else if let Some(output) = run.output {
task.outputs.insert(run.operation, output);
}
}
task.state = "interrupted".into();
task.error =
Some("Server stopped during this import. Retry resumes from saved results.".into());
task.finished = Some(now());
write_task(&*self.catalog.db()?, &task)?;
}
Ok(())
}
pub fn submit_import(&self, mut request: ImportRequest) -> Result<ImportTask> {
validate_release(&request.release)?;
match &mut request.source {
ImportSource::Local { paths } => {
ensure!(
!paths.is_empty() && paths.len() <= 32,
"Choose 1–32 files, or one folder"
);
let mut names = BTreeSet::new();
let multiple = paths.len() > 1;
for path in paths {
*path = self.allowed_source(path)?;
ensure!(
path.is_file() || (!multiple && path.is_dir()),
"Choose files or one complete folder"
);
ensure!(
names.insert(path.file_name().context("Missing filename")?.to_owned()),
"Selected files must have different names"
);
}
}
ImportSource::Url { url } => {
ensure!(
url.len() <= 8192
&& (url.starts_with("https://") || url.starts_with("http://")),
"Use an HTTP or HTTPS file URL"
);
}
ImportSource::Archived {
workspace,
snapshot,
paths,
} => {
let tree = self.workspace(workspace)?;
ensure!(
tree.file_snapshot == *snapshot && tree.release == request.release,
"Choose a program from the current working tree"
);
ensure!(
!request.options.extract && !request.options.media && request.options.analyze,
"Archived selection is for code analysis"
);
ensure!(!paths.is_empty(), "Choose at least one program");
paths.sort();
paths.dedup();
let available = self.executable_paths(snapshot)?;
ensure!(
paths.iter().all(|p| available.contains(p)),
"Selection contains an unsupported program"
);
}
}
ensure!(
!request.options.media || request.options.extract,
"Asset decoding requires extraction"
);
let mut configs = BTreeMap::new();
for (name, needed) in [
("import-extract", request.options.extract),
("media-extract", request.options.media),
("media-preview", request.options.media),
("ghidra", request.options.analyze),
] {
if needed {
let cfg = self
.config
.plugins
.get(name)
.with_context(|| format!("{name} is not configured"))?;
configs.insert(name.to_owned(), self.freeze_tool_config(cfg)?);
}
}
let root = match &request.source {
ImportSource::Archived { workspace, .. } => Some(workspace.clone()),
_ => None,
};
let task = ImportTask {
id: uuid::Uuid::new_v4().to_string(),
request,
state: "queued".into(),
stage: "Waiting to import".into(),
created: now(),
started: None,
finished: None,
root,
run: None,
error: None,
outputs: BTreeMap::new(),
configs,
source_receipt: None,
warnings: vec![],
};
let db = self.catalog.db()?;
let pending: i64 = db.query_row(
"SELECT count(*) FROM imports WHERE state IN ('queued','running')",
[],
|r| r.get(0),
)?;
ensure!(pending < 32, "Import queue is full");
write_task(&db, &task)?;
Ok(task)
}
pub fn control_import(&self, id: &str, action: &str) -> Result<ImportTask> {
let db = self.catalog.db()?;
let mut task = read_task(&db, id)?;
match action {
"cancel" => {
ensure!(
["queued", "running"].contains(&task.state.as_str()),
"Import is no longer active"
);
if task.state == "queued" {
task.state = "cancelled".into();
task.finished = Some(now());
} else {
self.activity.lock().unwrap().cancelled.insert(id.into());
}
}
"retry" => {
ensure!(
["failed", "cancelled", "interrupted"].contains(&task.state.as_str()),
"Only stopped imports can be retried"
);
for (name, config) in &mut task.configs {
if !task.outputs.contains_key(name)
&& let Some(current) = self.config.plugins.get(name)
{
*config = self.freeze_tool_config(current)?;
}
}
task.state = "queued".into();
task.stage = "Waiting to resume".into();
task.error = None;
task.finished = None;
task.run = None;
let mut live = self.activity.lock().unwrap();
live.cancelled.remove(id);
live.progress.remove(id);
}
_ => anyhow::bail!("Unknown import action"),
}
write_task(&db, &task)?;
Ok(task)
}
pub(crate) fn import_context(&self) -> Option<String> {
self.activity
.lock()
.unwrap()
.context
.as_ref()
.filter(|(thread, _)| *thread == std::thread::current().id())
.map(|(_, id)| id.clone())
}
pub(crate) fn check_interruption(&self) -> Result<()> {
ensure!(!self.is_stopping(), "Interrupted by server shutdown");
if let Some(id) = self.import_context() {
ensure!(
!self.activity.lock().unwrap().cancelled.contains(&id),
"Import cancelled"
);
}
Ok(())
}
pub(crate) fn track_run(&self, run: &Run, attach: bool) -> Result<()> {
self.set_progress(
&run.id,
json!({"stage":run.stage,"completed":run.bytes_processed,"unit":"bytes"}),
);
if attach && let Some(id) = self.import_context() {
let db = self.catalog.db()?;
let mut task = read_task(&db, &id)?;
task.run = Some(run.id.clone());
write_task(&db, &task)?;
}
Ok(())
}
pub(crate) fn set_progress(&self, key: &str, progress: Value) {
let context = self.import_context();
let mut live = self.activity.lock().unwrap();
live.progress.insert(key.into(), progress.clone());
if let Some(id) = context {
live.progress.insert(id, progress);
}
}
pub(crate) fn read_progress(&self, key: &str, file: &Path) {
if fs::metadata(file).is_ok_and(|m| m.len() <= 65536)
&& let Ok(raw) = fs::read(file)
&& let Ok(value) = serde_json::from_slice::<Value>(&raw)
&& value["stage"].is_string()
{
self.set_progress(key, value);
}
}
/// The worker executes only steps explicitly requested by this import, even when legacy jobs are paused.
pub fn work_import_one(&self) -> Result<bool> {
if self.is_stopping() {
return Ok(false);
}
let mut task = {
let mut db = self.catalog.db()?;
let tx = db.transaction()?;
let active: i64 = tx.query_row(
"SELECT count(*) FROM imports WHERE state='running'",
[],
|r| r.get(0),
)?;
if active > 0 {
return Ok(false);
}
let id: Option<String> = tx
.query_row(
"SELECT id FROM imports WHERE state='queued' ORDER BY created,rowid LIMIT 1",
[],
|r| r.get(0),
)
.optional()?;
let Some(id) = id else { return Ok(false) };
let mut task = read_task(&tx, &id)?;
task.state = "running".into();
task.started = Some(now());
write_task(&tx, &task)?;
tx.commit()?;
task
};
self.activity.lock().unwrap().context =
Some((std::thread::current().id(), task.id.clone()));
let result = self.run_import(&mut task);
if let Err(error) = result {
task.state = if self.is_stopping() {
"interrupted"
} else if self.activity.lock().unwrap().cancelled.contains(&task.id) {
"cancelled"
} else {
"failed"
}
.into();
task.error = Some(format!("{error:#}"));
} else {
task.state = "completed".into();
task.stage = "Ready to browse".into();
}
task.finished = Some(now());
// start_run updates the run association while a step executes.
task.run = read_task(&*self.catalog.db()?, &task.id)?.run;
write_task(&*self.catalog.db()?, &task)?;
let mut live = self.activity.lock().unwrap();
live.context = None;
live.progress.remove(&task.id);
live.cancelled.remove(&task.id);
Ok(true)
}
fn import_stage(&self, task: &mut ImportTask, stage: &str) -> Result<()> {
self.check_interruption()?;
task.stage = stage.into();
task.run = None;
write_task(&*self.catalog.db()?, task)?;
self.set_progress(&task.id, json!({"stage":stage}));
Ok(())
}
fn run_import(&self, task: &mut ImportTask) -> Result<()> {
if task.root.is_none() {
crate::workspace::check_scratch(&self.config)?;
self.import_stage(task, "Reading source")?;
let transfer = tempfile::Builder::new()
.prefix("download-")
.tempdir_in(&self.config.workspace)?;
let paths = match &task.request.source {
ImportSource::Archived { .. } => {
anyhow::bail!("Archived analysis is missing its working tree")
}
ImportSource::Local { paths } => paths
.iter()
.map(|p| self.allowed_source(p))
.collect::<Result<Vec<_>>>()?,
ImportSource::Url { url } => {
let receipt=self.source_helper(json!({"mode":"download","url":url,"directory":transfer.path(),"limit":self.config.workspace_bytes/2,"logical_limit":128u64*1024*1024*1024}),Some(&task.id))?;
let path =
PathBuf::from(receipt["path"].as_str().context("Download has no path")?);
task.source_receipt = Some(receipt);
vec![path]
}
};
self.import_stage(task, "Retaining original files")?;
let root = self.import_managed_sources(&paths, task.request.release.clone())?;
task.root = Some(root.id);
write_task(&*self.catalog.db()?, task)?;
// Release downloaded bytes before extraction needs the scratch allowance.
drop(transfer);
}
let root = task.root.clone().context("Missing original")?;
let files = if let ImportSource::Archived { snapshot, .. } = &task.request.source {
snapshot.clone()
} else if task.request.options.extract {
self.import_step(
task,
"import-extract",
&root,
"Unpacking and detecting formats",
)?
} else {
root
};
if task.request.options.media {
let media = self.import_step(task, "media-extract", &files, "Decoding assets")?;
self.import_step(task, "media-preview", &media, "Preparing media previews")?;
}
if task.request.options.analyze {
let paths = if let ImportSource::Archived { paths, .. } = &task.request.source {
paths.clone()
} else {
self.default_analysis_paths(&files)?
};
if paths.is_empty() {
task.warnings
.push("No supported ELF or PE programs were found for code analysis.".into());
} else {
let cfg = task
.configs
.get_mut("ghidra")
.context("Missing analysis configuration")?;
cfg.settings["paths"] = serde_json::to_value(paths)?;
self.import_step(task, "ghidra", &files, "Analyzing programs")?;
}
}
Ok(())
}
fn import_step(
&self,
task: &mut ImportTask,
tool: &str,
input: &str,
stage: &str,
) -> Result<String> {
if let Some(id) = task.outputs.get(tool) {
return Ok(id.clone());
}
self.import_stage(task, stage)?;
let cfg = task.configs.get(tool).context("Missing import tool")?;
let output = self.process_config(input, tool, cfg, None)?;
task.outputs.insert(tool.into(), output.id.clone());
task.warnings.extend(output.warnings);
task.run = Some(output.run);
write_task(&*self.catalog.db()?, task)?;
Ok(output.id)
}
pub fn inspect_import_url(&self, url: &str) -> Result<Value> {
ensure!(url.len() <= 8192, "URL is too long");
self.source_helper(json!({"mode":"inspect","url":url}), None)
}
fn source_helper(&self, mut request: Value, task: Option<&str>) -> Result<Value> {
let work = tempfile::Builder::new()
.prefix("source-")
.tempdir_in(&self.config.workspace)?;
let script = work.path().join("source.py");
fs::write(&script, include_str!("../plugins/import_source.py"))?;
let control = work.path().join("request.json");
let result = work.path().join("result.json");
let progress = work.path().join("progress.json");
request["result_file"] = json!(result);
request["progress_file"] = json!(progress);
fs::write(&control, serde_json::to_vec(&request)?)?;
let mut cmd = Command::new("python3");
cmd.arg(&script)
.arg(&control)
.env("TMPDIR", work.path())
.env("PYTHONDONTWRITEBYTECODE", "1")
.stdin(Stdio::null())
.stdout(Stdio::null())
.stderr(Stdio::null());
use std::os::unix::process::CommandExt;
cmd.process_group(0);
let mut child = cmd.spawn()?;
let _group = crate::plugins::ProcessGroup(child.id() as i32);
let started = Instant::now();
let timeout = if task.is_some() { 24 * 3600 } else { 60 };
let status = loop {
if let Err(error) = self.check_interruption() {
let _ = child.kill();
let _ = child.wait();
return Err(error);
}
if let Some(id) = task {
self.read_progress(id, &progress);
}
if let Some(status) = child.try_wait()? {
break status;
}
if started.elapsed().as_secs() > timeout {
let _ = child.kill();
let _ = child.wait();
anyhow::bail!("Source request timed out");
}
std::thread::sleep(Duration::from_millis(200));
};
ensure!(
fs::metadata(&result).is_ok_and(|m| m.len() <= 8 * 1024 * 1024),
"Source request failed before returning a result"
);
let value: Value = serde_json::from_slice(&fs::read(result)?)?;
if let Some(error) = value["error"].as_str() {
anyhow::bail!("{error}");
}
ensure!(status.success(), "Source request failed");
Ok(value)
}
pub fn browse_import_source(
&self,
path: Option<&Path>,
search: &str,
page: usize,
) -> Result<Value> {
let roots = &self.config.import_roots;
let Some(path) = path else {
return Ok(json!({"roots":roots,"items":[],"total":0,"page":0}));
};
let path = self.allowed_source(path)?;
ensure!(path.is_dir(), "Choose a folder to browse");
let mut items = Vec::new();
let search = search.to_lowercase();
for entry in fs::read_dir(&path)? {
let entry = entry?;
let name = entry.file_name().to_string_lossy().into_owned();
if !name.to_lowercase().contains(&search) {
continue;
}
let Ok(allowed) = self.allowed_source(&entry.path()) else {
continue;
};
let meta = fs::metadata(&allowed)?;
if !meta.is_file() && !meta.is_dir() {
continue;
}
items.push(json!({"name":name,"path":entry.path(),"directory":meta.is_dir(),"bytes":if meta.is_file(){Some(meta.len())}else{None}}));
}
items.sort_by(|a, b| {
b["directory"]
.as_bool()
.cmp(&a["directory"].as_bool())
.then(a["name"].as_str().cmp(&b["name"].as_str()))
});
let total = items.len();
let page = page.min(total.saturating_sub(1) / 200);
let parent = path.parent().filter(|p| self.allowed_source(p).is_ok());
Ok(
json!({"roots":roots,"path":path,"parent":parent,"items":items.into_iter().skip(page*200).take(200).collect::<Vec<_>>(),"total":total,"page":page,"page_size":200}),
)
}
pub fn activity_view(&self) -> Result<Value> {
let mut items = Vec::new();
for task in self.import_tasks()? {
let progress = self
.activity
.lock()
.unwrap()
.progress
.get(&task.id)
.cloned();
let cancelling = self.activity.lock().unwrap().cancelled.contains(&task.id);
items.push(json!({"kind":"import","id":task.id,"release":task.request.release,"source":task.request.source,"state":task.state,"stage":task.stage,"created":task.created,"started":task.started,"finished":task.finished,"root":task.root,"run":task.run,"error":task.error,"warnings":task.warnings,"progress":progress,"cancelling":cancelling}));
}
for job in self.jobs()? {
let raw: String = self.catalog.db()?.query_row(
"SELECT json_extract(summary,'$.release') FROM snapshots WHERE id=?1",
[&job.input],
|r| r.get(0),
)?;
let release: Release = serde_json::from_str(&raw)?;
let run = job
.run
.as_ref()
.map(|id| self.catalog.get_run(id))
.transpose()?
.flatten();
let progress = job
.run
.as_ref()
.and_then(|id| self.activity.lock().unwrap().progress.get(id).cloned())
.or_else(|| {
run.as_ref().map(
|r| json!({"stage":r.stage,"completed":r.bytes_processed,"unit":"bytes"}),
)
});
items.push(json!({"kind":"processing","id":job.id,"release":release,"state":job.state,"stage":job.tool,"created":job.created,"started":run.as_ref().map(|r|r.started_at),"run":job.run,"error":job.error,"progress":progress}));
}
items.sort_by(|a, b| b["created"].as_u64().cmp(&a["created"].as_u64()));
Ok(json!({"items":items,"automatic_processing":self.config.processing_enabled}))
}
}
src/jobs.rs
+4 -1
View File
@@ -239,7 +239,7 @@ impl Archive {
}
/// Claims and executes at most one job; CLI callers can drain without a service.
pub fn work_one(&self) -> Result<Option<Job>> {
if self.is_stopping() {
if self.is_stopping() || !self.config.processing_enabled {
return Ok(None);
}
let job = {
@@ -290,6 +290,9 @@ impl Archive {
Ok(Some(job))
}
pub fn queue_automatic(&self, snapshot: &Snapshot) -> Result<()> {
if !self.config.processing_enabled || self.import_context().is_some() {
return Ok(());
}
// No speculative generation inference. Configured suffixes declare applicability.
for (name, cfg) in &self.config.plugins {
if name.contains("ghidra") || cfg.auto_extensions.is_empty() {
src/lib.rs
+4
View File
@@ -2,12 +2,16 @@
pub mod analysis;
pub mod archive;
pub mod catalog;
pub mod catalog_management;
pub mod http;
pub mod imports;
pub mod jobs;
pub mod model;
pub mod plugins;
pub mod signatures;
pub mod storage;
pub use archive::Archive;
pub use model::*;
mod tool_sources;
pub mod workspace;
src/model.rs
+9
View File
@@ -14,6 +14,12 @@ pub fn now() -> u64 {
pub struct Config {
pub archive: PathBuf,
pub workspace: PathBuf,
/// Refuse disk-backed scratch storage; publication is the persistence boundary.
#[serde(default = "enabled")]
pub require_ram_workspace: bool,
/// Pause automatic submission and the background worker without losing history.
#[serde(default = "enabled")]
pub processing_enabled: bool,
pub import_roots: Vec<PathBuf>,
#[serde(default = "default_budget")]
pub workspace_bytes: u64,
@@ -22,6 +28,9 @@ pub struct Config {
#[serde(default)]
pub plugins: BTreeMap<String, PluginConfig>,
}
fn enabled() -> bool {
true
}
fn default_budget() -> u64 {
100 * 1024 * 1024 * 1024
}
src/plugins.rs
+41 -19
View File
@@ -72,6 +72,7 @@ impl Archive {
self.attach_job_run(id, &run.id)?;
}
let result = (|| {
crate::workspace::check_scratch(&self.config)?;
let mut staged_input = input.clone();
if plugin.contains("ghidra")
&& let Some(paths) = cfg.settings["paths"].as_array()
@@ -93,10 +94,6 @@ impl Archive {
);
staged_input.logical_bytes = staged_input.entries.iter().map(|e| e.size).sum();
}
ensure!(
staged_input.logical_bytes < self.config.workspace_bytes / 2,
"input exceeds plugin workspace allowance (half of total budget)"
);
// Ghidra rejects dot-prefixed project path components. Keep plugin
// workspaces filesystem-shaped without tempfile's default `.tmp` prefix.
let work = tempfile::Builder::new()
@@ -104,10 +101,18 @@ impl Archive {
.tempdir_in(&self.config.workspace)?;
let inputs = work.path().join("input");
let outputs = work.path().join("output");
let scratch = work.path().join("tmp");
fs::create_dir(&scratch)?;
fs::create_dir(&inputs)?;
fs::create_dir(&outputs)?;
self.materialize(&staged_input, &inputs)?;
self.materialize_progress(&staged_input, &inputs, &run.id)?;
let input_allocation = workspace_size(&inputs)?;
ensure!(
input_allocation < self.config.workspace_bytes / 2,
"input exceeds plugin workspace allowance (half of total budget)"
);
let result_file = work.path().join("result.json");
let progress_file = work.path().join("progress.json");
let request_file = work.path().join("request.json");
let request = PluginRequest {
protocol: 1,
@@ -115,7 +120,7 @@ impl Archive {
output_dir: &outputs,
result_file: &result_file,
settings: &cfg.settings,
workspace_bytes: self.config.workspace_bytes - staged_input.logical_bytes,
workspace_bytes: self.config.workspace_bytes - input_allocation,
};
serde_json::to_writer(File::create(&request_file)?, &request)?;
run.stage = "processing".into();
@@ -125,6 +130,12 @@ impl Archive {
.args(&cfg.command[1..])
.arg(&request_file)
.current_dir(work.path())
.env("TMPDIR", &scratch)
.env("TMP", &scratch)
.env("TEMP", &scratch)
.env("XDG_CACHE_HOME", work.path().join("cache"))
.env("PYTHONDONTWRITEBYTECODE", "1")
.env("VERSTACK_PROGRESS_FILE", &progress_file)
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
@@ -149,10 +160,14 @@ impl Archive {
let _process_group = ProcessGroup(child.id() as i32);
let started = Instant::now();
let status = loop {
ensure!(
!self.is_stopping(),
"processing interrupted by server shutdown"
);
if let Err(error) = self.check_interruption() {
drop(_process_group);
let _ = child.wait();
let _ = stdout.join();
let _ = stderr.join();
return Err(error);
}
self.read_progress(&run.id, &progress_file);
if let Some(status) = child.try_wait()? {
break status;
}
@@ -272,7 +287,7 @@ impl Archive {
result
}
}
struct ProcessGroup(i32);
pub(crate) struct ProcessGroup(pub(crate) i32);
impl Drop for ProcessGroup {
fn drop(&mut self) {
let _ = nix::sys::signal::killpg(
@@ -281,8 +296,10 @@ impl Drop for ProcessGroup {
);
}
}
fn workspace_size(path: &Path) -> Result<u64> {
pub(crate) fn workspace_size(path: &Path) -> Result<u64> {
use std::os::unix::fs::MetadataExt;
let mut size = 0u64;
let mut seen = std::collections::BTreeSet::new();
for e in walkdir::WalkDir::new(path).follow_links(false) {
let e = match e {
Ok(e) => e,
@@ -297,7 +314,11 @@ fn workspace_size(path: &Path) -> Result<u64> {
};
if e.file_type().is_file() {
match e.metadata() {
Ok(metadata) => size = size.saturating_add(metadata.len()),
Ok(metadata) => {
if seen.insert((metadata.dev(), metadata.ino())) {
size = size.saturating_add(metadata.blocks().saturating_mul(512));
}
}
Err(error)
if error
.io_error()
@@ -315,9 +336,7 @@ fn capture_log(
) -> std::thread::JoinHandle<()> {
std::thread::spawn(move || {
use std::io::Write;
let Ok(mut file) = File::create(path) else {
return;
};
let mut captured = Vec::with_capacity(256 * 1024);
let mut buffer = [0; 4096];
let mut remaining = 256 * 1024;
loop {
@@ -326,15 +345,18 @@ fn capture_log(
Ok(n) => {
let count = n.min(remaining);
if count > 0 {
let _ = file.write_all(&buffer[..count]);
captured.extend_from_slice(&buffer[..count]);
remaining -= count;
}
}
}
}
if remaining == 0 {
let _ = file.write_all(b"\n[log truncated at 256 KiB]\n");
captured.extend_from_slice(b"\n[log truncated at 256 KiB]\n");
}
if let Ok(mut file) = File::create(path) {
let _ = file.write_all(&captured);
let _ = file.sync_all();
}
let _ = file.sync_all();
})
}
src/signatures.rs
+625
View File
@@ -0,0 +1,625 @@
//! Saved function observations and explicit FLIRT libraries. Matches are evidence,
//! never automatic renames or storage identities.
use crate::{Archive, catalog::Catalog};
use anyhow::{Context, Result, ensure};
use rusqlite::{Connection, OptionalExtension, params};
use serde_json::{Value, json};
use std::collections::{BTreeMap, BTreeSet};
fn unhex(value: &str) -> Result<Vec<u8>> {
ensure!(
value.len() % 2 == 0 && value.is_ascii(),
"Invalid signature bytes"
);
(0..value.len())
.step_by(2)
.map(|i| Ok(u8::from_str_radix(&value[i..i + 2], 16)?))
.collect()
}
fn safe_name(name: &str) -> String {
name.chars()
.map(|c| if c.is_ascii_graphic() { c } else { '_' })
.take(512)
.collect()
}
fn pattern_line(function: &Value) -> Result<String> {
let pattern = function["signature_pattern"].as_str().unwrap_or("");
if pattern.len() < 64 {
return Ok(String::new());
}
ensure!(
pattern.len() <= 1024 && pattern.len() % 2 == 0 && pattern.is_ascii(),
"Invalid FLIRT pattern length"
);
let raw = unhex(function["signature_bytes"].as_str().unwrap_or(""))?;
ensure!(
raw.len() * 2 == pattern.len(),
"Pattern and bytes differ in length"
);
let tokens = pattern.as_bytes().chunks_exact(2).collect::<Vec<_>>();
for (i, t) in tokens.iter().enumerate() {
ensure!(
*t == b".." || u8::from_str_radix(std::str::from_utf8(t)?, 16)? == raw[i],
"Pattern literal differs from saved bytes"
);
}
let crc_len = tokens
.iter()
.skip(32)
.take(255)
.take_while(|t| **t != b"..")
.count();
let tail_end = (32 + crc_len + 64).min(tokens.len());
let name = safe_name(function["name"].as_str().unwrap_or("unnamed"));
Ok(format!(
"{} {:02X} {:04X} {:04X} :0000 {}{}",
&pattern[..64],
crc_len,
fast_flirt::crc16(&raw[32..32 + crc_len]),
function["size"].as_u64().unwrap_or(0),
if name.is_empty() { "unnamed" } else { &name },
if tail_end > 32 + crc_len {
format!(" {}", &pattern[(32 + crc_len) * 2..tail_end * 2])
} else {
String::new()
}
))
}
pub(crate) fn index_program(
db: &Connection,
snapshot: &str,
path: &str,
facts: &Value,
) -> Result<()> {
let operation:Option<String>=db.query_row("SELECT json_extract(r.payload,'$.operation') FROM snapshots s JOIN runs r ON r.id=s.run WHERE s.id=?1",[snapshot],|r|r.get(0)).optional()?;
if !operation.is_some_and(|v| v.contains("ghidra")) {
return Ok(());
}
let input = facts["input_sha256"].as_str().unwrap_or("").to_lowercase();
let language = facts["language"].as_str().unwrap_or("");
let mut metadata = facts.clone();
metadata
.as_object_mut()
.context("Invalid facts")?
.remove("functions");
db.execute(
"INSERT OR REPLACE INTO code_programs VALUES(?1,?2,?3,?4,?5,?6)",
params![
snapshot,
path,
input,
facts["source_path"].as_str().unwrap_or("Program"),
language,
metadata.to_string()
],
)?;
for f in facts["functions"].as_array().context("Missing functions")? {
let hash = f["body_sha256"].as_str().unwrap_or("").to_lowercase();
let valid = hash.len() == 64 && hash.bytes().all(|b| b.is_ascii_hexdigit());
let size = f["size"].as_u64().unwrap_or(0).min(i64::MAX as u64) as i64;
let eligible = valid
&& size >= 32
&& f["thunk"] != true
&& f["external"] != true
&& facts["analysis_timed_out"] != true;
let pattern = pattern_line(f)?;
let bytes = unhex(f["signature_bytes"].as_str().unwrap_or(""))?;
db.execute(
"INSERT OR REPLACE INTO function_signatures VALUES(?1,?2,?3,?4,?5,?6,?7,?8,?9,?10)",
params![
snapshot,
path,
f["address"].as_str().unwrap_or(""),
input,
language,
if valid { hash } else { String::new() },
size,
eligible,
pattern,
bytes
],
)?;
}
Ok(())
}
impl Archive {
pub(crate) fn workspace_code(&self, parents: &[String], prefix: &str) -> Result<Vec<Value>> {
let db = self.catalog.db()?;
let mut stmt=db.prepare("SELECT p.snapshot,p.path,p.input,p.source_path,p.metadata,s.parent,s.manifest FROM code_programs p JOIN snapshots s ON s.id=p.snapshot JOIN runs r ON r.id=s.run WHERE json_extract(r.payload,'$.state')='complete' ORDER BY s.rowid DESC,p.path")?;
let mut seen = BTreeSet::new();
let mut result = Vec::new();
for row in stmt.query_map([], |r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, String>(3)?,
r.get::<_, String>(4)?,
r.get::<_, Option<String>>(5)?,
r.get::<_, String>(6)?,
))
})? {
let (snapshot, path, input, source, metadata, parent, manifest) = row?;
if !parent.as_ref().is_some_and(|p| parents.contains(p)) || !seen.insert(input.clone())
{
continue;
}
let meta: Value = serde_json::from_str(&metadata)?;
let saved: Value = serde_json::from_str(&manifest)?;
let source_size = meta["source_size"].as_u64().unwrap_or_else(|| {
parent
.as_ref()
.and_then(|p| {
db.query_row("SELECT manifest FROM snapshots WHERE id=?1", [p], |r| {
r.get::<_, String>(0)
})
.ok()
})
.and_then(|raw| serde_json::from_str::<crate::Snapshot>(&raw).ok())
.and_then(|s| s.entries.into_iter().find(|e| e.path == source))
.map_or(0, |e| e.size)
});
result.push(json!({"snapshot":snapshot,"path":path,"input":input,"name":source.strip_prefix(prefix).unwrap_or(&source),"source_path":source,"size":source_size,"language":meta["language"],"release":saved["release"]}));
}
result.sort_by_key(|p| {
(
!crate::analysis::is_game_program(p["name"].as_str().unwrap_or("")),
std::cmp::Reverse(p["size"].as_u64().unwrap_or(0)),
p["name"].as_str().unwrap_or("").to_owned(),
)
});
Ok(result)
}
pub fn workspace_programs(&self, id: &str) -> Result<Value> {
let tree = self.workspace(id)?;
let snapshot = self.snapshot(&tree.file_snapshot)?;
let paths = self.executable_paths(&snapshot.id)?;
let defaults = self.default_analysis_paths(&snapshot.id)?;
let mut items = Vec::new();
for path in paths {
let e = self.file_entry(&snapshot, &path)?;
let analysis = tree.code.iter().find(|s| s["source_path"] == path).cloned();
items.push(json!({"path":path,"size":e.size,"game":crate::analysis::is_game_program(&path),"recommended":defaults.contains(&path),"analysis":analysis}));
}
items.sort_by_key(|p| {
(
!p["game"].as_bool().unwrap_or(false),
std::cmp::Reverse(p["size"].as_u64().unwrap_or(0)),
p["path"].as_str().unwrap_or("").to_owned(),
)
});
Ok(json!({"snapshot":snapshot.id,"items":items}))
}
}
#[derive(serde::Deserialize, Default)]
pub struct SignatureQuery {
#[serde(default)]
pub snapshot: String,
#[serde(default)]
pub path: String,
#[serde(default)]
pub search: String,
#[serde(default)]
pub page: usize,
#[serde(default)]
pub reused: bool,
}
impl Catalog {
pub fn signature_library(&self, q: &SignatureQuery) -> Result<Value> {
let db = self.db()?;
let stats:Value=db.query_row("SELECT count(*),count(DISTINCT input),sum(eligible),sum(pattern!='') FROM function_signatures",[],|r|Ok(json!({"observations":r.get::<_,i64>(0)?,"executables":r.get::<_,i64>(1)?,"eligible":r.get::<_,Option<i64>>(2)?.unwrap_or(0),"flirt_patterns":r.get::<_,Option<i64>>(3)?.unwrap_or(0)})))?;
let mut programs = Vec::new();
let mut stmt=db.prepare("SELECT p.snapshot,p.path,p.input,p.source_path,p.language,p.metadata,s.repository,s.edition,s.version,(SELECT count(*) FROM function_signatures f WHERE f.snapshot=p.snapshot AND f.path=p.path) FROM code_programs p JOIN snapshots s ON s.id=p.snapshot ORDER BY s.repository,s.version,p.source_path,s.rowid DESC")?;
for row in stmt.query_map([],|r|Ok(json!({"snapshot":r.get::<_,String>(0)?,"path":r.get::<_,String>(1)?,"input":r.get::<_,String>(2)?,"name":r.get::<_,String>(3)?,"language":r.get::<_,String>(4)?,"release":{"repository":r.get::<_,String>(6)?,"edition":r.get::<_,String>(7)?,"version":r.get::<_,String>(8)?},"functions":r.get::<_,i64>(9)?})))? {programs.push(row?);}
let filter = "FROM function_signatures f JOIN symbols s ON s.snapshot=f.snapshot AND s.path=f.path AND s.address=f.address WHERE (?1='' OR f.snapshot=?1) AND (?2='' OR f.path=?2) AND instr(s.search,?3)>0 AND (?4=0 OR (f.eligible=1 AND EXISTS(SELECT 1 FROM function_signatures other WHERE other.language=f.language AND other.body_hash=f.body_hash AND other.size=f.size AND other.eligible=1 AND other.input!=f.input)))";
let search = q.search.to_lowercase();
let total: i64 = db.query_row(
&format!("SELECT count(*) {filter}"),
params![q.snapshot, q.path, search, q.reused],
|r| r.get(0),
)?;
let page = q.page.min((total.max(1) - 1) as usize / 100);
let mut stmt=db.prepare(&format!("SELECT s.payload,f.snapshot,f.path,f.eligible,f.pattern!='',(SELECT count(DISTINCT other.input) FROM function_signatures other WHERE f.eligible=1 AND other.eligible=1 AND other.language=f.language AND other.body_hash=f.body_hash AND other.size=f.size AND other.input!=f.input) {filter} ORDER BY s.name,f.snapshot,f.address LIMIT 100 OFFSET ?5"))?;
let items = stmt
.query_map(
params![q.snapshot, q.path, search, q.reused, (page * 100) as i64],
|r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, bool>(3)?,
r.get::<_, bool>(4)?,
r.get::<_, i64>(5)?,
))
},
)?
.map(|row| {
let (raw, snapshot, path, eligible, flirt, seen) = row?;
let mut f: Value = serde_json::from_str(&raw)?;
f.as_object_mut().unwrap().remove("signature_bytes");
f.as_object_mut().unwrap().remove("signature_pattern");
f["snapshot"] = json!(snapshot);
f["path"] = json!(path);
f["eligible"] = json!(eligible);
f["flirt"] = json!(flirt);
f["seen_in"] = json!(seen);
Ok(f)
})
.collect::<Result<Vec<_>>>()?;
let mut stmt=db.prepare("SELECT id,name,language,format,patterns,created,length(data) FROM flirt_libraries ORDER BY created DESC")?;
let libraries=stmt.query_map([],|r|Ok(json!({"id":r.get::<_,String>(0)?,"name":r.get::<_,String>(1)?,"language":r.get::<_,String>(2)?,"format":r.get::<_,String>(3)?,"patterns":r.get::<_,i64>(4)?,"created":r.get::<_,i64>(5)?,"bytes":r.get::<_,i64>(6)?})))?.collect::<std::result::Result<Vec<_>,_>>()?;
Ok(
json!({"stats":stats,"programs":programs,"items":items,"total":total,"page":page,"page_size":100,"libraries":libraries}),
)
}
pub fn function_reuse(&self, snapshot: &str, path: &str, address: &str) -> Result<Value> {
let db = self.db()?;
let (input,language,hash,size,eligible,bytes):(String,String,String,i64,bool,Vec<u8>)=db.query_row("SELECT input,language,body_hash,size,eligible,bytes FROM function_signatures WHERE snapshot=?1 AND path=?2 AND address=?3",params![snapshot,path,address],|r|Ok((r.get(0)?,r.get(1)?,r.get(2)?,r.get(3)?,r.get(4)?,r.get(5)?)))?;
let mut stmt=db.prepare("SELECT f.input,f.snapshot,f.path,f.address,s.payload,p.source_path,v.repository,v.edition,v.version FROM function_signatures f JOIN symbols s ON s.snapshot=f.snapshot AND s.path=f.path AND s.address=f.address JOIN code_programs p ON p.snapshot=f.snapshot AND p.path=f.path JOIN snapshots v ON v.id=f.snapshot WHERE ?1=1 AND f.eligible=1 AND f.language=?2 AND f.body_hash=?3 AND f.size=?4 AND f.input!=?5 ORDER BY v.rowid DESC,f.address")?;
let mut seen = BTreeSet::new();
let mut occurrences = Vec::new();
for row in stmt.query_map(params![eligible, language, hash, size, input], |r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, String>(3)?,
r.get::<_, String>(4)?,
r.get::<_, String>(5)?,
r.get::<_, String>(6)?,
r.get::<_, String>(7)?,
r.get::<_, String>(8)?,
))
})? {
let (identity, snapshot, path, address, symbol, name, game, edition, version) = row?;
if seen.insert(identity.clone()) {
occurrences.push(json!({"input":identity,"source":{"snapshot":snapshot,"path":path,"name":name},"symbol":serde_json::from_str::<Value>(&symbol)?,"address":address,"release":{"repository":game,"edition":edition,"version":version}}));
}
}
drop(stmt);
let mut libraries = Vec::new();
let mut stmt =
db.prepare("SELECT id,name,format,data FROM flirt_libraries WHERE language=?1")?;
for row in stmt.query_map([&language], |r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, Vec<u8>>(3)?,
))
})? {
libraries.push(row?);
}
drop(stmt);
let mut saved = Vec::new();
let mut stmt=db.prepare("SELECT f.pattern,f.input,f.snapshot,f.path,s.payload,p.source_path,v.repository,v.edition,v.version FROM function_signatures f JOIN symbols s ON s.snapshot=f.snapshot AND s.path=f.path AND s.address=f.address JOIN code_programs p ON p.snapshot=f.snapshot AND p.path=f.path JOIN snapshots v ON v.id=f.snapshot WHERE f.language=?1 AND f.input!=?2 AND f.eligible=1 AND f.pattern!='' ORDER BY v.rowid DESC LIMIT 500000")?;
for row in stmt.query_map(params![language,input],|r|Ok((r.get::<_,String>(0)?,json!({"input":r.get::<_,String>(1)?,"source":{"snapshot":r.get::<_,String>(2)?,"path":r.get::<_,String>(3)?,"name":r.get::<_,String>(5)?},"symbol":serde_json::from_str::<Value>(&r.get::<_,String>(4)?).unwrap_or(Value::Null),"release":{"repository":r.get::<_,String>(6)?,"edition":r.get::<_,String>(7)?,"version":r.get::<_,String>(8)?}}))))?{saved.push(row?);}
drop(stmt);
drop(db);
let mut candidates = Vec::new();
let mut pattern_occurrences = Vec::new();
if eligible && !bytes.is_empty() {
let mut patterns = String::new();
for (i, (line, _)) in saved.iter().enumerate() {
let mut fields = line
.split_whitespace()
.map(str::to_owned)
.collect::<Vec<_>>();
if fields.len() >= 6 {
fields[5] = format!("observation_{i}");
patterns.push_str(&fields.join(" "));
patterns.push('\n');
}
}
if !patterns.is_empty() {
patterns.push_str("---\n");
let set = fast_flirt::pat::parse(&patterns)?;
let mut matched = BTreeSet::new();
for p in set.matches(&bytes) {
if let Some(i) = p
.public_name()
.and_then(|n| n.strip_prefix("observation_"))
.and_then(|n| n.parse::<usize>().ok())
&& let Some((_, row)) = saved.get(i)
&& matched.insert(row["input"].as_str().unwrap_or("").to_owned())
{
pattern_occurrences.push(row.clone());
}
}
}
for (id, name, format, data) in libraries {
let set = parse_library(&format, &data)?;
for pattern in set.matches(&bytes).into_iter().take(100) {
let references = pattern
.names()
.filter(|n| matches!(n, fast_flirt::Symbol::Reference(_)))
.map(|n| n.name().name.to_owned())
.collect::<Vec<_>>();
candidates.push(json!({"library":id,"library_name":name,"name":pattern.public_name(),"references":references,"status":"candidate","bytes_checked":pattern.min_input_len()}));
}
}
}
Ok(
json!({"eligible":eligible,"seen_in":occurrences.len(),"occurrences":occurrences,"pattern_occurrences":pattern_occurrences,"flirt_candidates":candidates,"flirt_available":!bytes.is_empty(),"method":"exact-body-sha256","note":"Distinct executable identities; repeated imports and re-analysis do not increase this count. FLIRT pattern hits are candidates; names and referenced functions are not automatically accepted."}),
)
}
pub fn import_flirt(&self, name: &str, language: &str, data: &[u8]) -> Result<Value> {
ensure!(
!name.trim().is_empty() && name.len() <= 255,
"Choose a library name"
);
ensure!(
!language.is_empty() && language.len() <= 128,
"Choose the target program language"
);
ensure!(
!data.is_empty() && data.len() <= 16 * 1024 * 1024,
"Signature files must be between 1 byte and 16 MiB"
);
let format = if data.starts_with(b"IDASGN") {
"sig"
} else {
"pat"
};
let set = parse_library(format, data)?;
ensure!(
!set.is_empty() && set.len() <= 500_000,
"Library must contain 1–500,000 patterns"
);
let id = blake3::hash(&[language.as_bytes(), b"\0", data].concat())
.to_hex()
.to_string();
self.db()?.execute(
"INSERT OR IGNORE INTO flirt_libraries VALUES(?1,?2,?3,?4,?5,?6,?7)",
params![
id,
name,
language,
format,
set.len() as i64,
crate::model::now() as i64,
data
],
)?;
Ok(json!({"id":id,"patterns":set.len(),"format":format}))
}
pub fn export_flirt(&self, snapshot: &str, path: &str, format: &str) -> Result<Vec<u8>> {
let db = self.db()?;
let (language, name): (String, String) = db.query_row(
"SELECT language,source_path FROM code_programs WHERE snapshot=?1 AND path=?2",
params![snapshot, path],
|r| Ok((r.get(0)?, r.get(1)?)),
)?;
let mut stmt=db.prepare("SELECT DISTINCT pattern FROM function_signatures WHERE snapshot=?1 AND path=?2 AND pattern!='' AND eligible=1 ORDER BY pattern")?;
let rows = stmt
.query_map(params![snapshot, path], |r| r.get::<_, String>(0))?
.collect::<std::result::Result<Vec<_>, _>>()?;
ensure!(
!rows.is_empty(),
"This saved analysis predates FLIRT exports. Select this program and choose Analyze selected to collect its patterns."
);
let pat = format!("{}\n---\n", rows.join("\n"));
match format {
"pat" => Ok(pat.into_bytes()),
"sig" => write_sig(&fast_flirt::pat::parse(&pat)?, &language, &name),
_ => anyhow::bail!("Choose pat or sig"),
}
}
pub fn downloaded_flirt(&self, id: &str) -> Result<(String, Vec<u8>)> {
Ok(self.db()?.query_row(
"SELECT format,data FROM flirt_libraries WHERE id=?1",
[id],
|r| Ok((r.get(0)?, r.get(1)?)),
)?)
}
}
fn parse_library(format: &str, data: &[u8]) -> Result<fast_flirt::FlirtSet> {
Ok(if format == "sig" {
fast_flirt::sig::parse(data)?
} else {
fast_flirt::pat::parse(std::str::from_utf8(data)?)?
})
}
// FLIRT v9 encoder for the patterns we generate. Round-trip validation uses
// fast-flirt's independent reader. Imported files are retained byte-for-byte.
fn vint(out: &mut Vec<u8>, value: u32) {
if value < 0x80 {
out.push(value as u8)
} else if value < 0x4000 {
out.extend_from_slice(&((value as u16) | 0x8000).to_be_bytes())
} else if value < 0x20000000 {
out.extend_from_slice(&(value | 0xc0000000).to_be_bytes())
} else {
out.push(0xe0);
out.extend_from_slice(&value.to_be_bytes())
}
}
fn word16(out: &mut Vec<u8>, value: u16) {
if value < 128 {
out.push(value as u8)
} else {
out.extend_from_slice(&(value | 0x8000).to_be_bytes())
}
}
fn write_sig(set: &fast_flirt::FlirtSet, language: &str, name: &str) -> Result<Vec<u8>> {
let processor = if language.starts_with("x86:") {
0
} else if language.starts_with("ARM:") || language.starts_with("AARCH64:") {
13
} else {
anyhow::bail!("SIG export currently supports x86 and ARM; use PAT for this architecture")
};
let name = safe_name(name);
let name = &name.as_bytes()[..name.len().min(255)];
let mut out = b"IDASGN".to_vec();
out.extend_from_slice(&[9, processor]);
out.extend_from_slice(&u32::MAX.to_le_bytes());
out.extend_from_slice(&u16::MAX.to_le_bytes());
out.extend_from_slice(&u16::MAX.to_le_bytes());
out.extend_from_slice(&[0; 18]);
out.push(name.len() as u8);
out.extend_from_slice(&[0; 2]);
out.extend_from_slice(&(set.len() as u32).to_le_bytes());
out.extend_from_slice(&32u16.to_le_bytes());
out.extend_from_slice(name);
fn node(out: &mut Vec<u8>, patterns: &[fast_flirt::Pattern<'_>], depth: usize) -> Result<()> {
if depth < 32 {
let mut groups: BTreeMap<Option<u8>, Vec<fast_flirt::Pattern<'_>>> = BTreeMap::new();
for p in patterns {
groups
.entry(if p.is_wildcard(depth) {
None
} else {
Some(p.leading()[depth])
})
.or_default()
.push(*p);
}
word16(out, groups.len() as u16);
for (byte, group) in groups {
out.push(1);
out.push(if byte.is_none() { 1 } else { 0 });
if let Some(value) = byte {
out.push(value);
}
node(out, &group, depth + 1)?;
}
return Ok(());
}
out.push(0);
for (i, p) in patterns.iter().enumerate() {
out.push(p.crc_len());
out.extend_from_slice(&p.crc16().to_be_bytes());
vint(out, p.module_len());
vint(out, 0);
out.extend_from_slice(
safe_name(p.public_name().context("Pattern has no public name")?).as_bytes(),
);
let tail = p
.tail()
.iter()
.enumerate()
.filter(|(j, _)| !p.is_tail_wildcard(*j))
.map(|(j, b)| ((32 + p.crc_len() as usize + j) as u32, *b))
.collect::<Vec<_>>();
out.push(
if i + 1 < patterns.len() { 0x10 } else { 0 } | if tail.is_empty() { 0 } else { 2 },
);
if !tail.is_empty() {
vint(out, tail.len() as u32);
for (offset, byte) in tail {
vint(out, offset);
out.push(byte);
}
}
}
Ok(())
}
let patterns = set.patterns().collect::<Vec<_>>();
ensure!(
patterns.iter().all(|p| p.leading().len() == 32),
"Expected a 32-byte FLIRT head"
);
node(&mut out, &patterns, 0)?;
let parsed = fast_flirt::sig::parse(&out)?;
ensure!(parsed.len() == set.len(), "SIG round-trip failed");
Ok(out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn flirt_crc_masks_tail_and_sig_round_trip() -> Result<()> {
let bytes = (0..120u8).collect::<Vec<_>>();
let encoded = bytes.iter().map(|b| format!("{b:02X}")).collect::<String>();
let mut masked = encoded.clone();
masked.replace_range(8..10, "..");
masked.replace_range(100..102, "..");
let f =
json!({"name":"known","size":120,"signature_bytes":encoded,"signature_pattern":masked});
let pat = pattern_line(&f)? + "\n---\n";
let parsed = fast_flirt::pat::parse(&pat)?;
let sig = write_sig(&parsed, "x86:LE:64:default", "fixture")?;
let compiled = fast_flirt::sig::parse(&sig)?;
assert_eq!(compiled.matches(&bytes).len(), 1);
let mut relocated = bytes.clone();
relocated[4] ^= 1;
relocated[50] ^= 1;
assert_eq!(compiled.matches(&relocated).len(), 1);
relocated[40] ^= 1;
assert!(
compiled.matches(&relocated).is_empty(),
"CRC bytes must be checked"
);
let mut bad_tail = bytes.clone();
bad_tail[60] ^= 1;
assert!(
compiled.matches(&bad_tail).is_empty(),
"tail bytes must be checked"
);
assert!(compiled.matches(&bytes[..70]).is_empty());
Ok(())
}
#[test]
fn reuse_counts_distinct_executable_contents_and_preserves_libraries() -> Result<()> {
let tmp = tempfile::tempdir()?;
let catalog = Catalog::open(&tmp.path().join("catalog.sqlite3"))?;
let bytes = "55".repeat(64);
let mut first = None;
for (snapshot, input) in [("one", "a"), ("two", "b"), ("two-repeated", "b")] {
{
let db = catalog.db()?;
db.execute(
"INSERT INTO snapshots VALUES(?1,'Game','1','LE','',NULL,?1,'{}','{}')",
[snapshot],
)?;
db.execute(
"INSERT INTO runs VALUES(?1,?2)",
params![
snapshot,
json!({"operation":"ghidra","state":"complete"}).to_string()
],
)?;
}
let f = json!({"address":"1000","name":"known","size":64,"thunk":false,"external":false,"body_sha256":"c".repeat(64),"signature_bytes":bytes,"signature_pattern":bytes});
let facts = json!({"input_sha256":input.repeat(64),"source_path":"game","language":"x86:LE:64:default","analysis_timed_out":false,"functions":[f]});
catalog.index_code(snapshot, "program/functions.json", &facts)?;
if first.is_none() {
first = Some(catalog.export_flirt(snapshot, "program/functions.json", "sig")?);
}
}
let reuse = catalog.function_reuse("one", "program/functions.json", "1000")?;
assert_eq!(reuse["seen_in"], 1);
assert_eq!(reuse["pattern_occurrences"].as_array().unwrap().len(), 1);
let sig = first.unwrap();
let saved = catalog.import_flirt("fixture.sig", "x86:LE:64:default", &sig)?;
assert_eq!(
catalog.downloaded_flirt(saved["id"].as_str().unwrap())?.1,
sig
);
assert!(
!catalog.function_reuse("one", "program/functions.json", "1000")?["flirt_candidates"]
.as_array()
.unwrap()
.is_empty()
);
assert!(
catalog
.import_flirt("broken.sig", "x86:LE:64:default", b"IDASGNtruncated")
.is_err()
);
drop(catalog);
let reopened = Catalog::open(&tmp.path().join("catalog.sqlite3"))?;
assert_eq!(
reopened.function_reuse("one", "program/functions.json", "1000")?["seen_in"],
1
);
Ok(())
}
}
src/storage.rs
+54
View File
@@ -26,6 +26,13 @@ pub struct Commit {
pub unique: u64,
pub packed: u64,
}
#[derive(Default)]
pub(crate) struct StorageUsage {
pub chunks: std::collections::BTreeMap<String, (u64, u64)>,
pub snapshots: std::collections::BTreeMap<String, std::collections::BTreeSet<String>>,
pub backend_snapshots: std::collections::BTreeSet<String>,
pub packed_bytes: u64,
}
pub struct RusticStore {
root: PathBuf,
indexed: Mutex<Option<Arc<Repository<IndexedFullStatus>>>>,
@@ -39,6 +46,53 @@ struct DirectoryCache {
nodes: usize,
}
impl RusticStore {
/// Inspect tree/index metadata only; never decode or restore file payloads.
pub(crate) fn usage(&self, snapshots: &[crate::Snapshot]) -> Result<StorageUsage> {
use rustic_core::repofile::{BlobType, IndexFile};
let repo = self.repo()?;
let mut usage = StorageUsage::default();
for file in repo.stream_files::<IndexFile>()? {
for pack in file?.1.packs {
for blob in pack.blobs.into_iter().filter(|b| b.tpe == BlobType::Data) {
usage.chunks.entry(blob.id.to_hex().to_string()).or_insert((
u64::from(blob.location.data_length()),
u64::from(blob.location.length),
));
}
}
}
usage.backend_snapshots = repo
.get_all_snapshots()?
.iter()
.map(|s| s.id.to_hex().to_string())
.collect();
for snapshot in snapshots {
let mut chunks = std::collections::BTreeSet::new();
for entry in snapshot
.entries
.iter()
.filter(|e| e.kind == crate::EntryKind::File)
{
let node = self.node(&repo, &snapshot.backend_snapshot, &entry.path)?;
for id in node.content.unwrap_or_default() {
let key = id.to_hex().to_string();
ensure!(
usage.chunks.contains_key(&key),
"Archived chunk is missing from the index"
);
chunks.insert(key);
}
}
usage.snapshots.insert(snapshot.id.clone(), chunks);
}
for entry in walkdir::WalkDir::new(self.root.join("store/data")) {
let entry = entry?;
if entry.file_type().is_file() {
usage.packed_bytes += entry.metadata()?.len();
}
}
Ok(usage)
}
pub(crate) fn reset_read_cache(&self) -> Result<()> {
*self
.indexed
src/workspace.rs
+434
View File
@@ -0,0 +1,434 @@
//! User-facing working trees. Processing outputs are resolved behind release identities.
use crate::{Archive, Config, EntryKind, Layer, Snapshot};
use anyhow::{Context, Result, ensure};
use serde::{Deserialize, Serialize};
use serde_json::{Value, json};
use std::{
collections::{BTreeMap, BTreeSet},
sync::Arc,
};
pub fn check_scratch(config: &Config) -> Result<()> {
if !config.require_ram_workspace {
return Ok(());
}
#[cfg(target_os = "linux")]
{
use std::os::unix::ffi::OsStrExt;
let path = std::ffi::CString::new(config.workspace.as_os_str().as_bytes())?;
let mut stat = std::mem::MaybeUninit::<nix::libc::statfs>::uninit();
// statfs initializes the struct on success; the pathname remains alive through the call.
ensure!(
unsafe { nix::libc::statfs(path.as_ptr(), stat.as_mut_ptr()) } == 0,
"cannot inspect workspace filesystem"
);
let stat = unsafe { stat.assume_init() };
ensure!(
stat.f_type == 0x01021994 || stat.f_type == 0x858458f6,
"workspace must be on tmpfs/ramfs; configure a RAM-backed path (for example /tmp/verstack-workspace on a tmpfs mount)"
);
ensure!(
fs2::available_space(&config.workspace)? >= config.workspace_bytes,
"RAM workspace has less free space than its configured budget; free scratch space or reduce workspace_bytes"
);
Ok(())
}
#[cfg(not(target_os = "linux"))]
anyhow::bail!("RAM workspace verification is currently supported on Linux")
}
#[derive(Clone, Serialize)]
pub struct WorkspaceIndex {
pub id: String,
pub release: crate::Release,
pub extracted: bool,
pub warnings: Vec<String>,
pub sources: Vec<Value>,
pub code: Vec<Value>,
pub file_snapshot: String,
#[serde(skip)]
pub rows: BTreeMap<String, Value>,
#[serde(skip)]
pub profiles: BTreeMap<String, String>,
}
#[derive(Default, Deserialize)]
pub struct BrowseQuery {
#[serde(default)]
pub before: String,
#[serde(default)]
pub prefix: String,
#[serde(default)]
pub search: String,
#[serde(default)]
pub kind: String,
#[serde(default)]
pub change: String,
#[serde(default)]
pub page: usize,
#[serde(default)]
pub directory: bool,
}
fn common_root(s: &Snapshot) -> String {
let mut paths = s
.entries
.iter()
.filter(|e| e.kind == EntryKind::File && !e.path.ends_with("evidence.json"));
let Some(first) = paths.next() else {
return String::new();
};
let mut prefix: Vec<_> = first.path.split('/').collect();
prefix.pop();
for e in paths {
let p: Vec<_> = e.path.split('/').collect();
let n = prefix
.iter()
.zip(p.iter())
.take_while(|(a, b)| a == b)
.count();
prefix.truncate(n);
}
if prefix.is_empty() {
String::new()
} else {
format!("{}/", prefix.join("/"))
}
}
fn relative<'a>(path: &'a str, prefix: &str) -> &'a str {
path.strip_prefix(prefix).unwrap_or(path)
}
impl Archive {
pub fn workspaces(&self) -> Result<Vec<Value>> {
Ok(self
.catalog
.library()?
.into_iter()
.filter(|s| s["parent"].is_null())
.map(|s| json!({"id":s["id"],"release":s["release"],"imported_at":s["imported_at"]}))
.collect())
}
pub fn workspace(&self, id: &str) -> Result<Arc<WorkspaceIndex>> {
let root = self.catalog.get_snapshot(id)?;
ensure!(
root.parent.is_none(),
"choose a release working tree, not an intermediate processing output"
);
let library = self.catalog.library()?;
let active: BTreeSet<String> = {
let db = self.catalog.db()?;
let mut stmt = db.prepare("SELECT output FROM active_outputs")?;
stmt.query_map([], |r| r.get(0))?
.collect::<std::result::Result<_, _>>()?
};
let signature = format!("{}:{:?}", library.len(), active);
if let Some((saved, index)) = self.workspace_cache.lock().unwrap().get(id) {
if saved == &signature {
return Ok(index.clone());
}
}
let runs = self.catalog.runs()?;
let publication_order: BTreeMap<String, i64> = {
let db = self.catalog.db()?;
let mut stmt = db.prepare("SELECT id,rowid FROM snapshots")?;
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?)))?
.collect::<std::result::Result<_, _>>()?
};
let mut chosen: BTreeMap<(String, String), &Value> = BTreeMap::new();
for s in &library {
let (Some(parent), Some(sid)) = (s["parent"].as_str(), s["id"].as_str()) else {
continue;
};
let Some(run) = runs
.iter()
.find(|r| Some(r.id.as_str()) == s["run"].as_str())
else {
continue;
};
if !matches!(run.state, crate::RunState::Complete) {
continue;
}
let k = (parent.to_owned(), run.operation.clone());
let replace = chosen.get(&k).is_none_or(|old| {
let old_id = old["id"].as_str().unwrap_or("");
if active.contains(sid) != active.contains(old_id) {
active.contains(sid)
} else {
(s["imported_at"].as_u64(), publication_order.get(sid))
> (old["imported_at"].as_u64(), publication_order.get(old_id))
}
});
if replace {
chosen.insert(k, s);
}
}
let mut selected = vec![(root.clone(), 0usize)];
let mut i = 0;
while i < selected.len() {
let (parent, depth) = (selected[i].0.id.clone(), selected[i].1);
for ((p, _), s) in &chosen {
if *p == parent {
selected.push((
self.catalog.get_snapshot(s["id"].as_str().unwrap())?,
depth + 1,
));
}
}
i += 1;
}
let files = selected
.iter()
.filter(|(s, _)| s.layer == Layer::Extracted)
.max_by_key(|(_, d)| *d)
.map(|(s, _)| s)
.unwrap_or(&root);
let prefix = if files.layer == Layer::Extracted {
common_root(files)
} else {
String::new()
};
let mut index = WorkspaceIndex {
id: id.into(),
release: root.release.clone(),
extracted: files.layer == Layer::Extracted,
warnings: vec![],
sources: vec![],
code: vec![],
file_snapshot: files.id.clone(),
rows: BTreeMap::new(),
profiles: BTreeMap::new(),
};
if !index.extracted {
index.warnings.push(
"This release has no extracted filesystem yet. Files shows the imported contents."
.into(),
);
}
for (s, _) in &selected {
let run = runs.iter().find(|r| r.id == s.run);
let operation = run.map(|r| r.operation.as_str()).unwrap_or("import");
index.sources.push(json!({"snapshot":s.id,"operation":operation,"layer":s.layer,"warnings":s.warnings}));
let include_files = s.id == files.id;
let include_media =
operation.contains("media-extract") || operation.contains("node-extract");
if !include_files && !include_media {
continue;
}
if include_media {
index.profiles.insert(
operation.into(),
run.and_then(|r| r.processing.as_ref())
.map(|p| {
format!(
"{}:{}:{}",
p.revision,
p.output_schema,
run.map(|r| r.settings.to_string()).unwrap_or_default()
)
})
.unwrap_or_else(|| "legacy".into()),
);
index.warnings.extend(
s.warnings
.iter()
.filter(|w| !w.starts_with("Plugin coverage:"))
.cloned(),
);
}
let payloads: Vec<Value> = {
let db = self.catalog.db()?;
let mut stmt=db.prepare("SELECT payload FROM artifacts WHERE snapshot=?1 AND coalesce(json_extract(payload,'$.is_preview'),0)=0 AND coalesce(json_extract(payload,'$.source_copy'),0)=0")?;
stmt.query_map([&s.id], |r| r.get::<_, String>(0))?
.map(|r| Ok(serde_json::from_str(&r?)?))
.collect::<Result<_>>()?
};
let identities: BTreeMap<_, _> = s
.entries
.iter()
.map(|e| (e.path.as_str(), e.artifact.clone()))
.collect();
for mut asset in payloads {
if asset["entry"]["kind"] != "file" {
continue;
}
let path = asset["entry"]["path"]
.as_str()
.context("missing path")?
.to_owned();
if operation == "import-extract" && path == "import-evidence.json" {
continue;
}
let mime = asset["media_type"].as_str().unwrap_or("").to_owned();
if !include_files && mime.is_empty() {
continue;
}
let logical = if include_files {
format!("Files/{}", relative(&path, &prefix))
} else {
let provenance = &asset["provenance"];
let source = provenance["source"].as_str().unwrap_or("");
if let Some(entry) = provenance["entry"].as_str() {
format!("Assets/{}/{}", relative(source, &prefix), entry)
} else {
let generated = path.strip_prefix("assets/").unwrap_or(&path);
format!("Assets/{}", relative(generated, &prefix))
}
};
let comparison_identity = asset["provenance"]["original"]
.as_str()
.and_then(|p| identities.get(p))
.and_then(|p| p.as_deref())
.or(asset["entry"]["artifact"].as_str())
.map(str::to_owned);
asset["identity"] = json!(comparison_identity);
asset["path"] = json!(logical);
asset["kind"] = json!(
mime.split('/')
.next()
.filter(|k| ["image", "audio", "video"].contains(k))
.unwrap_or("file")
);
asset["profile"] = json!(if include_media {
operation
} else {
"filesystem"
});
ensure!(
!index.rows.contains_key(&logical),
"ambiguous logical asset path: {logical}"
);
index.rows.insert(logical, asset);
}
}
// Separate selections analyze siblings. Keep all their saved programs,
// choosing the newest export per binary instead of hiding earlier selections.
index.code = self.workspace_code(
&selected
.iter()
.map(|(s, _)| s.id.clone())
.collect::<Vec<_>>(),
&prefix,
)?;
index.warnings.sort();
index.warnings.dedup();
let index = Arc::new(index);
let mut cache = self.workspace_cache.lock().unwrap();
if cache.len() >= 8 {
cache.clear();
}
cache.insert(id.into(), (signature, index.clone()));
Ok(index)
}
pub fn browse_workspace(&self, id: &str, q: &BrowseQuery) -> Result<Value> {
let after = self.workspace(id)?;
let before = if q.before.is_empty() {
None
} else {
Some(self.workspace(&q.before)?)
};
if let Some(b) = &before {
ensure!(
b.release.repository == after.release.repository,
"choose releases of the same game"
);
}
let mut keys: BTreeSet<&String> = after.rows.keys().collect();
if let Some(b) = &before {
keys.extend(b.rows.keys());
}
let mut rows = Vec::new();
let mut counts: BTreeMap<String, usize> = BTreeMap::new();
let search = q.search.to_lowercase();
for key in keys {
let a = after.rows.get(key);
let b = before.as_ref().and_then(|b| b.rows.get(key));
let asset = a.or(b).unwrap();
let profile = asset["profile"].as_str().unwrap_or("");
let comparable = before.as_ref().is_none_or(|b| {
if profile == "filesystem" {
b.extracted == after.extracted
} else {
b.profiles.get(profile).is_some()
&& b.profiles.get(profile) == after.profiles.get(profile)
}
});
let change = if before.is_none() {
""
} else if !comparable {
"unknown"
} else {
match (a, b) {
(Some(a), Some(b)) => {
if a["identity"] == b["identity"] {
"unchanged"
} else {
"changed"
}
}
(Some(_), None) => "added",
_ => "removed",
}
};
if !q.kind.is_empty() && q.kind != "all" && asset["kind"] != q.kind {
continue;
}
if !key.to_lowercase().contains(&search) {
continue;
}
if !q.prefix.is_empty()
&& !key.starts_with(&format!("{}/", q.prefix.trim_end_matches('/')))
{
continue;
}
*counts.entry(change.into()).or_default() += 1;
if !q.change.is_empty()
&& q.change != "all"
&& (if q.change == "differences" {
change == "unchanged" || change.is_empty()
} else {
change != q.change
})
{
continue;
}
let mut row = asset.clone();
row["change"] = json!(change);
row["before"] = json!(b);
row["after"] = json!(a);
rows.push(row);
}
if q.directory {
let prefix = if q.prefix.is_empty() {
String::new()
} else {
format!("{}/", q.prefix.trim_end_matches('/'))
};
let mut children = BTreeMap::new();
for row in rows {
let tail = row["path"].as_str().unwrap().strip_prefix(&prefix).unwrap();
if let Some((folder, _)) = tail.split_once('/') {
let path = format!("{prefix}{folder}");
children
.entry(path.clone())
.or_insert_with(|| json!({"path":path,"name":folder,"directory":true}));
} else {
children.insert(row["path"].as_str().unwrap().into(), row);
}
}
rows = children.into_values().collect();
rows.sort_by_key(|r| {
(
!r["directory"].as_bool().unwrap_or(false),
r["path"].as_str().unwrap_or("").to_owned(),
)
});
}
let total = rows.len();
let size = 120;
let page = q.page.min(total.saturating_sub(1) / size);
let items: Vec<_> = rows.into_iter().skip(page * size).take(size).collect();
Ok(
json!({"items":items,"total":total,"page":page,"page_size":size,"counts":counts,"before":before.as_ref().map(|b|&b.release),"after":after.release,"profiles_compatible":before.as_ref().is_none_or(|b| b.profiles==after.profiles && b.extracted==after.extracted)}),
)
}
}
tests/archive.rs
+2
View File
@@ -8,6 +8,8 @@ fn setup() -> (TempDir, Config) {
let config = Config {
archive: tmp.path().join("archive"),
workspace: tmp.path().join("work"),
require_ram_workspace: false,
processing_enabled: true,
import_roots: vec![tmp.path().join("inputs")],
workspace_bytes: 64 * 1024 * 1024,
bind: "127.0.0.1:0".into(),
tests/catalog_jobs.rs
+2
View File
@@ -7,6 +7,8 @@ fn setup() -> anyhow::Result<(tempfile::TempDir, Config)> {
let config = Config {
archive: root.join("archive"),
workspace: root.join("work"),
require_ram_workspace: false,
processing_enabled: true,
import_roots: vec![root.join("inputs")],
workspace_bytes: 64 * 1024 * 1024,
bind: "127.0.0.1:0".into(),
tests/catalog_management.rs
+239
View File
@@ -0,0 +1,239 @@
use std::{collections::BTreeMap, fs};
use verstack::{
Archive, Config, Release,
catalog_management::DeleteScope,
imports::{ImportOptions, ImportRequest, ImportSource},
};
fn setup() -> anyhow::Result<(tempfile::TempDir, Config)> {
let t = tempfile::tempdir()?;
fs::create_dir(t.path().join("inputs"))?;
let c = Config {
archive: t.path().join("archive"),
workspace: t.path().join("ram"),
import_roots: vec![t.path().join("inputs")],
workspace_bytes: 64 * 1024 * 1024,
require_ram_workspace: false,
processing_enabled: false,
bind: "127.0.0.1:0".into(),
plugins: BTreeMap::new(),
};
Ok((t, c))
}
fn release(game: &str, version: &str) -> Release {
Release {
repository: game.into(),
version: version.into(),
edition: "LE".into(),
generation: "3".into(),
released_at: None,
}
}
fn scope(game: &str, version: Option<&str>) -> DeleteScope {
DeleteScope {
repository: game.into(),
version: version.map(str::to_owned),
edition: version.map(|_| "LE".into()),
generation: version.map(|_| "3".into()),
}
}
fn bytes(seed: u64, n: usize) -> Vec<u8> {
let mut v = seed;
(0..n)
.map(|_| {
v ^= v << 13;
v ^= v >> 7;
v ^= v << 17;
v as u8
})
.collect()
}
#[test]
fn release_deletion_and_collection_preserve_other_games_and_versions() -> anyhow::Result<()> {
let (t, c) = setup()?;
let input = t.path().join("inputs");
let shared = bytes(737, 2 * 1024 * 1024);
for name in ["a1", "a2", "b"] {
fs::create_dir(input.join(name))?;
fs::write(input.join(name).join("shared.bin"), &shared)?;
fs::write(
input.join(name).join("unique.bin"),
bytes(
name.as_bytes().iter().map(|b| *b as u64).sum::<u64>() + 7,
128 * 1024,
),
)?;
}
let a = Archive::open(c.clone())?;
let a1 = a.import(&input.join("a1"), release("A", "1"))?;
let a2 = a.import(&input.join("a2"), release("A", "2"))?;
let b = a.import(&input.join("b"), release("B", "1"))?;
let stats = a.catalog_overview(true)?;
assert_eq!(stats["totals"]["releases"], 3);
assert!(stats["totals"]["exact_duplicate_bytes"].as_u64().unwrap() >= 2 * shared.len() as u64);
let plan = a.preview_delete(&scope("A", Some("1")))?;
assert!(plan["shared_data_preserved_bytes"].as_u64().unwrap() > 0);
assert!(plan["eligible_for_gc_bytes"].as_u64().unwrap() > 0);
a.delete_cataloged(&scope("A", Some("1")), plan["token"].as_str().unwrap())?;
assert!(a.snapshot(&a1.id).is_err());
assert!(input.join("a1/shared.bin").exists());
let dangling = a.catalog_overview(true)?;
assert!(
dangling["totals"]["storage"]["unreferenced_snapshots"]
.as_u64()
.unwrap()
> 0
);
assert!(
dangling["totals"]["storage"]["unreferenced_data_bytes"]
.as_u64()
.unwrap()
> 0
);
a.cleanup_orphans()?;
a.verify(&a2.id)?;
a.verify(&b.id)?;
assert_eq!(a.read_range(&b.id, "shared.bin", 0, shared.len())?, shared);
let plan = a.preview_delete(&scope("A", None))?;
a.delete_cataloged(&scope("A", None), plan["token"].as_str().unwrap())?;
a.cleanup_orphans()?;
a.verify(&b.id)?;
assert_eq!(a.catalog_overview(true)?["totals"]["releases"], 1);
drop(a);
let reopened = Archive::open(c)?;
reopened.verify(&b.id)?;
assert_eq!(reopened.snapshots()?.len(), 1);
Ok(())
}
#[test]
fn deletion_review_detects_changes_rejects_active_work_and_recovers() -> anyhow::Result<()> {
let (t, c) = setup()?;
let file = t.path().join("inputs/one.bin");
fs::write(&file, bytes(5, 2048))?;
let a = Archive::open(c.clone())?;
let original = a.import(&file, release("A", "1"))?;
let manifest = fs::read(
c.archive
.join("snapshots")
.join(format!("{}.json", original.id)),
)?;
let selection = scope("A", Some("1"));
let stale = a.preview_delete(&selection)?;
let other = a.import(&file, release("B", "1"))?;
assert!(
a.delete_cataloged(&selection, stale["token"].as_str().unwrap())
.is_err()
);
let task = a.submit_import(ImportRequest {
source: ImportSource::Local {
paths: vec![file.clone()],
},
release: release("A", "1"),
options: ImportOptions {
extract: false,
media: false,
analyze: false,
},
})?;
assert!(a.preview_delete(&selection).is_err());
assert!(a.cleanup_orphans().is_err());
a.control_import(&task.id, "cancel")?;
// A version's failed/canceled imports can be older than the activity page.
for _ in 0..201 {
let recent = a.submit_import(ImportRequest {
source: ImportSource::Local {
paths: vec![file.clone()],
},
release: release("B", "1"),
options: ImportOptions {
extract: false,
media: false,
analyze: false,
},
})?;
a.control_import(&recent.id, "cancel")?;
}
let plan = a.preview_delete(&selection)?;
a.delete_cataloged(&selection, plan["token"].as_str().unwrap())?;
let db = rusqlite::Connection::open(c.archive.join("catalog.sqlite3"))?;
let remaining: i64 = db.query_row(
"SELECT count(*) FROM imports WHERE id=?1",
[&task.id],
|r| r.get(0),
)?;
assert_eq!(remaining, 0);
drop(db);
drop(a);
// Replay a committed deletion with an incompletely removed manifest.
fs::write(
c.archive
.join("snapshots")
.join(format!("{}.json", original.id)),
manifest,
)?;
let journal = fs::read_dir(c.archive.join("deletions"))?
.next()
.unwrap()?
.path();
let mut record: serde_json::Value = serde_json::from_slice(&fs::read(&journal)?)?;
record["state"] = "committed".into();
fs::write(journal, serde_json::to_vec(&record)?)?;
let a = Archive::open(c.clone())?;
assert!(a.snapshot(&original.id).is_err());
assert_eq!(a.snapshots()?.len(), 1);
a.verify(&other.id)?;
drop(a);
// Rebuilding the index must never resurrect deleted versions.
fs::remove_file(c.archive.join("catalog.sqlite3"))?;
let a = Archive::open(c)?;
assert_eq!(a.catalog_overview(false)?["totals"]["releases"], 1);
a.verify(&other.id)?;
Ok(())
}
#[test]
fn deleting_a_baseline_removes_dependent_report_but_keeps_target() -> anyhow::Result<()> {
let (t, c) = setup()?;
let file = t.path().join("inputs/functions.json");
let facts = serde_json::json!({"schema":1,"processing_revision":1,"input_sha256":"test","language":"x86","compiler":"gcc","ghidra_version":"test","analysis_timed_out":false,"settings":{},"functions":[]});
fs::write(&file, serde_json::to_vec(&facts)?)?;
let a = Archive::open(c)?;
let before = a.import(&file, release("A", "1"))?;
let after = a.import(&file, release("B", "1"))?;
let report =
a.save_function_comparison(&before.id, "functions.json", &after.id, "functions.json")?;
let plan = a.preview_delete(&scope("A", None))?;
assert_eq!(plan["dependent_comparisons"], 1);
a.delete_cataloged(&scope("A", None), plan["token"].as_str().unwrap())?;
assert!(a.snapshot(&report.id).is_err());
a.cleanup_orphans()?;
a.verify(&after.id)?;
Ok(())
}
#[test]
fn collection_of_last_release_and_active_reader_protection() -> anyhow::Result<()> {
let (t, c) = setup()?;
let file = t.path().join("inputs/image.png");
let data = bytes(555, 256 * 1024);
fs::write(&file, &data)?;
let a = Archive::open(c)?;
let root = a.import(&file, release("A", "1"))?;
let inventory = a.catalog_files("image", "A", usize::MAX)?;
assert_eq!(inventory["page"], 0);
assert_eq!(inventory["items"][0]["path"], "image.png");
assert_eq!(inventory["items"][0]["kind"], "image");
let reader = a.reader(&root.id, "image.png")?;
let review = a.preview_delete(&scope("A", None))?;
a.delete_cataloged(&scope("A", None), review["token"].as_str().unwrap())?;
assert!(
a.cleanup_orphans().is_err(),
"streaming readers must veto physical collection"
);
drop(reader);
let cleanup = a.cleanup_orphans()?;
assert!(cleanup["reclaimed_packed_bytes"].as_u64().unwrap() > 0);
let empty = a.catalog_overview(true)?;
assert_eq!(empty["totals"]["releases"], 0);
assert_eq!(empty["totals"]["storage"]["unreferenced_snapshots"], 0);
assert_eq!(empty["totals"]["storage"]["unreferenced_data_bytes"], 0);
Ok(())
}
tests/imports.rs
+194
View File
@@ -0,0 +1,194 @@
use std::{
collections::BTreeMap,
fs,
path::Path,
sync::Arc,
time::{Duration, Instant},
};
use tempfile::TempDir;
use verstack::{
Archive, Config, PluginConfig, Release,
imports::{ImportOptions, ImportRequest, ImportSource},
};
fn setup() -> (TempDir, Config) {
let tmp = tempfile::tempdir().unwrap();
fs::create_dir(tmp.path().join("input")).unwrap();
let cfg = Config {
archive: tmp.path().join("archive"),
workspace: tmp.path().join("ram"),
require_ram_workspace: false,
processing_enabled: false,
import_roots: vec![tmp.path().join("input")],
workspace_bytes: 16 * 1024 * 1024,
bind: "127.0.0.1:0".into(),
plugins: BTreeMap::new(),
};
(tmp, cfg)
}
fn request(path: &Path, extract: bool) -> ImportRequest {
ImportRequest {
source: ImportSource::Local {
paths: vec![path.to_owned()],
},
release: Release {
repository: "Fixture".into(),
edition: "LE".into(),
version: "0.81.0".into(),
generation: String::new(),
released_at: None,
},
options: ImportOptions {
extract,
media: false,
analyze: false,
},
}
}
fn plugin(path: &Path) -> PluginConfig {
PluginConfig {
command: vec!["python3".into(), path.to_string_lossy().into_owned()],
version: "test/1".into(),
settings: serde_json::json!({}),
timeout_seconds: 60,
processing_revision: 1,
output_schema: 1,
source_digest: String::new(),
source_root: None,
auto_extensions: vec![],
}
}
#[test]
fn browse_multi_file_import_works_with_automatic_jobs_paused() -> anyhow::Result<()> {
let (_tmp, cfg) = setup();
let input = cfg.import_roots[0].clone();
fs::write(input.join("one.gba"), b"rom one")?;
fs::write(input.join("two.txt"), b"text two")?;
fs::create_dir(input.join("folder"))?;
std::os::unix::fs::symlink("/etc", input.join("outside"))?;
let a = Archive::open(cfg)?;
let listing = a.browse_import_source(Some(&input), "", 0)?;
assert_eq!(listing["total"], 3);
assert_eq!(listing["items"][0]["name"], "folder");
assert!(
a.browse_import_source(Some(Path::new("/etc")), "", 0)
.is_err()
);
let mut req = request(&input.join("one.gba"), false);
req.source = ImportSource::Local {
paths: vec![input.join("one.gba"), input.join("two.txt")],
};
let task = a.submit_import(req)?;
assert!(a.work_import_one()?);
assert!(a.jobs()?.is_empty());
assert!(a.work_one()?.is_none());
let saved = a.import_tasks()?.pop().unwrap();
assert_eq!(saved.id, task.id);
assert_eq!(saved.state, "completed");
let root = saved.root.unwrap();
assert_eq!(a.read_range(&root, "one.gba", 0, 100)?, b"rom one");
assert_eq!(a.read_range(&root, "two.txt", 0, 100)?, b"text two");
Ok(())
}
#[test]
fn retry_reuses_retained_original_and_pending_cancel_survives_restart() -> anyhow::Result<()> {
let (tmp, mut cfg) = setup();
let input = cfg.import_roots[0].join("rom.bin");
fs::write(&input, b"original")?;
let script = tmp.path().join("plugin.py");
let flag = tmp.path().join("ready");
fs::write(
&script,
format!(
"import json,pathlib,sys\nr=json.loads(pathlib.Path(sys.argv[1]).read_text())\nassert pathlib.Path({:?}).exists(),'retry me'\nout=pathlib.Path(r['output_dir'])/'asset.txt';out.write_text('recovered')\npathlib.Path(r['result_file']).write_text(json.dumps(dict(protocol=1,layer='extracted',coverage='complete',warnings=[],files=['asset.txt'])))\n",
flag.to_string_lossy()
),
)?;
cfg.plugins.insert("import-extract".into(), plugin(&script));
let a = Archive::open(cfg.clone())?;
let task = a.submit_import(request(&input, true))?;
a.work_import_one()?;
let failed = a.import_tasks()?.pop().unwrap();
assert_eq!(failed.state, "failed");
let root = failed.root.clone().unwrap();
fs::remove_file(&input)?;
fs::write(flag, b"yes")?;
a.control_import(&task.id, "retry")?;
a.work_import_one()?;
let completed = a.import_tasks()?.pop().unwrap();
assert_eq!(completed.state, "completed");
assert_eq!(completed.root.as_deref(), Some(root.as_str()));
assert_eq!(a.snapshots()?.len(), 2);
fs::write(&input, b"again")?;
let pending = a.submit_import(request(&input, false))?;
a.control_import(&pending.id, "cancel")?;
drop(a);
let reopened = Archive::open(cfg)?;
assert!(!reopened.work_import_one()?);
assert!(
reopened
.import_tasks()?
.iter()
.any(|t| t.id == pending.id && t.state == "cancelled")
);
Ok(())
}
#[test]
fn running_import_reports_progress_and_cancels_process_group() -> anyhow::Result<()> {
let (tmp, mut cfg) = setup();
let input = cfg.import_roots[0].join("rom.bin");
fs::write(&input, b"original")?;
let script = tmp.path().join("slow.py");
fs::write(
&script,
"import os,json,pathlib,time,subprocess\np=pathlib.Path(os.environ['VERSTACK_PROGRESS_FILE']);p.write_text(json.dumps(dict(stage='Exporting functions',completed=4,total=10,unit='functions')))\nsubprocess.Popen(['sleep','60'])\ntime.sleep(60)\n",
)?;
cfg.plugins.insert("import-extract".into(), plugin(&script));
let a = Arc::new(Archive::open(cfg)?);
let task = a.submit_import(request(&input, true))?;
let worker = a.clone();
let handle = std::thread::spawn(move || worker.work_import_one());
let start = Instant::now();
loop {
let view = a.activity_view()?;
if view["items"]
.as_array()
.unwrap()
.iter()
.any(|t| t["progress"]["completed"] == 4)
{
break;
}
assert!(start.elapsed() < Duration::from_secs(8));
std::thread::sleep(Duration::from_millis(50));
}
a.control_import(&task.id, "cancel")?;
assert!(handle.join().unwrap()?);
let task = a.import_tasks()?.pop().unwrap();
assert_eq!(task.state, "cancelled");
assert!(task.root.is_some());
assert!(start.elapsed() < Duration::from_secs(10));
Ok(())
}
#[test]
fn sparse_import_keeps_exact_bytes_beyond_physical_ram_budget() -> anyhow::Result<()> {
use std::io::{Seek, SeekFrom, Write};
let (_tmp, cfg) = setup();
let path = cfg.import_roots[0].join("card.raw");
let mut file = fs::File::create(&path)?;
file.write_all(b"header")?;
file.seek(SeekFrom::Start(20 * 1024 * 1024))?;
file.write_all(b"end")?;
drop(file);
let a = Archive::open(cfg)?;
a.submit_import(request(&path, false))?;
a.work_import_one()?;
let task = a.import_tasks()?.pop().unwrap();
assert_eq!(task.state, "completed");
let root = task.root.unwrap();
assert_eq!(
a.read_range(&root, "card.raw", 20 * 1024 * 1024, 3)?,
b"end"
);
assert_eq!(a.read_range(&root, "card.raw", 1024, 16)?, vec![0; 16]);
Ok(())
}
tests/test_import_sources.py
+99
View File
@@ -0,0 +1,99 @@
import hashlib
import http.server
import io
import json
import pathlib
import socketserver
import struct
import subprocess
import sys
import tempfile
import threading
import unittest
import zipfile
from unittest.mock import patch
sys.path.insert(0,str(pathlib.Path(__file__).resolve().parents[1]/'plugins'))
import import_extract
import import_source
import fat_files
class Extraction(unittest.TestCase):
def setUp(self):
self.temp=tempfile.TemporaryDirectory();self.addCleanup(self.temp.cleanup);self.root=pathlib.Path(self.temp.name)
self.source=self.root/'input';self.source.mkdir();self.output=self.root/'output';self.output.mkdir()
def run_extract(self):
return import_extract.run({'protocol':1,'input_dir':str(self.source),'output_dir':str(self.output),'workspace_bytes':128*1024**2,'settings':{}})
def test_nested_zip_and_unknown_rom(self):
inner=io.BytesIO()
with zipfile.ZipFile(inner,'w') as z:z.writestr('art/logo.png',b'original pixels');z.writestr('game.gba',b'rom bytes')
with zipfile.ZipFile(self.source/'release.zip','w') as z:z.writestr('sdcard.zip',inner.getvalue())
result=self.run_extract()
self.assertEqual((self.output/'game.gba').read_bytes(),b'rom bytes')
self.assertEqual((self.output/'art/logo.png').read_bytes(),b'original pixels')
self.assertIn('import-evidence.json',result['files'])
def test_traversal_and_duplicate_names_rejected(self):
for name in ('../escape','/absolute','folder/../../escape','back\\slash'):
with self.subTest(name=name):
with zipfile.ZipFile(self.source/'bad.zip','w') as z:z.writestr(name,b'bad')
with self.assertRaises(ValueError):self.run_extract()
self.assertFalse((self.root/'escape').exists())
def test_sparse_image_uses_allocated_budget(self):
target=self.root/'sparse.raw';data=b'nonzero'+bytes(4*1024**2-7)
import_extract.sparse_copy(io.BytesIO(data),target,len(data),1024**2,'fixture')
self.assertEqual(target.stat().st_size,len(data));self.assertLessEqual(target.stat().st_blocks*512,1024**2)
self.assertEqual(hashlib.sha256(target.read_bytes()).digest(),hashlib.sha256(data).digest())
with self.assertRaises(ValueError):import_extract.sparse_copy(io.BytesIO(b'x'*2048),self.root/'too-big',2048,1024,'fixture')
def test_mbr_ext_image_inside_zip(self):
files=self.root/'files';files.mkdir();(files/'new.txt').write_text('new asset');(files/'link').symlink_to('/etc/passwd')
image=self.root/'fs.img'
subprocess.run(['truncate','-s','16M',str(image)],check=True)
subprocess.run(['/usr/sbin/mke2fs','-q','-t','ext4','-F','-d',str(files),str(image)],check=True)
raw=self.root/'card.raw';header=bytearray(512);header[510:]=b'\x55\xaa';header[450]=0x83;struct.pack_into('<II',header,454,2048,image.stat().st_size//512)
with raw.open('wb') as f:f.write(header);f.seek(1024**2);f.write(image.read_bytes())
with zipfile.ZipFile(self.source/'release.zip','w',compression=zipfile.ZIP_DEFLATED) as z:z.write(raw,'version-0.81.raw')
result=self.run_extract();self.assertEqual((self.output/'partition-01/new.txt').read_text(),'new asset');self.assertFalse((self.output/'partition-01/link').exists());self.assertEqual(result['coverage'],'partial')
def test_bad_partition_bounds(self):
data=bytearray(512);data[510:]=b'\x55\xaa';data[450]=0x83;struct.pack_into('<II',data,454,2048,500)
image=self.root/'bad.raw';image.write_bytes(data)
with self.assertRaisesRegex(ValueError,'bounds'):import_extract.partitions(image)
def test_fat12(self):
image=bytearray(2880*512);image[:3]=b'\xeb\x3c\x90';struct.pack_into('<H',image,11,512);image[13]=1;struct.pack_into('<H',image,14,1);image[16]=2;struct.pack_into('<H',image,17,224);struct.pack_into('<H',image,19,2880);image[21]=0xf0;struct.pack_into('<H',image,22,9);image[54:62]=b'FAT12 ';image[510:512]=b'\x55\xaa'
for sector in (1,10):image[sector*512:sector*512+5]=b'\xf0\xff\xff\xff\x0f'
root=19*512;image[root:root+11]=b'README TXT';image[root+11]=0x20;struct.pack_into('<H',image,root+26,2);struct.pack_into('<I',image,root+28,5);image[33*512:33*512+5]=b'hello'
path=self.source/'floppy.img';path.write_bytes(image);self.run_extract();self.assertEqual((self.output/'README.TXT').read_bytes(),b'hello')
class Sources(unittest.TestCase):
def setUp(self):
class Handler(http.server.BaseHTTPRequestHandler):
def log_message(self,*args):pass
def do_HEAD(self):self.do_GET(True)
def do_GET(self,head=False):
if self.path=='/redirect':self.send_response(302);self.send_header('Location','/file.bin');self.end_headers();return
payload=b'content'*500000 if self.path=='/large' else b'hello world'
self.send_response(200)
if self.path!='/unknown':self.send_header('Content-Length',str(len(payload)+20 if self.path=='/truncated' else len(payload)))
self.send_header('Content-Type','text/html' if self.path=='/page' else 'application/octet-stream');self.end_headers()
if not head:
try:self.wfile.write(payload)
except (BrokenPipeError,ConnectionResetError):pass
self.server=socketserver.TCPServer(('127.0.0.1',0),Handler);thread=threading.Thread(target=self.server.serve_forever,daemon=True);thread.start()
self.addCleanup(self.server.server_close);self.addCleanup(self.server.shutdown)
self.url=f'http://127.0.0.1:{self.server.server_address[1]}'
self.temp=tempfile.TemporaryDirectory();self.addCleanup(self.temp.cleanup);self.root=pathlib.Path(self.temp.name)
def download(self,path,limit=1024):return import_source.download({'url':self.url+path,'allow_private':True,'directory':str(self.root),'limit':limit,'progress_file':str(self.root/'progress.json')})
def test_redirect_and_receipt(self):
result=self.download('/redirect');self.assertEqual(result['sha256'],hashlib.sha256(b'hello world').hexdigest());self.assertEqual(pathlib.Path(result['path']).read_bytes(),b'hello world');self.assertEqual(json.loads((self.root/'progress.json').read_text())['completed'],11)
def test_unknown_size(self):self.assertEqual(self.download('/unknown')['bytes'],11)
def test_html_and_oversized_and_truncated(self):
for path in ('/page','/large','/truncated'):
with self.subTest(path=path),self.assertRaises((ValueError,http.client.IncompleteRead)):self.download(path)
def test_private_addresses_blocked(self):
with self.assertRaisesRegex(ValueError,'public internet'):import_source.inspect(self.url+'/file.bin')
def test_ia_metadata_picker(self):
class Response:
def read(self,limit):return json.dumps({'metadata':{'title':'Example release'},'files':[{'name':'card.raw.zip','source':'original','size':'1234'},{'name':'item_meta.xml','source':'original'},{'name':'derived.mp4','source':'derivative'}]}).encode()
class Connection:
def close(self):pass
with patch.object(import_source,'connect',return_value=(Connection(),Response(),'')):
result=import_source.inspect('https://archive.org/details/example');self.assertEqual(len(result['files']),1);self.assertEqual(result['files'][0]['size'],1234)
if __name__=='__main__':unittest.main()
tests/test_parallel_media.py
+57
View File
@@ -0,0 +1,57 @@
import hashlib,json,pathlib,struct,tempfile,unittest
from unittest.mock import patch
import media_extract,media_preview,parallelism
from PIL import Image
def inventory(root):return {p.relative_to(root).as_posix():hashlib.sha256(p.read_bytes()).hexdigest() for p in root.rglob('*') if p.is_file()}
class ParallelMediaTests(unittest.TestCase):
def test_ordered_delta_chain_and_failed_frames_with_parallel_png(self):
count=96;head=[0]*7;head[3]=56;head[2]=56+count*8;data=bytearray(head[2]);struct.pack_into('<7Q',data,0,*head)
for i in range(count):
data.extend(bytes(-len(data)%8));struct.pack_into('<Q',data,56+i*8,len(data))
mode,payload=(0,bytes(range(64))) if i==0 else (9,b'\xc0\0')
if i==count-1:mode,payload=2,b'\0'
data.extend(struct.pack('<IIHHB',i,4,8,8,mode)+payload)
with tempfile.TemporaryDirectory() as tmp:
for budget in [2**20,600]:
reference=None
for workers in [1,4]:
root=pathlib.Path(tmp)/f'{budget}-{workers}';root.mkdir();out=media_extract.Output(root,budget)
with patch('media_extract.worker_count',return_value=workers):media_extract.extract_bitmaps(data,out,'dmd','game',{'cpu_threads':workers})
current=(out.records,inventory(root),out.left)
if reference is None:reference=current
else:self.assertEqual(current,reference)
self.assertGreaterEqual(out.left,0)
self.assertEqual(out.records[-1]['status'],'coverage_failure')
def test_thumbnail_parallel_output_order_and_shared_budget(self):
with tempfile.TemporaryDirectory() as tmp:
root=pathlib.Path(tmp);source=root/'input';source.mkdir()
for i in range(12):Image.new('RGB',(100+i,80),(i*16,128,255)).save(source/f'{i:02}.png')
(source/'13-broken.png').write_bytes(b'\x89PNG\r\n\x1a\ntruncated')
for budget in [1024,1024*1024]:
reference=None
for workers in [1,4]:
target=root/f'{budget}-{workers}';target.mkdir()
with patch('media_preview.worker_count',return_value=workers):media_preview.run({'input_dir':str(source),'output_dir':str(target),'workspace_bytes':budget})
files=inventory(target)
if reference is None:reference=files
else:self.assertEqual(files,reference)
self.assertLess(sum(p.stat().st_size for p in target.rglob('*.webp')),budget//2)
def test_resources_reserve_service_cpus_and_bound_memory(self):
with patch('parallelism.available_cpus',return_value=54),patch('parallelism.cgroup_dirs',return_value=[]):
self.assertEqual(parallelism.cpu_threads({}),52)
self.assertEqual(parallelism.cpu_threads({'cpu_threads':80}),54)
self.assertEqual(parallelism.worker_count({'workspace_bytes':4*1024**3},100),4)
self.assertEqual(parallelism.worker_count({'workspace_bytes':4*1024**3},2),2)
with self.assertRaises(ValueError):parallelism.cpu_threads({'cpu_threads':-1})
def test_cgroup_cpu_and_memory_limits(self):
files={'/sys/fs/cgroup/task/cpu.max':'200000 100000','/sys/fs/cgroup/task/memory.max':str(5*1024**3),'/sys/fs/cgroup/task/memory.current':str(1024**3),'/proc/meminfo':'MemAvailable: 100000000 kB'}
def read(path,*args,**kwargs):
if str(path) not in files:raise FileNotFoundError(path)
return files[str(path)]
with patch('parallelism.os.sched_getaffinity',return_value=set(range(54))),patch('parallelism.cgroup_dirs',return_value=[pathlib.Path('/sys/fs/cgroup/task')]),patch('parallelism.pathlib.Path.read_text',read):
self.assertEqual(parallelism.available_cpus(),2)
self.assertEqual(parallelism.worker_count({'workspace_bytes':64*1024**3},100),1)
files['/sys/fs/cgroup/task/cpu.max']='max 100000'
self.assertEqual(parallelism.worker_count({'workspace_bytes':64*1024**3},100),4)
if __name__=='__main__':unittest.main()
tests/test_ram_workspace.py
+25
View File
@@ -0,0 +1,25 @@
import pathlib
import sys
import tempfile
import unittest
from unittest.mock import patch
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1] / 'scripts'))
from ram_workspace import ram_workspace, scratch_environment
class RamScratchTests(unittest.TestCase):
def test_child_environment_uses_run_directory(self):
with tempfile.TemporaryDirectory() as root:
env = scratch_environment(root)
self.assertEqual(env['TMPDIR'], str(pathlib.Path(root).resolve() / 'tmp'))
self.assertEqual(env['TMPDIR'], env['TEMP'])
self.assertEqual(env['PYTHONDONTWRITEBYTECODE'], '1')
def test_disk_backed_path_is_rejected(self):
with tempfile.TemporaryDirectory() as root, patch.object(pathlib.Path, 'read_text', return_value='1 0 0:1 / / rw - ext4 /dev/disk rw'):
with self.assertRaisesRegex(ValueError, 'tmpfs/ramfs'):
ram_workspace({'workspace':root,'workspace_bytes':1024})
def test_budget_cannot_overrun_available_ram(self):
with tempfile.TemporaryDirectory() as root, patch.object(pathlib.Path, 'read_text', return_value='1 0 0:1 / / rw - tmpfs none rw'):
with self.assertRaisesRegex(ValueError, 'free RAM'):
ram_workspace({'workspace':root,'workspace_bytes':2**64})
tests/ui-catalog-management.mjs
+88
View File
@@ -0,0 +1,88 @@
// Starts its own disposable archive. Every API request is redirected there;
// no deletion, collection, import or analysis is sent to the live workbench API.
import {createRequire} from 'node:module';
import {mkdtemp, mkdir, writeFile} from 'node:fs/promises';
import {tmpdir} from 'node:os';
import {join, resolve} from 'node:path';
import {randomBytes} from 'node:crypto';
import {spawn, spawnSync} from 'node:child_process';
import {createServer} from 'node:net';
import {once} from 'node:events';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);
const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const root=await mkdtemp(join(tmpdir(),'verstack-catalog-ui-'));
const inputs=join(root,'inputs');await mkdir(inputs);
const socket=createServer();socket.listen(0,'127.0.0.1');await once(socket,'listening');
const port=socket.address().port;await new Promise(resolve=>socket.close(resolve));
const base=`http://127.0.0.1:${port}`;
const config=join(root,'config.json');
await writeFile(config,JSON.stringify({archive:join(root,'archive'),workspace:join(root,'ram'),import_roots:[inputs],workspace_bytes:64*1024**2,require_ram_workspace:true,processing_enabled:false,bind:`127.0.0.1:${port}`,plugins:{}}));
const executable=resolve(process.env.VERSTACK_TEST_BINARY||'target/debug/verstack');
const shared=randomBytes(1024*1024),roots=[];
for(const [index,game,version] of [[0,'Catalog fixture A','1'],[1,'Catalog fixture A','2'],[2,'Catalog fixture B','1']]){
const directory=join(inputs,String(index));await mkdir(directory);
await writeFile(join(directory,'shared.bin'),shared);await writeFile(join(directory,'unique.bin'),randomBytes(256*1024));
await writeFile(join(directory,'preview.png'),Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aShcAAAAASUVORK5CYII=','base64'));
if(index===0){for(let n=0;n<110;n++){await writeFile(join(directory,`note-${String(n).padStart(3,'0')}.txt`),`Pagination sample ${n}\n`);}}
const imported=spawnSync(executable,['--config',config,'import',directory,'--repository',game,'--version',version,'--edition','LE','--generation','3'],{encoding:'utf8'});
assert.equal(imported.status,0,imported.stderr);roots.push(JSON.parse(imported.stdout).id);
}
const backend=spawn(executable,['--config',config,'serve'],{stdio:['ignore','pipe','pipe']});
let backendLog='';backend.stderr.on('data',chunk=>backendLog+=chunk);
let browser;
try{
for(let n=0;;n++){
try{const response=await fetch(base+'/api/catalog');assert(response.ok);break;}
catch(error){if(n>=100||backend.exitCode!==null){throw new Error(`Fixture did not start: ${backendLog}`,{cause:error});}await new Promise(resolve=>setTimeout(resolve,100));}
}
const initial=await (await fetch(base+'/api/catalog?measure=true')).json();assert.equal(initial.totals.releases,3);assert(initial.totals.exact_duplicate_bytes>=2*shared.length);
browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const page=await browser.newPage({viewport:{width:1720,height:1080}}),errors=[],writes=[];
page.on('pageerror',e=>errors.push(e.message));
await page.route('**/api/**',async route=>{
const request=route.request(),url=new URL(request.url());url.host=new URL(base).host;url.protocol='http:';
if(request.method()!=='GET'){
assert.equal(request.method(),'POST');assert(['/api/catalog/delete','/api/catalog/collect'].includes(url.pathname),'Unexpected mutation');
if(url.pathname.endsWith('/delete')){assert.equal(request.postDataJSON().scope.repository,'Catalog fixture A');}
writes.push(url.pathname);
}
const response=await route.fetch({url:url.href,headers:{...request.headers(),host:url.host,origin:base}});await route.fulfill({response});
});
await page.goto(process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3000');
await page.getByRole('button',{name:'Catalog & storage',exact:true}).waitFor({timeout:90000});await page.getByRole('button',{name:'Catalog & storage',exact:true}).click();
const view=page.locator('#verstack\\.catalog');
await page.waitForFunction(()=>document.querySelectorAll('#verstack\\.catalog .vs-catalog-table tbody tr').length===3);
await view.getByRole('button',{name:'Measure storage',exact:true}).click();await view.getByRole('columnheader',{name:'Shared with other releases',exact:true}).waitFor();
await page.screenshot({path:join(root,'catalog-storage.png')});
await view.getByRole('button',{name:'All cataloged files',exact:true}).click();const pager=view.getByLabel('Catalog files page',{exact:true});
await page.waitForFunction(()=>document.querySelector('[aria-label="Catalog files page"]')?.disabled===false);
await pager.fill('2');await pager.press('Enter');await page.waitForFunction(()=>document.querySelector('[aria-label="Catalog files page"]')?.value==='2'&&!document.querySelector('[aria-label="Catalog files page"]')?.disabled);
assert.equal(await view.locator('.vs-table tbody tr').count(),19);
await view.getByLabel('Filter catalog').fill('preview.png');await page.waitForFunction(()=>document.querySelectorAll('#verstack\\.catalog .vs-table tbody tr').length===3);
await view.getByRole('button',{name:'preview.png',exact:true}).first().click();await page.locator('.vs-media-editor img').waitFor();
await page.waitForFunction(()=>Array.from(document.querySelectorAll('.vs-media-editor img')).some(img=>img.complete&&img.naturalWidth>0));
await page.locator('.lm-TabBar-tabLabel').filter({hasText:/^Catalog & storage$/}).first().click();
await view.getByLabel('Filter catalog').fill('');await view.getByRole('button',{name:'Releases',exact:true}).click();
const row=view.locator('.vs-catalog-table tbody tr').filter({hasText:'Catalog fixture A'}).filter({hasText:'LE / 1'});
await row.getByRole('button',{name:'Delete version…',exact:true}).click();await page.getByRole('button',{name:'Keep files',exact:true}).waitFor();
await page.screenshot({path:join(root,'delete-review.png')});await page.getByRole('button',{name:'Keep files',exact:true}).click();assert.equal(writes.length,0,'Cancel must preserve the catalog');
await row.getByRole('button',{name:'Delete version…',exact:true}).click();await page.getByRole('button',{name:'Delete version',exact:true}).click();
await page.waitForFunction(()=>document.querySelectorAll('#verstack\\.catalog .vs-catalog-table tbody tr').length===2);
assert.equal((await (await fetch(base+'/api/catalog')).json()).totals.releases,2);
await view.getByLabel('Catalog game',{exact:true}).selectOption('Catalog fixture A');
await view.getByRole('button',{name:'Delete all versions of Catalog fixture A…',exact:true}).click();await page.getByRole('button',{name:'Delete entire game',exact:true}).click();
await view.getByText('Deleted all versions of Catalog fixture A.',{exact:false}).waitFor();
await page.waitForFunction(()=>!document.querySelector('#verstack\\.catalog .vs-catalog-progress'));
assert.equal((await (await fetch(base+'/api/catalog')).json()).totals.releases,1);
await view.getByRole('button',{name:'Garbage collect…',exact:true}).click();await page.getByRole('button',{name:'Collect unreferenced data',exact:true}).click();
await view.getByText('Garbage collection reclaimed',{exact:false}).waitFor({timeout:60000});
const after=await (await fetch(base+'/api/catalog?measure=true')).json();assert.equal(after.totals.releases,1);assert.equal(after.totals.storage.unreferenced_snapshots,0);
const response=await fetch(`${base}/api/file/${roots[2]}?path=shared.bin`);assert(response.ok);assert.deepEqual(Buffer.from(await response.arrayBuffer()),shared,'Retained game must still restore identical shared bytes');
assert.deepEqual(writes,['/api/catalog/delete','/api/catalog/delete','/api/catalog/collect']);assert.deepEqual(errors,[]);
console.log(`PASS: catalog measurement, page entry, image opening, canceled/confirmed version and game deletion, GC, retained shared bytes. Fixture and screenshots: ${root}`);
}finally{
if(browser){await browser.close();}
if(backend.exitCode===null){const exited=once(backend,'exit');backend.kill('SIGTERM');await exited;}
}
tests/ui-import-workflow.mjs
+29
View File
@@ -0,0 +1,29 @@
// Uses a disposable backend with synthetic input files; never run against the live archive.
import {createRequire} from 'node:module';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const origin=process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3002';
const browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const game=`Import fixture ${Date.now()}`;
const page=await browser.newPage({viewport:{width:1536,height:1040}});const errors=[];page.on('pageerror',e=>errors.push(e.message));
try{
const info=await (await page.request.get(origin+'/api/info')).json();assert(info.import_roots.every(p=>p.startsWith('/tmp/verstack-import-check/')),'Only disposable import fixture may receive test mutations');
await page.goto(origin);await page.getByRole('button',{name:'Import release',exact:true}).first().waitFor({timeout:90000});await page.getByRole('button',{name:'Import release',exact:true}).first().click();
const view=page.locator('#verstack\\.import');await view.getByRole('tab',{name:'URL / Internet Archive'}).click();await view.getByLabel('File URL or Internet Archive item link').fill('https://archive.org/details/pokemon_le-0_81_0.Release.64G.sdcard-secure.raw');await view.getByRole('button',{name:'Browse URL',exact:true}).click();
await view.getByRole('radio',{name:'Choose pokemon_le-0_81_0.Release.64G.sdcard-secure.raw.zip',exact:true}).waitFor({timeout:70000});
assert.match(await view.innerText(),/3\.1 GiB/);await page.screenshot({path:'/home/jordan/verstack/data/validation/native-workbench/import-url-picker.png'});
await view.getByRole('tab',{name:'Server filesystem'}).click();await view.getByRole('button',{name:info.import_roots[0],exact:true}).click();await view.getByRole('checkbox',{name:'Select fixture-release.zip',exact:true}).check();
await view.getByLabel('Game or collection',{exact:true}).fill(game);await view.getByLabel('Version',{exact:true}).fill('0.81.0');await view.getByLabel('Analyze code',{exact:false}).check();
await page.screenshot({path:'/home/jordan/verstack/data/validation/native-workbench/import-server-browser.png'});
await view.getByRole('button',{name:'Import release',exact:true}).click();await Promise.race([view.getByRole('status').waitFor(),view.getByRole('alert').waitFor().then(async()=>{throw Error(await view.getByRole('alert').innerText());})]);
const jobs=page.locator('#verstack\\.jobs');await jobs.getByText(`${game} · LE / 0.81.0`,{exact:true}).waitFor();
await jobs.getByRole('progressbar').first().waitFor({timeout:30000});assert(await jobs.getByRole('button',{name:'Cancel',exact:true}).count()>0);
await page.screenshot({path:'/home/jordan/verstack/data/validation/native-workbench/import-job-progress.png'});
await page.reload();await page.getByRole('button',{name:'Jobs',exact:true}).waitFor({timeout:90000});await page.getByRole('button',{name:'Jobs',exact:true}).click();await jobs.getByText(`${game} · LE / 0.81.0`,{exact:true}).waitFor();
const started=Date.now();let task;
while(Date.now()-started<150000){const activity=await (await page.request.get(origin+'/api/activity')).json();task=activity.items.find(t=>t.kind==='import'&&t.release.repository===game);if(['completed','failed','cancelled','interrupted'].includes(task?.state)){break;}await new Promise(r=>setTimeout(r,1000));}
assert.equal(task?.state,'completed',task?.error||'import did not finish');assert.deepEqual(await (await page.request.get(origin+'/api/jobs')).json(),[],'No legacy/background jobs were queued');
await jobs.getByRole('button',{name:'Open release',exact:true}).first().click();await page.getByRole('button',{name:'Images',exact:true}).waitFor();await page.getByRole('button',{name:'Images',exact:true}).click();await page.locator('.vs-asset-name').filter({hasText:'icon.png'}).first().waitFor();
const workspace=await(await page.request.get(origin+`/api/workspaces/${task.root}`)).json();assert(workspace.code.length>0,'Ghidra code is available');
assert.deepEqual(errors,[]);console.log('PASS: real IA metadata picker without firmware download, server ZIP selection, async import, live progress, reload persistence, extraction/media/code, and opening assets.');
}catch(error){await page.screenshot({path:'/tmp/verstack-import-check/ui-failure.png'});throw error;}finally{await browser.close();}
tests/ui-native-workbench.mjs
+56
View File
@@ -0,0 +1,56 @@
// Read-only regression: safe against the live archive. Never submits jobs or imports.
import {createRequire} from 'node:module';
import {mkdir} from 'node:fs/promises';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);
const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const origin=process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3001';
const output=process.env.VERSTACK_SCREENSHOTS||'/tmp/verstack-browser-check';await mkdir(output,{recursive:true});
const browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const page=await browser.newPage({viewport:{width:1536,height:960}});const errors=[];const mutations=[];
page.on('pageerror',e=>errors.push(e.message));page.on('console',m=>{if(m.type()==='error'&&m.text().includes('root ERROR'))errors.push(m.text());});page.on('request',r=>{if(r.method()==='POST'&&r.url().includes('/api/'))mutations.push(r.url());});
try {
await page.goto(origin+'/?game=Pok%C3%A9mon');
await page.getByLabel('Working tree',{exact:true}).waitFor({timeout:90000});
await page.waitForFunction(()=>document.querySelectorAll('.vs-asset').length>0);
assert(await page.getByLabel('Working tree',{exact:true}).isVisible(),'Explorer opens visibly');
assert.equal(await page.locator('.stern-workbench').count(),0,'legacy application is not mounted');
const labels=await page.locator('#working-tree option').allTextContents();assert(labels.every(l=>!l.match(/extracted|derived|[0-9a-f]{8}-/)));
await page.getByRole('button',{name:'Images',exact:true}).click();
await page.waitForFunction(()=>Array.from(document.querySelectorAll('.vs-thumbnail img')).some(i=>i.complete&&i.naturalWidth>0));
await page.screenshot({path:output+'/native-assets.png'});
await page.locator('.vs-asset-name').first().click();
await page.locator('.vs-media-editor img').waitFor();
await page.waitForFunction(()=>document.querySelector('.vs-media-editor img')?.naturalWidth>0);
await page.screenshot({path:output+'/native-image-editor.png'});
await page.getByRole('button',{name:'Browse assets',exact:true}).click();
await page.getByRole('button',{name:'Audio',exact:true}).click();
await page.locator('.vs-thumbnail audio').first().waitFor();await page.locator('.vs-thumbnail audio').first().evaluate(a=>a.play());await page.waitForFunction(()=>document.querySelector('.vs-thumbnail audio')?.currentTime>0);
await page.getByRole('button',{name:'Videos',exact:true}).click();
await page.locator('.vs-thumbnail video').first().waitFor();await page.locator('.vs-thumbnail video').first().evaluate(v=>v.play());await page.waitForFunction(()=>document.querySelector('.vs-thumbnail video')?.currentTime>0.1);
await page.getByRole('button',{name:'Compare releases',exact:true}).click();await page.getByLabel('Compare against (older)').waitFor();
assert((await page.locator('.vs-panel').allTextContents()).some(t=>t.includes('Only this release is imported')));
await page.screenshot({path:output+'/native-comparison.png'});
// Return to Explorer through Theia's activity tab.
await page.locator('.theia-sidebar-menu .codicon-files, .theia-app-left .codicon-files').first().click();
await page.getByLabel('Working tree',{exact:true}).selectOption({label:'Game of Thrones · Pro / 1.37.0'});
await page.getByRole('button',{name:'Open code browser',exact:true}).click();
await page.locator('.vs-symbols button').first().waitFor({timeout:30000});
await page.getByLabel('Go to symbol or address').fill('FUN_');
await page.locator('.vs-symbols button').first().click();
await page.waitForFunction(()=>document.querySelectorAll('.theia-editor .monaco-editor').length>=2,{},{timeout:30000});
await page.getByText('Outgoing references · click a target to follow').waitFor();
await page.screenshot({path:output+'/native-code.png'});
const branch=page.locator('.vs-table tbody button').first();if(await branch.count()){await branch.click();assert.equal(await page.locator('.vs-error').count(),0,'instruction reference navigation succeeds');}
await page.locator('.theia-sidebar-menu .codicon-files, .theia-app-left .codicon-files').first().click();
await page.getByRole('button',{name:'Compare',exact:true}).click();
await page.getByLabel('Compare against (older)').selectOption({label:'LE / 1.37.0'});
await page.getByRole('button',{name:'Compare code exports',exact:true}).click();
await page.getByRole('button',{name:'Compare code',exact:true}).click();
await page.getByText(/5,526 exact matches/).waitFor({timeout:60000});
await page.getByRole('button',{name:'Open difference',exact:true}).first().click();
await page.locator('.monaco-diff-editor').waitFor({timeout:30000});
await page.screenshot({path:output+'/native-code-diff.png'});
assert.equal(await page.locator('.stern-dialog').count(),0);assert.deepEqual(errors,[]);assert.deepEqual(mutations,[]);
console.log('PASS: native Explorer, release labels, media editor, audio/video, comparison empty state, split native code editors; zero API writes.');
}finally{await page.screenshot({path:output+'/native-last-state.png'});await browser.close();}
tests/ui-release-comparison.mjs
+35
View File
@@ -0,0 +1,35 @@
// Read-only UI regression against the disposable comparison fixture service on port 3002.
import {createRequire} from 'node:module';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const origin=process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3002';
const browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const page=await browser.newPage({viewport:{width:1440,height:960}});const errors=[],writes=[];page.on('pageerror',e=>errors.push(e.message));page.on('request',r=>{if(r.method()==='POST'&&r.url().includes('/api/'))writes.push(r.url());});
try{
await page.goto(origin);await page.getByLabel('Working tree',{exact:true}).waitFor({timeout:90000});
await page.getByLabel('Working tree',{exact:true}).selectOption({label:'Comparison fixture · LE / 0.85.0'});
await page.getByRole('button',{name:'Compare',exact:true}).click();
await page.getByLabel('Compare against (older)').selectOption({label:'LE / 0.81.0'});
await page.locator('.vs-change-counts').waitFor();
assert.match(await page.locator('.vs-change-counts').innerText(),/added\s+2/i);assert.match(await page.locator('.vs-change-counts').innerText(),/removed\s+2/i);
await page.locator('.vs-change-counts button').filter({hasText:'added'}).click();
await page.waitForFunction(()=>document.querySelectorAll('.vs-asset').length===2);
assert((await page.locator('.vs-asset-name').allTextContents()).every(n=>n.startsWith('added.')));
await page.getByRole('button',{name:'Images',exact:true}).click();
await page.waitForFunction(()=>document.querySelectorAll('.vs-asset').length===1);
await page.locator('#verstack\\.assets').getByLabel('Show changes').selectOption('removed');
await page.waitForFunction(()=>document.querySelector('.vs-asset-name')?.textContent==='removed.png');
await page.screenshot({path:'/tmp/verstack-browser-check/removed-assets.png'});
await page.locator('.vs-asset-name').click();await page.locator('.vs-media-editor img').waitFor();
const url=await page.locator('.vs-media-editor img').getAttribute('src');
const workspaces=await (await page.request.get(origin+'/api/workspaces')).json();const before=workspaces.find(w=>w.release.version==='0.81.0');assert(url.includes(before.id),'removed preview reads the older release');
await page.locator('.theia-sidebar-menu .codicon-files, .theia-app-left .codicon-files').first().click();
await page.getByRole('button',{name:'Browse assets',exact:true}).click();
await page.getByRole('button',{name:'Files',exact:true}).click();
await page.locator('#verstack\\.assets').getByLabel('Show changes').selectOption('changed');
await page.waitForFunction(()=>document.querySelector('.vs-asset-name')?.textContent==='changed.txt');
await page.locator('.vs-asset-name').click();await page.locator('.monaco-diff-editor').waitFor({timeout:30000});
await page.screenshot({path:'/tmp/verstack-browser-check/native-file-diff.png'});
assert.deepEqual(errors,[]);assert.deepEqual(writes,[]);
console.log('PASS: edition/version comparison, Added-only assets, Removed-only image from older release, native changed-text diff; zero API writes.');
}finally{await page.screenshot({path:'/tmp/verstack-browser-check/comparison-last-state.png'});await browser.close();}
tests/ui-video-preview.mjs
+17
View File
@@ -0,0 +1,17 @@
// Read-only regression for first-frame video thumbnails; does not generate preview files.
import {createRequire} from 'node:module';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const page=await browser.newPage({viewport:{width:1536,height:960}});const writes=[],errors=[];page.on('request',r=>{if(r.method()==='POST'&&r.url().includes('/api/'))writes.push(r.url());});page.on('pageerror',e=>errors.push(e.message));
try{
await page.goto((process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3000')+'/?game=Pok%C3%A9mon');
await page.getByRole('button',{name:'Videos',exact:true}).waitFor({timeout:90000});await page.getByRole('button',{name:'Videos',exact:true}).click();
await page.locator('.vs-thumbnail video').first().waitFor();
await page.waitForFunction(()=>{const videos=Array.from(document.querySelectorAll('.vs-thumbnail video')).slice(0,3);return videos.length===3&&videos.every(v=>v.paused&&v.currentTime<0.01&&v.readyState>=2&&v.videoWidth>0);},null,{timeout:45000});
assert(await page.locator('.vs-thumbnail video[preload="none"]').count()>0,'offscreen videos remain unloaded');
await page.screenshot({path:'/home/jordan/verstack/data/validation/native-workbench/video-first-frames.png'});
await page.locator('.vs-thumbnail video').first().evaluate(v=>v.play());await page.waitForFunction(()=>document.querySelector('.vs-thumbnail video').currentTime>0.1);
assert.deepEqual(writes,[]);assert.deepEqual(errors,[]);
console.log('PASS: visible videos show decoded first frames while paused, offscreen videos defer loading, playback works; zero API writes.');
}finally{await browser.close();}
tests/ui-view-options.mjs
+30
View File
@@ -0,0 +1,30 @@
// Read-only live check: viewing preferences and pagination must not submit work.
import {createRequire} from 'node:module';
import assert from 'node:assert/strict';
const require=createRequire(import.meta.url);
const {chromium}=require(process.env.VERSTACK_PLAYWRIGHT||'/tmp/verstack-browser-check/node_modules/playwright');
const browser=await chromium.launch({executablePath:process.env.VERSTACK_CHROMIUM||'/home/jordan/.cache/ms-playwright/chromium-1243/chrome-linux64/chrome',headless:true,args:['--no-sandbox']});
const page=await browser.newPage({viewport:{width:1536,height:960}});const writes=[],errors=[],requests=[];
page.on('pageerror',e=>errors.push(e.message));page.on('request',r=>{if(r.url().includes('/api/')){if(r.method()!=='GET')writes.push(r.url());if(r.url().includes('/files?'))requests.push(r.url());}});
try{
await page.goto((process.env.VERSTACK_TEST_URL||'http://127.0.0.1:3000')+'/?game=Game%20of%20Thrones');
await page.getByRole('button',{name:'Images',exact:true}).waitFor({timeout:90000});await page.getByRole('button',{name:'Images',exact:true}).click();
const pager=page.getByLabel('Assets page',{exact:true});await pager.waitFor();await page.waitForFunction(()=>!document.querySelector('[aria-label="Assets page"]')?.disabled);
await pager.fill('10');await pager.press('Enter');await page.waitForFunction(()=>document.querySelector('[aria-label="Assets page"]')?.value==='10'&&!document.querySelector('[aria-label="Assets page"]')?.disabled);
assert.equal(new URL(requests.at(-1)).searchParams.get('page'),'9');const requestCount=requests.length;
await page.getByRole('button',{name:'Asset view options',exact:true}).click();await page.getByText('DMD orange tint',{exact:true}).click();
await page.waitForFunction(()=>document.querySelector('.vs-thumbnail img')?.classList.contains('vs-dmd-tint'));
assert.equal(await pager.inputValue(),'10');assert.equal(requests.length,requestCount,'tint must not refetch or reset the asset page');
await page.screenshot({path:'/tmp/verstack-dmd-gallery.png'});
await page.locator('.vs-asset-name').first().click();await page.locator('.vs-media-editor img.vs-dmd-tint').waitFor();
await page.getByRole('button',{name:'Image view options',exact:true}).click();await page.getByText('DMD orange tint',{exact:true}).click();
assert.equal(await page.locator('.vs-media-editor img.vs-dmd-tint').count(),0);assert.equal(await page.locator('.vs-thumbnail img.vs-dmd-tint').count(),0);
await page.locator('.lm-TabBar-tabLabel').filter({hasText:/^Assets$/}).first().click();
const afterOpen=requests.length;await pager.fill('0');await pager.press('Enter');assert.equal(requests.length,afterOpen,'invalid page must not request data');
await pager.press('Escape');assert.equal(await pager.inputValue(),'10');
await page.getByRole('button',{name:'Asset view options',exact:true}).click();await page.getByText('Details',{exact:true}).click();
await page.locator('#verstack\\.assets .vs-table tbody tr').first().waitFor();assert.equal(await pager.inputValue(),'10');
await page.reload();await page.locator('#verstack\\.assets .vs-table tbody tr').first().waitFor({timeout:90000});
assert.deepEqual(JSON.parse(await page.evaluate(()=>localStorage.getItem('verstack.asset-view'))),{layout:'list',dmdTint:false});
if(!await page.getByLabel('Working tree',{exact:true}).isVisible()){await page.locator('.theia-sidebar-menu .codicon-files, .theia-app-left .codicon-files').first().click();}await page.getByLabel('Working tree',{exact:true}).selectOption({label:'Pokémon · LE / 0.85.0'});await page.getByRole('button',{name:'Browse assets',exact:true}).click();await page.getByRole('button',{name:'Images',exact:true}).click();await page.getByRole('button',{name:'Asset view options',exact:true}).click();await page.getByText('Thumbnails',{exact:true}).click();await page.getByRole('button',{name:'Asset view options',exact:true}).click();await page.getByText('DMD orange tint',{exact:true}).click();await page.locator('.vs-thumbnail img').first().waitFor();assert.equal(await page.locator('.vs-thumbnail img.vs-dmd-tint').count(),0,'ordinary images must keep their original colors');assert.deepEqual(writes,[]);assert.deepEqual(errors,[]);console.log('PASS: native View menu, shared DMD filter, direct page jump, invalid-page bounds, saved preferences, zero API writes.');
}finally{await browser.close();}
tests/workspace.rs
+256
View File
@@ -0,0 +1,256 @@
use std::{collections::BTreeMap, fs};
use verstack::{Archive, Config, Release, workspace::BrowseQuery};
fn setup() -> anyhow::Result<(tempfile::TempDir, Config)> {
let tmp = tempfile::tempdir()?;
fs::create_dir(tmp.path().join("inputs"))?;
let config = Config {
archive: tmp.path().join("archive"),
workspace: tmp.path().join("scratch"),
require_ram_workspace: false,
processing_enabled: false,
import_roots: vec![tmp.path().join("inputs")],
workspace_bytes: 64 * 1024 * 1024,
bind: "127.0.0.1:0".into(),
plugins: BTreeMap::new(),
};
Ok((tmp, config))
}
fn release(version: &str) -> Release {
Release {
repository: "Game".into(),
version: version.into(),
edition: "LE".into(),
generation: "3".into(),
released_at: None,
}
}
#[test]
fn release_comparison_filters_and_opens_removed_originals() -> anyhow::Result<()> {
let (tmp, config) = setup()?;
let old = tmp.path().join("inputs/old");
let new = tmp.path().join("inputs/new");
fs::create_dir(&old)?;
fs::create_dir(&new)?;
for (name, content) in [
("same.txt", "same"),
("changed.txt", "before"),
("removed.txt", "removed bytes"),
] {
fs::write(old.join(name), content)?;
}
for (name, content) in [
("same.txt", "same"),
("changed.txt", "after"),
("added.txt", "new"),
] {
fs::write(new.join(name), content)?;
}
let archive = Archive::open(config)?;
let before = archive.import(&old, release("0.81.0"))?;
let after = archive.import(&new, release("0.85.0"))?;
let result = archive.browse_workspace(
&after.id,
&BrowseQuery {
before: before.id.clone(),
..Default::default()
},
)?;
for kind in ["added", "removed", "changed", "unchanged"] {
assert_eq!(result["counts"][kind], 1);
}
let removed = archive.browse_workspace(
&after.id,
&BrowseQuery {
before: before.id.clone(),
change: "removed".into(),
..Default::default()
},
)?;
assert_eq!(removed["total"], 1);
assert_eq!(removed["items"][0]["snapshot"], before.id);
assert!(removed["items"][0]["after"].is_null());
assert_eq!(
archive.read_range(&before.id, "removed.txt", 0, 100)?,
b"removed bytes"
);
let tree = archive.browse_workspace(
&after.id,
&BrowseQuery {
directory: true,
before: before.id.clone(),
change: "added".into(),
..Default::default()
},
)?;
assert_eq!(tree["items"][0]["path"], "Files");
let files = archive.browse_workspace(
&after.id,
&BrowseQuery {
directory: true,
prefix: "Files".into(),
before: before.id.clone(),
change: "added".into(),
..Default::default()
},
)?;
assert_eq!(files["items"][0]["path"], "Files/added.txt");
assert_eq!(archive.workspaces()?.len(), 2);
assert!(archive.jobs()?.is_empty());
Ok(())
}
#[test]
fn pause_survives_reopening_without_queuing() -> anyhow::Result<()> {
let (tmp, mut config) = setup()?;
config.plugins.insert(
"zip-extract".into(),
verstack::PluginConfig {
command: vec!["python3".into()],
version: "1".into(),
settings: serde_json::json!({}),
timeout_seconds: 10,
processing_revision: 1,
output_schema: 1,
source_digest: String::new(),
source_root: None,
auto_extensions: vec![".zip".into()],
},
);
fs::write(tmp.path().join("inputs/game.zip"), b"fake")?;
{
let archive = Archive::open(config.clone())?;
archive.import(&tmp.path().join("inputs/game.zip"), release("1"))?;
assert!(archive.jobs()?.is_empty());
}
let archive = Archive::open(config)?;
assert!(archive.jobs()?.is_empty());
assert!(archive.work_one()?.is_none());
Ok(())
}
#[test]
fn ram_policy_rejects_disk_and_oversized_budget() -> anyhow::Result<()> {
let (_tmp, mut config) = setup()?;
fs::create_dir_all(&config.workspace)?;
config.require_ram_workspace = true;
config.workspace_bytes = u64::MAX;
assert!(verstack::workspace::check_scratch(&config).is_err());
config.require_ram_workspace = false;
assert!(verstack::workspace::check_scratch(&config).is_ok());
Ok(())
}
#[test]
fn working_tree_resolves_nested_extraction_and_checks_decoder_profiles() -> anyhow::Result<()> {
let (tmp, mut config) = setup()?;
let extract = r#"import json,sys,pathlib,os,tempfile
r=json.load(open(sys.argv[1]));o=pathlib.Path(r['output_dir']);i=pathlib.Path(r['input_dir']);version=next(i.iterdir()).read_text();p=o/('package-'+version)/'game';p.mkdir(parents=True)
(p/'main.pck').write_text('container');(p/'keep.txt').write_text('same')
# Child tools must inherit the run's RAM scratch directory, not a disk /tmp fallback.
assert pathlib.Path(tempfile.gettempdir()).parent==o.parent
pathlib.Path(r['result_file']).write_text(json.dumps({'protocol':1,'layer':'extracted','coverage':'complete','files':[str(f.relative_to(o)) for f in o.rglob('*') if f.is_file()],'warnings':[]}))"#;
let media = r#"import json,sys,pathlib
r=json.load(open(sys.argv[1]));o=pathlib.Path(r['output_dir']);i=pathlib.Path(r['input_dir']);source=str(next(i.rglob('main.pck')).relative_to(i));p=o/'assets'/source;p.mkdir(parents=True)
original=p/'original.bin';original.write_bytes(b'original payload');preview=p/'asset.png';preview.write_bytes(b'preview'+str(r['settings']).encode())
evidence={'assets':[{'source':source,'entry':'textures/title.png','original':str(original.relative_to(o)),'preview':str(preview.relative_to(o)),'status':'decoded'}]}
(o/'media-evidence.json').write_text(json.dumps(evidence))
pathlib.Path(r['result_file']).write_text(json.dumps({'protocol':1,'layer':'derived','coverage':'complete','files':[str(f.relative_to(o)) for f in o.rglob('*') if f.is_file()],'warnings':[]}))"#;
let plugin = |script: &str| verstack::PluginConfig {
command: vec!["python3".into(), "-c".into(), script.into()],
version: "fixture/1".into(),
settings: serde_json::json!({}),
timeout_seconds: 10,
processing_revision: 1,
output_schema: 1,
source_digest: String::new(),
source_root: None,
auto_extensions: vec![],
};
config.plugins.insert("extract".into(), plugin(extract));
config.plugins.insert("media-extract".into(), plugin(media));
fs::write(tmp.path().join("inputs/old.bin"), "0.81")?;
fs::write(tmp.path().join("inputs/new.bin"), "0.85")?;
let archive = Archive::open(config.clone())?;
let old = archive.import(&tmp.path().join("inputs/old.bin"), release("0.81"))?;
let new = archive.import(&tmp.path().join("inputs/new.bin"), release("0.85"))?;
let old_files = archive.process(&old.id, "extract")?;
let new_files = archive.process(&new.id, "extract")?;
archive.process(&old_files.id, "media-extract")?;
archive.process(&new_files.id, "media-extract")?;
assert_eq!(
archive.workspaces()?.len(),
2,
"processing outputs never become release choices"
);
let diff = archive.browse_workspace(
&new.id,
&BrowseQuery {
before: old.id.clone(),
..Default::default()
},
)?;
assert_eq!(
diff["counts"]["unchanged"], 3,
"release wrapper names are removed for both files and assets"
);
assert!(archive.workspace(&new_files.id).is_err());
drop(archive);
config
.plugins
.get_mut("media-extract")
.unwrap()
.processing_revision = 2;
let archive = Archive::open(config)?;
archive.process(&new_files.id, "media-extract")?;
let incompatible = archive.browse_workspace(
&new.id,
&BrowseQuery {
before: old.id,
kind: "image".into(),
..Default::default()
},
)?;
assert_eq!(
incompatible["counts"]["unknown"], 1,
"decoder changes cannot be called publisher changes"
);
assert_eq!(incompatible["profiles_compatible"], false);
Ok(())
}
#[test]
fn discovers_all_games_without_an_arbitrary_program_count_limit() -> anyhow::Result<()> {
let (tmp, config) = setup()?;
let input = tmp.path().join("inputs");
for (name, size) in [("a/game", 2048), ("b/game.exe", 4096), ("c/game.bin", 3072)] {
let path = input.join(name);
fs::create_dir_all(path.parent().unwrap())?;
let mut bytes = vec![0; size];
bytes[..4].copy_from_slice(b"\x7fELF");
fs::write(path, bytes)?;
}
for i in 0..24 {
let mut bytes = vec![0; 8192];
bytes[..4].copy_from_slice(b"\x7fELF");
fs::write(input.join(format!("helper{i}")), bytes)?;
}
let archive = Archive::open(config)?;
let root = archive.import(&input, release("1"))?;
assert_eq!(archive.executable_paths(&root.id)?.len(), 27);
assert_eq!(
archive.default_analysis_paths(&root.id)?,
vec!["b/game.exe", "c/game.bin", "a/game"]
);
let programs = archive.workspace_programs(&root.id)?;
assert_eq!(programs["items"][0]["path"], "b/game.exe");
assert_eq!(
programs["items"]
.as_array()
.unwrap()
.iter()
.filter(|p| p["recommended"] == true)
.count(),
3
);
assert!(archive.import_tasks()?.is_empty());
assert!(archive.jobs()?.is_empty());
Ok(())
}
workbench/README.md
+28 -13
View File
@@ -1,24 +1,39 @@
# Stern game workbench
# Release workbench
Browser Eclipse Theia 1.75 application, scaffolded with the official widget generator and extended with a React game catalog and read-only Monaco viewers. The Rust archive service owns all data and processing. This application proxies `/api` to it on the same origin.
The Eclipse Theia 1.75 application uses a native virtualized `TreeWidget` Explorer, dockable views, and Theia's `EditorManager` and resource resolver for read-only C, assembly, text, hex previews and differences. Assets open as editor tabs. The former single-widget web application is no longer registered.
Run the archive service with an optimized build (`cargo build --release`, then `target/release/verstack --config config.json serve`). Debug builds are unsuitable for firmware restoration and gallery verification.
Choose a working tree such as **Pokémon · LE / 0.85.0** in Explorer. The selector and native status bar identify releases, never intermediate snapshots. Files, decoded media and completed code exports resolve from retained processing history. Snapshot IDs and extraction details remain in Properties and Processing.
Requires Node 22+, npm, and Theia's Linux native build dependencies. From this directory:
**Compare releases** selects an older release of the same game. Added, Removed, Changed and Unchanged filters apply to both the asset viewer and Explorer; removed assets open from the older release. File and container paths are compared after removing the common package wrapper. Media uses retained original asset hashes, not preview hashes. Decoder revision/schema/settings mismatches appear as Coverage differs. Structural record numbers are not semantic identifiers; moved records may appear added/removed. Unsupported or unextracted content is outside the decoded-asset comparison.
**Code browser** opens decompiled C and disassembly in separate native editors, with syntax highlighting and an outgoing-reference panel. Branch references within the current function reveal their instruction; exported function targets open that function. Back/Forward retraces function navigation. Code comparison reads existing exports and opens native difference editors. Exact body matching remains conservative: unmatched functions are not claimed as publisher additions/removals. This is a saved-analysis browser; it does not implement an interactive debugger or control-flow graph. General file previews are bounded to 64 KiB; Properties downloads complete originals.
**Import release** accepts server files/folders under allowed import roots, direct URLs, or an Internet Archive item with selectable downloads. The import options control extraction, previews and code analysis. Code analysis selects every supported `game` executable, with the largest as the default, or the largest supported executable if none is named `game`. Explicit imports work while the older automatic queue is paused. Browsing, comparison and code navigation do not submit jobs.
The code browser's program selector includes binaries without saved analysis and offers **Analyze selected** or **Analyze game programs**. **Function signatures** lists saved observations and reuse across distinct executable contents, with IDA FLIRT `.pat`/`.sig` import/export. Re-analysis does not inflate reuse counts.
The native **View** dropdown controls thumbnails/details and **DMD orange tint**. Tint applies only to images labeled **Approximate bitmap**. Pagination accepts a typed page number followed by Enter or Go.
**Catalog & storage** lists releases and all cataloged files. **Measure storage** shows exact-file and shared-chunk savings, compression, and per-release shared/exclusive usage. **Delete version** and the game filter's **Delete all versions** show a review before removing references. **Garbage collect** separately reclaims unreferenced objects while preserving shared data. Maintenance requires queued/running work to finish or be canceled. Original server files are retained.
## Build and run
Use the optimized Rust build for real firmware. Node 22+, npm, and Theia's Linux native dependencies are required.
```sh
npm ci
npm --prefix stern-catalog run build
npm run build:browser
VERSTACK_API_PORT=8080 npm run start:browser -- --hostname 127.0.0.1 --port 3000
cargo build --release
npm --prefix workbench/stern-catalog run build
npm --prefix workbench run build:browser
```
For direct access from your LAN, start Theia with `--hostname 0.0.0.0`; the Rust backend can remain bound to loopback. The default Rust endpoint remains available independently. Set `VERSTACK_API_PORT` to the Rust port; the proxy always connects to loopback. Use personal/local-network deployments only.
The deployment units in `../deploy/` serve Theia on LAN port 3000 and Rust on loopback port 8080. The same-origin `/api` proxy connects to `VERSTACK_API_PORT` (default 8080). For a separate local UI preview:
Games open in bookmarkable query URLs. Overview, Images, Movies, Sounds, Files, Code and Analysis share the persistent edition/version controls. Compare selects immutable snapshot revisions within the same game and preservation layer; it does not equate an update-package omission with absence from an installed game. Decompiled functions retain symbol provenance and exact-body comparison rules.
```sh
VERSTACK_API_PORT=8080 node workbench/browser-app/lib/backend/main.js --hostname 127.0.0.1 --port 3001
```
Run `npm test` for extension type checking, then `npm run build:browser` for the browser bundle. The Playwright regression is `../tests/ui-workbench.mjs`; it requires a disposable Rust archive on 18765, a separate workbench connected with `VERSTACK_API_PORT=18765`, and synthetic fixtures under `data/validation/workbench/inputs`. Set `VERSTACK_TEST_URL=http://127.0.0.1:3001` when the live workbench occupies port 3000. The test imports fixtures into the target archive; never point it at the real collection. Captures go to `data/validation/workbench/screenshots`.
Use `npm --prefix workbench/stern-catalog test` for type checking. `tests/ui-native-workbench.mjs` is a read-only real-archive browser regression; it asserts zero API writes. `tests/workspace.rs` covers release aggregation, comparison filters, removed-file readback, nested wrapper normalization, incompatible decoder profiles, inherited scratch locations and processing pause. The earlier `ui-workbench.mjs` targets the retired UI and is retained as historical evidence.
Full-corpus acceptance is still in progress; see `../docs/redesign-status.md` for evidence and remaining blockers.
Pokémon 0.81 is not imported. Its sources are recorded in `/srv/firmware/MORE_FIRMWARE.txt`; the Compare view links the archive item and explains that another release must be imported. Existing full-corpus and media-decoding limitations still apply; see `docs/redesign-status.md`.
`../tests/ui-function-comparison.mjs` is a read-only comparison regression. Supply `VERSTACK_TEST_URL`, `VERSTACK_COMPARE_A` and `VERSTACK_COMPARE_B` for two complete, compatible Ghidra outputs within one game; optionally supply `VERSTACK_EXPECTED_MATCHES`. It checks paired Monaco code, unmatched inspection, search and revision changes.
Import releases from the **Import release** link in Explorer. Browse server roots or paste a direct URL/Internet Archive item link; select source files, name the release, and choose extraction, previews, and optional code analysis. **Jobs** shows active imports and processing history with measured or indeterminate progress, cancel/retry controls, and **Open release**. Starting an explicit import does not resume the older paused processing queue.
workbench/package-lock.json
Generated
+2 -1
View File
@@ -14422,7 +14422,8 @@
"license": "UNLICENSED",
"dependencies": {
"@theia/core": "1.75.0",
"@theia/monaco": "1.75.0"
"@theia/monaco": "1.75.0",
"@theia/editor": "1.75.0"
},
"devDependencies": {
"@types/react": "^19.0.0",
workbench/stern-catalog/package.json
+2 -1
View File
@@ -12,7 +12,8 @@
],
"dependencies": {
"@theia/core": "1.75.0",
"@theia/monaco": "1.75.0"
"@theia/monaco": "1.75.0",
"@theia/editor": "1.75.0"
},
"devDependencies": {
"rimraf": "^5.0.0",
workbench/stern-catalog/src/browser/catalog-view.tsx
+46
View File
@@ -0,0 +1,46 @@
import * as React from 'react';
import { injectable, inject, postConstruct } from '@theia/core/shared/inversify';
import { ReactWidget } from '@theia/core/lib/browser/widgets/react-widget';
import { ConfirmDialog } from '@theia/core/lib/browser/dialogs';
import { WorkbenchState, Asset, Release, Workspace, api, releaseLabel } from './workbench-state';
import { Pagination } from './view-controls';
type Storage={packed_data_bytes:number;shared_data_bytes:number;exclusive_data_bytes:number;chunk_reuse_bytes:number;compression_bytes:number};
type CatalogRow={key:string;release:Release;roots:string[];files:number;snapshots:number;logical_bytes:number;unique_file_bytes:number;exact_duplicate_bytes:number;storage?:Storage};
type Catalog={items:CatalogRow[];measured:boolean;note:string;totals:{releases:number;files:number;logical_bytes:number;exact_duplicate_bytes:number;storage?:{packed_bytes_on_disk:number;active_packed_data_bytes:number;chunk_reuse_bytes:number;compression_bytes:number;unreferenced_data_bytes:number;unreferenced_snapshots:number}}};
type Scope={repository:string;version?:string;edition?:string;generation?:string};
type Deletion={scope:Scope;token:string;files:number;snapshots:number;logical_bytes:number;dependent_comparisons:number;eligible_for_gc_bytes:number;shared_data_preserved_bytes:number;note:string};
type CatalogFile=Asset&{release:Release;layer:string;identical_copies:number};
async function mutate<T>(url:string,body:unknown={}):Promise<T>{const response=await fetch(url,{method:'POST',headers:{'X-Verstack-Client':'1','Content-Type':'application/json'},body:JSON.stringify(body)});if(!response.ok){throw new Error(await response.text());}return response.json();}
function bytes(n?:number):string {
if(n===undefined){return '—';}
const units=['B','KiB','MiB','GiB','TiB'];let unit=0;
while(n>=1024&&unit<units.length-1){n/=1024;unit++;}
return `${n.toLocaleString(undefined,{maximumFractionDigits:unit?1:0})} ${units[unit]}`;
}
function CatalogView({state}:{state:WorkbenchState}):React.ReactElement {
const [data,setData]=React.useState<Catalog>(),[mode,setMode]=React.useState('releases'),[repository,setRepository]=React.useState(''),[search,setSearch]=React.useState(''),[page,setPage]=React.useState(0),[files,setFiles]=React.useState<{items:CatalogFile[];total:number;page:number}>(),[busy,setBusy]=React.useState(''),[filesLoading,setFilesLoading]=React.useState(false),[error,setError]=React.useState(''),[message,setMessage]=React.useState('');
async function refresh(measure=false):Promise<Catalog>{setBusy(measure?'Measuring archive storage…':'Loading catalog…');setError('');try{const result=await api<Catalog>(`/api/catalog?measure=${measure}`);setData(result);setRepository(selected=>selected&&!result.items.some(row=>row.release.repository===selected)?'':selected);return result;}finally{setBusy('');}}
React.useEffect(()=>{void refresh().catch(e=>setError(String(e)));},[]);
React.useEffect(()=>{if(mode!=='files'){setFilesLoading(false);return;}let alive=true;setFilesLoading(true);const timer=setTimeout(()=>{api<typeof files>(`/api/catalog/files?${new URLSearchParams({repository,search,page:String(page)})}`).then(r=>{if(alive){setFiles(r);}}).catch(e=>alive&&setError(String(e))).finally(()=>alive&&setFilesLoading(false));},150);return()=>{alive=false;clearTimeout(timer);};},[mode,repository,search,page,data]);
async function open(root:string):Promise<void>{state.workspaces=await api<Workspace[]>('/api/workspaces');await state.select(root);state.showView.fire('assets');}
async function remove(scope:Scope):Promise<void>{setError('');setMessage('');setBusy('Preparing deletion review…');try{
const params=new URLSearchParams(Object.fromEntries(Object.entries(scope).filter(([,v])=>v!==undefined)) as Record<string,string>);const review=await api<Deletion>(`/api/catalog/delete-preview?${params}`);
setBusy('');const target=scope.version?`${scope.repository} · ${scope.edition||''} / ${scope.version}`:`all versions of ${scope.repository}`;
const msg=document.createElement('div');msg.className='vs-delete-review';
for(const text of [`Delete ${target}?`,`${review.files.toLocaleString()} cataloged files and ${review.snapshots.toLocaleString()} saved processing results will be removed.`,`${bytes(review.shared_data_preserved_bytes)} of shared data will remain protected. ${bytes(review.eligible_for_gc_bytes)} will become eligible for garbage collection.`,review.dependent_comparisons?`${review.dependent_comparisons} dependent comparison reports will also be removed.`:'',review.note]){if(text){const p=document.createElement('p');p.textContent=text;msg.appendChild(p);}}
const confirmed=await new ConfirmDialog({title:scope.version?'Delete version':'Delete entire game',msg,ok:scope.version?'Delete version':'Delete entire game',cancel:'Keep files'}).open();if(!confirmed){return;}
setBusy('Deleting catalog references…');await mutate('/api/catalog/delete',{scope,token:review.token});setMessage(`Deleted ${target}. Run garbage collection when you want to reclaim unreferenced data.`);
state.workspaces=await api<Workspace[]>('/api/workspaces');if(state.current&&!state.workspaces.some(w=>w.id===state.current!.id)){state.current=undefined;state.baseline='';state.changed.fire();}
await refresh(!!data?.measured);
}catch(e){setError(String(e));}finally{setBusy('');}}
async function collect():Promise<void>{setError('');setMessage('');try{const current=await refresh(true);const storage=current.totals.storage!;const confirmed=await new ConfirmDialog({title:'Collect unreferenced archive data',msg:`${bytes(storage.unreferenced_data_bytes)} of data chunks and ${storage.unreferenced_snapshots} unreferenced archive snapshots are eligible. Objects still used by cataloged releases will be preserved. Actual space reclaimed can differ because chunks share packs.`,ok:'Collect unreferenced data',cancel:'Cancel'}).open();if(!confirmed){return;}setBusy('Collecting unreferenced data…');const result=await mutate<{reclaimed_packed_bytes:number}>('/api/catalog/collect');setMessage(`Garbage collection reclaimed ${bytes(result.reclaimed_packed_bytes)}. Referenced objects were preserved.`);await refresh(true);}catch(e){setError(String(e));}finally{setBusy('');}}
const games=Array.from(new Set(data?.items.map(r=>r.release.repository)||[]));const rows=data?.items.filter(r=>(!repository||r.release.repository===repository)&&releaseLabel(r.release).toLowerCase().includes(search.toLowerCase()))||[];
return <div className="vs-view"><div className="vs-toolbar"><b>Catalog &amp; storage</b><span className="vs-spacer"/><button disabled={!!busy} onClick={()=>void refresh(!!data?.measured).catch(e=>setError(String(e)))}>Refresh</button><button disabled={!!busy} onClick={()=>void refresh(true).catch(e=>setError(String(e)))}>Measure storage</button><button disabled={!!busy} onClick={()=>void collect()}>Garbage collect…</button></div><div className="vs-signature-stats">{data&&<><div><b>{data.totals.releases.toLocaleString()}</b><span>releases</span></div><div><b>{data.totals.files.toLocaleString()}</b><span>cataloged files</span></div><div><b>{bytes(data.totals.logical_bytes)}</b><span>logical contents</span></div>{data.totals.storage&&<div><b>{bytes(data.totals.storage.packed_bytes_on_disk)}</b><span>archive packs on disk</span></div>}</>}</div><div className="vs-toolbar"><div className="vs-segments"><button aria-pressed={mode==='releases'} onClick={()=>{setMode('releases');setPage(0);}}>Releases</button><button aria-pressed={mode==='files'} onClick={()=>{setMode('files');setPage(0);}}>All cataloged files</button></div><select className="theia-select" aria-label="Catalog game" value={repository} onChange={e=>{setRepository(e.target.value);setPage(0);}}><option value="">All games</option>{games.map(g=><option key={g}>{g}</option>)}</select><input className="theia-input" aria-label="Filter catalog" placeholder={mode==='files'?'Name or path':'Game, edition or version'} value={search} onChange={e=>{setSearch(e.target.value);setPage(0);}}/>{repository&&<button className="vs-danger" disabled={!!busy} onClick={()=>void remove({repository})}>Delete all versions of {repository}…</button>}</div>{(busy||filesLoading)&&<div className="vs-catalog-progress" role="status"><progress/>{busy||'Loading files…'}</div>}{error&&<p role="alert" className="vs-error">{error}</p>}{message&&<p role="status" className="vs-note">{message}</p>}<div className="vs-scroll vs-panel">{mode==='releases'?<><table className="vs-table vs-catalog-table"><thead><tr><th>Release</th><th>Files</th><th>Logical size</th><th>Exact duplicates saved</th>{data?.measured&&<><th>Shared chunks saved</th><th>Packed data used</th><th>Shared with other releases</th><th>Exclusive to this release</th></>}<th>Actions</th></tr></thead><tbody>{rows.map(r=><tr key={r.key}><td><b>{r.release.repository}</b><small>{r.release.edition} / {r.release.version}</small></td><td>{r.files.toLocaleString()}</td><td>{bytes(r.logical_bytes)}</td><td>{bytes(r.exact_duplicate_bytes)}</td>{data?.measured&&<><td>{bytes(r.storage?.chunk_reuse_bytes)}</td><td>{bytes(r.storage?.packed_data_bytes)}</td><td>{bytes(r.storage?.shared_data_bytes)}</td><td>{bytes(r.storage?.exclusive_data_bytes)}</td></>}<td><div className="vs-catalog-actions">{r.roots[0]&&<button onClick={()=>void open(r.roots[0]).catch(e=>setError(String(e)))}>Open release</button>}<button className="vs-danger" disabled={!!busy} onClick={()=>void remove({repository:r.release.repository,version:r.release.version,edition:r.release.edition,generation:r.release.generation})}>Delete version…</button></div></td></tr>)}</tbody></table>{!rows.length&&!busy&&<p>No releases match this view.</p>}<details className="vs-storage-explanation" open><summary>How storage is counted</summary><p>{data?.note}</p>{data?.totals.storage&&<dl><dt>Exact duplicate files saved across the catalog</dt><dd>{bytes(data.totals.exact_duplicate_bytes)}</dd><dt>Additional savings from shared byte ranges</dt><dd>{bytes(data.totals.storage.chunk_reuse_bytes)}</dd><dt>Compression savings on unique chunks</dt><dd>{bytes(data.totals.storage.compression_bytes)}</dd><dt>Unreferenced data eligible for collection</dt><dd>{bytes(data.totals.storage.unreferenced_data_bytes)}</dd></dl>}</details></>:<><p className="vs-note">This inventory includes original files and saved processing results. Identical-copy counts refer to the same exact file contents; shared data is stored once.</p><table className="vs-table"><thead><tr><th>File</th><th>Release</th><th>Content</th><th>Size</th><th>Identical copies</th></tr></thead><tbody>{files?.items.map(f=><tr key={f.snapshot+f.path}><td><button onClick={()=>state.openAsset.fire(f)}>{f.entry.path}</button></td><td>{releaseLabel(f.release)}</td><td>{f.layer==='original'?'Original':f.layer==='extracted'?'Extracted contents':'Saved analysis / preview'}</td><td>{bytes(f.entry.size)}</td><td>{f.identical_copies.toLocaleString()}</td></tr>)}</tbody></table></>}</div>{mode==='files'&&<div className="vs-footer"><Pagination label="Catalog files" page={files?.page||0} total={files?.total||0} pageSize={100} disabled={!!busy||filesLoading} onPage={setPage}/></div>}</div>;
}
@injectable()
export class CatalogWidget extends ReactWidget {
static readonly ID='verstack.catalog';@inject(WorkbenchState) readonly state!:WorkbenchState;
@postConstruct() protected init():void{this.id=CatalogWidget.ID;this.title.label='Catalog & storage';this.title.iconClass='codicon codicon-database';this.title.closable=true;this.addClass('vs-widget');this.update();}
protected render():React.ReactNode{return <CatalogView state={this.state}/>;}
}
workbench/stern-catalog/src/browser/code-languages.ts
+8
View File
@@ -0,0 +1,8 @@
import * as monaco from '@theia/monaco-editor-core';
// Syntax highlighting only. The retained Ghidra output remains the source of code facts.
export function registerCodeLanguages():void {
monaco.languages.register({id:'verstack-c',extensions:['.c','.h'],aliases:['Decompiled C']});
monaco.languages.setMonarchTokensProvider('verstack-c',{tokenizer:{root:[[/\/\*/, 'comment','@comment'],[/\/\/.*$/,'comment'],[/"([^"\\]|\\.)*"/,'string'],[/\b(?:if|else|while|for|do|switch|case|default|break|continue|return|goto|sizeof)\b/,'keyword'],[/\b(?:void|char|short|int|long|float|double|unsigned|signed|bool|struct|union|enum|const|static|uint|ulong|byte|undefined\d*|int\d+_t|uint\d+_t)\b/,'type'],[/\b(?:0x[\da-fA-F]+|\d+)\b/,'number'],[/[{}()[\]]/,'delimiter.bracket']],comment:[[/[^*]+/,'comment'],[/\*\//,'comment','@pop'],[/\*/,'comment']]}});
monaco.languages.register({id:'verstack-asm',extensions:['.asm'],aliases:['Disassembly']});
monaco.languages.setMonarchTokensProvider('verstack-asm',{tokenizer:{root:[[/^[\da-fA-F]+/,'number.hex'],[/\b(?:r\d+|sp|lr|pc|[re]?(?:ax|bx|cx|dx|si|di|bp|sp)|x\d+|w\d+)\b/,'variable.predefined'],[/\b(?:0x[\da-fA-F]+|\d+)\b/,'number'],[/\b(?:b\w*|call\w*|j\w*|ret\w*)\b/,'keyword.control'],[/\b[a-z][\w.]*\b/,'keyword'],[/[;@].*$/,'comment']]}});
}
workbench/stern-catalog/src/browser/signature-library.tsx
+33
View File
@@ -0,0 +1,33 @@
import * as React from 'react';
import { injectable, inject, postConstruct } from '@theia/core/shared/inversify';
import { ReactWidget } from '@theia/core/lib/browser/widgets/react-widget';
import { WorkbenchState, CodeSource, SymbolRow, Release, api, releaseLabel, basename } from './workbench-state';
import { Pagination } from './view-controls';
type Observation=SymbolRow&{snapshot:string;path:string;eligible:boolean;flirt:boolean;seen_in:number};
type Program=CodeSource&{release:Release;input:string;language:string;functions:number};
type Library={id:string;name:string;language:string;format:string;patterns:number;bytes:number};
type Inventory={stats:{observations:number;executables:number;eligible:number;flirt_patterns:number};programs:Program[];items:Observation[];total:number;page:number;page_size:number;libraries:Library[]};
type Reuse={eligible:boolean;seen_in:number;flirt_available:boolean;occurrences:Array<{input:string;source:CodeSource;symbol:SymbolRow;release:Release}>;pattern_occurrences:Reuse['occurrences'];flirt_candidates:Array<{library_name:string;name:string;references:string[];status:string}>;note:string};
export function FunctionReuse({state,source,symbol}:{state:WorkbenchState;source:CodeSource;symbol:SymbolRow}):React.ReactElement {
const [data,setData]=React.useState<Reuse>(),[error,setError]=React.useState('');
React.useEffect(()=>{let alive=true;setData(undefined);setError('');api<Reuse>(`/api/signatures/function?${new URLSearchParams({snapshot:source.snapshot,path:source.path,address:symbol.address})}`).then(r=>alive&&setData(r)).catch(e=>alive&&setError(String(e)));return()=>{alive=false;};},[source.snapshot,source.path,symbol.address]);
return <div className="vs-function-reuse">{error?<p role="alert" className="vs-error">{error}</p>:!data?<p role="status">Checking saved signatures…</p>:<><b>{data.eligible?`Seen in ${data.seen_in.toLocaleString()} other executable${data.seen_in===1?'':'s'} · exact function bytes`:'Saved observation · body is too small, a thunk, or unavailable for reliable identification'}</b>{!!data.occurrences.length&&<details><summary>Show matching executables</summary>{data.occurrences.map(o=><div key={o.input}><button className="vs-link" onClick={()=>state.openCode.fire({source:o.source,symbol:o.symbol})}>{releaseLabel(o.release)} · {basename(o.source.name)} · {o.symbol.name}</button></div>)}</details>}{!!data.pattern_occurrences?.length&&<details><summary>Similar FLIRT pattern in {data.pattern_occurrences.length} other executables · candidates</summary>{data.pattern_occurrences.map(o=><div key={o.input}><button className="vs-link" onClick={()=>state.openCode.fire({source:o.source,symbol:o.symbol})}>{releaseLabel(o.release)} · {basename(o.source.name)} · {o.symbol.name}</button></div>)}</details>}{!!data.flirt_candidates.length&&<details open><summary>{data.flirt_candidates.length} FLIRT candidates</summary>{data.flirt_candidates.map((c,i)=><p key={i}><b>{c.name}</b> · {c.library_name}{c.references.length?' · References need verification':''}</p>)}</details>}<small>{data.note}</small></>}</div>;
}
function SignatureLibrary({state}:{state:WorkbenchState}):React.ReactElement {
const [data,setData]=React.useState<Inventory>(),[program,setProgram]=React.useState(''),[search,setSearch]=React.useState(''),[reused,setReused]=React.useState(false),[page,setPage]=React.useState(0),[revision,refresh]=React.useReducer(n=>n+1,0),[selected,setSelected]=React.useState<Observation>(),[error,setError]=React.useState(''),[busy,setBusy]=React.useState(false),[message,setMessage]=React.useState('');
const [file,setFile]=React.useState<File>(),[language,setLanguage]=React.useState('');
const programRow=data?.programs.find(p=>p.snapshot+'|'+p.path===program);
React.useEffect(()=>{let alive=true;setBusy(true);setError('');const [snapshot='',path='']=program.split('|');const timer=setTimeout(()=>{api<Inventory>(`/api/signatures?${new URLSearchParams({snapshot,path,search,page:String(page),reused:String(reused)})}`).then(r=>{if(alive){setData(r);setPage(r.page);}}).catch(e=>alive&&setError(String(e))).finally(()=>alive&&setBusy(false));},150);return()=>{alive=false;clearTimeout(timer);};},[program,search,page,reused,revision]);
async function download(format:string):Promise<void>{if(!programRow){return;}setError('');try{const response=await fetch(`/api/signatures/export?${new URLSearchParams({snapshot:programRow.snapshot,path:programRow.path,format})}`);if(!response.ok){throw new Error(await response.text());}const url=URL.createObjectURL(await response.blob());const link=document.createElement('a');link.href=url;link.download=`${basename(programRow.name)}.${format}`;link.click();setTimeout(()=>URL.revokeObjectURL(url),1000);}catch(e){setError(String(e));}}
async function upload(event:React.FormEvent):Promise<void>{event.preventDefault();if(!file||!language){return;}setBusy(true);setError('');try{const response=await fetch(`/api/signatures/libraries?${new URLSearchParams({name:file.name,language})}`,{method:'POST',headers:{'X-Verstack-Client':'1','Content-Type':'application/octet-stream'},body:file});if(!response.ok){throw new Error(await response.text());}const result=await response.json();setMessage(`Imported ${result.patterns.toLocaleString()} patterns from ${file.name}.`);refresh();}catch(e){setError(String(e));}finally{setBusy(false);}}
const languages=Array.from(new Set(data?.programs.map(p=>p.language)||[]));const selectedSource=selected&&data?.programs.find(p=>p.snapshot===selected.snapshot&&p.path===selected.path);
return <div className="vs-view"><div className="vs-toolbar"><b>Function signatures</b><span className="vs-spacer"/><button onClick={()=>refresh()} disabled={busy}>Refresh</button></div><div className="vs-signature-stats">{data&&[[data.stats.observations,'saved functions'],[data.stats.executables,'distinct executables'],[data.stats.eligible,'eligible exact signatures'],[data.stats.flirt_patterns,'FLIRT patterns']].map(([n,label])=><div key={label}><b>{Number(n).toLocaleString()}</b><span>{label}</span></div>)}</div><div className="vs-toolbar"><select className="theia-select" aria-label="Signature program" value={program} onChange={e=>{setProgram(e.target.value);setPage(0);setSelected(undefined);}}><option value="">All saved programs</option>{data?.programs.map(p=><option key={p.snapshot+p.path} value={p.snapshot+'|'+p.path}>{releaseLabel(p.release)} · {p.name}</option>)}</select><input className="theia-input" aria-label="Find saved function" placeholder="Function name or address" value={search} onChange={e=>{setSearch(e.target.value);setPage(0);}}/><label><input type="checkbox" checked={reused} onChange={e=>{setReused(e.target.checked);setPage(0);}}/> Seen elsewhere</label><button disabled={!programRow} onClick={()=>void download('pat')}>Export .pat</button><button disabled={!programRow} onClick={()=>void download('sig')}>Export .sig</button></div>{error&&<p role="alert" className="vs-error">{error}</p>}<div className="vs-scroll vs-panel"><p className="vs-note">Every completed decompilation contributes observations. Counts use distinct executable contents, so repeated imports and re-analysis do not inflate reuse. FLIRT matches are candidates; function names remain as originally analysed.</p><table className="vs-table"><thead><tr><th>Function</th><th>Program</th><th>Size</th><th>Other executables</th><th>FLIRT</th></tr></thead><tbody>{data?.items.map(f=>{const p=data.programs.find(p=>p.snapshot===f.snapshot&&p.path===f.path);return <tr key={f.snapshot+f.path+f.address}><td><button onClick={()=>setSelected(f)}>{f.name}</button><small>{f.address}</small></td><td>{p&&releaseLabel(p.release)}<small>{p?.name}</small></td><td>{f.size.toLocaleString()} B</td><td>{f.eligible?f.seen_in.toLocaleString():'Not eligible'}</td><td>{f.flirt?'Saved pattern':'Not available'}</td></tr>;})}</tbody></table>{data&&!data.items.length&&<p>No saved functions match this view.</p>}{selected&&selectedSource&&<section className="vs-signature-detail"><div className="vs-toolbar"><b>{selected.name}</b><button onClick={()=>state.openCode.fire({source:selectedSource,symbol:selected})}>Open decompiled function</button><button onClick={()=>setSelected(undefined)}>Close details</button></div><FunctionReuse state={state} source={selectedSource} symbol={selected}/></section>}<details className="vs-flirt-import" open><summary>IDA FLIRT libraries</summary><form className="vs-toolbar" onSubmit={e=>void upload(e)}><input type="file" aria-label="FLIRT signature file" accept=".pat,.sig" onChange={e=>setFile(e.target.files?.[0])}/><select className="theia-select" aria-label="FLIRT target language" required value={language} onChange={e=>setLanguage(e.target.value)}><option value="">Choose target language…</option>{languages.map(l=><option key={l}>{l}</option>)}</select><button className="theia-button" disabled={!file||!language||busy}>Import signatures</button></form><p className="vs-muted">Choose a .pat or .sig file and its target language. Pattern candidates appear when you inspect functions. Existing library files can be downloaded unchanged.</p>{message&&<p role="status">{message}</p>}<table className="vs-table"><thead><tr><th>Library</th><th>Language</th><th>Patterns</th><th>File</th></tr></thead><tbody>{data?.libraries.map(l=><tr key={l.id}><td>{l.name}</td><td>{l.language}</td><td>{l.patterns.toLocaleString()}</td><td><a href={`/api/signatures/libraries/${l.id}`}>Download .{l.format}</a></td></tr>)}</tbody></table></details></div><div className="vs-footer"><Pagination label="Saved functions" page={data?.page||0} total={data?.total||0} pageSize={100} disabled={busy} onPage={setPage}/></div></div>;
}
@injectable()
export class SignatureLibraryWidget extends ReactWidget {
static readonly ID='verstack.signatures';@inject(WorkbenchState) readonly state!:WorkbenchState;
@postConstruct() protected init():void{this.id=SignatureLibraryWidget.ID;this.title.label='Function signatures';this.title.iconClass='codicon codicon-symbol-method';this.title.closable=true;this.addClass('vs-widget');this.update();}
protected render():React.ReactNode{return <SignatureLibrary state={this.state}/>;}
}
workbench/stern-catalog/src/browser/stern-catalog-contribution.ts
+141 -11
View File
@@ -1,13 +1,143 @@
import { CatalogWidget } from './catalog-view';
import { SignatureLibraryWidget } from './signature-library';
import { injectable, inject } from '@theia/core/shared/inversify';
import { ThemeService } from '@theia/core/lib/browser/theming';
import { SternCatalogWidget } from './stern-catalog-widget';
import { AbstractViewContribution, FrontendApplication } from '@theia/core/lib/browser';
@injectable()
export class SternCatalogContribution extends AbstractViewContribution<SternCatalogWidget> {
@inject(ThemeService) protected readonly themes!: ThemeService;
constructor() {
super({widgetId: SternCatalogWidget.ID, widgetName: SternCatalogWidget.LABEL,
defaultWidgetOptions: { area: 'main' }, toggleCommandId: 'stern-catalog:open'});
}
async onStart(_app: FrontendApplication): Promise<void> { this.themes.setCurrentTheme('dark'); await this.openView({activate:true,reveal:true}); }
import { FrontendApplication, FrontendApplicationContribution, ApplicationShell, WidgetManager, Widget } from '@theia/core/lib/browser';
import { CommandContribution, CommandRegistry } from '@theia/core/lib/common/command';
import { MenuContribution, MenuModelRegistry } from '@theia/core/lib/common/menu';
import { Resource, ResourceResolver } from '@theia/core/lib/common/resource';
import URI from '@theia/core/lib/common/uri';
import { EditorManager } from '@theia/editor/lib/browser/editor-manager';
import { StatusBar, StatusBarAlignment } from '@theia/core/lib/browser/status-bar/status-bar';
import { EditorWidget } from '@theia/editor/lib/browser/editor-widget';
import { DiffUris } from '@theia/core/lib/browser/diff-uris';
import { registerCodeLanguages } from './code-languages';
import { ASSET_VIEW_MENU, IMAGE_VIEW_MENU } from './view-controls';
import { WorkbenchState, Asset, CodeSource, CodeDetail, SymbolRow, api, fileURL, basename } from './workbench-state';
import { WorkspaceTreeWidget } from './workspace-tree';
import { AssetsWidget, AssetEditorWidget, ComparisonWidget, CodeNavigatorWidget, DetailsWidget, ReferencesWidget, JobsWidget, ImportWidget, FunctionDiffWidget } from './workspace-views';
export function contentURI(snapshot:string,path:string,name:string,view='file'):URI {
return new URI(`verstack:/${encodeURIComponent(snapshot)}/${encodeURIComponent(name)}`).withQuery(new URLSearchParams({snapshot,path,view}).toString());
}
@injectable()
export class ArchiveResourceResolver implements ResourceResolver {
async resolve(uri:URI):Promise<Resource> {
if(uri.scheme!=='verstack'){throw new Error('Unsupported URI');}
const q=new URLSearchParams(uri.query);
return {uri,readOnly:true,dispose:()=>{},readContents:async()=>{
const view=q.get('view')||'file';if(view==='empty'){return '';}
const snapshot=q.get('snapshot')||'',path=q.get('path')||'';
if(view==='decompiled'||view==='disassembly') {
const detail=await api<CodeDetail>(fileURL(snapshot,path));
return view==='decompiled'?`/* ${detail.address} · decompilation: ${detail.decompile_status} */\n${detail.decompiled}`:detail.disassembly;
}
const response=await fetch(fileURL(snapshot,path),{headers:{Range:'bytes=0-65535'}});
if(!response.ok){throw new Error(await response.text());}
const data=new Uint8Array(await response.arrayBuffer());
const text=new TextDecoder('utf-8',{fatal:false}).decode(data);
const binary=data.includes(0)||text.includes('\ufffd');
if(!binary){return text+(response.headers.get('content-range')?.split('/')[1]&&Number(response.headers.get('content-range')!.split('/')[1])>data.length?'\n\n[Preview limited to 64 KiB. Download original for the complete file.]':'');}
const lines=['Hex preview · first 64 KiB · original bytes available in Properties','Offset Bytes ASCII'];
for(let i=0;i<data.length;i+=16){const row=Array.from(data.slice(i,i+16));lines.push(`${i.toString(16).padStart(8,'0')} ${row.map(v=>v.toString(16).padStart(2,'0')).join(' ').padEnd(47,' ')} ${row.map(v=>v>=32&&v<127?String.fromCharCode(v):'.').join('')}`);}
return lines.join('\n');
}};
}
}
@injectable()
export class SternCatalogContribution implements FrontendApplicationContribution, CommandContribution, MenuContribution {
@inject(WorkbenchState) protected readonly state!:WorkbenchState;
@inject(ApplicationShell) protected readonly shell!:ApplicationShell;
@inject(WidgetManager) protected readonly widgets!:WidgetManager;
@inject(EditorManager) protected readonly editors!:EditorManager;
@inject(StatusBar) protected readonly statusBar!:StatusBar;
private decompiledEditor?:EditorWidget;private assemblyEditor?:EditorWidget;
initialize():void {registerCodeLanguages();}
private history:Array<{source:CodeSource;symbol:SymbolRow}>=[];private position=-1;private codeEpoch=0;
registerCommands(commands:CommandRegistry):void {
for(const [id,label,view] of [['verstack:assets','Verstack: Browse assets','assets'],['verstack:compare','Verstack: Compare releases','compare'],['verstack:code','Verstack: Open code browser','code'],['verstack:jobs','Verstack: Jobs','jobs'],['verstack:import','Verstack: Import release','import'],['verstack:signatures','Verstack: Function signatures','signatures'],['verstack:catalog','Verstack: Catalog and storage','catalog']]) {
commands.registerCommand({id,label},{execute:()=>this.show(view as 'assets'|'compare'|'code'|'jobs'|'import'|'functionDiff'|'signatures'|'catalog')});
}
commands.registerCommand({id:'verstack:explorer',label:'Verstack: Show Explorer'},{execute:()=>this.open(WorkspaceTreeWidget.ID,'left')});
for(const [layout,label] of [['grid','Thumbnails'],['list','Details']] as const){
commands.registerCommand({id:`verstack:asset-layout-${layout}`,label},{execute:()=>this.state.setViewOptions({layout}),isToggled:()=>this.state.viewOptions.layout===layout});
}
commands.registerCommand({id:'verstack:dmd-tint',label:'DMD orange tint'},{execute:()=>this.state.setViewOptions({dmdTint:!this.state.viewOptions.dmdTint}),isToggled:()=>this.state.viewOptions.dmdTint});
}
registerMenus(menus:MenuModelRegistry):void {
menus.registerMenuAction([...ASSET_VIEW_MENU,'1-layout'],{commandId:'verstack:asset-layout-grid',order:'1'});
menus.registerMenuAction([...ASSET_VIEW_MENU,'1-layout'],{commandId:'verstack:asset-layout-list',order:'2'});
for(const path of [ASSET_VIEW_MENU,IMAGE_VIEW_MENU]){menus.registerMenuAction([...path,'2-display'],{commandId:'verstack:dmd-tint'});}
}
async onDidInitializeLayout(_app:FrontendApplication):Promise<void> {
this.state.showView.event(view=>void this.show(view).catch(e=>this.state.fail(e)));
this.state.openCodeDiff.event(value=>{void (async()=>{
const uris:URI[]=[];
for(const [i,symbol] of [value.before,value.after].entries()){
if(!symbol){continue;}const source=value.sources[i];
const found=await api<{items:SymbolRow[]}>(`/api/code?${new URLSearchParams({snapshot:source.snapshot,path:source.path,search:symbol.address})}`);
const row=found.items.find(s=>s.address===symbol.address);if(!row?.code_path){throw new Error('This symbol has no saved code export.');}
uris.push(contentURI(source.snapshot,source.path.replace(/functions\.json$/,row.code_path),`${symbol.name}.${value.view==='decompiled'?'c':'asm'}`,value.view));
}
if(uris.length===2){await this.editors.open(DiffUris.encode(uris[0],uris[1],`${value.before?.name} — ${value.view} comparison`));}
else if(uris[0]){await this.editors.open(uris[0]);}
})().catch(e=>this.state.fail(e));});
this.state.openAsset.event(asset=>void this.openAsset(asset).catch(e=>this.state.fail(e)));
this.state.openCode.event(value=>void this.openFunction(value.source,value.symbol).catch(e=>this.state.fail(e)));
this.state.changed.event(()=>{this.codeEpoch++;const r=this.state.current?.release;void this.statusBar.setElement('verstack.release',{text:r?`$(versions) ${r.repository} · ${r.edition} / ${r.version}`:'Open a working tree',alignment:StatusBarAlignment.LEFT,priority:100,command:'verstack:explorer'});});
await this.open(WorkspaceTreeWidget.ID,'left');this.shell.resize(310,'left');
await this.open(AssetsWidget.ID,'main');
await this.state.initialize();await this.shell.revealWidget(WorkspaceTreeWidget.ID);this.shell.resize(310,'left');
}
private async open<T extends Widget>(id:string,area:ApplicationShell.Area,activate=true):Promise<T> {
const widget=await this.widgets.getOrCreateWidget<T>(id);
if(!widget.isAttached){await this.shell.addWidget(widget,{area});}
if(activate){await this.shell.activateWidget(widget.id);}
return widget;
}
private async show(view:'assets'|'compare'|'code'|'jobs'|'import'|'functionDiff'|'signatures'|'catalog'):Promise<void> {
const id={assets:AssetsWidget.ID,compare:ComparisonWidget.ID,code:CodeNavigatorWidget.ID,jobs:JobsWidget.ID,import:ImportWidget.ID,functionDiff:FunctionDiffWidget.ID,signatures:SignatureLibraryWidget.ID,catalog:CatalogWidget.ID}[view];
await this.open(id,view==='compare'||view==='code'?'left':view==='jobs'?'bottom':'main');
if(view==='compare'||view==='code'){this.shell.resize(340,'left');}
}
private async openAsset(asset:Asset):Promise<void> {
await this.open(DetailsWidget.ID,'right',false);this.state.inspectAsset.fire(asset);
if(['image','audio','video'].includes(asset.kind)) {
const widget=await this.widgets.getOrCreateWidget<AssetEditorWidget>('verstack.media',{asset});
if(!widget.isAttached){await this.shell.addWidget(widget,{area:'main'});}await this.shell.activateWidget(widget.id);return;
}
const uri=(a:Asset)=>contentURI(a.snapshot,a.entry.path,basename(a.path));
if(asset.change==='changed'&&asset.before&&asset.after){await this.editors.open(DiffUris.encode(uri(asset.before),uri(asset.after),`${basename(asset.path)} — release comparison`));}
else{await this.editors.open(uri(asset));}
}
private async openFunction(source:CodeSource,symbol:SymbolRow,record=true):Promise<void> {
const epoch=++this.codeEpoch;
if(!symbol.code_path){throw new Error('This saved analysis has no code export. Select an analysis with exported code.');}
const path=source.path.replace(/functions\.json$/,symbol.code_path);
const detail=await api<CodeDetail>(fileURL(source.snapshot,path));if(epoch!==this.codeEpoch){return;}
if(record){this.history=this.history.slice(0,this.position+1);this.history.push({source,symbol});this.position=this.history.length-1;}
const decompiled=contentURI(source.snapshot,path,`${symbol.name}.c`,'decompiled');
const assembly=contentURI(source.snapshot,path,`${symbol.name}.asm`,'disassembly');
const left=this.decompiledEditor&&!this.decompiledEditor.isDisposed?this.decompiledEditor:undefined;
const right=this.assemblyEditor&&!this.assemblyEditor.isDisposed?this.assemblyEditor:undefined;
const editor=await this.editors.open(decompiled,left?{widgetOptions:{area:'main',mode:'tab-after',ref:left}}:undefined);
this.decompiledEditor=editor;editor.editor.setLanguage('verstack-c');
this.assemblyEditor=await this.editors.open(assembly,{widgetOptions:{area:'main',mode:right?'tab-after':'split-right',ref:right||editor}});this.assemblyEditor.editor.setLanguage('verstack-asm');
const refs=await this.open<ReferencesWidget>(ReferencesWidget.ID,'bottom',false);
refs.canBack=this.position>0;refs.canForward=this.position<this.history.length-1;
const travel=(delta:number)=>{const target=this.history[this.position+delta];if(target){this.position+=delta;void this.openFunction(target.source,target.symbol,false).catch(e=>this.state.fail(e));}};
refs.back=()=>travel(-1);refs.forward=()=>travel(1);
refs.navigate=address=>{
const normalize=(v:string)=>v.replace(/^0x/,'').replace(/^0+/,'').toLowerCase();
const line=detail.disassembly.split('\n').findIndex(l=>normalize(l.trim().split(/\s+/)[0])===normalize(address));
if(line>=0&&this.assemblyEditor){const position={line,character:0};this.assemblyEditor.editor.cursor=position;this.assemblyEditor.editor.revealPosition(position);void this.shell.activateWidget(this.assemblyEditor.id);}
else{void this.seek(source,address).catch(e=>this.state.fail(e));}
};
refs.set(source,symbol,detail);await this.shell.revealWidget(refs.id);this.shell.resize(220,'bottom');
}
private async seek(source:CodeSource,address:string):Promise<void> {
const rows=await api<{items:SymbolRow[]}>(`/api/code?${new URLSearchParams({snapshot:source.snapshot,path:source.path,search:address})}`);
const target=rows.items.find(s=>s.address.replace(/^0+/,'').toLowerCase()===address.replace(/^0+/,'').toLowerCase());
if(!target){throw new Error(`${address} is not an exported function entry. This may be a data reference or a location inside a function.`);}
await this.openFunction(source,target);
}
}
workbench/stern-catalog/src/browser/stern-catalog-frontend-module.ts
+22 -12
View File
@@ -1,16 +1,26 @@
import { CatalogWidget } from './catalog-view';
import { SignatureLibraryWidget } from './signature-library';
import { ContainerModule } from '@theia/core/shared/inversify';
import { SternCatalogWidget } from './stern-catalog-widget';
import { SternCatalogContribution } from './stern-catalog-contribution';
import { bindViewContribution, FrontendApplicationContribution, WidgetFactory } from '@theia/core/lib/browser';
import { FrontendApplicationContribution, WidgetFactory } from '@theia/core/lib/browser';
import { CommandContribution } from '@theia/core/lib/common/command';
import { MenuContribution } from '@theia/core/lib/common/menu';
import { ResourceResolver } from '@theia/core/lib/common/resource';
import { createTreeContainer, TreeWidget } from '@theia/core/lib/browser/tree';
import { WorkbenchState, Asset } from './workbench-state';
import { WorkspaceTree, WorkspaceTreeWidget } from './workspace-tree';
import { AssetsWidget, AssetEditorWidget, ComparisonWidget, CodeNavigatorWidget, DetailsWidget, ReferencesWidget, JobsWidget, ImportWidget, FunctionDiffWidget } from './workspace-views';
import { ArchiveResourceResolver, SternCatalogContribution } from './stern-catalog-contribution';
import '../../src/browser/style/index.css';
export default new ContainerModule(bind => {
bindViewContribution(bind, SternCatalogContribution);
export default new ContainerModule(bind=>{
bind(WorkbenchState).toSelf().inSingletonScope();
bind(SternCatalogContribution).toSelf().inSingletonScope();
bind(FrontendApplicationContribution).toService(SternCatalogContribution);
bind(SternCatalogWidget).toSelf();
bind(WidgetFactory).toDynamicValue(ctx => ({
id: SternCatalogWidget.ID,
createWidget: () => ctx.container.get<SternCatalogWidget>(SternCatalogWidget)
})).inSingletonScope();
bind(CommandContribution).toService(SternCatalogContribution);
bind(MenuContribution).toService(SternCatalogContribution);
bind(ArchiveResourceResolver).toSelf().inSingletonScope();bind(ResourceResolver).toService(ArchiveResourceResolver);
bind(WidgetFactory).toDynamicValue(ctx=>({id:WorkspaceTreeWidget.ID,createWidget:()=>createTreeContainer(ctx.container,{tree:WorkspaceTree,widget:WorkspaceTreeWidget,props:{search:true,virtualized:true,multiSelect:false}}).get(TreeWidget)})).inSingletonScope();
for(const type of [AssetsWidget,ComparisonWidget,CodeNavigatorWidget,DetailsWidget,ReferencesWidget,JobsWidget,ImportWidget,FunctionDiffWidget,SignatureLibraryWidget,CatalogWidget]) {
bind(type).toSelf();bind(WidgetFactory).toDynamicValue(ctx=>({id:type.ID,createWidget:()=>ctx.container.get(type)})).inSingletonScope();
}
bind(AssetEditorWidget).toSelf();bind(WidgetFactory).toDynamicValue(ctx=>({id:'verstack.media',createWidget:({asset}:{asset:Asset})=>{const widget=ctx.container.get(AssetEditorWidget);widget.configure(asset);return widget;}})).inSingletonScope();
});
workbench/stern-catalog/src/browser/stern-catalog-widget.tsx
+6 -5
View File
@@ -14,6 +14,7 @@ type FunctionReport = {matches:Array<{before:SymbolRow;after:SymbolRow}>;unmatch
const tabs=['Overview','Images','Movies','Sounds','Files','Code','Analysis'];
const key=(r:Release)=>JSON.stringify([r.repository,r.version,r.edition,r.generation]);
const label=(r:Release)=>[r.edition||'Edition unknown',r.version,(/^[123]$/.test(r.generation)?`SPIKE ${r.generation}`:r.generation)||'Generation unknown'].join(' · ');
function outputKind(snapshot:Snapshot,jobs:Job[]):string {const job=jobs.find(j=>j.output===snapshot.id);return job?`${job.tool} · revision ${job.config.processing_revision}`:snapshot.layer;}
function assetLabel(asset:Asset):string { const source=asset.provenance?.entry||asset.provenance?.source||asset.entry.path;const name=source.split('/').pop()||source;return /^(image\.bin|main\.pck)$/.test(name)?asset.entry.path.split('/').pop()!:name.replace(/-[a-f0-9]{32}\.ctex$/i,''); }
const fileURL=(s:string,p:string,inline=false)=>`/api/file/${encodeURIComponent(s)}?${new URLSearchParams({path:p,inline:String(inline)})}`;
async function api<T>(path:string,body?:unknown):Promise<T> {
@@ -78,7 +79,7 @@ function ToolCard({name,tool,snapshots,jobs,processing,refresh,showText}:{name:s
React.useEffect(()=>{if(!ghidra||!input){return;}let alive=true;setLoading(true);api<string[]>(`/api/executables/${input}`).then(p=>{if(alive){setExecutables(p);setExecutable(p.find(x=>x.endsWith('/game'))||p[0]||'');}}).catch(e=>alive&&setError(String(e))).finally(()=>alive&&setLoading(false));return ()=>{alive=false;};},[input,ghidra]);
async function queue(retry=false):Promise<void>{try{setError('');await api('/api/jobs',{snapshot:input,plugin:name,retry,...(ghidra?{paths:[executable]}:{})});await refresh();}catch(e){setError(String(e));}}
return <article className="tool-card"><h3>{name} <span>{status?.status.replace(/_/g,' ')||'Checking status…'}</span></h3><p>Processing revision {tool.processing_revision} · executable {tool.version} · schema {tool.output_schema}</p>
<select aria-label={`${name} input`} value={input} onChange={e=>setInput(e.target.value)}>{inputs.map(s=><option key={s.id} value={s.id}>{s.layer} · {s.id.slice(0,8)}</option>)}</select>
<select aria-label={`${name} input`} value={input} onChange={e=>setInput(e.target.value)}>{inputs.map(s=><option key={s.id} value={s.id}>{outputKind(s,jobs)} · {s.id.slice(0,8)}</option>)}</select>
{ghidra&&<label>Executable <select aria-label="Ghidra executable" value={executable} onChange={e=>setExecutable(e.target.value)}>{executables.map(p=><option key={p} value={p}>{p}</option>)}</select></label>}
<button disabled={!input||!inputs.length||loading||(ghidra&&!executable)} onClick={()=>queue()}>Queue {name}</button>
{ghidra&&!loading&&!executable&&<p>Blocked: no ELF or PE executable in this input.</p>}{error&&<p role="alert">{error}</p>}
@@ -145,18 +146,18 @@ function Catalog():React.ReactElement {
<main className="game-content">
{releaseJobs.some(j=>['queued','running'].includes(j.state))&&<div className="processing-notice" role="status"><strong>Processing this release</strong><p>{releaseJobs.filter(j=>['queued','running'].includes(j.state)).map(j=>`${j.tool}: ${j.state}${j.run?` (${processing?.runs.find(r=>r.id===j.run)?.stage||'starting'})`:''}`).join(' · ')}</p><p>Completed assets stay available while new extraction and code exports finish.</p></div>}
{tab==='Overview'&&<><h2>{release?label(release):'Release unavailable'}</h2><div className="metrics"><article><strong>{selected.length}</strong><span>Retained snapshots</span></article><article><strong>{selected.reduce((n,s)=>n+s.entry_count,0).toLocaleString()}</strong><span>Indexed entries</span></article><article><strong>{releaseJobs.filter(j=>j.state==='queued'||j.state==='running').length}</strong><span>Pending jobs</span></article></div><h2>Preservation and coverage</h2><p>Original packages and processing outputs have independent, immutable identities. This archive contains update packages; a file missing from an update may still exist in an installed game.</p>{selected.map(s=><article className="snapshot-row" key={s.id}><b>{s.layer}</b><code>{s.id}</code><span>{s.entry_count} entries</span><ul>{s.warnings.map((w,i)=><li key={i}>{w}</li>)}</ul></article>)}</>}
{['Images','Movies','Sounds','Files','Code'].includes(tab)&&<><div className="filters"><input aria-label="Search assets or symbols" placeholder={tab==='Code'?'Search symbol or address':'Search paths'} value={search} onChange={e=>navigate({search:e.target.value,page:''})}/>{tab!=='Code'&&<select aria-label="Output revision" value={route.get('source')||''} onChange={e=>navigate({source:e.target.value,page:''})}><option value="">Current active outputs</option><option value="history">All retained history</option>{selected.map(s=><option key={s.id} value={s.id}>{s.layer} · {s.id.slice(0,8)}</option>)}</select>}<span aria-live="polite">{busy?'Loading…':`${tab==='Code'?symbolTotal:total} results`}</span></div>
{['Images','Movies','Sounds','Files','Code'].includes(tab)&&<><div className="filters"><input aria-label="Search assets or symbols" placeholder={tab==='Code'?'Search symbol or address':'Search paths'} value={search} onChange={e=>navigate({search:e.target.value,page:''})}/>{tab!=='Code'&&<select aria-label="Output revision" value={route.get('source')||''} onChange={e=>navigate({source:e.target.value,page:''})}><option value="">Current active outputs</option><option value="history">All retained history</option>{selected.map(s=><option key={s.id} value={s.id}>{outputKind(s,jobs)} · {s.id.slice(0,8)}</option>)}</select>}<span aria-live="polite">{busy?'Loading…':`${tab==='Code'?symbolTotal:total} results`}</span></div>
{!coverageComplete&&<p className="coverage-note">Extraction coverage is incomplete or unverified. Empty results do not establish that this release has no {tab.toLowerCase()}.</p>}
{tab==='Code'?<>{!codeSource&&!busy&&<div className="empty-state"><h2>Code analysis is not available yet</h2><p>{releaseJobs.some(j=>j.tool.includes('ghidra')&&['queued','running'].includes(j.state))?'Ghidra is queued or running for this release. Symbols and code will appear here when the export is verified.':'Select an executable in Analysis and run Ghidra to browse its symbols, disassembly, and decompiled code.'}</p><button onClick={()=>navigate({tab:'Analysis'})}>Open Analysis</button></div>}<select aria-label="Executable analysis" value={codeSource} onChange={e=>{setCodeSource(e.target.value);navigate({code:e.target.value,page:''});}}>{assets.map(a=><option key={a.snapshot+a.entry.path} value={JSON.stringify([a.snapshot,a.entry.path])}>{releaseJobs.find(j=>j.output===a.snapshot)?.config.processing_revision?`Revision ${releaseJobs.find(j=>j.output===a.snapshot)!.config.processing_revision}`:'Legacy analysis'} · {a.entry.path} · {a.snapshot.slice(0,8)}</option>)}</select><div className="symbol-list">{symbols.map(s=><button key={s.address} onClick={()=>act(async()=>{if(!s.code_path){setText('This analysis predates code export. Run the current Ghidra revision to obtain disassembly and decompiled text.');return;}const [snapshot,path]=JSON.parse(codeSource);const detail=await api<{decompiled:string;disassembly:string;references:unknown[]}>(fileURL(snapshot,path.replace(/functions\.json$/,s.code_path)));setText(`${detail.decompiled}\n\n/* Disassembly */\n${detail.disassembly}\n\n/* References */\n${JSON.stringify(detail.references,null,2)}`);})}><code>{s.address}</code><b>{s.name}</b><span>{s.symbol_source}</span></button>)}</div></>:<div className={tab==='Files'?'file-list':'asset-grid'}>{assets.map(a=><article className="asset-card" key={a.snapshot+a.entry.path}>{a.media_type?<Media asset={a}/>:<span className="file-mark">FILE</span>}<button className="asset-name" title={a.entry.path} onClick={()=>act(()=>inspect(a))}>{tab==='Files'?a.entry.path:assetLabel(a)}</button>{(a.provenance?.status==='approximate_indexed_preview'||a.provenance?.preview_accuracy==='approximate_palette')&&<p className="coverage-note">Approximate bitmap palette</p>}<button onClick={()=>setText(JSON.stringify({snapshot:a.snapshot,artifact:a.entry.artifact,provenance:a.provenance||'Original or legacy output; inspect the run receipt'},null,2))}>Provenance</button><div className="asset-meta"><span>{(a.entry.size/1024).toFixed(1)} KB · {a.snapshot.slice(0,8)}</span><a href={fileURL(a.snapshot,a.entry.path)}>Original ↓</a></div></article>)}</div>}
{!busy&&(tab==='Code'?symbols:assets).length===0&&<p>No matching results. {tab==='Code'?'Choose an analysis or run Ghidra from Analysis.':'Try a different search or inspect processing coverage in Analysis.'}</p>}
<div className="pagination"><button disabled={page<=0} onClick={()=>navigate({page:String(page-1)})}>← Previous</button><span>Page {page+1}</span><button disabled={(page+1)*(tab==='Code'?100:24)>=(tab==='Code'?symbolTotal:total)} onClick={()=>navigate({page:String(page+1)})}>Next →</button></div></>}
{tab==='Analysis'&&<><h2>Processing tools</h2><p>Tools run against an immutable snapshot. Successful results remain available when a replacement fails.</p>{Object.entries(tools).map(([name,tool])=><ToolCard key={selectedKey+name} name={name} tool={tool} snapshots={selected} jobs={releaseJobs} processing={processing} refresh={reload} showText={setText}/>)}</>}
</main><footer className="release-bar"><label>Edition / version<select aria-label="Edition and version" value={release?key(release):''} onChange={e=>navigate({release:e.target.value,source:'',page:''})}>{releases.map(r=><option key={key(r)} value={key(r)}>{label(r)}</option>)}</select></label><button onClick={()=>{const a=selected.find(s=>s.layer==='extracted')||selected.find(s=>s.layer==='original');setBefore(a?.id||'');setAfter(gameSnapshots.find(s=>release&&key(s.release)!==key(release)&&s.layer===a?.layer)?.id||'');setDiff([]);setCompare(true);}}>Compare releases</button><button onClick={()=>act(async()=>{const found=await api<Array<Asset&{processing_revision:number|null;release:Release}>>(`/api/exports?${new URLSearchParams({...release!})}`);if(!found.length){throw new Error('No verified Ghidra export for this release. Run Ghidra from Analysis.');}setExportAssets(found);})}>Ghidra export ↓</button></footer></>}
</main><footer className="release-bar"><label>Edition / version<select aria-label="Edition and version" value={release?key(release):''} onChange={e=>navigate({release:e.target.value,source:'',page:''})}>{releases.map(r=><option key={key(r)} value={key(r)}>{label(r)}</option>)}</select></label><button onClick={()=>{const a=selected.find(s=>s.layer==='extracted')||selected.find(s=>s.layer==='original');setBefore(a?.id||'');setAfter(gameSnapshots.find(s=>release&&key(s.release)!==key(release)&&s.layer===a?.layer)?.id||'');setDiff([]);setCompare(true);}}>Compare releases</button><button onClick={()=>act(async()=>{let found=await api<Array<Asset&{processing_revision:number|null;release:Release}>>(`/api/exports?${new URLSearchParams({...release!})}`);if(tab==='Code'&&codeSource){const [snapshot,path]=JSON.parse(codeSource);found=found.filter(a=>a.snapshot===snapshot&&a.entry.path===path.replace(/functions\.json$/,'program.gzf'));}if(!found.length){throw new Error('No verified Ghidra export for the selected analysis. Run Ghidra from Analysis.');}setExportAssets(found.sort((a,b)=>(b.processing_revision||0)-(a.processing_revision||0)));})}>Ghidra export ↓</button></footer></>}
<dialog ref={modalRef} className="stern-dialog" onCancel={close}>{error&&<p role="alert" className="error">{error}</p>}<button className="close-dialog" aria-label="Close dialog" onClick={close}>×</button>
{exportAssets.length>0&&<><h2>Verified Ghidra exports</h2>{exportAssets.map(a=><article className="snapshot-row" key={a.snapshot+a.entry.path}><p>{label(a.release)} · analysis revision {a.processing_revision??'legacy'}</p><code>{a.entry.path}</code><p>Snapshot {a.snapshot} · Run {a.run}</p><a href={fileURL(a.snapshot,a.entry.path)}>Download verified .gzf</a></article>)}</>}
{lightbox&&<><h2>{lightbox.entry.path}</h2><Media asset={lightbox} large/><p>Snapshot <code>{lightbox.snapshot}</code> · Run <code>{lightbox.run}</code></p><a href={fileURL(lightbox.snapshot,lightbox.entry.path)}>Download original</a></>}
{lightbox&&<><h2 title={lightbox.entry.path}>{assetLabel(lightbox)}</h2>{lightbox.preview&&<p>Browser preview · original source retained</p>}<Media asset={lightbox} large/><p>Snapshot <code>{lightbox.snapshot}</code> · Run <code>{lightbox.run}</code></p><a href={fileURL(lightbox.snapshot,lightbox.entry.path)}>Download original</a></>}
{text!==null&&<><h2>Read-only inspection</h2><Monaco text={text}/></>}
{compare&&<><h2>Compare {game} releases</h2><p>Select exact output revisions. Coverage differences are not proof of publisher changes.</p><div className="compare-select">{[before,after].map((value,i)=><label key={i}>Release {i===0?'A':'B'}<select disabled={busy} aria-label={`Release ${i===0?'A':'B'}`} value={value} onChange={e=>i===0?setBefore(e.target.value):setAfter(e.target.value)}><option value="">Select output…</option>{gameSnapshots.map(s=><option key={s.id} value={s.id}>{label(s.release)} · {s.layer} · {s.id.slice(0,8)}</option>)}</select></label>)}</div><button disabled={!before||!after||busy||gameSnapshots.find(s=>s.id===before)?.layer!==gameSnapshots.find(s=>s.id===after)?.layer} onClick={()=>act(async()=>setDiff(await api(`/api/compare?${new URLSearchParams({before,after})}`)))}>Compare selected outputs</button><button disabled={!before||!after||busy} onClick={()=>act(async()=>{const [a,b]=await Promise.all([api<{entries:Array<{path:string}>}>(`/api/snapshots/${before}`),api<{entries:Array<{path:string}>}>(`/api/snapshots/${after}`)]);const ap=a.entries.find(e=>/(^|\/)functions\.json$/.test(e.path)),bp=b.entries.find(e=>/(^|\/)functions\.json$/.test(e.path));if(!ap||!bp){throw new Error('Select two Ghidra analysis outputs for function comparison.');}const result=await api<FunctionReport>(`/api/functions/compare?${new URLSearchParams({before,before_path:ap.path,after,after_path:bp.path})}`);setFunctionReport(result);})}>Compare functions</button>{functionReport&&<FunctionComparison key={before+after} report={functionReport}/>}<p>{gameSnapshots.find(s=>s.id===before)?.layer!==gameSnapshots.find(s=>s.id===after)?.layer?'Choose outputs from the same preservation layer.':`${diff.length} file differences`}</p>{fileDiff&&<MonacoDiff before={fileDiff.before} after={fileDiff.after}/>}{diff.map(d=><article key={d.path} className="diff-row"><b>{d.kind}</b><span>{d.path}</span><div className="paired-media">{[before,after].map((snapshot,i)=>((i===0&&d.kind==='added')||(i===1&&d.kind==='removed'))?<span key={i}>{i===0?'A':'B'}: absent from this output</span>:<div key={i}>{mime(d.path)&&<Media asset={{snapshot,entry:{path:d.path,size:0,artifact:''},media_type:mime(d.path),run:''}}/>}<a href={fileURL(snapshot,d.path)}>{i===0?'A':'B'} original</a></div>)}</div>{!mime(d.path)&&<button onClick={()=>act(async()=>{const read=async(snapshot:string)=>{const r=await fetch(fileURL(snapshot,d.path),{headers:{Range:'bytes=0-65535'}});return r.ok?await r.text():'Absent or not a regular file';};const [a,b]=await Promise.all([read(before),read(after)]);setFileDiff({before:a,after:b});})}>Inspect difference</button>}</article>)}</>}
{compare&&<><h2>Compare {game} releases</h2><p>Select exact output revisions. Coverage differences are not proof of publisher changes.</p><div className="compare-select">{[before,after].map((value,i)=><label key={i}>Release {i===0?'A':'B'}<select disabled={busy} aria-label={`Release ${i===0?'A':'B'}`} value={value} onChange={e=>i===0?setBefore(e.target.value):setAfter(e.target.value)}><option value="">Select output…</option>{gameSnapshots.map(s=><option key={s.id} value={s.id}>{label(s.release)} · {outputKind(s,jobs)} · {s.id.slice(0,8)}</option>)}</select></label>)}</div><button disabled={!before||!after||busy||gameSnapshots.find(s=>s.id===before)?.layer!==gameSnapshots.find(s=>s.id===after)?.layer} onClick={()=>act(async()=>setDiff(await api(`/api/compare?${new URLSearchParams({before,after})}`)))}>Compare selected outputs</button><button disabled={!before||!after||busy} onClick={()=>act(async()=>{const [a,b]=await Promise.all([api<{entries:Array<{path:string}>}>(`/api/snapshots/${before}`),api<{entries:Array<{path:string}>}>(`/api/snapshots/${after}`)]);const ap=a.entries.find(e=>/(^|\/)functions\.json$/.test(e.path)),bp=b.entries.find(e=>/(^|\/)functions\.json$/.test(e.path));if(!ap||!bp){throw new Error('Select two Ghidra analysis outputs for function comparison.');}const result=await api<FunctionReport>(`/api/functions/compare?${new URLSearchParams({before,before_path:ap.path,after,after_path:bp.path})}`);setFunctionReport(result);})}>Compare functions</button>{functionReport&&<FunctionComparison key={before+after} report={functionReport}/>}<p>{gameSnapshots.find(s=>s.id===before)?.layer!==gameSnapshots.find(s=>s.id===after)?.layer?'Choose outputs from the same preservation layer.':`${diff.length} file differences`}</p>{fileDiff&&<MonacoDiff before={fileDiff.before} after={fileDiff.after}/>}{diff.map(d=><article key={d.path} className="diff-row"><b>{d.kind}</b><span>{d.path}</span><div className="paired-media">{[before,after].map((snapshot,i)=>((i===0&&d.kind==='added')||(i===1&&d.kind==='removed'))?<span key={i}>{i===0?'A':'B'}: absent from this output</span>:<div key={i}>{mime(d.path)&&<Media asset={{snapshot,entry:{path:d.path,size:0,artifact:''},media_type:mime(d.path),run:''}}/>}<a href={fileURL(snapshot,d.path)}>{i===0?'A':'B'} original</a></div>)}</div>{!mime(d.path)&&<button onClick={()=>act(async()=>{const read=async(snapshot:string)=>{const r=await fetch(fileURL(snapshot,d.path),{headers:{Range:'bytes=0-65535'}});return r.ok?await r.text():'Absent or not a regular file';};const [a,b]=await Promise.all([read(before),read(after)]);setFileDiff({before:a,after:b});})}>Inspect difference</button>}</article>)}</>}
{importOpen&&<form onSubmit={e=>{e.preventDefault();act(async()=>{await api('/api/import',{path:importPath,release:{repository:importGame,version:importVersion,edition:importEdition,generation:importGeneration}});await reload();close();navigate({game:importGame,tab:'Overview',release:key({repository:importGame,version:importVersion,edition:importEdition,generation:importGeneration}),source:'',search:'',page:'',code:''});});}}><h2>Import package</h2><p>Select a path within the server’s configured import roots.</p>{[['Server path',importPath,setImportPath],['Game',importGame,setImportGame],['Version',importVersion,setImportVersion],['Edition',importEdition,setImportEdition],['Generation',importGeneration,setImportGeneration]].map(([name,value,setter])=><label key={name as string}>{name as string}<input required={['Server path','Game','Version'].includes(name as string)} value={value as string} onChange={e=>(setter as React.Dispatch<React.SetStateAction<string>>)(e.target.value)}/></label>)}<button disabled={busy} type="submit">{busy?'Importing…':'Import and queue processing'}</button></form>}
</dialog>
</div>;
workbench/stern-catalog/src/browser/style/index.css
+83 -3
View File
File diff suppressed because one or more lines are too long
workbench/stern-catalog/src/browser/view-controls.tsx
+36
View File
@@ -0,0 +1,36 @@
import * as React from 'react';
import { ContextMenuAccess, ContextMenuRenderer } from '@theia/core/lib/browser/context-menu-renderer';
import { WorkbenchState, AssetViewOptions } from './workbench-state';
export const ASSET_VIEW_MENU=['verstack-asset-view'];
export const IMAGE_VIEW_MENU=['verstack-image-view'];
export function useViewOptions(state:WorkbenchState):AssetViewOptions {
const [,update]=React.useReducer(n=>n+1,0);
React.useEffect(()=>{const listener=state.viewChanged.event(()=>update());return()=>listener.dispose();},[state]);
return state.viewOptions;
}
export function ViewOptionsButton({state,menus,imageOnly=false}:{state:WorkbenchState;menus:ContextMenuRenderer;imageOnly?:boolean}):React.ReactElement {
const options=useViewOptions(state);const [open,setOpen]=React.useState(false);const menu=React.useRef<ContextMenuAccess|undefined>(undefined);
React.useEffect(()=>()=>menu.current?.dispose(),[]);
return <button type="button" aria-label={imageOnly?'Image view options':'Asset view options'} aria-haspopup="menu" aria-expanded={open} onClick={event=>{
if(menu.current&&!menu.current.disposed){menu.current.dispose();return;}
const button=event.currentTarget,bounds=button.getBoundingClientRect();setOpen(true);
menu.current=menus.render({menuPath:imageOnly?IMAGE_VIEW_MENU:ASSET_VIEW_MENU,anchor:{x:bounds.left,y:bounds.bottom},context:button,onHide:()=>{menu.current=undefined;setOpen(false);}});
}}>View{options.dmdTint?' · DMD':''} <span className="codicon codicon-chevron-down" aria-hidden="true"/></button>;
}
export function Pagination({page,total,pageSize,disabled=false,onPage,label}:{page:number;total:number;pageSize:number;disabled?:boolean;onPage:(page:number)=>void;label:string}):React.ReactElement {
const pages=Math.max(1,Math.ceil(total/pageSize));
const [draft,setDraft]=React.useState(String(page+1));const id=React.useId();
React.useEffect(()=>setDraft(String(page+1)),[page,total,pageSize]);
return <div className="vs-pagination" role="group" aria-label={`${label} pagination`}>
<button type="button" disabled={disabled||page<=0} onClick={()=>onPage(page-1)}>Previous</button>
<form onSubmit={event=>{event.preventDefault();const selected=Number(draft);if(!disabled&&Number.isInteger(selected)&&selected>=1&&selected<=pages){onPage(selected-1);}}}>
<label htmlFor={id}>Page</label><input id={id} aria-label={`${label} page`} className="theia-input" type="number" inputMode="numeric" min={1} max={pages} step={1} required disabled={disabled||!total} value={draft} onChange={event=>setDraft(event.target.value)} onKeyDown={event=>{if(event.key==='Escape'){setDraft(String(page+1));}}}/>
<span>of {pages.toLocaleString()}</span><button type="submit" disabled={disabled||!total}>Go</button>
</form>
<button type="button" disabled={disabled||page+1>=pages} onClick={()=>onPage(page+1)}>Next</button>
</div>;
}
workbench/stern-catalog/src/browser/workbench-state.ts
+62
View File
@@ -0,0 +1,62 @@
import { injectable } from '@theia/core/shared/inversify';
import { Emitter } from '@theia/core/lib/common/event';
export type Release = {repository:string;version:string;edition:string;generation:string};
export type Workspace = {id:string;release:Release;extracted?:boolean;file_snapshot?:string;warnings:string[];sources:Array<{snapshot:string;operation:string;warnings:string[]}>;code:CodeSource[]};
export type CodeSource = {snapshot:string;path:string;name:string;source_path?:string;size?:number;language?:string};
export type Asset = {path:string;name?:string;directory?:boolean;kind:string;change?:string;snapshot:string;entry:{path:string;size:number;artifact:string};media_type?:string;preview?:{snapshot:string;path:string};provenance?:Record<string,unknown>;before?:Asset;after?:Asset};
export type Page = {items:Asset[];total:number;page:number;page_size:number;counts:Record<string,number>;profiles_compatible:boolean};
export type AssetViewOptions = {layout:'grid'|'list';dmdTint:boolean};
function savedViewOptions():AssetViewOptions {
try {const saved=JSON.parse(localStorage.getItem('verstack.asset-view')||'{}');return {layout:saved.layout==='list'?'list':'grid',dmdTint:saved.dmdTint===true};}
catch {return {layout:'grid',dmdTint:false};}
}
export type SymbolRow = {name:string;address:string;size:number;code_path?:string;symbol_source:string};
export type CodeDetail = {address:string;decompiled:string;disassembly:string;decompile_status:string;references:Array<{from:string;to:string;type:string}>};
export const releaseLabel=(r:Release)=>`${r.repository} · ${r.edition} / ${r.version}`;
export const fileURL=(s:string,p:string,inline=false)=>`/api/file/${encodeURIComponent(s)}?${new URLSearchParams({path:p,inline:String(inline)})}`;
export async function api<T>(url:string):Promise<T> {const response=await fetch(url,{signal:AbortSignal.timeout(60000)});if(!response.ok){throw new Error(await response.text());}return response.json();}
export function mediaURL(a:Asset):string {return a.preview?fileURL(a.preview.snapshot,a.preview.path,true):fileURL(a.snapshot,a.entry.path,true);}
export const basename=(p:string)=>p.split('/').pop()||p;
export const sizeLabel=(size:number)=>size>=1048576?`${(size/1048576).toFixed(1)} MB`:size>=1024?`${(size/1024).toFixed(1)} KB`:`${size} B`;
@injectable()
export class WorkbenchState {
workspaces:Workspace[]=[];
current?:Workspace;
baseline='';prefix='';kind='';change='';search='';error='';busy=false;
readonly changed=new Emitter<void>();
readonly viewChanged=new Emitter<void>();
viewOptions:AssetViewOptions=savedViewOptions();
readonly openAsset=new Emitter<Asset>();
readonly inspectAsset=new Emitter<Asset>();
readonly openCodeDiff=new Emitter<{sources:CodeSource[];before?:SymbolRow;after?:SymbolRow;view:string}>();
readonly openCode=new Emitter<{source:CodeSource;symbol:SymbolRow}>();
readonly showView=new Emitter<'assets'|'compare'|'code'|'jobs'|'import'|'functionDiff'|'signatures'|'catalog'>();
private epoch=0;
async initialize():Promise<void> {
try {
this.workspaces=await api<Workspace[]>('/api/workspaces');
const params=new URLSearchParams(location.search);
const remembered=localStorage.getItem('verstack.workspace');
const selected=this.workspaces.find(w=>w.id===params.get('workspace'))||this.workspaces.find(w=>w.id===remembered)||this.workspaces.find(w=>w.release.repository===params.get('game'))||this.workspaces[this.workspaces.length-1];
if(selected){await this.select(selected.id);}
} catch(e) {this.fail(e);}
}
async select(id:string):Promise<void> {
const epoch=++this.epoch;this.busy=true;this.error='';this.current=undefined;this.baseline='';this.prefix='';this.change='';this.search='';this.kind='';this.changed.fire();
try {const workspace=await api<Workspace>(`/api/workspaces/${id}`);if(epoch!==this.epoch){return;}this.current=workspace;localStorage.setItem('verstack.workspace',id);const url=new URL(location.href);url.searchParams.set('workspace',id);url.searchParams.delete('release');history.replaceState({},'',url);}
catch(e){if(epoch===this.epoch){this.error=String(e);}}
finally{if(epoch===this.epoch){this.busy=false;this.changed.fire();}}
}
set(values:Partial<Pick<WorkbenchState,'baseline'|'prefix'|'kind'|'change'|'search'>>):void {Object.assign(this,values);this.error='';this.changed.fire();}
setViewOptions(values:Partial<AssetViewOptions>):void {
this.viewOptions={...this.viewOptions,...values};
try {localStorage.setItem('verstack.asset-view',JSON.stringify(this.viewOptions));}catch {/* The current view still works when browser storage is unavailable. */}
this.viewChanged.fire();
}
fail(error:unknown):void {this.error=String(error);this.changed.fire();}
async files(options:Record<string,string|number|boolean>={}):Promise<Page> {
if(!this.current){return {items:[],total:0,page:0,page_size:120,counts:{},profiles_compatible:true};}
const query=new URLSearchParams({before:this.baseline,prefix:this.prefix,kind:this.kind,change:this.change,search:this.search,...Object.fromEntries(Object.entries(options).map(([k,v])=>[k,String(v)]))});
return api<Page>(`/api/workspaces/${this.current.id}/files?${query}`);
}
}
workbench/stern-catalog/src/browser/workspace-tree.tsx
+46
View File
@@ -0,0 +1,46 @@
import * as React from 'react';
import { injectable, inject, postConstruct } from '@theia/core/shared/inversify';
import { TreeImpl, TreeNode, CompositeTreeNode, TreeWidget, NodeProps } from '@theia/core/lib/browser/tree';
import { WorkbenchState, Asset, basename, releaseLabel } from './workbench-state';
interface FileNode extends TreeNode {asset?:Asset;folder?:string;offset?:number;}
@injectable()
export class WorkspaceTree extends TreeImpl {
@inject(WorkbenchState) protected readonly state!:WorkbenchState;
protected async resolveChildren(parent:CompositeTreeNode):Promise<TreeNode[]> {
const node=parent as CompositeTreeNode&FileNode;
if(!this.state.current){return [];}
const epoch=this.state.current.id;
try {
const page=await this.state.files({directory:true,prefix:node.folder||'',page:node.offset||0});
if(this.state.current?.id!==epoch){return [];}
const nodes:TreeNode[]=page.items.map(asset=>({id:`${epoch}:${asset.path}`,name:basename(asset.path),parent,selected:false,...(asset.directory?{children:[],expanded:false,folder:asset.path}:{asset})}));
if((page.page+1)*page.page_size<page.total){nodes.push({id:`${node.id}:more:${page.page+1}`,name:'More files…',parent,selected:false,children:[],expanded:false,folder:node.folder||'',offset:page.page+1} as CompositeTreeNode&FileNode);}
return nodes;
}catch(e){this.state.fail(e);return [];}
}
}
@injectable()
export class WorkspaceTreeWidget extends TreeWidget {
static readonly ID='verstack.explorer';
@inject(WorkbenchState) protected readonly state!:WorkbenchState;
@postConstruct() protected init():void {
super.init();
this.id=WorkspaceTreeWidget.ID;this.title.label='Explorer';this.title.caption='Release files and assets';this.title.iconClass='codicon codicon-files';this.addClass('vs-explorer');
const reset=()=>{this.model.root={id:'workspace-root',name:'',parent:undefined,visible:false,children:[],expanded:true} as CompositeTreeNode;this.update();};
this.toDispose.push(this.state.changed.event(reset));
this.toDispose.push(this.model.onOpenNode(n=>{const node=n as FileNode;if(node.asset){this.state.openAsset.fire(node.asset);}else if(node.folder && !node.offset){this.state.set({prefix:node.folder});this.state.showView.fire('assets');}}));
this.toDispose.push(this.model.onSelectionChanged(nodes=>{const node=nodes[0] as FileNode|undefined;if(node?.asset){this.state.inspectAsset.fire(node.asset);}}));
reset();
}
protected renderIcon(node:TreeNode,_props:NodeProps):React.ReactNode {
const n=node as FileNode;const icon=n.folder!==undefined?'folder':({image:'file-media',audio:'unmute',video:'device-camera-video'} as Record<string,string>)[n.asset?.kind||'']||'file';
return <span className={`codicon codicon-${icon} vs-tree-icon`} aria-hidden="true"/>;
}
protected renderTailDecorations(node:TreeNode,props:NodeProps):React.ReactNode {
const change=(node as FileNode).asset?.change;
return change?<span className={`vs-change ${change}`} title={change}>{({added:'A',removed:'D',changed:'M',unknown:'?' } as Record<string,string>)[change]||''}</span>:super.renderTailDecorations(node,props);
}
protected render():React.ReactNode {
return <div className="vs-tree-layout"><div className="vs-explorer-header"><label htmlFor="working-tree">Working tree</label><select id="working-tree" className="theia-select" value={this.state.current?.id||''} disabled={this.state.busy} onChange={e=>void this.state.select(e.target.value)}><option value="" disabled>{this.state.busy?'Opening…':'Select release'}</option>{this.state.workspaces.map(w=><option key={w.id} value={w.id}>{releaseLabel(w.release)}</option>)}</select><div className="vs-actions"><button className="theia-button secondary" onClick={()=>{this.state.set({prefix:'',kind:''});this.state.showView.fire('assets');}}>Browse assets</button><button className="theia-button secondary" onClick={()=>this.state.showView.fire('compare')}>Compare</button></div><div className="vs-actions"><button className="vs-link" onClick={()=>this.state.showView.fire('import')}>Import release</button></div><div className="vs-actions"><button className="vs-link" onClick={()=>this.state.showView.fire('code')}>Open code browser</button><button className="vs-link" onClick={()=>this.state.showView.fire('jobs')}>Jobs</button></div><div className="vs-actions"><button className="vs-link" onClick={()=>this.state.showView.fire('signatures')}>Function signatures</button><button className="vs-link" onClick={()=>this.state.showView.fire('catalog')}>Catalog &amp; storage</button></div>{this.state.baseline&&<div className="vs-note">Comparison filter: {this.state.change||'all files'} <button className="vs-link" onClick={()=>this.state.set({baseline:'',change:''})}>Clear</button></div>}{this.state.error&&<p role="alert" className="vs-error">{this.state.error}</p>}</div><div className="vs-tree-body">{super.render()}</div></div>;
}
}
workbench/stern-catalog/src/browser/workspace-views.tsx
+188
View File
File diff suppressed because one or more lines are too long