verstack/docs/redesign-status.md

Catalog and workbench redesign

This implementation is not yet accepted against the full 73-package plan. The current static UI remains available; the Theia browser application runs separately. Local evidence lives under ignored data/validation/ and contains firmware-derived material that must not be committed.

Implemented

• Local Git source baseline 0b1a104; archive/tool/credential/build exclusions.
• Full pre-migration archive backup at data/archive-before-catalog, 221 files and 5,423,176,170 bytes verified by SHA-256. Receipt: archive-backup-receipt.json.
• WAL SQLite catalog (schema 5) with restartable JSON reconciliation, indexed library/artifacts/symbols, durable jobs, configured automatic extraction/preview scheduling, frozen processing settings and semantic revisions, deduplication, explicit retries, successful active-output replacement and startup queue-gap repair. Configured source trees are captured and readback-verified before queuing; pending jobs execute their captured Python/Java dependencies.
• Migrated real archive: 11 snapshots and 14 runs; identities verified in catalog-migration-receipt.json. Baseline executable retained locally in tools/bin/verstack-before-catalog.
• Same-origin Theia/React game workbench with seven tabs, persistent release controls, immutable snapshot comparison, media galleries, Monaco code/difference panels, tool/run history and verified GZF downloads.
• ZIP/split SquashFS, SPKS and existing LUKS/ext4 adapters, plus gzip-bootstrap/SPKS dispatch. All four previously rejected gzip packages now pass extraction/readback (*.tar-receipt.json).
• Node Intel HEX adapter, Radium PCM/indexed previews, structurally selected ELF sample-rate evidence, Godot PCK/embedded WebP/RSCC/font resources, BC1/BC3 scene textures, SPIKE1 PCM and FFmpeg previews. Exact source assets and format/provenance evidence remain retained. Bitmap revision 4 decodes raw, whitespace, delta and four-bit indices; see hash-bound evidence. Node address metadata and SPIKE2 audio source offsets are indexed alongside media provenance.
• Ghidra revision 2 exports per-function decompilation, disassembly and references; reopen verification compares exported symbols and input identity. Synthetic native export verified 16 exported symbols (ghidra-revision2/).
• Reference-aware orphan cleanup, durable recovery journal and measured physical reclamation; tests retain shared content across games.

Verification

Rust passes 29 integration tests and an inherited-lock-descriptor regression test covering captured-source execution, migration/recovery, deduplication, semantic revision replacement, and shared-reference cleanup. The 33 Python fixture checks cover extraction bounds/checksums/paths, PCM metadata, malformed DMD records and gzip/tar validation. Frontend build/type checks pass. The browser regression passed import → gallery → audio/video playback → history → compare → analysis → Monaco code → verified GZF download at desktop and 390px width; captures are under data/validation/workbench/screenshots/.

All 73 packages have SHA-256 inventory evidence (corpus-content.json). The completed package/media pass had 73 successful extractions, 67 partial media results and six complete results within that pass's scope. Receipts under corpus-extraction/ reported media revision 3; a revision 4 refresh completed 14 packages and is paused while the live archive is repaired. The GOT Pro refresh decoded 17,159 bitmap records and 1,940 PCM tracks, with menu/non-audio coverage still incomplete. Do not interpret these receipts as archive imports or full media acceptance. The optional --all-families validator also runs node and SPIKE2 audio adapters and preserves hash-bound representative FFmpeg readback samples; it has passed a fixture orchestration check but has not completed the corpus.

Real Ghidra revision 2 exports reopened with 11,549 Pro and 11,724 LE exported symbols verified. Their compatible Game of Thrones 1.37 comparison is retained in the disposable data/validation/edition-comparison/ archive: 5,526 unique exact function-body matches, 6,023 unmatched Pro functions and 6,198 unmatched LE functions. Unmatched is not evidence of additions/removals or semantic correspondence.

Pokémon LE 0.85 validation emitted 2,545 WAV outputs across game and menu, with 44,100 Hz proven by explicit ALSA calls in the matching ELF binaries. Approximate indexed bitmap previews are labelled; they are not confirmed final palettes. Reference wiki counts apply only to the exact binary hashes cited there. The local wiki is /home/jordan/pokemon-triage-wiki and remains unchanged.

Acceptance blockers

• Finish and rerun all 73 extraction inventories with the latest decoder revisions; validate every exposed media family, not just its file inventory. Final indexed palette semantics, menu layouts, encrypted SPIKE2 containers and other unsupported records remain visible blockers.
• Aerosmith main-image audio decoded all 1,522 sounds, but its menu codec remains unsupported. Broaden real node, audio and preview evidence across the full corpus; preserve the explicit upstream node4 limitation.
• Wrapper retirement is deliberately disabled: receipts currently set wrapper_removal_authorized=false. Full payload coverage, retirement/reimport state and crash-safe reclamation of published wrappers are not implemented. Orphan cleanup does not reclaim retained wrappers.
• Galleries default to current active outputs and hide exact source copies and derivative thumbnails; retained history remains selectable. Validate database browsing/resource limits against the full corpus before replacing the default UI.
• Workspace limits are polled, the import pipeline runs stages in order, logs are capped, and decoder workers/FFmpeg threads are bounded. DMD compression and image thumbnails run concurrently within a stage. These are not kernel-enforced memory/disk quotas.

Operation and recovery

Back up the entire archive while the service is idle/stopped, including its key, store, SQLite files, manifests, runs and logs. Do not remove SQLite casually: manifests rebuild browsing, but queued jobs and active-output scheduling state live in the database. Archive::open transactionally reconciles immutable manifests, resolves interrupted runs/jobs and repairs automatic submission gaps. Failed or interrupted processing requires explicit retry.

Generate a reviewable pipeline config using python3 scripts/configure_pipeline.py config.json pipeline.local.json. It reuses existing tool/key-file references; it does not embed credential contents. Inspect paths and resource settings before starting the service with it. An explicit import with code analysis enabled selects game programs automatically; opening existing releases does not submit analysis.

Use jobs, queue INPUT TOOL [--retry] [--priority -10..10], and work for CLI processing while the service is stopped. cleanup-orphans preserves all published manifests and backend snapshots they reference. A cleanup interrupted after its journal is written resumes on next open. Preserve original filesystem source collections.

Run the broader disposable validation with the pinned decoder environment:

tools/decoder-env/bin/python scripts/validate_corpus.py /srv/firmware/images/stern_game_code data/validation/corpus-all-families --all-families --refresh-stale

Run one corpus worker at a time. This mode decodes all supported SPIKE2 sounds, checks node records, and samples one detected media type per decoder output for FFmpeg readback. Sampled playback is recorded explicitly and does not certify unsampled assets. Retained source bundles are part of archive backup/recovery; cleanup currently does not reclaim unused tool bundles.

Live rollout corrections

The first rollout left the live archive on its old plugin configuration. Decoder validation outputs in disposable archives were not sufficient evidence that the user could browse those assets. The live service now uses the configured revision 4 media decoder and an optimized Rust build. Pokémon LE 0.85.0 has 4,916 indexed images and 2,545 audio assets; Game of Thrones 1.37.0 Pro and LE each have 17,159 bitmap previews and 1,940 audio assets. Approximate palettes remain labelled. Pokémon's hash-bound main/menu PCM and bitmap counts are recorded locally in data/validation/pokemon-live-wiki-counts.json.

Processing status uses manifest-only executable candidates instead of opening every archived asset. Explicit executable selection verifies file magic. Indexed file lookup, buffered manifest/asset reads and a bounded directory cache remove the repeated full-manifest and directory parsing that delayed gallery downloads. Readback verification clears the storage cache first. Regression checks cover status without payload access, priority persistence, initial legacy-profile freshness, and interruption of descendant processes during shutdown.

SIGINT/SIGTERM stop processing descendants and retain interrupted job history; retry is explicit. Selected Ghidra analysis stages only the selected executables. The workspace monitor tolerates temporary files disappearing during enumeration, while final inventories still require readable outputs. Compatible H.264/AAC videos receive complete-decode verification without reencoding; identical inputs reuse a conversion. Original source assets remain downloadable.

Function comparison now displays searchable, paginated exact-match and unmatched lists. Exact matches open paired Monaco decompilation/disassembly/reference views; unmatched functions open individually. Changing either immutable output selection clears previous comparison evidence. The read-only browser regression verifies 5,526 matches using complete real GoT Pro/LE exports in a disposable archive.

The live Pokémon preview revision 3 completed with 274 verified conversion records and 30 verified browser-compatible originals, with zero failed records. Identical inputs share 255 converted files. All 291 visible movie assets have either their verified preview attached (261 HEVC assets) or a compatible H.264 original (30). The live Chromium check passes images, audio, gallery video and enlarged video; local evidence is under data/validation/preview-v3/ and the workbench screenshots.

Both live GoT edition exports are now published and verified: 11,549 Pro symbols and 11,724 LE symbols, including reopened GZF exports. The read-only comparison regression also passes against the live archive with 5,526 exact matches. Live Code checks verify symbol search, Monaco views, and the selected GZF download. Pokémon's current Ghidra job is still analyzing; its Code export is pending.

On September 13, the user authorized persistent LAN services. Both deployment units are enabled and user lingering is enabled. The workbench is running under systemd on port 3000, with the reported Pokémon processing URL returning HTTP 200. A one-time systemd handover task waits for the existing backend queue to become idle before gracefully stopping that process and starting the persistent backend. This preserves the active Pokémon analysis. Handover status is available with systemctl --user status verstack-backend-handover.

The 73-package revision-4 media refresh remains in progress in a disposable archive. Unresolved media families remain acceptance blockers; these live fixes do not establish full-corpus completion.

Native working-tree redesign — September 13 follow-up

The earlier seven-tab React application is no longer mounted in Theia. The live workbench now uses a native virtualized Explorer, separate Assets/Compare/Code/ Properties/Processing views, dockable media editor tabs, and Theia resource-backed text, hex, C, assembly and difference editors. The release selector and native status bar show game + edition/version (for example Pokémon · LE / 0.85.0). Intermediate extraction snapshots appear only in provenance and history.

Release APIs resolve the retained extraction chain and current successful outputs. Added/Removed/Changed/Unchanged filters apply to both Explorer and the asset view; removed items read from the older release. Common package wrapper paths are removed. Media comparisons use original asset identities and mark incompatible decoder revision/schema/settings as unknown coverage. Unmatched code remains unmatched rather than being mislabelled as publisher additions/removals.

Validation: all 34 Rust tests pass; three Python RAM-policy checks pass; TypeScript and Theia bundle builds pass. The read-only native browser regression verifies Pokémon's 4,916 images, native media editors, audio/video playback, C/assembly editors, branch navigation, and GoT's 5,526 exact function matches in native diff editors, with zero API mutations. A separate tiny RAM-only fixture archive verifies Added-only, Removed-only image readback from 0.81.0, and changed-text differences against 0.85.0. These fixture versions are synthetic, not Pokémon release evidence. Scripts: tests/ui-native-workbench.mjs, tests/ui-release-comparison.mjs, tests/workspace.rs, tests/test_ram_workspace.py. Screenshots are retained locally under data/validation/native-workbench/.

The live configuration now uses /tmp/verstack-workspace on the host's tmpfs with a 64 GiB allowance. RAM scratch is required and verified at startup and processing; there is no automatic disk fallback. Child temporary/cache paths stay inside the run; Python bytecode writes are disabled; bounded logs buffer in memory before saving. The backend service sets MemorySwapMax=0 so its scratch pages cannot spill to swap. Archive publication, manifests, job history and completed logs remain durable. Standalone corpus validation also checks RAM storage and uses it for all extraction/preview work, persisting only receipts. For standalone validation, use a scope with MemorySwapMax=0 if swap must be prohibited as well.

processing_enabled=false pauses automatic submission and worker execution while preserving history. No new live jobs were submitted during this redesign. Pokémon's previous Ghidra analysis is interrupted; it has no completed live code export. Pokémon LE 0.81 source links were located in /srv/firmware/MORE_FIRMWARE.txt; that release was not downloaded or extracted. Its real comparison with 0.85 remains unverified until it is imported. The Code browser displays retained static analysis; a debugger, control-flow graph, strings/import indexing and semantic matching are not implemented. Existing decoder/full-corpus acceptance limits above remain.

Video thumbnails now ask the browser decoder for the first frame when they enter the visible area. They remain paused and retain playback controls; offscreen videos defer loading. This uses the existing compatible video preview/original and creates no new extraction job or saved thumbnail file. The targeted tests/ui-video-preview.mjs regression covers paused frame readiness, deferred loading and playback.

Import workflow (September 13, 2026)

The native Import release view now browses configured server roots, supports multiple files or a whole folder, and inspects direct HTTP(S) URLs and Internet Archive item links. Archive items show original downloadable filenames and sizes; inspection does not start a download. Release names remain game, edition, version.

Explicit imports use a durable queue independent of the paused automatic pipeline. The selected workflow retains originals, extracts recognized containers, decodes assets/prepares previews, and optionally analyzes supported ELF/PE programs. Jobs polls live progress, keeps history across navigation/reloads, supports cancel and retry of unfinished steps, and opens the resulting release. Retry preserves completed original/extraction steps. A restarted running import is interrupted until explicitly retried. Existing automatic jobs remain paused.

Progress is measured for download bytes/speed, archive bytes, file iteration, LUKS decryption, and exported Ghidra functions. Steps with no measurable total show an indeterminate bar and elapsed time. Live progress control files and logs stay in RAM; durable task state is written at transitions, not on every progress tick. Input staging also avoids per-file/per-tick persistent progress writes.

The new import-extract plugin handles nested ZIPs, recognized SPK wrappers, MBR/GPT disk partitions, ext filesystems, FAT12/16/32, and configured LUKS2 volumes. It does not mount images or execute imported programs. Other files, including unsupported console ROMs, remain cataloged; Ghidra import automation currently selects ELF/PE only. Extended MBR partition chains, unsupported encryption and unknown filesystems are not decoded. The original is retained on failure, with an explicit error and retry action. Extraction limits include eight nested layers, 200,000 files, 128 GiB per logical file, and the configured RAM allocation budget. Sparse zero regions avoid allocating empty SD-card space in tmpfs. Archive roots and source links are bounded to configured import roots; URL redirects resolve only public addresses. Network credentials are not accepted in URLs.

Verification uses tests/imports.rs, tests/test_import_sources.py, and tests/ui-import-workflow.mjs. The browser fixture imports a synthetic ZIP with an ELF, image and video, checks live progress/reload persistence, and opens its assets/code. Internet Archive metadata for Pokémon 0.81 was inspected, but no real Pokémon ROM was downloaded or imported during this work. Full extraction of that release remains to be tested through the new UI.

Activation and final checks: the live backend now runs the import endpoints and configured import-extract adapter. Theia serves the rebuilt frontend on refresh. The broader filesystem-root change was rejected by automatic approval review; the approved activation preserved /srv/firmware/images/stern_game_code and /home/jordan/verstack/samples, all existing service/network/RAM settings, and processing_enabled=false, restarting only the idle backend. There were still 13 historical jobs, zero queued/running jobs, and zero import tasks after the live smoke tests. RAM scratch is 64 GiB with MemorySwapMax=0.

All 38 Rust tests and 47 Python plugin tests passed, with targeted rechecks after final changes. The disposable browser import completed extraction, previews and Ghidra, survived a browser reload, and opened the resulting assets/code. Live read-only native-workbench and video-first-frame regressions also passed. The live import smoke check browsed the existing server root and the real Internet Archive file list without submitting an import. Activity polling returned in 17 ms in the live check; it reads release summaries, not full file manifests.

Bitmap display, bounded parallelism, and program discovery

• Native Theia View menus control layout and DMD tint. The orange filter applies only to assets marked Approximate bitmap, in thumbnails and image editors; originals and ordinary images are unchanged. The preferences persist in the browser.
• Asset, symbol, comparison, server-file, and signature pagination accept a page number (Enter or Go).
• DMD index decoding retains keyframe order, uses bulk byte operations, and compresses PNGs concurrently. Thumbnails run concurrently with ordered, budget-checked publication. Workers respect CPU affinity, cgroup CPU/memory limits, available memory, and the RAM workspace budget. On the current 54-CPU / 64-GiB-workspace setup the upper bounds are 52 DMD/SPK verification workers, 32 thumbnail workers, and 8 FFmpeg codec threads. SPK file writes, dependent DMD frames, and the overall import pipeline remain ordered.
• Synthetic RAM benchmark: 1,600 128×32 DMD frames, 14.536s before vs 1.783s automatic; 384 480×240 grayscale thumbnails, 10.705s before vs 1.114s automatic. All decoded indices, PNGs, WebPs, and evidence records matched the baseline byte-for-byte. These are stage timings, not a promised whole-import speedup.
• Imports with code analysis select every supported executable named game (including .exe, .elf, and .bin), largest first. With no game program, the largest supported executable is selected. The code browser discovers all supported program candidates, shows analysis availability and sizes, and offers explicit single-program or game-program analysis. Unrelated automatic jobs remain paused.
• Completed Ghidra exports contribute exact body hashes and bounded, operand-masked FLIRT patterns to a persistent function library. Reuse counts use distinct executable SHA-256 identities, so repeated imports and re-analysis do not increase counts. Older exports supply their already-saved exact hashes without analysis jobs; collecting FLIRT patterns for them requires an explicit re-analysis.
• Function signatures supports importing .pat and .sig, retaining original library bytes, and exporting generated .pat or FLIRT v9 .sig files (SIG processor headers currently cover x86 and ARM/AArch64). The pinned fast-flirt 0.2.2 parser checks imports and generated exports. Pattern matches are candidates; referenced names and behavior are not assumed equivalent and symbols are not automatically renamed. Saved match windows are bounded to 512 bytes; patterns needing more bytes cannot match those observations.

Catalog storage and removal

Catalog & storage is a native Theia view available from Explorer and the command menu. It lists every release and a searchable, paged inventory of original files and saved processing results. Images and other files open in the existing native asset editors.

Measure storage reads stored file/chunk metadata without extracting payloads. It separates exact-file deduplication, additional reuse of byte ranges, and compression. Release rows show packed data, bytes shared with other releases, and exclusive bytes. Shared row totals are not additive. Visual similarity and function recognition do not merge storage, so those matches are not counted as disk savings.

Delete version removes every retained extraction and analysis belonging to that edition/version/generation. Delete all versions removes the selected game across editions. The confirmation previews file counts, dependent comparison reports, protected shared data, and data eligible for collection. A review token rejects a deletion if the catalog changes before confirmation. Queued/running work prevents maintenance. Durable deletion journals are replayed before manifest reconciliation, including after an index rebuild; save them with archive backups.

Deletion removes catalog references, function observations, and related activity records, while retaining chunks until explicit Garbage collect. The collector preserves every remaining snapshot reference and refuses collection while files are being streamed. It removes orphan backend snapshots and prunes unreferenced chunks, reporting actual packed bytes reclaimed. Original source files and separately imported FLIRT libraries remain available.

Validation: the Rust suite has 45 passing tests, including shared data across games/versions, deleting the last release, dependent comparisons, stale reviews, active work/readers, and interrupted-deletion recovery. An additional history test covers imports older than the 200-entry activity page. All 51 Python tests pass. tests/ui-catalog-management.mjs creates its own disposable archive and redirects all browser API traffic to it for storage measurement, file paging/opening, cancellation and confirmation, version/game deletion, garbage collection, and byte-for-byte retained-file verification. It never mutates the live archive.

The disposable catalog browser workflow passed, including switching between file inventory and releases during loading. The live page was then checked with GET requests only: four releases, 168,519 cataloged files, no browser errors, automatic processing still paused, and zero queued/running tasks. Initial storage measurement took 11.42 seconds and is cached until archive contents change. No live versions were deleted and no live garbage collection was performed. The rebuilt backend and Theia bundle are active; refresh existing browser tabs.

Program/signature validation used an isolated import with 26 executable candidates and two compiled game programs. Actual Ghidra analysis selected those two, exposed all candidates in the native picker, and populated exact and FLIRT observations. Repeated analysis retained two distinct executable identities. Generated PAT/SIG files round-tripped through the independent FLIRT parser and matched imported libraries; the installed IDA application itself was not used for verification. The live DMD/page-entry browser check passed with zero API writes and confirmed ordinary Pokémon images keep their original colors.