verstack/docs/architecture.md

Architecture and implementation status

Decisions carried forward

• One authoritative Linux instance with shared physical storage across logical game repositories; no clustering or mandatory cloud services.
• Immutable original snapshots and separate extraction/analysis revisions; out-of-order releases work without predecessor reconstruction dependencies.
• Storage reduction first, native/function research second, asset-history browsing third.
• Exact executable and asset bytes are authoritative; decoded representations and research exports are separate artifacts.
• Trusted executable plugins in any suitable language. Core operations do not require HTTP; the UI consumes the same Rust library through Axum.
• Stern SPIKE packages are the pilot, with structure-based identification needed for all three generations. Game version, Pro/Premium/LE edition, and generation are separate metadata.
• Ghidra is a local optional tool installation, with retained per-program GZF exports. AI assistance is optional and not present in the initial implementation.
• Retain releases indefinitely until an explicit deletion facility is implemented. No source deletion or GC in this development slice.
• Local disk persistence, configurable filesystem workspaces, 100 GiB starting budget, no authentication requirement for initial trusted-LAN use.

Module boundaries

model defines versioned records. archive owns import, publication, verification, restoration, and file comparison. storage::ArtifactStorage isolates backend details. plugins manages external steps and validates their output. analysis interprets retained function facts. http and the CLI are adapters. web/ is a locally served static client.

The current logical repository is the Release.repository field; standalone repository/collection records, annotations, and explicit release relationships need a later schema. JSON manifests and runs are authoritative durable records. A future SQLite search index must be rebuildable from those records.

Publication order: captured input → backend snapshot → read-back file verification → immutable domain manifest → terminal run update. Reopening reconciles a crash between the last two steps. A failure before domain publication leaves an incomplete run and may leave backend content for later reclamation. Source mutation checks catch ordinary concurrent file changes; imports are not atomic filesystem snapshots of a live-changing directory tree.

The backend remains restic-compatible. Framework BLAKE3 IDs do not replace Rustic's internal content IDs or chunker. Function matches are evidence relationships and never storage equality. The current exact-body function comparison is intentionally a baseline for later Version Tracking/BSim adapters, not a replacement for them.

What the storage feasibility spike establishes

The runnable synthetic spike exercises independent snapshot reads, cross-game exact reuse, content-defined reuse after an insertion, whole-file identities, compressed backing storage, and verified restoration. Integration tests also cover damaged packs, bounded import failures, process ownership, out-of-order versions, and plugin revisions.

It does not establish performance at 2.4 TB, startup/index RSS, media-heavy compressibility, concurrency capacity, recovery under actual power loss, network filesystem suitability, or safe retention. Keep Rustic behind the interface while testing those requirements. A custom store has not been justified by current evidence.

Remaining acceptance milestones

1. Real package support: the local corpus, SPKS extraction, and structural wrapper probe have been exercised. LUKS2/ext4 update extraction has passed the Pokémon LE pilot. Implement generation evidence and remaining proprietary asset decoders. Keep README evidence distinct from inferred metadata and component versions.
2. Real Ghidra validation: x86/ARM analysis and GZF reopening pass. Exact comparison reports now retain both source inventories and their release metadata. Check known structural function changes. Add selected strings/imports and decompiled text, then Version Tracking and global BSim feasibility measurements.
3. Workflow engine: versioned DAG definitions, resolved dependency identities, idempotent cache keys, durable queued jobs, retry policy, resource reservations, progress events, and a UI workflow editor. Current plugins are explicit single steps; HTTP requests wait for their completion while run status is separately readable.
4. Scale and browsing: resumable/bundled uploads, server-path browser, paginated catalog queries, reusable SQLite indexes, richer byte/script/config comparisons, timeline-neighbor selection, preview cache, and accurate instance-wide physical/workspace statistics. Current listing loads JSON records directly.
5. Retention and recovery: coverage-based source cleanup, pins, cross-project reachability, safe GC, backup/restore drills, format migrations, annotations, and project-only exports. No pruning/reset endpoint exists yet.

Before source cleanup can be enabled, prove that the retained outputs cover the selected preservation policy. A plugin's success status and byte verification alone do not establish container coverage.