verstack/docs/job-queue.md

Import and analysis scheduling

The service runs one import/extraction worker and two Ghidra analysis workers by default. Set analysis_workers in the service configuration to an integer from 1 through 8 to change the analysis limit. An omitted setting means 2. preparation_workers controls concurrent imports, extraction, and media decoding (1 through 8; defaults to 1 for compatibility with small workspaces). This host is configured for 3 preparation workers. Settings take effect on service restart.

An import retains its original, extracts files, and prepares requested assets on the import worker. It then saves an analysis_input snapshot reference and enters Waiting for code analysis. This durable handoff frees the import worker before Ghidra starts. Analysis workers claim these tasks independently. A queued handoff survives restart; retrying an interrupted task reuses its saved original and stage outputs. Active Ghidra computation itself must restart after interruption.

Legacy jobs use the same execution slots. Each worker alternates between imports and legacy jobs when both have work, with job priority applied within its lane. processing_enabled: false pauses legacy jobs and automatic submission; explicitly requested imports and their analyses still run. Jobs with the same input and tool do not execute concurrently. Direct API/CLI processing also reserves a slot.

External analysis and file preparation run outside the archive writer lock. Archive commit, readback verification, manifest publication, and indexing remain serialized. Deletion and garbage collection reserve all execution slots and reject active or pending work. Progress and cancellation are attributed to each import's worker thread, and shutdown interrupts every active plugin process group.

Resource allowances

Scratch capacity is partitioned, rather than giving each concurrent operation the entire configured workspace_bytes allowance:

• Each analysis gets min(6 GiB, workspace_bytes / (analysis_workers + 2)).
• Each import/extraction slot gets the remainder after reserving every analysis slot, divided by preparation_workers. Increasing this count reduces the maximum scratch allowance for an individual large image.
• Input materialization uses at most half a slot's allowance. Original import staging also uses half, leaving room for downloaded or uploaded source bytes.
• Plugin execution and output capture are checked against that slot's allowance.

For a 64 GiB workspace and two analysis workers, this gives 6 GiB per analysis and 52 GiB for import/extraction. These are scratch allowances, not JVM heap limits or OS memory isolation. Ghidra's configured max_heap and max_cpu apply separately to each analysis process. Trusted plugins are monitored for excess scratch use; the monitor is not a hard filesystem quota. Reserve host RAM for JVMs and other processes in addition to scratch, and limit individual extraction tools' threads when CPU contention warrants it.

GET /api/info exposes analysis_workers, preparation_workers, analysis_workspace_bytes, and import_workspace_bytes. Existing activity endpoints show running imports and the Waiting for code analysis handoff without a frontend upgrade.

The synchronous work_import_one() helper remains available for callers that want to prepare an import and attempt its analysis in one call. The service uses separate preparation and analysis lanes so a long analysis cannot hold up imports.

Verification

cargo test --tests --offline covers ordinary archive behavior. The queue_concurrency integration tests additionally exercise overlapping analyses, imports during analysis, admission limits, isolated cancellation, shutdown and retry, durable handoff across restart, legacy job scheduling, and per-analysis scratch limits. Test plugins use explicit release signals to prove overlap.

Recovery

POST /api/imports/{id}/retry refreshes unfinished stages with the current configured tools and reuses successful stage outputs. For an extraction that previously treated a container as opaque, POST /api/imports/{id}/reextract also accepts completed imports and reruns the pipeline from the retained original. It resets the task's output references and warnings; published snapshots remain archived. Active imports cannot be reset.

The extractor recognizes ImageUSB .bin backups by their UTF-16 header and validates the payload length before passing the disk after the 512-byte wrapper to the filesystem extractor. Partitioned backups are read directly at that offset, avoiding a second whole-image copy during decryption. Sparse padding remains sparse in scratch storage.

Ghidra adapter revision 3 exports functions using up to max_cpu independent decompilers, with bounded pending work and deterministic output ordering. An auto-analysis timeout aborts before decompilation and database export. Successful outputs still require reopening the database and verifying the exported symbols. Adapter version 4 additionally supports an optional checkpoint_dir for recovery of completed program exports. Checkpoints verify the input hash, analysis settings, producer script hashes, and every retained file hash. Reuse always reopens the saved GZF and checks its symbols again; the source checkpoint receipt is archived alongside the new output. Corrupt or incompatible checkpoints fall back to fresh analysis. Checkpoint directories are operator-managed scratch storage, separate from the archive, and may be removed when no jobs use them. Their location does not change function-comparison compatibility.

Repeated imports can reuse a completed analysis of the same input snapshot when the tool command, version, source digest, revision, schema, and settings match and the saved inventory covers every selected program. Selecting a subset does not repeat a verified superset analysis. Missing programs or changed profiles run the tool again. Function inventories up to 512 MiB can be indexed, synchronized, and compared; media metadata retains its separate 64 MiB limit. Plugin result manifests allow up to 32 MiB so inventories with over 100,000 function-code paths can be published. Output paths are still checked against the complete on-disk inventory before capture.

Large code-export throughput

Verification now retains a directory index even above the old 65,536-node / 32 MiB soft cache limit. Previously, a ~125,000-function export bypassed that cache and reparsed the same large tree for each file. An oversized directory replaces the prior cache and remains resident until the next directory; memory is bounded by the soft budget or one largest decoded directory.

Each verification owns a fresh backend read view and directory cache, so normal browsing and other verifications cannot evict its index. Bytes are still reread and BLAKE3-verified. The archive writer covers the backend commit only; completed output verification and catalog indexing can proceed outside that writer lock. Maintenance remains excluded while executions are active.

Progress distinguishes waiting for the archive writer, saving, verifying (with file counts), and indexing. Progress updates carry a timestamp that changes only when the reported progress changes. Activity reads use a separate read-only WAL connection, so a large symbol-index transaction cannot freeze the Jobs panel. The Jobs view defaults to active/queued tasks, puts running tasks first, and provides status, release, search and compact-view controls. History is retained, not deleted.

On the 54-CPU / 172-GiB host, the current configuration uses four analysis slots, 8 Ghidra CPUs and an 8-GiB JVM heap per analysis. The 80-GiB scratch allocation reserves 6 GiB per analysis and divides the remaining 56 GiB across three preparation slots (about 18.7 GiB each), fitting the host’s 87-GiB RAM filesystem with headroom for other temporary files. Python plugins receive a per-job CPU budget of 7 threads on this host, rather than each automatically using nearly every CPU. These are cooperative limits, not per-job OS quotas. Deep analysis of distinct large games still takes time; adding more queue entries does not increase the number of simultaneous Ghidra processes.

Regression checks include a real 65,537-file backend directory, archive corruption detection, queue concurrency/cancellation, and the Jobs filter UI.

Signature synchronization uses existing analysis

Signature sync indexes saved function exports and compares those observations. It does not enumerate all SD executables or schedule fresh Ghidra auto-analysis. Previously analyzed libraries remain eligible for signature comparison. New analysis is requested explicitly through imports or Choose programs.

Legacy include_missing and include_system API flags are accepted for client compatibility but cannot enable automatic analysis. On resuming an old active sync plan, the worker cancels its queued owned jobs, preserves running and independently requested analyses, and rebuilds coverage from saved results.

The activity response reports actual analysis/preparation slot occupancy in execution; a planned or queued task does not occupy an execution slot. Active imports remain visible even when more than 200 completed imports exist.

TODO: resource governor

Replace static slot allocation with admission based on observed CPU use, process memory, tmpfs scratch headroom, archive free space, and I/O pressure.

• Lend idle analysis capacity to extraction/decoding without multiplying nested plugin thread pools beyond a shared host budget. Account for Ghidra's explicit max_cpu and heap settings, which are independent of Python worker limits.
• Estimate scratch demand from SD/container sizes; let large images reserve more capacity and run fewer at once instead of failing a smaller per-slot allowance.
• Pause new admissions before archive storage fills. Include pending publications and VM exports when estimating disk demand; tmpfs capacity alone is insufficient.
• Use bounded adjustments and hysteresis to avoid oscillation. Keep manual caps, prioritize fairly across releases, and expose the limiting resource in Jobs.
• Validate mixed 10–15-ROM queues, cancellation, maintenance exclusion, and recovery under memory/disk pressure before enabling automatic scaling.

The current change is static configuration only; it does not implement these adaptive controls. Archive publication remains serialized.