diff --git a/docs/vm-export-acceptance.md b/docs/vm-export-acceptance.md index ea8ed20..e6b16cf 100644 --- a/docs/vm-export-acceptance.md +++ b/docs/vm-export-acceptance.md @@ -10,14 +10,14 @@ export is not a claim that every game version is fully emulated. | Package the selected game, SD runtime, kernel, initramfs and Mac supporting scripts | Published 0.83 `723fe5e4d9024f2ab195d28d50271d12` and 0.85 `e4359dddb21946689d3ded1dbab63648` include these artifacts. ZIP and image checks are recorded in `docs/emulation.md`. Standard launch requires host Python and QEMU; supported local IC profiles are prepared during export and need no Docker on launch. | | Use the release's own SD before any donor | The current 0.83 ZIP manifest selects its own runtime ID and `own-sd`. `test_bundle_selection.py` also rejects substitution when an SD release's own system is missing. | | Otherwise choose the closest same-title SD | The 0.85 standard export and actual quick build select the 0.83 SD. Numeric ordering, tie-breaking and title priority have policy tests. | -| Otherwise choose a same-generation SD | The policy excludes the wrong architecture and borrowed update runtimes. The live catalog selects King Kong's SD for other SPIKE 2 titles. This is selection evidence; new SPIKE 2 ZIP exports are disabled. | +| Otherwise choose a same-generation SD | The policy excludes the wrong architecture and borrowed update runtimes. The live catalog selects King Kong's SD for other SPIKE 2 titles. SPIKE 2 standard and quick exports are now enabled; King Kong was built with its own SD. | | Quick export connects to the catalog and caches components | Actual downloaded client fetched game/runtime components and reused verified SHA-256 caches. The current downloaded client matched the source byte-for-byte; receipt `/tmp/verstack-quick-cleanup-validation.json`. | | Quick export updates and switches versions | Actual 0.83 and 0.85 builds, game/libc comparisons, filesystem checks and switching without rebuild are recorded in `/tmp/verstack-universal-build-result.json` and `/tmp/verstack-universal-switch-result.json`. Those runs predate local IC provisioning; they prove build/switch behavior, not current native boot of both versions. Recipe changes invalidate images; incomplete builds cannot become active. | | Explicit shared runtime per generation | Actual quick 0.83 build used the 0.81 SD override. Browser test selects that override, and policy tests reject a cross-generation override. `--default-runtime` returns to automatic selection. | | Retain the supplied SPIKE 3 nodebus and local switch panel | Both vendored netbridge implementations and keyboard relays match the hashes in `SOURCE.json`. The selected game's node firmware version is advertised while retaining the board model. The current ZIP's 28 JavaScript input tests pass. Native menu navigation remains pending. | -| Preserve experimental SPIKE 2 work | Existing emulator code and proxy remain; `verstack-emulator` was confirmed running. King Kong's imported SD remains available. This phase has not established a downloadable ARM32 VM or repaired its earlier game-startup limitations. | -| Do not interrupt active catalog jobs | Export worker is separate and serialized with CPU/RAM limits. Only isolated validation VMs were stopped. Backend activation watcher remains live and waits for active sound decoding to finish before restarting the backend. Three preparation workers are not yet confirmed active. | -| Clean up build and validation scratch | Standard success/failure cleanup is tested. Quick failure now removes incomplete images and source extraction while preserving the previous active VM and component cache. All 20 export tests pass. The recent headless validation disk was removed after saving small logs; published ZIPs remain. | +| Preserve experimental SPIKE 2 work | Existing emulator code and proxy remain; `verstack-emulator` was confirmed running. King Kong's imported SD remains available. The downloadable ARM64 appliance now contains the ARM32 worker and selected SD. Boot, diagnostic screen/audio and service controls passed; game-startup limitations remain. | +| Do not interrupt active catalog jobs | Export worker is separate and serialized with CPU/RAM limits. An erroneous ARM64 interpreter registration later blocked host execution and required a reboot; this violated the intended uninterrupted operation. After recovery, Star Wars was completed, Deadpool was running, and the backend reported three preparation workers. See the incident note below. | +| Clean up build and validation scratch | Standard success/failure cleanup is tested. Quick failure now removes incomplete images and source extraction while preserving the previous active VM and component cache. All 21 export tests pass. The recent headless validation disk was removed after saving small logs; published ZIPs remain. | ## Native confirmation and remaining limits @@ -41,7 +41,47 @@ event-routing tests using the existing Pillow environment. Together with its not the game's response on macOS. Logs remain in `/tmp/verstack-packaged-controls-audit`; the extracted test source was removed. -The existing three-worker backend activation request remains pending at a safe -stopping point. Its live watcher is `verstack-preparation-activation.service`; -the required final evidence is `/api/info` reporting `preparation_workers: 3` -and `/api/activity` reporting `execution.preparation_limit: 3`. +After the recovery reboot, `/api/info` reported `preparation_workers: 3` and +`analysis_workers: 4`; `/api/activity` reported `execution.preparation_limit: 3`. +The old activation watcher is inactive and is no longer needed. + + +## ARM64 build-host registration incident + +A manually constructed binfmt registration wrote raw NUL bytes instead of escaped +text. The kernel parser truncated its ELF signature and intercepted native +executables. The [kernel format documentation](https://docs.kernel.org/admin-guide/binfmt-misc.html) +requires NUL bytes in registration text to be escaped. New host commands failed with `Exec format error`. The registration +was absent after the VM reboot; catalog and emulator services recovered, while +the export service needed to be started again. The first SPIKE 2 VM had not yet +been built. This was an implementation error, not an ARM32 game failure. + +`scripts/enable-arm64-builder.py` now reads Debian's installed definition, checks +the complete ELF machine field, rejects native/x86/ARM32 matches, and preserves +escaped registration text. Installation performs readback and a native execution +check in the same live process, with rollback on failure. Its default invocation +only validates; `--install` changes the mounted host binfmt filesystem. Two tests +cover textual encoding, exclusion of the host, and truncated signatures. The user +authorized leaving the corrected registration enabled. It is not configured to +reinstall itself at boot. + +## SPIKE 2 download validation + +Corrected King Kong standard export `79f02d81e9fc48dc83ab4f00b28f6342` is +4,629,361,401 bytes, SHA-256 +`87c1b420237d8ef9d351d9f811b2325f84a199ce072f6231c94c626de503f15a`. +The catalog API reuses this completed export and serves ZIP range requests. +The launcher disables the optional virtio network ROM, so it does not depend on +a host firmware file. Both standard and actual quick builds preserve the selected +King Kong executable and SD libc byte-for-byte. Filesystem checks passed. The +downloaded quick client's second Update.command reused its completed image +without changing the disk. + +An isolated QEMU boot of the exported disk exposed the selected ARM32 runtime. +`tests/ui-spike2-vm.mjs` passed diagnostic screen rendering, nonzero audio and all +four service buttons. A separate 302-second actual-game observation produced no +frames and remained in runtime preparation on the Intel test host. This verifies +downloadability and appliance operation, not playable King Kong or native Mac +speed. The validation VM was stopped; temporary image/component copies were +removed after retaining small receipts in `/tmp/verstack-spike2-validation`. +Published downloads and intentional component/build caches remain. diff --git a/emulator/bundles/README.md b/emulator/bundles/README.md index c24cda0..ec0275b 100644 --- a/emulator/bundles/README.md +++ b/emulator/bundles/README.md @@ -3,10 +3,15 @@ The catalog's Download VM tab exports Apple Silicon macOS bundles based on `~/pokemon_emulator`. `vendor/pokemon_emulator/SOURCE.json` records the upstream commit and original source hashes. Local adaptations live alongside that source. -SPIKE 2's existing experimental emulator is preserved; this export builder -currently accepts SPIKE 3 ARM64 games. +SPIKE 2 exports package the existing experimental ARM32 worker inside an ARM64 +Linux appliance. Its local browser provides the screen, audio and switch panel; +the game still uses the selected SPIKE 2 SD libraries. Standard and quick exports +are available. Launch.command opens localhost:8095; click Start session and Enable +audio. The appliance passed boot, diagnostic display/audio and service-button +checks. King Kong gameplay remains experimental: the Intel validation run did not +reach a game frame within five minutes. Native Mac gameplay is untested. -Standard export packages the selected game and SD system into an ext4 VM with +SPIKE 3 standard export packages the selected game and SD system into an ext4 VM with its kernel, initramfs, Mac launcher, video bridge, and local switch dashboard. Install Python 3.11+ and QEMU, unzip, and run `Launch.command`. The launcher installs Pillow in a private environment if necessary. Controls are served on @@ -26,7 +31,9 @@ with Docker; `Launch-universal.command` runs the selected image. `--list`, explicit same-generation runtime reuse. Components are cached by SHA-256. The persistent export service is `verstack-exports.service` on localhost:8096, -proxied through the workbench. Builds have a separate serialized worker and +proxied through the workbench. Install it with +`systemctl --user enable --now /home/jordan/verstack/emulator/bundles/verstack-exports.service`. +Builds have a separate serialized worker and 2-CPU/4-GiB Docker limits. Standard builds use temporary export scratch (`VERSTACK_EXPORT_SCRATCH`, default `/tmp/verstack-vm-exports`), check free space with reserves, and remove staging after success or failure. Only the final diff --git a/emulator/bundles/builder.py b/emulator/bundles/builder.py index 5e4e99b..bed358a 100644 --- a/emulator/bundles/builder.py +++ b/emulator/bundles/builder.py @@ -223,8 +223,11 @@ def build_image(destination, *, docker=('docker',), log=None): """Use the original image builder in a separate, bounded Linux container.""" destination=Path(destination).resolve() metadata=json.loads((destination/'bundle.json').read_text()) + if metadata['architecture'] == 'armhf': + from .spike2 import build_image as build_spike2 + return build_spike2(destination,docker=docker,log=log) if metadata['architecture'] != 'aarch64': - raise ValueError('ARM32 VM image support is still experimental and not wired into this builder') + raise ValueError('Unsupported game architecture') repair=Path(__file__).with_name('repair-rootfs.sh').resolve() command=[*docker,'run','--rm','--cpus','2','--memory','4g', '--mount',f'type=bind,source={destination},target=/workspace', diff --git a/emulator/bundles/jobs.py b/emulator/bundles/jobs.py index 0f9174e..f4f6510 100644 --- a/emulator/bundles/jobs.py +++ b/emulator/bundles/jobs.py @@ -71,6 +71,8 @@ def quick_bundle(destination, request): destination.mkdir(parents=True,exist_ok=False) shutil.copytree(Path(__file__).parent,destination/'export-tools/bundles', ignore=shutil.ignore_patterns('__pycache__','*.pyc','web')) + from .spike2 import copy_worker + copy_worker(destination/'export-tools/bundles/spike2/worker') atomic_json(destination/'connection.json',dict(catalog_url=request['catalog_url'], game_id=request['game_id'],shared_runtime_id=request.get('shared_runtime_id'))) for name in ('Update.command','Launch-universal.command'): @@ -93,6 +95,13 @@ def source_revision(mode='quick'): paths=(list((root/'vendor').rglob('*')) + [root/name for name in ('builder.py','runtime_selection.py','repair-rootfs.sh','Launch.command')] if mode=='standard' else list(root.rglob('*'))) + from .spike2 import worker_source, WORKER_FILES + if mode=='standard': + paths += [root/'spike2.py', *list((root/'spike2').rglob('*'))] + worker=worker_source() + for path in sorted([worker/name for name in WORKER_FILES]+list((worker/'vendor').rglob('*'))): + if path.is_file() and '__pycache__' not in path.parts: + revision.update(str(path.relative_to(worker)).encode()+path.read_bytes()) revision.update(('artifact-v2:'+mode).encode()) for path in sorted(paths): if path.is_file() and '__pycache__' not in path.parts: @@ -103,8 +112,6 @@ def source_revision(mode='quick'): def create_job(store, runtime_root, request): game, selection=builder.plan(runtime_root,request['game_id'],request.get('shared_runtime_id')) - if game['architecture']!='aarch64': - raise ValueError('SPIKE 2 VM export is not ready; its existing experimental emulator is retained') if request.get('mode') not in ('standard','quick','components'): raise ValueError('Choose standard or quick export') revision=source_revision(request['mode']) diff --git a/emulator/bundles/server.py b/emulator/bundles/server.py index 9f0862c..64b458c 100644 --- a/emulator/bundles/server.py +++ b/emulator/bundles/server.py @@ -38,8 +38,8 @@ class ExportServer(ThreadingHTTPServer): for row in rows: try: game,selection=builder.plan(self.runtime_root,row['id']) - note='' if game['architecture']=='aarch64' else 'SPIKE 2 export is preliminary; the existing test emulator is retained.' - result.append(dict(id=row['id'],label=row['label'],exportable=row['architecture']=='aarch64', + note='' if game['architecture']=='aarch64' else 'Experimental SPIKE 2 VM: local browser screen and controls; game compatibility remains preliminary.' + result.append(dict(id=row['id'],label=row['label'],exportable=row['architecture'] in ('aarch64','armhf'), architecture=row['architecture'],runtime_label=selection['runtime']['label'],note=note, runtimes=[dict(id=r['id'],label=r['label']) for r in rows if r['is_sd'] and r['architecture']==row['architecture']])) except ValueError as error: diff --git a/emulator/bundles/spike2.py b/emulator/bundles/spike2.py new file mode 100644 index 0000000..a1efd2d --- /dev/null +++ b/emulator/bundles/spike2.py @@ -0,0 +1,99 @@ +"""Package the existing ARM32 worker and selected SD inside an ARM64 appliance.""" +import hashlib +import json +from pathlib import Path +import shutil +import subprocess +import tempfile + +ROOT=Path(__file__).parent +WORKER_FILES=('server.py','runner.py','runtimes.py','client.js','index.html','table.json','table.svg') + + +def worker_source(): + packaged=ROOT/'spike2/worker' + return packaged if packaged.is_dir() else ROOT.parent + + +def copy_worker(destination): + source=worker_source() + destination.mkdir(parents=True) + for name in WORKER_FILES:shutil.copy2(source/name,destination/name) + shutil.copytree(source/'vendor',destination/'vendor',ignore=shutil.ignore_patterns('__pycache__','*.pyc')) + # A downloadable game opens in game mode; diagnostics remain selectable. + page=destination/'index.html' + page.write_text(page.read_text().replace('value="game"','value="game" selected')) + + +def build_image(destination, *, docker, log=None): + destination=Path(destination).resolve() + metadata=json.loads((destination/'bundle.json').read_text()) + if metadata['architecture']!='armhf':raise ValueError('SPIKE 2 requires an ARM32 game') + game=destination/'source-root/games'/metadata['game_directory']/'game' + metadata['game_sha256']=hashlib.sha256(game.read_bytes()).hexdigest() + with tempfile.TemporaryDirectory(prefix='spike2-build-',dir=destination) as temporary: + context=Path(temporary) + copy_worker(context/'worker') + shutil.copy2(ROOT/'spike2/Dockerfile',context/'Dockerfile') + # Content tag keeps old exported clients from silently using a new worker. + digest=hashlib.sha256() + for path in sorted(context.rglob('*')): + if path.is_file():digest.update(str(path.relative_to(context)).encode()+path.read_bytes()) + tag='verstack-spike2-vm:'+digest.hexdigest()[:16] + exists=subprocess.run([*docker,'image','inspect',tag],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL) + if exists.returncode: + subprocess.run([*docker,'build','--cpu-quota','200000','--memory','4g','--platform','linux/arm64','-t',tag,str(context)],check=True,stdout=log,stderr=subprocess.STDOUT) + architecture=subprocess.check_output([*docker,'image','inspect','--format','{{.Architecture}}',tag],text=True).strip() + if architecture!='arm64':raise ValueError('SPIKE 2 appliance base must be ARM64') + container=subprocess.check_output([*docker,'create','--platform','linux/arm64',tag],text=True).strip() + try:subprocess.run([*docker,'export','-o',str(context/'base.tar'),container],check=True,stdout=log,stderr=subprocess.STDOUT) + finally:subprocess.run([*docker,'rm',container],check=True,stdout=log,stderr=subprocess.STDOUT) + manifest=dict(schema=2,architecture='armhf',system_root='system', + game_root='system/games/'+metadata['game_directory'],targets={'game':'game'}, + release=metadata['release'],snapshot=metadata['game_id'],system_snapshot=metadata['runtime_id']) + (context/'runtime.json').write_text(json.dumps(manifest)) + shutil.copy2(ROOT/'spike2/emu-init',context/'emu-init') + script='''set -eu +root=/workspace/appliance-root +cleanup() { + rm -rf "$root" + if [ -d /workspace/emulation/work ]; then + chown -R --reference=/workspace /workspace/emulation/work + fi +} +trap cleanup EXIT +mkdir -p "$root" +tar -xf /build/base.tar -C "$root" +mkdir -p "$root/runtime/selected" "$root/usr/local/spike-emu/bin" "$root/lib" +mv /workspace/source-root "$root/runtime/selected/system" +cp /build/runtime.json "$root/runtime/selected/runtime.json" +cp /build/emu-init "$root/usr/local/spike-emu/bin/emu-init" +chmod 0755 "$root/usr/local/spike-emu/bin/emu-init" +cp -a /opt/spike3-kernel/modules/lib/modules "$root/lib/" +for modules in "$root"/lib/modules/*; do depmod -b "$root" "$(basename "$modules")"; done +mkdir -p /workspace/emulation/work/macos-runtime +cp /opt/spike3-kernel/vmlinuz /opt/spike3-kernel/initrd.img /workspace/emulation/work/macos-runtime/ +size=$(du -sm "$root" | cut -f1) +truncate -s "$((size + 3072))M" /workspace/emulation/work/rootfs.ext4 +mkfs.ext4 -q -F -d "$root" /workspace/emulation/work/rootfs.ext4 +e2fsck -fn /workspace/emulation/work/rootfs.ext4 +chmod 0644 /workspace/emulation/work/rootfs.ext4 /workspace/emulation/work/macos-runtime/* +rm -rf "$root" +''' + subprocess.run([*docker,'run','--rm','--cpus','2','--memory','4g', + '--mount',f'type=bind,source={destination},target=/workspace', + '--mount',f'type=bind,source={context},target=/build', + '--entrypoint','bash','verstack-vm-builder:local','-ec',script],check=True,stdout=log,stderr=subprocess.STDOUT) + for name in ('Launch.command','launch-spike2.py'):shutil.copy2(ROOT/'spike2'/name,destination/name) + (destination/'Launch.command').chmod(0o755) + metadata.update(guest_architecture='aarch64',execution='experimental-spike2-worker', + display='local-browser',local_ic_supported=False) + (destination/'bundle.json').write_text(json.dumps(metadata,indent=2)+'\n') + (destination/'README.txt').write_text( + 'Experimental SPIKE 2 VM\n\nInstall Python 3 and QEMU (macOS: brew install python qemu).\n' + 'Run Launch.command. The local browser panel provides the game screen, audio and switches.\n' + 'Click Start session and Enable audio. No catalog connection or Docker is needed to launch.\n' + 'The ARM32 game uses the preserved experimental worker inside an ARM64 Linux VM.\n' + 'Game startup and hardware compatibility remain experimental; this does not repair the known King Kong startup issues.\n' + 'Selected game and SD libraries are retained together inside the disk image.\n' + 'Sessions use temporary state, as in the existing worker. Stop the VM with Ctrl+C.\n') diff --git a/emulator/bundles/spike2/Dockerfile b/emulator/bundles/spike2/Dockerfile new file mode 100644 index 0000000..3c9c1a5 --- /dev/null +++ b/emulator/bundles/spike2/Dockerfile @@ -0,0 +1,26 @@ +FROM arm64v8/debian:bookworm-slim +RUN test "$(dpkg --print-architecture)" = arm64 +ENV DEBIAN_FRONTEND=noninteractive PYTHONUNBUFFERED=1 +RUN dpkg --add-architecture arm64 && dpkg --add-architecture armhf && apt-get update && apt-get install -y --no-install-recommends \ + python3 python3-aiohttp python3-pil b3sum bubblewrap busybox-static \ + ca-certificates ffmpeg binutils-aarch64-linux-gnu gcc-aarch64-linux-gnu \ + qemu-user-static libegl-mesa0:arm64 libegl1:arm64 libgbm1:arm64 \ + libgl1-mesa-dri:arm64 libglapi-mesa:arm64 libgles2:arm64 libglvnd0:arm64 \ + && rm -rf /var/lib/apt/lists/* +RUN apt-get update && apt-get install -y --no-install-recommends libc6-dev-arm64-cross \ + && rm -rf /var/lib/apt/lists/* +RUN apt-get update && apt-get install -y --no-install-recommends \ + libavcodec-dev:arm64 libavutil-dev:arm64 libswscale-dev:arm64 \ + libgbm-dev:arm64 libyuv-dev:arm64 \ + && rm -rf /var/lib/apt/lists/* +RUN apt-get update && apt-get install -y --no-install-recommends \ + binutils-arm-linux-gnueabihf gcc-arm-linux-gnueabihf libc6-dev-armhf-cross \ + libegl-mesa0:armhf libegl1:armhf libgbm1:armhf libgl1-mesa-dri:armhf \ + libglapi-mesa:armhf libgles2:armhf libglvnd0:armhf \ + libavcodec-dev:armhf libavutil-dev:armhf libswscale-dev:armhf \ + libgbm-dev:armhf libyuv-dev:armhf \ + && rm -rf /var/lib/apt/lists/* +RUN apt-get update && apt-get install -y --no-install-recommends iproute2 kmod && rm -rf /var/lib/apt/lists/* +WORKDIR /app +COPY worker/ /app +CMD ["python3", "server.py"] diff --git a/emulator/bundles/spike2/Launch.command b/emulator/bundles/spike2/Launch.command new file mode 100644 index 0000000..e1b8bde --- /dev/null +++ b/emulator/bundles/spike2/Launch.command @@ -0,0 +1,4 @@ +#!/bin/sh +set -eu +bundle_dir=$(CDPATH= cd -- "$(dirname "$0")" && pwd) +exec python3 "$bundle_dir/launch-spike2.py" diff --git a/emulator/bundles/spike2/emu-init b/emulator/bundles/spike2/emu-init new file mode 100644 index 0000000..13d8b23 --- /dev/null +++ b/emulator/bundles/spike2/emu-init @@ -0,0 +1,22 @@ +#!/bin/sh +export PATH=/usr/sbin:/usr/bin:/sbin:/bin +mount -t proc proc /proc +mount -t sysfs sysfs /sys +mount -t devtmpfs devtmpfs /dev +mkdir -p /dev/pts /dev/shm /run /sessions +mount -t devpts devpts /dev/pts +mount -t tmpfs tmpfs /run +mount -t tmpfs -o size=2g,mode=0700 tmpfs /sessions +mount -t tmpfs tmpfs /dev/shm +modprobe virtio_net +ip link set lo up +ip link set eth0 up +ip addr add 10.0.2.15/24 dev eth0 +ip route add default via 10.0.2.2 dev eth0 +export VERSTACK_EMULATOR_MAX_SESSIONS=1 +export PYTHONUNBUFFERED=1 +echo 'Experimental SPIKE 2 VM: open the local browser control panel.' +cd /app +python3 server.py +echo 'Emulator service exited. Diagnostic shell follows.' +exec /bin/sh diff --git a/emulator/bundles/spike2/launch-spike2.py b/emulator/bundles/spike2/launch-spike2.py new file mode 100644 index 0000000..5485cbf --- /dev/null +++ b/emulator/bundles/spike2/launch-spike2.py @@ -0,0 +1,61 @@ +"""Launch the preserved SPIKE 2 worker in a self-contained Linux VM.""" +from pathlib import Path +import os +import platform +import shutil +import signal +import socket +import subprocess +import time +import urllib.request +import webbrowser + + +def main(): + root=Path(__file__).resolve().parent + qemu=shutil.which('qemu-system-aarch64') + if not qemu: + raise SystemExit('Install QEMU first (on macOS: brew install qemu).') + port=int(os.environ.get('VERSTACK_VM_PORT','8095')) + # Refuse to open another running instance's controls by accident. + with socket.socket() as probe: + probe.bind(('127.0.0.1',port)) + native=platform.system()=='Darwin' and platform.machine()=='arm64' + work=root/'emulation/work';logs=work/'logs';logs.mkdir(exist_ok=True) + command=[qemu,'-machine','virt,accel='+('hvf' if native else 'tcg'), + '-cpu','host' if native else 'cortex-a72','-m','4096','-smp','4', + '-kernel',str(work/'macos-runtime/vmlinuz'), + '-initrd',str(work/'macos-runtime/initrd.img'), + '-append','console=ttyAMA0 root=/dev/vda rw net.ifnames=0', + '-snapshot','-drive','if=none,file='+str(work/'rootfs.ext4')+',format=raw,id=rootfs', + '-device','virtio-blk-pci,drive=rootfs', + '-netdev',f'user,id=net0,restrict=on,hostfwd=tcp:127.0.0.1:{port}-:8095', + '-device','virtio-net-pci,netdev=net0,romfile=','-display','none', + '-serial','file:'+str(logs/'serial.log'),'-monitor','none'] + url=f'http://127.0.0.1:{port}/emulator/' + print('Starting experimental SPIKE 2 VM. Screen, audio and controls: '+url,flush=True) + print('Keep this terminal open. Press Ctrl+C to stop the VM.',flush=True) + process=subprocess.Popen(command) + def stop(*_): + if process.poll() is None:process.terminate() + signal.signal(signal.SIGTERM,stop) + try: + for _ in range(180): + if process.poll() is not None:raise SystemExit('QEMU exited; check emulation/work/logs/serial.log.') + try: + with urllib.request.urlopen(url,timeout=1) as response: + if response.status==200:break + except OSError:time.sleep(1) + else:raise SystemExit('VM startup timed out; check emulation/work/logs/serial.log.') + webbrowser.open(url) + process.wait() + except KeyboardInterrupt: + pass + finally: + stop() + try:process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill();process.wait() + + +if __name__=='__main__':main() diff --git a/emulator/bundles/universal.py b/emulator/bundles/universal.py index 48e22cd..642e6db 100644 --- a/emulator/bundles/universal.py +++ b/emulator/bundles/universal.py @@ -123,12 +123,16 @@ def update_image(bundle, selected, runtime_archive, game_archive): metadata={k:v for k,v in selected.items() if not k.endswith('_component')} metadata.update(schema=1,target='macos-arm64') (destination/'bundle.json').write_text(json.dumps(metadata,indent=2)+'\n') + if selected.get('architecture')=='armhf': + # Standard staging restores modes omitted by SD extraction; + # quick component builds must do the same before packaging. + builder.restore_executables(destination/'source-root') if subprocess.run(['docker','image','inspect','verstack-vm-builder:local'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode: subprocess.run(['docker','build','-t','verstack-vm-builder:local','-f', str(builder.VENDOR/'emulation/docker/Dockerfile'),str(builder.VENDOR)],check=True) builder.build_image(destination) # The runnable image now contains these immutable source bytes. - shutil.rmtree(destination/'source-root') + if (destination/'source-root').exists():shutil.rmtree(destination/'source-root') complete.write_text(json.dumps({'recipe':recipe})+'\n') except BaseException: # Only this incomplete recipe directory belongs to this attempt. diff --git a/emulator/bundles/verstack-exports.service b/emulator/bundles/verstack-exports.service index 19cbdc1..b79a0b9 100644 --- a/emulator/bundles/verstack-exports.service +++ b/emulator/bundles/verstack-exports.service @@ -4,7 +4,6 @@ After=network.target docker.service [Service] Type=simple -User=jordan WorkingDirectory=/home/jordan/verstack Environment=PYTHONPATH=/home/jordan/verstack/emulator ExecStart=/usr/bin/python3 -u -m bundles.server --store /home/jordan/verstack/data/emulator/exports --runtimes /home/jordan/verstack/data/emulator/runtime @@ -15,4 +14,4 @@ Nice=10 KillMode=process [Install] -WantedBy=multi-user.target +WantedBy=default.target diff --git a/emulator/test_bundle_api.py b/emulator/test_bundle_api.py index f661195..349637f 100644 --- a/emulator/test_bundle_api.py +++ b/emulator/test_bundle_api.py @@ -7,17 +7,24 @@ import unittest import urllib.request import urllib.error import zipfile +from unittest.mock import patch from bundles.server import ExportServer class ExportAPI(unittest.TestCase): def test_quick_export_and_component_plan_end_to_end(self): + self.check_export('aarch64') + + def test_spike2_quick_export_and_component_plan_end_to_end(self): + self.check_export('armhf') + + def check_export(self, architecture): with tempfile.TemporaryDirectory() as temporary: root=Path(temporary);runtime=root/'runtimes/sd';(runtime/'partition-02/lib').mkdir(parents=True) (runtime/'partition-02/lib/runtime').write_text('SD libraries') (runtime/'game').mkdir();(runtime/'game/game').write_text('exact game') (runtime/'runtime.json').write_text(json.dumps(dict(schema=2,snapshot='sd',system_snapshot='sd', - architecture='aarch64',system_root='partition-02',game_root='game',targets={'game':'game'}, + architecture=architecture,system_root='partition-02',game_root='game',targets={'game':'game'}, release=dict(repository='Fixture',version='1.0.0')))) server=ExportServer(('127.0.0.1',0),root/'exports',root/'runtimes') thread=threading.Thread(target=server.serve_forever);thread.start() @@ -25,6 +32,15 @@ class ExportAPI(unittest.TestCase): def get(path):return json.load(urllib.request.urlopen(base+'/emulator/exports'+path)) try: self.assertEqual(get('/catalog')[0]['runtime_label'],'Fixture 1.0.0') + self.assertTrue(get('/catalog')[0]['exportable']) + if architecture=='armhf': + standard=urllib.request.Request(base+'/emulator/exports', + data=json.dumps(dict(game_id='sd',mode='standard',catalog_url=base)).encode(), + headers={'Content-Type':'application/json','X-Verstack-Client':'1'}) + with patch('bundles.jobs.spawn'): + accepted=json.load(urllib.request.urlopen(standard)) + self.assertEqual(accepted['status'],'queued') + self.assertEqual(get('/'+accepted['id'])['runtime_id'],'sd') req=urllib.request.Request(base+'/emulator/exports',data=json.dumps(dict(game_id='sd',mode='quick',catalog_url=base)).encode(),headers={'Content-Type':'application/json','X-Verstack-Client':'1'}) job=json.load(urllib.request.urlopen(req));deadline=time.monotonic()+20 while time.monotonic(){ + window.__vmAudioBuffers=0; + const start=AudioBufferSourceNode.prototype.start; + AudioBufferSourceNode.prototype.start=function(...args){ + if(this.buffer?.getChannelData(0).some(v=>Math.abs(v)>.001))window.__vmAudioBuffers++; + return start.apply(this,args); + }; +}); +try { + await page.goto(base+'/emulator/'); + await page.locator('#runtime option').waitFor({state:'attached',timeout:90000}); + assert.equal(await page.locator('#mode').inputValue(),'game'); + assert.match(await page.locator('#runtime').innerText(),/King Kong.*SPIKE 2/); + await page.locator('#mode').selectOption('diagnostic'); + await page.getByRole('button',{name:'Start session',exact:true}).click(); + await page.locator('#media-status').filter({hasText:'Receiving screen'}).waitFor({timeout:90000}); + await page.waitForFunction(()=>{ + try{return !!JSON.parse(document.querySelector('#machine').textContent).spi;}catch{return false;} + },{},{timeout:90000}); + await page.getByRole('button',{name:'Enable audio',exact:true}).click(); + await page.waitForFunction(()=>window.__vmAudioBuffers>2,{},{timeout:90000}); + for (const name of ['service_enter','service_back','service_up','service_down']) { + const response=page.waitForResponse(r=>r.url().endsWith('/event')&&r.request().postDataJSON()?.name===name); + await page.locator(`[data-switch="${name}"]`).click(); + const reply=await response; + assert.equal(reply.status(),200,await reply.text()); + } + assert.equal(await page.locator('#error').innerText(),''); + await page.screenshot({path:process.env.VERSTACK_VM_SCREENSHOT||'/tmp/verstack-spike2-vm.png'}); + await page.getByRole('button',{name:'Stop session',exact:true}).click(); + await page.locator('#status').filter({hasText:'Session stopped'}).waitFor(); + console.log('PASS: exported VM browser screen and audio, default game selection, cabinet service controls, session cleanup.'); +} finally { + try {if(await page.locator('#stop').isEnabled())await page.locator('#stop').click();}catch{} + await browser.close(); +}