Files
LithosAnanake/FABRIC-3.10.8.md
T
Robert Allan JamesandClaude Opus 5.5 3a2ba5149f FABRIC-3.10.8 MC-1b: aarch64's second core online and parked
Cores from the DTB /cpus or, on QEMU (ACPI, no DTB), the MADT GICC
entries; the MADT lookup is now shared (hal/acpi.c) with amd64. PSCI
CPU_ON through the discovered conduit into ap_entry.S, which copies the
boot core's MMU/control state, sets TPIDR_EL1 and parks. Core 1 online
at EL1 with its own block; its vCPU thread 0.0% on the host. QEMU aarch64
-smp 2 -name debug-threads=on. 3-ISA acceptance clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 13:16:33 -04:00

79 KiB
Raw Blame History

FABRIC-3.10.8.md — Multi-core: execution plan and log

Status: IN PROGRESS. Design and rulings are in FABRIC-3.10.0.md (ruled 2026-09-25, §7). Multi-core runs ahead of O-D (MC-R6 as ruled); O-D relaunches afterwards with row path A.

Each stage:

  • needs Captain Bob's explicit go;
  • gets its own commit;
  • gets its own three-architecture acceptance (clean qemu; amd64, then aarch64, then riscv64; foreground; visible framebuffer; one at a time; coupled images reset before each).

Evidence is named before each stage runs.

The standing acceptance sequence, unchanged from FABRIC-3.9.8.md:

  • a boot to [zuse@Hera] ok> with zero UNKNOWN WORD, faults and frame corruption;
  • the dict_hash set identical across ISAs;
  • the Stage G interleave (A/B), byte-identical across ISAs while there is one worker core; by counts (A = 40, B = 40) once there are more (MC-R4);
  • the O-A checks (observer precedence, console responsive under load);
  • mint → WIREBIND → USE → 1 2 + . → 3;
  • 0 bytes of idle log growth over 20 s.

No DoE execution until everything is closed (Captain Bob, 2026-09-25: "until we close everything, do not do ant DoE eecution."). MC-4's campaigns wait for that too. Stage evidence is the standard sequence plus probes, never a campaign.

Never make clean an ISA while its QEMU runs: it deletes that QEMU's QMP and serial sockets (found 2026-09-25).

1. What MC-0 must cover (traced 2026-09-25)

Everything that would break if two cores ran kernel or VM code at the same moment.

Kernel subsystems (src/starkernel/), one lock each (MC-R2), except where noted:

Subsystem Shared state Treatment
kmalloc, pmm, vmm free lists, bitmaps, page tables heap lock; PMM lock; VMM lock
kernel-Hermes arena directory and segments, queue table, channel table, ledger, event counters one lock
Stadium reservoirs, cells, quotas, word rows per-dispatch path partitioned by quota slot (a VM's own row and cells need no lock while it runs on its one core); reservoir arithmetic atomic; cross-slot operations (admission evicting another VM's cell, grants, audits) take the Stadium lock
VM registry, birth, kill, reap registry list, names registry lock; KILL refuses a VM running on another core
switch-signal table slots, stamps, decision-due flag compare-and-swap for claim/release (MC-3); lock for register/unregister
current VM, frame stack g_switch_current_vm and friends move to the per-core block
console UART, framebuffer VT100 state, line prefix state one line lock
log region, DoE region, block subsystem rings, block chain, caches a lock each
heartbeat tick counters per-core counters (MC-3); one core until then

Shared VM code (src/, vendored; runs inside every VM):

File Global Treatment
dictionary_management.c sf_fc_list/sf_fc_count/sf_fc_cap/sf_cached_latest: one dictionary lookup index for all VMs, rebuilt whenever a different VM's dictionary is searched move into VM (per-VM index). Also stops the rebuild thrash on every switch today.
word_source/starforth_words.c g_prng_state: one PRNG stream for all VMs (accepted at Stage H) an atomic step, so the stream stays one stream (order is nondeterministic across cores anyway, MC-R4)
physics_runtime.c g_runtime inventory in MC-0; per-VM or atomic
word_source/dictionary_manipulation_words.c state_variable (STATE) inventory in MC-0; per-VM
physics_hotwords_cache.c g_hotwords_cache retired under __STARKERNEL__ (item 4.1); confirm unused
word_source/ttf_words.c font and raster cache core 0 only (Hestia's framebuffer)

2. Staged plan

Stage Change Evidence (named in advance)
MC-0a The per-core block, on one core. A per-core structure reached through each ISA's per-CPU register (GS base / TPIDR_EL1 / tp), set up for core 0 at boot. g_switch_current_vm moves into it. Standard sequence unchanged. A probe reads the per-core pointer from a normal kernel path and from an interrupt handler, and gets core 0's block both times on all three ISAs.
MC-0b Real spinlocks. A kernel spinlock primitive (atomic exchange, interrupts masked while held on the owning core). sf_mutex_* stop being no-ops. The §1 kernel locks are added. Standard sequence unchanged. A probe counts every lock's acquisitions and releases through a boot and the standard sequence: counts balance, and no lock is held across a context switch or on return to the idle point.
MC-0b2 The block subsystem's lock (split from MC-0b, Captain Bob 2026-09-25). The vendored block_subsystem.c calls its own public functions in cycles (blk_set_meta → migration check → relocate → blk_set_meta), so it is refactored into locking public functions over unlocked internals, under __STARKERNEL__. Standard sequence unchanged. The MC-0b probe, extended to the block lock: counts balance, 0 recursions.
MC-0c Shared VM code made per-VM or atomic (§1, second table). The dictionary lookup index moves into VM. The dict_hash set unchanged (the index is not hashed). A probe: two VMs searched alternately never rebuild each other's index. Standard sequence.
MC-0d Stadium's dispatch path partitioned (§1). Standard sequence; conservation holds (stadium_conserved() on every VM, the ledger audit). A probe proves the dispatch path takes no shared lock.
MC-1 Bring up the extra cores and park them halted, per ISA: amd64 (MADT, INIT-SIPI-SIPI, a position-independent trampoline below 1 MiB), aarch64 (DTB /cpus, PSCI CPU_ON), riscv64 (DTB harts, SBI HSM hart_start). QEMU gets -smp 2. The boot reports each core online with its own per-core block; the extra cores sit halted (0 idle growth, no spin); the standard sequence unchanged.
MC-2 Observers get core-0 affinity. With one worker core, behaviour identical to today. A probe shows no observer ever claimed off core 0.
MC-3 Worker cores claim through the shared decision: running_on with compare-and-swap, per-core idle contexts, per-core beats, wake by inter-processor interrupt. Two workers genuinely overlapping in time on two cores. A probe counts double claims: 0. Conservation holds. G by counts. Idle worker cores halted (0 idle growth).
MC-4 DoE with cores as a factor (after O-D): per-ISA campaigns at 1, 2 and 4 cores (MC-R5). Per ISA and core count: 18 MU-TRIAL markers, fail=0, MU-CAMPAIGN: complete.

Stop conditions (re-plan rather than push on):

  • dict_hash diverges across ISAs;
  • conservation goes false;
  • a lock held across a switch, or an unbalanced lock count;
  • a double claim;
  • an idle core spinning.

3. Execution log

3.1 MC-0a — the per-core block, on one core (2026-09-25)

Change.

  • include/starkernel/percpu.h (new): sk_percpu_t { self, core_id, current_vm }, with self first.
    • sk_percpu() reads this core's block: movq %gs:0 on amd64, mrs tpidr_el1 on aarch64, mv tp on riscv64.
    • sk_percpu_install() writes the register: IA32_GS_BASE via wrmsr, TPIDR_EL1, or tp.
  • src/starkernel/hal/percpu.c (new): the boot core's block is one static object, not a table sized to a core count. Secondary cores get theirs as they come online (MC-1).
  • kernel_main.c: sk_percpu_boot_init() runs straight after arch_early_init(), because on amd64 that call reloads GS, which would reset the base.
  • switch.c: the global g_switch_current_vm is gone.
    • sk_vm_switch_current_vm(), the setter, the idle point and sk_vm_context_switch() use sk_percpu()->current_vm.
    • After resume, the context switch re-reads the block rather than trusting a pointer cached before the switch.

Probe (temporary; removed before commit).

  • kernel_main.c checked, from the normal path after the Tripod came up, that:
    • sk_percpu()->self == sk_percpu();
    • core_id == 0;
    • current_vm is Hera;
    • sk_vm_switch_current_vm() agrees.
  • heartbeat_tick() ran the same self-check from interrupt context on every beat.
ISA log normal path core_id current_vm is Hera switch.c agrees interrupt path ok / bad
amd64 logs/20260925-130335 ok 0 1 1 889 / 0
aarch64 logs/20260925-130905 ok 0 1 1 1877 / 0
riscv64 logs/20260925-131337 ok 0 1 1 1625 / 0

Standard sequence (clean build, with the probe reverted; the tree was checked byte-for-byte against the saved stage diff).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260925-131805 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260925-133049 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260925-134357 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The dict_hash values (Hera 0x00340a8b2c842eb7, Hestia 0x78bf28439e4a643f, Artemis 0x66886d174d5f3b5a) are identical across ISAs and unchanged from O-C.
  • The ledger line matches across ISAs: held=2048, pulled=258048, returned=255263, consumed=737, conserved=true.
  • No stop condition was hit. MC-0a done.

3.2 MC-0b — real spinlocks (2026-09-25)

The primitive (include/starkernel/spinlock.h, src/starkernel/hal/spinlock.c, new).

  • One lock word per lock: 0 = free, core_id + 1 = held.

    • Taken by compare-and-swap. The atomics are inline on every ISA (nm -u is empty on all three ELFs).
    • Interrupts stay masked on the owning core while it holds any lock.
  • The mask state lives in the per-core block, not in the lock (lock_depth, saved_irq in sk_percpu_t):

    • the first lock taken saves the interrupt state and masks;
    • the last lock released restores it;
    • so release order doesn't matter.
  • The rule: no lock crosses a switch or reaches the idle point. Each of these sites checks it and counts any violation:

    • sk_vm_context_switch(), when leaving and on resume;
    • sk_vm_idle_point().
  • Failures are counted and logged at the idle point (LOCK: lines, LOG_ERROR):

    • recursion (a core takes a lock it already holds): let through, so the core doesn't hang, and counted. It is never legitimate, because on one core "the owner is me" can't tell a nested call from a second VM on the same core;
    • release of a lock not held;
    • a lock held at a switch or idle point;
    • a VM mutex destroyed while held.
  • The locks are no-ops until the boot core's block is installed (sk_percpu_ready). This was found during the probe runs:

    • riscv64 is built monolithic, and its arch_early_init() prints through the console before sk_percpu_boot_init();
    • the first lock therefore read tp while it still held the firmware's value, and the load hit OpenSBI memory (stval = tp+8, a firmware load access fault);
    • early boot is one core with interrupts off, so skipping the locks there is correct.

    That crashed run's log was deleted as flawed.

  • sf_mutex_* (the vendored VM's dict_lock and tuning_lock, shim.c) now use the same lock word. They are no longer no-ops.

The kernel locks, in acquisition order (spinlock.h; MC-3's deadlock freedom rests on this order; checked by reading the code, not yet by a counted check; MC-0b2 inserts block between log and console, §3.3):

  1. hermes
  2. switch
  3. registry
  4. doe
  5. log
  6. console
  7. heap
  8. vmm
  9. pmm

Stadium's lock comes with MC-0d, below Hermes. Stadium today runs inside dict_lock (vm_core.c's heat bump calls stadium_word_dispatch()); MC-0d's order must account for that.

Lock Covers Shape
heap kmalloc/kfree, stats, censuses kmalloc() delegates to kmalloc_aligned(), and only the latter locks
vmm page-table walks *_locked bodies. The map-conflict print moved outside the lock (the console lock sits above it). Takes pmm for table pages
pmm bitmap pmm_alloc_page()/pmm_free_page() delegate to the contiguous forms
console output path and line state (name, line start, pending line close, the framebuffer terminal) A string goes out under one hold. Unlocked on purpose: console_get_vm_name() (the prefix provider calls it under the lock), the boot-time console_fb_init()/console_fb_enable_ttf() (they log), and input
log, doe control-block read-modify-write plus the slot write *_locked bodies
registry node list, entry copies, single-field writes Never held while calling out. KILL and reap take it only for their reads and final writes. A birth fills its embryo unlocked (private until LIVE). kill_all_nonmama (cold-reset path) walks unlocked
switch slot table, stamps, decision-due flag, counters Never held while calling out. The decision, the Stage H floor and the reap sweep copy one slot under the lock (slot_copy()) and judge it outside
hermes arena, ledger, channels, queues, event counters Public functions lock over *_locked statics. Never held across vm_interpret_kind() (the drain) or a policy word. The switch notifications after a send go out once the lock is dropped (the floor reads Hermes queues back). Publish and event walk members one copied member at a time. sk_hermes_event()'s static payload moved onto the stack

The block subsystem is split out as MC-0b2 (Captain Bob, 2026-09-25), because its internal call cycles need a locked/unlocked refactor of vendored code. See §2.

Probe (temporary, removed before commit). A MC0B-PROBE word in Hera's and every child's vocabulary printed every counter, snapshotted before printing:

  • each lock's acquisitions and releases, and its owner;
  • the VM mutexes (the core total minus the kernel locks);
  • the core-0 total and its depth;
  • the number of switch/idle checks and violations;
  • recursions, bad releases and destroyed-while-held.

It ran at Hera's prompt (after O-A's G, PREC and LOAD), and again on rajames after MINT, WIREBIND, USE and 1 2 + .. The table shows the final (rajames) reading:

ISA log total acquired = released depth switch/idle checks held at switch recursions / bad releases / destroyed held
amd64 logs/20260925-150735 1,503,458,645 0 51,137,748 0 0 / 0 / 0
aarch64 logs/20260925-152109 235,854,532 0 7,565,646 0 0 / 0 / 0
riscv64 logs/20260925-155048 566,894,806 0 19,029,507 0 0 / 0 / 0
  • Every lock balances individually on every ISA, at both readings.
  • The amd64 and aarch64 probe runs predate the sk_percpu_ready guard. That guard only changes behaviour before sk_percpu_boot_init(), and the acceptance runs below are on the final code.
  • The riscv64 vblk_io timed out count, watched because known defect 3 sits in code that now runs with interrupts masked, is 2 in the probe run, the same as MC-0a's acceptance run. It is 0 in this stage's acceptance run.

Finding, for MC-3. Hera's input-poll idle loop runs the switch decision without pause, and every pass takes the registry, switch and Hermes locks once per slot. That is the hundreds of millions of acquisitions above (amd64: registry 379 M, switch 858 M, Hermes 266 M). It is harmless on one core. With several cores, core 0 spinning through these same locks would contend with every worker's claim. MC-3 should settle how that loop waits before cores share these locks.

Standard sequence (clean build, with the probe reverted; checked against the saved stage diff, which differs only by the guard added after it).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260925-160434 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260925-161720 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260925-163024 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The G string is byte-identical to MC-0a on all three ISAs, so masking interrupts under the locks didn't move the interleave.
  • The dict_hash set and the ledger line are unchanged from MC-0a.
  • No stop condition was hit. MC-0b done (the block subsystem follows as MC-0b2).

3.3 MC-0b2 — the block subsystem's lock (2026-09-25)

Change. src/block_subsystem.c (vendored) is split into bodies and wrappers:

  • Every public function's body is now a static *_locked function.

  • Bodies call each other only through the _locked forms. That breaks the internal cycles:

    • relocate → get buffer / update / mark free;
    • meta relocate → relocate;
    • migration scan → meta relocate;
    • firsttouch → get meta / set meta / mark allocated;
    • the six meta accessors.
  • A block of thin public wrappers at the end of the file takes the lock around each body.

    • Under __STARKERNEL__ that lock is g_lock_block.
    • The hosted build has no lock, and the vendored hosted make still builds clean.
  • The lock covers the device chain, every slot's cache and BAM, the relocation table, and the device I/O issued on their behalf.

  • Its place in the order is below log and doe (both regions write through the block subsystem) and above console (block logs), heap (calloc/free), vmm and pmm:

    1. hermes
    2. switch
    3. registry
    4. doe
    5. log
    6. block
    7. console
    8. heap
    9. vmm
    10. pmm

    Stadium (MC-0d) sits above block, because STADIUM_BEHAVIOUR_MIGRATE calls blk_flush().

Not covered, and not coverable by this lock: blk_get_buffer() returns a pointer into RAM, a ramdrive, or a device cache slot. The caller (BLOCK/BUFFER, Artemis) reads and writes through that pointer after the lock is released. On one core nothing can evict the slot in between; with worker cores (MC-3) another core's block access can. Block-buffer ownership across cores is an open item for MC-3.

The masking risk, measured. virtio_blk.c records (2026-09-09) that masking interrupts around vblk_io() once made missed completions worse. All block I/O now runs with interrupts masked (the MC-0b primitive), so the probe runs counted vblk_io timed out on every ISA:

Where amd64 aarch64 riscv64
This stage's probe runs 0 0 4
This stage's acceptance runs 0 0 2

The riscv64 counts sit inside the pre-lock baseline: the 24 most recent riscv64 logs, from 2026-09-24 (no locks at all) through this stage, range from 0 to 6, with 4, 5 and 6 all seen before MC-0b. No regression is visible at this sample size. The masking stays as MC-0b ruled.

Probe (temporary, removed before commit). This is MC-0b's MC0B-PROBE word, extended with the block lock. Same sequence: O-A, then Hera's prompt, then MINT, WIREBIND, USE and 1 2 + ., then rajames. The table shows the final (rajames) reading:

ISA log block acquired = released total acquired = released depth switch/idle checks held at switch recursions / bad releases / destroyed held
amd64 logs/20260925-173901 2,379 1,387,861,111 0 46,795,001 0 0 / 0 / 0
aarch64 logs/20260925-175215 2,057 247,761,816 0 8,014,413 0 0 / 0 / 0
riscv64 logs/20260925-180546 2,300 555,650,314 0 18,447,619 0 0 / 0 / 0

Every lock balances individually on every ISA, at both readings. 0 recursions means no public-to-public call inside block_subsystem.c was missed.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260925-181908 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260925-183151 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260925-184454 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The G string is byte-identical to MC-0b and MC-0a on all three ISAs.
  • The dict_hash set and the ledger line are unchanged.
  • No stop condition was hit. MC-0b2 done.

3.4 MC-0c — shared VM code made per-VM or atomic (2026-09-25)

Scope: §1's second table, plus five further items found by a sweep of every file-scope and function-local mutable variable in the vendored VM sources the kernel compiles. Captain Bob ruled all of it into this stage (2026-09-25).

§1's table:

Item Finding Change
dictionary lookup index (sf_fc_list/count/cap, sf_cached_latest) One index for the whole system, holding whichever VM's dictionary was searched last: every switch between VMs that look words up rebuilt it. dict_reorganize_buckets_by_heat() reordered that shared index, which could be another VM's. Per VM: struct SfFcIndex (dictionary_heat_optimization.h), vm->fc_index, allocated on first lookup and freed by both vm_cleanup()s. An allocation failure is counted and logged, and the lookup walks the dictionary newest-first instead (still correct). The heat-aware lookup and the heat reorganization use the VM's own index. The index's unused global reset is gone, as are the benchmark word's stale externs. The 256 buckets are one per possible first byte (the whole domain, not a capacity).
g_prng_state (starforth_words.c) One stream for all VMs, as accepted at Stage H; plain read-modify-write. Still one stream. The step is a compare-and-swap loop, and SEED is a single atomic store.
g_runtime (physics_runtime.c) The kernel build (STARFORTH_MINIMAL) compiles stubs; g_runtime exists only hosted. None.
state_variable (dictionary_manipulation_words.c) Written by INTERPRET, never read. STATE is vm->mode, already per VM. Removed (dead).
g_hotwords_cache (physics_hotwords_cache.c) Defined, never referenced; the live cache is per VM (vm->hotwords_cache). None.
ttf_words.c font and raster cache Hestia's framebuffer. None; core 0 only, per plan.

Found by the sweep:

Item Finding Change
heartbeat_export.c, three function-local statics heartbeat_capture_tick_snapshot() (every beat, vm_runtime.c) took each VM's deltas (beats, words executed, ANOVA early exits) against whichever VM was sampled last. Wrong across VMs on one core today. The snapshot feeds doe_log_tick_row() only, not any decision. The previous values move into the VM's HeartbeatState (export_last_*).
TOP-WORDS (starforth_words.c) A static DictEntry *[1000]: shared by every VM, and silently cut at 1000 words. Sized to the heated words and allocated per call. A refusal sets vm->error and is logged.
inference_outputs_to_string() Returned a static 256-byte buffer. No callers. Takes a caller's buffer.
Failure counters: word-id refusals, transition-grow failures, WORD too long Plain ++. Atomic increments and loads.
block_subsystem.c crc64 table Lazy init with no publish ordering; compute_crc64() also runs outside the block lock. Published with release/acquire.

Left as system-wide by design: system_running and forth_79_standard (system_words.c, system flags). profiler_state and vm_debug's g_vm are hosted/diagnostic; vm_debug is stubbed in the kernel.

The vendored hosted make builds clean.

Probe (temporary, removed before commit). An MC0C-PROBE word in Hera's and every child's vocabulary:

  • listed every VM's index, and how many times it has ever been rebuilt;
  • then looked up DUP, SWAP, EMIT, BLOCK and USE in two VMs alternately, 1,000 rounds each (5,000 lookups per VM);
  • checked that every result belongs to the dictionary of the VM that was searched.

It ran at Hera's prompt after O-A, and again on rajames after the standard sequence. The results are identical on all three ISAs:

ISA log every VM has its own index Hera rebuilds (her own defines/forgets) every other VM's rebuilds alternating 2×5000: rebuilds / found / wrong-VM alloc failures
amd64 logs/20260925-194705 yes (7 VMs) 85 2 0+0 / 10000 / 0 0
aarch64 logs/20260925-200019 yes (7 VMs) 85 2 0+0 / 10000 / 0 0
riscv64 logs/20260925-201350 yes (7 VMs) 85 2 0+0 / 10000 / 0 0

These runs' dict_hash values differ from the table below only because the probe adds a word.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260925-202657 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260925-203935 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260925-205238 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The dict_hash set is unchanged, as the plan's evidence requires (the index is not hashed): Hera 0x00340a8b2c842eb7, Hestia 0x78bf28439e4a643f, Artemis 0x66886d174d5f3b5a.
  • The G string is byte-identical to MC-0b2, MC-0b and MC-0a on all three ISAs.
  • The ledger line is unchanged.
  • No stop condition was hit. MC-0c done.

3.5 MC-0d — Stadium partitioned by quota slot (2026-09-25)

Ruling (Captain Bob, 2026-09-25): one lock per quota slot. FABRIC-3.10.0 §3.6 said a VM's own Stadium row and cells need no lock, with a Stadium lock only for cross-slot operations. That can't be made sound: a lock taken only on the cross-slot side can't exclude an owner that takes none. Concrete race:

  • a receiver's drain runs sk_hermes_release() → stadium_evict() on the sender's message cell, which rewrites the sender's free list, resident list and reservoir;
  • meanwhile the sender, running on another core, is popping that same free list in stadium_admit().

stadium_grant_quota() splits a donor's free list the same way. The amendment is recorded at FABRIC-3.10.0 §3.6.

Second ruling: the block table takes the shared Stadium lock. stadium_blocks.c keeps one open-addressed table for every VM, keyed by (slot, LBN). stadium_block_dispatch() therefore takes g_lock_stadium. It is the one shared-lock Stadium path, recorded for MC-3 to measure; block access is rare next to word dispatch.

Change.

  • The slot lock.

    • StadiumVMQuota gains lock, a spinlock.h lock word.
    • It covers the slot's free list, resident list, reservoir, consumed, word row and statistics, and the slot's cells: their headers, owner entries and resident-list links.
    • Every public Stadium function locks the slot it touches. A cell operation locks the cell's owning slot (lock_cell_owner() re-reads the owner once locked).
  • Word dispatch (stadium_word_dispatch()) holds its own VM's slot lock for the whole dispatch: row, cell, cooling, pull, and admission including its eviction fallback. It uses the new *_in_slot forms, so it takes no other lock (a row growth's kmalloc excepted).

    • The reservoir peek-then-pull is now one step under that lock.
  • Global part: g_lock_stadium.

    • Quota-slot allocation in grants. Each new slot is filled first, then published with a release store of in_use; quota_slot_for_vm() reads in_use with acquire.
    • The shared block table.
    • It is taken before a slot lock, never after.
  • The residency bitmap: a byte covers 8 cells, which can belong to different slots, so set and clear are atomic OR/AND.

  • Cross-slot writers moved into Stadium, under the owner's lock:

    • kernel-Hermes's decay: stadium_cell_decay(), the same q48_mul arithmetic, consumed recorded in the same step;
    • Hermes's release ledger: stadium_evict_credit() returns the heat actually credited, read in the same step as the eviction, where it used to be read from the cell beforehand;
    • session_set_pinned()'s pin bit: stadium_cell_set_pin().
  • stadium_conserved() reads its three terms under one hold, so the sum is of one instant.

  • Owner check added to both dispatch paths' cell lookups: the cell must still be this slot's.

    • Without it, a word or block cell evicted to this slot's free list could move to another VM in a grant and be admitted there with a matching identity. Word ids and LBNs are shared across VMs, so the old row would then heat the other VM's cell.
    • This is latent on one core.
  • Unpartitioned reads that remain: sk_hermes_scan_held() (reads message cells under the Hermes lock), session.c's bounds check, and kernel_main.c's decay self-test. None writes.

  • Lock order (spinlock.h):

    1. hermes
    2. switch
    3. registry
    4. stadium
    5. a slot lock
    6. doe
    7. log
    8. block
    9. console
    10. heap
    11. vmm
    12. pmm

    A VM's dict_lock sits above Stadium: vm_core.c's heat bump dispatches the word under it. Nothing in the block path calls into Stadium, so MIGRATE's blk_flush() edge doesn't close a cycle.

Probe (temporary, removed before commit). It counted:

  • the per-core lock acquisitions made during each stadium_word_dispatch();
  • slot-lock acquisitions made for a slot other than the running VM's own;
  • admission-fallback evictions of message (DELIVER) cells;
  • quota slots in use.

An MC0D-PROBE word in every VM printed these, with stadium_conserved() on every live VM and the Hermes audit-failure count. The results are identical on all three ISAs, count for count:

ISA log word dispatches exactly one lock (own slot) more than one …all at a row growth conserved Hermes audit failures DELIVER-cell fallback evictions
amd64 logs/20260925-212159 10,746,616 10,746,599 17 17 7/7 0 0
aarch64 logs/20260925-213527 10,746,616 10,746,599 17 17 7/7 0 0
riscv64 logs/20260925-214855 10,746,616 10,746,599 17 17 7/7 0 0

The table shows the final (rajames) reading. At Hera's prompt, before MINT, the readings were 9,687,103 dispatches with 14 exceptions (all row growths), 5/5 conserved, and 268 cross-slot slot locks.

Findings, reported and not fixed:

  1. The running VM isn't always the core's current VM. Cross-slot slot locks went from 268 to 1,059,738 across MINT, WIREBIND, USE and 1 2 + ., the same growth as word dispatches in that window.
    • A birth runs the child's capsule on the parent's stack (capsule_birth.c), and USE runs another VM's lines, so these dispatches are for a VM other than sk_percpu()->current_vm.
    • The Stadium locking is correct either way: it locks the dispatching VM's own slot.
    • But MC-3's claim (running_on) must cover a VM executed this way, not only one switched to.
  2. Quota slots are never released. stadium.c has no release path: in_use is set by grants and cleared only at boot.
    • The probe counted 13 slots in use against 7 registry entries. Boot self-tests grant quotas to test ids (kernel_main.c), and a dead VM keeps its slot and cells.
    • A long-running system eventually exhausts the RAM-sized slot table.
  3. An owner's word admission can evict a live message cell. The admission fallback evicts the owner's least-dense resident, skipping only pinned and containing cells. Hermes admits messages unpinned into the sender's quota.
    • After such an eviction, msg->stadium_cell is stale: the release would evict whatever occupies that cell next.
    • This run counted 0 such evictions (the quotas never ran dry of free cells), so it is latent.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260925-220202 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260925-221440 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260925-222740 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The G string is byte-identical to MC-0c and earlier on all three ISAs, so the O(ncells) audit scan now running under a slot lock didn't move it.
  • The dict_hash set is unchanged, and the ledger line reads stadium_conserved(self)=true.
  • No stop condition was hit. MC-0d done, and with it MC-0: one core, fully locked, ready for MC-1.

4. Closing the known issues (plan, 2026-09-26)

Captain Bob, 2026-09-26: "CONTINUE BY CLOSING KNOWN ISSUES" -- ahead of MC-1. The list is FABRIC-3.10.0 §8's ten known defects plus what MC-0 found. Stages are grouped by subsystem; each gets its own commit and three-architecture acceptance (§ header rules), with evidence named here before it runs. No DoE.

Rulings taken for this plan (2026-09-26):

  • Released quota cells go to the best donor (stadium_best_donor(): the live slot with the most free cells).
  • The hosted make stops copying its binary into lfs/ (it rewrote the tracked lfs/amd64/starforth on every compile-sanity build).
Stage Closes Change Evidence (named in advance)
K-1 §8 defects 5, 6, 7, 8, 9; plus the kernel sf_realloc() The kernel sf_realloc() becomes a real realloc: copy min(old, new) bytes (old size from a new kmalloc_usable_size()), free the old block, and free-and-return-NULL on size 0. It used to allocate fresh, copy nothing and free nothing. That makes the dictionary index's bucket growth stop leaking (defect 9), and fixes vm_state_push(), whose call-state stack lost every saved state past depth 8. The shim's realloc() copies min(old, new); it used to over-read the old block when growing. The heat-percentile snapshot is heap-sized to the dictionary (defect 5). A word whose transition metrics can't be allocated is refused whole (defect 6). Both vm_cleanup()s free the metrics' inner arrays (defect 7). The most-likely-next cache updates incrementally (defect 8; compiled out of the kernel today). Standard sequence unchanged. A probe shows: sf_realloc() growth preserves contents and returns the old block (heap used bytes back to baseline); a call-state stack pushed to depth 20 pops back intact; kernel heap used bytes after a 20-VM birth/KILL churn return to within one allocation of the pre-churn figure.
K-2 §8 defects 1, 2 COLD returns the dictionary to its post-boot fence through FORGET's own path (latest, HERE, word ids and Stadium cells together); it used to rewind HERE only, leaving live words to be compiled over. PAD moves below the block I/O window (BLK_VM_WINDOW_BASE - 512), outside both the dictionary and the window. Standard sequence unchanged. A probe on a worker: define three words, COLD, and each is gone (not findable), HERE equals the fence, a new definition works; PAD plus 511 lies below BLK_VM_WINDOW_BASE and above the dictionary limit.
K-3 §8 defect 4 The two self-tests that wait for the removed Hermes VM (drain; channel-open policy) are retargeted to live Tripod VMs with the same properties. Both self-tests print PASS on all three ISAs; no SKIPPED line for either.
K-4 MC-0d findings 2, 3 Quota slots are released: on KILL and reap, and at the end of each boot self-test that granted one. Resident patrons are evicted, the cells join the best donor's free list, and the slot is cleared for reuse. The admission fallback no longer evicts message (DELIVER) cells; a message leaves Stadium only through its release. Standard sequence unchanged; conservation holds on every live VM. A probe: quota slots in use equal live VMs after boot and after a 20-VM churn; a forced admission fallback with a pending message leaves the message cell resident.
K-5 §8 defect 10 The xHCI Input Context is allocated full size (33 contexts) once per slot at Address Device and never swapped, mirroring the Output Device Context fix. Configure Endpoint used to allocate a fresh one on every attach and never free the old. Standard sequence unchanged (WIREBIND attaches twice). A probe: kernel heap used bytes identical after the second attach as after the first.
K-6 §8 defect 3 riscv64 vblk_io timeouts: investigate first. A probe measures completion latency (poll iterations) and whether a timed-out request's completion later arrives. If it does, a timeout leaves a late completion that the next request would take as its own, while the device may still be reading reused descriptors. The fix is chosen from what the probe shows. Named after the investigation, before the fix: completion-latency distribution per ISA, late completions observed, and after the fix 0 misattributed completions.
K-7 the lfs/ rewrite The hosted Makefile no longer copies the binary into lfs/. After make, git status shows no change under lfs/.

MC-1 follows once these are closed.

4.1 K-1 — the allocator and word metrics (2026-09-26)

Change.

  • The kernel sf_realloc() is a real realloc. It allocates the new block, copies min(old, new) bytes, frees the old block, and on size 0 frees and returns NULL. The old size comes from a new kmalloc_usable_size(); on failure the old block is untouched.
    • It used to allocate fresh and do nothing else, while its callers assumed standard realloc.
    • Found while closing defect 9: vm_state_push()'s call-state stack (BIRTH, VM-EXEC and START nesting) grows through sf_realloc(), so past depth 8 every saved state was lost, and every growth leaked.
  • Shim realloc() copies min(old, new); it used to copy size bytes, over-reading the old block whenever it grew.
  • Defect 9: the dictionary-index bucket growth drops its kernel-only copy patch, which would now read freed memory, and stops leaking the old block. It also calls sf_realloc() directly instead of sf_xrealloc(), which freed the old block on failure while the index still pointed at it. sf_xrealloc() is gone.
  • Defect 5: the heat-percentile snapshot is heap-sized to the dictionary, allocated per call (was heats[DICTIONARY_SIZE], cut at 1024 words). An allocation failure is logged and the previous thresholds stand.
  • Defect 6: a word whose transition metrics can't be allocated is refused whole: vm->error set, logged at ERROR, counted with the word-id refusals. It used to be defined anyway, with a WARN.
  • Defect 7: both vm_cleanup()s call transition_metrics_cleanup() before freeing each word's metrics, so the inner arrays are freed.
  • Defect 8: transition_metrics_record() keeps the most-likely-next word current (highest count, lowest id on a tie), and transition_metrics_update_cache() only refreshes its probability instead of scanning every id.
    • Compiled out of the kernel today (ENABLE_PIPELINING off).
    • One by-construction difference is documented at the function: the scan compared floored Q48.16 probabilities.
    • Also fixed: a NULL dereference in that function's guard.

Probe (temporary, removed before commit). A K1-PROBE word:

  1. sf_malloc(64) with a pattern, sf_realloc to 4096 then to 32, checking the pattern each time, then free, reading heap used bytes before and after;
  2. a scratch VM's call-state stack pushed to depth 20 and popped, checking every saved field;
  3. heap used bytes.
ISA log realloc grow / shrink heap used before = after call-state depth 20
amd64 logs/20260926-011757 kept / kept yes (3 readings) intact, capacity 32
aarch64 logs/20260926-014005 kept / kept yes intact, capacity 32
riscv64 logs/20260926-014626 kept / kept yes intact, capacity 32

The churn half of the named evidence is not met. On amd64 the probe ran churn.sh (28 worker births, jobs and KILLs) twice, with the probe word before, between and after:

  • heap used: 28,871,243 → 29,196,203 (+324,960) → 29,397,899 (+201,696);
  • so about 7 KB is still retained per churned VM.
  • Most likely cause: the per-VM state K-4 releases. A dead VM keeps its quota slot, including its Stadium word row (up to about 16 KB at a 1024-id row). Registry nodes are kept by design. K-4 re-measures this.

In the same run, after the second churn, rajames's 1 2 + . was refused by kernel-Hermes ("refused CONSOLE-CMD-EVENT send").

  • Most likely cause: the quota leak. Every KILL strands its slot's cells, and each birth halves the best donor, so after 56 churned VMs the newborn console VM's quota can't admit a message cell.
  • K-1 doesn't touch Stadium or Hermes. K-4's probe re-runs this double churn followed by 1 2 + ., which is the direct test.
  • The clean acceptance runs below have no churn, and CONSOLE-CMD works in all of them.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff; the hosted make builds).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260926-015235 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260926-020511 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260926-021813 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The dict_hash set, the G string and the ledger line are unchanged.
  • K-1 closes §8 defects 5–9 and the sf_realloc()/call-state-stack bug. Its churn-leak criterion carries into K-4.

4.2 K-2 — COLD and PAD (2026-09-26)

Change.

  • Defect 1 (COLD). reset_vm_state()'s cold branch now calls a new vm_dictionary_forget_to_fence() (dictionary_management.c). It forgets every word defined since the fence exactly as FORGET does, and rewinds HERE to the fence:

    • untrack (the id recycled, the Stadium cell reclaimed);
    • free the transition metrics and their arrays;
    • free the entry.

    It used to rewind HERE alone, to 1024, leaving latest at live words that the next definitions compiled over.

    • The fence is set after vm_init() allots the input buffers, so HERE can't land below them.
    • What COLD forgets is therefore what FORGET could forget. For a worker that includes its workload capsule's words (the fence is set at vm_init()). Hera's SDK capsule raises FENCE, so her base and SDK words survive.
    • FORGET itself is unchanged.
  • Defect 2 (PAD). PAD is now BLK_VM_WINDOW_BASE - 512, the 512 bytes just below the block I/O window. It was VM_MEMORY_SIZE - 512, inside the window's top slot. The span between the 2 MB dictionary limit and the window is otherwise unused.

Probe (plain FORTH, k2.sh, run on the clean build after O-A; no temporary code):

  • 7770 . PAD . PAD 511 + . on Hera;
  • a worker kc defines KA, KB and KC and prints whether KA and KC are findable, then HERE;
  • then COLD, FIND for each of the three, HERE, and a new definition : KD 4 ; KD ..

The results are identical on all three ISAs:

ISA log PAD … PAD+511 window base / dictionary limit before COLD: KA, KC found; HERE after COLD: KA, KB, KC; HERE; KD
amd64 logs/20260926-023300 5,238,272 … 5,238,783 5,238,784 / 2,097,152 yes, yes; 1856 0, 0, 0; 1424 (the fence); 4
aarch64 logs/20260926-024629 same same same same
riscv64 logs/20260926-030013 same same same same

Standard sequence (the same runs, after the probe).

  • All three ISAs pass: hash 0x99b01134c3d3049c, G byte-identical (A=40 B=40, run before the probe), MINT / WIREBIND / USE / 1 2 + . → 3, 0 B idle growth, 0 UNKNOWN and 0 faults.
  • The ledger totals (pulled 262144 / returned 259359) are higher than the usual line only because the probe worker's messages are counted; held and consumed are unchanged, and stadium_conserved(self)=true.
  • The hosted make builds.
  • K-2 closes §8 defects 1 and 2.

4.3 K-3 — the two skipped self-tests (2026-09-26)

Change (kernel_main.c). Both self-tests looked up a VM named "Hermes", which hasn't existed since the Phase 4 reshuffle, so both printed SKIPPED on every boot. Both now use Hestia:

  • Drain self-test: the target is Hestia. The nested depth-2 check (which must not drain) VM-EXECs a harmless 1 DROP into her from Hera, in place of Hermes's WELCOME word. The outermost drain must take exactly the one message.
  • Channel-open policy self-test: Hestia is the no-policy VM. Like the old Hermes VM she never loads ACL.4th (capsules/hestia/init.4th loads fabric, font and lib only), so she has no HERMES-CHANNEL-OPEN? and must be refused closed.
  • A missing target now prints FAIL, not SKIPPED.

Evidence: both self-tests print PASS on all three ISAs, and no SKIPPED line appears for either.

Standard sequence. All three ISAs pass: hash 0x99b01134c3d3049c, G byte-identical, MINT / WIREBIND / USE / 1 2 + . → 3, 0 B idle growth, 0 UNKNOWN and 0 faults.

ISA log
amd64 logs/20260926-031446
aarch64 logs/20260926-032735
riscv64 logs/20260926-034037

The ledger line's baseline moves from here on: pulled 262144 and returned 259359, from 258048 and 255263. The drain self-test now really publishes and drains one message, which adds one message cell's pull and return. Held (2048) and consumed (737) are unchanged, and conservation holds.

Found, not fixed: three more references to the retired Hermes VM.

  1. capsules/artemis/init.4th block 4129, ARTEMIS-ANNOUNCE-READY, VM-EXECs 2 ENQUEUE-READY into "Hermes".
  2. doe_log.c:207 reads a per-row "Hermes" heat column (doe_log_heat_by_name("Hermes")).
  3. The CONNECT-HERMES word (mama_forth_words.c ~2257) still births and names a Hermes VM.

K-3 closes §8 defect 4.

4.4 K-4 — quota slots released; message cells kept out of the admission fallback (2026-09-26)

Change (stadium.c; ruling: released cells go to the best donor). Release happens in two steps, because a dead VM's slot can still hold cells for messages it sent that sit in other VMs' queues.

  1. stadium_release_quota(vm_id), under g_lock_stadium, then the slot's lock:

    • every resident patron except pending messages (DELIVER) is evicted, its heat back to the slot's reservoir; pins are cleared first, since a dead VM's pins no longer matter;
    • the slot is marked retiring, and is never a donor while retiring;
    • a patron eviction refuses (one holding another) stays resident and is logged.
  2. stadium_retire_sweep() runs at release, and after any eviction while a slot is retiring, which is how a retiring slot's last message leaving completes its release. It is one atomic load when nothing is retiring. For each retiring slot with nothing resident:

    • detach the free list;
    • clear its word row (stadium_words_slot_released(), under the slot lock) and its block-table entries (stadium_blocks_slot_released(), under the global lock);
    • hand the cells to the best donor (one slot lock at a time, each cell's owner set to the donor);
    • clear the slot and publish in_use = 0.

    The dead VM's Q48_ONE share leaves with the slot. The donor gains free cells only, with no heat, so its own sum is unchanged.

  • Callers:
    • KILL and reap release the VM's slot after its Hermes departure;
    • kernel_main.c releases the nine synthetic self-test ids (hi 0, lo 1, 3, 7, 11, 13, 15, 16, 19, 20) after the last self-test, and prints Stadium: self-test quotas released=;
    • Hera's slot is never released.
  • MC-0d finding 3: the admission fallback skips message (DELIVER) cells. A message leaves Stadium only through its release.

Probe (temporary K4-PROBE / K4-FALLBACK words, removed before commit). They print:

  • quota slots in use (and retiring) against live VMs, conservation on every live VM, slots released so far, and heap used;
  • the forced fallback:
    1. grant a synthetic slot T, and send one real message from it to a synthetic receiver;
    2. fill T's free list with zero-heat fillers until admission refuses;
    3. admit a denser candidate (heat 1, pulled from T's reservoir) and check the message cell;
    4. clean up: the receiver departs (its mail released), then T is released.

Then two churn.sh rounds (28 worker births, jobs and KILLs each), K4-PROBE after each, and the standard sequence on the churned system, the direct test of K-1's post-churn refusal.

ISA log self-test quotas released slots in use = live VMs (boot / churn 1 / churn 2) fallback: denser admitted, message cell resident and intact post-churn 1 2 + .
amd64 logs/20260926-035738 9 3=3 / 3=3 / 4=4 † yes, yes, yes 3
aarch64 logs/20260926-042229 9 3=3 / 3=3 / 3=3 yes, yes, yes 3
riscv64 logs/20260926-044430 9 3=3 / 3=3 / 3=3 yes, yes, yes 3
  • Every live VM was conserved at every reading, with 0 Hermes audit failures and 0 slots left retiring.
  • † amd64: my script's timeout cut the second churn short (36 kills). I ran a third full round, 64 churned VMs in total, past K-1's 56; one worker from the interrupted round was still alive at the final reading.

K-1's carried criterion (heap), re-measured:

Heap growth per 28-VM churn round round 1 round 2
amd64, K-1 (before K-4) +324,960 +201,696
aarch64, K-4 +95,584 +95,200
riscv64, K-4 +95,584 +95,200

The per-round growth halved, but about 3.4 KB is still retained per churned VM, so the "within one allocation" criterion is still not met. Registry nodes are kept by design (a node per VM ever born); the rest is unidentified. It stays open as K-8: find the ~3.4 KB/VM residual.

K-1's post-churn Hermes refusal is gone: after 56–64 churned VMs, rajames's 1 2 + . returns 3 on every ISA. That confirms the refusal was the quota leak.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log O-A (G, PREC, LOAD) M7.1a hash MINT / WIREBIND / USE / 1 2 + . idle growth UNKNOWN / faults
amd64 logs/20260926-050559 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
aarch64 logs/20260926-051834 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
riscv64 logs/20260926-053135 A=40 B=40, HHH, ok 0x99b01134c3d3049c all ok, 3 0 B / 20 s 0 / 0
  • The G string and the ledger line (the K-3 baseline) are unchanged.
  • K-4 closes MC-0d findings 2 and 3. K-1's heap criterion carries on as K-8.

4.5 K-5 — the xHCI Input Context leak (2026-09-26)

Change.

  • New XHCI_INPUT_CTX_BYTES (xhci.h): the Input Control Context plus a Slot Context and all 31 Endpoint Contexts.
  • Address Device allocates the slot's Input Context at that full size, once per xHCI slot, mirroring the Output Device Context fix.
  • Configure Endpoint reuses it, bounds-checking its span. It used to allocate a new block on every attach and never free the old one.

Evidence (the named criterion, counted directly). Heap bytes would mix in the WIREBIND births that attach triggers, so a temporary counter recorded Input Context allocations against Configure Endpoint calls, read before and after the standard sequence:

  • at Hera's prompt, before the sequence: 1 allocation / 1 configure, the Zuse thumbdrive attached at boot;
  • after the sequence (Bob's drive attached, detached and attached again): 2 allocations / 3 configures, one per xHCI slot used. Before K-5 every configure allocated a fresh block.

The counts are identical on all three ISAs (probe logs: amd64 logs/20260926-054615, aarch64 logs/20260926-055249, riscv64 logs/20260926-055939).

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log result
amd64 logs/20260926-060605 hash 0x99b01134c3d3049c, G A=40 B=40, all ok 3, 0 B idle, 0 / 0
aarch64 logs/20260926-061840 same
riscv64 logs/20260926-063140 same

The ledger line is unchanged. K-5 closes §8 defect 10.

4.6 K-7 — the hosted build no longer copies into lfs/ (2026-09-26)

Change (ruled 2026-09-26). The hosted Makefile's all target no longer copies the binary to lfs/<arch>/starforth. In this repo the hosted build is a compile-sanity check only, and the copy rewrote the tracked lfs/amd64/starforth on every build.

Evidence: a forced hosted relink (touch src/main.c; make) builds, and git status shows no change under lfs/. There is no kernel change, so no QEMU acceptance (the hosted build isn't an acceptance target, CLAUDE.md).

4.7 K-6 — riscv64 vblk_io timeouts: investigated, root-caused, fixed (2026-09-26)

Investigation (temporary instrumentation in vblk_io(); two riscv64 boots, each running O-A and the standard sequence).

  1. Poll iterations per completion (logs/20260926-064553, 2,442 I/Os): 413 took ≤10, 20 took ≤1k, 956 took ≤10k, 172 took ≤100k, 878 took 100k–1M, and 3 timed out. The slowest successful completion took 975,256 iterations, against a bound of 1,048,576. 394 requests started with an unconsumed completion already in the used ring.
  2. Time per completion (logs/20260926-070007, 1,808 I/Os, measured on the raw hardware counter): every completion finished within 100 ms, the slowest at 15.1 ms; 6 timed out. 1M poll iterations take only about 14 ms of guest time on riscv64 TCG.

Root cause.

  • The completions are slow, not lost. The iteration bound (about 14 ms on riscv64) sat inside the device's latency tail (up to 15 ms), so the slow tail timed out.
  • Each timeout started a misattribution chain. A timed-out request stayed in the device, but the driver forgot it. The next request found that request's used entry, took it as its own (the wait loop was "used index changed"), and returned before its own I/O finished. Every later request did the same (394 in one boot).
    • Reads could return data before the device wrote it.
    • The next request rewrote descriptors and buffers the device was still reading, which can hand QEMU a malformed chain it drops.
  • This matches the FABRIC-3.md 2026-09-09 report of "a request that never gets a used-ring completion", and why disabling interrupts "made it strictly worse" (interrupts off slows the device-side progress, widening the tail).
  • amd64 and aarch64 were never affected in practice: none of their completions reached the bound.

Evidence named before the fix: 0 timeouts on all three ISAs over O-A and the standard sequence; 0 requests starting with an unaccounted completion pending; the completion-latency distribution recorded per ISA.

Change (virtio_blk.c).

  • A time bound, VBLK_TIMEOUT_MS (2 s), measured on the raw hardware counter (heartbeat_read_counter() / timer_tsc_hz(), readable with interrupts masked). That is over a hundred times the slowest completion measured. A large iteration bound remains as a fallback only if the counter rate is unknown.
  • Exact accounting.
    • Each request waits for exactly one more used entry (last_used_idx + 1), not "any change".
    • A request that times out stays outstanding (late). The next request first drains its completion, discards it and counts it, before reusing anything.
    • If the late completion still hasn't come by the deadline, the new request is refused without touching the queue, logged and counted.
  • New counters: virtio_blk_timeouts(), virtio_blk_late_consumed(), virtio_blk_refused_busy().

Evidence, after the fix (instrumented builds, O-A plus the standard sequence, 1,912 I/Os on each ISA):

ISA log timeouts / late drained / refused unaccounted at entry slowest ≤100 µs / ≤1 ms / ≤10 ms / ≤1 s
riscv64 logs/20260926-071524 0 / 0 / 0 0 7.7 ms 810 / 120 / 982 / 0
amd64 logs/20260926-072824 0 / 0 / 0 0 189 ms (one outlier) 817 / 189 / 905 / 1
aarch64 logs/20260926-074136 0 / 0 / 0 0 7.0 ms 774 / 201 / 937 / 0

riscv64's slowest completion halved (15.1 → 7.7 ms), consistent with requests no longer overlapping in the device.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log result
amd64 logs/20260926-075501 hash 0x99b01134c3d3049c, G A=40 B=40, all ok 3, 0 B idle, 0 / 0
aarch64 logs/20260926-080740 same
riscv64 logs/20260926-082042 same; 0 timed out lines in the log

K-6 closes §8 defect 3.

4.8 K-8 — the ~3.4 KB/VM residual: found and freed (2026-09-26)

Investigation (temporary probe: a kmalloc heap walk and two Mama words, K8-SNAP (record every used block) and K8-DIFF (list the used blocks that are new since the snapshot, with their first 32 bytes)). amd64. Each run did two warm-up worker cycles (WORKER-BIRTH, SEND, KILL), then snapshot, one cycle, diff, twice.

  • Before the fix (logs/20260926-083533): each cycle left 84 new blocks, 3,400 bytes, identical in both cycles. That is exactly K-4's 95,200 bytes per 28-VM round.
  • What they were: the dead VM's vocabulary search indexes. These are the first-character bucket arrays for CONTEXT and FORTH (vm->vocab.ctx_fc / forth_fc), built lazily by fc_rebuild() and never freed by vm_cleanup().
  • Reading vm_cleanup() turned up one more: vm->last_inference_outputs was never freed either.

Change.

  • New vocabulary_free_indexes(vm) (vocabulary_words.c) frees both indexes' buckets and clears the cached heads.
  • Both vm_cleanup()s (kernel vm_core.c, hosted vm_bootstrap.c) call it after the dictionary index, then free last_inference_outputs.

Evidence, after the fix (same probe, logs/20260926-084642): each cycle leaves 2 blocks, 232 bytes, identical in both cycles. Both are keyed by the dead VM's id:

Block Size What Status
registry node (vm_node_t), state DEAD 168 B the VM's registry entry kept by design: nodes are never freed after boot, so entry pointers stay valid
vm_physics_node_t 64 B the VM's fleet-physics node the MC-0 finding "vm_physics nodes are never removed"; open, awaiting a ruling
  • The unexplained residual is gone: 3,400 → 232 bytes per VM, all of it identified.
  • K-1's "within one allocation" criterion, as written, cannot be met while those two nodes stay after death. The registry node is a design decision; the physics node depends on the ruling above.

Standard sequence (clean build, with the probe reverted; only the four fix files differ from K-6).

ISA log result
amd64 logs/20260926-085034 hash 0x99b01134c3d3049c, G A=40 B=40, HHH, all ok 3, ledger unchanged, 0 B idle, 0 / 0
aarch64 logs/20260926-090043 same
riscv64 logs/20260926-091122 same; 0 timed out lines

K-8 closes the unexplained part of K-1's heap criterion.

4.9 Three more stages, from the rulings on K-8's report (plan, 2026-09-26)

Rulings (Captain Bob, 2026-09-26):

  1. A dead VM's vm_physics node is removed at KILL and reap.
  2. The stale Hermes-VM references are removed. The DoE CSV's hermes_heat_q48 column becomes Hestia's heat (hestia_heat_q48): same position, same column count.
  3. The lock order gets a runtime rank check.

Found while planning K-9: vm_physics_retire() (the dying VM's heat goes to Hera) is called only by the FORTH KILL word. capsule_vm_kill() called from WIREBIND/EJECT/UNCLEAN, and capsule_vm_force_reap(), never retire. A VM that dies those ways keeps a live physics node still holding heat. That heat still counts in the fleet sum, so K stays conserved, but a dead VM holds it. K-9 moves the retirement into the two kernel death paths, so every death retires and then removes.

Stage Change Evidence (named in advance)
K-9 New vm_physics_remove(vm_id): retire (heat to the root), unlink, free. Called from capsule_vm_kill() and capsule_vm_force_reap(), before the registry entry goes DEAD (the root walk reads the registry). The KILL word's own retire call goes; Hera is refused, as today. Standard sequence unchanged; fleet conserved. K-8's heap probe: each worker cycle leaves 1 block, 168 bytes (the registry node only). VM-PHYSICS lists only live VMs after a churn.
K-10 CONNECT-HERMES (births the removed hermes:init.4th, so it always fails) is removed with both registrations and its docs/lithosananke/DICTIONARY.md row. ARTEMIS-ANNOUNCE-READY (capsules/artemis/init.4th block 4129, no callers) is removed. doe_log.c column 17 reads Hestia's heat, named hestia_heat_q48, and the column comments are corrected. Standard sequence passes. The M7.1a hash and word count, Hera's and Artemis's dict hashes change once and are recorded as the new baseline; Hestia's is unchanged. The CSV header reads hestia_heat_q48, and its rows carry nonzero Hestia heat.
K-11 Every lock gets a rank, in the header's order: hermes < switch < registry < VM dict_lock < VM tuning_lock < stadium < slot < doe < log < block < console < heap < vmm < pmm. Each core counts the locks it holds per rank. Taking a lock whose rank is not above every rank already held counts an order violation, recording both locks and the call site. A same-rank pair, such as two Stadium slots or two VMs' dict_locks, is a violation too. It is logged by sk_lock_report() like the other lock failures. A probe that takes heap and then registry logs exactly one LOCK: order violation, naming both. With the probe reverted, O-A, the standard sequence and a 20-VM churn log 0 order violations on all three ISAs, or else each one is reported as a finding (not fixed unrequested).

Found, not changed: ARTEMIS-READY-ACK (block 4129) was only ever called by the Hermes VM and has no caller now. experiments/std79-doe/{analysis,report}-20260912/{combine.py,analysis.R} read hermes_heat_q48 from their own historical data; they are left as they are.

4.10 K-9 — a dead VM's physics node is removed (2026-09-26)

Change.

  • New vm_physics_remove(vm_id) (capsule_vm_physics.c): retire (heat to the root, as before), then unlink and free the node. Hera is refused.
  • capsule_vm_kill() and capsule_vm_force_reap() call it after releasing the quota slot and before the entry goes DEAD. That covers every death: the KILL word, WIREBIND/EJECT/UNCLEAN, and the reap.
  • The KILL word's own vm_physics_retire() call is gone; it kept its console reset.

Evidence (K-8's heap probe, reapplied temporarily, amd64, logs/20260926-092847).

  • Each worker cycle leaves 1 block, 168 bytes, the registry node, in both measured cycles (K-8: 2 blocks, 232 bytes).
  • After 10 worker kills, VM-PHYSICS-STATUS lists Hera, Hestia and Artemis only. fleet_heat_sum=65536, CONSERVED; Hera holds all of it.

Standard sequence (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).

ISA log result
amd64 logs/20260926-093320 hash 0x99b01134c3d3049c, G A=40 B=40, HHH, all ok 3, ledger unchanged, 0 B idle, 0 / 0
aarch64 logs/20260926-094315 same
riscv64 logs/20260926-095353 same

K-9 closes the MC-0 finding "vm_physics nodes are never removed". Each dead VM now leaves only its registry node, 168 bytes, by design.

4.11 K-10 — the stale Hermes-VM references removed (2026-09-26)

Change.

  • CONNECT-HERMES: removed from mama_forth_words.c along with both of its registrations (FORTH and MAMA vocabularies). It births hermes:init.4th, which no longer exists, so every call failed. Its docs/lithosananke/DICTIONARY.md row and a caller list in capsule_runcap.c were updated too.
  • ARTEMIS-ANNOUNCE-READY: removed from capsules/artemis/init.4th block 4129. It had no callers. mkcapsule --lint passes on all 37 files.
  • doe_log.c column 17 is now hestia_heat_q48 (Hestia's heat, looked up by name like columns 16 and 18), in the same position with the same column count. The comments on the slot index (22) and slot-1 readiness (24) no longer name Hermes.

The new baseline, identical on all three ISAs:

before (K-9) after
M7.1a word_count / latest_id 546 / 715 544 / 713 (one word in each of two vocabularies)
M7.1a hash 0x99b01134c3d3049c 0x927513aabd543433
Hera dict_hash 0x00340a8b2c842eb7 0x4953a2e2d11fff34
Artemis dict_hash 0x66886d174d5f3b5a 0xc5a8cff4b805b6a0
Hestia dict_hash 0x78bf28439e4a643f unchanged

Only these two FABRIC docs record the old hashes; nothing in code or scripts checks them.

CSV spot check (aarch64, HB-ON around one worker job plus one Hestia message, then HB-OFF; not a DoE run). The extracted CSV (experiments/bare_metal/runs/doe-aarch64-20260926-101553.csv, 48 rows) has the header hera_heat_q48,hestia_heat_q48,artemis_heat_q48, and K = 65536 in every row.

  • Not as named in advance: Hestia's column reads 0 in every row, not a nonzero value. That is her actual heat. VM-PHYSICS-STATUS on amd64 (logs/20260926-100501) also shows Hestia heat_q48=0 while the two workers hold 5240 each: heat moves only on VM touches and beat charges, and message delivery to Hestia is neither. The column is live, but in these runs it has nothing to show.

Standard sequence.

ISA log result
amd64 logs/20260926-100501 new hashes above, G A=40 B=40, HHH, all ok 3, ledger unchanged, 0 B idle, 0 / 0
aarch64 logs/20260926-101553 same hashes and results. Ledger pulled 266240 / returned 263455 (+4096 each): the two extra SENDs of the spot check, each a message cell pulled and returned. Held and consumed unchanged; conserved
riscv64 logs/20260926-102708 same as amd64

K-10 closes the three stale Hermes-VM references found at K-3. ARTEMIS-READY-ACK (no caller) and the switch_hermes_readiness CSV column name (always 0, kept with its two siblings pending FABRIC-3.8.md Stage I) remain, as noted in §4.9.

4.12 K-11 — the lock order checked at run time (2026-09-26)

Change.

  • Ranks (spinlock.h, sk_lock_rank_t): hermes < switch < registry < VM dict_lock < VM tuning_lock < stadium < Stadium slot < doe < log < block < console < heap < vmm < pmm.
    • sk_spinlock_t carries its rank (SK_SPINLOCK_INIT(rank)).
    • The raw-lockword calls take one: sk_lockword_acquire/release(word, rank) for Stadium slots and sf_mutex.
    • The kernel sf_mutex_t gains a rank. sf_mutex_init() sets the dict rank, and the kernel vm_init() re-ranks tuning_lock.
  • The check (spinlock.c).
    • Each core counts the locks it holds per rank (sk_percpu_t.held_rank[]).
    • Taking a lock whose rank is not above the highest held rank counts an order violation, recording both ranks and the call site. A same-rank pair counts too.
    • A recursion is still counted as a recursion only.
    • New sk_lock_order_violations().
  • The report (sk_lock_report(), at the idle point) prints the full line to the console. The log line keeps only the count, because log_message() cuts at LOG_LINE_MAX (64 bytes): the first probe run showed ...while holding he (logs/20260926-103947). This follows parity.c's INCOMPLETE report.

Evidence.

  • Probe (temporary K11-PROBE word: heap, then registry; amd64, logs/20260926-104239). Nothing is reported through boot. After the probe, exactly one line: LOCK: order violation x1 (first: took registry while holding heap, from 0x000000003d71a62c).
  • Clean (probe reverted; the tree is byte-identical to the saved stage diff). O-A, then a 28-VM churn (32 births, 28 kills), then the standard sequence:
ISA log order violations / any LOCK: line result
amd64 logs/20260926-104446 0 / none K-10 hashes, G A=40 B=40, HHH, all ok 3, 0 B idle, 0 / 0
aarch64 logs/20260926-110203 0 / none same
riscv64 logs/20260926-112011 0 / none same; 0 timed out

The ledger reads pulled 319488 / returned 316703 on all three. That is the K-3 baseline plus 28 × 2048: the churn's 28 SENDs, each a message cell pulled and returned. Held and consumed are unchanged, and conservation holds.

Found, not fixed: MC-0b's three older lock reports (recursion, bad release, held at switch) go through log_message() only, so they are cut at 64 bytes and lose the lock name and site. None fired in any run here.

K-11 closes the "lock order checked by inspection only" finding. With K-9..K-11, every ruling from K-8's report is carried out.

5. MC-1 — bringing up the extra cores (plan, 2026-09-26)

Rulings (Captain Bob, 2026-09-26):

  • Multi-core (MC-1..MC-3) comes before the O-D DoE (confirming MC-R6).
  • MC-1 is split per ISA: MC-1a (the shared core table, plus amd64), MC-1b (aarch64), MC-1c (riscv64). Each gets its own commit and 3-ISA acceptance. QEMU's -smp 2 is turned on for an ISA only when its bring-up lands.

Common to every ISA (built in MC-1a):

  • Core ids are dense and logical. Core 0 is the boot core, whatever its APIC id, MPIDR or hartid. The hardware id is recorded in the block beside the logical one and never reaches the lock owner word (core_id + 1).
  • The core table (hal/smp.c) is sized from enumeration and heap-allocated. It has no MAX_CPUS. Each extra core gets a heap-allocated sk_percpu_t and a 16 KiB stack.
  • The block is handed to the core in its entry path, and the per-CPU register is set before any C code. sk_percpu_ready is already 1, so it cannot guard a secondary core.
  • In MC-1 a secondary core does no console, log or heap work. It stores a readback of its per-CPU register into its block, marks itself online with a release store, then parks: interrupts masked, a hlt/wfi loop that counts every wake.
  • The boot core brings the cores up one at a time. It waits under a time bound (1 s, on the raw counter, as K-6 does), then prints the core's line, or prints that it did not come online and carries on.
  • Park wakes are reported at the idle point only when they change, like the lock report: a core that never wakes prints nothing.
  • No new FORTH word. The dict_hash set stays at the K-10 baseline.
  • Bring-up runs in kernel_main_deep() right after the heartbeat init, with interrupts still off on the boot core.

MC-1a, amd64:

  • Enumeration: the MADT's enabled Local APIC entries (type 0, and type 9 x2APIC). The boot core is skipped by its own LAPIC ID.
  • Trampoline: one position-independent 4 KiB blob, copied into a page chosen from the UEFI memory map (conventional memory below 1 MiB, never page 0). The SIPI vector is that page's number.
    • Data block: patched at run time with the trampoline's own GDT (flat 32-bit code and data, 64-bit code) and its far pointers.
    • Control state copied from the boot core: CR0, CR3, CR4, EFER (for NXE), IA32_PAT, and XCR0 when CR4.OSXSAVE is set.
    • Also patched in: the kernel's GDTR and IDTR, the core's block, its stack, and the C entry point.
    • Path: real mode → protected mode → long mode. It loads the kernel GDT and IDT, reloads the segment registers, then writes IA32_GS_BASE (in that order, because a GS reload clears the base) and calls the C entry.
    • Limit, recorded for SER5: CR3 must lie below 4 GiB, because it is loaded in 32-bit mode.
  • Start sequence: INIT, wait 10 ms, SIPI, wait 200 µs, SIPI, polling the ICR delivery-status bit (xAPIC MMIO, as the timer already uses).
  • QEMU: -smp 2 on the make qemu amd64 invocation only. qemu-esp and qemu-gdb are unchanged.

MC-1a evidence (named in advance):

  • amd64, -smp 2: the boot prints 2 cores found in the MADT. Core 1's line gives its logical id 1, its APIC id, its block address, and its GS readback, which equals its own block and differs from core 0's. It came online within the bound, with the time taken.
  • Parked, not spinning: no park-wake line in the whole run. On the host, core 1's QEMU vCPU thread sits near 0% CPU (ps -L), if TCG gives it its own thread.
  • aarch64 and riscv64: they report 1 core and are otherwise unchanged.
  • Standard sequence on all three: the K-10 dict_hash set, G A=40 B=40 byte-identical (still one worker core), all ok 3, 0 B idle growth.

5.1 MC-1a — the core table, and amd64's second core online and parked (2026-09-26)

Change.

  • include/starkernel/smp.h, hal/smp.c: the core table. It holds dense logical ids, is heap-sized from enumeration, and gives each extra core a heap block and a 16 KiB stack.
    • sk_smp_init() brings the cores up one at a time, each waited for under a 1 s bound on the raw counter.
    • sk_smp_secondary_main() records the per-CPU register readback, marks the core online with a release store, and parks.
    • sk_smp_report() runs at the idle point, after the lock report, and logs park wakes only when they change.
    • aarch64 and riscv64 report their boot core only, through stubs in hal/smp.c that MC-1b/c replace. Their "hw id 0x0" is that stub, not a real MPIDR or hartid.
  • sk_percpu_t gains hw_id, reg_readback, online and park_wakes.
  • arch/amd64/smp.c:
    • MADT enumeration: enabled type-0 entries, then type-9 entries with ids up to 255, de-duplicated.
    • The trampoline page: the lowest conventional page in [4 KiB, 1 MiB) from the UEFI memory map.
    • The control-state copy: CR0/CR3/CR4, CR4 without PCIDE for the 32-bit step, EFER without LMA, PAT, and XCR0 when OSXSAVE is set. The kernel GDTR and IDTR are copied too.
    • INIT, 10 ms, STARTUP, 200 µs, then a second STARTUP only if the core isn't online yet.
  • arch/amd64/ap_trampoline.S: the relocation-free blob (§5 path). apic_send_ipi() in apic.c writes ICR high then low, and polls delivery status.
  • Wiring: kernel_main_deep() calls sk_smp_init() after the heartbeat init. The Makefile builds the two amd64 files into the loader, and -smp 2 is on the make qemu amd64 line only.

Two failed debug boots, found and fixed; their logs are deleted as flawed:

  1. Core 1 triple-faulted and reset the machine. QEMU runs with -no-reboot, so it exited just after the "2 core(s) found" line.
  2. A rerun with QEMU_EXTRA="-d int,cpu_reset" showed core 1 had reached 64-bit mode and loaded the kernel GDT. It then page-faulted on push 0x08, the far return's first push, with RSP still 0 (CR2 0xfffffffffffffff8).
    • The fix loads the stack before the far return.
    • EFER.LMA is also masked out of the copied value, since only the CPU may set it.
  3. The traced rerun after the fix (logs/20260926-120808, kept): 0 exceptions on either core over the boot.

Evidence (clean builds, no trace; the tracked diff is byte-identical to the saved stage diff):

ISA log SMP lines standard sequence
amd64 (-smp 2) logs/20260926-121008 SMP: 2 core(s) found; core 0 is the boot core, hw id 0x0; SMP: core 1 online: hw id 0x1, block 0x0000000001780130, per-CPU register 0x0000000001780130 (its own block), 59 us, parked; SMP: 2 core(s) online K-10 hashes, G A=40 B=40 byte-identical, HHH, all ok 3, ledger unchanged, 0 B idle, 0 / 0; 0 park-wake lines, 0 LOCK: lines
aarch64 logs/20260926-122030 1 core found, 1 online same
riscv64 logs/20260926-123103 1 core found, 1 online same
  • The 59 µs is measured from the end of arch_smp_start(), which already includes the 10 ms INIT wait and the 200 µs STARTUP wait. The core arrived during the second STARTUP's delivery, not 59 µs after INIT.
  • Not as named in advance: the host-side check (core 1's vCPU thread near 0% in ps -L) could not be made. QEMU names its threads per vCPU only with -name debug-threads=on, which is not on the command line. The park-wake counter (0 wakes) is the evidence of record. Adding -name debug-threads=on is a candidate for MC-1b's run line.

MC-1a is complete. amd64 now boots two cores, and the second is parked with interrupts masked. Next: MC-1b (aarch64, PSCI CPU_ON).

5.2 MC-1b — aarch64's second core online and parked (2026-09-26)

Go: Captain Bob, 2026-09-26, "keep going".

Change.

  • Enumeration (arch/aarch64/smp.c): the devicetree's /cpus children with device_type = "cpu" (their reg, per #address-cells) when the firmware hands one over, as on the RPi5. Otherwise, the ACPI MADT's enabled GICC entries (type 0x0B, MPIDR at +68).
    • QEMU's AAVMF hands over ACPI and no DTB (PSCI: no DTB -- using HVC), so on QEMU the cores come from the MADT.
    • The hardware id is the MPIDR affinity (Aff3..Aff0). The boot core's is read from MPIDR_EL1.
  • The MADT lookup is now shared (hal/acpi.c, acpi_find_table()). amd64's own copy in arch/amd64/smp.c is removed and uses it.
  • Start: PSCI CPU_ON (SMC64 0xC4000003) through the conduit arch.c already discovers (aarch64_psci_cpu_on(); HVC on QEMU). Entry is ap_entry.S, with x0 = the core's boot block, heap-allocated and kept.
    • The block, the core's per-core block, and the first 256 bytes of the entry code are cleaned to the point of coherency (dc cvac, dsb sy) before the call, because the core starts with its MMU and caches off. TCG does not model this; the RPi5 will.
    • A non-zero PSCI return is printed and the start refused.
  • ap_entry.S:
    • It masks DAIF and copies the boot core's MAIR, TCR, TTBR0/1, VBAR and CPACR (FP/SIMD on). Then it does tlbi vmalle1 and ic iallu, and writes SCTLR, which turns the MMU and caches on at the same identity-mapped address.
    • It sets TPIDR_EL1 to its block, loads its stack, records CurrentEL in its boot block, and calls sk_smp_secondary_main().
    • It references no symbol: every address comes from the block, because the file is also assembled for the PE/COFF loader.
  • Park: msr daifset, #0xf and wfi, each wake counted.
  • arch_smp_arrival() is a per-ISA phrase in the online line: "EL1" on aarch64 (from the recorded CurrentEL), "long mode" on amd64.
  • QEMU: -smp 2 -name debug-threads=on on the make qemu aarch64 line only. The thread names make the host-side check possible.

Evidence (clean builds; the source diff is identical to the saved stage diff):

ISA log SMP lines standard sequence
aarch64 (-smp 2) logs/20260926-124518 SMP: 2 core(s) found; SMP: core 1 online: hw id 0x1, block 0x0000000048000130, per-CPU register 0x0000000048000130 (its own block), EL1, 226 us, parked; SMP: 2 core(s) online K-10 hashes, G A=40 B=40 byte-identical, HHH, all ok 3, ledger unchanged, 0 B idle, 0 / 0; 0 park-wake lines, 0 LOCK: lines
amd64 (-smp 2) logs/20260926-125631 as MC-1a, now with "long mode": core 1 online: hw id 0x1, ... (its own block), long mode, 500 us, parked same; 0 park wakes, 0 LOCK:
riscv64 logs/20260926-130637 1 core found, 1 online same

Host side, aarch64 (ps -L on the QEMU process after O-A, then 20 s later):

thread %CPU CPU time 20 s later
CPU 0/TCG 63.5 5:10 63.8, 5:24
CPU 1/TCG 0.0 0:00 0.0, 0:00

Core 1 is halted, not spinning.

Found, not changed: core 0's vCPU burns about 64% of a host CPU while the system is idle. The boot core's idle loop (the console poll and idle point) polls rather than halting. This predates MC-1. MC-3's criterion "idle worker cores halted, not spinning" applies to the new cores; whether core 0 should halt when idle too is a separate question for MC-3's design.

MC-1b is complete. Next: MC-1c (riscv64, SBI HSM hart_start).