LithosAnanake/FABRIC-3.10.0.md

FABRIC-3.10.0.md — Multi-core: observers pinned, workers claimed by the physics

Status: DESIGN, for Captain Bob's review. Nothing here authorizes an edit. Proposed to start after O-D and the merge to master (ruling MC-R6). An execution plan (FABRIC-3.10.8.md) follows once the rulings are made. Part II (§8) records the OS's other open gaps as of this writing.

Provenance. Opened 2026-09-25, while O-D (the DoE relaunch) was running on amd64. Captain Bob, 2026-09-25:

• "I'm interested im the idea of pinning hera, hesta and artimes to cpu cores whre possibl."
• On placing workers on cores at birth: "Not sure, static allocation of worker vms seems anti starshipos given there isn't a fied number of vm's. investigate how to pin observers and allow multithreading across cores forload ballancing."

Static placement of workers is therefore ruled out. A worker is never assigned a core. It runs wherever the physics next chooses it.

---

1. The idea

• Observers are pinned. The Tripod (Hera, Hestia, Artemis) are a fixed, known set, already outside the compudynamic comparison (FABRIC-3.9.0.md §2). They get affinity to a system core, core 0, where there is more than one core, and share the one core exactly as today where there isn't.
• Workers are placed nowhere. Every other core is a worker core. At each clean boundary, a worker core asks the existing compudynamic decision for the coolest participant with work that no other core is running, and claims it atomically. A worker runs wherever it is next chosen. It migrates for free, always at a clean boundary, and the load balances through the heat itself: a hot worker is not the coolest, so free cores take the cooler ones.
• No scheduler, no fixed count. Each core runs the same physical question the single core runs today. Nothing counts VMs or cores in advance, so the design holds for any number of VMs on any number of cores.

A worker VM stays single-threaded: one interpreter, one data stack, one native stack, one core at a time. Parallelism comes from many workers running at once. Splitting one VM across cores would need a multi-threaded interpreter sharing one dictionary and one set of stacks, which is a different machine. A job that wants more parallelism is split into more workers, as the DoE already does.

---

2. What exists today (traced 2026-09-25)

Area	Today
QEMU	No -smp on any ISA (Makefile.starkernel): every run is one CPU.
Secondary core start	None on any ISA: no INIT-SIPI (amd64), no PSCI CPU_ON (aarch64), no SBI HSM hart_start (riscv64). arch/riscv64/interrupts.c says "the current single-hart kernel".
Groundwork to reuse	amd64 already parses the ACPI MADT (arch/amd64/ioapic.c, for the I/O APIC); its LAPIC entries (type 0) list the cores. aarch64 already discovers the PSCI conduit from the DTB (aarch64_psci_conduit_init(), for reset); CPU_ON goes through the same conduit. All three have halt primitives (hlt/wfi, arch_halt()).
Locks	sf_mutex_* are deliberate no-ops in the kernel (vm/host/shim.c: "single-threaded kernel"). The shared VM code already takes them at the right places (dict_lock, tuning_lock), so they can become real spinlocks without touching callers.
Atomics	Two helpers in vm/vm_runtime.c (__atomic_load_n/__atomic_store_n).
Global mutable state	Written as "single-writer mainline" throughout: kernel-Hermes (arena, queues, channels, ledger), Stadium (reservoirs, cells, quotas), kmalloc's free list, the VM registry, the switch-signal table, console output, the log region, the block subsystem.
Per-core state	None. g_switch_current_vm (vm/switch.c) is one global "who is running". The heartbeat is one global beat, driven by the boot CPU's timer ISR.
Stacks	Each VM has its own native stack; a parked VM holds only its trampoline context (Stage D sk_vm_parked_idle()). Hera runs on the boot stack.

---

3. Design

3.1 Observers on the system core

• Affinity is a role fact, set at birth, from the same is_fleet_foundation fact that sets the observer flag (O-A). Nothing a VM executes can change it.
• Core 0 runs the Tripod under exactly today's rules: the O-A tiers, the O-B idle-point drain, the console, USB polling (sk_repl_idle()), and Artemis's storage.
• Worker cores never claim an observer. Hera never runs on a worker core.
• One core means today's behaviour: core 0 is also the only worker core, and every participant runs there as now.

3.2 Workers: claimed through the shared decision

• The switch-signal slot gains running_on (core index, or none).
• At every clean boundary (the trampoline's between-messages point, the in-word checkpoint, a worker core's idle loop), a worker core runs sk_vm_switch_decide() over participants with running_on = none or = this core. It then claims the winner with one atomic compare-and-swap on running_on. If it loses the race, it decides again.
• The claim is released when the worker parks (IDLE frame) or switches out.
• Tie-break: among equally cool candidates, prefer the one that last ran on this core (cache affinity), then the existing incumbent rule, then the oldest wait stamp.
• Migration is implicit. A parked worker resumes on whichever core claims it next. Its stack holds nothing tied to a core; per-core values live behind the per-core pointer (§3.5), never on a VM's stack.
• "Nobody has work" changes on worker cores. Today a trampoline with no work switches to Hera. On a worker core it switches to that core's own idle context, a small per-core stack that halts (§3.3). Hera remains the idle root on core 0 only.

3.3 Idle and wake

• An idle worker core halts (hlt/wfi). Nothing polls.
• Wake by inter-processor interrupt. When sk_hermes_send_one() admits mail, or an event (M-B), for a participant that no core is running, it wakes one halted worker core. That core's decision picks the work up.
• Core 0 wakes on its own inputs as now: its timer, console input, and USB polling at its idle point.

3.4 Physics across cores

• Each core has its own timer (the LAPIC timer on amd64, the generic timer on aarch64, the SBI timer on riscv64) and charges beats to the VM it is running. Beats stay execution-paced per VM, as today.
• Conservation stays exact. Stadium reservoirs, consumed and kernel-Hermes's held/pulled/returned/consumed become atomic, or per-core accumulators merged at audit time. The integer audits (stadium_conserved(), sk_hermes_audit()) must hold exactly, as today.
• Observers stay uncharged (O-2).

3.5 Per-core state

• One per-core block per core, reached through each ISA's per-CPU register: GS base on amd64, TPIDR_EL1 on aarch64, tp on riscv64. The kernel owns tp; there is no user mode.
• It holds: the current VM (replacing the global g_switch_current_vm), the core's idle context, the running frame stack, local beat counters, and the "decision due" flag.

3.6 What has to be made safe

These are the shared structures and what each needs (FABRIC-3.10.8.md will inventory them line by line):

Structure	Needs
sf_mutex_*	Real spinlocks (with interrupts masked while held on the owning core).
kmalloc	A heap lock.
kernel-Hermes (arena, queues, channels, ledger, events)	A lock (one to start; per-queue later if contention shows).
Stadium (reservoirs, cells, quotas, word rows)	Not one lock — the per-dispatch path must not contend (see below).
VM registry, birth, kill	A registry lock. KILL of a VM whose running_on is another core is refused, the multi-core form of today's "it is the VM running" refusal.
Switch-signal table	Compare-and-swap for claim and release; a lock for register and unregister.
Console output	A line lock, so lines from different cores never interleave mid-line.
Log region, block subsystem	A lock each. Artemis's storage stays on core 0, but workers can use BLOCK words.

The Stadium hot path decides whether multi-core speeds anything up (added 2026-09-25, after Captain Bob asked whether this should give a significant performance boost).

• stadium_word_dispatch() runs on every word dispatch of every VM (vm_core.c), for word heat: resolve the word's cell, cool it, pull admission heat from the VM's reservoir, and admit or evict a cell.
• Behind a single Stadium lock, every worker core would contend on every word, and the speedup would collapse towards 1×. kmalloc and kernel-Hermes are touched far less often (on definitions and messages), so one lock each is fine there.
• Requirement: Stadium's per-dispatch state is partitioned by the VM's quota slot, which is already how the word rows are keyed (word_slots[slot][word_id], S-1b).
  • A VM runs on one core at a time, so its own row and cells need no lock while it runs.
  • Its reservoir takes atomic arithmetic.
  • Only operations that cross slots (admission evicting another VM's cell, quota grants, conservation audits) take the Stadium lock.
• This is part of MC-0, and MC-0's evidence includes it: a per-dispatch path that takes no shared lock.

3.7 Bringing up the cores, per ISA

• amd64: enumerate cores from the MADT's LAPIC entries. Start each with INIT-SIPI-SIPI through the LAPIC ICR, into a trampoline copied below 1 MiB (real mode → long mode). The trampoline must be position-independent: the kernel is relocated by a whole number of pages plus 0x440, differently on every boot (found at O-C).
• aarch64: enumerate cores from the DTB's /cpus. Start each with PSCI CPU_ON through the already-discovered conduit (HVC on QEMU virt). Each core initialises its GIC CPU interface or redistributor and its generic timer.
• riscv64: enumerate harts from the DTB. Start each with SBI HSM hart_start. Each hart arms its own SBI timer. IPIs go through the SBI IPI extension, or ACLINT where present.

---

4. Consequences

• Execution order stops being deterministic once there is more than one worker core.
  • Byte-identical evidence such as the Stage G interleave string gives way to count evidence (A = 40, B = 40).
  • The dict_hash set and the integer conservation audits stay exact.
• Cores become a DoE factor. A core count is fixed per boot (-smp N), so it is a factor across boots. Measuring what compudynamics does with 1 core versus N is probably the most interesting result this makes possible.
• QEMU: each virtual CPU is a host thread. Host-side contention shows up in the timing the DoE measures, so the one-QEMU-at-a-time rule matters more, not less.
• Hardware: SER5 (8 cores) and the Raspberry Pi 5 (4 cores) are the real targets this opens.

---

5. Staging (proposed)

Each step keeps the system shippable and gets its own 3-ISA acceptance.

Stage	Change	Evidence (to be named in full in 3.10.8)
MC-0	Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind sf_mutex; the §3.6 locking applied, including Stadium's per-dispatch path partitioned by quota slot.	Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch, and no shared lock taken on the word-dispatch path.
MC-1	Bring up the extra cores and park them halted; QEMU -smp 2.	The boot reports N cores online; the standard sequence is unchanged, with the extra cores idle.
MC-2	Observers get system-core affinity (core 0).	With one worker core, behaviour identical to today; observers are never claimed off core 0.
MC-3	Worker cores claim through the shared decision, with per-core idle contexts and wake by interrupt.	Two workers genuinely overlapping in time on two cores. A probe proves the claim never lets two cores run one worker. Conservation holds throughout. G by counts. Idle cores halted, not spinning (0 idle log growth).
MC-4	DoE with cores as a factor.	A per-ISA campaign at each ruled core count.

---

6. Known limits and open questions

• Worker cores and devices: only core 0 services USB. A worker core never touches xHCI.
• Locking granularity: one lock per subsystem to start (MC-R2) may contend under many workers. It is measured before it is refined. Stadium's per-dispatch path is the exception, partitioned from the start (§3.6).
• Expected gain. Close to linear throughput for many independent compute-bound workers on real hardware, once the dispatch path is partitioned. None for a single worker (a VM is single-threaded). Little for anything bound by serial output, the current DoE's limit: every core's heartbeat rows funnel through one console. MC-3/MC-4 measure it: throughput for a fixed batch of workers at 1, 2 and 4 cores.
• Interrupt routing: device IRQs stay on core 0 (the I/O APIC, GIC SPIs and PLIC contexts route there). Only inter-processor wakes reach worker cores.

---

7. Rulings needed from Captain Bob

1. MC-R1: the system core. All three observers together on core 0 (proposed), or a core each where cores allow.
2. MC-R2: lock granularity to start. One lock per subsystem (proposed), or a single kernel lock first, or fine-grained from the start.
3. MC-R3: tie-break affinity. Prefer the worker's last core among equally cool candidates (proposed), or no affinity.
4. MC-R4: evidence under concurrency. Count-based checks replace byte-identical interleave strings once there is more than one worker core (proposed).
5. MC-R5: DoE core counts. Which counts per ISA (proposed: 1, 2, 4).
6. MC-R6: timing. Start after O-D and the merge (proposed).

---

8. Part II — the OS's other open gaps (2026-09-25)

Ruled out by Captain Bob, 2026-09-25:

• A filesystem: never. "Filesystem? NEVER! anti POSIX!" Storage is FORTH blocks, Artemis's disk, content-addressed capsules and identity thumbdrives. That is the model, not a gap.
• Networking: deferred. "Not iterested in networking yet." There is no virtio-net and no stack today; it is not scheduled.

Open, in proposed order:

#	Gap	State
1	O-D and the merge. The DoE relaunch on all three ISAs, then this branch to master.	amd64 launched 2026-09-25 11:29:29 (logs/20260925-112529/amd64/), about 2.5–3 days, bound by serial row output.
2	Multi-core (this document).	Designed; rulings needed (§7).
3	Interrupt-driven devices (O-4 option (b)). USB is polled at Hera's idle point.	Deferred at O-4; the system core (MC-2) eases it.
4	The rest of the fixed-size audit (FABRIC-3.9.0.md §8). S-1 and M-0 are done. Still fixed: WIREBIND identities attached at once (13), block relocations (500), executable regions (8), the run log (1024), the kernel RAM disk, control-flow nesting (64), and the RAM-sized VM/queue/channel tables (class D′).	S-3 order ruled; S-2 (Stadium's cell pool) still needs a ruling. The WIREBIND cap matters most for multi-user.
5	Security policy. The ACL mechanism works, but ACL-RECHECK allows every STRICT word; INJECT's gate is in place with no policy behind it. ACL Phase 8 (PKI challenge-response) is marked open, though MINT/WIREBIND covers part of it.	Policy to write in ACL.4th.
6	Real hardware. SER5 and Raspberry Pi 5 tracks exist; nothing is verified on hardware. LITHOS_VERSION stays 2.0.0 until it is.	Unstarted this line.
7	Milestones M8 (REPL keyboard input) and M9 (block storage). Listed as planned; completion not independently verified.	To verify.
8	Failure counters readable from FORTH. §8.1 asks for this. S-1a/S-1b/M-0 expose C accessors only; EV-DROPPED is the one FORTH counter so far (its name unruled).	Ruling needed: adding words changes the dict_hash set.

Known defects, reported and not fixed (from FABRIC-3.9.8.md §2 unless noted):

1. COLD rewinds HERE to 1024 without rewinding latest: live words get compiled over (S-1c).
2. PAD returns VM_MEMORY_SIZE - 512, inside the top block I/O window slot (S-1c).
3. riscv64 intermittent virtio_blk: vblk_io timed out on disk writes under emulation (M-A).
4. Two boot self-tests silently SKIPPED: drain, and channel-open policy. Both still wait for the removed Hermes VM (M-0).
5. dictionary_heat_optimization.c: the heat snapshot is heats[1024], silently truncated (S-1b).
6. vm_create_word(): a failed transition-metrics allocation still defines the word (S-1b).
7. vm_cleanup() frees neither transition_heat nor context_window inside each word's metrics (S-1b).
8. transition_metrics_update_cache() scans the whole id space on every recorded transition (S-1b).
9. The kernel's FC-bucket grow never frees the old block after sf_realloc() (S-1b).
10. New, found at the xHCI fix: xhci_cmd_configure_endpoint() allocates a new Input Context on every configure, and never frees the old one. It is a per-attach leak, but harmless to the controller, which reads the Input Context only during the command.