LithosAnanake/FABRIC-3.10.0.md

# FABRIC-3.10.0.md — Multi-core: observers pinned, workers claimed by the physics

**Status: RULED 2026-09-25 (§7); execution in `FABRIC-3.10.8.md`.** Multi-core starts now, ahead
of O-D (MC-R6 as ruled). Part II (§8) records the OS's other open gaps as of this writing.

**Provenance.** Opened 2026-09-25, while O-D (the DoE relaunch) was running on amd64. Captain Bob,
2026-09-25:
- *"I'm interested im the idea of pinning hera, hesta and artimes to cpu cores whre possibl."*
- On placing workers on cores at birth: *"Not sure, static allocation of worker vms seems anti
  starshipos given there isn't a fied number of vm's. investigate how to pin observers and allow
  multithreading across cores forload ballancing."*

**Static placement of workers is therefore ruled out.** A worker is never assigned a core. It
runs wherever the physics next chooses it.

---

## 1. The idea

- **Observers are pinned.** The Tripod (Hera, Hestia, Artemis) are a fixed, known set, already
  outside the compudynamic comparison (`FABRIC-3.9.0.md` §2). They get affinity to a **system
  core**, core 0, where there is more than one core, and share the one core exactly as today where
  there isn't.
- **Workers are placed nowhere.** Every other core is a worker core. At each clean boundary, a
  worker core asks the existing compudynamic decision for the coolest participant with work that
  no other core is running, and claims it atomically. A worker runs wherever it is next chosen. It
  migrates for free, always at a clean boundary, and the load balances through the heat itself: a
  hot worker is not the coolest, so free cores take the cooler ones.
- **No scheduler, no fixed count.** Each core runs the same physical question the single core
  runs today. Nothing counts VMs or cores in advance, so the design holds for any number of VMs on
  any number of cores.

A worker VM stays single-threaded: one interpreter, one data stack, one native stack, one core
at a time. Parallelism comes from many workers running at once. Splitting one VM across cores
would need a multi-threaded interpreter sharing one dictionary and one set of stacks, which is a
different machine. A job that wants more parallelism is split into more workers, as the DoE
already does.

---

## 2. What exists today (traced 2026-09-25)

| Area | Today |
|---|---|
| QEMU | No `-smp` on any ISA (`Makefile.starkernel`): every run is one CPU. |
| Secondary core start | None on any ISA: no INIT-SIPI (amd64), no PSCI `CPU_ON` (aarch64), no SBI HSM `hart_start` (riscv64). `arch/riscv64/interrupts.c` says "the current single-hart kernel". |
| Groundwork to reuse | amd64 already parses the ACPI MADT (`arch/amd64/ioapic.c`, for the I/O APIC); its LAPIC entries (type 0) list the cores. aarch64 already discovers the PSCI conduit from the DTB (`aarch64_psci_conduit_init()`, for reset); `CPU_ON` goes through the same conduit. All three have halt primitives (`hlt`/`wfi`, `arch_halt()`). |
| Locks | `sf_mutex_*` are deliberate no-ops in the kernel (`vm/host/shim.c`: "single-threaded kernel"). The shared VM code already takes them at the right places (`dict_lock`, `tuning_lock`), so they can become real spinlocks without touching callers. |
| Atomics | Two helpers in `vm/vm_runtime.c` (`__atomic_load_n`/`__atomic_store_n`). |
| Global mutable state | Written as "single-writer mainline" throughout: kernel-Hermes (arena, queues, channels, ledger), Stadium (reservoirs, cells, quotas), `kmalloc`'s free list, the VM registry, the switch-signal table, console output, the log region, the block subsystem. |
| Per-core state | None. `g_switch_current_vm` (`vm/switch.c`) is one global "who is running". The heartbeat is one global beat, driven by the boot CPU's timer ISR. |
| Stacks | Each VM has its own native stack; a parked VM holds only its trampoline context (Stage D `sk_vm_parked_idle()`). Hera runs on the boot stack. |

---

## 3. Design

### 3.1 Observers on the system core

- **Affinity is a role fact, set at birth**, from the same `is_fleet_foundation` fact that sets
  the observer flag (O-A). Nothing a VM executes can change it.
- **Core 0 runs the Tripod** under exactly today's rules: the O-A tiers, the O-B idle-point drain,
  the console, USB polling (`sk_repl_idle()`), and Artemis's storage.
- **Worker cores never claim an observer.** Hera never runs on a worker core.
- **One core means today's behaviour:** core 0 is also the only worker core, and every
  participant runs there as now.

### 3.2 Workers: claimed through the shared decision

- The switch-signal slot gains **`running_on`** (core index, or none).
- At every clean boundary (the trampoline's between-messages point, the in-word checkpoint, a
  worker core's idle loop), a worker core runs `sk_vm_switch_decide()` over participants with
  `running_on` = none or = this core. It then claims the winner with **one atomic compare-and-swap
  on `running_on`**. If it loses the race, it decides again.
- The claim is **released** when the worker parks (IDLE frame) or switches out.
- **Tie-break:** among equally cool candidates, prefer the one that last ran on this core (cache
  affinity), then the existing incumbent rule, then the oldest wait stamp.
- **Migration is implicit.** A parked worker resumes on whichever core claims it next. Its stack
  holds nothing tied to a core; per-core values live behind the per-core pointer (§3.5), never on
  a VM's stack.
- **"Nobody has work" changes on worker cores.** Today a trampoline with no work switches to
  Hera. On a worker core it switches to that **core's own idle context**, a small per-core stack
  that halts (§3.3). Hera remains the idle root on core 0 only.

### 3.3 Idle and wake

- **An idle worker core halts** (`hlt`/`wfi`). Nothing polls.
- **Wake by inter-processor interrupt.** When `sk_hermes_send_one()` admits mail, or an event
  (M-B), for a participant that no core is running, it wakes one halted worker core. That core's
  decision picks the work up.
- **Core 0 wakes on its own inputs as now:** its timer, console input, and USB polling at its
  idle point.

### 3.4 Physics across cores

- **Each core has its own timer** (the LAPIC timer on amd64, the generic timer on aarch64, the
  SBI timer on riscv64) and charges beats to the VM it is running. Beats stay execution-paced per
  VM, as today.
- **Conservation stays exact.** Stadium reservoirs, `consumed` and kernel-Hermes's
  held/pulled/returned/consumed become atomic, or per-core accumulators merged at audit time. The
  integer audits (`stadium_conserved()`, `sk_hermes_audit()`) must hold exactly, as today.
- **Observers stay uncharged** (O-2).

### 3.5 Per-core state

- **One per-core block per core**, reached through each ISA's per-CPU register: GS base on amd64,
  `TPIDR_EL1` on aarch64, `tp` on riscv64. The kernel owns `tp`; there is no user mode.
- **It holds:** the current VM (replacing the global `g_switch_current_vm`), the core's idle
  context, the running frame stack, local beat counters, and the "decision due" flag.

### 3.6 What has to be made safe

These are the shared structures and what each needs (`FABRIC-3.10.8.md` will inventory them line
by line):

| Structure | Needs |
|---|---|
| `sf_mutex_*` | Real spinlocks (with interrupts masked while held on the owning core). |
| `kmalloc` | A heap lock. |
| kernel-Hermes (arena, queues, channels, ledger, events) | A lock (one to start; per-queue later if contention shows). |
| Stadium (reservoirs, cells, quotas, word rows) | **Not one lock — the per-dispatch path must not contend** (see below). |
| VM registry, birth, kill | A registry lock. `KILL` of a VM whose `running_on` is another core is refused, the multi-core form of today's "it is the VM running" refusal. |
| Switch-signal table | Compare-and-swap for claim and release; a lock for register and unregister. |
| Console output | A line lock, so lines from different cores never interleave mid-line. |
| Log region, block subsystem | A lock each. Artemis's storage stays on core 0, but workers can use `BLOCK` words. |

**The Stadium hot path decides whether multi-core speeds anything up** (added 2026-09-25, after
Captain Bob asked whether this should give a significant performance boost).
- `stadium_word_dispatch()` runs on **every word dispatch** of every VM (`vm_core.c`), for word
  heat: resolve the word's cell, cool it, pull admission heat from the VM's reservoir, and admit
  or evict a cell.
- Behind a single Stadium lock, every worker core would contend on every word, and the speedup
  would collapse towards 1×. `kmalloc` and kernel-Hermes are touched far less often (on
  definitions and messages), so one lock each is fine there.
- **Requirement: Stadium's per-dispatch state is partitioned by the VM's quota slot,** which is
  already how the word rows are keyed (`word_slots[slot][word_id]`, S-1b).
  - A VM runs on one core at a time, so its own row and cells need no lock while it runs.
  - Its reservoir takes atomic arithmetic.
  - Only operations that cross slots (admission evicting another VM's cell, quota grants,
    conservation audits) take the Stadium lock.
- This is part of MC-0, and MC-0's evidence includes it: a per-dispatch path that takes no
  shared lock.

### 3.7 Bringing up the cores, per ISA

- **amd64:** enumerate cores from the MADT's LAPIC entries. Start each with INIT-SIPI-SIPI through
  the LAPIC ICR, into a trampoline copied below 1 MiB (real mode → long mode). The trampoline must
  be position-independent: the kernel is relocated by a whole number of pages plus 0x440,
  differently on every boot (found at O-C).
- **aarch64:** enumerate cores from the DTB's `/cpus`. Start each with PSCI `CPU_ON` through the
  already-discovered conduit (HVC on QEMU virt). Each core initialises its GIC CPU interface or
  redistributor and its generic timer.
- **riscv64:** enumerate harts from the DTB. Start each with SBI HSM `hart_start`. Each hart
  arms its own SBI timer. IPIs go through the SBI IPI extension, or ACLINT where present.

---

## 4. Consequences

- **Execution order stops being deterministic** once there is more than one worker core.
  - Byte-identical evidence such as the Stage G interleave string gives way to count evidence
    (A = 40, B = 40).
  - The `dict_hash` set and the integer conservation audits stay exact.
- **Cores become a DoE factor.** A core count is fixed per boot (`-smp N`), so it is a factor
  across boots. Measuring what compudynamics does with 1 core versus N is probably the most
  interesting result this makes possible.
- **QEMU:** each virtual CPU is a host thread. Host-side contention shows up in the timing the
  DoE measures, so the one-QEMU-at-a-time rule matters more, not less.
- **Hardware:** SER5 (8 cores) and the Raspberry Pi 5 (4 cores) are the real targets this opens.

---

## 5. Staging (proposed)

Each step keeps the system shippable and gets its own 3-ISA acceptance.

| Stage | Change | Evidence (to be named in full in 3.10.8) |
|---|---|---|
| **MC-0** | Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind `sf_mutex`; the §3.6 locking applied, including Stadium's per-dispatch path partitioned by quota slot. | Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch, and **no shared lock taken on the word-dispatch path**. |
| **MC-1** | Bring up the extra cores and park them halted; QEMU `-smp 2`. | The boot reports N cores online; the standard sequence is unchanged, with the extra cores idle. |
| **MC-2** | Observers get system-core affinity (core 0). | With one worker core, behaviour identical to today; observers are never claimed off core 0. |
| **MC-3** | Worker cores claim through the shared decision, with per-core idle contexts and wake by interrupt. | Two workers genuinely overlapping in time on two cores. A probe proves the claim never lets two cores run one worker. Conservation holds throughout. G by counts. Idle cores halted, not spinning (0 idle log growth). |
| **MC-4** | DoE with cores as a factor. | A per-ISA campaign at each ruled core count. |

---

## 6. Known limits and open questions

- **Worker cores and devices:** only core 0 services USB. A worker core never touches xHCI.
- **Locking granularity:** one lock per subsystem to start (MC-R2) may contend under many
  workers. It is measured before it is refined. Stadium's per-dispatch path is the exception,
  partitioned from the start (§3.6).
- **Expected gain.** Close to linear throughput for many independent compute-bound workers on
  real hardware, once the dispatch path is partitioned. None for a single worker (a VM is
  single-threaded). Little for anything bound by serial output, the current DoE's limit: every
  core's heartbeat rows funnel through one console. MC-3/MC-4 measure it: throughput for a fixed
  batch of workers at 1, 2 and 4 cores.
- **Interrupt routing:** device IRQs stay on core 0 (the I/O APIC, GIC SPIs and PLIC contexts
  route there). Only inter-processor wakes reach worker cores.

---

## 7. Rulings needed from Captain Bob

1. **MC-R1: the system core.** All three observers together on core 0 (proposed), or a core
   each where cores allow.
2. **MC-R2: lock granularity to start.** One lock per subsystem (proposed), or a single kernel
   lock first, or fine-grained from the start.
3. **MC-R3: tie-break affinity.** Prefer the worker's last core among equally cool candidates
   (proposed), or no affinity.
4. **MC-R4: evidence under concurrency.** Count-based checks replace byte-identical interleave
   strings once there is more than one worker core (proposed).
5. **MC-R5: DoE core counts.** Which counts per ISA (proposed: 1, 2, 4).
6. **MC-R6: timing.** Start after O-D and the merge (proposed).

**RULED 2026-09-25, Captain Bob.** First, redirecting from the DoE row-path question: *"pin
observr and dynmically utilize worker VM's."* Then, confirming:
- MC-R1 to MC-R5 go **as proposed**: all three observers on core 0; one lock per subsystem, with
  Stadium's dispatch path partitioned; the last-core tie-break; count-based evidence; DoE at 1, 2
  and 4 cores.
- **MC-R6 is changed:** multi-core starts **now, ahead of O-D**. O-D waits until it lands.
- For O-D's relaunch, the row path is **A**: DoE rows go to serial only, skipping the
  framebuffer, whose full-screen scroll on every row was the rows-on bottleneck. The 2026-09-25
  amd64 launch (`logs/20260925-112529/amd64/`) was stopped for this after 81 minutes and 131,537
  rows, with no trial yet complete.

---

## 8. Part II — the OS's other open gaps (2026-09-25)

**Ruled out by Captain Bob, 2026-09-25:**
- **A filesystem: never.** *"Filesystem? NEVER! anti POSIX!"* Storage is FORTH blocks, Artemis's
  disk, content-addressed capsules and identity thumbdrives. That is the model, not a gap.
- **Networking: deferred.** *"Not iterested in networking yet."* There is no virtio-net and no
  stack today; it is not scheduled.

**Open, in proposed order:**

| # | Gap | State |
|---|---|---|
| 1 | **O-D and the merge.** The DoE relaunch on all three ISAs, then this branch to `master`. | amd64 launched 2026-09-25 11:29:29 (`logs/20260925-112529/amd64/`), about 2.5–3 days, bound by serial row output. |
| 2 | **Multi-core** (this document). | Designed; rulings needed (§7). |
| 3 | **Interrupt-driven devices** (O-4 option (b)). USB is polled at Hera's idle point. | Deferred at O-4; the system core (MC-2) eases it. |
| 4 | **The rest of the fixed-size audit** (`FABRIC-3.9.0.md` §8). S-1 and M-0 are done. Still fixed: WIREBIND identities attached at once (13), block relocations (500), executable regions (8), the run log (1024), the kernel RAM disk, control-flow nesting (64), and the RAM-sized VM/queue/channel tables (class D′). | S-3 order ruled; **S-2** (Stadium's cell pool) still needs a ruling. The WIREBIND cap matters most for multi-user. |
| 5 | **Security policy.** The ACL mechanism works, but `ACL-RECHECK` allows every STRICT word; `INJECT`'s gate is in place with no policy behind it. ACL Phase 8 (PKI challenge-response) is marked open, though MINT/WIREBIND covers part of it. | Policy to write in `ACL.4th`. |
| 6 | **Real hardware.** SER5 and Raspberry Pi 5 tracks exist; nothing is verified on hardware. `LITHOS_VERSION` stays 2.0.0 until it is. | Unstarted this line. |
| 7 | **Milestones M8 (REPL keyboard input) and M9 (block storage).** Listed as planned; completion not independently verified. | To verify. |
| 8 | **Failure counters readable from FORTH.** §8.1 asks for this. S-1a/S-1b/M-0 expose C accessors only; `EV-DROPPED` is the one FORTH counter so far (its name unruled). | Ruling needed: adding words changes the `dict_hash` set. |

**Known defects, reported and not fixed** (from `FABRIC-3.9.8.md` §2 unless noted):

1. `COLD` rewinds `HERE` to 1024 without rewinding `latest`: live words get compiled over (S-1c).
2. `PAD` returns `VM_MEMORY_SIZE - 512`, inside the top block I/O window slot (S-1c).
3. riscv64 intermittent `virtio_blk: vblk_io timed out` on disk writes under emulation (M-A).
4. Two boot self-tests silently `SKIPPED`: drain, and channel-open policy. Both still wait for the
   removed Hermes VM (M-0).
5. `dictionary_heat_optimization.c`: the heat snapshot is `heats[1024]`, silently truncated (S-1b).
6. `vm_create_word()`: a failed transition-metrics allocation still defines the word (S-1b).
7. `vm_cleanup()` frees neither `transition_heat` nor `context_window` inside each word's metrics
   (S-1b).
8. `transition_metrics_update_cache()` scans the whole id space on every recorded transition
   (S-1b).
9. The kernel's FC-bucket grow never frees the old block after `sf_realloc()` (S-1b).
10. **New, found at the xHCI fix:** `xhci_cmd_configure_endpoint()` allocates a new Input Context
    on every configure, and never frees the old one. It is a per-attach leak, but harmless to the
    controller, which reads the Input Context only during the command.