diff --git a/FABRIC-3.10.0.md b/FABRIC-3.10.0.md new file mode 100644 index 00000000..d29d0d4e --- /dev/null +++ b/FABRIC-3.10.0.md @@ -0,0 +1,233 @@ +# FABRIC-3.10.0.md — Multi-core: observers pinned, workers claimed by the physics + +**Status: DESIGN, for Captain Bob's review. Nothing here authorizes an edit.** Proposed to start +after O-D and the merge to `master` (ruling MC-R6). An execution plan (`FABRIC-3.10.8.md`) follows +once the rulings are made. Part II (§8) records the OS's other open gaps as of this writing. + +**Provenance.** Opened 2026-09-25, while O-D (the DoE relaunch) was running on amd64. Captain Bob, +2026-09-25: +- *"I'm interested im the idea of pinning hera, hesta and artimes to cpu cores whre possibl."* +- On placing workers on cores at birth: *"Not sure, static allocation of worker vms seems anti + starshipos given there isn't a fied number of vm's. investigate how to pin observers and allow + multithreading across cores forload ballancing."* + +**Static placement of workers is therefore ruled out.** A worker is never assigned a core. It +runs wherever the physics next chooses it. + +--- + +## 1. The idea + +- **Observers are pinned.** The Tripod (Hera, Hestia, Artemis) are a fixed, known set, already + outside the compudynamic comparison (`FABRIC-3.9.0.md` §2). They get affinity to a **system + core**, core 0, where there is more than one core, and share the one core exactly as today where + there isn't. +- **Workers are placed nowhere.** Every other core is a worker core. At each clean boundary, a + worker core asks the existing compudynamic decision for the coolest participant with work that + no other core is running, and claims it atomically. A worker runs wherever it is next chosen. It + migrates for free, always at a clean boundary, and the load balances through the heat itself: a + hot worker is not the coolest, so free cores take the cooler ones. +- **No scheduler, no fixed count.** Each core runs the same physical question the single core + runs today. Nothing counts VMs or cores in advance, so the design holds for any number of VMs on + any number of cores. + +A worker VM stays single-threaded: one interpreter, one data stack, one native stack, one core +at a time. Parallelism comes from many workers running at once. Splitting one VM across cores +would need a multi-threaded interpreter sharing one dictionary and one set of stacks, which is a +different machine. A job that wants more parallelism is split into more workers, as the DoE +already does. + +--- + +## 2. What exists today (traced 2026-09-25) + +| Area | Today | +|---|---| +| QEMU | No `-smp` on any ISA (`Makefile.starkernel`): every run is one CPU. | +| Secondary core start | None on any ISA: no INIT-SIPI (amd64), no PSCI `CPU_ON` (aarch64), no SBI HSM `hart_start` (riscv64). `arch/riscv64/interrupts.c` says "the current single-hart kernel". | +| Groundwork to reuse | amd64 already parses the ACPI MADT (`arch/amd64/ioapic.c`, for the I/O APIC); its LAPIC entries (type 0) list the cores. aarch64 already discovers the PSCI conduit from the DTB (`aarch64_psci_conduit_init()`, for reset); `CPU_ON` goes through the same conduit. All three have halt primitives (`hlt`/`wfi`, `arch_halt()`). | +| Locks | `sf_mutex_*` are deliberate no-ops in the kernel (`vm/host/shim.c`: "single-threaded kernel"). The shared VM code already takes them at the right places (`dict_lock`, `tuning_lock`), so they can become real spinlocks without touching callers. | +| Atomics | Two helpers in `vm/vm_runtime.c` (`__atomic_load_n`/`__atomic_store_n`). | +| Global mutable state | Written as "single-writer mainline" throughout: kernel-Hermes (arena, queues, channels, ledger), Stadium (reservoirs, cells, quotas), `kmalloc`'s free list, the VM registry, the switch-signal table, console output, the log region, the block subsystem. | +| Per-core state | None. `g_switch_current_vm` (`vm/switch.c`) is one global "who is running". The heartbeat is one global beat, driven by the boot CPU's timer ISR. | +| Stacks | Each VM has its own native stack; a parked VM holds only its trampoline context (Stage D `sk_vm_parked_idle()`). Hera runs on the boot stack. | + +--- + +## 3. Design + +### 3.1 Observers on the system core + +- **Affinity is a role fact, set at birth**, from the same `is_fleet_foundation` fact that sets + the observer flag (O-A). Nothing a VM executes can change it. +- **Core 0 runs the Tripod** under exactly today's rules: the O-A tiers, the O-B idle-point drain, + the console, USB polling (`sk_repl_idle()`), and Artemis's storage. +- **Worker cores never claim an observer.** Hera never runs on a worker core. +- **One core means today's behaviour:** core 0 is also the only worker core, and every + participant runs there as now. + +### 3.2 Workers: claimed through the shared decision + +- The switch-signal slot gains **`running_on`** (core index, or none). +- At every clean boundary (the trampoline's between-messages point, the in-word checkpoint, a + worker core's idle loop), a worker core runs `sk_vm_switch_decide()` over participants with + `running_on` = none or = this core. It then claims the winner with **one atomic compare-and-swap + on `running_on`**. If it loses the race, it decides again. +- The claim is **released** when the worker parks (IDLE frame) or switches out. +- **Tie-break:** among equally cool candidates, prefer the one that last ran on this core (cache + affinity), then the existing incumbent rule, then the oldest wait stamp. +- **Migration is implicit.** A parked worker resumes on whichever core claims it next. Its stack + holds nothing tied to a core; per-core values live behind the per-core pointer (§3.5), never on + a VM's stack. +- **"Nobody has work" changes on worker cores.** Today a trampoline with no work switches to + Hera. On a worker core it switches to that **core's own idle context**, a small per-core stack + that halts (§3.3). Hera remains the idle root on core 0 only. + +### 3.3 Idle and wake + +- **An idle worker core halts** (`hlt`/`wfi`). Nothing polls. +- **Wake by inter-processor interrupt.** When `sk_hermes_send_one()` admits mail, or an event + (M-B), for a participant that no core is running, it wakes one halted worker core. That core's + decision picks the work up. +- **Core 0 wakes on its own inputs as now:** its timer, console input, and USB polling at its + idle point. + +### 3.4 Physics across cores + +- **Each core has its own timer** (the LAPIC timer on amd64, the generic timer on aarch64, the + SBI timer on riscv64) and charges beats to the VM it is running. Beats stay execution-paced per + VM, as today. +- **Conservation stays exact.** Stadium reservoirs, `consumed` and kernel-Hermes's + held/pulled/returned/consumed become atomic, or per-core accumulators merged at audit time. The + integer audits (`stadium_conserved()`, `sk_hermes_audit()`) must hold exactly, as today. +- **Observers stay uncharged** (O-2). + +### 3.5 Per-core state + +- **One per-core block per core**, reached through each ISA's per-CPU register: GS base on amd64, + `TPIDR_EL1` on aarch64, `tp` on riscv64. The kernel owns `tp`; there is no user mode. +- **It holds:** the current VM (replacing the global `g_switch_current_vm`), the core's idle + context, the running frame stack, local beat counters, and the "decision due" flag. + +### 3.6 What has to be made safe + +These are the shared structures and what each needs (`FABRIC-3.10.8.md` will inventory them line +by line): + +| Structure | Needs | +|---|---| +| `sf_mutex_*` | Real spinlocks (with interrupts masked while held on the owning core). | +| `kmalloc` | A heap lock. | +| kernel-Hermes (arena, queues, channels, ledger, events) | A lock (one to start; per-queue later if contention shows). | +| Stadium (reservoirs, cells, quotas, word rows) | A lock, plus atomic reservoir arithmetic. | +| VM registry, birth, kill | A registry lock. `KILL` of a VM whose `running_on` is another core is refused, the multi-core form of today's "it is the VM running" refusal. | +| Switch-signal table | Compare-and-swap for claim and release; a lock for register and unregister. | +| Console output | A line lock, so lines from different cores never interleave mid-line. | +| Log region, block subsystem | A lock each. Artemis's storage stays on core 0, but workers can use `BLOCK` words. | + +### 3.7 Bringing up the cores, per ISA + +- **amd64:** enumerate cores from the MADT's LAPIC entries. Start each with INIT-SIPI-SIPI through + the LAPIC ICR, into a trampoline copied below 1 MiB (real mode → long mode). The trampoline must + be position-independent: the kernel is relocated by a whole number of pages plus 0x440, + differently on every boot (found at O-C). +- **aarch64:** enumerate cores from the DTB's `/cpus`. Start each with PSCI `CPU_ON` through the + already-discovered conduit (HVC on QEMU virt). Each core initialises its GIC CPU interface or + redistributor and its generic timer. +- **riscv64:** enumerate harts from the DTB. Start each with SBI HSM `hart_start`. Each hart + arms its own SBI timer. IPIs go through the SBI IPI extension, or ACLINT where present. + +--- + +## 4. Consequences + +- **Execution order stops being deterministic** once there is more than one worker core. + - Byte-identical evidence such as the Stage G interleave string gives way to count evidence + (A = 40, B = 40). + - The `dict_hash` set and the integer conservation audits stay exact. +- **Cores become a DoE factor.** A core count is fixed per boot (`-smp N`), so it is a factor + across boots. Measuring what compudynamics does with 1 core versus N is probably the most + interesting result this makes possible. +- **QEMU:** each virtual CPU is a host thread. Host-side contention shows up in the timing the + DoE measures, so the one-QEMU-at-a-time rule matters more, not less. +- **Hardware:** SER5 (8 cores) and the Raspberry Pi 5 (4 cores) are the real targets this opens. + +--- + +## 5. Staging (proposed) + +Each step keeps the system shippable and gets its own 3-ISA acceptance. + +| Stage | Change | Evidence (to be named in full in 3.10.8) | +|---|---|---| +| **MC-0** | Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind `sf_mutex`; the §3.6 locking applied. | Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch. | +| **MC-1** | Bring up the extra cores and park them halted; QEMU `-smp 2`. | The boot reports N cores online; the standard sequence is unchanged, with the extra cores idle. | +| **MC-2** | Observers get system-core affinity (core 0). | With one worker core, behaviour identical to today; observers are never claimed off core 0. | +| **MC-3** | Worker cores claim through the shared decision, with per-core idle contexts and wake by interrupt. | Two workers genuinely overlapping in time on two cores. A probe proves the claim never lets two cores run one worker. Conservation holds throughout. G by counts. Idle cores halted, not spinning (0 idle log growth). | +| **MC-4** | DoE with cores as a factor. | A per-ISA campaign at each ruled core count. | + +--- + +## 6. Known limits and open questions + +- **Worker cores and devices:** only core 0 services USB. A worker core never touches xHCI. +- **Locking granularity:** one lock per subsystem to start (MC-R2) may contend under many + workers. It is measured before it is refined. +- **Interrupt routing:** device IRQs stay on core 0 (the I/O APIC, GIC SPIs and PLIC contexts + route there). Only inter-processor wakes reach worker cores. + +--- + +## 7. Rulings needed from Captain Bob + +1. **MC-R1: the system core.** All three observers together on core 0 (proposed), or a core + each where cores allow. +2. **MC-R2: lock granularity to start.** One lock per subsystem (proposed), or a single kernel + lock first, or fine-grained from the start. +3. **MC-R3: tie-break affinity.** Prefer the worker's last core among equally cool candidates + (proposed), or no affinity. +4. **MC-R4: evidence under concurrency.** Count-based checks replace byte-identical interleave + strings once there is more than one worker core (proposed). +5. **MC-R5: DoE core counts.** Which counts per ISA (proposed: 1, 2, 4). +6. **MC-R6: timing.** Start after O-D and the merge (proposed). + +--- + +## 8. Part II — the OS's other open gaps (2026-09-25) + +**Ruled out by Captain Bob, 2026-09-25:** +- **A filesystem: never.** *"Filesystem? NEVER! anti POSIX!"* Storage is FORTH blocks, Artemis's + disk, content-addressed capsules and identity thumbdrives. That is the model, not a gap. +- **Networking: deferred.** *"Not iterested in networking yet."* There is no virtio-net and no + stack today; it is not scheduled. + +**Open, in proposed order:** + +| # | Gap | State | +|---|---|---| +| 1 | **O-D and the merge.** The DoE relaunch on all three ISAs, then this branch to `master`. | amd64 launched 2026-09-25 11:29:29 (`logs/20260925-112529/amd64/`), about 2.5–3 days, bound by serial row output. | +| 2 | **Multi-core** (this document). | Designed; rulings needed (§7). | +| 3 | **Interrupt-driven devices** (O-4 option (b)). USB is polled at Hera's idle point. | Deferred at O-4; the system core (MC-2) eases it. | +| 4 | **The rest of the fixed-size audit** (`FABRIC-3.9.0.md` §8). S-1 and M-0 are done. Still fixed: WIREBIND identities attached at once (13), block relocations (500), executable regions (8), the run log (1024), the kernel RAM disk, control-flow nesting (64), and the RAM-sized VM/queue/channel tables (class D′). | S-3 order ruled; **S-2** (Stadium's cell pool) still needs a ruling. The WIREBIND cap matters most for multi-user. | +| 5 | **Security policy.** The ACL mechanism works, but `ACL-RECHECK` allows every STRICT word; `INJECT`'s gate is in place with no policy behind it. ACL Phase 8 (PKI challenge-response) is marked open, though MINT/WIREBIND covers part of it. | Policy to write in `ACL.4th`. | +| 6 | **Real hardware.** SER5 and Raspberry Pi 5 tracks exist; nothing is verified on hardware. `LITHOS_VERSION` stays 2.0.0 until it is. | Unstarted this line. | +| 7 | **Milestones M8 (REPL keyboard input) and M9 (block storage).** Listed as planned; completion not independently verified. | To verify. | +| 8 | **Failure counters readable from FORTH.** §8.1 asks for this. S-1a/S-1b/M-0 expose C accessors only; `EV-DROPPED` is the one FORTH counter so far (its name unruled). | Ruling needed: adding words changes the `dict_hash` set. | + +**Known defects, reported and not fixed** (from `FABRIC-3.9.8.md` §2 unless noted): + +1. `COLD` rewinds `HERE` to 1024 without rewinding `latest`: live words get compiled over (S-1c). +2. `PAD` returns `VM_MEMORY_SIZE - 512`, inside the top block I/O window slot (S-1c). +3. riscv64 intermittent `virtio_blk: vblk_io timed out` on disk writes under emulation (M-A). +4. Two boot self-tests silently `SKIPPED`: drain, and channel-open policy. Both still wait for the + removed Hermes VM (M-0). +5. `dictionary_heat_optimization.c`: the heat snapshot is `heats[1024]`, silently truncated (S-1b). +6. `vm_create_word()`: a failed transition-metrics allocation still defines the word (S-1b). +7. `vm_cleanup()` frees neither `transition_heat` nor `context_window` inside each word's metrics + (S-1b). +8. `transition_metrics_update_cache()` scans the whole id space on every recorded transition + (S-1b). +9. The kernel's FC-bucket grow never frees the old block after `sf_realloc()` (S-1b). +10. **New, found at the xHCI fix:** `xhci_cmd_configure_endpoint()` allocates a new Input Context + on every configure, and never frees the old one. It is a per-attach leak, but harmless to the + controller, which reads the Input Context only during the command.