From 1777ac59a1a90e9a5aeac68ff89d257be938fd5f Mon Sep 17 00:00:00 2001 From: Robert Allan James Date: Fri, 25 Sep 2026 12:44:26 -0400 Subject: [PATCH] FABRIC-3.10.0: Stadium's per-dispatch path must be partitioned by quota slot, not locked (the multi-core speedup depends on it); expected-gain note Co-Authored-By: Claude Opus 5.5 --- FABRIC-3.10.0.md | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/FABRIC-3.10.0.md b/FABRIC-3.10.0.md index d29d0d4e..5c1081a4 100644 --- a/FABRIC-3.10.0.md +++ b/FABRIC-3.10.0.md @@ -119,12 +119,29 @@ by line): | `sf_mutex_*` | Real spinlocks (with interrupts masked while held on the owning core). | | `kmalloc` | A heap lock. | | kernel-Hermes (arena, queues, channels, ledger, events) | A lock (one to start; per-queue later if contention shows). | -| Stadium (reservoirs, cells, quotas, word rows) | A lock, plus atomic reservoir arithmetic. | +| Stadium (reservoirs, cells, quotas, word rows) | **Not one lock — the per-dispatch path must not contend** (see below). | | VM registry, birth, kill | A registry lock. `KILL` of a VM whose `running_on` is another core is refused, the multi-core form of today's "it is the VM running" refusal. | | Switch-signal table | Compare-and-swap for claim and release; a lock for register and unregister. | | Console output | A line lock, so lines from different cores never interleave mid-line. | | Log region, block subsystem | A lock each. Artemis's storage stays on core 0, but workers can use `BLOCK` words. | +**The Stadium hot path decides whether multi-core speeds anything up** (added 2026-09-25, after +Captain Bob asked whether this should give a significant performance boost). +- `stadium_word_dispatch()` runs on **every word dispatch** of every VM (`vm_core.c`), for word + heat: resolve the word's cell, cool it, pull admission heat from the VM's reservoir, and admit + or evict a cell. +- Behind a single Stadium lock, every worker core would contend on every word, and the speedup + would collapse towards 1×. `kmalloc` and kernel-Hermes are touched far less often (on + definitions and messages), so one lock each is fine there. +- **Requirement: Stadium's per-dispatch state is partitioned by the VM's quota slot,** which is + already how the word rows are keyed (`word_slots[slot][word_id]`, S-1b). + - A VM runs on one core at a time, so its own row and cells need no lock while it runs. + - Its reservoir takes atomic arithmetic. + - Only operations that cross slots (admission evicting another VM's cell, quota grants, + conservation audits) take the Stadium lock. +- This is part of MC-0, and MC-0's evidence includes it: a per-dispatch path that takes no + shared lock. + ### 3.7 Bringing up the cores, per ISA - **amd64:** enumerate cores from the MADT's LAPIC entries. Start each with INIT-SIPI-SIPI through @@ -160,7 +177,7 @@ Each step keeps the system shippable and gets its own 3-ISA acceptance. | Stage | Change | Evidence (to be named in full in 3.10.8) | |---|---|---| -| **MC-0** | Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind `sf_mutex`; the §3.6 locking applied. | Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch. | +| **MC-0** | Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind `sf_mutex`; the §3.6 locking applied, including Stadium's per-dispatch path partitioned by quota slot. | Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch, and **no shared lock taken on the word-dispatch path**. | | **MC-1** | Bring up the extra cores and park them halted; QEMU `-smp 2`. | The boot reports N cores online; the standard sequence is unchanged, with the extra cores idle. | | **MC-2** | Observers get system-core affinity (core 0). | With one worker core, behaviour identical to today; observers are never claimed off core 0. | | **MC-3** | Worker cores claim through the shared decision, with per-core idle contexts and wake by interrupt. | Two workers genuinely overlapping in time on two cores. A probe proves the claim never lets two cores run one worker. Conservation holds throughout. G by counts. Idle cores halted, not spinning (0 idle log growth). | @@ -172,7 +189,13 @@ Each step keeps the system shippable and gets its own 3-ISA acceptance. - **Worker cores and devices:** only core 0 services USB. A worker core never touches xHCI. - **Locking granularity:** one lock per subsystem to start (MC-R2) may contend under many - workers. It is measured before it is refined. + workers. It is measured before it is refined. Stadium's per-dispatch path is the exception, + partitioned from the start (§3.6). +- **Expected gain.** Close to linear throughput for many independent compute-bound workers on + real hardware, once the dispatch path is partitioned. None for a single worker (a VM is + single-threaded). Little for anything bound by serial output, the current DoE's limit: every + core's heartbeat rows funnel through one console. MC-3/MC-4 measure it: throughput for a fixed + batch of workers at 1, 2 and 4 cores. - **Interrupt routing:** device IRQs stay on core 0 (the I/O APIC, GIC SPIs and PLIC contexts route there). Only inter-processor wakes reach worker cores.