FABRIC-3.10.0: multi-core design (observers pinned to a system core, workers claimed by the physics) and the OS's open gaps
Design only, for Captain Bob's review; nothing authorizes an edit.
- Observers (the Tripod) get system-core affinity where cores allow;
workers are placed nowhere -- every worker core asks the existing
compudynamic decision at clean boundaries and claims the winner with one
atomic compare-and-swap. Static worker placement ruled out (Captain Bob:
"anti starshipos given there isn't a fixed number of vm's").
- Traces what exists (no SMP bring-up; MADT/PSCI groundwork; no-op kernel
mutexes; single-writer globals), the per-ISA bring-up, the locking
inventory, idle/wake by IPI, per-core physics, staging MC-0..MC-4, and
rulings MC-R1..MC-R6.
- Part II: the open gaps. Filesystem ruled out ("NEVER! anti POSIX!"),
networking deferred; O-D/merge, multi-core, interrupt-driven devices,
the fixed-size audit remainder, security policy, hardware, M8/M9,
FORTH-readable counters, and ten known defects (one new: xHCI input
context leak per configure).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
ac42cf36b5
commit
85b97ea467
@@ -0,0 +1,233 @@
|
||||
# FABRIC-3.10.0.md — Multi-core: observers pinned, workers claimed by the physics
|
||||
|
||||
**Status: DESIGN, for Captain Bob's review. Nothing here authorizes an edit.** Proposed to start
|
||||
after O-D and the merge to `master` (ruling MC-R6). An execution plan (`FABRIC-3.10.8.md`) follows
|
||||
once the rulings are made. Part II (§8) records the OS's other open gaps as of this writing.
|
||||
|
||||
**Provenance.** Opened 2026-09-25, while O-D (the DoE relaunch) was running on amd64. Captain Bob,
|
||||
2026-09-25:
|
||||
- *"I'm interested im the idea of pinning hera, hesta and artimes to cpu cores whre possibl."*
|
||||
- On placing workers on cores at birth: *"Not sure, static allocation of worker vms seems anti
|
||||
starshipos given there isn't a fied number of vm's. investigate how to pin observers and allow
|
||||
multithreading across cores forload ballancing."*
|
||||
|
||||
**Static placement of workers is therefore ruled out.** A worker is never assigned a core. It
|
||||
runs wherever the physics next chooses it.
|
||||
|
||||
---
|
||||
|
||||
## 1. The idea
|
||||
|
||||
- **Observers are pinned.** The Tripod (Hera, Hestia, Artemis) are a fixed, known set, already
|
||||
outside the compudynamic comparison (`FABRIC-3.9.0.md` §2). They get affinity to a **system
|
||||
core**, core 0, where there is more than one core, and share the one core exactly as today where
|
||||
there isn't.
|
||||
- **Workers are placed nowhere.** Every other core is a worker core. At each clean boundary, a
|
||||
worker core asks the existing compudynamic decision for the coolest participant with work that
|
||||
no other core is running, and claims it atomically. A worker runs wherever it is next chosen. It
|
||||
migrates for free, always at a clean boundary, and the load balances through the heat itself: a
|
||||
hot worker is not the coolest, so free cores take the cooler ones.
|
||||
- **No scheduler, no fixed count.** Each core runs the same physical question the single core
|
||||
runs today. Nothing counts VMs or cores in advance, so the design holds for any number of VMs on
|
||||
any number of cores.
|
||||
|
||||
A worker VM stays single-threaded: one interpreter, one data stack, one native stack, one core
|
||||
at a time. Parallelism comes from many workers running at once. Splitting one VM across cores
|
||||
would need a multi-threaded interpreter sharing one dictionary and one set of stacks, which is a
|
||||
different machine. A job that wants more parallelism is split into more workers, as the DoE
|
||||
already does.
|
||||
|
||||
---
|
||||
|
||||
## 2. What exists today (traced 2026-09-25)
|
||||
|
||||
| Area | Today |
|
||||
|---|---|
|
||||
| QEMU | No `-smp` on any ISA (`Makefile.starkernel`): every run is one CPU. |
|
||||
| Secondary core start | None on any ISA: no INIT-SIPI (amd64), no PSCI `CPU_ON` (aarch64), no SBI HSM `hart_start` (riscv64). `arch/riscv64/interrupts.c` says "the current single-hart kernel". |
|
||||
| Groundwork to reuse | amd64 already parses the ACPI MADT (`arch/amd64/ioapic.c`, for the I/O APIC); its LAPIC entries (type 0) list the cores. aarch64 already discovers the PSCI conduit from the DTB (`aarch64_psci_conduit_init()`, for reset); `CPU_ON` goes through the same conduit. All three have halt primitives (`hlt`/`wfi`, `arch_halt()`). |
|
||||
| Locks | `sf_mutex_*` are deliberate no-ops in the kernel (`vm/host/shim.c`: "single-threaded kernel"). The shared VM code already takes them at the right places (`dict_lock`, `tuning_lock`), so they can become real spinlocks without touching callers. |
|
||||
| Atomics | Two helpers in `vm/vm_runtime.c` (`__atomic_load_n`/`__atomic_store_n`). |
|
||||
| Global mutable state | Written as "single-writer mainline" throughout: kernel-Hermes (arena, queues, channels, ledger), Stadium (reservoirs, cells, quotas), `kmalloc`'s free list, the VM registry, the switch-signal table, console output, the log region, the block subsystem. |
|
||||
| Per-core state | None. `g_switch_current_vm` (`vm/switch.c`) is one global "who is running". The heartbeat is one global beat, driven by the boot CPU's timer ISR. |
|
||||
| Stacks | Each VM has its own native stack; a parked VM holds only its trampoline context (Stage D `sk_vm_parked_idle()`). Hera runs on the boot stack. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Design
|
||||
|
||||
### 3.1 Observers on the system core
|
||||
|
||||
- **Affinity is a role fact, set at birth**, from the same `is_fleet_foundation` fact that sets
|
||||
the observer flag (O-A). Nothing a VM executes can change it.
|
||||
- **Core 0 runs the Tripod** under exactly today's rules: the O-A tiers, the O-B idle-point drain,
|
||||
the console, USB polling (`sk_repl_idle()`), and Artemis's storage.
|
||||
- **Worker cores never claim an observer.** Hera never runs on a worker core.
|
||||
- **One core means today's behaviour:** core 0 is also the only worker core, and every
|
||||
participant runs there as now.
|
||||
|
||||
### 3.2 Workers: claimed through the shared decision
|
||||
|
||||
- The switch-signal slot gains **`running_on`** (core index, or none).
|
||||
- At every clean boundary (the trampoline's between-messages point, the in-word checkpoint, a
|
||||
worker core's idle loop), a worker core runs `sk_vm_switch_decide()` over participants with
|
||||
`running_on` = none or = this core. It then claims the winner with **one atomic compare-and-swap
|
||||
on `running_on`**. If it loses the race, it decides again.
|
||||
- The claim is **released** when the worker parks (IDLE frame) or switches out.
|
||||
- **Tie-break:** among equally cool candidates, prefer the one that last ran on this core (cache
|
||||
affinity), then the existing incumbent rule, then the oldest wait stamp.
|
||||
- **Migration is implicit.** A parked worker resumes on whichever core claims it next. Its stack
|
||||
holds nothing tied to a core; per-core values live behind the per-core pointer (§3.5), never on
|
||||
a VM's stack.
|
||||
- **"Nobody has work" changes on worker cores.** Today a trampoline with no work switches to
|
||||
Hera. On a worker core it switches to that **core's own idle context**, a small per-core stack
|
||||
that halts (§3.3). Hera remains the idle root on core 0 only.
|
||||
|
||||
### 3.3 Idle and wake
|
||||
|
||||
- **An idle worker core halts** (`hlt`/`wfi`). Nothing polls.
|
||||
- **Wake by inter-processor interrupt.** When `sk_hermes_send_one()` admits mail, or an event
|
||||
(M-B), for a participant that no core is running, it wakes one halted worker core. That core's
|
||||
decision picks the work up.
|
||||
- **Core 0 wakes on its own inputs as now:** its timer, console input, and USB polling at its
|
||||
idle point.
|
||||
|
||||
### 3.4 Physics across cores
|
||||
|
||||
- **Each core has its own timer** (the LAPIC timer on amd64, the generic timer on aarch64, the
|
||||
SBI timer on riscv64) and charges beats to the VM it is running. Beats stay execution-paced per
|
||||
VM, as today.
|
||||
- **Conservation stays exact.** Stadium reservoirs, `consumed` and kernel-Hermes's
|
||||
held/pulled/returned/consumed become atomic, or per-core accumulators merged at audit time. The
|
||||
integer audits (`stadium_conserved()`, `sk_hermes_audit()`) must hold exactly, as today.
|
||||
- **Observers stay uncharged** (O-2).
|
||||
|
||||
### 3.5 Per-core state
|
||||
|
||||
- **One per-core block per core**, reached through each ISA's per-CPU register: GS base on amd64,
|
||||
`TPIDR_EL1` on aarch64, `tp` on riscv64. The kernel owns `tp`; there is no user mode.
|
||||
- **It holds:** the current VM (replacing the global `g_switch_current_vm`), the core's idle
|
||||
context, the running frame stack, local beat counters, and the "decision due" flag.
|
||||
|
||||
### 3.6 What has to be made safe
|
||||
|
||||
These are the shared structures and what each needs (`FABRIC-3.10.8.md` will inventory them line
|
||||
by line):
|
||||
|
||||
| Structure | Needs |
|
||||
|---|---|
|
||||
| `sf_mutex_*` | Real spinlocks (with interrupts masked while held on the owning core). |
|
||||
| `kmalloc` | A heap lock. |
|
||||
| kernel-Hermes (arena, queues, channels, ledger, events) | A lock (one to start; per-queue later if contention shows). |
|
||||
| Stadium (reservoirs, cells, quotas, word rows) | A lock, plus atomic reservoir arithmetic. |
|
||||
| VM registry, birth, kill | A registry lock. `KILL` of a VM whose `running_on` is another core is refused, the multi-core form of today's "it is the VM running" refusal. |
|
||||
| Switch-signal table | Compare-and-swap for claim and release; a lock for register and unregister. |
|
||||
| Console output | A line lock, so lines from different cores never interleave mid-line. |
|
||||
| Log region, block subsystem | A lock each. Artemis's storage stays on core 0, but workers can use `BLOCK` words. |
|
||||
|
||||
### 3.7 Bringing up the cores, per ISA
|
||||
|
||||
- **amd64:** enumerate cores from the MADT's LAPIC entries. Start each with INIT-SIPI-SIPI through
|
||||
the LAPIC ICR, into a trampoline copied below 1 MiB (real mode → long mode). The trampoline must
|
||||
be position-independent: the kernel is relocated by a whole number of pages plus 0x440,
|
||||
differently on every boot (found at O-C).
|
||||
- **aarch64:** enumerate cores from the DTB's `/cpus`. Start each with PSCI `CPU_ON` through the
|
||||
already-discovered conduit (HVC on QEMU virt). Each core initialises its GIC CPU interface or
|
||||
redistributor and its generic timer.
|
||||
- **riscv64:** enumerate harts from the DTB. Start each with SBI HSM `hart_start`. Each hart
|
||||
arms its own SBI timer. IPIs go through the SBI IPI extension, or ACLINT where present.
|
||||
|
||||
---
|
||||
|
||||
## 4. Consequences
|
||||
|
||||
- **Execution order stops being deterministic** once there is more than one worker core.
|
||||
- Byte-identical evidence such as the Stage G interleave string gives way to count evidence
|
||||
(A = 40, B = 40).
|
||||
- The `dict_hash` set and the integer conservation audits stay exact.
|
||||
- **Cores become a DoE factor.** A core count is fixed per boot (`-smp N`), so it is a factor
|
||||
across boots. Measuring what compudynamics does with 1 core versus N is probably the most
|
||||
interesting result this makes possible.
|
||||
- **QEMU:** each virtual CPU is a host thread. Host-side contention shows up in the timing the
|
||||
DoE measures, so the one-QEMU-at-a-time rule matters more, not less.
|
||||
- **Hardware:** SER5 (8 cores) and the Raspberry Pi 5 (4 cores) are the real targets this opens.
|
||||
|
||||
---
|
||||
|
||||
## 5. Staging (proposed)
|
||||
|
||||
Each step keeps the system shippable and gets its own 3-ISA acceptance.
|
||||
|
||||
| Stage | Change | Evidence (to be named in full in 3.10.8) |
|
||||
|---|---|---|
|
||||
| **MC-0** | Per-core infrastructure on one core: the per-CPU pointer and block; real spinlocks behind `sf_mutex`; the §3.6 locking applied. | Standard sequence unchanged on one core. A probe shows every lock taken and released, never held across a switch. |
|
||||
| **MC-1** | Bring up the extra cores and park them halted; QEMU `-smp 2`. | The boot reports N cores online; the standard sequence is unchanged, with the extra cores idle. |
|
||||
| **MC-2** | Observers get system-core affinity (core 0). | With one worker core, behaviour identical to today; observers are never claimed off core 0. |
|
||||
| **MC-3** | Worker cores claim through the shared decision, with per-core idle contexts and wake by interrupt. | Two workers genuinely overlapping in time on two cores. A probe proves the claim never lets two cores run one worker. Conservation holds throughout. G by counts. Idle cores halted, not spinning (0 idle log growth). |
|
||||
| **MC-4** | DoE with cores as a factor. | A per-ISA campaign at each ruled core count. |
|
||||
|
||||
---
|
||||
|
||||
## 6. Known limits and open questions
|
||||
|
||||
- **Worker cores and devices:** only core 0 services USB. A worker core never touches xHCI.
|
||||
- **Locking granularity:** one lock per subsystem to start (MC-R2) may contend under many
|
||||
workers. It is measured before it is refined.
|
||||
- **Interrupt routing:** device IRQs stay on core 0 (the I/O APIC, GIC SPIs and PLIC contexts
|
||||
route there). Only inter-processor wakes reach worker cores.
|
||||
|
||||
---
|
||||
|
||||
## 7. Rulings needed from Captain Bob
|
||||
|
||||
1. **MC-R1: the system core.** All three observers together on core 0 (proposed), or a core
|
||||
each where cores allow.
|
||||
2. **MC-R2: lock granularity to start.** One lock per subsystem (proposed), or a single kernel
|
||||
lock first, or fine-grained from the start.
|
||||
3. **MC-R3: tie-break affinity.** Prefer the worker's last core among equally cool candidates
|
||||
(proposed), or no affinity.
|
||||
4. **MC-R4: evidence under concurrency.** Count-based checks replace byte-identical interleave
|
||||
strings once there is more than one worker core (proposed).
|
||||
5. **MC-R5: DoE core counts.** Which counts per ISA (proposed: 1, 2, 4).
|
||||
6. **MC-R6: timing.** Start after O-D and the merge (proposed).
|
||||
|
||||
---
|
||||
|
||||
## 8. Part II — the OS's other open gaps (2026-09-25)
|
||||
|
||||
**Ruled out by Captain Bob, 2026-09-25:**
|
||||
- **A filesystem: never.** *"Filesystem? NEVER! anti POSIX!"* Storage is FORTH blocks, Artemis's
|
||||
disk, content-addressed capsules and identity thumbdrives. That is the model, not a gap.
|
||||
- **Networking: deferred.** *"Not iterested in networking yet."* There is no virtio-net and no
|
||||
stack today; it is not scheduled.
|
||||
|
||||
**Open, in proposed order:**
|
||||
|
||||
| # | Gap | State |
|
||||
|---|---|---|
|
||||
| 1 | **O-D and the merge.** The DoE relaunch on all three ISAs, then this branch to `master`. | amd64 launched 2026-09-25 11:29:29 (`logs/20260925-112529/amd64/`), about 2.5–3 days, bound by serial row output. |
|
||||
| 2 | **Multi-core** (this document). | Designed; rulings needed (§7). |
|
||||
| 3 | **Interrupt-driven devices** (O-4 option (b)). USB is polled at Hera's idle point. | Deferred at O-4; the system core (MC-2) eases it. |
|
||||
| 4 | **The rest of the fixed-size audit** (`FABRIC-3.9.0.md` §8). S-1 and M-0 are done. Still fixed: WIREBIND identities attached at once (13), block relocations (500), executable regions (8), the run log (1024), the kernel RAM disk, control-flow nesting (64), and the RAM-sized VM/queue/channel tables (class D′). | S-3 order ruled; **S-2** (Stadium's cell pool) still needs a ruling. The WIREBIND cap matters most for multi-user. |
|
||||
| 5 | **Security policy.** The ACL mechanism works, but `ACL-RECHECK` allows every STRICT word; `INJECT`'s gate is in place with no policy behind it. ACL Phase 8 (PKI challenge-response) is marked open, though MINT/WIREBIND covers part of it. | Policy to write in `ACL.4th`. |
|
||||
| 6 | **Real hardware.** SER5 and Raspberry Pi 5 tracks exist; nothing is verified on hardware. `LITHOS_VERSION` stays 2.0.0 until it is. | Unstarted this line. |
|
||||
| 7 | **Milestones M8 (REPL keyboard input) and M9 (block storage).** Listed as planned; completion not independently verified. | To verify. |
|
||||
| 8 | **Failure counters readable from FORTH.** §8.1 asks for this. S-1a/S-1b/M-0 expose C accessors only; `EV-DROPPED` is the one FORTH counter so far (its name unruled). | Ruling needed: adding words changes the `dict_hash` set. |
|
||||
|
||||
**Known defects, reported and not fixed** (from `FABRIC-3.9.8.md` §2 unless noted):
|
||||
|
||||
1. `COLD` rewinds `HERE` to 1024 without rewinding `latest`: live words get compiled over (S-1c).
|
||||
2. `PAD` returns `VM_MEMORY_SIZE - 512`, inside the top block I/O window slot (S-1c).
|
||||
3. riscv64 intermittent `virtio_blk: vblk_io timed out` on disk writes under emulation (M-A).
|
||||
4. Two boot self-tests silently `SKIPPED`: drain, and channel-open policy. Both still wait for the
|
||||
removed Hermes VM (M-0).
|
||||
5. `dictionary_heat_optimization.c`: the heat snapshot is `heats[1024]`, silently truncated (S-1b).
|
||||
6. `vm_create_word()`: a failed transition-metrics allocation still defines the word (S-1b).
|
||||
7. `vm_cleanup()` frees neither `transition_heat` nor `context_window` inside each word's metrics
|
||||
(S-1b).
|
||||
8. `transition_metrics_update_cache()` scans the whole id space on every recorded transition
|
||||
(S-1b).
|
||||
9. The kernel's FC-bucket grow never frees the old block after `sf_realloc()` (S-1b).
|
||||
10. **New, found at the xHCI fix:** `xhci_cmd_configure_endpoint()` allocates a new Input Context
|
||||
on every configure, and never frees the old one. It is a per-attach leak, but harmless to the
|
||||
controller, which reads the Input Context only during the command.
|
||||
Reference in New Issue
Block a user