Stage MC-0b: real kernel spinlocks; sf_mutex no longer a no-op (FABRIC-3.10.8 §3.2)

- New spinlock primitive (spinlock.h/.c): CAS on an owner word (core+1),
  interrupts masked while any lock is held (state kept in the per-core
  block, so release order doesn't matter). Recursion, release-not-held,
  held-at-switch and destroyed-while-held are counted and logged, never
  silent. Locks are no-ops until the boot core's block is installed
  (riscv64's monolithic early boot prints before sk_percpu_boot_init()).
- Kernel locks in a documented order: hermes, switch, registry, doe, log,
  console, heap, vmm, pmm. sf_mutex (dict_lock/tuning_lock) uses the same
  lock word.
- Hermes: public functions over *_locked internals; never held across the
  drain's vm_interpret_kind(); switch notifications after dropping the
  lock; event payload moved off static storage. Switch table and registry
  never call out while held (slot/entry copied under the lock).
- No lock crosses a switch: checked at context switch (leave, resume) and
  the idle point.
- Probe (removed): every lock balances on all three ISAs; 0 held at
  switch over 7.5M-51M checks; 0 recursions, 0 bad releases.
- Standard sequence on amd64/aarch64/riscv64: hashes and G byte-identical
  to MC-0a, MINT/WIREBIND/USE ok, conservation true, 0 idle growth.
- Block subsystem split out as MC-0b2 (Captain Bob's ruling).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Robert Allan James
2026-09-25 16:43:19 -04:00
co-authored by Claude Opus 5.5
parent 8409e376cf
commit 5bc7b4dee7
28 changed files with 56110 additions and 274 deletions
+86
View File
@@ -62,6 +62,7 @@ Everything that would break if two cores ran kernel or VM code at the same momen
|---|---|---| |---|---|---|
| **MC-0a** | **The per-core block, on one core.** A per-core structure reached through each ISA's per-CPU register (GS base / `TPIDR_EL1` / `tp`), set up for core 0 at boot. `g_switch_current_vm` moves into it. | Standard sequence unchanged. A probe reads the per-core pointer from a normal kernel path and from an interrupt handler, and gets core 0's block both times on all three ISAs. | | **MC-0a** | **The per-core block, on one core.** A per-core structure reached through each ISA's per-CPU register (GS base / `TPIDR_EL1` / `tp`), set up for core 0 at boot. `g_switch_current_vm` moves into it. | Standard sequence unchanged. A probe reads the per-core pointer from a normal kernel path and from an interrupt handler, and gets core 0's block both times on all three ISAs. |
| **MC-0b** | **Real spinlocks.** A kernel spinlock primitive (atomic exchange, interrupts masked while held on the owning core). `sf_mutex_*` stop being no-ops. The §1 kernel locks are added. | Standard sequence unchanged. A probe counts every lock's acquisitions and releases through a boot and the standard sequence: counts balance, and no lock is held across a context switch or on return to the idle point. | | **MC-0b** | **Real spinlocks.** A kernel spinlock primitive (atomic exchange, interrupts masked while held on the owning core). `sf_mutex_*` stop being no-ops. The §1 kernel locks are added. | Standard sequence unchanged. A probe counts every lock's acquisitions and releases through a boot and the standard sequence: counts balance, and no lock is held across a context switch or on return to the idle point. |
| **MC-0b2** | **The block subsystem's lock** (split from MC-0b, Captain Bob 2026-09-25). The vendored `block_subsystem.c` calls its own public functions in cycles (`blk_set_meta` → migration check → relocate → `blk_set_meta`), so it is refactored into locking public functions over unlocked internals, under `__STARKERNEL__`. | Standard sequence unchanged. The MC-0b probe, extended to the block lock: counts balance, 0 recursions. |
| **MC-0c** | **Shared VM code made per-VM or atomic** (§1, second table). The dictionary lookup index moves into `VM`. | The `dict_hash` set unchanged (the index is not hashed). A probe: two VMs searched alternately never rebuild each other's index. Standard sequence. | | **MC-0c** | **Shared VM code made per-VM or atomic** (§1, second table). The dictionary lookup index moves into `VM`. | The `dict_hash` set unchanged (the index is not hashed). A probe: two VMs searched alternately never rebuild each other's index. Standard sequence. |
| **MC-0d** | **Stadium's dispatch path partitioned** (§1). | Standard sequence; conservation holds (`stadium_conserved()` on every VM, the ledger audit). A probe proves the dispatch path takes no shared lock. | | **MC-0d** | **Stadium's dispatch path partitioned** (§1). | Standard sequence; conservation holds (`stadium_conserved()` on every VM, the ledger audit). A probe proves the dispatch path takes no shared lock. |
| **MC-1** | **Bring up the extra cores and park them halted**, per ISA: amd64 (MADT, INIT-SIPI-SIPI, a position-independent trampoline below 1 MiB), aarch64 (DTB `/cpus`, PSCI `CPU_ON`), riscv64 (DTB harts, SBI HSM `hart_start`). QEMU gets `-smp 2`. | The boot reports each core online with its own per-core block; the extra cores sit halted (0 idle growth, no spin); the standard sequence unchanged. | | **MC-1** | **Bring up the extra cores and park them halted**, per ISA: amd64 (MADT, INIT-SIPI-SIPI, a position-independent trampoline below 1 MiB), aarch64 (DTB `/cpus`, PSCI `CPU_ON`), riscv64 (DTB harts, SBI HSM `hart_start`). QEMU gets `-smp 2`. | The boot reports each core online with its own per-core block; the extra cores sit halted (0 idle growth, no spin); the standard sequence unchanged. |
@@ -115,3 +116,88 @@ Everything that would break if two cores ran kernel or VM code at the same momen
- The `dict_hash` values (Hera `0x00340a8b2c842eb7`, Hestia `0x78bf28439e4a643f`, Artemis `0x66886d174d5f3b5a`) are identical across ISAs and unchanged from O-C. - The `dict_hash` values (Hera `0x00340a8b2c842eb7`, Hestia `0x78bf28439e4a643f`, Artemis `0x66886d174d5f3b5a`) are identical across ISAs and unchanged from O-C.
- The ledger line matches across ISAs: held=2048, pulled=258048, returned=255263, consumed=737, conserved=true. - The ledger line matches across ISAs: held=2048, pulled=258048, returned=255263, consumed=737, conserved=true.
- No stop condition was hit. **MC-0a done.** - No stop condition was hit. **MC-0a done.**
### 3.2 MC-0b — real spinlocks (2026-09-25)
**The primitive** (`include/starkernel/spinlock.h`, `src/starkernel/hal/spinlock.c`, new).
- One lock word per lock: 0 = free, `core_id + 1` = held.
- Taken by compare-and-swap. The atomics are inline on every ISA (`nm -u` is empty on all three ELFs).
- Interrupts stay masked on the owning core while it holds any lock.
- The mask state lives in the per-core block, not in the lock (`lock_depth`, `saved_irq` in `sk_percpu_t`):
- the first lock taken saves the interrupt state and masks;
- the last lock released restores it;
- so release order doesn't matter.
- **The rule: no lock crosses a switch or reaches the idle point.** Each of these sites checks it and counts any violation:
- `sk_vm_context_switch()`, when leaving and on resume;
- `sk_vm_idle_point()`.
- Failures are counted and logged at the idle point (`LOCK:` lines, LOG_ERROR):
- **recursion** (a core takes a lock it already holds): let through, so the core doesn't hang, and counted. It is never legitimate, because on one core "the owner is me" can't tell a nested call from a second VM on the same core;
- release of a lock not held;
- a lock held at a switch or idle point;
- a VM mutex destroyed while held.
- The locks are no-ops until the boot core's block is installed (`sk_percpu_ready`). This was found during the probe runs:
- riscv64 is built monolithic, and its `arch_early_init()` prints through the console before `sk_percpu_boot_init()`;
- the first lock therefore read `tp` while it still held the firmware's value, and the load hit OpenSBI memory (`stval` = `tp`+8, a firmware load access fault);
- early boot is one core with interrupts off, so skipping the locks there is correct.
That crashed run's log was deleted as flawed.
- `sf_mutex_*` (the vendored VM's `dict_lock` and `tuning_lock`, `shim.c`) now use the same lock word. They are no longer no-ops.
**The kernel locks, in acquisition order** (spinlock.h; MC-3's deadlock freedom rests on this order):
1. `hermes`
2. `switch`
3. `registry`
4. `doe`
5. `log`
6. `console`
7. `heap`
8. `vmm`
9. `pmm`
Stadium's lock comes with MC-0d, below Hermes. Stadium today runs inside `dict_lock` (`vm_core.c`'s heat bump calls `stadium_word_dispatch()`); MC-0d's order must account for that.
| Lock | Covers | Shape |
|---|---|---|
| heap | `kmalloc`/`kfree`, stats, censuses | `kmalloc()` delegates to `kmalloc_aligned()`, and only the latter locks |
| vmm | page-table walks | `*_locked` bodies. The map-conflict print moved outside the lock (the console lock sits above it). Takes `pmm` for table pages |
| pmm | bitmap | `pmm_alloc_page()`/`pmm_free_page()` delegate to the contiguous forms |
| console | output path and line state (name, line start, pending line close, the framebuffer terminal) | A string goes out under one hold. Unlocked on purpose: `console_get_vm_name()` (the prefix provider calls it under the lock), the boot-time `console_fb_init()`/`console_fb_enable_ttf()` (they log), and input |
| log, doe | control-block read-modify-write plus the slot write | `*_locked` bodies |
| registry | node list, entry copies, single-field writes | Never held while calling out. KILL and reap take it only for their reads and final writes. A birth fills its embryo unlocked (private until LIVE). `kill_all_nonmama` (cold-reset path) walks unlocked |
| switch | slot table, stamps, decision-due flag, counters | Never held while calling out. The decision, the Stage H floor and the reap sweep copy one slot under the lock (`slot_copy()`) and judge it outside |
| hermes | arena, ledger, channels, queues, event counters | Public functions lock over `*_locked` statics. Never held across `vm_interpret_kind()` (the drain) or a policy word. The switch notifications after a send go out once the lock is dropped (the floor reads Hermes queues back). Publish and event walk members one copied member at a time. `sk_hermes_event()`'s static payload moved onto the stack |
The block subsystem is **split out as MC-0b2** (Captain Bob, 2026-09-25), because its internal call cycles need a locked/unlocked refactor of vendored code. See §2.
**Probe** (temporary, removed before commit). A `MC0B-PROBE` word in Hera's and every child's vocabulary printed every counter, snapshotted before printing:
- each lock's acquisitions and releases, and its owner;
- the VM mutexes (the core total minus the kernel locks);
- the core-0 total and its depth;
- the number of switch/idle checks and violations;
- recursions, bad releases and destroyed-while-held.
It ran at Hera's prompt (after O-A's G, PREC and LOAD), and again on rajames after MINT, WIREBIND, `USE` and `1 2 + .`. The table shows the final (rajames) reading:
| ISA | log | total acquired = released | depth | switch/idle checks | held at switch | recursions / bad releases / destroyed held |
|---|---|---|---|---|---|---|
| amd64 | `logs/20260925-150735` | 1,503,458,645 | 0 | 51,137,748 | 0 | 0 / 0 / 0 |
| aarch64 | `logs/20260925-152109` | 235,854,532 | 0 | 7,565,646 | 0 | 0 / 0 / 0 |
| riscv64 | `logs/20260925-155048` | 566,894,806 | 0 | 19,029,507 | 0 | 0 / 0 / 0 |
- Every lock balances individually on every ISA, at both readings.
- The amd64 and aarch64 probe runs predate the `sk_percpu_ready` guard. That guard only changes behaviour before `sk_percpu_boot_init()`, and the acceptance runs below are on the final code.
- The riscv64 `vblk_io timed out` count, which the advisor flagged (defect 3 sits in the region now masked), is 2 in the probe run, the same as MC-0a's acceptance run. It is 0 in this stage's acceptance run.
**Finding, for MC-3.** Hera's input-poll idle loop runs the switch decision without pause, and every pass takes the registry, switch and Hermes locks once per slot. That is the hundreds of millions of acquisitions above (amd64: registry 379 M, switch 858 M, Hermes 266 M). It is harmless on one core. With several cores, core 0 spinning through these same locks would contend with every worker's claim. MC-3 should settle how that loop waits before cores share these locks.
**Standard sequence** (clean build, with the probe reverted; checked against the saved stage diff, which differs only by the guard added after it).
| ISA | log | O-A (G, PREC, LOAD) | M7.1a hash | MINT / WIREBIND / USE / `1 2 + .` | idle growth | UNKNOWN / faults |
|---|---|---|---|---|---|---|
| amd64 | `logs/20260925-160434` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
| aarch64 | `logs/20260925-161720` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
| riscv64 | `logs/20260925-163024` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
- The G string is byte-identical to MC-0a on all three ISAs, so masking interrupts under the locks didn't move the interleave.
- The `dict_hash` set and the ledger line are unchanged from MC-0a.
- No stop condition was hit. **MC-0b done** (the block subsystem follows as MC-0b2).
+16
View File
@@ -49,6 +49,14 @@ typedef struct sk_percpu {
struct sk_percpu *self; /* must stay first: amd64 reads it as %gs:0 */ struct sk_percpu *self; /* must stay first: amd64 reads it as %gs:0 */
uint32_t core_id; /* 0 = the boot core (the system core, MC-R1) */ uint32_t core_id; /* 0 = the boot core (the system core, MC-R1) */
struct VM *current_vm; /* who is running on this core (switch.c) */ struct VM *current_vm; /* who is running on this core (switch.c) */
/* MC-0b (spinlock.c): locks this core holds right now, and the
* interrupt state to restore when that count returns to 0. Kept here,
* not in each lock, so interrupts come back correctly whatever order
* the locks are released in. */
uint32_t lock_depth;
uint64_t saved_irq;
uint64_t lock_acquired; /* every acquire on this core, ever */
uint64_t lock_released; /* every release on this core, ever */
} sk_percpu_t; } sk_percpu_t;
/* Installs the boot core's block in this core's per-CPU register. Call /* Installs the boot core's block in this core's per-CPU register. Call
@@ -56,6 +64,14 @@ typedef struct sk_percpu {
* arch_early_init(): on amd64 that reloads GS, which would reset the base). */ * arch_early_init(): on amd64 that reloads GS, which would reset the base). */
void sk_percpu_boot_init(void); void sk_percpu_boot_init(void);
/* 1 once the boot core's block is installed (MC-0b). Before that the
* per-CPU register still holds whatever the firmware left there -- on
* riscv64 the monolithic loader's kernel_main() prints from
* arch_early_init() with tp still the firmware's -- so the spinlocks
* (spinlock.c) do nothing until it is set: early boot is one core with
* interrupts off. */
extern volatile int sk_percpu_ready;
/* This core's block. Valid from sk_percpu_boot_init() on. */ /* This core's block. Valid from sk_percpu_boot_init() on. */
static inline sk_percpu_t *sk_percpu(void) static inline sk_percpu_t *sk_percpu(void)
{ {
+111
View File
@@ -0,0 +1,111 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* spinlock.h - Kernel spinlocks (FABRIC-3.10.8 Stage MC-0b).
*
* One lock word: 0 = free, core_id + 1 = held by that core. Taken by
* compare-and-swap; interrupts are masked on the owning core while it
* holds any lock (the first lock taken saves the interrupt state in the
* per-core block, the last one released restores it -- percpu.h).
*
* The rule that makes this correct: **no lock is held across a context
* switch or into the idle point.** Every switch and every idle point
* checks it (sk_lock_check_none_held()) and counts a violation.
*
* Failures are counted, never silent (sk_lock_report() logs new ones):
* - recursion: this core takes a lock it already holds. On one core,
* "the owner is this core" can't tell a nested call from a second VM
* on the same core, so it is never treated as legitimate. The call is
* let through (spinning would hang the core) and counted;
* - bad release: releasing a lock this core doesn't hold;
* - held at switch: a switch or idle point reached with a lock held.
*
* sf_mutex_t (the vendored VM's dict_lock/tuning_lock) uses the same lock
* word through sk_lockword_acquire()/sk_lockword_release() (shim.c).
*
* THE KERNEL LOCKS, in acquisition order (a lock may be taken while
* holding only locks above it; MC-3's deadlock freedom rests on this):
*
* g_lock_hermes kernel-Hermes arena, queues, channels, ledger
* g_lock_switch switch-signal slot table and stamps
* (capsule_vm_switch_signal.c)
* g_lock_registry VM registry list and entries (capsule_birth.c)
* g_lock_doe DoE region (doe_region.c)
* g_lock_log log region (log_region.c)
* g_lock_console console output and line state (console.c)
* g_lock_heap kmalloc heap
* g_lock_vmm page tables (allocates tables from the PMM)
* g_lock_pmm physical page bitmap
*
* The switch table and the registry never call out while held: their walks
* copy one entry under the lock and judge it outside. Hermes hands the
* switch module its notifications after dropping its own lock (they read
* Hermes queues back). Stadium's lock comes with MC-0d; it sits below
* Hermes (Hermes admits and evicts cells).
*/
#ifndef STARKERNEL_SPINLOCK_H
#define STARKERNEL_SPINLOCK_H
#include <stdint.h>
typedef struct sk_spinlock {
volatile uint32_t owner; /* 0 = free, else core_id + 1 */
uint64_t acquired; /* updated while held, so exact */
uint64_t released;
} sk_spinlock_t;
#define SK_SPINLOCK_INIT { 0, 0, 0 }
void sk_lockword_acquire(volatile uint32_t *word);
void sk_lockword_release(volatile uint32_t *word);
void sk_spin_lock(sk_spinlock_t *l);
void sk_spin_unlock(sk_spinlock_t *l);
/* 1 if this core holds `l`. For paths that must not spin on a lock their
* own core may hold (the fatal exception printers). */
int sk_spin_held_here(const sk_spinlock_t *l);
/* Every switch and idle point: counts a violation if this core holds any
* lock. `where` names the site in the log. */
void sk_lock_check_none_held(const char *where);
/* Logs any failure counted since the last report. Call only with no lock
* held (the idle point does). */
void sk_lock_report(void);
uint64_t sk_lock_recursions(void);
uint64_t sk_lock_bad_releases(void);
uint64_t sk_lock_held_at_switch(void);
extern sk_spinlock_t g_lock_registry;
extern sk_spinlock_t g_lock_switch;
extern sk_spinlock_t g_lock_hermes;
extern sk_spinlock_t g_lock_doe;
extern sk_spinlock_t g_lock_log;
extern sk_spinlock_t g_lock_console;
extern sk_spinlock_t g_lock_heap;
extern sk_spinlock_t g_lock_pmm;
extern sk_spinlock_t g_lock_vmm;
#endif /* STARKERNEL_SPINLOCK_H */
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+110 -42
View File
@@ -44,6 +44,7 @@
#include "word_source/include/framebuffer_words.h" /* FABRIC-3.6.md task 1.8 -- register_framebuffer_words() */ #include "word_source/include/framebuffer_words.h" /* FABRIC-3.6.md task 1.8 -- register_framebuffer_words() */
#include "vm.h" #include "vm.h"
#include "platform_alloc.h" #include "platform_alloc.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_registry */
/* No LOG_LINE_MAX include-order constraint anymore: vm.h's own /* No LOG_LINE_MAX include-order constraint anymore: vm.h's own
* LOG_LINE_MAX (persistent block-log, 64) and log.h's in-memory line * LOG_LINE_MAX (persistent block-log, 64) and log.h's in-memory line
* length (renamed LOG_MSG_LINE_MAX, 256) are distinct names, so include * length (renamed LOG_MSG_LINE_MAX, 256) are distinct names, so include
@@ -101,7 +102,18 @@ static int vm_name_eq(const char *a, const char *b) {
return *a == *b; return *a == *b;
} }
/* Internal: return mutable pointer into registry node for vm_id */ /* FABRIC-3.10.8 MC-0b: g_lock_registry covers the node list (head, links,
* count) and every copy out of or single-field write into an entry. It is
* never held while calling out (Stadium, Hermes, the switch table, the
* console, vm_cleanup()): a long operation on an entry -- birth, KILL,
* reap -- runs unlocked and takes the lock only for its reads and final
* writes. Nodes are never freed after boot, so an entry pointer stays
* valid. A birth fills its embryo's entry unlocked: nothing else touches
* an entry before it is LIVE. capsule_vm_kill_all_nonmama() runs only on
* the way to a cold reset and walks unlocked. */
/* Internal: return mutable pointer into registry node for vm_id.
* g_lock_registry held. */
static VMRegistryEntry *vm_find_entry_ptr(VMUuid vm_id) { static VMRegistryEntry *vm_find_entry_ptr(VMUuid vm_id) {
vm_node_t *node = vm_registry_head; vm_node_t *node = vm_registry_head;
while (node) { while (node) {
@@ -145,16 +157,17 @@ void capsule_vm_registry_init(void *mama_vm_ptr) {
vm_node_t *mama; vm_node_t *mama;
/* Free any nodes from a previous init (defensive) */ /* Free any nodes from a previous init (defensive) */
sk_spin_lock(&g_lock_registry);
node = vm_registry_head; node = vm_registry_head;
vm_registry_head = (void *)0;
vm_registry_count = 0;
sk_spin_unlock(&g_lock_registry);
while (node) { while (node) {
next = node->next; next = node->next;
kfree(node); kfree(node);
node = next; node = next;
} }
vm_registry_head = (void *)0;
vm_registry_count = 0;
/* Mama is always the reserved, fixed vm_uuid_hera() id (item 3.8) */ /* Mama is always the reserved, fixed vm_uuid_hera() id (item 3.8) */
mama = (vm_node_t *)kmalloc(sizeof(vm_node_t)); mama = (vm_node_t *)kmalloc(sizeof(vm_node_t));
if (!mama) return; if (!mama) return;
@@ -175,8 +188,10 @@ void capsule_vm_registry_init(void *mama_vm_ptr) {
vm_name_copy(mama->entry.name, "Hera"); vm_name_copy(mama->entry.name, "Hera");
mama->next = (void *)0; mama->next = (void *)0;
sk_spin_lock(&g_lock_registry);
vm_registry_head = mama; vm_registry_head = mama;
vm_registry_count = 1; vm_registry_count = 1;
sk_spin_unlock(&g_lock_registry);
/* From this point on all console output is prefixed [Hera] */ /* From this point on all console output is prefixed [Hera] */
console_set_vm_name("Hera"); console_set_vm_name("Hera");
@@ -218,6 +233,7 @@ static VMRegistryEntry *vm_registry_alloc(void) {
node->next = (void *)0; node->next = (void *)0;
/* Append to tail */ /* Append to tail */
sk_spin_lock(&g_lock_registry);
if (!vm_registry_head) { if (!vm_registry_head) {
vm_registry_head = node; vm_registry_head = node;
} else { } else {
@@ -227,16 +243,18 @@ static VMRegistryEntry *vm_registry_alloc(void) {
} }
vm_registry_count++; vm_registry_count++;
sk_spin_unlock(&g_lock_registry);
return &node->entry; return &node->entry;
} }
int capsule_vm_registry_get(VMUuid vm_id, VMRegistryEntry *out) { int capsule_vm_registry_get(VMUuid vm_id, VMRegistryEntry *out) {
VMRegistryEntry *entry; VMRegistryEntry *entry;
if (!out) return -1; if (!out) return -1;
sk_spin_lock(&g_lock_registry);
entry = vm_find_entry_ptr(vm_id); entry = vm_find_entry_ptr(vm_id);
if (!entry) return -1; if (entry) *out = *entry;
*out = *entry; sk_spin_unlock(&g_lock_registry);
return 0; return entry ? 0 : -1;
} }
uint32_t capsule_vm_registry_count(void) { uint32_t capsule_vm_registry_count(void) {
@@ -244,15 +262,19 @@ uint32_t capsule_vm_registry_count(void) {
} }
int capsule_vm_registry_get_by_index(uint32_t index, VMRegistryEntry *out) { int capsule_vm_registry_get_by_index(uint32_t index, VMRegistryEntry *out) {
vm_node_t *node = vm_registry_head; vm_node_t *node;
uint32_t i = 0; uint32_t i = 0;
int rc = -1;
if (!out) return -1; if (!out) return -1;
sk_spin_lock(&g_lock_registry);
node = vm_registry_head;
while (node) { while (node) {
if (i == index) { *out = node->entry; return 0; } if (i == index) { *out = node->entry; rc = 0; break; }
node = node->next; node = node->next;
i++; i++;
} }
return -1; sk_spin_unlock(&g_lock_registry);
return rc;
} }
/* Live population, distinct from vm_registry_count above: vm_registry_count /* Live population, distinct from vm_registry_count above: vm_registry_count
@@ -265,27 +287,34 @@ int capsule_vm_registry_get_by_index(uint32_t index, VMRegistryEntry *out) {
* "K soak" check, capsule_vm_physics.c). Not exposed in the public header: * "K soak" check, capsule_vm_physics.c). Not exposed in the public header:
* only capsule_birth_baby's bound check needs it today. */ * only capsule_birth_baby's bound check needs it today. */
static uint32_t vm_registry_live_count(void) { static uint32_t vm_registry_live_count(void) {
vm_node_t *node = vm_registry_head; vm_node_t *node;
uint32_t live = 0; uint32_t live = 0;
sk_spin_lock(&g_lock_registry);
node = vm_registry_head;
while (node) { while (node) {
if (node->entry.state == VM_STATE_LIVE) live++; if (node->entry.state == VM_STATE_LIVE) live++;
node = node->next; node = node->next;
} }
sk_spin_unlock(&g_lock_registry);
return live; return live;
} }
int capsule_vm_find_by_name(const char *name, VMRegistryEntry *out) { int capsule_vm_find_by_name(const char *name, VMRegistryEntry *out) {
vm_node_t *node; vm_node_t *node;
int rc = -1;
if (!name || !out) return -1; if (!name || !out) return -1;
sk_spin_lock(&g_lock_registry);
node = vm_registry_head; node = vm_registry_head;
while (node) { while (node) {
if (vm_name_eq(node->entry.name, name)) { if (vm_name_eq(node->entry.name, name)) {
*out = node->entry; *out = node->entry;
return 0; rc = 0;
break;
} }
node = node->next; node = node->next;
} }
return -1; sk_spin_unlock(&g_lock_registry);
return rc;
} }
/* Fold ASCII letter to lowercase (no libc) */ /* Fold ASCII letter to lowercase (no libc) */
@@ -319,26 +348,36 @@ static int vm_name_prefix_eq_nocase(const char *str, const char *prefix) {
int capsule_vm_find_by_name_nocase(const char *name, VMRegistryEntry *out) { int capsule_vm_find_by_name_nocase(const char *name, VMRegistryEntry *out) {
vm_node_t *node; vm_node_t *node;
int rc = -1;
if (!name || !out) return -1; if (!name || !out) return -1;
sk_spin_lock(&g_lock_registry);
node = vm_registry_head; node = vm_registry_head;
while (node) { while (node) {
if (vm_name_eq_nocase(node->entry.name, name)) { if (vm_name_eq_nocase(node->entry.name, name)) {
*out = node->entry; *out = node->entry;
return 0; rc = 0;
break;
} }
node = node->next; node = node->next;
} }
return -1; sk_spin_unlock(&g_lock_registry);
return rc;
} }
void capsule_vm_set_state(VMUuid vm_id, uint32_t state) { void capsule_vm_set_state(VMUuid vm_id, uint32_t state) {
VMRegistryEntry *entry = vm_find_entry_ptr(vm_id); VMRegistryEntry *entry;
sk_spin_lock(&g_lock_registry);
entry = vm_find_entry_ptr(vm_id);
if (entry) entry->state = state; if (entry) entry->state = state;
sk_spin_unlock(&g_lock_registry);
} }
void capsule_vm_set_pending_reap(VMUuid vm_id, int pending) { void capsule_vm_set_pending_reap(VMUuid vm_id, int pending) {
VMRegistryEntry *entry = vm_find_entry_ptr(vm_id); VMRegistryEntry *entry;
sk_spin_lock(&g_lock_registry);
entry = vm_find_entry_ptr(vm_id);
if (entry) entry->pending_reap = pending; if (entry) entry->pending_reap = pending;
sk_spin_unlock(&g_lock_registry);
} }
/* A dead VM leaves kernel-Hermes: every channel, and its undelivered mail /* A dead VM leaves kernel-Hermes: every channel, and its undelivered mail
@@ -363,21 +402,28 @@ static void capsule_vm_hermes_depart(VMUuid vm_id, const char *name, int was_obs
} }
void capsule_vm_force_reap(VMUuid vm_id) { void capsule_vm_force_reap(VMUuid vm_id) {
VMRegistryEntry *entry = vm_find_entry_ptr(vm_id); VMRegistryEntry *entry;
if (!entry) return; VM *vm;
size_t patron_cell;
sk_spin_lock(&g_lock_registry);
entry = vm_find_entry_ptr(vm_id);
/* Idempotent, matching capsule_vm_kill()'s own convention -- the /* Idempotent, matching capsule_vm_kill()'s own convention -- the
* Stage 3 checkpoint that calls this cannot itself know whether some * Stage 3 checkpoint that calls this cannot itself know whether some
* other path (a later re-attach's own teardown, say) already reaped * other path (a later re-attach's own teardown, say) already reaped
* this exact VM before the checkpoint got to it. */ * this exact VM before the checkpoint got to it.
if (entry->state == VM_STATE_DEAD) return; *
* Hera is never a switch-signal participant subject to pending_reap
/* Hera is never a switch-signal participant subject to pending_reap
* in the first place, but refuse on principle anyway -- matches * in the first place, but refuse on principle anyway -- matches
* capsule_vm_kill()'s own unconditional rule. */ * capsule_vm_kill()'s own unconditional rule. */
if (vm_uuid_is_hera(entry->vm_id)) return; if (!entry || entry->state == VM_STATE_DEAD || vm_uuid_is_hera(entry->vm_id)) {
sk_spin_unlock(&g_lock_registry);
VM *vm = (VM *)entry->vm_ptr; return;
}
vm = (VM *)entry->vm_ptr;
patron_cell = entry->stadium_patron_cell;
entry->stadium_patron_cell = STADIUM_CELL_NONE;
sk_spin_unlock(&g_lock_registry);
/* Deliberately NOT capsule_vm_kill()'s VM_STATE_SWITCHED_OUT refusal /* Deliberately NOT capsule_vm_kill()'s VM_STATE_SWITCHED_OUT refusal
* -- that guard exists precisely for the case this function handles: * -- that guard exists precisely for the case this function handles:
@@ -386,9 +432,8 @@ void capsule_vm_force_reap(VMUuid vm_id) {
* native-stack context will never be resumed. Freeing it here is the * native-stack context will never be resumed. Freeing it here is the
* deferred half of the mark-and-defer design (FABRIC-3.md §XXVIII * deferred half of the mark-and-defer design (FABRIC-3.md §XXVIII
* Stage 4) -- see VMRegistryEntry.pending_reap's own doc comment. */ * Stage 4) -- see VMRegistryEntry.pending_reap's own doc comment. */
if (entry->stadium_patron_cell != STADIUM_CELL_NONE) { if (patron_cell != STADIUM_CELL_NONE) {
(void)stadium_evict(entry->stadium_patron_cell); (void)stadium_evict(patron_cell);
entry->stadium_patron_cell = STADIUM_CELL_NONE;
} }
if (vm) { if (vm) {
@@ -411,20 +456,30 @@ void capsule_vm_force_reap(VMUuid vm_id) {
console_puts(entry->name[0] ? entry->name : "(unnamed)"); console_puts(entry->name[0] ? entry->name : "(unnamed)");
console_println(" tombstone reclaimed"); console_println(" tombstone reclaimed");
sk_spin_lock(&g_lock_registry);
entry->vm_ptr = (void *)0; entry->vm_ptr = (void *)0;
entry->state = VM_STATE_DEAD; entry->state = VM_STATE_DEAD;
entry->pending_reap = 0; entry->pending_reap = 0;
for (uint32_t i = 0; i < VM_NAME_MAX; i++) entry->name[i] = '\0'; for (uint32_t i = 0; i < VM_NAME_MAX; i++) entry->name[i] = '\0';
sk_spin_unlock(&g_lock_registry);
} }
void capsule_vm_registry_set_name(VMUuid vm_id, const char *name) { void capsule_vm_registry_set_name(VMUuid vm_id, const char *name) {
VMRegistryEntry *entry; VMRegistryEntry *entry;
int first_naming; int first_naming, live;
char named[VM_NAME_MAX];
if (!name) return; if (!name) return;
sk_spin_lock(&g_lock_registry);
entry = vm_find_entry_ptr(vm_id); entry = vm_find_entry_ptr(vm_id);
if (!entry) return; if (!entry) {
sk_spin_unlock(&g_lock_registry);
return;
}
first_naming = (entry->name[0] == '\0'); first_naming = (entry->name[0] == '\0');
vm_name_copy(entry->name, name); vm_name_copy(entry->name, name);
vm_name_copy(named, entry->name);
live = (entry->state == VM_STATE_LIVE);
sk_spin_unlock(&g_lock_registry);
/* FABRIC-3.9.8 Stage M-B (§7.3): EV-BORN. capsule_birth_baby() leaves /* FABRIC-3.9.8 Stage M-B (§7.3): EV-BORN. capsule_birth_baby() leaves
* a VM unnamed; every birth path (BIRTH, WORKER-BIRTH, identity births, * a VM unnamed; every birth path (BIRTH, WORKER-BIRTH, identity births,
@@ -432,9 +487,8 @@ void capsule_vm_registry_set_name(VMUuid vm_id, const char *name) {
* first naming of a LIVE VM is the moment it's born, under the name * first naming of a LIVE VM is the moment it's born, under the name
* its subscribers know it by. Funded by the newborn (M-5), whose quota * its subscribers know it by. Funded by the newborn (M-5), whose quota
* is granted during birth; participants only (M-4). */ * is granted during birth; participants only (M-4). */
if (first_naming && entry->state == VM_STATE_LIVE && if (first_naming && live && !sk_vm_switch_signal_is_observer(vm_id)) {
!sk_vm_switch_signal_is_observer(vm_id)) { (void)sk_hermes_event(SK_HERMES_CHANNEL_EV_BORN, vm_id, vm_id, named);
(void)sk_hermes_event(SK_HERMES_CHANNEL_EV_BORN, vm_id, vm_id, entry->name);
} }
} }
@@ -494,8 +548,10 @@ int capsule_vm_kill(const char *name) {
/* Locate by name (case-insensitive) */ /* Locate by name (case-insensitive) */
{ {
vm_node_t *node = vm_registry_head; vm_node_t *node;
entry = (VMRegistryEntry *)0; entry = (VMRegistryEntry *)0;
sk_spin_lock(&g_lock_registry);
node = vm_registry_head;
while (node) { while (node) {
if (vm_name_eq_nocase(node->entry.name, name)) { if (vm_name_eq_nocase(node->entry.name, name)) {
entry = &node->entry; entry = &node->entry;
@@ -503,6 +559,7 @@ int capsule_vm_kill(const char *name) {
} }
node = node->next; node = node->next;
} }
sk_spin_unlock(&g_lock_registry);
} }
if (!entry) { if (!entry) {
@@ -544,16 +601,22 @@ int capsule_vm_kill(const char *name) {
} }
} }
{
size_t patron_cell;
sk_spin_lock(&g_lock_registry);
vm_id = entry->vm_id; vm_id = entry->vm_id;
vm = (VM *)entry->vm_ptr; vm = (VM *)entry->vm_ptr;
patron_cell = entry->stadium_patron_cell;
entry->stadium_patron_cell = STADIUM_CELL_NONE;
sk_spin_unlock(&g_lock_registry);
/* FABRIC-2.md SS B, VM-COOL: reap this VM's own Stadium patron cell for /* FABRIC-2.md SS B, VM-COOL: reap this VM's own Stadium patron cell for
* real, dispatching COOL. Refusal (already naturally reclaimed by * real, dispatching COOL. Refusal (already naturally reclaimed by
* unrelated quota pressure, or never admitted) is silently tolerated -- * unrelated quota pressure, or never admitted) is silently tolerated --
* KILL tears the VM down unconditionally either way. */ * KILL tears the VM down unconditionally either way. */
if (entry->stadium_patron_cell != STADIUM_CELL_NONE) { if (patron_cell != STADIUM_CELL_NONE) {
(void)stadium_evict(entry->stadium_patron_cell); (void)stadium_evict(patron_cell);
entry->stadium_patron_cell = STADIUM_CELL_NONE; }
} }
/* Tear down and free */ /* Tear down and free */
@@ -572,9 +635,11 @@ int capsule_vm_kill(const char *name) {
capsule_vm_hermes_depart(vm_id, entry->name, was_observer); capsule_vm_hermes_depart(vm_id, entry->name, was_observer);
} }
sk_spin_lock(&g_lock_registry);
entry->vm_ptr = (void *)0; entry->vm_ptr = (void *)0;
entry->state = VM_STATE_DEAD; entry->state = VM_STATE_DEAD;
for (i = 0; i < VM_NAME_MAX; i++) entry->name[i] = '\0'; for (i = 0; i < VM_NAME_MAX; i++) entry->name[i] = '\0';
sk_spin_unlock(&g_lock_registry);
capsule_parity_log_kill(vm_id, name); capsule_parity_log_kill(vm_id, name);
@@ -700,11 +765,14 @@ CapsuleRunResult capsule_birth_mama(
post_dict_hash); post_dict_hash);
{ {
VMRegistryEntry *mama_entry = vm_find_entry_ptr(vm_uuid_hera()); VMRegistryEntry *mama_entry;
sk_spin_lock(&g_lock_registry);
mama_entry = vm_find_entry_ptr(vm_uuid_hera());
if (mama_entry) { if (mama_entry) {
mama_entry->birth_capsule_id = mama_cap->capsule_id; mama_entry->birth_capsule_id = mama_cap->capsule_id;
mama_entry->birth_dict_hash = post_dict_hash; mama_entry->birth_dict_hash = post_dict_hash;
} }
sk_spin_unlock(&g_lock_registry);
} }
/* item 3.8: seed the vm_uuid pool now that the Mama capsule's content /* item 3.8: seed the vm_uuid pool now that the Mama capsule's content
@@ -995,9 +1063,9 @@ CapsuleRunResult capsule_birth_baby(
* capsule booted twice produces the same dict hash" determinism * capsule booted twice produces the same dict hash" determinism
* invariant this codebase relies on elsewhere. */ * invariant this codebase relies on elsewhere. */
{ {
VMRegistryEntry *parent_entry = vm_find_entry_ptr(parent); VMRegistryEntry parent_entry;
if (parent_entry && parent_entry->vm_ptr) { if (capsule_vm_registry_get(parent, &parent_entry) == 0 && parent_entry.vm_ptr) {
dictionary_snapshot_acl_from_parent(new_vm, (VM *)parent_entry->vm_ptr); dictionary_snapshot_acl_from_parent(new_vm, (VM *)parent_entry.vm_ptr);
} }
} }
@@ -39,6 +39,7 @@
#include "starkernel/vm/vm_frame.h" /* top frame kind -- in progress vs parked idle */ #include "starkernel/vm/vm_frame.h" /* top frame kind -- in progress vs parked idle */
#include "starkernel/vm/bootstrap/sk_vm_bootstrap.h" /* sk_get_mama_vm() */ #include "starkernel/vm/bootstrap/sk_vm_bootstrap.h" /* sk_get_mama_vm() */
#include "starkernel/repl.h" /* sk_console_key_available() -- Hera's mail (C-5) */ #include "starkernel/repl.h" /* sk_console_key_available() -- Hera's mail (C-5) */
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_switch */
#include <stddef.h> #include <stddef.h>
/* FABRIC-3.6.md task 3.1 (2026-09-21, ruled B2 / FABRIC-3.5.md §XLV.2): /* FABRIC-3.6.md task 3.1 (2026-09-21, ruled B2 / FABRIC-3.5.md §XLV.2):
@@ -113,6 +114,15 @@ static void console_put_u64(uint64_t v) {
console_puts(&buf[i]); console_puts(&buf[i]);
} }
/* FABRIC-3.10.8 MC-0b: g_lock_switch covers the slot table, the stamp
* counter, the decision-due flag and the switch counters. Nothing here
* calls out while holding it: the walks (the decision, the Stage H floor,
* the reap sweep) copy one slot under the lock with slot_copy() and judge
* it -- registry, Hermes queues, heat -- outside. A slot removed or shifted
* between two copies is simply judged at its new index or missed until
* the next decision, the same as a VM unregistered just before a walk. */
/* g_lock_switch held. */
static int slot_for_vm_id(VMUuid vm_id) { static int slot_for_vm_id(VMUuid vm_id) {
int i; int i;
for (i = 0; i < g_slot_count; i++) { for (i = 0; i < g_slot_count; i++) {
@@ -121,6 +131,16 @@ static int slot_for_vm_id(VMUuid vm_id) {
return -1; return -1;
} }
/* Copies slot i out; 0 once i is past the end. */
static int slot_copy(int i, sk_switch_slot_entry_t *out) {
int ok;
sk_spin_lock(&g_lock_switch);
ok = i < g_slot_count;
if (ok) *out = g_slots[i];
sk_spin_unlock(&g_lock_switch);
return ok;
}
int sk_vm_switch_signal_boot_init(void) { int sk_vm_switch_signal_boot_init(void) {
size_t max_vm_count = stadium_max_vm_count(); size_t max_vm_count = stadium_max_vm_count();
sk_switch_slot_entry_t *slots; sk_switch_slot_entry_t *slots;
@@ -137,9 +157,11 @@ int sk_vm_switch_signal_boot_init(void) {
slots[i].observer = 0; slots[i].observer = 0;
} }
sk_spin_lock(&g_lock_switch);
g_slots = slots; g_slots = slots;
g_slot_capacity = (int)max_vm_count; g_slot_capacity = (int)max_vm_count;
g_slot_count = 0; g_slot_count = 0;
sk_spin_unlock(&g_lock_switch);
console_puts("Switch-signal: "); console_puts("Switch-signal: ");
console_put_u64((uint64_t)max_vm_count); console_put_u64((uint64_t)max_vm_count);
@@ -152,64 +174,96 @@ int sk_vm_switch_signal_register(VMUuid vm_id) {
/* FABRIC-3.8.md §3.6 (Stage E): idempotent. Registration now happens /* FABRIC-3.8.md §3.6 (Stage E): idempotent. Registration now happens
* once, at birth; a second call returns the existing slot rather than * once, at birth; a second call returns the existing slot rather than
* allocating a duplicate that the decision would walk twice. */ * allocating a duplicate that the decision would walk twice. */
int existing = slot_for_vm_id(vm_id); int slot;
if (existing >= 0) return existing; sk_spin_lock(&g_lock_switch);
if (g_slot_count >= g_slot_capacity) return -1; slot = slot_for_vm_id(vm_id);
if (slot < 0 && g_slot_count < g_slot_capacity) {
g_slots[g_slot_count].vm_id = vm_id; g_slots[g_slot_count].vm_id = vm_id;
g_slots[g_slot_count].parked_stamp = 0; g_slots[g_slot_count].parked_stamp = 0;
g_slots[g_slot_count].observer = 0; g_slots[g_slot_count].observer = 0;
return g_slot_count++; slot = g_slot_count++;
}
sk_spin_unlock(&g_lock_switch);
return slot;
} }
void sk_vm_switch_signal_set_observer(VMUuid vm_id) { void sk_vm_switch_signal_set_observer(VMUuid vm_id) {
int slot = slot_for_vm_id(vm_id); int slot;
sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(vm_id);
if (slot >= 0) g_slots[slot].observer = 1; if (slot >= 0) g_slots[slot].observer = 1;
sk_spin_unlock(&g_lock_switch);
} }
int sk_vm_switch_signal_is_observer(VMUuid vm_id) { int sk_vm_switch_signal_is_observer(VMUuid vm_id) {
int slot = slot_for_vm_id(vm_id); int slot, obs;
return slot >= 0 && g_slots[slot].observer; sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(vm_id);
obs = slot >= 0 && g_slots[slot].observer;
sk_spin_unlock(&g_lock_switch);
return obs;
} }
int sk_vm_switch_signal_unregister(VMUuid vm_id) { int sk_vm_switch_signal_unregister(VMUuid vm_id) {
int slot = slot_for_vm_id(vm_id); int slot;
if (slot < 0) return -1; sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(vm_id);
if (slot >= 0) {
for (int i = slot; i < g_slot_count - 1; i++) { for (int i = slot; i < g_slot_count - 1; i++) {
g_slots[i] = g_slots[i + 1]; g_slots[i] = g_slots[i + 1];
} }
g_slot_count--; g_slot_count--;
}
sk_spin_unlock(&g_lock_switch);
/* No pending target to clear any more (Stage C-b): the decision keeps /* No pending target to clear any more (Stage C-b): the decision keeps
* no state between calls, so a removed VM simply stops being walked. */ * no state between calls, so a removed VM simply stops being walked. */
return 0; return slot >= 0 ? 0 : -1;
} }
int sk_vm_switch_signal_is_participant(VMUuid vm_id) { int sk_vm_switch_signal_is_participant(VMUuid vm_id) {
return slot_for_vm_id(vm_id) >= 0; int slot;
sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(vm_id);
sk_spin_unlock(&g_lock_switch);
return slot >= 0;
} }
uint64_t sk_vm_switch_signal_next_stamp(void) { uint64_t sk_vm_switch_signal_next_stamp(void) {
return g_stamp_next++; uint64_t s;
sk_spin_lock(&g_lock_switch);
s = g_stamp_next++;
sk_spin_unlock(&g_lock_switch);
return s;
} }
void sk_vm_switch_signal_note_parked(VMUuid vm_id) { void sk_vm_switch_signal_note_parked(VMUuid vm_id) {
int slot = slot_for_vm_id(vm_id); int slot;
sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(vm_id);
if (slot >= 0) g_slots[slot].parked_stamp = g_stamp_next++; if (slot >= 0) g_slots[slot].parked_stamp = g_stamp_next++;
sk_spin_unlock(&g_lock_switch);
} }
void sk_vm_switch_signal_note_beat(void) { void sk_vm_switch_signal_note_beat(void) {
sk_spin_lock(&g_lock_switch);
g_beats_since_switch++; g_beats_since_switch++;
g_decision_due = 1; g_decision_due = 1;
sk_spin_unlock(&g_lock_switch);
} }
void sk_vm_switch_signal_note_mail(void) { void sk_vm_switch_signal_note_mail(void) {
sk_spin_lock(&g_lock_switch);
g_decision_due = 1; g_decision_due = 1;
sk_spin_unlock(&g_lock_switch);
} }
int sk_vm_switch_signal_take_decision_due(void) { int sk_vm_switch_signal_take_decision_due(void) {
int due = g_decision_due; int due;
sk_spin_lock(&g_lock_switch);
due = g_decision_due;
g_decision_due = 0; g_decision_due = 0;
sk_spin_unlock(&g_lock_switch);
return due; return due;
} }
@@ -274,6 +328,7 @@ VM *sk_vm_switch_decide(VM *self, int self_competes) {
VM *mama = (VM *)sk_get_mama_vm(); VM *mama = (VM *)sk_get_mama_vm();
VM *best = (VM *)0; VM *best = (VM *)0;
uint64_t best_heat = 0, best_stamp = 0; uint64_t best_heat = 0, best_stamp = 0;
sk_switch_slot_entry_t s;
int i, self_obs = self && sk_vm_switch_signal_is_observer(self->stadium_vm_id); int i, self_obs = self && sk_vm_switch_signal_is_observer(self->stadium_vm_id);
/* FABRIC-3.9.0.md §2.4, ruling O-1 (Stage O-A): an observer acting on /* FABRIC-3.9.0.md §2.4, ruling O-1 (Stage O-A): an observer acting on
@@ -287,38 +342,38 @@ VM *sk_vm_switch_decide(VM *self, int self_competes) {
/* Observers with events go ahead of every participant, at this clean /* Observers with events go ahead of every participant, at this clean
* boundary; among themselves, the one waiting longest (unique stamps, * boundary; among themselves, the one waiting longest (unique stamps,
* so still independent of walk order). No heat involved. */ * so still independent of walk order). No heat involved. */
for (i = 0; i < g_slot_count; i++) { for (i = 0; slot_copy(i, &s); i++) {
VMRegistryEntry e; VMRegistryEntry e;
VM *v; VM *v;
uint64_t stamp; uint64_t stamp;
if (!g_slots[i].observer) continue; if (!s.observer) continue;
if (self && vm_uuid_equal(g_slots[i].vm_id, self->stadium_vm_id)) continue; if (self && vm_uuid_equal(s.vm_id, self->stadium_vm_id)) continue;
if (capsule_vm_registry_get(g_slots[i].vm_id, &e) != 0) continue; if (capsule_vm_registry_get(s.vm_id, &e) != 0) continue;
v = (VM *)e.vm_ptr; v = (VM *)e.vm_ptr;
if (!v || v == self || e.pending_reap) continue; if (!v || v == self || e.pending_reap) continue;
if (e.state == VM_STATE_DEAD || e.state == VM_STATE_STILLBORN) continue; if (e.state == VM_STATE_DEAD || e.state == VM_STATE_STILLBORN) continue;
if (v->native_stack_top == 0) continue; if (v->native_stack_top == 0) continue;
if (!observer_event(v, mama, g_slots[i].parked_stamp, &stamp)) continue; if (!observer_event(v, mama, s.parked_stamp, &stamp)) continue;
if (!best || stamp < best_stamp) { best = v; best_stamp = stamp; } if (!best || stamp < best_stamp) { best = v; best_stamp = stamp; }
} }
if (best) return best; if (best) return best;
/* Participants: the FABRIC-3.8 compudynamics, unchanged -- observers /* Participants: the FABRIC-3.8 compudynamics, unchanged -- observers
* are never heat candidates. */ * are never heat candidates. */
for (i = 0; i < g_slot_count; i++) { for (i = 0; slot_copy(i, &s); i++) {
VMRegistryEntry e; VMRegistryEntry e;
VM *v; VM *v;
uint64_t heat, stamp; uint64_t heat, stamp;
if (g_slots[i].observer) continue; if (s.observer) continue;
if (self && vm_uuid_equal(g_slots[i].vm_id, self->stadium_vm_id)) continue; if (self && vm_uuid_equal(s.vm_id, self->stadium_vm_id)) continue;
if (capsule_vm_registry_get(g_slots[i].vm_id, &e) != 0) continue; if (capsule_vm_registry_get(s.vm_id, &e) != 0) continue;
v = (VM *)e.vm_ptr; v = (VM *)e.vm_ptr;
if (!v || v == self || e.pending_reap) continue; if (!v || v == self || e.pending_reap) continue;
if (e.state == VM_STATE_DEAD || e.state == VM_STATE_STILLBORN) continue; if (e.state == VM_STATE_DEAD || e.state == VM_STATE_STILLBORN) continue;
if (v->native_stack_top == 0) continue; if (v->native_stack_top == 0) continue;
if (!candidate_standing(v, mama, g_slots[i].parked_stamp, &heat, &stamp)) continue; if (!candidate_standing(v, mama, s.parked_stamp, &heat, &stamp)) continue;
/* Coolest wins; among equally cool, the one waiting longest. /* Coolest wins; among equally cool, the one waiting longest.
* Stamps are unique, so the winner doesn't depend on the order * Stamps are unique, so the winner doesn't depend on the order
@@ -347,6 +402,7 @@ void sk_vm_switch_signal_note_work_gained(VMUuid vm_id) {
VM *mama = (VM *)sk_get_mama_vm(); VM *mama = (VM *)sk_get_mama_vm();
VMRegistryEntry e; VMRegistryEntry e;
VM *v; VM *v;
sk_switch_slot_entry_t s;
uint64_t floor_q48 = 0; uint64_t floor_q48 = 0;
int have = 0, i; int have = 0, i;
@@ -371,17 +427,17 @@ void sk_vm_switch_signal_note_work_gained(VMUuid vm_id) {
floor_q48 = vm_physics_work_heat_of(current->stadium_vm_id); floor_q48 = vm_physics_work_heat_of(current->stadium_vm_id);
have = 1; have = 1;
} }
for (i = 0; i < g_slot_count; i++) { for (i = 0; slot_copy(i, &s); i++) {
VMRegistryEntry o; VMRegistryEntry o;
VM *ov; VM *ov;
uint64_t heat, stamp; uint64_t heat, stamp;
if (g_slots[i].observer) continue; if (s.observer) continue;
if (vm_uuid_equal(g_slots[i].vm_id, vm_id)) continue; if (vm_uuid_equal(s.vm_id, vm_id)) continue;
if (capsule_vm_registry_get(g_slots[i].vm_id, &o) != 0) continue; if (capsule_vm_registry_get(s.vm_id, &o) != 0) continue;
ov = (VM *)o.vm_ptr; ov = (VM *)o.vm_ptr;
if (!ov || ov == current || o.pending_reap) continue; if (!ov || ov == current || o.pending_reap) continue;
if (o.state == VM_STATE_DEAD || o.state == VM_STATE_STILLBORN) continue; if (o.state == VM_STATE_DEAD || o.state == VM_STATE_STILLBORN) continue;
if (!candidate_standing(ov, mama, g_slots[i].parked_stamp, &heat, &stamp)) continue; if (!candidate_standing(ov, mama, s.parked_stamp, &heat, &stamp)) continue;
if (!have || heat < floor_q48) { floor_q48 = heat; have = 1; } if (!have || heat < floor_q48) { floor_q48 = heat; have = 1; }
} }
if (have) vm_physics_work_heat_floor(vm_id, floor_q48); if (have) vm_physics_work_heat_floor(vm_id, floor_q48);
@@ -389,14 +445,15 @@ void sk_vm_switch_signal_note_work_gained(VMUuid vm_id) {
void sk_vm_switch_reap_pending(void) { void sk_vm_switch_reap_pending(void) {
VM *current = sk_vm_switch_current_vm(); VM *current = sk_vm_switch_current_vm();
sk_switch_slot_entry_t s;
int i; int i;
/* One per call: capsule_vm_force_reap() unregisters the VM, which /* One per call: capsule_vm_force_reap() unregisters the VM, which
* compacts this table under the walk. Every safe point calls this, so * compacts this table under the walk. Every safe point calls this, so
* any further pending reaps are picked up at the next one. */ * any further pending reaps are picked up at the next one. */
for (i = 0; i < g_slot_count; i++) { for (i = 0; slot_copy(i, &s); i++) {
VMRegistryEntry e; VMRegistryEntry e;
if (capsule_vm_registry_get(g_slots[i].vm_id, &e) == 0 && if (capsule_vm_registry_get(s.vm_id, &e) == 0 &&
e.pending_reap && e.vm_ptr != (void *)current) { e.pending_reap && e.vm_ptr != (void *)current) {
capsule_vm_force_reap(e.vm_id); capsule_vm_force_reap(e.vm_id);
return; return;
@@ -407,8 +464,10 @@ void sk_vm_switch_reap_pending(void) {
void sk_vm_switch_signal_note_switch_performed(VMUuid target_id) { void sk_vm_switch_signal_note_switch_performed(VMUuid target_id) {
/* Stage C-b: counters only -- there is no readiness left to reset. */ /* Stage C-b: counters only -- there is no readiness left to reset. */
(void)target_id; (void)target_id;
sk_spin_lock(&g_lock_switch);
g_switch_count++; g_switch_count++;
g_beats_since_switch = 0; g_beats_since_switch = 0;
sk_spin_unlock(&g_lock_switch);
} }
uint64_t sk_vm_switch_signal_switch_count(void) { uint64_t sk_vm_switch_signal_switch_count(void) {
@@ -421,7 +480,13 @@ uint32_t sk_vm_switch_signal_ticks_since_switch(void) {
int sk_vm_switch_signal_current_slot(void) { int sk_vm_switch_signal_current_slot(void) {
VM *current = sk_vm_switch_current_vm(); VM *current = sk_vm_switch_current_vm();
return current ? slot_for_vm_id(current->stadium_vm_id) : -1; int slot = -1;
if (current) {
sk_spin_lock(&g_lock_switch);
slot = slot_for_vm_id(current->stadium_vm_id);
sk_spin_unlock(&g_lock_switch);
}
return slot;
} }
int sk_vm_switch_signal_slot_count(void) { int sk_vm_switch_signal_slot_count(void) {
+26 -2
View File
@@ -36,6 +36,7 @@
#include <string.h> #include <string.h>
#include "block_subsystem.h" /* blk_meta_zone_read/write, compute_crc64 */ #include "block_subsystem.h" /* blk_meta_zone_read/write, compute_crc64 */
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_doe */
static uint64_t doe_region_ctrl_compute_crc(const doe_region_ctrl_t *ctrl) { static uint64_t doe_region_ctrl_compute_crc(const doe_region_ctrl_t *ctrl) {
size_t crc_span = offsetof(doe_region_ctrl_t, hdr_crc); size_t crc_span = offsetof(doe_region_ctrl_t, hdr_crc);
@@ -119,7 +120,9 @@ static int doe_region_read_slot(uint32_t slot_index, doe_trial_slot_t *out) {
return 0; return 0;
} }
int doe_region_append(uint32_t run_id, uint32_t cfg, uint32_t rep, uint32_t nw, /* FABRIC-3.10.8 MC-0b: append and count run under g_lock_doe (the public
* functions below); these are their bodies. */
static int doe_region_append_locked(uint32_t run_id, uint32_t cfg, uint32_t rep, uint32_t nw,
uint32_t mode, uint32_t fail_count, uint32_t mode, uint32_t fail_count,
uint64_t fleet_k_q48, uint32_t fleet_conserved, uint64_t fleet_k_q48, uint32_t fleet_conserved,
uint64_t switch_count_cumulative, uint64_t switch_count_cumulative,
@@ -174,7 +177,20 @@ int doe_region_append(uint32_t run_id, uint32_t cfg, uint32_t rep, uint32_t nw,
return doe_region_ctrl_write(&ctrl) == 0 ? 0 : -1; return doe_region_ctrl_write(&ctrl) == 0 ? 0 : -1;
} }
uint32_t doe_region_count_for_isa(const char *isa, uint32_t isa_len) { int doe_region_append(uint32_t run_id, uint32_t cfg, uint32_t rep, uint32_t nw,
uint32_t mode, uint32_t fail_count,
uint64_t fleet_k_q48, uint32_t fleet_conserved,
uint64_t switch_count_cumulative,
const char *isa, uint32_t isa_len) {
int rc;
sk_spin_lock(&g_lock_doe);
rc = doe_region_append_locked(run_id, cfg, rep, nw, mode, fail_count, fleet_k_q48,
fleet_conserved, switch_count_cumulative, isa, isa_len);
sk_spin_unlock(&g_lock_doe);
return rc;
}
static uint32_t doe_region_count_for_isa_locked(const char *isa, uint32_t isa_len) {
doe_region_ctrl_t ctrl; doe_region_ctrl_t ctrl;
if (doe_region_ctrl_read(&ctrl) != 0) return 0; if (doe_region_ctrl_read(&ctrl) != 0) return 0;
@@ -202,3 +218,11 @@ uint32_t doe_region_count_for_isa(const char *isa, uint32_t isa_len) {
} }
return count; return count;
} }
uint32_t doe_region_count_for_isa(const char *isa, uint32_t isa_len) {
uint32_t n;
sk_spin_lock(&g_lock_doe);
n = doe_region_count_for_isa_locked(isa, isa_len);
sk_spin_unlock(&g_lock_doe);
return n;
}
+49 -5
View File
@@ -49,6 +49,7 @@
#include "framebuffer.h" #include "framebuffer.h"
#include "vt100.h" #include "vt100.h"
#include "arch.h" #include "arch.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_console */
#if defined(__x86_64__) || defined(__i386__) #if defined(__x86_64__) || defined(__i386__)
#define SERIAL_SUPPORTED 1 #define SERIAL_SUPPORTED 1
@@ -178,18 +179,32 @@ static const char *g_active_vm_name = (void *)0;
static int g_line_start = 1; static int g_line_start = 1;
static uint64_t g_console_tx_count; /* monotonic console_putc() counter (BSS) */ static uint64_t g_console_tx_count; /* monotonic console_putc() counter (BSS) */
/* FABRIC-3.10.8 MC-0b: g_lock_console covers the output path and the line
* state above (name, line start, pending line close, the framebuffer
* terminal behind it). A string goes out under one hold, so lines from two
* cores never interleave mid-call. Unlocked on purpose:
* - console_get_vm_name(): a pointer read, and the prefix provider
* (repl.c) calls it from inside emit_prefix(), i.e. under the lock;
* - console_fb_init()/console_fb_enable_ttf(): boot-time, and they log
* (log_message() prints through this lock);
* - input (console_poll()/console_getc()). */
void console_set_vm_name(const char *name) { void console_set_vm_name(const char *name) {
/* Empty string treated the same as NULL: console_save_vm_name() /* Empty string treated the same as NULL: console_save_vm_name()
* writes "" for "there was no active name," so this keeps that * writes "" for "there was no active name," so this keeps that
* round-trip correct (save-empty then restore-empty must mean * round-trip correct (save-empty then restore-empty must mean
* "still no prefix," not "prefix is now the empty string"). */ * "still no prefix," not "prefix is now the empty string"). */
if (!name || !name[0]) { g_active_vm_name = (void *)0; return; } sk_spin_lock(&g_lock_console);
if (!name || !name[0]) {
g_active_vm_name = (void *)0;
} else {
size_t i; size_t i;
for (i = 0; i < CONSOLE_VM_NAME_BUF - 1u && name[i]; i++) for (i = 0; i < CONSOLE_VM_NAME_BUF - 1u && name[i]; i++)
g_active_vm_name_buf[i] = name[i]; g_active_vm_name_buf[i] = name[i];
g_active_vm_name_buf[i] = '\0'; g_active_vm_name_buf[i] = '\0';
g_active_vm_name = g_active_vm_name_buf; g_active_vm_name = g_active_vm_name_buf;
} }
sk_spin_unlock(&g_lock_console);
}
const char *console_get_vm_name(void) { const char *console_get_vm_name(void) {
return g_active_vm_name; return g_active_vm_name;
@@ -210,10 +225,12 @@ void console_set_user_prefix_provider(console_user_prefix_fn fn) {
void console_save_vm_name(char *out, size_t cap) { void console_save_vm_name(char *out, size_t cap) {
if (!out || cap == 0) return; if (!out || cap == 0) return;
size_t i = 0; size_t i = 0;
sk_spin_lock(&g_lock_console);
if (g_active_vm_name) { if (g_active_vm_name) {
for (; i < cap - 1u && g_active_vm_name[i]; i++) for (; i < cap - 1u && g_active_vm_name[i]; i++)
out[i] = g_active_vm_name[i]; out[i] = g_active_vm_name[i];
} }
sk_spin_unlock(&g_lock_console);
out[i] = '\0'; out[i] = '\0';
} }
@@ -302,7 +319,8 @@ static void console_putc_inner(char c);
* its own. */ * its own. */
static int g_pending_line_close = 0; static int g_pending_line_close = 0;
void console_putc(char c) { /* console_putc() with g_lock_console already held. */
static void console_putc_locked(char c) {
if (g_pending_line_close) { if (g_pending_line_close) {
g_pending_line_close = 0; g_pending_line_close = 0;
g_console_tx_count++; g_console_tx_count++;
@@ -312,6 +330,12 @@ void console_putc(char c) {
console_putc_inner(c); console_putc_inner(c);
} }
void console_putc(char c) {
sk_spin_lock(&g_lock_console);
console_putc_locked(c);
sk_spin_unlock(&g_lock_console);
}
static void console_putc_inner(char c) static void console_putc_inner(char c)
{ {
/* --- serial UART path (always on) --- */ /* --- serial UART path (always on) --- */
@@ -337,9 +361,11 @@ static void console_putc_inner(char c)
void console_puts(const char *s) { void console_puts(const char *s) {
if (!s) return; if (!s) return;
sk_spin_lock(&g_lock_console);
while (*s) { while (*s) {
console_putc(*s++); console_putc_locked(*s++);
} }
sk_spin_unlock(&g_lock_console);
} }
uint64_t console_tx_count(void) uint64_t console_tx_count(void)
@@ -349,23 +375,31 @@ uint64_t console_tx_count(void)
void console_ensure_line_start(void) void console_ensure_line_start(void)
{ {
sk_spin_lock(&g_lock_console);
if (!g_line_start) if (!g_line_start)
{ {
g_pending_line_close = 1; g_pending_line_close = 1;
} }
sk_spin_unlock(&g_lock_console);
} }
void console_cancel_deferred_line_start(void) void console_cancel_deferred_line_start(void)
{ {
sk_spin_lock(&g_lock_console);
g_pending_line_close = 0; g_pending_line_close = 0;
sk_spin_unlock(&g_lock_console);
} }
/** /**
* Write a string with newline to serial console * Write a string with newline to serial console
*/ */
void console_println(const char *s) { void console_println(const char *s) {
console_puts(s); sk_spin_lock(&g_lock_console);
console_putc('\n'); if (s) {
while (*s) console_putc_locked(*s++);
}
console_putc_locked('\n');
sk_spin_unlock(&g_lock_console);
} }
/** /**
@@ -409,37 +443,47 @@ void console_fb_enable_ttf(void)
void console_fb_scroll_back(uint32_t n) void console_fb_scroll_back(uint32_t n)
{ {
sk_spin_lock(&g_lock_console);
if (fb_is_available()) { if (fb_is_available()) {
vt100_scroll_back(n); vt100_scroll_back(n);
} }
sk_spin_unlock(&g_lock_console);
} }
void console_fb_scroll_fwd(uint32_t n) void console_fb_scroll_fwd(uint32_t n)
{ {
sk_spin_lock(&g_lock_console);
if (fb_is_available()) { if (fb_is_available()) {
vt100_scroll_fwd(n); vt100_scroll_fwd(n);
} }
sk_spin_unlock(&g_lock_console);
} }
void console_fb_toggle_graphics(void) void console_fb_toggle_graphics(void)
{ {
sk_spin_lock(&g_lock_console);
if (fb_is_available()) { if (fb_is_available()) {
vt100_toggle_graphics(); vt100_toggle_graphics();
} }
sk_spin_unlock(&g_lock_console);
} }
void console_fb_draw_cursor(void) void console_fb_draw_cursor(void)
{ {
sk_spin_lock(&g_lock_console);
if (fb_is_available()) { if (fb_is_available()) {
vt100_draw_cursor(); vt100_draw_cursor();
} }
sk_spin_unlock(&g_lock_console);
} }
void console_fb_erase_cursor(void) void console_fb_erase_cursor(void)
{ {
sk_spin_lock(&g_lock_console);
if (fb_is_available()) { if (fb_is_available()) {
vt100_erase_cursor(); vt100_erase_cursor();
} }
sk_spin_unlock(&g_lock_console);
} }
/** /**
+7
View File
@@ -31,10 +31,17 @@
* Secondary cores get theirs as they come online (MC-1). */ * Secondary cores get theirs as they come online (MC-1). */
static sk_percpu_t g_percpu_boot; static sk_percpu_t g_percpu_boot;
volatile int sk_percpu_ready;
void sk_percpu_boot_init(void) void sk_percpu_boot_init(void)
{ {
g_percpu_boot.self = &g_percpu_boot; g_percpu_boot.self = &g_percpu_boot;
g_percpu_boot.core_id = 0; g_percpu_boot.core_id = 0;
g_percpu_boot.current_vm = (struct VM *)0; g_percpu_boot.current_vm = (struct VM *)0;
g_percpu_boot.lock_depth = 0;
g_percpu_boot.saved_irq = 0;
g_percpu_boot.lock_acquired = 0;
g_percpu_boot.lock_released = 0;
sk_percpu_install(&g_percpu_boot); sk_percpu_install(&g_percpu_boot);
sk_percpu_ready = 1;
} }
+227
View File
@@ -0,0 +1,227 @@
/*
StarForth — Steady-State Virtual Machine Runtime
Copyright (c) 2023–2025 Robert A. James
All rights reserved.
This file is part of the StarForth project.
Licensed under the StarForth License, Version 1.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at:
https://github.com/star.4th@proton.me/StarForth/LICENSE.txt
This software is provided "AS IS", WITHOUT WARRANTY OF ANY KIND,
express or implied, including but not limited to the warranties of
merchantability, fitness for a particular purpose, and noninfringement.
See the License for the specific language governing permissions and
limitations under the License.
*/
/**
* spinlock.c - Kernel spinlocks (FABRIC-3.10.8 Stage MC-0b). See
* starkernel/spinlock.h for the rules and the lock order.
*/
#include "starkernel/spinlock.h"
#include "starkernel/percpu.h"
#include "starkernel/arch.h"
#include "log.h"
sk_spinlock_t g_lock_registry = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_switch = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_hermes = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_doe = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_log = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_console = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_heap = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_pmm = SK_SPINLOCK_INIT;
sk_spinlock_t g_lock_vmm = SK_SPINLOCK_INIT;
/* Failure counters (spinlock.h). Written with atomics: a failure can be
* counted on any core. The first site of each kind is kept for the log. */
static uint64_t g_recursions;
static uint64_t g_bad_releases;
static uint64_t g_held_at_switch;
static const volatile uint32_t *g_recursion_word;
static void *g_recursion_site;
static const volatile uint32_t *g_bad_release_word;
static void *g_bad_release_site;
static const char *g_held_at_switch_where;
static uint64_t g_reported_rec, g_reported_bad, g_reported_held;
/* ---- interrupt state: save-and-mask / restore, per ISA ---------------- */
static inline uint64_t irq_save_mask(void)
{
uint64_t f;
#if defined(__x86_64__)
__asm__ volatile ("pushfq; popq %0; cli" : "=r"(f) : : "memory");
#elif defined(__aarch64__)
__asm__ volatile ("mrs %0, daif; msr daifset, #2" : "=r"(f) : : "memory");
#elif defined(__riscv)
__asm__ volatile ("csrrci %0, sstatus, 2" : "=r"(f) : : "memory");
#else
#error "spinlock.c: unsupported architecture"
#endif
return f;
}
static inline void irq_restore(uint64_t f)
{
#if defined(__x86_64__)
if (f & 0x200u) __asm__ volatile ("sti" : : : "memory"); /* IF */
#elif defined(__aarch64__)
__asm__ volatile ("msr daif, %0" : : "r"(f) : "memory");
#elif defined(__riscv)
if (f & 0x2u) __asm__ volatile ("csrsi sstatus, 2" : : : "memory"); /* SIE */
#endif
}
/* ---- the lock word ---------------------------------------------------- */
static void lockword_acquire(volatile uint32_t *w, void *site)
{
sk_percpu_t *pc;
uint32_t me;
uint64_t f;
if (!sk_percpu_ready) return; /* early boot: one core, interrupts off */
pc = sk_percpu();
me = pc->core_id + 1u;
f = irq_save_mask();
if (pc->lock_depth++ == 0) pc->saved_irq = f;
pc->lock_acquired++;
if (*w == me) {
/* Already ours: spinning would hang this core for good. Let it
* through and count it (spinlock.h). */
if (__atomic_fetch_add(&g_recursions, 1, __ATOMIC_RELAXED) == 0) {
g_recursion_word = w;
g_recursion_site = site;
}
return;
}
for (;;) {
uint32_t expected = 0;
if (__atomic_compare_exchange_n(w, &expected, me, 0,
__ATOMIC_ACQUIRE, __ATOMIC_RELAXED)) {
return;
}
while (*w != 0) arch_relax();
}
}
static void lockword_release(volatile uint32_t *w, void *site)
{
sk_percpu_t *pc;
uint32_t me;
if (!sk_percpu_ready) return; /* early boot: see lockword_acquire() */
pc = sk_percpu();
me = pc->core_id + 1u;
if (*w == me) {
__atomic_store_n(w, 0u, __ATOMIC_RELEASE);
} else if (__atomic_fetch_add(&g_bad_releases, 1, __ATOMIC_RELAXED) == 0) {
g_bad_release_word = w;
g_bad_release_site = site;
}
pc->lock_released++;
if (pc->lock_depth == 0) return; /* counted above: nothing was held */
if (--pc->lock_depth == 0) irq_restore(pc->saved_irq);
}
void sk_lockword_acquire(volatile uint32_t *word)
{
lockword_acquire(word, __builtin_return_address(0));
}
void sk_lockword_release(volatile uint32_t *word)
{
lockword_release(word, __builtin_return_address(0));
}
void sk_spin_lock(sk_spinlock_t *l)
{
lockword_acquire(&l->owner, __builtin_return_address(0));
l->acquired++;
}
void sk_spin_unlock(sk_spinlock_t *l)
{
l->released++;
lockword_release(&l->owner, __builtin_return_address(0));
}
int sk_spin_held_here(const sk_spinlock_t *l)
{
return sk_percpu_ready && l->owner == sk_percpu()->core_id + 1u;
}
void sk_lock_check_none_held(const char *where)
{
if (!sk_percpu_ready || sk_percpu()->lock_depth == 0) return;
if (__atomic_fetch_add(&g_held_at_switch, 1, __ATOMIC_RELAXED) == 0) {
g_held_at_switch_where = where;
}
}
uint64_t sk_lock_recursions(void) { return g_recursions; }
uint64_t sk_lock_bad_releases(void) { return g_bad_releases; }
uint64_t sk_lock_held_at_switch(void) { return g_held_at_switch; }
/* ---- reporting -------------------------------------------------------- */
static const char *lock_name(const volatile uint32_t *w)
{
if (w == &g_lock_registry.owner) return "registry";
if (w == &g_lock_switch.owner) return "switch";
if (w == &g_lock_hermes.owner) return "hermes";
if (w == &g_lock_doe.owner) return "doe";
if (w == &g_lock_log.owner) return "log";
if (w == &g_lock_console.owner) return "console";
if (w == &g_lock_heap.owner) return "heap";
if (w == &g_lock_pmm.owner) return "pmm";
if (w == &g_lock_vmm.owner) return "vmm";
return "vm sf_mutex";
}
/* Freestanding kvsnprintf has no %llx: hex by hand. */
static void hex64(char out[19], uint64_t v)
{
int i;
out[0] = '0'; out[1] = 'x';
for (i = 17; i >= 2; i--) {
uint32_t n = (uint32_t)(v & 0xFu);
out[i] = (char)(n < 10u ? '0' + n : 'a' + (n - 10u));
v >>= 4;
}
out[18] = '\0';
}
void sk_lock_report(void)
{
char a[19];
if (g_recursions != g_reported_rec) {
g_reported_rec = g_recursions;
hex64(a, (uint64_t)(uintptr_t)g_recursion_site);
log_message(LOG_ERROR, "LOCK: recursion x%llu (first: %s lock, taken again from %s)",
(unsigned long long)g_recursions, lock_name(g_recursion_word), a);
}
if (g_bad_releases != g_reported_bad) {
g_reported_bad = g_bad_releases;
hex64(a, (uint64_t)(uintptr_t)g_bad_release_site);
log_message(LOG_ERROR, "LOCK: release of a lock not held x%llu (first: %s lock, from %s)",
(unsigned long long)g_bad_releases, lock_name(g_bad_release_word), a);
}
if (g_held_at_switch != g_reported_held) {
g_reported_held = g_held_at_switch;
log_message(LOG_ERROR, "LOCK: lock held at a switch or idle point x%llu (first: %s)",
(unsigned long long)g_held_at_switch,
g_held_at_switch_where ? g_held_at_switch_where : "?");
}
}
+14 -1
View File
@@ -37,6 +37,7 @@
#include <string.h> #include <string.h>
#include "block_subsystem.h" /* blk_meta_zone_read/write, compute_crc64 */ #include "block_subsystem.h" /* blk_meta_zone_read/write, compute_crc64 */
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_log */
static uint64_t log_region_ctrl_compute_crc(const log_region_ctrl_t *ctrl) { static uint64_t log_region_ctrl_compute_crc(const log_region_ctrl_t *ctrl) {
size_t crc_span = offsetof(log_region_ctrl_t, hdr_crc); size_t crc_span = offsetof(log_region_ctrl_t, hdr_crc);
@@ -106,7 +107,9 @@ static int log_region_write_slot(uint32_t slot_index, const log_slot_t *slot) {
return blk_meta_zone_write(dft, devblock_buf) == 0 ? 0 : -1; return blk_meta_zone_write(dft, devblock_buf) == 0 ? 0 : -1;
} }
int log_region_append(uint8_t level, uint64_t timestamp, /* FABRIC-3.10.8 MC-0b: the control block's read-modify-write and the slot
* write are one step under g_lock_log (log_region_append() below). */
static int log_region_append_locked(uint8_t level, uint64_t timestamp,
const char *source, uint32_t source_len, const char *source, uint32_t source_len,
const char *msg, uint32_t msg_len) { const char *msg, uint32_t msg_len) {
log_region_ctrl_t ctrl; log_region_ctrl_t ctrl;
@@ -176,3 +179,13 @@ int log_region_append(uint8_t level, uint64_t timestamp,
return log_region_ctrl_write(&ctrl); return log_region_ctrl_write(&ctrl);
} }
int log_region_append(uint8_t level, uint64_t timestamp,
const char *source, uint32_t source_len,
const char *msg, uint32_t msg_len) {
int rc;
sk_spin_lock(&g_lock_log);
rc = log_region_append_locked(level, timestamp, source, source_len, msg, msg_len);
sk_spin_unlock(&g_lock_log);
return rc;
}
+27 -20
View File
@@ -52,6 +52,7 @@
#include <stddef.h> #include <stddef.h>
#include "kmalloc.h" #include "kmalloc.h"
#include "pmm.h" #include "pmm.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_heap */
#define KMALLOC_DEFAULT_HEAP_SIZE (2ULL * 1024ULL * 1024ULL * 1024ULL) /* 2 GiB — floor for 256+ baby VMs */ #define KMALLOC_DEFAULT_HEAP_SIZE (2ULL * 1024ULL * 1024ULL * 1024ULL) /* 2 GiB — floor for 256+ baby VMs */
#define KMALLOC_MIN_ALIGN 16u #define KMALLOC_MIN_ALIGN 16u
@@ -292,21 +293,20 @@ void *kmalloc_aligned(size_t size, size_t align)
return NULL; return NULL;
} }
heap_block_t *cur = heap_head; heap_block_t *cur;
void *ptr = NULL;
size_t effective_align = normalize_alignment(align); size_t effective_align = normalize_alignment(align);
while (cur) { sk_spin_lock(&g_lock_heap);
for (cur = heap_head; cur; cur = cur->next) {
if (cur->free) { if (cur->free) {
void *ptr = allocate_from_block(cur, size, effective_align); ptr = allocate_from_block(cur, size, effective_align);
if (ptr) { if (ptr) break;
}
}
sk_spin_unlock(&g_lock_heap);
return ptr; return ptr;
} }
}
cur = cur->next;
}
return NULL;
}
void kfree(void *ptr) void kfree(void *ptr)
{ {
@@ -320,31 +320,34 @@ void kfree(void *ptr)
kmalloc_prefix_t *prefix = kmalloc_prefix_t *prefix =
(kmalloc_prefix_t *)((uint8_t *)ptr - sizeof(kmalloc_prefix_t)); (kmalloc_prefix_t *)((uint8_t *)ptr - sizeof(kmalloc_prefix_t));
if (prefix->magic != KMALLOC_MAGIC || !prefix->block) {
return;
}
sk_spin_lock(&g_lock_heap);
if (prefix->magic == KMALLOC_MAGIC && prefix->block && !prefix->block->free) {
heap_block_t *block = prefix->block; heap_block_t *block = prefix->block;
if (block->free) {
return;
}
account_free(block->requested); account_free(block->requested);
block->requested = 0; block->requested = 0;
block->free = 1; block->free = 1;
coalesce_neighbors(block); coalesce_neighbors(block);
} }
sk_spin_unlock(&g_lock_heap);
}
kmalloc_stats_t kmalloc_get_stats(void) kmalloc_stats_t kmalloc_get_stats(void)
{ {
return heap_stats; kmalloc_stats_t s;
sk_spin_lock(&g_lock_heap);
s = heap_stats;
sk_spin_unlock(&g_lock_heap);
return s;
} }
void kmalloc_debug_census(size_t *out_largest_free, size_t *out_free_count, void kmalloc_debug_census(size_t *out_largest_free, size_t *out_free_count,
size_t *out_used_count, size_t *out_total_blocks) size_t *out_used_count, size_t *out_total_blocks)
{ {
size_t largest = 0, free_count = 0, used_count = 0, total = 0; size_t largest = 0, free_count = 0, used_count = 0, total = 0;
heap_block_t *cur = heap_head; heap_block_t *cur;
sk_spin_lock(&g_lock_heap);
cur = heap_head;
while (cur) { while (cur) {
total++; total++;
if (cur->free) { if (cur->free) {
@@ -355,6 +358,7 @@ void kmalloc_debug_census(size_t *out_largest_free, size_t *out_free_count,
} }
cur = cur->next; cur = cur->next;
} }
sk_spin_unlock(&g_lock_heap);
if (out_largest_free) *out_largest_free = largest; if (out_largest_free) *out_largest_free = largest;
if (out_free_count) *out_free_count = free_count; if (out_free_count) *out_free_count = free_count;
if (out_used_count) *out_used_count = used_count; if (out_used_count) *out_used_count = used_count;
@@ -364,7 +368,9 @@ void kmalloc_debug_census(size_t *out_largest_free, size_t *out_free_count,
void kmalloc_debug_census_bytes(size_t *out_total_free_bytes, size_t *out_total_used_bytes) void kmalloc_debug_census_bytes(size_t *out_total_free_bytes, size_t *out_total_used_bytes)
{ {
size_t free_bytes = 0, used_bytes = 0; size_t free_bytes = 0, used_bytes = 0;
heap_block_t *cur = heap_head; heap_block_t *cur;
sk_spin_lock(&g_lock_heap);
cur = heap_head;
while (cur) { while (cur) {
if (cur->free) { if (cur->free) {
free_bytes += cur->size; free_bytes += cur->size;
@@ -373,6 +379,7 @@ void kmalloc_debug_census_bytes(size_t *out_total_free_bytes, size_t *out_total_
} }
cur = cur->next; cur = cur->next;
} }
sk_spin_unlock(&g_lock_heap);
if (out_total_free_bytes) *out_total_free_bytes = free_bytes; if (out_total_free_bytes) *out_total_free_bytes = free_bytes;
if (out_total_used_bytes) *out_total_used_bytes = used_bytes; if (out_total_used_bytes) *out_total_used_bytes = used_bytes;
} }
+15 -21
View File
@@ -51,6 +51,7 @@
#include "pmm.h" #include "pmm.h"
#include "console.h" #include "console.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_pmm */
#define PMM_MAX_BITMAP_BYTES (2u * 1024u * 1024u) /* Tracks up to ~64 GiB */ #define PMM_MAX_BITMAP_BYTES (2u * 1024u * 1024u) /* Tracks up to ~64 GiB */
#define PMM_BITMAP_BITS (PMM_MAX_BITMAP_BYTES * 8ull) #define PMM_BITMAP_BITS (PMM_MAX_BITMAP_BYTES * 8ull)
@@ -313,38 +314,27 @@ int pmm_is_initialized(void) {
} }
uint64_t pmm_alloc_page(void) { uint64_t pmm_alloc_page(void) {
if (!pmm_initialized) { return pmm_alloc_contiguous(1);
return 0;
}
uint64_t start = find_contiguous_free(1);
if (start == UINT64_MAX) {
return 0;
}
bitmap_mark_range(start, 1, 1);
return start * (uint64_t)PMM_PAGE_SIZE;
} }
uint64_t pmm_alloc_contiguous(uint64_t num_pages) { uint64_t pmm_alloc_contiguous(uint64_t num_pages) {
uint64_t start;
if (!pmm_initialized || num_pages == 0) { if (!pmm_initialized || num_pages == 0) {
return 0; return 0;
} }
uint64_t start = find_contiguous_free(num_pages); sk_spin_lock(&g_lock_pmm);
if (start == UINT64_MAX) { start = find_contiguous_free(num_pages);
return 0; if (start != UINT64_MAX) {
}
bitmap_mark_range(start, num_pages, 1); bitmap_mark_range(start, num_pages, 1);
return start * (uint64_t)PMM_PAGE_SIZE; }
sk_spin_unlock(&g_lock_pmm);
return start == UINT64_MAX ? 0 : start * (uint64_t)PMM_PAGE_SIZE;
} }
void pmm_free_page(uint64_t paddr) { void pmm_free_page(uint64_t paddr) {
if (!pmm_initialized || (paddr % PMM_PAGE_SIZE) != 0) {
return;
}
pmm_free_contiguous(paddr, 1); pmm_free_contiguous(paddr, 1);
} }
@@ -362,13 +352,17 @@ void pmm_free_contiguous(uint64_t paddr, uint64_t num_pages) {
num_pages = pmm_tracked_pages - start_page; num_pages = pmm_tracked_pages - start_page;
} }
sk_spin_lock(&g_lock_pmm);
bitmap_mark_range(start_page, num_pages, 0); bitmap_mark_range(start_page, num_pages, 0);
sk_spin_unlock(&g_lock_pmm);
} }
pmm_stats_t pmm_get_stats(void) { pmm_stats_t pmm_get_stats(void) {
pmm_stats_t stats = {0}; pmm_stats_t stats = {0};
sk_spin_lock(&g_lock_pmm);
stats.total_pages = pmm_total_pages; stats.total_pages = pmm_total_pages;
stats.free_pages = pmm_free_pages; stats.free_pages = pmm_free_pages;
sk_spin_unlock(&g_lock_pmm);
stats.used_pages = (pmm_total_pages > pmm_free_pages) ? (pmm_total_pages - pmm_free_pages) : 0; stats.used_pages = (pmm_total_pages > pmm_free_pages) ? (pmm_total_pages - pmm_free_pages) : 0;
stats.total_bytes = stats.total_pages * (uint64_t)PMM_PAGE_SIZE; stats.total_bytes = stats.total_pages * (uint64_t)PMM_PAGE_SIZE;
stats.used_bytes = stats.used_pages * (uint64_t)PMM_PAGE_SIZE; stats.used_bytes = stats.used_pages * (uint64_t)PMM_PAGE_SIZE;
+44 -10
View File
@@ -53,6 +53,7 @@
#include "pmm.h" #include "pmm.h"
#include "console.h" #include "console.h"
#include "uefi.h" #include "uefi.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_vmm */
#define PTE_PRESENT (1ull << 0) #define PTE_PRESENT (1ull << 0)
#define PTE_WRITABLE (1ull << 1) #define PTE_WRITABLE (1ull << 1)
@@ -176,7 +177,11 @@ static uint64_t *resolve_pt(uint64_t vaddr, uint16_t *idx_out) {
return pt; return pt;
} }
int vmm_map_page(uint64_t vaddr, uint64_t paddr, uint64_t flags) { /* FABRIC-3.10.8 MC-0b: every table walk runs under g_lock_vmm. The walk
* bodies are the *_locked functions; the public ones take the lock around
* them. -2 from map_page_locked means a mapping conflict, printed by the
* caller once the lock is dropped (the console lock sits above this one). */
static int map_page_locked(uint64_t vaddr, uint64_t paddr, uint64_t flags) {
if (vmm_root_pml4_phys == 0) { if (vmm_root_pml4_phys == 0) {
return -1; return -1;
} }
@@ -204,8 +209,7 @@ int vmm_map_page(uint64_t vaddr, uint64_t paddr, uint64_t flags) {
return 0; return 0;
} }
} }
console_println("VMM map conflict"); return -2;
return -1;
} }
pt[idx] = make_pte(paddr, flags | VMM_FLAG_PRESENT); pt[idx] = make_pte(paddr, flags | VMM_FLAG_PRESENT);
@@ -213,7 +217,19 @@ int vmm_map_page(uint64_t vaddr, uint64_t paddr, uint64_t flags) {
return 0; return 0;
} }
int vmm_unmap_page(uint64_t vaddr) { int vmm_map_page(uint64_t vaddr, uint64_t paddr, uint64_t flags) {
int rc;
sk_spin_lock(&g_lock_vmm);
rc = map_page_locked(vaddr, paddr, flags);
sk_spin_unlock(&g_lock_vmm);
if (rc == -2) {
console_println("VMM map conflict");
rc = -1;
}
return rc;
}
static int unmap_page_locked(uint64_t vaddr) {
if (vmm_root_pml4_phys == 0) { if (vmm_root_pml4_phys == 0) {
return -1; return -1;
} }
@@ -238,7 +254,15 @@ int vmm_unmap_page(uint64_t vaddr) {
return 0; return 0;
} }
uint64_t vmm_get_paddr(uint64_t vaddr) { int vmm_unmap_page(uint64_t vaddr) {
int rc;
sk_spin_lock(&g_lock_vmm);
rc = unmap_page_locked(vaddr);
sk_spin_unlock(&g_lock_vmm);
return rc;
}
static uint64_t get_paddr_locked(uint64_t vaddr) {
if (vmm_root_pml4_phys == 0) { if (vmm_root_pml4_phys == 0) {
return 0; return 0;
} }
@@ -261,6 +285,14 @@ uint64_t vmm_get_paddr(uint64_t vaddr) {
return (pt[idx] & ADDR_MASK) | (vaddr & 0xFFF); return (pt[idx] & ADDR_MASK) | (vaddr & 0xFFF);
} }
uint64_t vmm_get_paddr(uint64_t vaddr) {
uint64_t pa;
sk_spin_lock(&g_lock_vmm);
pa = get_paddr_locked(vaddr);
sk_spin_unlock(&g_lock_vmm);
return pa;
}
int vmm_map_range(uint64_t vaddr, uint64_t paddr, uint64_t size, uint64_t flags) { int vmm_map_range(uint64_t vaddr, uint64_t paddr, uint64_t size, uint64_t flags) {
uint64_t pages = (size + VMM_PAGE_SIZE - 1) / VMM_PAGE_SIZE; uint64_t pages = (size + VMM_PAGE_SIZE - 1) / VMM_PAGE_SIZE;
for (uint64_t i = 0; i < pages; ++i) { for (uint64_t i = 0; i < pages; ++i) {
@@ -279,12 +311,14 @@ int vmm_query_page(uint64_t vaddr, vmm_page_info_t *info) {
} }
uint16_t idx = 0; uint16_t idx = 0;
uint64_t *pt = resolve_pt(vaddr, &idx); uint64_t entry = 0;
if (!pt) { uint64_t *pt;
return 0;
} sk_spin_lock(&g_lock_vmm);
pt = resolve_pt(vaddr, &idx);
if (pt) entry = pt[idx];
sk_spin_unlock(&g_lock_vmm);
uint64_t entry = pt[idx];
if (!(entry & PTE_PRESENT)) { if (!(entry & PTE_PRESENT)) {
return 0; return 0;
} }
+34 -30
View File
@@ -58,6 +58,7 @@
#endif #endif
#include "platform_time.h" #include "platform_time.h"
#include "platform_lock.h" #include "platform_lock.h"
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- sf_mutex is a real lock */
/* No LOG_LINE_MAX include-order constraint anymore: vm.h's own /* No LOG_LINE_MAX include-order constraint anymore: vm.h's own
* LOG_LINE_MAX (persistent block-log line size, 64) and log.h's in-memory * LOG_LINE_MAX (persistent block-log line size, 64) and log.h's in-memory
* message line length (renamed LOG_MSG_LINE_MAX, 256) no longer share a * message line length (renamed LOG_MSG_LINE_MAX, 256) no longer share a
@@ -381,14 +382,18 @@ void log_test_result(const char *word_name, TestResult result) {
void log_set_vm(struct VM *vm) { (void)vm; } void log_set_vm(struct VM *vm) { (void)vm; }
/* ----------------------------------------------------------------------------- /* -----------------------------------------------------------------------------
* Mutex stubs (single-threaded kernel) * Mutexes: kernel spinlocks (FABRIC-3.10.8 Stage MC-0b)
* ---------------------------------------------------------------------------*/ * ---------------------------------------------------------------------------*/
/* Until MC-0b these were no-ops ("single-threaded kernel"). They are now
* the kernel spinlock (starkernel/spinlock.h) on sf_mutex_t's one lock
* word: 0 = free, core_id + 1 = held. Interrupts are masked on the owning
* core while held, which also settles the old concern recorded here
* (FABRIC-0.md §21.2, §25.1 item 0.9): an interrupt can no longer arrive
* on a core inside one of these sections, so it can't spin against its
* own core. */
/** /**
* @brief Initialise a kernel mutex (sets state to 0 — unlocked). * @brief Initialise a kernel mutex (unlocked).
*
* The kernel VM runs on a single thread with no preemption at the VM level,
* so the full spinlock/pthread path is unnecessary. This stub zeros the
* @c state field in @p mutex to leave it in a consistent unlocked state.
* *
* @param mutex Mutex to initialise; returns -1 immediately if @c NULL. * @param mutex Mutex to initialise; returns -1 immediately if @c NULL.
* @return 0 on success, -1 if @p mutex is @c NULL. * @return 0 on success, -1 if @p mutex is @c NULL.
@@ -399,47 +404,46 @@ int sf_mutex_init(sf_mutex_t *mutex) {
return 0; return 0;
} }
static uint64_t g_mutex_destroyed_held;
/** /**
* @brief Destroy a kernel mutex (no-op). * @brief Destroy a kernel mutex.
* *
* Nothing to release in the kernel stub — the mutex holds only a single * Nothing to release. A mutex destroyed while held is counted and logged
* @c int field and has no OS resources. * (the VM it belongs to is going away with a critical section open).
* *
* @param mutex Ignored. * @param mutex Mutex to destroy; ignored if @c NULL.
*/ */
void sf_mutex_destroy(sf_mutex_t *mutex) { void sf_mutex_destroy(sf_mutex_t *mutex) {
(void)mutex; if (!mutex) return;
if (mutex->state != 0) {
g_mutex_destroyed_held++;
log_message(LOG_ERROR, "LOCK: VM mutex destroyed while held x%llu",
(unsigned long long)g_mutex_destroyed_held);
}
mutex->state = 0;
} }
uint64_t sf_mutex_destroyed_held_count(void) { return g_mutex_destroyed_held; }
/** /**
* @brief Acquire a kernel mutex (no-op). * @brief Acquire a kernel mutex (spins; interrupts masked while held).
* *
* Single-threaded kernel: no contention is possible at the VM level, so * @param mutex Mutex to lock; ignored if @c NULL.
* acquiring the mutex is always immediate and guaranteed to succeed.
*
* This no-op is correct only as long as nothing running in interrupt
* context mutates the structure a given lock protects (FABRIC-0.md §21.2,
* §25.1 item 0.9). If that ever stops being true, making this a real
* spinlock is not the fix: a single hart taking an interrupt while
* already holding the lock would spin against itself and deadlock. The
* correct fix in that case is disabling interrupts around the critical
* section, not adding a spinlock here.
*
* @param mutex Ignored.
*/ */
void sf_mutex_lock(sf_mutex_t *mutex) { void sf_mutex_lock(sf_mutex_t *mutex) {
(void)mutex; if (!mutex) return;
sk_lockword_acquire(&mutex->state);
} }
/** /**
* @brief Release a kernel mutex (no-op). * @brief Release a kernel mutex.
* *
* Single-threaded kernel: matching @c sf_mutex_lock() no-op. * @param mutex Mutex to unlock; ignored if @c NULL.
*
* @param mutex Ignored.
*/ */
void sf_mutex_unlock(sf_mutex_t *mutex) { void sf_mutex_unlock(sf_mutex_t *mutex) {
(void)mutex; if (!mutex) return;
sk_lockword_release(&mutex->state);
} }
/* ----------------------------------------------------------------------------- /* -----------------------------------------------------------------------------
+223 -77
View File
@@ -54,6 +54,21 @@
#include "log.h" /* Stage M-0: grow failures are logged at LOG_ERROR */ #include "log.h" /* Stage M-0: grow failures are logged at LOG_ERROR */
#include "vm.h" /* task 3.4 -- VM struct fields, vm_interpret() */ #include "vm.h" /* task 3.4 -- VM struct fields, vm_interpret() */
#include "starkernel/capsule_vm_switch_signal.h" /* FABRIC-3.8.md §3.13.3 -- wait stamps */ #include "starkernel/capsule_vm_switch_signal.h" /* FABRIC-3.8.md §3.13.3 -- wait stamps */
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- g_lock_hermes */
/* FABRIC-3.10.8 MC-0b: g_lock_hermes covers everything in this file -- the
* arena directory and segments, the ledger and event counters, the channel
* table and member lists, the pending queues. Public functions take it;
* the *_locked statics run with it held and are what public functions
* compose from, so it is never taken twice. It is never held across:
* - vm_interpret_kind() (a drained message's FORTH can SEND straight
* back in) or a policy word (sk_hermes_channel_open_policy());
* - the switch module's notifications after a send (the Stage H floor
* reads Hermes queues back) -- sk_hermes_send_one() makes them once
* the lock is dropped. The wait stamp itself is taken under this lock
* (the switch lock sits below it, spinlock.h).
* Publish and event walk a channel's members one copied member at a time,
* sending each through sk_hermes_send_one(). */
#include "starkernel/capsule_run.h" /* VM_NAME_MAX -- Stage M-B event payloads */ #include "starkernel/capsule_run.h" /* VM_NAME_MAX -- Stage M-B event payloads */
/* Freestanding: no libc printf. Prints an unsigned decimal, no leading /* Freestanding: no libc printf. Prints an unsigned decimal, no leading
@@ -138,10 +153,12 @@ static uint64_t sk_hermes_returned = 0;
static uint64_t sk_hermes_consumed = 0; static uint64_t sk_hermes_consumed = 0;
void sk_hermes_ledger(uint64_t *held, uint64_t *pulled, uint64_t *returned, uint64_t *consumed) { void sk_hermes_ledger(uint64_t *held, uint64_t *pulled, uint64_t *returned, uint64_t *consumed) {
sk_spin_lock(&g_lock_hermes);
if (held) *held = sk_hermes_held; if (held) *held = sk_hermes_held;
if (pulled) *pulled = sk_hermes_pulled; if (pulled) *pulled = sk_hermes_pulled;
if (returned) *returned = sk_hermes_returned; if (returned) *returned = sk_hermes_returned;
if (consumed) *consumed = sk_hermes_consumed; if (consumed) *consumed = sk_hermes_consumed;
sk_spin_unlock(&g_lock_hermes);
} }
int sk_hermes_audit_values(uint64_t held, uint64_t pulled, uint64_t returned, uint64_t consumed) { int sk_hermes_audit_values(uint64_t held, uint64_t pulled, uint64_t returned, uint64_t consumed) {
@@ -151,7 +168,7 @@ int sk_hermes_audit_values(uint64_t held, uint64_t pulled, uint64_t returned, ui
static uint64_t sk_hermes_audit_failures = 0; static uint64_t sk_hermes_audit_failures = 0;
int sk_hermes_audit(void) { static int audit_locked(void) {
int ok = sk_hermes_audit_values(sk_hermes_held, sk_hermes_pulled, int ok = sk_hermes_audit_values(sk_hermes_held, sk_hermes_pulled,
sk_hermes_returned, sk_hermes_consumed); sk_hermes_returned, sk_hermes_consumed);
if (!ok) { if (!ok) {
@@ -161,11 +178,19 @@ int sk_hermes_audit(void) {
return ok; return ok;
} }
int sk_hermes_audit(void) {
int ok;
sk_spin_lock(&g_lock_hermes);
ok = audit_locked();
sk_spin_unlock(&g_lock_hermes);
return ok;
}
uint64_t sk_hermes_audit_failure_count(void) { uint64_t sk_hermes_audit_failure_count(void) {
return sk_hermes_audit_failures; return sk_hermes_audit_failures;
} }
uint64_t sk_hermes_scan_held(size_t *live_count) { static uint64_t scan_held_locked(size_t *live_count) {
uint64_t sum = 0; uint64_t sum = 0;
size_t n = 0; size_t n = 0;
size_t i, len = sk_hermes_arena_capacity(); size_t i, len = sk_hermes_arena_capacity();
@@ -180,8 +205,20 @@ uint64_t sk_hermes_scan_held(size_t *live_count) {
return sum; return sum;
} }
uint64_t sk_hermes_scan_held(size_t *live_count) {
uint64_t sum;
sk_spin_lock(&g_lock_hermes);
sum = scan_held_locked(live_count);
sk_spin_unlock(&g_lock_hermes);
return sum;
}
int sk_hermes_scan_check(void) { int sk_hermes_scan_check(void) {
return sk_hermes_scan_held((size_t *)0) == sk_hermes_held; int ok;
sk_spin_lock(&g_lock_hermes);
ok = scan_held_locked((size_t *)0) == sk_hermes_held;
sk_spin_unlock(&g_lock_hermes);
return ok;
} }
/* Lowest free index; grows the arena by a segment when every slot is in /* Lowest free index; grows the arena by a segment when every slot is in
@@ -195,7 +232,7 @@ static long sk_hermes_find_free_slot(void) {
return (long)len; /* first slot of the new segment */ return (long)len; /* first slot of the new segment */
} }
int sk_hermes_alloc(VMUuid vm_id, SkHermesMessage **out_msg) { static int alloc_locked(VMUuid vm_id, SkHermesMessage **out_msg) {
long slot; long slot;
SkHermesMessage *m; SkHermesMessage *m;
uint64_t pulled; uint64_t pulled;
@@ -260,12 +297,20 @@ int sk_hermes_alloc(VMUuid vm_id, SkHermesMessage **out_msg) {
* once every refusal path above has already returned. */ * once every refusal path above has already returned. */
sk_hermes_held += pulled; sk_hermes_held += pulled;
sk_hermes_pulled += pulled; sk_hermes_pulled += pulled;
sk_hermes_audit(); audit_locked();
return 0; return 0;
} }
int sk_hermes_release(SkHermesMessage *msg) { int sk_hermes_alloc(VMUuid vm_id, SkHermesMessage **out_msg) {
int rc;
sk_spin_lock(&g_lock_hermes);
rc = alloc_locked(vm_id, out_msg);
sk_spin_unlock(&g_lock_hermes);
return rc;
}
static int release_locked(SkHermesMessage *msg) {
int rc; int rc;
uint64_t remaining; uint64_t remaining;
@@ -302,18 +347,30 @@ int sk_hermes_release(SkHermesMessage *msg) {
if (rc == 0) { if (rc == 0) {
sk_hermes_held -= remaining; sk_hermes_held -= remaining;
sk_hermes_returned += remaining; sk_hermes_returned += remaining;
sk_hermes_audit(); audit_locked();
} }
memset(msg, 0, sizeof(*msg)); memset(msg, 0, sizeof(*msg));
return rc; return rc;
} }
int sk_hermes_release(SkHermesMessage *msg) {
int rc;
sk_spin_lock(&g_lock_hermes);
rc = release_locked(msg);
sk_spin_unlock(&g_lock_hermes);
return rc;
}
int sk_hermes_decay(SkHermesMessage *msg) { int sk_hermes_decay(SkHermesMessage *msg) {
uint64_t before, after; uint64_t before, after;
StadiumCell *cell; StadiumCell *cell;
if (!msg || !msg->in_use || msg->stadium_cell < 0) return -1; sk_spin_lock(&g_lock_hermes);
if (!msg || !msg->in_use || msg->stadium_cell < 0) {
sk_spin_unlock(&g_lock_hermes);
return -1;
}
cell = &stadium_cells()[msg->stadium_cell]; cell = &stadium_cells()[msg->stadium_cell];
before = cell->header.heat; before = cell->header.heat;
@@ -326,7 +383,8 @@ int sk_hermes_decay(SkHermesMessage *msg) {
sk_hermes_consumed += before - after; sk_hermes_consumed += before - after;
sk_hermes_held -= before - after; sk_hermes_held -= before - after;
stadium_consumed_record(msg->owner, before - after); stadium_consumed_record(msg->owner, before - after);
sk_hermes_audit(); audit_locked();
sk_spin_unlock(&g_lock_hermes);
return 0; return 0;
} }
@@ -361,8 +419,10 @@ int sk_hermes_channels_boot_init(void) {
table[SK_HERMES_CHANNEL_EV_DIED].in_use = 1; table[SK_HERMES_CHANNEL_EV_DIED].in_use = 1;
} }
sk_spin_lock(&g_lock_hermes);
sk_hermes_channels = table; sk_hermes_channels = table;
sk_hermes_channel_capacity_val = (int)max_vm_count; sk_hermes_channel_capacity_val = (int)max_vm_count;
sk_spin_unlock(&g_lock_hermes);
console_puts("Kernel-Hermes: "); console_puts("Kernel-Hermes: ");
console_put_u64((uint64_t)max_vm_count); console_put_u64((uint64_t)max_vm_count);
@@ -383,40 +443,47 @@ static int channel_valid(int channel_id) {
} }
int sk_hermes_channel_create(void) { int sk_hermes_channel_create(void) {
int i; int i, ch = -1;
if (!sk_hermes_channels) return -1;
sk_spin_lock(&g_lock_hermes);
/* Index 0 is always in_use (the common channel), so this scan never /* Index 0 is always in_use (the common channel), so this scan never
* returns it -- start at 1 purely as a scan-cost micro-optimisation, * returns it -- start at 1 purely as a scan-cost micro-optimisation,
* not for correctness (the in_use check alone would already skip it). */ * not for correctness (the in_use check alone would already skip it). */
for (i = 1; i < sk_hermes_channel_capacity_val; i++) { for (i = 1; sk_hermes_channels && i < sk_hermes_channel_capacity_val; i++) {
if (!sk_hermes_channels[i].in_use) { if (!sk_hermes_channels[i].in_use) {
memset(&sk_hermes_channels[i], 0, sizeof(SkHermesChannel)); memset(&sk_hermes_channels[i], 0, sizeof(SkHermesChannel));
sk_hermes_channels[i].in_use = 1; sk_hermes_channels[i].in_use = 1;
return i; ch = i;
break;
} }
} }
return -1; sk_spin_unlock(&g_lock_hermes);
return ch;
} }
int sk_hermes_channel_destroy(int channel_id) { int sk_hermes_channel_destroy(int channel_id) {
int rc = -1;
/* Permanent: channel 0 (SXLV.1) and, since Stage M-B, the event /* Permanent: channel 0 (SXLV.1) and, since Stage M-B, the event
* channels 1..3. */ * channels 1..3. */
if (channel_id >= SK_HERMES_CHANNEL_COMMON && if (channel_id >= SK_HERMES_CHANNEL_COMMON &&
channel_id <= SK_HERMES_CHANNEL_PERMANENT_LAST) return -1; channel_id <= SK_HERMES_CHANNEL_PERMANENT_LAST) return -1;
if (!channel_valid(channel_id)) return -1;
sk_spin_lock(&g_lock_hermes);
if (channel_valid(channel_id)) {
/* Stage M-0: the member list is heap-grown -- free it, or create()'s /* Stage M-0: the member list is heap-grown -- free it, or create()'s
* memset of this slot would leak it. */ * memset of this slot would leak it. */
if (sk_hermes_channels[channel_id].membership.members) { if (sk_hermes_channels[channel_id].membership.members) {
kfree(sk_hermes_channels[channel_id].membership.members); kfree(sk_hermes_channels[channel_id].membership.members);
} }
memset(&sk_hermes_channels[channel_id], 0, sizeof(SkHermesChannel)); memset(&sk_hermes_channels[channel_id], 0, sizeof(SkHermesChannel));
return 0; rc = 0;
}
sk_spin_unlock(&g_lock_hermes);
return rc;
} }
int sk_hermes_channel_subscribe(int channel_id, VMUuid vm_id) { static int subscribe_locked(int channel_id, VMUuid vm_id) {
SkHermesMembership *m; SkHermesMembership *m;
size_t i; size_t i;
@@ -446,7 +513,7 @@ int sk_hermes_channel_subscribe(int channel_id, VMUuid vm_id) {
return 0; return 0;
} }
int sk_hermes_channel_unsubscribe(int channel_id, VMUuid vm_id) { static int unsubscribe_locked(int channel_id, VMUuid vm_id) {
SkHermesMembership *m; SkHermesMembership *m;
size_t i; size_t i;
@@ -466,29 +533,56 @@ int sk_hermes_channel_unsubscribe(int channel_id, VMUuid vm_id) {
return -1; /* not a member */ return -1; /* not a member */
} }
int sk_hermes_channel_subscribe(int channel_id, VMUuid vm_id) {
int rc;
sk_spin_lock(&g_lock_hermes);
rc = subscribe_locked(channel_id, vm_id);
sk_spin_unlock(&g_lock_hermes);
return rc;
}
int sk_hermes_channel_unsubscribe(int channel_id, VMUuid vm_id) {
int rc;
sk_spin_lock(&g_lock_hermes);
rc = unsubscribe_locked(channel_id, vm_id);
sk_spin_unlock(&g_lock_hermes);
return rc;
}
int sk_hermes_channel_is_member(int channel_id, VMUuid vm_id) { int sk_hermes_channel_is_member(int channel_id, VMUuid vm_id) {
SkHermesMembership *m; SkHermesMembership *m;
size_t i; size_t i;
int member = 0;
if (!channel_valid(channel_id)) return 0; sk_spin_lock(&g_lock_hermes);
if (channel_valid(channel_id)) {
m = &sk_hermes_channels[channel_id].membership; m = &sk_hermes_channels[channel_id].membership;
for (i = 0; i < m->count; i++) { for (i = 0; i < m->count; i++) {
if (vm_uuid_equal(m->members[i], vm_id)) return 1; if (vm_uuid_equal(m->members[i], vm_id)) { member = 1; break; }
} }
return 0; }
sk_spin_unlock(&g_lock_hermes);
return member;
} }
int sk_hermes_channel_member_count(int channel_id) { int sk_hermes_channel_member_count(int channel_id) {
if (!channel_valid(channel_id)) return -1; int n = -1;
return (int)sk_hermes_channels[channel_id].membership.count; sk_spin_lock(&g_lock_hermes);
if (channel_valid(channel_id)) n = (int)sk_hermes_channels[channel_id].membership.count;
sk_spin_unlock(&g_lock_hermes);
return n;
} }
int sk_hermes_channel_member_at(int channel_id, size_t i, VMUuid *out) { int sk_hermes_channel_member_at(int channel_id, size_t i, VMUuid *out) {
if (!channel_valid(channel_id) || !out) return -1; int rc = -1;
if (i >= sk_hermes_channels[channel_id].membership.count) return -1; if (!out) return -1;
sk_spin_lock(&g_lock_hermes);
if (channel_valid(channel_id) && i < sk_hermes_channels[channel_id].membership.count) {
*out = sk_hermes_channels[channel_id].membership.members[i]; *out = sk_hermes_channels[channel_id].membership.members[i];
return 0; rc = 0;
}
sk_spin_unlock(&g_lock_hermes);
return rc;
} }
/* Task 3.3: per-subscriber pending queue table. See kernel_hermes.h's own /* Task 3.3: per-subscriber pending queue table. See kernel_hermes.h's own
@@ -526,8 +620,10 @@ int sk_hermes_queues_boot_init(void) {
memset(&table[i], 0, sizeof(SkHermesPendingQueue)); memset(&table[i], 0, sizeof(SkHermesPendingQueue));
} }
sk_spin_lock(&g_lock_hermes);
sk_hermes_queues = table; sk_hermes_queues = table;
sk_hermes_queue_capacity_val = (int)max_vm_count; sk_hermes_queue_capacity_val = (int)max_vm_count;
sk_spin_unlock(&g_lock_hermes);
console_puts("Kernel-Hermes: "); console_puts("Kernel-Hermes: ");
console_put_u64((uint64_t)max_vm_count); console_put_u64((uint64_t)max_vm_count);
@@ -606,19 +702,15 @@ static int queue_push(SkHermesPendingQueue *q, uint32_t msg_index) {
return 0; return 0;
} }
int sk_hermes_send_one(VMUuid from, VMUuid to, uint32_t type, uint32_t channel_id, /* sk_hermes_send_one() under g_lock_hermes. *first_out: the recipient's
void *payload_addr, uint32_t payload_len) { * queue went from empty to one message (the Stage H floor applies). */
static int send_one_locked(VMUuid from, VMUuid to, uint32_t type, uint32_t channel_id,
void *payload_addr, uint32_t payload_len, int *first_out) {
SkHermesMessage *msg; SkHermesMessage *msg;
SkHermesPendingQueue *q; SkHermesPendingQueue *q;
uint32_t idx; uint32_t idx;
/* Task 3.5: the one-block bound, enforced uniformly here whether or if (alloc_locked(from, &msg) != 0) return -1;
* not payload_addr happens to be a chunk carrier -- see
* kernel_hermes.h's own sizing note on why SK_HERMES_CHUNK_MAX_SLICE
* (not SK_HERMES_CHUNK_MAX_PAYLOAD) is a chunk's own content size. */
if (payload_len > SK_HERMES_CHUNK_MAX_PAYLOAD) return -1;
if (sk_hermes_alloc(from, &msg) != 0) return -1;
/* Real defect, found 2026-09-22 (FABRIC-3.6.md task 3.8's own /* Real defect, found 2026-09-22 (FABRIC-3.6.md task 3.8's own
* findings log): this used to store the caller's own payload_addr * findings log): this used to store the caller's own payload_addr
@@ -645,21 +737,41 @@ int sk_hermes_send_one(VMUuid from, VMUuid to, uint32_t type, uint32_t channel_i
q = find_or_create_queue(to); q = find_or_create_queue(to);
if (!q) { if (!q) {
sk_hermes_release(msg); /* roll back -- no destination queue */ release_locked(msg); /* roll back -- no destination queue */
return -1; return -1;
} }
idx = msg->index; idx = msg->index;
if (queue_push(q, idx) != 0) { if (queue_push(q, idx) != 0) {
sk_hermes_release(msg); /* roll back -- queue couldn't grow (logged, counted) */ release_locked(msg); /* roll back -- queue couldn't grow (logged, counted) */
return -1; return -1;
} }
/* FABRIC-3.8.md §3.13.3 (Stage C-b): stamped only once admitted, and /* FABRIC-3.8.md §3.13.3 (Stage C-b): stamped only once admitted. */
* the switch decision told something changed -- the recipient may now
* be the coolest VM with work. */
msg->wait_stamp = sk_vm_switch_signal_next_stamp(); msg->wait_stamp = sk_vm_switch_signal_next_stamp();
if (q->count == 1) sk_vm_switch_signal_note_work_gained(to); /* Stage H floor */ *first_out = (q->count == 1);
return 0;
}
int sk_hermes_send_one(VMUuid from, VMUuid to, uint32_t type, uint32_t channel_id,
void *payload_addr, uint32_t payload_len) {
int rc, first = 0;
/* Task 3.5: the one-block bound, enforced uniformly here whether or
* not payload_addr happens to be a chunk carrier -- see
* kernel_hermes.h's own sizing note on why SK_HERMES_CHUNK_MAX_SLICE
* (not SK_HERMES_CHUNK_MAX_PAYLOAD) is a chunk's own content size. */
if (payload_len > SK_HERMES_CHUNK_MAX_PAYLOAD) return -1;
sk_spin_lock(&g_lock_hermes);
rc = send_one_locked(from, to, type, channel_id, payload_addr, payload_len, &first);
sk_spin_unlock(&g_lock_hermes);
if (rc != 0) return rc;
/* The switch decision told something changed -- the recipient may now
* be the coolest VM with work. After the lock: the floor reads Hermes
* queues back. */
if (first) sk_vm_switch_signal_note_work_gained(to); /* Stage H floor */
sk_vm_switch_signal_note_mail(); sk_vm_switch_signal_note_mail();
return 0; return 0;
@@ -667,20 +779,19 @@ int sk_hermes_send_one(VMUuid from, VMUuid to, uint32_t type, uint32_t channel_i
int sk_hermes_publish(VMUuid from, int channel_id, uint32_t type, int sk_hermes_publish(VMUuid from, int channel_id, uint32_t type,
void *payload_addr, uint32_t payload_len) { void *payload_addr, uint32_t payload_len) {
SkHermesMembership *m; VMUuid to;
size_t i; size_t i;
int delivered = 0; int delivered = 0;
if (!channel_valid(channel_id)) return -1; if (sk_hermes_channel_member_count(channel_id) < 0) return -1; /* not a channel */
if (payload_len > SK_HERMES_CHUNK_MAX_PAYLOAD) return -1; /* see sk_hermes_send_one()'s if (payload_len > SK_HERMES_CHUNK_MAX_PAYLOAD) return -1; /* see sk_hermes_send_one()'s
* own check -- duplicated here * own check -- duplicated here
* so publish() itself returns * so publish() itself returns
* -1 (not a silent 0) on an * -1 (not a silent 0) on an
* oversized payload */ * oversized payload */
m = &sk_hermes_channels[channel_id].membership; for (i = 0; sk_hermes_channel_member_at(channel_id, i, &to) == 0; i++) {
for (i = 0; i < m->count; i++) { if (sk_hermes_send_one(from, to, type, (uint32_t)channel_id,
if (sk_hermes_send_one(from, m->members[i], type, (uint32_t)channel_id,
payload_addr, payload_len) == 0) { payload_addr, payload_len) == 0) {
delivered++; delivered++;
} }
@@ -696,9 +807,11 @@ uint64_t sk_hermes_events_delivered(void) { return sk_hermes_ev_delivered; }
uint64_t sk_hermes_events_dropped(void) { return sk_hermes_ev_dropped; } uint64_t sk_hermes_events_dropped(void) { return sk_hermes_ev_dropped; }
int sk_hermes_event(int ev_channel, VMUuid about, VMUuid funder, const char *name) { int sk_hermes_event(int ev_channel, VMUuid about, VMUuid funder, const char *name) {
static char payload[VM_NAME_MAX + 32]; /* MC-0b: on this stack, not static -- two cores can publish events at
* once. */
char payload[VM_NAME_MAX + 32];
const char *handler; const char *handler;
SkHermesMembership *m; VMUuid to;
size_t i; size_t i;
int len, delivered = 0, dropped = 0; int len, delivered = 0, dropped = 0;
@@ -708,17 +821,15 @@ int sk_hermes_event(int ev_channel, VMUuid about, VMUuid funder, const char *nam
case SK_HERMES_CHANNEL_EV_DIED: handler = "ON-DIED"; break; case SK_HERMES_CHANNEL_EV_DIED: handler = "ON-DIED"; break;
default: return 0; default: return 0;
} }
if (!channel_valid(ev_channel) || !name) return 0; if (sk_hermes_channel_member_count(ev_channel) < 0 || !name) return 0;
len = snprintf(payload, sizeof payload, "S\" %s\" %s", name, handler); len = snprintf(payload, sizeof payload, "S\" %s\" %s", name, handler);
if (len <= 0 || (size_t)len >= sizeof payload) return 0; if (len <= 0 || (size_t)len >= sizeof payload) return 0;
/* Copied by sk_hermes_send_one(), so one static payload serves every /* Copied by sk_hermes_send_one(), so one payload serves every member.
* member. The member list is read by index each time: a send never * The member list is read by index each time: a send never changes
* changes channel membership. */ * channel membership. */
m = &sk_hermes_channels[ev_channel].membership; for (i = 0; sk_hermes_channel_member_at(ev_channel, i, &to) == 0; i++) {
for (i = 0; i < m->count; i++) {
VMUuid to = m->members[i];
if (vm_uuid_equal(to, about)) continue; /* never to the VM it's about */ if (vm_uuid_equal(to, about)) continue; /* never to the VM it's about */
if (sk_hermes_send_one(funder, to, SK_HERMES_MSG_TYPE_EVENT, (uint32_t)ev_channel, if (sk_hermes_send_one(funder, to, SK_HERMES_MSG_TYPE_EVENT, (uint32_t)ev_channel,
payload, (uint32_t)(len + 1)) == 0) { payload, (uint32_t)(len + 1)) == 0) {
@@ -728,10 +839,12 @@ int sk_hermes_event(int ev_channel, VMUuid about, VMUuid funder, const char *nam
} }
} }
sk_spin_lock(&g_lock_hermes);
sk_hermes_ev_delivered += (uint64_t)delivered; sk_hermes_ev_delivered += (uint64_t)delivered;
sk_hermes_ev_dropped += (uint64_t)dropped;
sk_spin_unlock(&g_lock_hermes);
if (dropped > 0) { if (dropped > 0) {
/* M-6: dropped, counted, logged -- never blocking. */ /* M-6: dropped, counted, logged -- never blocking. */
sk_hermes_ev_dropped += (uint64_t)dropped;
console_puts("Kernel-Hermes: event "); console_puts("Kernel-Hermes: event ");
console_puts(handler); console_puts(handler);
console_puts(" for "); console_puts(" for ");
@@ -744,24 +857,13 @@ int sk_hermes_event(int ev_channel, VMUuid about, VMUuid funder, const char *nam
return delivered; return delivered;
} }
int sk_hermes_pending_count(VMUuid vm_id) { static SkHermesMessage *pending_peek_locked(VMUuid vm_id) {
SkHermesPendingQueue *q = find_queue(vm_id);
return q ? (int)q->count : 0;
}
SkHermesMessage *sk_hermes_pending_peek(VMUuid vm_id) {
SkHermesPendingQueue *q = find_queue(vm_id); SkHermesPendingQueue *q = find_queue(vm_id);
if (!q || q->count == 0) return (SkHermesMessage *)0; if (!q || q->count == 0) return (SkHermesMessage *)0;
return msg_at(q->slots[q->head]); return msg_at(q->slots[q->head]);
} }
SkHermesMessage *sk_hermes_pending_at(VMUuid vm_id, size_t i) { static int pending_pop_locked(VMUuid vm_id) {
SkHermesPendingQueue *q = find_queue(vm_id);
if (!q || i >= q->count) return (SkHermesMessage *)0;
return msg_at(q->slots[(q->head + i) % q->cap]);
}
int sk_hermes_pending_pop(VMUuid vm_id) {
SkHermesPendingQueue *q = find_queue(vm_id); SkHermesPendingQueue *q = find_queue(vm_id);
if (!q || q->count == 0) return -1; if (!q || q->count == 0) return -1;
q->head = (q->head + 1) % q->cap; q->head = (q->head + 1) % q->cap;
@@ -770,14 +872,53 @@ int sk_hermes_pending_pop(VMUuid vm_id) {
return 0; return 0;
} }
int sk_hermes_pending_count(VMUuid vm_id) {
SkHermesPendingQueue *q;
int n;
sk_spin_lock(&g_lock_hermes);
q = find_queue(vm_id);
n = q ? (int)q->count : 0;
sk_spin_unlock(&g_lock_hermes);
return n;
}
/* The message pointer stays valid after the lock drops (messages never
* move, kernel_hermes.h); only the queue's owner pops it. */
SkHermesMessage *sk_hermes_pending_peek(VMUuid vm_id) {
SkHermesMessage *m;
sk_spin_lock(&g_lock_hermes);
m = pending_peek_locked(vm_id);
sk_spin_unlock(&g_lock_hermes);
return m;
}
SkHermesMessage *sk_hermes_pending_at(VMUuid vm_id, size_t i) {
SkHermesPendingQueue *q;
SkHermesMessage *m = (SkHermesMessage *)0;
sk_spin_lock(&g_lock_hermes);
q = find_queue(vm_id);
if (q && i < q->count) m = msg_at(q->slots[(q->head + i) % q->cap]);
sk_spin_unlock(&g_lock_hermes);
return m;
}
int sk_hermes_pending_pop(VMUuid vm_id) {
int rc;
sk_spin_lock(&g_lock_hermes);
rc = pending_pop_locked(vm_id);
sk_spin_unlock(&g_lock_hermes);
return rc;
}
int sk_hermes_vm_departed(VMUuid vm_id, int *channels_left, int *released) { int sk_hermes_vm_departed(VMUuid vm_id, int *channels_left, int *released) {
SkHermesPendingQueue *q; SkHermesPendingQueue *q;
SkHermesMessage *m; SkHermesMessage *m;
int ch, left = 0, rel = 0; int ch, left = 0, rel = 0;
sk_spin_lock(&g_lock_hermes);
/* Every channel it belonged to, channel 0 included. */ /* Every channel it belonged to, channel 0 included. */
for (ch = 0; ch < sk_hermes_channel_capacity_val; ch++) { for (ch = 0; ch < sk_hermes_channel_capacity_val; ch++) {
if (channel_valid(ch) && sk_hermes_channel_unsubscribe(ch, vm_id) == 0) left++; if (channel_valid(ch) && unsubscribe_locked(ch, vm_id) == 0) left++;
} }
/* Its undelivered mail: each message released (heat back to whoever /* Its undelivered mail: each message released (heat back to whoever
@@ -785,12 +926,13 @@ int sk_hermes_vm_departed(VMUuid vm_id, int *channels_left, int *released) {
* freed for reuse, keeping its grown ring (find_or_create_queue()). */ * freed for reuse, keeping its grown ring (find_or_create_queue()). */
q = find_queue(vm_id); q = find_queue(vm_id);
if (q) { if (q) {
while ((m = sk_hermes_pending_peek(vm_id)) != (SkHermesMessage *)0) { while ((m = pending_peek_locked(vm_id)) != (SkHermesMessage *)0) {
if (sk_hermes_release(m) == 0) rel++; if (release_locked(m) == 0) rel++;
sk_hermes_pending_pop(vm_id); pending_pop_locked(vm_id);
} }
q->in_use = 0; q->in_use = 0;
} }
sk_spin_unlock(&g_lock_hermes);
if (channels_left) *channels_left = left; if (channels_left) *channels_left = left;
if (released) *released = rel; if (released) *released = rel;
@@ -896,8 +1038,10 @@ int sk_hermes_drain_checkpoint(VM *vm) {
* enclosing execution */ * enclosing execution */
vm->abort_requested = saved_abort; vm->abort_requested = saved_abort;
sk_hermes_release(msg); sk_spin_lock(&g_lock_hermes);
sk_hermes_pending_pop(self); release_locked(msg);
pending_pop_locked(self);
sk_spin_unlock(&g_lock_hermes);
return drain_error ? -1 : 1; return drain_error ? -1 : 1;
} }
@@ -925,7 +1069,9 @@ int sk_hermes_reassemble(SkHermesMessage **chunks, int n_chunks,
present = (int *)kmalloc((size_t)n_chunks * sizeof(int)); present = (int *)kmalloc((size_t)n_chunks * sizeof(int));
if (!present) { if (!present) {
sk_spin_lock(&g_lock_hermes); /* the failure counter */
sk_hermes_grow_failed("reassembly scratch"); sk_hermes_grow_failed("reassembly scratch");
sk_spin_unlock(&g_lock_hermes);
return -1; return -1;
} }
rc = reassemble_with(chunks, n_chunks, out_buf, out_buf_cap, out_len, present); rc = reassemble_with(chunks, n_chunks, out_buf, out_buf_cap, out_len, present);
+9
View File
@@ -35,6 +35,7 @@
#include "starkernel/capsule_vm_switch_signal.h" /* FABRIC-3.8.md §3.13.3 -- the decision */ #include "starkernel/capsule_vm_switch_signal.h" /* FABRIC-3.8.md §3.13.3 -- the decision */
#include "starkernel/vm/bootstrap/sk_vm_bootstrap.h" /* sk_get_mama_vm() -- the idle root */ #include "starkernel/vm/bootstrap/sk_vm_bootstrap.h" /* sk_get_mama_vm() -- the idle root */
#include "starkernel/percpu.h" /* FABRIC-3.10.8 MC-0a -- this core's current VM */ #include "starkernel/percpu.h" /* FABRIC-3.10.8 MC-0a -- this core's current VM */
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0b -- no lock crosses a switch */
#include <stdint.h> #include <stdint.h>
/* Arch-specific (switch.S in each arch dir). sk_vm_switch_prime() builds /* Arch-specific (switch.S in each arch dir). sk_vm_switch_prime() builds
@@ -226,6 +227,11 @@ void sk_vm_idle_point(int may_drain_self) {
VM *self = sk_percpu()->current_vm; VM *self = sk_percpu()->current_vm;
VM *next; VM *next;
/* FABRIC-3.10.8 MC-0b: nothing is held on return to the idle point,
* which makes this the place to log any lock failure counted since. */
sk_lock_check_none_held("idle point");
sk_lock_report();
if (!self || self != (VM *)sk_get_mama_vm()) return; if (!self || self != (VM *)sk_get_mama_vm()) return;
if (self->frames.count != 0) return; /* inside a word (KEY/EXPECT): not idle */ if (self->frames.count != 0) return; /* inside a word (KEY/EXPECT): not idle */
@@ -280,7 +286,10 @@ int sk_vm_context_switch(VM *from, VM *to) {
sk_vm_switch_signal_note_parked(from->stadium_vm_id); /* §3.13.3 wait stamp */ sk_vm_switch_signal_note_parked(from->stadium_vm_id); /* §3.13.3 wait stamp */
sk_percpu()->current_vm = to; sk_percpu()->current_vm = to;
/* FABRIC-3.10.8 MC-0b: no lock crosses a switch (spinlock.h). */
sk_lock_check_none_held("context switch, leaving");
sk_vm_switch_to(&from->native_stack_saved_sp, new_sp); sk_vm_switch_to(&from->native_stack_saved_sp, new_sp);
sk_lock_check_none_held("context switch, resumed");
/* Resumes here only once something later switches back to `from` -- /* Resumes here only once something later switches back to `from` --
* from `from`'s own point of view, this call simply took a while. * from `from`'s own point of view, this call simply took a while.