Stage MC-0d: Stadium partitioned by quota slot, one lock per slot (FABRIC-3.10.8 §3.5)
- Ruling (Captain Bob): per-slot locks, not "own path needs no lock" -- a lock only on the cross-slot side can't exclude an unlocked owner (Hermes release evicting a sender's cell while the sender admits). FABRIC-3.10.0 §3.6 amended. - StadiumVMQuota.lock covers the slot's lists, reservoir, consumed, word row and its cells. Word dispatch holds only its own slot lock for the whole dispatch (*_in_slot forms); peek-then-pull is one step. - g_lock_stadium: grant slot allocation (new slot published with release in_use) and the shared block table (ruled: the one shared-lock Stadium path). Taken before a slot lock. - Bitmap bytes set/cleared atomically (8 cells, possibly 8 slots). - Cross-slot writers moved under the owner's lock: Hermes decay (stadium_cell_decay), Hermes release ledger (stadium_evict_credit), session pin (stadium_cell_set_pin). stadium_conserved() one hold. - Word and block cell lookups check the cell is still this slot's. - Probe (removed), identical on all three ISAs: 10,746,599 of 10,746,616 word dispatches took exactly one lock (own slot), the 17 others all row growths; every live VM conserved; Hermes audit failures 0. - Findings reported: births/USE run a VM's words while another VM is the core's current VM (MC-3 claim must cover it); quota slots never released; owner admission can evict a live message cell (latent, 0). - Standard sequence on amd64/aarch64/riscv64: dict_hash set and G byte-identical, MINT/WIREBIND/USE ok, conservation true, 0 idle growth. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
355909d46f
commit
02c4c7e620
@@ -140,6 +140,11 @@ Captain Bob asked whether this should give a significant performance boost).
|
||||
conservation audits) take the Stadium lock.
|
||||
- This is part of MC-0, and MC-0's evidence includes it: a per-dispatch path that takes no
|
||||
shared lock.
|
||||
- **Amended at MC-0d (Captain Bob, 2026-09-25): one lock per quota slot**, not "no lock". A
|
||||
lock taken only by cross-slot operations can't exclude an owner that takes none. Example: a
|
||||
receiver's drain evicts the sender's message cell while the sender, on another core, pops
|
||||
the same free list. The owner's dispatch takes its own slot's lock, which no other VM's
|
||||
dispatch contends, so the path still takes no shared lock. See `FABRIC-3.10.8.md` §3.5.
|
||||
|
||||
### 3.7 Bringing up the cores, per ISA
|
||||
|
||||
|
||||
@@ -319,3 +319,90 @@ These runs' `dict_hash` values differ from the table below only because the prob
|
||||
- The G string is byte-identical to MC-0b2, MC-0b and MC-0a on all three ISAs.
|
||||
- The ledger line is unchanged.
|
||||
- No stop condition was hit. **MC-0c done.**
|
||||
|
||||
### 3.5 MC-0d — Stadium partitioned by quota slot (2026-09-25)
|
||||
|
||||
**Ruling (Captain Bob, 2026-09-25): one lock per quota slot.** FABRIC-3.10.0 §3.6 said a VM's own Stadium row and cells need no lock, with a Stadium lock only for cross-slot operations. That can't be made sound: a lock taken only on the cross-slot side can't exclude an owner that takes none. Concrete race:
|
||||
- a receiver's drain runs `sk_hermes_release()` → `stadium_evict()` on the **sender's** message cell, which rewrites the sender's free list, resident list and reservoir;
|
||||
- meanwhile the sender, running on another core, is popping that same free list in `stadium_admit()`.
|
||||
|
||||
`stadium_grant_quota()` splits a donor's free list the same way. The amendment is recorded at FABRIC-3.10.0 §3.6.
|
||||
|
||||
**Second ruling: the block table takes the shared Stadium lock.** `stadium_blocks.c` keeps one open-addressed table for every VM, keyed by (slot, LBN). `stadium_block_dispatch()` therefore takes `g_lock_stadium`. It is the one shared-lock Stadium path, recorded for MC-3 to measure; block access is rare next to word dispatch.
|
||||
|
||||
**Change.**
|
||||
- **The slot lock.**
|
||||
- `StadiumVMQuota` gains `lock`, a spinlock.h lock word.
|
||||
- It covers the slot's free list, resident list, reservoir, consumed, word row and statistics, and the slot's cells: their headers, owner entries and resident-list links.
|
||||
- Every public Stadium function locks the slot it touches. A cell operation locks the cell's owning slot (`lock_cell_owner()` re-reads the owner once locked).
|
||||
- **Word dispatch** (`stadium_word_dispatch()`) holds its own VM's slot lock for the whole dispatch: row, cell, cooling, pull, and admission including its eviction fallback. It uses the new `*_in_slot` forms, so it takes no other lock (a row growth's `kmalloc` excepted).
|
||||
- The reservoir peek-then-pull is now one step under that lock.
|
||||
- **Global part: `g_lock_stadium`.**
|
||||
- Quota-slot allocation in grants. Each new slot is filled first, then published with a release store of `in_use`; `quota_slot_for_vm()` reads `in_use` with acquire.
|
||||
- The shared block table.
|
||||
- It is taken before a slot lock, never after.
|
||||
- **The residency bitmap:** a byte covers 8 cells, which can belong to different slots, so set and clear are atomic OR/AND.
|
||||
- **Cross-slot writers moved into Stadium, under the owner's lock:**
|
||||
- kernel-Hermes's decay: `stadium_cell_decay()`, the same `q48_mul` arithmetic, consumed recorded in the same step;
|
||||
- Hermes's release ledger: `stadium_evict_credit()` returns the heat actually credited, read in the same step as the eviction, where it used to be read from the cell beforehand;
|
||||
- `session_set_pinned()`'s pin bit: `stadium_cell_set_pin()`.
|
||||
- **`stadium_conserved()`** reads its three terms under one hold, so the sum is of one instant.
|
||||
- **Owner check** added to both dispatch paths' cell lookups: the cell must still be this slot's.
|
||||
- Without it, a word or block cell evicted to this slot's free list could move to another VM in a grant and be admitted there with a matching identity. Word ids and LBNs are shared across VMs, so the old row would then heat the other VM's cell.
|
||||
- This is latent on one core.
|
||||
- **Unpartitioned reads that remain:** `sk_hermes_scan_held()` (reads message cells under the Hermes lock), `session.c`'s bounds check, and `kernel_main.c`'s decay self-test. None writes.
|
||||
- **Lock order** (spinlock.h):
|
||||
1. `hermes`
|
||||
2. `switch`
|
||||
3. `registry`
|
||||
4. `stadium`
|
||||
5. a slot lock
|
||||
6. `doe`
|
||||
7. `log`
|
||||
8. `block`
|
||||
9. `console`
|
||||
10. `heap`
|
||||
11. `vmm`
|
||||
12. `pmm`
|
||||
|
||||
A VM's `dict_lock` sits above Stadium: `vm_core.c`'s heat bump dispatches the word under it. Nothing in the block path calls into Stadium, so MIGRATE's `blk_flush()` edge doesn't close a cycle.
|
||||
|
||||
**Probe** (temporary, removed before commit). It counted:
|
||||
- the per-core lock acquisitions made during each `stadium_word_dispatch()`;
|
||||
- slot-lock acquisitions made for a slot other than the running VM's own;
|
||||
- admission-fallback evictions of message (DELIVER) cells;
|
||||
- quota slots in use.
|
||||
|
||||
An `MC0D-PROBE` word in every VM printed these, with `stadium_conserved()` on **every** live VM and the Hermes audit-failure count. The results are identical on all three ISAs, count for count:
|
||||
|
||||
| ISA | log | word dispatches | exactly one lock (own slot) | more than one | …all at a row growth | conserved | Hermes audit failures | DELIVER-cell fallback evictions |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| amd64 | `logs/20260925-212159` | 10,746,616 | 10,746,599 | 17 | 17 | 7/7 | 0 | 0 |
|
||||
| aarch64 | `logs/20260925-213527` | 10,746,616 | 10,746,599 | 17 | 17 | 7/7 | 0 | 0 |
|
||||
| riscv64 | `logs/20260925-214855` | 10,746,616 | 10,746,599 | 17 | 17 | 7/7 | 0 | 0 |
|
||||
|
||||
The table shows the final (rajames) reading. At Hera's prompt, before MINT, the readings were 9,687,103 dispatches with 14 exceptions (all row growths), 5/5 conserved, and 268 cross-slot slot locks.
|
||||
|
||||
**Findings, reported and not fixed:**
|
||||
1. **The running VM isn't always the core's current VM.** Cross-slot slot locks went from 268 to 1,059,738 across MINT, WIREBIND, `USE` and `1 2 + .`, the same growth as word dispatches in that window.
|
||||
- A birth runs the child's capsule on the parent's stack (capsule_birth.c), and `USE` runs another VM's lines, so these dispatches are for a VM other than `sk_percpu()->current_vm`.
|
||||
- The Stadium locking is correct either way: it locks the dispatching VM's own slot.
|
||||
- But MC-3's claim (`running_on`) must cover a VM executed this way, not only one switched to.
|
||||
2. **Quota slots are never released.** `stadium.c` has no release path: `in_use` is set by grants and cleared only at boot.
|
||||
- The probe counted 13 slots in use against 7 registry entries. Boot self-tests grant quotas to test ids (`kernel_main.c`), and a dead VM keeps its slot and cells.
|
||||
- A long-running system eventually exhausts the RAM-sized slot table.
|
||||
3. **An owner's word admission can evict a live message cell.** The admission fallback evicts the owner's least-dense resident, skipping only pinned and containing cells. Hermes admits messages unpinned into the sender's quota.
|
||||
- After such an eviction, `msg->stadium_cell` is stale: the release would evict whatever occupies that cell next.
|
||||
- This run counted 0 such evictions (the quotas never ran dry of free cells), so it is latent.
|
||||
|
||||
**Standard sequence** (clean build, with the probe reverted; the tree is byte-identical to the saved stage diff).
|
||||
|
||||
| ISA | log | O-A (G, PREC, LOAD) | M7.1a hash | MINT / WIREBIND / USE / `1 2 + .` | idle growth | UNKNOWN / faults |
|
||||
|---|---|---|---|---|---|---|
|
||||
| amd64 | `logs/20260925-220202` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
|
||||
| aarch64 | `logs/20260925-221440` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
|
||||
| riscv64 | `logs/20260925-222740` | A=40 B=40, HHH, ok | `0x99b01134c3d3049c` | all ok, `3` | 0 B / 20 s | 0 / 0 |
|
||||
|
||||
- The G string is byte-identical to MC-0c and earlier on all three ISAs, so the O(ncells) audit scan now running under a slot lock didn't move it.
|
||||
- The `dict_hash` set is unchanged, and the ledger line reads `stadium_conserved(self)=true`.
|
||||
- No stop condition was hit. **MC-0d done, and with it MC-0: one core, fully locked, ready for MC-1.**
|
||||
|
||||
@@ -50,6 +50,11 @@
|
||||
* g_lock_switch switch-signal slot table and stamps
|
||||
* (capsule_vm_switch_signal.c)
|
||||
* g_lock_registry VM registry list and entries (capsule_birth.c)
|
||||
* g_lock_stadium Stadium's global part: quota-slot allocation
|
||||
* (grants) and the shared block table (MC-0d)
|
||||
* (slot lock) one Stadium quota slot: its lists, reservoir,
|
||||
* consumed, word row and cells (stadium.c's
|
||||
* StadiumVMQuota.lock; at most one held at a time)
|
||||
* g_lock_doe DoE region (doe_region.c)
|
||||
* g_lock_log log region (log_region.c)
|
||||
* g_lock_block block subsystem: device chain, caches, BAM,
|
||||
@@ -63,8 +68,10 @@
|
||||
* The switch table and the registry never call out while held: their walks
|
||||
* copy one entry under the lock and judge it outside. Hermes hands the
|
||||
* switch module its notifications after dropping its own lock (they read
|
||||
* Hermes queues back). Stadium's lock comes with MC-0d; it sits below
|
||||
* Hermes (Hermes admits and evicts cells).
|
||||
* Hermes queues back). Stadium sits below Hermes (Hermes admits, evicts and
|
||||
* decays cells) and below a VM's dict_lock (vm_core.c's heat bump dispatches
|
||||
* the word under it), and above block (MIGRATE calls blk_flush()); nothing
|
||||
* in the block path calls into Stadium.
|
||||
*/
|
||||
#ifndef STARKERNEL_SPINLOCK_H
|
||||
#define STARKERNEL_SPINLOCK_H
|
||||
@@ -102,6 +109,7 @@ uint64_t sk_lock_bad_releases(void);
|
||||
uint64_t sk_lock_held_at_switch(void);
|
||||
|
||||
extern sk_spinlock_t g_lock_registry;
|
||||
extern sk_spinlock_t g_lock_stadium;
|
||||
extern sk_spinlock_t g_lock_switch;
|
||||
extern sk_spinlock_t g_lock_hermes;
|
||||
extern sk_spinlock_t g_lock_doe;
|
||||
|
||||
@@ -584,6 +584,44 @@ uint64_t stadium_cell_heat_get(VMUuid vm_id, size_t cell_index);
|
||||
*/
|
||||
int stadium_cell_heat_set(VMUuid vm_id, size_t cell_index, uint64_t new_heat);
|
||||
|
||||
/*
|
||||
* FABRIC-3.10.8 Stage MC-0d: Stadium is partitioned by quota slot, one lock
|
||||
* per slot (Captain Bob, 2026-09-25). Every public function above locks the
|
||||
* slot it touches; a cell's operations lock the cell's owning slot. The word
|
||||
* and block dispatch paths hold their own VM's slot lock across a whole
|
||||
* dispatch and use the *_in_slot forms below, so another VM's dispatch never
|
||||
* contends them. g_lock_stadium (spinlock.h) serializes grants and the
|
||||
* block table; it is taken before a slot lock, never after.
|
||||
*/
|
||||
|
||||
/* Locks vm_id's quota slot and returns it, or -1 (no quota; nothing locked). */
|
||||
int stadium_slot_lock_vm(VMUuid vm_id);
|
||||
void stadium_slot_unlock(int slot);
|
||||
|
||||
/* Caller holds `slot`'s lock. */
|
||||
size_t stadium_admit_in_slot(int slot, const StadiumPatronHeader *candidate);
|
||||
int stadium_evict_in_slot(int slot, size_t cell_index); /* -1 unless slot owns the cell */
|
||||
uint64_t stadium_reservoir_pull_in_slot(int slot, uint64_t amount);
|
||||
void stadium_reservoir_push_in_slot(int slot, uint64_t amount);
|
||||
uint64_t stadium_reservoir_peek_in_slot(int slot);
|
||||
|
||||
/* The quota slot that owns cell_index (-1 out of range). A resident cell's
|
||||
* owner never changes. */
|
||||
int stadium_cell_owner_slot(size_t cell_index);
|
||||
|
||||
/* stadium_evict(), and *credited receives the heat returned to the owner's
|
||||
* reservoir, read under the owner's lock (kernel-Hermes's release ledger). */
|
||||
int stadium_evict_credit(size_t cell_index, uint64_t *credited);
|
||||
|
||||
/* kernel-Hermes's message decay, under the owning slot's lock: heat becomes
|
||||
* q48_mul(heat, factor_q48); the difference is recorded as the slot's
|
||||
* consumed heat and returned in *consumed_out. -1 if the cell isn't resident. */
|
||||
int stadium_cell_decay(size_t cell_index, uint64_t factor_q48, uint64_t *consumed_out);
|
||||
|
||||
/* Sets or clears STADIUM_FLAG_PIN under the owning slot's lock (session.c's
|
||||
* session_set_pinned(), the sole pin choke point). -1 if out of range. */
|
||||
int stadium_cell_set_pin(size_t cell_index, int pinned);
|
||||
|
||||
#endif /* __STARKERNEL__ */
|
||||
|
||||
#endif /* STARKERNEL_VM_STADIUM_H */
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -31,6 +31,7 @@
|
||||
#include "log.h"
|
||||
|
||||
sk_spinlock_t g_lock_registry = SK_SPINLOCK_INIT;
|
||||
sk_spinlock_t g_lock_stadium = SK_SPINLOCK_INIT;
|
||||
sk_spinlock_t g_lock_switch = SK_SPINLOCK_INIT;
|
||||
sk_spinlock_t g_lock_hermes = SK_SPINLOCK_INIT;
|
||||
sk_spinlock_t g_lock_doe = SK_SPINLOCK_INIT;
|
||||
@@ -180,6 +181,7 @@ uint64_t sk_lock_held_at_switch(void) { return g_held_at_switch; }
|
||||
static const char *lock_name(const volatile uint32_t *w)
|
||||
{
|
||||
if (w == &g_lock_registry.owner) return "registry";
|
||||
if (w == &g_lock_stadium.owner) return "stadium";
|
||||
if (w == &g_lock_switch.owner) return "switch";
|
||||
if (w == &g_lock_hermes.owner) return "hermes";
|
||||
if (w == &g_lock_doe.owner) return "doe";
|
||||
@@ -189,7 +191,7 @@ static const char *lock_name(const volatile uint32_t *w)
|
||||
if (w == &g_lock_heap.owner) return "heap";
|
||||
if (w == &g_lock_pmm.owner) return "pmm";
|
||||
if (w == &g_lock_vmm.owner) return "vmm";
|
||||
return "vm sf_mutex";
|
||||
return "vm sf_mutex or Stadium slot";
|
||||
}
|
||||
|
||||
/* Freestanding kvsnprintf has no %llx: hex by hand. */
|
||||
|
||||
@@ -321,24 +321,20 @@ static int release_locked(SkHermesMessage *msg) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* Read the patron's current heat BEFORE eviction -- stadium_evict()
|
||||
* zeroes the header as part of returning the cell to the free list,
|
||||
* and its own return value is a success code, not the amount
|
||||
* credited. Today (before task 2.5's decay exists) this always
|
||||
* equals exactly what was pulled at allocation; once decay exists,
|
||||
* this is the message's true remaining heat, which is what actually
|
||||
* flows back to the reservoir -- reading it here rather than
|
||||
* assuming the original pulled amount is what keeps this correct
|
||||
* without changes once task 2.5 lands. */
|
||||
remaining = stadium_cells()[msg->stadium_cell].header.heat;
|
||||
|
||||
/* stadium_evict() returns the departing patron's remaining heat to
|
||||
* its owner's reservoir itself (stadium.c: "the departing patron's
|
||||
* remaining heat must flow back to its owner's reservoir before the
|
||||
* cell returns to the free list") -- release does not touch the
|
||||
* reservoir directly, the eviction path does it, matching
|
||||
* MSG-FREE-NODE's own shape ("DUP 5 CELLS + @ STADIUM-EVICT DROP"). */
|
||||
rc = stadium_evict((size_t)msg->stadium_cell);
|
||||
/* The patron's remaining heat is what flows back to the reservoir:
|
||||
* what was pulled at allocation, less any decay. FABRIC-3.10.8 MC-0d:
|
||||
* stadium_evict_credit() reports exactly what it credited, read under
|
||||
* the owning slot's lock in the same step as the eviction -- was read
|
||||
* here beforehand, separately, straight out of the cell.
|
||||
*
|
||||
* The eviction returns that heat to its owner's reservoir itself
|
||||
* (stadium.c: "the departing patron's remaining heat must flow back to
|
||||
* its owner's reservoir before the cell returns to the free list") --
|
||||
* release does not touch the reservoir directly, the eviction path does
|
||||
* it, matching MSG-FREE-NODE's own shape ("DUP 5 CELLS + @ STADIUM-EVICT
|
||||
* DROP"). */
|
||||
remaining = 0;
|
||||
rc = stadium_evict_credit((size_t)msg->stadium_cell, &remaining);
|
||||
|
||||
/* Task 2.4: the one site where held/returned change on release. Only
|
||||
* on successful eviction -- a refused eviction leaves the patron
|
||||
@@ -363,8 +359,7 @@ int sk_hermes_release(SkHermesMessage *msg) {
|
||||
}
|
||||
|
||||
int sk_hermes_decay(SkHermesMessage *msg) {
|
||||
uint64_t before, after;
|
||||
StadiumCell *cell;
|
||||
uint64_t delta;
|
||||
|
||||
sk_spin_lock(&g_lock_hermes);
|
||||
if (!msg || !msg->in_use || msg->stadium_cell < 0) {
|
||||
@@ -372,17 +367,20 @@ int sk_hermes_decay(SkHermesMessage *msg) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
cell = &stadium_cells()[msg->stadium_cell];
|
||||
before = cell->header.heat;
|
||||
after = (uint64_t)q48_mul((q48_16_t)before, (q48_16_t)SK_HERMES_Q_DECAY);
|
||||
|
||||
cell->header.heat = after;
|
||||
/* FABRIC-3.10.8 MC-0d: the decay itself (heat *= SK_HERMES_Q_DECAY, the
|
||||
* difference recorded as the sender's consumed heat) happens in Stadium,
|
||||
* under the sender's slot lock -- was a direct write into the sender's
|
||||
* cell from here. Refused (-1, ledger untouched) if the cell isn't
|
||||
* resident. */
|
||||
if (stadium_cell_decay((size_t)msg->stadium_cell, (uint64_t)SK_HERMES_Q_DECAY, &delta) != 0) {
|
||||
sk_spin_unlock(&g_lock_hermes);
|
||||
return -1;
|
||||
}
|
||||
|
||||
/* Task 2.5: the one site where consumed increments (and where held
|
||||
* pays for it). SXXXVII.3: decay's delta is exact and in hand here. */
|
||||
sk_hermes_consumed += before - after;
|
||||
sk_hermes_held -= before - after;
|
||||
stadium_consumed_record(msg->owner, before - after);
|
||||
sk_hermes_consumed += delta;
|
||||
sk_hermes_held -= delta;
|
||||
audit_locked();
|
||||
sk_spin_unlock(&g_lock_hermes);
|
||||
|
||||
|
||||
@@ -140,11 +140,9 @@ void session_set_pinned(VMUuid vm_id, int pinned)
|
||||
cells = stadium_cells();
|
||||
if (!cells) return;
|
||||
|
||||
if (pinned) {
|
||||
cells[s->stadium_cell].header.flags |= STADIUM_FLAG_PIN;
|
||||
} else {
|
||||
cells[s->stadium_cell].header.flags &= (uint8_t)~STADIUM_FLAG_PIN;
|
||||
}
|
||||
/* FABRIC-3.10.8 MC-0d: under the cell's owning quota-slot lock (was a
|
||||
* direct write into the cell header from here). */
|
||||
(void)stadium_cell_set_pin((size_t)s->stadium_cell, pinned);
|
||||
}
|
||||
|
||||
int session_is_pinned(VMUuid vm_id)
|
||||
|
||||
+237
-55
@@ -43,6 +43,7 @@
|
||||
#include "vm.h" /* VM_MEMORY_SIZE -- the per-VM footprint stadium_max_vm_count() budgets against */
|
||||
#include "block_subsystem.h" /* blk_flush() -- STADIUM_BEHAVIOUR_MIGRATE's real write-back action */
|
||||
#include "starkernel/session.h" /* session_register()/session_set_pinned() -- FABRIC-2.md §H.12 step 4 */
|
||||
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0d -- per-slot locks, g_lock_stadium */
|
||||
|
||||
static StadiumCell *stadium_cell_array = (StadiumCell *)0;
|
||||
static uint8_t *stadium_bitmap = (uint8_t *)0;
|
||||
@@ -112,13 +113,39 @@ typedef struct {
|
||||
* have decayed away (recorded by kernel-Hermes,
|
||||
* FABRIC-3.6.md task 2.7). Full invariant:
|
||||
* Σ(resident) + reservoir + consumed == Q48_ONE. */
|
||||
volatile uint32_t lock; /* FABRIC-3.10.8 MC-0d: this slot's own lock (a
|
||||
* spinlock.h lock word). Covers every field above,
|
||||
* and this slot's cells: their headers, owner and
|
||||
* resident-list entries. Taken by the owning VM's
|
||||
* dispatch and by anything else touching the slot --
|
||||
* another VM's dispatch never contends it. */
|
||||
} StadiumVMQuota;
|
||||
|
||||
/* FABRIC-3.10.8 MC-0d (Captain Bob, 2026-09-25: per-slot locks).
|
||||
*
|
||||
* Stadium is partitioned by quota slot. The word-dispatch path takes only its
|
||||
* own VM's slot lock, so it takes no shared lock. Cross-slot work takes the
|
||||
* other slot's lock:
|
||||
* - kernel-Hermes evicting or decaying a sender's message cell from the
|
||||
* receiver's drain;
|
||||
* - KILL/reap evicting a dead VM's cell;
|
||||
* - conservation audits of another VM.
|
||||
* g_lock_stadium serializes what is truly global:
|
||||
* - quota-slot allocation (grants);
|
||||
* - stadium_blocks.c's one block table, shared by every VM (ruled: a shared
|
||||
* lock on the block path, recorded for MC-3 to measure).
|
||||
* Order: g_lock_stadium, then at most one slot lock (spinlock.h). The lock
|
||||
* helpers, slot_lock()/slot_unlock(), follow the quota table below. */
|
||||
|
||||
/* kmalloc'd at stadium_boot_init() to stadium_max_vm_count_val entries,
|
||||
* replacing the old static StadiumVMQuota stadium_quotas[STADIUM_MAX_VM_COUNT]
|
||||
* (2026-08-15: the bound is computed from RAM, not a compile-time constant). */
|
||||
static StadiumVMQuota *stadium_quotas = (StadiumVMQuota *)0;
|
||||
|
||||
/* MC-0d: one quota slot's lock (see StadiumVMQuota.lock). */
|
||||
static void slot_lock(int slot) { sk_lockword_acquire(&stadium_quotas[slot].lock); }
|
||||
static void slot_unlock(int slot) { sk_lockword_release(&stadium_quotas[slot].lock); }
|
||||
|
||||
/* Doubly-linked resident-list threading, one entry per cell, parallel to
|
||||
* stadium_owner[] -- kmalloc'd at stadium_boot_init() to stadium_ncells
|
||||
* entries each. Only meaningful for cells currently resident (bitmap bit
|
||||
@@ -162,7 +189,10 @@ static void resident_list_remove(int slot, size_t cell_index) {
|
||||
static int quota_slot_for_vm(VMUuid vm_id) {
|
||||
size_t i;
|
||||
for (i = 0; i < stadium_max_vm_count_val; i++) {
|
||||
if (stadium_quotas[i].in_use && vm_uuid_equal(stadium_quotas[i].vm_id, vm_id)) return i;
|
||||
/* MC-0d: in_use is published last, with release (stadium_grant_quota()),
|
||||
* so a slot seen in use has its vm_id and lists already in place. */
|
||||
if (__atomic_load_n(&stadium_quotas[i].in_use, __ATOMIC_ACQUIRE) &&
|
||||
vm_uuid_equal(stadium_quotas[i].vm_id, vm_id)) return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
@@ -281,6 +311,7 @@ int stadium_boot_init(void) {
|
||||
quotas[i].resident_head = STADIUM_CELL_NONE;
|
||||
quotas[i].reservoir = 0;
|
||||
quotas[i].consumed = 0;
|
||||
quotas[i].lock = 0;
|
||||
}
|
||||
}
|
||||
quotas[0].vm_id = vm_uuid_hera();
|
||||
@@ -424,16 +455,22 @@ uint64_t stadium_density(size_t cell_index) {
|
||||
/* STADIUM_FLAG_PIN moved to stadium.h (FABRIC-2.md §H.12 step 3) -- see its
|
||||
* doc comment there for why. */
|
||||
|
||||
/* FABRIC-3.10.8 MC-0d: one bitmap byte covers 8 cells, which can belong to
|
||||
* different quota slots, and each slot's lock covers only its own cells --
|
||||
* so the byte is changed with atomic OR/AND, never a plain read-modify-write. */
|
||||
static int bitmap_get(size_t cell_index) {
|
||||
return (stadium_bitmap[cell_index / 8u] >> (cell_index % 8u)) & 1u;
|
||||
return (__atomic_load_n(&stadium_bitmap[cell_index / 8u], __ATOMIC_ACQUIRE)
|
||||
>> (cell_index % 8u)) & 1u;
|
||||
}
|
||||
|
||||
static void bitmap_set(size_t cell_index) {
|
||||
stadium_bitmap[cell_index / 8u] |= (uint8_t)(1u << (cell_index % 8u));
|
||||
__atomic_fetch_or(&stadium_bitmap[cell_index / 8u],
|
||||
(uint8_t)(1u << (cell_index % 8u)), __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
static void bitmap_clear(size_t cell_index) {
|
||||
stadium_bitmap[cell_index / 8u] &= (uint8_t)~(1u << (cell_index % 8u));
|
||||
__atomic_fetch_and(&stadium_bitmap[cell_index / 8u],
|
||||
(uint8_t)~(1u << (cell_index % 8u)), __ATOMIC_RELEASE);
|
||||
}
|
||||
|
||||
/*
|
||||
@@ -447,7 +484,9 @@ static void bitmap_clear(size_t cell_index) {
|
||||
* purpose, so a wrongly-cleared pin cannot silently swallow the violation
|
||||
* via the ordinary refusal path instead of surfacing it.
|
||||
*/
|
||||
int stadium_evict(size_t cell_index) {
|
||||
/* MC-0d: the body, with the cell's owning slot locked. *credited (if given)
|
||||
* receives the heat returned to the owner's reservoir. */
|
||||
static int evict_locked(size_t cell_index, uint64_t *credited) {
|
||||
StadiumPatronHeader *header;
|
||||
uint16_t slot;
|
||||
|
||||
@@ -473,6 +512,7 @@ int stadium_evict(size_t cell_index) {
|
||||
* Q48_ONE. Captured BEFORE the zero-fill below, which would otherwise
|
||||
* destroy it. */
|
||||
stadium_quotas[slot].reservoir += header->heat;
|
||||
if (credited) *credited = header->heat;
|
||||
|
||||
{
|
||||
uint8_t *raw = (uint8_t *)header;
|
||||
@@ -493,6 +533,34 @@ int stadium_evict(size_t cell_index) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* Locks the slot that owns cell_index. A resident cell's owner never changes
|
||||
* (grants move only free cells), and a free cell's only under its donor's
|
||||
* lock -- so the owner is re-read once locked, and the lock retaken if it
|
||||
* moved in between. Returns the locked slot, or -1 (out of range). */
|
||||
static int lock_cell_owner(size_t cell_index) {
|
||||
int slot;
|
||||
if (!stadium_initialized || cell_index >= stadium_ncells) return -1;
|
||||
for (;;) {
|
||||
slot = (int)__atomic_load_n(&stadium_owner[cell_index], __ATOMIC_ACQUIRE);
|
||||
slot_lock(slot);
|
||||
if ((int)stadium_owner[cell_index] == slot) return slot;
|
||||
slot_unlock(slot);
|
||||
}
|
||||
}
|
||||
|
||||
int stadium_evict(size_t cell_index) {
|
||||
return stadium_evict_credit(cell_index, (uint64_t *)0);
|
||||
}
|
||||
|
||||
int stadium_evict_credit(size_t cell_index, uint64_t *credited) {
|
||||
int slot = lock_cell_owner(cell_index);
|
||||
int rc;
|
||||
if (slot < 0) return -1;
|
||||
rc = evict_locked(cell_index, credited);
|
||||
slot_unlock(slot);
|
||||
return rc;
|
||||
}
|
||||
|
||||
/*
|
||||
* FABRIC-0.md §19.3, §22.3, item 3.7: admit into vm_id's own quota. Pops that
|
||||
* VM's free-list head first (O(1), no comparison needed -- §19.3's density
|
||||
@@ -503,13 +571,26 @@ int stadium_evict(size_t cell_index) {
|
||||
* evicting it only if the candidate is strictly denser.
|
||||
*/
|
||||
size_t stadium_admit(VMUuid vm_id, const StadiumPatronHeader *candidate) {
|
||||
int slot;
|
||||
int slot;
|
||||
size_t idx;
|
||||
|
||||
if (!stadium_initialized || !candidate) return STADIUM_CELL_NONE;
|
||||
slot = quota_slot_for_vm(vm_id);
|
||||
if (slot < 0) return STADIUM_CELL_NONE;
|
||||
slot_lock(slot);
|
||||
idx = stadium_admit_in_slot(slot, candidate);
|
||||
slot_unlock(slot);
|
||||
return idx;
|
||||
}
|
||||
|
||||
/* MC-0d: stadium_admit() with `slot` already locked by the caller. */
|
||||
size_t stadium_admit_in_slot(int slot, const StadiumPatronHeader *candidate) {
|
||||
size_t idx;
|
||||
size_t least_dense_index = STADIUM_CELL_NONE;
|
||||
uint64_t least_dense_value = 0;
|
||||
uint64_t candidate_density;
|
||||
|
||||
if (!stadium_initialized || !candidate) return STADIUM_CELL_NONE;
|
||||
if (!stadium_initialized || !candidate || slot < 0) return STADIUM_CELL_NONE;
|
||||
|
||||
/* Multi-cell patrons need a continuation chain, and no header field is
|
||||
* documented as carrying one's first index -- see this function's doc
|
||||
@@ -524,9 +605,6 @@ size_t stadium_admit(VMUuid vm_id, const StadiumPatronHeader *candidate) {
|
||||
if (candidate->contains != STADIUM_CONTAINS_NONE &&
|
||||
candidate->contains >= stadium_ncells) return STADIUM_CELL_NONE;
|
||||
|
||||
slot = quota_slot_for_vm(vm_id);
|
||||
if (slot < 0) return STADIUM_CELL_NONE;
|
||||
|
||||
if (stadium_quotas[slot].free_head != STADIUM_CELL_NONE) {
|
||||
idx = stadium_quotas[slot].free_head;
|
||||
stadium_quotas[slot].free_head = link_to_size(stadium_cell_array[idx].header.link);
|
||||
@@ -582,13 +660,12 @@ size_t stadium_admit(VMUuid vm_id, const StadiumPatronHeader *candidate) {
|
||||
candidate_density = candidate->heat / (uint64_t)candidate->mass; /* mass == 1, guaranteed above */
|
||||
if (candidate_density <= least_dense_value) return STADIUM_CELL_NONE;
|
||||
|
||||
if (stadium_evict(least_dense_index) != 0) return STADIUM_CELL_NONE;
|
||||
if (evict_locked(least_dense_index, (uint64_t *)0) != 0) return STADIUM_CELL_NONE;
|
||||
|
||||
/* stadium_evict() just pushed least_dense_index onto quotas[slot]'s free
|
||||
/* evict_locked() just pushed least_dense_index onto quotas[slot]'s free
|
||||
* list -- its owner is `slot`, the same quota we scoped the search to.
|
||||
* Single-threaded today (§21.1/§21.2: real concurrency is step-one, not
|
||||
* yet built), so nothing else can have touched the list meanwhile; pop
|
||||
* it straight back off. */
|
||||
* MC-0d: the slot has stayed locked throughout, so nothing else can have
|
||||
* touched the list meanwhile; pop it straight back off. */
|
||||
idx = stadium_quotas[slot].free_head;
|
||||
stadium_quotas[slot].free_head = link_to_size(stadium_cell_array[idx].header.link);
|
||||
stadium_quotas[slot].free_count--;
|
||||
@@ -616,19 +693,28 @@ int stadium_grant_quota(VMUuid new_vm_id, VMUuid from_vm_id) {
|
||||
size_t half, idx, last_new, new_head, remainder_head;
|
||||
|
||||
if (!stadium_initialized) return -1;
|
||||
if (quota_slot_for_vm(new_vm_id) >= 0) return -1;
|
||||
|
||||
/* MC-0d: slot allocation is global (g_lock_stadium: two grants must not
|
||||
* pick the same free slot); the split is the donor's (its slot lock). */
|
||||
sk_spin_lock(&g_lock_stadium);
|
||||
if (quota_slot_for_vm(new_vm_id) >= 0) { sk_spin_unlock(&g_lock_stadium); return -1; }
|
||||
|
||||
from_slot = quota_slot_for_vm(from_vm_id);
|
||||
if (from_slot < 0) return -1;
|
||||
if (from_slot < 0) { sk_spin_unlock(&g_lock_stadium); return -1; }
|
||||
|
||||
new_slot = -1;
|
||||
for (i = 0; i < (int)stadium_max_vm_count_val; i++) {
|
||||
if (!stadium_quotas[i].in_use) { new_slot = i; break; }
|
||||
}
|
||||
if (new_slot < 0) return -1;
|
||||
if (new_slot < 0) { sk_spin_unlock(&g_lock_stadium); return -1; }
|
||||
|
||||
slot_lock(from_slot);
|
||||
half = stadium_quotas[from_slot].free_count / 2;
|
||||
if (half == 0) return -1; /* fewer than 2 free cells -- nothing to split */
|
||||
if (half == 0) { /* fewer than 2 free cells -- nothing to split */
|
||||
slot_unlock(from_slot);
|
||||
sk_spin_unlock(&g_lock_stadium);
|
||||
return -1;
|
||||
}
|
||||
|
||||
new_head = stadium_quotas[from_slot].free_head;
|
||||
idx = new_head;
|
||||
@@ -644,15 +730,19 @@ int stadium_grant_quota(VMUuid new_vm_id, VMUuid from_vm_id) {
|
||||
stadium_cell_array[last_new].header.link = size_to_link(STADIUM_CELL_NONE);
|
||||
stadium_quotas[from_slot].free_head = remainder_head;
|
||||
stadium_quotas[from_slot].free_count -= half;
|
||||
slot_unlock(from_slot);
|
||||
|
||||
/* Not yet visible to anyone (in_use is 0 until the release store below). */
|
||||
stadium_quotas[new_slot].vm_id = new_vm_id;
|
||||
stadium_quotas[new_slot].in_use = 1;
|
||||
stadium_quotas[new_slot].free_head = new_head;
|
||||
stadium_quotas[new_slot].free_count = half;
|
||||
stadium_quotas[new_slot].resident_head = STADIUM_CELL_NONE;
|
||||
stadium_quotas[new_slot].reservoir = Q48_ONE;
|
||||
stadium_quotas[new_slot].consumed = 0;
|
||||
stadium_quotas[new_slot].lock = 0;
|
||||
__atomic_store_n(&stadium_quotas[new_slot].in_use, 1, __ATOMIC_RELEASE);
|
||||
|
||||
sk_spin_unlock(&g_lock_stadium);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@@ -703,69 +793,143 @@ static int owned_resident_slot(VMUuid vm_id, size_t cell_index) {
|
||||
}
|
||||
|
||||
uint64_t stadium_cell_heat_get(VMUuid vm_id, size_t cell_index) {
|
||||
if (owned_resident_slot(vm_id, cell_index) < 0) return 0;
|
||||
return stadium_cell_array[cell_index].header.heat;
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
uint64_t heat = 0;
|
||||
if (slot < 0) return 0;
|
||||
slot_lock(slot);
|
||||
if (owned_resident_slot(vm_id, cell_index) >= 0) heat = stadium_cell_array[cell_index].header.heat;
|
||||
slot_unlock(slot);
|
||||
return heat;
|
||||
}
|
||||
|
||||
int stadium_cell_heat_set(VMUuid vm_id, size_t cell_index, uint64_t new_heat) {
|
||||
int slot = owned_resident_slot(vm_id, cell_index);
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
uint64_t old_heat, delta, pulled;
|
||||
int rc = 0;
|
||||
|
||||
if (slot < 0) return -1;
|
||||
slot_lock(slot);
|
||||
if (owned_resident_slot(vm_id, cell_index) < 0) { slot_unlock(slot); return -1; }
|
||||
|
||||
old_heat = stadium_cell_array[cell_index].header.heat;
|
||||
if (new_heat == old_heat) return 0;
|
||||
|
||||
if (new_heat > old_heat) {
|
||||
delta = new_heat - old_heat;
|
||||
pulled = (delta > stadium_quotas[slot].reservoir) ? stadium_quotas[slot].reservoir : delta;
|
||||
if (pulled < delta) return -1; /* insufficient -- no partial credit, no mutation */
|
||||
stadium_quotas[slot].reservoir -= pulled;
|
||||
} else {
|
||||
if (pulled < delta) {
|
||||
rc = -1; /* insufficient -- no partial credit, no mutation */
|
||||
} else {
|
||||
stadium_quotas[slot].reservoir -= pulled;
|
||||
stadium_cell_array[cell_index].header.heat = new_heat;
|
||||
}
|
||||
} else if (new_heat < old_heat) {
|
||||
delta = old_heat - new_heat;
|
||||
stadium_quotas[slot].reservoir += delta;
|
||||
stadium_cell_array[cell_index].header.heat = new_heat;
|
||||
}
|
||||
slot_unlock(slot);
|
||||
return rc;
|
||||
}
|
||||
|
||||
stadium_cell_array[cell_index].header.heat = new_heat;
|
||||
/* MC-0d: kernel-Hermes's message decay (was a direct write to the sender's
|
||||
* cell from kernel_hermes.c). Under the owning slot's lock: the cell's heat
|
||||
* becomes q48_mul(heat, factor_q48), and the difference is recorded as that
|
||||
* slot's consumed heat -- the same arithmetic sk_hermes_decay() did inline.
|
||||
* *consumed_out receives the difference. -1 if the cell isn't resident. */
|
||||
int stadium_cell_decay(size_t cell_index, uint64_t factor_q48, uint64_t *consumed_out) {
|
||||
int slot = lock_cell_owner(cell_index);
|
||||
uint64_t before, after;
|
||||
if (slot < 0) return -1;
|
||||
if (!bitmap_get(cell_index)) { slot_unlock(slot); return -1; }
|
||||
before = stadium_cell_array[cell_index].header.heat;
|
||||
after = (uint64_t)q48_mul((q48_16_t)before, (q48_16_t)factor_q48);
|
||||
stadium_cell_array[cell_index].header.heat = after;
|
||||
stadium_quotas[slot].consumed += before - after;
|
||||
slot_unlock(slot);
|
||||
if (consumed_out) *consumed_out = before - after;
|
||||
return 0;
|
||||
}
|
||||
|
||||
uint64_t stadium_reservoir_pull(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
uint64_t pulled;
|
||||
/* MC-0d: session.c's pin (was a direct write into the cell header). */
|
||||
int stadium_cell_set_pin(size_t cell_index, int pinned) {
|
||||
int slot = lock_cell_owner(cell_index);
|
||||
if (slot < 0) return -1;
|
||||
if (pinned) stadium_cell_array[cell_index].header.flags |= STADIUM_FLAG_PIN;
|
||||
else stadium_cell_array[cell_index].header.flags &= (uint8_t)~STADIUM_FLAG_PIN;
|
||||
slot_unlock(slot);
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (slot < 0) return 0;
|
||||
|
||||
pulled = (amount > stadium_quotas[slot].reservoir) ? stadium_quotas[slot].reservoir : amount;
|
||||
/* The *_in_slot forms: caller holds `slot`'s lock (stadium_slot_lock()). */
|
||||
uint64_t stadium_reservoir_pull_in_slot(int slot, uint64_t amount) {
|
||||
uint64_t pulled = (amount > stadium_quotas[slot].reservoir) ? stadium_quotas[slot].reservoir : amount;
|
||||
stadium_quotas[slot].reservoir -= pulled;
|
||||
return pulled;
|
||||
}
|
||||
|
||||
void stadium_reservoir_push(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
|
||||
if (slot < 0) return;
|
||||
void stadium_reservoir_push_in_slot(int slot, uint64_t amount) {
|
||||
stadium_quotas[slot].reservoir += amount;
|
||||
}
|
||||
|
||||
uint64_t stadium_reservoir_peek(VMUuid vm_id) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
|
||||
if (slot < 0) return 0;
|
||||
uint64_t stadium_reservoir_peek_in_slot(int slot) {
|
||||
return stadium_quotas[slot].reservoir;
|
||||
}
|
||||
|
||||
int stadium_evict_in_slot(int slot, size_t cell_index) {
|
||||
if (cell_index >= stadium_ncells || (int)stadium_owner[cell_index] != slot) return -1;
|
||||
return evict_locked(cell_index, (uint64_t *)0);
|
||||
}
|
||||
|
||||
int stadium_cell_owner_slot(size_t cell_index) {
|
||||
if (cell_index >= stadium_ncells) return -1;
|
||||
return (int)__atomic_load_n(&stadium_owner[cell_index], __ATOMIC_ACQUIRE);
|
||||
}
|
||||
|
||||
int stadium_slot_lock_vm(VMUuid vm_id) {
|
||||
int slot;
|
||||
if (!stadium_initialized) return -1;
|
||||
slot = quota_slot_for_vm(vm_id);
|
||||
if (slot >= 0) slot_lock(slot);
|
||||
return slot;
|
||||
}
|
||||
|
||||
void stadium_slot_unlock(int slot) {
|
||||
if (slot >= 0) slot_unlock(slot);
|
||||
}
|
||||
|
||||
uint64_t stadium_reservoir_pull(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
uint64_t pulled;
|
||||
if (slot < 0) return 0;
|
||||
pulled = stadium_reservoir_pull_in_slot(slot, amount);
|
||||
slot_unlock(slot);
|
||||
return pulled;
|
||||
}
|
||||
|
||||
void stadium_reservoir_push(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) return;
|
||||
stadium_reservoir_push_in_slot(slot, amount);
|
||||
slot_unlock(slot);
|
||||
}
|
||||
|
||||
uint64_t stadium_reservoir_peek(VMUuid vm_id) {
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
uint64_t r;
|
||||
if (slot < 0) return 0;
|
||||
r = stadium_reservoir_peek_in_slot(slot);
|
||||
slot_unlock(slot);
|
||||
return r;
|
||||
}
|
||||
|
||||
int stadium_quota_slot_for_vm(VMUuid vm_id) {
|
||||
return quota_slot_for_vm(vm_id);
|
||||
}
|
||||
|
||||
uint64_t stadium_resident_sum(VMUuid vm_id) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
/* The full-array scan (the stronger audit: it doesn't trust the resident
|
||||
* list). Caller holds `slot`'s lock. */
|
||||
static uint64_t resident_sum_locked(int slot) {
|
||||
uint64_t sum = 0;
|
||||
size_t i;
|
||||
|
||||
if (slot < 0) return 0;
|
||||
|
||||
for (i = 0; i < stadium_ncells; i++) {
|
||||
if (!bitmap_get(i)) continue;
|
||||
if (stadium_owner[i] != (uint16_t)slot) continue;
|
||||
@@ -774,23 +938,41 @@ uint64_t stadium_resident_sum(VMUuid vm_id) {
|
||||
return sum;
|
||||
}
|
||||
|
||||
void stadium_consumed_record(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
uint64_t stadium_resident_sum(VMUuid vm_id) {
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
uint64_t sum;
|
||||
if (slot < 0) return 0;
|
||||
sum = resident_sum_locked(slot);
|
||||
slot_unlock(slot);
|
||||
return sum;
|
||||
}
|
||||
|
||||
void stadium_consumed_record(VMUuid vm_id, uint64_t amount) {
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) return;
|
||||
stadium_quotas[slot].consumed += amount;
|
||||
slot_unlock(slot);
|
||||
}
|
||||
|
||||
uint64_t stadium_consumed_peek(VMUuid vm_id) {
|
||||
int slot = quota_slot_for_vm(vm_id);
|
||||
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
uint64_t c;
|
||||
if (slot < 0) return 0;
|
||||
return stadium_quotas[slot].consumed;
|
||||
c = stadium_quotas[slot].consumed;
|
||||
slot_unlock(slot);
|
||||
return c;
|
||||
}
|
||||
|
||||
/* MC-0d: the three terms read under one hold of the slot's lock, so the sum
|
||||
* is of one instant (read separately, a concurrent pull or eviction between
|
||||
* them could make a conserved slot read as drifted). */
|
||||
int stadium_conserved(VMUuid vm_id) {
|
||||
return (stadium_resident_sum(vm_id) + stadium_reservoir_peek(vm_id) +
|
||||
stadium_consumed_peek(vm_id)) == (uint64_t)Q48_ONE;
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
uint64_t total;
|
||||
if (slot < 0) return 0; /* no quota: 0 + 0 + 0 != Q48_ONE, as before */
|
||||
total = resident_sum_locked(slot) + stadium_quotas[slot].reservoir + stadium_quotas[slot].consumed;
|
||||
slot_unlock(slot);
|
||||
return total == (uint64_t)Q48_ONE;
|
||||
}
|
||||
|
||||
/*
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
#include "starkernel/console.h"
|
||||
#include "starkernel/kmalloc.h"
|
||||
#include "starkernel/q48_16.h" /* Q48_ONE */
|
||||
#include "starkernel/spinlock.h" /* FABRIC-3.10.8 MC-0d -- g_lock_stadium (the shared block table) */
|
||||
#include "starforth_config.h" /* STADIUM_BLOCK_HEAT_QUANTUM, STADIUM_BLOCK_COOL_RATE_Q48,
|
||||
STADIUM_BLOCK_TRACK_CAP_MULT */
|
||||
|
||||
@@ -196,24 +197,40 @@ static size_t table_probe(int16_t quota_slot, uint32_t lbn, int *out_found) {
|
||||
* pulls) depend on. Own copy, not shared code across files -- same
|
||||
* decision stadium_words.c's own version already made.
|
||||
*/
|
||||
static uint64_t block_dispatch_pull(VMUuid vm_id, uint64_t want) {
|
||||
uint64_t available = stadium_reservoir_peek(vm_id);
|
||||
static uint64_t block_dispatch_pull(int slot, uint64_t want) {
|
||||
/* MC-0d: `slot` is locked by the caller, so the peek and the pull are
|
||||
* one step. */
|
||||
uint64_t available = stadium_reservoir_peek_in_slot(slot);
|
||||
uint64_t floor = Q48_ONE / 3;
|
||||
uint64_t pullable = (available > floor) ? (available - floor) : 0;
|
||||
uint64_t capped = (want < pullable) ? want : pullable;
|
||||
return stadium_reservoir_pull(vm_id, capped);
|
||||
return stadium_reservoir_pull_in_slot(slot, capped);
|
||||
}
|
||||
|
||||
/* FABRIC-3.10.8 MC-0d: block_slots[] is one table shared by every VM, so a
|
||||
* block dispatch takes g_lock_stadium for the table (the one shared-lock
|
||||
* Stadium path, as ruled 2026-09-25 -- block access is rare next to word
|
||||
* dispatch), then this VM's own slot lock for its cells and reservoir. */
|
||||
static void block_dispatch_locked(int slot, uint32_t lbn, uint64_t heartbeat_ticks);
|
||||
|
||||
void stadium_block_dispatch(VMUuid vm_id, uint32_t lbn, uint64_t heartbeat_ticks) {
|
||||
int slot;
|
||||
int found;
|
||||
size_t idx;
|
||||
int stale = 0;
|
||||
int slot;
|
||||
|
||||
if (!blocks_initialized) return;
|
||||
|
||||
slot = stadium_quota_slot_for_vm(vm_id);
|
||||
if (slot < 0) return;
|
||||
sk_spin_lock(&g_lock_stadium);
|
||||
slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot >= 0) {
|
||||
block_dispatch_locked(slot, lbn, heartbeat_ticks);
|
||||
stadium_slot_unlock(slot);
|
||||
}
|
||||
sk_spin_unlock(&g_lock_stadium);
|
||||
}
|
||||
|
||||
static void block_dispatch_locked(int slot, uint32_t lbn, uint64_t heartbeat_ticks) {
|
||||
int found;
|
||||
size_t idx;
|
||||
int stale = 0;
|
||||
|
||||
idx = table_probe((int16_t)slot, lbn, &found);
|
||||
|
||||
@@ -223,7 +240,8 @@ void stadium_block_dispatch(VMUuid vm_id, uint32_t lbn, uint64_t heartbeat_ticks
|
||||
|
||||
if (cell == STADIUM_CELL_NONE || cell >= stadium_cell_count() || !cell_is_resident(cell)) {
|
||||
stale = 1;
|
||||
} else if (stadium_cells()[cell].header.identity != (uint64_t)lbn) {
|
||||
} else if (stadium_cell_owner_slot(cell) != slot || /* MC-0d: still this slot's cell */
|
||||
stadium_cells()[cell].header.identity != (uint64_t)lbn) {
|
||||
stale = 1;
|
||||
}
|
||||
|
||||
@@ -237,12 +255,12 @@ void stadium_block_dispatch(VMUuid vm_id, uint32_t lbn, uint64_t heartbeat_ticks
|
||||
if (cooled > h->heat) cooled = h->heat;
|
||||
if (cooled > 0) {
|
||||
h->heat -= cooled;
|
||||
stadium_reservoir_push(vm_id, cooled);
|
||||
stadium_reservoir_push_in_slot(slot, cooled);
|
||||
}
|
||||
bs->last_decay_tick = heartbeat_ticks;
|
||||
}
|
||||
|
||||
h->heat += block_dispatch_pull(vm_id, (uint64_t)STADIUM_BLOCK_HEAT_QUANTUM);
|
||||
h->heat += block_dispatch_pull(slot, (uint64_t)STADIUM_BLOCK_HEAT_QUANTUM);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -254,7 +272,7 @@ void stadium_block_dispatch(VMUuid vm_id, uint32_t lbn, uint64_t heartbeat_ticks
|
||||
if (idx >= block_slots_cap) return; /* table full: skip tracking this touch */
|
||||
|
||||
{
|
||||
uint64_t pulled = block_dispatch_pull(vm_id, (uint64_t)STADIUM_BLOCK_HEAT_QUANTUM);
|
||||
uint64_t pulled = block_dispatch_pull(slot, (uint64_t)STADIUM_BLOCK_HEAT_QUANTUM);
|
||||
StadiumPatronHeader candidate;
|
||||
uint8_t *raw = (uint8_t *)&candidate;
|
||||
size_t i;
|
||||
@@ -270,9 +288,9 @@ void stadium_block_dispatch(VMUuid vm_id, uint32_t lbn, uint64_t heartbeat_ticks
|
||||
candidate.flags = 0; /* unpinned */
|
||||
candidate.behaviour = (uint8_t)STADIUM_BEHAVIOUR_MIGRATE;
|
||||
|
||||
cell_idx = stadium_admit(vm_id, &candidate);
|
||||
cell_idx = stadium_admit_in_slot(slot, &candidate);
|
||||
if (cell_idx == STADIUM_CELL_NONE) {
|
||||
stadium_reservoir_push(vm_id, pulled); /* rollback: preserve conservation */
|
||||
stadium_reservoir_push_in_slot(slot, pulled); /* rollback: preserve conservation */
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -304,13 +322,22 @@ static void console_put_u64(uint64_t v) {
|
||||
}
|
||||
|
||||
void stadium_blocks_print_boot_diagnostics(VMUuid vm_id) {
|
||||
int slot = stadium_quota_slot_for_vm(vm_id);
|
||||
int slot;
|
||||
uint64_t promotions = 0, evictions = 0;
|
||||
uint64_t resident_sum = 0;
|
||||
uint64_t reservoir;
|
||||
size_t i;
|
||||
|
||||
if (!blocks_initialized || slot < 0) {
|
||||
if (!blocks_initialized) {
|
||||
console_println("Stadium blocks: not initialized or vm_id holds no quota");
|
||||
return;
|
||||
}
|
||||
/* MC-0d: the table (g_lock_stadium), then this slot's cells. Printing
|
||||
* waits until both are released. */
|
||||
sk_spin_lock(&g_lock_stadium);
|
||||
slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) {
|
||||
sk_spin_unlock(&g_lock_stadium);
|
||||
console_println("Stadium blocks: not initialized or vm_id holds no quota");
|
||||
return;
|
||||
}
|
||||
@@ -334,7 +361,9 @@ void stadium_blocks_print_boot_diagnostics(VMUuid vm_id) {
|
||||
resident_sum += stadium_cells()[cell].header.heat;
|
||||
}
|
||||
|
||||
reservoir = stadium_reservoir_peek(vm_id);
|
||||
reservoir = stadium_reservoir_peek_in_slot(slot);
|
||||
stadium_slot_unlock(slot);
|
||||
sk_spin_unlock(&g_lock_stadium);
|
||||
|
||||
console_puts("Stadium blocks: promotions=");
|
||||
console_put_u64(promotions);
|
||||
|
||||
@@ -92,7 +92,7 @@ static int grow_row(int slot, uint32_t word_id) {
|
||||
}
|
||||
row = (StadiumWordSlot *)kmalloc((size_t)new_cap * sizeof(StadiumWordSlot));
|
||||
if (!row) {
|
||||
row_grow_failures++;
|
||||
__atomic_add_fetch(&row_grow_failures, 1, __ATOMIC_RELAXED); /* MC-0d: any slot's dispatch */
|
||||
log_message(LOG_ERROR, "Stadium word row can't grow: not admitted");
|
||||
return -1;
|
||||
}
|
||||
@@ -168,12 +168,14 @@ static int cell_is_resident(size_t idx) {
|
||||
* NOT floored -- only word-execution admission respects this ceiling on
|
||||
* its own consumption.
|
||||
*/
|
||||
static uint64_t word_dispatch_pull(VMUuid vm_id, uint64_t want) {
|
||||
uint64_t available = stadium_reservoir_peek(vm_id);
|
||||
static uint64_t word_dispatch_pull(int slot, uint64_t want) {
|
||||
/* MC-0d: `slot` is locked by the caller, so the peek and the pull are
|
||||
* one step. */
|
||||
uint64_t available = stadium_reservoir_peek_in_slot(slot);
|
||||
uint64_t floor = Q48_ONE / 3;
|
||||
uint64_t pullable = (available > floor) ? (available - floor) : 0;
|
||||
uint64_t capped = (want < pullable) ? want : pullable;
|
||||
return stadium_reservoir_pull(vm_id, capped);
|
||||
return stadium_reservoir_pull_in_slot(slot, capped);
|
||||
}
|
||||
|
||||
/*
|
||||
@@ -195,8 +197,14 @@ static size_t resolve_resident_cell(int slot, uint32_t word_id) {
|
||||
stat_evictions[slot]++;
|
||||
return STADIUM_CELL_NONE;
|
||||
}
|
||||
/* FABRIC-3.10.8 MC-0d: the cell must also still be this slot's. A word
|
||||
* cell evicted to this slot's free list can move to another VM in a
|
||||
* grant and be admitted there -- identity is a word id, which another
|
||||
* VM's words share -- and this row would then heat the other VM's cell
|
||||
* (and, with worker cores, touch it without that slot's lock). */
|
||||
cells = stadium_cells();
|
||||
if (cells[cell].header.identity != (uint64_t)word_id) {
|
||||
if (stadium_cell_owner_slot(cell) != slot ||
|
||||
cells[cell].header.identity != (uint64_t)word_id) {
|
||||
word_slots[slot][word_id].cell_index = STADIUM_CELL_NONE;
|
||||
stat_evictions[slot]++;
|
||||
return STADIUM_CELL_NONE;
|
||||
@@ -211,7 +219,7 @@ uint64_t stadium_words_resident_heat(VMUuid vm_id) {
|
||||
StadiumCell *cells;
|
||||
|
||||
if (!words_initialized) return 0;
|
||||
slot = stadium_quota_slot_for_vm(vm_id);
|
||||
slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) return 0;
|
||||
|
||||
cells = stadium_cells();
|
||||
@@ -220,18 +228,30 @@ uint64_t stadium_words_resident_heat(VMUuid vm_id) {
|
||||
if (cell == STADIUM_CELL_NONE) continue;
|
||||
sum += cells[cell].header.heat;
|
||||
}
|
||||
stadium_slot_unlock(slot);
|
||||
return sum;
|
||||
}
|
||||
|
||||
/* FABRIC-3.10.8 MC-0d: the whole dispatch runs under this VM's own quota
|
||||
* slot lock and nothing else (the row, the cell, the reservoir, admission
|
||||
* and its eviction fallback are all this slot's) -- no shared lock on the
|
||||
* word path. A row grow's kmalloc is the one exception (the heap lock). */
|
||||
static void word_dispatch_locked(int slot, uint32_t word_id, uint64_t heartbeat_ticks);
|
||||
|
||||
void stadium_word_dispatch(VMUuid vm_id, uint32_t word_id, uint64_t heartbeat_ticks) {
|
||||
int slot;
|
||||
size_t cell;
|
||||
int slot;
|
||||
|
||||
if (!words_initialized) return;
|
||||
if (word_id == WORD_ID_INVALID) return;
|
||||
|
||||
slot = stadium_quota_slot_for_vm(vm_id);
|
||||
slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) return;
|
||||
word_dispatch_locked(slot, word_id, heartbeat_ticks);
|
||||
stadium_slot_unlock(slot);
|
||||
}
|
||||
|
||||
static void word_dispatch_locked(int slot, uint32_t word_id, uint64_t heartbeat_ticks) {
|
||||
size_t cell;
|
||||
|
||||
if (word_id >= row_cap[slot] && grow_row(slot, word_id) != 0) return;
|
||||
|
||||
@@ -251,19 +271,19 @@ void stadium_word_dispatch(VMUuid vm_id, uint32_t word_id, uint64_t heartbeat_ti
|
||||
if (cooled > h->heat) cooled = h->heat;
|
||||
if (cooled > 0) {
|
||||
h->heat -= cooled;
|
||||
stadium_reservoir_push(vm_id, cooled);
|
||||
stadium_reservoir_push_in_slot(slot, cooled);
|
||||
}
|
||||
word_slots[slot][word_id].last_decay_tick = heartbeat_ticks;
|
||||
}
|
||||
|
||||
h->heat += word_dispatch_pull(vm_id, (uint64_t)STADIUM_WORD_HEAT_QUANTUM);
|
||||
h->heat += word_dispatch_pull(slot, (uint64_t)STADIUM_WORD_HEAT_QUANTUM);
|
||||
return;
|
||||
}
|
||||
|
||||
/* Not resident: Option B starter-grant admission (§17.7). execution_heat
|
||||
* plays no role -- density is decided entirely by the pulled quantum. */
|
||||
{
|
||||
uint64_t pulled = word_dispatch_pull(vm_id, (uint64_t)STADIUM_WORD_HEAT_QUANTUM);
|
||||
uint64_t pulled = word_dispatch_pull(slot, (uint64_t)STADIUM_WORD_HEAT_QUANTUM);
|
||||
StadiumPatronHeader candidate;
|
||||
uint8_t *raw = (uint8_t *)&candidate;
|
||||
size_t i;
|
||||
@@ -279,9 +299,9 @@ void stadium_word_dispatch(VMUuid vm_id, uint32_t word_id, uint64_t heartbeat_ti
|
||||
candidate.flags = 0; /* unpinned -- words carry no pin exception (§17.3) */
|
||||
candidate.behaviour = (uint8_t)STADIUM_BEHAVIOUR_COOL;
|
||||
|
||||
idx = stadium_admit(vm_id, &candidate);
|
||||
idx = stadium_admit_in_slot(slot, &candidate);
|
||||
if (idx == STADIUM_CELL_NONE) {
|
||||
stadium_reservoir_push(vm_id, pulled); /* rollback: preserve conservation */
|
||||
stadium_reservoir_push_in_slot(slot, pulled); /* rollback: preserve conservation */
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -298,25 +318,25 @@ void stadium_word_forget(VMUuid vm_id, uint32_t word_id) {
|
||||
if (!words_initialized) return;
|
||||
if (word_id == WORD_ID_INVALID) return;
|
||||
|
||||
slot = stadium_quota_slot_for_vm(vm_id);
|
||||
slot = stadium_slot_lock_vm(vm_id);
|
||||
if (slot < 0) return;
|
||||
|
||||
if (word_id >= row_cap[slot]) return; /* never dispatched here -- can't be resident */
|
||||
|
||||
cell = resolve_resident_cell(slot, word_id);
|
||||
if (cell == STADIUM_CELL_NONE) return;
|
||||
|
||||
if (stadium_evict(cell) == 0) {
|
||||
word_slots[slot][word_id].cell_index = STADIUM_CELL_NONE;
|
||||
stat_evictions[slot]++;
|
||||
if (word_id < row_cap[slot]) { /* else never dispatched here -- can't be resident */
|
||||
cell = resolve_resident_cell(slot, word_id);
|
||||
if (cell != STADIUM_CELL_NONE && stadium_evict_in_slot(slot, cell) == 0) {
|
||||
word_slots[slot][word_id].cell_index = STADIUM_CELL_NONE;
|
||||
stat_evictions[slot]++;
|
||||
}
|
||||
}
|
||||
stadium_slot_unlock(slot);
|
||||
}
|
||||
|
||||
void stadium_words_stats(VMUuid vm_id, uint64_t *promotions, uint64_t *evictions) {
|
||||
int slot = stadium_quota_slot_for_vm(vm_id);
|
||||
int slot = stadium_slot_lock_vm(vm_id);
|
||||
|
||||
if (promotions) *promotions = (slot >= 0) ? stat_promotions[slot] : 0;
|
||||
if (evictions) *evictions = (slot >= 0) ? stat_evictions[slot] : 0;
|
||||
stadium_slot_unlock(slot);
|
||||
}
|
||||
|
||||
/* Freestanding: no libc printf. Prints an unsigned decimal, no leading
|
||||
|
||||
Reference in New Issue
Block a user