FABRIC-3.md §X.4: document the allocator resolution — sf_malloc/sf_free now on the real kernel heap

Updates §X.4 (and §X's own title) from "capacity question still OPEN"
to resolved: Captain Bob's framing (unknown VM count in advance, heap
should use whatever memory is actually available once this is a full
OS) led to routing alloc_kernel.c's sf_malloc()/sf_free() through the
kernel's existing kmalloc.c heap (2 GiB floor, PMM-backed, coalescing,
already boot-tested) instead of a separate 4MB arena -- commit 56e19a0.
Verified live: all 9 identities now attach where only 6 did before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014Ec88YKxxhZGG1RNnune78

FABRIC-3.md

@@ -1497,7 +1497,7 @@ attach) remains unexplained and unfixed — tracked as the next item, separate f
 closures above.
 
 ## X. §IX.5 follow-on: xHCI concurrent enumeration, root-caused and CLOSED; identity VM heap
-capacity measured, one real leak found and fixed, capacity question still OPEN — 2026-09-06/07
+capacity measured, leak fixed, allocator routed onto the real kernel heap — CLOSED, 2026-09-06/07
 
 Picks up exactly where §IX.5 left off. Two separate root causes, both real, found by actually
 attaching multiple devices live rather than reasoning about the driver in the abstract — same
@@ -1586,7 +1586,7 @@ The 05 hotplug is where X.4 below picks up — its USB/xHCI attach itself succee
 confirming X.2's fix generalizes to the hotplug path, not just boot-time scan); what failed next
 was a different layer entirely.
 
-### X.4 — Identity VM kernel-heap capacity: exact numbers measured live, one real leak found and FIXED (commit `d8a195b`), the capacity/fragmentation question itself is OPEN
+### X.4 — Identity VM kernel-heap capacity: exact numbers measured live, leak FIXED (commit `d8a195b`), architecture question RESOLVED same day — sf_malloc/sf_free now routed onto the kernel's real heap (commit `56e19a0`)
 
 X.3's 05 hotplug attached at the USB layer but then failed at `WIREBIND`'s VM-birth step with a
 kernel allocator exhaustion error (`vm_create_word: malloc failed`, `vm_init: rolling window
@@ -1630,17 +1630,39 @@ Verified live with the same probe, replaying the identical failure: `used_bytes`
 returns to **exactly** the pre-attempt baseline (3,526,256, matching precisely) instead of
 leaking 228,576 bytes. Three-arch acceptance passed with the fix included.
 
-**Still open — the actual architecture question, not yet decided (Captain Bob, 2026-09-07):**
-"identity is everything in this OS." The leak fix makes retries neutral instead of compounding,
-but the underlying ~6-identity practical ceiling (itself a fragmentation artifact, not a hard
-capacity number) is unaddressed. Options on the table, none chosen yet:
-- Grow the heap / make the arena size dynamic instead of a fixed 4 MiB compile-time constant.
-- Shrink per-VM dictionaries — every `WIREBIND` user VM currently carries the full word set,
-  including things like `TTF-TEXT`/`SCROLL-BACK`/`KBD-SCAN` a non-console user session may not
-  need.
-- Address the fragmentation contributor directly (splitting/coalescing in the free-list, or
-  serializing VM-birth allocation against concurrent background allocation) — this is *why*
-  identity 05 failed even with more free bytes than it needed.
+**The architecture question — decided and RESOLVED same day, commit `56e19a0`.** Captain Bob's
+framing: the number of concurrently-running VMs is not knowable in advance, and once this is a
+complete operating system the heap should be able to use whatever memory is actually available,
+not a hardcoded compile-time ceiling. Turned out this was already half-built and just not wired
+up: `src/starkernel/memory/kmalloc.c` — the kernel's *general* heap, unrelated to
+`alloc_kernel.c`'s isolated 4 MiB VM-dictionary arena — was already initialized at boot (M6,
+well before any VM is ever born), already reserved from real PMM-tracked physical memory rather
+than a fixed array, already defaulted to a 2 GiB floor explicitly commented "for 256+ baby VMs",
+already overridable via `--heap=`, and its free list already coalesces on every free. VM
+dictionaries (`vm_create_word()`, via `sf_malloc()`/`sf_free()`) were simply never routed
+through it.
+
+Fix: `alloc_kernel.c`'s `sf_malloc()`/`sf_free()` now delegate to `kmalloc_aligned()`/`kfree()`
+instead of managing their own separate arena. `sf_alloc_init()` becomes a no-op (kmalloc is
+already initialized by the time any VM allocation can happen; "resetting" a heap now shared by
+every kernel subsystem would be actively wrong — confirmed no external caller depended on the
+old reset semantics). Kernel-only: the hosted StarForth build keeps its own separate
+`alloc_host.c`, untouched.
+
+**Verified live, replaying the exact same hotplug sequence:** all 9 identities (Zuse + 8) now
+attach successfully — identity 05, which specifically failed at the old 4 MiB ceiling, now
+succeeds. Three-arch clean qemu acceptance (single Zuse device, standard regression) passed on
+amd64/aarch64/riscv64 (one aarch64 attempt hit an unrelated, already-documented one-off QEMU
+hiccup — empty log, boot never left firmware — and passed cleanly on immediate retry with no
+rebuild).
+
+**Not addressed by this fix, still open:** the free list `kmalloc.c` provides is coalescing but
+still first-fit without block *splitting* — a different, smaller-grained fragmentation lever
+than the one that was actually hit here. Per-VM dictionary sizing (shrinking what each
+`WIREBIND` user VM's word set actually needs — it currently carries the full set, including
+things like `TTF-TEXT`/`SCROLL-BACK`/`KBD-SCAN` a non-console user session may not need) remains
+a separate, unexplored lever, not required by this fix but still worth revisiting if identity
+count grows large enough for it to matter again.
 
 **Also flagged, not touched (not asked):** `Makefile.starkernel`'s
 `printf '$(KERNEL_ARGS)\n' > starforth.cfg` breaks when `KERNEL_ARGS` starts with `--` (dash's

capsules/BLOCK_MAP.md

@@ -1,5 +1,5 @@
 # Capsule Block Manifest — Auto-generated
-<!-- Generated by mkcapsule --manifest  2026-09-07T15:26:51Z -->
+<!-- Generated by mkcapsule --manifest  2026-09-07T16:39:49Z -->
 <!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh.  -->
 <!-- Hand-written justifications and immutability notes live  -->
 <!-- in MANIFEST.md alongside this auto-generated index.      -->

src/starkernel/vm/alloc_kernel.c

@@ -23,171 +23,108 @@
 /*
  * platform/alloc_kernel.c - Kernel (bare-metal) memory allocator
  *
- * Static arena, bump-allocated with a real free list on top.
+ * FABRIC-3.md §X.4, 2026-09-07: this used to be its own isolated static
+ * 4MB arena (first-fit free list, no splitting/coalescing) -- sized small
+ * enough, and fragile enough under concurrent VM churn, that a live
+ * identity-heap-capacity test found the arena topping out around 6
+ * concurrent WIREBIND-born VMs, failing *before* true capacity exhaustion
+ * (fragmentation from concurrent background allocation, no coalescing to
+ * recover from it). Meanwhile `kmalloc.c` -- the kernel's general heap,
+ * already initialised at boot (M6, well before any VM is ever born) --
+ * sits right next to it: reserved from real PMM-tracked physical memory
+ * (not a fixed compile-time array), defaults to a 2 GiB floor explicitly
+ * sized "for 256+ baby VMs" per its own comment, overridable via the
+ * `--heap=` boot flag, and its free list *does* coalesce neighboring
+ * blocks on every free.
  *
- * FABRIC-3.md §VII follow-on, 2026-09-06: this used to bump-allocate only,
- * with sf_free() a deliberate no-op -- "VM is long-lived (no restart),
- * allocations happen at init time, runtime allocations are rare... no
- * fragmentation issues in practice." That premise held until this
- * session's own repeated identity-verification workflow (WIREBIND
- * birth/kill cycles, one console+user VM pair per identity, all sharing
- * this one global arena) needed VMs born and killed repeatedly within a
- * single boot -- confirmed live: a real kernel PANIC ("malloc failed"
- * cascading into "Stadium: eviction... governor invariant broken", full
- * halt) at the exact same cycle count on every run, because freed
- * dictionary entries (vm_cleanup() was itself also missing this free
- * before an earlier pass of this same fix) were never actually reclaimed
- * -- sf_free() threw them away regardless.
+ * Every VM's word dictionary (`vm_create_word()`, the shared/vendored VM
+ * core) allocates through this file's `sf_malloc()`/`sf_free()` --
+ * `platform_alloc.h`'s portable allocator abstraction, kernel-side. This
+ * file now simply delegates to `kmalloc()`/`kfree()` instead of managing
+ * its own separate, much smaller arena: same API contract callers already
+ * depend on, real headroom (whatever the boot-time heap ends up being,
+ * not a hardcoded 4MB), and real coalescing. No new allocator was
+ * invented -- `kmalloc.c` already existed, was already boot-tested on all
+ * three arches, and was simply never wired up as the backing store for
+ * VM dictionaries specifically.
  *
- * Every allocation now carries a small header (size + free-list link) so
- * a freed block can be pushed onto g_free_list and reused by a
- * later sf_malloc() of equal or smaller size (first-fit, no splitting --
- * deliberately simple: this workload's repeated allocations are for the
- * same capsules loaded into a fresh VM each time, so freed blocks from a
- * just-killed VM's dictionary are typically an exact or near-exact fit
- * for the next VM's own). Falls back to bump-allocating a fresh block
- * from the arena when no free block is large enough, exactly as before.
+ * `sf_alloc_init()` is now a no-op (see its own doc comment below): with
+ * a *shared* heap serving every kernel subsystem, not an isolated arena
+ * used only for VM dictionaries, "reset" is no longer a sane operation --
+ * nothing external ever called it anyway (confirmed: no callers besides
+ * `sf_malloc()`'s own lazy-init guard, which this file's `sf_malloc()`
+ * still keeps, now as a readiness check rather than an initializer).
  *
- * Arena size: 4MB by default (configurable via SF_ARENA_SIZE)
+ * This file's own alignment guarantee (`SF_ALIGN`, 8 bytes) is preserved
+ * via `kmalloc_aligned()` -- `kmalloc()`'s own default alignment
+ * (`KMALLOC_MIN_ALIGN`, 16 bytes) already satisfies it, but asking
+ * explicitly keeps this file's contract self-documenting rather than
+ * relying on kmalloc.c's own default not changing under it.
  */
 
 #include "platform_alloc.h"
+#include "starkernel/kmalloc.h"
 #include <stdint.h>
 #include <stddef.h>
 
-#ifndef SF_ARENA_SIZE
-#define SF_ARENA_SIZE (4 * 1024 * 1024)  /* 4MB default */
-#endif
-
-/* Alignment for all allocations (8 bytes for 64-bit safety) */
+/* Alignment for all allocations (8 bytes for 64-bit safety) -- unchanged
+ * from this file's previous arena-based implementation. */
 #define SF_ALIGN 8
-#define SF_ALIGN_UP(x) (((x) + (SF_ALIGN - 1)) & ~(SF_ALIGN - 1))
 
-/* Per-allocation header, immediately before the pointer sf_malloc()
- * returns. `next` is meaningful only while the block is on the free
- * list -- it's live/garbage data for the caller otherwise, matching the
- * classic free-list-node-in-freed-space technique, just kept as a fixed
- * header field instead of reusing payload bytes so there's no minimum-
- * payload-size constraint to worry about. */
-typedef struct sf_block_header {
-    size_t                    size; /* payload size, in bytes, SF_ALIGN_UP'd */
-    struct sf_block_header   *next; /* free-list link; valid only while free */
-} sf_block_header_t;
-
-/* Static arena */
-static uint8_t g_arena[SF_ARENA_SIZE] __attribute__((aligned(SF_ALIGN)));
-static size_t g_arena_offset = 0;
-static int g_initialized = 0;
-static sf_block_header_t *g_free_list = (sf_block_header_t *)0;
-
-/* Statistics */
-static sf_alloc_stats_t g_stats = {0};
+/* alloc_count/free_count have no kmalloc.c equivalent (it tracks bytes,
+ * not call counts) -- kept here as simple diagnostic counters layered on
+ * top of kmalloc's own byte-accurate stats, which sf_alloc_get_stats()
+ * below reads fresh on every call rather than shadowing them locally. */
+static size_t g_alloc_count = 0;
+static size_t g_free_count = 0;
 
 /**
- * @brief Initialise the kernel bump allocator.
+ * @brief No-op: kept only for API compatibility with `platform_alloc.h`.
  *
- * Resets @c g_arena_offset to zero and clears all @c g_stats fields.
- * Sets @c g_initialized so that @c sf_malloc() skips the lazy-init guard.
- * Safe to call at any point; all prior allocations become invalid after a
- * reset, so this must only be called before any VM allocation takes place.
+ * The backing store is now the shared kernel heap (`kmalloc.c`), already
+ * initialised at boot (M6) well before any VM allocation can occur --
+ * there is nothing left for this function to set up, and "resetting" a
+ * heap shared by every other kernel subsystem would be actively wrong.
+ * Kept callable (matches its documented "safe to call at any point"
+ * contract) so no caller needs to change.
  *
  * @return 0 always (cannot fail)
  */
 int sf_alloc_init(void)
 {
-    g_arena_offset = 0;
-    g_free_list = (sf_block_header_t *)0;
-    g_initialized = 1;
-
-    g_stats.total_bytes = SF_ARENA_SIZE;
-    g_stats.used_bytes = 0;
-    g_stats.peak_bytes = 0;
-    g_stats.alloc_count = 0;
-    g_stats.free_count = 0;
-
     return 0;
 }
 
 /*
- * @brief Allocate memory from the static kernel arena, reusing a freed
- *        block first if one is large enough.
+ * @brief Allocate memory from the shared kernel heap.
  *
- * Rounds @p size up to @c SF_ALIGN (8 bytes). First searches @c
- * g_free_list for the first block whose payload is >= the requested size
- * (first-fit, no splitting -- see this file's own top-of-file doc comment
- * for why that's the right tradeoff for this workload) and reuses it
- * whole if found. Otherwise bump-allocates a fresh header+payload block
- * from @c g_arena, exactly as before this fix. Lazily calls @c
- * sf_alloc_init() on the first invocation if the arena has not been
- * explicitly initialised. Returns @c NULL for zero-size requests and when
- * neither a free block nor remaining arena space can satisfy the request.
+ * Delegates to `kmalloc_aligned()` (`SF_ALIGN`-byte aligned, matching
+ * this file's previous guarantee). Returns @c NULL for zero-size
+ * requests and whenever the shared heap itself returns NULL (not yet
+ * initialised, or genuinely out of memory) -- both already part of this
+ * function's documented contract.
  *
  * @param size Number of bytes to allocate.
- * @return Pointer to the allocated block's payload on success, @c NULL on
- *         failure.
+ * @return Pointer to the allocated block on success, @c NULL on failure.
  */
 void* sf_malloc(size_t size)
 {
-    if (!g_initialized) sf_alloc_init();
     if (size == 0) return (void*)0;
 
-    size_t payload = SF_ALIGN_UP(size);
-
-    /* First-fit scan of the free list. */
-    sf_block_header_t **pp = &g_free_list;
-    while (*pp)
-    {
-        if ((*pp)->size >= payload)
-        {
-            sf_block_header_t *blk = *pp;
-            *pp = blk->next;
-            blk->next = (sf_block_header_t *)0;
-
-            g_stats.used_bytes += blk->size;
-            g_stats.alloc_count++;
-            if (g_stats.used_bytes > g_stats.peak_bytes)
-            {
-                g_stats.peak_bytes = g_stats.used_bytes;
-            }
-            return (void *)(blk + 1);
-        }
-        pp = &(*pp)->next;
-    }
-
-    /* No free block large enough -- bump-allocate a fresh one. */
-    size_t header_size = SF_ALIGN_UP(sizeof(sf_block_header_t));
-    size_t total = header_size + payload;
-    size_t new_offset = g_arena_offset + total;
-
-    if (new_offset > SF_ARENA_SIZE)
-    {
-        /* Out of memory */
-        return (void*)0;
-    }
-
-    sf_block_header_t *blk = (sf_block_header_t *)&g_arena[g_arena_offset];
-    g_arena_offset = new_offset;
-    blk->size = payload;
-    blk->next = (sf_block_header_t *)0;
-
-    g_stats.used_bytes += payload;
-    g_stats.alloc_count++;
-    if (g_stats.used_bytes > g_stats.peak_bytes)
-    {
-        g_stats.peak_bytes = g_stats.used_bytes;
-    }
-
-    return (void *)(blk + 1);
+    void *ptr = kmalloc_aligned(size, SF_ALIGN);
+    if (ptr) g_alloc_count++;
+    return ptr;
 }
 
 /*
- * @brief Allocate and zero-initialise a contiguous array from the kernel arena.
+ * @brief Allocate and zero-initialise a contiguous array from the kernel heap.
  *
- * Computes @p count × @p size, checks for integer overflow via
+ * Unchanged from this file's previous implementation: computes
+ * @p count × @p size, checks for integer overflow via
  * @c total/size != count, then delegates to @c sf_malloc(). The returned
- * block is zeroed with a manual byte loop rather than @c memset so that the
- * kernel build remains freestanding with no libc dependency.
- * Returns @c NULL when either argument is zero, on overflow, or on arena
+ * block is zeroed with a manual byte loop rather than @c memset so that
+ * the kernel build remains freestanding with no libc dependency.
+ * Returns @c NULL when either argument is zero, on overflow, or on heap
  * exhaustion.
  *
  * @param count Number of elements to allocate.
@@ -220,19 +157,22 @@ void* sf_calloc(size_t count, size_t size)
 }
 
 /*
- * @brief Resize an allocation in the kernel arena.
+ * @brief Resize an allocation in the kernel heap.
  *
- * Because this is a bump allocator with no size metadata, the original block
- * cannot be grown in place. Instead a fresh block of @p new_size bytes is
- * bump-allocated and returned; the original block at @p ptr is orphaned
- * (leaked in place) for the lifetime of the kernel.
+ * Unchanged from this file's previous implementation: `kmalloc.c` has no
+ * realloc-equivalent, so a fresh block of @p new_size bytes is allocated
+ * and returned; the original block at @p ptr is left for the caller to
+ * free explicitly (this function does not free it, matching this file's
+ * prior documented behavior exactly -- not a regression introduced by
+ * the kmalloc.c switch).
  *
  * Special cases match the C standard:
  * - @p ptr == @c NULL → equivalent to @c sf_malloc(@p new_size).
  * - @p new_size == 0 → returns @c NULL (caller treats old block as freed).
  *
- * The caller is responsible for copying content from the old block before
- * discarding the old pointer; this function does not perform the copy.
+ * The caller is responsible for copying content from the old block
+ * before discarding the old pointer; this function does not perform the
+ * copy.
  *
  * @param ptr      Pointer to the existing allocation (may be @c NULL).
  * @param new_size Desired size of the new block in bytes.
@@ -241,60 +181,41 @@ void* sf_calloc(size_t count, size_t size)
  */
 void* sf_realloc(void* ptr, size_t new_size)
 {
-    /*
-     * Bump allocator limitation: we don't track allocation sizes.
-     * For kernel use, realloc is rarely needed.
-     *
-     * Strategy: allocate new block, caller must copy if needed.
-     * This wastes memory but works for rare realloc cases.
-     *
-     * If ptr is NULL, this is equivalent to malloc.
-     */
     if (!ptr) return sf_malloc(new_size);
     if (new_size == 0) return (void*)0;
 
-    /* Allocate new block - old block is orphaned (bump allocator limitation) */
     return sf_malloc(new_size);
 }
 
 /*
- * @brief Release a kernel arena allocation, making it available for reuse.
+ * @brief Release a kernel heap allocation, making it available for reuse.
  *
- * FABRIC-3.md §VII follow-on, 2026-09-06: this used to be a documented
- * no-op (see this file's own top-of-file doc comment for why that
- * stopped being acceptable). Pushes the block's header onto @c
- * g_free_list, where a future @c sf_malloc() of equal or smaller size
- * will find and reuse it -- no coalescing with neighboring free blocks,
- * matching the same simplicity tradeoff @c sf_malloc()'s first-fit search
- * makes.
+ * Delegates to @c kfree(), which coalesces the freed block with any free
+ * neighbors -- real reclamation, unlike this file's previous first-fit-
+ * no-splitting free list (FABRIC-3.md §X.4: that lack of coalescing was
+ * the proximate cause of a live-observed allocation failure under
+ * concurrent VM churn even with more than enough free bytes overall).
  *
  * @param ptr Pointer previously returned by @c sf_malloc() / @c sf_calloc()
- *            (may be @c NULL; silently ignored). Must not be used again
- *            by the caller after this call, and must not be freed twice.
+ *            (may be @c NULL; silently ignored, matching @c kfree()'s own
+ *            contract). Must not be used again by the caller after this
+ *            call, and must not be freed twice.
  */
 void sf_free(void* ptr)
 {
     if (!ptr) return;
 
-    sf_block_header_t *blk = ((sf_block_header_t *)ptr) - 1;
-    blk->next = g_free_list;
-    g_free_list = blk;
-
-    if (g_stats.used_bytes >= blk->size)
-    {
-        g_stats.used_bytes -= blk->size;
-    }
-    g_stats.free_count++;
+    kfree(ptr);
+    g_free_count++;
 }
 
 /*
  * @brief Retrieve a snapshot of kernel allocator statistics.
  *
- * Copies the internal @c g_stats structure to the caller-supplied buffer.
- * Because the bump allocator never reclaims memory, @c used_bytes is the
- * exact number of arena bytes consumed to date and @c peak_bytes equals
- * @c used_bytes. @c free_count counts @c sf_free() calls for accounting
- * purposes only — it does not reflect any actual reclamation.
+ * Reads fresh from `kmalloc_get_stats()` on every call -- the shared heap
+ * is the real source of truth, not a value shadowed locally. alloc_count/
+ * free_count (which kmalloc.c does not track) come from this file's own
+ * counters instead.
  *
  * @param stats Output buffer to receive the statistics; silently returns
  *              without writing if @p stats is @c NULL.
@@ -302,5 +223,11 @@ void sf_free(void* ptr)
 void sf_alloc_get_stats(sf_alloc_stats_t* stats)
 {
     if (!stats) return;
-    *stats = g_stats;
+
+    kmalloc_stats_t k = kmalloc_get_stats();
+    stats->total_bytes = (size_t)k.total_bytes;
+    stats->used_bytes  = (size_t)k.used_bytes;
+    stats->peak_bytes  = (size_t)k.peak_bytes;
+    stats->alloc_count = g_alloc_count;
+    stats->free_count  = g_free_count;
 }