Stage S-1b: dictionary word ids grow with the dictionary; a word that can't get an id is refused whole (FABRIC-3.9.0 §8.1)

- word_id_map / recycled_word_ids are heap tables (one shared capacity, 64 doubling), grown all-or-nothing by allocate-copy-free. Numbering is unchanged, so boot parity is unchanged. - vm_create_word() takes the id before linking; on failure it frees the entry, sets vm->error, logs LOG_ERROR and counts (vm_dictionary_word_id_refusals()). No word runs with WORD_ID_INVALID. - Readers of the id space bound by vm->next_word_id, not DICTIONARY_SIZE. - transition_heat gets its own capacity and grows on record (was sized once -- a heap overflow under a dynamic bound); every reader bounds by it. - Stadium id->cell rows start empty and grow on dispatch; a failed grow is logged, counted and not admitted. S-2 (the cell pool) stays open. Probe (reverted), identical on amd64/aarch64/riscv64: +2000 words get real ids (capacity 4096); a word at id 2738 lands 200x in the rolling window and is Stadium-admitted; FORGET recycles across the old boundary; an injected grow failure refuses the word whole (NULL, error, counter, dictionary unchanged) and the retry succeeds. 3-ISA acceptance passes; boot parity (531, 0x572daac83b797332) and the dict_hash set unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 13:59:43 -04:00
parent ee3db96ea8
commit 1dbd0c74ed
26 changed files with 55442 additions and 83 deletions
@@ -223,3 +223,100 @@ Evidence is named **before** each stage runs.
        `1 2 + .` gives `3` mid-stream.
      - Mint → WIREBIND → `USE` → `1 2 + .` → `3`. Idle log growth 0. Zero `UNKNOWN WORD` or
        faults. QEMU exits cleanly.
+- [x] **Stage S-1b — dictionary word ids grow with the dictionary (`FABRIC-3.9.0.md` §8.1, class C).**
+      2026-09-24. Captain Bob: *"go ahead with S-1b."*
+
+      **The defect.** Stable word ids came from fixed `word_id_map[1024]` /
+      `recycled_word_ids[1024]` tables in the `VM` struct. Word 1025 onward got `WORD_ID_INVALID`
+      and a `LOG_WARN` ("out of stable IDs, disabling metrics"), but was still defined. It then ran
+      outside the physics:
+      - the rolling window and the pipelining hooks skipped it;
+      - Stadium never admitted it;
+      - parity hashed it as `0xFFFFFFFF`.
+
+      Nothing counted it and nothing refused it.
+
+      **The change:**
+      - **`vm.h` / `dictionary_management.c`: the core.**
+        - `word_id_map` and `recycled_word_ids` are heap tables sharing one `word_id_capacity`.
+        - They start at 64 and double when a fresh id reaches capacity.
+        - Growth is all-or-nothing: both new tables are allocated before either old one is touched.
+          It uses allocate-copy-free, because the kernel's `sf_realloc()` does not copy.
+        - Numbering is unchanged (a recycled id first, else `next_word_id++`). That is why boot
+          parity is unchanged.
+        - Both `vm_cleanup()`s free the tables.
+      - **Refusal is whole.**
+        - `vm_dictionary_track_entry()` now returns a status.
+        - `vm_create_word()` takes the id *before* linking the entry. On failure it frees the entry
+          and its transition metrics, sets `vm->error`, and returns NULL.
+        - The failure is logged at `LOG_ERROR` ("word id table can't grow: '…' refused") and counted
+          (`vm_dictionary_word_id_refusals()`).
+        - Every caller already handled NULL from the existing malloc-failure and pinned-shadow paths.
+      - **The bound for every reader of the id space is `vm->next_word_id`,** not `DICTIONARY_SIZE`.
+        Every issued id, live or recycled, is below it:
+        - `lookup_by_word_id`, `untrack`;
+        - both interpreters' physics hooks (`physics_execution_hooks.c`, `vm_core.c`);
+        - the rolling-window seed (its frequency array is now sized to the bound);
+        - `inference_engine.c`;
+        - the pipelining diagnostics.
+      - **`transition_heat` (`physics_pipelining_metrics.c`).** Each word's array was sized once, at
+        whatever size its first record passed in. With a dynamic bound, that would have been a heap
+        overflow on the first transition to a newer id.
+        - It now has its own `transition_heat_cap` and grows on record (to the bound, at least
+          doubling).
+        - Every reader bounds by that field. `get_probability` had no bounds check at all.
+        - A failed grow drops that one transition (the word already ran; nothing is left to refuse).
+          It is logged and counted (`transition_metrics_grow_failures()`).
+        - Pipelining is compiled out of the kernel (`ENABLE_PIPELINING=0`), so this path is live
+          only in hosted builds.
+      - **Stadium's id→cell rows (`stadium_words.c`).** These were `[slot][DICTIONARY_SIZE]`.
+        - Each row now starts empty and grows on dispatch (`row_cap[slot]`), not at definition,
+          because a quota slot can be granted after a VM's words exist, and slots are reused across
+          VMs.
+        - A failed grow is logged and counted (`stadium_words_row_grow_failures()`). The word is not
+          admitted that dispatch, which is the same, already-conserving outcome as a full Stadium.
+        - This is the id→cell map, not the cell pool, so **S-2 stays open**.
+      - `DICTIONARY_SIZE` stays defined for its one remaining user, reported below.
+
+      **Evidence, named before the run.** A probe (reverted) at the end of boot, byte-identical on
+      all three ISAs (38 lines; `logs/20260924-130702/amd64/`, `-131140/aarch64/`,
+      `-131624/riscv64/`):
+
+      | Check | Result |
+      |---|---|
+      | Start | `next_word_id=738`, capacity 1024 (grown 64 → 1024 during boot) |
+      | +2000 words | `next_word_id=2738`, capacity 4096, `PZ1999 word_id=2737`, refusals 0, and no "out of stable IDs" line. Parity OK, `word_count=2563`, `latest_id=2737` (was `4294967295` at S-1a), `hash=0xc873c20962d728ed`. The hash differs from S-1a's `0xcf85…` as expected: 1,539 words now hash real ids instead of `0xFFFFFFFF`. |
+      | Physics past id 1024 | `: PZHOT 1 DROP ; : PZRUN 200 0 DO PZHOT LOOP ; PZRUN` gives PZHOT id 2738 and PZRUN id 2739; `lookup(2738)==PZHOT`; **200** PZHOT entries in the rolling window (the hooks used to skip any id ≥ 1024); Stadium promotions +2 (both words admitted); row-grow failures 0. |
+      | FORGET/recycle | `FORGET PZHOT` leaves 2 recycled ids and `lookup(2738)==NULL`. `: PZNEW ;` reuses id 2738, and `lookup` returns PZNEW. |
+      | Forced grow failure | After filling the tables to capacity (4096), one allocation failure was injected into the grow. `PZFAIL` then gives NULL, `vm->error=1`, refusals 1, and `[HADES][ERROR] word id table can't grow: 'PZFAIL' refused`. `latest`, `word_count`, capacity and `next_word_id` are all unchanged, and `PZFAIL` is not findable. |
+      | Retry | With the injection cleared, `PZOK` is defined: id 4096, capacity 8192. |
+      | Final parity | `word_count=3922`, `hash=0x5af2d7e820d21624`, `incomplete_count=0` |
+
+      **Acceptance** on the clean tree, with the probe reverted (`logs/20260924-132055/amd64/`,
+      `-133339/aarch64/`, `-134638/riscv64/`):
+      - Boot M7.1a is unchanged (`word_count=531`, `latest_id=700`, `0x572daac83b797332`), and so is
+        the `dict_hash` set (Hera `0xda9742ce9c3376a2`, Hestia `0x413e7d086f342d8b`, Artemis
+        `0xdfd68316d5f5ba85`).
+      - The O-A checks pass: G interleave A = 40/B = 40/48 runs, `HHH` precedence, and a typed
+        `1 2 + .` gives `3` mid-stream.
+      - Mint → WIREBIND → `USE` → `3`. Idle log growth 0. Zero `UNKNOWN WORD` or faults. QEMU exits
+        cleanly.
+      - Clean kernel builds on all three ISAs, plus the hosted build.
+
+      **Found, reported, not fixed:**
+      1. `dictionary_heat_optimization.c:123`: `cell_t heats[DICTIONARY_SIZE]` snapshots at most
+         1024 words' heat and silently ignores the rest. That is class C, keyed on the dictionary
+         count, not on ids.
+      2. `vm_create_word()`: a failed transition-metrics malloc logs a WARN and defines the word
+         anyway, with no pipelining metrics. That is a partial.
+      3. Neither `vm_cleanup()` frees `transition_heat` (or `context_window`) inside each word's
+         metrics, only the metrics struct itself. That is a leak on every VM kill with pipelining on.
+      4. `transition_metrics_update_cache()` scans the whole id space on every recorded transition.
+         The hot-path cost grows with the dictionary, now that the dictionary can grow.
+      5. The kernel's FC-bucket grow (`sf_fc_reserve`) copies after `sf_realloc()`, which is correct,
+         but never frees the old block. Each bucket grow leaks it.
+      6. §8.1 asks that each failure counter be readable by a FORTH word or DoE row. S-1a and S-1b
+         expose C accessors only (`sk_parity_incomplete_count()`,
+         `vm_dictionary_word_id_refusals()`, `transition_metrics_grow_failures()`,
+         `stadium_words_row_grow_failures()`). Adding a word changes the `dict_hash` set, so this
+         awaits Captain Bob's ruling.
@@ -154,11 +154,15 @@ typedef struct WordTransitionMetrics {
     * transition_heat[i]: Number of times word with index i has been
     *                     executed immediately after this word.
     *
-     * Dynamically allocated array of size DICTIONARY_SIZE.
+     * Dynamically allocated, transition_heat_cap entries long.
     * Initialized to NULL; allocated on first use to save memory
-     * for words that are never executed.
+     * for words that are never executed, and grown when a word id at or
+     * past transition_heat_cap is recorded (Stage S-1b: word ids are no
+     * longer capped at DICTIONARY_SIZE, so no fixed length can hold them).
+     * Every reader bounds by transition_heat_cap, never by a global size.
     */
    uint64_t *transition_heat;
+    uint32_t transition_heat_cap;

    /** Total number of transitions observed from this word */
    uint64_t total_transitions;
@@ -264,11 +268,14 @@ int transition_metrics_init(WordTransitionMetrics *metrics);
 *
 * @param metrics Metrics for the current word
 * @param next_word_id Index of word that executed next
- * @param dict_size Total number of words in dictionary (for array bounds)
- * @return 0 on success, -1 on invalid word_id
+ * @param dict_size The VM's word-id bound (vm->next_word_id): every issued id is below it
+ * @return 0 on success, -1 on invalid word_id or a failed grow (logged, counted)
 */
 int transition_metrics_record(WordTransitionMetrics *metrics, uint32_t next_word_id, uint32_t dict_size);

+/* S-1b: transitions dropped because a transition_heat array couldn't grow. */
+uint64_t transition_metrics_grow_failures(void);
+
 /*
 * Get the probability (Q48.16 format) of a specific word following this word.
 *
@@ -48,7 +48,8 @@

 /*
 * stadium_words_init - Allocates and zeroes the word_id -> cell_index map
- * (DICTIONARY_SIZE entries per VM quota slot, stadium_max_vm_count() slots,
+ * (one row per VM quota slot, stadium_max_vm_count() slots; since S-1b each
+ * row starts empty and grows on dispatch, no longer DICTIONARY_SIZE entries;
 * kmalloc'd -- was a flat static array before 2026-08-15, when the VM count
 * bound became RAM-derived rather than a compile-time constant). Must be
 * called after stadium_boot_init() and stadium_birth_hera(), before any word
@@ -97,8 +98,10 @@ void stadium_words_init(void);
 * the failed attempt). On success, records the mapping and increments the
 * promotion counter.
 *
- * No-op if word_id == WORD_ID_INVALID, word_id >= DICTIONARY_SIZE, or vm_id
- * holds no Stadium quota.
+ * No-op if word_id == WORD_ID_INVALID or vm_id holds no Stadium quota. A
+ * word_id past the slot's row grows the row first (S-1b); if that fails the
+ * dispatch is not admitted, logged and counted
+ * (stadium_words_row_grow_failures()).
 *
 * @param vm_id           Owning VM -- vm->stadium_vm_id at every call site.
 *                        Scopes the word_id -> cell_index lookup to this
@@ -140,7 +143,7 @@ void stadium_word_forget(VMUuid vm_id, uint32_t word_id);
 * Bob's ruling 2026-08-06) can add this as an explicit term instead of
 * silently omitting word-execution heat it has no other way to see.
 *
- * Walks all DICTIONARY_SIZE word_slots for vm_id's quota slot; each
+ * Walks all of the word_slots row for vm_id's quota slot; each
 * resident entry contributes its cell's current heat, verified live against
 * the discriminator bitmap (same self-healing pattern as
 * resolve_resident_cell() -- a stale map entry contributes 0, not garbage).
@@ -161,6 +164,9 @@ uint64_t stadium_words_resident_heat(VMUuid vm_id);
 */
 void stadium_words_stats(VMUuid vm_id, uint64_t *promotions, uint64_t *evictions);

+/* S-1b: dispatches not admitted because a slot's word row couldn't grow. */
+uint64_t stadium_words_row_grow_failures(void);
+
 /*
 * stadium_words_print_boot_diagnostics - Console output satisfying item
 * 4.1's "observable via a diagnostic word or boot console output"
@@ -510,8 +510,15 @@ typedef struct VM
    uint8_t* memory; /**< Unified VM memory buffer */
    uintptr_t here; /**< Next free memory location (byte offset or absolute pointer if M7) */
    DictEntry* latest; /**< Most recent word */
-    DictEntry* word_id_map[DICTIONARY_SIZE]; /**< Stable ID → entry map for speculation */
-    uint32_t recycled_word_ids[DICTIONARY_SIZE]; /**< Reusable ID stack for FORGET */
+    /* FABRIC-3.9.0.md §8.1 (Stage S-1b): the stable-id tables grow with the
+     * dictionary (were fixed [DICTIONARY_SIZE] arrays -- word 1025 onward got
+     * WORD_ID_INVALID and ran outside the physics). Both share one capacity:
+     * the recycle stack can never hold more ids than were ever issued. Every
+     * issued id, live or recycled, is < next_word_id -- that is the bound for
+     * any reader of the id space. */
+    DictEntry** word_id_map; /**< Stable ID → entry map for speculation (word_id_capacity entries) */
+    uint32_t* recycled_word_ids; /**< Reusable ID stack for FORGET (word_id_capacity entries) */
+    uint32_t word_id_capacity; /**< Allocated length of both tables */
    uint32_t recycled_word_id_count; /**< Depth of recycled stack */
    uint32_t next_word_id; /**< Next fresh ID when recycle stack empty */

@@ -767,7 +774,12 @@ DictEntry* vm_dictionary_lookup_by_word_id(VM* vm, uint32_t word_id);
 * popped off the data stack (EXECUTE, ENTROPY@/!, etc). */
 int vm_dict_entry_ok(VM* vm, DictEntry* candidate);

-void vm_dictionary_track_entry(VM* vm, DictEntry* entry);
+/* Assign `entry` a stable word id, growing the id tables if needed (S-1b).
+ * Returns 0, or -1 if the tables can't grow: logged at LOG_ERROR, counted
+ * in vm_dictionary_word_id_refusals(), and the caller must refuse the word. */
+int vm_dictionary_track_entry(VM* vm, DictEntry* entry);
+
+uint32_t vm_dictionary_word_id_refusals(void);

 void vm_dictionary_untrack_entry(VM* vm, DictEntry* entry);

@@ -75,6 +75,61 @@ size_t sf_fc_cap[SF_FC_BUCKETS];

 static DictEntry *sf_cached_latest = NULL; /* last head we indexed */

+/* FABRIC-3.9.0.md §8.1 (Stage S-1b): stable word ids are no longer capped at
+ * DICTIONARY_SIZE. The id tables grow with the dictionary; a word that can't
+ * get an id is refused whole (vm_create_word() returns NULL, vm->error set),
+ * logged at LOG_ERROR and counted -- it is never defined with WORD_ID_INVALID
+ * to run silently outside the physics. */
+#define WORD_ID_TABLE_MIN 64u
+
+static uint32_t g_word_id_refusals;
+
+uint32_t vm_dictionary_word_id_refusals(void) { return g_word_id_refusals; }
+
+/* Grow both id tables to at least `need` entries, all-or-nothing: both new
+ * tables are allocated before either old one is touched, so a failure leaves
+ * the VM exactly as it was. Allocate-copy-free, not sf_realloc(): the
+ * kernel's sf_realloc() does not copy (alloc_kernel.c). */
+static int vm_dictionary_grow_word_ids(VM *vm, uint32_t need) {
+    uint32_t   new_cap = vm->word_id_capacity ? vm->word_id_capacity : WORD_ID_TABLE_MIN;
+    DictEntry **new_map;
+    uint32_t  *new_recycled;
+    uint32_t   i;
+
+    while (new_cap < need) {
+        /* Ids run 0..WORD_ID_INVALID-1, so a table never needs more than
+         * WORD_ID_INVALID entries. */
+        new_cap = (new_cap > WORD_ID_INVALID / 2u) ? WORD_ID_INVALID : new_cap * 2u;
+    }
+
+    new_map      = (DictEntry **)sf_malloc((size_t)new_cap * sizeof(DictEntry *));
+    new_recycled = (uint32_t *)sf_malloc((size_t)new_cap * sizeof(uint32_t));
+    if (!new_map || !new_recycled) {
+        sf_free(new_map);
+        sf_free(new_recycled);
+        return -1;
+    }
+
+    for (i = 0; i < vm->word_id_capacity; i++) {
+        new_map[i] = vm->word_id_map[i];
+    }
+    for (; i < new_cap; i++) {
+        new_map[i] = NULL;
+    }
+    for (i = 0; i < vm->recycled_word_id_count; i++) {
+        new_recycled[i] = vm->recycled_word_ids[i];
+    }
+
+    sf_free(vm->word_id_map);
+    sf_free(vm->recycled_word_ids);
+    vm->word_id_map       = new_map;
+    vm->recycled_word_ids = new_recycled;
+    vm->word_id_capacity  = new_cap;
+    return 0;
+}
+
+/* Numbering is unchanged from the fixed-table version: a recycled id first,
+ * else the next fresh one. Only the fresh path can need a grow. */
 static uint32_t vm_dictionary_acquire_word_id(VM *vm) {
    if (!vm) {
        return WORD_ID_INVALID;
@@ -84,29 +139,35 @@ static uint32_t vm_dictionary_acquire_word_id(VM *vm) {
        return vm->recycled_word_ids[--vm->recycled_word_id_count];
    }

-    if (vm->next_word_id < DICTIONARY_SIZE) {
-        return vm->next_word_id++;
+    if (vm->next_word_id == WORD_ID_INVALID) {
+        return WORD_ID_INVALID; /* every id is issued -- a real limit, refused the same way */
    }

-    return WORD_ID_INVALID;
+    if (vm->next_word_id >= vm->word_id_capacity &&
+        vm_dictionary_grow_word_ids(vm, vm->next_word_id + 1u) != 0) {
+        return WORD_ID_INVALID;
+    }
+
+    return vm->next_word_id++;
 }

-void vm_dictionary_track_entry(VM *vm, DictEntry *entry) {
+int vm_dictionary_track_entry(VM *vm, DictEntry *entry) {
    if (!vm || !entry) {
-        return;
+        return -1;
    }

    uint32_t word_id = vm_dictionary_acquire_word_id(vm);
    entry->word_id = word_id;

-    if (word_id == WORD_ID_INVALID || word_id >= DICTIONARY_SIZE) {
-        log_message(LOG_WARN,
-                    "vm_dictionary_track_entry: out of stable IDs, disabling metrics for '%.*s'",
+    if (word_id == WORD_ID_INVALID) {
+        g_word_id_refusals++;
+        log_message(LOG_ERROR, "word id table can't grow: '%.*s' refused",
                    (int)entry->name_len, entry->name);
-        return;
+        return -1;
    }

    vm->word_id_map[word_id] = entry;
+    return 0;
 }

 void vm_dictionary_untrack_entry(VM *vm, DictEntry *entry) {
@@ -125,7 +186,7 @@ void vm_dictionary_untrack_entry(VM *vm, DictEntry *entry) {
 #endif

    uint32_t word_id = entry->word_id;
-    if (word_id == WORD_ID_INVALID || word_id >= DICTIONARY_SIZE) {
+    if (word_id == WORD_ID_INVALID || word_id >= vm->next_word_id) {
        return;
    }

@@ -142,7 +203,9 @@ void vm_dictionary_untrack_entry(VM *vm, DictEntry *entry) {
        vm->word_id_map[word_id] = NULL;
    }

-    if (vm->recycled_word_id_count < DICTIONARY_SIZE) {
+    /* Always fits: at most next_word_id ids were ever issued, and the tables
+     * hold word_id_capacity >= next_word_id entries. */
+    if (vm->recycled_word_id_count < vm->word_id_capacity) {
        vm->recycled_word_ids[vm->recycled_word_id_count++] = word_id;
    }

@@ -150,7 +213,7 @@ void vm_dictionary_untrack_entry(VM *vm, DictEntry *entry) {
 }

 DictEntry *vm_dictionary_lookup_by_word_id(VM *vm, uint32_t word_id) {
-    if (!vm || word_id >= DICTIONARY_SIZE) {
+    if (!vm || word_id >= vm->next_word_id) {
        return NULL;
    }

@@ -467,8 +530,18 @@ DictEntry *vm_create_word(VM *vm, const char *name, size_t len, word_func_t func
    }
    *(cell_t *) (((uint8_t *) entry) + df_off) = 0;

+    /* S-1b: take the stable id BEFORE linking, so a refusal leaves the
+     * dictionary exactly as it was -- nothing half-defined. */
+    if (vm_dictionary_track_entry(vm, entry) != 0) {
+        vm->error = 1;
+        if (entry->transition_metrics) {
+            sf_free(entry->transition_metrics);
+        }
+        sf_free(entry);
+        entry = NULL;
+        goto create_cleanup;
+    }
    vm->latest = entry;
-    vm_dictionary_track_entry(vm, entry);

    /* Cache coherence: this definition may shadow an older word of the same
     * name that the hot-words cache is still serving. Evict the name so the
@@ -184,7 +184,7 @@ static uint64_t* extract_heat_trajectory(
    for (uint64_t i = 0; i < span; i++) {
        uint32_t word_id = word_ids[start + i];
        uint64_t heat = 0;
-        if (word_id < DICTIONARY_SIZE) {
+        if (word_id < vm->next_word_id) { /* S-1b: was DICTIONARY_SIZE */
            DictEntry *entry = vm_dictionary_lookup_by_word_id(vm, word_id);
            if (entry) {
                heat = (uint64_t)entry->execution_heat;
@@ -92,7 +92,7 @@ void physics_pre_execute(VM* vm, DictEntry* word, DictEntry* prev)
    physics_execution_heat_increment(word);

    uint32_t word_id = word->word_id;
-    if (word_id >= DICTIONARY_SIZE) return;
+    if (word_id >= vm->next_word_id) return; /* S-1b: ids are bounded by next_word_id, not DICTIONARY_SIZE */

    /* Loop #2: Rolling Window of Truth */
    rolling_window_record_execution(&vm->rolling_window, word_id);
@@ -116,15 +116,15 @@ void physics_pre_execute(VM* vm, DictEntry* word, DictEntry* prev)
    }

    /* Record transition from previous word to current word */
-    transition_metrics_record(prev->transition_metrics, word_id, DICTIONARY_SIZE);
+    transition_metrics_record(prev->transition_metrics, word_id, vm->next_word_id);

    /* Update probability cache to find most likely next word */
-    transition_metrics_update_cache(prev->transition_metrics, DICTIONARY_SIZE);
+    transition_metrics_update_cache(prev->transition_metrics, vm->next_word_id);

    uint32_t speculated_word_id = prev->transition_metrics->most_likely_next_word_id;

    /* Speculative prefetch of most likely next word */
-    if (speculated_word_id < DICTIONARY_SIZE &&
+    if (speculated_word_id < vm->next_word_id &&
        transition_metrics_should_speculate(prev->transition_metrics, speculated_word_id))
    {
        DictEntry* spec_entry = vm_dictionary_lookup_by_word_id(vm, speculated_word_id);
@@ -52,6 +52,7 @@

 #include "physics_pipelining_metrics.h"
 #include "vm.h"
+#include "log.h" /* S-1b: a failed transition_heat grow is logged */
 #include <stdlib.h>
 #include <string.h>
 #include <stdio.h>
@@ -149,6 +150,7 @@ int transition_metrics_init(WordTransitionMetrics *metrics) {
    if (!metrics) return -1;

    metrics->transition_heat = NULL;  /* Lazy allocation on first use */
+    metrics->transition_heat_cap = 0;
    metrics->total_transitions = 0;
    metrics->prefetch_attempts = 0;
    metrics->prefetch_hits = 0;
@@ -208,17 +210,41 @@ int transition_metrics_init(WordTransitionMetrics *metrics) {
 * @return              0 on success; −1 on NULL @p metrics, out-of-bounds
 *                      @p next_word_id, or allocation failure.
 */
+static uint64_t g_transition_grow_failures;
+
+uint64_t transition_metrics_grow_failures(void) { return g_transition_grow_failures; }
+
 int transition_metrics_record(WordTransitionMetrics *metrics, uint32_t next_word_id, uint32_t dict_size) {
    if (!metrics || next_word_id >= dict_size) {
        return -1;
    }

-    /* Lazy allocation: only allocate transition_heat array when first transition occurs */
-    if (!metrics->transition_heat) {
-        metrics->transition_heat = (uint64_t *)calloc(dict_size, sizeof(uint64_t));
-        if (!metrics->transition_heat) {
-            return -1;  /* Memory allocation failure */
+    /* Lazy allocation on the first transition, and growth when a word id at
+     * or past the current length arrives (S-1b: ids are no longer capped, so
+     * an array sized once can be outgrown). Grown to the VM's whole id bound,
+     * at least doubling, so a steady stream of new words doesn't regrow on
+     * each one. A failed grow drops this one transition -- the word itself
+     * already ran; there is nothing to refuse -- and is logged and counted. */
+    if (next_word_id >= metrics->transition_heat_cap) {
+        uint32_t  new_cap = dict_size;
+        uint64_t *grown;
+        if (metrics->transition_heat_cap > new_cap / 2u) {
+            new_cap = (metrics->transition_heat_cap > UINT32_MAX / 2u)
+                          ? UINT32_MAX : metrics->transition_heat_cap * 2u;
        }
+        grown = (uint64_t *)calloc(new_cap, sizeof(uint64_t));
+        if (!grown) {
+            g_transition_grow_failures++;
+            log_message(LOG_ERROR, "transition_heat can't grow: dropped");
+            return -1;
+        }
+        if (metrics->transition_heat) {
+            memcpy(grown, metrics->transition_heat,
+                   (size_t)metrics->transition_heat_cap * sizeof(uint64_t));
+            free(metrics->transition_heat);
+        }
+        metrics->transition_heat     = grown;
+        metrics->transition_heat_cap = new_cap;
    }

    /* Record the transition */
@@ -265,7 +291,7 @@ int64_t transition_metrics_get_probability_q48(const WordTransitionMetrics *metr
    /* Safety: Bounds check on target_word_id to prevent out-of-bounds read
     * Note: transition_heat is allocated with dict_size passed to transition_metrics_record()
     * Callers must ensure target_word_id < dict_size used during allocation */
-    if (!metrics->transition_heat) {
+    if (!metrics->transition_heat || target_word_id >= metrics->transition_heat_cap) {
        return 0;
    }

@@ -311,8 +337,10 @@ void transition_metrics_update_cache(WordTransitionMetrics *metrics, uint32_t di
    int64_t max_prob_q48 = 0;
    uint32_t max_idx = 0;

-    /* Find word with highest transition count */
-    for (uint32_t i = 0; i < dict_size; i++) {
+    /* Find word with highest transition count. Bounded by the array's own
+     * length (S-1b): it may lag dict_size until the next grow. */
+    uint32_t limit = (dict_size < metrics->transition_heat_cap) ? dict_size : metrics->transition_heat_cap;
+    for (uint32_t i = 0; i < limit; i++) {
        if (metrics->transition_heat[i] > 0) {
            int64_t prob_q48 = transition_metrics_get_probability_q48(metrics, i);
            if (prob_q48 > max_prob_q48) {
@@ -511,6 +539,7 @@ void transition_metrics_reset(WordTransitionMetrics *metrics) {
        free(metrics->transition_heat);
        metrics->transition_heat = NULL;
    }
+    metrics->transition_heat_cap = 0;

    metrics->total_transitions = 0;
    metrics->prefetch_attempts = 0;
@@ -559,6 +588,7 @@ void transition_metrics_cleanup(WordTransitionMetrics *metrics) {
        free(metrics->transition_heat);
        metrics->transition_heat = NULL;
    }
+    metrics->transition_heat_cap = 0;

    /* Phase 1 Extension: Clean up context window */
    if (metrics->context_window) {
@@ -705,7 +705,7 @@ void rolling_window_cleanup(RollingWindowOfTruth* window)
 * that the first interactive command benefits from cache entries derived from
 * real POST execution data rather than starting cold. The algorithm:
 *
- * 1. Allocates a zero-filled frequency array of @c DICTIONARY_SIZE @c uint32_t
+ * 1. Allocates a zero-filled frequency array of @c vm->next_word_id @c uint32_t
 *    counters via @c sf_calloc.
 * 2. Obtains the current snapshot and tallies occurrences of each word ID
 *    over all @c ROLLING_WINDOW_SIZE history slots.
@@ -733,8 +733,11 @@ void rolling_window_seed_hotwords_cache(const RollingWindowOfTruth* window,
        return; /* Window not ready or no cache */
    }

-    /* Count frequency of each word in rolling window */
-    uint32_t *freq = (uint32_t *)sf_calloc(DICTIONARY_SIZE, sizeof(uint32_t));
+    /* Count frequency of each word in rolling window. Sized to the VM's
+     * word-id bound (S-1b: ids are no longer capped at DICTIONARY_SIZE). */
+    const uint32_t id_bound = vm->next_word_id;
+    if (id_bound == 0) return;
+    uint32_t *freq = (uint32_t *)sf_calloc(id_bound, sizeof(uint32_t));
    if (!freq) return;

    rolling_window_publish_snapshot_if_needed((RollingWindowOfTruth*)window);
@@ -749,7 +752,7 @@ void rolling_window_seed_hotwords_cache(const RollingWindowOfTruth* window,
    for (uint32_t i = 0; i < ROLLING_WINDOW_SIZE; i++)
    {
        uint32_t word_id = view.history[i];
-        if (word_id < DICTIONARY_SIZE)
+        if (word_id < id_bound)
        {
            freq[word_id]++;
        }
@@ -760,7 +763,7 @@ void rolling_window_seed_hotwords_cache(const RollingWindowOfTruth* window,
    uint32_t sum_freq = 0;
    uint32_t count_nonzero = 0;

-    for (uint32_t i = 0; i < DICTIONARY_SIZE; i++)
+    for (uint32_t i = 0; i < id_bound; i++)
    {
        if (freq[i] > 0)
        {
@@ -773,7 +776,7 @@ void rolling_window_seed_hotwords_cache(const RollingWindowOfTruth* window,
    uint32_t threshold = (count_nonzero > 0) ? (sum_freq / count_nonzero) : 1;

    sf_mutex_lock(&vm->dict_lock);
-    for (uint32_t i = 0; i < DICTIONARY_SIZE; i++)
+    for (uint32_t i = 0; i < id_bound; i++)
    {
        if (freq[i] > threshold)
        {
@@ -866,7 +869,7 @@ void rolling_window_seed_pipelining_context(const RollingWindowOfTruth* window,
        uint32_t current_id = view.history[i];
        uint32_t next_id = view.history[(i + 1) % ROLLING_WINDOW_SIZE];

-        if (current_id < DICTIONARY_SIZE && next_id < DICTIONARY_SIZE)
+        if (current_id < vm->next_word_id && next_id < vm->next_word_id) /* S-1b: was DICTIONARY_SIZE */
        {
            /* Lookup dictionary entry for current word by stable ID */
            DictEntry *current_entry = NULL;
@@ -882,7 +885,7 @@ void rolling_window_seed_pipelining_context(const RollingWindowOfTruth* window,
                    context,
                    context_window_size,
                    next_id,
-                    DICTIONARY_SIZE);
+                    vm->next_word_id);

                seeded_transitions++;

@@ -34,7 +34,8 @@
 #include "starkernel/console.h"
 #include "starkernel/kmalloc.h"
 #include "starkernel/q48_16.h" /* Q48_ONE -- diagnostic print only */
-#include "vm.h" /* DICTIONARY_SIZE, WORD_ID_INVALID */
+#include "vm.h" /* WORD_ID_INVALID */
+#include "log.h" /* S-1b: a failed row grow is logged at LOG_ERROR */

 /*
 * StadiumWordSlot - the kernel-side word_id -> cell_index map (decided
@@ -56,19 +57,60 @@ typedef struct {
    uint64_t last_decay_tick;
 } StadiumWordSlot;

-/* Row-per-quota-slot, kmalloc'd at stadium_words_init() to stadium_max_vm_
- * count() rows of DICTIONARY_SIZE entries each -- replaces the old static
+/* Row-per-quota-slot: stadium_max_vm_count() rows -- replaces the old static
 * word_slots[STADIUM_MAX_VM_COUNT][DICTIONARY_SIZE] (2026-08-15: the VM
 * count bound is computed from RAM, not a compile-time constant, so this
- * can no longer be a flat static array). */
+ * can no longer be a flat static array).
+ *
+ * FABRIC-3.9.0.md §8.1 (Stage S-1b): each row is no longer DICTIONARY_SIZE
+ * entries -- word ids aren't capped any more. A row starts empty and grows
+ * (row_cap[slot]) when a word id at or past its length is dispatched. Grown
+ * at dispatch, not at definition: a slot can be granted after the VM's
+ * words exist, and slots are reused across VMs. A failed grow is logged and
+ * counted and the word is simply not admitted this dispatch -- the same,
+ * already-conserving outcome as a full Stadium. This is the id -> cell map,
+ * not the cell pool (S-2 stays open). */
 static StadiumWordSlot **word_slots        = (StadiumWordSlot **)0;
+static uint32_t         *row_cap           = (uint32_t *)0;
 static int               words_initialized = 0;
 static uint64_t         *stat_promotions   = (uint64_t *)0;
 static uint64_t         *stat_evictions    = (uint64_t *)0;
+static uint64_t          row_grow_failures = 0;
+
+uint64_t stadium_words_row_grow_failures(void) { return row_grow_failures; }
+
+/* Grow `slot`'s row to cover word_id, at least doubling. All-or-nothing:
+ * the old row stays in place if the new one can't be allocated. */
+static int grow_row(int slot, uint32_t word_id) {
+    uint32_t         old_cap = row_cap[slot];
+    uint32_t         new_cap = old_cap ? old_cap : 64u;
+    StadiumWordSlot *row;
+    uint32_t         i;
+
+    while (new_cap <= word_id) {
+        new_cap = (new_cap > UINT32_MAX / 2u) ? UINT32_MAX : new_cap * 2u;
+    }
+    row = (StadiumWordSlot *)kmalloc((size_t)new_cap * sizeof(StadiumWordSlot));
+    if (!row) {
+        row_grow_failures++;
+        log_message(LOG_ERROR, "Stadium word row can't grow: not admitted");
+        return -1;
+    }
+    for (i = 0; i < old_cap; i++) {
+        row[i] = word_slots[slot][i];
+    }
+    for (; i < new_cap; i++) {
+        row[i].cell_index      = STADIUM_CELL_NONE;
+        row[i].last_decay_tick = 0;
+    }
+    if (word_slots[slot]) kfree(word_slots[slot]);
+    word_slots[slot] = row;
+    row_cap[slot]    = new_cap;
+    return 0;
+}

 void stadium_words_init(void) {
    size_t   slot;
-    uint32_t i;
    size_t   count = stadium_max_vm_count();

    if (words_initialized) return; /* not re-entrant -- see stadium_words.h */
@@ -79,42 +121,26 @@ void stadium_words_init(void) {
    }

    word_slots      = (StadiumWordSlot **)kmalloc(count * sizeof(StadiumWordSlot *));
+    row_cap         = (uint32_t *)kmalloc(count * sizeof(uint32_t));
    stat_promotions = (uint64_t *)kmalloc(count * sizeof(uint64_t));
    stat_evictions  = (uint64_t *)kmalloc(count * sizeof(uint64_t));
-    if (!word_slots || !stat_promotions || !stat_evictions) {
+    if (!word_slots || !row_cap || !stat_promotions || !stat_evictions) {
        console_println("Stadium words: kmalloc failed for per-VM tables");
        if (word_slots)      kfree(word_slots);
+        if (row_cap)         kfree(row_cap);
        if (stat_promotions) kfree(stat_promotions);
        if (stat_evictions)  kfree(stat_evictions);
        word_slots = (StadiumWordSlot **)0;
+        row_cap = (uint32_t *)0;
        stat_promotions = (uint64_t *)0;
        stat_evictions  = (uint64_t *)0;
        return;
    }

+    /* S-1b: rows start empty and grow on dispatch (grow_row()). */
    for (slot = 0; slot < count; slot++) {
-        word_slots[slot] = (StadiumWordSlot *)kmalloc(DICTIONARY_SIZE * sizeof(StadiumWordSlot));
-        if (!word_slots[slot]) {
-            console_println("Stadium words: kmalloc failed for a word-slot row");
-            /* Free everything allocated so far, including this row's
-             * predecessors, and bail the same all-or-nothing way
-             * stadium_boot_init() does. */
-            {
-                size_t j;
-                for (j = 0; j < slot; j++) kfree(word_slots[j]);
-            }
-            kfree(word_slots);
-            kfree(stat_promotions);
-            kfree(stat_evictions);
-            word_slots = (StadiumWordSlot **)0;
-            stat_promotions = (uint64_t *)0;
-            stat_evictions  = (uint64_t *)0;
-            return;
-        }
-        for (i = 0; i < DICTIONARY_SIZE; i++) {
-            word_slots[slot][i].cell_index      = STADIUM_CELL_NONE;
-            word_slots[slot][i].last_decay_tick = 0;
-        }
+        word_slots[slot]      = (StadiumWordSlot *)0;
+        row_cap[slot]         = 0;
        stat_promotions[slot] = 0;
        stat_evictions[slot]  = 0;
    }
@@ -189,7 +215,7 @@ uint64_t stadium_words_resident_heat(VMUuid vm_id) {
    if (slot < 0) return 0;

    cells = stadium_cells();
-    for (i = 0; i < DICTIONARY_SIZE; i++) {
+    for (i = 0; i < row_cap[slot]; i++) {
        size_t cell = resolve_resident_cell(slot, i);
        if (cell == STADIUM_CELL_NONE) continue;
        sum += cells[cell].header.heat;
@@ -202,11 +228,13 @@ void stadium_word_dispatch(VMUuid vm_id, uint32_t word_id, uint64_t heartbeat_ti
    size_t cell;

    if (!words_initialized) return;
-    if (word_id == WORD_ID_INVALID || word_id >= DICTIONARY_SIZE) return;
+    if (word_id == WORD_ID_INVALID) return;

    slot = stadium_quota_slot_for_vm(vm_id);
    if (slot < 0) return;

+    if (word_id >= row_cap[slot] && grow_row(slot, word_id) != 0) return;
+
    cell = resolve_resident_cell(slot, word_id);

    if (cell != STADIUM_CELL_NONE) {
@@ -268,11 +296,13 @@ void stadium_word_forget(VMUuid vm_id, uint32_t word_id) {
    size_t cell;

    if (!words_initialized) return;
-    if (word_id == WORD_ID_INVALID || word_id >= DICTIONARY_SIZE) return;
+    if (word_id == WORD_ID_INVALID) return;

    slot = stadium_quota_slot_for_vm(vm_id);
    if (slot < 0) return;

+    if (word_id >= row_cap[slot]) return; /* never dispatched here -- can't be resident */
+
    cell = resolve_resident_cell(slot, word_id);
    if (cell == STADIUM_CELL_NONE) return;

@@ -357,6 +357,15 @@ void vm_cleanup(VM* vm)
        vm->latest = NULL;
    }

+    /* S-1b: the stable-id tables are heap-grown now (dictionary_management.c). */
+    sf_free(vm->word_id_map);
+    sf_free(vm->recycled_word_ids);
+    vm->word_id_map = NULL;
+    vm->recycled_word_ids = NULL;
+    vm->word_id_capacity = 0;
+    vm->recycled_word_id_count = 0;
+    vm->next_word_id = 0;
+
    if (vm->memory)
    {
        vm_host_free(vm, vm->memory);
@@ -780,7 +789,7 @@ void execute_colon_word(VM* vm)
            stadium_word_dispatch(vm->stadium_vm_id, w->word_id, vm->heartbeat.tick_count);

            uint32_t word_id = w->word_id;
-            if (word_id < DICTIONARY_SIZE)
+            if (word_id < vm->next_word_id) /* S-1b: was DICTIONARY_SIZE */
            {
                /* L8 FINAL INTEGRATION: Loop #2 always-on - Rolling Window of Truth */
                rolling_window_record_execution(&vm->rolling_window, word_id);
@@ -798,15 +807,15 @@ void execute_colon_word(VM* vm)
                    }

                    /* Record transition from previous word to current word */
-                    transition_metrics_record(prev_word->transition_metrics, word_id, DICTIONARY_SIZE);
+                    transition_metrics_record(prev_word->transition_metrics, word_id, vm->next_word_id);

                    /* Update probability cache to find most likely next word */
-                    transition_metrics_update_cache(prev_word->transition_metrics, DICTIONARY_SIZE);
+                    transition_metrics_update_cache(prev_word->transition_metrics, vm->next_word_id);

                    uint32_t speculated_word_id = prev_word->transition_metrics->most_likely_next_word_id;

                    /* Check if we should speculatively prefetch the most likely next word */
-                    if (speculated_word_id < DICTIONARY_SIZE &&
+                    if (speculated_word_id < vm->next_word_id &&
                        transition_metrics_should_speculate(prev_word->transition_metrics, speculated_word_id))
                    {
                        /* Speculation decision: the most likely next word has high probability
@@ -418,6 +418,15 @@ void vm_cleanup(VM* vm)
        vm->latest = NULL;
    }

+    /* S-1b: the stable-id tables are heap-grown now (dictionary_management.c). */
+    sf_free(vm->word_id_map);
+    sf_free(vm->recycled_word_ids);
+    vm->word_id_map = NULL;
+    vm->recycled_word_ids = NULL;
+    vm->word_id_capacity = 0;
+    vm->recycled_word_id_count = 0;
+    vm->next_word_id = 0;
+
    if (vm->memory)
    {
        sf_free(vm->memory);
@@ -206,7 +206,7 @@ static void forth_PIPELINING_SHOW_TOP_TRANSITIONS(VM *vm) {
    memset(top, 0, sizeof(TransitionPair) * top_count);

    /* Scan all words and keep top N */
-    for (uint32_t i = 0; i < DICTIONARY_SIZE; i++) {
+    for (uint32_t i = 0; i < m->transition_heat_cap; i++) {
        uint64_t count = m->transition_heat[i];
        if (count == 0) continue;

@@ -373,9 +373,10 @@ static void forth_PIPELINING_ANALYZE_WORD(VM *vm) {
    printf("=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "=" "\n");

    /* Calculate entropy / distribution spread */
-    double avg_count = (double)m->total_transitions / DICTIONARY_SIZE;
+    /* S-1b: the id space is vm->next_word_id, no longer a fixed DICTIONARY_SIZE. */
+    double avg_count = (double)m->total_transitions / (vm->next_word_id ? vm->next_word_id : 1u);
    double max_count = 0;
-    for (uint32_t i = 0; i < DICTIONARY_SIZE; i++) {
+    for (uint32_t i = 0; i < m->transition_heat_cap; i++) {
        if (m->transition_heat && m->transition_heat[i] > max_count) {
            max_count = (double)m->transition_heat[i];
        }