LithosAnanake/FABRIC-3.9.8.md

FABRIC-3.9.8.md — Observers and participants: execution plan and log

Status: PLANNED. No stage started. The design and its open rulings (O-1 to O-4, plus the discard question) are in FABRIC-3.9.0.md.

Each stage:

• needs Captain Bob's explicit go;
• gets its own commit;
• gets its own three-architecture acceptance (clean qemu; amd64, then aarch64, then riscv64; foreground; visible framebuffer; one at a time; coupled images reset before each).

Evidence is named before each stage runs.

The standing acceptance sequence for every stage, unchanged from FABRIC-3.8.md:

• a boot to [zuse@Hera] ok> with zero UNKNOWN WORD, faults and frame corruption;
• the dict_hash set identical across ISAs;
• the Stage G interleave (two workers, A/B), byte-identical across ISAs;
• mint → WIREBIND → USE → 1 2 + . → 3;
• 0 bytes of idle log growth over 20 s.

1. Staged plan

Stage	Change	Evidence (named in advance)
O-A	Observer role (FABRIC-3.9.0.md §2.1). The absolute-heat-share charge is discarded (ruling 5). Observers are left out of the decision's candidates, the work-heat charge and the floor, and (O-2) the absolute charge. Observers with events go ahead at clean boundaries, among themselves by wait stamp (§2.4, per O-1).	(Corrected before starting.) The standard sequence, with the G interleave unchanged. Console responsive under load: 1 2 + . typed while wa/wb run prints 3 in the middle of the A/B stream. Observer precedence: a message to Hestia sent in the same line as the two worker jobs runs before any A/B. Not run: the busy-wait DoE driver. Under O-1, Hera running DOE is an observer acting on an event, and nothing displaces her, so MU-WAIT-ALL would wait forever. The originally named evidence ("Hera's share falls on the unchanged busy-wait driver") contradicts the design. DOE stays unrun until O-C replaces the driver.
M-0 (first; a defect against the standing no-fixed-bounds rule)	Kernel-Hermes arena, per-VM queues and channel member lists grow on demand; Q_SLOT decoupled from arena size (FABRIC-3.9.0.md §7.0).	The standard sequence. The existing kernel-Hermes boot self-tests (allocator, publish, drain, chunk, negotiation) still PASS, with expected_n now set by the reservoir, not the arena. A probe (reverted) admits more than 32 messages system-wide, spread over senders with reservoir to pay, and more than 32 to one queue, with conservation and the ledger holding throughout. A channel takes more than 64 members. Nothing moves or corrupts across a grow: messages queued before a grow drain intact after it.
M-A (ahead of O-B, per Captain Bob: "close the gap first")	Manual messaging words over the existing C API (FABRIC-3.9.0.md §7.2, names per M-1).	The standard sequence. At the console: CHANNEL, then JOIN-VM of wa and wb, then PUBLISH of a marker payload delivers to both (n = 2, both print). LEAVE-VM then PUBLISH delivers to one. CHANNEL-CLOSE succeeds, and refuses channel 0. INJECT lands in a queue that MAIL?/.MAIL show before the recipient runs. .CHANNELS lists what was made. The dict_hash set changes by the new words and is identical across ISAs.
M-B	Autonomous events (§7.3): event channels created at boot; EV-DONE/EV-BORN/EV-DIED fired from the existing kernel points, for participants, funded per M-5, dropped and counted when full (M-6).	The standard sequence. A VM that JOINs EV-BORN/EV-DONE/EV-DIED and defines the ON-… words sees one event for each WORKER-BIRTH, each job finished, and each KILL, with no SEND anywhere in the job text. A VM that didn't join sees nothing. The dropped-event counter reads 0 in normal use; a probe that floods the arena shows events dropped and counted, never blocking.
O-B	Hera's own mail drained at sk_vm_idle_point(), with the readline n > 0 gate; the timer-paced pump drain removed (§2.5).	The standard sequence, plus: a message sent to Hera while she sits idle at the prompt is acted on without waiting for the idle beat. A probe records the delay in beats from admission to drain; it must be well under SK_IDLE_BEAT_INTERVAL. The relay and BLK-ATTACH flows are unchanged.
O-C	An event-driven DoE driver (§3, as revised by §7.4 once M-B lands): DOE starts trial 1 and returns to the prompt; Hera's ON-DONE (fed by EV-DONE) counts, ends trials and starts the next; MU-WAIT-ALL and the job text's own completion SEND are deleted. Capsule only.	A 1-rep smoke campaign on amd64 runs all 6 cells to MU-CAMPAIGN: complete, with every trial fail=0 and N/N KILL … dead. Hera's rows during trials are only event handling. The prompt answers typed input while trials run. Then the standard sequence on all three ISAs.
O-D	The DoE relaunch: 12345 3 S" <isa>" DOE, amd64, then aarch64, then riscv64, with the ring extracted between ISAs and artemis.img not reset in between.	A fresh time estimate from O-C's measured row rate, before launch. Then per ISA: 18 MU-TRIAL markers, fail=0, MU-CAMPAIGN: complete.
Merge	This branch to master, once all three ISAs have completed O-D.	Per this project's own merge practice, at Captain Bob's word.

Stop conditions (re-plan rather than push on):

• dict_hash diverges across ISAs;
• conservation (vm_physics_conserved(), stadium_conserved()) goes false;
• an observer is ever returned as a heat candidate;
• a switch storm: log growth at an idle prompt, with QEMU pinned;
• a trial that never produces its marker.

2. Execution log

• Stage O-A — the observer role (FABRIC-3.9.0.md §2). 2026-09-24. Captain Bob: "all as proposed, go ahead with O-A" (O-1 to O-4 and the discard, all as proposed). - Discarded: the uncommitted absolute-heat-share charge (git checkout of capsule_vm_physics.c/.h), per ruling 5. - capsule_vm_switch_signal.c/.h: - Each slot carries an observer flag, with sk_vm_switch_signal_set_observer() and is_observer(). - sk_vm_switch_decide() now works in tiers: 1. An observer that is itself in progress (a live non-IDLE frame) or holding an event keeps the CPU. 2. Otherwise, an observer with an event, meaning mail (or console input, for Hera), goes ahead of every participant. Among observers, the oldest wait stamp wins. 3. Otherwise, participants are compared by work heat exactly as in FABRIC-3.8. Observers are never heat candidates. - note_work_gained() never floors an observer and leaves observers out of the floor set. The running VM counts only if it is a participant. - capsule_birth.c: is_fleet_foundation is hoisted to function scope. Registration marks the Tripod as observers from that same birth-time fact. - kernel_main.c: Hera is marked an observer where she registers. - vm_runtime.c: observers are not charged at all (O-2). - Nothing a VM executes can set or clear the role.

  **Evidence, as corrected before starting (§1).** Ordinary words only, typed at
  `[zuse@Hera] ok>`, byte-identical on all three ISAs (`logs/20260924-084904/amd64/`,
  `-090107/aarch64/`, `-091424/riscv64/`):
  - **G interleave unchanged:** A = 40, B = 40, 48 runs.
  - **Observer precedence:** `S" LA" S" wa" SEND . S" LB" S" wb" SEND . S" 72 EMIT 72 EMIT
    72 EMIT" S" Hestia" SEND .` gives `HHHABBAABBAA…`. Hestia's mail ran before any
    participant output once Hera went idle.
  - **Console responsive under load:** with `wa`/`wb` on 200 markers × 20,000-iteration
    loops, `1 2 + .` typed mid-run appears as `…ABABAB1 2 + .` / `3  ok` / `ok> ABABBA…`.
    Hera, an observer with console input, got the CPU at the next worker boundary, ran the
    line, and the workers resumed. This is the C-4 (α) starvation recorded at C-b, fixed.
    *(The first amd64 attempt at this check used loops that finished before the line was
    typed, so it proved nothing. It was re-run with the heavy loops, and the script was fixed
    for the other two ISAs.)*
  
  **Acceptance** (same boots):
  - All reach `[zuse@Hera] ok>` with zero `UNKNOWN WORD`, faults and frame corruption.
  - The `dict_hash` set is unchanged from Stage I (word_count 531, M7.1a
    `0x572daac83b797332`), since no words were added or removed.
  - Mint → WIREBIND → `USE` → `1 2 + .` → `3`.
  - Idle log growth: 0 bytes.
  - Clean build on all three plus hosted, no new warnings.
  
  **Not run, by design:** the busy-wait DoE driver, which would deadlock under O-1 (§1).
  Don't run `DOE` until O-C.
  

• Stack-convention fix: out-of-bounds stack writes in four vendored word files (found during S-1a, 2026-09-24). Captain Bob: "Fix now, own commit."

  **The defect.** The VM's `dsp` is the index of the top item (−1 = empty):
  `vm_pop` reads `data_stack[dsp--]`. Four vendored files instead treated `dsp` as a count,
  pushing with `data_stack[vm->dsp++] = x` (11 sites) and popping with
  `data_stack[--vm->dsp]` (17 sites):
  - `physics_freeze_words.c`: 7 pushes, 12 pops;
  - `dictionary_heat_diagnostic_words.c`: 4 pushes, 2 pops;
  - `physics_diagnostic_words.c`: 1 pop;
  - `physics_benchmark_words.c`: 1 pop.
  
  **The consequences:**
  - On an empty stack a push wrote `data_stack[-1]`, **8 bytes before the VM struct**. POST's
    own `DECAY-RATE@.basic` test does exactly that on every boot.
  - Otherwise a push overwrote the top item and left `dsp` one slot off.
  - The pops read the item *below* the top, and on the last pop, `data_stack[-1]`.
  - Four one-item words also guarded with `dsp < 1`, which demands two items, so
    `LOOKUP-STRATEGY!`, `COMPARE-LOOKUPS`, `PHYSICS-BURN` and `BENCH-DICT-LOOKUP` always
    errored on valid input.
  
  **How it was found.** The S-1a parity probe's incomplete counter read one too many on
  aarch64 only.
  - A per-test hook in the test runner named `DECAY-RATE@.basic` as the test after which
    the counter moved (`logs/20260924-104424/aarch64/`).
  - Writing a marker into `mama->data_stack[-1]` changed the counter to the marker
    (`-105120`).
  - Printing runtime addresses (`-105729`) showed why aarch64 only: the aarch64 kernel runs
    inside the `lld-link` PE, whose `.bss` puts `parity.c`'s counter directly before
    `sk_mama_vm`.
  - On amd64 and riscv64 the same write lands in the last cell of the POST block-RAM buffer,
    silently.
  - An interim hypothesis of memory aliasing, drawn from ELF symbol addresses that don't
    describe the aarch64 runtime image, was wrong and is withdrawn.
  
  **The fix:**
  - Every site now uses the VM's own `vm_pop(vm)` / `vm_push(vm, x)`, which check bounds.
  - The four one-item guards are now `dsp < 0`.
  - The two- and three-item guards (`dsp < 1`, `dsp < 2`) were already correct for the
    top-index convention and are unchanged.
  
  **Evidence.** A probe (reverted) compared the 8 bytes before the VM struct across POST,
  plus console checks, on all three ISAs (`logs/20260924-111124/amd64/`, `-112508/aarch64/`,
  `-113949/riscv64/`):
  - `STACKFIX PROBE: … UNCHANGED across POST`;
  - `DECAY-RATE@ DEPTH . .` gives `1 1`;
  - `S" DUP" FROZEN? DEPTH . .` gives `1 0`;
  - `HEAT@` leaves 1 item, and `HEAT-PERCENTILES` leaves 3;
  - `1 LOOKUP-STRATEGY! LOOKUP-STRATEGY@ .` gives `1`, depth 0, where it used to error;
    and `0 …` gives `0`.
  
  Acceptance in the same boots:
  - the O-A checks: G interleave A = 40/B = 40/48 runs, `HHH` precedence, and the typed line
    running mid-stream;
  - relay `3`, idle growth 0;
  - `dict_hash` unchanged (531, `0x572daac83b797332`);
  - POST passes;
  - clean kernel and hosted builds.
  

• Stage S-1a — parity hashing covers the whole dictionary (FABRIC-3.9.0.md §8.1, class C). 2026-09-24. Captain Bob: "all as proposed, commit and start S-1."

  **The change** (`src/starkernel/vm/parity.c`, `include/starkernel/vm/parity.h`):
  - **`MAX_DICT_ENTRIES` (2048) removed.** The walk covers the whole link chain. The entry list
    is allocated to the dictionary's actual size (`kmalloc` in the kernel, `malloc` hosted) and
    freed after.
  - **Cycle check replaces the cap.** The only thing the cap guarded against, a corrupt cyclic
    chain, is now detected directly (Floyd: a second pointer moving two links per step).
  - **The colon-body walk is bounded by `vm->here`,** not a 1024-cell cap. A body that reaches
    `HERE` without `EXIT` or a NULL cell is marked incomplete, not quietly truncated.
  - **Failures follow §8.1's rules.** Every failure (cyclic chain, entry list not allocated,
    malformed body) is:
    - printed in full to the console, `PARITY:INCOMPLETE #n -- <why>`. `log_message`'s line
      buffer is 64 bytes and cut every reason off mid-sentence.
    - logged at `LOG_ERROR` for the persisted record;
    - counted (`sk_parity_incomplete_count()`).
  - **An incomplete hash is marked as such.** The record prints `hash=INCOMPLETE` (new
    `ParityPacket.hash_complete`) and `PARITY:FAIL code=4`
    (`SK_BOOTSTRAP_PARITY_INCOMPLETE`), never a value that looks valid.
  - `sk_dict_word_count` returns 0 on a cyclic chain, logged and counted.
  
  **Two pre-existing defects found by the probe, both fixed in the walker:**
  1. **The colon-body walker dereferenced inline data.** It treated every cell of a body as a
     `DictEntry*` and followed it. Only `LIT`'s operand was known to be data. `S"` string
     bytes, branch offsets and other inline operands were followed too.
     - This was harmless while parity only ever ran at early boot, before any colon bodies
       existed.
     - The first hash of a mature dictionary took a #GP (vector 13) and halted the kernel. This
       is the "interrupt exception" Captain Bob reported (`logs/20260924-094537/amd64/`,
       `-095154/amd64/`).
     - The **original** code faults the same way (`logs/20260924-095730/amd64/`), so the
       defect predates S-1a.
     - **Fix:** a cell is dereferenced only if it is a known dictionary entry, found by binary
       search in a sorted copy of the entry list. Any other cell is hashed as its raw value and
       never followed.
  2. **A `LIT` operand holding an entry pointer was hashed by address.** `['] WORD` compiles
     `LIT <DictEntry*>`, a native address that differs per ISA (seen in `ACL-BOOT`'s
     `ACL-PIN` sequences). So the mature-dictionary hash differed across ISAs:
     - amd64 `0x945a76bc6ace1fdc`, aarch64 `0x8cc1255e6a40c12e` (`-100252`, `-100732`).
     - **Fix:** rule 2 (word_id, never address) now applies to such an operand. The walker
       hashes a tag byte (`0xE7`) plus the entry's `word_id`, so it can't collide with a plain
       number. Any other literal is still hashed as its value.
  
  A third defect, an incomplete counter one too high on aarch64 only (`-101803`, `-102504`,
  `-103207`, `-103820`), was not in parity. It was the out-of-bounds stack write fixed in its
  own commit (the entry above).
  
  **Evidence.** A probe (reverted) at the end of boot:
  1. hashed Hera's full dictionary;
  2. registered 2,000 more words (`PZ0000`…) and hashed again;
  3. made the chain cyclic and asked for both the word count and the hash.
  
  The results are byte-identical on all three ISAs (`logs/20260924-115524/amd64/`,
  `-115954/aarch64/`, `-120533/riscv64/`):
  
  | Probe step | Result |
  |---|---|
  | before | `word_count=563 hash=0x224007f9af328111`, `PARITY:OK` |
  | after +2000 | `word_count=2563 hash=0xcf85876337e75a84`, `PARITY:OK`. Past the old 2048 cap: the whole dictionary is hashed. |
  | cyclic | `PARITY:INCOMPLETE #1`/`#2 -- dictionary link chain is cyclic (corrupt)`, two `[HADES][ERROR]` lines, `word_count=0 hash=INCOMPLETE`, `PARITY:FAIL code=4` |
  | counter | `incomplete_count=2` (word count + hash) |
  
  The "after" record shows `latest_id=4294967295`. That is `WORD_ID_INVALID`: the 2,000
  probe words ran past the 1024 stable word ids, and each logged "out of stable IDs, disabling
  metrics". That is S-1b's defect, not S-1a's. The §8.1 row now notes that this path does log
  a warning, but neither counts nor refuses.
  
  **Acceptance** on the clean tree, with the probe reverted (`logs/20260924-121207/amd64/`,
  `-122500/aarch64/`, `-123759/riscv64/`):
  - Boot M7.1a is unchanged: `word_count=531 hash=0x572daac83b797332`. At boot there are no
    colon bodies or entry-pointer literals yet, so neither fix changes the boot hash.
  - The `dict_hash` set is unchanged: Hera `0xda9742ce9c3376a2`, Hestia `0x413e7d086f342d8b`,
    Artemis `0xdfd68316d5f5ba85`.
  - The O-A checks pass: G interleave A = 40/B = 40/48 runs, `HHH` precedence, and a typed
    `1 2 + .` gives `3` mid-stream.
  - Mint → WIREBIND → `USE` → `1 2 + .` → `3`. Idle log growth 0. Zero `UNKNOWN WORD` or
    faults. QEMU exits cleanly.
  

• Stage S-1b — dictionary word ids grow with the dictionary (FABRIC-3.9.0.md §8.1, class C). 2026-09-24. Captain Bob: "go ahead with S-1b."

  **The defect.** Stable word ids came from fixed `word_id_map[1024]` /
  `recycled_word_ids[1024]` tables in the `VM` struct. Word 1025 onward got `WORD_ID_INVALID`
  and a `LOG_WARN` ("out of stable IDs, disabling metrics"), but was still defined. It then ran
  outside the physics:
  - the rolling window and the pipelining hooks skipped it;
  - Stadium never admitted it;
  - parity hashed it as `0xFFFFFFFF`.
  
  Nothing counted it and nothing refused it.
  
  **The change:**
  - **`vm.h` / `dictionary_management.c`: the core.**
    - `word_id_map` and `recycled_word_ids` are heap tables sharing one `word_id_capacity`.
    - They start at 64 and double when a fresh id reaches capacity.
    - Growth is all-or-nothing: both new tables are allocated before either old one is touched.
      It uses allocate-copy-free, because the kernel's `sf_realloc()` does not copy.
    - Numbering is unchanged (a recycled id first, else `next_word_id++`). That is why boot
      parity is unchanged.
    - Both `vm_cleanup()`s free the tables.
  - **Refusal is whole.**
    - `vm_dictionary_track_entry()` now returns a status.
    - `vm_create_word()` takes the id *before* linking the entry. On failure it frees the entry
      and its transition metrics, sets `vm->error`, and returns NULL.
    - The failure is logged at `LOG_ERROR` ("word id table can't grow: '…' refused") and counted
      (`vm_dictionary_word_id_refusals()`).
    - Every caller already handled NULL from the existing malloc-failure and pinned-shadow paths.
  - **The bound for every reader of the id space is `vm->next_word_id`,** not `DICTIONARY_SIZE`.
    Every issued id, live or recycled, is below it:
    - `lookup_by_word_id`, `untrack`;
    - both interpreters' physics hooks (`physics_execution_hooks.c`, `vm_core.c`);
    - the rolling-window seed (its frequency array is now sized to the bound);
    - `inference_engine.c`;
    - the pipelining diagnostics.
  - **`transition_heat` (`physics_pipelining_metrics.c`).** Each word's array was sized once, at
    whatever size its first record passed in. With a dynamic bound, that would have been a heap
    overflow on the first transition to a newer id.
    - It now has its own `transition_heat_cap` and grows on record (to the bound, at least
      doubling).
    - Every reader bounds by that field. `get_probability` had no bounds check at all.
    - A failed grow drops that one transition (the word already ran; nothing is left to refuse).
      It is logged and counted (`transition_metrics_grow_failures()`).
    - Pipelining is compiled out of the kernel (`ENABLE_PIPELINING=0`), so this path is live
      only in hosted builds.
  - **Stadium's id→cell rows (`stadium_words.c`).** These were `[slot][DICTIONARY_SIZE]`.
    - Each row now starts empty and grows on dispatch (`row_cap[slot]`), not at definition,
      because a quota slot can be granted after a VM's words exist, and slots are reused across
      VMs.
    - A failed grow is logged and counted (`stadium_words_row_grow_failures()`). The word is not
      admitted that dispatch, which is the same, already-conserving outcome as a full Stadium.
    - This is the id→cell map, not the cell pool, so **S-2 stays open**.
  - `DICTIONARY_SIZE` stays defined for its one remaining user, reported below.
  
  **Evidence, named before the run.** A probe (reverted) at the end of boot, byte-identical on
  all three ISAs (38 lines; `logs/20260924-130702/amd64/`, `-131140/aarch64/`,
  `-131624/riscv64/`):
  
  | Check | Result |
  |---|---|
  | Start | `next_word_id=738`, capacity 1024 (grown 64 → 1024 during boot) |
  | +2000 words | `next_word_id=2738`, capacity 4096, `PZ1999 word_id=2737`, refusals 0, and no "out of stable IDs" line. Parity OK, `word_count=2563`, `latest_id=2737` (was `4294967295` at S-1a), `hash=0xc873c20962d728ed`. The hash differs from S-1a's `0xcf85…` as expected: 1,539 words now hash real ids instead of `0xFFFFFFFF`. |
  | Physics past id 1024 | `: PZHOT 1 DROP ; : PZRUN 200 0 DO PZHOT LOOP ; PZRUN` gives PZHOT id 2738 and PZRUN id 2739; `lookup(2738)==PZHOT`; **200** PZHOT entries in the rolling window (the hooks used to skip any id ≥ 1024); Stadium promotions +2 (both words admitted); row-grow failures 0. |
  | FORGET/recycle | `FORGET PZHOT` leaves 2 recycled ids and `lookup(2738)==NULL`. `: PZNEW ;` reuses id 2738, and `lookup` returns PZNEW. |
  | Forced grow failure | After filling the tables to capacity (4096), one allocation failure was injected into the grow. `PZFAIL` then gives NULL, `vm->error=1`, refusals 1, and `[HADES][ERROR] word id table can't grow: 'PZFAIL' refused`. `latest`, `word_count`, capacity and `next_word_id` are all unchanged, and `PZFAIL` is not findable. |
  | Retry | With the injection cleared, `PZOK` is defined: id 4096, capacity 8192. |
  | Final parity | `word_count=3922`, `hash=0x5af2d7e820d21624`, `incomplete_count=0` |
  
  **Acceptance** on the clean tree, with the probe reverted (`logs/20260924-132055/amd64/`,
  `-133339/aarch64/`, `-134638/riscv64/`):
  - Boot M7.1a is unchanged (`word_count=531`, `latest_id=700`, `0x572daac83b797332`), and so is
    the `dict_hash` set (Hera `0xda9742ce9c3376a2`, Hestia `0x413e7d086f342d8b`, Artemis
    `0xdfd68316d5f5ba85`).
  - The O-A checks pass: G interleave A = 40/B = 40/48 runs, `HHH` precedence, and a typed
    `1 2 + .` gives `3` mid-stream.
  - Mint → WIREBIND → `USE` → `3`. Idle log growth 0. Zero `UNKNOWN WORD` or faults. QEMU exits
    cleanly.
  - Clean kernel builds on all three ISAs, plus the hosted build.
  
  **Found, reported, not fixed:**
  1. `dictionary_heat_optimization.c:123`: `cell_t heats[DICTIONARY_SIZE]` snapshots at most
     1024 words' heat and silently ignores the rest. That is class C, keyed on the dictionary
     count, not on ids.
  2. `vm_create_word()`: a failed transition-metrics malloc logs a WARN and defines the word
     anyway, with no pipelining metrics. That is a partial.
  3. Neither `vm_cleanup()` frees `transition_heat` (or `context_window`) inside each word's
     metrics, only the metrics struct itself. That is a leak on every VM kill with pipelining on.
  4. `transition_metrics_update_cache()` scans the whole id space on every recorded transition.
     The hot-path cost grows with the dictionary, now that the dictionary can grow.
  5. The kernel's FC-bucket grow (`sf_fc_reserve`) copies after `sf_realloc()`, which is correct,
     but never frees the old block. Each bucket grow leaks it.
  6. §8.1 asks that each failure counter be readable by a FORTH word or DoE row. S-1a and S-1b
     expose C accessors only (`sk_parity_incomplete_count()`,
     `vm_dictionary_word_id_refusals()`, `transition_metrics_grow_failures()`,
     `stadium_words_row_grow_failures()`). Adding a word changes the `dict_hash` set, so this
     awaits Captain Bob's ruling.