Files
LithosAnanke/docs/v4.0.0/plans/2026-11-13-engine-step4-phase7-the-loops-and-parity.md
T

227 lines
18 KiB
Markdown

# ENGINE Step 4, Phase 7: the physics loops on the word records, and v3's whole POST -- Plan
> **For agentic workers:** steps use checkbox (`- [ ]`) syntax. One QEMU at a time, in the foreground,
> every boot under `timeout 570` (`.claude/CLAUDE.md`, `.claude/memory/rules.md`); every log and CSV
> is committed. Test first: each change starts with a failing test.
**Goal (#189):** every loop v3 runs before its prompt runs on v4, on the kernel's word records
(Phase 2), on every node; and v3's whole POST passes on v4.
**Parent plan:** `2026-10-20-engine-step4-node-in-the-vms-place.md`, Phase 7; after Phase 2
(`2026-11-13-engine-step4-phase2-word-accounts.md`, protocol PASS on e8e7ec8b).
## Rulings (Captain Bob, 2026-11-13)
1. **Every call, as v3.** The per-call loops (decay on touch, the rolling window, pipelining) run at
every `call`, from the hook, as v3's run at every execution -- not replayed from a trace each tick.
Measured, kept or reverted on the numbers (as HEAT-OP was).
2. **Rewrite for v4.** The loops are written afresh for v4's records, with v3's code
(`v3/src/rolling_window_of_truth.c`, `physics_pipelining_metrics.c`, `inference_engine.c`,
`physics_metadata.c`, `physics_execution_hooks.c`, `compudynamics.c`, `ssm_jacquard.c`, and the
kernel's `vm_runtime.c` tick) and v3's POST as the specification.
3. **Accepted on v3's whole POST.** Not the loops' own modules alone: every module of v3's POST passes
on v4.
4. **Words in F18 and StarForth, not C** (2026-11-13, on the Mama module: "do not compose words in
C99, use F18 and StarForth"). Every word this phase adds -- the loops' (`FREEZE-WORD`, `HEAT@`,
the inference words, the entropy words), Mama's, the StarForth words -- is written in the system's
F18 or in StarForth. What only the kernel holds (a word's record, the window, the registry) is asked
for by a request on port 0, as `KERNEL-WORD` words do. The loops themselves run in the kernel, on its
records (Q4).
5. **Mama's 20 cases** run as they are, on v4 words with v3's names mapped to v4's equivalents; **the
C modules** (ACL, break_me) are rewritten for v4. Inventory and rulings: `docs/v4.0.0/POST-PARITY.md`.
6. **Pipelining and the hot-words cache: "Built, off as v3"** (2026-11-13). Asked first "Build a
hot-words cache" (in front of FIND, in F18/StarForth, as v3 promotes into its own); then it was found
that v3 as built has both off (`ENABLE_PIPELINING`, `ENABLE_HOTWORDS_CACHE`: `n` in
`Kconfig.physics` and `kernel/Makefile`), so v3's boot records no transitions and uses no cache.
Ruled: both written and tested, behind the same two build options, off by default as in v3.
7. **Compudynamics on the F18 opcodes as well as the composed words** (2026-11-14: "we're supposed to be
working on f18 opcode as well as our composed f18 dictionary"; ruled "#1 and an easy revert if opcodes
are too granular"). Every loop runs on both: the 32 opcodes are a second set of records on each node,
fed at every retired instruction (a hook at retire, as Part A's at `call`) -- heat, decay, temperature,
an opcode window, opcode transitions (under L4, as the words'), background decay and inference in the
cycle. L8 scores from the words, as v3, and its card's gates govern both. The tick stays paced by word
calls, as v3's. **The revert:** one build option, `OPCODE_PHYSICS` (`Kconfig.physics`, default y);
off, the opcodes are counted and charged to the Stadium as in Phase 1, and nothing more.
**Where the arithmetic lives:** compudynamics is the Stadium/fabric's (2026-11-14: "The compudynamics
is stadium/fabric"): the kernel computes; a word that reads or sets it is F18 that asks the kernel.
## What ruling 3 brings in
v3's POST is 26 modules (`v3/src/test_runner/modules/`, about 1,050 cases; v3's bare-metal boot says
1050). v4's POST is v3's cases for the FORTH-79 Required Word Set, 539 (`v4/tools/mkpost.py`). The rest
need words v4 does not have yet, which the parent plan put in other phases:
| v3's modules | Needs | Parent plan's phase |
|---|---|---|
| physics_freeze, inference | the loops, and the words that read and set them (`FREEZE-WORD`, ...) | 7 |
| starforth, q48, format, double (beyond the Required set), string, system, vocabulary, ... | the StarForth words: `init.4th`, `starforth.4th` | 3 |
| acl | `ACL.4th` on v4; the word card (the four ACL fields) on the record; the countdown at `call` | 4 |
| block | the blocks the kernel serves; writes | 5 (in part) |
| mama_forth, integration, break_me, stress | to be read case by case (Task 0): some name v3's VMs and fleet | -- |
So, by ruling 3, Phase 7 is accepted only when Phases 3 and 4 (and the block cases of 5) are in. The
ruled order (Part A, Phase 1, Phase 2, Phase 7) stands for the loops; the words of 3 and 4 come in
under this plan, as tasks of it.
## The design
**The records grow v3's physics.** A word's record (`v4_mesh_account`, Phase 2) gains what a
`DictEntry` carries for the loops: temperature, the last decay tick, mass, the state flags (frozen,
pinned), and its transition metrics (by next word ID). Per node: one rolling window, one pipelining
context, one L8 mode, one inference state.
**At every `call`** (the hook; ruling 1), in v3's order (`V3-PARITY.md` 2.1): decay the word by the
ticks since it was touched (Loop #3, unless frozen or L8 has it off); its heat +1 (Loop #1, done); the
Stadium (Phase 2 charges per tick: kept); the rolling window records its ID (Loop #2); the transition
from the word before is recorded and the likely next worked out (Loop #4); the ACL countdown (with
Phase 4).
**At every heartbeat tick**, for every node: the window tuner (Loop #5), the slope validator
(Loop #6), background decay, L8's mode update, the inference engine, the snapshot -- as
`vm_runtime.c` does for v3's VM.
## Tasks
### Task 0: the inventory
- [x] Every case of v3's POST, by module, in v3's own order: kept as it is, kept with v4's address
units (D-1), needs a word of Phase 3, 4, 5 or 7, or names v3's VMs (Mama, the fleet) -- the last
listed for Captain Bob's ruling, case by case, before anything is written for them.
- [x] `docs/v4.0.0/POST-PARITY.md`: the table, and the count v4 must reach (about 1,050; 539 today).
Mama and the C modules ruled (above).
### Task 1: the records' physics fields, and the per-node state
- [x] Test first (`test_mesh_boot.c`): a new word's heat, temperature, last decay tick and state flags
are 0; each node starts with v3's decay slope (21845, Q48.16) and Loop #3 on; a word's state
flags as v3 derives them (IMMEDIATE read from its header).
- [x] `v4_mesh_account` gains `heat` (v3's `execution_heat`), `last_decay_tick`, `temperature_q8`,
`state_flags` (the kernel's own: pinned, frozen); `v4_mesh_physics` per node (tick, decay slope,
L8's Loop #3 gate); `v4_mesh_state_flags`. Immediate and hidden are read from the node's header
when asked, so they are never stale. v3's `mass_bytes` is only ever set, never read, in v3: not
kept until a word reads it.
### Task 2a: the per-call loops at `call` -- decay on touch, heat, temperature, the rolling window -- measured
- [x] Test first (`test_mesh_boot.c`, 56 checks): heat +1 a call; decay by the ticks since at v3's
slope and arithmetic (6 ticks: 1; 3 ticks: 0); none for a frozen word, none while L8 has Loop #3
off; the temperature; the window.
- [x] `mesh.c`'s call hook, in v3's order; the kernel's heartbeat read at each call (`ticks`). Until
Task 3 brings L8, Loop #2 and #3 are on (v3's boot sets both off and lets its card decide).
- [x] Measured: amd64 boot 146 s, against Phase 2's 142 s (about +3%).
### Task 2b: pipelining (transitions, the likely next word, speculation) and the hot-words cache in front of FIND, behind `ENABLE_PIPELINING` and `ENABLE_HOTWORDS_CACHE`, off as v3
- [x] Test first: `test_mesh_boot.c` built twice -- as the build is (off: no transitions, an empty
cache; 57 checks) and with both on (`make -C v4 mesh-boot-on`, 65 checks): PA's 12 transitions to
PB, its likeliest next; speculation after 10 puts PB in the node's cache; hits; FIND through the
cache; PB defined again is taken out; a word past heat 50 goes in; FORGET takes the rest out.
- [x] The kernel (`mesh.c`), at every call: v3's hit check, the transition, the likeliest next (kept as
each count rises, not a scan of 1024 a call: the same word but in a tie v3's Q48.16 rounding
makes, recorded in the code), speculation at v3's thresholds (1/2, 10 samples). Counters from
the owner's heap on a word's first transition; given back when it goes.
- [x] The node (`dict.f18`, `(LOOKUP)`): `(HOT)`'s xts tried first, in F18, calling only `C@` as before;
the kernel fills it as v3's promote, and takes out entries that go or whose name is defined again.
Empty unless the kernel's build has `ENABLE_HOTWORDS_CACHE`.
- [x] One departure, ruled by default (Captain Bob did not choose; recorded): v3 promotes a hot word
when a lookup finds it in the dictionary; v4's heat is the kernel's, so it is promoted at the call
where it is hot.
- [x] Measured: amd64 174 s and 181 s; 2a's own commit booted again just after, 175 s (it was 146 s an
hour before). The host slowed; 2b costs nothing measurable, as the host mesh boot also shows.
- **Reported, not changed (v3's own):** with the cache full, v3's promote takes the next slot in turn
without looking for the word already there, so a few hot words churn through all 32 slots
(`hotwords_cache_promote`, `v3/src/physics_hotwords_cache.c`). v4 does the same. And v3 never calls
`transition_metrics_record_prefetch_miss`: its misses are always 0, as v4's.
### Task 3: the per-tick loops (tuner, slope, background decay, L8, inference, snapshot)
- [x] **The tick, by execution** (Captain Bob, 2026-11-13: "By execution, as v3"). v3's kernel runs a VM's
heartbeat cycle every 256 word executions (`HEARTBEAT_CHECK_FREQUENCY`, `kernel/src/vm/vm_core.c:912`
and `:1210`), and its tick is the count of cycles; the timer heartbeat is not the VM's tick. Task 2a
had used the kernel's timer: corrected. Each node counts its word calls; every 256th runs its cycle
at the call. Test first (`test_mesh_boot.c`, 59 and 67 checks).
**Superseded 2026-11-17** by `2026-11-17-one-adaptive-heartbeat.md` (Captain Bob: "ONE ADAPTIVE
HEARTBEAT. EVERY VM USES THAT ONE CLOCK"). The tick is now the kernel's one heartbeat, and every node
ticks on it in lockstep. The count of word calls, the nodes' Loop #7 and Hera's re-arming of the timer
are removed. The heartbeat is held through POST; FABRIC-3 XLVIII.25.
- [x] Background decay (v3's `vm_tick_apply_background_decay`): 64 words a cycle from a cursor down the
dictionary, so a word not called cools. The window's adaptive pass (v3's
`rolling_window_run_adaptive_pass`): diversity against the last; halve to 256, double to 4096.
Test first (`test_mesh_boot.c`: 62 checks; 70 on).
- [x] Inference every 1000 ticks (v3's `vm_tick_inference_engine`): `v4/system/inference.c`, v3's
`inference_engine.c` and Q48.16 arithmetic ported to the bit -- `v4/tests/test_inference.c` links
v3's own and compares them on POST's cases, 20000 random multiply and log pairs, and 400
trajectories (`make -C v4 inference`). The trajectory from the window's newest words' heat; the
window width and slope applied unless it exits early (as v3, the last inferred slope re-applied
on a full run even with L6 off). Loop #7: the heartrate 25% slower when stable, 20% faster when
not, within 1/4..4x; Hera's re-arms the kernel's heartbeat (superseded 2026-11-17: Loop #7 is the kernel heartbeat's own, on the Stadium's resident heat). Test first (72 checks; 80 on).
amd64 PASS, 163 s.
- [x] L8's update (v3's `vm_heartbeat_update_l8` with its table, `ssm_jacquard.c`, `compudynamics.c`): the
regime (window entropy >= 0.75, prediction CV >= 0.15, locality >= 1/2, as exact integer
comparisons), the card scored (ANOVA from inference; joint convergence when L5 or L6), the next by
UCB1, its gates applied. Every gate off and card 8 at the start, as v3's kernel boot. Applied
through `card_bits` -- v3 applies the card's number, one bit off (#196; ruled "v4 correct, v3 as
is"). `v4_mesh_l8_force` as v3's `ssm_l8_force_config`. Test first (67 checks; 75 on). amd64 PASS,
156 s.
- [x] The snapshot (v3's `heartbeat_publish_snapshot`): each cycle, the tick, window, slope and the
dictionary's heat as inference counted it (v3's `*_at_check`). One copy: v3 double-buffers for its
heartbeat thread, which v4 has no counterpart of. Test first (73 checks; 81 on).
### Task 3b: the loops on the 32 opcodes (ruling 7), behind `OPCODE_PHYSICS` -- measured
- [x] The engine: a hook at retire (`exec.h`, `on_retire`), as Part A's at `call`; test first (`test_exec.c`).
- [x] The mesh: 32 opcode records per node beside the words', an opcode window, opcode transitions (L4);
at every retire decay at the opcodes' own slope, heat, temperature, the window, the transition; in
the cycle all 32 cooled, the opcode window's service, inference on its heat (their own width and
slope). L8 scores from the words; its gates govern both. The window refactored to one type for both.
Test first, three builds: as built (77 checks), pipelining on (86), the revert `OPCODE_PHYSICS=0` (74).
- [x] `OPCODE_PHYSICS` in `Kconfig.physics` (default y) and the kernel's flags; `OPCODE_PHYSICS=0` checked
to reach the build (the revert's kernel has none of the opcode code).
- [x] Measured, back to back on amd64: 219 s on, 196 s off -- about +12% under TCG. Kept, as ruled; the
revert is the one switch.
### Task 4: the words that read and set them (`FREEZE-WORD`, the inference words, ...), and their POST
- [x] `capsules/v4/physics.4th` (blocks 8200-8201), a kernel capsule on Hera: v3's physics-freeze words
(`FREEZE-WORD` `UNFREEZE-WORD` `FROZEN?` `HEAT!` `HEAT@` `DECAY-RATE@`) and inference words
(`Q.VARIANCE` `INFER-DECAY-SLOPE` `INFER-WINDOW-WIDTH` `INFER-RUN` `INFER-WINDOW@` `INFER-DECAY@`
`INFER-VARIANCE@` `INFER-FIT@` `INFER-EARLY-EXIT@`), each `KERNEL-WORD` -- requests -21 to -34, which the
kernel serves on any node ("compudynamics is stadium/fabric"): a word named as v3's, by the newest
visible entry of CONTEXT then FORTH; the arithmetic `v4/system/inference.c`'s. A Q48.16 result is v4's
double (qmath.f18: low cell, then high). `HEAT!` starts the word's decay from now (a word never
touched would otherwise decay at once by every tick since its definition). Test first, v3's POST
cases with v4's address units (`test_mesh_boot.c`: 86 checks; 95 on; 83 without opcode physics).
- [x] **The tick, counted as v3's** (Captain Bob, 2026-11-14: "v3"; "we can always change it late"). v3 counts
one execution per token its interpreter takes and one per word its definitions run; its interpreter is
C. v4's is F18, and counting every call made ~700 counts to v3's one on a typed line (measured on 8
lines) but 1:1 in compiled code, so no single HEARTBEAT_CHECK_FREQUENCY could match (the knob ruling,
withdrawn on that measurement). Now: `INTERPRET` counts its tokens in `(TOKENS)` (one F18 store); the
kernel adds them in at each call, and counts a call -- and gives it physics -- only when it comes from
code defined at run time (dictionary space), or is INTERPRET's or EXECUTE's dispatch; the system's own
calls among its words, none. `3 4 + .` counts 4, as v3. Test first (90 checks; 99 on; 87 without
opcode physics). amd64 PASS: promotions=136 (fewer: the interpreter's own calls are no longer words run).
- [x] v3's StarForth module's physics words, in `physics.4th` (blocks 8202-8204): `ENTROPY@` `ENTROPY!`
(a word's heat by the address ' leaves; an error for what is not a word, as v3's) and
`RESET-ENTROPY` -- requests -35..-37; `TOP-WORDS` and `WORD-ENTROPY` in StarForth, printing what the
kernel gives them (-38..-40: how many have heat, the i-th hottest in v3's stable order, a word's
heat), in v3's form; `(-` in StarForth, skipping to the matching close paren with nesting as v3's
code does. Test first (101 checks; 110 on; 98 without opcode physics). Listing: 403 live words.
**For Task 8:** WORD-ENTROPY prints "Total words", and v4's dictionary is not v3's size -- that line
cannot print v3's number; TOP-WORDS' hottest words are v4's system words, not v3's.
### Task 5: the StarForth words (Phase 3's: `init.4th`, `starforth.4th`), and their POST
### Task 6: the ACL (Phase 4's: `ACL.4th`, the word card, the countdown), and its POST
### Task 7: the block cases (Phase 5's part)
### Task 8: `mkpost.py` beyond the Required set
- [x] `mkpost.py --all --out FILE`: the whole POST as a table, measured on Hera in the fabric -- 1020 cases,
849 pass, 171 fail (`POST-PARITY.md` section 5).
- [x] Ruled (2026-11-14): "v3's table, not v3's bugs" -- v3's self-disagreements held to its table, cases
naming v3's internals held to their form with v4's values; both listed.
- [x] The 171, by kind: address units (rules); what v4 lacks (`SP@` `SP!` `CASE` `SKIP` `SCAN` `SEARCH`
`COMPARE` `ORDER` `THRU` `FLUSH` ...); v3's self-disagreements and v3-internal output (by the ruling);
the rest case by case. Then the whole table becomes v4's POST and its floor.: v3's whole POST as v4's; the floor raised
**Done 2026-11-14** (`POST-PARITY.md` section 6): 990 cases, all pass on Hera; each case rewritten is
listed with its reason in `POST.md`; the floor (`v4/system/POST_FLOOR`) is 990.
### Task 9: acceptance -- the full protocol; ENGINE.md, V3-PARITY.md, FABRIC-3
Each task gets its steps when the one before it is accepted, test first, as Phase 2's did.
## Review focus
1. **Cost at `call`** (ruling 1): the boot's time under TCG before and after each per-call loop.
2. **Order:** the window and the transitions see calls in the order they happen, across nodes kept
apart.
3. **Recycled IDs:** a word ID FORGET gave back starts with no physics (decay, window, transitions).
4. **The 32-bit build:** the same results at 32-bit cells.
5. **What names v3's VMs** (Task 0): ruled, not guessed.