Files
LithosAnanke/docs/v4.0.0/plans/2026-11-17-the-regression-at-046ce0be.md
T
Claude 45b11aec25 feat(v4, ci): the fault dump on a node's stop, and the EMERGENCY diagnostic build after a failed stage
Captain Bob's CI directive (2026-11-17) and the design he ruled point by point
(docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md).

- v4/system/dump.c, v4/include/v4/dump.h: what a node was doing when it stopped -- the fault's kind,
  address and count; P A B; the data stack, T first; the return stack, R first, each entry named by
  the word it falls in (the backtrace; ROM and below-every-word said so); the running line. Bounded
  dictionary walk; no C library. Ruled: every build, on a stop only (a handled fault keeps its line).
- kernel/src/v4/sk_v4.c: the dump before each of the three halts (not started; POST's blocks not
  let go; a node stopped on a fault) -- Hera and any stopped node. kernel/Makefile links dump.o.
- scripts/nightly.sh diagnose [<stage>]: after a failed stage, that configuration built again with
  EMERGENCY_CONSOLE_ENABLED=1 -- build only, nothing booted (ruled) -- its artifacts kept in
  logs/<TS>/diagnose/ only from a build that passed, with a README saying how to launch it with
  --log-level=debug. Always exits 0: evidence, never a result. The protocol runs it after a failed
  stage and still exits with that stage's code; acceptance.yml, nightly.yml and release.yml run it
  `if: failure()`, before the logs are kept. Any failed stage triggers it (ruled).
- capsules/BLOCK_MAP.md: as the build writes it -- the v4:physics.4th row, missing since 28ffed8f
  (ruled: commit the regenerated map).
- Test first: test_dump.c failed with no dump.c (logs/20261117-dump), then 15 checks at both widths
  and under ASan+UBSan. The diagnose stage: a real EMERGENCY amd64 build; a failing build exits 0 and
  keeps no artifacts (a defect found and fixed this way); the protocol path keeps the stage's exit.
- Also the write-up of the regression at 046ce0be, for ruling.

Not yet: acceptance on a clean tree (the next run); the kernel's stop wiring is compile-tested only
-- no boot reaches a stop on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
2026-11-17 19:14:40 +00:00

160 lines
7.8 KiB
Markdown

# The regression at 046ce0be -- write-up, for ruling
Written 2026-11-17, after reading the record again in full (notes: `.claude/memory/reading-2026-11-17.md`).
**Nothing here is built, reverted or reset.** Every decision below waits for Captain Bob.
## 1. What and when
v4's boot became 1.5 to 2.4 times slower at **046ce0be** ("the mesh boot to spec -- Hera born empty in
the kernel's fabric; ... every node stepping together", 2026-11-13). The last good commit is
**f10d2bc8**, the commit before it.
Both commits ran the same POST, 539 cases, and both protocols passed. Each stage's time is taken from
the start of its QEMU log to the start of the next stage's log. A stage is a `clean` build, the boot and
the session.
| Stage | f10d2bc8 (`logs/20261113-142432`) | 046ce0be (`logs/20261113-170011`) | Ratio |
|---|---|---|---|
| amd64 | 14:25:04 to 14:26:51: **107 s** | 17:00:39 to 17:03:43: **184 s** | 1.7 |
| aarch64 | 14:26:51 to 14:30:20: **209 s** | 17:03:43 to 17:12:02: **499 s** | 2.4 |
| riscv64 | 14:30:20 to 14:32:05: **105 s** | 17:12:02 to 17:14:45: **163 s** | 1.6 |
**What these numbers do not prove on their own:**
- There is one run of each commit.
- The time includes the build.
- This host varies: the same commit took 146 s, then 175 s an hour later (Phase 7 plan, Task 2b).
Section 5 proposes the proof.
Two more measurements were taken in the previous session but are **not committed evidence**:
- A gprof profile of the host mesh boot, with `v4_fabric_step` at about 40% of the time spent in its own code (`v4/gmon.out`, untracked).
- A loop of a million, on hosted v3 and on Hera in the mesh:
| Run | Time |
|---|---|
| hosted v3 | about 172 ms |
| Hera in the mesh | about 1362 ms |
## 2. The mechanism, in the code
- **f10d2bc8.** One node runs the boot. For each instruction word, `boot_exec`
(`git show f10d2bc8:v4/system/boot.c`, lines 200-209) does three things:
- sets the node's have-mask;
- calls `v4_exec_step_word` once;
- checks one flag.
- **046ce0be onward.** Every instruction word Hera executes is one `v4_fabric_step` (`v4/src/fabric.c`,
lines 423-490), and the step walks all 16 places of the kernel's fabric (`SK_V4_PLACES`,
`kernel/src/v4/sk_v4.c:63`) in ten passes:
1. the have-mask, with an 8-port loop for each awake node;
2. `look_about` for each node that is not blocked, with another 8-port loop;
3. one `v4_exec_step_word` for each awake node, blocked ones included;
4. clearing `have_fetched`;
5. and 6. the two operation passes;
7. handing over writes;
8. handing over reads, which asks the console device on every step that Hera reads;
9. the gone-error pass.
After the step, `takes()` polls the console's queue (`v4/system/mesh.c`, `v4_mesh_step`, `v4_mesh_tell`).
So the cost of a step grows with how many places there are, not with how many nodes run. In the boot one
node runs, so nearly all of each step is overhead.
## 3. What the record says. I was wrong about this before.
**The rotation is your ruled design. It is not a scheduler.**
- ENGINE.md §3d, ruled "A" on 2026-10-05: "Every node that is not blocked runs; on one processor that is
each in rotation, an instruction word at a time, which is what 'all at once' is there and is not a
policy."
- MESH.md §2, ruling 7, and §4.3: "There is no choice of whose turn it is."
- `fabric.h` says the same.
Calling the lock-step itself a scheduler was my mistake. The scheduler I actually offered was the
withdrawn ruling "Hera by compudynamics now" (Phase 1 plan, 2026-11-13). The rotation stays as ruled.
**What the change broke is the cost.**
- **"An idle node does not spin"** (ENGINE §3d; MESH ruling 7). The node does not, but the fabric now
spins over it at every instruction word. MESH 9a.6 tested only the node: "its anti-clock ... does not
move over thousands of steps". It never tested what the step costs.
- **FABRIC-3.5 §XXII and §XXXV.2** rule out per-tick walks over every VM as a scaling wall: "we really will
be giving birth to potentially hundreds of VMs".
- **ENGINE §6** still lists as open "the node's safe moment for message delivery and switching". 046ce0be
did not settle it. It made every instruction word that moment, for every place.
**The slowdown was seen at each step and explained away.** It was never measured against f10d2bc8.
- **Phase 1 plan:** "The amd64 boot: 160 s (about 90 s before -- POST and the lines now go as messages)".
- **Phase 2:** "142 s (Phase 1: 153 s)", compared with the boot that was already slow.
- **Phase 7, Task 2b:** "The host slowed."
- **POST-PARITY §6:** "about 3 m 15 s at 539" was taken as the baseline.
Your standing order, "Find the TCG cost first" (POST-PARITY §6), is what this answers.
## 4. What it costs to go back to f10d2bc8
There are 33 commits after f10d2bc8:
| Commits | Where they are |
|---|---|
| 046ce0be | **pushed**, in PR 164 |
| Phase 2, e8e7ec8b | **pushed**, in PR 164 |
| The remote's head, ee28530b | **pushed** |
| 28 more: Phase 7 (the loops, the opcode physics, the physics words, the whole POST of 990 cases) and my five | **local only** |
So starting over from f10d2bc8 changes pushed history. That needs your explicit word. The choices are:
- reset the branch, with a force push;
- add revert commits;
- keep the history and redesign the step forward from HEAD.
## 5. Proposed proof, before anything is changed
**Intent:** show that the gap is the commit, not the host.
**How:**
- Use `git archive` to build f10d2bc8 and 046ce0be into the scratchpad. Nothing on the branch changes.
- Boot each one, back to back, on amd64, then on aarch64. That is four boots.
- Each boot runs in the foreground, one QEMU at a time, under `timeout 570`.
**The observation that proves it**, named before the run: 046ce0be's boot time is above f10d2bc8's by more
than the host's variance on both ISAs (about 1.7 times on amd64 and 2.4 times on aarch64), and the POST
line is the same.
**The logs** are committed and labelled.
## 6. What escaped, and how (a CAPA candidate, to rule)
- **What got through:** a 1.6 to 2.4 times regression passed the protocol.
- **Why:** no stage compares the boot's time with the last good commit, and acceptance has no criterion for cost.
- **Proposed control (not built):** the protocol records each stage's time and compares it with the last
passing PROTOCOL.md. The threshold is yours to set.
## 7. The uncommitted work you rejected, still untouched
**Untracked or modified files:**
- `v4/system/post_all_cases.c`, `post_v3_cases.c`, `post_cases.c`, `POST_FLOOR`
- `v4/tests/test_mesh_boot.c`, `v4/Makefile`
- `v4/include/v4/post.h`, `mesh.h`, `v4/system/mesh.c`
- `v4/tools/mkpost.py`, `post_all_rules.py`
- `docs/v4.0.0/POST.md`, `docs/v4.0.0/POST-PARITY.md`
- `logs/20261117-135508/aarch64/README.md`
**Logs not yet committed:**
- `logs/20261114-064855`, `-065851`, `-081210`, `-090737`, and `logs/20261117-135508`
- two DoE CSVs
**One false statement:** the addition to POST-PARITY §6 says "Ruled ... Whole POST in tests; boot runs
less". You withdrew that. It must not be committed as a ruling.
## 8. For ruling
1. **The write-up:** commit it locally as it stands, or correct it first?
2. **The proof of §5:** run it, yes or no?
3. **How far back:** start over from f10d2bc8, and if so by reset or by revert? Or redesign the step from HEAD?
4. **The step's design, whichever way 3 goes.** Keep the rotation as ruled, but make the step's work grow
with what runs and what moves, not with how many places there are. v3's own answers are on record:
- FABRIC-3 §XXVIII: a flag set, not a walk;
- FABRIC-0 GAP-A1: a virtual tick;
- FABRIC-3.5 §XLIII: the target takes its own messages on its own turn.
The design is to be written first, in the record (ENGINE §6, "the node's safe moment"), for your ruling.
5. **The rejected work (§7):** discard it, or keep any part of it? And which logs get committed as labelled evidence?
6. **The CAPA (§6):** record it now, or later?