feat(v4, ci): the fault dump on a node's stop, and the EMERGENCY diagnostic build after a failed stage

Captain Bob's CI directive (2026-11-17) and the design he ruled point by point
(docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md).

- v4/system/dump.c, v4/include/v4/dump.h: what a node was doing when it stopped -- the fault's kind,
  address and count; P A B; the data stack, T first; the return stack, R first, each entry named by
  the word it falls in (the backtrace; ROM and below-every-word said so); the running line. Bounded
  dictionary walk; no C library. Ruled: every build, on a stop only (a handled fault keeps its line).
- kernel/src/v4/sk_v4.c: the dump before each of the three halts (not started; POST's blocks not
  let go; a node stopped on a fault) -- Hera and any stopped node. kernel/Makefile links dump.o.
- scripts/nightly.sh diagnose [<stage>]: after a failed stage, that configuration built again with
  EMERGENCY_CONSOLE_ENABLED=1 -- build only, nothing booted (ruled) -- its artifacts kept in
  logs/<TS>/diagnose/ only from a build that passed, with a README saying how to launch it with
  --log-level=debug. Always exits 0: evidence, never a result. The protocol runs it after a failed
  stage and still exits with that stage's code; acceptance.yml, nightly.yml and release.yml run it
  `if: failure()`, before the logs are kept. Any failed stage triggers it (ruled).
- capsules/BLOCK_MAP.md: as the build writes it -- the v4:physics.4th row, missing since 28ffed8f
  (ruled: commit the regenerated map).
- Test first: test_dump.c failed with no dump.c (logs/20261117-dump), then 15 checks at both widths
  and under ASan+UBSan. The diagnose stage: a real EMERGENCY amd64 build; a failing build exits 0 and
  keeps no artifacts (a defect found and fixed this way); the protocol path keeps the stage's exit.
- Also the write-up of the regression at 046ce0be, for ruling.

Not yet: acceptance on a clean tree (the next run); the kernel's stop wiring is compile-tested only
-- no boot reaches a stop on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
This commit is contained in:
Claude
2026-11-17 19:14:40 +00:00
parent 3376255930
commit 45b11aec25
13 changed files with 632 additions and 7 deletions
@@ -0,0 +1,81 @@
# The failure-triggered EMERGENCY diagnostic build -- design, for ruling
Written 2026-11-17 from Captain Bob's CI directive ("Emergency Console -- Separate Build
Configuration") and his order: "inspect the debug build first, then write the design".
**Nothing here is built.** The open points are put to Captain Bob one at a time, as he ruled the same
day.
## 1. What exists today (inspected 2026-11-17, read only)
| What | Where | What it does |
|---|---|---|
| `EMERGENCY_CONSOLE_ENABLED` (build parameter) | `Kconfig.heartbeat:70`, default n; `kernel/Makefile:429` | Gives a console at boot with no login, and makes an error recover instead of halting. On v4 the screen is live from the start and there is no headless wait (`kernel/src/v4/sk_v4.c:258, 348, 366`; `sk_v4_identity.c:463, 504`). **It adds no diagnostics.** |
| `--log-level=debug` (boot argument) | `kernel/src/boot/cmdline.c:135`; `kernel_args.h:25` | v3 only: a trace of every word (`ECW: w=... func=... 'NAME'`, `kernel/src/vm/vm_core.c:872`). This is very slow. **Nothing on the v4 path reads `log_level`.** |
| `SK_PARITY_DEBUG` (build parameter) | `kernel/Makefile:524` | v3's parity debugging. |
| The CPU's fault handlers (every build) | `kernel/src/arch/*/interrupts.c` | Print the vector or cause, then halt. amd64 prints RIP, CR2, RSP, RBP, the word on the stack, and the executing word on a #GP. aarch64 prints ELR_EL1; riscv64 prints sepc. |
| A v4 node that faults | `v4/include/v4/node.h:129-134`; `kernel/src/v4/sk_v4.c:407` | The node records its fault's kind, its address and a count. The kernel prints only "StarForth v4: the node stopped on a fault" and halts. **Nothing prints P, A, B, the stacks, the return stack (the node's backtrace) or the fault's kind and address.** |
| Acceptance and nightly on failure | `.gitea/workflows/acceptance.yml`, `nightly.yml`; `scripts/nightly.sh` | The failed step fails the job and the logs are kept. **No diagnostic build follows.** |
So the directive's diagnostics (stack states and contents, pointers and execution state, backtraces,
full logs) exist on v4 today only as fields in the node that nothing prints. On v3 they exist only as
the per-word trace, which a boot argument turns on.
## 2. The design
**A. The trigger.** A normal build or validation fails. Its step has failed, so the job's result is
already failed.
- A following step runs, `if: failure()` in the workflows, and the same call in
`scripts/nightly.sh protocol`: `scripts/nightly.sh diagnose <what failed>`.
- It builds the failed configuration again with `EMERGENCY_CONSOLE_ENABLED=1` and diagnostic logging
on (B).
- It keeps the result and the artifacts under `logs/<TS>/diagnose/`:
- the ELF with symbols;
- the ISO or the image;
- the `.config`;
- the capsule map;
- the build log;
- its own result.
- Nothing it does can change the job's result: the step only gathers evidence, and the failed step
stays the reported failure.
**B. What the diagnostic build carries** (it needs building, and a ruling):
- On a node's fault or stop, the kernel prints:
- the fault's kind and address, and the fault count;
- P, A, B, T and S;
- the whole data stack;
- the whole return stack, with each address named by the dictionary word it falls in. That list is
the node's backtrace.
- the line that was running.
- Diagnostic logging on the v4 path: what the node is doing, word by word, as v3's per-word trace is.
It is allowed to be slow.
**C. Building is not launching.** The diagnostic step builds and keeps the artifacts. It does not boot
an emergency console. Launching one is a separate act: a person's, or a stage made for it.
**D. Failure categories.** Each is its own named step and its own line in PROTOCOL.md:
1. source: a `.f18` that does not assemble or a `.4th` that does not compile (POST);
2. the kernel's build and unit tests;
3. ASan and UBSan;
4. QEMU, per architecture;
5. the emergency diagnostic build.
## 3. Open points, to be put one at a time
1. ~~Diagnostic logging is a boot argument today (`--log-level`). Should the diagnostic build turn it
on as a build parameter instead?~~ **Withdrawn 2026-11-17. It was not a discrepancy.**
- The directive forbids only a boot argument standing in for the EMERGENCY *build* (`--sos`).
- `--log-level` stands in for nothing: it is how logging has always been chosen.
- I made a problem out of established practice by reading the directive too widely.
- The diagnostic build is EMERGENCY at build time, with `--log-level=debug` as it has always been.
2. ~~Should the diagnostic step build only (C), or also boot that build once?~~ **Ruled 2026-11-17
(Captain Bob): build only.** The step builds the diagnostic configuration and keeps its artifacts.
Launching it is a person's act.
3. ~~Should B be printed in every build, or only in the EMERGENCY build?~~ **Ruled 2026-11-17 (Captain
Bob): every build.** The fault dump prints in any build on a node's fault, as the CPU's fault
handlers already do. The slow word-by-word log stays behind `--log-level=debug`.
4. ~~Which stage does the trigger cover?~~ **Ruled 2026-11-17 (Captain Bob): any failed stage.** The
diagnostic build of that configuration follows whatever fails.
All four open points are settled. The design waits for Captain Bob's word to build it.
@@ -0,0 +1,159 @@
# The regression at 046ce0be -- write-up, for ruling
Written 2026-11-17, after reading the record again in full (notes: `.claude/memory/reading-2026-11-17.md`).
**Nothing here is built, reverted or reset.** Every decision below waits for Captain Bob.
## 1. What and when
v4's boot became 1.5 to 2.4 times slower at **046ce0be** ("the mesh boot to spec -- Hera born empty in
the kernel's fabric; ... every node stepping together", 2026-11-13). The last good commit is
**f10d2bc8**, the commit before it.
Both commits ran the same POST, 539 cases, and both protocols passed. Each stage's time is taken from
the start of its QEMU log to the start of the next stage's log. A stage is a `clean` build, the boot and
the session.
| Stage | f10d2bc8 (`logs/20261113-142432`) | 046ce0be (`logs/20261113-170011`) | Ratio |
|---|---|---|---|
| amd64 | 14:25:04 to 14:26:51: **107 s** | 17:00:39 to 17:03:43: **184 s** | 1.7 |
| aarch64 | 14:26:51 to 14:30:20: **209 s** | 17:03:43 to 17:12:02: **499 s** | 2.4 |
| riscv64 | 14:30:20 to 14:32:05: **105 s** | 17:12:02 to 17:14:45: **163 s** | 1.6 |
**What these numbers do not prove on their own:**
- There is one run of each commit.
- The time includes the build.
- This host varies: the same commit took 146 s, then 175 s an hour later (Phase 7 plan, Task 2b).
Section 5 proposes the proof.
Two more measurements were taken in the previous session but are **not committed evidence**:
- A gprof profile of the host mesh boot, with `v4_fabric_step` at about 40% of the time spent in its own code (`v4/gmon.out`, untracked).
- A loop of a million, on hosted v3 and on Hera in the mesh:
| Run | Time |
|---|---|
| hosted v3 | about 172 ms |
| Hera in the mesh | about 1362 ms |
## 2. The mechanism, in the code
- **f10d2bc8.** One node runs the boot. For each instruction word, `boot_exec`
(`git show f10d2bc8:v4/system/boot.c`, lines 200-209) does three things:
- sets the node's have-mask;
- calls `v4_exec_step_word` once;
- checks one flag.
- **046ce0be onward.** Every instruction word Hera executes is one `v4_fabric_step` (`v4/src/fabric.c`,
lines 423-490), and the step walks all 16 places of the kernel's fabric (`SK_V4_PLACES`,
`kernel/src/v4/sk_v4.c:63`) in ten passes:
1. the have-mask, with an 8-port loop for each awake node;
2. `look_about` for each node that is not blocked, with another 8-port loop;
3. one `v4_exec_step_word` for each awake node, blocked ones included;
4. clearing `have_fetched`;
5. and 6. the two operation passes;
7. handing over writes;
8. handing over reads, which asks the console device on every step that Hera reads;
9. the gone-error pass.
After the step, `takes()` polls the console's queue (`v4/system/mesh.c`, `v4_mesh_step`, `v4_mesh_tell`).
So the cost of a step grows with how many places there are, not with how many nodes run. In the boot one
node runs, so nearly all of each step is overhead.
## 3. What the record says. I was wrong about this before.
**The rotation is your ruled design. It is not a scheduler.**
- ENGINE.md §3d, ruled "A" on 2026-10-05: "Every node that is not blocked runs; on one processor that is
each in rotation, an instruction word at a time, which is what 'all at once' is there and is not a
policy."
- MESH.md §2, ruling 7, and §4.3: "There is no choice of whose turn it is."
- `fabric.h` says the same.
Calling the lock-step itself a scheduler was my mistake. The scheduler I actually offered was the
withdrawn ruling "Hera by compudynamics now" (Phase 1 plan, 2026-11-13). The rotation stays as ruled.
**What the change broke is the cost.**
- **"An idle node does not spin"** (ENGINE §3d; MESH ruling 7). The node does not, but the fabric now
spins over it at every instruction word. MESH 9a.6 tested only the node: "its anti-clock ... does not
move over thousands of steps". It never tested what the step costs.
- **FABRIC-3.5 §XXII and §XXXV.2** rule out per-tick walks over every VM as a scaling wall: "we really will
be giving birth to potentially hundreds of VMs".
- **ENGINE §6** still lists as open "the node's safe moment for message delivery and switching". 046ce0be
did not settle it. It made every instruction word that moment, for every place.
**The slowdown was seen at each step and explained away.** It was never measured against f10d2bc8.
- **Phase 1 plan:** "The amd64 boot: 160 s (about 90 s before -- POST and the lines now go as messages)".
- **Phase 2:** "142 s (Phase 1: 153 s)", compared with the boot that was already slow.
- **Phase 7, Task 2b:** "The host slowed."
- **POST-PARITY §6:** "about 3 m 15 s at 539" was taken as the baseline.
Your standing order, "Find the TCG cost first" (POST-PARITY §6), is what this answers.
## 4. What it costs to go back to f10d2bc8
There are 33 commits after f10d2bc8:
| Commits | Where they are |
|---|---|
| 046ce0be | **pushed**, in PR 164 |
| Phase 2, e8e7ec8b | **pushed**, in PR 164 |
| The remote's head, ee28530b | **pushed** |
| 28 more: Phase 7 (the loops, the opcode physics, the physics words, the whole POST of 990 cases) and my five | **local only** |
So starting over from f10d2bc8 changes pushed history. That needs your explicit word. The choices are:
- reset the branch, with a force push;
- add revert commits;
- keep the history and redesign the step forward from HEAD.
## 5. Proposed proof, before anything is changed
**Intent:** show that the gap is the commit, not the host.
**How:**
- Use `git archive` to build f10d2bc8 and 046ce0be into the scratchpad. Nothing on the branch changes.
- Boot each one, back to back, on amd64, then on aarch64. That is four boots.
- Each boot runs in the foreground, one QEMU at a time, under `timeout 570`.
**The observation that proves it**, named before the run: 046ce0be's boot time is above f10d2bc8's by more
than the host's variance on both ISAs (about 1.7 times on amd64 and 2.4 times on aarch64), and the POST
line is the same.
**The logs** are committed and labelled.
## 6. What escaped, and how (a CAPA candidate, to rule)
- **What got through:** a 1.6 to 2.4 times regression passed the protocol.
- **Why:** no stage compares the boot's time with the last good commit, and acceptance has no criterion for cost.
- **Proposed control (not built):** the protocol records each stage's time and compares it with the last
passing PROTOCOL.md. The threshold is yours to set.
## 7. The uncommitted work you rejected, still untouched
**Untracked or modified files:**
- `v4/system/post_all_cases.c`, `post_v3_cases.c`, `post_cases.c`, `POST_FLOOR`
- `v4/tests/test_mesh_boot.c`, `v4/Makefile`
- `v4/include/v4/post.h`, `mesh.h`, `v4/system/mesh.c`
- `v4/tools/mkpost.py`, `post_all_rules.py`
- `docs/v4.0.0/POST.md`, `docs/v4.0.0/POST-PARITY.md`
- `logs/20261117-135508/aarch64/README.md`
**Logs not yet committed:**
- `logs/20261114-064855`, `-065851`, `-081210`, `-090737`, and `logs/20261117-135508`
- two DoE CSVs
**One false statement:** the addition to POST-PARITY §6 says "Ruled ... Whole POST in tests; boot runs
less". You withdrew that. It must not be committed as a ruling.
## 8. For ruling
1. **The write-up:** commit it locally as it stands, or correct it first?
2. **The proof of §5:** run it, yes or no?
3. **How far back:** start over from f10d2bc8, and if so by reset or by revert? Or redesign the step from HEAD?
4. **The step's design, whichever way 3 goes.** Keep the rotation as ruled, but make the step's work grow
with what runs and what moves, not with how many places there are. v3's own answers are on record:
- FABRIC-3 §XXVIII: a flag set, not a walk;
- FABRIC-0 GAP-A1: a virtual tick;
- FABRIC-3.5 §XLIII: the target takes its own messages on its own turn.
The design is to be written first, in the record (ENGINE §6, "the node's safe moment"), for your ruling.
5. **The rejected work (§7):** discard it, or keep any part of it? And which logs get committed as labelled evidence?
6. **The CAPA (§6):** record it now, or later?