feat(v4, ci): the fault dump on a node's stop, and the EMERGENCY diagnostic build after a failed stage
Captain Bob's CI directive (2026-11-17) and the design he ruled point by point (docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md). - v4/system/dump.c, v4/include/v4/dump.h: what a node was doing when it stopped -- the fault's kind, address and count; P A B; the data stack, T first; the return stack, R first, each entry named by the word it falls in (the backtrace; ROM and below-every-word said so); the running line. Bounded dictionary walk; no C library. Ruled: every build, on a stop only (a handled fault keeps its line). - kernel/src/v4/sk_v4.c: the dump before each of the three halts (not started; POST's blocks not let go; a node stopped on a fault) -- Hera and any stopped node. kernel/Makefile links dump.o. - scripts/nightly.sh diagnose [<stage>]: after a failed stage, that configuration built again with EMERGENCY_CONSOLE_ENABLED=1 -- build only, nothing booted (ruled) -- its artifacts kept in logs/<TS>/diagnose/ only from a build that passed, with a README saying how to launch it with --log-level=debug. Always exits 0: evidence, never a result. The protocol runs it after a failed stage and still exits with that stage's code; acceptance.yml, nightly.yml and release.yml run it `if: failure()`, before the logs are kept. Any failed stage triggers it (ruled). - capsules/BLOCK_MAP.md: as the build writes it -- the v4:physics.4th row, missing since28ffed8f(ruled: commit the regenerated map). - Test first: test_dump.c failed with no dump.c (logs/20261117-dump), then 15 checks at both widths and under ASan+UBSan. The diagnose stage: a real EMERGENCY amd64 build; a failing build exits 0 and keeps no artifacts (a defect found and fixed this way); the protocol path keeps the stage's exit. - Also the write-up of the regression at046ce0be, for ruling. Not yet: acceptance on a clean tree (the next run); the kernel's stop wiring is compile-tested only -- no boot reaches a stop on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
This commit is contained in:
@@ -0,0 +1,81 @@
|
||||
# The failure-triggered EMERGENCY diagnostic build -- design, for ruling
|
||||
|
||||
Written 2026-11-17 from Captain Bob's CI directive ("Emergency Console -- Separate Build
|
||||
Configuration") and his order: "inspect the debug build first, then write the design".
|
||||
|
||||
**Nothing here is built.** The open points are put to Captain Bob one at a time, as he ruled the same
|
||||
day.
|
||||
|
||||
## 1. What exists today (inspected 2026-11-17, read only)
|
||||
|
||||
| What | Where | What it does |
|
||||
|---|---|---|
|
||||
| `EMERGENCY_CONSOLE_ENABLED` (build parameter) | `Kconfig.heartbeat:70`, default n; `kernel/Makefile:429` | Gives a console at boot with no login, and makes an error recover instead of halting. On v4 the screen is live from the start and there is no headless wait (`kernel/src/v4/sk_v4.c:258, 348, 366`; `sk_v4_identity.c:463, 504`). **It adds no diagnostics.** |
|
||||
| `--log-level=debug` (boot argument) | `kernel/src/boot/cmdline.c:135`; `kernel_args.h:25` | v3 only: a trace of every word (`ECW: w=... func=... 'NAME'`, `kernel/src/vm/vm_core.c:872`). This is very slow. **Nothing on the v4 path reads `log_level`.** |
|
||||
| `SK_PARITY_DEBUG` (build parameter) | `kernel/Makefile:524` | v3's parity debugging. |
|
||||
| The CPU's fault handlers (every build) | `kernel/src/arch/*/interrupts.c` | Print the vector or cause, then halt. amd64 prints RIP, CR2, RSP, RBP, the word on the stack, and the executing word on a #GP. aarch64 prints ELR_EL1; riscv64 prints sepc. |
|
||||
| A v4 node that faults | `v4/include/v4/node.h:129-134`; `kernel/src/v4/sk_v4.c:407` | The node records its fault's kind, its address and a count. The kernel prints only "StarForth v4: the node stopped on a fault" and halts. **Nothing prints P, A, B, the stacks, the return stack (the node's backtrace) or the fault's kind and address.** |
|
||||
| Acceptance and nightly on failure | `.gitea/workflows/acceptance.yml`, `nightly.yml`; `scripts/nightly.sh` | The failed step fails the job and the logs are kept. **No diagnostic build follows.** |
|
||||
|
||||
So the directive's diagnostics (stack states and contents, pointers and execution state, backtraces,
|
||||
full logs) exist on v4 today only as fields in the node that nothing prints. On v3 they exist only as
|
||||
the per-word trace, which a boot argument turns on.
|
||||
|
||||
## 2. The design
|
||||
|
||||
**A. The trigger.** A normal build or validation fails. Its step has failed, so the job's result is
|
||||
already failed.
|
||||
|
||||
- A following step runs, `if: failure()` in the workflows, and the same call in
|
||||
`scripts/nightly.sh protocol`: `scripts/nightly.sh diagnose <what failed>`.
|
||||
- It builds the failed configuration again with `EMERGENCY_CONSOLE_ENABLED=1` and diagnostic logging
|
||||
on (B).
|
||||
- It keeps the result and the artifacts under `logs/<TS>/diagnose/`:
|
||||
- the ELF with symbols;
|
||||
- the ISO or the image;
|
||||
- the `.config`;
|
||||
- the capsule map;
|
||||
- the build log;
|
||||
- its own result.
|
||||
- Nothing it does can change the job's result: the step only gathers evidence, and the failed step
|
||||
stays the reported failure.
|
||||
|
||||
**B. What the diagnostic build carries** (it needs building, and a ruling):
|
||||
- On a node's fault or stop, the kernel prints:
|
||||
- the fault's kind and address, and the fault count;
|
||||
- P, A, B, T and S;
|
||||
- the whole data stack;
|
||||
- the whole return stack, with each address named by the dictionary word it falls in. That list is
|
||||
the node's backtrace.
|
||||
- the line that was running.
|
||||
- Diagnostic logging on the v4 path: what the node is doing, word by word, as v3's per-word trace is.
|
||||
It is allowed to be slow.
|
||||
|
||||
**C. Building is not launching.** The diagnostic step builds and keeps the artifacts. It does not boot
|
||||
an emergency console. Launching one is a separate act: a person's, or a stage made for it.
|
||||
|
||||
**D. Failure categories.** Each is its own named step and its own line in PROTOCOL.md:
|
||||
1. source: a `.f18` that does not assemble or a `.4th` that does not compile (POST);
|
||||
2. the kernel's build and unit tests;
|
||||
3. ASan and UBSan;
|
||||
4. QEMU, per architecture;
|
||||
5. the emergency diagnostic build.
|
||||
|
||||
## 3. Open points, to be put one at a time
|
||||
|
||||
1. ~~Diagnostic logging is a boot argument today (`--log-level`). Should the diagnostic build turn it
|
||||
on as a build parameter instead?~~ **Withdrawn 2026-11-17. It was not a discrepancy.**
|
||||
- The directive forbids only a boot argument standing in for the EMERGENCY *build* (`--sos`).
|
||||
- `--log-level` stands in for nothing: it is how logging has always been chosen.
|
||||
- I made a problem out of established practice by reading the directive too widely.
|
||||
- The diagnostic build is EMERGENCY at build time, with `--log-level=debug` as it has always been.
|
||||
2. ~~Should the diagnostic step build only (C), or also boot that build once?~~ **Ruled 2026-11-17
|
||||
(Captain Bob): build only.** The step builds the diagnostic configuration and keeps its artifacts.
|
||||
Launching it is a person's act.
|
||||
3. ~~Should B be printed in every build, or only in the EMERGENCY build?~~ **Ruled 2026-11-17 (Captain
|
||||
Bob): every build.** The fault dump prints in any build on a node's fault, as the CPU's fault
|
||||
handlers already do. The slow word-by-word log stays behind `--log-level=debug`.
|
||||
4. ~~Which stage does the trigger cover?~~ **Ruled 2026-11-17 (Captain Bob): any failed stage.** The
|
||||
diagnostic build of that configuration follows whatever fails.
|
||||
|
||||
All four open points are settled. The design waits for Captain Bob's word to build it.
|
||||
@@ -0,0 +1,159 @@
|
||||
# The regression at 046ce0be -- write-up, for ruling
|
||||
|
||||
Written 2026-11-17, after reading the record again in full (notes: `.claude/memory/reading-2026-11-17.md`).
|
||||
**Nothing here is built, reverted or reset.** Every decision below waits for Captain Bob.
|
||||
|
||||
## 1. What and when
|
||||
|
||||
v4's boot became 1.5 to 2.4 times slower at **046ce0be** ("the mesh boot to spec -- Hera born empty in
|
||||
the kernel's fabric; ... every node stepping together", 2026-11-13). The last good commit is
|
||||
**f10d2bc8**, the commit before it.
|
||||
|
||||
Both commits ran the same POST, 539 cases, and both protocols passed. Each stage's time is taken from
|
||||
the start of its QEMU log to the start of the next stage's log. A stage is a `clean` build, the boot and
|
||||
the session.
|
||||
|
||||
| Stage | f10d2bc8 (`logs/20261113-142432`) | 046ce0be (`logs/20261113-170011`) | Ratio |
|
||||
|---|---|---|---|
|
||||
| amd64 | 14:25:04 to 14:26:51: **107 s** | 17:00:39 to 17:03:43: **184 s** | 1.7 |
|
||||
| aarch64 | 14:26:51 to 14:30:20: **209 s** | 17:03:43 to 17:12:02: **499 s** | 2.4 |
|
||||
| riscv64 | 14:30:20 to 14:32:05: **105 s** | 17:12:02 to 17:14:45: **163 s** | 1.6 |
|
||||
|
||||
**What these numbers do not prove on their own:**
|
||||
- There is one run of each commit.
|
||||
- The time includes the build.
|
||||
- This host varies: the same commit took 146 s, then 175 s an hour later (Phase 7 plan, Task 2b).
|
||||
|
||||
Section 5 proposes the proof.
|
||||
|
||||
Two more measurements were taken in the previous session but are **not committed evidence**:
|
||||
- A gprof profile of the host mesh boot, with `v4_fabric_step` at about 40% of the time spent in its own code (`v4/gmon.out`, untracked).
|
||||
- A loop of a million, on hosted v3 and on Hera in the mesh:
|
||||
|
||||
| Run | Time |
|
||||
|---|---|
|
||||
| hosted v3 | about 172 ms |
|
||||
| Hera in the mesh | about 1362 ms |
|
||||
|
||||
## 2. The mechanism, in the code
|
||||
|
||||
- **f10d2bc8.** One node runs the boot. For each instruction word, `boot_exec`
|
||||
(`git show f10d2bc8:v4/system/boot.c`, lines 200-209) does three things:
|
||||
- sets the node's have-mask;
|
||||
- calls `v4_exec_step_word` once;
|
||||
- checks one flag.
|
||||
- **046ce0be onward.** Every instruction word Hera executes is one `v4_fabric_step` (`v4/src/fabric.c`,
|
||||
lines 423-490), and the step walks all 16 places of the kernel's fabric (`SK_V4_PLACES`,
|
||||
`kernel/src/v4/sk_v4.c:63`) in ten passes:
|
||||
1. the have-mask, with an 8-port loop for each awake node;
|
||||
2. `look_about` for each node that is not blocked, with another 8-port loop;
|
||||
3. one `v4_exec_step_word` for each awake node, blocked ones included;
|
||||
4. clearing `have_fetched`;
|
||||
5. and 6. the two operation passes;
|
||||
7. handing over writes;
|
||||
8. handing over reads, which asks the console device on every step that Hera reads;
|
||||
9. the gone-error pass.
|
||||
|
||||
After the step, `takes()` polls the console's queue (`v4/system/mesh.c`, `v4_mesh_step`, `v4_mesh_tell`).
|
||||
|
||||
So the cost of a step grows with how many places there are, not with how many nodes run. In the boot one
|
||||
node runs, so nearly all of each step is overhead.
|
||||
|
||||
## 3. What the record says. I was wrong about this before.
|
||||
|
||||
**The rotation is your ruled design. It is not a scheduler.**
|
||||
- ENGINE.md §3d, ruled "A" on 2026-10-05: "Every node that is not blocked runs; on one processor that is
|
||||
each in rotation, an instruction word at a time, which is what 'all at once' is there and is not a
|
||||
policy."
|
||||
- MESH.md §2, ruling 7, and §4.3: "There is no choice of whose turn it is."
|
||||
- `fabric.h` says the same.
|
||||
|
||||
Calling the lock-step itself a scheduler was my mistake. The scheduler I actually offered was the
|
||||
withdrawn ruling "Hera by compudynamics now" (Phase 1 plan, 2026-11-13). The rotation stays as ruled.
|
||||
|
||||
**What the change broke is the cost.**
|
||||
- **"An idle node does not spin"** (ENGINE §3d; MESH ruling 7). The node does not, but the fabric now
|
||||
spins over it at every instruction word. MESH 9a.6 tested only the node: "its anti-clock ... does not
|
||||
move over thousands of steps". It never tested what the step costs.
|
||||
- **FABRIC-3.5 §XXII and §XXXV.2** rule out per-tick walks over every VM as a scaling wall: "we really will
|
||||
be giving birth to potentially hundreds of VMs".
|
||||
- **ENGINE §6** still lists as open "the node's safe moment for message delivery and switching". 046ce0be
|
||||
did not settle it. It made every instruction word that moment, for every place.
|
||||
|
||||
**The slowdown was seen at each step and explained away.** It was never measured against f10d2bc8.
|
||||
- **Phase 1 plan:** "The amd64 boot: 160 s (about 90 s before -- POST and the lines now go as messages)".
|
||||
- **Phase 2:** "142 s (Phase 1: 153 s)", compared with the boot that was already slow.
|
||||
- **Phase 7, Task 2b:** "The host slowed."
|
||||
- **POST-PARITY §6:** "about 3 m 15 s at 539" was taken as the baseline.
|
||||
|
||||
Your standing order, "Find the TCG cost first" (POST-PARITY §6), is what this answers.
|
||||
|
||||
## 4. What it costs to go back to f10d2bc8
|
||||
|
||||
There are 33 commits after f10d2bc8:
|
||||
|
||||
| Commits | Where they are |
|
||||
|---|---|
|
||||
| 046ce0be | **pushed**, in PR 164 |
|
||||
| Phase 2, e8e7ec8b | **pushed**, in PR 164 |
|
||||
| The remote's head, ee28530b | **pushed** |
|
||||
| 28 more: Phase 7 (the loops, the opcode physics, the physics words, the whole POST of 990 cases) and my five | **local only** |
|
||||
|
||||
So starting over from f10d2bc8 changes pushed history. That needs your explicit word. The choices are:
|
||||
- reset the branch, with a force push;
|
||||
- add revert commits;
|
||||
- keep the history and redesign the step forward from HEAD.
|
||||
|
||||
## 5. Proposed proof, before anything is changed
|
||||
|
||||
**Intent:** show that the gap is the commit, not the host.
|
||||
|
||||
**How:**
|
||||
- Use `git archive` to build f10d2bc8 and 046ce0be into the scratchpad. Nothing on the branch changes.
|
||||
- Boot each one, back to back, on amd64, then on aarch64. That is four boots.
|
||||
- Each boot runs in the foreground, one QEMU at a time, under `timeout 570`.
|
||||
|
||||
**The observation that proves it**, named before the run: 046ce0be's boot time is above f10d2bc8's by more
|
||||
than the host's variance on both ISAs (about 1.7 times on amd64 and 2.4 times on aarch64), and the POST
|
||||
line is the same.
|
||||
|
||||
**The logs** are committed and labelled.
|
||||
|
||||
## 6. What escaped, and how (a CAPA candidate, to rule)
|
||||
|
||||
- **What got through:** a 1.6 to 2.4 times regression passed the protocol.
|
||||
- **Why:** no stage compares the boot's time with the last good commit, and acceptance has no criterion for cost.
|
||||
- **Proposed control (not built):** the protocol records each stage's time and compares it with the last
|
||||
passing PROTOCOL.md. The threshold is yours to set.
|
||||
|
||||
## 7. The uncommitted work you rejected, still untouched
|
||||
|
||||
**Untracked or modified files:**
|
||||
- `v4/system/post_all_cases.c`, `post_v3_cases.c`, `post_cases.c`, `POST_FLOOR`
|
||||
- `v4/tests/test_mesh_boot.c`, `v4/Makefile`
|
||||
- `v4/include/v4/post.h`, `mesh.h`, `v4/system/mesh.c`
|
||||
- `v4/tools/mkpost.py`, `post_all_rules.py`
|
||||
- `docs/v4.0.0/POST.md`, `docs/v4.0.0/POST-PARITY.md`
|
||||
- `logs/20261117-135508/aarch64/README.md`
|
||||
|
||||
**Logs not yet committed:**
|
||||
- `logs/20261114-064855`, `-065851`, `-081210`, `-090737`, and `logs/20261117-135508`
|
||||
- two DoE CSVs
|
||||
|
||||
**One false statement:** the addition to POST-PARITY §6 says "Ruled ... Whole POST in tests; boot runs
|
||||
less". You withdrew that. It must not be committed as a ruling.
|
||||
|
||||
## 8. For ruling
|
||||
|
||||
1. **The write-up:** commit it locally as it stands, or correct it first?
|
||||
2. **The proof of §5:** run it, yes or no?
|
||||
3. **How far back:** start over from f10d2bc8, and if so by reset or by revert? Or redesign the step from HEAD?
|
||||
4. **The step's design, whichever way 3 goes.** Keep the rotation as ruled, but make the step's work grow
|
||||
with what runs and what moves, not with how many places there are. v3's own answers are on record:
|
||||
- FABRIC-3 §XXVIII: a flag set, not a walk;
|
||||
- FABRIC-0 GAP-A1: a virtual tick;
|
||||
- FABRIC-3.5 §XLIII: the target takes its own messages on its own turn.
|
||||
|
||||
The design is to be written first, in the record (ENGINE §6, "the node's safe moment"), for your ruling.
|
||||
5. **The rejected work (§7):** discard it, or keep any part of it? And which logs get committed as labelled evidence?
|
||||
6. **The CAPA (§6):** record it now, or later?
|
||||
Reference in New Issue
Block a user