feat(v4, ci): the fault dump on a node's stop, and the EMERGENCY diagnostic build after a failed stage

Captain Bob's CI directive (2026-11-17) and the design he ruled point by point
(docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md).

- v4/system/dump.c, v4/include/v4/dump.h: what a node was doing when it stopped -- the fault's kind,
  address and count; P A B; the data stack, T first; the return stack, R first, each entry named by
  the word it falls in (the backtrace; ROM and below-every-word said so); the running line. Bounded
  dictionary walk; no C library. Ruled: every build, on a stop only (a handled fault keeps its line).
- kernel/src/v4/sk_v4.c: the dump before each of the three halts (not started; POST's blocks not
  let go; a node stopped on a fault) -- Hera and any stopped node. kernel/Makefile links dump.o.
- scripts/nightly.sh diagnose [<stage>]: after a failed stage, that configuration built again with
  EMERGENCY_CONSOLE_ENABLED=1 -- build only, nothing booted (ruled) -- its artifacts kept in
  logs/<TS>/diagnose/ only from a build that passed, with a README saying how to launch it with
  --log-level=debug. Always exits 0: evidence, never a result. The protocol runs it after a failed
  stage and still exits with that stage's code; acceptance.yml, nightly.yml and release.yml run it
  `if: failure()`, before the logs are kept. Any failed stage triggers it (ruled).
- capsules/BLOCK_MAP.md: as the build writes it -- the v4:physics.4th row, missing since 28ffed8f
  (ruled: commit the regenerated map).
- Test first: test_dump.c failed with no dump.c (logs/20261117-dump), then 15 checks at both widths
  and under ASan+UBSan. The diagnose stage: a real EMERGENCY amd64 build; a failing build exits 0 and
  keeps no artifacts (a defect found and fixed this way); the protocol path keeps the stage's exit.
- Also the write-up of the regression at 046ce0be, for ruling.

Not yet: acceptance on a clean tree (the next run); the kernel's stop wiring is compile-tested only
-- no boot reaches a stop on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
This commit is contained in:
Claude
2026-11-17 19:14:40 +00:00
parent 3376255930
commit 45b11aec25
13 changed files with 632 additions and 7 deletions
+4
View File
@@ -71,6 +71,10 @@ jobs:
- name: Acceptance 32-bit build
run: scripts/nightly.sh test-32
- name: EMERGENCY diagnostic build (on failure; evidence, not a result)
if: failure()
run: scripts/nightly.sh diagnose # build only; the failed step stays the job's failure
- name: Keep the logs and DoE CSVs
if: always()
uses: actions/upload-artifact@v3
+4
View File
@@ -94,6 +94,10 @@ jobs:
if: always()
run: scripts/nightly.sh proofs
- name: EMERGENCY diagnostic build (on failure; evidence, not a result)
if: failure()
run: scripts/nightly.sh diagnose # build only; the failed step stays the job's failure
- name: Keep the logs and DoE CSVs
if: always()
uses: actions/upload-artifact@v3
+4
View File
@@ -107,6 +107,10 @@ jobs:
build/nightly/
build/nightly-docs/
- name: EMERGENCY diagnostic build (on failure; evidence, not a result)
if: failure()
run: scripts/nightly.sh diagnose # build only; the failed step stays the job's failure
- name: Keep the logs and DoE CSVs
if: always()
uses: actions/upload-artifact@v3
+7 -1
View File
@@ -32,6 +32,7 @@
| `v4:centre.4th` | 8100, 8101, 8102, 8104, 8105, 8106, 8107 | `0xccd1b28a101c78b2` | n/a |
| `v4:forth79.4th` | 6000, 6001, 6002 | `0x4055641ee17d176b` | n/a |
| `v4:hera.4th` | 8000, 8001 | `0x85300ad4e6e8b86b` | n/a |
| `v4:physics.4th` | 8200, 8201, 8202, 8203, 8204 | `0x747d89ca8d57e29c` | n/a |
| `v4:starshipos.4th` | 7000, 7001 | `0xb74bfa51f8d4719a` | n/a |
| `workload-0.4th` | 2200, 2201 | `0x93f86f60aeba8feb` | n/a |
| `workload-1-lite.4th` | 5058, 5059 | `0x44a7a7e3176dcc8d` | n/a |
@@ -379,10 +380,15 @@
| 8105 | `v4:centre.4th` | `0xccd1b28a101c78b2` | ok |
| 8106 | `v4:centre.4th` | `0xccd1b28a101c78b2` | ok |
| 8107 | `v4:centre.4th` | `0xccd1b28a101c78b2` | ok |
| 8200 | `v4:physics.4th` | `0x747d89ca8d57e29c` | ok |
| 8201 | `v4:physics.4th` | `0x747d89ca8d57e29c` | ok |
| 8202 | `v4:physics.4th` | `0x747d89ca8d57e29c` | ok |
| 8203 | `v4:physics.4th` | `0x747d89ca8d57e29c` | ok |
| 8204 | `v4:physics.4th` | `0x747d89ca8d57e29c` | ok |
## Conflicts
None.
---
*39 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.*
*40 capsule(s) scanned. Re-run `mkcapsule --manifest <dir>` to refresh.*
@@ -0,0 +1,81 @@
# The failure-triggered EMERGENCY diagnostic build -- design, for ruling
Written 2026-11-17 from Captain Bob's CI directive ("Emergency Console -- Separate Build
Configuration") and his order: "inspect the debug build first, then write the design".
**Nothing here is built.** The open points are put to Captain Bob one at a time, as he ruled the same
day.
## 1. What exists today (inspected 2026-11-17, read only)
| What | Where | What it does |
|---|---|---|
| `EMERGENCY_CONSOLE_ENABLED` (build parameter) | `Kconfig.heartbeat:70`, default n; `kernel/Makefile:429` | Gives a console at boot with no login, and makes an error recover instead of halting. On v4 the screen is live from the start and there is no headless wait (`kernel/src/v4/sk_v4.c:258, 348, 366`; `sk_v4_identity.c:463, 504`). **It adds no diagnostics.** |
| `--log-level=debug` (boot argument) | `kernel/src/boot/cmdline.c:135`; `kernel_args.h:25` | v3 only: a trace of every word (`ECW: w=... func=... 'NAME'`, `kernel/src/vm/vm_core.c:872`). This is very slow. **Nothing on the v4 path reads `log_level`.** |
| `SK_PARITY_DEBUG` (build parameter) | `kernel/Makefile:524` | v3's parity debugging. |
| The CPU's fault handlers (every build) | `kernel/src/arch/*/interrupts.c` | Print the vector or cause, then halt. amd64 prints RIP, CR2, RSP, RBP, the word on the stack, and the executing word on a #GP. aarch64 prints ELR_EL1; riscv64 prints sepc. |
| A v4 node that faults | `v4/include/v4/node.h:129-134`; `kernel/src/v4/sk_v4.c:407` | The node records its fault's kind, its address and a count. The kernel prints only "StarForth v4: the node stopped on a fault" and halts. **Nothing prints P, A, B, the stacks, the return stack (the node's backtrace) or the fault's kind and address.** |
| Acceptance and nightly on failure | `.gitea/workflows/acceptance.yml`, `nightly.yml`; `scripts/nightly.sh` | The failed step fails the job and the logs are kept. **No diagnostic build follows.** |
So the directive's diagnostics (stack states and contents, pointers and execution state, backtraces,
full logs) exist on v4 today only as fields in the node that nothing prints. On v3 they exist only as
the per-word trace, which a boot argument turns on.
## 2. The design
**A. The trigger.** A normal build or validation fails. Its step has failed, so the job's result is
already failed.
- A following step runs, `if: failure()` in the workflows, and the same call in
`scripts/nightly.sh protocol`: `scripts/nightly.sh diagnose <what failed>`.
- It builds the failed configuration again with `EMERGENCY_CONSOLE_ENABLED=1` and diagnostic logging
on (B).
- It keeps the result and the artifacts under `logs/<TS>/diagnose/`:
- the ELF with symbols;
- the ISO or the image;
- the `.config`;
- the capsule map;
- the build log;
- its own result.
- Nothing it does can change the job's result: the step only gathers evidence, and the failed step
stays the reported failure.
**B. What the diagnostic build carries** (it needs building, and a ruling):
- On a node's fault or stop, the kernel prints:
- the fault's kind and address, and the fault count;
- P, A, B, T and S;
- the whole data stack;
- the whole return stack, with each address named by the dictionary word it falls in. That list is
the node's backtrace.
- the line that was running.
- Diagnostic logging on the v4 path: what the node is doing, word by word, as v3's per-word trace is.
It is allowed to be slow.
**C. Building is not launching.** The diagnostic step builds and keeps the artifacts. It does not boot
an emergency console. Launching one is a separate act: a person's, or a stage made for it.
**D. Failure categories.** Each is its own named step and its own line in PROTOCOL.md:
1. source: a `.f18` that does not assemble or a `.4th` that does not compile (POST);
2. the kernel's build and unit tests;
3. ASan and UBSan;
4. QEMU, per architecture;
5. the emergency diagnostic build.
## 3. Open points, to be put one at a time
1. ~~Diagnostic logging is a boot argument today (`--log-level`). Should the diagnostic build turn it
on as a build parameter instead?~~ **Withdrawn 2026-11-17. It was not a discrepancy.**
- The directive forbids only a boot argument standing in for the EMERGENCY *build* (`--sos`).
- `--log-level` stands in for nothing: it is how logging has always been chosen.
- I made a problem out of established practice by reading the directive too widely.
- The diagnostic build is EMERGENCY at build time, with `--log-level=debug` as it has always been.
2. ~~Should the diagnostic step build only (C), or also boot that build once?~~ **Ruled 2026-11-17
(Captain Bob): build only.** The step builds the diagnostic configuration and keeps its artifacts.
Launching it is a person's act.
3. ~~Should B be printed in every build, or only in the EMERGENCY build?~~ **Ruled 2026-11-17 (Captain
Bob): every build.** The fault dump prints in any build on a node's fault, as the CPU's fault
handlers already do. The slow word-by-word log stays behind `--log-level=debug`.
4. ~~Which stage does the trigger cover?~~ **Ruled 2026-11-17 (Captain Bob): any failed stage.** The
diagnostic build of that configuration follows whatever fails.
All four open points are settled. The design waits for Captain Bob's word to build it.
@@ -0,0 +1,159 @@
# The regression at 046ce0be -- write-up, for ruling
Written 2026-11-17, after reading the record again in full (notes: `.claude/memory/reading-2026-11-17.md`).
**Nothing here is built, reverted or reset.** Every decision below waits for Captain Bob.
## 1. What and when
v4's boot became 1.5 to 2.4 times slower at **046ce0be** ("the mesh boot to spec -- Hera born empty in
the kernel's fabric; ... every node stepping together", 2026-11-13). The last good commit is
**f10d2bc8**, the commit before it.
Both commits ran the same POST, 539 cases, and both protocols passed. Each stage's time is taken from
the start of its QEMU log to the start of the next stage's log. A stage is a `clean` build, the boot and
the session.
| Stage | f10d2bc8 (`logs/20261113-142432`) | 046ce0be (`logs/20261113-170011`) | Ratio |
|---|---|---|---|
| amd64 | 14:25:04 to 14:26:51: **107 s** | 17:00:39 to 17:03:43: **184 s** | 1.7 |
| aarch64 | 14:26:51 to 14:30:20: **209 s** | 17:03:43 to 17:12:02: **499 s** | 2.4 |
| riscv64 | 14:30:20 to 14:32:05: **105 s** | 17:12:02 to 17:14:45: **163 s** | 1.6 |
**What these numbers do not prove on their own:**
- There is one run of each commit.
- The time includes the build.
- This host varies: the same commit took 146 s, then 175 s an hour later (Phase 7 plan, Task 2b).
Section 5 proposes the proof.
Two more measurements were taken in the previous session but are **not committed evidence**:
- A gprof profile of the host mesh boot, with `v4_fabric_step` at about 40% of the time spent in its own code (`v4/gmon.out`, untracked).
- A loop of a million, on hosted v3 and on Hera in the mesh:
| Run | Time |
|---|---|
| hosted v3 | about 172 ms |
| Hera in the mesh | about 1362 ms |
## 2. The mechanism, in the code
- **f10d2bc8.** One node runs the boot. For each instruction word, `boot_exec`
(`git show f10d2bc8:v4/system/boot.c`, lines 200-209) does three things:
- sets the node's have-mask;
- calls `v4_exec_step_word` once;
- checks one flag.
- **046ce0be onward.** Every instruction word Hera executes is one `v4_fabric_step` (`v4/src/fabric.c`,
lines 423-490), and the step walks all 16 places of the kernel's fabric (`SK_V4_PLACES`,
`kernel/src/v4/sk_v4.c:63`) in ten passes:
1. the have-mask, with an 8-port loop for each awake node;
2. `look_about` for each node that is not blocked, with another 8-port loop;
3. one `v4_exec_step_word` for each awake node, blocked ones included;
4. clearing `have_fetched`;
5. and 6. the two operation passes;
7. handing over writes;
8. handing over reads, which asks the console device on every step that Hera reads;
9. the gone-error pass.
After the step, `takes()` polls the console's queue (`v4/system/mesh.c`, `v4_mesh_step`, `v4_mesh_tell`).
So the cost of a step grows with how many places there are, not with how many nodes run. In the boot one
node runs, so nearly all of each step is overhead.
## 3. What the record says. I was wrong about this before.
**The rotation is your ruled design. It is not a scheduler.**
- ENGINE.md §3d, ruled "A" on 2026-10-05: "Every node that is not blocked runs; on one processor that is
each in rotation, an instruction word at a time, which is what 'all at once' is there and is not a
policy."
- MESH.md §2, ruling 7, and §4.3: "There is no choice of whose turn it is."
- `fabric.h` says the same.
Calling the lock-step itself a scheduler was my mistake. The scheduler I actually offered was the
withdrawn ruling "Hera by compudynamics now" (Phase 1 plan, 2026-11-13). The rotation stays as ruled.
**What the change broke is the cost.**
- **"An idle node does not spin"** (ENGINE §3d; MESH ruling 7). The node does not, but the fabric now
spins over it at every instruction word. MESH 9a.6 tested only the node: "its anti-clock ... does not
move over thousands of steps". It never tested what the step costs.
- **FABRIC-3.5 §XXII and §XXXV.2** rule out per-tick walks over every VM as a scaling wall: "we really will
be giving birth to potentially hundreds of VMs".
- **ENGINE §6** still lists as open "the node's safe moment for message delivery and switching". 046ce0be
did not settle it. It made every instruction word that moment, for every place.
**The slowdown was seen at each step and explained away.** It was never measured against f10d2bc8.
- **Phase 1 plan:** "The amd64 boot: 160 s (about 90 s before -- POST and the lines now go as messages)".
- **Phase 2:** "142 s (Phase 1: 153 s)", compared with the boot that was already slow.
- **Phase 7, Task 2b:** "The host slowed."
- **POST-PARITY §6:** "about 3 m 15 s at 539" was taken as the baseline.
Your standing order, "Find the TCG cost first" (POST-PARITY §6), is what this answers.
## 4. What it costs to go back to f10d2bc8
There are 33 commits after f10d2bc8:
| Commits | Where they are |
|---|---|
| 046ce0be | **pushed**, in PR 164 |
| Phase 2, e8e7ec8b | **pushed**, in PR 164 |
| The remote's head, ee28530b | **pushed** |
| 28 more: Phase 7 (the loops, the opcode physics, the physics words, the whole POST of 990 cases) and my five | **local only** |
So starting over from f10d2bc8 changes pushed history. That needs your explicit word. The choices are:
- reset the branch, with a force push;
- add revert commits;
- keep the history and redesign the step forward from HEAD.
## 5. Proposed proof, before anything is changed
**Intent:** show that the gap is the commit, not the host.
**How:**
- Use `git archive` to build f10d2bc8 and 046ce0be into the scratchpad. Nothing on the branch changes.
- Boot each one, back to back, on amd64, then on aarch64. That is four boots.
- Each boot runs in the foreground, one QEMU at a time, under `timeout 570`.
**The observation that proves it**, named before the run: 046ce0be's boot time is above f10d2bc8's by more
than the host's variance on both ISAs (about 1.7 times on amd64 and 2.4 times on aarch64), and the POST
line is the same.
**The logs** are committed and labelled.
## 6. What escaped, and how (a CAPA candidate, to rule)
- **What got through:** a 1.6 to 2.4 times regression passed the protocol.
- **Why:** no stage compares the boot's time with the last good commit, and acceptance has no criterion for cost.
- **Proposed control (not built):** the protocol records each stage's time and compares it with the last
passing PROTOCOL.md. The threshold is yours to set.
## 7. The uncommitted work you rejected, still untouched
**Untracked or modified files:**
- `v4/system/post_all_cases.c`, `post_v3_cases.c`, `post_cases.c`, `POST_FLOOR`
- `v4/tests/test_mesh_boot.c`, `v4/Makefile`
- `v4/include/v4/post.h`, `mesh.h`, `v4/system/mesh.c`
- `v4/tools/mkpost.py`, `post_all_rules.py`
- `docs/v4.0.0/POST.md`, `docs/v4.0.0/POST-PARITY.md`
- `logs/20261117-135508/aarch64/README.md`
**Logs not yet committed:**
- `logs/20261114-064855`, `-065851`, `-081210`, `-090737`, and `logs/20261117-135508`
- two DoE CSVs
**One false statement:** the addition to POST-PARITY §6 says "Ruled ... Whole POST in tests; boot runs
less". You withdrew that. It must not be committed as a ruling.
## 8. For ruling
1. **The write-up:** commit it locally as it stands, or correct it first?
2. **The proof of §5:** run it, yes or no?
3. **How far back:** start over from f10d2bc8, and if so by reset or by revert? Or redesign the step from HEAD?
4. **The step's design, whichever way 3 goes.** Keep the rotation as ruled, but make the step's work grow
with what runs and what moves, not with how many places there are. v3's own answers are on record:
- FABRIC-3 §XXVIII: a flag set, not a walk;
- FABRIC-0 GAP-A1: a virtual tick;
- FABRIC-3.5 §XLIII: the target takes its own messages on its own turn.
The design is to be written first, in the record (ENGINE §6, "the node's safe moment"), for your ruling.
5. **The rejected work (§7):** discard it, or keep any part of it? And which logs get committed as labelled evidence?
6. **The CAPA (§6):** record it now, or later?
+2 -2
View File
@@ -738,8 +738,8 @@ KERNEL_EXTRA_SRCS += $(KERNEL_SRC)/v4/sk_v4.c $(KERNEL_SRC)/v4/sk_v4_identity.c
# v4/system/boot.c is the boot the hosted v4 system runs too: the ROM
# assembles the system, then the capsules from this kernel's own capsule
# directory, then the prompt.
LOADER_V4_OBJS := $(patsubst v4/src/%.c,$(LOADER_OBJ_DIR)/v4engine/%.o,$(V4_ENGINE_SRCS)) $(LOADER_OBJ_DIR)/v4engine/v4_rom.o $(LOADER_OBJ_DIR)/v4engine/v4_map.o $(LOADER_OBJ_DIR)/v4engine/v4_mesh_rom.o $(LOADER_OBJ_DIR)/v4system/mesh.o $(LOADER_OBJ_DIR)/v4system/inference.o $(LOADER_OBJ_DIR)/v4system/boot.o $(LOADER_OBJ_DIR)/v4system/blocks.o $(LOADER_OBJ_DIR)/v4system/post.o $(LOADER_OBJ_DIR)/v4system/post_cases.o
KERNEL_V4_OBJS := $(patsubst v4/src/%.c,$(KERNEL_OBJ_DIR)/v4engine/%.o,$(V4_ENGINE_SRCS)) $(KERNEL_OBJ_DIR)/v4engine/v4_rom.o $(KERNEL_OBJ_DIR)/v4engine/v4_map.o $(KERNEL_OBJ_DIR)/v4engine/v4_mesh_rom.o $(KERNEL_OBJ_DIR)/v4system/mesh.o $(KERNEL_OBJ_DIR)/v4system/inference.o $(KERNEL_OBJ_DIR)/v4system/boot.o $(KERNEL_OBJ_DIR)/v4system/blocks.o $(KERNEL_OBJ_DIR)/v4system/post.o $(KERNEL_OBJ_DIR)/v4system/post_cases.o
LOADER_V4_OBJS := $(patsubst v4/src/%.c,$(LOADER_OBJ_DIR)/v4engine/%.o,$(V4_ENGINE_SRCS)) $(LOADER_OBJ_DIR)/v4engine/v4_rom.o $(LOADER_OBJ_DIR)/v4engine/v4_map.o $(LOADER_OBJ_DIR)/v4engine/v4_mesh_rom.o $(LOADER_OBJ_DIR)/v4system/mesh.o $(LOADER_OBJ_DIR)/v4system/inference.o $(LOADER_OBJ_DIR)/v4system/boot.o $(LOADER_OBJ_DIR)/v4system/blocks.o $(LOADER_OBJ_DIR)/v4system/post.o $(LOADER_OBJ_DIR)/v4system/post_cases.o $(LOADER_OBJ_DIR)/v4system/dump.o
KERNEL_V4_OBJS := $(patsubst v4/src/%.c,$(KERNEL_OBJ_DIR)/v4engine/%.o,$(V4_ENGINE_SRCS)) $(KERNEL_OBJ_DIR)/v4engine/v4_rom.o $(KERNEL_OBJ_DIR)/v4engine/v4_map.o $(KERNEL_OBJ_DIR)/v4engine/v4_mesh_rom.o $(KERNEL_OBJ_DIR)/v4system/mesh.o $(KERNEL_OBJ_DIR)/v4system/inference.o $(KERNEL_OBJ_DIR)/v4system/boot.o $(KERNEL_OBJ_DIR)/v4system/blocks.o $(KERNEL_OBJ_DIR)/v4system/post.o $(KERNEL_OBJ_DIR)/v4system/post_cases.o $(KERNEL_OBJ_DIR)/v4system/dump.o
endif
LOADER_SRCS := $(LOADER_SRCS_BASE) $(LOADER_EXTRA_SRCS)
+24
View File
@@ -34,6 +34,7 @@
#include "starkernel/timer.h"
#include "v4/boot.h"
#include "v4/fabric.h"
#include "v4/dump.h"
#include "v4/manage.h"
#include "v4/mesh.h"
#include "starkernel/kmalloc.h"
@@ -196,6 +197,26 @@ static void sk_v4_out(const char *text, unsigned len)
for (i = 0; i < len; i++) console_putc(text[i]);
}
/* The fault dump (v4/dump.h; plan 2026-11-17-emergency-diagnostic-build, ruled by Captain Bob 2026-11-17:
* every build, on a stop only): before the kernel halts on a node that stopped, it says what Hera -- and
* any other node that is stopped -- was doing: the fault, P A B, both stacks, the return stack named by
* word (the backtrace), and the line that was running. */
static void sk_v4_dump_out(void *self, const char *s, unsigned n) { (void)self; sk_v4_out(s, n); }
static void sk_v4_dump(const char *line, unsigned len)
{
unsigned k;
for (k = 0; k < SK_V4_PLACES; k++) {
v4_fabric_node *x = v4_fabric_node_at(&sk_v4_mesh.f, k);
if (!x || (k != sk_v4_mesh.hera_place && !x->n.stopped)) continue;
console_puts("V4 DUMP: the node at place ");
console_putc((char)('0' + (int)(k / 10u))); console_putc((char)('0' + (int)(k % 10u)));
console_puts(k == sk_v4_mesh.hera_place ? " (Hera)\n" : "\n");
v4_dump_node(&x->n, &sk_v4_image, k == sk_v4_mesh.hera_place ? line : 0,
k == sk_v4_mesh.hera_place ? len : 0u, sk_v4_dump_out, 0);
}
}
/* a character for a line that reads the keyboard, when there is one: the
* serial line or any keyboard (docs/v4.0.0/STICK.md 5.4) */
static int sk_v4_key(void)
@@ -333,6 +354,7 @@ void sk_v4_boot(void (*chain)(void), const BootDisk *boot_disk)
blk_subsys_init(sk_v4_post_blk_ram, sizeof(sk_v4_post_blk_ram));
if (!v4_mesh_boot(&sk_v4_mesh)) {
console_println("StarForth v4: not started");
sk_v4_dump(0, 0u);
sk_v4_stop(); /* #177: halts, not a 100% spin */
}
if (chain) {
@@ -341,6 +363,7 @@ void sk_v4_boot(void (*chain)(void), const BootDisk *boot_disk)
chain();
if (sk_v4_tell("EMPTY-BUFFERS", 13) != V4_TEXT_COMPLETED) {
console_println("StarForth v4: the node did not let go of POST's blocks");
sk_v4_dump("EMPTY-BUFFERS", 13u);
sk_v4_stop(); /* #177: halts, not a 100% spin */
}
}
@@ -405,6 +428,7 @@ void sk_v4_prompt(void)
else if (how == V4_BOOT_LINE_REFUSED) console_puts("Message refused\n ERROR\n"); /* the node's wire is full: error 19's words */
else if (how == V4_BOOT_LINE_STOPPED) {
console_println("StarForth v4: the node stopped on a fault");
sk_v4_dump(sk_v4_text, (unsigned)len);
sk_v4_stop(); /* #177: halts, not a 100% spin */
}
}
+68
View File
@@ -42,6 +42,16 @@
# run's serial logs and DoE CSVs -- the evidence for the rest; and the proof log, when the
# proofs ran before it, as a release runs them).
#
# scripts/nightly.sh diagnose [<stage that failed>]
# After a failed stage: the EMERGENCY diagnostic build (docs/v4.0.0/plans/
# 2026-11-17-emergency-diagnostic-build.md; Captain Bob, 2026-11-17). The failed configuration
# is built again with EMERGENCY_CONSOLE_ENABLED=1 -- the architecture of a failed accept, or all
# three for any other stage -- and its artifacts kept in logs/<TS>/diagnose/: the loader, the
# kernel ELF with its symbols, the capsule map, each build's log and result, and a README that
# says how to launch it with diagnostic logging (--log-level=debug, as always). BUILD ONLY:
# nothing is booted (ruled). It is evidence, never a result: it always exits 0, whatever its own
# build did, and the stage that failed stays the run's failure.
#
# The documents (a separate pipeline):
#
# scripts/nightly.sh docs
@@ -83,6 +93,7 @@ usage() {
echo "Usage: scripts/nightly.sh accept <amd64|aarch64|riscv64|stick-high> | test-32 | v4-checks | proofs | images" >&2
echo " | publish <URL> | docs | publish-docs <URL> | gitea-release <tag>" >&2
echo " | protocol [<stage>...] | protocol-report | dict-hash | post-count" >&2
echo " | diagnose [<stage that failed>]" >&2
echo " protocol stages: $PROTOCOL_STAGES" >&2
exit 2
}
@@ -308,6 +319,62 @@ publish-docs)
"$DOCS/vol1.pdf" pdf vol1 "$DOCS/vol2.pdf" pdf vol2 "$DOCS/vol3.pdf" pdf vol3 \
"$DOCS/api.pdf" pdf api
;;
diagnose)
# The EMERGENCY diagnostic build: see the header. Evidence only -- it exits 0 whatever happens.
failed=${2:-}
if [ -z "$failed" ]; then # the workflows: the first accept that did not pass
for b in amd64 aarch64 riscv64 stick-high; do
grep -q '^RESULT: PASS' "$OUT/accept-$b.txt" 2>/dev/null || { failed=accept-$b; break; }
done
failed=${failed:-unknown}
fi
case $failed in
accept-stick-high|accept-amd64) archs=amd64 ;;
accept-aarch64) archs=aarch64 ;;
accept-riscv64) archs=riscv64 ;;
*) archs="amd64 aarch64 riscv64" ;;
esac
ts=$(date -u +%Y%m%d-%H%M%S); d=logs/$ts/diagnose; mkdir -p "$d"
echo "=== nightly: diagnose after $failed: the EMERGENCY build of $archs (build only)"
{
echo "# EMERGENCY diagnostic build -- evidence, not a result"
echo
echo "After: \`$failed\` failed (that failure is the run's result). Commit: $(git rev-parse HEAD)."
echo "Built with EMERGENCY_CONSOLE_ENABLED=1; nothing was booted (build only, ruled 2026-11-17)."
echo
echo "To launch it by hand, with diagnostic logging, one QEMU at a time, in the foreground:"
echo
for a in $archs; do
echo " timeout 570 make -f kernel/Makefile ARCH=$a EMERGENCY_CONSOLE_ENABLED=1 KERNEL_ARGS=--log-level=debug qemu"
done
echo
echo "| Architecture | Build | Log |"
echo "|---|---|---|"
} > "$d/README.md"
for a in $archs; do
set +e
make --no-print-directory -f kernel/Makefile ARCH=$a EMERGENCY_CONSOLE_ENABLED=1 clean all > "$d/build-$a.log" 2>&1
rc=$?
set -e
if [ $rc = 0 ]; then st=PASS; else st="FAIL (exit $rc)"; fi
# the artifacts only of a build that passed: what is in build/$a after a failed one is an older
# build's, and must not pass for this one
if [ $rc = 0 ]; then
mkdir -p "$d/$a"
for f in starkernel_loader.efi starkernel_kernel.elf; do
[ -f "build/$a/kernel/$f" ] && cp "build/$a/kernel/$f" "$d/$a/" || true
done
[ -f "build/$a/.config" ] && cp "build/$a/.config" "$d/$a/config" || true
cp capsules/BLOCK_MAP.md "$d/$a/" 2>/dev/null || true
else
st="$st -- no artifacts (the build did not complete)"
fi
echo "$rc" > "$d/build-$a.rc"
echo "| $a | $st | \`$d/build-$a.log\` |" >> "$d/README.md"
echo "=== nightly: diagnose $a: $st"
done
exit 0
;;
protocol)
# The developer protocol (#135): the stages a PR, the nightly and a release run, never publish.
shift
@@ -337,6 +404,7 @@ protocol)
if [ "$rc" != 0 ]; then
echo "nightly: protocol stage $s failed (exit $rc)" >&2
[ $all = 0 ] || "$self" protocol-report || true
"$self" diagnose "$s" || true # evidence only: the stage's failure stays the result
exit "$rc"
fi
done
+7 -4
View File
@@ -91,6 +91,9 @@ $(BINDIR)/san-v3blocks.o: $(V3_BLOCK_SRCS) $(wildcard $(ROOT_DIR)/v3/include/*.h
# docs/v4.0.0/NUCLEUS.md 6.3), which are the kernel's too.
blocks_src = $(if $(call uses_blocks,$(1)),$(BLOCKS_SRC) $(HERE)/system/post.c $(HERE)/system/post_cases.c)
blocks_objs = $(if $(call uses_blocks,$(1)),$(BINDIR)/v3blocks.o)
# the fault dump of a node that stopped (system/dump.c; plan 2026-11-17-emergency-diagnostic-build): the
# kernel's, linked with its own test
dump_src = $(if $(findstring /test_dump,$(1)),$(HERE)/system/dump.c)
san_blocks_objs = $(if $(call uses_blocks,$(1)),$(BINDIR)/san-v3blocks.o)
# The system's source, capsules/v4/sys (MACHINE.md 3): F18 assembler as text
@@ -318,16 +321,16 @@ endif
#
# Both binaries depend on this Makefile, so a flag change rebuilds them.
define TEST_RULE
$(BINDIR)/$(1)-$(notdir $(2)): $(2) $$(SRCS) $(call blocks_src,$(2)) $(call blocks_objs,$(2)) $(call rom_src,$(2),$(1)) $$(wildcard $(HERE)/include/v4/*.h) $$(wildcard $(HERE)/tests/*.h) $(HERE)/Makefile
$(BINDIR)/$(1)-$(notdir $(2)): $(2) $$(SRCS) $(call blocks_src,$(2)) $(call dump_src,$(2)) $(call blocks_objs,$(2)) $(call rom_src,$(2),$(1)) $$(wildcard $(HERE)/include/v4/*.h) $$(wildcard $(HERE)/tests/*.h) $(HERE)/Makefile
@mkdir -p $(BINDIR)
$$(CC) $$(CFLAGS) -I$(HERE)/include $(call blocks_inc,$(2)) -DV4_CELL_BITS=$(1) $(call node_size,$(2)) $(capsule_dir) $(call rom_sym,$(2),$(1)) \
$(2) $$(SRCS) $(call blocks_src,$(2)) $(call blocks_objs,$(2)) $(call rom_src,$(2),$(1)) -o $$@
$(2) $$(SRCS) $(call blocks_src,$(2)) $(call dump_src,$(2)) $(call blocks_objs,$(2)) $(call rom_src,$(2),$(1)) -o $$@
$(BINDIR)/san-$(1)-$(notdir $(2)): $(2) $$(SRCS) $(call blocks_src,$(2)) $(call san_blocks_objs,$(2)) $(call rom_src,$(2),$(1)) $$(wildcard $(HERE)/include/v4/*.h) $$(wildcard $(HERE)/tests/*.h) $(HERE)/Makefile
$(BINDIR)/san-$(1)-$(notdir $(2)): $(2) $$(SRCS) $(call blocks_src,$(2)) $(call dump_src,$(2)) $(call san_blocks_objs,$(2)) $(call rom_src,$(2),$(1)) $$(wildcard $(HERE)/include/v4/*.h) $$(wildcard $(HERE)/tests/*.h) $(HERE)/Makefile
@mkdir -p $(BINDIR)
$$(CC) $$(CSTD) $$(WARN) -O1 -g -fno-omit-frame-pointer \
-fsanitize=address,undefined -fno-sanitize-recover=all -I$(HERE)/include $(call blocks_inc,$(2)) -DV4_CELL_BITS=$(1) $(call node_size,$(2)) $(capsule_dir) $(call rom_sym,$(2),$(1)) \
$(2) $$(SRCS) $(call blocks_src,$(2)) $(call san_blocks_objs,$(2)) $(call rom_src,$(2),$(1)) -o $$@
$(2) $$(SRCS) $(call blocks_src,$(2)) $(call dump_src,$(2)) $(call san_blocks_objs,$(2)) $(call rom_src,$(2),$(1)) -o $$@
endef
# One run target per (width, test) and one prerequisite line per width, so
+32
View File
@@ -0,0 +1,32 @@
/* SPDX-License-Identifier: LicenseRef-Starship-1.0
* Copyright (c) 2023-2026 Robert A. James
*/
/* dump.h -- what a node was doing when it stopped (docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md;
* ruled by Captain Bob, 2026-11-17: in every build, on a stop only -- a fault the node's own handler takes
* keeps its one-line message).
*
* The dump says:
* - the latest fault's kind, its address for an address fault, how many there have been, and whether
* the node is stopped;
* - P, A and B;
* - the data stack, T first, every value it holds;
* - the return stack, R first, each entry named by the dictionary word whose code it falls in: the
* node's backtrace (an address in the ROM says so, and one below every word says so);
* - the line that was running, if one was.
*
* A word is found by walking the dictionary from LATEST (dict.f18: the link at xt-1, the name's word
* address at xt-2, its count byte first); the walk is bounded, so a damaged dictionary cannot hold the
* dump. An address is named by the newest word whose xt is at or below it: code with no header of its
* own that follows a word is named by that word. Nothing here uses the C library. */
#ifndef V4_DUMP_H
#define V4_DUMP_H
#include "v4/node.h"
#include "v4/image.h"
typedef void (*v4_dump_out)(void *self, const char *s, unsigned n);
void v4_dump_node(const v4_node *n, const v4_image *im, const char *line, unsigned len,
v4_dump_out out, void *self);
#endif /* V4_DUMP_H */
+119
View File
@@ -0,0 +1,119 @@
/* SPDX-License-Identifier: LicenseRef-Starship-1.0
* Copyright (c) 2023-2026 Robert A. James
*/
/* dump.c -- what a node was doing when it stopped. See v4/dump.h. */
#include "v4/dump.h"
#include "v4/stack.h"
static void say(v4_dump_out out, void *self, const char *s)
{
unsigned n = 0;
while (s[n]) n++;
out(self, s, n);
}
static void say_num(v4_dump_out out, void *self, v4_cell v)
{
char buf[24];
unsigned i = sizeof buf;
v4_ucell u = v < 0 ? (v4_ucell)0 - (v4_ucell)v : (v4_ucell)v;
do { buf[--i] = (char)('0' + (int)(u % 10u)); u /= 10u; } while (u && i > 1);
if (v < 0) buf[--i] = '-';
out(self, buf + i, (unsigned)(sizeof buf - i));
}
static unsigned byte_at(const v4_node *n, v4_cell nfa, unsigned i)
{
v4_cell w = nfa + (v4_cell)(i / 4u);
if (w < 0 || w >= (v4_cell)V4_NODE_WORDS) return 0;
return (unsigned)(((v4_ucell)n->mem[w] >> (8u * (i % 4u))) & 0xFFu);
}
/* the newest word whose xt is at or below addr, or 0 */
static v4_cell word_of(const v4_node *n, const v4_image *im, v4_cell addr)
{
v4_cell xt;
unsigned k;
if (!im || im->latest <= 0 || im->latest >= (v4_cell)V4_NODE_WORDS) return 0;
for (xt = n->mem[im->latest], k = 0; xt > 4 && xt < (v4_cell)V4_NODE_WORDS && k < V4_NODE_WORDS;
xt = n->mem[xt - 1], k++)
if (xt <= addr) return xt;
return 0;
}
static void say_where(v4_dump_out out, void *self, const v4_node *n, const v4_image *im, v4_cell addr)
{
v4_cell xt, nfa;
unsigned len, i;
char c;
say_num(out, self, addr);
if (addr >= V4_ROM_BASE) { say(out, self, " in ROM"); return; }
xt = addr >= 0 && addr < (v4_cell)V4_NODE_WORDS ? word_of(n, im, addr) : 0;
if (xt == 0) { say(out, self, " below every word"); return; }
nfa = n->mem[xt - 2];
if (nfa < 0 || nfa >= xt) { say(out, self, " in a word with no name"); return; }
len = byte_at(n, nfa, 0);
if (len > 31u) len = 31u;
say(out, self, " in ");
for (i = 1; i <= len; i++) { c = (char)byte_at(n, nfa, i); out(self, &c, 1); }
}
static const char *kind_name(unsigned k)
{
switch (k) {
case V4_FAULT_ADDRESS: return "address fault";
case V4_FAULT_DATA_OVER: return "data overflow";
case V4_FAULT_DATA_UNDER: return "data underflow";
case V4_FAULT_RET_OVER: return "return overflow";
case V4_FAULT_RET_UNDER: return "return underflow";
case V4_FAULT_RAISED: return "error raised";
default: return "unknown fault";
}
}
void v4_dump_node(const v4_node *n, const v4_image *im, const char *line, unsigned len,
v4_dump_out out, void *self)
{
unsigned i, h;
say(out, self, "V4 DUMP: ");
if (n->faults == 0) say(out, self, "no fault");
else {
say(out, self, kind_name(n->fault_kind));
if (n->fault_kind == V4_FAULT_ADDRESS) { say(out, self, " at "); say_num(out, self, n->fault_addr); }
}
say(out, self, " faults="); say_num(out, self, (v4_cell)n->faults);
say(out, self, n->stopped ? " stopped\n" : " running\n");
say(out, self, "V4 DUMP: P="); say_num(out, self, n->p);
say(out, self, " A="); say_num(out, self, n->a);
say(out, self, " B="); say_num(out, self, n->b);
say(out, self, "\n");
say(out, self, "V4 DUMP: data stack depth="); say_num(out, self, (v4_cell)n->ds.depth); say(out, self, ":");
for (i = 0, h = n->ds.head; i < n->ds.depth; i++) {
v4_cell v;
if (i == 0) v = n->ds.t;
else if (i == 1) v = n->ds.s;
else { h = (h + V4_DATA_RING - 1u) % V4_DATA_RING; v = n->ds.ring[h]; }
say(out, self, " "); say_num(out, self, v);
}
say(out, self, "\n");
say(out, self, "V4 DUMP: return stack depth="); say_num(out, self, (v4_cell)n->rs.depth); say(out, self, ", R first:\n");
for (i = 0, h = n->rs.head; i < n->rs.depth; i++) {
v4_cell v;
if (i == 0) v = n->rs.r;
else { h = (h + V4_RET_RING - 1u) % V4_RET_RING; v = n->rs.ring[h]; }
say(out, self, "V4 DUMP: ");
say_where(out, self, n, im, v);
say(out, self, "\n");
}
if (line && len) {
say(out, self, "V4 DUMP: line: ");
out(self, line, len);
say(out, self, "\n");
}
}
+121
View File
@@ -0,0 +1,121 @@
/* SPDX-License-Identifier: LicenseRef-Starship-1.0
* Copyright (c) 2023-2026 Robert A. James
*/
/* test_dump.c -- the fault dump of a node that stopped (v4/dump.h;
* docs/v4.0.0/plans/2026-11-17-emergency-diagnostic-build.md, ruled by Captain Bob 2026-11-17:
* every build, on a stop only).
*
* A node is set by hand to a known state: registers, both stacks, a fault, and
* a dictionary of two words laid out as dict.f18 lays an entry (the name's
* word address at xt-2, the link at xt-1, the count byte first). The dump
* must say all of it: the fault's kind, address and count; P, A, B; every
* value on the data stack, T first; every return-stack entry, R first, each
* named by the word it falls in (the backtrace), the ROM's as ROM, one below
* every word as such; and the line that was running.
*
* Built and run at both V4_CELL_BITS=32 and 64; see v4/Makefile.
*/
#include "v4/dump.h"
#include "v4/node.h"
#include "v4/stack.h"
#include "v4/image.h"
#include <stdio.h>
#include <string.h>
static int failures = 0;
static int checks = 0;
#define CHECK(cond, ...) \
do { \
checks++; \
if (!(cond)) { \
failures++; \
printf(" FAIL %s:%d: ", __FILE__, __LINE__); \
printf(__VA_ARGS__); \
printf("\n"); \
} \
} while (0)
static char out[8192];
static size_t used;
static void capture(void *self, const char *s, unsigned n)
{
(void)self;
if (used + n >= sizeof out) n = (unsigned)(sizeof out - 1 - used);
memcpy(out + used, s, n);
used += n;
out[used] = 0;
}
static v4_node node;
/* an entry as dict.f18 lays one: name at nfa (count byte first, four to a cell, low byte first) */
static void entry(v4_cell xt, v4_cell nfa, v4_cell link, const char *name)
{
unsigned len = (unsigned)strlen(name), i;
for (i = 0; i <= len; i++) {
unsigned b = i == 0 ? len : (unsigned char)name[i - 1];
node.mem[nfa + (v4_cell)(i / 4u)] |= (v4_cell)((v4_ucell)b << (8u * (i % 4u)));
}
node.mem[xt - 3] = 0;
node.mem[xt - 2] = nfa;
node.mem[xt - 1] = link;
}
static int has(const char *s) { return strstr(out, s) != 0; }
int main(void)
{
v4_image im;
memset(&im, 0, sizeof im);
v4_node_reset(&node);
im.latest = 10;
entry(100, 90, 0, "ALPHA");
entry(200, 190, 100, "BETA");
node.mem[im.latest] = 200;
node.p = 123; node.a = 7; node.b = 8;
node.fault_kind = V4_FAULT_DATA_UNDER; node.fault_addr = 0; node.faults = 3; node.stopped = 1;
v4_dstack_push(&node.ds, 11);
v4_dstack_push(&node.ds, 22);
v4_dstack_push(&node.ds, 33);
v4_rstack_push(&node.rs, 50); /* below every word */
v4_rstack_push(&node.rs, (v4_cell)V4_ROM_BASE + 5); /* in the ROM */
v4_rstack_push(&node.rs, 205); /* in BETA */
v4_rstack_push(&node.rs, 150); /* in ALPHA: R */
used = 0; out[0] = 0;
v4_dump_node(&node, &im, "3 4 SPIN", 8u, capture, 0);
CHECK(has("data underflow"), "the fault's kind: %s", out);
CHECK(has("faults=3"), "the fault count: %s", out);
CHECK(has("stopped"), "it says the node stopped: %s", out);
CHECK(has("P=123") && has("A=7") && has("B=8"), "P, A, B: %s", out);
CHECK(has("data stack depth=3: 33 22 11"), "the data stack, T first: %s", out);
CHECK(has("return stack depth=4"), "the return stack's depth: %s", out);
CHECK(has("150 in ALPHA"), "R named by its word: %s", out);
CHECK(has("205 in BETA"), "the next named: %s", out);
CHECK(has("in ROM"), "a ROM address said so: %s", out);
CHECK(has("50 below every word"), "an address below the dictionary said so: %s", out);
CHECK(strstr(out, "150 in ALPHA") < strstr(out, "205 in BETA"), "R first, then down the stack: %s", out);
CHECK(has("line: 3 4 SPIN"), "the running line: %s", out);
/* an address fault says the address */
node.fault_kind = V4_FAULT_ADDRESS; node.fault_addr = 99999;
used = 0; out[0] = 0;
v4_dump_node(&node, &im, 0, 0u, capture, 0);
CHECK(has("address fault at 99999"), "the faulting address: %s", out);
CHECK(!has("line:"), "no line when none was running: %s", out);
/* a dictionary whose link goes wild does not run away: the walk is bounded */
node.mem[100 - 1] = 200; /* ALPHA links back to BETA: a loop */
used = 0; out[0] = 0;
v4_dump_node(&node, &im, 0, 0u, capture, 0);
CHECK(has("return stack depth=4"), "the dump still ends: %s", out);
printf("test_dump: %d checks, %d failures\n", checks, failures);
return failures ? 1 : 0;
}