CI's Acceptance fails on aarch64 at BIRTH on every head since 58d0e887.
The birth takes more than 300 s there and does not reproduce here: about
11 s, unsigned. The heartbeat spiral was ruled out by an instrumented
scratch kernel.
Captain Bob ruled "Diagnose on CI". On a failed check, the session script
asks QEMU's monitor for the registers three times, a second apart, and
prints the program counters, so CI's next failure shows where the kernel
is and whether it moves. A monitor that does not answer does not stop
the run.
Test first: scripts/test_qemu_session_snapshot.sh, 5 checks; 3 failed
before. The register text was checked on real, paused QEMU for aarch64,
riscv64 and x86.
The evidence and the measurements are in logs/20261119-ci-snapshot. The
local aarch64 timing run is in logs/20261118-235613.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
21 lines
1.3 KiB
Plaintext
21 lines
1.3 KiB
Plaintext
The CI aarch64 BIRTH stall, 2026-11-18/19 (Captain Bob: "Measure, then fix"; then "Diagnose on CI").
|
|
|
|
CI's Acceptance has failed on aarch64 at "BIRTH: node 11 born from Hera" on every head since 58d0e887
|
|
(jobs 828, 835, 840, 850): the birth does not finish within the session's 300 s wait. POST passes there.
|
|
|
|
Here it does not reproduce:
|
|
- host mesh boot: a birth is 0.045 s and 374,589 fabric steps (nucleus 7,515 cells);
|
|
- local aarch64 boots (unsigned): the birth in about 11 s, background or foreground
|
|
(local-aarch64-birth-timeline.txt; logs/20261118-235613/);
|
|
- an instrumented scratch kernel (counters in the mesh's heart()): one ticking node, the heartbeat at
|
|
most 0x203 ticks behind once at the prompt and then keeping up -- no catch-up spiral.
|
|
|
|
The one known difference: CI signs its capsules and checks each signature at open; a birth checks one
|
|
(forth79.4th), about 7 s on CI's aarch64 by its boot's timing -- not 300 s.
|
|
|
|
So qemu_session.py now prints where the CPUs are when a check fails (cpu_snapshot: the monitor's
|
|
registers three times, a second apart), and CI's next failure will show it.
|
|
Test first: scripts/test_qemu_session_snapshot.sh (test-before.log: 3 failures; test-after.log: PASS).
|
|
The monitor's register text was checked on real QEMU, paused with no kernel: aarch64 "PC=", riscv64
|
|
"pc", x86 "EIP"/"RIP" -- all matched.
|