Fix two catastrophically heavy workloads found running the real campaign
The full 360-trial campaign was launched, then killed after 4h39m of CPU time with zero trials completed -- still stuck on the very first touch of the very first trial. Root cause, quantified from the workload's own source, not estimated: RUN-CHAOS5 (workload-5.4th) is 1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE LOOP, multiplying out to ~2.4 trillion word executions for one call -- ~495 days at the observed rate. Never designed to be called to completion as a single touch in a repeatedly-birthed worker. A second problem found computing the fix rather than discovering it mid-run again: workload-1.4th's own birth-time self-execution (20 SQUARE-WAVE + 1500 SQUARE-BURST + 100000x MICRO-BURST, all three run automatically at capsule load) sums to ~875 million words -- ~4.3 hours just to birth one worker -- and worker index 1 always maps to it in heterogeneous mode, landing in half the campaign's cells regardless of concurrency level. Fix: two new capsules, workload-1-lite.4th and workload-5-lite.4th, carrying the same word bodies verbatim but a bounded self-execution tail, comparable in scale to the campaign's other workloads. The originals are untouched; only multiuser-doe.4th's own WL-CAPSULE/ WL-ENTRY index 1 and 5 mappings were repointed. Verified live before relaunching a third time: the actual worst case in isolation (5 0 998 MU-RUN-TRIAL, concurrency=8 heterogeneous, includes both fixed workloads plus RUN-OMNI) completed in ~20 minutes wall-clock, Hera stayed healthy throughout. Clean 3-arch qemu boot. Reps reduced 60 -> 30/cell (Bob's call after seeing the real per-trial cost) -- still matches the project's "rule of 3's" DoE convention (same as ACL-RWT's own 30 reps). 6 cfgs x 30 reps = 180 trials. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BWpNjdwPtFLuVLaAq44L9K
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
29f91f9531
commit
94215e8b47
+42
@@ -5090,3 +5090,45 @@ workloads like `RUN-CHAOS5`'s 1,000,000 iterations) -- a genuinely long-running
|
||||
TCG, deliberately not kicked off without explicit confirmation first. The WIREBIND-automation
|
||||
fixed arm also remains unbuilt.
|
||||
|
||||
### XXXIII.7 -- Two workloads too heavy to touch as-is, found live at real cost
|
||||
|
||||
**Approved and launched, then killed after finding a real problem the design review never
|
||||
caught:** the full campaign ran for 4h39m of CPU time without completing even the first
|
||||
touch of the first trial. Root cause quantified directly from the workload's own source
|
||||
(`workload-5.4th`), not estimated: `RUN-CHAOS5` is `1000000 0 DO CHAOS-FIELD CHAOS-RIPPLE
|
||||
LOOP`, and `CHAOS-FIELD` alone is `24 0 DO CHAOS-BLOCK LOOP` with `CHAOS-BLOCK` itself ~100K
|
||||
word executions -- multiplying out to **~2.4 trillion word executions for one call**. At the
|
||||
observed rate (~56,300 words/sec), that's roughly 495 days for a single touch. This workload
|
||||
was never designed to be called to completion as one touch in a repeatedly-birthed-and-killed
|
||||
worker; whatever its role in the original single-VM L8 DoE, it doesn't fit this campaign's
|
||||
shape at all.
|
||||
|
||||
**A second, related problem found computing the fix, not by running it first:** `workload-
|
||||
1.4th`'s own birth-time self-execution (`20 SQUARE-WAVE` + `1500 SQUARE-BURST` + `100000x
|
||||
MICRO-BURST`, all three run automatically at capsule load) sums to roughly **875 million word
|
||||
executions -- ~4.3 hours just to birth one worker**, and worker index 1 always maps to this
|
||||
capsule in heterogeneous mode, so the cost lands in half the campaign's cells regardless of
|
||||
concurrency level. Caught by computing the real per-workload word counts from every capsule's
|
||||
own source before relaunching a third time, not by discovering it mid-run again.
|
||||
|
||||
**Fix, matching the pattern Captain Bob had already authorized ("workloads are not carved in
|
||||
stone, create new extended workloads where needed"):** two new capsules,
|
||||
`workload-5-lite.4th` and `workload-1-lite.4th`, carrying the *same* word bodies verbatim
|
||||
(`CHAOS-BLOCK`/`CHAOS-FIELD`; `COMPUTE`/`BURST`/`MICRO-BURST`) but a bounded self-execution
|
||||
tail (`CHAOS-FIELD` once; `1000x MICRO-BURST`) -- comparable in scale to the campaign's other
|
||||
workloads (~2-30M words), not the original files' unbounded repeat counts. The originals
|
||||
(`workload-1.4th`, `workload-5.4th`) are untouched, still real, still available for whatever
|
||||
their original context needs -- only `multiuser-doe.4th`'s own `WL-CAPSULE`/`WL-ENTRY` index
|
||||
1 and 5 mappings were repointed to the lite variants.
|
||||
|
||||
**Verified live before relaunching the full campaign a third time:** ran the actual worst
|
||||
case in isolation -- `5 0 998 MU-RUN-TRIAL` (cfg=5: concurrency=8, heterogeneous, so it
|
||||
includes both previously-catastrophic workloads plus `RUN-OMNI`'s own ~30M words) -- and it
|
||||
completed in ~20 minutes wall-clock (`MU-TRIAL run=998 cfg=5 rep=0 nw=8 mode=1 fail=0`), Hera
|
||||
stayed healthy throughout. Clean 3-arch qemu boot on the real committed capsule changes.
|
||||
|
||||
**Reps reduced from 60 to 30 per cell**, Captain Bob's own call after seeing the real per-
|
||||
trial cost -- still matches the project's "rule of 3's" DoE convention exactly (the same 30
|
||||
reps as the ACL-RWT campaign). 6 cfgs x 30 reps = 180 main-block trials, roughly half the
|
||||
prior 360-trial estimate.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user