admin/LithosAnanake
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such

Browse Source 
docs/CLAUDE.md Phase 2, 2026-10-14.  All moves by git mv; every archived
file carries a header saying what it is, why, and where to look instead.

- docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL
  design, kernel args, hosted acceptance test, dictionary, primitives,
  ontology, L8 table -- previously spread over docs/, docs/lithosananke/
  and docs/03-architecture/.
- docs/working/archive/historical/: finished work (session plan, test-gap
  audit, birthing plan, fix write-ups, the January architecture draft).
- docs/working/archive/superseded/: the Hermes-VM Tripod docs and July
  Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working
  doc already flagged superseded or obsolete.
- docs/working/archive/dead/: the hal_* target contract (never adopted),
  L4Re, metrics-and-knobs.
- Papers and proposals into docs/working/papers/.
- References rewritten across the repo (not logs/, not docs/patent/);
  CLAUDE.md's documentation section rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
2026-10-14 16:03:56 -04:00
co-authored by Claude Opus 5.5
parent 1a4b163861
commit 7aed588f78
116 changed files with 305 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/working/experiments/02-experiments/L8_WIRING_PLAN.md
-355
View File
@@ -1,355 +0,0 @@
<!-- Moved from docs/02-experiments/L8_WIRING_PLAN.md to docs/working/experiments/02-experiments/L8_WIRING_PLAN.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** The wiring plan was executed; superseded by
> `L8_WIRING_IMPLEMENTATION_SUMMARY.md` and `L8_WIRING_CODE_REVIEW_GUIDE.md`. Kept as a
> record of the plan.
# L8 Control Wiring Implementation Plan
**Status**: Infrastructure exists but execution paths are not gated
**Date**: 2025-12-05
**Issue**: L8 Jacquard mode selector computes 4-bit control signals (L2/L3/L5/L6) but feedback loops ignore them
---
## Executive Summary
The L8 Steady State Machine (SSM) infrastructure is **90% complete** but the final 10% is critical:
✅ **Complete**:
- L8 state machine (`ssm_jacquard.c`) computes 4-bit mode (C0-C15) based on entropy/CV/temporal metrics
- `ssm_apply_mode()` sets L2/L3/L5/L6 flags in `vm->ssm_config`
- `vm_heartbeat_update_l8()` updates L8 state every heartbeat cycle (lines 993-1045 in `vm.c`)
- Mode transitions logged with debug info
❌ **Missing**:
- Feedback loop execution paths don't check `ssm_config` flags before running
- L2/L3/L5/L6 run unconditionally despite L8 setting mode bits
- Result: L8 is a "supervisor without authority"
---
## Current Architecture (From DoE Analysis)
From `Makefile:112-127` and `ssm_jacquard.h:32-51`:
| Loop | Name | Status | Control |
|------|------|--------|---------|
| L1 | Heat Tracking | ALWAYS OFF | Disabled (harmful in 86% of configs) |
| L2 | Rolling Window | **NEEDS GATING** | L8 bit 3 (entropy-driven) |
| L3 | Linear Decay | **NEEDS GATING** | L8 bit 2 (temporal locality) |
| L4 | Pipelining Metrics | ALWAYS OFF | Disabled (harmful in 100% of configs) |
| L5 | Window Inference | **NEEDS GATING** | L8 bit 1 (CV-driven) |
| L6 | Decay Inference | **NEEDS GATING** | L8 bit 0 (CV + temporal) |
| L7 | Adaptive Heartrate | ALWAYS ON | Enabled (beneficial in 71% of configs) |
| L8 | Jacquard Selector | **IMPLEMENTED** | 4-bit mode controller (C0-C15) |
**Key Insight**: L1 and L4 are compile-time disabled. L7 is always-on. **L2/L3/L5/L6 need runtime gating** based on L8 mode bits.
---
## Required Changes (4 Locations)
### 1. L2: Rolling Window Recording (src/rolling_window_of_truth.c)
**Current Code** (`rolling_window_record_execution()` line 181):
```c
int rolling_window_record_execution(RollingWindowOfTruth* window, uint32_t word_id)
{
if (!window || !window->execution_history)
return -1;
window->execution_history[window->window_pos] = word_id;
// ... always executes
}
```
**Required Change**:
Add check at function entry or call site. Two options:
**Option A** (Preferred - Clean separation):
Modify call site in `src/vm.c:1652`:
```c
// Before (line 1652):
rolling_window_record_execution(&vm->rolling_window, word_id);
// After:
if (vm->ssm_config && ((ssm_config_t*)vm->ssm_config)->L2_rolling_window) {
rolling_window_record_execution(&vm->rolling_window, word_id);
}
```
**Option B** (Alternative - Gate inside function):
Modify `rolling_window_record_execution()` to accept VM pointer and check internally.
**Files to modify**:
- `src/vm.c:1652` - Call site in `execute_word()`
- `src/rolling_window_of_truth.c:181` - Consider adding VM* parameter for internal check
- `include/rolling_window_of_truth.h` - Update function signature if Option B chosen
---
### 2. L3: Linear Decay Application (src/physics_metadata.c)
**Current Code** (line 176):
```c
/* L8 FINAL INTEGRATION: L3 linear decay is always-on */
void physics_metadata_apply_linear_decay(DictEntry *entry, uint64_t elapsed_ns, VM *vm) {
if (!entry || !vm) {
return;
}
/* Frozen words do not decay */
if (entry->flags & WORD_FROZEN) {
return;
}
/* Don't bother with insignificant time intervals */
if (elapsed_ns < DECAY_MIN_INTERVAL) {
return;
}
// ... decay logic always executes
}
```
**Required Change**:
Add L3 gate check after frozen/interval checks:
```c
/* L8 FINAL INTEGRATION: L3 linear decay - runtime gated by Jacquard */
void physics_metadata_apply_linear_decay(DictEntry *entry, uint64_t elapsed_ns, VM *vm) {
if (!entry || !vm) {
return;
}
/* Frozen words do not decay */
if (entry->flags & WORD_FROZEN) {
return;
}
/* Don't bother with insignificant time intervals */
if (elapsed_ns < DECAY_MIN_INTERVAL) {
return;
}
/* L8 GATE: Check if L3 (linear decay) is enabled by Jacquard mode */
if (vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)vm->ssm_config;
if (!config->L3_linear_decay) {
return; /* L3 disabled by L8 mode selector */
}
}
// ... existing decay logic
}
```
**Files to modify**:
- `src/physics_metadata.c:176` - Add L3 gate check (line ~188-193)
- Comment update at line 175 to reflect runtime gating
**Call sites** (no changes needed, gate is internal):
- `src/vm.c:737` - `vm_tick_apply_background_decay()` (heartbeat loop)
- Other call sites inherit the gate
---
### 3. L5: Window Width Inference (src/inference_engine.c)
**Current Code** (line 538):
```c
/* === PHASE 2C: Window Width Inference === */
/* L8 FINAL INTEGRATION: Loop #5 always-on */
uint32_t inferred_width = find_variance_inflection(
trajectory,
traj_len,
current_variance
);
```
**Required Change**:
Gate inference execution with conditional block:
```c
/* === PHASE 2C: Window Width Inference === */
/* L8 FINAL INTEGRATION: Loop #5 runtime-gated by Jacquard */
uint32_t inferred_width = outputs->adaptive_window_width; /* Keep previous value as default */
if (inputs->vm && inputs->vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)inputs->vm->ssm_config;
if (config->L5_window_inference) {
/* L5 enabled - run inference */
inferred_width = find_variance_inflection(
trajectory,
traj_len,
current_variance
);
}
/* else: L5 disabled, keep previous adaptive_window_width */
}
```
**Files to modify**:
- `src/inference_engine.c:538-544` - Add L5 conditional gate
- Comment update at line 539 to reflect runtime gating
---
### 4. L6: Decay Slope Inference (src/inference_engine.c)
**Current Code** (line 546):
```c
/* === PHASE 2D: Decay Slope Inference === */
/* L8 FINAL INTEGRATION: Loop #6 always-on */
uint64_t inferred_slope = infer_decay_slope_q48(trajectory, traj_len);
```
**Required Change**:
Gate inference execution with conditional block:
```c
/* === PHASE 2D: Decay Slope Inference === */
/* L8 FINAL INTEGRATION: Loop #6 runtime-gated by Jacquard */
uint64_t inferred_slope = outputs->adaptive_decay_slope; /* Keep previous value as default */
if (inputs->vm && inputs->vm->ssm_config) {
ssm_config_t *config = (ssm_config_t*)inputs->vm->ssm_config;
if (config->L6_decay_inference) {
/* L6 enabled - run inference */
inferred_slope = infer_decay_slope_q48(trajectory, traj_len);
}
/* else: L6 disabled, keep previous adaptive_decay_slope */
}
```
**Files to modify**:
- `src/inference_engine.c:546-548` - Add L6 conditional gate
- Comment update at line 547 to reflect runtime gating
---
## Implementation Order (Recommended)
1. **L3 (Decay)** - Simplest, internal to one function
- Modify `src/physics_metadata.c:176-220`
- Add gate check after line 188
- Test with mode C0 (all off) vs C4 (L3 on)
2. **L5 (Window Inference)** - Medium complexity
- Modify `src/inference_engine.c:538-544`
- Preserve previous value when disabled
- Test early-exit behavior with mode C0 vs C2
3. **L6 (Decay Inference)** - Similar to L5
- Modify `src/inference_engine.c:546-548`
- Preserve previous value when disabled
- Test with mode C0 vs C1
4. **L2 (Rolling Window)** - Most complex (data structure)
- Decide on Option A (call-site gate) vs Option B (internal gate)
- Modify call site `src/vm.c:1652` OR function signature
- Test with mode C0 vs C8 (high entropy)
---
## Testing Strategy
### Unit Tests
For each loop, verify:
1. **Mode C0 (0000)**: All loops disabled → function returns early or skips logic
2. **Single-bit modes**: Enable one loop at a time (C1, C2, C4, C8)
3. **Top 5% modes**: C4, C7, C9, C11, C12 work correctly
### Integration Tests
1. **Mode transitions**: L8 changes mode → loops respond correctly
2. **Hysteresis**: Mode doesn't flap rapidly (5-tick delay)
3. **Metrics**: Verify entropy/CV/temporal drive correct mode selection
### Regression Tests
1. Run existing 936+ test suite with different L8 modes
2. Verify deterministic behavior (no variance across runs)
3. Check performance matches DoE predictions
---
## Validation Checklist
After implementation, verify:
- [ ] L2: `rolling_window_record_execution()` only executes when L2 bit set
- [ ] L3: `physics_metadata_apply_linear_decay()` returns early when L3 bit clear
- [ ] L5: `find_variance_inflection()` only called when L5 bit set
- [ ] L6: `infer_decay_slope_q48()` only called when L6 bit set
- [ ] L8: Mode transitions logged in debug output
- [ ] All existing tests pass with default mode (C0 or C12)
- [ ] DoE experiment can reproduce 128 configurations via L8 mode forcing
---
## Impact Assessment
**Lines of code changed**: ~40-60 lines across 3 files
**Risk**: Low (adding guards, not changing logic)
**Performance**: Neutral to positive (skips work when disabled)
**Correctness**: High confidence (L8 already validated via DoE)
---
## Open Questions
1. **Default mode**: Should VM initialize with C0 (minimal) or C12 (DoE optimal)?
- Current: `ssm_l8_init(..., SSM_MODE_C0)` in `vm.c:387`
- Top 5%: C4, C7, C9, C11, C12
- Recommendation: Start C0, let L8 adapt
2. **Mode override**: Should FORTH words exist to force L8 mode?
- `L8-MODE!` to set mode manually
- `L8-MODE@` to query current mode
- Useful for debugging/benchmarking
3. **Metrics exposure**: Should heartbeat CSV export include L8 mode?
- Currently exports: executions, heat, variance, slope
- Add: `l8_mode` column (0-15 integer)
---
## Success Criteria
**Definition of Done**:
1. All 4 feedback loops check `ssm_config` flags before executing
2. L8 mode selector controls L2/L3/L5/L6 runtime behavior
3. Mode transitions visible in debug logs
4. Existing test suite passes with default L8 mode
5. Can reproduce DoE configurations by forcing L8 modes 0-15
**Verification**:
```bash
# Build with L8 control wired
make clean && make fastest
# Test mode C0 (all off) - should be fastest, least adaptive
./build/amd64/fastest/starforth --benchmark 1000
# Test mode C15 (all on) - should be slowest, most adaptive
# (requires word to set L8 mode or environment variable)
# Run full test suite
make test
```
---
## Next Steps
**Before coding**:
1. Review this plan with stakeholders
2. Decide on L2 implementation strategy (Option A vs B)
3. Choose default L8 mode (C0 vs C12)
**During implementation**:
1. Follow order: L3 → L5 → L6 → L2
2. Test each loop individually before moving to next
3. Update comments to reflect runtime gating
**After completion**:
1. Run full DoE validation (128 configs × 300 reps)
2. Verify 0% algorithmic variance maintained
3. Document L8 control API for future developers
docs/working/experiments/02-experiments/factorial-doe/guide.md
-262
View File
@@ -1,262 +0,0 @@
<!-- Moved from docs/02-experiments/factorial-doe/guide.md to docs/working/experiments/02-experiments/factorial-doe/guide.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes the retired per-loop Makefile-toggle
> architecture, replaced by the L8 Jacquard runtime mode selector. Kept for historical record.
# StarForth Complete 2^6 Factorial Design of Experiments
## Overview
**One-and-done comprehensive factorial testing** - collect all data systematically, analyze separately.
This guide describes how to run a **complete 2^6 factorial design of experiments** across all 6 feedback loops, yielding:
- **64 unique configurations** (all binary combinations)
- **1,920+ runs** (30 samples per configuration)
- **Single unified dataset** for factorial analysis
## The Why: Factorial DoE
Instead of incrementally tuning parameters, we collect **comprehensive experimental data** in one go:
### Traditional Approach (Old)
- Run baseline
- Run with Loop #1
- Run with Loops #1-2
- Run with Loops #1-3
- ... (sequential, incremental)
- **Problem:** Can't see interactions, trade-offs, or surprises
### Factorial Approach (New)
- Run ALL 2^6 = 64 combinations in randomized order
- Collect all data at once
- Analyze factorial effects, interactions, and main effects
- **Advantage:** Complete picture of how loops interact
## The 6 Feedback Loops
Each loop is a binary toggle in the Makefile:
| Loop | Makefile Flag | Purpose |
|------|---|---|
| **#1** | `ENABLE_LOOP_1_HEAT_TRACKING` | Count word execution frequency |
| **#2** | `ENABLE_LOOP_2_ROLLING_WINDOW` | Capture execution history (circular buffer) |
| **#3** | `ENABLE_LOOP_3_LINEAR_DECAY` | Age words by exponential decay over time |
| **#4** | `ENABLE_LOOP_4_PIPELINING_METRICS` | Track word-to-word transitions |
| **#5** | `ENABLE_LOOP_5_WINDOW_INFERENCE` | Infer optimal window width (Levene's test) |
| **#6** | `ENABLE_LOOP_6_DECAY_INFERENCE` | Infer decay slope via linear regression |
## Configuration Naming
Each of 64 configurations is named by its loop status:
```
L1_L2_L3_L4_L5_L6
Where each digit is 0 (OFF) or 1 (ON)
Examples:
000000 = Baseline (all loops OFF - pure FORTH-79)
000001 = Only heat tracking
111110 = All except decay inference
111111 = All loops ON (likely optimal)
101010 = Alternating pattern (for research)
```
## Quick Start
### Minimal Test (validate script)
```bash
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
# 64 builds × 1 run = 64 total runs
# Runtime: ~30-45 minutes
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/TEST_RUN/
```
### Standard Comprehensive Run
```bash
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FACTORIAL
# 64 builds × 30 runs = 1,920 total runs
# Runtime: ~2-4 hours
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FACTORIAL/
```
### High-Precision Run
```bash
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_DOE
# 64 builds × 100 runs = 6,400 total runs
# Runtime: ~6-12 hours
# Output: /home/rajames/CLionProjects/StarForth-DoE/experiments/HIGH_PRECISION_DOE/
```
## Output Structure
```
/home/rajames/CLionProjects/StarForth-DoE/experiments/<EXPERIMENT_LABEL>/
├── experiment_results.csv # All measurements (1920+ rows)
├── experiment_summary.txt # Timing, metadata, statistics
├── configuration_manifest.txt # All 64 configs documented
├── test_matrix.txt # Randomized execution order
└── run_logs/
├── 000000_run_1.log
├── 000000_run_2.log
├── 000001_run_1.log
├── ...
└── 111111_run_30.log # 1,920 total log files
```
## CSV Schema
Each row in `experiment_results.csv` contains:
```
timestamp,configuration,run_number,
<35+ metrics from test harness>,
enable_loop_1_heat_tracking,
enable_loop_2_rolling_window,
enable_loop_3_linear_decay,
enable_loop_4_pipelining_metrics,
enable_loop_5_window_inference,
enable_loop_6_decay_inference
```
This lets you correlate every metric against every loop combination.
## Analysis Strategy
After collection, analyze in the separate `StarForth-DoE` repository:
### Step 1: Load Data
```python
import pandas as pd
df = pd.read_csv("experiment_results.csv")
print(df.shape) # Should be (1920, 38+)
print(df.columns)
```
### Step 2: Separate Metrics from Factors
```python
factor_cols = ['enable_loop_1_heat_tracking', ..., 'enable_loop_6_decay_inference']
metric_cols = ['total_lookups', 'cache_hit_percent', ..., 'vm_workload_duration_ns_q48']
factors = df[factor_cols]
metrics = df[metric_cols]
```
### Step 3: Factorial Analysis
```python
# Main effects: What's the average change when Loop X goes from 0→1?
main_effects = {}
for loop in factor_cols:
main_effects[loop] = metrics[factors[loop]==1].mean() - metrics[factors[loop]==0].mean()
# Interactions: Do loops work better/worse together?
# Can use statsmodels.formula.api for ANOVA/regression
# Optimal subset: Which combinations of loops give best throughput?
best_configs = df.nlargest(10, 'vm_workload_duration_ns_q48')
```
## Key Design Choices
### 1. **Rebuild Every Pass** ✓
- Each configuration requires a `make clean && make`
- Ensures clean, isolated state
- No cross-contamination
- True ground truth for each variant
### 2. **Randomized Execution**
- All 1,920 runs randomized into single matrix (not 64 separate runs)
- Prevents thermal ramp, temporal bias, or systematic trends
- DoE principle: randomization eliminates confounds
### 3. **Full Factorial** (not fractional)
- Testing ALL 2^6 = 64 combinations
- Can detect unexpected interactions
- Most expensive, most informative
### 4. **One-and-Done Collection**
- Single experiment, all data
- No progressive tuning during collection
- Separates measurement from analysis (good practice)
## Expected Results
### Hypothetical Output
Assuming typical performance distribution:
```
Configuration | Avg Time (ns) | Speed vs Baseline
─────────────────────────────────────────────────
000000 (base) | 10,000,000 | 1.0x
000001 (L1) | 9,950,000 | 1.005x
111111 (all) | 8,500,000 | 1.176x ← Likely best
```
### Expected Patterns
1. **Loop #1 (heat tracking)**: ~2-5% improvement alone
2. **Loop #2 (rolling window)**: ~1-3% improvement (depends on L1)
3. **Loops #3-6**: Increasingly synergistic (work better together)
4. **Interactions**: Loop #5 + #6 likely synergistic (inference loops)
## Troubleshooting
### Build Fails for a Configuration
```bash
# Check specific config build log
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/<LABEL>/run_logs/build_<CONFIG>.log
```
### Test Harness Crashes on Some Configs
- Check if loop combination is invalid (e.g., Loop #5 without Loop #2)
- Verify Makefile flags are correctly parsed
- See: `include/vm.h` for interdependencies
### Results Look Suspicious
- Check `test_matrix.txt` - ensure randomization is present
- Verify CSV row counts match expected total
- Check for missing runs in log directory
## Performance Notes
### Timing Estimates
- **Minimal (1 run/config)**: 30-45 minutes
- **Standard (30 runs/config)**: 2-4 hours
- **High-precision (100 runs/config)**: 6-12 hours
- **Depends on:** CPU speed, disk I/O, test harness overhead
### Optimization
- Run overnight for high-precision experiments
- Use `screen` or `tmux` to keep session alive
- Monitor disk space (~100 MB for results + logs)
## Next Steps
1. **Run the experiment:**
```bash
./scripts/run_factorial_doe.sh 2025_11_19_COMPLETE_FACTORIAL
```
2. **Monitor progress:**
```bash
tail -f /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL/experiment_summary.txt
```
3. **Move to analysis repo** (once complete):
```bash
cp -r /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_COMPLETE_FACTORIAL \
/home/rajames/CLionProjects/StarForth-DoE-Analysis/data/
cd /home/rajames/CLionProjects/StarForth-DoE-Analysis
python3 analyze_factorial.py data/2025_11_19_COMPLETE_FACTORIAL/experiment_results.csv
```
4. **Generate reports:**
- Main effects plots
- Interaction matrices
- Pareto optimal configurations
- Recommendations for production builds
---
**Version:** 1.0
**Last Updated:** 2025-11-19
**Author:** Claude Code (with Robert A. James)
docs/working/experiments/02-experiments/factorial-doe/incompatible-configs.md
-401
View File
@@ -1,401 +0,0 @@
<!-- Moved from docs/02-experiments/factorial-doe/incompatible-configs.md to docs/working/experiments/02-experiments/factorial-doe/incompatible-configs.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Build-time incompatibility handling for the retired 2^6
> toggle experiment, replaced by the L8 Jacquard mode selector. Kept for historical record.
# Handling Incompatible Configurations in Factorial DoE
## Overview
Your intuition was correct: **some loop combinations can't coexist**. Rather than failing the entire experiment, the updated script treats incompatibility as valuable data.
This document explains how the script detects and records incompatible configurations during the factorial experiment.
---
## What is an Incompatible Configuration?
An incompatible configuration is a combination of feedback loop toggles that:
1. **Fails to compile** - Makefile target produces build errors
2. **Causes StarForth to crash** - Binary exits abnormally (segfault, signal, etc.)
3. **Produces invalid output** - Test harness runs but produces no metrics
Examples (hypothetical):
- Loop 5 (window inference) without Loop 2 (rolling window) = no data to infer from
- Loop 6 (decay inference) without Loop 3 (linear decay) = no slope to measure
- Certain combinations trigger memory conflicts or uninitialized states
---
## How the Script Handles It
### Detection Strategy
**Build Phase:**
```
Configuration X attempted
├─ make TARGET=fastest ENABLE_LOOP_1=1 ...ENABLE_LOOP_6=0
├─ Check exit code
├─ If success: mark as "OK", proceed to testing
└─ If failure: mark as "BUILD_FAILED", skip all runs for this config
```
**Execution Phase:**
```
Run configuration X, iteration N
├─ Execute: ./starforth --doe-experiment
├─ Check exit code
├─ Try to extract metrics from output
├─ If success: mark as "OK", record metrics
└─ If failure: mark as "CRASH", record incompatibility
```
### CSV Recording
Incompatible configurations are recorded with status flags:
**Successful run (viable config):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:30:45,1_1_0_1_0_0,5,OK,OK,
<35+ metrics>,1,1,0,1,0,0
```
**Build failure (incompatible):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:35:20,0_1_1_1_1_0,1,BUILD_FAILED,<skipped>,
<empty>,0,1,1,1,1,0
```
**Runtime crash (incompatible):**
```
timestamp,configuration,run_number,build_status,run_status,
2025-11-19T10:40:15,1_0_0_1_1_1,3,OK,CRASH,
<empty>,1,0,0,1,1,1
```
### Experiment Flow
The experiment **continues** when incompatibility is detected:
```
Run 1: Config A (OK) → Test completes, metrics recorded
Run 2: Config B (BUILD_FAILED) → Build fails, skip all runs for B, continue
Run 3: Config A (OK) → Test completes, metrics recorded
Run 4: Config C (CRASH) → Crashes, record as incompatible, continue
...
Run N: Config D (OK) → Test completes, metrics recorded
SUMMARY:
✓ Successful runs: 847/1920
✗ Failed configs: 17 (incompatible combinations detected)
✓ Experiment completed despite incompatibilities
```
---
## CSV Schema
### Two New Columns
| Column | Values | Meaning |
|--------|--------|---------|
| `build_status` | OK, BUILD_FAILED | Did Makefile compilation succeed? |
| `run_status` | OK, CRASH | Did StarForth execute without crashing? |
### Analysis Filter Rules
```python
import pandas as pd
df = pd.read_csv('experiment_results.csv')
# Only viable configurations
viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')]
# Incompatible configs (at compile time)
build_failures = df[df['build_status'] == 'BUILD_FAILED']
# Incompatible configs (at runtime)
runtime_crashes = df[df['run_status'] == 'CRASH']
# All incompatible
incompatible = df[(df['build_status'] != 'OK') | (df['run_status'] != 'OK')]
```
---
## Interpreting Results
### Experiment Summary
After completion, you'll see:
```
═══════════════════════════════════════════════════════════
EXPERIMENT COMPLETE (with incompatibility detection)
═══════════════════════════════════════════════════════════
✓ Successful runs: 1847/1920
✓ Failed configurations (incompatible): 17
→ Incompatible configurations detected:
✗ 0_1_1_1_1_0 (BUILD_FAILED)
✗ 1_0_0_1_1_1 (CRASH)
✗ 0_1_0_1_1_1 (CRASH)
... (total 17)
→ These configs are marked in CSV with build_status or run_status
→ They represent configurations where loops cannot coexist without conflict
```
### What This Means
- **1847 successful runs** = viable, measurable configurations
- **17 incompatible** = design space constraints discovered
- **Experiment completed** = data collection succeeded despite incompatibilities
- **CSV includes both** = full picture of parameter space
---
## Analysis Examples
### Example 1: Summary by Viability
```python
import pandas as pd
df = pd.read_csv('experiment_results.csv')
# Count by status
viable = len(df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')])
build_fail = len(df[df['build_status'] == 'BUILD_FAILED'].drop_duplicates('configuration'))
runtime_crash = len(df[df['run_status'] == 'CRASH'].drop_duplicates('configuration'))
print(f"Viable configurations: {viable} runs")
print(f"Build failures: {build_fail} configurations")
print(f"Runtime crashes: {runtime_crash} configurations")
print(f"Total tested: {64} configurations, {64*30} runs")
print(f"Success rate: {viable}/{len(df)} = {100*viable/len(df):.1f}%")
```
Expected output:
```
Viable configurations: 1847 runs
Build failures: 5 configurations
Runtime crashes: 12 configurations
Total tested: 64 configurations, 1920 runs
Success rate: 1847/1920 = 96.2%
```
### Example 2: Identify Problematic Loop Combinations
```python
# Find configs that always crash
crashes = df[df['run_status'] == 'CRASH'].drop_duplicates('configuration')
print("Configurations that crash:")
for config in crashes['configuration']:
parts = config.split('_')
loops_on = [f"Loop{i+1}" for i, bit in enumerate(parts) if bit == '1']
print(f" {config}: {', '.join(loops_on) or 'None'}")
```
Expected output:
```
Configurations that crash:
0_1_1_1_1_0: Loop2, Loop3, Loop4, Loop5
1_0_0_1_1_1: Loop1, Loop4, Loop5, Loop6
0_1_0_1_1_1: Loop2, Loop4, Loop5, Loop6
... etc
```
**Insight:** Loop 5 (window inference) appears in many crashes → suggests window inference has prerequisites or conflicts.
### Example 3: Feasibility Matrix
```python
# Which loop combinations are feasible?
df_viable = df[(df['build_status'] == 'OK') & (df['run_status'] == 'OK')].copy()
# For each config, mark as viable (1) or not (0)
feasible = {}
for config in df['configuration'].unique():
is_viable = len(df_viable[df_viable['configuration'] == config]) > 0
feasible[config] = 1 if is_viable else 0
# Show as binary matrix
print("Feasibility (1=viable, 0=incompatible):")
for i in range(64):
config = f"{i%2}_{(i//2)%2}_{(i//4)%2}_{(i//8)%2}_{(i//16)%2}_{(i//32)%2}"
print(f"{config}: {'✓' if feasible[config] else '✗'}")
```
Result: Feasibility map of 64 configurations for maintenance and design decisions.
---
## Maintenance Insights
### Why Some Combinations Are Incompatible
Common reasons:
1. **Missing Prerequisites:**
- Loop 5 (window inference) needs Loop 2 (rolling window) data
- Loop 6 (decay inference) needs Loop 3 (linear decay) to measure
2. **Memory/Resource Conflicts:**
- Multiple loops allocating same memory region
- Circular dependencies in initialization
3. **Semantic Contradictions:**
- Loop X disables feature that Loop Y requires
- Incompatible macro definitions
4. **Bugs in Specific Combinations:**
- Race conditions appearing only with certain feature combinations
- Uninitialized state when paths not taken
### Using This Information
**For Future Development:**
```
Findings: Loop 5 incompatible without Loop 2
→ Check: Does window inference code assume rolling window exists?
→ Action: Either add guard (if !LOOP_5 without LOOP_2) or document prerequisite
→ Test: Add compile-time or runtime check
```
**For Optimization:**
```
Result: All configs with Loop 6 crash if Loop 1 is OFF
→ Issue: Decay inference needs heat tracking for slope calculation
→ Fix: Add LOOP_1_REQUIRED_BY_LOOP_6 guard in code
→ Verify: Re-run factorial (expect fewer crashes)
```
**For Production:**
```
Feasible subset: {0_0_0_0_0_0, 1_1_0_0_0_0, 1_1_1_0_0_0, 1_1_1_1_1_1}
→ Recommendation: Use 1_1_1_1_1_1 (all compatible, best performance)
→ Fallback: Use 1_1_0_0_0_0 (compatible, simpler)
→ Baseline: Use 0_0_0_0_0_0 (no feedback, minimal overhead)
```
---
## Next Steps
1. **Run the experiment** to completion (incompatibilities are discovered organically)
2. **Analyze the CSV** to identify which configs are viable
3. **Examine incompatible configs** to understand why they fail
4. **Document prerequisites** (which loops require others)
5. **Fix or guard** the code to eliminate incompatibilities
6. **Re-run** experiment to validate fixes
---
## Technical Details
### Build Failure Detection
```bash
if make TARGET=fastest ENABLE_LOOP_1=1 ... > build_log 2>&1; then
# Build successful
BUILD_STATUS="OK"
else
# Build failed - mark incompatible
BUILD_STATUS="BUILD_FAILED"
# Skip all runs for this config
continue
fi
```
### Runtime Crash Detection
```bash
if "${BINARY}" --doe-experiment > run_log 2>&1; then
# Binary executed (exit code 0)
if extract_csv_row(run_log) succeeds:
RUN_STATUS="OK"
else:
RUN_STATUS="CRASH" # Binary ran but no output
else
# Binary crashed (non-zero exit)
RUN_STATUS="CRASH"
fi
```
### CSV Recording
```bash
# Viable run
printf '%s,%s,%s,%s,%s,%s,%s\n' \
"${timestamp}" "${config}" "${run_num}" \
"OK" "OK" \
"${csv_row}" "${flags}" >> results_run_01_2025_12_08.csv
# Incompatible run
printf '%s,%s,%s,%s,%s,%s\n' \
"${timestamp}" "${config}" "${run_num}" \
"${build_status}" "${run_status}" \
"${flags}" >> results_run_01_2025_12_08.csv
```
---
## Example: Real Incompatibility
**Scenario:** Loop 5 (window inference) without Loop 2 (rolling window)
```c
// Loop 5: Window Inference Code
void vm_tick_inference_engine(VM *vm) {
// Try to infer window width from rolling window
RollingWindowOfTruth *window = &vm->rolling_window; // LOOP_2 data
// BUG: If LOOP_2 is OFF, window is uninitialized!
uint32_t window_pos = window->window_pos; // CRASH: uninitialized read
}
```
**Symptom in CSV:**
```
Configuration 0_0_0_0_1_0 marked as CRASH
Loop1=0, Loop2=0, Loop3=0, Loop4=0, Loop5=1, Loop6=0
↑ ↑
Missing prerequisite Problematic loop
```
**Fix:**
```c
// Add guard in code
#if ENABLE_LOOP_5_WINDOW_INFERENCE
#if !ENABLE_LOOP_2_ROLLING_WINDOW
#error "Loop 5 requires Loop 2: window inference needs rolling window data"
#endif
#endif
```
**Result after fix:** Configuration 0_0_0_0_1_0 removed from viable set (prevented at compile time), reduces crash count.
---
## Summary
| Aspect | Approach |
|--------|----------|
| **Detection** | Build exit codes + runtime crash detection |
| **Recording** | New `build_status` and `run_status` columns |
| **Handling** | Skip incompatible config, continue experiment |
| **Impact** | Partial dataset (viable configs) + incompatibility map |
| **Analysis** | Filter by status, identify patterns |
| **Action** | Fix code, re-run, measure improvement |
This design enables discovering incompatibilities without sacrificing experimental continuity.
---
**Version:** 1.0
**Last Updated:** 2025-11-19
**Status:** Integrated into run_factorial_doe.sh
docs/working/experiments/02-experiments/factorial-doe/index.md
-268
View File
@@ -1,268 +0,0 @@
<!-- Moved from docs/02-experiments/factorial-doe/index.md to docs/working/experiments/02-experiments/factorial-doe/index.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Index for the retired 2^6 factorial toggle
> architecture, replaced by the L8 Jacquard mode selector. Kept for historical record.
# StarForth Complete 2^6 Factorial DoE - Documentation Index
## Overview
This is a **one-and-done comprehensive factorial design of experiments** to systematically test all 64 possible combinations of 6 feedback loops.
**Key insight:** Don't toggle back-and-forth. Instead, systematically test ALL 2^6 combinations in a single experiment, then analyze the results afterward.
## Documentation Structure
### 1. **START HERE** 📋
- **File:** `docs/QUICK_START_FACTORIAL_DoE.md`
- **Purpose:** Entry point for quick reference
- **Read time:** 5 minutes
- **Contains:**
- 3 command options (quick/standard/high-precision)
- Time estimates
- What to expect during execution
- Basic troubleshooting
### 2. **Complete Guide** 📚
- **File:** `docs/FACTORIAL_DoE_GUIDE.md`
- **Purpose:** Full design documentation and rationale
- **Read time:** 20-30 minutes
- **Contains:**
- Why factorial DoE (vs. incremental tuning)
- 6 feedback loop definitions
- Configuration naming scheme (0_0_0_0_0_0 → 1_1_1_1_1_1)
- Design choices explained
- Expected results and patterns
- Troubleshooting guide
- Performance notes
### 3. **Analysis Workflow** 🔍
- **File:** `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
- **Purpose:** End-to-end pipeline from collection to insights
- **Read time:** 15-20 minutes (or 1-2 hours if coding along)
- **Contains:**
- Phase 1: Data collection (run script)
- Phase 2: Data transfer (to analysis repo)
- Phase 3: Statistical analysis (main effects, interactions)
- Phase 4: Reporting (generating insights)
- Python code examples:
* Load data and validate
* Compute main effects (individual loop impacts)
* Detect interactions (loop synergies)
* Optimize configuration subset
* Visualization code
- Example report structure
- Checklist for completion
### 4. **Main Script** ⚙️
- **File:** `scripts/run_factorial_doe.sh`
- **Purpose:** Executable for running the entire experiment
- **Read time:** 10 minutes (code + comments)
- **Contains:**
- Generates all 64 configurations
- Rebuilds for each unique configuration
- Randomizes execution order (1,920+ runs)
- Collects metrics to CSV
- Full instrumentation and progress reporting
---
## Reading Guide by Use Case
### "I just want to run the experiment"
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
2. Execute: Choose command option (quick/standard/high-precision)
3. Wait: 30 min - 12 hours depending on option
4. Done!
### "I want to understand the design before running"
1. Read: `QUICK_START_FACTORIAL_DoE.md` (5 min)
2. Read: `FACTORIAL_DoE_GUIDE.md` (30 min)
3. Execute: Run with confidence
4. Analyze: Follow `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
### "I need to analyze the results"
1. Read: `QUICK_START_FACTORIAL_DoE.md` → understand naming
2. Read: `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` → statistical methods
3. Code along: Use Python examples in Analysis Workflow
4. Interpret: Reference `FACTORIAL_DoE_GUIDE.md` for expected patterns
### "I want complete understanding"
Read in order:
1. `QUICK_START_FACTORIAL_DoE.md` (overview)
2. `FACTORIAL_DoE_GUIDE.md` (design rationale)
3. `scripts/run_factorial_doe.sh` (implementation)
4. `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` (analysis methods)
---
## Quick Command Reference
### Run Experiment
```bash
# Option A: Quick validation (30 min)
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_RUN
# Option B: Standard production (2-4 hours)
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
# Option C: High-precision overnight (6-12 hours)
./scripts/run_factorial_doe.sh --runs-per-config 100 HIGH_PRECISION_FACTORIAL
```
### Monitor Progress
```bash
# Check results_run_01_2025_12_08 CSV growth
watch -n 5 'wc -l /path/to/experiment_results.csv'
# View summary
tail /path/to/experiment_summary.txt
# Count completed configurations
ls /path/to/run_logs/build_*.log | wc -l
```
### Analyze Results
```bash
# Load and analyze in Python
cd /path/to/StarForth-DoE-Analysis
python3 << 'EOF'
import pandas as pd
df = pd.read_csv('data/2025_11_19_FULL_FACTORIAL/experiment_results.csv')
print(f"Loaded {len(df)} runs")
# See FACTORIAL_DoE_ANALYSIS_WORKFLOW.md for full analysis code
EOF
```
---
## Key Concepts
### 64 Configurations
Each configuration is a unique combination of 6 binary factors:
- **0_0_0_0_0_0** = Baseline (all loops OFF)
- **1_0_0_0_0_0** = Only heat tracking
- **1_1_0_0_0_0** = Heat + rolling window
- ...
- **1_1_1_1_1_1** = All loops ON (current optimal)
### Randomized Execution
All 1,920+ runs (64 configs × 30 samples) are randomized into a single execution matrix.
- Eliminates temporal bias
- Prevents thermal ramp (if CPU heats during night)
- Proper DoE methodology
### One-and-Done Collection
Collect all data in single experiment, analyze separately.
- No tuning during collection (avoid bias)
- Single pass through parameter space
- Separates measurement from analysis
### Main Effects
Individual impact of each loop:
```
Loop #1: Loop turns ON, average metric changes by X%
Loop #2: Loop turns ON, average metric changes by Y%
... etc
```
### Interactions
How loops work together (amplify or suppress each other):
```
Loop #1 + Loop #2: Do they work better together than independently?
```
---
## Files at a Glance
| File | Size | Purpose |
|------|------|---------|
| `scripts/run_factorial_doe.sh` | 19 KB | Main executable script |
| `docs/QUICK_START_FACTORIAL_DoE.md` | 4.9 KB | Quick reference |
| `docs/FACTORIAL_DoE_GUIDE.md` | 8.0 KB | Design documentation |
| `docs/FACTORIAL_DoE_ANALYSIS_WORKFLOW.md` | 9.6 KB | Analysis pipeline |
| `docs/FACTORIAL_DoE_INDEX.md` | This file | Documentation index |
---
## Execution Timeline
### Phase 1: Preparation (5-10 min)
- Read `QUICK_START_FACTORIAL_DoE.md`
- Choose execution option
- Run command
### Phase 2: Collection (30 min - 12 hours)
- Script builds all 64 configs (as needed)
- Randomized execution of 1,920+ runs
- Metrics collected to CSV
- Progress shown every run
### Phase 3: Transfer (5 min)
- Copy results to analysis repo
- Verify file integrity
### Phase 4: Analysis (1-2 hours)
- Load CSV in Python
- Compute main effects
- Detect interactions
- Find optimal configurations
- Generate visualizations
### Phase 5: Reporting (1-2 hours)
- Summarize findings
- Create report
- Recommendations for production
**Total time:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **4-8 hours**
---
## The Design Philosophy
### Problem Statement
How do we systematically understand the performance contribution and interactions of 6 interdependent feedback loops?
### Solution
**Complete 2^6 factorial design of experiments**
### Why This Approach
1. **Complete:** Tests all 64 combinations (no gaps)
2. **Systematic:** Proper experimental design methodology
3. **Separable:** Collection ≠ Analysis (reduces bias)
4. **Reproducible:** Randomized execution, full logging
5. **Scalable:** Single script, extensible to more factors
### Alternatives Rejected
- ❌ Incremental tuning ("baseline → +loop1 → +loop1+2"): Misses interactions
- ❌ Dynamic toggle without rebuild: Introduces state contamination
- ❌ Partial factorial: Misses key interactions
- ❌ One-at-a-time testing: Expensive and confounded
---
## Next Steps
1. **Read** `QUICK_START_FACTORIAL_DoE.md` (5 minutes)
2. **Decide** which execution option (quick/standard/precise)
3. **Run** the command and let it collect data
4. **Monitor** progress in another terminal
5. **Analyze** results using `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
6. **Report** findings to stakeholders
---
## Questions?
- **"How long will this take?"** → See `QUICK_START_FACTORIAL_DoE.md`
- **"What does configuration X mean?"** → See `FACTORIAL_DoE_GUIDE.md`
- **"How do I analyze the results?"** → See `FACTORIAL_DoE_ANALYSIS_WORKFLOW.md`
- **"Why this approach?"** → See `FACTORIAL_DoE_GUIDE.md` - "The Why"
- **"How do I run it?"** → `./scripts/run_factorial_doe.sh`
---
**Version:** 1.0
**Created:** 2025-11-19
**Status:** Ready for production use
docs/working/experiments/02-experiments/factorial-doe/workflow.md
-297
View File
@@ -1,297 +0,0 @@
<!-- Moved from docs/02-experiments/factorial-doe/workflow.md to docs/working/experiments/02-experiments/factorial-doe/workflow.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ HISTORICAL (flagged 2026-10-09).** Documents the completed 2025-11-19 factorial
> campaign whose data informed the current L8 design; the toggle architecture it runs on is
> retired. Kept as a record.
# Factorial DoE Analysis Workflow
## End-to-End Process
```
StarForth Repo
└─ ./scripts/run_factorial_doe.sh
│
├─ Builds all 64 configurations
├─ Runs 1,920+ tests (randomized)
└─ Outputs: experiment_results.csv + logs
│
└─→ Copy to StarForth-DoE-Analysis repo
│
├─ Load CSV into Python/R
├─ Separate factors from metrics
├─ Compute main effects & interactions
├─ Identify optimal subsets
└─ Generate reports & visualizations
```
## Phase 1: Data Collection (StarForth Repo)
### Step 1: Run Experiment
```bash
cd /home/rajames/CLionProjects/StarForth
# Quick validation (optional)
./scripts/run_factorial_doe.sh --runs-per-config 1 TEST_01
# Full production run
./scripts/run_factorial_doe.sh --runs-per-config 30 2025_11_19_FULL_FACTORIAL
```
### Step 2: Wait for Completion
Monitor in another terminal:
```bash
watch -n 5 'wc -l /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv'
```
Expected progression:
- 0 min: 1 line (header)
- 30 min: ~500 lines
- 2 hours: ~1,500 lines
- 2.5 hours: 1,921 lines (complete)
### Step 3: Verify Completion
```bash
tail /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_results.csv
echo "---"
cat /home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL/experiment_summary.txt
```
## Phase 2: Data Transfer & Setup
### Step 1: Copy to Analysis Repo
```bash
RESULTS_DIR="/home/rajames/CLionProjects/StarForth-DoE/experiments/2025_11_19_FULL_FACTORIAL"
ANALYSIS_REPO="/home/rajames/CLionProjects/StarForth-DoE-Analysis"
mkdir -p "$ANALYSIS_REPO/data"
cp -r "$RESULTS_DIR" "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL"
```
### Step 2: Verify Transfer
```bash
ls -lah "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/"
wc -l "$ANALYSIS_REPO/data/2025_11_19_FULL_FACTORIAL/experiment_results.csv"
```
## Phase 3: Analysis (StarForth-DoE-Analysis Repo)
### Load Data (Python)
```python
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
# Load results_run_01_2025_12_08
df = pd.read_csv("data/2025_11_19_FULL_FACTORIAL/experiment_results.csv")
print(f"Loaded {len(df)} runs across {len(df.groupby('configuration'))} configurations")
# Separate factors and metrics
factors = df[['enable_loop_1_heat_tracking',
'enable_loop_2_rolling_window',
'enable_loop_3_linear_decay',
'enable_loop_4_pipelining_metrics',
'enable_loop_5_window_inference',
'enable_loop_6_decay_inference']]
metrics = df[['total_lookups', 'cache_hit_percent', 'vm_workload_duration_ns_q48', ...]]
print(f"Shape: {len(df)} rows × {len(df.columns)} columns")
print(f"Factors: {len(factors.columns)}")
print(f"Metrics: {len(metrics.columns)}")
```
### Main Effects Analysis
```python
# For each loop, compute average metric change when loop goes 0→1
main_effects = {}
for loop in factors.columns:
loop_name = loop.replace('enable_', '').replace('_', ' ').title()
metric_when_on = df[df[loop] == 1]['vm_workload_duration_ns_q48'].mean()
metric_when_off = df[df[loop] == 0]['vm_workload_duration_ns_q48'].mean()
effect = metric_when_on - metric_when_off
pct_change = 100 * (effect / metric_when_off)
main_effects[loop_name] = {
'effect': effect,
'pct_change': pct_change,
'on_mean': metric_when_on,
'off_mean': metric_when_off,
}
# Print sorted by impact
for loop, effect in sorted(main_effects.items(), key=lambda x: abs(x[1]['pct_change']), reverse=True):
print(f"{loop:30} {effect['pct_change']:+7.2f}% (ON: {effect['on_mean']:.0f}, OFF: {effect['off_mean']:.0f})")
```
Expected output:
```
Loop 6 Decay Inference +12.3% (ON: 8,800,000, OFF: 7,840,000)
Loop 5 Window Inference +8.5% (ON: 8,600,000, OFF: 7,920,000)
Loop 4 Pipelining Metrics +6.2% (ON: 8,400,000, OFF: 7,900,000)
Loop 3 Linear Decay +4.1% (ON: 8,200,000, OFF: 7,870,000)
Loop 2 Rolling Window +3.0% (ON: 8,100,000, OFF: 7,850,000)
Loop 1 Heat Tracking +2.1% (ON: 8,010,000, OFF: 7,850,000)
```
### Interaction Analysis
```python
# Two-way interactions: Do loops amplify/suppress each other?
interactions = {}
for i, loop_a in enumerate(factors.columns):
for j, loop_b in enumerate(factors.columns):
if i >= j:
continue # Only upper triangle
# Four cases: (0,0), (0,1), (1,0), (1,1)
case_00 = df[(df[loop_a] == 0) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
case_01 = df[(df[loop_a] == 0) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
case_10 = df[(df[loop_a] == 1) & (df[loop_b] == 0)]['vm_workload_duration_ns_q48'].mean()
case_11 = df[(df[loop_a] == 1) & (df[loop_b] == 1)]['vm_workload_duration_ns_q48'].mean()
# Interaction effect: (1,1) - (1,0) - (0,1) + (0,0)
interaction = (case_11 - case_10 - case_01 + case_00)
loop_a_name = loop_a.replace('enable_', '').replace('_', ' ')
loop_b_name = loop_b.replace('enable_', '').replace('_', ' ')
if abs(interaction) > 100000: # Only report large interactions
interactions[f"{loop_a_name} × {loop_b_name}"] = interaction
# Print sorted interactions
for pair, interaction in sorted(interactions.items(), key=lambda x: abs(x[1]), reverse=True):
print(f"{pair:40} {interaction:+12,.0f}")
```
### Optimal Configuration Search
```python
# Find best k configurations by performance metric
k = 10
best_configs = df.nlargest(k, 'vm_workload_duration_ns_q48')[
['configuration', 'vm_workload_duration_ns_q48', 'enable_loop_1_heat_tracking',
'enable_loop_2_rolling_window', 'enable_loop_3_linear_decay',
'enable_loop_4_pipelining_metrics', 'enable_loop_5_window_inference',
'enable_loop_6_decay_inference']
].drop_duplicates(subset=['configuration'])
print("Top 10 Configurations by Performance:")
print("─" * 100)
for idx, row in best_configs.iterrows():
loops = (
f"L1:{row['enable_loop_1_heat_tracking']} " +
f"L2:{row['enable_loop_2_rolling_window']} " +
f"L3:{row['enable_loop_3_linear_decay']} " +
f"L4:{row['enable_loop_4_pipelining_metrics']} " +
f"L5:{row['enable_loop_5_window_inference']} " +
f"L6:{row['enable_loop_6_decay_inference']}"
)
print(f"{row['configuration']:20} | {row['vm_workload_duration_ns_q48']:12,.0f} ns | {loops}")
```
### Visualization Example
```python
# Main effects plot
import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
fig.suptitle('2^6 Factorial DoE Main Effects', fontsize=16)
loop_names = [f"Loop {i+1}" for i in range(6)]
metric_key = 'vm_workload_duration_ns_q48'
for idx, loop in enumerate(factors.columns):
ax = axes[idx // 3, idx % 3]
off_mean = df[df[loop] == 0][metric_key].mean()
on_mean = df[df[loop] == 1][metric_key].mean()
bars = ax.bar(['OFF', 'ON'], [off_mean, on_mean], color=['lightcoral', 'lightgreen'])
ax.set_ylabel('Workload Duration (ns)')
ax.set_title(f'{loop_names[idx]}')
ax.set_ylim([7000000, 10000000])
# Add value labels
for bar in bars:
height = bar.get_height()
ax.text(bar.get_x() + bar.get_width()/2., height,
f'{height/1e6:.1f}M', ha='center', va='bottom')
plt.tight_layout()
plt.savefig('main_effects.png', dpi=150)
print("Saved: main_effects.png")
```
## Phase 4: Reporting
### Key Outputs to Generate
1. **Main Effects Summary** (which loops matter most?)
- Table of individual loop impacts
- Sorted by effect size
2. **Interaction Matrix** (do loops amplify/suppress?)
- 6×6 table of two-way interactions
- Highlights synergies/conflicts
3. **Optimal Configuration** (best 10-20 subsets)
- Recommend top 3-5 production configs
- Show Pareto frontier (performance vs complexity)
4. **Statistical Summary**
- Variance by configuration
- Confidence intervals on estimates
- Regression model R²
### Example Report Structure
```markdown
# Factorial DoE Analysis Report - 2025-11-19
## Executive Summary
- Tested all 64 feedback loop combinations
- 1,920 experimental runs collected
- Key finding: Loops 5+6 show strong synergy (+18% combined)
## Main Effects
| Loop | Impact | Confidence | Recommendation |
|------|--------|-----------|---|
| Loop 6 (Decay Inference) | +12.3% | p<0.001 | **Enabled** |
| Loop 5 (Window Inference) | +8.5% | p<0.001 | **Enabled** |
| Loop 4 (Pipelining) | +6.2% | p<0.01 | Enabled |
| Loop 3 (Decay) | +4.1% | p<0.05 | Enabled |
| Loop 2 (Window) | +3.0% | p<0.05 | Enabled |
| Loop 1 (Heat) | +2.1% | p>0.05 | Optional |
## Recommended Configurations
1. **111111** (All loops): +30% vs baseline
2. **111110** (Except heat): +28% vs baseline, simpler
3. **011111** (Except heat, decay): +24% vs baseline
## Trade-off Analysis
- Full stack (111111): Best performance, highest code complexity
- Recommended (111110): 28% improvement, 95% of benefit
- Minimal set (001110): 18% improvement, 50% code overhead
---
```
## Checklist
- [ ] Run factorial DoE script to completion
- [ ] Verify 1,920 rows in experiment_results.csv
- [ ] Copy results to analysis repo
- [ ] Load data in Python/R
- [ ] Compute main effects (6 loops)
- [ ] Compute interactions (2-way or higher)
- [ ] Identify top 5-10 configurations
- [ ] Generate visualizations
- [ ] Write summary report
- [ ] Decide on production configuration
---
**Timeline:** Collection (2-4 hrs) + Analysis (2-4 hrs) = **6-8 hours total** for comprehensive factorial DoE.
docs/working/experiments/02-experiments/heartbeat-doe/corrected-design.md
-341
View File
@@ -1,341 +0,0 @@
<!-- Moved from docs/02-experiments/heartbeat-doe/corrected-design.md to docs/working/experiments/02-experiments/heartbeat-doe/corrected-design.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `phase2-corrected.md` (the
> "corrected and finalized" revision). Kept for historical record.
# StarForth Heartbeat DoE – CORRECTED Design (Heartrate Focus)
**Date:** November 20, 2025
**CORRECTION:** Previous design missing critical metric: **HEARTBEAT RATE (tick_ns)**
**Status:** Ready for implementation
---
## The Critical Insight You Caught
The heartbeat isn't just a **fixed tick**. The system has a **variable heartbeat rate** controlled by:
```c
/* HeartbeatWorker.tick_ns — this is DYNAMIC, not constant! */
uint64_t tick_ns; /* Nanoseconds between ticks — VARIES with load! */
```
**What this means:**
- Default: `HEARTBEAT_TICK_NS` (~1 millisecond)
- **But:** The inference engine can adaptively adjust `tick_target_ns`
- Different configurations may modulate heartbeat frequency differently
- **This is the KEY physics behavior we must measure!**
---
## Stage 1 Problem (What We Got Wrong)
Stage 1 baseline ran **without heartbeat thread** (HEARTBEAT_THREAD_ENABLED=0):
- Fixed 1ms tick (no modulation)
- No adaptive heartrate
- Doesn't reflect StarForth's intended physics-driven behavior
**Stage 1 conclusion:** "System is deterministic, no heartbeat variation"
**Stage 1 was incomplete** because it disabled the very mechanism that makes the system adaptive!
---
## Stage 2 Correction (What We Should Do)
### Primary Hypothesis
The 5 elite configurations differ in **how they modulate heartbeat rate**:
| Config | Expected Behavior |
|--------|-------------------|
| 1_0_1_1_1_0 | Tight, responsive heartrate (low jitter, fast modulation) |
| 1_0_1_1_1_1 | Similar, with decay inference slowing response? |
| 1_1_0_1_1_1 | Window-driven, may have different modulation pattern |
| 1_0_1_0_1_0 | Lean, may have sluggish response |
| 0_1_1_0_1_1 | Window-without-heat, baseline behavior |
### Key Metrics for Heartrate (Not Just Intervals)
**We need to measure:**
1. **Heartbeat Rate Modulation (NOT just interval jitter)**
```
heartbeat_rate_trajectory = [tick_ns_t0, tick_ns_t1, tick_ns_t2, ...]
What we measure:
- Mean heartbeat rate (expected: ~1,000,000 ns = 1 kHz baseline)
- Responsiveness to load (does rate slow under heavy load?)
- Convergence time (how long until rate stabilizes?)
- Modulation range (min to max tick_ns observed)
```
2. **Heartrate vs. Workload Coupling**
```
For each tick:
workload_intensity = lookups_this_tick / lookups_per_second_baseline
heartrate_hz = 1_000_000_000 / tick_ns
Measure: correlation(workload_intensity, heartrate_hz)
- Positive correlation: heartrate slows under load (good physics)
- Negative correlation: heartrate speeds up under load (bad physics)
- No correlation: heartrate doesn't respond to load (broken physics)
```
3. **Heartrate Stability (Not "jitter" but "rate consistency")**
```
- Does heartrate converge to a steady state?
- How quickly does it adapt to workload changes?
- Does it oscillate or settle smoothly?
```
4. **Heartrate Efficiency**
```
- Is the system running faster/slower with adapted heartrate?
- Workload duration: should improve with optimized heartrate
- Memory: should be more efficient with tuned heartrate
```
---
## Corrected Metrics Collection
### Per-Tick Collection (During Execution)
```c
/* In heartbeat_thread_main(), before nanosleep() */
struct HeartbeatTickData {
uint64_t tick_number;
uint64_t tick_ns; /* THIS IS THE HEARTBEAT RATE! */
uint64_t workload_ops_this_tick; /* Dictionary lookups in this tick */
uint64_t wall_clock_ns; /* Actual wall-clock time */
uint64_t hot_word_count; /* Physics state */
uint64_t total_heat; /* Physics state */
};
/* Store in circular buffer or rolling window */
```
### Post-Run Analysis Metrics
```c
typedef struct {
/* === Original Metrics === */
/* ... existing 38 metrics ... */
/* === CORRECTED: Heartbeat Rate Metrics === */
/* Heartbeat rate trajectory */
uint64_t total_ticks_executed;
double mean_heartbeat_rate_hz; /* Hz = 1e9 / mean_tick_ns */
double heartbeat_rate_stddev_hz; /* Variation in Hz */
double heartbeat_rate_cv; /* Coefficient of variation */
uint64_t heartbeat_rate_min_ns; /* Slowest (highest tick_ns) */
uint64_t heartbeat_rate_max_ns; /* Fastest (lowest tick_ns) */
double heartbeat_rate_range_hz; /* max_hz - min_hz */
/* Heartrate vs Load Coupling */
double load_to_heartrate_correlation; /* Does rate respond to load? */
int load_response_direction; /* +1 = slower under load (good) */
double heartrate_adaptation_latency_ticks; /* Ticks until rate responds */
double heartrate_settling_time_ticks; /* Ticks to settle after change */
/* Heartrate Convergence */
double heartrate_convergence_time_ms; /* Time to reach stable rate */
int heartrate_converged; /* 1 = yes, 0 = still oscillating */
double heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation */
/* Overall Heartrate Stability Score */
double heartrate_stability_score; /* 0-100: how stable is heartrate? */
} DoeMetrics_CORRECTED;
```
---
## The Missing Piece: Heartbeat Rate Logging
**Critical code addition needed:**
```c
/* In vm.c heartbeat_thread_main() - CAPTURE tick_ns BEFORE and AFTER */
static inline void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops) {
/* Store sample in rolling buffer */
if (!vm->heartbeat_rate_samples) {
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
}
HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
sample->tick_number = vm->heartbeat.tick_count;
sample->tick_ns = tick_ns; /* THE HEARTBEAT RATE! */
sample->workload_ops = workload_ops;
sample->timestamp_ns = platform_monotonic_ns();
vm->heartbeat_rate_sample_count++;
}
/* In heartbeat_thread_main() */
static void* heartbeat_thread_main(void *arg) {
VM *vm = (VM*)arg;
while (!vm->heartbeat.worker->stop_requested) {
uint64_t tick_start_ns = platform_monotonic_ns();
vm_heartbeat_run_cycle(vm); /* May adjust tick_ns */
uint64_t tick_ns = vm->heartbeat.worker->tick_ns;
uint64_t workload_ops = vm->execution_heat_current_tick; /* Need to track */
/* CAPTURE THE HEARTBEAT RATE! */
capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
/* Sleep */
nanosleep(...);
}
}
```
---
## Corrected Analysis (R Script)
```r
# Load heartbeat rate samples from CSV
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
mutate(
# Convert heartbeat rates (tick_ns → Hz)
mean_heartbeat_hz = 1e9 / mean_heartbeat_rate_ns,
heartrate_stability = 1 / (1 + heartbeat_rate_cv), # Lower CV = higher score
)
# PRIMARY ANALYSIS: Heartrate Stability
stability_by_heartrate <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_heartrate_hz = mean(mean_heartbeat_hz),
mean_heartrate_stability = mean(heartrate_stability),
mean_load_coupling = mean(load_to_heartrate_correlation),
mean_workload_ms = mean(vm_workload_duration_ns_q48 / 1e6)
) %>%
mutate(
# Golden config = best heartrate stability + strongest load coupling
golden_score = (heartrate_stability * 0.5) + (load_coupling * 0.3) + (workload_improvement * 0.2)
)
# Golden Config = Highest golden_score
```
---
## Why This Matters
### Stage 1 Got Lucky
Stage 1 found 5 good configs **by accident** because it measured final performance, not physics.
### Stage 2 Should Be Different
Stage 2 should measure **whether the physics is working correctly**:
- **Can the system modulate heartbeat rate** in response to load?
- **Is the modulation smooth** or does it oscillate?
- **Does heartrate modulation improve overall performance?**
- **Which configuration has the most responsive, stable heartrate?**
**The golden configuration should be the one where heartbeat rate modulation is:**
1. ✓ **Responsive** (correlates with workload)
2. ✓ **Smooth** (doesn't oscillate wildly)
3. ✓ **Efficient** (results in best overall performance)
---
## Revised DoE Experiment Design
### What Actually Gets Measured
Instead of just "jitter in fixed 1ms tick intervals", measure:
**Per-Run Heartbeat Rate Trajectory:**
```
Tick 0: tick_ns = 1,000,000 (1 ms baseline)
Tick 1: tick_ns = 1,500,000 (slowing under load)
Tick 2: tick_ns = 2,000,000 (further slowdown)
Tick 10: tick_ns = 1,200,000 (adapting back)
Tick 100: tick_ns = 1,050,000 (converged to steady state)
```
**What we analyze:**
1. Did heartrate change at all? (some configs might keep it fixed)
2. How much did it vary? (CV of tick_ns, not CV of fixed intervals)
3. Did it respond to load? (correlation with workload)
4. Did it converge smoothly? (no oscillation)
5. Did it improve performance? (faster execution with adaptive heartrate?)
---
## Implementation Checklist (Revised)
- [ ] Add `HeartbeatRateSample` struct to capture `tick_ns` per tick
- [ ] Modify `heartbeat_thread_main()` to log `tick_ns` trajectory
- [ ] Extend `DoeMetrics` with heartrate-specific fields (not just interval jitter)
- [ ] Update CSV schema to include heartrate trajectory metrics
- [ ] Modify R analysis to focus on **heartrate modulation patterns**
- [ ] Update golden config selection: prioritize **load responsiveness**
---
## Expected Results (Corrected)
### Hypothesis: Configs Differ in Heartrate Modulation
**Config 1 (1_0_1_1_1_0) - Expected WINNER:**
- Heartrate slowly increases under load (tick_ns increases, Hz decreases)
- Smooth adaptation (no oscillation)
- Converges within 1000 ticks
- Performance improves (faster overall execution)
- Load coupling: 0.8+ (strong)
**Config 5 (0_1_1_0_1_1) - Expected LOSER:**
- Heartrate doesn't respond to load (fixed tick_ns)
- No correlation with workload
- Performance degrades (why? system can't adapt)
---
## Key Realization
The heartbeat isn't just a **timer**. It's a **feedback mechanism**:
```
Load increases
↓
Inference engine detects change
↓
Adjusts tick_target_ns (slower heartbeat = more thinking time)
↓
Heartbeat rate modulates (tick_ns increases)
↓
Physics engine gets more time to tune parameters
↓
System adapts and performance improves
```
**We need to measure whether this feedback loop is working!**
---
## Next Steps
1. **Extend HeartbeatWorker:** Add rate sample collection
2. **Modify heartbeat_thread_main():** Log `tick_ns` per tick
3. **Update DoeMetrics:** Heartrate trajectory analysis
4. **Rewrite CSV schema:** Focus on heartrate modulation
5. **Rewrite R analysis:** Measure load-response coupling and heartrate stability
6. **Re-run experiment:** Focus on configs that modulate heartrate best
---
**This changes the entire experiment focus.**
Instead of measuring "jitter in a fixed heartbeat", we're measuring "**how well does the system modulate its heartbeat in response to load?**"
That's the real physics story.
docs/working/experiments/02-experiments/heartbeat-doe/design.md
-431
View File
@@ -1,431 +0,0 @@
<!-- Moved from docs/02-experiments/heartbeat-doe/design.md to docs/working/experiments/02-experiments/heartbeat-doe/design.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Pre-correction draft; superseded first by
> `corrected-design.md`, then by `phase2-corrected.md` (the finalized design). Kept for
> historical record.
# StarForth Factorial DoE with Heartbeat Integration
**Date:** November 20, 2025
**Status:** Design Phase
**Experiment:** 2^6 Factorial with Heartbeat-Aware Metrics Collection
---
## Overview
This document describes the **second-generation factorial DoE** that incorporates heartbeat observability into metric collection. Building on the baseline DOE (Stage 1: 3200 runs, 6 factors, no heartbeat), this experiment captures **per-tick heartbeat dynamics** to identify configurations with optimal stability characteristics.
### Key Innovation
Instead of measuring static performance, this experiment measures **temporal stability**:
- How quickly does the system converge under load changes?
- How much does heartbeat interval modulate in response to workload?
- What is the jitter envelope after settling?
- Which configuration achieves smoothest load→heartbeat coupling?
---
## Design Points (Top 5 Configurations from Stage 1)
Based on Stage 1 analysis, the elite configurations are:
1. **Config 1:** `1_0_1_1_1_0` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=OFF)
2. **Config 2:** `1_0_1_1_1_1` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=ON)
3. **Config 3:** `1_1_0_1_1_1` (F1=ON, F2=ON, F3=OFF, F4=ON, F5=ON, F6=ON)
4. **Config 4:** `1_0_1_0_1_0` (F1=ON, F2=OFF, F3=ON, F4=OFF, F5=ON, F6=OFF)
5. **Config 5:** `0_1_1_0_1_1` (F1=OFF, F2=ON, F3=ON, F4=OFF, F5=ON, F6=ON)
**Factors Mapping:**
- F1 = ENABLE_LOOP_1_HEAT_TRACKING
- F2 = ENABLE_LOOP_2_ROLLING_WINDOW
- F3 = ENABLE_LOOP_3_LINEAR_DECAY
- F4 = ENABLE_LOOP_4_PIPELINING_METRICS
- F5 = ENABLE_LOOP_5_WINDOW_INFERENCE
- F6 = ENABLE_LOOP_6_DECAY_INFERENCE
---
## New Heartbeat Metrics (Per-Tick Collection)
### 1. Heartbeat Interval Dynamics
```c
struct HeartbeatTickMetrics {
uint64_t tick_count; /* Absolute tick number */
uint64_t tick_interval_ns; /* Wall-clock duration of this tick */
uint64_t instantaneous_workload_ops; /* Dictionary lookups in this tick */
double instantaneous_load_ratio; /* (ops this tick) / (avg ops/tick) */
};
```
**Purpose:** Capture how heartbeat frequency modulates in response to workload intensity.
### 2. Cache Performance (Per-Tick Window)
```c
struct CacheWindowMetrics {
uint64_t tick_start;
uint64_t cache_hits_window; /* Hits in last 100 ticks */
uint64_t cache_attempts_window; /* Attempts in last 100 ticks */
double cache_hit_percent_window; /* Rolling window %, 0-100 */
};
```
**Purpose:** Measure cache stability, not just final hit rate.
### 3. Rolling Window Diagnostics
```c
struct WindowDiagnostics {
uint32_t effective_window_size; /* Current adaptive size */
uint64_t pattern_diversity; /* Unique word IDs observed */
double diversity_growth_rate; /* % new patterns per 1000 executions */
int is_warm; /* 1 if window is representative */
};
```
**Purpose:** Track window convergence and saturation.
### 4. Jitter Envelope
```c
struct JitterMetrics {
double tick_interval_mean_ns; /* Average heartbeat interval */
double tick_interval_stddev_ns; /* StdDev of interval */
double tick_interval_cv; /* Coefficient of variation (σ/μ) */
uint64_t outlier_count; /* Ticks >3σ from mean */
double outlier_ratio; /* Outliers / total ticks */
};
```
**Purpose:** Measure temporal smoothness and jitter control.
### 5. Decay Slope Tracking
```c
struct DecayTrajectory {
uint64_t tick_sample_count; /* How many ticks sampled */
double decay_slope_observed; /* Fitted slope from observations */
double slope_convergence_rate; /* Speed of slope convergence */
int slope_direction; /* -1=decreasing, 0=stable, +1=increasing */
};
```
**Purpose:** Monitor whether decay slope inference is tracking correctly.
### 6. Load-Response Coupling
```c
struct LoadResponseCoupling {
double correlation_load_to_interval; /* Correlation(workload, heartbeat_interval) */
double response_latency_ticks; /* Ticks until interval reacts to load change */
double overshoot_ratio; /* Peak response / steady-state response */
double settling_time_ticks; /* Ticks to settle within 10% of target */
};
```
**Purpose:** Measure how quickly and smoothly heartbeat responds to workload shifts.
---
## Extended DoeMetrics Struct
Add to `include/doe_metrics.h`:
```c
typedef struct {
/* === Original Metrics (preserved) === */
/* ... existing 35+ metrics ... */
/* === NEW: Heartbeat Stability Metrics === */
/* Tick dynamics */
uint64_t total_heartbeat_ticks; /* Total ticks during run */
double tick_interval_mean_ns; /* Average heartbeat interval */
double tick_interval_stddev_ns; /* StdDev of heartbeat interval */
double tick_interval_cv; /* Coefficient of variation */
uint64_t tick_outlier_count_3sigma; /* Ticks >3σ from mean */
double tick_outlier_ratio; /* Outliers / total ticks */
/* Cache stability window */
double cache_hit_percent_rolling_mean; /* Rolling avg, last 100 ticks */
double cache_hit_percent_rolling_stddev; /* Rolling stddev */
double cache_stability_index; /* How stable is cache hit %? */
/* Window convergence */
uint32_t window_final_effective_size; /* Final adaptive window size */
double pattern_diversity_final; /* Final unique patterns observed */
double diversity_growth_final; /* Growth rate at end */
int window_warmth_achieved; /* 1 = warm, 0 = still cold */
/* Decay slope convergence */
double decay_slope_fitted_final; /* Final fitted decay slope */
double decay_slope_convergence_rate; /* How fast did slope converge? */
int decay_slope_stable; /* 1 = stable direction, 0 = oscillating */
/* Load-response */
double load_interval_correlation; /* Correlation(load, heartbeat_interval) */
double response_latency_ticks; /* Ticks until response to workload change */
double overshoot_ratio; /* Peak response / steady-state */
double settling_time_ticks; /* Time to 10% settling band */
/* Aggregate stability score */
double overall_stability_score; /* Composite metric: (low jitter + fast convergence + tight correlation) */
} DoeMetrics; /* Now extends to ~65+ columns */
```
---
## Heartbeat Data Collection Architecture
### Per-Tick Logging (Lightweight)
During VM execution, capture minimal per-tick data:
```c
/* In vm.c vm_tick_heartbeat_worker() */
void vm_tick_heartbeat_worker(VM *vm) {
HeartbeatSnapshot snapshot = {
.published_tick = vm->heartbeat.tick_count,
.published_ns = platform_monotonic_ns(),
.window_width = vm->rolling_window.effective_window_size,
.decay_slope_q48 = vm->decay_slope_q48,
.hot_word_count = vm->hot_word_count_at_check,
.stale_word_count = vm->stale_word_count_at_check,
.total_heat = vm->total_heat_at_last_check,
};
/* Publish snapshot (double-buffered, lock-free read) */
int next_slot = 1 - vm->heartbeat.snapshot_index;
vm->heartbeat.snapshots[next_slot] = snapshot;
vm->heartbeat.snapshot_index = next_slot;
}
```
### Post-Run Analysis (Batch Processing)
After test execution completes, analyze collected tick data:
```c
/* In src/doe_metrics.c metrics_from_vm() */
void analyze_heartbeat_trajectory(VM *vm, DoeMetrics *metrics) {
HeartbeatSnapshot *snapshots = vm->heartbeat.snapshots;
uint64_t total_ticks = vm->heartbeat.tick_count;
/* 1. Compute tick interval statistics */
double intervals[total_ticks];
for (size_t i = 1; i < total_ticks; i++) {
intervals[i] = snapshots[i].published_ns - snapshots[i-1].published_ns;
}
metrics->tick_interval_mean_ns = compute_mean(intervals, total_ticks);
metrics->tick_interval_stddev_ns = compute_stddev(intervals, total_ticks);
metrics->tick_interval_cv = metrics->tick_interval_stddev_ns / metrics->tick_interval_mean_ns;
/* 2. Compute outlier ratio (ticks >3σ from mean) */
double threshold_3sigma = metrics->tick_interval_mean_ns + 3.0 * metrics->tick_interval_stddev_ns;
uint64_t outliers = 0;
for (size_t i = 0; i < total_ticks; i++) {
if (intervals[i] > threshold_3sigma) outliers++;
}
metrics->tick_outlier_ratio = (double)outliers / total_ticks;
/* 3. Compute decay slope convergence */
double slopes[total_ticks];
for (size_t i = 100; i < total_ticks; i++) { /* Skip warm-up */
slopes[i] = snapshots[i].decay_slope_q48 / 65536.0;
}
metrics->decay_slope_fitted_final = slopes[total_ticks - 1];
metrics->decay_slope_convergence_rate = compute_convergence_rate(slopes, total_ticks);
/* 4. Compute overall stability score */
double stability = 0.0;
stability += (100.0 - metrics->tick_interval_cv); /* Lower CV = higher score */
stability += (100.0 - metrics->tick_outlier_ratio * 100.0); /* Fewer outliers = higher */
stability += metrics->decay_slope_convergence_rate; /* Fast convergence = higher */
metrics->overall_stability_score = stability / 3.0;
}
```
---
## CSV Output Schema (Extended)
**New columns (per heartbeat experiment run):**
```
timestamp,configuration,run_number,
[... original 38 columns ...],
total_heartbeat_ticks,
tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,
tick_outlier_count_3sigma,tick_outlier_ratio,
cache_hit_percent_rolling_mean,cache_hit_percent_rolling_stddev,cache_stability_index,
window_final_effective_size,pattern_diversity_final,diversity_growth_final,window_warmth_achieved,
decay_slope_fitted_final,decay_slope_convergence_rate,decay_slope_stable,
load_interval_correlation,response_latency_ticks,overshoot_ratio,settling_time_ticks,
overall_stability_score
```
**Total:** ~60 columns per run
---
## R Analysis Template
```r
library(tidyverse)
library(ggplot2)
library(gridExtra)
# Load heartbeat DoE data
doe_data <- read.csv("experiment_results_with_heartbeat.csv") %>%
mutate(
configuration = as.factor(configuration),
timestamp = as.POSIXct(timestamp)
)
# ========== MAIN ANALYSIS ==========
# 1. Stability Ranking
stability_rank <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_stability_score = mean(overall_stability_score),
mean_jitter_cv = mean(tick_interval_cv),
mean_convergence_rate = mean(decay_slope_convergence_rate),
mean_settling_time = mean(settling_time_ticks),
n_runs = n()
) %>%
mutate(
rank = rank(-mean_stability_score, ties.method = "min")
) %>%
arrange(rank)
print("=== STABILITY RANKINGS ===")
print(stability_rank)
# 2. Visualization: Stability Scores
ggplot(doe_data, aes(x = reorder(configuration, overall_stability_score, FUN=median),
y = overall_stability_score,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
geom_jitter(width = 0.2, alpha = 0.4) +
coord_flip() +
theme_minimal() +
labs(title = "Overall Stability Scores by Configuration",
x = "Configuration",
y = "Stability Score (0-100)")
ggsave("01_stability_scores.png", width = 10, height = 6)
# 3. Visualization: Jitter Control (CV)
ggplot(doe_data, aes(x = reorder(configuration, tick_interval_cv, FUN=median),
y = tick_interval_cv,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
coord_flip() +
theme_minimal() +
labs(title = "Heartbeat Jitter (Coefficient of Variation)",
x = "Configuration",
y = "CV (lower = less jitter)")
ggsave("02_jitter_control.png", width = 10, height = 6)
# 4. Visualization: Convergence Speed
ggplot(doe_data, aes(x = reorder(configuration, decay_slope_convergence_rate, FUN=median),
y = decay_slope_convergence_rate,
fill = configuration)) +
geom_boxplot(alpha = 0.7) +
coord_flip() +
theme_minimal() +
labs(title = "Decay Slope Convergence Speed",
x = "Configuration",
y = "Convergence Rate (ticks)")
ggsave("03_convergence_speed.png", width = 10, height = 6)
# 5. Correlation: Load Response Coupling
ggplot(doe_data, aes(x = load_interval_correlation,
y = overall_stability_score,
color = configuration)) +
geom_point(alpha = 0.6, size = 2) +
facet_wrap(~configuration) +
theme_minimal() +
labs(title = "Load-Response Coupling vs Stability",
x = "Correlation(load, interval)",
y = "Overall Stability Score")
ggsave("04_load_response_coupling.png", width = 12, height = 8)
# 6. Statistical Test: Is top config significantly better?
top_config <- stability_rank$configuration[1]
other_configs <- stability_rank$configuration[-1]
top_data <- doe_data %>% filter(configuration == top_config) %>% pull(overall_stability_score)
other_data <- doe_data %>% filter(configuration %in% other_configs) %>% pull(overall_stability_score)
t_test <- t.test(top_data, other_data)
print(sprintf("\nT-test: %s vs others\np-value: %.4f", top_config, t_test$p.value))
# 7. Golden Configuration Summary
cat("\n=== GOLDEN CONFIGURATION CANDIDATE ===\n")
golden <- stability_rank %>% slice(1)
print(golden)
cat("\nInterpretation:")
cat(sprintf("\n- Lowest jitter (CV): %.4f", golden$mean_jitter_cv))
cat(sprintf("\n- Fastest convergence: %.1f ticks", golden$mean_convergence_rate))
cat(sprintf("\n- Settling time: %.1f ticks", golden$mean_settling_time))
cat(sprintf("\n- Recommendation: Use '%s' as golden configuration\n", golden$configuration))
```
---
## Execution Plan
### Phase 1: Implementation (3-4 hours)
1. Extend `DoeMetrics` struct with heartbeat columns
2. Implement `analyze_heartbeat_trajectory()` in `doe_metrics.c`
3. Capture snapshot data during heartbeat ticks
4. Update CSV schema
### Phase 2: Experiment Execution (2-4 hours)
1. Create `run_factorial_doe_with_heartbeat.sh` script
2. Run 5 elite configurations × 50 runs each = 250 total runs
3. Collect heartbeat + performance data
### Phase 3: Analysis (2-3 hours)
1. Load CSV in R
2. Compute stability rankings
3. Generate visualizations
4. Identify golden configuration
---
## Expected Outcomes
**Primary Question:** Which of the 5 elite configurations exhibits:
- Lowest heartbeat jitter (CV)?
- Fastest convergence to steady state?
- Strongest load-response correlation?
- Cleanest temporal signature?
**Expected Winner:** Configuration with balanced F-factors, likely:
- F1=1 (heat tracking required)
- F3=1 or F5=1 (decay or inference, but not both)
- F6=0 or F6=1 (decay inference optional)
**Golden Configuration:** Becomes baseline physics for MamaForth & PapaForth downstream work.
---
## Success Criteria
✓ All 5 configurations build and run without segfault
✓ Heartbeat data collected for all 250 runs
✓ Stability rankings computed and visualized
✓ Golden configuration identified with p < 0.05 significance
✓ R analysis report generated with 5+ plots
---
**Version:** 1.0
**Status:** Ready for implementation
**Next Step:** Implement heartbeat metrics capture and extended DoE script