docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such
docs/CLAUDE.md Phase 2, 2026-10-14. All moves by git mv; every archived file carries a header saying what it is, why, and where to look instead. - docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL design, kernel args, hosted acceptance test, dictionary, primitives, ontology, L8 table -- previously spread over docs/, docs/lithosananke/ and docs/03-architecture/. - docs/working/archive/historical/: finished work (session plan, test-gap audit, birthing plan, fix write-ups, the January architecture draft). - docs/working/archive/superseded/: the Hermes-VM Tripod docs and July Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working doc already flagged superseded or obsolete. - docs/working/archive/dead/: the hal_* target contract (never adopted), L4Re, metrics-and-knobs. - Papers and proposals into docs/working/papers/. - References rewritten across the repo (not logs/, not docs/patent/); CLAUDE.md's documentation section rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1a4b163861
commit
7aed588f78
@@ -1,341 +0,0 @@
|
||||
<!-- Moved from docs/02-experiments/heartbeat-doe/corrected-design.md to docs/working/experiments/02-experiments/heartbeat-doe/corrected-design.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `phase2-corrected.md` (the
|
||||
> "corrected and finalized" revision). Kept for historical record.
|
||||
|
||||
# StarForth Heartbeat DoE – CORRECTED Design (Heartrate Focus)
|
||||
|
||||
**Date:** November 20, 2025
|
||||
**CORRECTION:** Previous design missing critical metric: **HEARTBEAT RATE (tick_ns)**
|
||||
**Status:** Ready for implementation
|
||||
|
||||
---
|
||||
|
||||
## The Critical Insight You Caught
|
||||
|
||||
The heartbeat isn't just a **fixed tick**. The system has a **variable heartbeat rate** controlled by:
|
||||
|
||||
```c
|
||||
/* HeartbeatWorker.tick_ns — this is DYNAMIC, not constant! */
|
||||
uint64_t tick_ns; /* Nanoseconds between ticks — VARIES with load! */
|
||||
```
|
||||
|
||||
**What this means:**
|
||||
- Default: `HEARTBEAT_TICK_NS` (~1 millisecond)
|
||||
- **But:** The inference engine can adaptively adjust `tick_target_ns`
|
||||
- Different configurations may modulate heartbeat frequency differently
|
||||
- **This is the KEY physics behavior we must measure!**
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 Problem (What We Got Wrong)
|
||||
|
||||
Stage 1 baseline ran **without heartbeat thread** (HEARTBEAT_THREAD_ENABLED=0):
|
||||
- Fixed 1ms tick (no modulation)
|
||||
- No adaptive heartrate
|
||||
- Doesn't reflect StarForth's intended physics-driven behavior
|
||||
|
||||
**Stage 1 conclusion:** "System is deterministic, no heartbeat variation"
|
||||
|
||||
**Stage 1 was incomplete** because it disabled the very mechanism that makes the system adaptive!
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 Correction (What We Should Do)
|
||||
|
||||
### Primary Hypothesis
|
||||
|
||||
The 5 elite configurations differ in **how they modulate heartbeat rate**:
|
||||
|
||||
| Config | Expected Behavior |
|
||||
|--------|-------------------|
|
||||
| 1_0_1_1_1_0 | Tight, responsive heartrate (low jitter, fast modulation) |
|
||||
| 1_0_1_1_1_1 | Similar, with decay inference slowing response? |
|
||||
| 1_1_0_1_1_1 | Window-driven, may have different modulation pattern |
|
||||
| 1_0_1_0_1_0 | Lean, may have sluggish response |
|
||||
| 0_1_1_0_1_1 | Window-without-heat, baseline behavior |
|
||||
|
||||
### Key Metrics for Heartrate (Not Just Intervals)
|
||||
|
||||
**We need to measure:**
|
||||
|
||||
1. **Heartbeat Rate Modulation (NOT just interval jitter)**
|
||||
```
|
||||
heartbeat_rate_trajectory = [tick_ns_t0, tick_ns_t1, tick_ns_t2, ...]
|
||||
|
||||
What we measure:
|
||||
- Mean heartbeat rate (expected: ~1,000,000 ns = 1 kHz baseline)
|
||||
- Responsiveness to load (does rate slow under heavy load?)
|
||||
- Convergence time (how long until rate stabilizes?)
|
||||
- Modulation range (min to max tick_ns observed)
|
||||
```
|
||||
|
||||
2. **Heartrate vs. Workload Coupling**
|
||||
```
|
||||
For each tick:
|
||||
workload_intensity = lookups_this_tick / lookups_per_second_baseline
|
||||
heartrate_hz = 1_000_000_000 / tick_ns
|
||||
|
||||
Measure: correlation(workload_intensity, heartrate_hz)
|
||||
- Positive correlation: heartrate slows under load (good physics)
|
||||
- Negative correlation: heartrate speeds up under load (bad physics)
|
||||
- No correlation: heartrate doesn't respond to load (broken physics)
|
||||
```
|
||||
|
||||
3. **Heartrate Stability (Not "jitter" but "rate consistency")**
|
||||
```
|
||||
- Does heartrate converge to a steady state?
|
||||
- How quickly does it adapt to workload changes?
|
||||
- Does it oscillate or settle smoothly?
|
||||
```
|
||||
|
||||
4. **Heartrate Efficiency**
|
||||
```
|
||||
- Is the system running faster/slower with adapted heartrate?
|
||||
- Workload duration: should improve with optimized heartrate
|
||||
- Memory: should be more efficient with tuned heartrate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Corrected Metrics Collection
|
||||
|
||||
### Per-Tick Collection (During Execution)
|
||||
|
||||
```c
|
||||
/* In heartbeat_thread_main(), before nanosleep() */
|
||||
struct HeartbeatTickData {
|
||||
uint64_t tick_number;
|
||||
uint64_t tick_ns; /* THIS IS THE HEARTBEAT RATE! */
|
||||
uint64_t workload_ops_this_tick; /* Dictionary lookups in this tick */
|
||||
uint64_t wall_clock_ns; /* Actual wall-clock time */
|
||||
uint64_t hot_word_count; /* Physics state */
|
||||
uint64_t total_heat; /* Physics state */
|
||||
};
|
||||
|
||||
/* Store in circular buffer or rolling window */
|
||||
```
|
||||
|
||||
### Post-Run Analysis Metrics
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
/* === Original Metrics === */
|
||||
/* ... existing 38 metrics ... */
|
||||
|
||||
/* === CORRECTED: Heartbeat Rate Metrics === */
|
||||
|
||||
/* Heartbeat rate trajectory */
|
||||
uint64_t total_ticks_executed;
|
||||
double mean_heartbeat_rate_hz; /* Hz = 1e9 / mean_tick_ns */
|
||||
double heartbeat_rate_stddev_hz; /* Variation in Hz */
|
||||
double heartbeat_rate_cv; /* Coefficient of variation */
|
||||
uint64_t heartbeat_rate_min_ns; /* Slowest (highest tick_ns) */
|
||||
uint64_t heartbeat_rate_max_ns; /* Fastest (lowest tick_ns) */
|
||||
double heartbeat_rate_range_hz; /* max_hz - min_hz */
|
||||
|
||||
/* Heartrate vs Load Coupling */
|
||||
double load_to_heartrate_correlation; /* Does rate respond to load? */
|
||||
int load_response_direction; /* +1 = slower under load (good) */
|
||||
double heartrate_adaptation_latency_ticks; /* Ticks until rate responds */
|
||||
double heartrate_settling_time_ticks; /* Ticks to settle after change */
|
||||
|
||||
/* Heartrate Convergence */
|
||||
double heartrate_convergence_time_ms; /* Time to reach stable rate */
|
||||
int heartrate_converged; /* 1 = yes, 0 = still oscillating */
|
||||
double heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation */
|
||||
|
||||
/* Overall Heartrate Stability Score */
|
||||
double heartrate_stability_score; /* 0-100: how stable is heartrate? */
|
||||
|
||||
} DoeMetrics_CORRECTED;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The Missing Piece: Heartbeat Rate Logging
|
||||
|
||||
**Critical code addition needed:**
|
||||
|
||||
```c
|
||||
/* In vm.c heartbeat_thread_main() - CAPTURE tick_ns BEFORE and AFTER */
|
||||
|
||||
static inline void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops) {
|
||||
/* Store sample in rolling buffer */
|
||||
if (!vm->heartbeat_rate_samples) {
|
||||
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
|
||||
}
|
||||
|
||||
HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
|
||||
sample->tick_number = vm->heartbeat.tick_count;
|
||||
sample->tick_ns = tick_ns; /* THE HEARTBEAT RATE! */
|
||||
sample->workload_ops = workload_ops;
|
||||
sample->timestamp_ns = platform_monotonic_ns();
|
||||
|
||||
vm->heartbeat_rate_sample_count++;
|
||||
}
|
||||
|
||||
/* In heartbeat_thread_main() */
|
||||
static void* heartbeat_thread_main(void *arg) {
|
||||
VM *vm = (VM*)arg;
|
||||
|
||||
while (!vm->heartbeat.worker->stop_requested) {
|
||||
uint64_t tick_start_ns = platform_monotonic_ns();
|
||||
|
||||
vm_heartbeat_run_cycle(vm); /* May adjust tick_ns */
|
||||
|
||||
uint64_t tick_ns = vm->heartbeat.worker->tick_ns;
|
||||
uint64_t workload_ops = vm->execution_heat_current_tick; /* Need to track */
|
||||
|
||||
/* CAPTURE THE HEARTBEAT RATE! */
|
||||
capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
|
||||
|
||||
/* Sleep */
|
||||
nanosleep(...);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Corrected Analysis (R Script)
|
||||
|
||||
```r
|
||||
# Load heartbeat rate samples from CSV
|
||||
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
|
||||
mutate(
|
||||
# Convert heartbeat rates (tick_ns → Hz)
|
||||
mean_heartbeat_hz = 1e9 / mean_heartbeat_rate_ns,
|
||||
heartrate_stability = 1 / (1 + heartbeat_rate_cv), # Lower CV = higher score
|
||||
)
|
||||
|
||||
# PRIMARY ANALYSIS: Heartrate Stability
|
||||
stability_by_heartrate <- doe_data %>%
|
||||
group_by(configuration) %>%
|
||||
summarize(
|
||||
mean_heartrate_hz = mean(mean_heartbeat_hz),
|
||||
mean_heartrate_stability = mean(heartrate_stability),
|
||||
mean_load_coupling = mean(load_to_heartrate_correlation),
|
||||
mean_workload_ms = mean(vm_workload_duration_ns_q48 / 1e6)
|
||||
) %>%
|
||||
mutate(
|
||||
# Golden config = best heartrate stability + strongest load coupling
|
||||
golden_score = (heartrate_stability * 0.5) + (load_coupling * 0.3) + (workload_improvement * 0.2)
|
||||
)
|
||||
|
||||
# Golden Config = Highest golden_score
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why This Matters
|
||||
|
||||
### Stage 1 Got Lucky
|
||||
Stage 1 found 5 good configs **by accident** because it measured final performance, not physics.
|
||||
|
||||
### Stage 2 Should Be Different
|
||||
Stage 2 should measure **whether the physics is working correctly**:
|
||||
|
||||
- **Can the system modulate heartbeat rate** in response to load?
|
||||
- **Is the modulation smooth** or does it oscillate?
|
||||
- **Does heartrate modulation improve overall performance?**
|
||||
- **Which configuration has the most responsive, stable heartrate?**
|
||||
|
||||
**The golden configuration should be the one where heartbeat rate modulation is:**
|
||||
1. ✓ **Responsive** (correlates with workload)
|
||||
2. ✓ **Smooth** (doesn't oscillate wildly)
|
||||
3. ✓ **Efficient** (results in best overall performance)
|
||||
|
||||
---
|
||||
|
||||
## Revised DoE Experiment Design
|
||||
|
||||
### What Actually Gets Measured
|
||||
|
||||
Instead of just "jitter in fixed 1ms tick intervals", measure:
|
||||
|
||||
**Per-Run Heartbeat Rate Trajectory:**
|
||||
```
|
||||
Tick 0: tick_ns = 1,000,000 (1 ms baseline)
|
||||
Tick 1: tick_ns = 1,500,000 (slowing under load)
|
||||
Tick 2: tick_ns = 2,000,000 (further slowdown)
|
||||
Tick 10: tick_ns = 1,200,000 (adapting back)
|
||||
Tick 100: tick_ns = 1,050,000 (converged to steady state)
|
||||
```
|
||||
|
||||
**What we analyze:**
|
||||
1. Did heartrate change at all? (some configs might keep it fixed)
|
||||
2. How much did it vary? (CV of tick_ns, not CV of fixed intervals)
|
||||
3. Did it respond to load? (correlation with workload)
|
||||
4. Did it converge smoothly? (no oscillation)
|
||||
5. Did it improve performance? (faster execution with adaptive heartrate?)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Checklist (Revised)
|
||||
|
||||
- [ ] Add `HeartbeatRateSample` struct to capture `tick_ns` per tick
|
||||
- [ ] Modify `heartbeat_thread_main()` to log `tick_ns` trajectory
|
||||
- [ ] Extend `DoeMetrics` with heartrate-specific fields (not just interval jitter)
|
||||
- [ ] Update CSV schema to include heartrate trajectory metrics
|
||||
- [ ] Modify R analysis to focus on **heartrate modulation patterns**
|
||||
- [ ] Update golden config selection: prioritize **load responsiveness**
|
||||
|
||||
---
|
||||
|
||||
## Expected Results (Corrected)
|
||||
|
||||
### Hypothesis: Configs Differ in Heartrate Modulation
|
||||
|
||||
**Config 1 (1_0_1_1_1_0) - Expected WINNER:**
|
||||
- Heartrate slowly increases under load (tick_ns increases, Hz decreases)
|
||||
- Smooth adaptation (no oscillation)
|
||||
- Converges within 1000 ticks
|
||||
- Performance improves (faster overall execution)
|
||||
- Load coupling: 0.8+ (strong)
|
||||
|
||||
**Config 5 (0_1_1_0_1_1) - Expected LOSER:**
|
||||
- Heartrate doesn't respond to load (fixed tick_ns)
|
||||
- No correlation with workload
|
||||
- Performance degrades (why? system can't adapt)
|
||||
|
||||
---
|
||||
|
||||
## Key Realization
|
||||
|
||||
The heartbeat isn't just a **timer**. It's a **feedback mechanism**:
|
||||
|
||||
```
|
||||
Load increases
|
||||
↓
|
||||
Inference engine detects change
|
||||
↓
|
||||
Adjusts tick_target_ns (slower heartbeat = more thinking time)
|
||||
↓
|
||||
Heartbeat rate modulates (tick_ns increases)
|
||||
↓
|
||||
Physics engine gets more time to tune parameters
|
||||
↓
|
||||
System adapts and performance improves
|
||||
```
|
||||
|
||||
**We need to measure whether this feedback loop is working!**
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Extend HeartbeatWorker:** Add rate sample collection
|
||||
2. **Modify heartbeat_thread_main():** Log `tick_ns` per tick
|
||||
3. **Update DoeMetrics:** Heartrate trajectory analysis
|
||||
4. **Rewrite CSV schema:** Focus on heartrate modulation
|
||||
5. **Rewrite R analysis:** Measure load-response coupling and heartrate stability
|
||||
6. **Re-run experiment:** Focus on configs that modulate heartrate best
|
||||
|
||||
---
|
||||
|
||||
**This changes the entire experiment focus.**
|
||||
|
||||
Instead of measuring "jitter in a fixed heartbeat", we're measuring "**how well does the system modulate its heartbeat in response to load?**"
|
||||
|
||||
That's the real physics story.
|
||||
@@ -1,431 +0,0 @@
|
||||
<!-- Moved from docs/02-experiments/heartbeat-doe/design.md to docs/working/experiments/02-experiments/heartbeat-doe/design.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ SUPERSEDED (flagged 2026-10-09).** Pre-correction draft; superseded first by
|
||||
> `corrected-design.md`, then by `phase2-corrected.md` (the finalized design). Kept for
|
||||
> historical record.
|
||||
|
||||
# StarForth Factorial DoE with Heartbeat Integration
|
||||
|
||||
**Date:** November 20, 2025
|
||||
**Status:** Design Phase
|
||||
**Experiment:** 2^6 Factorial with Heartbeat-Aware Metrics Collection
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the **second-generation factorial DoE** that incorporates heartbeat observability into metric collection. Building on the baseline DOE (Stage 1: 3200 runs, 6 factors, no heartbeat), this experiment captures **per-tick heartbeat dynamics** to identify configurations with optimal stability characteristics.
|
||||
|
||||
### Key Innovation
|
||||
|
||||
Instead of measuring static performance, this experiment measures **temporal stability**:
|
||||
- How quickly does the system converge under load changes?
|
||||
- How much does heartbeat interval modulate in response to workload?
|
||||
- What is the jitter envelope after settling?
|
||||
- Which configuration achieves smoothest load→heartbeat coupling?
|
||||
|
||||
---
|
||||
|
||||
## Design Points (Top 5 Configurations from Stage 1)
|
||||
|
||||
Based on Stage 1 analysis, the elite configurations are:
|
||||
|
||||
1. **Config 1:** `1_0_1_1_1_0` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=OFF)
|
||||
2. **Config 2:** `1_0_1_1_1_1` (F1=ON, F2=OFF, F3=ON, F4=ON, F5=ON, F6=ON)
|
||||
3. **Config 3:** `1_1_0_1_1_1` (F1=ON, F2=ON, F3=OFF, F4=ON, F5=ON, F6=ON)
|
||||
4. **Config 4:** `1_0_1_0_1_0` (F1=ON, F2=OFF, F3=ON, F4=OFF, F5=ON, F6=OFF)
|
||||
5. **Config 5:** `0_1_1_0_1_1` (F1=OFF, F2=ON, F3=ON, F4=OFF, F5=ON, F6=ON)
|
||||
|
||||
**Factors Mapping:**
|
||||
- F1 = ENABLE_LOOP_1_HEAT_TRACKING
|
||||
- F2 = ENABLE_LOOP_2_ROLLING_WINDOW
|
||||
- F3 = ENABLE_LOOP_3_LINEAR_DECAY
|
||||
- F4 = ENABLE_LOOP_4_PIPELINING_METRICS
|
||||
- F5 = ENABLE_LOOP_5_WINDOW_INFERENCE
|
||||
- F6 = ENABLE_LOOP_6_DECAY_INFERENCE
|
||||
|
||||
---
|
||||
|
||||
## New Heartbeat Metrics (Per-Tick Collection)
|
||||
|
||||
### 1. Heartbeat Interval Dynamics
|
||||
|
||||
```c
|
||||
struct HeartbeatTickMetrics {
|
||||
uint64_t tick_count; /* Absolute tick number */
|
||||
uint64_t tick_interval_ns; /* Wall-clock duration of this tick */
|
||||
uint64_t instantaneous_workload_ops; /* Dictionary lookups in this tick */
|
||||
double instantaneous_load_ratio; /* (ops this tick) / (avg ops/tick) */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Capture how heartbeat frequency modulates in response to workload intensity.
|
||||
|
||||
### 2. Cache Performance (Per-Tick Window)
|
||||
|
||||
```c
|
||||
struct CacheWindowMetrics {
|
||||
uint64_t tick_start;
|
||||
uint64_t cache_hits_window; /* Hits in last 100 ticks */
|
||||
uint64_t cache_attempts_window; /* Attempts in last 100 ticks */
|
||||
double cache_hit_percent_window; /* Rolling window %, 0-100 */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Measure cache stability, not just final hit rate.
|
||||
|
||||
### 3. Rolling Window Diagnostics
|
||||
|
||||
```c
|
||||
struct WindowDiagnostics {
|
||||
uint32_t effective_window_size; /* Current adaptive size */
|
||||
uint64_t pattern_diversity; /* Unique word IDs observed */
|
||||
double diversity_growth_rate; /* % new patterns per 1000 executions */
|
||||
int is_warm; /* 1 if window is representative */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Track window convergence and saturation.
|
||||
|
||||
### 4. Jitter Envelope
|
||||
|
||||
```c
|
||||
struct JitterMetrics {
|
||||
double tick_interval_mean_ns; /* Average heartbeat interval */
|
||||
double tick_interval_stddev_ns; /* StdDev of interval */
|
||||
double tick_interval_cv; /* Coefficient of variation (σ/μ) */
|
||||
uint64_t outlier_count; /* Ticks >3σ from mean */
|
||||
double outlier_ratio; /* Outliers / total ticks */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Measure temporal smoothness and jitter control.
|
||||
|
||||
### 5. Decay Slope Tracking
|
||||
|
||||
```c
|
||||
struct DecayTrajectory {
|
||||
uint64_t tick_sample_count; /* How many ticks sampled */
|
||||
double decay_slope_observed; /* Fitted slope from observations */
|
||||
double slope_convergence_rate; /* Speed of slope convergence */
|
||||
int slope_direction; /* -1=decreasing, 0=stable, +1=increasing */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Monitor whether decay slope inference is tracking correctly.
|
||||
|
||||
### 6. Load-Response Coupling
|
||||
|
||||
```c
|
||||
struct LoadResponseCoupling {
|
||||
double correlation_load_to_interval; /* Correlation(workload, heartbeat_interval) */
|
||||
double response_latency_ticks; /* Ticks until interval reacts to load change */
|
||||
double overshoot_ratio; /* Peak response / steady-state response */
|
||||
double settling_time_ticks; /* Ticks to settle within 10% of target */
|
||||
};
|
||||
```
|
||||
|
||||
**Purpose:** Measure how quickly and smoothly heartbeat responds to workload shifts.
|
||||
|
||||
---
|
||||
|
||||
## Extended DoeMetrics Struct
|
||||
|
||||
Add to `include/doe_metrics.h`:
|
||||
|
||||
```c
|
||||
typedef struct {
|
||||
/* === Original Metrics (preserved) === */
|
||||
/* ... existing 35+ metrics ... */
|
||||
|
||||
/* === NEW: Heartbeat Stability Metrics === */
|
||||
|
||||
/* Tick dynamics */
|
||||
uint64_t total_heartbeat_ticks; /* Total ticks during run */
|
||||
double tick_interval_mean_ns; /* Average heartbeat interval */
|
||||
double tick_interval_stddev_ns; /* StdDev of heartbeat interval */
|
||||
double tick_interval_cv; /* Coefficient of variation */
|
||||
uint64_t tick_outlier_count_3sigma; /* Ticks >3σ from mean */
|
||||
double tick_outlier_ratio; /* Outliers / total ticks */
|
||||
|
||||
/* Cache stability window */
|
||||
double cache_hit_percent_rolling_mean; /* Rolling avg, last 100 ticks */
|
||||
double cache_hit_percent_rolling_stddev; /* Rolling stddev */
|
||||
double cache_stability_index; /* How stable is cache hit %? */
|
||||
|
||||
/* Window convergence */
|
||||
uint32_t window_final_effective_size; /* Final adaptive window size */
|
||||
double pattern_diversity_final; /* Final unique patterns observed */
|
||||
double diversity_growth_final; /* Growth rate at end */
|
||||
int window_warmth_achieved; /* 1 = warm, 0 = still cold */
|
||||
|
||||
/* Decay slope convergence */
|
||||
double decay_slope_fitted_final; /* Final fitted decay slope */
|
||||
double decay_slope_convergence_rate; /* How fast did slope converge? */
|
||||
int decay_slope_stable; /* 1 = stable direction, 0 = oscillating */
|
||||
|
||||
/* Load-response */
|
||||
double load_interval_correlation; /* Correlation(load, heartbeat_interval) */
|
||||
double response_latency_ticks; /* Ticks until response to workload change */
|
||||
double overshoot_ratio; /* Peak response / steady-state */
|
||||
double settling_time_ticks; /* Time to 10% settling band */
|
||||
|
||||
/* Aggregate stability score */
|
||||
double overall_stability_score; /* Composite metric: (low jitter + fast convergence + tight correlation) */
|
||||
|
||||
} DoeMetrics; /* Now extends to ~65+ columns */
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Heartbeat Data Collection Architecture
|
||||
|
||||
### Per-Tick Logging (Lightweight)
|
||||
|
||||
During VM execution, capture minimal per-tick data:
|
||||
|
||||
```c
|
||||
/* In vm.c vm_tick_heartbeat_worker() */
|
||||
void vm_tick_heartbeat_worker(VM *vm) {
|
||||
HeartbeatSnapshot snapshot = {
|
||||
.published_tick = vm->heartbeat.tick_count,
|
||||
.published_ns = platform_monotonic_ns(),
|
||||
.window_width = vm->rolling_window.effective_window_size,
|
||||
.decay_slope_q48 = vm->decay_slope_q48,
|
||||
.hot_word_count = vm->hot_word_count_at_check,
|
||||
.stale_word_count = vm->stale_word_count_at_check,
|
||||
.total_heat = vm->total_heat_at_last_check,
|
||||
};
|
||||
|
||||
/* Publish snapshot (double-buffered, lock-free read) */
|
||||
int next_slot = 1 - vm->heartbeat.snapshot_index;
|
||||
vm->heartbeat.snapshots[next_slot] = snapshot;
|
||||
vm->heartbeat.snapshot_index = next_slot;
|
||||
}
|
||||
```
|
||||
|
||||
### Post-Run Analysis (Batch Processing)
|
||||
|
||||
After test execution completes, analyze collected tick data:
|
||||
|
||||
```c
|
||||
/* In src/doe_metrics.c metrics_from_vm() */
|
||||
void analyze_heartbeat_trajectory(VM *vm, DoeMetrics *metrics) {
|
||||
HeartbeatSnapshot *snapshots = vm->heartbeat.snapshots;
|
||||
uint64_t total_ticks = vm->heartbeat.tick_count;
|
||||
|
||||
/* 1. Compute tick interval statistics */
|
||||
double intervals[total_ticks];
|
||||
for (size_t i = 1; i < total_ticks; i++) {
|
||||
intervals[i] = snapshots[i].published_ns - snapshots[i-1].published_ns;
|
||||
}
|
||||
metrics->tick_interval_mean_ns = compute_mean(intervals, total_ticks);
|
||||
metrics->tick_interval_stddev_ns = compute_stddev(intervals, total_ticks);
|
||||
metrics->tick_interval_cv = metrics->tick_interval_stddev_ns / metrics->tick_interval_mean_ns;
|
||||
|
||||
/* 2. Compute outlier ratio (ticks >3σ from mean) */
|
||||
double threshold_3sigma = metrics->tick_interval_mean_ns + 3.0 * metrics->tick_interval_stddev_ns;
|
||||
uint64_t outliers = 0;
|
||||
for (size_t i = 0; i < total_ticks; i++) {
|
||||
if (intervals[i] > threshold_3sigma) outliers++;
|
||||
}
|
||||
metrics->tick_outlier_ratio = (double)outliers / total_ticks;
|
||||
|
||||
/* 3. Compute decay slope convergence */
|
||||
double slopes[total_ticks];
|
||||
for (size_t i = 100; i < total_ticks; i++) { /* Skip warm-up */
|
||||
slopes[i] = snapshots[i].decay_slope_q48 / 65536.0;
|
||||
}
|
||||
metrics->decay_slope_fitted_final = slopes[total_ticks - 1];
|
||||
metrics->decay_slope_convergence_rate = compute_convergence_rate(slopes, total_ticks);
|
||||
|
||||
/* 4. Compute overall stability score */
|
||||
double stability = 0.0;
|
||||
stability += (100.0 - metrics->tick_interval_cv); /* Lower CV = higher score */
|
||||
stability += (100.0 - metrics->tick_outlier_ratio * 100.0); /* Fewer outliers = higher */
|
||||
stability += metrics->decay_slope_convergence_rate; /* Fast convergence = higher */
|
||||
metrics->overall_stability_score = stability / 3.0;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CSV Output Schema (Extended)
|
||||
|
||||
**New columns (per heartbeat experiment run):**
|
||||
|
||||
```
|
||||
timestamp,configuration,run_number,
|
||||
[... original 38 columns ...],
|
||||
total_heartbeat_ticks,
|
||||
tick_interval_mean_ns,tick_interval_stddev_ns,tick_interval_cv,
|
||||
tick_outlier_count_3sigma,tick_outlier_ratio,
|
||||
cache_hit_percent_rolling_mean,cache_hit_percent_rolling_stddev,cache_stability_index,
|
||||
window_final_effective_size,pattern_diversity_final,diversity_growth_final,window_warmth_achieved,
|
||||
decay_slope_fitted_final,decay_slope_convergence_rate,decay_slope_stable,
|
||||
load_interval_correlation,response_latency_ticks,overshoot_ratio,settling_time_ticks,
|
||||
overall_stability_score
|
||||
```
|
||||
|
||||
**Total:** ~60 columns per run
|
||||
|
||||
---
|
||||
|
||||
## R Analysis Template
|
||||
|
||||
```r
|
||||
library(tidyverse)
|
||||
library(ggplot2)
|
||||
library(gridExtra)
|
||||
|
||||
# Load heartbeat DoE data
|
||||
doe_data <- read.csv("experiment_results_with_heartbeat.csv") %>%
|
||||
mutate(
|
||||
configuration = as.factor(configuration),
|
||||
timestamp = as.POSIXct(timestamp)
|
||||
)
|
||||
|
||||
# ========== MAIN ANALYSIS ==========
|
||||
|
||||
# 1. Stability Ranking
|
||||
stability_rank <- doe_data %>%
|
||||
group_by(configuration) %>%
|
||||
summarize(
|
||||
mean_stability_score = mean(overall_stability_score),
|
||||
mean_jitter_cv = mean(tick_interval_cv),
|
||||
mean_convergence_rate = mean(decay_slope_convergence_rate),
|
||||
mean_settling_time = mean(settling_time_ticks),
|
||||
n_runs = n()
|
||||
) %>%
|
||||
mutate(
|
||||
rank = rank(-mean_stability_score, ties.method = "min")
|
||||
) %>%
|
||||
arrange(rank)
|
||||
|
||||
print("=== STABILITY RANKINGS ===")
|
||||
print(stability_rank)
|
||||
|
||||
# 2. Visualization: Stability Scores
|
||||
ggplot(doe_data, aes(x = reorder(configuration, overall_stability_score, FUN=median),
|
||||
y = overall_stability_score,
|
||||
fill = configuration)) +
|
||||
geom_boxplot(alpha = 0.7) +
|
||||
geom_jitter(width = 0.2, alpha = 0.4) +
|
||||
coord_flip() +
|
||||
theme_minimal() +
|
||||
labs(title = "Overall Stability Scores by Configuration",
|
||||
x = "Configuration",
|
||||
y = "Stability Score (0-100)")
|
||||
ggsave("01_stability_scores.png", width = 10, height = 6)
|
||||
|
||||
# 3. Visualization: Jitter Control (CV)
|
||||
ggplot(doe_data, aes(x = reorder(configuration, tick_interval_cv, FUN=median),
|
||||
y = tick_interval_cv,
|
||||
fill = configuration)) +
|
||||
geom_boxplot(alpha = 0.7) +
|
||||
coord_flip() +
|
||||
theme_minimal() +
|
||||
labs(title = "Heartbeat Jitter (Coefficient of Variation)",
|
||||
x = "Configuration",
|
||||
y = "CV (lower = less jitter)")
|
||||
ggsave("02_jitter_control.png", width = 10, height = 6)
|
||||
|
||||
# 4. Visualization: Convergence Speed
|
||||
ggplot(doe_data, aes(x = reorder(configuration, decay_slope_convergence_rate, FUN=median),
|
||||
y = decay_slope_convergence_rate,
|
||||
fill = configuration)) +
|
||||
geom_boxplot(alpha = 0.7) +
|
||||
coord_flip() +
|
||||
theme_minimal() +
|
||||
labs(title = "Decay Slope Convergence Speed",
|
||||
x = "Configuration",
|
||||
y = "Convergence Rate (ticks)")
|
||||
ggsave("03_convergence_speed.png", width = 10, height = 6)
|
||||
|
||||
# 5. Correlation: Load Response Coupling
|
||||
ggplot(doe_data, aes(x = load_interval_correlation,
|
||||
y = overall_stability_score,
|
||||
color = configuration)) +
|
||||
geom_point(alpha = 0.6, size = 2) +
|
||||
facet_wrap(~configuration) +
|
||||
theme_minimal() +
|
||||
labs(title = "Load-Response Coupling vs Stability",
|
||||
x = "Correlation(load, interval)",
|
||||
y = "Overall Stability Score")
|
||||
ggsave("04_load_response_coupling.png", width = 12, height = 8)
|
||||
|
||||
# 6. Statistical Test: Is top config significantly better?
|
||||
top_config <- stability_rank$configuration[1]
|
||||
other_configs <- stability_rank$configuration[-1]
|
||||
|
||||
top_data <- doe_data %>% filter(configuration == top_config) %>% pull(overall_stability_score)
|
||||
other_data <- doe_data %>% filter(configuration %in% other_configs) %>% pull(overall_stability_score)
|
||||
|
||||
t_test <- t.test(top_data, other_data)
|
||||
print(sprintf("\nT-test: %s vs others\np-value: %.4f", top_config, t_test$p.value))
|
||||
|
||||
# 7. Golden Configuration Summary
|
||||
cat("\n=== GOLDEN CONFIGURATION CANDIDATE ===\n")
|
||||
golden <- stability_rank %>% slice(1)
|
||||
print(golden)
|
||||
|
||||
cat("\nInterpretation:")
|
||||
cat(sprintf("\n- Lowest jitter (CV): %.4f", golden$mean_jitter_cv))
|
||||
cat(sprintf("\n- Fastest convergence: %.1f ticks", golden$mean_convergence_rate))
|
||||
cat(sprintf("\n- Settling time: %.1f ticks", golden$mean_settling_time))
|
||||
cat(sprintf("\n- Recommendation: Use '%s' as golden configuration\n", golden$configuration))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution Plan
|
||||
|
||||
### Phase 1: Implementation (3-4 hours)
|
||||
1. Extend `DoeMetrics` struct with heartbeat columns
|
||||
2. Implement `analyze_heartbeat_trajectory()` in `doe_metrics.c`
|
||||
3. Capture snapshot data during heartbeat ticks
|
||||
4. Update CSV schema
|
||||
|
||||
### Phase 2: Experiment Execution (2-4 hours)
|
||||
1. Create `run_factorial_doe_with_heartbeat.sh` script
|
||||
2. Run 5 elite configurations × 50 runs each = 250 total runs
|
||||
3. Collect heartbeat + performance data
|
||||
|
||||
### Phase 3: Analysis (2-3 hours)
|
||||
1. Load CSV in R
|
||||
2. Compute stability rankings
|
||||
3. Generate visualizations
|
||||
4. Identify golden configuration
|
||||
|
||||
---
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
**Primary Question:** Which of the 5 elite configurations exhibits:
|
||||
- Lowest heartbeat jitter (CV)?
|
||||
- Fastest convergence to steady state?
|
||||
- Strongest load-response correlation?
|
||||
- Cleanest temporal signature?
|
||||
|
||||
**Expected Winner:** Configuration with balanced F-factors, likely:
|
||||
- F1=1 (heat tracking required)
|
||||
- F3=1 or F5=1 (decay or inference, but not both)
|
||||
- F6=0 or F6=1 (decay inference optional)
|
||||
|
||||
**Golden Configuration:** Becomes baseline physics for MamaForth & PapaForth downstream work.
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✓ All 5 configurations build and run without segfault
|
||||
✓ Heartbeat data collected for all 250 runs
|
||||
✓ Stability rankings computed and visualized
|
||||
✓ Golden configuration identified with p < 0.05 significance
|
||||
✓ R analysis report generated with 5+ plots
|
||||
|
||||
---
|
||||
|
||||
**Version:** 1.0
|
||||
**Status:** Ready for implementation
|
||||
**Next Step:** Implement heartbeat metrics capture and extended DoE script
|
||||
Reference in New Issue
Block a user