docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such
docs/CLAUDE.md Phase 2, 2026-10-14. All moves by git mv; every archived file carries a header saying what it is, why, and where to look instead. - docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL design, kernel args, hosted acceptance test, dictionary, primitives, ontology, L8 table -- previously spread over docs/, docs/lithosananke/ and docs/03-architecture/. - docs/working/archive/historical/: finished work (session plan, test-gap audit, birthing plan, fix write-ups, the January architecture draft). - docs/working/archive/superseded/: the Hermes-VM Tripod docs and July Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working doc already flagged superseded or obsolete. - docs/working/archive/dead/: the hal_* target contract (never adopted), L4Re, metrics-and-knobs. - Papers and proposals into docs/working/papers/. - References rewritten across the repo (not logs/, not docs/patent/); CLAUDE.md's documentation section rewritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
1a4b163861
commit
7aed588f78
@@ -1,362 +0,0 @@
|
||||
<!-- Moved from docs/03-architecture/physics-engine/feedback-loops-analysis.md to docs/working/architecture/03-architecture/physics-engine/feedback-loops-analysis.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ SUPERSEDED (flagged 2026-10-09).** Precursor bug-hunt audit covering six feedback loops;
|
||||
> the canonical, current reference is `feedback-loops.md` (all seven loops, tuning knobs, and
|
||||
> source files). Kept for historical record.
|
||||
|
||||
# StarForth Feedback Loops & Self-Optimization Analysis
|
||||
|
||||
## Executive Summary
|
||||
|
||||
StarForth implements **6 feedback loops** for automatic self-tuning during execution. Of these:
|
||||
- **3 fully wired AND operational** ✅ (actively driving optimization decisions)
|
||||
1. ✅ Hot-words cache promotion (1.78× speedup measured)
|
||||
2. ✅ Rolling window adaptive shrinking (FIXED 2025-11-08, enforcement now works)
|
||||
4. ✅ Pipelining transition speculation (WIRED 2025-11-08, speculative prefetch active)
|
||||
- **1 wired AND utilized, but NOT validated** ⚠️ (running, but unsupported design choices)
|
||||
3. ⚠️ Heat decay of execution history (arbitrary linear function, no tuning knob, unvalidated)
|
||||
- **1 wired but NOT utilized** ⏳ (Phase 2 experimental, infrastructure ready)
|
||||
5. Context-aware window tuning (binary chop algorithm stub, needs VM-level metrics aggregation)
|
||||
- **1 observability-only** (metrics collected for formal verification)
|
||||
6. Physics metadata temperature tracking
|
||||
|
||||
**CURRENT STATUS:**
|
||||
- A_BASELINE, A_B_CACHE, A_C_FULL, A_B_C_FULL configurations now test complete optimization stack:
|
||||
- Loop #1 (cache) enabled via A_B_CACHE flag
|
||||
- Loop #2 (window shrinking) always enabled, active in all configs
|
||||
- Loop #3 (decay) always enabled, active in all configs
|
||||
- Loop #4 (speculation) disabled by default (ENABLE_PIPELINING=0), ready for A_C_FULL testing
|
||||
- **BUT:** Loop #3 (heat decay) lacks principled design—slope is arbitrary, function choice unvalidated
|
||||
- **Gap:** No measurements showing decay actually helps; DECAY_RATE has NO DoE knob to tune it
|
||||
|
||||
---
|
||||
|
||||
## FULLY WIRED & UTILIZED FEEDBACK LOOPS ✅
|
||||
|
||||
### ~~1. Hot-Words Cache Promotion Feedback Loop~~ ✅ DONE
|
||||
|
||||
**What's Measured:**
|
||||
- `execution_heat` counter on each dictionary entry (incremented per execution)
|
||||
|
||||
**Trigger Condition:**
|
||||
- When `execution_heat > HOTWORDS_EXECUTION_HEAT_THRESHOLD` (default: 50)
|
||||
|
||||
**Automatic Action:**
|
||||
- `hotwords_cache_promote()` moves word to 32-entry LRU cache
|
||||
- Subsequent lookups hit cache (1-2 ns vs 20-30 ns in bucket)
|
||||
|
||||
**Code Locations:**
|
||||
- Metric tracking: `src/dictionary_management.c:170`
|
||||
- Threshold check: `src/physics_hotwords_cache.c:155`
|
||||
- Promotion logic: `src/physics_hotwords_cache.c:420-450`
|
||||
|
||||
**Observed Impact:**
|
||||
- 1.78× speedup on cache-hit paths (measured in DoE phase 1)
|
||||
- **Status: FULLY OPERATIONAL** ✅
|
||||
|
||||
---
|
||||
|
||||
### ~~2. Rolling Window Adaptive Shrinking Feedback Loop~~ ✅ FIXED
|
||||
|
||||
**What's Measured:**
|
||||
- Pattern diversity: count of unique word transitions in execution history
|
||||
|
||||
**Measurement Function:**
|
||||
```c
|
||||
// src/rolling_window_of_truth.c:546-577 (FIXED)
|
||||
uint64_t rolling_window_measure_diversity(const RollingWindowOfTruth* window)
|
||||
{
|
||||
// Count unique adjacent transitions (word_a → word_b)
|
||||
// NOW RESPECTS effective_window_size - only scans recent entries
|
||||
}
|
||||
```
|
||||
|
||||
**Trigger Condition:**
|
||||
- When pattern diversity growth rate < `ADAPTIVE_GROWTH_THRESHOLD` (default: 1%)
|
||||
- Checked every `ADAPTIVE_CHECK_FREQUENCY` executions (default: 256)
|
||||
|
||||
**Automatic Action:**
|
||||
- Shrink `effective_window_size` to `ADAPTIVE_SHRINK_RATE` percent of current (default: 75%)
|
||||
- Window shrinks: 4096 → 3072 → 2304 → 1728 → ... → `ADAPTIVE_MIN_WINDOW_SIZE` (256)
|
||||
|
||||
**Code Locations:**
|
||||
- Check function: `src/rolling_window_of_truth.c:589-645`
|
||||
- Shrink calculation: Line 626
|
||||
- Knobs: `include/rolling_window_knobs.h`
|
||||
|
||||
**Integration:**
|
||||
- Called from `rolling_window_record_execution()` on every word execution
|
||||
- **Status: FULLY OPERATIONAL** ✅
|
||||
|
||||
**Fix Applied (2025-11-08):**
|
||||
- Line 559: Now limits scan to `effective_window_size` when window is warm
|
||||
- Lines 561-564: Circular indexing ensures only recent entries within effective window are scanned
|
||||
- **Effect:** New data arrives every tick, old data falls off after `effective_window_size` ticks (was: always 4096)
|
||||
|
||||
---
|
||||
|
||||
### 3. Linear Decay of Execution Heat Feedback Loop
|
||||
|
||||
**What's Measured:**
|
||||
- `execution_heat` value per word (time-sensitive metric)
|
||||
|
||||
**Decay Formula (INTENTIONALLY BASELINE):**
|
||||
- `H(t) = max(0, H_0 - decay_rate × Δt)` — **LINEAR decay**
|
||||
- Configurable decay rate: `DECAY_RATE_PER_US_Q16` (default: 1 = 1/65536 heat/µs)
|
||||
- Derived half-life: ~6-7 seconds for 100-heat word
|
||||
|
||||
**Automatic Action:**
|
||||
- Stale words automatically lose heat weight
|
||||
- Only frequently-executed words stay above promotion threshold
|
||||
|
||||
**Code Locations:**
|
||||
- Decay logic: `src/physics_metadata.c:164-203` (`physics_metadata_apply_linear_decay()`)
|
||||
- Integration: `src/vm.c:524` (called before each execution in inner loop)
|
||||
|
||||
**Design Rationale (From Phase 2 Specification):**
|
||||
- Chose LINEAR over exponential for:
|
||||
1. Performance (integer arithmetic, no floating-point)
|
||||
2. Predictability (bounded convergence time)
|
||||
3. Simplicity (fewer lines, easier to verify formally)
|
||||
4. StarshipOS applicability (models "task context switch forces heat reset")
|
||||
- **Explicitly noted as baseline:** "Can always switch to halflife after validating linear approach"
|
||||
|
||||
**STATUS: WIRED & UTILIZED, BUT NOT VALIDATED** ⚠️
|
||||
|
||||
**CRITICAL GAPS:**
|
||||
1. **No empirical validation**: Does linear decay actually help? No measurements.
|
||||
2. **Arbitrary initial slope**: Why 1 µs⁻¹? No evidence this is optimal.
|
||||
3. **Wrong function?**: Could exponential, log, parabolic, or sinusoidal be better?
|
||||
4. **Tuning blind**: DECAY_RATE_PER_US_Q16 has NO KNOB in DoE configurations
|
||||
5. **No feedback loop closure**: Decay happens, but no measurement of whether it improved anything
|
||||
|
||||
---
|
||||
|
||||
## ✅ IMPLEMENTATION FIXES APPLIED
|
||||
|
||||
### Rolling Window Shrinking - FIXED
|
||||
|
||||
**The Bug (Now Fixed):**
|
||||
The adaptive shrinking mechanism was shrinking the reported metric without enforcing the limit.
|
||||
|
||||
**Old Code (Line 557):**
|
||||
```c
|
||||
for (uint32_t i = 0; i < ROLLING_WINDOW_SIZE; i++) // ← ALWAYS full size (4096)
|
||||
{
|
||||
uint32_t current = window->execution_history[i];
|
||||
// Count unique transitions...
|
||||
}
|
||||
```
|
||||
|
||||
**Fixed Code (Lines 559-574):**
|
||||
```c
|
||||
uint32_t scan_limit = (window->is_warm) ? window->effective_window_size : ROLLING_WINDOW_SIZE;
|
||||
|
||||
for (uint32_t i = 0; i < scan_limit; i++)
|
||||
{
|
||||
/* Index into the recent window (window_pos is the next write position) */
|
||||
uint32_t idx = (window->window_pos + ROLLING_WINDOW_SIZE - scan_limit + i) % ROLLING_WINDOW_SIZE;
|
||||
uint32_t current = window->execution_history[idx];
|
||||
// Count unique transitions in recent entries only
|
||||
}
|
||||
```
|
||||
|
||||
**What Changed:**
|
||||
- Now limits scan to `effective_window_size` when window is warm
|
||||
- Uses circular indexing to ensure only RECENT entries are scanned
|
||||
- New data arrives every tick, old data falls off after `effective_window_size` ticks
|
||||
|
||||
**User's Requirement Met:**
|
||||
> "The window should advance with every tick so new data comes in and stale data falls off."
|
||||
|
||||
✅ Now data falls off after `effective_window_size` ticks, not always 4096
|
||||
|
||||
---
|
||||
|
||||
## ~~4. Pipelining Transition Metrics~~ ✅ NOW FUNCTIONAL
|
||||
|
||||
**What's Measured:**
|
||||
- Transition patterns: (word_A → word_B) frequencies via `transition_metrics_record()`
|
||||
- Probability calculations: P(B | A) in Q48.16 fixed-point format
|
||||
- Prefetch accuracy: Hits vs. misses from speculative cache promotions
|
||||
|
||||
**Trigger Conditions:**
|
||||
- `SPECULATION_THRESHOLD_Q48` (50% minimum confidence)
|
||||
- `MIN_SAMPLES_FOR_SPECULATION` (10 minimum observations)
|
||||
|
||||
**Automatic Action (NOW WIRED):**
|
||||
- After recording each transition, update probability cache
|
||||
- Check: Does next word probability exceed threshold?
|
||||
- If yes: Find dictionary entry and pre-promote to hotwords cache
|
||||
- Effect: When next word is actually looked up, it's cache-warm
|
||||
- Record: Track prefetch_attempts for feedback measurement
|
||||
|
||||
**Code Locations:**
|
||||
- Metric collection: `src/physics_pipelining_metrics.c:92-109` (`transition_metrics_record()`)
|
||||
- Probability calculation: `src/physics_pipelining_metrics.c:112-123`
|
||||
- Decision logic: `src/physics_pipelining_metrics.c:164-186` (`transition_metrics_should_speculate()`)
|
||||
- **Action wiring: `src/vm.c:544-583`** (execute_colon_word, speculative prefetch logic)
|
||||
|
||||
**Status: FULLY WIRED & FUNCTIONAL** ✅
|
||||
|
||||
**Wiring Details (2025-11-08):**
|
||||
- Line 548: `transition_metrics_record()` - record the transition
|
||||
- Line 551: `transition_metrics_update_cache()` - find most likely next word
|
||||
- Line 554-555: `transition_metrics_should_speculate()` - check confidence threshold
|
||||
- Lines 560-581: If confident, look up predicted word and promote to hotwords cache
|
||||
- Line 580: Record the prefetch attempt for metrics feedback
|
||||
|
||||
**Loop Closure:**
|
||||
1. ✅ Metrics collected: transition_heat array counts (word_B | word_A)
|
||||
2. ✅ Probability calculated: Q48.16 format
|
||||
3. ✅ Decision made: Confidence check against threshold
|
||||
4. ✅ Action taken: Speculative promotion to hotwords cache
|
||||
5. ✅ Feedback recorded: prefetch_attempts counter
|
||||
|
||||
### 5. Context-Aware Window Size Tuning (Phase 2 Placeholder)
|
||||
|
||||
**What's Measured:**
|
||||
- Multi-word context patterns (sequences of 2-4 consecutive words)
|
||||
|
||||
**Intended Action (Not Connected):**
|
||||
- Use binary search to find optimal `effective_window_size` for given workload
|
||||
- Function: `transition_metrics_binary_chop_suggest_window()` (lines 340-357)
|
||||
|
||||
**Why Not Utilized:**
|
||||
- Phase 2 research feature
|
||||
- Experimental algorithm not yet validated
|
||||
- No integration point in main execution path
|
||||
|
||||
**Status: PROTOTYPE, NOT INTEGRATED** ⏳
|
||||
|
||||
---
|
||||
|
||||
## OBSERVABILITY-ONLY (Not Decision-Driving)
|
||||
|
||||
### 6. Physics Metadata Temperature Tracking
|
||||
|
||||
**What's Measured:**
|
||||
- `temperature_q8`: Exponential moving average of execution_heat
|
||||
- Used for observability and formal verification model tracing
|
||||
|
||||
**Current Use:**
|
||||
- Profiling and debugging (not driving optimization decisions)
|
||||
- Forms foundation for Isabelle/HOL physics state machine proofs
|
||||
|
||||
**Status: WORKING AS INTENDED** (observability, not optimization)
|
||||
|
||||
---
|
||||
|
||||
## Bootstrap Seeding Mechanisms
|
||||
|
||||
These run once at startup to initialize feedback loop state:
|
||||
|
||||
### POST (Power-On Self-Test) Seeding
|
||||
- **Function**: `rolling_window_seed_hotwords_cache()` (lines 258-329)
|
||||
- **Action**: Records POST execution, promotes high-heat words to cache
|
||||
- **Effect**: Cache is "warm" when user workload begins
|
||||
|
||||
### Pipelining Context Seeding
|
||||
- **Function**: `rolling_window_seed_pipelining_context()` (lines 341-415)
|
||||
- **Action**: Records POST sequences, establishes transition baseline
|
||||
- **Effect**: Pipelining has initial pattern knowledge
|
||||
|
||||
---
|
||||
|
||||
## Configuration Knobs for Feedback Loops
|
||||
|
||||
All knobs are **tunable via Makefile**:
|
||||
|
||||
```bash
|
||||
# Hot-words cache promotion
|
||||
make HOTWORDS_EXECUTION_HEAT_THRESHOLD=75
|
||||
|
||||
# Adaptive window shrinking
|
||||
make ADAPTIVE_SHRINK_RATE=75 # Keep 75%, shrink by 25%
|
||||
make ADAPTIVE_MIN_WINDOW_SIZE=256 # Never shrink below 256
|
||||
make ADAPTIVE_CHECK_FREQUENCY=256 # Check every 256 executions
|
||||
make ADAPTIVE_GROWTH_THRESHOLD=1 # Shrink when growth < 1%
|
||||
|
||||
# Heat decay
|
||||
make DECAY_RATE_PER_US_Q16=16384 # Decay rate (units: heat/μs)
|
||||
make DECAY_MIN_INTERVAL=1000000 # Min interval between decays (ns)
|
||||
|
||||
# Pipelining (Phase 2, disabled by default)
|
||||
make ENABLE_PIPELINING=1
|
||||
make SPECULATION_THRESHOLD_Q48=$((50 << 16)) # 50% confidence
|
||||
make MIN_SAMPLES_FOR_SPECULATION=10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary Table: All Feedback Loops
|
||||
|
||||
| Loop | Measured | Threshold | Action | Wired? | Utilized? | Status |
|
||||
|------|----------|-----------|--------|--------|-----------|--------|
|
||||
| ~~1. Hot-words promotion~~ | `execution_heat` | > 50 | Add to cache | ✅ | ✅ | ✅ VALIDATED |
|
||||
| ~~2. Adaptive shrinking~~ | `pattern_diversity` | growth < 1% | Shrink window | ✅ | ✅ | ✅ FIXED |
|
||||
| 3. Heat decay | Time since last execution | ~6-7s half-life | Reduce heat | ✅ | ✅ | ⚠️ NOT VALIDATED |
|
||||
| 4. Pipelining transitions | `transition_heat[]` | > 50% confidence | Speculative prefetch | ✅ | ❌ | ⏳ Phase 2 |
|
||||
| 5. Context window tuning | Multi-word sequences | N/A | Binary chop | ✅ | ❌ | ⏳ Phase 2 |
|
||||
| 6. Temperature tracking | `temperature_q8` | N/A | Observability only | ✅ | ✅ | ✅ OBSERVABILITY |
|
||||
|
||||
---
|
||||
|
||||
## Completed Actions ✅
|
||||
|
||||
### ✅ DONE: Fix Rolling Window Shrinking Implementation
|
||||
|
||||
**Status: APPLIED (2025-11-08)**
|
||||
|
||||
Modified `src/rolling_window_of_truth.c:546-577` (function `rolling_window_measure_diversity()`):
|
||||
- Now respects `effective_window_size` when window is warm
|
||||
- Uses circular indexing to scan only recent entries
|
||||
- New data arrives every tick, old data falls off after `effective_window_size` ticks (not always 4096)
|
||||
- A_B_C_FULL configuration now tests both hot-words cache + adaptive window shrinking together
|
||||
|
||||
---
|
||||
|
||||
## Next Steps for Phase 2 (If Proceeding)
|
||||
|
||||
### OPTIONAL 1: Connect Pipelining Decision Logic
|
||||
|
||||
**When `ENABLE_PIPELINING=1`:**
|
||||
- Call `transition_metrics_should_speculate()` from vm inner loop
|
||||
- If returns true: Pre-load next word from context prediction
|
||||
- Measure cache miss reduction vs. A_BASELINE (Phase 2 experimental validation)
|
||||
|
||||
**Status:** Metrics collection ready, decision logic not wired
|
||||
|
||||
### OPTIONAL 2: Validate Context Window Tuning (Phase 2)
|
||||
|
||||
**For adaptive window size optimization:**
|
||||
- Implement binary chop search algorithm in main execution path
|
||||
- Tune `effective_window_size` dynamically based on context complexity
|
||||
- Measure pattern capture efficiency vs. memory overhead
|
||||
|
||||
**Status:** Binary chop function exists, integration point not defined
|
||||
|
||||
---
|
||||
|
||||
## Files & Line Numbers Reference
|
||||
|
||||
| Purpose | File | Lines |
|
||||
|---------|------|-------|
|
||||
| Rolling window structure | `include/vm.h` | 58-70 |
|
||||
| Rolling window implementation | `src/rolling_window_of_truth.c` | - |
|
||||
| Adaptive shrinking | `src/rolling_window_of_truth.c` | 583-639 |
|
||||
| **Bug location** | `src/rolling_window_of_truth.c` | **557** |
|
||||
| Diversity measurement | `src/rolling_window_of_truth.c` | 546-571 |
|
||||
| Shrinking knobs | `include/rolling_window_knobs.h` | - |
|
||||
| Hot-words cache | `src/physics_hotwords_cache.c` | - |
|
||||
| Promotion threshold | `src/physics_hotwords_cache.c` | 155 |
|
||||
| Heat decay | `src/physics_metadata.c` | 164-203 |
|
||||
| Pipelining metrics | `src/physics_pipelining_metrics.c` | - |
|
||||
| Pipelining decision | `src/physics_pipelining_metrics.c` | 164-186 |
|
||||
| Main integration | `src/vm.c` | 524, 542-548 |
|
||||
|
||||
---
|
||||
|
||||
**Analysis Complete:** 2025-11-08
|
||||
**User Request:** Investigate feedback loops, identify implementation problems
|
||||
**Finding:** 3 active + 2 dormant + 1 observability feedback loops; 1 critical bug in rolling window shrinking
|
||||
@@ -1,145 +0,0 @@
|
||||
<!-- Moved from docs/03-architecture/physics-engine/metrics-and-knobs.md to docs/working/architecture/03-architecture/physics-engine/metrics-and-knobs.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
# StarForth Metrics & Knobs: GAPS ONLY
|
||||
|
||||
## What's NOT Wired (Things to Add)
|
||||
|
||||
### DEAD CODE / NEVER CALLED
|
||||
|
||||
| What | Location | Status | Comment |
|
||||
|------|----------|--------|---------|
|
||||
| Bucket reordering | `physics_hotwords_cache.c:201-215` | Defined but NEVER CALLED | Bubble sort exists but no automatic trigger. Is this dead code or forgotten? |
|
||||
|
||||
---
|
||||
|
||||
### STACK METRICS (ZERO COLLECTION)
|
||||
|
||||
| Metric | File | Missing Since | Impact | Knob Needed |
|
||||
|--------|------|---|---|---|
|
||||
| Data stack peak depth | `stack_management.c` | Day 1 | Can't tell if workload needs larger stack | STACK_SIZE tuning blind |
|
||||
| Return stack peak depth | `stack_management.c` | Day 1 | Can't measure colon definition nesting | Tail-call optimization impossible |
|
||||
| Underflow attempts | `arithmetic_words.c:59-62` | Day 1 | Can't detect fragile code paths | Could optimize checks if zero underflows observed |
|
||||
| Overflow attempts | `arithmetic_words.c` | Day 1 | Can't detect stack exhaustion | Early warning system missing |
|
||||
|
||||
---
|
||||
|
||||
### DICTIONARY METRICS (ZERO VISIBILITY)
|
||||
|
||||
| Metric | File | Missing Since | Impact | Why Matters |
|
||||
|--------|------|---|---|---|
|
||||
| Bucket load histogram | `vm.c: dict lookup` | Day 1 | First-char bucketing (26 buckets) may be unbalanced | Affects lookup latency, collision clustering |
|
||||
| Dictionary growth rate | `memory_management.c` | Day 1 | No visibility into `here` pointer velocity | Can't predict "dictionary full" events |
|
||||
| FORGET usage frequency | `dictionary_words.c` | Day 1 | No trace of word deletion patterns | Fragmentation unknown |
|
||||
| Lookup failure rate | `vm.c` | Day 1 | How often is a word not found? | Error path frequency unknown |
|
||||
| Word name length distribution | `vm.c` | Day 1 | Average name length unknown | Affects memory efficiency |
|
||||
|
||||
---
|
||||
|
||||
### BLOCK I/O METRICS (COMPLETELY DARK)
|
||||
|
||||
| Metric | File | Missing Since | Impact | Phase |
|
||||
|--------|------|---|---|---|
|
||||
| Block read frequency | `block_subsystem.c` | Day 1 | How often is BLOCK called? | DoE phase 1 |
|
||||
| Block write frequency | `block_subsystem.c` | Day 1 | How often is UPDATE/FLUSH called? | DoE phase 1 |
|
||||
| Writeback frequency | `block_subsystem.c:198-200` | Day 1 | When are dirty blocks flushed? | DoE phase 1 |
|
||||
| Cache hit rate (disk) | `block_subsystem.c` | Day 1 | Blocks served from RAM cache % | DoE phase 1 |
|
||||
| Dirty block ratio | `block_subsystem.c` | Day 1 | Blocks pending flush at time T | DoE phase 1 |
|
||||
| I/O latency | `block_subsystem.c` | Day 1 | Time per read/write operation | DoE phase 2 |
|
||||
| Fragmentation ratio | `block_subsystem.c` | Day 1 | Gaps in allocated blocks | DoE phase 2 |
|
||||
|
||||
---
|
||||
|
||||
### PER-WORD EXECUTION METRICS
|
||||
|
||||
| Metric | File | Missing Since | Impact | Use Case |
|
||||
|--------|------|---|---|---|
|
||||
| Per-word execution frequency | `vm.c: inner loop` | Day 1 | Which words are actually hot? | Profiling, hotspot detection |
|
||||
| Per-word latency distribution | `vm.c` | Day 1 | Median/P99 time per word | Performance regression detection |
|
||||
| Word category breakdown | `word_source/*.c` | Day 1 | Arithmetic vs. Stack vs. Control dominance | Workload characterization |
|
||||
| Colon vs. native ratio | `vm.c` | Day 1 | User-defined word execution frequency | Code complexity analysis |
|
||||
|
||||
---
|
||||
|
||||
### HEAT DYNAMICS (PHASE 2 NOT ACTIVE)
|
||||
|
||||
| Metric | File | Status | Comment |
|
||||
|--------|------|--------|---------|
|
||||
| Heat decay trace | `physics_metadata.c: TODO` | Phase 2 placeholder | Need to measure if decay causes pattern loss |
|
||||
| Heat percentile distribution | `physics_metadata.c` | Not updating dynamic | Stored but never dynamically recalculated |
|
||||
| Heat concentration ratio | `physics_metadata.c` | Not tracked | (Top 10 words heat) / (total heat) |
|
||||
| Heat velocity | `physics_metadata.c` | Not tracked | Rate of heat change per unit time |
|
||||
| Heat plateau detection | `physics_metadata.c` | Not tracked | When does word heat level off? |
|
||||
|
||||
---
|
||||
|
||||
### PIPELINING METRICS (PHASE 1 INSTRUMENTATION, NOT USED)
|
||||
|
||||
| Metric | File | Status | Phase 2 Use |
|
||||
|--------|------|--------|---|
|
||||
| Prediction accuracy by depth | `physics_pipelining_metrics.c` | Not measured | Binary chop TRANSITION_WINDOW_SIZE |
|
||||
| Context transition frequency | `physics_pipelining_metrics.c` | Counted but not reported | Validate context patterns |
|
||||
| Speculation ROI histogram | `physics_pipelining_metrics.c` | Not aggregated | Tune SPECULATION_THRESHOLD |
|
||||
| Misprediction cost (actual) | `physics_pipelining_metrics.c` | Measured but not validated | Calibrate cost model |
|
||||
| Pipeline stall reduction | `physics_pipelining_metrics.c` | Not measured | Throughput impact unknown |
|
||||
|
||||
---
|
||||
|
||||
### CACHE POLLUTION & CYCLES
|
||||
|
||||
| Metric | File | Missing Since | Impact |
|
||||
|--------|------|---|---|
|
||||
| Cache eviction/re-promotion cycles | `physics_hotwords_cache.c` | Day 1 | Are we thrashing? |
|
||||
| False negatives (words evicted before promotion) | `physics_hotwords_cache.c` | Day 1 | Should promotion threshold be lower? |
|
||||
| Cache eviction candidate value | `physics_hotwords_cache.c` | Day 1 | Was LRU the right choice? |
|
||||
| Bucket scan depth (position of hit) | `physics_hotwords_cache.c` | Day 1 | Is bucket order actually helping? |
|
||||
| Promotion wait time | `physics_hotwords_cache.c` | Day 1 | How long does word stay in bucket before cache? |
|
||||
|
||||
---
|
||||
|
||||
## Priority for "Twisting the Dragon's Tail"
|
||||
|
||||
### 🔴 **CRITICAL (Do First)**
|
||||
1. **Stack peak depth** - 5 minutes to add, massive insight
|
||||
2. **Block I/O metrics** - Writeback behavior unknown (could be performance killer)
|
||||
3. **Dictionary bucket load** - Unbalanced bucketing = latency tail
|
||||
|
||||
### 🟠 **HIGH (Phase 1 DoE)**
|
||||
4. **Per-word execution frequency** - Validation for cache/window decisions
|
||||
5. **Heat decay trace** - Validate Phase 2 feature before enabling
|
||||
6. **Block I/O hit rate** - Complete the I/O picture
|
||||
|
||||
### 🟡 **MEDIUM (Phase 2 DoE)**
|
||||
7. **Pipelining accuracy by depth** - Input to window size binary chop
|
||||
8. **Colon definition nesting depth** - Tail-call optimization opportunities
|
||||
9. **Dictionary growth velocity** - Predict "full" events
|
||||
|
||||
### 🟢 **NICE-TO-HAVE (Phase 3)**
|
||||
10. Heat concentration ratio
|
||||
11. Word category breakdown
|
||||
12. Speculation ROI histogram
|
||||
|
||||
---
|
||||
|
||||
## Implementation Effort Estimate
|
||||
|
||||
| Category | Files | LOC | Time | Priority |
|
||||
|----------|-------|-----|------|----------|
|
||||
| Stack metrics | stack_management.c, vm.h | ~30 | 30 min | 🔴 |
|
||||
| Dictionary metrics | vm.c, memory_management.c, vm.h | ~50 | 45 min | 🔴 |
|
||||
| Block I/O metrics | block_subsystem.c, vm.h | ~80 | 1 hr | 🔴 |
|
||||
| Per-word metrics | vm.c, physics_metadata.c, vm.h | ~100 | 1.5 hr | 🟠 |
|
||||
| Heat dynamics | physics_metadata.c, vm.h | ~60 | 1 hr | 🟠 |
|
||||
| Pipelining accuracy | physics_pipelining_metrics.c, vm.h | ~40 | 45 min | 🟡 |
|
||||
|
||||
---
|
||||
|
||||
## Bucket Reordering Mystery
|
||||
|
||||
**File**: `physics_hotwords_cache.c:201-215`
|
||||
**Code**: `hotwords_bucket_reorder()` - bubble sort by execution_heat
|
||||
**Status**: Function defined but **NEVER CALLED** from lookup path
|
||||
**Question**: Is this dead code or forgot to hook it?
|
||||
|
||||
**If enabled, metrics needed:**
|
||||
- Reorder frequency
|
||||
- Cost (bubble sort latency)
|
||||
- Benefit (does reordering actually improve hit latency?)
|
||||
@@ -1,220 +0,0 @@
|
||||
<!-- Moved from docs/03-architecture/physics-engine/steady-state-machine.md to docs/working/architecture/03-architecture/physics-engine/steady-state-machine.md on 2026-06-16 (docs reorg Phase 2) -->
|
||||
|
||||
> **⚠ OBSOLETE (flagged 2026-10-09).** Conceptual precursor to the L8 Jacquard mode-selector
|
||||
> system; it never mentions Jacquard/L8, the attractor buckets, or the six named modes, and
|
||||
> was never reconciled with the implemented `ssm_jacquard.c` system. The canonical current
|
||||
> reference for the physics/feedback model is `feedback-loops.md`. Kept for historical record.
|
||||
|
||||
# 1. Introduction
|
||||
|
||||
A Steady State Machine (SSM) is a novel computational model in which the system’s operational state is defined not by discrete symbolic transitions, but by convergence to an optimal runtime equilibrium.
|
||||
|
||||
Unlike traditional models such as Finite State Machines (FSMs), Pushdown Automata, or Turing Machines — where “state” is explicit and symbolic — a Steady State Machine derives its behavior from continuous thermodynamic variables such as:
|
||||
|
||||
* execution_heat
|
||||
|
||||
* entropy (diversity)
|
||||
|
||||
* pipeline readiness
|
||||
|
||||
* heat decay
|
||||
|
||||
* steady-state slope
|
||||
|
||||
* localized temperature gradients
|
||||
|
||||
* adaptive strategy selection
|
||||
|
||||
An SSM observes itself, adapts, and settles into stable performance basins called steady states.
|
||||
|
||||
This makes the Steady State Machine the first computational model where the attractor defines the machine, not the symbol.
|
||||
|
||||
# 2. Core Definition
|
||||
Definition 1 — Steady State Machine
|
||||
|
||||
> A Steady State Machine is a computational system defined by:
|
||||
|
||||
A set of internal physical metrics
|
||||
|
||||
* 𝐻 = execution_heats
|
||||
|
||||
* 𝐸 = entropy/diversity measures
|
||||
|
||||
* 𝑃 = pipeline activation thresholds
|
||||
|
||||
* Θ = heat decay profile
|
||||
|
||||
* 𝑊 = sliding execution window
|
||||
|
||||
A set of adaptive policies
|
||||
|
||||
* lookup strategies
|
||||
|
||||
* caching regimes
|
||||
|
||||
* bucket reorganizations
|
||||
|
||||
* pipeline decisions
|
||||
|
||||
* inference-based mode switches
|
||||
|
||||
* A state-space composed of attractor basins, not symbolic states
|
||||
|
||||
* steady attractor
|
||||
|
||||
* quasi-steady attractor
|
||||
|
||||
* transient attractor
|
||||
|
||||
* chaotic attractor
|
||||
|
||||
* A governing dynamic
|
||||
|
||||
|
||||
### 𝑀𝑡+1=𝑓(𝑀𝑡,Δ𝐻,Δ𝑊,Δ𝐸) where 𝑓 pushes the machine toward a stable basin.
|
||||
|
||||
The machine is defined by its convergence behavior, not its instruction sequence.
|
||||
|
||||
# 3. The Fundamental Insight
|
||||
|
||||
In an SSM, performance is the state.
|
||||
|
||||
The system is constantly measuring:
|
||||
|
||||
* how hot words are
|
||||
|
||||
* how often patterns recur
|
||||
|
||||
* how pipeline-able a sequence is
|
||||
|
||||
* how diverse the window is
|
||||
|
||||
* how much the dictionary should reorganize
|
||||
|
||||
And as these stabilize, the machine settles into a steady state — a self-maintaining optimal regime.
|
||||
|
||||
This regime is preserved until the environment changes (e.g., new workloads, diversity spikes), at which point the SSM moves into a transient or chaotic regime and eventually finds a new equilibrium.
|
||||
|
||||
# 4. Phases of a Steady State Machine
|
||||
## 1. Transient Phase
|
||||
|
||||
- System warms up
|
||||
|
||||
- Execution patterns are sporadic
|
||||
|
||||
- Heat gradients are noisy
|
||||
|
||||
- Pipeline inactive or unstable
|
||||
|
||||
## 2. Quasi-Steady Phase
|
||||
|
||||
- Some patterns emerge
|
||||
|
||||
- Lookups begin to bias
|
||||
|
||||
- Hot words form local attractors
|
||||
|
||||
- Dictionary reorders stabilize
|
||||
|
||||
## 3. True Steady State
|
||||
|
||||
- Heat surface smooth
|
||||
|
||||
- Pipeline active
|
||||
|
||||
- Diversity window predictable
|
||||
|
||||
- Lookups converge to optimal
|
||||
|
||||
- Runtime > baseline VM
|
||||
|
||||
- System remains stable unless perturbed
|
||||
|
||||
## 4. Chaotic/Disruption Phase
|
||||
|
||||
### Triggered by:
|
||||
|
||||
- major workload shift
|
||||
|
||||
- dictionary mutation
|
||||
|
||||
- extreme entropy spike
|
||||
|
||||
- cold-cache shock
|
||||
|
||||
- SSM falls out of equilibrium temporarily.
|
||||
|
||||
## 5. Axioms of Steady State Machines
|
||||
>Axiom 1 — Convergence
|
||||
Every sustained workload induces the SSM to converge toward a stable attractor unless external entropy forces divergence.
|
||||
|
||||
>Axiom 2 — Locality of Heat
|
||||
Execution heat reflects both locality and temporal relevance. Locality produces stability.
|
||||
|
||||
>Axiom 3 — Adaptive Reordering
|
||||
Reordering is not symbolic mutation; it is thermodynamic self-optimization.
|
||||
|
||||
>Axiom 4 — Stability Maximizes Throughput
|
||||
The steady state is always faster than the cold state, and usually faster than the naive baseline VM.
|
||||
|
||||
>Axiom 5 — Perturbation Response
|
||||
When disrupted, the SSM seeks a new steady state; it does not thrash indefinitely.
|
||||
|
||||
## 6. Relationship to Other Computational Models
|
||||
>FSM vs SSM
|
||||
FSM: discrete symbolic states
|
||||
SSM: emergent attractors in thermal space
|
||||
|
||||
>Turing Machine vs SSM
|
||||
TM: behavior defined by tape symbols and transitions
|
||||
SSM: behavior defined by adaptation toward equilibrium
|
||||
|
||||
>Markov Chain vs SSM
|
||||
MC: probabilistic transitions between explicit states
|
||||
SSM: deterministic drift toward stable basins, driven by observation
|
||||
|
||||
>Neural Network vs SSM
|
||||
NN: learns weights
|
||||
SSM: reorganizes runtime structures, not parameters
|
||||
|
||||
## 7. Why This Breaks New Ground
|
||||
|
||||
### A Steady State Machine is the first:
|
||||
|
||||
- runtime-thermodynamic computing model
|
||||
|
||||
- adaptive VM with convergence properties
|
||||
|
||||
- load-dependent reordering machine
|
||||
|
||||
- self-optimizing dictionary machine
|
||||
|
||||
- phase-shifting execution system
|
||||
|
||||
- state-machine whose state is emergent, not explicit
|
||||
|
||||
- This is a fundamental new category of computation.
|
||||
|
||||
## 8. Canonical Example: StarForth
|
||||
|
||||
### StarForth implements the first real SSM with:
|
||||
|
||||
- execution_heat
|
||||
|
||||
- rolling diversity windows
|
||||
|
||||
- phase-based lookup strategies
|
||||
|
||||
- adaptive dictionary ordering
|
||||
|
||||
- pipeline activation
|
||||
|
||||
- DOE-calibrated inference loop
|
||||
|
||||
- background physics monitor
|
||||
|
||||
- It is, quite literally, the reference implementation.
|
||||
|
||||
## 9. Formal Summary Statement
|
||||
|
||||
>A Steady State Machine (SSM) is a computational system in which the operational state is defined by attractor convergence in thermodynamic runtime space rather than by discrete symbolic transitions. An SSM continuously measures workload characteristics, adjusts internal structures, and stabilizes into performance-optimized steady states. When perturbed, it transitions through transient or chaotic phases before finding a new equilibrium. This model defines a new class of adaptive computational systems with properties distinct from finite state machines, pushdown automata, and Turing Machines.
|
||||
Reference in New Issue
Block a user