admin/LithosAnanake
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reorganise -- current reference gathered, finished work archived, superseded and dead ideas marked as such

Browse Source 
docs/CLAUDE.md Phase 2, 2026-10-14.  All moves by git mv; every archived
file carries a header saying what it is, why, and where to look instead.

- docs/reference/ (new): roadmap and versioning policy, M7.1, word-ACL
  design, kernel args, hosted acceptance test, dictionary, primitives,
  ontology, L8 table -- previously spread over docs/, docs/lithosananke/
  and docs/03-architecture/.
- docs/working/archive/historical/: finished work (session plan, test-gap
  audit, birthing plan, fix write-ups, the January architecture draft).
- docs/working/archive/superseded/: the Hermes-VM Tripod docs and July
  Hermes designs (-> FABRIC-3.5/3.6), the StarForth roadmap, every working
  doc already flagged superseded or obsolete.
- docs/working/archive/dead/: the hal_* target contract (never adopted),
  L4Re, metrics-and-knobs.
- Papers and proposals into docs/working/papers/.
- References rewritten across the repo (not logs/, not docs/patent/);
  CLAUDE.md's documentation section rewritten.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
2026-10-14 16:03:56 -04:00
co-authored by Claude Opus 5.5
parent 1a4b163861
commit 7aed588f78
116 changed files with 305 additions and 175 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/working/architecture/03-architecture/README.md
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- Moved from docs/03-architecture/README.md to docs/working/architecture/03-architecture/README.md on 2026-06-16 (docs reorg Phase 2) -->
<!-- Moved from docs/working/archive/superseded/2026-08-tripod/03-architecture-README.md to docs/working/architecture/03-architecture/README.md on 2026-06-16 (docs reorg Phase 2) -->
# Architecture
This directory contains system architecture documentation for StarForth's core subsystems and the path to StarKernel and StarshipOS.
docs/working/architecture/03-architecture/adaptive-systems/loop-5-sketch.md
-275
View File
@@ -1,275 +0,0 @@
<!-- Moved from docs/03-architecture/adaptive-systems/loop-5-sketch.md to docs/working/architecture/03-architecture/adaptive-systems/loop-5-sketch.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `window-inference-redesign.md`
> (Levene's-test window inference — the shipped Loop #5 mechanism). This undated sketch ended
> in an open question and was never approved as designed. Kept for historical record.
# Loop #5: Context-Aware Window Tuning - Design Sketch
## The Challenge
**Current state:** Window size shrinks based on pattern diversity, but we don't know if shrinking improves or hurts **pipelining prediction accuracy**.
**Goal:** Use actual prefetch hit rate to guide window size tuning via binary chop search.
---
## What Loop #5 Should Do
### High-Level Flow
1. **Measure**: Track global pipelining metrics across ALL words
- Total speculative prefetch attempts
- Successful prefetch hits
- Current accuracy = hits / attempts
2. **Trigger**: Periodically (e.g., every 1000 executions when warm)
- Calculate prediction accuracy at current effective_window_size
- Call binary chop to suggest next size to try
3. **Adapt**: Shrink or grow effective_window_size based on accuracy trend
- If smaller window → better accuracy: keep shrinking
- If smaller window → worse accuracy: grow back up
- Converge to sweet spot
4. **Measure**: Record the accuracy at each window size tried
- Build history of (window_size → accuracy) pairs
- Use this history to accelerate convergence
---
## Architecture Needed
### 1. VM-Level Metrics (New)
**File:** `include/vm.h` (add to VM struct)
```c
typedef struct {
uint64_t prefetch_attempts; /* Total speculative prefetch calls */
uint64_t prefetch_hits; /* Successful hits (word was looked up next) */
uint64_t window_tuning_checks; /* How many times we've checked window size */
uint32_t last_checked_window_size;
double last_checked_accuracy;
uint32_t suggested_next_size; /* What binary chop recommends trying */
} PipelineGlobalMetrics;
```
**Add to VM struct:**
```c
struct VM {
// ... existing fields ...
PipelineGlobalMetrics pipeline_metrics; /* NEW: aggregated metrics */
};
```
### 2. Update Prefetch Feedback (in vm.c)
**When speculative prefetch succeeds:**
```c
// In execute_colon_word(), after promotion:
if (spec_entry && vm->hotwords_cache && ENABLE_HOTWORDS_CACHE)
{
spec_entry->execution_heat = HOTWORDS_EXECUTION_HEAT_THRESHOLD + 1;
hotwords_cache_promote(vm->hotwords_cache, spec_entry);
// NEW: Record the prefetch attempt globally
vm->pipeline_metrics.prefetch_attempts++;
prev_word->transition_metrics->prefetch_attempts++; // Per-word too
}
```
**When word is actually looked up (check if it was speculated):**
```c
// In vm_interpret_word(), during dictionary lookup:
if (entry && ENABLE_PIPELINING && vm->hotwords_cache)
{
// NEW: Check if this word was in speculated cache (dirty bit or timestamp)
if (word_was_speculatively_promoted_recently(entry))
{
vm->pipeline_metrics.prefetch_hits++; // Global metric
// per_word_metrics->prefetch_hits++; // Already done in prefetch logic
}
}
```
### 3. Window Tuning Decision Loop
**New function:** `physics_pipelining_metrics.c`
```c
/**
* Suggest window size via binary chop based on prefetch accuracy trend.
* Called periodically (every 1000 executions when pipelining is enabled).
*/
uint32_t loop_5_binary_chop_suggest_window(
const PipelineGlobalMetrics *global,
uint32_t current_window_size,
uint32_t min_window,
uint32_t max_window)
{
if (global->prefetch_attempts == 0)
return current_window_size; /* Not enough data yet */
double current_accuracy = (double)global->prefetch_hits / (double)global->prefetch_attempts;
/* Binary chop: try smaller window first (aggressive shrinking)
* If accuracy drops, back off and try larger windows */
if (global->window_tuning_checks == 0)
{
/* First check: try shrinking by 25% */
return (current_window_size * 75) / 100;
}
/* Compare current accuracy to last check */
double accuracy_delta = current_accuracy - global->last_checked_accuracy;
if (accuracy_delta > 0.01) /* Improvement threshold: 1% */
{
/* Accuracy improved! Try shrinking more */
uint32_t smaller = (current_window_size * 75) / 100;
return (smaller > min_window) ? smaller : min_window;
}
else if (accuracy_delta < -0.01)
{
/* Accuracy degraded. Try growing instead */
uint32_t larger = (current_window_size * 133) / 100; /* Grow by ~33% */
return (larger < max_window) ? larger : max_window;
}
else
{
/* Plateau: stick with current size */
return current_window_size;
}
}
```
### 4. Integration Point
**In rolling_window_of_truth.c: `rolling_window_check_adaptive_shrink()`**
```c
void rolling_window_check_adaptive_shrink(RollingWindowOfTruth* window)
{
// ... existing adaptive shrinking code ...
/* Loop #5: Binary chop window tuning (Phase 2 IMPLEMENTATION) */
if (ENABLE_PIPELINING && window->is_warm)
{
window->adaptive_check_count++;
/* Check every 1000 executions for tuning opportunity */
if (window->adaptive_check_count % 1000 == 0)
{
PipelineGlobalMetrics *metrics = &vm->pipeline_metrics;
/* Calculate current accuracy */
double current_accuracy = (metrics->prefetch_attempts > 0)
? (double)metrics->prefetch_hits / (double)metrics->prefetch_attempts
: 0.0;
/* Ask binary chop for next window size to try */
uint32_t suggested_size = loop_5_binary_chop_suggest_window(
metrics,
window->effective_window_size,
ADAPTIVE_MIN_WINDOW_SIZE,
ROLLING_WINDOW_SIZE
);
/* Apply if different */
if (suggested_size != window->effective_window_size)
{
log_message(LOG_INFO,
"Loop #5: Binary chop tuning: %u → %u (accuracy %.2f%%, %lu/%lu hits)",
window->effective_window_size,
suggested_size,
current_accuracy * 100.0,
metrics->prefetch_hits,
metrics->prefetch_attempts);
window->effective_window_size = suggested_size;
}
/* Record this check for next iteration */
metrics->last_checked_window_size = window->effective_window_size;
metrics->last_checked_accuracy = current_accuracy;
metrics->window_tuning_checks++;
}
}
}
```
---
## Key Design Decisions
### A. Accuracy Measurement
- **Count**: Successful speculative promotions that were actually used
- **Challenge**: How to know if lookup was "predicted" vs "natural"?
- Option 1: Mark speculatively-promoted entries with a timestamp/flag
- Option 2: Track word IDs predicted and check if they match next lookup
- Option 3: Use per-word prefetch metrics already collected
### B. Tuning Frequency
- **Every 1000 executions** (when window is warm)
- Could be tunable: `make WINDOW_TUNING_FREQUENCY=1000`
### C. Search Strategy
- **Greedy shrinking first**: Try smaller windows aggressively
- **Backoff on degradation**: If accuracy drops, grow back up
- **Convergence**: Stop when oscillating around sweet spot
### D. Knobs Needed (Makefile)
```bash
make ENABLE_PIPELINING=1 # Must be enabled for Loop #5
make WINDOW_TUNING_FREQUENCY=1000 # How often to check
make WINDOW_TUNING_ACCURACY_THRESHOLD=1 # % improvement to continue shrinking
```
---
## Validation Approach
Once Loop #5 is wired, DoE would test:
- **A_BASELINE**: No optimizations
- **A_B_CACHE**: Cache only
- **A_C_FULL**: Window tuning with pipelining only
- **A_B_C_FULL**: Cache + window tuning + pipelining
**Metrics to collect:**
- Final effective_window_size (what did it converge to?)
- Final prefetch accuracy (%)
- Throughput improvement vs baseline
---
## Risks & Unknowns
1. **Overhead**: Checking accuracy every 1000 executions adds latency
2. **Noise**: If accuracy is noisy, binary chop might oscillate
3. **Workload-dependent**: Optimal window size might vary by workload
4. **False positives**: Speculative lookup might not correlate with prediction
---
## Sketch Summary
**Loop #5 would:**
1. Track global prefetch accuracy (hits/attempts)
2. Every 1000 executions, ask binary chop: "Is current window size optimal?"
3. Binary chop suggests shrinking, holding, or growing based on accuracy trend
4. Apply suggestion and record for next iteration
5. Over time, converge to window size that maximizes prediction accuracy
**Code locations if implemented:**
- VM struct: Add PipelineGlobalMetrics field
- vm.c: Update global metrics on prefetch attempts/hits
- physics_pipelining_metrics.c: `loop_5_binary_chop_suggest_window()`
- rolling_window_of_truth.c: Call binary chop from adaptive shrink check
**Effort**: ~150 LOC across 3-4 files
---
**Question for user:** Does this architecture align with what you had in mind for Loop #5?
docs/working/architecture/03-architecture/hal/README.md
-243
View File
@@ -1,243 +0,0 @@
<!-- Moved from docs/03-architecture/hal/README.md to docs/working/architecture/03-architecture/hal/README.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This document family
> describes a proposed `hal_*`-named interface and a `src/platform/<platform>/` layout that
> was never adopted as written. The implemented kernel HAL lives in `src/starkernel/hal/`
> with different (time-backend-style) naming. Treat all five documents here as design intent,
> not the current API. For current architecture see `.claude/CLAUDE.md`.
# Hardware Abstraction Layer (HAL) Documentation
This directory contains comprehensive documentation for StarForth's Hardware Abstraction Layer (HAL), which enables the evolution from hosted VM to StarKernel to StarshipOS.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references throughout this directory's docs describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
---
## Document Organization
Read in this order for full understanding:
### 1. **[overview.md](overview.md)** - Start Here
**What:** High-level HAL architecture, design principles, and rationale
**Who:** Everyone (developers, architects, contributors)
**Why read:** Understand the problem HAL solves and how it fits into StarForth → StarKernel → StarshipOS
**Key sections:**
- The Problem (why HAL is necessary)
- Architecture Overview (3-layer model)
- Design Principles (VM purity, contract-first, testability)
- HAL Subsystems (time, interrupts, memory, console, CPU)
- HAL and Physics Subsystems
- Success Criteria
---
### 2. **[interfaces.md](interfaces.md)** - The Contract
**What:** Detailed HAL interface specifications (function signatures, semantics, guarantees)
**Who:** VM developers, platform implementers
**Why read:** Understand exactly what each HAL function does and requires
**Key sections:**
- Time & Timers interface (`hal_time.h`)
- Interrupts interface (`hal_interrupt.h`)
- Memory interface (`hal_memory.h`)
- Console interface (`hal_console.h`)
- CPU interface (`hal_cpu.h`)
- Panic/Error interface (`hal_panic.h`)
- Interface safety summary (ISR-safe, thread-safe)
**Critical details:**
- Function semantics (what it does, guaranteed behavior)
- Error handling (when it can fail, how to handle)
- Performance expectations (latency, overhead)
- Concurrency model (thread-safe? ISR-safe?)
---
### 3. **[platform-implementations.md](platform-implementations.md)** - How to Implement
**What:** Platform-specific implementation examples and guidance
**Who:** Platform developers adding HAL support for new targets
**Why read:** Learn how to implement HAL for a specific platform
**Key sections:**
- Platform directory structure
- Linux HAL implementation (reference platform)
- Kernel HAL implementation (StarKernel)
- Platform testing strategy
- Common implementation pitfalls
**Includes complete code examples for:**
- Linux (POSIX, hosted)
- Kernel (freestanding, bare metal)
---
### 4. **[migration-plan.md](migration-plan.md)** - Refactoring Guide
**What:** Step-by-step plan for refactoring existing StarForth to use HAL
**Who:** Core developers doing the migration work
**Why read:** Execute the HAL migration safely with zero regressions
**Key sections:**
- 7-phase migration strategy
- Incremental refactoring approach
- Test validation after each phase
- Rollback strategy
- Timeline estimates (2-4 weeks)
**Phases:**
1. Define HAL interfaces (headers only)
2. Implement HAL for Linux
3. Migrate VM core to use HAL
4. Migrate physics subsystems to use HAL
5. Migrate REPL and word implementations
6. Implement HAL for L4Re (optional validation)
7. Validate deterministic behavior
---
### 5. **[starkernel-integration.md](starkernel-integration.md)** - Kernel Details
**What:** StarKernel-specific HAL implementation and boot sequence
**Who:** Kernel developers building StarKernel
**Why read:** Understand UEFI boot, freestanding C, hardware initialization
**Key sections:**
- StarKernel architecture (boot sequence)
- Directory structure (`src/platform/kernel/`)
- Boot sequence detail (UEFI → StarKernel → VM)
- HAL implementation details (time, interrupts, memory, console, CPU)
- Build system (freestanding toolchain, linker script)
- Testing on QEMU/OVMF
- Debugging techniques
- Roadmap to `ok` prompt
**Includes:**
- Complete UEFI loader implementation
- Kernel entry point
- Hardware initialization sequence
- QEMU testing guide
- Debugging strategies
---
## Quick Reference
### For Different Roles
**I'm a VM developer:**
- Read: `overview.md`, `interfaces.md`
- Use HAL functions in VM code
- Never call platform-specific APIs directly
**I'm adding a new platform:**
- Read: `overview.md`, `interfaces.md`, `platform-implementations.md`
- Implement all HAL interfaces for your platform
- Test against VM test suite (936+ tests must pass)
**I'm migrating existing code to HAL:**
- Read: `migration-plan.md`
- Follow 7-phase incremental approach
- Test after each phase
**I'm building StarKernel:**
- Read: All documents, especially `starkernel-integration.md`
- Start with UEFI boot + serial output
- Build incrementally to `ok` prompt
---
## Key Concepts
### HAL Principles
1. **VM Purity:** VM code is platform-agnostic, never knows which platform it's on
2. **Contract-First:** HAL interfaces are contracts with precise semantics
3. **Testability:** Develop/test on Linux, deploy on kernel
4. **Zero Overhead:** HAL inlines to direct hardware access when optimized
5. **Fail-Fast:** Validate platform assumptions at init, not during execution
### Success Criteria
The HAL is successful if:
- ✅ VM core has zero platform-specific code
- ✅ All 936+ tests pass on all platforms
- ✅ 0% algorithmic variance maintained
- ✅ No measurable performance regression
- ✅ StarKernel boots to `ok` prompt
- ✅ Physics subsystems work identically on all platforms
---
## Document Status
| Document | Status | Last Updated |
|----------|--------|--------------|
| overview.md | ✅ Complete | 2025-12-14 |
| interfaces.md | ✅ Complete | 2025-12-14 |
| platform-implementations.md | ✅ Complete | 2025-12-14 |
| migration-plan.md | ✅ Complete | 2025-12-14 |
| starkernel-integration.md | ✅ Complete | 2025-12-14 |
---
## Next Steps
### Immediate (HAL Migration)
1. Review and approve HAL interface specifications
2. Execute migration plan Phase 1 (define interfaces)
3. Execute migration plan Phase 2 (implement Linux HAL)
4. Continue through Phase 7 (validation)
### Near-Term (StarKernel)
1. Implement UEFI boot loader
2. Implement kernel HAL subsystems
3. Boot to `ok` prompt on QEMU
4. Validate physics subsystems on bare metal
### Long-Term (StarshipOS)
1. Add storage drivers (AHCI, NVMe)
2. Add networking stack (TCP/IP)
3. Implement process model (Forth tasks)
4. Build device model (block/net/char)
---
## References
### StarForth Documentation
- `docs/CLAUDE.md` - Project overview and build instructions
- `docs/03-architecture/heartbeat-system/` - Heartbeat subsystem details
- `docs/FEEDBACK_LOOPS.md` - Physics feedback loops
- `README.md` - Project quick start
### External Resources
- [UEFI Specification](https://uefi.org/specifications)
- [GNU-EFI](https://sourceforge.net/projects/gnu-efi/)
- [OSDev Wiki](https://wiki.osdev.org/)
- [Intel Software Developer Manual](https://www.intel.com/sdm)
- [ACPI Specification](https://uefi.org/acpi)
---
## Contributing
When updating HAL documentation:
1. **Keep documents synchronized** - Changes to interfaces require updates to implementation guides
2. **Maintain code examples** - Examples must compile and work
3. **Update this README** - Reflect new sections or document reorganization
4. **Version documentation** - Note "Last Updated" dates
5. **Test instructions** - Verify all commands and code snippets work
---
## Questions?
For questions about HAL architecture or StarKernel implementation, see:
- GitHub Issues: https://github.com/anthropics/starforth/issues (if public)
- Project maintainer: (contact info)
- Documentation feedback: (preferred method)
---
*The HAL is not just a kernel bootstrapping tool—it's the architectural foundation for StarForth → StarKernel → StarshipOS.*
docs/working/architecture/03-architecture/hal/interfaces.md
-710
View File
@@ -1,710 +0,0 @@
<!-- Moved from docs/03-architecture/hal/interfaces.md to docs/working/architecture/03-architecture/hal/interfaces.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This document defines the
> proposed `hal_*` interface contract; it was never adopted as written. The implemented kernel
> HAL is `src/starkernel/hal/` with different naming. See `README.md` in this directory and
> `.claude/CLAUDE.md` for what is actually built.
# HAL Interface Specifications
## Overview
This document defines the **contract** for each HAL subsystem. These interfaces are platform-agnostic and must be implemented identically (semantically) on all platforms.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Critical:** These are not optional "convenience wrappers"—they are the **only** way VM and physics code access platform resources.
---
## 1. Time & Timers (`include/hal/hal_time.h`)
### Purpose
Provide monotonic time, periodic/oneshot timers, and calibration for physics subsystems.
### Interface
```c
#ifndef HAL_TIME_H
#define HAL_TIME_H
#include <stdint.h>
/**
* Initialize time subsystem.
* Must be called before any other hal_time_* functions.
*
* Responsibilities:
* - Calibrate high-resolution timer (TSC, HPET, etc.)
* - Validate monotonicity
* - Set up timer interrupt sources (if needed)
*
* Error handling:
* - hal_panic() if no suitable time source available
* - hal_panic() if calibration fails
*/
void hal_time_init(void);
/**
* Get current monotonic time in nanoseconds.
*
* Guarantees:
* - MUST be monotonic (never goes backward)
* - SHOULD have sub-microsecond resolution
* - MUST be callable from interrupt context (ISR-safe)
* - MUST be callable concurrently (thread-safe)
*
* Performance:
* - Target: < 50 cycles on x86_64
* - Acceptable: < 200 cycles
*
* Returns:
* - Nanoseconds since arbitrary epoch (typically boot time)
*/
uint64_t hal_time_now_ns(void);
/**
* Busy-wait delay for specified nanoseconds.
* Used for calibration and short delays.
*
* Guarantees:
* - MUST delay for at least `ns` nanoseconds
* - MAY delay longer (but not by more than 10%)
* - MUST be callable from interrupt context
*
* Parameters:
* - ns: Nanoseconds to delay (0 = no delay)
*
* Notes:
* - This is a busy-wait, not a sleep
* - For delays > 1ms, prefer hal_timer_oneshot()
*/
void hal_time_delay_ns(uint64_t ns);
/**
* Timer callback function type.
*
* Parameters:
* - ctx: User-provided context pointer
*
* Context:
* - MAY be called from interrupt context (ISR)
* - MUST NOT block or sleep
* - MUST complete quickly (< 10µs target)
*/
typedef void (*hal_timer_callback_t)(void *ctx);
/**
* Schedule a oneshot timer to fire after specified nanoseconds.
*
* Guarantees:
* - Callback will fire at least `ns` nanoseconds in the future
* - MAY fire up to 10% late (jitter allowed)
* - Callback runs in interrupt context (ISR)
*
* Parameters:
* - ns: Nanoseconds until callback fires (0 = ASAP)
* - callback: Function to call when timer fires
* - ctx: User context passed to callback
*
* Error handling:
* - Returns 0 on success
* - Returns -1 if timer resources exhausted
*
* Concurrency:
* - Thread-safe (can be called from multiple threads)
* - NOT reentrant from same timer callback
*/
int hal_timer_oneshot(uint64_t ns, hal_timer_callback_t callback, void *ctx);
/**
* Schedule a periodic timer to fire every `period_ns` nanoseconds.
*
* Guarantees:
* - Callback will fire repeatedly at ~period_ns intervals
* - Jitter: < 10% of period (target: < 1%)
* - Callback runs in interrupt context (ISR)
* - Timer continues until hal_timer_cancel() called
*
* Parameters:
* - period_ns: Period in nanoseconds (minimum: 1000ns = 1µs)
* - callback: Function to call on each period
* - ctx: User context passed to callback
*
* Returns:
* - Timer ID (> 0) on success
* - -1 if timer resources exhausted
* - -2 if period_ns < minimum
*
* Notes:
* - This is the PRIMARY mechanism for heartbeat ISR
* - Platform MUST minimize jitter for experimental validity
*/
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx);
/**
* Cancel a periodic timer.
*
* Parameters:
* - timer_id: ID returned from hal_timer_periodic()
*
* Guarantees:
* - Callback will not fire after this call returns
* - Safe to call even if timer already fired or was cancelled
*
* Returns:
* - 0 on success
* - -1 if timer_id invalid
*/
int hal_timer_cancel(int timer_id);
/**
* Get timer frequency in Hz.
* Used for diagnostics and experiment metadata.
*
* Returns:
* - Frequency of underlying time source (e.g., TSC frequency)
* - 0 if frequency is variable or unknown
*/
uint64_t hal_time_frequency_hz(void);
#endif /* HAL_TIME_H */
```
### Platform Implementation Notes
**Linux (`platform/linux/hal_time.c`):**
- `hal_time_now_ns()` → `clock_gettime(CLOCK_MONOTONIC)`
- `hal_timer_periodic()` → `timer_create()` with `SIGRTMIN` + signal handler
- Challenge: Signal-based timers have ~1-10µs latency jitter
**L4Re (`platform/l4re/hal_time.c`):**
- `hal_time_now_ns()` → `l4re_kip_clock()`
- `hal_timer_periodic()` → L4Re IPC-based timer + IRQ object
- Challenge: IPC overhead, but lower jitter than Linux
**Kernel (`platform/kernel/hal_time.c`):**
- `hal_time_now_ns()` → `rdtsc()` calibrated against HPET
- `hal_timer_periodic()` → APIC timer interrupt
- Challenge: TSC drift on old hardware, calibration complexity
---
## 2. Interrupts (`include/hal/hal_interrupt.h`)
### Purpose
Enable/disable interrupts, register ISRs, query interrupt context.
### Interface
```c
#ifndef HAL_INTERRUPT_H
#define HAL_INTERRUPT_H
#include <stdint.h>
/**
* Initialize interrupt subsystem.
* Must be called before any other hal_interrupt_* functions.
*
* Responsibilities:
* - Set up interrupt controller (PIC/APIC on x86_64)
* - Initialize IDT (on kernel platforms)
* - Prepare for ISR registration
*
* Error handling:
* - hal_panic() if interrupt controller unavailable
*/
void hal_interrupt_init(void);
/**
* Enable interrupts globally.
*
* Guarantees:
* - Interrupts will be delivered after this call
* - Safe to call multiple times (idempotent)
* - NOT callable from interrupt context
*
* Notes:
* - On x86_64 kernel: sti
* - On Linux: no-op (interrupts always enabled)
*/
void hal_irq_enable(void);
/**
* Disable interrupts globally.
*
* Guarantees:
* - Interrupts will NOT be delivered after this call
* - Safe to call multiple times (idempotent)
* - Callable from interrupt context
*
* Returns:
* - Previous interrupt state (for hal_irq_restore)
*
* Notes:
* - On x86_64 kernel: cli
* - On Linux: no-op (interrupts always enabled)
*/
unsigned long hal_irq_disable(void);
/**
* Restore previous interrupt state.
*
* Parameters:
* - state: Value returned from hal_irq_disable()
*
* Use case:
* unsigned long flags = hal_irq_disable();
* // critical section
* hal_irq_restore(flags);
*/
void hal_irq_restore(unsigned long state);
/**
* Interrupt service routine function type.
*
* Parameters:
* - irq: IRQ number that triggered
* - ctx: User-provided context pointer
*
* Context:
* - ALWAYS called from interrupt context
* - MUST NOT block or sleep
* - MUST complete quickly (< 10µs target, < 100µs acceptable)
* - MUST NOT call non-ISR-safe HAL functions
*/
typedef void (*hal_isr_t)(unsigned int irq, void *ctx);
/**
* Register an interrupt service routine.
*
* Parameters:
* - irq: IRQ number to register (platform-specific numbering)
* - isr: Function to call when IRQ fires
* - ctx: User context passed to ISR
*
* Guarantees:
* - ISR will be called when IRQ fires
* - Only one ISR per IRQ (replaces previous if registered)
* - ISR runs with interrupts disabled (not reentrant)
*
* Returns:
* - 0 on success
* - -1 if IRQ number invalid
* - -2 if ISR registration failed
*
* Notes:
* - Used by hal_timer_periodic() internally
* - Used by device drivers for hardware IRQs
*/
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx);
/**
* Unregister an interrupt service routine.
*
* Parameters:
* - irq: IRQ number to unregister
*
* Returns:
* - 0 on success
* - -1 if IRQ was not registered
*/
int hal_irq_unregister(unsigned int irq);
/**
* Check if currently executing in interrupt context.
*
* Returns:
* - 1 if in interrupt context (ISR)
* - 0 if in normal execution context
*
* Use case:
* - Validate ISR-only functions (e.g., heartbeat sampling)
* - Conditional behavior (ISR vs. normal path)
*
* Performance:
* - MUST be fast (< 10 cycles)
* - Often implemented as checking a per-CPU flag
*/
int hal_in_interrupt_context(void);
#endif /* HAL_INTERRUPT_H */
```
### Platform Implementation Notes
**Linux:**
- `hal_irq_enable/disable()` → no-op (can't disable signals)
- `hal_irq_register()` → `sigaction()` + signal handler
- `hal_in_interrupt_context()` → thread-local flag set by signal handler
**Kernel:**
- `hal_irq_enable/disable()` → `sti`/`cli` (x86_64)
- `hal_irq_register()` → IDT entry + APIC configuration
- `hal_in_interrupt_context()` → check per-CPU interrupt nesting count
---
## 3. Memory (`include/hal/hal_memory.h`)
### Purpose
Allocate/free memory, map pages, manage heap.
### Interface
```c
#ifndef HAL_MEMORY_H
#define HAL_MEMORY_H
#include <stddef.h>
#include <stdint.h>
/**
* Initialize memory subsystem.
* Must be called before any other hal_mem_* functions.
*
* Responsibilities:
* - Set up heap allocator
* - Initialize page tables (kernel platforms)
* - Parse memory map (kernel platforms)
*
* Error handling:
* - hal_panic() if insufficient memory
*/
void hal_mem_init(void);
/**
* Allocate memory from heap.
*
* Parameters:
* - size: Bytes to allocate (0 = returns NULL)
*
* Guarantees:
* - Returns aligned pointer (at least 8-byte aligned)
* - Memory is zero-initialized
* - Thread-safe (can be called concurrently)
* - NOT callable from interrupt context
*
* Returns:
* - Pointer to allocated memory
* - NULL if allocation failed
*
* Notes:
* - On Linux: wrapper around malloc()
* - On kernel: custom heap allocator (kmalloc)
*/
void *hal_mem_alloc(size_t size);
/**
* Free memory allocated by hal_mem_alloc().
*
* Parameters:
* - ptr: Pointer returned from hal_mem_alloc() (NULL = no-op)
*
* Guarantees:
* - Thread-safe
* - NOT callable from interrupt context
* - Safe to call with NULL
*/
void hal_mem_free(void *ptr);
/**
* Allocate contiguous physical pages.
* Used for DMA buffers, page tables, large allocations.
*
* Parameters:
* - count: Number of pages to allocate (page size is platform-specific)
*
* Returns:
* - Physical address of first page
* - 0 if allocation failed
*
* Notes:
* - Pages are NOT mapped into virtual address space
* - Caller must call hal_mem_map() to access
* - On Linux: may use mmap() with MAP_ANONYMOUS
* - On kernel: physical memory manager (PMM)
*/
uint64_t hal_mem_alloc_pages(size_t count);
/**
* Free physical pages allocated by hal_mem_alloc_pages().
*
* Parameters:
* - paddr: Physical address returned from hal_mem_alloc_pages()
* - count: Number of pages to free
*/
void hal_mem_free_pages(uint64_t paddr, size_t count);
/**
* Map physical memory into virtual address space.
*
* Parameters:
* - vaddr: Virtual address to map (must be page-aligned)
* - paddr: Physical address to map (must be page-aligned)
* - size: Size in bytes (must be page-multiple)
* - flags: Mapping flags (HAL_MEM_*)
*
* Returns:
* - 0 on success
* - -1 if mapping failed
*
* Flags:
* - HAL_MEM_READ: Readable
* - HAL_MEM_WRITE: Writable
* - HAL_MEM_EXEC: Executable
* - HAL_MEM_NOCACHE: Uncached (for MMIO)
*
* Notes:
* - On Linux: no-op (identity mapping assumed)
* - On kernel: update page tables, flush TLB
*/
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags);
/* Mapping flags */
#define HAL_MEM_READ (1 << 0)
#define HAL_MEM_WRITE (1 << 1)
#define HAL_MEM_EXEC (1 << 2)
#define HAL_MEM_NOCACHE (1 << 3)
/**
* Get page size in bytes.
*
* Returns:
* - Page size (typically 4096 on x86_64)
*/
size_t hal_mem_page_size(void);
#endif /* HAL_MEMORY_H */
```
---
## 4. Console (`include/hal/hal_console.h`)
### Purpose
Character I/O for REPL and diagnostics.
### Interface
```c
#ifndef HAL_CONSOLE_H
#define HAL_CONSOLE_H
/**
* Initialize console subsystem.
* Must be called before any other hal_console_* functions.
*
* Responsibilities:
* - Initialize UART (kernel platforms)
* - Set up framebuffer (if available)
* - Configure terminal settings (Linux)
*
* Error handling:
* - hal_panic() if no console available
*/
void hal_console_init(void);
/**
* Write a single character to console.
*
* Parameters:
* - c: Character to write
*
* Guarantees:
* - Character will be visible on console
* - Blocking (waits if output buffer full)
* - Callable from interrupt context (for panic messages)
*
* Performance:
* - May be slow (UART is ~1µs per char)
* - Buffering recommended for bulk output
*/
void hal_console_putc(char c);
/**
* Write a null-terminated string to console.
*
* Parameters:
* - s: String to write (NULL = no-op)
*
* Notes:
* - Convenience wrapper around hal_console_putc()
*/
void hal_console_puts(const char *s);
/**
* Read a single character from console.
*
* Guarantees:
* - Blocking (waits until character available)
* - NOT callable from interrupt context
*
* Returns:
* - Character read (0-255)
* - -1 on error or EOF
*/
int hal_console_getc(void);
/**
* Check if input is available (non-blocking).
*
* Returns:
* - 1 if character available (hal_console_getc() will not block)
* - 0 if no character available
*/
int hal_console_has_input(void);
#endif /* HAL_CONSOLE_H */
```
---
## 5. CPU (`include/hal/hal_cpu.h`)
### Purpose
CPU identification, relax/halt, SMP coordination.
### Interface
```c
#ifndef HAL_CPU_H
#define HAL_CPU_H
#include <stdint.h>
/**
* Initialize CPU subsystem.
* Must be called before any other hal_cpu_* functions.
*
* Responsibilities:
* - Detect CPU features (SSE, AVX, etc.)
* - Set up per-CPU storage
* - Bring up secondary CPUs (SMP)
*
* Error handling:
* - hal_panic() if CPU unsupported
*/
void hal_cpu_init(void);
/**
* Get current CPU ID.
*
* Returns:
* - CPU ID (0 = boot processor, 1+ = application processors)
*
* Guarantees:
* - Callable from interrupt context
* - Fast (< 10 cycles)
*
* Notes:
* - On x86_64 kernel: APIC ID
* - On Linux: thread ID (not true CPU ID)
*/
unsigned int hal_cpu_id(void);
/**
* Hint to CPU to relax (pause in spin-wait loop).
*
* Guarantees:
* - Callable from interrupt context
* - No functional effect (pure performance hint)
*
* Notes:
* - On x86_64: pause instruction
* - On ARM: yield instruction
* - On Linux: sched_yield()
*/
void hal_cpu_relax(void);
/**
* Halt CPU until next interrupt.
*
* Guarantees:
* - CPU enters low-power state
* - Wakes on next interrupt
* - NOT callable from interrupt context
*
* Notes:
* - On x86_64 kernel: hlt instruction
* - On Linux: select() or nanosleep()
*/
void hal_cpu_halt(void);
/**
* Get number of CPUs in system.
*
* Returns:
* - Number of CPUs (1 = single-core, 2+ = multi-core)
*/
unsigned int hal_cpu_count(void);
#endif /* HAL_CPU_H */
```
---
## 6. Panic/Error Handling (`include/hal/hal_panic.h`)
### Purpose
Fatal error handling (system halt or abort).
### Interface
```c
#ifndef HAL_PANIC_H
#define HAL_PANIC_H
/**
* Fatal error: print message and halt system.
*
* Parameters:
* - msg: Error message (NULL = "panic")
*
* Guarantees:
* - NEVER returns
* - Message printed to console (if available)
* - System halted (kernel) or abort() (hosted)
* - Callable from interrupt context
*
* Notes:
* - Use for unrecoverable errors only
* - On kernel: disable interrupts + hlt loop
* - On Linux: fprintf(stderr) + abort()
*/
void hal_panic(const char *msg) __attribute__((noreturn));
#endif /* HAL_PANIC_H */
```
---
## Interface Summary
| Subsystem | Key Functions | ISR-Safe | Thread-Safe |
|-----------|---------------|----------|-------------|
| **Time** | `hal_time_now_ns()` | ✅ | ✅ |
| | `hal_timer_periodic()` | ❌ | ✅ |
| **Interrupt** | `hal_in_interrupt_context()` | ✅ | ✅ |
| | `hal_irq_disable()` | ✅ | ✅ |
| **Memory** | `hal_mem_alloc()` | ❌ | ✅ |
| | `hal_mem_map()` | ❌ | ⚠️ |
| **Console** | `hal_console_putc()` | ✅ | ⚠️ |
| | `hal_console_getc()` | ❌ | ❌ |
| **CPU** | `hal_cpu_id()` | ✅ | ✅ |
| | `hal_cpu_relax()` | ✅ | ✅ |
**Legend:**
- ✅ = Safe
- ❌ = Unsafe
- ⚠️ = Platform-dependent
---
## Next Steps
See companion documentation:
- `platform-implementations.md` - How to implement HAL for a platform
- `migration-plan.md` - Refactoring existing code to use HAL
- `starkernel-integration.md` - StarKernel-specific HAL implementation
docs/working/architecture/03-architecture/hal/migration-plan.md
-657
View File
@@ -1,657 +0,0 @@
<!-- Moved from docs/03-architecture/hal/migration-plan.md to docs/working/architecture/03-architecture/hal/migration-plan.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ TARGET HAL — NOT THE BUILT INTERFACE (flagged 2026-10-09).** This migration plan targets
> the proposed `hal_*` interface, which was never adopted as written. The kernel grew its own
> HAL directly in `src/starkernel/hal/`. Kept for design rationale; see `.claude/CLAUDE.md`
> for the built architecture.
# HAL Migration Plan
## Overview
This document provides a **step-by-step plan** for refactoring StarForth to use the Hardware Abstraction Layer (HAL). The migration preserves all existing functionality while preparing the codebase for StarKernel.
> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Goal:** Zero functional regressions, all 936+ tests pass, 0% algorithmic variance maintained.
---
## Migration Strategy
### Principles
1. **Incremental refactoring** - One subsystem at a time
2. **Test after each step** - Build + test suite must pass
3. **Platform parity** - Linux and L4Re work throughout migration
4. **No big bang** - HAL interfaces defined first, then adopted gradually
### Phases
```
Phase 1: Define HAL Interfaces (headers only)
↓
Phase 2: Implement HAL for Linux (refactor existing code)
↓
Phase 3: Migrate VM Core to use HAL
↓
Phase 4: Migrate Physics Subsystems to use HAL
↓
Phase 5: Migrate REPL and Word Implementations
↓
Phase 6: Implement HAL for L4Re (optional validation)
↓
Phase 7: Validate Deterministic Behavior
```
---
## Phase 1: Define HAL Interfaces
**Duration:** 1-2 days
**Risk:** Low (no code changes, only headers)
### Tasks
1. **Create HAL header directory**
```bash
mkdir -p include/hal
```
2. **Write HAL interface headers**
- `include/hal/hal_time.h`
- `include/hal/hal_interrupt.h`
- `include/hal/hal_memory.h`
- `include/hal/hal_console.h`
- `include/hal/hal_cpu.h`
- `include/hal/hal_panic.h`
See `interfaces.md` for full specifications.
3. **Update Makefile to include HAL headers**
```makefile
INCLUDES += -Iinclude/hal
```
4. **Compile check (headers only)**
```bash
make clean && make PLATFORM=linux
```
Should compile without errors (HAL functions not yet called).
### Success Criteria
✅ All HAL headers compile without errors
✅ No changes to VM code yet
✅ Documentation reviewed and approved
---
## Phase 2: Implement HAL for Linux
**Duration:** 3-5 days
**Risk:** Medium (refactoring existing platform code)
### Current Platform Code Locations
Audit existing platform-specific code:
```bash
# Find POSIX-specific calls
grep -r "clock_gettime\|pthread\|signal\|malloc\|printf" src/
```
**Expected locations:**
- Timing: `src/platform/linux/time.c` (already exists)
- Memory: Direct `malloc()` calls in VM code (needs refactoring)
- Console: Direct `printf()` calls in REPL (needs refactoring)
### Tasks
1. **Create Linux HAL directory**
```bash
mkdir -p src/platform/linux/hal
```
2. **Implement Linux HAL subsystems**
**2a. `hal_time.c`** - Refactor `src/platform/linux/time.c`
```c
/* Before: */
uint64_t platform_time_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ...;
}
/* After: */
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
```
**2b. `hal_memory.c`** - Wrap malloc/free
```c
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
void *ptr = malloc(size);
if (ptr) memset(ptr, 0, size);
return ptr;
}
void hal_mem_free(void *ptr) {
free(ptr);
}
```
**2c. `hal_console.c`** - Wrap stdio
```c
void hal_console_putc(char c) {
putchar(c);
fflush(stdout);
}
int hal_console_getc(void) {
return getchar();
}
```
**2d. `hal_interrupt.c`** - Signal-based interrupts
See `platform-implementations.md` for full implementation.
**2e. `hal_cpu.c`** - CPU info
```c
unsigned int hal_cpu_id(void) {
return 0; /* Single-threaded */
}
void hal_cpu_relax(void) {
sched_yield();
}
```
**2f. `hal_panic.c`** - Error handling
```c
void hal_panic(const char *msg) {
fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown");
abort();
}
```
3. **Update Makefile**
```makefile
ifeq ($(PLATFORM),linux)
PLATFORM_SOURCES = \
src/platform/linux/hal_time.c \
src/platform/linux/hal_interrupt.c \
src/platform/linux/hal_memory.c \
src/platform/linux/hal_console.c \
src/platform/linux/hal_cpu.c \
src/platform/linux/hal_panic.c
endif
```
4. **Build test (HAL code compiles)**
```bash
make clean && make PLATFORM=linux
```
### Success Criteria
✅ Linux HAL compiles without errors
✅ Existing VM code still builds (not yet using HAL)
✅ No functional changes yet
---
## Phase 3: Migrate VM Core to use HAL
**Duration:** 2-3 days
**Risk:** High (core VM changes)
### Files to Modify
- `src/vm.c` - Core interpreter loop
- `src/vm_api.c` - External API
- `src/memory_management.c` - Dictionary allocator
- `include/vm.h` - VM struct
### Tasks
1. **Replace direct malloc/free in VM core**
**Before:**
```c
/* src/memory_management.c */
void *mem = malloc(size);
```
**After:**
```c
#include "hal/hal_memory.h"
void *mem = hal_mem_alloc(size);
```
2. **Replace timing calls in heartbeat**
**Before:**
```c
/* src/vm.c */
#include <time.h>
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t now = ts.tv_sec * 1000000000ULL + ts.tv_nsec;
```
**After:**
```c
#include "hal/hal_time.h"
uint64_t now = hal_time_now_ns();
```
3. **Add HAL initialization to VM startup**
**In `src/main.c`:**
```c
int main(int argc, char **argv) {
/* Initialize HAL subsystems */
hal_time_init();
hal_interrupt_init();
hal_mem_init();
hal_console_init();
hal_cpu_init();
/* Continue with existing VM initialization */
VM *vm = vm_create();
...
}
```
4. **Remove platform-specific #ifdefs**
**Before:**
```c
#ifdef PLATFORM_LINUX
clock_gettime(CLOCK_MONOTONIC, &ts);
#elif PLATFORM_L4RE
l4re_kip_clock(kip);
#endif
```
**After:**
```c
uint64_t now = hal_time_now_ns(); /* Works on all platforms */
```
5. **Build and test**
```bash
make clean && make PLATFORM=linux
make test
```
### Expected Failures
- **Compilation errors:** Missing `#include "hal/hal_*.h"`
- **Link errors:** HAL functions not implemented
- **Runtime errors:** HAL not initialized before VM
**Fix incrementally:** Add includes, implement missing HAL functions, ensure init order.
### Success Criteria
✅ VM builds using HAL interfaces
✅ All 936+ tests pass
✅ No platform-specific code in `src/vm.c`
---
## Phase 4: Migrate Physics Subsystems to use HAL
**Duration:** 2-3 days
**Risk:** Medium (determinism must be preserved)
### Files to Modify
- `src/dictionary_heat_optimization.c`
- `src/rolling_window_of_truth.c`
- `src/physics_hotwords_cache.c`
- `src/physics_pipelining_metrics.c`
- `src/inference_engine.c`
- `src/heartbeat.c`
### Tasks
1. **Audit timing calls in physics subsystems**
```bash
grep -n "clock_gettime\|time\.h" src/*physics*.c src/heartbeat.c src/inference*.c
```
2. **Replace timing calls**
**Example: `src/rolling_window_of_truth.c`**
```c
/* Before: */
#include <time.h>
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t timestamp = ...;
/* After: */
#include "hal/hal_time.h"
uint64_t timestamp = hal_time_now_ns();
```
3. **Migrate heartbeat to HAL timer**
**Before:** `src/heartbeat.c` uses pthread + sleep
```c
void *heartbeat_thread(void *arg) {
while (running) {
usleep(period_us);
vm_tick(vm);
}
}
```
**After:** Use HAL periodic timer
```c
static VM *heartbeat_vm = NULL;
static void heartbeat_isr(void *ctx) {
(void)ctx;
if (heartbeat_vm) {
vm_tick_isr(heartbeat_vm); /* ISR-safe VM tick */
}
}
void heartbeat_start(VM *vm, uint64_t rate_hz) {
heartbeat_vm = vm;
uint64_t period_ns = 1000000000ULL / rate_hz;
hal_timer_periodic(period_ns, heartbeat_isr, NULL);
}
```
**Critical:** `vm_tick_isr()` must be ISR-safe:
- No malloc/free
- No blocking I/O
- Only lock-free ring buffer updates
4. **Test deterministic behavior**
```bash
make fastest PLATFORM=linux
./build/amd64/fastest/starforth --doe > results_after_hal.csv
```
Compare with baseline (before HAL migration):
```bash
diff results_before_hal.csv results_after_hal.csv
```
**Expected:** Identical results (0% algorithmic variance).
### Success Criteria
✅ Physics subsystems use HAL for all timing
✅ Heartbeat runs via HAL periodic timer
✅ 0% algorithmic variance maintained
✅ All tests pass
---
## Phase 5: Migrate REPL and Word Implementations
**Duration:** 1-2 days
**Risk:** Low (mostly console I/O)
### Files to Modify
- `src/repl.c` - Read-eval-print loop
- `src/word_source/*_words.c` - Words that do I/O (emit, key, etc.)
### Tasks
1. **Replace stdio in REPL**
**Before:** `src/repl.c`
```c
#include <stdio.h>
char c = getchar();
printf("ok\n");
```
**After:**
```c
#include "hal/hal_console.h"
char c = hal_console_getc();
hal_console_puts("ok\n");
```
2. **Update I/O words**
**`src/word_source/io_words.c`:**
```c
/* EMIT ( c -- ) */
static void word_emit(VM *vm) {
int c = vm_pop(vm);
hal_console_putc((char)c);
}
/* KEY ( -- c ) */
static void word_key(VM *vm) {
int c = hal_console_getc();
vm_push(vm, c);
}
```
3. **Test REPL interactively**
```bash
./build/amd64/standard/starforth
> 1 2 + .
3 ok
> BYE
```
### Success Criteria
✅ REPL works identically to before
✅ All I/O words use HAL
✅ No direct stdio calls in VM code
---
## Phase 6: Implement HAL for L4Re (Optional)
**Duration:** 3-5 days
**Risk:** Low (validates HAL portability)
This phase is **optional** but highly recommended to validate that the HAL abstraction actually works across platforms.
### Tasks
1. **Create L4Re HAL directory**
```bash
mkdir -p src/platform/l4re/hal
```
2. **Implement L4Re HAL subsystems**
- `hal_time.c` → L4Re clock API
- `hal_memory.c` → L4Re dataspaces
- `hal_console.c` → L4Re console service
- `hal_interrupt.c` → L4Re IRQ objects
See L4Re documentation for API details.
3. **Build for L4Re**
```bash
make PLATFORM=l4re
```
4. **Run tests on L4Re**
```bash
make PLATFORM=l4re test
```
### Success Criteria
✅ L4Re HAL compiles
✅ VM runs on L4Re without source changes
✅ Tests pass on L4Re
---
## Phase 7: Validate Deterministic Behavior
**Duration:** 1 day
**Risk:** High (final validation)
### Tasks
1. **Run DoE on Linux (HAL)**
```bash
make fastest PLATFORM=linux
for i in {1..10}; do
./build/amd64/fastest/starforth --doe > results_linux_$i.csv
done
```
2. **Verify 0% variance across runs**
```bash
# All CSV files should be identical
md5sum results_linux_*.csv
```
3. **Compare with pre-HAL baseline**
```bash
diff results_before_hal.csv results_linux_1.csv
```
**Expected:** Identical (HAL added zero overhead).
4. **Run full test suite**
```bash
make test PLATFORM=linux
```
**Expected:** All 936+ tests pass.
5. **Benchmark performance**
```bash
make bench PLATFORM=linux
```
**Expected:** < 5% performance delta from pre-HAL baseline.
### Success Criteria
✅ 0% algorithmic variance maintained
✅ All tests pass
✅ < 5% performance regression
✅ HAL abstraction validated
---
## Rollback Strategy
If migration fails at any phase:
1. **Git branch strategy**
```bash
git checkout -b hal-migration
# Work on branch, test each phase
git commit -m "Phase N complete"
# If phase fails, revert:
git reset --hard HEAD~1
```
2. **Incremental commits**
- Commit after each phase passes tests
- Never commit broken code
- Each commit should build and pass tests
3. **Feature flag (if needed)**
```c
#ifdef ENABLE_HAL
uint64_t now = hal_time_now_ns();
#else
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
uint64_t now = ...;
#endif
```
Enable HAL incrementally, fall back if issues arise.
---
## Post-Migration Cleanup
After successful migration:
1. **Remove old platform code**
```bash
# If src/platform/linux/time.c was fully replaced by hal_time.c
git rm src/platform/linux/time.c
```
2. **Update documentation**
- `docs/CLAUDE.md` - Mention HAL architecture
- `README.md` - Update build instructions
- `docs/DEVELOPER.md` - Add HAL section
3. **Update .gitignore**
```
# Build artifacts for all platforms
build/linux/
build/l4re/
build/kernel/
```
---
## Timeline Estimate
| Phase | Duration | Cumulative |
|-------|----------|------------|
| 1. Define HAL Interfaces | 1-2 days | 1-2 days |
| 2. Implement Linux HAL | 3-5 days | 4-7 days |
| 3. Migrate VM Core | 2-3 days | 6-10 days |
| 4. Migrate Physics | 2-3 days | 8-13 days |
| 5. Migrate REPL | 1-2 days | 9-15 days |
| 6. L4Re HAL (optional) | 3-5 days | 12-20 days |
| 7. Validation | 1 day | 13-21 days |
**Total:** 2-4 weeks (depending on L4Re inclusion)
---
## Success Metrics
The HAL migration is successful if:
1. ✅ **All tests pass** - 936+ tests on Linux (and L4Re if implemented)
2. ✅ **0% variance** - DoE results identical pre/post migration
3. ✅ **No regressions** - Performance < 5% slower
4. ✅ **Zero platform code in VM** - No #ifdef PLATFORM_X in core
5. ✅ **Clean abstractions** - HAL interfaces well-documented
6. ✅ **Ready for StarKernel** - Platform layer can be replaced
---
## Next Steps
After successful HAL migration:
1. **Implement StarKernel platform** (`src/platform/kernel/`)
2. **UEFI boot loader** (`src/platform/kernel/boot/uefi_loader.c`)
3. **Boot to `ok` prompt** on QEMU/OVMF
4. **Validate physics on bare metal** (determinism on real hardware)
See `starkernel-integration.md` for StarKernel-specific implementation details.
docs/working/architecture/03-architecture/hal/overview.md
-346
View File
@@ -1,346 +0,0 @@
<!-- Moved from docs/03-architecture/hal/overview.md to docs/working/architecture/03-architecture/hal/overview.md on 2026-06-16 (docs reorg Phase 2) -->
# Hardware Abstraction Layer (HAL) Architecture
> **⚠️ Important Note**: This document describes the **target HAL architecture** with standardized `hal_*` function naming. The current codebase uses `sf_time_backend_t` and similar abstractions with different naming conventions. See [migration-plan.md](migration-plan.md) for the refactoring steps to migrate existing platform code to the HAL interfaces described here.
>
> **HISTORICAL — L4Re/Fiasco.OC**: this document's L4Re references (platform comparison table, microkernel discussion, etc.) describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. Treat every L4Re mention below as historical design context, not a currently buildable platform.
## Executive Summary
The Hardware Abstraction Layer (HAL) is the architectural foundation that enables StarForth to evolve from a hosted VM into StarKernel while preserving the physics-driven adaptive runtime's deterministic behavior across all platforms.
**Critical insight:** The HAL is not an afterthought—it's the linchpin that makes StarKernel → StarshipOS possible without compromising StarForth's experimental integrity.
---
## The Problem
StarForth currently runs on multiple platforms:
- **Linux** (POSIX, hosted)
- **L4Re/Fiasco.OC** (microkernel)
- **Bare metal** (limited, experimental)
Each platform has its own:
- Timing mechanisms (POSIX timers vs. hardware timers)
- Interrupt handling (signals vs. IDT/APIC)
- Memory allocation (malloc vs. physical page frames)
- I/O mechanisms (stdin/stdout vs. UART/framebuffer)
**Without a HAL:** Platform-specific code bleeds into the VM core, physics subsystems, and word implementations. This creates:
- ❌ Fragile #ifdef PLATFORM_X conditionals throughout codebase
- ❌ Platform-specific bugs in supposedly portable code
- ❌ Inability to test kernel code on hosted platforms
- ❌ Risk to deterministic behavior guarantees (0% algorithmic variance)
**With a HAL:** Clean separation between VM logic and platform implementation:
- ✅ VM and physics subsystems are platform-agnostic
- ✅ Platform code is isolated and testable
- ✅ New platforms (StarKernel) can be added without touching VM core
- ✅ Deterministic behavior guaranteed by HAL contract, not platform quirks
---
## Architecture Overview
```
┌────────────────────────────────────────────────────────────┐
│ StarForth VM Core + Physics Subsystems │
│ • Interpreter loop (vm.c) │
│ • Execution heat tracking │
│ • Rolling window of truth │
│ • Hot-words cache │
│ • Pipelining metrics │
│ • Inference engine │
│ • Heartbeat coordination │
│ │
│ ↓ Calls HAL interfaces (platform-agnostic) │
├────────────────────────────────────────────────────────────┤
│ Hardware Abstraction Layer (HAL) │
│ • hal_time.h - Monotonic time, timers, calibration │
│ • hal_interrupt.h - IRQ management, ISR registration │
│ • hal_memory.h - Allocation, page mapping, heap │
│ • hal_console.h - Character I/O (serial, framebuffer) │
│ • hal_cpu.h - CPU ID, relax/halt, SMP coordination │
│ │
│ ↓ Platform-specific implementations │
├────────────────────────────────────────────────────────────┤
│ Platform Implementations │
│ ┌──────────────┬──────────────┬──────────────────────┐ │
│ │ Linux │ L4Re │ Kernel (StarKernel) │ │
│ │ (POSIX) │ (microkernel)│ (freestanding) │ │
│ ├──────────────┼──────────────┼──────────────────────┤ │
│ │ clock_gettime│ L4Re::Clock │ TSC + HPET + APIC │ │
│ │ timerfd │ L4Re::IrqEoi │ IDT + APIC IRQ │ │
│ │ malloc/free │ dataspaces │ PMM + VMM + kmalloc │ │
│ │ stdin/stdout │ L4Re::Console│ UART + framebuffer │ │
│ │ pthread │ L4Re::Thread │ SMP bring-up │ │
│ └──────────────┴──────────────┴──────────────────────┘ │
└────────────────────────────────────────────────────────────┘
```
---
## Design Principles
### 1. **VM Purity**
The VM core must never know which platform it's running on. All platform awareness lives in HAL implementations.
**Anti-pattern (current):**
```c
#ifdef PLATFORM_LINUX
clock_gettime(CLOCK_MONOTONIC, &ts);
#elif PLATFORM_L4RE
l4re_kip_clock(kip);
#elif PLATFORM_KERNEL
rdtsc();
#endif
```
**Correct pattern (HAL):**
```c
// In VM code (platform-agnostic)
uint64_t now = hal_time_now_ns();
// In platform/linux/hal_time.c
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
// In platform/kernel/hal_time.c
uint64_t hal_time_now_ns(void) {
return tsc_to_ns(rdtsc());
}
```
### 2. **Contract-First Design**
HAL interfaces are **contracts**, not convenience wrappers. Each HAL function has:
- **Precise semantics** - What it does, guaranteed across all platforms
- **Error handling** - When it can fail and how
- **Performance expectations** - Allowed latency/overhead
- **Concurrency model** - Thread-safe? ISR-safe?
### 3. **Testability on Hosted Platforms**
StarKernel code must be developable and testable on Linux/L4Re before deploying to bare metal.
**Example:** Heartbeat ISR development
- **Kernel implementation:** APIC timer interrupt → ISR → ring buffer
- **Linux test implementation:** timerfd + signal handler → ISR → ring buffer
- **Same HAL interface, same VM code, different platform layer**
### 4. **Zero Overhead When Possible**
HAL calls should compile to direct hardware access on kernel platforms, not add abstraction tax.
**Good:** `hal_time_now_ns()` inlines to `rdtsc()` with `-O2`
**Bad:** Function pointer indirection adds 5-10 cycles per call
### 5. **Fail-Fast Validation**
HAL implementations validate platform assumptions at init time, not during execution.
**Example:**
```c
void hal_time_init(void) {
// Calibrate TSC frequency at boot
tsc_calibrate_hpet();
// Validate monotonicity
uint64_t t1 = hal_time_now_ns();
hal_cpu_relax();
uint64_t t2 = hal_time_now_ns();
if (t2 < t1) {
hal_panic("hal_time: TSC not monotonic!");
}
}
```
---
## HAL Subsystems
### 1. **Time & Timers** (`hal_time.h`)
**Purpose:** Monotonic time, periodic/oneshot timers, calibration
**Critical for StarForth:** The heartbeat subsystem and physics feedback loops depend on precise, jitter-free timing. HAL must guarantee:
- Monotonic time (never goes backward)
- Sub-microsecond resolution
- Calibrated frequency (for TSC-based timing)
**Platform challenges:**
- **Linux:** `clock_gettime()` is good, but signal-based timers have latency
- **Kernel:** TSC drift, HPET/PIT fallback, per-core calibration
### 2. **Interrupts** (`hal_interrupt.h`)
**Purpose:** Enable/disable IRQs, register ISRs, query interrupt context
**Critical for StarForth:** Heartbeat ISR must run at precise intervals without VM involvement.
**Platform challenges:**
- **Linux:** Signals are "interrupt-like" but not true IRQs
- **Kernel:** IDT setup, APIC configuration, spurious interrupt handling
### 3. **Memory** (`hal_memory.h`)
**Purpose:** Allocate/free memory, page mapping, heap management
**Critical for StarForth:** Dictionary allocation, stack allocation, heap allocator must work identically across platforms.
**Platform challenges:**
- **Linux:** malloc/free are simple
- **Kernel:** Physical memory manager, virtual memory manager, heap allocator—all from scratch
### 4. **Console** (`hal_console.h`)
**Purpose:** Character I/O for REPL and diagnostics
**Critical for StarForth:** The REPL must work on all platforms for interactive experimentation.
**Platform challenges:**
- **Linux:** stdin/stdout are perfect
- **Kernel:** UART 16550 is well-documented but framebuffer is tricky
### 5. **CPU** (`hal_cpu.h`)
**Purpose:** CPU ID, relax/halt, SMP coordination
**Critical for StarForth:** Per-core execution heat tracking (future), SMP scalability (future).
**Platform challenges:**
- **Linux:** pthread local storage
- **Kernel:** Local APIC, per-core stacks, CPU-local storage
---
## HAL and the Physics Subsystems
The physics-driven adaptive runtime is the **primary beneficiary** of the HAL:
| Physics Subsystem | HAL Dependency | Why |
|-------------------|----------------|-----|
| **Execution Heat** | None (pure VM state) | Tracks word execution frequency |
| **Rolling Window** | `hal_time_now_ns()` | Timestamps for window entries |
| **Hot-Words Cache** | None (pure dictionary state) | Frequency-based reordering |
| **Pipelining Metrics** | None (pure transition tracking) | Word-to-word prediction |
| **Inference Engine** | `hal_time_now_ns()` | Calibration timing, ANOVA |
| **Heartbeat** | `hal_timer_periodic()`, `hal_interrupt.*` | ISR-based sampling |
**Key insight:** Only 2 of 6 subsystems touch the HAL directly, and only via clean interfaces. This preserves deterministic behavior while enabling kernel deployment.
---
## HAL and StarKernel
StarKernel is a **new platform implementation** of the HAL:
```
src/platform/kernel/
├── boot/
│ └── uefi_loader.c # UEFI entry point → BootInfo handoff
├── hal_time.c # TSC + HPET + APIC timer
├── hal_interrupt.c # IDT + Local APIC + IOAPIC
├── hal_memory.c # PMM + VMM + kmalloc
├── hal_console.c # UART 16550 + framebuffer
├── hal_cpu.c # SMP + CPU-local storage
└── drivers/ # PCI, AHCI, NVMe, VirtIO, etc.
```
**StarKernel does NOT modify:**
- VM core (`src/vm.c`)
- Physics subsystems (`src/dictionary_heat_optimization.c`, etc.)
- Word implementations (`src/word_source/*.c`)
**StarKernel ONLY implements:**
- HAL interfaces (`hal_*.c`)
- UEFI boot loader (`boot/uefi_loader.c`)
- Device drivers (`drivers/*.c`)
This is the **proof** that the HAL abstraction works: StarKernel is a pure platform layer, not a VM fork.
---
## HAL and StarshipOS
StarshipOS builds on StarKernel by adding:
- **Process model** - Forth tasks vs. traditional processes
- **Filesystem** - FAT32, ext2, or log-structured
- **Networking** - TCP/IP stack, DHCP, DNS
- **Device model** - Unified block/net/char device interfaces
- **Security model** - Capabilities, ACL, Forth-based access control
All of these **still use the HAL** for low-level access:
- Filesystem → `hal_memory` for caching, `hal_interrupt` for async I/O
- Networking → `hal_interrupt` for packet RX, `hal_time` for timeouts
- Device model → HAL as the common substrate
**The HAL is not just a kernel bootstrapping tool—it's the foundation for the entire OS.**
---
## Migration Strategy
Current StarForth codebase must be refactored to introduce the HAL:
### Phase 1: Define HAL Interfaces
- Write `include/hal/*.h` headers with contracts
- Document semantics, error handling, performance expectations
- **No implementation yet**
### Phase 2: Refactor Existing Platforms
- Create `src/platform/linux/hal_*.c` implementing HAL
- Create `src/platform/l4re/hal_*.c` implementing HAL
- Update VM code to call HAL instead of platform-specific APIs
- **VM must still build and pass all 936+ tests**
### Phase 3: Validate HAL
- Build and test on Linux
- Build and test on L4Re (if available)
- Verify deterministic behavior (0% algorithmic variance) still holds
- Benchmark: HAL must not add measurable overhead
### Phase 4: Implement StarKernel Platform
- Create `src/platform/kernel/` with HAL implementations
- UEFI boot loader
- Basic MM, console, timing
- **Goal:** Boot to `ok` prompt on QEMU/OVMF
### Phase 5: Full StarKernel
- Complete HAL implementations (interrupts, SMP, drivers)
- Heartbeat ISR running at kernel level
- Physics subsystems operational
- **Goal:** Reproduce DoE results on bare metal
### Phase 6: StarshipOS
- Process model, filesystem, networking
- Forth as native control plane
- **Goal:** Self-hosting OS
---
## Success Criteria
The HAL is successful if:
1. ✅ **VM core has zero platform-specific code**
2. ✅ **All 936+ tests pass on Linux, L4Re, and StarKernel**
3. ✅ **0% algorithmic variance maintained across platforms**
4. ✅ **No measurable performance regression from HAL abstraction**
5. ✅ **StarKernel boots to `ok` prompt and runs REPL**
6. ✅ **Heartbeat subsystem works identically on all platforms**
7. ✅ **New platforms can be added without touching VM code**
---
## Next Steps
See companion documentation:
- `interfaces.md` - Detailed HAL interface specifications
- `platform-implementations.md` - How to implement HAL for new platforms
- `migration-plan.md` - Step-by-step refactoring guide
- `starkernel-integration.md` - StarKernel-specific implementation details
---
## References
- StarForth VM core: `src/vm.c`
- Heartbeat system: `docs/03-architecture/heartbeat-system/`
- Physics feedback loops: `docs/FEEDBACK_LOOPS.md`
- Platform abstraction (current): `src/platform/`
docs/working/architecture/03-architecture/hal/platform-implementations.md
-738
View File
@@ -1,738 +0,0 @@
<!-- Moved from docs/03-architecture/hal/platform-implementations.md to docs/working/architecture/03-architecture/hal/platform-implementations.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Assumes a `src/platform/<platform>/hal_*.c` layout
> whose platform names ("Linux", "L4Re", "StarKernel") and interface never matched the built
> tree: the actual kernel HAL is `src/starkernel/hal/`, L4Re is no longer an active target,
> and the kernel is well past a hypothetical "Boot + Serial Output" starting point. Do not use
> as a platform porting guide. See the TARGET-HAL caveat in `README.md` and
> `.claude/CLAUDE.md`.
# HAL Platform Implementation Guide
## Overview
This document explains how to implement the HAL for a new platform. It provides detailed examples for Linux, L4Re, and StarKernel (freestanding) implementations.
> **HISTORICAL — L4Re/Fiasco.OC**: the L4Re examples below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build.
**Target audience:** Platform developers adding HAL support for a new target.
---
## Platform Directory Structure
```
src/platform/<platform>/
├── hal_time.c # Time & timer HAL implementation
├── hal_interrupt.c # Interrupt HAL implementation
├── hal_memory.c # Memory HAL implementation
├── hal_console.c # Console HAL implementation
├── hal_cpu.c # CPU HAL implementation
├── hal_panic.c # Panic/error HAL implementation
└── platform_init.c # Platform-specific initialization (optional)
```
### Build Integration
Each platform is compiled independently via Makefile:
```makefile
# Platform selection
PLATFORM ?= linux # linux | l4re | kernel
# Platform-specific source files
PLATFORM_SOURCES = \
src/platform/$(PLATFORM)/hal_time.c \
src/platform/$(PLATFORM)/hal_interrupt.c \
src/platform/$(PLATFORM)/hal_memory.c \
src/platform/$(PLATFORM)/hal_console.c \
src/platform/$(PLATFORM)/hal_cpu.c \
src/platform/$(PLATFORM)/hal_panic.c
# Platform-specific flags
ifeq ($(PLATFORM),kernel)
CFLAGS += -ffreestanding -nostdlib -mno-red-zone
LDFLAGS += -nostdlib -static
endif
```
---
## Implementation Examples
### 1. Linux Platform (`src/platform/linux/`)
Linux is the **reference platform** for development and testing. It provides the richest debugging environment.
#### `hal_time.c` (Linux)
```c
#include "hal/hal_time.h"
#include <time.h>
#include <signal.h>
#include <string.h>
#include <stdlib.h>
/* Timer state */
typedef struct {
timer_t timer_id;
hal_timer_callback_t callback;
void *ctx;
} linux_timer_t;
static linux_timer_t *periodic_timers = NULL;
static int timer_count = 0;
void hal_time_init(void) {
/* Linux: clock_gettime is always available */
struct timespec ts;
if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0) {
hal_panic("hal_time_init: CLOCK_MONOTONIC unavailable");
}
}
uint64_t hal_time_now_ns(void) {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
void hal_time_delay_ns(uint64_t ns) {
struct timespec req;
req.tv_sec = ns / 1000000000ULL;
req.tv_nsec = ns % 1000000000ULL;
nanosleep(&req, NULL);
}
/* Signal handler for periodic timer */
static void timer_signal_handler(int signo, siginfo_t *si, void *uc) {
(void)signo;
(void)uc;
linux_timer_t *timer = (linux_timer_t *)si->si_value.sival_ptr;
if (timer && timer->callback) {
timer->callback(timer->ctx);
}
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
/* Allocate timer structure */
linux_timer_t *timer = malloc(sizeof(linux_timer_t));
if (!timer) return -1;
timer->callback = callback;
timer->ctx = ctx;
/* Set up signal handler */
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_flags = SA_SIGINFO;
sa.sa_sigaction = timer_signal_handler;
sigemptyset(&sa.sa_mask);
sigaction(SIGRTMIN, &sa, NULL);
/* Create POSIX timer */
struct sigevent sev;
memset(&sev, 0, sizeof(sev));
sev.sigev_notify = SIGEV_SIGNAL;
sev.sigev_signo = SIGRTMIN;
sev.sigev_value.sival_ptr = timer;
if (timer_create(CLOCK_MONOTONIC, &sev, &timer->timer_id) != 0) {
free(timer);
return -1;
}
/* Arm the timer */
struct itimerspec its;
its.it_value.tv_sec = period_ns / 1000000000ULL;
its.it_value.tv_nsec = period_ns % 1000000000ULL;
its.it_interval = its.it_value; /* Periodic */
if (timer_settime(timer->timer_id, 0, &its, NULL) != 0) {
timer_delete(timer->timer_id);
free(timer);
return -1;
}
return timer_count++; /* Return timer ID */
}
int hal_timer_cancel(int timer_id) {
/* TODO: Implement timer cancellation */
(void)timer_id;
return 0;
}
uint64_t hal_time_frequency_hz(void) {
/* Linux: CLOCK_MONOTONIC is variable frequency, return 0 */
return 0;
}
```
#### `hal_interrupt.c` (Linux)
```c
#include "hal/hal_interrupt.h"
#include <signal.h>
/* Linux uses signals as "interrupts" */
static __thread int in_signal_handler = 0;
void hal_interrupt_init(void) {
/* No initialization needed on Linux */
}
void hal_irq_enable(void) {
/* Linux: interrupts (signals) always enabled */
}
unsigned long hal_irq_disable(void) {
/* Linux: can't disable signals, return dummy state */
return 0;
}
void hal_irq_restore(unsigned long state) {
(void)state;
}
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) {
/* Linux: IRQs are handled via signal mechanism in hal_timer_periodic() */
(void)irq;
(void)isr;
(void)ctx;
return 0;
}
int hal_irq_unregister(unsigned int irq) {
(void)irq;
return 0;
}
int hal_in_interrupt_context(void) {
return in_signal_handler;
}
```
#### `hal_memory.c` (Linux)
```c
#include "hal/hal_memory.h"
#include <stdlib.h>
#include <string.h>
void hal_mem_init(void) {
/* malloc is always available on Linux */
}
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
void *ptr = malloc(size);
if (ptr) {
memset(ptr, 0, size); /* Zero-initialize */
}
return ptr;
}
void hal_mem_free(void *ptr) {
free(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
/* Linux: just allocate from heap (no separate PMM) */
size_t size = count * hal_mem_page_size();
return (uint64_t)hal_mem_alloc(size);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
(void)count;
hal_mem_free((void *)paddr);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
/* Linux: identity mapping, no-op */
(void)vaddr;
(void)paddr;
(void)size;
(void)flags;
return 0;
}
size_t hal_mem_page_size(void) {
return 4096; /* Standard page size */
}
```
#### `hal_console.c` (Linux)
```c
#include "hal/hal_console.h"
#include <stdio.h>
#include <unistd.h>
#include <poll.h>
void hal_console_init(void) {
/* stdin/stdout always available */
}
void hal_console_putc(char c) {
putchar(c);
fflush(stdout); /* Ensure character is visible */
}
void hal_console_puts(const char *s) {
if (s) {
fputs(s, stdout);
fflush(stdout);
}
}
int hal_console_getc(void) {
return getchar();
}
int hal_console_has_input(void) {
struct pollfd pfd;
pfd.fd = STDIN_FILENO;
pfd.events = POLLIN;
return poll(&pfd, 1, 0) > 0;
}
```
#### `hal_cpu.c` (Linux)
```c
#include "hal/hal_cpu.h"
#include <unistd.h>
#include <sched.h>
void hal_cpu_init(void) {
/* No initialization needed */
}
unsigned int hal_cpu_id(void) {
/* Linux: return thread ID (not true CPU ID) */
return 0; /* Single-threaded for now */
}
void hal_cpu_relax(void) {
sched_yield();
}
void hal_cpu_halt(void) {
pause(); /* Wait for signal */
}
unsigned int hal_cpu_count(void) {
return sysconf(_SC_NPROCESSORS_ONLN);
}
```
#### `hal_panic.c` (Linux)
```c
#include "hal/hal_panic.h"
#include <stdio.h>
#include <stdlib.h>
void hal_panic(const char *msg) {
fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown error");
fflush(stderr);
abort();
}
```
---
### 2. Kernel Platform (`src/platform/kernel/`)
The kernel platform is **freestanding**—no libc, no OS, direct hardware access.
#### `hal_time.c` (Kernel)
```c
#include "hal/hal_time.h"
#include "hal/hal_interrupt.h"
#include "kernel/apic.h"
#include "kernel/hpet.h"
#include "kernel/tsc.h"
/* TSC calibration state */
static uint64_t tsc_frequency_hz = 0;
static uint64_t tsc_offset_ns = 0;
void hal_time_init(void) {
/* Calibrate TSC using HPET */
tsc_frequency_hz = tsc_calibrate_hpet();
if (tsc_frequency_hz == 0) {
hal_panic("hal_time_init: TSC calibration failed");
}
/* Validate monotonicity */
uint64_t t1 = hal_time_now_ns();
for (volatile int i = 0; i < 1000; i++);
uint64_t t2 = hal_time_now_ns();
if (t2 < t1) {
hal_panic("hal_time_init: TSC not monotonic");
}
}
uint64_t hal_time_now_ns(void) {
/* Read TSC and convert to nanoseconds */
uint64_t tsc = rdtsc();
return tsc_to_ns(tsc) - tsc_offset_ns;
}
void hal_time_delay_ns(uint64_t ns) {
uint64_t end = hal_time_now_ns() + ns;
while (hal_time_now_ns() < end) {
hal_cpu_relax();
}
}
/* APIC timer state */
static hal_timer_callback_t apic_timer_callback = NULL;
static void *apic_timer_ctx = NULL;
/* APIC timer ISR (called from IDT) */
void apic_timer_isr(void) {
if (apic_timer_callback) {
apic_timer_callback(apic_timer_ctx);
}
apic_eoi(); /* Send End-Of-Interrupt */
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
if (period_ns < 1000) return -2; /* Minimum 1µs */
apic_timer_callback = callback;
apic_timer_ctx = ctx;
/* Configure APIC timer */
uint64_t ticks = (period_ns * apic_timer_frequency_hz()) / 1000000000ULL;
apic_timer_periodic(ticks);
return 1; /* Timer ID */
}
int hal_timer_cancel(int timer_id) {
(void)timer_id;
apic_timer_stop();
apic_timer_callback = NULL;
return 0;
}
uint64_t hal_time_frequency_hz(void) {
return tsc_frequency_hz;
}
```
#### `hal_interrupt.c` (Kernel)
```c
#include "hal/hal_interrupt.h"
#include "kernel/idt.h"
#include "kernel/apic.h"
/* Per-CPU interrupt nesting count */
static __thread unsigned int interrupt_depth = 0;
void hal_interrupt_init(void) {
/* Initialize IDT */
idt_init();
/* Initialize Local APIC */
apic_init();
/* Disable legacy PIC (use APIC instead) */
pic_disable();
}
void hal_irq_enable(void) {
__asm__ volatile("sti");
}
unsigned long hal_irq_disable(void) {
unsigned long flags;
__asm__ volatile(
"pushfq\n"
"popq %0\n"
"cli\n"
: "=r"(flags)
);
return flags;
}
void hal_irq_restore(unsigned long state) {
__asm__ volatile(
"pushq %0\n"
"popfq\n"
:
: "r"(state)
);
}
/* ISR table (256 entries for x86_64 IDT) */
static hal_isr_t isr_table[256] = {0};
static void *isr_ctx_table[256] = {0};
/* Common ISR entry point (called from IDT stubs) */
void common_isr_handler(unsigned int vector) {
interrupt_depth++;
if (isr_table[vector]) {
isr_table[vector](vector, isr_ctx_table[vector]);
}
interrupt_depth--;
apic_eoi();
}
int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) {
if (irq >= 256) return -1;
isr_table[irq] = isr;
isr_ctx_table[irq] = ctx;
/* Configure IOAPIC to route IRQ to Local APIC */
ioapic_route_irq(irq, 0); /* Route to CPU 0 */
return 0;
}
int hal_irq_unregister(unsigned int irq) {
if (irq >= 256) return -1;
isr_table[irq] = NULL;
isr_ctx_table[irq] = NULL;
return 0;
}
int hal_in_interrupt_context(void) {
return interrupt_depth > 0;
}
```
#### `hal_memory.c` (Kernel)
```c
#include "hal/hal_memory.h"
#include "kernel/pmm.h" /* Physical Memory Manager */
#include "kernel/vmm.h" /* Virtual Memory Manager */
#include "kernel/kmalloc.h" /* Kernel heap allocator */
void hal_mem_init(void) {
/* Initialize physical memory manager (parse UEFI memory map) */
pmm_init();
/* Initialize virtual memory manager (set up page tables) */
vmm_init();
/* Initialize kernel heap allocator */
kmalloc_init();
}
void *hal_mem_alloc(size_t size) {
if (size == 0) return NULL;
return kmalloc(size); /* Kernel heap allocator */
}
void hal_mem_free(void *ptr) {
kfree(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
return pmm_alloc_pages(count);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
pmm_free_pages(paddr, count);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
/* Translate HAL flags to page table flags */
unsigned int pt_flags = 0;
if (flags & HAL_MEM_WRITE) pt_flags |= PAGE_WRITE;
if (flags & HAL_MEM_EXEC) pt_flags |= PAGE_EXEC;
if (flags & HAL_MEM_NOCACHE) pt_flags |= PAGE_NOCACHE;
return vmm_map(vaddr, paddr, size, pt_flags);
}
size_t hal_mem_page_size(void) {
return 4096;
}
```
#### `hal_console.c` (Kernel)
```c
#include "hal/hal_console.h"
#include "kernel/uart.h"
#include "kernel/framebuffer.h"
void hal_console_init(void) {
/* Initialize UART 16550 (serial console) */
uart_init(0x3F8, 115200); /* COM1, 115200 baud */
/* Initialize framebuffer (if available from UEFI GOP) */
fb_init();
}
void hal_console_putc(char c) {
/* Output to both serial and framebuffer */
uart_putc(c);
fb_putc(c);
}
void hal_console_puts(const char *s) {
if (!s) return;
while (*s) {
hal_console_putc(*s++);
}
}
int hal_console_getc(void) {
/* Read from UART (blocking) */
return uart_getc();
}
int hal_console_has_input(void) {
return uart_has_data();
}
```
#### `hal_cpu.c` (Kernel)
```c
#include "hal/hal_cpu.h"
#include "kernel/apic.h"
static unsigned int cpu_count = 1;
void hal_cpu_init(void) {
/* Detect CPU features */
cpu_detect_features();
/* Bring up secondary CPUs (SMP) */
cpu_count = smp_init();
}
unsigned int hal_cpu_id(void) {
/* Get Local APIC ID */
return apic_get_id();
}
void hal_cpu_relax(void) {
__asm__ volatile("pause");
}
void hal_cpu_halt(void) {
__asm__ volatile("hlt");
}
unsigned int hal_cpu_count(void) {
return cpu_count;
}
```
---
## Platform Testing Strategy
### 1. **Unit Tests (Per-Platform)**
Each platform should have unit tests for HAL functions:
```c
/* test/platform/linux/hal_time_test.c */
void test_hal_time_monotonic(void) {
hal_time_init();
uint64_t t1 = hal_time_now_ns();
hal_time_delay_ns(1000000); /* 1ms */
uint64_t t2 = hal_time_now_ns();
assert(t2 > t1);
assert((t2 - t1) >= 1000000);
assert((t2 - t1) < 2000000); /* Allow 100% jitter for test */
}
```
### 2. **Cross-Platform VM Tests**
VM tests should run identically on all platforms:
```bash
# Linux
make PLATFORM=linux test
# L4Re
make PLATFORM=l4re test
# Kernel (QEMU)
make PLATFORM=kernel test-qemu
```
**Success criteria:** All 936+ tests pass on all platforms.
### 3. **Determinism Validation**
Physics subsystems must show 0% algorithmic variance:
```bash
# Run DoE on each platform
for platform in linux l4re kernel; do
make PLATFORM=$platform fastest
./build/$ARCH/fastest/starforth --doe > results_$platform.csv
done
# Compare results (should be identical)
diff results_linux.csv results_kernel.csv
```
---
## Common Implementation Pitfalls
### 1. **Non-Monotonic Time**
**Symptom:** Physics metrics show negative deltas, rolling window corrupted.
**Cause:** TSC goes backward (multi-core, frequency scaling).
**Fix:** Use `rdtscp` instead of `rdtsc`, synchronize TSC across cores, or fall back to HPET.
### 2. **Timer Jitter**
**Symptom:** Heartbeat intervals vary by 10-50%, metrics show high variance.
**Cause:** Signal latency (Linux), interrupt latency (kernel).
**Fix:** Minimize ISR work, use high-priority timer interrupt, disable frequency scaling.
### 3. **Memory Leaks**
**Symptom:** `hal_mem_alloc()` calls grow without matching `hal_mem_free()`.
**Cause:** Missing free in error paths, lost pointers.
**Fix:** Valgrind on Linux, manual leak tracking on kernel.
### 4. **Interrupt Context Violations**
**Symptom:** Crashes, hangs, or corruption when calling blocking HAL functions from ISR.
**Cause:** Calling `hal_mem_alloc()` or `hal_console_getc()` from timer ISR.
**Fix:** Assert `!hal_in_interrupt_context()` in blocking functions.
---
## Next Steps
- See `migration-plan.md` for refactoring existing code
- See `starkernel-integration.md` for kernel-specific details
docs/working/architecture/03-architecture/hal/starkernel-integration.md
-770
View File
@@ -1,770 +0,0 @@
<!-- Moved from docs/03-architecture/hal/starkernel-integration.md to docs/working/architecture/03-architecture/hal/starkernel-integration.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Describes implementing `src/platform/kernel/` and a
> "Milestone 1: Boot + Serial Output" starting point. True architecture is far ahead: the
> kernel lives in `src/starkernel/` with M0–M7 complete and M7.1 live. Kept for historical
> context; for the real boot/kernel story see `.claude/CLAUDE.md` and `docs/fabric/`.
# StarKernel HAL Integration Guide
## Overview
This document provides **kernel-specific implementation details** for the HAL platform layer that becomes StarKernel. It covers UEFI boot, freestanding C environment, hardware initialization, and the path to a working `ok` prompt.
**Audience:** Kernel developers implementing `src/platform/kernel/`
---
## StarKernel Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ UEFI Firmware │
│ • Initializes hardware │
│ • Provides boot services (memory map, ACPI, GOP) │
│ • Loads BOOTX64.EFI │
└─────────────────────┬───────────────────────────────────────┘
│
↓ ExitBootServices()
┌─────────────────────────────────────────────────────────────┐
│ StarKernel Boot Loader (stage 0) │
│ • Collects BootInfo (memory map, ACPI tables, framebuffer) │
│ • Sets up initial page tables │
│ • Exits UEFI boot services │
│ • Jumps to StarKernel proper │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ StarKernel HAL (stage 1) │
│ • CPU initialization (GDT, IDT, interrupts) │
│ • Memory subsystem (PMM, VMM, heap) │
│ • Time subsystem (TSC, HPET, APIC timer) │
│ • Console (UART + framebuffer) │
└─────────────────────┬───────────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ StarForth VM │
│ • vm_create() │
│ • Initialize physics subsystems │
│ • Start REPL │
│ • Print "ok" prompt │
└─────────────────────────────────────────────────────────────┘
```
---
## Directory Structure
```
src/platform/kernel/
├── boot/
│ ├── uefi_loader.c # UEFI entry point (BOOTX64.EFI)
│ ├── bootinfo.h # BootInfo struct definition
│ └── handoff.S # ASM trampoline (UEFI → StarKernel)
│
├── cpu/
│ ├── gdt.c # Global Descriptor Table
│ ├── idt.c # Interrupt Descriptor Table
│ ├── isr.S # Interrupt service routine stubs
│ └── smp.c # SMP bring-up (future)
│
├── mm/
│ ├── pmm.c # Physical Memory Manager
│ ├── vmm.c # Virtual Memory Manager
│ ├── kmalloc.c # Kernel heap allocator
│ └── paging.S # Page table manipulation ASM
│
├── time/
│ ├── tsc.c # Time Stamp Counter
│ ├── hpet.c # High Precision Event Timer
│ ├── apic_timer.c # Local APIC timer
│ └── pit.c # Programmable Interval Timer (fallback)
│
├── drivers/
│ ├── uart.c # UART 16550 serial console
│ ├── framebuffer.c # UEFI GOP framebuffer
│ ├── apic.c # Local APIC + IOAPIC
│ ├── acpi.c # ACPI table parsing
│ └── pci.c # PCI enumeration (future)
│
├── hal_time.c # HAL time implementation
├── hal_interrupt.c # HAL interrupt implementation
├── hal_memory.c # HAL memory implementation
├── hal_console.c # HAL console implementation
├── hal_cpu.c # HAL CPU implementation
├── hal_panic.c # HAL panic implementation
│
└── kernel_main.c # StarKernel entry point (after UEFI)
```
---
## Boot Sequence Detail
### 1. UEFI Loader (`boot/uefi_loader.c`)
**Responsibilities:**
- Collect system information from UEFI
- Allocate kernel memory
- Exit UEFI boot services
- Jump to StarKernel
**Implementation:**
```c
/* boot/uefi_loader.c */
#include <efi.h>
#include <efilib.h>
#include "bootinfo.h"
/* BootInfo passed to StarKernel */
typedef struct {
uint64_t memory_map_addr;
uint64_t memory_map_size;
uint64_t memory_map_descriptor_size;
uint64_t acpi_rsdp_addr;
uint64_t framebuffer_addr;
uint32_t framebuffer_width;
uint32_t framebuffer_height;
uint32_t framebuffer_pitch;
} BootInfo;
EFI_STATUS EFIAPI efi_main(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable) {
InitializeLib(ImageHandle, SystemTable);
BootInfo *boot_info = (BootInfo *)AllocatePool(sizeof(BootInfo));
/* 1. Get memory map */
UINTN map_size = 0, map_key, descriptor_size;
UINT32 descriptor_version;
EFI_MEMORY_DESCRIPTOR *memory_map = NULL;
uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key,
&descriptor_size, &descriptor_version);
map_size += 2 * descriptor_size; /* Extra space for ExitBootServices */
memory_map = (EFI_MEMORY_DESCRIPTOR *)AllocatePool(map_size);
uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key,
&descriptor_size, &descriptor_version);
boot_info->memory_map_addr = (uint64_t)memory_map;
boot_info->memory_map_size = map_size;
boot_info->memory_map_descriptor_size = descriptor_size;
/* 2. Get ACPI RSDP */
EFI_GUID acpi_20_guid = ACPI_20_TABLE_GUID;
void *rsdp = NULL;
for (UINTN i = 0; i < ST->NumberOfTableEntries; i++) {
if (CompareGuid(&ST->ConfigurationTable[i].VendorGuid, &acpi_20_guid) == 0) {
rsdp = ST->ConfigurationTable[i].VendorTable;
break;
}
}
boot_info->acpi_rsdp_addr = (uint64_t)rsdp;
/* 3. Get framebuffer from GOP (Graphics Output Protocol) */
EFI_GUID gop_guid = EFI_GRAPHICS_OUTPUT_PROTOCOL_GUID;
EFI_GRAPHICS_OUTPUT_PROTOCOL *gop = NULL;
uefi_call_wrapper(BS->LocateProtocol, 3, &gop_guid, NULL, (void **)&gop);
if (gop) {
boot_info->framebuffer_addr = gop->Mode->FrameBufferBase;
boot_info->framebuffer_width = gop->Mode->Info->HorizontalResolution;
boot_info->framebuffer_height = gop->Mode->Info->VerticalResolution;
boot_info->framebuffer_pitch = gop->Mode->Info->PixelsPerScanLine * 4;
}
/* 4. Exit boot services */
uefi_call_wrapper(BS->ExitBootServices, 2, ImageHandle, map_key);
/* 5. Jump to StarKernel */
extern void kernel_main(BootInfo *boot_info);
kernel_main(boot_info);
/* Never returns */
while (1) __asm__ volatile("hlt");
return EFI_SUCCESS;
}
```
### 2. StarKernel Entry Point (`kernel_main.c`)
**Responsibilities:**
- Parse BootInfo
- Initialize CPU (GDT, IDT)
- Initialize memory subsystem
- Initialize HAL
- Call `main()` to start VM
**Implementation:**
```c
/* kernel_main.c */
#include "bootinfo.h"
#include "cpu/gdt.h"
#include "cpu/idt.h"
#include "mm/pmm.h"
#include "mm/vmm.h"
#include "drivers/uart.h"
#include "drivers/framebuffer.h"
/* Global BootInfo (available to all kernel code) */
BootInfo *g_boot_info = NULL;
void kernel_main(BootInfo *boot_info) {
g_boot_info = boot_info;
/* 1. Initialize serial console ASAP (for debug output) */
uart_early_init(0x3F8, 115200); /* COM1 */
uart_puts("StarKernel booting...\r\n");
/* 2. Initialize CPU structures */
gdt_init();
idt_init();
/* 3. Initialize physical memory manager */
pmm_init(boot_info);
uart_puts("PMM initialized\r\n");
/* 4. Initialize virtual memory manager */
vmm_init();
uart_puts("VMM initialized\r\n");
/* 5. Initialize kernel heap */
kmalloc_init();
uart_puts("Heap initialized\r\n");
/* 6. Initialize framebuffer */
fb_init(boot_info);
/* 7. Initialize HAL */
hal_time_init();
hal_interrupt_init();
hal_mem_init();
hal_console_init();
hal_cpu_init();
uart_puts("HAL initialized\r\n");
/* 8. Call standard main() to start VM */
extern int main(int argc, char **argv);
char *argv[] = {"starforth", NULL};
main(1, argv);
/* Never returns, but halt if it does */
hal_panic("main() returned unexpectedly");
}
```
---
## HAL Implementation Details
### 1. HAL Time (`hal_time.c`)
**Hardware:** TSC (Time Stamp Counter), HPET (High Precision Event Timer), APIC timer
**Key challenges:**
- TSC calibration against HPET
- TSC frequency varies on old CPUs (need invariant TSC check)
- APIC timer setup for periodic interrupts
**Implementation outline:**
```c
/* hal_time.c */
#include "hal/hal_time.h"
#include "time/tsc.h"
#include "time/hpet.h"
#include "time/apic_timer.h"
static uint64_t tsc_hz = 0;
static uint64_t boot_tsc = 0;
void hal_time_init(void) {
/* Check for invariant TSC */
if (!tsc_is_invariant()) {
hal_panic("hal_time: TSC not invariant, falling back to HPET");
/* TODO: Use HPET directly if TSC unreliable */
}
/* Calibrate TSC frequency using HPET */
tsc_hz = tsc_calibrate_hpet();
if (tsc_hz == 0) {
hal_panic("hal_time: TSC calibration failed");
}
boot_tsc = rdtsc();
/* Initialize APIC timer for periodic interrupts */
apic_timer_init();
}
uint64_t hal_time_now_ns(void) {
uint64_t tsc = rdtsc() - boot_tsc;
return (tsc * 1000000000ULL) / tsc_hz;
}
int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) {
return apic_timer_periodic(period_ns, callback, ctx);
}
```
**See:** `time/tsc.c`, `time/hpet.c`, `time/apic_timer.c` for low-level implementations.
---
### 2. HAL Interrupt (`hal_interrupt.c`)
**Hardware:** IDT (Interrupt Descriptor Table), Local APIC, IOAPIC
**Key challenges:**
- Setting up 256 IDT entries
- Routing hardware IRQs through IOAPIC
- Tracking interrupt nesting depth (for `hal_in_interrupt_context()`)
**Implementation outline:**
```c
/* hal_interrupt.c */
#include "hal/hal_interrupt.h"
#include "cpu/idt.h"
#include "drivers/apic.h"
/* Per-CPU interrupt depth (thread-local on SMP) */
static __thread unsigned int irq_depth = 0;
void hal_interrupt_init(void) {
/* Set up IDT with 256 entries */
idt_init();
/* Disable legacy PIC (use APIC instead) */
outb(0x21, 0xFF); /* Mask all PIC1 IRQs */
outb(0xA1, 0xFF); /* Mask all PIC2 IRQs */
/* Initialize Local APIC */
apic_init();
/* Initialize IOAPIC */
ioapic_init();
}
void hal_irq_enable(void) {
__asm__ volatile("sti");
}
unsigned long hal_irq_disable(void) {
unsigned long flags;
__asm__ volatile("pushfq; popq %0; cli" : "=r"(flags));
return flags;
}
void hal_irq_restore(unsigned long flags) {
__asm__ volatile("pushq %0; popfq" :: "r"(flags));
}
int hal_in_interrupt_context(void) {
return irq_depth > 0;
}
/* Common ISR entry (called from IDT stubs) */
void isr_common_handler(uint64_t vector, uint64_t error_code) {
irq_depth++;
/* Dispatch to registered handler */
extern void isr_dispatch(uint64_t vector, uint64_t error_code);
isr_dispatch(vector, error_code);
irq_depth--;
/* Send EOI to APIC */
apic_eoi();
}
```
**See:** `cpu/idt.c`, `cpu/isr.S`, `drivers/apic.c` for low-level implementations.
---
### 3. HAL Memory (`hal_memory.c`)
**Components:**
- **PMM (Physical Memory Manager):** Tracks free/used physical page frames
- **VMM (Virtual Memory Manager):** Manages page tables (4-level paging on x86_64)
- **kmalloc:** Kernel heap allocator (slab or buddy allocator)
**Implementation outline:**
```c
/* hal_memory.c */
#include "hal/hal_memory.h"
#include "mm/pmm.h"
#include "mm/vmm.h"
#include "mm/kmalloc.h"
void hal_mem_init(void) {
/* PMM and VMM initialized in kernel_main() before HAL */
/* Just verify they're ready */
if (!pmm_is_initialized()) {
hal_panic("hal_mem_init: PMM not initialized");
}
}
void *hal_mem_alloc(size_t size) {
return kmalloc(size);
}
void hal_mem_free(void *ptr) {
kfree(ptr);
}
uint64_t hal_mem_alloc_pages(size_t count) {
return pmm_alloc_pages(count);
}
void hal_mem_free_pages(uint64_t paddr, size_t count) {
pmm_free_pages(paddr, count);
}
int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) {
return vmm_map_range(vaddr, paddr, size, flags);
}
size_t hal_mem_page_size(void) {
return 4096;
}
```
**PMM details** (`mm/pmm.c`):
- Bitmap allocator (1 bit per 4KB page)
- Parse UEFI memory map to mark usable vs. reserved regions
- Reserve kernel image, ACPI tables, framebuffer
**VMM details** (`mm/vmm.c`):
- 4-level page tables (PML4 → PDPT → PD → PT)
- Recursive mapping trick for page table access
- Identity-map kernel, higher-half kernel optional
**kmalloc details** (`mm/kmalloc.c`):
- Slab allocator for common sizes (16, 32, 64, ..., 4096 bytes)
- Buddy allocator for large allocations
- Zero-initialization required by HAL contract
---
### 4. HAL Console (`hal_console.c`)
**Hardware:** UART 16550 (serial), UEFI GOP framebuffer (video)
**Implementation outline:**
```c
/* hal_console.c */
#include "hal/hal_console.h"
#include "drivers/uart.h"
#include "drivers/framebuffer.h"
void hal_console_init(void) {
/* UART already initialized in kernel_main() for early debug */
/* Framebuffer initialized from BootInfo */
}
void hal_console_putc(char c) {
uart_putc(c); /* Always output to serial */
fb_putc(c); /* Also output to framebuffer if available */
}
void hal_console_puts(const char *s) {
while (*s) {
hal_console_putc(*s++);
}
}
int hal_console_getc(void) {
/* Block until UART has data */
return uart_getc();
}
int hal_console_has_input(void) {
return uart_has_data();
}
```
**UART details** (`drivers/uart.c`):
- 16550 compatible (COM1 = 0x3F8, COM2 = 0x2F8)
- 115200 baud default
- Polling mode (no interrupts for simplicity)
**Framebuffer details** (`drivers/framebuffer.c`):
- Linear framebuffer from UEFI GOP
- 32-bit RGB pixels
- Software text rendering (8x16 font)
- Scrolling, cursor, ANSI escape codes (optional)
---
### 5. HAL CPU (`hal_cpu.c`)
**Implementation outline:**
```c
/* hal_cpu.c */
#include "hal/hal_cpu.h"
#include "drivers/apic.h"
static unsigned int num_cpus = 1;
void hal_cpu_init(void) {
/* Detect CPU features (CPUID) */
/* TODO: Bring up secondary CPUs (SMP) */
}
unsigned int hal_cpu_id(void) {
return apic_get_id(); /* Local APIC ID */
}
void hal_cpu_relax(void) {
__asm__ volatile("pause");
}
void hal_cpu_halt(void) {
__asm__ volatile("hlt");
}
unsigned int hal_cpu_count(void) {
return num_cpus;
}
```
---
## Build System
### Toolchain Requirements
**Freestanding C environment:**
- `gcc` or `clang` with `-ffreestanding`
- `ld` with custom linker script
- `objcopy` to create PE32+ executable (for UEFI)
### Makefile Additions
```makefile
# Platform: kernel
ifeq ($(PLATFORM),kernel)
CC = gcc
LD = ld
OBJCOPY = objcopy
CFLAGS += -ffreestanding -nostdlib -mno-red-zone -mcmodel=large
CFLAGS += -mno-mmx -mno-sse -mno-sse2 # No FP in kernel
CFLAGS += -I/usr/include/efi -I/usr/include/efi/x86_64
LDFLAGS += -nostdlib -static -T src/platform/kernel/linker.ld
# Platform sources
PLATFORM_SOURCES = \
src/platform/kernel/boot/uefi_loader.c \
src/platform/kernel/kernel_main.c \
src/platform/kernel/cpu/gdt.c \
src/platform/kernel/cpu/idt.c \
src/platform/kernel/cpu/isr.S \
src/platform/kernel/mm/pmm.c \
src/platform/kernel/mm/vmm.c \
src/platform/kernel/mm/kmalloc.c \
src/platform/kernel/drivers/uart.c \
src/platform/kernel/drivers/framebuffer.c \
src/platform/kernel/drivers/apic.c \
src/platform/kernel/hal_time.c \
src/platform/kernel/hal_interrupt.c \
src/platform/kernel/hal_memory.c \
src/platform/kernel/hal_console.c \
src/platform/kernel/hal_cpu.c \
src/platform/kernel/hal_panic.c
# Build UEFI PE32+ executable
starforth.efi: $(OBJECTS)
$(LD) $(LDFLAGS) -o starforth.so $(OBJECTS)
$(OBJCOPY) -j .text -j .data -j .rodata -j .reloc \
--target=efi-app-x86_64 starforth.so $@
endif
```
### Linker Script (`src/platform/kernel/linker.ld`)
```ld
OUTPUT_FORMAT("elf64-x86-64")
ENTRY(efi_main)
SECTIONS {
. = 0x100000; /* Load at 1MB (standard kernel load address) */
.text : {
*(.text .text.*)
}
.rodata : {
*(.rodata .rodata.*)
}
.data : {
*(.data .data.*)
}
.bss : {
*(COMMON)
*(.bss .bss.*)
}
/DISCARD/ : {
*(.eh_frame)
}
}
```
---
## Testing on QEMU
### QEMU + OVMF Setup
```bash
# Install OVMF (UEFI firmware for QEMU)
sudo apt install ovmf
# Create ESP (EFI System Partition)
mkdir -p esp/EFI/BOOT
cp starforth.efi esp/EFI/BOOT/BOOTX64.EFI
# Run QEMU
qemu-system-x86_64 \
-bios /usr/share/ovmf/OVMF.fd \
-drive file=fat:rw:esp/,format=raw \
-serial stdio \
-m 512M \
-enable-kvm
```
### Expected Output
```
StarKernel booting...
PMM initialized
VMM initialized
Heap initialized
HAL initialized
StarForth VM v1.0
ok
```
---
## Debugging
### Serial Debugging
```c
/* Early debug output via UART */
void debug_puts(const char *s) {
while (*s) {
while (!(inb(0x3FD) & 0x20)); /* Wait for UART ready */
outb(0x3F8, *s++); /* Write character */
}
}
```
### GDB Remote Debugging
```bash
# Start QEMU with GDB server
qemu-system-x86_64 ... -s -S
# In another terminal
gdb starforth.elf
(gdb) target remote :1234
(gdb) break kernel_main
(gdb) continue
```
### Panic Handler
```c
/* hal_panic.c */
void hal_panic(const char *msg) {
hal_irq_disable();
hal_console_puts("\n*** KERNEL PANIC ***\n");
hal_console_puts(msg ? msg : "unknown error");
hal_console_puts("\nSystem halted.\n");
/* Halt all CPUs */
while (1) {
hal_cpu_halt();
}
}
```
---
## Roadmap to `ok` Prompt
### Milestone 1: Boot + Serial Output
- UEFI loader runs
- Kernel prints "StarKernel booting..." to serial
- System doesn't triple-fault
### Milestone 2: Memory Works
- PMM tracks physical pages
- VMM maps kernel
- kmalloc/kfree work
### Milestone 3: HAL Initialized
- All `hal_*_init()` functions complete
- HAL functions callable (even if not fully functional)
### Milestone 4: VM Starts
- `vm_create()` succeeds
- Dictionary allocated
- No crashes in VM init
### Milestone 5: REPL Runs
- REPL prints "`ok`" prompt
- Can type characters (echoed to serial)
- Can execute simple words (`1 2 + .` → `3`)
### Milestone 6: Physics Subsystems Work
- Execution heat tracking operational
- Heartbeat timer fires
- Rolling window captures execution history
### Milestone 7: Full Test Suite
- All 936+ tests pass on kernel
- DoE mode works
- 0% algorithmic variance on bare metal
---
## Future Work (StarshipOS)
After achieving `ok` prompt on StarKernel:
1. **Storage:** AHCI/NVMe drivers, FAT32 filesystem
2. **Networking:** VirtIO-net driver, TCP/IP stack
3. **Process model:** Forth tasks, scheduling, IPC
4. **Device model:** Unified block/net/char subsystem
5. **Security:** Capabilities, ACL, Forth-based access control
**The HAL remains the foundation throughout.**
---
## References
- UEFI Specification: https://uefi.org/specifications
- GNU-EFI: https://sourceforge.net/projects/gnu-efi/
- OSDev Wiki: https://wiki.osdev.org/
- Intel SDM: https://www.intel.com/sdm
- ACPI Specification: https://uefi.org/acpi
docs/working/architecture/03-architecture/heartbeat-system/implementation-plan.md
-542
View File
@@ -1,542 +0,0 @@
<!-- Moved from docs/03-architecture/heartbeat-system/implementation-plan.md to docs/working/architecture/03-architecture/heartbeat-system/implementation-plan.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Superseded by `instrumentation-plan.md`. Proposes the
> earlier `HeartbeatRateSample` design; the live target is `HeartbeatTickSnapshot`
> (`include/vm.h`) with a `tick_buffer` and `heartbeat_export_csv()`. Kept for historical
> record.
# Implementation Plan: Heartbeat Rate Logging
**Date:** November 20, 2025
**Priority:** BLOCKING — Must implement before running Phase 2 DoE
**Complexity:** Medium (requires code additions, CSV schema update)
---
## Overview
We need to capture **tick_ns trajectory** (the actual heartbeat rate changes) during execution, then analyze how each configuration modulates heartrate in response to workload.
---
## Step 1: Add Heartbeat Rate Sample Struct
### File: `include/vm.h`
Add after `HeartbeatSnapshot` definition:
```c
/* === Heartbeat Rate Trajectory Collection === */
#define HEARTBEAT_RATE_SAMPLE_MAX 100000 /* Maximum samples per run */
typedef struct {
uint64_t tick_number; /* Absolute tick count */
uint64_t tick_ns; /* ← HEARTBEAT RATE (nanoseconds) */
uint64_t workload_ops_this_tick; /* Dictionary lookups this tick */
uint64_t timestamp_ns; /* Wall-clock timestamp */
uint64_t hot_word_count; /* Physics state: hot words */
uint64_t total_heat; /* Physics state: total execution heat */
} HeartbeatRateSample;
```
### File: `include/vm.h` (in VM struct)
Add to the HeartbeatState section:
```c
/** @name Heartbeat Rate Trajectory (Phase 2 DoE)
* @{
*/
HeartbeatRateSample* heartbeat_rate_samples; /* Circular buffer of rate samples */
uint64_t heartbeat_rate_sample_count; /* Number of samples collected */
/** @} */
```
---
## Step 2: Initialize Heartbeat Rate Collection
### File: `src/vm.c` in `vm_init()`
Add after heartbeat initialization:
```c
/* Initialize heartbeat rate sampling for Phase 2 DoE */
vm->heartbeat_rate_samples = calloc(HEARTBEAT_RATE_SAMPLE_MAX, sizeof(HeartbeatRateSample));
vm->heartbeat_rate_sample_count = 0;
```
### File: `src/vm.c` in `vm_cleanup()`
Add cleanup:
```c
if (vm->heartbeat_rate_samples) {
free(vm->heartbeat_rate_samples);
vm->heartbeat_rate_samples = NULL;
}
```
---
## Step 3: Capture Heartbeat Rate Per Tick
### File: `src/vm.c` - Add helper function
```c
/**
* @brief Capture current heartbeat rate sample for analysis
*
* Called by heartbeat_thread_main() to log the tick_ns (heartbeat rate)
* along with contemporaneous workload metrics.
*
* @param vm Pointer to VM
* @param tick_ns Current heartbeat interval (nanoseconds)
* @param workload_ops Dictionary lookups executed this tick
*/
static void capture_heartbeat_rate_sample(VM *vm, uint64_t tick_ns, uint64_t workload_ops)
{
if (!vm || !vm->heartbeat_rate_samples)
return;
if (vm->heartbeat_rate_sample_count >= HEARTBEAT_RATE_SAMPLE_MAX)
return; /* Buffer full, stop sampling */
HeartbeatRateSample *sample = &vm->heartbeat_rate_samples[vm->heartbeat_rate_sample_count];
sample->tick_number = vm->heartbeat.tick_count;
sample->tick_ns = tick_ns;
sample->workload_ops_this_tick = workload_ops;
sample->timestamp_ns = platform_monotonic_ns();
/* Capture current physics state */
sample->hot_word_count = vm->hot_word_count_at_check;
sample->total_heat = vm->total_heat_at_last_check;
vm->heartbeat_rate_sample_count++;
}
```
---
## Step 4: Log Workload Per Tick
### File: `src/vm.c` - Track lookups per tick
Add to VM struct (in `include/vm.h`):
```c
/** @name Per-Tick Workload Metrics
* @{
*/
uint64_t workload_ops_current_tick; /* Dictionary lookups in current tick */
/** @} */
```
### File: `src/vm.c` - In `vm_heartbeat_run_cycle()`
**Before calling inference engine:**
```c
void vm_heartbeat_run_cycle(VM *vm)
{
if (!vm || !vm->heartbeat.heartbeat_enabled)
return;
vm->heartbeat.tick_count++;
/* Reset per-tick workload counter */
uint64_t lookups_this_tick = vm->workload_ops_current_tick;
vm->workload_ops_current_tick = 0; /* Reset for next tick */
/* ... existing inference logic ... */
}
```
### File: `src/vm.c` - Increment lookup counter
In the main word execution loop (in `vm_interpret()` or inner interpreter):
```c
/* After successful dictionary lookup */
vm->workload_ops_current_tick++;
```
---
## Step 5: Capture in Heartbeat Thread
### File: `src/vm.c` in `heartbeat_thread_main()`
Replace the existing sleep logic:
```c
static void* heartbeat_thread_main(void *arg)
{
VM *vm = (VM*)arg;
if (!vm || !vm->heartbeat.worker)
return NULL;
HeartbeatWorker *worker = vm->heartbeat.worker;
worker->running = 1;
while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled)
{
vm_heartbeat_run_cycle(vm);
/* GET THE CURRENT HEARTBEAT RATE */
uint64_t tick_ns = worker->tick_ns ? worker->tick_ns : HEARTBEAT_TICK_NS;
uint64_t workload_ops = vm->workload_ops_current_tick;
/* ↓ ADD THIS ↓ CAPTURE HEARTBEAT RATE SAMPLE */
capture_heartbeat_rate_sample(vm, tick_ns, workload_ops);
/* ↑ ADD THIS ↑ */
struct timespec req;
req.tv_sec = (time_t)(tick_ns / 1000000000ULL);
req.tv_nsec = (long)(tick_ns % 1000000000ULL);
struct timespec rem;
while (!worker->stop_requested && vm->heartbeat.heartbeat_enabled &&
nanosleep(&req, &rem) == -1 && errno == EINTR)
{
req = rem;
}
}
worker->running = 0;
return NULL;
}
```
---
## Step 6: Extend DoeMetrics
### File: `include/doe_metrics.h`
Add to `DoeMetrics` struct:
```c
/* === HEARTBEAT RATE MODULATION METRICS (NEW) === */
/* Per-run heartbeat rate statistics */
uint64_t total_heartbeat_ticks;
double mean_heartbeat_rate_hz; /* Average frequency (Hz) */
double heartbeat_rate_stddev_hz; /* Variation in Hz */
double heartbeat_rate_cv; /* CV of heartbeat rate */
uint64_t heartbeat_rate_min_ns; /* Slowest tick observed */
uint64_t heartbeat_rate_max_ns; /* Fastest tick observed */
double heartbeat_rate_modulation_range_hz; /* max_hz - min_hz */
/* Load-Heartrate Response */
double load_to_heartrate_correlation; /* Correlation(workload, heartrate) */
int load_response_direction; /* +1 if slower under load, -1 if faster */
double heartrate_adaptation_latency_ticks; /* Ticks until rate responds to load */
double heartrate_settling_time_ticks; /* Ticks to settle after adaptation */
/* Heartrate Convergence */
double heartrate_convergence_time_ms; /* Time to reach stable rate */
int heartrate_converged; /* 1 = converged, 0 = oscillating */
double heartrate_oscillation_amplitude_hz; /* Peak-to-peak variation in final state */
/* Overall Heartrate Quality Score */
double heartrate_responsiveness_score; /* 0-100: how well does it respond? */
double heartrate_stability_score; /* 0-100: how smooth is adaptation? */
```
---
## Step 7: Implement Metrics Extraction
### File: `src/doe_metrics.c`
Add new function:
```c
/**
* @brief Analyze heartbeat rate trajectory and compute metrics
*
* @param vm Pointer to VM with heartbeat_rate_samples
* @param metrics Pointer to DoeMetrics to fill
*/
static void analyze_heartbeat_rate_trajectory(VM *vm, DoeMetrics *metrics)
{
if (!vm || !vm->heartbeat_rate_samples || vm->heartbeat_rate_sample_count == 0) {
metrics->mean_heartbeat_rate_hz = 0;
metrics->load_to_heartrate_correlation = 0;
return;
}
uint64_t count = vm->heartbeat_rate_sample_count;
HeartbeatRateSample *samples = vm->heartbeat_rate_samples;
/* 1. Compute heartbeat rate statistics (tick_ns → Hz) */
double *heartrate_hz = malloc(count * sizeof(double));
double *workload_intensity = malloc(count * sizeof(double));
double min_ns = samples[0].tick_ns;
double max_ns = samples[0].tick_ns;
double sum_hz = 0;
double sum_workload = 0;
for (uint64_t i = 0; i < count; i++) {
double tick_ns = samples[i].tick_ns;
heartrate_hz[i] = 1e9 / tick_ns;
workload_intensity[i] = samples[i].workload_ops_this_tick;
sum_hz += heartrate_hz[i];
sum_workload += workload_intensity[i];
if (tick_ns < min_ns) min_ns = tick_ns;
if (tick_ns > max_ns) max_ns = tick_ns;
}
/* Mean heartbeat rate */
metrics->mean_heartbeat_rate_hz = sum_hz / count;
metrics->heartbeat_rate_min_ns = (uint64_t)min_ns;
metrics->heartbeat_rate_max_ns = (uint64_t)max_ns;
metrics->heartbeat_rate_modulation_range_hz = (1e9 / min_ns) - (1e9 / max_ns);
/* 2. Compute standard deviation */
double variance = 0;
for (uint64_t i = 0; i < count; i++) {
double delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
variance += delta * delta;
}
metrics->heartbeat_rate_stddev_hz = sqrt(variance / count);
metrics->heartbeat_rate_cv = metrics->heartbeat_rate_stddev_hz / metrics->mean_heartbeat_rate_hz;
/* 3. Compute correlation(workload, heartrate) — THE KEY METRIC */
double workload_mean = sum_workload / count;
double cov = 0;
double workload_var = 0;
for (uint64_t i = 0; i < count; i++) {
double workload_delta = workload_intensity[i] - workload_mean;
double heartrate_delta = heartrate_hz[i] - metrics->mean_heartbeat_rate_hz;
cov += workload_delta * heartrate_delta;
workload_var += workload_delta * workload_delta;
}
if (workload_var > 0 && metrics->heartbeat_rate_stddev_hz > 0) {
metrics->load_to_heartrate_correlation = cov / sqrt(workload_var *
(metrics->heartbeat_rate_stddev_hz * metrics->heartbeat_rate_stddev_hz * count));
}
/* 4. Determine if heartrate responds correctly to load */
metrics->load_response_direction = (metrics->load_to_heartrate_correlation > 0.3) ? 1 : -1;
/* 5. Estimate convergence time (when heartrate stabilizes) */
double threshold = 0.15; /* 15% of final value */
metrics->heartrate_converged = 1;
for (uint64_t i = count / 2; i < count; i++) {
double current_cv = 0;
/* Compute CV of last N samples */
for (uint64_t j = i; j < (i + 100 < count ? i + 100 : count); j++) {
double delta = heartrate_hz[j] - metrics->mean_heartbeat_rate_hz;
current_cv += delta * delta;
}
if (sqrt(current_cv) > threshold * metrics->mean_heartbeat_rate_hz) {
metrics->heartrate_converged = 0;
metrics->heartrate_convergence_time_ms = i; /* Approximate */
break;
}
}
/* 6. Compute overall responsiveness score */
/* Higher correlation + faster convergence + lower CV = higher score */
metrics->heartrate_responsiveness_score =
(metrics->load_to_heartrate_correlation + 1.0) * 50 *
(1.0 / (1.0 + metrics->heartbeat_rate_cv));
metrics->heartrate_stability_score =
(100.0 * (1.0 - metrics->heartbeat_rate_cv)) *
(metrics->heartrate_converged ? 1.0 : 0.5);
free(heartrate_hz);
free(workload_intensity);
}
```
### Update `metrics_from_vm()`
Call the new analysis function:
```c
DoeMetrics metrics_from_vm(VM *vm, ...) {
/* ... existing metrics extraction ... */
/* NEW: Analyze heartbeat rate trajectory */
analyze_heartbeat_rate_trajectory(vm, &metrics);
return metrics;
}
```
---
## Step 8: Update CSV Schema
### File: `src/doe_metrics.c` in `metrics_write_csv_header()`
Add new columns:
```c
void metrics_write_csv_header(FILE *out) {
{
printf '[...existing columns...]');
printf('total_heartbeat_ticks,mean_heartbeat_rate_hz,heartbeat_rate_stddev_hz,heartbeat_rate_cv,'
'heartbeat_rate_min_ns,heartbeat_rate_max_ns,heartbeat_rate_modulation_range_hz,'
'load_to_heartrate_correlation,load_response_direction,'
'heartrate_convergence_time_ms,heartrate_converged,'
'heartrate_responsiveness_score,heartrate_stability_score\n');
} > output_csv;
}
```
### File: `src/doe_metrics.c` in `metrics_write_csv_row()`
Add the values:
```c
void metrics_write_csv_row(FILE *out, const DoeMetrics *metrics) {
fprintf(out, "[...existing fields...],"
"%lu,%lf,%lf,%lf,%lu,%lu,%lf,%lf,%d,%lu,%d,%lf,%lf\n",
metrics->total_heartbeat_ticks,
metrics->mean_heartbeat_rate_hz,
metrics->heartbeat_rate_stddev_hz,
metrics->heartbeat_rate_cv,
metrics->heartbeat_rate_min_ns,
metrics->heartbeat_rate_max_ns,
metrics->heartbeat_rate_modulation_range_hz,
metrics->load_to_heartrate_correlation,
metrics->load_response_direction,
metrics->heartrate_convergence_time_ms,
metrics->heartrate_converged,
metrics->heartrate_responsiveness_score,
metrics->heartrate_stability_score);
}
```
---
## Step 9: Updated R Analysis
### File: `scripts/analyze_heartbeat_stability_REVISED.R`
```r
library(tidyverse)
# Load data
doe_data <- read.csv("experiment_results_heartbeat.csv") %>%
mutate(
configuration = as.factor(configuration),
# New metrics available now
load_response = ifelse(load_to_heartrate_correlation > 0.5, "Responsive", "Sluggish")
)
# PRIMARY METRIC: Load-Heartrate Correlation
# This tells us which config best responds to workload changes
ranking <- doe_data %>%
group_by(configuration) %>%
summarize(
mean_correlation = mean(load_to_heartrate_correlation, na.rm = TRUE),
mean_responsiveness = mean(heartrate_responsiveness_score, na.rm = TRUE),
mean_stability = mean(heartrate_stability_score, na.rm = TRUE),
mean_convergence_time = mean(heartrate_convergence_time_ms, na.rm = TRUE),
golden_score = (mean(load_to_heartrate_correlation) * 0.5) +
(mean(heartrate_responsiveness_score) * 0.3) +
(mean(heartrate_stability_score) * 0.2)
) %>%
arrange(-golden_score)
print("=== HEARTBEAT RATE RESPONSIVENESS RANKING ===")
print(ranking)
# Plot 1: Load-Heartrate Correlation (KEY METRIC)
ggplot(doe_data, aes(x = reorder(configuration, load_to_heartrate_correlation, FUN=median),
y = load_to_heartrate_correlation,
fill = configuration)) +
geom_boxplot() +
coord_flip() +
labs(title = "Load-Heartrate Correlation (Higher = Better)",
y = "Correlation(workload, heartrate)")
ggsave("01_load_heartrate_correlation.png")
# Plot 2: Heartrate Responsiveness Score
ggplot(doe_data, aes(x = reorder(configuration, heartrate_responsiveness_score, FUN=median),
y = heartrate_responsiveness_score,
fill = configuration)) +
geom_boxplot() +
coord_flip() +
labs(title = "Heartrate Responsiveness Score (0-100)")
ggsave("02_heartrate_responsiveness.png")
```
---
## Compilation Steps
1. **Update headers:** Add structs and fields to `include/vm.h`, `include/doe_metrics.h`
2. **Implement collection:** Add capture function in `src/vm.c`
3. **Implement analysis:** Add analysis function in `src/doe_metrics.c`
4. **Update CSV:** Modify CSV header/row writers
5. **Compile:** `make clean && make fastest`
6. **Test:** `./build/starforth -c "1 2 + . BYE"`
---
## Verification
After implementation, verify:
✓ Binary compiles without errors
✓ Heartbeat rate samples are captured (check sample_count > 0 after run)
✓ CSV contains new heartrate columns
✓ Correlation metrics are computed
✓ R analysis generates ranking by load-response
---
## Expected Results
When properly implemented, you'll see:
```
=== HEARTBEAT RATE RESPONSIVENESS RANKING ===
configuration mean_correlation mean_responsiveness golden_score
1_0_1_1_1_0 0.82 78.5 0.795
1_0_1_1_1_1 0.78 75.2 0.761
1_1_0_1_1_1 0.65 62.3 0.641
1_0_1_0_1_0 0.45 48.1 0.451
0_1_1_0_1_1 0.12 15.8 0.148
```
**Golden config:** 1_0_1_1_1_0 (strongest load-heartrate coupling)
---
## Timeline
- **Implementation:** 2-3 hours
- **Testing:** 1 hour
- **Re-run Phase 2:** 2-4 hours
- **Analysis:** 30 min
- **Total:** ~6-9 hours
---
## Files Modified
- `include/vm.h` — Add HeartbeatRateSample struct and VM fields
- `include/doe_metrics.h` — Add heartrate metrics to DoeMetrics
- `src/vm.c` — Add sample capture, logging, initialization
- `src/doe_metrics.c` — Add analysis function, CSV updates
- `scripts/analyze_heartbeat_stability_REVISED.R` — Rewritten for load-response focus
---
**This implementation captures the actual heartbeat rate modulation and measures how well each config responds to load.**
This is the correct physics story. 🚀
docs/working/architecture/03-architecture/physics-engine/feedback-loops-analysis.md
-362
View File
@@ -1,362 +0,0 @@
<!-- Moved from docs/03-architecture/physics-engine/feedback-loops-analysis.md to docs/working/architecture/03-architecture/physics-engine/feedback-loops-analysis.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ SUPERSEDED (flagged 2026-10-09).** Precursor bug-hunt audit covering six feedback loops;
> the canonical, current reference is `feedback-loops.md` (all seven loops, tuning knobs, and
> source files). Kept for historical record.
# StarForth Feedback Loops & Self-Optimization Analysis
## Executive Summary
StarForth implements **6 feedback loops** for automatic self-tuning during execution. Of these:
- **3 fully wired AND operational** ✅ (actively driving optimization decisions)
1. ✅ Hot-words cache promotion (1.78× speedup measured)
2. ✅ Rolling window adaptive shrinking (FIXED 2025-11-08, enforcement now works)
4. ✅ Pipelining transition speculation (WIRED 2025-11-08, speculative prefetch active)
- **1 wired AND utilized, but NOT validated** ⚠️ (running, but unsupported design choices)
3. ⚠️ Heat decay of execution history (arbitrary linear function, no tuning knob, unvalidated)
- **1 wired but NOT utilized** ⏳ (Phase 2 experimental, infrastructure ready)
5. Context-aware window tuning (binary chop algorithm stub, needs VM-level metrics aggregation)
- **1 observability-only** (metrics collected for formal verification)
6. Physics metadata temperature tracking
**CURRENT STATUS:**
- A_BASELINE, A_B_CACHE, A_C_FULL, A_B_C_FULL configurations now test complete optimization stack:
- Loop #1 (cache) enabled via A_B_CACHE flag
- Loop #2 (window shrinking) always enabled, active in all configs
- Loop #3 (decay) always enabled, active in all configs
- Loop #4 (speculation) disabled by default (ENABLE_PIPELINING=0), ready for A_C_FULL testing
- **BUT:** Loop #3 (heat decay) lacks principled design—slope is arbitrary, function choice unvalidated
- **Gap:** No measurements showing decay actually helps; DECAY_RATE has NO DoE knob to tune it
---
## FULLY WIRED & UTILIZED FEEDBACK LOOPS ✅
### ~~1. Hot-Words Cache Promotion Feedback Loop~~ ✅ DONE
**What's Measured:**
- `execution_heat` counter on each dictionary entry (incremented per execution)
**Trigger Condition:**
- When `execution_heat > HOTWORDS_EXECUTION_HEAT_THRESHOLD` (default: 50)
**Automatic Action:**
- `hotwords_cache_promote()` moves word to 32-entry LRU cache
- Subsequent lookups hit cache (1-2 ns vs 20-30 ns in bucket)
**Code Locations:**
- Metric tracking: `src/dictionary_management.c:170`
- Threshold check: `src/physics_hotwords_cache.c:155`
- Promotion logic: `src/physics_hotwords_cache.c:420-450`
**Observed Impact:**
- 1.78× speedup on cache-hit paths (measured in DoE phase 1)
- **Status: FULLY OPERATIONAL** ✅
---
### ~~2. Rolling Window Adaptive Shrinking Feedback Loop~~ ✅ FIXED
**What's Measured:**
- Pattern diversity: count of unique word transitions in execution history
**Measurement Function:**
```c
// src/rolling_window_of_truth.c:546-577 (FIXED)
uint64_t rolling_window_measure_diversity(const RollingWindowOfTruth* window)
{
// Count unique adjacent transitions (word_a → word_b)
// NOW RESPECTS effective_window_size - only scans recent entries
}
```
**Trigger Condition:**
- When pattern diversity growth rate < `ADAPTIVE_GROWTH_THRESHOLD` (default: 1%)
- Checked every `ADAPTIVE_CHECK_FREQUENCY` executions (default: 256)
**Automatic Action:**
- Shrink `effective_window_size` to `ADAPTIVE_SHRINK_RATE` percent of current (default: 75%)
- Window shrinks: 4096 → 3072 → 2304 → 1728 → ... → `ADAPTIVE_MIN_WINDOW_SIZE` (256)
**Code Locations:**
- Check function: `src/rolling_window_of_truth.c:589-645`
- Shrink calculation: Line 626
- Knobs: `include/rolling_window_knobs.h`
**Integration:**
- Called from `rolling_window_record_execution()` on every word execution
- **Status: FULLY OPERATIONAL** ✅
**Fix Applied (2025-11-08):**
- Line 559: Now limits scan to `effective_window_size` when window is warm
- Lines 561-564: Circular indexing ensures only recent entries within effective window are scanned
- **Effect:** New data arrives every tick, old data falls off after `effective_window_size` ticks (was: always 4096)
---
### 3. Linear Decay of Execution Heat Feedback Loop
**What's Measured:**
- `execution_heat` value per word (time-sensitive metric)
**Decay Formula (INTENTIONALLY BASELINE):**
- `H(t) = max(0, H_0 - decay_rate × Δt)` — **LINEAR decay**
- Configurable decay rate: `DECAY_RATE_PER_US_Q16` (default: 1 = 1/65536 heat/µs)
- Derived half-life: ~6-7 seconds for 100-heat word
**Automatic Action:**
- Stale words automatically lose heat weight
- Only frequently-executed words stay above promotion threshold
**Code Locations:**
- Decay logic: `src/physics_metadata.c:164-203` (`physics_metadata_apply_linear_decay()`)
- Integration: `src/vm.c:524` (called before each execution in inner loop)
**Design Rationale (From Phase 2 Specification):**
- Chose LINEAR over exponential for:
1. Performance (integer arithmetic, no floating-point)
2. Predictability (bounded convergence time)
3. Simplicity (fewer lines, easier to verify formally)
4. StarshipOS applicability (models "task context switch forces heat reset")
- **Explicitly noted as baseline:** "Can always switch to halflife after validating linear approach"
**STATUS: WIRED & UTILIZED, BUT NOT VALIDATED** ⚠️
**CRITICAL GAPS:**
1. **No empirical validation**: Does linear decay actually help? No measurements.
2. **Arbitrary initial slope**: Why 1 µs⁻¹? No evidence this is optimal.
3. **Wrong function?**: Could exponential, log, parabolic, or sinusoidal be better?
4. **Tuning blind**: DECAY_RATE_PER_US_Q16 has NO KNOB in DoE configurations
5. **No feedback loop closure**: Decay happens, but no measurement of whether it improved anything
---
## ✅ IMPLEMENTATION FIXES APPLIED
### Rolling Window Shrinking - FIXED
**The Bug (Now Fixed):**
The adaptive shrinking mechanism was shrinking the reported metric without enforcing the limit.
**Old Code (Line 557):**
```c
for (uint32_t i = 0; i < ROLLING_WINDOW_SIZE; i++) // ← ALWAYS full size (4096)
{
uint32_t current = window->execution_history[i];
// Count unique transitions...
}
```
**Fixed Code (Lines 559-574):**
```c
uint32_t scan_limit = (window->is_warm) ? window->effective_window_size : ROLLING_WINDOW_SIZE;
for (uint32_t i = 0; i < scan_limit; i++)
{
/* Index into the recent window (window_pos is the next write position) */
uint32_t idx = (window->window_pos + ROLLING_WINDOW_SIZE - scan_limit + i) % ROLLING_WINDOW_SIZE;
uint32_t current = window->execution_history[idx];
// Count unique transitions in recent entries only
}
```
**What Changed:**
- Now limits scan to `effective_window_size` when window is warm
- Uses circular indexing to ensure only RECENT entries are scanned
- New data arrives every tick, old data falls off after `effective_window_size` ticks
**User's Requirement Met:**
> "The window should advance with every tick so new data comes in and stale data falls off."
✅ Now data falls off after `effective_window_size` ticks, not always 4096
---
## ~~4. Pipelining Transition Metrics~~ ✅ NOW FUNCTIONAL
**What's Measured:**
- Transition patterns: (word_A → word_B) frequencies via `transition_metrics_record()`
- Probability calculations: P(B | A) in Q48.16 fixed-point format
- Prefetch accuracy: Hits vs. misses from speculative cache promotions
**Trigger Conditions:**
- `SPECULATION_THRESHOLD_Q48` (50% minimum confidence)
- `MIN_SAMPLES_FOR_SPECULATION` (10 minimum observations)
**Automatic Action (NOW WIRED):**
- After recording each transition, update probability cache
- Check: Does next word probability exceed threshold?
- If yes: Find dictionary entry and pre-promote to hotwords cache
- Effect: When next word is actually looked up, it's cache-warm
- Record: Track prefetch_attempts for feedback measurement
**Code Locations:**
- Metric collection: `src/physics_pipelining_metrics.c:92-109` (`transition_metrics_record()`)
- Probability calculation: `src/physics_pipelining_metrics.c:112-123`
- Decision logic: `src/physics_pipelining_metrics.c:164-186` (`transition_metrics_should_speculate()`)
- **Action wiring: `src/vm.c:544-583`** (execute_colon_word, speculative prefetch logic)
**Status: FULLY WIRED & FUNCTIONAL** ✅
**Wiring Details (2025-11-08):**
- Line 548: `transition_metrics_record()` - record the transition
- Line 551: `transition_metrics_update_cache()` - find most likely next word
- Line 554-555: `transition_metrics_should_speculate()` - check confidence threshold
- Lines 560-581: If confident, look up predicted word and promote to hotwords cache
- Line 580: Record the prefetch attempt for metrics feedback
**Loop Closure:**
1. ✅ Metrics collected: transition_heat array counts (word_B | word_A)
2. ✅ Probability calculated: Q48.16 format
3. ✅ Decision made: Confidence check against threshold
4. ✅ Action taken: Speculative promotion to hotwords cache
5. ✅ Feedback recorded: prefetch_attempts counter
### 5. Context-Aware Window Size Tuning (Phase 2 Placeholder)
**What's Measured:**
- Multi-word context patterns (sequences of 2-4 consecutive words)
**Intended Action (Not Connected):**
- Use binary search to find optimal `effective_window_size` for given workload
- Function: `transition_metrics_binary_chop_suggest_window()` (lines 340-357)
**Why Not Utilized:**
- Phase 2 research feature
- Experimental algorithm not yet validated
- No integration point in main execution path
**Status: PROTOTYPE, NOT INTEGRATED** ⏳
---
## OBSERVABILITY-ONLY (Not Decision-Driving)
### 6. Physics Metadata Temperature Tracking
**What's Measured:**
- `temperature_q8`: Exponential moving average of execution_heat
- Used for observability and formal verification model tracing
**Current Use:**
- Profiling and debugging (not driving optimization decisions)
- Forms foundation for Isabelle/HOL physics state machine proofs
**Status: WORKING AS INTENDED** (observability, not optimization)
---
## Bootstrap Seeding Mechanisms
These run once at startup to initialize feedback loop state:
### POST (Power-On Self-Test) Seeding
- **Function**: `rolling_window_seed_hotwords_cache()` (lines 258-329)
- **Action**: Records POST execution, promotes high-heat words to cache
- **Effect**: Cache is "warm" when user workload begins
### Pipelining Context Seeding
- **Function**: `rolling_window_seed_pipelining_context()` (lines 341-415)
- **Action**: Records POST sequences, establishes transition baseline
- **Effect**: Pipelining has initial pattern knowledge
---
## Configuration Knobs for Feedback Loops
All knobs are **tunable via Makefile**:
```bash
# Hot-words cache promotion
make HOTWORDS_EXECUTION_HEAT_THRESHOLD=75
# Adaptive window shrinking
make ADAPTIVE_SHRINK_RATE=75 # Keep 75%, shrink by 25%
make ADAPTIVE_MIN_WINDOW_SIZE=256 # Never shrink below 256
make ADAPTIVE_CHECK_FREQUENCY=256 # Check every 256 executions
make ADAPTIVE_GROWTH_THRESHOLD=1 # Shrink when growth < 1%
# Heat decay
make DECAY_RATE_PER_US_Q16=16384 # Decay rate (units: heat/μs)
make DECAY_MIN_INTERVAL=1000000 # Min interval between decays (ns)
# Pipelining (Phase 2, disabled by default)
make ENABLE_PIPELINING=1
make SPECULATION_THRESHOLD_Q48=$((50 << 16)) # 50% confidence
make MIN_SAMPLES_FOR_SPECULATION=10
```
---
## Summary Table: All Feedback Loops
| Loop | Measured | Threshold | Action | Wired? | Utilized? | Status |
|------|----------|-----------|--------|--------|-----------|--------|
| ~~1. Hot-words promotion~~ | `execution_heat` | > 50 | Add to cache | ✅ | ✅ | ✅ VALIDATED |
| ~~2. Adaptive shrinking~~ | `pattern_diversity` | growth < 1% | Shrink window | ✅ | ✅ | ✅ FIXED |
| 3. Heat decay | Time since last execution | ~6-7s half-life | Reduce heat | ✅ | ✅ | ⚠️ NOT VALIDATED |
| 4. Pipelining transitions | `transition_heat[]` | > 50% confidence | Speculative prefetch | ✅ | ❌ | ⏳ Phase 2 |
| 5. Context window tuning | Multi-word sequences | N/A | Binary chop | ✅ | ❌ | ⏳ Phase 2 |
| 6. Temperature tracking | `temperature_q8` | N/A | Observability only | ✅ | ✅ | ✅ OBSERVABILITY |
---
## Completed Actions ✅
### ✅ DONE: Fix Rolling Window Shrinking Implementation
**Status: APPLIED (2025-11-08)**
Modified `src/rolling_window_of_truth.c:546-577` (function `rolling_window_measure_diversity()`):
- Now respects `effective_window_size` when window is warm
- Uses circular indexing to scan only recent entries
- New data arrives every tick, old data falls off after `effective_window_size` ticks (not always 4096)
- A_B_C_FULL configuration now tests both hot-words cache + adaptive window shrinking together
---
## Next Steps for Phase 2 (If Proceeding)
### OPTIONAL 1: Connect Pipelining Decision Logic
**When `ENABLE_PIPELINING=1`:**
- Call `transition_metrics_should_speculate()` from vm inner loop
- If returns true: Pre-load next word from context prediction
- Measure cache miss reduction vs. A_BASELINE (Phase 2 experimental validation)
**Status:** Metrics collection ready, decision logic not wired
### OPTIONAL 2: Validate Context Window Tuning (Phase 2)
**For adaptive window size optimization:**
- Implement binary chop search algorithm in main execution path
- Tune `effective_window_size` dynamically based on context complexity
- Measure pattern capture efficiency vs. memory overhead
**Status:** Binary chop function exists, integration point not defined
---
## Files & Line Numbers Reference
| Purpose | File | Lines |
|---------|------|-------|
| Rolling window structure | `include/vm.h` | 58-70 |
| Rolling window implementation | `src/rolling_window_of_truth.c` | - |
| Adaptive shrinking | `src/rolling_window_of_truth.c` | 583-639 |
| **Bug location** | `src/rolling_window_of_truth.c` | **557** |
| Diversity measurement | `src/rolling_window_of_truth.c` | 546-571 |
| Shrinking knobs | `include/rolling_window_knobs.h` | - |
| Hot-words cache | `src/physics_hotwords_cache.c` | - |
| Promotion threshold | `src/physics_hotwords_cache.c` | 155 |
| Heat decay | `src/physics_metadata.c` | 164-203 |
| Pipelining metrics | `src/physics_pipelining_metrics.c` | - |
| Pipelining decision | `src/physics_pipelining_metrics.c` | 164-186 |
| Main integration | `src/vm.c` | 524, 542-548 |
---
**Analysis Complete:** 2025-11-08
**User Request:** Investigate feedback loops, identify implementation problems
**Finding:** 3 active + 2 dormant + 1 observability feedback loops; 1 critical bug in rolling window shrinking
docs/working/architecture/03-architecture/physics-engine/metrics-and-knobs.md
-145
View File
@@ -1,145 +0,0 @@
<!-- Moved from docs/03-architecture/physics-engine/metrics-and-knobs.md to docs/working/architecture/03-architecture/physics-engine/metrics-and-knobs.md on 2026-06-16 (docs reorg Phase 2) -->
# StarForth Metrics & Knobs: GAPS ONLY
## What's NOT Wired (Things to Add)
### DEAD CODE / NEVER CALLED
| What | Location | Status | Comment |
|------|----------|--------|---------|
| Bucket reordering | `physics_hotwords_cache.c:201-215` | Defined but NEVER CALLED | Bubble sort exists but no automatic trigger. Is this dead code or forgotten? |
---
### STACK METRICS (ZERO COLLECTION)
| Metric | File | Missing Since | Impact | Knob Needed |
|--------|------|---|---|---|
| Data stack peak depth | `stack_management.c` | Day 1 | Can't tell if workload needs larger stack | STACK_SIZE tuning blind |
| Return stack peak depth | `stack_management.c` | Day 1 | Can't measure colon definition nesting | Tail-call optimization impossible |
| Underflow attempts | `arithmetic_words.c:59-62` | Day 1 | Can't detect fragile code paths | Could optimize checks if zero underflows observed |
| Overflow attempts | `arithmetic_words.c` | Day 1 | Can't detect stack exhaustion | Early warning system missing |
---
### DICTIONARY METRICS (ZERO VISIBILITY)
| Metric | File | Missing Since | Impact | Why Matters |
|--------|------|---|---|---|
| Bucket load histogram | `vm.c: dict lookup` | Day 1 | First-char bucketing (26 buckets) may be unbalanced | Affects lookup latency, collision clustering |
| Dictionary growth rate | `memory_management.c` | Day 1 | No visibility into `here` pointer velocity | Can't predict "dictionary full" events |
| FORGET usage frequency | `dictionary_words.c` | Day 1 | No trace of word deletion patterns | Fragmentation unknown |
| Lookup failure rate | `vm.c` | Day 1 | How often is a word not found? | Error path frequency unknown |
| Word name length distribution | `vm.c` | Day 1 | Average name length unknown | Affects memory efficiency |
---
### BLOCK I/O METRICS (COMPLETELY DARK)
| Metric | File | Missing Since | Impact | Phase |
|--------|------|---|---|---|
| Block read frequency | `block_subsystem.c` | Day 1 | How often is BLOCK called? | DoE phase 1 |
| Block write frequency | `block_subsystem.c` | Day 1 | How often is UPDATE/FLUSH called? | DoE phase 1 |
| Writeback frequency | `block_subsystem.c:198-200` | Day 1 | When are dirty blocks flushed? | DoE phase 1 |
| Cache hit rate (disk) | `block_subsystem.c` | Day 1 | Blocks served from RAM cache % | DoE phase 1 |
| Dirty block ratio | `block_subsystem.c` | Day 1 | Blocks pending flush at time T | DoE phase 1 |
| I/O latency | `block_subsystem.c` | Day 1 | Time per read/write operation | DoE phase 2 |
| Fragmentation ratio | `block_subsystem.c` | Day 1 | Gaps in allocated blocks | DoE phase 2 |
---
### PER-WORD EXECUTION METRICS
| Metric | File | Missing Since | Impact | Use Case |
|--------|------|---|---|---|
| Per-word execution frequency | `vm.c: inner loop` | Day 1 | Which words are actually hot? | Profiling, hotspot detection |
| Per-word latency distribution | `vm.c` | Day 1 | Median/P99 time per word | Performance regression detection |
| Word category breakdown | `word_source/*.c` | Day 1 | Arithmetic vs. Stack vs. Control dominance | Workload characterization |
| Colon vs. native ratio | `vm.c` | Day 1 | User-defined word execution frequency | Code complexity analysis |
---
### HEAT DYNAMICS (PHASE 2 NOT ACTIVE)
| Metric | File | Status | Comment |
|--------|------|--------|---------|
| Heat decay trace | `physics_metadata.c: TODO` | Phase 2 placeholder | Need to measure if decay causes pattern loss |
| Heat percentile distribution | `physics_metadata.c` | Not updating dynamic | Stored but never dynamically recalculated |
| Heat concentration ratio | `physics_metadata.c` | Not tracked | (Top 10 words heat) / (total heat) |
| Heat velocity | `physics_metadata.c` | Not tracked | Rate of heat change per unit time |
| Heat plateau detection | `physics_metadata.c` | Not tracked | When does word heat level off? |
---
### PIPELINING METRICS (PHASE 1 INSTRUMENTATION, NOT USED)
| Metric | File | Status | Phase 2 Use |
|--------|------|--------|---|
| Prediction accuracy by depth | `physics_pipelining_metrics.c` | Not measured | Binary chop TRANSITION_WINDOW_SIZE |
| Context transition frequency | `physics_pipelining_metrics.c` | Counted but not reported | Validate context patterns |
| Speculation ROI histogram | `physics_pipelining_metrics.c` | Not aggregated | Tune SPECULATION_THRESHOLD |
| Misprediction cost (actual) | `physics_pipelining_metrics.c` | Measured but not validated | Calibrate cost model |
| Pipeline stall reduction | `physics_pipelining_metrics.c` | Not measured | Throughput impact unknown |
---
### CACHE POLLUTION & CYCLES
| Metric | File | Missing Since | Impact |
|--------|------|---|---|
| Cache eviction/re-promotion cycles | `physics_hotwords_cache.c` | Day 1 | Are we thrashing? |
| False negatives (words evicted before promotion) | `physics_hotwords_cache.c` | Day 1 | Should promotion threshold be lower? |
| Cache eviction candidate value | `physics_hotwords_cache.c` | Day 1 | Was LRU the right choice? |
| Bucket scan depth (position of hit) | `physics_hotwords_cache.c` | Day 1 | Is bucket order actually helping? |
| Promotion wait time | `physics_hotwords_cache.c` | Day 1 | How long does word stay in bucket before cache? |
---
## Priority for "Twisting the Dragon's Tail"
### 🔴 **CRITICAL (Do First)**
1. **Stack peak depth** - 5 minutes to add, massive insight
2. **Block I/O metrics** - Writeback behavior unknown (could be performance killer)
3. **Dictionary bucket load** - Unbalanced bucketing = latency tail
### 🟠 **HIGH (Phase 1 DoE)**
4. **Per-word execution frequency** - Validation for cache/window decisions
5. **Heat decay trace** - Validate Phase 2 feature before enabling
6. **Block I/O hit rate** - Complete the I/O picture
### 🟡 **MEDIUM (Phase 2 DoE)**
7. **Pipelining accuracy by depth** - Input to window size binary chop
8. **Colon definition nesting depth** - Tail-call optimization opportunities
9. **Dictionary growth velocity** - Predict "full" events
### 🟢 **NICE-TO-HAVE (Phase 3)**
10. Heat concentration ratio
11. Word category breakdown
12. Speculation ROI histogram
---
## Implementation Effort Estimate
| Category | Files | LOC | Time | Priority |
|----------|-------|-----|------|----------|
| Stack metrics | stack_management.c, vm.h | ~30 | 30 min | 🔴 |
| Dictionary metrics | vm.c, memory_management.c, vm.h | ~50 | 45 min | 🔴 |
| Block I/O metrics | block_subsystem.c, vm.h | ~80 | 1 hr | 🔴 |
| Per-word metrics | vm.c, physics_metadata.c, vm.h | ~100 | 1.5 hr | 🟠 |
| Heat dynamics | physics_metadata.c, vm.h | ~60 | 1 hr | 🟠 |
| Pipelining accuracy | physics_pipelining_metrics.c, vm.h | ~40 | 45 min | 🟡 |
---
## Bucket Reordering Mystery
**File**: `physics_hotwords_cache.c:201-215`
**Code**: `hotwords_bucket_reorder()` - bubble sort by execution_heat
**Status**: Function defined but **NEVER CALLED** from lookup path
**Question**: Is this dead code or forgot to hook it?
**If enabled, metrics needed:**
- Reorder frequency
- Cost (bubble sort latency)
- Benefit (does reordering actually improve hit latency?)
docs/working/architecture/03-architecture/physics-engine/steady-state-machine.md
-220
View File
@@ -1,220 +0,0 @@
<!-- Moved from docs/03-architecture/physics-engine/steady-state-machine.md to docs/working/architecture/03-architecture/physics-engine/steady-state-machine.md on 2026-06-16 (docs reorg Phase 2) -->
> **⚠ OBSOLETE (flagged 2026-10-09).** Conceptual precursor to the L8 Jacquard mode-selector
> system; it never mentions Jacquard/L8, the attractor buckets, or the six named modes, and
> was never reconciled with the implemented `ssm_jacquard.c` system. The canonical current
> reference for the physics/feedback model is `feedback-loops.md`. Kept for historical record.
# 1. Introduction
A Steady State Machine (SSM) is a novel computational model in which the system’s operational state is defined not by discrete symbolic transitions, but by convergence to an optimal runtime equilibrium.
Unlike traditional models such as Finite State Machines (FSMs), Pushdown Automata, or Turing Machines — where “state” is explicit and symbolic — a Steady State Machine derives its behavior from continuous thermodynamic variables such as:
* execution_heat
* entropy (diversity)
* pipeline readiness
* heat decay
* steady-state slope
* localized temperature gradients
* adaptive strategy selection
An SSM observes itself, adapts, and settles into stable performance basins called steady states.
This makes the Steady State Machine the first computational model where the attractor defines the machine, not the symbol.
# 2. Core Definition
Definition 1 — Steady State Machine
> A Steady State Machine is a computational system defined by:
A set of internal physical metrics
* 𝐻 = execution_heats
* 𝐸 = entropy/diversity measures
* 𝑃 = pipeline activation thresholds
* Θ = heat decay profile
* 𝑊 = sliding execution window
A set of adaptive policies
* lookup strategies
* caching regimes
* bucket reorganizations
* pipeline decisions
* inference-based mode switches
* A state-space composed of attractor basins, not symbolic states
* steady attractor
* quasi-steady attractor
* transient attractor
* chaotic attractor
* A governing dynamic
### 𝑀𝑡+1=𝑓(𝑀𝑡,Δ𝐻,Δ𝑊,Δ𝐸) where 𝑓 pushes the machine toward a stable basin.
The machine is defined by its convergence behavior, not its instruction sequence.
# 3. The Fundamental Insight
In an SSM, performance is the state.
The system is constantly measuring:
* how hot words are
* how often patterns recur
* how pipeline-able a sequence is
* how diverse the window is
* how much the dictionary should reorganize
And as these stabilize, the machine settles into a steady state — a self-maintaining optimal regime.
This regime is preserved until the environment changes (e.g., new workloads, diversity spikes), at which point the SSM moves into a transient or chaotic regime and eventually finds a new equilibrium.
# 4. Phases of a Steady State Machine
## 1. Transient Phase
- System warms up
- Execution patterns are sporadic
- Heat gradients are noisy
- Pipeline inactive or unstable
## 2. Quasi-Steady Phase
- Some patterns emerge
- Lookups begin to bias
- Hot words form local attractors
- Dictionary reorders stabilize
## 3. True Steady State
- Heat surface smooth
- Pipeline active
- Diversity window predictable
- Lookups converge to optimal
- Runtime > baseline VM
- System remains stable unless perturbed
## 4. Chaotic/Disruption Phase
### Triggered by:
- major workload shift
- dictionary mutation
- extreme entropy spike
- cold-cache shock
- SSM falls out of equilibrium temporarily.
## 5. Axioms of Steady State Machines
>Axiom 1 — Convergence
Every sustained workload induces the SSM to converge toward a stable attractor unless external entropy forces divergence.
>Axiom 2 — Locality of Heat
Execution heat reflects both locality and temporal relevance. Locality produces stability.
>Axiom 3 — Adaptive Reordering
Reordering is not symbolic mutation; it is thermodynamic self-optimization.
>Axiom 4 — Stability Maximizes Throughput
The steady state is always faster than the cold state, and usually faster than the naive baseline VM.
>Axiom 5 — Perturbation Response
When disrupted, the SSM seeks a new steady state; it does not thrash indefinitely.
## 6. Relationship to Other Computational Models
>FSM vs SSM
FSM: discrete symbolic states
SSM: emergent attractors in thermal space
>Turing Machine vs SSM
TM: behavior defined by tape symbols and transitions
SSM: behavior defined by adaptation toward equilibrium
>Markov Chain vs SSM
MC: probabilistic transitions between explicit states
SSM: deterministic drift toward stable basins, driven by observation
>Neural Network vs SSM
NN: learns weights
SSM: reorganizes runtime structures, not parameters
## 7. Why This Breaks New Ground
### A Steady State Machine is the first:
- runtime-thermodynamic computing model
- adaptive VM with convergence properties
- load-dependent reordering machine
- self-optimizing dictionary machine
- phase-shifting execution system
- state-machine whose state is emergent, not explicit
- This is a fundamental new category of computation.
## 8. Canonical Example: StarForth
### StarForth implements the first real SSM with:
- execution_heat
- rolling diversity windows
- phase-based lookup strategies
- adaptive dictionary ordering
- pipeline activation
- DOE-calibrated inference loop
- background physics monitor
- It is, quite literally, the reference implementation.
## 9. Formal Summary Statement
>A Steady State Machine (SSM) is a computational system in which the operational state is defined by attractor convergence in thermodynamic runtime space rather than by discrete symbolic transitions. An SSM continuously measures workload characteristics, adjusts internal structures, and stabilizes into performance-optimized steady states. When perturbed, it transitions through transient or chaotic phases before finding a new equilibrium. This model defines a new class of adaptive computational systems with properties distinct from finite state machines, pushdown automata, and Turing Machines.
docs/working/architecture/03-architecture/word-acl/DESIGN.md
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- Moved from docs/03-architecture/word-acl/DESIGN.md to docs/working/architecture/03-architecture/word-acl/DESIGN.md on 2026-06-16 (docs reorg Phase 2) -->
<!-- Moved from docs/reference/word-acl/DESIGN.md to docs/working/architecture/03-architecture/word-acl/DESIGN.md on 2026-06-16 (docs reorg Phase 2) -->
# Word-Level ACL System
**Status:** Implemented through Phase 6 — Phase 7 (LithosAnanke parity) remaining