diff --git a/README.md b/README.md index 11fd7db7..299f45d9 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ -# LithosAnanke v2.0.1 +# LithosAnanke v2.1.0 -**UEFI-bootable FORTH microkernel.** Boots from firmware, initialises memory and interrupts, then runs the StarForth VM as its sole userspace runtime. No libc. No OS. Just stone and necessity. +**UEFI-bootable FORTH microkernel.** Boots from firmware, initialises memory and interrupts, then +runs StarForth as its sole userspace runtime. No libc. No OS. Just stone and necessity. *Lithos* (foundation) + *Ananke* (necessity) — the kernel under StarshipOS. @@ -13,54 +14,52 @@ StarForth v4 moves the engine onto a 32-instruction core derived from Chuck Moor | Product | What it is | |---|---| -| **Hosted StarForth F18** | The v4 engine as a native Linux build, for amd64, arm64 and riscv64, on real hardware. | -| **FPGA StarForth F18** | A 32-bit build of the same engine loaded into the FPGA. It is the gateway to the rest: the foundation everything else is built up from. | -| **StarshipOS** | The bare-metal product: LithosAnanke running the v4 F18 engine. | +| **Hosted StarForth F18** | The v4 engine as a native Linux build, for amd64, arm64 and riscv64. | +| **FPGA StarForth F18** | A 32-bit build of the same engine loaded into the FPGA: the gateway to the rest, the foundation everything else is built up from. | +| **StarshipOS** | The bare-metal product: LithosAnanke running StarForth. | FORTH-79 is recomposed on the F18 engine and stored as a capsule; the StarshipOS-specific -vocabulary gets the same treatment. The source tree will be reorganised around this split. - -v3 remains the reference system until v4 meets its acceptance criteria -(`docs/v4.0.0/JUSTIFICATION.md` §16). Everything below describes v3 as it stands today. +vocabulary gets the same treatment. v3 remains the reference system until v4 meets its +acceptance criteria (`docs/v4.0.0/JUSTIFICATION.md` §16). **Both run on bare metal now:** every +board image carries v3 and v4, and one word in its boot configuration chooses (see below). --- -## Status — M7.1 (Capsule System · Multi-VM Fleet) +## Status | Milestone | | Status | |---|---|---| -| M0–M6 | UEFI boot · PMM · VMM · IDT · APIC · heap · framebuffer VT100 console (v1.5.1-FINAL) | ✅ Complete | +| M0–M6 | UEFI boot · PMM · VMM · IDT · APIC · heap · framebuffer VT100 console | ✅ Complete | | M7 | StarForth VM integration + parity validation | ✅ Complete | -| **M7.1** | **Capsule birth protocol · Mama FORTH vocabulary · Tripod multi-VM fleet (Hermes/Artemis) · Word-level ACL (Phases 1–7)** | 🔄 **In Progress** | -| M8 | REPL — keyboard input, interactive Forth | Planned | -| M9 | Block storage — AHCI driver | Planned | +| M7.1 | Capsule birth · Mama FORTH vocabulary · the Tripod fleet · kernel-Hermes · word-level ACL | ✅ Live | +| M8 | Keyboard input to the REPL — PS/2, virtio and **USB** keyboards (a full keyboard: Caps/Num Lock, keypad) | ✅ In QEMU, all three ISAs | +| M9 | Block storage — Artemis's disk on virtio-blk; identity thumbdrives on USB mass storage | ✅ In QEMU · real disks: no driver yet (`boards/BRINGUP.md` A4) | +| Bare metal | SER5 (amd64), Raspberry Pi 5 (aarch64), Milk-V Mars (riscv64); Zynq-7020 (ARMv7) not yet ported | Images build; first boots on the bench — `boards/BRINGUP.md` | -POST at boot: **parity hash verified across amd64/aarch64/riscv64 · Mama capsule dictionary: 453 words** +POST at boot, on all three ISAs: **1052 tests**, parity hash verified — +`PARITY:M7.1a word_count=526 hash=0x82f4ae7cb2e2d424`. The v4 system's POST: 538 tests. -### What's live in M7.1 +### What's live -- **Tripod** — a named multi-VM fleet (Hera the Mama VM, Artemis, two Hermes - instances) births, runs, and re-births independently, verified booting - live pre-REPL on all three architectures. -- **Hermes** — a 17-block inter-VM messaging/channel layer between fleet - members, with async delivery and channel negotiation. -- **Artemis** — a Block Allocation Map (BAM) storage subsystem with Q48.16 - block-heat tracking and cooldown/reclamation (`ART-COOL`/`ART-REAP`). -- **Word-level ACL** — every dictionary entry carries a TTL/allow/mode/pin - access-control record. Strict, TTL, and pinned modes; two console layers - (emergency `ok>` and superuser `zuse)ok>`). Phases 1–7 complete (C - infrastructure, FORTH policy layer, `zuse` bootstrap superuser, Isabelle - proof stubs, kernel parity); Phase 8 (Ed25519 PKI / thumbdrive - challenge-response) is the only item remaining. Measured overhead once - active on every check: **+0.0054%–+0.0088%**, CV = **0.000%**, across a - 3×3 Latin-square DoE campaign (architecture × seed × 30 replicates) — - three orders of magnitude below the measurement floor. -- **VM Fleet Attractor physics** — the L8 Jacquard mode selector now has a - real per-VM heat channel into fleet-wide tuning, replacing hardcoded - compudynamics constants with a dynamically-inferred rate. -- **Kconfig build configuration** — every physics/heartbeat/pipelining/ - kernel-only tuning knob (~40 total) is now a discoverable, optional - Kconfig symbol shared with the hosted VM build. See **Quick Start** below. +- **The Tripod** — a named multi-VM fleet: **Hera** (the Mama VM, the governor), **Artemis** + (block storage: a Block Allocation Map with Q48.16 block heat, cooldown and reclamation) and + **Hestia**, birthed, run and re-birthed independently on all three architectures. +- **kernel-Hermes** — messaging moved into the kernel (the Hermes VM is gone, + `docs/fabric/FABRIC-3.5.md`): every VM takes its own messages on its own turn, under a + cooperative switcher that every VM takes part in. +- **Identities on thumbdrives** — Zuse, the superuser, and every user is a minted USB + thumbdrive (Ed25519, Zuse-signed); plugging one in births that user's VMs, pulling it out + ends them. There is only the drive: no password, no PIN. +- **Word-level ACL** — every dictionary entry carries a TTL/allow/mode/pin access-control + record; strict, TTL and pinned modes; two console layers (emergency `ok>` and superuser + `zuse)ok>`). Measured overhead of the live ACL-TTL mechanism: **+0.0603%**, CV 0.000%, + architecture-independent (`docs/fabric/FABRIC-1.md` §T). +- **Logs that persist** — every VM's log records go to a ring on Artemis's disk (`LOG-FLUSH`, + `LOG-SHOW`). +- **The physics-driven runtime** — seven feedback loops and the L8 Jacquard mode selector, with + a per-VM heat channel into fleet-wide tuning (VM Fleet Attractor). +- **Kconfig build configuration** — about 40 physics, heartbeat, pipelining and kernel-only + knobs, shared with the hosted build. --- @@ -71,45 +70,41 @@ POST at boot: **parity hash verified across amd64/aarch64/riscv64 · Mama capsul make boot_image TARGET=SER5 # Beelink SER5 -- GPT/ESP, BOOTX64.EFI make boot_image TARGET=RASPI # Raspberry Pi 5 -- FAT, config.txt + kernel_2712.img + DTB make boot_image TARGET=MILKV # Milk-V Mars -- GPT/ESP, BOOTRISCV64.EFI (U-Boot bootefi) -make boot_image TARGET=ZYNQ7020 # Zynq-7020 -- ARMv7 port in progress -make boards # list boards +make boards # list boards (the Zynq-7020 has no ARMv7 port yet) + +# Each image carries v3 and v4. As built it boots v3; to boot v4, change the one word +# system=v3 -> system=v4 +# in starforth.cfg on the SER5's stick or the Mars's card, or cmdline.txt on the Pi 5's card. # The same TARGET drives all, docs and clean (make help) make # every board with a kernel port + hosted v3 + the book -make TARGET=RASPI # one board's image + build/docs/LithosAnanke-raspi.pdf make docs # build/docs/LithosAnanke.pdf (LaTeX master: docs/book/main.tex) make clean TARGET=RASPI # that board's outputs and build/aarch64/kernel -make clean # all of build/ except build/cache/ -# Build kernel (requires cross-compilation toolchain, or native gcc) -make -f kernel/Makefile ARCH=amd64 +# The kernel in QEMU (OVMF / AAVMF / EDK2), one architecture at a time +make -f kernel/Makefile ARCH=amd64 clean qemu +make -f kernel/Makefile ARCH=aarch64 clean qemu +make -f kernel/Makefile ARCH=riscv64 clean qemu +make -f kernel/Makefile ARCH=amd64 STARFORTH_V4=1 clean qemu # the v4 system -# Run in QEMU with OVMF -make -f kernel/Makefile qemu - -# Other architectures -make -f kernel/Makefile ARCH=aarch64 qemu -make -f kernel/Makefile ARCH=riscv64 qemu +# The Pi 5's native boot path, rehearsed on QEMU's virt machine at EL2 +make -f kernel/Makefile ARCH=aarch64 RPI5_NATIVE_BASE=0x40080000 rpi5-rehearsal ``` -Artifacts: `build/amd64/kernel/starkernel_loader.efi` · `build/amd64/kernel/starkernel_kernel.elf` +Artifacts: `build//kernel/starkernel_loader.efi` · `build//kernel/starkernel_kernel.elf` Tree: `kernel/` (LithosAnanke: `Makefile`, `src/`, `include/starkernel/`, `linker/`), -`v3/` (StarForth v3 engine and hosted build: `Makefile`, `src/`, `include/`), -`v4/` (StarForth v4, see `docs/v4.0.0/`), `boards//` (`board.mk` + boot files). -Every makefile runs from the repo root; the root `Makefile` forwards any other goal to -`v3/Makefile` (hosted build). +`v3/` (StarForth v3 engine and hosted build), `v4/` (StarForth v4, see `docs/v4.0.0/`), +`boards//` (`board.mk` + boot files), `capsules/` (the FORTH capsules). Every makefile +runs from the repo root; the root `Makefile` forwards any other goal to `v3/Makefile`. -For the hosted VM by itself (Linux, no cross-compiler needed, no bare-metal tooling): see -the separate **StarForth** repository — LithosAnanke used to be a branch inside that repo, -now it's its own project with its own `master`. +For the hosted v3 VM by itself, see the separate **StarForth** repository — LithosAnanke used to +be a branch inside it, and is now its own project. ### Build configuration (optional) -Every kernel-only knob (`STARFORTH_ENABLE_VM`, `PARITY_MODE`, the shared -physics/heartbeat family, etc.) is an optional Kconfig symbol — a plain -`make -f kernel/Makefile` uses the same defaults it always has unless -you opt in: +Every kernel-only knob is an optional Kconfig symbol; a plain `make -f kernel/Makefile` uses +the defaults unless you opt in: ```bash make -f kernel/Makefile ARCH=amd64 menuconfig @@ -120,15 +115,21 @@ make -f kernel/Makefile ARCH=amd64 kernel_amd64_defconfig ## Documentation +**Start at [docs/README.md](docs/README.md)** — the map: where each kind of document lives and +which to believe. Among them: + | | | |---|---| -| [System Architecture](docs/working/archive/historical/2026-01-27-system-architecture-draft.md) | Full kernel + VM design | -| [HAL Reference](kernel/src/hal/README.md) | Hardware abstraction layer, as built | -| [Capsule System — M7.1](docs/reference/M7.1.md) | Capsule birth protocol design | -| [VM Fleet Attractor design log](docs/working/architecture/VM-FLEET-ATTRACTOR-DESIGN-20260705.md) | Tripod/Hermes/Artemis physics + build-system history | -| [Getting Started / Kconfig reference](docs/working/architecture/getting-started/DEVELOPER.md#build-configuration-kconfig) | Full symbol reference for both build targets | +| [The FABRIC record](docs/fabric/) | Every design decision and the work as it happened (`FABRIC-3.md` is the living one) | +| [StarForth v4](docs/v4.0.0/) | The F18-derived engine | +| [Board bring-up](boards/BRINGUP.md) | Getting onto the SER5, the Pi 5 and the Mars | +| [Roadmap and versioning](docs/reference/ROADMAP.md) | Milestone plan and release-versioning policy | +| [Capsule System — M7.1](docs/reference/M7.1.md) | Capsule birth protocol | +| [Word-level ACL](docs/reference/word-acl/DESIGN.md) | The access-control design | +| [HAL, as built](kernel/src/hal/README.md) | Hardware abstraction layer | +| [Developer guide / Kconfig reference](docs/working/architecture/getting-started/DEVELOPER.md#build-configuration-kconfig) | Full symbol reference for both builds | +| [Dead ideas and revised claims](docs/DEAD-AND-REVISED.md) | What is no longer true, and since when | | [Changelog](CHANGELOG.md) | Milestone-level history | -| [Roadmap](docs/reference/ROADMAP.md) | Milestone plan through self-hosting | --- diff --git a/docs/DEAD-AND-REVISED.md b/docs/DEAD-AND-REVISED.md index e0fa93c3..59004657 100644 --- a/docs/DEAD-AND-REVISED.md +++ b/docs/DEAD-AND-REVISED.md @@ -43,6 +43,7 @@ document is superseded or a claim corrected. | The old elevation path passed a raw cross-VM address | **it copied text** (the narrower real gap was a `"` in a name) | 2026-09-22 | `fabric/FABRIC-3.7.md` (its own correction) | | `.claude/ARTEMIS.md`, `HERMES.md`, `TRIPOD.md`, `CONSOLE.md` describe the system | **frozen as record**, superseded by FABRIC-3.5/3.6 | 2026-09-22 | `.claude/CLAUDE.md` (top) | -**Open discrepancy, not decided here:** the top-level `README.md` is titled "LithosAnanke v2.0.1", -but the kernel is 2.1.0, and the versioning policy reserves 2.0.1 for the SER5 hardware line -(`reference/ROADMAP.md`). Captain Bob's call which it should say. +**Resolved 2026-10-14:** the top-level `README.md` was titled "LithosAnanke v2.0.1" (the SER5 +hardware line's version) while the kernel is 2.1.0; Captain Bob ruled it rewritten -- it now +describes the system as it is (Tripod Hera/Artemis/Hestia, kernel-Hermes, 1052-test POST, both +systems on every board image). diff --git a/docs/DELETE-CANDIDATES.md b/docs/DELETE-CANDIDATES.md index b8e683c7..185c8371 100644 --- a/docs/DELETE-CANDIDATES.md +++ b/docs/DELETE-CANDIDATES.md @@ -1,7 +1,9 @@ # docs/DELETE-CANDIDATES.md -**Listed:** 2026-10-14 (docs triage). **Status:** awaiting Captain Bob's ruling -- nothing here has -been deleted (docs/CLAUDE.md: never delete without confirmation). +**Listed:** 2026-10-14 (docs triage). **Status: deleted 2026-10-14 on Captain Bob's ruling +("delete").** The list is kept as the record of what was removed and why; nothing is awaiting a +ruling now. The formal tier's `%% SOURCE:` comments that named two of these were repointed at +the maintained copies. Every one is a duplicate: either byte-identical to another file in the same archive, or an older copy of a document whose maintained copy is named in the reason. Deleting them loses nothing. @@ -24,4 +26,3 @@ an AI session note -- are no longer in the tree: they were removed at some point | `docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/run_logs/test_matrix.txt` | Byte-identical to the same file in ../../../full_90_run_comprehensive/run_logs/. | | `docs/working/papers/ONTOLOGY.md` | Older copy (v1.0) of docs/reference/ONTOLOGY.md (v1.1, 2026-08-13). | -To act: "delete the delete candidates", or name the ones to keep. diff --git a/docs/INDEX.md b/docs/INDEX.md index ccbef4cf..0fb4045a 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -2,8 +2,8 @@ Generated 2026-10-14 by the docs triage (docs/CLAUDE.md, Phase 1), from the tracked tree after the reorganisation. Start at **[README.md](README.md)** for the map; abandoned ideas and -revised claims are in **[DEAD-AND-REVISED.md](DEAD-AND-REVISED.md)**; files awaiting a delete -ruling are in **[DELETE-CANDIDATES.md](DELETE-CANDIDATES.md)**. Logs (`logs/`) and DoE run +revised claims are in **[DEAD-AND-REVISED.md](DEAD-AND-REVISED.md)**; the duplicates deleted +2026-10-14 are recorded in **[DELETE-CANDIDATES.md](DELETE-CANDIDATES.md)**. Logs (`logs/`) and DoE run CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by `docs/formal/CLAUDE.md`), are listed one row per folder. @@ -14,7 +14,7 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by | HISTORICAL | Accurate for its time; kept as record | 90 | | SUPERSEDED | Replaced -- its header says by what | 26 | | DEAD | An idea or design that was abandoned or never adopted | 12 | -| DELETE | Proposed for deletion, awaiting a ruling | 12 | +| DELETE | Proposed for deletion, awaiting a ruling | 0 | | RECORD | Experiment data and results | 50 | | FORMAL | The press-ready LaTeX tier | 19 | | LEGAL HOLD | Patent-adjacent; not touched without Captain Bob | 57 | @@ -81,7 +81,7 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by |---|---|---|---| | docs/CLAUDE.md | CLAUDE.md — StarshipOS Documentation Root | CURRENT | Rules for documentation work. | | docs/DEAD-AND-REVISED.md | Dead ideas and revised claims | CURRENT | Register of abandoned ideas and revised claims. | -| docs/DELETE-CANDIDATES.md | docs/DELETE-CANDIDATES.md | CURRENT | Awaiting Captain Bob's ruling. | +| docs/DELETE-CANDIDATES.md | docs/DELETE-CANDIDATES.md | CURRENT | Record of the duplicates deleted 2026-10-14. | | docs/INDEX.md | docs/INDEX.md — every document, and its status | CURRENT | This index. | | docs/README.md | StarForth / LithosAnanke documentation — the map | CURRENT | The map: start here. | | docs/SSM_DARPA_Summary_Slide.pptx | SSM_DARPA_Summary_Slide.pptx | CURRENT | | @@ -133,17 +133,6 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by | docs/formal/vol2-kernel/ | 8 files | FORMAL | Press tier; see docs/formal/CLAUDE.md. | | docs/formal/vol3-research/ | 11 files | FORMAL | Press tier; see docs/formal/CLAUDE.md. | -## docs/lithosananke - -| Path | Title | Status | Notes | -|---|---|---|---| -| docs/lithosananke/hal/README.md | LithosAnanke Hardware Abstraction Layer (HAL) | DELETE | Duplicate of docs/working/archive/dead/hal-target-contract/README.md, without that copy's 2026-10-09 "never adopted" flags. | -| docs/lithosananke/hal/interfaces.md | HAL Interface Specifications | DELETE | Duplicate (older, unflagged) of docs/working/archive/dead/hal-target-contract/interfaces.md. | -| docs/lithosananke/hal/lithosananke-integration.md | LithosAnanke HAL Integration Guide | DELETE | The same document as docs/working/archive/dead/hal-target-contract/starkernel-integration.md under the kernel's newer name, without its "obsolete" flag. | -| docs/lithosananke/hal/migration-plan.md | HAL Migration Plan | DELETE | Duplicate (older, unflagged) of docs/working/archive/dead/hal-target-contract/migration-plan.md. | -| docs/lithosananke/hal/overview.md | Hardware Abstraction Layer (HAL) Architecture | DELETE | Duplicate (older) of docs/working/archive/dead/hal-target-contract/overview.md. | -| docs/lithosananke/hal/platform-implementations.md | HAL Platform Implementation Guide | DELETE | Duplicate (older, unflagged) of docs/working/archive/dead/hal-target-contract/platform-implementations.md. | - ## docs/patent | Path | Title | Status | Notes | @@ -270,7 +259,6 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by | docs/working/architecture/03-architecture/pipelining/design.md | FORTH VM Pipelining & Speculative Execution Design | WORKING | | | docs/working/architecture/03-architecture/pipelining/phase-1-instrumentation.md | Phase 1: Pipelining Instrumentation | WORKING | | | docs/working/architecture/03-architecture/pipelining/wired-not-utilized.md | Pipelining Feedback Loop: WIRED BUT NOT UTILIZED (Critical Issue) | WORKING | | -| docs/working/architecture/03-architecture/word-acl/DESIGN.md | Word-Level ACL System | DELETE | Older copy (2026-06) of docs/reference/word-acl/DESIGN.md, which is the maintained one (status through Phase 7). | | docs/working/architecture/ARTEMIS-BAM-ACCEPTANCE-20260703.md | Artemis BAM — Phase 1 Acceptance Report | WORKING | | | docs/working/architecture/ARTEMIS-BLOCK-PHYSICS-DESIGN-20260708.md | Artemis Block Heat Physics — Design Doc | WORKING | | | docs/working/architecture/ARTEMIS-FREEMAP-CAPSULE-RESERVATION-GAP-20260710.md | Artemis Free-Map Doesn't Reserve Capsule-Claimed Blocks | WORKING | | @@ -352,10 +340,6 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by | docs/working/archive/legacy/phase-1/Reference/codex_capa_list.md | StarForth Governance – CAPA Register (Fresh Audit) | HISTORICAL | | | docs/working/archive/legacy/phase-1/Reference/github_backlog_items.md | GitHub Project Backlog Prep | HISTORICAL | | | docs/working/archive/legacy/phase-1/Reference/physics_experiment/ | 35 files | HISTORICAL | Grouped: bulk data or a self-governed tree. | -| docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/FORMAL_CLAIMS_FOR_REVIEWERS.txt | 0% / 70% | DELETE | Byte-identical to ../../FORMAL_CLAIMS_FOR_REVIEWERS.txt in the same archive. | -| docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/QUICK_REFERENCE.txt | Before mitigations (baseline) | DELETE | Byte-identical to ../../QUICK_REFERENCE.txt in the same archive. | -| docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/experiment_summary.txt | experiment_summary.txt | DELETE | Byte-identical to ../../../full_90_run_comprehensive/experiment_summary.txt in the same archive. | -| docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/run_logs/test_matrix.txt | test_matrix.txt | DELETE | Byte-identical to the same file in ../../../full_90_run_comprehensive/run_logs/. | | docs/working/archive/legacy/phase-1/Reference/toolchain/ | 2 files | HISTORICAL | Grouped: bulk data or a self-governed tree. | | docs/working/archive/operations/README.md | Operations | HISTORICAL | | | docs/working/archive/operations/backlog-gatekeeper.md | Backlog Gatekeeper Procedures | HISTORICAL | | @@ -456,7 +440,6 @@ CSVs are not listed. Bulk data folders, and the formal LaTeX tree (governed by | docs/working/papers/FORMAL_CLAIM_TABLE.md | Formal Claim Table: StarForth Adaptive Runtime | WORKING | | | docs/working/papers/NEGATIVE_RESULTS.md | Negative Results: Where StarForth Fails | WORKING | | | docs/working/papers/NULL_HYPOTHESIS.md | Null Hypothesis and Falsification Criteria | WORKING | | -| docs/working/papers/ONTOLOGY.md | StarForth Ontology, Taxonomy, and Lexicon | DELETE | Older copy (v1.0) of docs/reference/ONTOLOGY.md (v1.1, 2026-08-13). | | docs/working/papers/ONTOLOGY_README.md | StarForth Ontology Documentation | WORKING | | | docs/working/papers/README.md | docs/working/papers/ | WORKING | | | docs/working/papers/REPLICATION_INVITE.md | Invitation to Independent Replication | WORKING | | diff --git a/docs/README.md b/docs/README.md index aad7581a..faaf1557 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,8 +8,8 @@ its own header. - **[INDEX.md](INDEX.md)** — every document in the repository, with its status. - **[DEAD-AND-REVISED.md](DEAD-AND-REVISED.md)** — the ideas that were abandoned, and the claims that were revised, each with the date and where the decision is recorded. -- **[DELETE-CANDIDATES.md](DELETE-CANDIDATES.md)** — duplicates proposed for deletion, awaiting a - ruling. +- **[DELETE-CANDIDATES.md](DELETE-CANDIDATES.md)** — what was deleted as duplicate, and why + (2026-10-14). ## What to believe @@ -60,8 +60,6 @@ Where two documents disagree, the one higher in this table wins, and code wins o - **[pptx/](pptx/)** — slide decks for different audiences. - **[patent/](patent/)** — provisional patent material. **Legal hold: ask Captain Bob before touching anything here** (also `formal/patent/`, `formal/scraps/legal/`, `working/legal/`). -- `lithosananke/hal/` — the last of the old `lithosananke/` folder: duplicates of the dead HAL - contract, listed in DELETE-CANDIDATES.md. ## Adding a document diff --git a/docs/formal/cookbook/main.tex b/docs/formal/cookbook/main.tex index 33df75d7..29bbf914 100644 --- a/docs/formal/cookbook/main.tex +++ b/docs/formal/cookbook/main.tex @@ -100,7 +100,7 @@ \part{Security and ACL Recipes} \chapter{Word-Level ACL Basics} -%% SOURCE: docs/working/architecture/03-architecture/word-acl/DESIGN.md +%% SOURCE: docs/reference/word-acl/DESIGN.md %% TODO(bob): promote ACL recipe: loading ACL.4th, granting/revoking modes, %% TTL-based expiry. Do NOT draft independent claim language. \input{../scraps/architecture/03-architecture/word-acl/DESIGN} diff --git a/docs/formal/scraps/architecture/03-architecture/word-acl/DESIGN.tex b/docs/formal/scraps/architecture/03-architecture/word-acl/DESIGN.tex index 9c5a5aea..31f72972 100644 --- a/docs/formal/scraps/architecture/03-architecture/word-acl/DESIGN.tex +++ b/docs/formal/scraps/architecture/03-architecture/word-acl/DESIGN.tex @@ -1,5 +1,5 @@ %% SCRAP: architecture/03-architecture/word-acl/DESIGN -%% SOURCE: docs/working/architecture/03-architecture/word-acl/DESIGN.md +%% SOURCE: docs/reference/word-acl/DESIGN.md %% STATUS: CURRENT %% FITS: dev-guide/ch-acl, cookbook/ch-acl %% EDITORIAL: lifted — prose rewritten to press voice diff --git a/docs/formal/scraps/papers/ONTOLOGY.tex b/docs/formal/scraps/papers/ONTOLOGY.tex index 2d4581da..8bf11f04 100644 --- a/docs/formal/scraps/papers/ONTOLOGY.tex +++ b/docs/formal/scraps/papers/ONTOLOGY.tex @@ -1,5 +1,5 @@ %% SCRAP: papers/ONTOLOGY -%% SOURCE: docs/working/papers/ONTOLOGY.md +%% SOURCE: docs/reference/ONTOLOGY.md %% STATUS: CURRENT %% FITS: ssrn/ch-ontology, vol3-research/ch-ontology %% EDITORIAL: lifted — prose rewritten to press voice; thermodynamic-as-metaphor framing preserved throughout diff --git a/docs/formal/ssrn/main.tex b/docs/formal/ssrn/main.tex index ea147e98..20cfe545 100644 --- a/docs/formal/ssrn/main.tex +++ b/docs/formal/ssrn/main.tex @@ -2,7 +2,7 @@ %% SOURCE: docs/working/papers/published/James_Steady-State_Convergence_Adaptive_Runtime.pdf %% docs/working/papers/SSRN_companion/Math_Companion_SSRN.tex %% docs/working/papers/FORMAL_CLAIM_TABLE.md -%% docs/working/papers/ONTOLOGY.md +%% docs/reference/ONTOLOGY.md %% NOTE: The SSRN paper itself is a published PDF — it is NOT re-typeset here. %% This volume collects the companion material: Math Companion (full LaTeX), %% formal claim tables, ontology, and academic wording guidelines. @@ -84,7 +84,7 @@ The canonical paper PDF is at: \part{Ontology of Terms} \chapter{Thermodynamic Metaphor Ontology} -%% SOURCE: docs/working/papers/ONTOLOGY.md +%% SOURCE: docs/reference/ONTOLOGY.md %% NOTE: Thermodynamic terms are modeling tools, not physics claims. %% Always introduce with domain qualifier: "using X as a proxy for Y, …" \input{../scraps/papers/ONTOLOGY} diff --git a/docs/formal/vol2-kernel/ch03-acl-security.tex b/docs/formal/vol2-kernel/ch03-acl-security.tex index 06d17a0d..6ebad172 100644 --- a/docs/formal/vol2-kernel/ch03-acl-security.tex +++ b/docs/formal/vol2-kernel/ch03-acl-security.tex @@ -1,5 +1,5 @@ %% Vol II, Chapter 3 — ACL and Security Model -%% SOURCE: docs/working/architecture/03-architecture/word-acl/DESIGN.md +%% SOURCE: docs/reference/word-acl/DESIGN.md %% .claude/CLAUDE.md (Word-Level ACL System section) %% capsules/ACL.4th, capsules/zuse.4th @@ -19,7 +19,7 @@ \section{Two-Console Model} \label{vol2:sec:acl:consoles} -%% SOURCE: docs/working/architecture/03-architecture/word-acl/DESIGN.md +%% SOURCE: docs/reference/word-acl/DESIGN.md %% TODO(bob): describe the two permanent console layers: %% - emergency console (ok>) — fault handler active %% - zuse console (zuse)ok>) — superuser authenticated diff --git a/docs/formal/vol3-research/ch06-ontology.tex b/docs/formal/vol3-research/ch06-ontology.tex index 8d474b20..76887819 100644 --- a/docs/formal/vol3-research/ch06-ontology.tex +++ b/docs/formal/vol3-research/ch06-ontology.tex @@ -1,5 +1,5 @@ %% Vol III, Chapter 6 — Ontology of Thermodynamic Terms -%% SOURCE: docs/working/papers/ONTOLOGY.md +%% SOURCE: docs/reference/ONTOLOGY.md %% docs/working/papers/ONTOLOGY_README.md \chapter{Ontology of Thermodynamic Terms} diff --git a/docs/lithosananke/hal/README.md b/docs/lithosananke/hal/README.md deleted file mode 100644 index f338711e..00000000 --- a/docs/lithosananke/hal/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# LithosAnanke Hardware Abstraction Layer (HAL) - -This directory contains HAL documentation for the LithosAnanke kernel branch. - -> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references throughout this directory's docs describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. - ---- - -## Document Overview - -| Document | Purpose | -|----------|---------| -| [overview.md](overview.md) | HAL architecture and design principles | -| [interfaces.md](interfaces.md) | HAL function specifications and contracts | -| [lithosananke-integration.md](lithosananke-integration.md) | Kernel-specific HAL implementation | -| [platform-implementations.md](platform-implementations.md) | How to implement HAL for a platform | -| [migration-plan.md](migration-plan.md) | Migration from hosted to kernel | - ---- - -## Reading Order - -1. **overview.md** — Understand why HAL exists -2. **interfaces.md** — Learn the contracts -3. **lithosananke-integration.md** — See kernel implementation -4. **platform-implementations.md** — Reference implementations - ---- - -## HAL Subsystems - -``` -hal_time.h — Monotonic time, timers -hal_interrupt.h — IRQ management, ISR registration -hal_memory.h — Allocation, page mapping -hal_console.h — Character I/O (UART, framebuffer) -hal_cpu.h — CPU ID, relax/halt, SMP -hal_panic.h — Fatal error handling -``` - ---- - -## Current State (lithosananke branch) - -The HAL is **partially implemented** in LithosAnanke: - -| Subsystem | Status | Location | -|-----------|--------|----------| -| Time | Implemented | `src/starkernel/arch/amd64/timer.c` | -| Interrupts | Implemented | `src/starkernel/arch/amd64/interrupts.c` | -| Memory | Implemented | `src/starkernel/memory/` | -| Console | Implemented | `src/starkernel/hal/console.c` | -| CPU | Partial | `src/starkernel/arch/amd64/apic.c` | -| Panic | Implemented | `src/starkernel/hal/hal.c` | - ---- - -## Relationship to Hosted StarForth - -The HAL enables the same VM code to run on: - -- **Linux** (hosted, POSIX) -- **LithosAnanke** (freestanding, bare metal) -- **L4Re** (microkernel, optional) - -``` -┌─────────────────────────────────────┐ -│ StarForth VM + Physics Subsystems │ -│ (platform-agnostic) │ -├─────────────────────────────────────┤ -│ HAL Layer │ -├──────────┬──────────┬───────────────┤ -│ Linux │ L4Re │ LithosAnanke │ -│ (POSIX) │ (micro) │ (kernel) │ -└──────────┴──────────┴───────────────┘ -``` - ---- - -## See Also - -- [../README.md](../README.md) — LithosAnanke overview -- [../ROADMAP.md](../ROADMAP.md) — Milestone roadmap -- [../M7.1.md](../M7.1.md) — Init Capsule Architecture diff --git a/docs/lithosananke/hal/interfaces.md b/docs/lithosananke/hal/interfaces.md deleted file mode 100644 index 5cea95d0..00000000 --- a/docs/lithosananke/hal/interfaces.md +++ /dev/null @@ -1,703 +0,0 @@ -# HAL Interface Specifications - -## Overview - -This document defines the **contract** for each HAL subsystem. These interfaces are platform-agnostic and must be implemented identically (semantically) on all platforms. - -> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. - -**Critical:** These are not optional "convenience wrappers"—they are the **only** way VM and physics code access platform resources. - ---- - -## 1. Time & Timers (`include/hal/hal_time.h`) - -### Purpose -Provide monotonic time, periodic/oneshot timers, and calibration for physics subsystems. - -### Interface - -```c -#ifndef HAL_TIME_H -#define HAL_TIME_H - -#include - -/** - * Initialize time subsystem. - * Must be called before any other hal_time_* functions. - * - * Responsibilities: - * - Calibrate high-resolution timer (TSC, HPET, etc.) - * - Validate monotonicity - * - Set up timer interrupt sources (if needed) - * - * Error handling: - * - hal_panic() if no suitable time source available - * - hal_panic() if calibration fails - */ -void hal_time_init(void); - -/** - * Get current monotonic time in nanoseconds. - * - * Guarantees: - * - MUST be monotonic (never goes backward) - * - SHOULD have sub-microsecond resolution - * - MUST be callable from interrupt context (ISR-safe) - * - MUST be callable concurrently (thread-safe) - * - * Performance: - * - Target: < 50 cycles on x86_64 - * - Acceptable: < 200 cycles - * - * Returns: - * - Nanoseconds since arbitrary epoch (typically boot time) - */ -uint64_t hal_time_now_ns(void); - -/** - * Busy-wait delay for specified nanoseconds. - * Used for calibration and short delays. - * - * Guarantees: - * - MUST delay for at least `ns` nanoseconds - * - MAY delay longer (but not by more than 10%) - * - MUST be callable from interrupt context - * - * Parameters: - * - ns: Nanoseconds to delay (0 = no delay) - * - * Notes: - * - This is a busy-wait, not a sleep - * - For delays > 1ms, prefer hal_timer_oneshot() - */ -void hal_time_delay_ns(uint64_t ns); - -/** - * Timer callback function type. - * - * Parameters: - * - ctx: User-provided context pointer - * - * Context: - * - MAY be called from interrupt context (ISR) - * - MUST NOT block or sleep - * - MUST complete quickly (< 10µs target) - */ -typedef void (*hal_timer_callback_t)(void *ctx); - -/** - * Schedule a oneshot timer to fire after specified nanoseconds. - * - * Guarantees: - * - Callback will fire at least `ns` nanoseconds in the future - * - MAY fire up to 10% late (jitter allowed) - * - Callback runs in interrupt context (ISR) - * - * Parameters: - * - ns: Nanoseconds until callback fires (0 = ASAP) - * - callback: Function to call when timer fires - * - ctx: User context passed to callback - * - * Error handling: - * - Returns 0 on success - * - Returns -1 if timer resources exhausted - * - * Concurrency: - * - Thread-safe (can be called from multiple threads) - * - NOT reentrant from same timer callback - */ -int hal_timer_oneshot(uint64_t ns, hal_timer_callback_t callback, void *ctx); - -/** - * Schedule a periodic timer to fire every `period_ns` nanoseconds. - * - * Guarantees: - * - Callback will fire repeatedly at ~period_ns intervals - * - Jitter: < 10% of period (target: < 1%) - * - Callback runs in interrupt context (ISR) - * - Timer continues until hal_timer_cancel() called - * - * Parameters: - * - period_ns: Period in nanoseconds (minimum: 1000ns = 1µs) - * - callback: Function to call on each period - * - ctx: User context passed to callback - * - * Returns: - * - Timer ID (> 0) on success - * - -1 if timer resources exhausted - * - -2 if period_ns < minimum - * - * Notes: - * - This is the PRIMARY mechanism for heartbeat ISR - * - Platform MUST minimize jitter for experimental validity - */ -int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx); - -/** - * Cancel a periodic timer. - * - * Parameters: - * - timer_id: ID returned from hal_timer_periodic() - * - * Guarantees: - * - Callback will not fire after this call returns - * - Safe to call even if timer already fired or was cancelled - * - * Returns: - * - 0 on success - * - -1 if timer_id invalid - */ -int hal_timer_cancel(int timer_id); - -/** - * Get timer frequency in Hz. - * Used for diagnostics and experiment metadata. - * - * Returns: - * - Frequency of underlying time source (e.g., TSC frequency) - * - 0 if frequency is variable or unknown - */ -uint64_t hal_time_frequency_hz(void); - -#endif /* HAL_TIME_H */ -``` - -### Platform Implementation Notes - -**Linux (`platform/linux/hal_time.c`):** -- `hal_time_now_ns()` → `clock_gettime(CLOCK_MONOTONIC)` -- `hal_timer_periodic()` → `timer_create()` with `SIGRTMIN` + signal handler -- Challenge: Signal-based timers have ~1-10µs latency jitter - -**L4Re (`platform/l4re/hal_time.c`):** -- `hal_time_now_ns()` → `l4re_kip_clock()` -- `hal_timer_periodic()` → L4Re IPC-based timer + IRQ object -- Challenge: IPC overhead, but lower jitter than Linux - -**Kernel (`platform/kernel/hal_time.c`):** -- `hal_time_now_ns()` → `rdtsc()` calibrated against HPET -- `hal_timer_periodic()` → APIC timer interrupt -- Challenge: TSC drift on old hardware, calibration complexity - ---- - -## 2. Interrupts (`include/hal/hal_interrupt.h`) - -### Purpose -Enable/disable interrupts, register ISRs, query interrupt context. - -### Interface - -```c -#ifndef HAL_INTERRUPT_H -#define HAL_INTERRUPT_H - -#include - -/** - * Initialize interrupt subsystem. - * Must be called before any other hal_interrupt_* functions. - * - * Responsibilities: - * - Set up interrupt controller (PIC/APIC on x86_64) - * - Initialize IDT (on kernel platforms) - * - Prepare for ISR registration - * - * Error handling: - * - hal_panic() if interrupt controller unavailable - */ -void hal_interrupt_init(void); - -/** - * Enable interrupts globally. - * - * Guarantees: - * - Interrupts will be delivered after this call - * - Safe to call multiple times (idempotent) - * - NOT callable from interrupt context - * - * Notes: - * - On x86_64 kernel: sti - * - On Linux: no-op (interrupts always enabled) - */ -void hal_irq_enable(void); - -/** - * Disable interrupts globally. - * - * Guarantees: - * - Interrupts will NOT be delivered after this call - * - Safe to call multiple times (idempotent) - * - Callable from interrupt context - * - * Returns: - * - Previous interrupt state (for hal_irq_restore) - * - * Notes: - * - On x86_64 kernel: cli - * - On Linux: no-op (interrupts always enabled) - */ -unsigned long hal_irq_disable(void); - -/** - * Restore previous interrupt state. - * - * Parameters: - * - state: Value returned from hal_irq_disable() - * - * Use case: - * unsigned long flags = hal_irq_disable(); - * // critical section - * hal_irq_restore(flags); - */ -void hal_irq_restore(unsigned long state); - -/** - * Interrupt service routine function type. - * - * Parameters: - * - irq: IRQ number that triggered - * - ctx: User-provided context pointer - * - * Context: - * - ALWAYS called from interrupt context - * - MUST NOT block or sleep - * - MUST complete quickly (< 10µs target, < 100µs acceptable) - * - MUST NOT call non-ISR-safe HAL functions - */ -typedef void (*hal_isr_t)(unsigned int irq, void *ctx); - -/** - * Register an interrupt service routine. - * - * Parameters: - * - irq: IRQ number to register (platform-specific numbering) - * - isr: Function to call when IRQ fires - * - ctx: User context passed to ISR - * - * Guarantees: - * - ISR will be called when IRQ fires - * - Only one ISR per IRQ (replaces previous if registered) - * - ISR runs with interrupts disabled (not reentrant) - * - * Returns: - * - 0 on success - * - -1 if IRQ number invalid - * - -2 if ISR registration failed - * - * Notes: - * - Used by hal_timer_periodic() internally - * - Used by device drivers for hardware IRQs - */ -int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx); - -/** - * Unregister an interrupt service routine. - * - * Parameters: - * - irq: IRQ number to unregister - * - * Returns: - * - 0 on success - * - -1 if IRQ was not registered - */ -int hal_irq_unregister(unsigned int irq); - -/** - * Check if currently executing in interrupt context. - * - * Returns: - * - 1 if in interrupt context (ISR) - * - 0 if in normal execution context - * - * Use case: - * - Validate ISR-only functions (e.g., heartbeat sampling) - * - Conditional behavior (ISR vs. normal path) - * - * Performance: - * - MUST be fast (< 10 cycles) - * - Often implemented as checking a per-CPU flag - */ -int hal_in_interrupt_context(void); - -#endif /* HAL_INTERRUPT_H */ -``` - -### Platform Implementation Notes - -**Linux:** -- `hal_irq_enable/disable()` → no-op (can't disable signals) -- `hal_irq_register()` → `sigaction()` + signal handler -- `hal_in_interrupt_context()` → thread-local flag set by signal handler - -**Kernel:** -- `hal_irq_enable/disable()` → `sti`/`cli` (x86_64) -- `hal_irq_register()` → IDT entry + APIC configuration -- `hal_in_interrupt_context()` → check per-CPU interrupt nesting count - ---- - -## 3. Memory (`include/hal/hal_memory.h`) - -### Purpose -Allocate/free memory, map pages, manage heap. - -### Interface - -```c -#ifndef HAL_MEMORY_H -#define HAL_MEMORY_H - -#include -#include - -/** - * Initialize memory subsystem. - * Must be called before any other hal_mem_* functions. - * - * Responsibilities: - * - Set up heap allocator - * - Initialize page tables (kernel platforms) - * - Parse memory map (kernel platforms) - * - * Error handling: - * - hal_panic() if insufficient memory - */ -void hal_mem_init(void); - -/** - * Allocate memory from heap. - * - * Parameters: - * - size: Bytes to allocate (0 = returns NULL) - * - * Guarantees: - * - Returns aligned pointer (at least 8-byte aligned) - * - Memory is zero-initialized - * - Thread-safe (can be called concurrently) - * - NOT callable from interrupt context - * - * Returns: - * - Pointer to allocated memory - * - NULL if allocation failed - * - * Notes: - * - On Linux: wrapper around malloc() - * - On kernel: custom heap allocator (kmalloc) - */ -void *hal_mem_alloc(size_t size); - -/** - * Free memory allocated by hal_mem_alloc(). - * - * Parameters: - * - ptr: Pointer returned from hal_mem_alloc() (NULL = no-op) - * - * Guarantees: - * - Thread-safe - * - NOT callable from interrupt context - * - Safe to call with NULL - */ -void hal_mem_free(void *ptr); - -/** - * Allocate contiguous physical pages. - * Used for DMA buffers, page tables, large allocations. - * - * Parameters: - * - count: Number of pages to allocate (page size is platform-specific) - * - * Returns: - * - Physical address of first page - * - 0 if allocation failed - * - * Notes: - * - Pages are NOT mapped into virtual address space - * - Caller must call hal_mem_map() to access - * - On Linux: may use mmap() with MAP_ANONYMOUS - * - On kernel: physical memory manager (PMM) - */ -uint64_t hal_mem_alloc_pages(size_t count); - -/** - * Free physical pages allocated by hal_mem_alloc_pages(). - * - * Parameters: - * - paddr: Physical address returned from hal_mem_alloc_pages() - * - count: Number of pages to free - */ -void hal_mem_free_pages(uint64_t paddr, size_t count); - -/** - * Map physical memory into virtual address space. - * - * Parameters: - * - vaddr: Virtual address to map (must be page-aligned) - * - paddr: Physical address to map (must be page-aligned) - * - size: Size in bytes (must be page-multiple) - * - flags: Mapping flags (HAL_MEM_*) - * - * Returns: - * - 0 on success - * - -1 if mapping failed - * - * Flags: - * - HAL_MEM_READ: Readable - * - HAL_MEM_WRITE: Writable - * - HAL_MEM_EXEC: Executable - * - HAL_MEM_NOCACHE: Uncached (for MMIO) - * - * Notes: - * - On Linux: no-op (identity mapping assumed) - * - On kernel: update page tables, flush TLB - */ -int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags); - -/* Mapping flags */ -#define HAL_MEM_READ (1 << 0) -#define HAL_MEM_WRITE (1 << 1) -#define HAL_MEM_EXEC (1 << 2) -#define HAL_MEM_NOCACHE (1 << 3) - -/** - * Get page size in bytes. - * - * Returns: - * - Page size (typically 4096 on x86_64) - */ -size_t hal_mem_page_size(void); - -#endif /* HAL_MEMORY_H */ -``` - ---- - -## 4. Console (`include/hal/hal_console.h`) - -### Purpose -Character I/O for REPL and diagnostics. - -### Interface - -```c -#ifndef HAL_CONSOLE_H -#define HAL_CONSOLE_H - -/** - * Initialize console subsystem. - * Must be called before any other hal_console_* functions. - * - * Responsibilities: - * - Initialize UART (kernel platforms) - * - Set up framebuffer (if available) - * - Configure terminal settings (Linux) - * - * Error handling: - * - hal_panic() if no console available - */ -void hal_console_init(void); - -/** - * Write a single character to console. - * - * Parameters: - * - c: Character to write - * - * Guarantees: - * - Character will be visible on console - * - Blocking (waits if output buffer full) - * - Callable from interrupt context (for panic messages) - * - * Performance: - * - May be slow (UART is ~1µs per char) - * - Buffering recommended for bulk output - */ -void hal_console_putc(char c); - -/** - * Write a null-terminated string to console. - * - * Parameters: - * - s: String to write (NULL = no-op) - * - * Notes: - * - Convenience wrapper around hal_console_putc() - */ -void hal_console_puts(const char *s); - -/** - * Read a single character from console. - * - * Guarantees: - * - Blocking (waits until character available) - * - NOT callable from interrupt context - * - * Returns: - * - Character read (0-255) - * - -1 on error or EOF - */ -int hal_console_getc(void); - -/** - * Check if input is available (non-blocking). - * - * Returns: - * - 1 if character available (hal_console_getc() will not block) - * - 0 if no character available - */ -int hal_console_has_input(void); - -#endif /* HAL_CONSOLE_H */ -``` - ---- - -## 5. CPU (`include/hal/hal_cpu.h`) - -### Purpose -CPU identification, relax/halt, SMP coordination. - -### Interface - -```c -#ifndef HAL_CPU_H -#define HAL_CPU_H - -#include - -/** - * Initialize CPU subsystem. - * Must be called before any other hal_cpu_* functions. - * - * Responsibilities: - * - Detect CPU features (SSE, AVX, etc.) - * - Set up per-CPU storage - * - Bring up secondary CPUs (SMP) - * - * Error handling: - * - hal_panic() if CPU unsupported - */ -void hal_cpu_init(void); - -/** - * Get current CPU ID. - * - * Returns: - * - CPU ID (0 = boot processor, 1+ = application processors) - * - * Guarantees: - * - Callable from interrupt context - * - Fast (< 10 cycles) - * - * Notes: - * - On x86_64 kernel: APIC ID - * - On Linux: thread ID (not true CPU ID) - */ -unsigned int hal_cpu_id(void); - -/** - * Hint to CPU to relax (pause in spin-wait loop). - * - * Guarantees: - * - Callable from interrupt context - * - No functional effect (pure performance hint) - * - * Notes: - * - On x86_64: pause instruction - * - On ARM: yield instruction - * - On Linux: sched_yield() - */ -void hal_cpu_relax(void); - -/** - * Halt CPU until next interrupt. - * - * Guarantees: - * - CPU enters low-power state - * - Wakes on next interrupt - * - NOT callable from interrupt context - * - * Notes: - * - On x86_64 kernel: hlt instruction - * - On Linux: select() or nanosleep() - */ -void hal_cpu_halt(void); - -/** - * Get number of CPUs in system. - * - * Returns: - * - Number of CPUs (1 = single-core, 2+ = multi-core) - */ -unsigned int hal_cpu_count(void); - -#endif /* HAL_CPU_H */ -``` - ---- - -## 6. Panic/Error Handling (`include/hal/hal_panic.h`) - -### Purpose -Fatal error handling (system halt or abort). - -### Interface - -```c -#ifndef HAL_PANIC_H -#define HAL_PANIC_H - -/** - * Fatal error: print message and halt system. - * - * Parameters: - * - msg: Error message (NULL = "panic") - * - * Guarantees: - * - NEVER returns - * - Message printed to console (if available) - * - System halted (kernel) or abort() (hosted) - * - Callable from interrupt context - * - * Notes: - * - Use for unrecoverable errors only - * - On kernel: disable interrupts + hlt loop - * - On Linux: fprintf(stderr) + abort() - */ -void hal_panic(const char *msg) __attribute__((noreturn)); - -#endif /* HAL_PANIC_H */ -``` - ---- - -## Interface Summary - -| Subsystem | Key Functions | ISR-Safe | Thread-Safe | -|-----------|---------------|----------|-------------| -| **Time** | `hal_time_now_ns()` | ✅ | ✅ | -| | `hal_timer_periodic()` | ❌ | ✅ | -| **Interrupt** | `hal_in_interrupt_context()` | ✅ | ✅ | -| | `hal_irq_disable()` | ✅ | ✅ | -| **Memory** | `hal_mem_alloc()` | ❌ | ✅ | -| | `hal_mem_map()` | ❌ | ⚠️ | -| **Console** | `hal_console_putc()` | ✅ | ⚠️ | -| | `hal_console_getc()` | ❌ | ❌ | -| **CPU** | `hal_cpu_id()` | ✅ | ✅ | -| | `hal_cpu_relax()` | ✅ | ✅ | - -**Legend:** -- ✅ = Safe -- ❌ = Unsafe -- ⚠️ = Platform-dependent - ---- - -## Next Steps - -See companion documentation: -- `platform-implementations.md` - How to implement HAL for a platform -- `migration-plan.md` - Refactoring existing code to use HAL -- `lithosananke-integration.md` - LithosAnanke-specific HAL implementation \ No newline at end of file diff --git a/docs/lithosananke/hal/lithosananke-integration.md b/docs/lithosananke/hal/lithosananke-integration.md deleted file mode 100644 index 81e7828b..00000000 --- a/docs/lithosananke/hal/lithosananke-integration.md +++ /dev/null @@ -1,763 +0,0 @@ -# LithosAnanke HAL Integration Guide - -## Overview - -This document provides **kernel-specific implementation details** for the HAL platform layer that becomes LithosAnanke. It covers UEFI boot, freestanding C environment, hardware initialization, and the path to a working `ok` prompt. - -**Audience:** Kernel developers implementing `src/platform/kernel/` - ---- - -## LithosAnanke Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ UEFI Firmware │ -│ • Initializes hardware │ -│ • Provides boot services (memory map, ACPI, GOP) │ -│ • Loads BOOTX64.EFI │ -└─────────────────────┬───────────────────────────────────────┘ - │ - ↓ ExitBootServices() -┌─────────────────────────────────────────────────────────────┐ -│ LithosAnanke Boot Loader (stage 0) │ -│ • Collects BootInfo (memory map, ACPI tables, framebuffer) │ -│ • Sets up initial page tables │ -│ • Exits UEFI boot services │ -│ • Jumps to LithosAnanke proper │ -└─────────────────────┬───────────────────────────────────────┘ - │ - ↓ -┌─────────────────────────────────────────────────────────────┐ -│ LithosAnanke HAL (stage 1) │ -│ • CPU initialization (GDT, IDT, interrupts) │ -│ • Memory subsystem (PMM, VMM, heap) │ -│ • Time subsystem (TSC, HPET, APIC timer) │ -│ • Console (UART + framebuffer) │ -└─────────────────────┬───────────────────────────────────────┘ - │ - ↓ -┌─────────────────────────────────────────────────────────────┐ -│ StarForth VM │ -│ • vm_create() │ -│ • Initialize physics subsystems │ -│ • Start REPL │ -│ • Print "ok" prompt │ -└─────────────────────────────────────────────────────────────┘ -``` - ---- - -## Directory Structure - -``` -src/platform/kernel/ -├── boot/ -│ ├── uefi_loader.c # UEFI entry point (BOOTX64.EFI) -│ ├── bootinfo.h # BootInfo struct definition -│ └── handoff.S # ASM trampoline (UEFI → LithosAnanke) -│ -├── cpu/ -│ ├── gdt.c # Global Descriptor Table -│ ├── idt.c # Interrupt Descriptor Table -│ ├── isr.S # Interrupt service routine stubs -│ └── smp.c # SMP bring-up (future) -│ -├── mm/ -│ ├── pmm.c # Physical Memory Manager -│ ├── vmm.c # Virtual Memory Manager -│ ├── kmalloc.c # Kernel heap allocator -│ └── paging.S # Page table manipulation ASM -│ -├── time/ -│ ├── tsc.c # Time Stamp Counter -│ ├── hpet.c # High Precision Event Timer -│ ├── apic_timer.c # Local APIC timer -│ └── pit.c # Programmable Interval Timer (fallback) -│ -├── drivers/ -│ ├── uart.c # UART 16550 serial console -│ ├── framebuffer.c # UEFI GOP framebuffer -│ ├── apic.c # Local APIC + IOAPIC -│ ├── acpi.c # ACPI table parsing -│ └── pci.c # PCI enumeration (future) -│ -├── hal_time.c # HAL time implementation -├── hal_interrupt.c # HAL interrupt implementation -├── hal_memory.c # HAL memory implementation -├── hal_console.c # HAL console implementation -├── hal_cpu.c # HAL CPU implementation -├── hal_panic.c # HAL panic implementation -│ -└── kernel_main.c # LithosAnanke entry point (after UEFI) -``` - ---- - -## Boot Sequence Detail - -### 1. UEFI Loader (`boot/uefi_loader.c`) - -**Responsibilities:** -- Collect system information from UEFI -- Allocate kernel memory -- Exit UEFI boot services -- Jump to LithosAnanke - -**Implementation:** - -```c -/* boot/uefi_loader.c */ -#include -#include -#include "bootinfo.h" - -/* BootInfo passed to LithosAnanke */ -typedef struct { - uint64_t memory_map_addr; - uint64_t memory_map_size; - uint64_t memory_map_descriptor_size; - uint64_t acpi_rsdp_addr; - uint64_t framebuffer_addr; - uint32_t framebuffer_width; - uint32_t framebuffer_height; - uint32_t framebuffer_pitch; -} BootInfo; - -EFI_STATUS EFIAPI efi_main(EFI_HANDLE ImageHandle, EFI_SYSTEM_TABLE *SystemTable) { - InitializeLib(ImageHandle, SystemTable); - - BootInfo *boot_info = (BootInfo *)AllocatePool(sizeof(BootInfo)); - - /* 1. Get memory map */ - UINTN map_size = 0, map_key, descriptor_size; - UINT32 descriptor_version; - EFI_MEMORY_DESCRIPTOR *memory_map = NULL; - - uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key, - &descriptor_size, &descriptor_version); - - map_size += 2 * descriptor_size; /* Extra space for ExitBootServices */ - memory_map = (EFI_MEMORY_DESCRIPTOR *)AllocatePool(map_size); - - uefi_call_wrapper(BS->GetMemoryMap, 5, &map_size, memory_map, &map_key, - &descriptor_size, &descriptor_version); - - boot_info->memory_map_addr = (uint64_t)memory_map; - boot_info->memory_map_size = map_size; - boot_info->memory_map_descriptor_size = descriptor_size; - - /* 2. Get ACPI RSDP */ - EFI_GUID acpi_20_guid = ACPI_20_TABLE_GUID; - void *rsdp = NULL; - for (UINTN i = 0; i < ST->NumberOfTableEntries; i++) { - if (CompareGuid(&ST->ConfigurationTable[i].VendorGuid, &acpi_20_guid) == 0) { - rsdp = ST->ConfigurationTable[i].VendorTable; - break; - } - } - boot_info->acpi_rsdp_addr = (uint64_t)rsdp; - - /* 3. Get framebuffer from GOP (Graphics Output Protocol) */ - EFI_GUID gop_guid = EFI_GRAPHICS_OUTPUT_PROTOCOL_GUID; - EFI_GRAPHICS_OUTPUT_PROTOCOL *gop = NULL; - uefi_call_wrapper(BS->LocateProtocol, 3, &gop_guid, NULL, (void **)&gop); - - if (gop) { - boot_info->framebuffer_addr = gop->Mode->FrameBufferBase; - boot_info->framebuffer_width = gop->Mode->Info->HorizontalResolution; - boot_info->framebuffer_height = gop->Mode->Info->VerticalResolution; - boot_info->framebuffer_pitch = gop->Mode->Info->PixelsPerScanLine * 4; - } - - /* 4. Exit boot services */ - uefi_call_wrapper(BS->ExitBootServices, 2, ImageHandle, map_key); - - /* 5. Jump to LithosAnanke */ - extern void kernel_main(BootInfo *boot_info); - kernel_main(boot_info); - - /* Never returns */ - while (1) __asm__ volatile("hlt"); - return EFI_SUCCESS; -} -``` - -### 2. LithosAnanke Entry Point (`kernel_main.c`) - -**Responsibilities:** -- Parse BootInfo -- Initialize CPU (GDT, IDT) -- Initialize memory subsystem -- Initialize HAL -- Call `main()` to start VM - -**Implementation:** - -```c -/* kernel_main.c */ -#include "bootinfo.h" -#include "cpu/gdt.h" -#include "cpu/idt.h" -#include "mm/pmm.h" -#include "mm/vmm.h" -#include "drivers/uart.h" -#include "drivers/framebuffer.h" - -/* Global BootInfo (available to all kernel code) */ -BootInfo *g_boot_info = NULL; - -void kernel_main(BootInfo *boot_info) { - g_boot_info = boot_info; - - /* 1. Initialize serial console ASAP (for debug output) */ - uart_early_init(0x3F8, 115200); /* COM1 */ - uart_puts("LithosAnanke booting...\r\n"); - - /* 2. Initialize CPU structures */ - gdt_init(); - idt_init(); - - /* 3. Initialize physical memory manager */ - pmm_init(boot_info); - uart_puts("PMM initialized\r\n"); - - /* 4. Initialize virtual memory manager */ - vmm_init(); - uart_puts("VMM initialized\r\n"); - - /* 5. Initialize kernel heap */ - kmalloc_init(); - uart_puts("Heap initialized\r\n"); - - /* 6. Initialize framebuffer */ - fb_init(boot_info); - - /* 7. Initialize HAL */ - hal_time_init(); - hal_interrupt_init(); - hal_mem_init(); - hal_console_init(); - hal_cpu_init(); - - uart_puts("HAL initialized\r\n"); - - /* 8. Call standard main() to start VM */ - extern int main(int argc, char **argv); - char *argv[] = {"starforth", NULL}; - main(1, argv); - - /* Never returns, but halt if it does */ - hal_panic("main() returned unexpectedly"); -} -``` - ---- - -## HAL Implementation Details - -### 1. HAL Time (`hal_time.c`) - -**Hardware:** TSC (Time Stamp Counter), HPET (High Precision Event Timer), APIC timer - -**Key challenges:** -- TSC calibration against HPET -- TSC frequency varies on old CPUs (need invariant TSC check) -- APIC timer setup for periodic interrupts - -**Implementation outline:** - -```c -/* hal_time.c */ -#include "hal/hal_time.h" -#include "time/tsc.h" -#include "time/hpet.h" -#include "time/apic_timer.h" - -static uint64_t tsc_hz = 0; -static uint64_t boot_tsc = 0; - -void hal_time_init(void) { - /* Check for invariant TSC */ - if (!tsc_is_invariant()) { - hal_panic("hal_time: TSC not invariant, falling back to HPET"); - /* TODO: Use HPET directly if TSC unreliable */ - } - - /* Calibrate TSC frequency using HPET */ - tsc_hz = tsc_calibrate_hpet(); - if (tsc_hz == 0) { - hal_panic("hal_time: TSC calibration failed"); - } - - boot_tsc = rdtsc(); - - /* Initialize APIC timer for periodic interrupts */ - apic_timer_init(); -} - -uint64_t hal_time_now_ns(void) { - uint64_t tsc = rdtsc() - boot_tsc; - return (tsc * 1000000000ULL) / tsc_hz; -} - -int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) { - return apic_timer_periodic(period_ns, callback, ctx); -} -``` - -**See:** `time/tsc.c`, `time/hpet.c`, `time/apic_timer.c` for low-level implementations. - ---- - -### 2. HAL Interrupt (`hal_interrupt.c`) - -**Hardware:** IDT (Interrupt Descriptor Table), Local APIC, IOAPIC - -**Key challenges:** -- Setting up 256 IDT entries -- Routing hardware IRQs through IOAPIC -- Tracking interrupt nesting depth (for `hal_in_interrupt_context()`) - -**Implementation outline:** - -```c -/* hal_interrupt.c */ -#include "hal/hal_interrupt.h" -#include "cpu/idt.h" -#include "drivers/apic.h" - -/* Per-CPU interrupt depth (thread-local on SMP) */ -static __thread unsigned int irq_depth = 0; - -void hal_interrupt_init(void) { - /* Set up IDT with 256 entries */ - idt_init(); - - /* Disable legacy PIC (use APIC instead) */ - outb(0x21, 0xFF); /* Mask all PIC1 IRQs */ - outb(0xA1, 0xFF); /* Mask all PIC2 IRQs */ - - /* Initialize Local APIC */ - apic_init(); - - /* Initialize IOAPIC */ - ioapic_init(); -} - -void hal_irq_enable(void) { - __asm__ volatile("sti"); -} - -unsigned long hal_irq_disable(void) { - unsigned long flags; - __asm__ volatile("pushfq; popq %0; cli" : "=r"(flags)); - return flags; -} - -void hal_irq_restore(unsigned long flags) { - __asm__ volatile("pushq %0; popfq" :: "r"(flags)); -} - -int hal_in_interrupt_context(void) { - return irq_depth > 0; -} - -/* Common ISR entry (called from IDT stubs) */ -void isr_common_handler(uint64_t vector, uint64_t error_code) { - irq_depth++; - - /* Dispatch to registered handler */ - extern void isr_dispatch(uint64_t vector, uint64_t error_code); - isr_dispatch(vector, error_code); - - irq_depth--; - - /* Send EOI to APIC */ - apic_eoi(); -} -``` - -**See:** `cpu/idt.c`, `cpu/isr.S`, `drivers/apic.c` for low-level implementations. - ---- - -### 3. HAL Memory (`hal_memory.c`) - -**Components:** -- **PMM (Physical Memory Manager):** Tracks free/used physical page frames -- **VMM (Virtual Memory Manager):** Manages page tables (4-level paging on x86_64) -- **kmalloc:** Kernel heap allocator (slab or buddy allocator) - -**Implementation outline:** - -```c -/* hal_memory.c */ -#include "hal/hal_memory.h" -#include "mm/pmm.h" -#include "mm/vmm.h" -#include "mm/kmalloc.h" - -void hal_mem_init(void) { - /* PMM and VMM initialized in kernel_main() before HAL */ - /* Just verify they're ready */ - if (!pmm_is_initialized()) { - hal_panic("hal_mem_init: PMM not initialized"); - } -} - -void *hal_mem_alloc(size_t size) { - return kmalloc(size); -} - -void hal_mem_free(void *ptr) { - kfree(ptr); -} - -uint64_t hal_mem_alloc_pages(size_t count) { - return pmm_alloc_pages(count); -} - -void hal_mem_free_pages(uint64_t paddr, size_t count) { - pmm_free_pages(paddr, count); -} - -int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) { - return vmm_map_range(vaddr, paddr, size, flags); -} - -size_t hal_mem_page_size(void) { - return 4096; -} -``` - -**PMM details** (`mm/pmm.c`): -- Bitmap allocator (1 bit per 4KB page) -- Parse UEFI memory map to mark usable vs. reserved regions -- Reserve kernel image, ACPI tables, framebuffer - -**VMM details** (`mm/vmm.c`): -- 4-level page tables (PML4 → PDPT → PD → PT) -- Recursive mapping trick for page table access -- Identity-map kernel, higher-half kernel optional - -**kmalloc details** (`mm/kmalloc.c`): -- Slab allocator for common sizes (16, 32, 64, ..., 4096 bytes) -- Buddy allocator for large allocations -- Zero-initialization required by HAL contract - ---- - -### 4. HAL Console (`hal_console.c`) - -**Hardware:** UART 16550 (serial), UEFI GOP framebuffer (video) - -**Implementation outline:** - -```c -/* hal_console.c */ -#include "hal/hal_console.h" -#include "drivers/uart.h" -#include "drivers/framebuffer.h" - -void hal_console_init(void) { - /* UART already initialized in kernel_main() for early debug */ - /* Framebuffer initialized from BootInfo */ -} - -void hal_console_putc(char c) { - uart_putc(c); /* Always output to serial */ - fb_putc(c); /* Also output to framebuffer if available */ -} - -void hal_console_puts(const char *s) { - while (*s) { - hal_console_putc(*s++); - } -} - -int hal_console_getc(void) { - /* Block until UART has data */ - return uart_getc(); -} - -int hal_console_has_input(void) { - return uart_has_data(); -} -``` - -**UART details** (`drivers/uart.c`): -- 16550 compatible (COM1 = 0x3F8, COM2 = 0x2F8) -- 115200 baud default -- Polling mode (no interrupts for simplicity) - -**Framebuffer details** (`drivers/framebuffer.c`): -- Linear framebuffer from UEFI GOP -- 32-bit RGB pixels -- Software text rendering (8x16 font) -- Scrolling, cursor, ANSI escape codes (optional) - ---- - -### 5. HAL CPU (`hal_cpu.c`) - -**Implementation outline:** - -```c -/* hal_cpu.c */ -#include "hal/hal_cpu.h" -#include "drivers/apic.h" - -static unsigned int num_cpus = 1; - -void hal_cpu_init(void) { - /* Detect CPU features (CPUID) */ - /* TODO: Bring up secondary CPUs (SMP) */ -} - -unsigned int hal_cpu_id(void) { - return apic_get_id(); /* Local APIC ID */ -} - -void hal_cpu_relax(void) { - __asm__ volatile("pause"); -} - -void hal_cpu_halt(void) { - __asm__ volatile("hlt"); -} - -unsigned int hal_cpu_count(void) { - return num_cpus; -} -``` - ---- - -## Build System - -### Toolchain Requirements - -**Freestanding C environment:** -- `gcc` or `clang` with `-ffreestanding` -- `ld` with custom linker script -- `objcopy` to create PE32+ executable (for UEFI) - -### Makefile Additions - -```makefile -# Platform: kernel -ifeq ($(PLATFORM),kernel) - CC = gcc - LD = ld - OBJCOPY = objcopy - - CFLAGS += -ffreestanding -nostdlib -mno-red-zone -mcmodel=large - CFLAGS += -mno-mmx -mno-sse -mno-sse2 # No FP in kernel - CFLAGS += -I/usr/include/efi -I/usr/include/efi/x86_64 - - LDFLAGS += -nostdlib -static -T src/platform/kernel/linker.ld - - # Platform sources - PLATFORM_SOURCES = \ - src/platform/kernel/boot/uefi_loader.c \ - src/platform/kernel/kernel_main.c \ - src/platform/kernel/cpu/gdt.c \ - src/platform/kernel/cpu/idt.c \ - src/platform/kernel/cpu/isr.S \ - src/platform/kernel/mm/pmm.c \ - src/platform/kernel/mm/vmm.c \ - src/platform/kernel/mm/kmalloc.c \ - src/platform/kernel/drivers/uart.c \ - src/platform/kernel/drivers/framebuffer.c \ - src/platform/kernel/drivers/apic.c \ - src/platform/kernel/hal_time.c \ - src/platform/kernel/hal_interrupt.c \ - src/platform/kernel/hal_memory.c \ - src/platform/kernel/hal_console.c \ - src/platform/kernel/hal_cpu.c \ - src/platform/kernel/hal_panic.c - - # Build UEFI PE32+ executable - starforth.efi: $(OBJECTS) - $(LD) $(LDFLAGS) -o starforth.so $(OBJECTS) - $(OBJCOPY) -j .text -j .data -j .rodata -j .reloc \ - --target=efi-app-x86_64 starforth.so $@ -endif -``` - -### Linker Script (`src/platform/kernel/linker.ld`) - -```ld -OUTPUT_FORMAT("elf64-x86-64") -ENTRY(efi_main) - -SECTIONS { - . = 0x100000; /* Load at 1MB (standard kernel load address) */ - - .text : { - *(.text .text.*) - } - - .rodata : { - *(.rodata .rodata.*) - } - - .data : { - *(.data .data.*) - } - - .bss : { - *(COMMON) - *(.bss .bss.*) - } - - /DISCARD/ : { - *(.eh_frame) - } -} -``` - ---- - -## Testing on QEMU - -### QEMU + OVMF Setup - -```bash -# Install OVMF (UEFI firmware for QEMU) -sudo apt install ovmf - -# Create ESP (EFI System Partition) -mkdir -p esp/EFI/BOOT -cp starforth.efi esp/EFI/BOOT/BOOTX64.EFI - -# Run QEMU -qemu-system-x86_64 \ - -bios /usr/share/ovmf/OVMF.fd \ - -drive file=fat:rw:esp/,format=raw \ - -serial stdio \ - -m 512M \ - -enable-kvm -``` - -### Expected Output - -``` -LithosAnanke booting... -PMM initialized -VMM initialized -Heap initialized -HAL initialized -StarForth VM v1.0 -ok -``` - ---- - -## Debugging - -### Serial Debugging - -```c -/* Early debug output via UART */ -void debug_puts(const char *s) { - while (*s) { - while (!(inb(0x3FD) & 0x20)); /* Wait for UART ready */ - outb(0x3F8, *s++); /* Write character */ - } -} -``` - -### GDB Remote Debugging - -```bash -# Start QEMU with GDB server -qemu-system-x86_64 ... -s -S - -# In another terminal -gdb starforth.elf -(gdb) target remote :1234 -(gdb) break kernel_main -(gdb) continue -``` - -### Panic Handler - -```c -/* hal_panic.c */ -void hal_panic(const char *msg) { - hal_irq_disable(); - - hal_console_puts("\n*** KERNEL PANIC ***\n"); - hal_console_puts(msg ? msg : "unknown error"); - hal_console_puts("\nSystem halted.\n"); - - /* Halt all CPUs */ - while (1) { - hal_cpu_halt(); - } -} -``` - ---- - -## Roadmap to `ok` Prompt - -### Milestone 1: Boot + Serial Output -- UEFI loader runs -- Kernel prints "LithosAnanke booting..." to serial -- System doesn't triple-fault - -### Milestone 2: Memory Works -- PMM tracks physical pages -- VMM maps kernel -- kmalloc/kfree work - -### Milestone 3: HAL Initialized -- All `hal_*_init()` functions complete -- HAL functions callable (even if not fully functional) - -### Milestone 4: VM Starts -- `vm_create()` succeeds -- Dictionary allocated -- No crashes in VM init - -### Milestone 5: REPL Runs -- REPL prints "`ok`" prompt -- Can type characters (echoed to serial) -- Can execute simple words (`1 2 + .` → `3`) - -### Milestone 6: Physics Subsystems Work -- Execution heat tracking operational -- Heartbeat timer fires -- Rolling window captures execution history - -### Milestone 7: Full Test Suite -- All 936+ tests pass on kernel -- DoE mode works -- 0% algorithmic variance on bare metal - ---- - -## Future Work (StarshipOS) - -After achieving `ok` prompt on LithosAnanke: - -1. **Storage:** AHCI/NVMe drivers, FAT32 filesystem -2. **Networking:** VirtIO-net driver, TCP/IP stack -3. **Process model:** Forth tasks, scheduling, IPC -4. **Device model:** Unified block/net/char subsystem -5. **Security:** Capabilities, ACL, Forth-based access control - -**The HAL remains the foundation throughout.** - ---- - -## References - -- UEFI Specification: https://uefi.org/specifications -- GNU-EFI: https://sourceforge.net/projects/gnu-efi/ -- OSDev Wiki: https://wiki.osdev.org/ -- Intel SDM: https://www.intel.com/sdm -- ACPI Specification: https://uefi.org/acpi \ No newline at end of file diff --git a/docs/lithosananke/hal/migration-plan.md b/docs/lithosananke/hal/migration-plan.md deleted file mode 100644 index 7c7f7cea..00000000 --- a/docs/lithosananke/hal/migration-plan.md +++ /dev/null @@ -1,650 +0,0 @@ -# HAL Migration Plan - -## Overview - -This document provides a **step-by-step plan** for refactoring StarForth to use the Hardware Abstraction Layer (HAL). The migration preserves all existing functionality while preparing the codebase for LithosAnanke. - -> **HISTORICAL — L4Re/Fiasco.OC**: L4Re references below (including large sections of this migration plan) describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. - -**Goal:** Zero functional regressions, all 936+ tests pass, 0% algorithmic variance maintained. - ---- - -## Migration Strategy - -### Principles - -1. **Incremental refactoring** - One subsystem at a time -2. **Test after each step** - Build + test suite must pass -3. **Platform parity** - Linux and L4Re work throughout migration -4. **No big bang** - HAL interfaces defined first, then adopted gradually - -### Phases - -``` -Phase 1: Define HAL Interfaces (headers only) - ↓ -Phase 2: Implement HAL for Linux (refactor existing code) - ↓ -Phase 3: Migrate VM Core to use HAL - ↓ -Phase 4: Migrate Physics Subsystems to use HAL - ↓ -Phase 5: Migrate REPL and Word Implementations - ↓ -Phase 6: Implement HAL for L4Re (optional validation) - ↓ -Phase 7: Validate Deterministic Behavior -``` - ---- - -## Phase 1: Define HAL Interfaces - -**Duration:** 1-2 days -**Risk:** Low (no code changes, only headers) - -### Tasks - -1. **Create HAL header directory** - ```bash - mkdir -p include/hal - ``` - -2. **Write HAL interface headers** - - `include/hal/hal_time.h` - - `include/hal/hal_interrupt.h` - - `include/hal/hal_memory.h` - - `include/hal/hal_console.h` - - `include/hal/hal_cpu.h` - - `include/hal/hal_panic.h` - - See `interfaces.md` for full specifications. - -3. **Update Makefile to include HAL headers** - ```makefile - INCLUDES += -Iinclude/hal - ``` - -4. **Compile check (headers only)** - ```bash - make clean && make PLATFORM=linux - ``` - - Should compile without errors (HAL functions not yet called). - -### Success Criteria - -✅ All HAL headers compile without errors -✅ No changes to VM code yet -✅ Documentation reviewed and approved - ---- - -## Phase 2: Implement HAL for Linux - -**Duration:** 3-5 days -**Risk:** Medium (refactoring existing platform code) - -### Current Platform Code Locations - -Audit existing platform-specific code: - -```bash -# Find POSIX-specific calls -grep -r "clock_gettime\|pthread\|signal\|malloc\|printf" src/ -``` - -**Expected locations:** -- Timing: `src/platform/linux/time.c` (already exists) -- Memory: Direct `malloc()` calls in VM code (needs refactoring) -- Console: Direct `printf()` calls in REPL (needs refactoring) - -### Tasks - -1. **Create Linux HAL directory** - ```bash - mkdir -p src/platform/linux/hal - ``` - -2. **Implement Linux HAL subsystems** - - **2a. `hal_time.c`** - Refactor `src/platform/linux/time.c` - ```c - /* Before: */ - uint64_t platform_time_ns(void) { - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - return ...; - } - - /* After: */ - uint64_t hal_time_now_ns(void) { - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec; - } - ``` - - **2b. `hal_memory.c`** - Wrap malloc/free - ```c - void *hal_mem_alloc(size_t size) { - if (size == 0) return NULL; - void *ptr = malloc(size); - if (ptr) memset(ptr, 0, size); - return ptr; - } - - void hal_mem_free(void *ptr) { - free(ptr); - } - ``` - - **2c. `hal_console.c`** - Wrap stdio - ```c - void hal_console_putc(char c) { - putchar(c); - fflush(stdout); - } - - int hal_console_getc(void) { - return getchar(); - } - ``` - - **2d. `hal_interrupt.c`** - Signal-based interrupts - See `platform-implementations.md` for full implementation. - - **2e. `hal_cpu.c`** - CPU info - ```c - unsigned int hal_cpu_id(void) { - return 0; /* Single-threaded */ - } - - void hal_cpu_relax(void) { - sched_yield(); - } - ``` - - **2f. `hal_panic.c`** - Error handling - ```c - void hal_panic(const char *msg) { - fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown"); - abort(); - } - ``` - -3. **Update Makefile** - ```makefile - ifeq ($(PLATFORM),linux) - PLATFORM_SOURCES = \ - src/platform/linux/hal_time.c \ - src/platform/linux/hal_interrupt.c \ - src/platform/linux/hal_memory.c \ - src/platform/linux/hal_console.c \ - src/platform/linux/hal_cpu.c \ - src/platform/linux/hal_panic.c - endif - ``` - -4. **Build test (HAL code compiles)** - ```bash - make clean && make PLATFORM=linux - ``` - -### Success Criteria - -✅ Linux HAL compiles without errors -✅ Existing VM code still builds (not yet using HAL) -✅ No functional changes yet - ---- - -## Phase 3: Migrate VM Core to use HAL - -**Duration:** 2-3 days -**Risk:** High (core VM changes) - -### Files to Modify - -- `src/vm.c` - Core interpreter loop -- `src/vm_api.c` - External API -- `src/memory_management.c` - Dictionary allocator -- `include/vm.h` - VM struct - -### Tasks - -1. **Replace direct malloc/free in VM core** - - **Before:** - ```c - /* src/memory_management.c */ - void *mem = malloc(size); - ``` - - **After:** - ```c - #include "hal/hal_memory.h" - - void *mem = hal_mem_alloc(size); - ``` - -2. **Replace timing calls in heartbeat** - - **Before:** - ```c - /* src/vm.c */ - #include - - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - uint64_t now = ts.tv_sec * 1000000000ULL + ts.tv_nsec; - ``` - - **After:** - ```c - #include "hal/hal_time.h" - - uint64_t now = hal_time_now_ns(); - ``` - -3. **Add HAL initialization to VM startup** - - **In `src/main.c`:** - ```c - int main(int argc, char **argv) { - /* Initialize HAL subsystems */ - hal_time_init(); - hal_interrupt_init(); - hal_mem_init(); - hal_console_init(); - hal_cpu_init(); - - /* Continue with existing VM initialization */ - VM *vm = vm_create(); - ... - } - ``` - -4. **Remove platform-specific #ifdefs** - - **Before:** - ```c - #ifdef PLATFORM_LINUX - clock_gettime(CLOCK_MONOTONIC, &ts); - #elif PLATFORM_L4RE - l4re_kip_clock(kip); - #endif - ``` - - **After:** - ```c - uint64_t now = hal_time_now_ns(); /* Works on all platforms */ - ``` - -5. **Build and test** - ```bash - make clean && make PLATFORM=linux - make test - ``` - -### Expected Failures - -- **Compilation errors:** Missing `#include "hal/hal_*.h"` -- **Link errors:** HAL functions not implemented -- **Runtime errors:** HAL not initialized before VM - -**Fix incrementally:** Add includes, implement missing HAL functions, ensure init order. - -### Success Criteria - -✅ VM builds using HAL interfaces -✅ All 936+ tests pass -✅ No platform-specific code in `src/vm.c` - ---- - -## Phase 4: Migrate Physics Subsystems to use HAL - -**Duration:** 2-3 days -**Risk:** Medium (determinism must be preserved) - -### Files to Modify - -- `src/dictionary_heat_optimization.c` -- `src/rolling_window_of_truth.c` -- `src/physics_hotwords_cache.c` -- `src/physics_pipelining_metrics.c` -- `src/inference_engine.c` -- `src/heartbeat.c` - -### Tasks - -1. **Audit timing calls in physics subsystems** - ```bash - grep -n "clock_gettime\|time\.h" src/*physics*.c src/heartbeat.c src/inference*.c - ``` - -2. **Replace timing calls** - - **Example: `src/rolling_window_of_truth.c`** - ```c - /* Before: */ - #include - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - uint64_t timestamp = ...; - - /* After: */ - #include "hal/hal_time.h" - uint64_t timestamp = hal_time_now_ns(); - ``` - -3. **Migrate heartbeat to HAL timer** - - **Before:** `src/heartbeat.c` uses pthread + sleep - ```c - void *heartbeat_thread(void *arg) { - while (running) { - usleep(period_us); - vm_tick(vm); - } - } - ``` - - **After:** Use HAL periodic timer - ```c - static VM *heartbeat_vm = NULL; - - static void heartbeat_isr(void *ctx) { - (void)ctx; - if (heartbeat_vm) { - vm_tick_isr(heartbeat_vm); /* ISR-safe VM tick */ - } - } - - void heartbeat_start(VM *vm, uint64_t rate_hz) { - heartbeat_vm = vm; - uint64_t period_ns = 1000000000ULL / rate_hz; - hal_timer_periodic(period_ns, heartbeat_isr, NULL); - } - ``` - - **Critical:** `vm_tick_isr()` must be ISR-safe: - - No malloc/free - - No blocking I/O - - Only lock-free ring buffer updates - -4. **Test deterministic behavior** - ```bash - make fastest PLATFORM=linux - ./build/amd64/fastest/starforth --doe > results_after_hal.csv - ``` - - Compare with baseline (before HAL migration): - ```bash - diff results_before_hal.csv results_after_hal.csv - ``` - - **Expected:** Identical results (0% algorithmic variance). - -### Success Criteria - -✅ Physics subsystems use HAL for all timing -✅ Heartbeat runs via HAL periodic timer -✅ 0% algorithmic variance maintained -✅ All tests pass - ---- - -## Phase 5: Migrate REPL and Word Implementations - -**Duration:** 1-2 days -**Risk:** Low (mostly console I/O) - -### Files to Modify - -- `src/repl.c` - Read-eval-print loop -- `src/word_source/*_words.c` - Words that do I/O (emit, key, etc.) - -### Tasks - -1. **Replace stdio in REPL** - - **Before:** `src/repl.c` - ```c - #include - - char c = getchar(); - printf("ok\n"); - ``` - - **After:** - ```c - #include "hal/hal_console.h" - - char c = hal_console_getc(); - hal_console_puts("ok\n"); - ``` - -2. **Update I/O words** - - **`src/word_source/io_words.c`:** - ```c - /* EMIT ( c -- ) */ - static void word_emit(VM *vm) { - int c = vm_pop(vm); - hal_console_putc((char)c); - } - - /* KEY ( -- c ) */ - static void word_key(VM *vm) { - int c = hal_console_getc(); - vm_push(vm, c); - } - ``` - -3. **Test REPL interactively** - ```bash - ./build/amd64/standard/starforth - > 1 2 + . - 3 ok - > BYE - ``` - -### Success Criteria - -✅ REPL works identically to before -✅ All I/O words use HAL -✅ No direct stdio calls in VM code - ---- - -## Phase 6: Implement HAL for L4Re (Optional) - -**Duration:** 3-5 days -**Risk:** Low (validates HAL portability) - -This phase is **optional** but highly recommended to validate that the HAL abstraction actually works across platforms. - -### Tasks - -1. **Create L4Re HAL directory** - ```bash - mkdir -p src/platform/l4re/hal - ``` - -2. **Implement L4Re HAL subsystems** - - `hal_time.c` → L4Re clock API - - `hal_memory.c` → L4Re dataspaces - - `hal_console.c` → L4Re console service - - `hal_interrupt.c` → L4Re IRQ objects - - See L4Re documentation for API details. - -3. **Build for L4Re** - ```bash - make PLATFORM=l4re - ``` - -4. **Run tests on L4Re** - ```bash - make PLATFORM=l4re test - ``` - -### Success Criteria - -✅ L4Re HAL compiles -✅ VM runs on L4Re without source changes -✅ Tests pass on L4Re - ---- - -## Phase 7: Validate Deterministic Behavior - -**Duration:** 1 day -**Risk:** High (final validation) - -### Tasks - -1. **Run DoE on Linux (HAL)** - ```bash - make fastest PLATFORM=linux - for i in {1..10}; do - ./build/amd64/fastest/starforth --doe > results_linux_$i.csv - done - ``` - -2. **Verify 0% variance across runs** - ```bash - # All CSV files should be identical - md5sum results_linux_*.csv - ``` - -3. **Compare with pre-HAL baseline** - ```bash - diff results_before_hal.csv results_linux_1.csv - ``` - - **Expected:** Identical (HAL added zero overhead). - -4. **Run full test suite** - ```bash - make test PLATFORM=linux - ``` - - **Expected:** All 936+ tests pass. - -5. **Benchmark performance** - ```bash - make bench PLATFORM=linux - ``` - - **Expected:** < 5% performance delta from pre-HAL baseline. - -### Success Criteria - -✅ 0% algorithmic variance maintained -✅ All tests pass -✅ < 5% performance regression -✅ HAL abstraction validated - ---- - -## Rollback Strategy - -If migration fails at any phase: - -1. **Git branch strategy** - ```bash - git checkout -b hal-migration - # Work on branch, test each phase - git commit -m "Phase N complete" - # If phase fails, revert: - git reset --hard HEAD~1 - ``` - -2. **Incremental commits** - - Commit after each phase passes tests - - Never commit broken code - - Each commit should build and pass tests - -3. **Feature flag (if needed)** - ```c - #ifdef ENABLE_HAL - uint64_t now = hal_time_now_ns(); - #else - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - uint64_t now = ...; - #endif - ``` - - Enable HAL incrementally, fall back if issues arise. - ---- - -## Post-Migration Cleanup - -After successful migration: - -1. **Remove old platform code** - ```bash - # If src/platform/linux/time.c was fully replaced by hal_time.c - git rm src/platform/linux/time.c - ``` - -2. **Update documentation** - - `docs/CLAUDE.md` - Mention HAL architecture - - `README.md` - Update build instructions - - `docs/DEVELOPER.md` - Add HAL section - -3. **Update .gitignore** - ``` - # Build artifacts for all platforms - build/linux/ - build/l4re/ - build/kernel/ - ``` - ---- - -## Timeline Estimate - -| Phase | Duration | Cumulative | -|-------|----------|------------| -| 1. Define HAL Interfaces | 1-2 days | 1-2 days | -| 2. Implement Linux HAL | 3-5 days | 4-7 days | -| 3. Migrate VM Core | 2-3 days | 6-10 days | -| 4. Migrate Physics | 2-3 days | 8-13 days | -| 5. Migrate REPL | 1-2 days | 9-15 days | -| 6. L4Re HAL (optional) | 3-5 days | 12-20 days | -| 7. Validation | 1 day | 13-21 days | - -**Total:** 2-4 weeks (depending on L4Re inclusion) - ---- - -## Success Metrics - -The HAL migration is successful if: - -1. ✅ **All tests pass** - 936+ tests on Linux (and L4Re if implemented) -2. ✅ **0% variance** - DoE results identical pre/post migration -3. ✅ **No regressions** - Performance < 5% slower -4. ✅ **Zero platform code in VM** - No #ifdef PLATFORM_X in core -5. ✅ **Clean abstractions** - HAL interfaces well-documented -6. ✅ **Ready for LithosAnanke** - Platform layer can be replaced - ---- - -## Next Steps - -After successful HAL migration: - -1. **Implement LithosAnanke platform** (`src/platform/kernel/`) -2. **UEFI boot loader** (`src/platform/kernel/boot/uefi_loader.c`) -3. **Boot to `ok` prompt** on QEMU/OVMF -4. **Validate physics on bare metal** (determinism on real hardware) - -See `lithosananke-integration.md` for LithosAnanke-specific implementation details. \ No newline at end of file diff --git a/docs/lithosananke/hal/overview.md b/docs/lithosananke/hal/overview.md deleted file mode 100644 index b5602070..00000000 --- a/docs/lithosananke/hal/overview.md +++ /dev/null @@ -1,345 +0,0 @@ -# Hardware Abstraction Layer (HAL) Architecture - -> **⚠️ Important Note**: This document describes the **target HAL architecture** with standardized `hal_*` function naming. The current codebase uses `sf_time_backend_t` and similar abstractions with different naming conventions. See [migration-plan.md](migration-plan.md) for the refactoring steps to migrate existing platform code to the HAL interfaces described here. -> -> **HISTORICAL — L4Re/Fiasco.OC**: this document's L4Re references (platform comparison table, microkernel discussion, etc.) describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. Treat every L4Re mention below as historical design context, not a currently buildable platform. - -## Executive Summary - -The Hardware Abstraction Layer (HAL) is the architectural foundation that enables StarForth to evolve from a hosted VM into LithosAnanke while preserving the physics-driven adaptive runtime's deterministic behavior across all platforms. - -**Critical insight:** The HAL is not an afterthought—it's the linchpin that makes LithosAnanke → StarshipOS possible without compromising StarForth's experimental integrity. - ---- - -## The Problem - -StarForth currently runs on multiple platforms: -- **Linux** (POSIX, hosted) -- **L4Re/Fiasco.OC** (microkernel) -- **Bare metal** (limited, experimental) - -Each platform has its own: -- Timing mechanisms (POSIX timers vs. hardware timers) -- Interrupt handling (signals vs. IDT/APIC) -- Memory allocation (malloc vs. physical page frames) -- I/O mechanisms (stdin/stdout vs. UART/framebuffer) - -**Without a HAL:** Platform-specific code bleeds into the VM core, physics subsystems, and word implementations. This creates: -- ❌ Fragile #ifdef PLATFORM_X conditionals throughout codebase -- ❌ Platform-specific bugs in supposedly portable code -- ❌ Inability to test kernel code on hosted platforms -- ❌ Risk to deterministic behavior guarantees (0% algorithmic variance) - -**With a HAL:** Clean separation between VM logic and platform implementation: -- ✅ VM and physics subsystems are platform-agnostic -- ✅ Platform code is isolated and testable -- ✅ New platforms (LithosAnanke) can be added without touching VM core -- ✅ Deterministic behavior guaranteed by HAL contract, not platform quirks - ---- - -## Architecture Overview - -``` -┌────────────────────────────────────────────────────────────┐ -│ StarForth VM Core + Physics Subsystems │ -│ • Interpreter loop (vm.c) │ -│ • Execution heat tracking │ -│ • Rolling window of truth │ -│ • Hot-words cache │ -│ • Pipelining metrics │ -│ • Inference engine │ -│ • Heartbeat coordination │ -│ │ -│ ↓ Calls HAL interfaces (platform-agnostic) │ -├────────────────────────────────────────────────────────────┤ -│ Hardware Abstraction Layer (HAL) │ -│ • hal_time.h - Monotonic time, timers, calibration │ -│ • hal_interrupt.h - IRQ management, ISR registration │ -│ • hal_memory.h - Allocation, page mapping, heap │ -│ • hal_console.h - Character I/O (serial, framebuffer) │ -│ • hal_cpu.h - CPU ID, relax/halt, SMP coordination │ -│ │ -│ ↓ Platform-specific implementations │ -├────────────────────────────────────────────────────────────┤ -│ Platform Implementations │ -│ ┌──────────────┬──────────────┬──────────────────────┐ │ -│ │ Linux │ L4Re │ Kernel (LithosAnanke) │ │ -│ │ (POSIX) │ (microkernel)│ (freestanding) │ │ -│ ├──────────────┼──────────────┼──────────────────────┤ │ -│ │ clock_gettime│ L4Re::Clock │ TSC + HPET + APIC │ │ -│ │ timerfd │ L4Re::IrqEoi │ IDT + APIC IRQ │ │ -│ │ malloc/free │ dataspaces │ PMM + VMM + kmalloc │ │ -│ │ stdin/stdout │ L4Re::Console│ UART + framebuffer │ │ -│ │ pthread │ L4Re::Thread │ SMP bring-up │ │ -│ └──────────────┴──────────────┴──────────────────────┘ │ -└────────────────────────────────────────────────────────────┘ -``` - ---- - -## Design Principles - -### 1. **VM Purity** -The VM core must never know which platform it's running on. All platform awareness lives in HAL implementations. - -**Anti-pattern (current):** -```c -#ifdef PLATFORM_LINUX - clock_gettime(CLOCK_MONOTONIC, &ts); -#elif PLATFORM_L4RE - l4re_kip_clock(kip); -#elif PLATFORM_KERNEL - rdtsc(); -#endif -``` - -**Correct pattern (HAL):** -```c -// In VM code (platform-agnostic) -uint64_t now = hal_time_now_ns(); - -// In platform/linux/hal_time.c -uint64_t hal_time_now_ns(void) { - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec; -} - -// In platform/kernel/hal_time.c -uint64_t hal_time_now_ns(void) { - return tsc_to_ns(rdtsc()); -} -``` - -### 2. **Contract-First Design** -HAL interfaces are **contracts**, not convenience wrappers. Each HAL function has: -- **Precise semantics** - What it does, guaranteed across all platforms -- **Error handling** - When it can fail and how -- **Performance expectations** - Allowed latency/overhead -- **Concurrency model** - Thread-safe? ISR-safe? - -### 3. **Testability on Hosted Platforms** -LithosAnanke code must be developable and testable on Linux/L4Re before deploying to bare metal. - -**Example:** Heartbeat ISR development -- **Kernel implementation:** APIC timer interrupt → ISR → ring buffer -- **Linux test implementation:** timerfd + signal handler → ISR → ring buffer -- **Same HAL interface, same VM code, different platform layer** - -### 4. **Zero Overhead When Possible** -HAL calls should compile to direct hardware access on kernel platforms, not add abstraction tax. - -**Good:** `hal_time_now_ns()` inlines to `rdtsc()` with `-O2` -**Bad:** Function pointer indirection adds 5-10 cycles per call - -### 5. **Fail-Fast Validation** -HAL implementations validate platform assumptions at init time, not during execution. - -**Example:** -```c -void hal_time_init(void) { - // Calibrate TSC frequency at boot - tsc_calibrate_hpet(); - - // Validate monotonicity - uint64_t t1 = hal_time_now_ns(); - hal_cpu_relax(); - uint64_t t2 = hal_time_now_ns(); - - if (t2 < t1) { - hal_panic("hal_time: TSC not monotonic!"); - } -} -``` - ---- - -## HAL Subsystems - -### 1. **Time & Timers** (`hal_time.h`) -**Purpose:** Monotonic time, periodic/oneshot timers, calibration - -**Critical for StarForth:** The heartbeat subsystem and physics feedback loops depend on precise, jitter-free timing. HAL must guarantee: -- Monotonic time (never goes backward) -- Sub-microsecond resolution -- Calibrated frequency (for TSC-based timing) - -**Platform challenges:** -- **Linux:** `clock_gettime()` is good, but signal-based timers have latency -- **Kernel:** TSC drift, HPET/PIT fallback, per-core calibration - -### 2. **Interrupts** (`hal_interrupt.h`) -**Purpose:** Enable/disable IRQs, register ISRs, query interrupt context - -**Critical for StarForth:** Heartbeat ISR must run at precise intervals without VM involvement. - -**Platform challenges:** -- **Linux:** Signals are "interrupt-like" but not true IRQs -- **Kernel:** IDT setup, APIC configuration, spurious interrupt handling - -### 3. **Memory** (`hal_memory.h`) -**Purpose:** Allocate/free memory, page mapping, heap management - -**Critical for StarForth:** Dictionary allocation, stack allocation, heap allocator must work identically across platforms. - -**Platform challenges:** -- **Linux:** malloc/free are simple -- **Kernel:** Physical memory manager, virtual memory manager, heap allocator—all from scratch - -### 4. **Console** (`hal_console.h`) -**Purpose:** Character I/O for REPL and diagnostics - -**Critical for StarForth:** The REPL must work on all platforms for interactive experimentation. - -**Platform challenges:** -- **Linux:** stdin/stdout are perfect -- **Kernel:** UART 16550 is well-documented but framebuffer is tricky - -### 5. **CPU** (`hal_cpu.h`) -**Purpose:** CPU ID, relax/halt, SMP coordination - -**Critical for StarForth:** Per-core execution heat tracking (future), SMP scalability (future). - -**Platform challenges:** -- **Linux:** pthread local storage -- **Kernel:** Local APIC, per-core stacks, CPU-local storage - ---- - -## HAL and the Physics Subsystems - -The physics-driven adaptive runtime is the **primary beneficiary** of the HAL: - -| Physics Subsystem | HAL Dependency | Why | -|-------------------|----------------|-----| -| **Execution Heat** | None (pure VM state) | Tracks word execution frequency | -| **Rolling Window** | `hal_time_now_ns()` | Timestamps for window entries | -| **Hot-Words Cache** | None (pure dictionary state) | Frequency-based reordering | -| **Pipelining Metrics** | None (pure transition tracking) | Word-to-word prediction | -| **Inference Engine** | `hal_time_now_ns()` | Calibration timing, ANOVA | -| **Heartbeat** | `hal_timer_periodic()`, `hal_interrupt.*` | ISR-based sampling | - -**Key insight:** Only 2 of 6 subsystems touch the HAL directly, and only via clean interfaces. This preserves deterministic behavior while enabling kernel deployment. - ---- - -## HAL and LithosAnanke - -LithosAnanke is a **new platform implementation** of the HAL: - -``` -src/platform/kernel/ -├── boot/ -│ └── uefi_loader.c # UEFI entry point → BootInfo handoff -├── hal_time.c # TSC + HPET + APIC timer -├── hal_interrupt.c # IDT + Local APIC + IOAPIC -├── hal_memory.c # PMM + VMM + kmalloc -├── hal_console.c # UART 16550 + framebuffer -├── hal_cpu.c # SMP + CPU-local storage -└── drivers/ # PCI, AHCI, NVMe, VirtIO, etc. -``` - -**LithosAnanke does NOT modify:** -- VM core (`src/vm.c`) -- Physics subsystems (`src/dictionary_heat_optimization.c`, etc.) -- Word implementations (`src/word_source/*.c`) - -**LithosAnanke ONLY implements:** -- HAL interfaces (`hal_*.c`) -- UEFI boot loader (`boot/uefi_loader.c`) -- Device drivers (`drivers/*.c`) - -This is the **proof** that the HAL abstraction works: LithosAnanke is a pure platform layer, not a VM fork. - ---- - -## HAL and StarshipOS - -StarshipOS builds on LithosAnanke by adding: -- **Process model** - Forth tasks vs. traditional processes -- **Filesystem** - FAT32, ext2, or log-structured -- **Networking** - TCP/IP stack, DHCP, DNS -- **Device model** - Unified block/net/char device interfaces -- **Security model** - Capabilities, ACL, Forth-based access control - -All of these **still use the HAL** for low-level access: -- Filesystem → `hal_memory` for caching, `hal_interrupt` for async I/O -- Networking → `hal_interrupt` for packet RX, `hal_time` for timeouts -- Device model → HAL as the common substrate - -**The HAL is not just a kernel bootstrapping tool—it's the foundation for the entire OS.** - ---- - -## Migration Strategy - -Current StarForth codebase must be refactored to introduce the HAL: - -### Phase 1: Define HAL Interfaces -- Write `include/hal/*.h` headers with contracts -- Document semantics, error handling, performance expectations -- **No implementation yet** - -### Phase 2: Refactor Existing Platforms -- Create `src/platform/linux/hal_*.c` implementing HAL -- Create `src/platform/l4re/hal_*.c` implementing HAL -- Update VM code to call HAL instead of platform-specific APIs -- **VM must still build and pass all 936+ tests** - -### Phase 3: Validate HAL -- Build and test on Linux -- Build and test on L4Re (if available) -- Verify deterministic behavior (0% algorithmic variance) still holds -- Benchmark: HAL must not add measurable overhead - -### Phase 4: Implement LithosAnanke Platform -- Create `src/platform/kernel/` with HAL implementations -- UEFI boot loader -- Basic MM, console, timing -- **Goal:** Boot to `ok` prompt on QEMU/OVMF - -### Phase 5: Full LithosAnanke -- Complete HAL implementations (interrupts, SMP, drivers) -- Heartbeat ISR running at kernel level -- Physics subsystems operational -- **Goal:** Reproduce DoE results on bare metal - -### Phase 6: StarshipOS -- Process model, filesystem, networking -- Forth as native control plane -- **Goal:** Self-hosting OS - ---- - -## Success Criteria - -The HAL is successful if: - -1. ✅ **VM core has zero platform-specific code** -2. ✅ **All 936+ tests pass on Linux, L4Re, and LithosAnanke** -3. ✅ **0% algorithmic variance maintained across platforms** -4. ✅ **No measurable performance regression from HAL abstraction** -5. ✅ **LithosAnanke boots to `ok` prompt and runs REPL** -6. ✅ **Heartbeat subsystem works identically on all platforms** -7. ✅ **New platforms can be added without touching VM code** - ---- - -## Next Steps - -See companion documentation: -- `interfaces.md` - Detailed HAL interface specifications -- `platform-implementations.md` - How to implement HAL for new platforms -- `migration-plan.md` - Step-by-step refactoring guide -- `lithosananke-integration.md` - LithosAnanke-specific implementation details - ---- - -## References - -- StarForth VM core: `src/vm.c` -- Heartbeat system: `docs/03-architecture/heartbeat-system/` -- Physics feedback loops: `docs/FEEDBACK_LOOPS.md` -- Platform abstraction (current): `src/platform/` \ No newline at end of file diff --git a/docs/lithosananke/hal/platform-implementations.md b/docs/lithosananke/hal/platform-implementations.md deleted file mode 100644 index 9ad4e538..00000000 --- a/docs/lithosananke/hal/platform-implementations.md +++ /dev/null @@ -1,729 +0,0 @@ -# HAL Platform Implementation Guide - -## Overview - -This document explains how to implement the HAL for a new platform. It provides detailed examples for Linux, L4Re, and LithosAnanke (freestanding) implementations. - -> **HISTORICAL — L4Re/Fiasco.OC**: the L4Re examples below describe a target that was a supported platform through mid-2026. L4Re support has since been removed as an active target — `src/platform/l4re/time.c` and related `#ifdef __l4__` code are retained for reference but no longer wired into any build. - -**Target audience:** Platform developers adding HAL support for a new target. - ---- - -## Platform Directory Structure - -``` -src/platform// -├── hal_time.c # Time & timer HAL implementation -├── hal_interrupt.c # Interrupt HAL implementation -├── hal_memory.c # Memory HAL implementation -├── hal_console.c # Console HAL implementation -├── hal_cpu.c # CPU HAL implementation -├── hal_panic.c # Panic/error HAL implementation -└── platform_init.c # Platform-specific initialization (optional) -``` - -### Build Integration - -Each platform is compiled independently via Makefile: - -```makefile -# Platform selection -PLATFORM ?= linux # linux | l4re | kernel - -# Platform-specific source files -PLATFORM_SOURCES = \ - src/platform/$(PLATFORM)/hal_time.c \ - src/platform/$(PLATFORM)/hal_interrupt.c \ - src/platform/$(PLATFORM)/hal_memory.c \ - src/platform/$(PLATFORM)/hal_console.c \ - src/platform/$(PLATFORM)/hal_cpu.c \ - src/platform/$(PLATFORM)/hal_panic.c - -# Platform-specific flags -ifeq ($(PLATFORM),kernel) - CFLAGS += -ffreestanding -nostdlib -mno-red-zone - LDFLAGS += -nostdlib -static -endif -``` - ---- - -## Implementation Examples - -### 1. Linux Platform (`src/platform/linux/`) - -Linux is the **reference platform** for development and testing. It provides the richest debugging environment. - -#### `hal_time.c` (Linux) - -```c -#include "hal/hal_time.h" -#include -#include -#include -#include - -/* Timer state */ -typedef struct { - timer_t timer_id; - hal_timer_callback_t callback; - void *ctx; -} linux_timer_t; - -static linux_timer_t *periodic_timers = NULL; -static int timer_count = 0; - -void hal_time_init(void) { - /* Linux: clock_gettime is always available */ - struct timespec ts; - if (clock_gettime(CLOCK_MONOTONIC, &ts) != 0) { - hal_panic("hal_time_init: CLOCK_MONOTONIC unavailable"); - } -} - -uint64_t hal_time_now_ns(void) { - struct timespec ts; - clock_gettime(CLOCK_MONOTONIC, &ts); - return (uint64_t)ts.tv_sec * 1000000000ULL + ts.tv_nsec; -} - -void hal_time_delay_ns(uint64_t ns) { - struct timespec req; - req.tv_sec = ns / 1000000000ULL; - req.tv_nsec = ns % 1000000000ULL; - nanosleep(&req, NULL); -} - -/* Signal handler for periodic timer */ -static void timer_signal_handler(int signo, siginfo_t *si, void *uc) { - (void)signo; - (void)uc; - - linux_timer_t *timer = (linux_timer_t *)si->si_value.sival_ptr; - if (timer && timer->callback) { - timer->callback(timer->ctx); - } -} - -int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) { - /* Allocate timer structure */ - linux_timer_t *timer = malloc(sizeof(linux_timer_t)); - if (!timer) return -1; - - timer->callback = callback; - timer->ctx = ctx; - - /* Set up signal handler */ - struct sigaction sa; - memset(&sa, 0, sizeof(sa)); - sa.sa_flags = SA_SIGINFO; - sa.sa_sigaction = timer_signal_handler; - sigemptyset(&sa.sa_mask); - sigaction(SIGRTMIN, &sa, NULL); - - /* Create POSIX timer */ - struct sigevent sev; - memset(&sev, 0, sizeof(sev)); - sev.sigev_notify = SIGEV_SIGNAL; - sev.sigev_signo = SIGRTMIN; - sev.sigev_value.sival_ptr = timer; - - if (timer_create(CLOCK_MONOTONIC, &sev, &timer->timer_id) != 0) { - free(timer); - return -1; - } - - /* Arm the timer */ - struct itimerspec its; - its.it_value.tv_sec = period_ns / 1000000000ULL; - its.it_value.tv_nsec = period_ns % 1000000000ULL; - its.it_interval = its.it_value; /* Periodic */ - - if (timer_settime(timer->timer_id, 0, &its, NULL) != 0) { - timer_delete(timer->timer_id); - free(timer); - return -1; - } - - return timer_count++; /* Return timer ID */ -} - -int hal_timer_cancel(int timer_id) { - /* TODO: Implement timer cancellation */ - (void)timer_id; - return 0; -} - -uint64_t hal_time_frequency_hz(void) { - /* Linux: CLOCK_MONOTONIC is variable frequency, return 0 */ - return 0; -} -``` - -#### `hal_interrupt.c` (Linux) - -```c -#include "hal/hal_interrupt.h" -#include - -/* Linux uses signals as "interrupts" */ -static __thread int in_signal_handler = 0; - -void hal_interrupt_init(void) { - /* No initialization needed on Linux */ -} - -void hal_irq_enable(void) { - /* Linux: interrupts (signals) always enabled */ -} - -unsigned long hal_irq_disable(void) { - /* Linux: can't disable signals, return dummy state */ - return 0; -} - -void hal_irq_restore(unsigned long state) { - (void)state; -} - -int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) { - /* Linux: IRQs are handled via signal mechanism in hal_timer_periodic() */ - (void)irq; - (void)isr; - (void)ctx; - return 0; -} - -int hal_irq_unregister(unsigned int irq) { - (void)irq; - return 0; -} - -int hal_in_interrupt_context(void) { - return in_signal_handler; -} -``` - -#### `hal_memory.c` (Linux) - -```c -#include "hal/hal_memory.h" -#include -#include - -void hal_mem_init(void) { - /* malloc is always available on Linux */ -} - -void *hal_mem_alloc(size_t size) { - if (size == 0) return NULL; - void *ptr = malloc(size); - if (ptr) { - memset(ptr, 0, size); /* Zero-initialize */ - } - return ptr; -} - -void hal_mem_free(void *ptr) { - free(ptr); -} - -uint64_t hal_mem_alloc_pages(size_t count) { - /* Linux: just allocate from heap (no separate PMM) */ - size_t size = count * hal_mem_page_size(); - return (uint64_t)hal_mem_alloc(size); -} - -void hal_mem_free_pages(uint64_t paddr, size_t count) { - (void)count; - hal_mem_free((void *)paddr); -} - -int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) { - /* Linux: identity mapping, no-op */ - (void)vaddr; - (void)paddr; - (void)size; - (void)flags; - return 0; -} - -size_t hal_mem_page_size(void) { - return 4096; /* Standard page size */ -} -``` - -#### `hal_console.c` (Linux) - -```c -#include "hal/hal_console.h" -#include -#include -#include - -void hal_console_init(void) { - /* stdin/stdout always available */ -} - -void hal_console_putc(char c) { - putchar(c); - fflush(stdout); /* Ensure character is visible */ -} - -void hal_console_puts(const char *s) { - if (s) { - fputs(s, stdout); - fflush(stdout); - } -} - -int hal_console_getc(void) { - return getchar(); -} - -int hal_console_has_input(void) { - struct pollfd pfd; - pfd.fd = STDIN_FILENO; - pfd.events = POLLIN; - return poll(&pfd, 1, 0) > 0; -} -``` - -#### `hal_cpu.c` (Linux) - -```c -#include "hal/hal_cpu.h" -#include -#include - -void hal_cpu_init(void) { - /* No initialization needed */ -} - -unsigned int hal_cpu_id(void) { - /* Linux: return thread ID (not true CPU ID) */ - return 0; /* Single-threaded for now */ -} - -void hal_cpu_relax(void) { - sched_yield(); -} - -void hal_cpu_halt(void) { - pause(); /* Wait for signal */ -} - -unsigned int hal_cpu_count(void) { - return sysconf(_SC_NPROCESSORS_ONLN); -} -``` - -#### `hal_panic.c` (Linux) - -```c -#include "hal/hal_panic.h" -#include -#include - -void hal_panic(const char *msg) { - fprintf(stderr, "PANIC: %s\n", msg ? msg : "unknown error"); - fflush(stderr); - abort(); -} -``` - ---- - -### 2. Kernel Platform (`src/platform/kernel/`) - -The kernel platform is **freestanding**—no libc, no OS, direct hardware access. - -#### `hal_time.c` (Kernel) - -```c -#include "hal/hal_time.h" -#include "hal/hal_interrupt.h" -#include "kernel/apic.h" -#include "kernel/hpet.h" -#include "kernel/tsc.h" - -/* TSC calibration state */ -static uint64_t tsc_frequency_hz = 0; -static uint64_t tsc_offset_ns = 0; - -void hal_time_init(void) { - /* Calibrate TSC using HPET */ - tsc_frequency_hz = tsc_calibrate_hpet(); - - if (tsc_frequency_hz == 0) { - hal_panic("hal_time_init: TSC calibration failed"); - } - - /* Validate monotonicity */ - uint64_t t1 = hal_time_now_ns(); - for (volatile int i = 0; i < 1000; i++); - uint64_t t2 = hal_time_now_ns(); - - if (t2 < t1) { - hal_panic("hal_time_init: TSC not monotonic"); - } -} - -uint64_t hal_time_now_ns(void) { - /* Read TSC and convert to nanoseconds */ - uint64_t tsc = rdtsc(); - return tsc_to_ns(tsc) - tsc_offset_ns; -} - -void hal_time_delay_ns(uint64_t ns) { - uint64_t end = hal_time_now_ns() + ns; - while (hal_time_now_ns() < end) { - hal_cpu_relax(); - } -} - -/* APIC timer state */ -static hal_timer_callback_t apic_timer_callback = NULL; -static void *apic_timer_ctx = NULL; - -/* APIC timer ISR (called from IDT) */ -void apic_timer_isr(void) { - if (apic_timer_callback) { - apic_timer_callback(apic_timer_ctx); - } - apic_eoi(); /* Send End-Of-Interrupt */ -} - -int hal_timer_periodic(uint64_t period_ns, hal_timer_callback_t callback, void *ctx) { - if (period_ns < 1000) return -2; /* Minimum 1µs */ - - apic_timer_callback = callback; - apic_timer_ctx = ctx; - - /* Configure APIC timer */ - uint64_t ticks = (period_ns * apic_timer_frequency_hz()) / 1000000000ULL; - apic_timer_periodic(ticks); - - return 1; /* Timer ID */ -} - -int hal_timer_cancel(int timer_id) { - (void)timer_id; - apic_timer_stop(); - apic_timer_callback = NULL; - return 0; -} - -uint64_t hal_time_frequency_hz(void) { - return tsc_frequency_hz; -} -``` - -#### `hal_interrupt.c` (Kernel) - -```c -#include "hal/hal_interrupt.h" -#include "kernel/idt.h" -#include "kernel/apic.h" - -/* Per-CPU interrupt nesting count */ -static __thread unsigned int interrupt_depth = 0; - -void hal_interrupt_init(void) { - /* Initialize IDT */ - idt_init(); - - /* Initialize Local APIC */ - apic_init(); - - /* Disable legacy PIC (use APIC instead) */ - pic_disable(); -} - -void hal_irq_enable(void) { - __asm__ volatile("sti"); -} - -unsigned long hal_irq_disable(void) { - unsigned long flags; - __asm__ volatile( - "pushfq\n" - "popq %0\n" - "cli\n" - : "=r"(flags) - ); - return flags; -} - -void hal_irq_restore(unsigned long state) { - __asm__ volatile( - "pushq %0\n" - "popfq\n" - : - : "r"(state) - ); -} - -/* ISR table (256 entries for x86_64 IDT) */ -static hal_isr_t isr_table[256] = {0}; -static void *isr_ctx_table[256] = {0}; - -/* Common ISR entry point (called from IDT stubs) */ -void common_isr_handler(unsigned int vector) { - interrupt_depth++; - - if (isr_table[vector]) { - isr_table[vector](vector, isr_ctx_table[vector]); - } - - interrupt_depth--; - apic_eoi(); -} - -int hal_irq_register(unsigned int irq, hal_isr_t isr, void *ctx) { - if (irq >= 256) return -1; - - isr_table[irq] = isr; - isr_ctx_table[irq] = ctx; - - /* Configure IOAPIC to route IRQ to Local APIC */ - ioapic_route_irq(irq, 0); /* Route to CPU 0 */ - - return 0; -} - -int hal_irq_unregister(unsigned int irq) { - if (irq >= 256) return -1; - - isr_table[irq] = NULL; - isr_ctx_table[irq] = NULL; - - return 0; -} - -int hal_in_interrupt_context(void) { - return interrupt_depth > 0; -} -``` - -#### `hal_memory.c` (Kernel) - -```c -#include "hal/hal_memory.h" -#include "kernel/pmm.h" /* Physical Memory Manager */ -#include "kernel/vmm.h" /* Virtual Memory Manager */ -#include "kernel/kmalloc.h" /* Kernel heap allocator */ - -void hal_mem_init(void) { - /* Initialize physical memory manager (parse UEFI memory map) */ - pmm_init(); - - /* Initialize virtual memory manager (set up page tables) */ - vmm_init(); - - /* Initialize kernel heap allocator */ - kmalloc_init(); -} - -void *hal_mem_alloc(size_t size) { - if (size == 0) return NULL; - return kmalloc(size); /* Kernel heap allocator */ -} - -void hal_mem_free(void *ptr) { - kfree(ptr); -} - -uint64_t hal_mem_alloc_pages(size_t count) { - return pmm_alloc_pages(count); -} - -void hal_mem_free_pages(uint64_t paddr, size_t count) { - pmm_free_pages(paddr, count); -} - -int hal_mem_map(uint64_t vaddr, uint64_t paddr, size_t size, unsigned int flags) { - /* Translate HAL flags to page table flags */ - unsigned int pt_flags = 0; - if (flags & HAL_MEM_WRITE) pt_flags |= PAGE_WRITE; - if (flags & HAL_MEM_EXEC) pt_flags |= PAGE_EXEC; - if (flags & HAL_MEM_NOCACHE) pt_flags |= PAGE_NOCACHE; - - return vmm_map(vaddr, paddr, size, pt_flags); -} - -size_t hal_mem_page_size(void) { - return 4096; -} -``` - -#### `hal_console.c` (Kernel) - -```c -#include "hal/hal_console.h" -#include "kernel/uart.h" -#include "kernel/framebuffer.h" - -void hal_console_init(void) { - /* Initialize UART 16550 (serial console) */ - uart_init(0x3F8, 115200); /* COM1, 115200 baud */ - - /* Initialize framebuffer (if available from UEFI GOP) */ - fb_init(); -} - -void hal_console_putc(char c) { - /* Output to both serial and framebuffer */ - uart_putc(c); - fb_putc(c); -} - -void hal_console_puts(const char *s) { - if (!s) return; - while (*s) { - hal_console_putc(*s++); - } -} - -int hal_console_getc(void) { - /* Read from UART (blocking) */ - return uart_getc(); -} - -int hal_console_has_input(void) { - return uart_has_data(); -} -``` - -#### `hal_cpu.c` (Kernel) - -```c -#include "hal/hal_cpu.h" -#include "kernel/apic.h" - -static unsigned int cpu_count = 1; - -void hal_cpu_init(void) { - /* Detect CPU features */ - cpu_detect_features(); - - /* Bring up secondary CPUs (SMP) */ - cpu_count = smp_init(); -} - -unsigned int hal_cpu_id(void) { - /* Get Local APIC ID */ - return apic_get_id(); -} - -void hal_cpu_relax(void) { - __asm__ volatile("pause"); -} - -void hal_cpu_halt(void) { - __asm__ volatile("hlt"); -} - -unsigned int hal_cpu_count(void) { - return cpu_count; -} -``` - ---- - -## Platform Testing Strategy - -### 1. **Unit Tests (Per-Platform)** - -Each platform should have unit tests for HAL functions: - -```c -/* test/platform/linux/hal_time_test.c */ -void test_hal_time_monotonic(void) { - hal_time_init(); - - uint64_t t1 = hal_time_now_ns(); - hal_time_delay_ns(1000000); /* 1ms */ - uint64_t t2 = hal_time_now_ns(); - - assert(t2 > t1); - assert((t2 - t1) >= 1000000); - assert((t2 - t1) < 2000000); /* Allow 100% jitter for test */ -} -``` - -### 2. **Cross-Platform VM Tests** - -VM tests should run identically on all platforms: - -```bash -# Linux -make PLATFORM=linux test - -# L4Re -make PLATFORM=l4re test - -# Kernel (QEMU) -make PLATFORM=kernel test-qemu -``` - -**Success criteria:** All 936+ tests pass on all platforms. - -### 3. **Determinism Validation** - -Physics subsystems must show 0% algorithmic variance: - -```bash -# Run DoE on each platform -for platform in linux l4re kernel; do - make PLATFORM=$platform fastest - ./build/$ARCH/fastest/starforth --doe > results_$platform.csv -done - -# Compare results (should be identical) -diff results_linux.csv results_kernel.csv -``` - ---- - -## Common Implementation Pitfalls - -### 1. **Non-Monotonic Time** - -**Symptom:** Physics metrics show negative deltas, rolling window corrupted. - -**Cause:** TSC goes backward (multi-core, frequency scaling). - -**Fix:** Use `rdtscp` instead of `rdtsc`, synchronize TSC across cores, or fall back to HPET. - -### 2. **Timer Jitter** - -**Symptom:** Heartbeat intervals vary by 10-50%, metrics show high variance. - -**Cause:** Signal latency (Linux), interrupt latency (kernel). - -**Fix:** Minimize ISR work, use high-priority timer interrupt, disable frequency scaling. - -### 3. **Memory Leaks** - -**Symptom:** `hal_mem_alloc()` calls grow without matching `hal_mem_free()`. - -**Cause:** Missing free in error paths, lost pointers. - -**Fix:** Valgrind on Linux, manual leak tracking on kernel. - -### 4. **Interrupt Context Violations** - -**Symptom:** Crashes, hangs, or corruption when calling blocking HAL functions from ISR. - -**Cause:** Calling `hal_mem_alloc()` or `hal_console_getc()` from timer ISR. - -**Fix:** Assert `!hal_in_interrupt_context()` in blocking functions. - ---- - -## Next Steps - -- See `migration-plan.md` for refactoring existing code -- See `lithosananke-integration.md` for kernel-specific details \ No newline at end of file diff --git a/docs/working/architecture/03-architecture/word-acl/DESIGN.md b/docs/working/architecture/03-architecture/word-acl/DESIGN.md deleted file mode 100644 index 4eb50dec..00000000 --- a/docs/working/architecture/03-architecture/word-acl/DESIGN.md +++ /dev/null @@ -1,393 +0,0 @@ - -# Word-Level ACL System - -**Status:** Implemented through Phase 6 — Phase 7 (LithosAnanke parity) remaining -**Target branch:** `master` complete; `lithosananke` parity next -**Implementation files:** `capsules/ACL.4th`, `capsules/zuse.4th`, `src/word_source/acl_words.c`, `src/test_runner/modules/acl_words_test.c` - ---- - -## Summary - -### Overview - -A word-level access control system for StarForth VMs, implemented as a -combination of a thin C infrastructure layer and a pure-FORTH policy capsule -(`ACL.4th`). The C layer provides four `DictEntry` fields for the hot path -(TTL counter + allow bit + mode + pin). All policy, all adaptive logic, all -inheritance rules live as colon definitions in `ACL.4th`. A bootstrap -superuser (`zuse.4th`) provides the sole authenticated escalation path. - ---- - -### ACL Disposition — Three States - -Each word in the dictionary has one of three enforcement dispositions: - -| State | Behavior | -|-------|---------| -| `STRICT` | ACL re-checked on every execution | -| `TTL` | ACL cached; re-checked only when TTL counter reaches zero | -| `PINNED` | Current mode and decision frozen permanently; one-way ratchet | - ---- - -### TTL — Statistically Adaptive - -The TTL is not a fixed value. It is derived and continuously updated from -the existing SSM execution physics: - -- **Heat** (`execution_heat`): hotter words earn longer TTLs; check cost - amortized over more executions -- **Rolling window**: sudden shift in caller pattern or access rate causes - TTL to shrink aggressively -- **Decay**: quiescent words pull their TTL back down so the next burst - re-validates early -- **Inference engine** (L5/L6): detects oscillating TTL and stabilizes it — - same CV threshold logic already used for window-width inference - -`PINNED` is the asymptote: the adaptive process converges, you thumbtack -it, and the accumulator freezes permanently. - -A FORTH security word selects enforcement mode per word: - -```forth -' MY-WORD ACL-STRICT \ check every execution -' MY-WORD ACL-TTL-MODE \ use adaptive TTL -' MY-WORD ACL-PIN \ freeze — mode and decision immutable forever -``` - ---- - -### The Thumbtack / Pin Flag - -`ACL-PIN ( xt -- )` is a one-way ratchet: - -- Transitions: `STRICT → PINNED` or `TTL → PINNED` — never back -- Once pinned, `acl_mode` and `acl_ttl` are immutable for that VM context -- Any attempt to change a pinned word's ACL is silently ignored (or errors, - depending on policy) -- Kernel primitive words (`BIRTH`, `EXEC`, `BYE`, etc.) are pinned by Mama - at boot before the first `BIRTH` fires - ---- - -### Inheritance at Birth - -When a child VM is born: - -``` -acl_mode = parent->acl_mode \ policy propagates (STRICT or TTL) -acl_pinned = 0 \ always clear — child owns its own lock -acl_ttl = default \ reset; child has no execution history -acl_decision = ALLOW \ re-evaluated on first access -``` - -Key properties: - -- **Pin is contextual, not viral.** A parent's pinned words do not force - the child to be pinned. The child inherits the mode as a starting point - and can tighten, relax, or re-pin freely. -- **Security lattice flows downward at birth only.** After birth, each VM's - ACL state is independent. A compromised child cannot bootstrap its way - back to Mama's pinned ACLs. - ---- - -### Interpreter Hook — Two-Level Check - -The C interpreter reads only two fields per `DictEntry`: - -```c -if (vm->emergency_console) goto execute; /* 100% bypass — physical access */ -if (entry->acl_ttl-- > 0) goto execute; /* TTL good — single decrement */ -acl_recheck(vm, entry); /* TTL=0: call FORTH ACL-RECHECK */ -if (!entry->acl_allow) goto reject; -``` - -**Hot path cost:** one decrement and a branch — essentially free. -**Cold path:** calls `ACL-RECHECK` in `ACL.4th`, which recomputes the -adaptive TTL, updates `acl_allow`, and resets the counter. The C side -never reasons about policy; it only reads the result. - ---- - -### Two-Console Architecture - -Two permanent, independent console layers exist at all times: - -#### 1. Emergency Console (always present) - -- `emergency_console` (`uint8_t`) flag in `VM` struct — C-only write; no - FORTH word sets it -- Checked first in the interpreter hot path — 100% ACL bypass -- `acl_recheck()` uses save/restore around it for re-entrancy protection -- `EMERGENCY_CONSOLE_ENABLED` Makefile build flag (default=1): when set to - 0, strips interactive fallthrough; `vm_fault_handler()` weak symbol is - called instead (override for hardware reset, watchdog, debug probe) -- `BYE` returns to the emergency console -- Only `panic` kills the VM -- Prompt: `ok>` - -#### 2. Zuse Console (omnipresent, awaiting authentication) - -- `zuse_session` (`uint8_t`) flag in `VM` struct — C-only write; no FORTH - word sets it -- Prompt: `zuse)ok>` -- Full ACL bypass when a Zuse session is active -- Completely independent of `emergency_console` — setting one does not - affect the other - ---- - -### Superuser: Zuse - -Named for Konrad Zuse, pioneer of programmable computers. Zuse is the -bootstrap superuser: the sole entity that can own the zuse console and mint -user credentials. - -- Defined in `capsules/zuse.4th`, loaded by `ACL.4th` at capsule boot -- Software-only for now (no thumbdrive); `zuse.4th` is a capsule citizen - from day one -- Zuse's words are pinned by `ACL-ZUSE-BOOT` at load time -- `ACL-BOOT` is called first, then `S" zuse.4th" EXEC` — both from within - `ACL.4th` itself (Block 2067) -- Future: replace with thumbdrive PKI (Ed25519 challenge-response); - `zuse.4th` becomes the bootstrapper that validates the physical drive - ---- - -### CA Root - -- Embedded in `ACL.4th` (Block 2066) as constants `ACL-CA-KEY-LO` / - `ACL-CA-KEY-HI` -- Capsule hash = root-of-trust fingerprint: any CA key change changes the - capsule hash and the birth protocol detects tampering -- Placeholder (zeros) until `tools/mkcapsule` embeds the real Ed25519 key - at build time - ---- - -### Security Model / No-Security Condition - -Security is opt-in via `init.4th`. The toggle is one commented-out line: - -```forth -\ S" ACL.4th" EXEC -``` - -| `ACL.4th` | `zuse.4th` | Result | -|-----------|------------|--------| -| absent | absent | No security — open dev mode | -| present | absent | ACL enforced; emergency REPL locked until Zuse provisioned | -| present | present | Full lockdown; Zuse owns the zuse console | - -Uncomment the line in `init.4th` to enable full lockdown. Leave it -commented out for development builds. - ---- - -### Self-Activating Capsule - -`ACL.4th` is self-activating — `init.4th` only needs one line: - -```forth -S" ACL.4th" EXEC -``` - -Internal structure of `ACL.4th`: -- **Block 2066**: CA root placeholder constants (`ACL-CA-KEY-LO` / - `ACL-CA-KEY-HI`) -- **Block 2067**: Calls `ACL-BOOT` then `S" zuse.4th" EXEC` - -`ACL-BOOT` stamps default ACL entries on all existing dictionary words. -After it runs, every subsequent `:` definition gets an ACL entry via the -defining-word hook. No word can exist without an ACL entry. - ---- - -### Emergency Console — 100% Bypass - -`vm->emergency_console` is set by the C layer when the physical `ok>` REPL -is active. It is checked first, before any ACL table lookup. Physical -presence always wins. The emergency console is never subject to ACL -constraints regardless of what `ACL.4th` defines. - -The `EMERGENCY_CONSOLE_ENABLED=0` build flag strips the interactive -fallthrough for production or embedded builds, routing faults to -`vm_fault_handler()` instead. - ---- - -### ACL.4th — Pure FORTH Implementation - -The ACL table is a `CREATE`d FORTH array indexed by execution token (XT). -`'` (tick) gives the XT of any word; XT is just a cell value usable as a -table key. - -Core words: - -| Word | Stack | Description | -|------|-------|-------------| -| `ACL-ENTRY` | `( xt -- addr )` | O(1) table lookup by XT | -| `ACL-MODE@` | `( xt -- mode )` | read enforcement mode | -| `ACL-MODE!` | `( mode xt -- )` | set mode (no-op if pinned) | -| `ACL-PINNED?` | `( xt -- flag )` | test pin bit | -| `ACL-PIN` | `( xt -- )` | set pin — one-way, irreversible | -| `ACL-STRICT` | `( xt -- )` | set STRICT mode (no-op if pinned) | -| `ACL-TTL-MODE` | `( xt -- )` | set TTL mode (no-op if pinned) | -| `ACL-TTL@` | `( xt -- n )` | read current TTL counter | -| `ACL-TTL!` | `( n xt -- )` | write TTL counter | -| `ACL-ALLOW@` | `( xt -- flag )` | read cached allow/deny decision | -| `ACL-ALLOW!` | `( flag xt -- )` | write allow/deny decision | -| `ACL-INHERIT` | `( src dst -- )` | birth inheritance: copy mode, clear pin, reset ttl+decision | -| `ACL-RECHECK` | `( xt -- )` | adaptive TTL recomputation; updates allow + new TTL | -| `ACL-INIT-PRIMITIVES` | `( -- )` | bulk-initialize ACL entries for all existing dictionary words | - -Example pin at boot in `init.4th`: - -```forth -' BIRTH ACL-STRICT ' BIRTH ACL-PIN -' EXEC ACL-STRICT ' EXEC ACL-PIN -' BYE ACL-STRICT ' BYE ACL-PIN -``` - ---- - -### Bootstrap Sequence - -```forth -\ In init.4th (one line, opt-in toggle — uncomment to enable): -S" ACL.4th" EXEC \ self-activating: calls ACL-BOOT then loads zuse.4th - -\ ACL.4th Block 2067 does internally: -ACL-BOOT \ stamp default ACLs on all existing words -S" zuse.4th" EXEC \ load and pin Zuse's words - -\ Subsequent capsule loads (e.g. doe.4th) get ACL entries via : hook -S" doe.4th" EXEC -``` - -Every `:` definition after `ACL-BOOT` runs creates its own ACL entry at -definition time via a hook in the defining word. No word is ever born -without an ACL entry. - ---- - -### Capsule Namespace - -| File | Role | -|------|------| -| `init.4th` | Mama VM personality (default); contains opt-in ACL toggle | -| `init-*.4th` | Alternate personalities | -| `doe.4th` | DoE experiment harness | -| `ACL.4th` | Word-level ACL subsystem — self-activating; loads `zuse.4th` | -| `zuse.4th` | Bootstrap superuser personality; words pinned by `ACL-ZUSE-BOOT` | -| `std-blob.4th` | Standard library layer (future) | - ---- - -## Implementation Punch List - -### Phase 1 — C Infrastructure (`master`) ✅ COMPLETE - -- [x] Add `acl_ttl` (`uint32_t`) to `DictEntry` in `include/vm.h` -- [x] Add `acl_allow` (`uint8_t`) to `DictEntry` -- [x] Add `acl_mode` (`uint8_t`: STRICT=0 / TTL=1) to `DictEntry` -- [x] Add `acl_pinned` (`uint8_t`) to `DictEntry` -- [x] Default-initialize all four fields in `word_register()`: - `ttl=ACL_OPEN, allow=1, mode=TTL, pinned=0` -- [x] Add `emergency_console` (`uint8_t`) flag to `VM` struct -- [x] Add `zuse_session` (`uint8_t`) flag to `VM` struct -- [x] Insert two-level ACL check into interpreter loop in `vm.c` -- [x] Implement `acl_recheck()` in `vm.c` — calls FORTH `ACL-RECHECK` when - TTL=0; save/restore `emergency_console` for re-entrancy protection -- [x] Add `:` definition hook to create ACL entry for each newly defined word - -### Phase 2 — C Primitive Words + `ACL.4th` Capsule (`master`) ✅ COMPLETE - -- [x] 12 C primitive words in `src/word_source/acl_words.c` -- [x] `CREATE ACL-TABLE` sized to max dictionary entries -- [x] `ACL-ENTRY ( xt -- addr )` — XT-indexed O(1) lookup -- [x] Field accessors: `ACL-MODE@`, `ACL-MODE!`, `ACL-PINNED?`, - `ACL-TTL@`, `ACL-TTL!`, `ACL-ALLOW@`, `ACL-ALLOW!` -- [x] `ACL-PIN ( xt -- )` — one-way ratchet; no-op if already pinned -- [x] `ACL-STRICT ( xt -- )` — set STRICT mode; no-op if pinned -- [x] `ACL-TTL-MODE ( xt -- )` — set TTL mode; no-op if pinned -- [x] `ACL-INHERIT ( src-xt dst-xt -- )` — copy mode, clear pin, - reset ttl+decision -- [x] `ACL-RECHECK ( xt -- )` — recompute adaptive TTL from heat and rolling - window; update `acl_allow` and reset counter -- [x] `ACL-INIT-PRIMITIVES` — walk dictionary, create default entry per word -- [x] Pin privileged words: `BIRTH`, `EXEC`, `BYE`, and other Mama-only words - -### Phase 3 — `capsules/ACL.4th` Self-Activating Capsule (`master`) ✅ COMPLETE - -- [x] `ACL.4th` is self-activating: Block 2067 calls `ACL-BOOT` then - `S" zuse.4th" EXEC` -- [x] Block 2066: CA root placeholder constants (`ACL-CA-KEY-LO` / - `ACL-CA-KEY-HI`) -- [x] `ACL-BOOT` stamps default ACL entries on all existing dictionary words -- [x] `capsules/zuse.4th` loaded at end of `ACL.4th` capsule -- [x] Zuse's words pinned by `ACL-ZUSE-BOOT` at load time - -### Phase 4 — `init.4th` Opt-In Toggle (`master`) ✅ COMPLETE - -- [x] `init.4th` has one commented-out line as the opt-in toggle: - `\ S" ACL.4th" EXEC` -- [x] Uncomment to enable full lockdown; leave commented for dev/open mode -- [x] No other changes to `init.4th` required — `ACL.4th` is self-contained - -### Phase 5 — POST Tests (`master`) ✅ COMPLETE - -POST tests implemented in `src/test_runner/modules/acl_words_test.c`, -registered in `src/test_runner/test_runner.c`. 800/800 passing. - -- [x] POST test: `ACL-PIN` is one-way — mode cannot change after pin set -- [x] POST test: inheritance — child entry has mode copied, pin cleared, - ttl and decision reset -- [x] POST test: emergency console bypass — `emergency_console=1` skips - ACL check entirely -- [x] POST test: STRICT mode — ACL re-evaluated on every execution -- [x] POST test: TTL hot path — TTL > 0 bypasses re-evaluation -- [x] POST test: adaptive accumulator — heat increase produces TTL increase -- [x] POST test: privileged words pinned at boot remain pinned after - `ACL-INIT-PRIMITIVES` - -### Phase 6 — Isabelle/HOL Formal Verification (`master`) ✅ COMPLETE - -Five `.thy` files in `proof/` alongside existing VM proofs: - -- [x] `ACL_Pin_Monotone.thy` — pin bit is set-only; no operation clears it - once set -- [x] `ACL_Inherit_Clears_Pin.thy` — `ACL-INHERIT` always produces an entry - with `acl_pinned = 0` regardless of source entry state -- [x] `ACL_TTL_Bounded.thy` — TTL counter is bounded above by - `ACL-TTL-COMPUTE` output; cannot grow unboundedly -- [x] `ACL_Emergency_Bypass.thy` — when `emergency_console = 1` the - allow/deny decision is never consulted -- [x] `ACL_No_Escalation.thy` — a child VM cannot produce a pinned entry - with higher privilege than its inherited mode - -### Phase 7 — LithosAnanke Parity (REMAINING) - -- [ ] Merge / port ACL subsystem to `lithosananke` branch -- [ ] Verify `ACL.4th` loads cleanly in kernel context (freestanding) -- [ ] `ACL-BOOT` runs at kernel boot before first `BIRTH` -- [ ] `vm->emergency_console` wired to kernel REPL active flag -- [ ] `vm->zuse_session` wired to kernel Zuse console authentication path -- [ ] Three-arch acceptance: amd64, aarch64, riscv64 boot to `ok>` with ACL - active and no regressions -- [ ] Commit acceptance logs - -### Phase 8 — PKI / Thumbdrive Authentication (FUTURE) - -- [ ] Ed25519 challenge-response authentication for Zuse thumbdrive -- [ ] `tools/mkcapsule` embeds real Ed25519 CA key into `ACL.4th` Block 2066 - at build time (replacing zero placeholder) -- [ ] User minting: admin creates thumbdrive with CA-signed certificate + home - block image -- [ ] Lose the drive → admin mints a new one; no software recovery path by - design -- [ ] `zuse.4th` becomes the bootstrapper that validates the physical drive - before granting Zuse session access diff --git a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/FORMAL_CLAIMS_FOR_REVIEWERS.txt b/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/FORMAL_CLAIMS_FOR_REVIEWERS.txt deleted file mode 100644 index 764128bb..00000000 --- a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/FORMAL_CLAIMS_FOR_REVIEWERS.txt +++ /dev/null @@ -1,345 +0,0 @@ -================================================================================ -FORMAL VERIFICATION CLAIMS: PHYSICS-DRIVEN SELF-ADAPTIVE FORTH-79 VM -================================================================================ - -For use in peer review submissions and formal verification papers. - -These claims are supported by the 90-run validation experiment and are -defensible against reviewer scrutiny. - -================================================================================ -CLAIM 1: ALGORITHMIC DETERMINISM -================================================================================ - -STATEMENT: -"The physics-driven adaptation model produces deterministic behavior, -enabling formal verification of the system's correctness." - -SUPPORTING EVIDENCE: -- Cache hit rates: 17.39% ± 0.00% (zero variance across 30 runs) -- Coefficient of variation (CV): 0% for all cache metrics -- Runtime behavior: Perfectly reproducible post-warmup - -FORMAL DEFINITION: -For all valid programs P and configurations C: - execute(P, C, t₁) ≡ execute(P, C, t₂) - -where both t₁ and t₂ are post-warmup (after 15+ runs) - -DEFENSE AGAINST REVIEWERS: -Q: "Isn't the variation just measurement noise?" -A: "We measured two types of variance: - (1) Cache hit rates: 0% internal variance - (2) Runtime: 60-70% variance (traced to OS scheduling, not algorithm) - - The separation proves the algorithm is deterministic." - -Q: "How do you know cache hits are truly consistent?" -A: "All 30 runs per configuration report identical hit rates (17.39%). - This is empirically proven, not assumed. Reproducible raw data available." - -METRIC FOR PAPER: "Algorithmic Consistency Guarantee: σ_cache = 0%" - -================================================================================ -CLAIM 2: ADAPTIVE CONVERGENCE -================================================================================ - -STATEMENT: -"The system demonstrates measurable convergence toward optimal configuration, -proving the adaptation mechanism is functional and effective." - -SUPPORTING EVIDENCE: -- C_FULL (adaptive): -25.4% improvement (early vs. late runs) -- B_CACHE (moderate): +0.5% stable (already optimal) -- A_BASELINE (non-adaptive): +6.4% degradation (no optimization possible) - -FORMAL DEFINITION: -Define: OptimalityGap(t) = P_optimal - P_actual(t) - -Convergence proof: - OptimalityGap(t₁₆₊) < OptimalityGap(t₁₋₁₅) - - Measured: 7.61 ms (late) < 10.20 ms (early) - Gap reduction: 2.59 ms (25.4% improvement) - -DEFENSE AGAINST REVIEWERS: -Q: "Isn't this just JIT warmup, not adaptive optimization?" -A: "Three points: - (1) Your warmup is already implemented (PRE/POST phases) - (2) A_BASELINE shows NO improvement despite same warmup - (3) B_CACHE shows minimal improvement - (4) ONLY C_FULL (with adaptation) shows 25% convergence - - This proves improvement is configuration-dependent." - -Q: "Why does A_BASELINE get slower?" -A: "Perfect! This validates our hypothesis: - - A_BASELINE has no adaptation mechanism → no improvement possible - - Slight degradation (+6.4%) is expected under sustained load - - This non-response proves the system is responding to its model, not random" - -Q: "Could this be better explained by cache warming?" -A: "Cache warming is included in PRE phase (your current implementation). - The C_FULL improvement occurs in LATE RUNS (16-30), not initial runs. - This is post-warmup adaptation, not initial cache fill." - -METRIC FOR PAPER: "Convergence Rate: ΔP/Δt = 25.4% / 15 runs ≈ 1.7% per run" - -================================================================================ -CLAIM 3: STABILITY UNDER ENVIRONMENTAL STRESS -================================================================================ - -STATEMENT: -"The algorithm maintains internal stability (0% variance) despite -external environmental noise (70% variance), proving robustness." - -SUPPORTING EVIDENCE: -- Internal variance (algorithm): Cache hits = 0% CV -- External variance (OS): Runtime = 60-70% CV -- Separation: Algorithm behavior independent of OS scheduling - -FORMAL DEFINITION: -Stability = Variance_internal / Variance_external - = 0% / 70% - = Perfect stability - -The algorithm output (cache hits) shows zero correlation with -environmental factors (OS scheduling), proving decoupling. - -DEFENSE AGAINST REVIEWERS: -Q: "Doesn't 70% variance make your system unpredictable?" -A: "No. This is the critical insight: - - 70% variance comes from OS scheduling (external) - - Algorithm itself has 0% variance (internal) - - The system has INFINITE stability margin. It is robust to - extreme environmental noise while maintaining algorithmic purity." - -Q: "How can you claim 0% variance when runtime varies 70%?" -A: "Excellent question. This shows the power of variance decomposition: - - Runtime variance = Algorithm variance + OS variance - 70% total CV = 0% algorithm + 70% OS - - The algorithm deterministically selects same cache configuration - on each run. OS scheduling then adds 70% variance on top. - - Analogy: A perfect algorithm running on a noisy OS." - -Q: "Isn't some of that 0% just lucky data?" -A: "We collected 60 data points for cache behavior (30 runs × B_CACHE + C_FULL). - All 60 show identical 17.39% hit rate. The probability of this by - chance is astronomically low." - -METRIC FOR PAPER: "Stability Margin: σ_algorithm / σ_environment = 0/70 = ∞" - -================================================================================ -CLAIM 4: QUANTIFIABLE & PREDICTABLE PERFORMANCE BOUNDS -================================================================================ - -STATEMENT: -"Performance is bounded within quantifiable ranges with 95% confidence, -enabling safe deployment under formal guarantees." - -SUPPORTING EVIDENCE: -- B_CACHE: 7.80 ± 2.52 ms → [5.28, 10.32] ms @ 95% confidence -- C_FULL: 8.91 ± 5.36 ms → [3.55, 14.27] ms @ 95% confidence -- Confidence intervals are mathematically justified (n=30 per config) - -FORMAL DEFINITION: -For any configuration C, performance P(C) satisfies: - P(|P(C) - μ(C)| ≤ 2σ(C)) ≥ 0.95 - -where μ and σ are empirically measured from 30 runs - -DEFENSE AGAINST REVIEWERS: -Q: "Why can you make 95% confidence claims on just 30 runs?" -A: "Two reasons: - (1) Sample size: Central Limit Theorem applies at n=30 - (2) Data distribution: Empirically approximately normal (verify with Q-Q plot) - (3) Mathematical backing: Chebyshev's inequality guarantees 75%, - normal distribution assumption gives 95% - - We're not claiming absolute truth, but 95% statistical confidence." - -Q: "Aren't confidence intervals just an illusion?" -A: "No. They're empirically grounded: - - Collected 30 independent samples per configuration - - Measured mean and variance directly - - Applied well-established statistical formulas - - Confidence interval reflects real data uncertainty - - If you rerun the experiment, your results should fall within - these bounds 95% of the time (on same hardware)." - -METRIC FOR PAPER: "Performance Bounds (95% confidence): - B_CACHE: 7.80 ± 5.04 ms - C_FULL: 8.91 ± 10.72 ms" - -================================================================================ -CLAIM 5: PHYSICS MODEL ENABLES FORMAL VERIFICATION -================================================================================ - -STATEMENT: -"The physics-based model (execution_heat tracking) is observable, -measurable, and enables formal verification of system properties." - -SUPPORTING EVIDENCE: -- Execution_heat is directly observable (word frequency counter) -- Cache promotion is deterministically triggered (threshold-based) -- Model state is reproducible (same program → same heat distribution) - -FORMAL DEFINITION: -Physics Model State Machine: - -State = {execution_heat[w] for all words w} -Input = {word sequence from program} -Output = {cache configuration} - -Transition rules: - For each word w executed: - execution_heat[w] ← execution_heat[w] + 1 - if execution_heat[w] ≥ THRESHOLD: - cache_config ← cache_config ∪ {w} - -DEFENSE AGAINST REVIEWERS: -Q: "This is just a simple counter - why call it 'physics-based'?" -A: "True, it's simple. But that's a feature, not a bug: - (1) Simplicity enables formal verification (you can axiomatize it) - (2) 'Physics-based' refers to the particle state machine framework - (3) Execution_heat is analogous to thermal energy (high heat → optimization) - (4) The model is deterministic and verifiable in Isabelle/HOL - - We chose simplicity for provability, not for complexity." - -Q: "Is this really novel?" -A: "The novelty is in the verification approach: - - Showing that a simple physics model can optimize a FORTH VM - - Empirically proving convergence and stability - - Formal framework connecting runtime model to observable behavior - - Variance decomposition as a verification technique - - Individual components (adaptive caching, frequency counting) are known. - The integrated verification story is novel." - -METRIC FOR PAPER: "Model Complexity: O(n) state, O(1) per transition" - -================================================================================ -SUMMARY: WHAT YOU CAN CLAIM TO REVIEWERS -================================================================================ - -✓ PROVABLE CLAIMS (Directly supported by data): - -1. "Our system is deterministic" - Support: 0% variance in algorithm output - -2. "Our system adapts toward optimal configuration" - Support: 25% improvement in C_FULL, no improvement in non-adaptive configs - -3. "Our system is stable" - Support: 0% internal variance despite 70% external variance - -4. "Our system has predictable performance" - Support: 95% confidence intervals empirically established - -5. "Our physics model is observable and verifiable" - Support: Execution_heat is directly measurable, formally definable - -✗ CLAIMS TO AVOID (Not directly supported): - -- "Our system is optimal" - (You've shown it converges toward optimal, not that it achieves it) - -- "Our system beats all other approaches" - (You've shown it works, not that it's better than alternatives) - -- "This is the first self-adaptive VM" - (Correct: credit prior work, focus on verification contribution) - -================================================================================ -PEER REVIEW RESPONSE TEMPLATE -================================================================================ - -When reviewers challenge your claims: - -1. Acknowledge valid concern -2. Point to specific data -3. Explain methodology -4. Offer clarification - -EXAMPLE EXCHANGE: - -Reviewer: "Your variance numbers don't seem right. How can you have 0% -variance when runtimes vary 70%?" - -Your Response: "Excellent point. This is the key insight of our analysis. -We decomposed variance into two components: - -INTERNAL variance (algorithm-controlled): - Cache hit rates: 17.39% ± 0.00% (CV = 0%) - -EXTERNAL variance (OS-controlled): - Runtime fluctuation: 60-70% CV (from scheduling) - -The algorithm selects identical cache configurations every run (0% variance). -The OS scheduler then adds its own 70% variance on top. - -See Table 3 in the paper for the full variance decomposition breakdown. -Raw data available in appendix for verification." - -================================================================================ -FINAL CHECKLIST FOR PEER REVIEW -================================================================================ - -Before submitting, verify you can defend: - -□ Claim 1 (Determinism): - - Raw data showing 0% cache hit rate variance ✓ - - Explanation of variance decomposition ✓ - - Defense against "measurement noise" charge ✓ - -□ Claim 2 (Convergence): - - Comparison of all three configurations ✓ - - Explanation why non-adaptive configs don't converge ✓ - - Defense against "this is just JIT" argument ✓ - -□ Claim 3 (Stability): - - Proof that algorithm variance ≠ runtime variance ✓ - - Explanation of robustness to external noise ✓ - - Defense against "70% variance is too high" ✓ - -□ Claim 4 (Predictability): - - Statistical justification for confidence intervals ✓ - - Sample size adequacy (n=30) ✓ - - Defense against "confidence intervals are meaningless" ✓ - -□ Claim 5 (Physics Model): - - Model description clear and formal ✓ - - Observable state definition ✓ - - Transition rules explicitly stated ✓ - -□ Supporting Evidence: - - All data reproducible from raw CSV ✓ - - Analysis scripts provided ✓ - - Methodology clearly documented ✓ - -================================================================================ -GO SUBMIT WITH CONFIDENCE -================================================================================ - -Your experimental work is solid. -Your data is clean. -Your analysis is rigorous. -Your claims are defensible. - -You have empirically proven: -✓ Physics model produces deterministic behavior -✓ System demonstrates adaptive convergence -✓ Algorithm is stable despite environmental noise -✓ Performance bounds are predictable - -This is formal verification through empirical validation. - -Ready for peer review. Good luck! - -================================================================================ diff --git a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/QUICK_REFERENCE.txt b/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/QUICK_REFERENCE.txt deleted file mode 100644 index babc0b4b..00000000 --- a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/02_SUPPORTING_DOCS/QUICK_REFERENCE.txt +++ /dev/null @@ -1,179 +0,0 @@ -================================================================================ -PHYSICS ENGINE VALIDATION - QUICK REFERENCE GUIDE -================================================================================ - -EXPERIMENTS COMPLETED: -✓ 90-run comprehensive validation (3 configs × 30 runs each) -✓ CPU thermal monitoring (captured temperature & frequency) -✓ Data integrity validation (all 90 rows clean) -✓ Thermal hypothesis testing (REFUTED - no thermal noise) -✓ Variance source analysis (identified OS scheduling + memory latency) - -CURRENT STATE: -Location: /home/rajames/CLionProjects/StarForth/physics_experiment/full_90_run_comprehensive/ -Results: experiment_results.csv (90 runs, all data validated) -Analysis Files: - - analyze_thermal_hypothesis.py (thermal analysis) - - analyze_variance_sources.py (variance decomposition) - - VARIANCE_ANALYSIS_SUMMARY.md (comprehensive findings) - -================================================================================ -KEY FINDINGS AT A GLANCE -================================================================================ - -THERMAL HYPOTHESIS: ✗ REFUTED - - CPU remained at exactly 27°C throughout all 90 runs - - No thermal drift, no frequency throttling - - Conclusion: Thermal noise is NOT a factor in your measurements - -VARIANCE SOURCE RANKING (Coefficient of Variation): - -1. OS SCHEDULING NOISE (PRIMARY) - 60-70% CV ⚠️ - A_BASELINE: 69.58% (range: 3.34-17.67ms, 305% span!) - C_FULL: 60.23% (range: 6.56-33.15ms, 299% span!) - B_CACHE: 32.35% (range: 6.27-19.26ms, 167% span) - - → This is the DOMINANT source of variance - → Simple configs are WORSE (more sensitive to preemption) - → Caused by: context switching, interrupts, page faults - -2. MEMORY LATENCY VARIABILITY (SECONDARY) - 54-103% CV ⚠️ - B_CACHE cache hit latency: 40.1 ns ± 41.4 ns (103.3% CV) - C_FULL cache hit latency: 37.9 ns ± 20.5 ns (54.0% CV) - - → Extreme variability in L1/L2 hit times - → Indicates cache coherency misses, NUMA effects - → 80ns swings in what should be 40ns hits - -3. CACHE HIT RATE (EXCELLENT) - 0% CV ✓ - B_CACHE: 17.39% ± 0.00% - C_FULL: 17.39% ± 0.00% - - → Perfect consistency! Cache behavior is deterministic - → Variance NOT from unpredictable cache misses - → Working set is stable across all runs - -4. WARMUP EFFECTS (INTERESTING) - -25% to +6% - A_BASELINE: +6.4% slower (degradation over time) - B_CACHE: +0.5% stable (excellent warmup) - C_FULL: -25.4% FASTER (improves significantly!) - - → C_FULL benefits from warming caches - → A_BASELINE shows slight system degradation - -================================================================================ -QUICK FIXES (DO THIS NOW) -================================================================================ - -STEP 1: CPU PINNING (Expected improvement: -30-40% variance) - Run your benchmark pinned to a single CPU core: - $ taskset -c 0 ./build/starforth ... - - Or for the experiment: - $ taskset -c 0 scripts/run_comprehensive_physics_experiment.sh ... - -STEP 2: SET PERFORMANCE MODE (Expected improvement: -10-15% variance) - Disable frequency scaling: - $ echo performance | sudo tee /sys/devices/system/cpu/*/cpufreq/scaling_governor - - Verify: - $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor - -STEP 3: USE ROBUST STATISTICS (Immediate improvement) - Don't use mean ± stdev when you have 69% CV! - - Instead, use trimmed mean: - - Sort runtimes - - Remove top/bottom 10% - - Average the middle 80% - - Or use median + IQR (interquartile range) - - Python example: - runtimes = [4.5, 4.7, 5.2, 17.6] # 17.6 is outlier - sorted_rt = sorted(runtimes) - trimmed = sorted_rt[1:-1] # Remove min and max - mean = sum(trimmed) / len(trimmed) - -COMBINED EXPECTED IMPROVEMENT: - Current: A_BASELINE ~70% CV - After Step 1+2: ~30-40% CV - After Step 3: Statistics properly account for outliers - - Result: Much clearer performance comparisons! - -================================================================================ -DETAILED MITIGATION PRIORITY -================================================================================ - -IMMEDIATE (This week): - [1] CPU pinning with taskset - [2] Performance governor setting - [3] Rerun experiment with CPU pinning - [4] Switch to trimmed mean analysis - -SHORT TERM (Next 1-2 weeks): - [1] Disable SMT (hyper-threading) - [2] Increase warmup phase (5 runs instead of current) - [3] Use perf to measure cache misses - $ perf stat -e cache-references,cache-misses \ - taskset -c 0 ./build/starforth ... - -MEDIUM TERM (If variance still >30%): - [1] Try SCHED_FIFO real-time scheduling (requires root) - [2] NUMA binding with numactl - [3] Isolate CPU cores (kernel parameter isolcpus) - [4] Profile with perf-record to find hotspots - -LONG TERM (For production benchmarks): - [1] Use real-time kernel (RT_PREEMPT) - [2] Dedicated measurement system - [3] Disable all background services - -================================================================================ -FILES TO READ -================================================================================ - -DETAILED ANALYSIS: - → /home/rajames/CLionProjects/StarForth/physics_experiment/VARIANCE_ANALYSIS_SUMMARY.md - (Full findings, root cause analysis, mitigation strategies) - -DATA FILES: - → /home/rajames/CLionProjects/StarForth/physics_experiment/full_90_run_comprehensive/ - experiment_results.csv (90 rows of clean data, all metrics) - -ANALYSIS SCRIPTS: - → analyze_thermal_hypothesis.py (thermal analysis) - → analyze_variance_sources.py (variance decomposition - rerun anytime) - -================================================================================ -QUICK TEST: VALIDATE IMPROVEMENTS -================================================================================ - -After applying mitigations, rerun and compare: - -# Before mitigations (baseline) -$ ./scripts/run_comprehensive_physics_experiment.sh before_fix -$ python3 analyze_variance_sources.py \ - before_fix/experiment_results.csv - -# After mitigations -$ taskset -c 0 ./scripts/run_comprehensive_physics_experiment.sh after_fix -$ python3 analyze_variance_sources.py \ - after_fix/experiment_results.csv - -Expected: Coefficient of Variation drops from 60-70% to <40% - -================================================================================ -BOTTOM LINE -================================================================================ - -Your data collection script: ✓ PERFECT (no issues found) -Your thermal hypothesis: ✗ NOT CONFIRMED (system is cool) -Your actual problem: → OS scheduling noise (expected, solvable) - -Next action: Pin to CPU + set performance mode, then rerun. -Expected result: Variance cut by 40% or more. - -For questions, see VARIANCE_ANALYSIS_SUMMARY.md for full technical details. -================================================================================ \ No newline at end of file diff --git a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/experiment_summary.txt b/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/experiment_summary.txt deleted file mode 100644 index 7f5fb54e..00000000 --- a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/experiment_summary.txt +++ /dev/null @@ -1,16 +0,0 @@ - -Experiment Start: 2025-11-07T06:40:43 -Output Directory: physics_experiment/full_90_run_comprehensive -Runs per Configuration: 30 -Total Runs: 90 -Benchmark Iterations: 100000 -Build Profile: fastest -Execution: Randomized order (eliminates temporal bias) - - -Experiment End: 2025-11-07T06:49:08 -Total Runtime: 8 minutes (505 seconds) -Results CSV: physics_experiment/full_90_run_comprehensive/experiment_results.csv -Run Logs: physics_experiment/full_90_run_comprehensive/run_logs/ -Test Matrix: physics_experiment/full_90_run_comprehensive/run_logs/test_matrix.txt - diff --git a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/run_logs/test_matrix.txt b/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/run_logs/test_matrix.txt deleted file mode 100644 index 93599a33..00000000 --- a/docs/working/archive/legacy/phase-1/Reference/physics_experiment/PEER_REVIEW_SUBMISSION/03_EXPERIMENTAL_DATA/full_90_run_comprehensive/run_logs/test_matrix.txt +++ /dev/null @@ -1,90 +0,0 @@ -A_BASELINE,0,0,8 -B_CACHE,1,0,21 -A_BASELINE,0,0,17 -B_CACHE,1,0,25 -C_FULL,1,1,23 -B_CACHE,1,0,7 -B_CACHE,1,0,16 -B_CACHE,1,0,13 -B_CACHE,1,0,30 -C_FULL,1,1,26 -A_BASELINE,0,0,28 -C_FULL,1,1,16 -B_CACHE,1,0,22 -A_BASELINE,0,0,24 -A_BASELINE,0,0,18 -B_CACHE,1,0,8 -C_FULL,1,1,12 -A_BASELINE,0,0,19 -C_FULL,1,1,4 -A_BASELINE,0,0,3 -C_FULL,1,1,14 -B_CACHE,1,0,9 -A_BASELINE,0,0,11 -A_BASELINE,0,0,10 -C_FULL,1,1,22 -B_CACHE,1,0,1 -B_CACHE,1,0,28 -C_FULL,1,1,15 -C_FULL,1,1,1 -C_FULL,1,1,30 -A_BASELINE,0,0,13 -C_FULL,1,1,29 -B_CACHE,1,0,11 -B_CACHE,1,0,3 -A_BASELINE,0,0,29 -C_FULL,1,1,6 -A_BASELINE,0,0,20 -A_BASELINE,0,0,15 -B_CACHE,1,0,14 -B_CACHE,1,0,5 -C_FULL,1,1,20 -A_BASELINE,0,0,16 -C_FULL,1,1,3 -A_BASELINE,0,0,23 -B_CACHE,1,0,6 -A_BASELINE,0,0,30 -A_BASELINE,0,0,7 -C_FULL,1,1,19 -C_FULL,1,1,5 -B_CACHE,1,0,15 -B_CACHE,1,0,29 -C_FULL,1,1,13 -C_FULL,1,1,9 -C_FULL,1,1,25 -A_BASELINE,0,0,4 -B_CACHE,1,0,17 -C_FULL,1,1,21 -C_FULL,1,1,17 -B_CACHE,1,0,20 -B_CACHE,1,0,26 -C_FULL,1,1,11 -A_BASELINE,0,0,27 -B_CACHE,1,0,18 -B_CACHE,1,0,12 -C_FULL,1,1,2 -A_BASELINE,0,0,25 -A_BASELINE,0,0,14 -C_FULL,1,1,24 -C_FULL,1,1,8 -B_CACHE,1,0,27 -B_CACHE,1,0,4 -C_FULL,1,1,7 -A_BASELINE,0,0,9 -A_BASELINE,0,0,2 -C_FULL,1,1,18 -B_CACHE,1,0,19 -A_BASELINE,0,0,21 -A_BASELINE,0,0,22 -C_FULL,1,1,28 -A_BASELINE,0,0,12 -A_BASELINE,0,0,26 -B_CACHE,1,0,24 -B_CACHE,1,0,10 -B_CACHE,1,0,2 -B_CACHE,1,0,23 -A_BASELINE,0,0,6 -A_BASELINE,0,0,5 -C_FULL,1,1,10 -C_FULL,1,1,27 -A_BASELINE,0,0,1 diff --git a/docs/working/papers/ONTOLOGY.md b/docs/working/papers/ONTOLOGY.md deleted file mode 100644 index 79069cbd..00000000 --- a/docs/working/papers/ONTOLOGY.md +++ /dev/null @@ -1,528 +0,0 @@ - -# StarForth Ontology, Taxonomy, and Lexicon - -**Version**: 1.0 -**Date**: 2025-12-13 -**Purpose**: Formal conceptual framework for precise academic discourse - ---- - -## I. ONTOLOGY (Conceptual Framework) - -### 1.1 Core Concepts - -``` -StarForth Adaptive Runtime System -├── Execution Metrics -│ ├── Execution Frequency (primary measurable) -│ ├── Temporal Decay (derived quantity) -│ └── Transition Probability (derived quantity) -├── Adaptive Mechanisms -│ ├── Frequency-Based Caching -│ ├── Window-Based Inference -│ └── Decay-Based Pruning -├── Convergence Properties -│ ├── Deterministic Behavior -│ ├── Steady-State Equilibrium -│ └── Variance Reduction -└── Analysis Frameworks - ├── Dynamical Systems View - ├── Statistical Inference View - └── Control Theory View -``` - -### 1.2 Conceptual Relationships - -``` -METAPHORICAL MAPPING (Thermodynamics → Execution): - Thermal Energy ≈ Execution Frequency (measurable count) - Heat Dissipation ≈ Exponential Decay (time-based reduction) - Thermal Equilibrium ≈ Steady-State Convergence (stable metrics) - Temperature ≈ Normalized Frequency Rank - Cooling Rate ≈ Decay Coefficient (λ) - -LITERAL IMPLEMENTATIONS (No Metaphor): - Frequency Counter → Integer increment on execution - Decay Function → f(t) = f₀ * e^(-λt) - Hot-Words Cache → Top-K frequency-sorted entries - Rolling Window → Circular buffer of execution records - ANOVA Test → Statistical variance analysis (Levene's test) -``` - ---- - -## II. TAXONOMY (Hierarchical Classification) - -### 2.1 Adaptive Runtime Components - -``` -1. Measurement Layer - 1.1 Execution Frequency Tracking - └── Dictionary Entry Counter (per-word increment) - 1.2 Temporal Recording - └── Rolling Window of Truth (circular buffer) - 1.3 Transition Tracking - └── Word-to-Word Transition Matrix - -2. Transformation Layer - 2.1 Decay Models - 2.1.1 Linear Decay (Loop #3) - └── Δf = -k * Δt - 2.1.2 Exponential Decay (Loop #6 Inference) - └── f(t) = f₀ * e^(-λt) - 2.2 Normalization - 2.2.1 Frequency Ranking - 2.2.2 Probability Calculation - -3. Inference Layer - 3.1 Window Width Inference (Loop #5) - 3.1.1 Variance Analysis (Levene's Test) - 3.1.2 Binary Search (Variance Inflection Point) - 3.2 Decay Slope Inference (Loop #6) - 3.2.1 Exponential Regression - 3.2.2 Least Squares Fitting - -4. Actuation Layer - 4.1 Hot-Words Cache (Loop #1) - 4.1.1 Frequency-Based Sorting - 4.1.2 Top-K Selection - 4.1.3 Fast-Path Dictionary Lookup - 4.2 Speculative Execution (Loop #4) - 4.2.1 Transition Probability Calculation - 4.2.2 Prefetch Decision - -5. Coordination Layer - 5.1 Heartbeat System (Loop #7) - 5.1.1 Time-Driven Tick Generation - 5.1.2 Loop Orchestration - 5.1.3 Adaptive Tick Rate -``` - -### 2.2 Feedback Loop Taxonomy - -``` -Feedback Loops (7 Total) -├── Positive Loops (Amplifying) -│ ├── Loop #1: Execution Heat Tracking -│ │ └── More executions → Higher rank → More cache hits → More executions -│ └── Loop #4: Pipelining Metrics -│ └── More transitions → Better prediction → More prefetch hits -├── Negative Loops (Stabilizing) -│ ├── Loop #3: Linear Decay -│ │ └── High frequency → Faster decay → Lower frequency → Slower decay -│ ├── Loop #5: Window Width Inference -│ │ └── High variance → Smaller window → Lower variance -│ └── Loop #6: Decay Slope Inference -│ └── Unstable metrics → Steeper decay → Faster stabilization -├── Neutral Loops (Monitoring) -│ └── Loop #2: Rolling Window History -│ └── Execution occurs → Record in window → Historical data available -└── Meta-Loop (Adaptive Coordination) - └── Loop #7: Adaptive Heartbeat - └── Stable system → Slower ticks → Less overhead -``` - -### 2.3 Convergence Taxonomy - -``` -Convergence Properties -├── Determinism -│ ├── Algorithmic Determinism (same inputs → same outputs) -│ ├── Temporal Determinism (time-invariant steady state) -│ └── Statistical Determinism (0% variance) -├── Steady-State Characteristics -│ ├── Fixed Point (attractors in phase space) -│ ├── Periodic Behavior (limit cycles) -│ └── Chaotic Behavior (sensitive dependence - avoided) -└── Variance Metrics - ├── Inter-Run Variance (across multiple executions) - ├── Intra-Run Variance (within single execution) - └── Temporal Variance (across time windows) -``` - ---- - -## III. LEXICON (Precise Definitions) - -### 3.1 Core Terms (Alphabetical) - -**Adaptive Heartbeat** -*Definition*: Time-driven coordination mechanism that orchestrates feedback loop execution at dynamically-adjusted intervals. -*Formal*: Thread T executing `vm_tick()` at frequency f_tick, where f_tick ∈ [f_min, f_max] adapts based on system stability. -*Measurement*: Tick period in nanoseconds (configurable: HEARTBEAT_TICK_NS). -*Category*: Coordination mechanism. - -**Attractor** -*Definition*: Stable equilibrium point or region in phase space toward which execution trajectories converge. -*Formal*: Fixed point x* where F(x*) = x* for dynamical system x_{t+1} = F(x_t). -*Measurement*: Coordinates in (window_size, decay_slope, variance) space. -*Category*: Dynamical systems concept. - -**Decay Coefficient (λ)** -*Definition*: Rate parameter controlling exponential reduction in execution frequency over time. -*Formal*: λ in f(t) = f₀ * e^(-λt), units of [1/time]. -*Measurement*: Derived via exponential regression; stored as Q48.16 fixed-point. -*Category*: Transformation parameter. - -**Deterministic Convergence** -*Definition*: Property whereby repeated executions of identical workloads produce statistically indistinguishable steady-state metrics. -*Formal*: ∀ executions i,j: |metric_i - metric_j| / σ < ε, where ε → 0 as t → ∞. -*Measurement*: Coefficient of variation CV = σ/μ → 0%. -*Category*: Convergence property. - -**Execution Frequency** -*Definition*: Count of times a dictionary entry has been executed since VM initialization, optionally adjusted by decay. -*Formal*: f = Σ executions - ∫ decay(t) dt. -*Measurement*: Unsigned 64-bit integer (`uint64_t execution_heat`). -*Category*: Primary measurable quantity. -*Note*: "Heat" is metaphorical naming; actual quantity is frequency. - -**Exponential Decay** -*Definition*: Mathematical function modeling reduction in execution frequency proportional to current value. -*Formal*: f(t) = f₀ * e^(-λt), where f₀ is initial frequency. -*Measurement*: Applied periodically by heartbeat system. -*Category*: Transformation function. -*Metaphor*: Analogous to radioactive decay or thermal dissipation. - -**Hot-Words Cache** -*Definition*: Fixed-size array storing pointers to the K most frequently executed dictionary entries for O(1) lookup acceleration. -*Formal*: Cache C = {e_1, e_2, ..., e_K} where f(e_i) ≥ f(e_{i+1}) ∀ i. -*Measurement*: Cache size K (configurable: HOTWORDS_CACHE_SIZE). -*Category*: Frequency-based optimization. - -**Levene's Test** -*Definition*: Non-parametric statistical test for homogeneity of variance across groups. -*Formal*: H₀: σ₁² = σ₂² = ... = σ_k² (variances equal). -*Measurement*: F-statistic with p-value threshold (typically 0.05). -*Category*: Statistical inference method. -*Application*: Used to detect variance changes when adjusting window size. - -**Phase Space** -*Definition*: Multi-dimensional coordinate system where each axis represents a system state variable. -*Formal*: Space S = {(w, λ, σ²) | w ∈ ℕ, λ ∈ ℝ⁺, σ² ∈ ℝ⁺}. -*Measurement*: Coordinates: (window_size, decay_slope_Q48, variance_Q48). -*Category*: Dynamical systems representation. - -**Rolling Window of Truth** -*Definition*: Circular buffer recording recent execution history for deterministic metric seeding. -*Formal*: Buffer B[i] = word_id at execution event i mod |B|. -*Measurement*: Buffer size (configurable: ROLLING_WINDOW_SIZE = 4096). -*Category*: Temporal recording mechanism. -*Purpose*: Ensures identical initial conditions for reproducibility. - -**Steady-State Equilibrium** -*Definition*: Condition where adaptive system metrics stabilize within bounded oscillation. -*Formal*: ∃ t_0: ∀ t > t_0, |x(t) - x*| < δ for small δ. -*Measurement*: Variance CV < 0.1% over 1000-tick window. -*Category*: Convergence property. -*Metaphor*: Analogous to thermodynamic equilibrium. - -**Thermodynamic Metaphor** -*Definition*: Conceptual mapping between thermodynamic quantities and execution metrics. -*Mapping*: - - Heat ↔ Execution Frequency - - Temperature ↔ Normalized Rank - - Cooling ↔ Exponential Decay - - Equilibrium ↔ Steady State -*Category*: Conceptual framework (not literal physics). -*Warning*: Must qualify as metaphor in academic writing. - -**Transition Probability** -*Definition*: Conditional probability that word B is executed immediately after word A. -*Formal*: P(B|A) = count(A→B) / count(A). -*Measurement*: Stored as Q48.16 fixed-point in transition matrix. -*Category*: Derived metric for speculative execution. - -**Variance Inflection Point** -*Definition*: Window size w* where variance begins to increase when window shrinks below w*. -*Formal*: w* = arg min_w { Var(w) | w < w_current }. -*Measurement*: Found via binary search with Levene's test. -*Category*: Inference target. -*Purpose*: Optimal window size for stable metrics. - -**Word Transition** -*Definition*: Sequential execution of word B immediately following word A in threaded code. -*Formal*: Event (A, B, t) where A executes at time t and B at time t+ε. -*Measurement*: Recorded in transition_metrics.history[] circular buffer. -*Category*: Execution event. - ---- - -### 3.2 Avoid/Deprecated Terms - -| ❌ Avoid | ✅ Use Instead | Reason | -|---------|---------------|--------| -| "Physics-based" | "Thermodynamically-inspired metaphor" | Not literal physics | -| "Execution heat" (in formal writing) | "Execution frequency with decay" | "Heat" is metaphorical | -| "Temperature" | "Normalized frequency rank" | No actual thermal quantity | -| "Quantum-inspired" | N/A | No quantum mechanics involved | -| "AI-driven" | "Statistically-inferred" | No neural networks or ML | -| "Learning" | "Adaptive inference" | Not machine learning | -| "Training" | "Convergence to steady state" | Not supervised learning | - ---- - -## IV. MATHEMATICAL FORMALISM - -### 4.1 Execution Frequency Evolution - -**Discrete-time update**: -``` -f[t+1] = f[t] + Δexec[t] - Δdecay[t] - -where: - Δexec[t] = 1 if word executed at tick t, else 0 - Δdecay[t] = λ * f[t] * Δt (exponential decay approximation) -``` - -**Continuous-time model**: -``` -df/dt = r(t) - λf(t) - -where: - r(t) = execution rate [executions/second] - λ = decay coefficient [1/second] - -Solution: - f(t) = e^(-λt) * [f₀ + ∫₀ᵗ r(τ)e^(λτ) dτ] -``` - -### 4.2 Hot-Words Cache Selection - -**Cache membership criterion**: -``` -e ∈ Cache ⟺ rank(e) ≤ K - -where: - rank(e) = |{e' ∈ Dictionary : f(e') > f(e)}| + 1 - K = cache size (constant) -``` - -### 4.3 Window Width Inference - -**Objective function**: -``` -w* = arg min { Var(w) : w ∈ [w_min, w_current] } - -Subject to: - Levene(w, w_current) → F-statistic - p-value(F) < α (typically α = 0.05) -``` - -### 4.4 Decay Slope Inference - -**Exponential regression**: -``` -Given: {(t_i, f_i)}_{i=1}^N from rolling window - -Model: f(t) = f₀ * e^(-λt) - -Log-transform: ln(f) = ln(f₀) - λt - -Least squares: - λ* = arg min Σ [ln(f_i) - (ln(f₀) - λt_i)]² -``` - -### 4.5 Transition Probability - -**Maximum likelihood estimate**: -``` -P(B|A) = count(A→B) / count(A) - -where: - count(A→B) = # times B executed immediately after A - count(A) = # times A executed -``` - -### 4.6 Convergence Metric - -**Coefficient of Variation**: -``` -CV = σ / μ - -where: - σ = √(Var[metric]) = standard deviation - μ = E[metric] = mean - -Convergence achieved when: CV → 0 -``` - ---- - -## V. RELATIONSHIP DIAGRAM - -### 5.1 Component Dependencies - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Execution Event │ -│ (word executed) │ -└──────────────────────┬──────────────────────────────────────┘ - │ - ┌───────────┴───────────┐ - │ │ - ▼ ▼ - ┌────────────────┐ ┌──────────────────┐ - │ Frequency │ │ Rolling Window │ - │ Increment │ │ Recording │ - │ (Loop #1) │ │ (Loop #2) │ - └────────┬───────┘ └─────────┬────────┘ - │ │ - │ │ - ▼ ▼ - ┌────────────────┐ ┌──────────────────┐ - │ Decay │ │ Inference │ - │ Application │ │ Engine │ - │ (Loop #3,#6) │ │ (Loop #5,#6) │ - └────────┬───────┘ └─────────┬────────┘ - │ │ - └───────────┬────────────┘ - │ - ▼ - ┌───────────────────────┐ - │ Hot-Words Cache │ - │ Reorganization │ - └───────────┬───────────┘ - │ - ▼ - ┌───────────────────────┐ - │ Optimized Lookup │ - │ (O(1) cache hit) │ - └───────────────────────┘ -``` - -### 5.2 Feedback Loop Interactions - -``` -Execution → Frequency ↑ → Cache Rank ↑ → Lookup Speed ↑ → More Execution - ▲ │ - │ │ - └─────────────── (Positive Feedback) ────────────┘ - -Frequency ↑ → Decay ↑ → Frequency ↓ → Decay ↓ → Frequency Stabilizes - ▲ │ - │ │ - └──── (Negative Feedback) ────┘ - -Variance ↑ → Window Shrink → Variance ↓ → Window Stable - ▲ │ - │ │ - └──── (Negative Feedback) ──┘ -``` - ---- - -## VI. ONTOLOGICAL COMMITMENTS - -### 6.1 Foundational Assumptions - -1. **Frequency as Proxy for Importance** - - Assumption: Frequently executed code is more important to optimize - - Justification: Empirical validation (Zipf's law in execution patterns) - -2. **Decay Models Temporal Relevance** - - Assumption: Recent executions are more relevant than distant past - - Justification: Locality of reference (temporal locality principle) - -3. **Determinism Through Convergence** - - Assumption: Adaptive systems can converge to deterministic steady states - - Justification: Fixed-point theorems for contractive mappings - -4. **Statistical Inference Validity** - - Assumption: Execution patterns are statistically analyzable - - Justification: Central Limit Theorem for large sample sizes - -### 6.2 Scope Limitations - -**What This Ontology DOES Cover**: -- ✓ Execution frequency measurement and decay -- ✓ Adaptive caching and inference mechanisms -- ✓ Dynamical systems characterization -- ✓ Statistical convergence properties - -**What This Ontology DOES NOT Cover**: -- ✗ Actual thermodynamic processes (metaphor only) -- ✗ Machine learning or neural networks (no training) -- ✗ Quantum computing (no quantum effects) -- ✗ Biological neural systems (no biomimicry) - ---- - -## VII. USAGE GUIDELINES - -### 7.1 Academic Writing - -**In Abstracts/Titles**: Use precise, non-metaphorical terms -``` -✅ "Thermodynamically-Inspired Adaptive Runtime" -❌ "Physics-Based Virtual Machine" -``` - -**In Technical Sections**: Qualify metaphors explicitly -``` -✅ "We employ a thermodynamic metaphor where execution frequency - is treated as 'heat' that dissipates over time..." -❌ "The physics model applies heat decay..." -``` - -**In Formalism**: Use mathematical definitions, not analogies -``` -✅ "Frequency evolves as f(t) = f₀ * e^(-λt)" -❌ "Heat cools exponentially like in Newton's law" -``` - -### 7.2 Code Comments - -**In Source Code**: Use concrete terms from lexicon -```c -// ✅ GOOD: Increment execution frequency counter -entry->execution_heat++; - -// ❌ BAD: Increase temperature of word -entry->execution_heat++; // heat up! -``` - -### 7.3 Presentation/Talks - -**Slides**: Use metaphor for intuition, then formalize -``` -Slide 1: "Think of execution frequency like heat..." -Slide 2: "Formally: f(t) = f₀ * e^(-λt)" -``` - ---- - -## VIII. REFERENCES - -### 8.1 Foundational Concepts - -- **Dynamical Systems**: Strogatz, S. (2015). *Nonlinear Dynamics and Chaos* -- **Control Theory**: Åström, K. & Murray, R. (2008). *Feedback Systems* -- **Statistical Inference**: Casella, G. & Berger, R. (2002). *Statistical Inference* -- **Exponential Decay**: Standard mathematical function (e^(-λt)) - -### 8.2 Related Work - -- **Trace-based JIT**: Bolz et al. (2009). "Tracing the Meta-Level: PyPy's Tracing JIT Compiler" -- **Adaptive Systems**: Garlan et al. (2004). "Rainbow: Architecture-Based Self-Adaptation" -- **Forth Optimization**: Ertl, M.A. (1996). "Stack Caching for Interpreters" - ---- - -## IX. VERSION HISTORY - -**v1.0** (2025-12-13): -- Initial ontology, taxonomy, and lexicon -- Formal mathematical definitions -- Relationship diagrams -- Usage guidelines - ---- - -## X. ACKNOWLEDGMENTS - -This ontology provides the formal conceptual framework for the StarForth project. -It aims to eliminate ambiguity and enable precise academic discourse while -acknowledging the metaphorical nature of certain conceptual mappings. - -**License**: CC0 / Public Domain