diff --git a/docs/v4.0.0/NUCLEUS.md b/docs/v4.0.0/NUCLEUS.md new file mode 100644 index 00000000..5da25b2a --- /dev/null +++ b/docs/v4.0.0/NUCLEUS.md @@ -0,0 +1,273 @@ +# StarForth v4.0.0 — Nucleus, FORTH-79 Capsule and POST + +Design, 2026-10-05. Ruled by Captain Bob in conversation the same day; this +file records those rulings. It sits beside `JUSTIFICATION.md` (why v4) and +`DECOMPOSITION.md` (every v3 word's fate), and changes how the vocabulary +`DECOMPOSITION.md` describes is delivered, not what the words do. + +## 1. Goal + +v4 comes up on amd64, aarch64 and riscv64, both as the bare-metal kernel +under QEMU and as the hosted Linux binary, in this order: + +1. start the assembled nucleus; +2. load FORTH-79 from a capsule; +3. run POST on FORTH-79; +4. reach the `ok>` prompt. + +That is all this work delivers. Both the kernel and the hosted binary are +products. + +## 2. What is wrong today + +- All 292 named words are written in F18 assembler text (`v4/capsule/*.v4`). + Only the editor and `SEE` are FORTH source. +- `v4/tools/mkimage.c` assembles and compiles everything on the build + machine. The kernel links the finished memory image. Nothing is loaded + from a capsule at boot. +- There is no v4 POST. +- No QEMU log of a v4 boot exists, so bare-metal v4 has never been accepted. + +## 3. Scope + +In scope: the FORTH-79 Required Word Set, as colon definitions in a capsule, +and a POST for it. + +Out of scope until this is accepted: the Double Number extension set, Q48 +arithmetic, logging, access control, the editor, `SEE`, `DEFER`, and every +other StarForth extension. Their `.v4` and `.fth` sources stay in the tree +but are not built into the nucleus image or loaded at boot. + +## 4. The nucleus + +The nucleus is the part of the vocabulary that stays in assembler. A word is +in the nucleus only if one of these holds. + +1. **It is an opcode or a register access.** v4 compiles colon definitions + to native instruction words, so FORTH source cannot write an opcode. + `DUP DROP OVER + AND XOR 2* 2/ @ ! >R R> R@`, the stack-depth registers, + and the console registers behind `KEY` and `EMIT`. +2. **The node needs it to read a capsule.** `WORD FIND NUMBER INTERPRET + QUIT`, the error trap, `HERE ALLOT ,`, `CREATE`, `:`, `;`, `IMMEDIATE`, + `LITERAL`, `[`, `]`, `(` and the code generator. +3. **It lays down code for a colon definition in the capsule.** The control + words become colon definitions, and they need the code generator's + words by name: `(OP,)`, `(LIT,)`, `(LABEL)`, `(BRANCH>)`, `(RESOLVE)`, + `(JUMP,)`, `(CALL,)`, and the control-flow stack words `>CF`, `CF>` and + `(PAIR)`. These get headers. They are not FORTH-79 words. + +Everything else in the Required Word Set is a colon definition in the +capsule. That includes words that are assembler today only for speed: +`SWAP ROT - * /MOD 0= < C@ C! CMOVE FILL . <# # #> TYPE COUNT VARIABLE +CONSTANT`, the control words, the vocabulary words and the block words. + +The capsule's definitions will run slower than the assembled ones they +replace. That is accepted: the goal is the minimum in assembler. + +The list in rule 2 is the starting boundary, not the final one. A word +leaves the nucleus whenever a colon definition of it passes POST (section +8, step 3). The nucleus is at its minimum when no remaining word can be +moved. + +## 5. Capsules + +### 5.1 Files + +| File | Capsule name | Contents | +|---|---|---| +| `capsules/v4/forth79.4th` | `v4:forth79.4th` | The Required Word Set as colon definitions | +| `capsules/v4/post79.4th` | `v4:post79.4th` | The POST harness and its cases | + +Both are ordinary `.4th` capsules: `Block N` headers, lines of at most 64 +characters, at most 16 lines to a block. `tools/mkcapsule.c` bakes them into +`capsule_generated.c` with every other file under `capsules/`, hashed, and +signed when the signing key is present. v3 does not load them: it runs +`init.4th` and only what that file `EXEC`s. + +Named blobs (any non-`.4th` file under `capsules/`) are baked the same way +and stay available to v4 by name. This work does not need one. + +### 5.2 The block-number rule + +`validate_forth_blocks` (`tools/mkcapsule.c:409`) accepts block numbers in +`[2048, 5120)`. The floor is real: blocks 0–2047 are the VM's fast RAM. The +ceiling matches no device. + +**Change:** the check becomes "block number is at least 2048". There is no +upper bound. Blocks 0–2047 are the only forbidden ones. The collision checks +against other capsules and `tools/capsule-reserved.txt` are unchanged. +`experiments/bare_metal/README.md` and `.claude/CLAUDE.md` state the old +range and are corrected with it. + +The v4 capsules take a free range found from `capsules/BLOCK_MAP.md` and +`tools/capsule-reserved.txt`. + +### 5.3 Loading + +A capsule reaches the node as source lines through its console input, the +way v3's `capsule_exec_payload` interprets a payload line by line. `LOAD` +is not used: the block words are themselves in the capsule. + +For each capsule, in order: + +1. find it by name (`capsule_find_by_name`); +2. recompute its hash (`capsule_validate`); a mismatch stops the boot; +3. check its signature (`capsule_verify_signature`); invalid stops the + boot, missing warns — v3's rule at `capsule_birth.c:585`; +4. split the payload on `Block N` headers and feed each line to the node, + running the node until it waits for input again; +5. require the node to answer ` ok` to every line. + +A line that is not accepted stops the boot and prints the capsule name, +block number, line number and what the node said. v3's retry, which skips a +failing line and runs the block again, is not carried over: a skipped line +in the standard's own capsule must not pass unnoticed. + +What the node prints while loading is not shown, except on failure. + +In v4 the block numbers are labels for now. They become storage addresses +when the block words exist and a capsule is copied to block storage. + +## 6. POST + +### 6.1 Source of the cases + +POST is v3's own test cases, ported. The cases are in +`v3/src/test_runner/modules/*.c`; each is a name, a line of FORTH and a +flag saying whether an error is expected. Only cases for Required Word Set +words are ported, chosen by word name against the standard's list. + +### 6.2 Judging + +v3 judges a case only by whether it raised an error +(`v3/src/test_runner/test_common.c:258-285`); the "expected" text is a +comment. A wrong result that raises no error passes. That is not enough for +POST's purpose here, which is to prove each colon definition against the +assembled word it replaces. + +**Ruling:** every ported case compares its result. The expected data stack +and printed output are taken from running the same line on the hosted v3 +binary, so v3 remains the reference. A case v3 expects to raise an error +must raise one on v4. + +Where v3 departs from FORTH-79 for a standard word, FORTH-79 wins (standing +ruling); the case's expected value is then the standard's, and the +departure is reported. + +### 6.3 Form + +`post79.4th` defines a small harness in FORTH — a case is written +`T{ 1 2 3 ROT -> 2 3 1 }T` — and a tally. Cases that check printed output +or an expected error use harness words for those. The harness needs one +thing from the nucleus that FORTH-79 does not provide: a way to run a case +and regain control if it raises an error. That is one nucleus word. + +After the last case POST prints the number of cases, passes and failures, +and names each failing case. + +### 6.4 Generating the expected values + +A script reads the v3 test modules, keeps the cases for Required Word Set +words, pipes each line through the hosted v3 binary, and writes the +`T{ ... }T` lines. It is a development tool, run when the cases change; its +output, `post79.4th`, is committed and reviewed like any source. + +## 7. Hashing, signing and parity + +Hashing and signing are the existing capsule mechanisms, used unchanged +(section 5.3). + +Parity needs one new function. `sk_dict_canonical_hash` walks v3's +`DictEntry` list and cannot hash a v4 node. The v4 dictionary hash is +FNV-1a (`fnv1a_64`, already in `kernel/src/vm/parity.c`) over the node's +memory from the start of code to `HERE`, and `LATEST`. Stacks, input +buffers, block buffers and heat are left out. + +The boot prints, in order: + +``` +PARITY:V4_NUCLEUS words=N image_hash=0x... +PARITY:V4_CAPSULE name=v4:forth79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x... +PARITY:V4_CAPSULE name=v4:post79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x... +PARITY:V4_POST tests=N pass=N fail=N +PARITY:OK +POST: PASSED +ok> +``` + +On any failure it prints `PARITY:FAIL` and `POST: FAILED` in place of the +last three lines and does not give a prompt. + +The tags are `PARITY:V4_*` so tooling that looks for v3's `PARITY:M7.1a` +is not misled. + +Every build uses 64-bit cells on the same engine, so every line above is +identical on all six builds of one commit (section 9). The dictionary hash +changes whenever a word moves between the nucleus and the capsule, so +there is no fixed golden hash until decomposition is finished. + +The nucleus is data linked into the binary and is trusted as the binary's +own code is. It is hashed for the parity line and not signed. + +## 8. Order of work + +1. **Capsule loading at boot**, hosted and bare metal, with the block-number + change. The capsule may be nearly empty at this step. +2. **POST against today's assembled words.** POST must pass here, on words + `make -C v4 test` already covers. This proves POST before it judges + anything new. +3. **Decompose group by group.** Move one group from `.v4` to + `forth79.4th` and run POST hosted: stack, arithmetic, comparison, + memory, strings, number output, control, defining, vocabulary, blocks. + A failure points at the group just moved. +4. **Acceptance** (section 9). + +## 9. Products and acceptance + +The hosted binary and the kernel run the same engine (`v4/src`), the same +nucleus image, the same baked capsule directory and the same boot sequence +(section 5.3 and section 7). Only the console differs: stdin and stdout +hosted, the serial console on bare metal. The hosted binary links +`capsule_generated.c` and reads nothing from the source tree. + +Acceptance is six boots of one commit: + +| | amd64 | aarch64 | riscv64 | +|---|---|---|---| +| Hosted Linux | native | user-mode QEMU | user-mode QEMU | +| Bare metal | `clean qemu` | `clean qemu` | `clean qemu` | + +Each must print identical `PARITY:V4_*` lines, pass POST and reach `ok>`. +Bare-metal runs follow the repository's QEMU rules: one instance at a time, +in the foreground, `clean` before `qemu`, logs kept under `logs/`. + +`make -C v4 test` also runs at 32-bit cells. That stays as a development +check for the FPGA gateway and is not part of this acceptance. + +## 10. What is reused and what is new + +| Need | Existing | Change | +|---|---|---| +| Bake capsules | `tools/mkcapsule.c`, `kernel/Makefile` capsule rule | Block-number check only (5.2) | +| Find, hash-check, verify | `capsule_find.c`, `capsule_validate.c`, `capsule_sig.c` | None; also compiled into the hosted binary | +| Split a payload into blocks and lines | `is_block_header` and the walk in `capsule_loader.c` | Moved to a file both loaders call | +| Start a node from an image | `v4/src/image.c`, `v4_image_boot` | None | +| Build the nucleus image | `v4/tools/mkimage.c` | File list shrinks as groups move; the step that types `editor.fth` and `tools.fth` is removed | +| Kernel entry | `kernel/src/v4/sk_v4.c`, Kconfig `STARFORTH_V4` | Calls the shared boot sequence before reading the keyboard | +| Hosted entry | `v4/tools/hosted.c` | Same | +| Parity lines and FNV-1a | `kernel/src/vm/parity.c` | v4 dictionary hash added | +| v3 cases | `v3/src/test_runner/modules/*.c` | Read only | +| Expected values | hosted v3 binary | Read only | +| Hosted builds per ISA | `v4/Makefile` | Cross-compiler targets for aarch64 and riscv64 | + +New: the shared v4 boot sequence (one file), the v4 dictionary hash, the +case-extraction script, `forth79.4th` and `post79.4th`. + +The v3 boot is untouched: `STARFORTH_V4` defaults to off. + +## 11. Not decided here + +- Where the v4 hosted binaries are installed. They are built under + `v4/build/`; `lfs/` holds v3's and is left alone. +- Signing the nucleus. It would have to become a named blob under + `capsules/`, which puts a generated file in a source directory.