docs(v4.0.0): nucleus, FORTH-79 capsule and POST design
The assembled vocabulary shrinks to a nucleus; the FORTH-79 Required Word Set is loaded at boot from a capsule as colon definitions, POST (v3's cases, ported, comparing results) runs on it, and the node reaches ok> -- hosted and bare metal, on amd64, aarch64 and riscv64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
rajames
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
c1bbcaaba4
commit
d4bd6e9601
@@ -0,0 +1,273 @@
|
||||
# StarForth v4.0.0 — Nucleus, FORTH-79 Capsule and POST
|
||||
|
||||
Design, 2026-10-05. Ruled by Captain Bob in conversation the same day; this
|
||||
file records those rulings. It sits beside `JUSTIFICATION.md` (why v4) and
|
||||
`DECOMPOSITION.md` (every v3 word's fate), and changes how the vocabulary
|
||||
`DECOMPOSITION.md` describes is delivered, not what the words do.
|
||||
|
||||
## 1. Goal
|
||||
|
||||
v4 comes up on amd64, aarch64 and riscv64, both as the bare-metal kernel
|
||||
under QEMU and as the hosted Linux binary, in this order:
|
||||
|
||||
1. start the assembled nucleus;
|
||||
2. load FORTH-79 from a capsule;
|
||||
3. run POST on FORTH-79;
|
||||
4. reach the `ok>` prompt.
|
||||
|
||||
That is all this work delivers. Both the kernel and the hosted binary are
|
||||
products.
|
||||
|
||||
## 2. What is wrong today
|
||||
|
||||
- All 292 named words are written in F18 assembler text (`v4/capsule/*.v4`).
|
||||
Only the editor and `SEE` are FORTH source.
|
||||
- `v4/tools/mkimage.c` assembles and compiles everything on the build
|
||||
machine. The kernel links the finished memory image. Nothing is loaded
|
||||
from a capsule at boot.
|
||||
- There is no v4 POST.
|
||||
- No QEMU log of a v4 boot exists, so bare-metal v4 has never been accepted.
|
||||
|
||||
## 3. Scope
|
||||
|
||||
In scope: the FORTH-79 Required Word Set, as colon definitions in a capsule,
|
||||
and a POST for it.
|
||||
|
||||
Out of scope until this is accepted: the Double Number extension set, Q48
|
||||
arithmetic, logging, access control, the editor, `SEE`, `DEFER`, and every
|
||||
other StarForth extension. Their `.v4` and `.fth` sources stay in the tree
|
||||
but are not built into the nucleus image or loaded at boot.
|
||||
|
||||
## 4. The nucleus
|
||||
|
||||
The nucleus is the part of the vocabulary that stays in assembler. A word is
|
||||
in the nucleus only if one of these holds.
|
||||
|
||||
1. **It is an opcode or a register access.** v4 compiles colon definitions
|
||||
to native instruction words, so FORTH source cannot write an opcode.
|
||||
`DUP DROP OVER + AND XOR 2* 2/ @ ! >R R> R@`, the stack-depth registers,
|
||||
and the console registers behind `KEY` and `EMIT`.
|
||||
2. **The node needs it to read a capsule.** `WORD FIND NUMBER INTERPRET
|
||||
QUIT`, the error trap, `HERE ALLOT ,`, `CREATE`, `:`, `;`, `IMMEDIATE`,
|
||||
`LITERAL`, `[`, `]`, `(` and the code generator.
|
||||
3. **It lays down code for a colon definition in the capsule.** The control
|
||||
words become colon definitions, and they need the code generator's
|
||||
words by name: `(OP,)`, `(LIT,)`, `(LABEL)`, `(BRANCH>)`, `(RESOLVE)`,
|
||||
`(JUMP,)`, `(CALL,)`, and the control-flow stack words `>CF`, `CF>` and
|
||||
`(PAIR)`. These get headers. They are not FORTH-79 words.
|
||||
|
||||
Everything else in the Required Word Set is a colon definition in the
|
||||
capsule. That includes words that are assembler today only for speed:
|
||||
`SWAP ROT - * /MOD 0= < C@ C! CMOVE FILL . <# # #> TYPE COUNT VARIABLE
|
||||
CONSTANT`, the control words, the vocabulary words and the block words.
|
||||
|
||||
The capsule's definitions will run slower than the assembled ones they
|
||||
replace. That is accepted: the goal is the minimum in assembler.
|
||||
|
||||
The list in rule 2 is the starting boundary, not the final one. A word
|
||||
leaves the nucleus whenever a colon definition of it passes POST (section
|
||||
8, step 3). The nucleus is at its minimum when no remaining word can be
|
||||
moved.
|
||||
|
||||
## 5. Capsules
|
||||
|
||||
### 5.1 Files
|
||||
|
||||
| File | Capsule name | Contents |
|
||||
|---|---|---|
|
||||
| `capsules/v4/forth79.4th` | `v4:forth79.4th` | The Required Word Set as colon definitions |
|
||||
| `capsules/v4/post79.4th` | `v4:post79.4th` | The POST harness and its cases |
|
||||
|
||||
Both are ordinary `.4th` capsules: `Block N` headers, lines of at most 64
|
||||
characters, at most 16 lines to a block. `tools/mkcapsule.c` bakes them into
|
||||
`capsule_generated.c` with every other file under `capsules/`, hashed, and
|
||||
signed when the signing key is present. v3 does not load them: it runs
|
||||
`init.4th` and only what that file `EXEC`s.
|
||||
|
||||
Named blobs (any non-`.4th` file under `capsules/`) are baked the same way
|
||||
and stay available to v4 by name. This work does not need one.
|
||||
|
||||
### 5.2 The block-number rule
|
||||
|
||||
`validate_forth_blocks` (`tools/mkcapsule.c:409`) accepts block numbers in
|
||||
`[2048, 5120)`. The floor is real: blocks 0–2047 are the VM's fast RAM. The
|
||||
ceiling matches no device.
|
||||
|
||||
**Change:** the check becomes "block number is at least 2048". There is no
|
||||
upper bound. Blocks 0–2047 are the only forbidden ones. The collision checks
|
||||
against other capsules and `tools/capsule-reserved.txt` are unchanged.
|
||||
`experiments/bare_metal/README.md` and `.claude/CLAUDE.md` state the old
|
||||
range and are corrected with it.
|
||||
|
||||
The v4 capsules take a free range found from `capsules/BLOCK_MAP.md` and
|
||||
`tools/capsule-reserved.txt`.
|
||||
|
||||
### 5.3 Loading
|
||||
|
||||
A capsule reaches the node as source lines through its console input, the
|
||||
way v3's `capsule_exec_payload` interprets a payload line by line. `LOAD`
|
||||
is not used: the block words are themselves in the capsule.
|
||||
|
||||
For each capsule, in order:
|
||||
|
||||
1. find it by name (`capsule_find_by_name`);
|
||||
2. recompute its hash (`capsule_validate`); a mismatch stops the boot;
|
||||
3. check its signature (`capsule_verify_signature`); invalid stops the
|
||||
boot, missing warns — v3's rule at `capsule_birth.c:585`;
|
||||
4. split the payload on `Block N` headers and feed each line to the node,
|
||||
running the node until it waits for input again;
|
||||
5. require the node to answer ` ok` to every line.
|
||||
|
||||
A line that is not accepted stops the boot and prints the capsule name,
|
||||
block number, line number and what the node said. v3's retry, which skips a
|
||||
failing line and runs the block again, is not carried over: a skipped line
|
||||
in the standard's own capsule must not pass unnoticed.
|
||||
|
||||
What the node prints while loading is not shown, except on failure.
|
||||
|
||||
In v4 the block numbers are labels for now. They become storage addresses
|
||||
when the block words exist and a capsule is copied to block storage.
|
||||
|
||||
## 6. POST
|
||||
|
||||
### 6.1 Source of the cases
|
||||
|
||||
POST is v3's own test cases, ported. The cases are in
|
||||
`v3/src/test_runner/modules/*.c`; each is a name, a line of FORTH and a
|
||||
flag saying whether an error is expected. Only cases for Required Word Set
|
||||
words are ported, chosen by word name against the standard's list.
|
||||
|
||||
### 6.2 Judging
|
||||
|
||||
v3 judges a case only by whether it raised an error
|
||||
(`v3/src/test_runner/test_common.c:258-285`); the "expected" text is a
|
||||
comment. A wrong result that raises no error passes. That is not enough for
|
||||
POST's purpose here, which is to prove each colon definition against the
|
||||
assembled word it replaces.
|
||||
|
||||
**Ruling:** every ported case compares its result. The expected data stack
|
||||
and printed output are taken from running the same line on the hosted v3
|
||||
binary, so v3 remains the reference. A case v3 expects to raise an error
|
||||
must raise one on v4.
|
||||
|
||||
Where v3 departs from FORTH-79 for a standard word, FORTH-79 wins (standing
|
||||
ruling); the case's expected value is then the standard's, and the
|
||||
departure is reported.
|
||||
|
||||
### 6.3 Form
|
||||
|
||||
`post79.4th` defines a small harness in FORTH — a case is written
|
||||
`T{ 1 2 3 ROT -> 2 3 1 }T` — and a tally. Cases that check printed output
|
||||
or an expected error use harness words for those. The harness needs one
|
||||
thing from the nucleus that FORTH-79 does not provide: a way to run a case
|
||||
and regain control if it raises an error. That is one nucleus word.
|
||||
|
||||
After the last case POST prints the number of cases, passes and failures,
|
||||
and names each failing case.
|
||||
|
||||
### 6.4 Generating the expected values
|
||||
|
||||
A script reads the v3 test modules, keeps the cases for Required Word Set
|
||||
words, pipes each line through the hosted v3 binary, and writes the
|
||||
`T{ ... }T` lines. It is a development tool, run when the cases change; its
|
||||
output, `post79.4th`, is committed and reviewed like any source.
|
||||
|
||||
## 7. Hashing, signing and parity
|
||||
|
||||
Hashing and signing are the existing capsule mechanisms, used unchanged
|
||||
(section 5.3).
|
||||
|
||||
Parity needs one new function. `sk_dict_canonical_hash` walks v3's
|
||||
`DictEntry` list and cannot hash a v4 node. The v4 dictionary hash is
|
||||
FNV-1a (`fnv1a_64`, already in `kernel/src/vm/parity.c`) over the node's
|
||||
memory from the start of code to `HERE`, and `LATEST`. Stacks, input
|
||||
buffers, block buffers and heat are left out.
|
||||
|
||||
The boot prints, in order:
|
||||
|
||||
```
|
||||
PARITY:V4_NUCLEUS words=N image_hash=0x...
|
||||
PARITY:V4_CAPSULE name=v4:forth79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
|
||||
PARITY:V4_CAPSULE name=v4:post79.4th capsule_id=0x... capsule_hash=0x... dict_hash=0x...
|
||||
PARITY:V4_POST tests=N pass=N fail=N
|
||||
PARITY:OK
|
||||
POST: PASSED
|
||||
ok>
|
||||
```
|
||||
|
||||
On any failure it prints `PARITY:FAIL` and `POST: FAILED` in place of the
|
||||
last three lines and does not give a prompt.
|
||||
|
||||
The tags are `PARITY:V4_*` so tooling that looks for v3's `PARITY:M7.1a`
|
||||
is not misled.
|
||||
|
||||
Every build uses 64-bit cells on the same engine, so every line above is
|
||||
identical on all six builds of one commit (section 9). The dictionary hash
|
||||
changes whenever a word moves between the nucleus and the capsule, so
|
||||
there is no fixed golden hash until decomposition is finished.
|
||||
|
||||
The nucleus is data linked into the binary and is trusted as the binary's
|
||||
own code is. It is hashed for the parity line and not signed.
|
||||
|
||||
## 8. Order of work
|
||||
|
||||
1. **Capsule loading at boot**, hosted and bare metal, with the block-number
|
||||
change. The capsule may be nearly empty at this step.
|
||||
2. **POST against today's assembled words.** POST must pass here, on words
|
||||
`make -C v4 test` already covers. This proves POST before it judges
|
||||
anything new.
|
||||
3. **Decompose group by group.** Move one group from `.v4` to
|
||||
`forth79.4th` and run POST hosted: stack, arithmetic, comparison,
|
||||
memory, strings, number output, control, defining, vocabulary, blocks.
|
||||
A failure points at the group just moved.
|
||||
4. **Acceptance** (section 9).
|
||||
|
||||
## 9. Products and acceptance
|
||||
|
||||
The hosted binary and the kernel run the same engine (`v4/src`), the same
|
||||
nucleus image, the same baked capsule directory and the same boot sequence
|
||||
(section 5.3 and section 7). Only the console differs: stdin and stdout
|
||||
hosted, the serial console on bare metal. The hosted binary links
|
||||
`capsule_generated.c` and reads nothing from the source tree.
|
||||
|
||||
Acceptance is six boots of one commit:
|
||||
|
||||
| | amd64 | aarch64 | riscv64 |
|
||||
|---|---|---|---|
|
||||
| Hosted Linux | native | user-mode QEMU | user-mode QEMU |
|
||||
| Bare metal | `clean qemu` | `clean qemu` | `clean qemu` |
|
||||
|
||||
Each must print identical `PARITY:V4_*` lines, pass POST and reach `ok>`.
|
||||
Bare-metal runs follow the repository's QEMU rules: one instance at a time,
|
||||
in the foreground, `clean` before `qemu`, logs kept under `logs/`.
|
||||
|
||||
`make -C v4 test` also runs at 32-bit cells. That stays as a development
|
||||
check for the FPGA gateway and is not part of this acceptance.
|
||||
|
||||
## 10. What is reused and what is new
|
||||
|
||||
| Need | Existing | Change |
|
||||
|---|---|---|
|
||||
| Bake capsules | `tools/mkcapsule.c`, `kernel/Makefile` capsule rule | Block-number check only (5.2) |
|
||||
| Find, hash-check, verify | `capsule_find.c`, `capsule_validate.c`, `capsule_sig.c` | None; also compiled into the hosted binary |
|
||||
| Split a payload into blocks and lines | `is_block_header` and the walk in `capsule_loader.c` | Moved to a file both loaders call |
|
||||
| Start a node from an image | `v4/src/image.c`, `v4_image_boot` | None |
|
||||
| Build the nucleus image | `v4/tools/mkimage.c` | File list shrinks as groups move; the step that types `editor.fth` and `tools.fth` is removed |
|
||||
| Kernel entry | `kernel/src/v4/sk_v4.c`, Kconfig `STARFORTH_V4` | Calls the shared boot sequence before reading the keyboard |
|
||||
| Hosted entry | `v4/tools/hosted.c` | Same |
|
||||
| Parity lines and FNV-1a | `kernel/src/vm/parity.c` | v4 dictionary hash added |
|
||||
| v3 cases | `v3/src/test_runner/modules/*.c` | Read only |
|
||||
| Expected values | hosted v3 binary | Read only |
|
||||
| Hosted builds per ISA | `v4/Makefile` | Cross-compiler targets for aarch64 and riscv64 |
|
||||
|
||||
New: the shared v4 boot sequence (one file), the v4 dictionary hash, the
|
||||
case-extraction script, `forth79.4th` and `post79.4th`.
|
||||
|
||||
The v3 boot is untouched: `STARFORTH_V4` defaults to off.
|
||||
|
||||
## 11. Not decided here
|
||||
|
||||
- Where the v4 hosted binaries are installed. They are built under
|
||||
`v4/build/`; `lfs/` holds v3's and is left alone.
|
||||
- Signing the nucleus. It would have to become a named blob under
|
||||
`capsules/`, which puts a generated file in a source directory.
|
||||
Reference in New Issue
Block a user