LithosAnanake/docs/v4.0.0/MESH.md

# StarForth v4.0.0 — Talking nodes

Design, 2026-10-06. Ruled by Captain Bob by question and answer on
2026-10-05 and -06; each ruling is recorded with his words in `ENGINE.md`
section 3d. This file is the design that follows from them. Where it goes
beyond a ruling it says so, and those parts are proposals.

## 1. What this step is

> The entire point is that ultimately we have F18 engines digesting
> capsules alone, and in a sense can be anything written in F18 assembler
> for our fabric — 144 someday as a 12x12 grid, but I want a 12^3 FPGA
> ultimately, where using a capsule digester like StarForth, it's more
> than an operating system.

> The next step is talking nodes sharing the common SSD, and [they] may or
> may not have block storage available.

More than one node, each born empty and made into something by the capsule
it takes in, talking to each other through ports, sharing a disk. It comes
before making v4 equal v3 on bare metal (`ENGINE.md` step 4), which waits.

## 2. The rulings

1. **The ports are the transport; the message is what is transported.**
   Node to node, a write blocks until the neighbour reads and a read blocks
   until the neighbour writes (`DECOMPOSITION.md` section 6). What travels
   is v3's Hermes message with what it carries, so v3's messaging rules are
   not dropped: they go with the message.
2. **Some nodes have storage of their own**, in addition to or instead of
   the common SSD. Some have none.
3. **The first set of nodes is "2x2 + 1 central"**: five.
4. **The geometry is data, not design.** A node has a number of ports, and
   that number is a parameter. Which port connects to what is a table that
   can change while the system runs. "Not constrained by a 3D world. Other
   geometries might be better."
5. **The first geometry: "Only the central node can connect to only another
   central node."** Four outer nodes and their centre are a unit. An outer
   node is wired only inside its unit. Units are joined centre to centre.
6. **It must scale at run time, adaptively.**
7. **Hera decides which nodes exist and are awake, not whose turn it is.**
   Every node that is not blocked runs. Cooperative is a node blocking
   itself at a port; preemptive is Hera putting a node to sleep or killing
   it between any two instruction words. An idle node does not spin: it is
   blocked reading its ports.
8. **A node is born empty.** It has nothing but the ability to listen. The
   first thing a neighbour sends it is a capsule of F18 code. StarForth's
   nucleus is such a capsule.
9. **Built in the shared engine, proven hosted on three ISAs first, then on
   bare metal.**

## 3. Acceptance (approved 2026-10-06)

1. **Five nodes come up from nothing.** Hera is born empty, takes in the
   nucleus capsule and the FORTH-79 capsule, and passes POST. She births
   the four outer nodes while running; each is born empty, takes in the
   same capsules through its port from Hera, and passes POST. Each prints a
   parity line; the four outer nodes' dictionary hashes are identical.
2. **They talk.** A line typed at the console reaches Hera as a message. A
   message from Hera runs on an outer node and its output comes back. A
   message between two outer nodes that are not wired to each other is
   forwarded by a node in between.
3. **They share the SSD.** A block written by one node is read by another.
   One outer node has a drive of its own and uses it. One has no storage,
   and `BLOCK` on it is an error with a message.
4. **It scales while running.** A second unit of five is born and joined
   centre to centre. A message crosses from one unit to the other. The
   second unit is removed and the first carries on.
5. **Hera manages.** An idle node executes nothing while it waits, which
   the anti-clock shows. Hera puts a node to sleep and wakes it. Hera kills
   a node that is stuck in an endless loop, and everything else keeps
   running.
6. **On every build.** The three hosted ISAs, then the three bare-metal
   ISAs under QEMU with the real console and disk.

**Left for the step after, by agreement:**

- Hera sleeping, waking and killing nodes by herself from each node's
  heat. Here she does it by command. The rule for it has not been given.
- v3's messaging rules checked at every hop. From this step a message
  carries its heat, TTL and ACL tag; checking them is the router's, and
  the checks await rulings.

## 4. The engine: ports

Everything in this section is the engine's (`v4/src`) and knows nothing of
StarForth or of any kernel.

### 4.1 A node's ports

A node has `V4_PORTS` ports. The number is a build parameter, as
`V4_CELL_BITS` and `V4_NODE_WORDS` are. (Proposal: 8 for now, which is what
the first geometry needs of a centre — four outer nodes, two devices, two
other centres — with nothing depending on the number.)

Ports are addresses, as `DECOMPOSITION.md` section 6 has them. Attached at
word address `base`:

| Address | What |
|---|---|
| `base` … `base + V4_PORTS − 1` | Port 0 … port `V4_PORTS − 1` |
| `base + V4_PORTS` | Any port: a read here takes from whichever port has a neighbour writing |
| `base + V4_PORTS + 1` | Which port the last read from "any" came from. Read only. |

- **A write to a port blocks the node until the neighbour has read it.**
  Built 2026-10-05 for one port; the opcode after the write runs when the
  write has been taken.
- **A read from a port blocks the node until the neighbour writes.** The
  fetch does not happen until there is something to fetch. This is new.
- **A read is any fetch**: `@`, `@b`, `@+`, the literal fetch `@p`, and the
  fetch of an instruction word when `P` is a port address. When `P` is a
  port, it is not advanced: the node goes on executing what arrives there.
  That is how an F18 node runs code from a port, and it is what makes
  ruling 8 need no code in a newborn node.
- A port with nothing on the other end blocks for ever, as on the fabric.

### 4.2 A node at reset

`P` is the "any port" address, both stacks are empty, memory is zero. The
node is blocked reading its ports. Nothing else is in it.

### 4.3 The fabric

`v4/src/fabric.c`: the nodes there are, how their ports are wired, and
time passing for all of them at once.

- **The nodes.** A set that grows and shrinks while the system runs
  (ruling 6). A node is added empty (4.2) and removed whole.
- **The wiring.** For each port of each node: nothing, or a port of
  another node, or a device. It is a table, changed at run time
  (ruling 4). The fabric does not know what shape it makes.
- **A device** is what is on the other end of a port that is not a node:
  two functions, one that takes a word the node writes and one that gives
  a word when the node reads, each able to say "not yet". The console, a
  disk, and the kernel that serves a node's requests (`ENGINE.md` 3.3) are
  devices.
- **A step.** Every node that is awake and not blocked executes one
  instruction word. Then every write that has a reader waiting on the
  other end of its wire is handed over, and both nodes are unblocked. That
  is all: there is no choice of whose turn it is (ruling 7).
- **Asleep.** A node that is asleep executes nothing and nothing is handed
  to it or taken from it. It is put to sleep and woken from outside, at
  any instruction word.

### 4.4 What is not in the engine

Messages, routing, capsules, the unit of five, Hera. Those are sections 5
to 8 and are made of capsule code and of the host that owns the devices.

## 5. A capsule of F18 code, and how an empty node takes it in

**As built.** A capsule of F18 code is not a format a node has to
understand. It is the words a neighbour writes to the node's port, in the
order they are written: for each stretch of memory that is not zero the
two instruction words below with their address and count and then the
words themselves, and at the end a jump to where to start. The file is
those words, each in a cell's bytes, low byte first, and its hash is the
hash of exactly what is sent. The nucleus is one, in the capsule directory
with a name, a hash and a signature like any other. `mkcapsule` was not
changed: the nucleus capsule is a built file kept under `capsules/v4/`.

A neighbour puts it into an empty node by writing to the port between
them, and the node executes what arrives (4.1). For each run it sends

```
@p a! @p push       \ then the address, then count − 1: executed from the port
@p !+ unext         \ then the words: each is fetched from the port and stored
```

and at the end `jump` to the start address. That is the F18's own way,
and it needs nothing in the node beforehand.

## 6. A message

**Proposal**, from `DECOMPOSITION.md` section 6.1 and v3's `SkHermesMessage`
(`kernel/include/starkernel/vm/kernel_hermes.h`), which it must be able to
carry whole:

| Word | Contents |
|---|---|
| 0 | To: the node it is for |
| 1 | From: the node that sent it |
| 2 | Type, and the channel |
| 3 | Heat and TTL |
| 4 | ACL tag: the sender's identity fingerprint |
| 5 | Sequence |
| 6 | Payload length in characters, 0 to 1024 |
| 7 … | Payload: FORTH text, four characters to a word |

It is written to a port a word at a time and read a word at a time. Words
3 and 4 are carried from this step on and are not yet checked
(section 3).

A node that is a StarForth digester, when it has nothing to do, reads a
message from "any port". If it is for this node, the payload is
interpreted, as a line is today (`ENGINE.md` 3.1). If it is for another,
it is written to the port that leads there (section 7).

What a node prints goes to its console, and its console is a place like
any other: for the node wired to the console device, that port; for any
other node, a message to the node that is. So what an outer node prints
comes back through its centre.

## 7. Finding the way

**Proposal.** Each node has a small table: for a destination, the port
that leads toward it; and one port for everything not in the table.
Whoever wires a node writes its table, and changes it when the wiring
changes. Under the first geometry an outer node's table is its two grid
neighbours and "everything else to my centre"; a centre's is its four
outer nodes, and for each other unit the port toward that unit's centre.

A different geometry is a different way of filling in the wiring table
and these tables. Nothing else changes.

## 7a. Two neighbours writing to each other (ruled 2026-10-06)

The fault is in section 10, step 4. **Ruled: a node writes only to a
neighbour that is already reading; a node keeps the messages it takes in
meanwhile, and one there is no room for is lost and counted.**

**As built.**

- *The engine.* Two more addresses follow a node's ports, as the F18's
  `io` register would give them: which ports have a neighbour waiting to
  write to this node, and which have one waiting to read from it, a bit to
  a port. Fetching them waits for nothing and changes nothing
  (`v4/src/node.c`; the fabric keeps them, `v4/src/fabric.c`). A device
  that takes what is written to it shows as waiting to read.
- *Looking before writing* (`(GATE)`, `v4/capsule/core.v4`). Before a node
  begins any message it takes in every message a neighbour is waiting to
  write to it; then it writes if the neighbour it means to write to is
  waiting to read; and if not, it looks again. Once a message is begun it
  is written to its end: the neighbour that took its first word reads the
  rest.
- *The messages waiting* (`(MQ)`). What a node takes in is kept in a ring
  of 400 cells of its own memory -- for each message its seven words, the
  port it came on, and its text -- and dealt with, oldest first, when the
  node has nothing else to do. A node with none waiting is blocked reading
  its ports, as before. One there is no room for is read to its end, let
  go, and counted in `(LOST)`.

**What had to be added to the ruling, and why.** As put to Captain Bob the
rule was "write only to a neighbour that is reading, and look again if it
is not". That is not enough: two neighbours each with a message for the
other would each look, see the other not reading, and look again, for
ever. No rule that treats both ends of a wire alike can get out of that.
So one end of every wire may write without waiting for a reader, and one
only: **the node with the lower number.** A node that waits to write is
then always waiting on a higher number, so no ring of nodes can all be
waiting on each other, and the highest of any that wait is not waiting to
write: it is looking, and takes in what is being written to it.

For this a node must know the number of the node on each of its ports.
Whoever wires it tells it, as it is told the ways: `NEIGHBOUR ( node port
-- )`, `v4/capsule/quit.v4`. A port it has not been told of -- a device's
-- is written to only when what is there is waiting to read. **Two nodes
wired together and not told of each other can still stop each other**,
each looking for ever; they execute, but nothing passes. Telling them is
part of wiring them. Put to Captain Bob after it was built, and ruled:
"1 is fine."

**What it costs.** A node that is flooded loses messages, of every kind:
text for it, answers to text it sent, and messages it was only passing
on. In the test three nodes send each other 800 messages at once; 287
arrive and 714 messages are let go (the count includes answers and text
that was to start a node sending). Six from each to each, at once, all
arrive. Nothing here makes a sender slow down or send again; that is
kernel-Hermes's work in v3 and is not decided for the mesh.

## 8. Storage

**Proposal.** A disk is a device on a port. Asking for a block is a
message to it — read block *n*, or write block *n* with 1024 characters —
and the answer is a message back; a block is exactly the most a message
carries. Under v3's one block-number space, each device has its own range
of numbers.

- A node wired to a disk asks it directly.
- A node not wired to one asks through the node that is: the message is
  forwarded like any other. So the common SSD is shared by every node that
  has a way to the node that holds it.
- A node with its own drive has that device on one of its own ports.
- A node with no way to any disk has no storage, and `BLOCK` says so.

Who may have which block is the block card's, decided where the disk is,
under the identity in the message's ACL tag (`V3-PARITY.md` 1d). Not
checked in this step.

## 9. Birth, and Hera

**Proposal, built as step 5 (section 10).** Hera asks, through the port
where her requests go (`ENGINE.md` 3.3), for a node to be added and wired;
she then sends the newborn its capsules through the port that joins them.
Putting a node to sleep, waking it, and removing it are requests of the
same kind. Only Hera's requests are honoured.

The unit rule (ruling 5) is Hera's, in capsule code: it is what she does
with those requests. The engine and the host do not know it.

## 10. Steps

Each is tested, committed and pushed before the next.

1. **Ports and the fabric (section 4).** Reads; "any port"; a node at
   reset; execution from a port; nodes added and removed; wiring changed;
   asleep and awake. Tests at the level of opcodes: two nodes exchange
   words; an empty node is filled through its port and runs what it was
   sent; a word is passed on by a node in between; a node waiting executes
   nothing; a node is put to sleep, woken, and removed while looping.
   **Done 2026-10-06.** `v4/src/node.c` (the ports, `v4_node_born`),
   `v4/src/exec.c` (a fetch from a port waits; execution from a port; the
   slot a blocked node goes on from), `v4/src/fabric.c`. `V4_PORTS` is 8.
   `v4/tests/test_fabric.c`, 53 checks at both cell widths and under ASan
   and UBSan: all of the above, with the empty node filled once by a device
   and once by another node holding the capsule in its own memory; the
   fabric given more room while nodes run; what cannot be wired refused.
   The single-node products are unchanged by it: `hosted-check` on three
   ISAs, and the three bare-metal boots with lines typed at each prompt
   (`logs/20261006-074907`, `-075150`, `-075532`).
2. **The nucleus as a capsule, and an empty node made a StarForth node
   through its port (section 5).**
   **Done 2026-10-06.** `v4/src/capsule.c`, `v4_capsule_write`: any
   node's memory as the words to send an empty node. `mkimage` writes the
   nucleus so, to `capsules/v4/nucleus-64.f18`, a built file kept in the
   tree as `BLOCK_MAP.md` is; `mkcapsule` is unchanged and bakes, hashes
   and signs it with the rest. The boot's node is born empty
   (`v4_image_born`) and is given the nucleus a word at a time as it reads
   its port, after the capsule's hash and signature are checked
   (`v4/system/boot.c`). Nothing of the nucleus is linked into either
   product any more. `test_fabric.c`: a memory with a programme and words
   here and there arrives word for word and runs. Both products boot so:
   `hosted-check` on three ISAs; `logs/20261006-102421`, `-102706`,
   `-103048`.
   **What a newborn node still has from its host:** the registers the
   nucleus expects — the console's three, the two stack registers, the
   error register, the fault table's address, the storage registers — are
   attached by the host when the node is born, from the description
   `mkimage` writes. They are the node's hardware as the golden model has
   it, at addresses the memory map (D-4, still open) will fix. The console
   and storage ones go when those become devices on ports (steps 3 and
   6).
3. **Messages (section 6):** a StarForth node that reads messages when
   idle; the console as a device; a line typed is a message.
   **Done 2026-10-06, but for the keyboard.** A node with nothing to do is
   blocked reading "any port" (`(IDLE)`, `v4/capsule/quit.v4`). What
   arrives is a message; text for this node is interpreted; what it prints
   is kept and sent back as messages to the sender, on the port the
   message came on, and then a message saying how the text ended (`EMIT`,
   `(FLUSH-OUT)`, `(HDR)` in `core.v4`; `(FINISH)` in `quit.v4`).
   `v4/src/message.c` is the same format for whatever is on the other end
   of a port and is not a node. The boot is that, on the node's port 1, as
   the console (`v4_boot_line`); the prompt tests are too. Handing a node a
   line by writing its input buffer and setting its `P` is gone
   (`v4_line_begin` and the rest). Nothing reads the `CONSOLE-TX` register
   any more.
   `EMIT` still needs one free cell of the data stack and no more, as
   before; it keeps its working values on the return stack. The full-stack
   tests hold at the same figures as before.
   All v4 tests pass at both widths and under ASan and UBSan;
   `hosted-check` on three ISAs; bare metal `logs/20261006-110551` (amd64),
   `-111621` (aarch64), `-111341` (riscv64). `-110837` is an aarch64 run
   that was ended by the test wrapper's limit while still in UEFI firmware,
   before the kernel had started; it shows nothing about v4.
   **Not done:** `KEY`, `EXPECT` and `QUERY` still read the console's two
   input registers, not a message. A message not for this node, or not
   text, is let go: passing it on is step 4.
4. **Finding the way (section 7).**
   **Built 2026-10-06.** Each node has a table of up to 16 destinations and the port
   toward each, and one port for everything else (`ROUTE ( node port -- )`,
   `DEFAULT-ROUTE ( port -- )`, `NO-ROUTES`; `(PORT-FOR)` in
   `v4/capsule/core.v4`). A message not for this node is written, whole,
   to the port its destination's entry names (`(PASS-ON)`,
   `v4/capsule/quit.v4`); with no entry and no port for everything else it
   is dropped and counted (`(LOST)`). What text prints, and how it ended,
   go back to the node the text came from by the same table, so an answer
   crosses as many nodes as the text did. `SEND ( baddr u node -- )` sends
   text to another node; `(ME)` is a node's own number; `(CONSOLE)`, when
   set, is where a node's printing goes instead of to the sender.
   `v4/tests/test_host_mesh.c`, 28 checks at both widths and under ASan
   and UBSan: three StarForth nodes in a row behind a console; text for
   the far one passes through the other two and its answer comes back;
   each node keeps its own dictionary; output longer than one message; a
   1024-character message passed on whole; a message with nowhere to go
   counted; a table changed while running. The single-node products are
   unchanged: `hosted-check` on three ISAs; bare metal
   `logs/20261006-115225` (amd64), `-115501` (aarch64), `-115849`
   (riscv64).
   **The fault found here, and put right 2026-10-06 (section 7a).** Two
   neighbours that wrote to each other at once waited for ever: a write
   blocks until the neighbour reads, and a node that is blocked writing is
   not reading. Found when the far node `SEND`s text to the middle one and
   then sends word of how its own text ended, which goes by the middle
   one, while the middle one answers the far one. Ruled: a node writes only
   to a neighbour that is reading, keeps what it takes in meanwhile, and
   loses and counts what it has no room for. Built so, with one thing
   added that the ruling needs (section 7a): of two neighbours the lower
   number may wait to write, and `NEIGHBOUR` tells a node who is on each
   port. `test_host_mesh.c`, 44 checks: the case that stopped the nodes
   now passes; every node sending every other six messages at once, all
   arrive; 800 at once, the nodes come to rest and every message either
   arrived or was counted. `test_fabric.c`: a node sees who is waiting to
   write to it and who to read from it, and looking disturbs neither.
   The prompt and `EMIT` take no more of the data stack than they did.
   All v4 tests at both widths and under ASan and UBSan; `hosted-check` on
   three ISAs; bare metal, with lines typed at each prompt,
   `logs/20261006-134817` (amd64), `-135044` (aarch64), `-135440`
   (riscv64).
   **Found on the way: POST was writing into the nucleus's own code.** On
   v4 `HERE` is a cell address and `C!`, `FILL` and `CMOVE` take byte
   addresses (D-1), so cases such as `65 HERE C!` wrote their bytes into
   the nucleus at cell `HERE/4`, read them back from there, and passed.
   `65 HERE C!` was watched changing cell 2223 during POST; the other
   cases of the same form were not watched, and the boots committed
   before this were not gone back to. It showed only when this step moved
   other code onto that cell and POST stopped. Ruled: the twelve cases
   are left out, marked OPEN (`v4/tools/post79_rules.py`,
   `docs/v4.0.0/POST79.md` section 4), and how `HERE` and the byte words
   are to agree is its own step. POST is 538 cases, not 550. `C!` and
   `FILL` now have fewer cases; nothing stops any other text doing what
   those cases did.
5. **Birth and Hera's requests (section 9); the unit of five.**
   **Done 2026-10-06, in the fabric under test; the products are still one
   node each until steps 8 and 9.** Section 9's proposal, as built:
   - *What Hera asks* (`v4/include/v4/manage.h`, `v4/src/manage.c`). Eleven
     requests, each a word on her made with `KERNEL-WORD`, written to the
     port her requests go to: `NODE-ME`, `NODE-BORN`, `NODE-WIRE`,
     `NODE-UNWIRE`, `NODE-SLEEP`, `NODE-WAKE`, `NODE-KILL`, `NODE-PARITY`;
     and `CAPSULE-OPEN`, `CAPSULE-CELL`, `CAPSULE-LINE`, by which she reads
     a capsule the host has found and checked. A node is named to the host
     by its place in the fabric; its number, which messages go by, is the
     nodes' own. Only the node the host has wired to this is answered.
   - *What a node needs to do it* (`v4/capsule/quit.v4`): `PORT! ( w port
     -- )`, a cell written to a port as it is; `SEND-ON ( baddr u node port
     -- )`, text by a port named, for a neighbour with no number yet;
     `AWAIT ( node -- how )`, blocked until that node's word of how text
     ended comes, keeping every other message for later; `(SEAL)`, what is
     in the dictionary now is the system.
   - *Hera's capsule* (`capsules/v4/hera.4th`, blocks 8000 to 8004), FORTH.
     `BIRTH ( number port -- place )`: a node is asked for and wired to
     that port; the nucleus is sent it cell by cell; it is told its number,
     its console and that Hera is on its port 2; FORTH-79 and POST are sent
     it a line at a time, each waited for; it seals; its parity is
     recorded. `UNIT ( n -- )`: four births, on ports 2 to 5, and the four
     joined in a square, each told of the two beside it. The unit rule is
     there and nowhere else.
   `v4/tests/test_host_unit.c`, 34 checks, 64-bit: Hera is born empty and
   takes the nucleus through her port, then FORTH-79, POST (538 of 538) and
   her capsule from the console; `10 UNIT`; four nodes are born, each
   passes POST, the host records four parities with one dictionary hash;
   all five wait and execute nothing; text for each outer node is done
   there and what it prints comes back; `COLD` on one comes back to the
   nucleus and FORTH-79; one outer node sends text to the one beside it,
   and to the one across from it by way of Hera. Acceptance 1 and 2, in
   the fabric. All v4 tests at both widths and under ASan and UBSan. The
   single-node products are unchanged but for the nucleus's new words:
   `hosted-check` on three ISAs; bare metal `logs/20261006-143934`
   (amd64), `-144204` (aarch64), `-144601` (riscv64).
   **What stands in, in the test only:** the capsules are read from the
   files the build bakes in, and the nucleus capsule is written from the
   nucleus as the test assembles it, not found in the baked directory with
   its hash and signature checked; that is the host's part and comes with
   step 8. Each node has a disk of its own attached at birth, as the boot's
   one node has; storage through the ports is step 6.
   **Not done, and open:**
   - The test is not run on 32-bit cells: the POST capsule's expected
     results are 64-bit v3's and the nucleus capsule is `nucleus-64`. POST
     has never been run at 32 bits.
   - A node that never answers leaves Hera waiting for ever in `AWAIT`;
     she cannot then kill it. What she does about a birth that does not
     finish is not decided.
   - What an outer node prints while Hera is giving birth waits in Hera's
     400 cells until she is idle, and is lost if there is more than they
     hold. POST prints one line.
   - An outer node has no port to the kernel, so no kernel words: no
     `BYE`, and nothing it asks is honoured, as ruled.
   - The suite now takes about four minutes, eight under the sanitizers:
     five POSTs.
6. **Storage (section 8).**
7. **A second unit; scaling while running; sleep, wake and kill by
   command.**
8. **The hosted product is the five nodes**, on three ISAs: acceptance 1
   to 5.
9. **The same on bare metal:** acceptance 6.

## 11. What becomes of what is there

- `v4/system/boot.c` and `v4/tools/hosted.c` hand one node a line and run
  it to the end. They stay until step 8, where the hosted product becomes
  the five nodes.
- `(LINE)` and `(IDLE)` (`v4/capsule/quit.v4`): `(LINE)` stays, as what
  interprets a message's payload. `(IDLE)` becomes the read of a message
  at step 3.
- The one port of 2026-10-05, where a node's requests go, is port 0 of the
  ports of 4.1.
- The nucleus linked into the binary goes at step 2.
- `DECOMPOSITION.md` section 6 says four ports; it is corrected at step 1.