Single-block relocation: RELOCATE-BLOCK, resolve_lbn(), persisted exception table
Implements the full design from the prior commit in one pass. resolve_lbn()
is the single choke point threaded through the ten public LBN-consuming
entry points (blk_get_buffer, blk_update, blk_flush, blk_is_allocated,
blk_mark_allocated, blk_mark_free, blk_is_valid, blk_get_meta, blk_set_meta,
plus blk_get_empty_buffer covered via delegation) -- an LBN->LBN redirect,
not a new storage allocator, since the LBN space is already unified across
every attached blkio_dev backend. VM window cache staleness across a
relocation reuses the existing blk_vm_check_epoch() mechanism from
Milestone 2h's hot-detach fix for free -- g.epoch bumps on relocation too.
Persistence lands in the same pass: two new uint32_t fields
(reloc_start/reloc_devblocks) appended after hdr_crc in blk_volume_meta_t,
carved from existing padding without moving any earlier field's byte
offset -- an old formatted volume's zeroed padding reads back as
reloc_devblocks=0 ("no reloc capacity"), gracefully, not a format-breaking
change. compute_totals_from_B() generalized to account for the new
reserved region. reloc_flush_to_disk()/reloc_load_from_disk() mirror the
BAM I/O functions' own absolute-devblock-addressing shape; the persisted
copy's owner is first_disk_slot() (already existed, already used for this
exact "which device is canonical" question by blk_get_volume_meta()).
blk_subsys_relocate_block() is a mechanical primitive only -- copies
content (staged through a local buffer, since obtaining the target's
blk_get_buffer() result can evict and invalidate the source's cache
pointer if they share a device), frees the source BAM entry, appends the
exception entry, bumps the epoch, flushes to disk. RELOCATE-BLOCK exposes
it to FORTH, no policy of its own (ACL's job, per this session's direction).
A first live-test attempt gave a false negative against disk/artemis.img
(predates reloc capacity, so relocation only ever existed in memory that
boot) -- traced to the test's own setup before being mistaken for a bug,
then re-verified correctly against a fresh volume (new fixture,
disk/artemis-reloc-test.img): relocated a RAMDRIVE block to the fresh
disk, confirmed live resolution through the redirect, then confirmed both
the redirect and the relocated content survived an abrupt QEMU kill and
full reboot. Also fixed three lingering "glibc" doc-comment
misattributions from Milestone 2h (the actual allocator is this kernel's
own kmalloc) that survived an earlier FABRIC-2.md-only correction. All
three architectures re-verified clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CXjAPTEKrgY2Mrk25KoLDn
This commit is contained in:
co-authored by
Claude Sonnet 5
parent
073dae4f56
commit
36d832ff47
+52
-7
@@ -4139,19 +4139,64 @@ mechanical primitive only — copies content, updates BAM/exception-table state,
|
|||||||
signal) and *validating* that a target LBN is genuinely owned by the right identity are ACL's
|
signal) and *validating* that a target LBN is genuinely owned by the right identity are ACL's
|
||||||
job, per this session's own direction — not something this function enforces on its own.
|
job, per this session's own direction — not something this function enforces on its own.
|
||||||
|
|
||||||
- [ ] Add `reloc_start`/`reloc_devblocks` to `blk_volume_meta_t`, adjust fresh-format geometry
|
- [x] Add `reloc_start`/`reloc_devblocks` to `blk_volume_meta_t`, adjust fresh-format geometry
|
||||||
to reserve one devblock for the reloc table
|
to reserve one devblock for the reloc table
|
||||||
- [ ] Implement `resolve_lbn()` and thread it through the ten public entry points listed above
|
- [x] Implement `resolve_lbn()` and thread it through the ten public entry points listed above
|
||||||
- [ ] Implement `reloc_flush_to_disk()`/`reloc_load_from_disk()` (mirroring the BAM I/O
|
- [x] Implement `reloc_flush_to_disk()`/`reloc_load_from_disk()` (mirroring the BAM I/O
|
||||||
functions' own shape), wired into `blk_subsys_attach_device()` (load once, from
|
functions' own shape), wired into `blk_subsys_attach_device()` (load once, from
|
||||||
`first_disk_slot()`) and `blk_commit_format()` (zero the region on fresh format)
|
`first_disk_slot()`) and `blk_commit_format()` (zero the region on fresh format)
|
||||||
- [ ] Implement `blk_subsys_relocate_block(home_lbn, target_lbn)`: copy content, free the
|
- [x] Implement `blk_subsys_relocate_block(home_lbn, target_lbn)`: copy content, free the
|
||||||
source BAM entry, append the exception entry, bump `blk_subsys_epoch()`, flush to disk
|
source BAM entry, append the exception entry, bump `blk_subsys_epoch()`, flush to disk
|
||||||
- [ ] Add a FORTH word exposing it (matching this project's "compose in FORTH first"
|
- [x] Add a FORTH word exposing it (matching this project's "compose in FORTH first"
|
||||||
convention — ACL/higher-level policy code needs a callable entry point)
|
convention — ACL/higher-level policy code needs a callable entry point). `RELOCATE-BLOCK
|
||||||
- [ ] Live-verify: relocate a block between two writable devices (Artemis disk ↔ RAMDRIVE),
|
( home target -- )`
|
||||||
|
- [x] Live-verify: relocate a block between two writable devices (Artemis disk ↔ RAMDRIVE),
|
||||||
confirm content and the redirect both survive an abrupt-kill/reboot cycle (same
|
confirm content and the redirect both survive an abrupt-kill/reboot cycle (same
|
||||||
methodology as Section V item 6's own persistence test)
|
methodology as Section V item 6's own persistence test)
|
||||||
|
|
||||||
|
**All done 2026-08-25, in one pass (design → implementation → live verification), all three
|
||||||
|
architectures re-verified. Full writeup:**
|
||||||
|
|
||||||
|
Implemented exactly as designed above, with one addition found necessary mid-implementation:
|
||||||
|
`compute_totals_from_B()` (the function that derives `total_blocks`/`tracked_blocks` from a
|
||||||
|
device's geometry) hardcoded its payload-region math as `total_devblocks - 1 - bam_devblocks`
|
||||||
|
(the `1` being the header devblock) — didn't yet know about the new reloc region at all.
|
||||||
|
Generalized to `total_devblocks - 1 - bam_devblocks - reloc_devblocks`, reading
|
||||||
|
`reloc_devblocks` off the same `blk_volume_meta_t` the caller already populated; single call
|
||||||
|
site, no other changes needed.
|
||||||
|
|
||||||
|
**A first live-test attempt gave a false negative, caught before being mistaken for a bug.**
|
||||||
|
The first persistence run used `disk/artemis.img` as the relocation target: attach, write,
|
||||||
|
relocate (LBN 2048 → 25000), live read-back correctly showed the written byte (`119`), abrupt
|
||||||
|
kill, reboot — read-back came back `0`, not `119`. Before concluding the mechanism was broken,
|
||||||
|
traced it to the test's own setup: `artemis.img` is a volume formatted *before* this session's
|
||||||
|
work, so its on-disk `reloc_devblocks` reads back as `0` from what was previously unused header
|
||||||
|
padding — exactly the documented, intentional "old volumes gracefully lack reloc capacity"
|
||||||
|
behavior from this design's own persistence section, not a defect. `reloc_flush_to_disk()`
|
||||||
|
correctly no-op'd (nothing to write to); the relocation only ever existed in memory for that one
|
||||||
|
boot, which is exactly why it didn't survive a reboot. Re-ran against a genuinely fresh volume
|
||||||
|
instead (`disk/artemis-reloc-test.img`, new fixture — see `disk/README.md`): confirmed the fresh
|
||||||
|
format via `BLK-CONFIRM-FORMAT`, wrote a marker byte to LBN 2048 (the volatile RAMDRIVE), ran
|
||||||
|
`2048 3200 RELOCATE-BLOCK`, confirmed live read-back through the redirect (`119`), killed QEMU
|
||||||
|
abruptly via QMP (no `BYE`, no `SAVE-BUFFERS`), rebooted with the same disk image, and read LBN
|
||||||
|
2048 back again: `119` — both the redirect and the relocated content survived the reboot.
|
||||||
|
Because LBN 2048's *original* backing (RAMDRIVE) is volatile and resets to zero every boot, a
|
||||||
|
correct `119` read-back specifically proves the *persisted* redirect was followed, not any
|
||||||
|
leftover RAMDRIVE state. `logs/20260825-185704/amd64/` + `logs/20260825-185849/amd64/` are the
|
||||||
|
false-negative pair against `artemis.img` (kept as evidence the "old volumes lack capacity"
|
||||||
|
behavior is real and was actually hit, not just theorized); `logs/20260825-190040/amd64/` +
|
||||||
|
`logs/20260825-190503/amd64/` are the successful pair against the fresh volume.
|
||||||
|
|
||||||
|
Also fixed while in the neighborhood: the "glibc's allocator" misattribution from Milestone 2h's
|
||||||
|
epoch-based cache-invalidation fix was still present in three in-code doc comments
|
||||||
|
(`block_subsystem.h`, `block_subsystem.c`, `block_words.c`) even after being corrected in this
|
||||||
|
file's own prose earlier in the session — corrected all three to name the actual allocator
|
||||||
|
(this kernel's own first-fit `kmalloc`, `src/starkernel/memory/kmalloc.c`).
|
||||||
|
|
||||||
|
All three architectures re-verified clean with the default `disk/artemis.img` (no relocation
|
||||||
|
performed on these boots — just confirming the final code path boots clean with an old,
|
||||||
|
reloc-incapable volume attached, the common case going forward until volumes get reformatted):
|
||||||
|
`logs/20260825-190851/amd64/`, `logs/20260825-191007/aarch64/`, `logs/20260825-191200/riscv64/`.
|
||||||
- [x] Implement the `sk_repl_idle()` body — the cheap "anything dirty? no? done" check
|
- [x] Implement the `sk_repl_idle()` body — the cheap "anything dirty? no? done" check
|
||||||
(Section V confirmed this hook is empty and ready right now, doesn't even need
|
(Section V confirmed this hook is empty and ready right now, doesn't even need
|
||||||
Milestone 2 to be written, only to be *tested end to end*). **Done 2026-08-25**, see
|
Milestone 2 to be written, only to be *tested end to end*). **Done 2026-08-25**, see
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
# Capsule Block Manifest — Auto-generated
|
# Capsule Block Manifest — Auto-generated
|
||||||
<!-- Generated by mkcapsule --manifest 2026-08-25T18:44:46Z -->
|
<!-- Generated by mkcapsule --manifest 2026-08-25T23:11:30Z -->
|
||||||
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
|
<!-- DO NOT EDIT — re-run mkcapsule --manifest to refresh. -->
|
||||||
<!-- Hand-written justifications and immutability notes live -->
|
<!-- Hand-written justifications and immutability notes live -->
|
||||||
<!-- in MANIFEST.md alongside this auto-generated index. -->
|
<!-- in MANIFEST.md alongside this auto-generated index. -->
|
||||||
|
|||||||
@@ -71,6 +71,17 @@ carrying timestamp noise in git history.
|
|||||||
silently replaying the old device's cached (all-zero) content, which is
|
silently replaying the old device's cached (all-zero) content, which is
|
||||||
exactly the class of bug a same-LBN-range device swap can cause if the
|
exactly the class of bug a same-LBN-range device swap can cause if the
|
||||||
VM block window cache (`vm->blk_vm_cbuf[]`) isn't re-validated on a hit.
|
VM block window cache (`vm->blk_vm_cbuf[]`) isn't re-validated on a hit.
|
||||||
|
- `artemis-reloc-test.img` — 64MB raw image, added 2026-08-25 for single-block
|
||||||
|
relocation (`RELOCATE-BLOCK`) persistence verification. Blank at creation;
|
||||||
|
a genuinely *fresh* volume was required (not `artemis.img`) because
|
||||||
|
relocation-table capacity (`reloc_start`/`reloc_devblocks` in
|
||||||
|
`blk_volume_meta_t`) is only reserved by `blk_compute_fresh_geometry()` on
|
||||||
|
a fresh format — `artemis.img` predates the feature and correctly reads
|
||||||
|
back `reloc_devblocks=0` from what was previously unused header padding.
|
||||||
|
Attach via `make -f Makefile.starkernel ARTDISK=disk/artemis-reloc-test.img
|
||||||
|
...` (the Makefile's `ARTDISK` var is `?=`-overridable). Not yet
|
||||||
|
`BLK-CONFIRM-FORMAT`-committed in the repo copy — commit that step live if
|
||||||
|
reusing this fixture for further reloc-table testing.
|
||||||
|
|
||||||
**Convention, standing as of 2026-08-22: every virtual disk/thumb-drive image
|
**Convention, standing as of 2026-08-22: every virtual disk/thumb-drive image
|
||||||
used for testing — Artemis persistence disks above, and USB Mass Storage
|
used for testing — Artemis persistence disks above, and USB Mass Storage
|
||||||
|
|||||||
Binary file not shown.
@@ -113,7 +113,7 @@ typedef struct {
|
|||||||
/* BAM placement (external 1-bit bitmap region, stored in 4 KiB pages) */
|
/* BAM placement (external 1-bit bitmap region, stored in 4 KiB pages) */
|
||||||
uint32_t bam_start; /* usually 1 */
|
uint32_t bam_start; /* usually 1 */
|
||||||
uint32_t bam_devblocks; /* number of 4 KiB pages used by BAM */
|
uint32_t bam_devblocks; /* number of 4 KiB pages used by BAM */
|
||||||
uint32_t devblock_base; /* first payload devblock = bam_start + bam_devblocks */
|
uint32_t devblock_base; /* first payload devblock = reloc_start + reloc_devblocks */
|
||||||
|
|
||||||
/* Capacity modeling (Forth 1 KiB blocks tracked/usable) */
|
/* Capacity modeling (Forth 1 KiB blocks tracked/usable) */
|
||||||
uint64_t tracked_blocks; /* 32768 * bam_devblocks (bits per 4 KiB page) */
|
uint64_t tracked_blocks; /* 32768 * bam_devblocks (bits per 4 KiB page) */
|
||||||
@@ -135,6 +135,15 @@ typedef struct {
|
|||||||
/* Optional integrity (unused yet) */
|
/* Optional integrity (unused yet) */
|
||||||
uint64_t hdr_crc;
|
uint64_t hdr_crc;
|
||||||
|
|
||||||
|
/* Relocation-exception table placement (Milestone 2h+ single-block relocation).
|
||||||
|
* Appended after hdr_crc, carved out of what was previously _pad[] -- appending
|
||||||
|
* (not inserting) preserves every earlier field's byte offset, so a pre-existing
|
||||||
|
* formatted volume's zeroed padding reads back here as reloc_devblocks=0 ("no
|
||||||
|
* reloc capacity"), gracefully, not a format-breaking change. See
|
||||||
|
* block_subsystem.c's reloc_flush_to_disk()/reloc_load_from_disk(). */
|
||||||
|
uint32_t reloc_start; /* usually bam_start + bam_devblocks */
|
||||||
|
uint32_t reloc_devblocks; /* number of 4 KiB pages used by the reloc table (0 = none) */
|
||||||
|
|
||||||
/* Padding to keep header ≤ 4096 bytes */
|
/* Padding to keep header ≤ 4096 bytes */
|
||||||
uint8_t _pad[4096 - (
|
uint8_t _pad[4096 - (
|
||||||
4 + 4 + /* magic, version */
|
4 + 4 + /* magic, version */
|
||||||
@@ -145,7 +154,8 @@ typedef struct {
|
|||||||
8 + 8 + /* first_free, last_allocated */
|
8 + 8 + /* first_free, last_allocated */
|
||||||
4 + 4 + /* reserved ranges */
|
4 + 4 + /* reserved ranges */
|
||||||
8 + 8 + /* timestamps */
|
8 + 8 + /* timestamps */
|
||||||
8 /* hdr_crc */
|
8 + /* hdr_crc */
|
||||||
|
4 + 4 /* reloc_start, reloc_devblocks */
|
||||||
)];
|
)];
|
||||||
} blk_volume_meta_t;
|
} blk_volume_meta_t;
|
||||||
|
|
||||||
@@ -236,15 +246,39 @@ int blk_subsys_detach_device(struct blkio_dev *dev);
|
|||||||
|
|
||||||
/* Monotonic counter, bumped on every attach/detach (Milestone 2h). A raw
|
/* Monotonic counter, bumped on every attach/detach (Milestone 2h). A raw
|
||||||
* pointer comparison against a blk_get_buffer() result cannot reliably
|
* pointer comparison against a blk_get_buffer() result cannot reliably
|
||||||
* detect a same-address device swap (glibc's allocator can hand back the
|
* detect a same-address device swap (this kernel's own first-fit kmalloc,
|
||||||
* exact address just free()'d by a detach to the very next attach's
|
* src/starkernel/memory/kmalloc.c, can hand back the exact address just
|
||||||
* calloc()) -- callers that cache a blk_get_buffer() result across calls
|
* free()'d by a detach to the very next attach's calloc() -- confirmed
|
||||||
|
* live) -- callers that cache a blk_get_buffer() result across calls
|
||||||
* (block_words.c's VM block window) must instead compare this epoch
|
* (block_words.c's VM block window) must instead compare this epoch
|
||||||
* against the value they last observed, invalidating their whole cache on
|
* against the value they last observed, invalidating their whole cache on
|
||||||
* any change rather than trusting a stored pointer's identity.
|
* any change rather than trusting a stored pointer's identity.
|
||||||
*/
|
*/
|
||||||
uint64_t blk_subsys_epoch(void);
|
uint64_t blk_subsys_epoch(void);
|
||||||
|
|
||||||
|
/* Single-block relocation. Copies home_lbn's current content to target_lbn
|
||||||
|
* (both must already be valid LBNs -- target_lbn is expected to be a block
|
||||||
|
* the caller's own identity already owns, on whichever device it's being
|
||||||
|
* relocated to; this function does not itself validate ownership, that's
|
||||||
|
* policy, left to the caller -- see this session's ACL-owns-policy
|
||||||
|
* direction in FABRIC-2.md), frees home_lbn's original BAM entry, records
|
||||||
|
* an LBN->LBN redirect so every future access to home_lbn transparently
|
||||||
|
* resolves to target_lbn instead, and bumps blk_subsys_epoch() so any VM's
|
||||||
|
* cached block window correctly invalidates. Persisted immediately to the
|
||||||
|
* relocation-owner device's on-disk table (see block_subsystem.c's
|
||||||
|
* reloc_flush_to_disk()) if one exists.
|
||||||
|
*
|
||||||
|
* Returns BLK_OK on success.
|
||||||
|
* Returns BLK_EINVAL if home_lbn == target_lbn, or home_lbn is already
|
||||||
|
* relocated (call again with a different target to re-relocate --
|
||||||
|
* not supported by simply calling this twice on the same home_lbn).
|
||||||
|
* Returns BLK_ERANGE if either LBN doesn't resolve to a valid device.
|
||||||
|
* Returns BLK_EIO if the content copy fails (e.g. target device refuses
|
||||||
|
* the write -- a read-only backend like blkio_usb.c today).
|
||||||
|
* Returns BLK_ENOMEM if the in-memory relocation table is full.
|
||||||
|
*/
|
||||||
|
int blk_subsys_relocate_block(uint32_t home_lbn, uint32_t target_lbn);
|
||||||
|
|
||||||
int blk_subsys_shutdown(void);
|
int blk_subsys_shutdown(void);
|
||||||
|
|
||||||
uint8_t *blk_get_buffer(uint32_t block_num, int writable);
|
uint8_t *blk_get_buffer(uint32_t block_num, int writable);
|
||||||
|
|||||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+192
-5
File diff suppressed because it is too large
Load Diff
@@ -159,9 +159,10 @@ void empty_all_buffers(VM *vm) {
|
|||||||
* it was last validated (Milestone 2h). A device hot-detach followed by a
|
* it was last validated (Milestone 2h). A device hot-detach followed by a
|
||||||
* later re-attach can reuse both the exact same LBN range
|
* later re-attach can reuse both the exact same LBN range
|
||||||
* (block_subsystem.c's chain always appends at the current tail) *and* the
|
* (block_subsystem.c's chain always appends at the current tail) *and* the
|
||||||
* exact same blk_get_buffer() return address (confirmed live: glibc's
|
* exact same blk_get_buffer() return address (confirmed live: this
|
||||||
* allocator hands the just-freed slot straight back to the very next
|
* kernel's own first-fit kmalloc, src/starkernel/memory/kmalloc.c, hands
|
||||||
* same-size calloc()), so neither LBN nor a cached pointer is a reliable
|
* the just-freed slot straight back to the very next same-size
|
||||||
|
* calloc()), so neither LBN nor a cached pointer is a reliable
|
||||||
* "still the same device" signal on its own. Any epoch change discards
|
* "still the same device" signal on its own. Any epoch change discards
|
||||||
* every cached slot outright -- no flush attempt, matching
|
* every cached slot outright -- no flush attempt, matching
|
||||||
* blk_subsys_detach_device()'s own reasoning: whatever device the stale
|
* blk_subsys_detach_device()'s own reasoning: whatever device the stale
|
||||||
@@ -353,6 +354,22 @@ void block_word_confirm_format(VM *vm) {
|
|||||||
if (blk_subsys_confirm_format((uint32_t) blk) != BLK_OK) { vm->error = 1; return; }
|
if (blk_subsys_confirm_format((uint32_t) blk) != BLK_OK) { vm->error = 1; return; }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* RELOCATE-BLOCK ( home target -- ) : relocate home's content to target,
|
||||||
|
* redirecting all future access to home transparently through target
|
||||||
|
* from now on. See blk_subsys_relocate_block()'s own doc comment
|
||||||
|
* (block_subsystem.h) for the full contract -- this word performs no
|
||||||
|
* policy validation of its own (is target actually free? does the
|
||||||
|
* caller actually own it?), matching this project's "ACL owns policy,
|
||||||
|
* this is a mechanical primitive" direction. */
|
||||||
|
void block_word_relocate(VM *vm) {
|
||||||
|
if (vm->dsp < 1) { vm->error = 1; return; }
|
||||||
|
cell_t target = vm_pop(vm);
|
||||||
|
cell_t home = vm_pop(vm);
|
||||||
|
if (blk_subsys_relocate_block((uint32_t) home, (uint32_t) target) != BLK_OK) {
|
||||||
|
vm->error = 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/* SAVE-BUFFERS ( -- ) : sync and write all dirty blocks */
|
/* SAVE-BUFFERS ( -- ) : sync and write all dirty blocks */
|
||||||
void block_word_save_buffers(VM *vm) {
|
void block_word_save_buffers(VM *vm) {
|
||||||
blk_vm_flush_all(vm);
|
blk_vm_flush_all(vm);
|
||||||
@@ -532,6 +549,7 @@ void register_block_words(VM *vm) {
|
|||||||
register_word(vm, "BUFFER", block_word_buffer);
|
register_word(vm, "BUFFER", block_word_buffer);
|
||||||
register_word(vm, "UPDATE", block_word_update);
|
register_word(vm, "UPDATE", block_word_update);
|
||||||
register_word(vm, "BLK-CONFIRM-FORMAT", block_word_confirm_format);
|
register_word(vm, "BLK-CONFIRM-FORMAT", block_word_confirm_format);
|
||||||
|
register_word(vm, "RELOCATE-BLOCK", block_word_relocate);
|
||||||
register_word(vm, "SAVE-BUFFERS", block_word_save_buffers);
|
register_word(vm, "SAVE-BUFFERS", block_word_save_buffers);
|
||||||
register_word(vm, "EMPTY-BUFFERS", block_word_empty_buffers);
|
register_word(vm, "EMPTY-BUFFERS", block_word_empty_buffers);
|
||||||
register_word(vm, "FLUSH", block_word_flush);
|
register_word(vm, "FLUSH", block_word_flush);
|
||||||
|
|||||||
Reference in New Issue
Block a user