Captain Bob, 2026-11-12: "Is it possible to include the FMEA in the pr review? That way the items are forced into a manual review beforehand?" -- "Yes, add it to the plan". Each FMEA row names its paths; the PR carries a review record with a decision for every row its diff touches; a check_fmea_review.sh beside check_protocol.sh refuses the PR without one, or with one for another diff. A sixth point to rule. Nothing built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RzZgrf7V7Zk2iczBXDRbm
124 lines
7.4 KiB
Markdown
124 lines
7.4 KiB
Markdown
# A CAPA & FMEA gate in the build pipelines -- plan, for ruling
|
|
|
|
Captain Bob, 2026-11-11: "I'm thinking about adding a CAPA & FMEA GATE to our build pipelines" --
|
|
then "Yes" to writing it up. **Nothing here is built.** Every choice below marked *To rule* waits for
|
|
Captain Bob; the rest are my calls, open to a ruling. Decide on paper first, then build in small
|
|
increments, each tested first and seen to fail.
|
|
|
|
## 1. What it is for
|
|
|
|
The gate (`scripts/nightly.sh` stages, the same by hand, in a PR and in the nightly) proves the
|
|
system works. It does not prove that **a fault which got past it once cannot get past it again**,
|
|
nor that **every known way the system can fail has something that would see it**. Silent failure is
|
|
this project's dominant failure mode; this gate is aimed at it.
|
|
|
|
- **CAPA** (Corrective and Preventive Action): a record for each *escaped* defect -- a fault the gate
|
|
that should have caught it let through. The correction fixes this one; the **preventive control**
|
|
is a check, wired into a stage, that would have caught it.
|
|
- **FMEA** (Failure Modes and Effects Analysis): a table of the ways each stage or subsystem can
|
|
fail -- effect, cause, Severity, Occurrence, Detection (1-10 each), their product the RPN -- and
|
|
the **control** that detects each. Detection is the project's rule "name, before running, the
|
|
observation that proves it worked", made a number.
|
|
|
|
## 2. The gate
|
|
|
|
A new stage, `scripts/nightly.sh quality`, run in `acceptance.yml` (so in the PR into `master`), the
|
|
nightly and the release. It fails when:
|
|
|
|
1. an issue labelled **`kind/escaped`** has no CAPA record, or its record's preventive control does
|
|
not exist;
|
|
2. a CAPA's control is not run by any stage -- including one that stopped running (a stage renamed
|
|
or removed);
|
|
3. an FMEA row with RPN at or over the threshold has no control in a gate stage;
|
|
4. a record is malformed (missing fields, scores out of range, a control naming nothing).
|
|
|
|
It checks the records against the tree; it does not judge them. Whether a root cause is the real one
|
|
is Captain Bob's ruling, in the PR.
|
|
|
|
## 3. The records
|
|
|
|
*To rule:* format. My call: **YAML** under `docs/quality/` -- `capa/CAPA-NNNN.yml`, one per escaped
|
|
defect, and `fmea.yml` -- because the gate must read them; a Markdown view generated from them for
|
|
the Book of Fabric. A control is named as `<stage>:<check>`, e.g. `accept:put-backs-tracked`, and
|
|
each check is a named function in the stage's script, so the gate can find it.
|
|
|
|
# CAPA-0001.yml
|
|
id: CAPA-0001
|
|
issue: 168 # the kind/escaped issue
|
|
found: 2026-11-11
|
|
escaped: acceptance # the gate it got past
|
|
defect: nightly.sh images stops: git checkout names the untracked version.h
|
|
root_cause: untracking a file without searching for every use; acceptance does not run images
|
|
correction: 9415d7ed
|
|
prevention:
|
|
control: accept:put-backs-tracked
|
|
added_in: <commit>
|
|
seen_to_fail: <log> # the control run against the defect, failing
|
|
|
|
## 4. Seeded from 2026-11-11 (to rule: which become CAPAs)
|
|
|
|
| CAPA | Escaped | Root cause | Correction | Preventive control (proposed) |
|
|
|---|---|---|---|---|
|
|
| 0001 | acceptance | `version.h` untracked while six put-backs still named it; acceptance does not run `images` | `9415d7ed` | every path in a put-back `git checkout` is tracked (or `images` in the gate) |
|
|
| 0002 | every gate, 3 months | LFS pointers committed without their objects (`a5ed8c3d`); nothing checks the server | `aec7ddb5` | `git lfs fsck` and every pointer's object present on the server |
|
|
| 0003 | the rules | acceptance run in the background; a rule nothing enforces | stopped, rerun | `qemu_session.py` refuses to run without a terminal (TTY) |
|
|
| 0004 | CI | a shared concurrency group cancels the waiting run (#140) | `88d5f777` | no run in the last N "cancelled after 0s" |
|
|
| 0005 | acceptance | 146 logs committed as plain text, the LFS filter off | `06f1d41a` | every path `.gitattributes` gives to LFS is a pointer |
|
|
|
|
The code-review findings (`.claude/memory/code-review.md`: MINT-ZUSE's false "nothing changed",
|
|
`manage.c`'s missing depth checks, the descriptor loop, ...) are FMEA rows, not CAPAs: none escaped a
|
|
gate into a fault seen in use.
|
|
|
|
## 5. The scales (to rule)
|
|
|
|
My call, 1-10 each: **Severity** 10 = data lost or a stick erased silently, 7 = a release or the
|
|
nightly fails, 4 = a wrong message, 1 = cosmetic. **Occurrence** from the tracker's history, not
|
|
guessed. **Detection** 10 = nothing would see it, 1 = a gate stage fails on it every time.
|
|
Threshold: RPN >= 100, or Severity >= 9 at any RPN.
|
|
|
|
## 6. Increments (once ruled)
|
|
|
|
1. The record formats and `nightly.sh quality` checking them (malformed refused) -- test first.
|
|
2. CAPA-0001..0005 and their controls, each control seen to fail against its defect.
|
|
3. `fmea.yml` seeded: the acceptance stages, the stick, LFS, CI; rows from the review findings.
|
|
4. `quality` in `acceptance.yml`, the nightly and the release; the Markdown view in the Book.
|
|
5. The `kind/escaped` label; the issue template asks for it.
|
|
|
|
## 7. To rule
|
|
|
|
1. Scope: CAPA for escaped defects only (my call), or every defect?
|
|
2. Format: YAML the gate checks (my call), or Markdown read by a person?
|
|
3. Where: a stage in `acceptance.yml` (my call), or a separate required check?
|
|
4. The scales and the threshold (section 5).
|
|
5. Which of section 4's five become CAPAs.
|
|
6. The FMEA in the PR review (section 8), or not.
|
|
|
|
Don't over-engineer: start with the escaped defects and an FMEA grown as defects arrive, not a
|
|
large table written up front.
|
|
|
|
## 8. The FMEA in the PR review: a manual review forced before the merge
|
|
|
|
Captain Bob, 2026-11-12: "Is it possible to include the FMEA in the pr review? That way the items
|
|
are forced into a manual review beforehand?" -- then "Yes, add it to the plan". As with the protocol's
|
|
evidence (#135), the review is a record committed in the PR, and the gate refuses the PR without it.
|
|
|
|
- **Each FMEA row names the paths it concerns** (`paths:` in `fmea.yml`), narrowly: the MINT-ZUSE row
|
|
`kernel/src/v4/sk_v4_mint.c`, the put-back row `scripts/nightly.sh` and the scripts it lists.
|
|
- **The PR carries `docs/quality/reviews/<TS>.yml`:** for every row whose paths the PR's diff against
|
|
`master` touches -- the row's id; the decision (scores unchanged / raised / lowered, a control
|
|
added); one line of why; and any failure mode the change brings, as a new row in `fmea.yml`.
|
|
- **The gate:** `scripts/check_fmea_review.sh` in `acceptance.yml`, beside `check_protocol.sh`, refuses
|
|
the PR when a touched row has no entry, an entry names a row that does not exist, or the review was
|
|
made for another diff (it names the base and head it reviewed, as PROTOCOL.md names its commit).
|
|
It prints exactly which rows the PR triggers, so the review is of those and no more.
|
|
- **The PR template** gets an "FMEA review" section pointing at the file: the review is done before
|
|
the PR is opened, not after.
|
|
- **What it forces, and what it cannot:** that every affected row was looked at and a decision
|
|
written down. Whether the thinking was honest stays a person's: Captain Bob's while there is one
|
|
developer; with a second, branch protection's required approval as well, the record telling the
|
|
reviewer what to check.
|
|
- **The risk:** rows mapped too coarsely make every PR touch dozens, and the review becomes ticking
|
|
boxes. Rows are mapped narrowly, and a row that triggers on most PRs is split.
|
|
|
|
*To rule:* a sixth point -- the FMEA review in the PR, as above, or not.
|