CI cleans up its litter: green, orange and red runs alike #129

New Issue

2026-11-07T17:58:42Z

rajames commented

2026-11-07 17:58:42 +00:00

What

CI cleans up after itself whatever a run's outcome: green (passed), orange (cancelled, skipped, superseded in the concurrency group) and red (failed). Today a job's container and network are removed when it ends, but the rest stays, run after run:

Litter	Where	Cleanup
Docker images and BuildKit cache (old runner images, pulled tags, layers) -- the largest	the server's Docker	weekly cron: `docker image prune -af --filter "until=168h"` and `docker builder prune -af --filter "until=168h"` (an image in use is kept)
Old `lithosananke-runner` versions (`72b6cf370db1` is already obsolete)	Gitea's package registry	Settings, Packages, Cleanup rules: container, `lithosananke-runner`, keep the newest 2 (one to roll back to)
Run artifacts (logs, images) of every run	Gitea	`[actions] ARTIFACT_RETENTION_DAYS = 14` in `app.ini`: what matters is committed or in Nexus
Nightly snapshots, ~60 MB a night	Nexus `starshipos-snapshots`	a cleanup policy (keep 7 days) and the scheduled "Compact blob store" task, which frees the disk; `starshipos-releases` untouched
Run logs	Gitea	small: the default retention (a year) stands

Every rule applies to green, orange and red runs alike: none is exempt because it passed, and none is kept forever because it failed. A red run's evidence lives as long as any other's (14 days of artifacts, a year of logs); anything worth keeping longer is committed (logs/) or filed with the issue it belongs to.

Why -- the ruling or the design it comes from

Captain Bob, 2026-11-07: "Will the CI clean up its litter when it's finished?" -- "Add an issue to clean up litter with both green builds, orange builds and red builds." The server has 3.8 GiB of memory and a disk shared by both wikis, Gitea, Nexus and CI; the runner-image rebuilds and the nightlies grow it every day.

Done when

The weekly Docker prune runs on the server (cron), and docker system df shows it reclaiming
Gitea's cleanup rule keeps the newest 2 lithosananke-runner versions, never the one the workflows name
ARTIFACT_RETENTION_DAYS = 14 in Gitea's app.ini
Nexus: the snapshot cleanup policy and the blob-store compaction scheduled
Verified after a week: a green, an orange and a red run's artifacts and images gone on schedule; disk use flat

Area

CI, the server

Priority

must -- high: Captain Bob, 2026-11-07, "Given my current server constraints, this is a high priority item"

🤖 Filed with Claude Code

### What CI cleans up after itself **whatever a run's outcome**: green (passed), orange (cancelled, skipped, superseded in the concurrency group) and red (failed). Today a job's container and network are removed when it ends, but the rest stays, run after run: | Litter | Where | Cleanup | |---|---|---| | Docker images and BuildKit cache (old runner images, pulled tags, layers) -- the largest | the server's Docker | weekly cron: `docker image prune -af --filter "until=168h"` and `docker builder prune -af --filter "until=168h"` (an image in use is kept) | | Old `lithosananke-runner` versions (`72b6cf370db1` is already obsolete) | Gitea's package registry | Settings, Packages, Cleanup rules: container, `lithosananke-runner`, keep the newest 2 (one to roll back to) | | Run artifacts (logs, images) of every run | Gitea | `[actions] ARTIFACT_RETENTION_DAYS = 14` in `app.ini`: what matters is committed or in Nexus | | Nightly snapshots, ~60 MB a night | Nexus `starshipos-snapshots` | a cleanup policy (keep 7 days) and the scheduled "Compact blob store" task, which frees the disk; `starshipos-releases` untouched | | Run logs | Gitea | small: the default retention (a year) stands | Every rule applies to green, orange and red runs alike: none is exempt because it passed, and none is kept forever because it failed. A red run's evidence lives as long as any other's (14 days of artifacts, a year of logs); anything worth keeping longer is committed (`logs/`) or filed with the issue it belongs to. ### Why -- the ruling or the design it comes from Captain Bob, 2026-11-07: "Will the CI clean up its litter when it's finished?" -- "Add an issue to clean up litter with both green builds, orange builds and red builds." The server has 3.8 GiB of memory and a disk shared by both wikis, Gitea, Nexus and CI; the runner-image rebuilds and the nightlies grow it every day. ### Done when - [ ] The weekly Docker prune runs on the server (cron), and `docker system df` shows it reclaiming - [ ] Gitea's cleanup rule keeps the newest 2 `lithosananke-runner` versions, never the one the workflows name - [ ] `ARTIFACT_RETENTION_DAYS = 14` in Gitea's `app.ini` - [ ] Nexus: the snapshot cleanup policy and the blob-store compaction scheduled - [ ] Verified after a week: a green, an orange and a red run's artifacts and images gone on schedule; disk use flat ### Area CI, the server ### Priority must -- high: Captain Bob, 2026-11-07, "Given my current server constraints, this is a high priority item" 🤖 Filed with [Claude Code](https://claude.com/claude-code)

rajames added the

labels 2026-11-07 17:58:42 +00:00

rajames added

and removed

labels 2026-11-07 18:01:19 +00:00

rajames commented

2026-11-07 19:16:20 +00:00

Done on the server (Captain Bob, 2026-11-07):

Weekly Docker prune: Sundays 03:00, logged to /var/log/docker-prune.log
Gitea artifact retention: 30 days (set by environment variable; this issue said 14)
Gitea package cleanup: a script, Sundays 03:15, logged to /var/log/gitea-cleanup.log, in place of a UI cleanup rule

Left: Nexus. 3.96 Community has no public cleanup-policy API, and the restricted account snapshot-admin gets 403 on the internal one. So in the UI: a cleanup policy (maven2, component age over 7 days) on starshipos-snapshots, applied by the existing nightly "Cleanup service" task (01:00); and a weekly "Admin - Compact blob store" task on the default blob store (Sunday 03:30). Then the week-later check.

Done on the server (Captain Bob, 2026-11-07): - Weekly Docker prune: Sundays 03:00, logged to `/var/log/docker-prune.log` - Gitea artifact retention: **30 days** (set by environment variable; this issue said 14) - Gitea package cleanup: a script, Sundays 03:15, logged to `/var/log/gitea-cleanup.log`, in place of a UI cleanup rule Left: Nexus. 3.96 Community has no public cleanup-policy API, and the restricted account `snapshot-admin` gets 403 on the internal one. So in the UI: a cleanup policy (`maven2`, component age over 7 days) on `starshipos-snapshots`, applied by the existing nightly "Cleanup service" task (01:00); and a weekly "Admin - Compact blob store" task on the `default` blob store (Sunday 03:30). Then the week-later check.

rajames referenced this issue

2026-11-07 20:15:18 +00:00

Rotate what was exposed on 2026-11-07 #134

rajames added this to the v4.0.0-alpha.4 milestone 2026-11-07 20:16:04 +00:00

rajames referenced this issue

2026-11-07 20:16:17 +00:00

alpha.4: the list #147

Sign in to join this conversation.