jerboa-prog-* temp artifacts leak on kill/crash (unbounded /tmp growth; ENOSPC incident) #76

Open
opened 2026-09-14 23:10:53 -04:00 by ober · 0 comments
Owner

Summary

jerboa-prog-XXXXXX temporary artifacts accumulate without bound when jerboa-built programs are killed or crash before normal exit. On our Bot Commons validation host this filled the jail rootfs (125,834 artifacts / 61 GB between Sep 11–13, 2026), drove the filesystem to 109% capacity, and an unrelated production SQLite database was truncated to 0 bytes by the resulting ENOSPC.

Creation sites (current master)

  • jerbuild.ss write_program_tmpfile() (~line 2762): mkstemp("%s/jerboa-prog-XXXXXX") boot file; unlink(prog_path) only runs after Sscheme_program returns normally (~line 2795).
  • support/build-binary.sh (~line 420) and support/multicall-main.c (~line 721): same pattern.
  • Multicall binaries additionally create jerboa-prog-* directories for extracted library bundles (ensure_extracted()), held for the process lifetime.

Why it leaks

Any supervisor that runs untrusted jerboa programs with a timeout (our validator kills submissions routinely) or any crash/SIGKILL skips the unlink, leaving one artifact per run. Long-lived services running many short-lived programs make this unbounded. Observed leak rate on our host: ~2,100/day.

Reproduction

  1. jerbuild binary any program; note it is a static multicall binary.
  2. Run it under timeout 1 with a workload longer than 1s (or kill -9 it).
  3. ls /tmp/jerboa-prog-* — the boot file remains.

Suggested fixes (any one closes the operational risk)

  1. Unlink the boot file immediately after it is consumed (POSIX keeps it readable via the open fd / after Sscheme_program has loaded it), rather than at process exit.
  2. Startup sweep: on boot, best-effort delete jerboa-prog-* artifacts older than e.g. 24h in the same tmpdir (owner check).
  3. Use O_TMPFILE where available.

Option 1 looks smallest: the boot file is fully read at heap-build time.

Environment

  • FreeBSD 14 (bastille jail), jerboa master circa 2a7c88a0..57aa69e3 (2026-09-13/14), binaries produced by jerbuild binary --static-native.

Happy to send a PR for option 1 if the approach sounds right.

## Summary `jerboa-prog-XXXXXX` temporary artifacts accumulate without bound when jerboa-built programs are killed or crash before normal exit. On our Bot Commons validation host this filled the jail rootfs (125,834 artifacts / 61 GB between Sep 11–13, 2026), drove the filesystem to 109% capacity, and an unrelated production SQLite database was truncated to 0 bytes by the resulting ENOSPC. ## Creation sites (current master) - `jerbuild.ss` `write_program_tmpfile()` (~line 2762): `mkstemp("%s/jerboa-prog-XXXXXX")` boot file; `unlink(prog_path)` only runs after `Sscheme_program` returns normally (~line 2795). - `support/build-binary.sh` (~line 420) and `support/multicall-main.c` (~line 721): same pattern. - Multicall binaries additionally create `jerboa-prog-*` directories for extracted library bundles (`ensure_extracted()`), held for the process lifetime. ## Why it leaks Any supervisor that runs untrusted jerboa programs with a timeout (our validator kills submissions routinely) or any crash/SIGKILL skips the `unlink`, leaving one artifact per run. Long-lived services running many short-lived programs make this unbounded. Observed leak rate on our host: ~2,100/day. ## Reproduction 1. `jerbuild binary` any program; note it is a static multicall binary. 2. Run it under `timeout 1` with a workload longer than 1s (or `kill -9` it). 3. `ls /tmp/jerboa-prog-*` — the boot file remains. ## Suggested fixes (any one closes the operational risk) 1. Unlink the boot file immediately after it is consumed (POSIX keeps it readable via the open fd / after `Sscheme_program` has loaded it), rather than at process exit. 2. Startup sweep: on boot, best-effort delete `jerboa-prog-*` artifacts older than e.g. 24h in the same tmpdir (owner check). 3. Use `O_TMPFILE` where available. Option 1 looks smallest: the boot file is fully read at heap-build time. ## Environment - FreeBSD 14 (bastille jail), jerboa master circa 2a7c88a0..57aa69e3 (2026-09-13/14), binaries produced by `jerbuild binary --static-native`. Happy to send a PR for option 1 if the approach sounds right.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ober/jerboa#76
No description provided.