jerbuild build hangs forever in [1/5] compile-program on FreeBSD amd64 (Chez 10.4.0 GC churn); same sources finish on macOS #78

Closed
opened 2026-09-16 21:39:52 -04:00 by ober · 1 comment
Owner

Summary

jerbuild build (via gmake binary in ober/jerboa-code) hangs forever on FreeBSD amd64. The build gets cleanly through everything up to the [1/5] Program compile in isolated subprocess (multicall self) step, then a single child process (jerboa runtime <objdir>/cross-wpo-compile.ss) spins at 100% CPU with multi-GB GC churn and zero output progress. The identical sources + identical jerboa version complete on macOS in ~2 minutes, so this is a FreeBSD-specific compiler bug in jerbuild's WPO/compile-program path (bundled Chez 10.4.0), not a source, Makefile, version, or resource problem.

Tested under both jerboa 0.12.6 and 0.12.9 — identical stall, so the recent update to 0.12.9 is unrelated.

Environment (reproducer host)

  • OS: FreeBSD 15.1 amd64 (uname -m = amd64, uname -r = 15.1), host nickname biggus
  • RAM: 133 GB (hw.physmem=132930727936) — resource ceiling ruled out
  • jerboa tested: 0.12.6 (freebsd-amd64 release tarball) and 0.12.9 (multicall ELF), both from ~/.local/bin; bundled Chez runtime 10.4.0
  • Repo: ober/jerboa-code @ HEAD 7cb9f19, VERSION 0.4.7
  • Toolchain env used for the runs:
    • PATH=~/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
    • gmake binary (Makefile binary target → jerbuild build --config .jerbuild --os-libs "-lm -lpthread -lutil -lncurses -L/usr/local/lib -liconv")
    • JERBOA_BINARY_DETERMINISTIC_LINK=0

Repro

ssh biggus
cd ~/mine/jerboa-code
export PATH="$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"
gmake binary

Expected: proceeds past [1/5]. Actual: stuck forever at:

==> [1/5] Program compile in isolated subprocess (multicall self)
    helper:  /tmp/jerbuild-binary-XXXX/cross-wpo-compile.ss
    cmd:     '/home/user/.local/bin/jerboa' runtime '/tmp/jerbuild-binary-XXXX/cross-wpo-compile.ss'

with the child jerboa runtime ... at ~99% CPU on one thread.

Hard evidence (measured on biggus)

  • Meanwhile child process state (0.12.9 run, WPO on default):
    • CPU 99%, single thread (procstat -k → one running thread)
    • RSS oscillated 4.5 → 5.5 GB across repeated samples (GC reclaiming; not monotonic leak, not OOM; also not a RAM ceiling — 133 GB machine)
    • Obj-dir output frozen: bundle/ stuck at 158 files, u-* at 234 files, across 90 s sampling windows
    • program.so stayed 0 bytes, program.wp.so never created
  • gdb attach (repeatedly, mid-spin) — always inside the collector, with Scheme mutator frames above:
#0  sweep_generation ()
#1  S_gc_ocd_entry ()
#2  S_do_gc ()
#3  <Scheme-generated code interior>
#4  ... S_call_help ()
  • lldb: single thread jerboa, frame #0 at 0x7341f41 in a Scheme-generated trampoline region (compiler's own compiled code), no C-stack deadlock.
  • JERBOA_BINARY_WPO=0 run (0.12.6): same spin, froze even earlier (bundle 79 / u-* 117, RSS climbed past 5.5 GB). So not WPO-fasl-emission specific — it's the compile-program expansion/codegen itself.
  • Every run reached [1/5] in under ~1 minute (the transpile, Rust crate build via cargo, and linker prep all completed fine).

Control experiment: macOS completes identically

On macOS arm64 (/Users/user/mine/jerboa-code, same HEAD 7cb9f19), jerboa 0.12.6 (macos-arm64 multicall), make binary:

==> [1/5] Program compile in isolated subprocess (multicall self)
    cmd: '/Users/user/.local/bin/jerboa' runtime '.../cross-wpo-compile.ss'
...
=== Build complete: /Users/user/mine/jerboa-code/./jcode ===   (~2 min total)

So: same sources, same jerboa version — macOS compiles the full closure to a working 17 MB jcode; FreeBSD amd64 never writes the first byte of program.so. The fault is specific to the FreeBSD/ta6fb bundled Chez build path.

Code path inside this repo

  • jerbuild.ss → do-binary-build → [1/5] branch selected at ~line 3390 when (or xpatch (getenv "JERBOA_SELF_EXE")) (multicall self → isolated subprocess).
  • cross-program-subprocess! (~line 3115–3186) writes cross-wpo-compile.ss and spawns a fresh jerboa runtime whose helper runs, in order:
    1. (library-directories '(...)) (bundle + user libs redirected under obj-dir)
    2. (compile-imported-libraries #t)
    3. (generate-wpo-files <binary-wpo?>)
    4. (compile-program <entry> <program.so>)
    5. (if WPO) second helper cross-wpo-whole.ss → (compile-whole-program ...)
  • The stall is step 4: ~158 bundled-stdlib .so/.wpo + ~234 project .so/.wpo are all emitted, then compile-program on main-binary.ss runs forever without touching program.so (0 bytes) and without ever reaching the whole-program helper. Last-successful module writes before the freeze cluster around jcode/ui/tui-*, jcode/tool/lsp, std/os/sysmon, std/misc/process — recent PRs #16–#19 in jerboa-code touched exactly these tui/lsp/eval modules.

What was ruled out

  • ❌ 0.12.9 regression — 0.12.6 stalls identically (also verified from the unpacked freebsd-amd64 release tree)
  • ❌ Stale artifacts / dirty checkout — fresh /tmp/jerbuild-binary-* obj-dir each run; toolchain bundle in ~/.cache/jerbuild complete (.complete marker, verified contents)
  • ❌ Missing network — the phase is a local subprocess; network only used earlier in ensure-jerboa-tools (which succeeded; the v0.2.8 fallback 404s but is never reached when jerbuild is on PATH)
  • ❌ Slow machine / RAM ceiling — 133 GB RAM, and frozen output with 100% CPU is not progress
  • ❌ JERBOA_BINARY_WPO (WPO on vs off) — both stall at compile-program
  • ❌ jcode sources themselves — identical sources build in ~2 min on macOS
  • ❌ JERBOA_BINARY_DETERMINISTIC_LINK — set (0) in all runs including the working macOS one

Hypotheses for the fixer (ranked)

  1. compile-program (R6RS top-level program mode) on the full jcode import closure blows up on the FreeBSD/ta6fb bundled Chez 10.4.0 — an optimization/expansion pass allocates a combinatorially growing structure (GC-churn signature ~ sweep_generation continuance is the giveaway: endless mark/sweep, RSS plateaus as the collector reclaims, output never advances). Likely a path/bug that is x86_64-freebsd-specific (macOS arm64 tarm64osx build unaffected). Prime suspects: a specific large module in the closure (see last-writes above), a huge literal/case/macro expansion, or letrec* closure-conversion on a large let-chain.
  2. Bisect the closure to isolate the triggering module: run the generated cross-wpo-compile.ss helper standalone against progressively larger subsets of the libdirs; once a minimal repro is found, reduce to bare Chez (chezscheme) compile-program for an upstream report.
  3. Verify against a locally built Chez on FreeBSD (JERBOA_WPO_SCHEME pointing at a source-built ta6fb scheme, see the jerboa-freebsd-build-from-source build recipe) to distinguish bundled-runtime build flags (dtrace probes, machine type, --enable-harden) from a genuine Chez 10.4.0 FreeBSD codegen bug. The bundled runtime is the current suspect; a plain gmake-built ta6fb scheme may dodge it.
  4. Instrument the stuck subprocess (add (collect-trip-bytes)/GC notify around compile-program, interrupt with gdb and unwind the Scheme continuation) to identify the exact pass/continuation.

Success criteria for a fix

  • gmake binary completes to === Build complete: .../jcode === on FreeBSD 15.1 amd64 with these exact sources and 0.12.6 and 0.12.9, OR
  • a faithful self-contained reproducer (repo → minimal .ss → chezscheme compile-program) that documents the FreeBSD-specific Chez bug, with the upstream workaround/flag documented.

Files/dirs referenced on the reproducer host

~/mine/jerboa-code                      # ober/jerboa-code @ 7cb9f19
~/.cache/jerbuild/<sha>/{bundle,lib}    # toolchain bundle (complete)
/tmp/jerbuild-binary-*/cross-wpo-compile.ss   # generated helper (survives, can be replayed)
/var/folders/.../jerbuild-binary-*/    # macOS equivalent (macOS run completed)
## Summary `jerbuild build` (via `gmake binary` in **ober/jerboa-code**) hangs **forever** on FreeBSD amd64. The build gets cleanly through everything up to the `[1/5] Program compile in isolated subprocess (multicall self)` step, then a single child process (`jerboa runtime <objdir>/cross-wpo-compile.ss`) spins at 100% CPU with multi-GB GC churn and **zero output progress**. The identical sources + identical jerboa version complete on macOS in ~2 minutes, so this is a FreeBSD-specific compiler bug in jerbuild's WPO/compile-program path (bundled Chez 10.4.0), **not** a source, Makefile, version, or resource problem. Tested under **both jerboa 0.12.6 and 0.12.9** — identical stall, so the recent update to 0.12.9 is unrelated. ## Environment (reproducer host) - OS: FreeBSD 15.1 amd64 (`uname -m` = amd64, `uname -r` = 15.1), host nickname `biggus` - RAM: 133 GB (`hw.physmem=132930727936`) — resource ceiling ruled out - jerboa tested: `0.12.6` (freebsd-amd64 release tarball) and `0.12.9` (multicall ELF), both from ~/.local/bin; bundled Chez runtime 10.4.0 - Repo: `ober/jerboa-code` @ HEAD `7cb9f19`, VERSION 0.4.7 - Toolchain env used for the runs: - `PATH=~/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin` - `gmake binary` (Makefile `binary` target → `jerbuild build --config .jerbuild --os-libs "-lm -lpthread -lutil -lncurses -L/usr/local/lib -liconv"`) - `JERBOA_BINARY_DETERMINISTIC_LINK=0` ## Repro ```sh ssh biggus cd ~/mine/jerboa-code export PATH="$HOME/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin" gmake binary ``` Expected: proceeds past `[1/5]`. Actual: stuck forever at: ``` ==> [1/5] Program compile in isolated subprocess (multicall self) helper: /tmp/jerbuild-binary-XXXX/cross-wpo-compile.ss cmd: '/home/user/.local/bin/jerboa' runtime '/tmp/jerbuild-binary-XXXX/cross-wpo-compile.ss' ``` with the child `jerboa runtime ...` at ~99% CPU on one thread. ## Hard evidence (measured on biggus) - Meanwhile child process state (0.12.9 run, WPO on default): - CPU 99%, single thread (`procstat -k` → one running thread) - RSS oscillated **4.5 → 5.5 GB** across repeated samples (GC reclaiming; not monotonic leak, not OOM; also not a RAM ceiling — 133 GB machine) - Obj-dir output **frozen**: `bundle/` stuck at **158 files**, `u-*` at **234 files**, across 90 s sampling windows - `program.so` stayed **0 bytes**, `program.wp.so` never created - gdb attach (repeatedly, mid-spin) — always inside the collector, with Scheme mutator frames above: ``` #0 sweep_generation () #1 S_gc_ocd_entry () #2 S_do_gc () #3 <Scheme-generated code interior> #4 ... S_call_help () ``` - lldb: single thread `jerboa`, frame #0 at `0x7341f41` in a Scheme-generated trampoline region (compiler's own compiled code), no C-stack deadlock. - `JERBOA_BINARY_WPO=0` run (0.12.6): same spin, froze even earlier (bundle 79 / u-* 117, RSS climbed past 5.5 GB). So **not** WPO-fasl-emission specific — it's the `compile-program` expansion/codegen itself. - Every run reached `[1/5]` in under ~1 minute (the transpile, Rust crate build via cargo, and linker prep all completed fine). ## Control experiment: macOS completes identically On macOS arm64 (`/Users/user/mine/jerboa-code`, same HEAD `7cb9f19`), jerboa **0.12.6** (macos-arm64 multicall), `make binary`: ``` ==> [1/5] Program compile in isolated subprocess (multicall self) cmd: '/Users/user/.local/bin/jerboa' runtime '.../cross-wpo-compile.ss' ... === Build complete: /Users/user/mine/jerboa-code/./jcode === (~2 min total) ``` So: same sources, same jerboa version — macOS compiles the full closure to a working 17 MB `jcode`; FreeBSD amd64 never writes the first byte of `program.so`. The fault is specific to the **FreeBSD/ta6fb bundled Chez build path**. ## Code path inside this repo - `jerbuild.ss` → `do-binary-build` → `[1/5]` branch selected at ~line 3390 when `(or xpatch (getenv "JERBOA_SELF_EXE"))` (multicall self → isolated subprocess). - `cross-program-subprocess!` (~line 3115–3186) writes `cross-wpo-compile.ss` and spawns a **fresh** `jerboa runtime` whose helper runs, in order: 1. `(library-directories '(...))` (bundle + user libs redirected under obj-dir) 2. `(compile-imported-libraries #t)` 3. `(generate-wpo-files <binary-wpo?>)` 4. `(compile-program <entry> <program.so>)` 5. (if WPO) second helper `cross-wpo-whole.ss` → `(compile-whole-program ...)` - The stall is **step 4**: ~158 bundled-stdlib `.so`/`.wpo` + ~234 project `.so`/`.wpo` are all emitted, then `compile-program` on `main-binary.ss` runs forever without touching `program.so` (0 bytes) and without ever reaching the whole-program helper. Last-successful module writes before the freeze cluster around `jcode/ui/tui-*`, `jcode/tool/lsp`, `std/os/sysmon`, `std/misc/process` — recent PRs #16–#19 in jerboa-code touched exactly these tui/lsp/eval modules. ## What was ruled out - ❌ 0.12.9 regression — 0.12.6 stalls identically (also verified from the unpacked freebsd-amd64 release tree) - ❌ Stale artifacts / dirty checkout — fresh `/tmp/jerbuild-binary-*` obj-dir each run; toolchain bundle in `~/.cache/jerbuild` complete (`.complete` marker, verified contents) - ❌ Missing network — the phase is a local subprocess; network only used earlier in `ensure-jerboa-tools` (which succeeded; the v0.2.8 fallback 404s but is never reached when `jerbuild` is on PATH) - ❌ Slow machine / RAM ceiling — 133 GB RAM, and frozen output with 100% CPU is not progress - ❌ `JERBOA_BINARY_WPO` (WPO on vs off) — both stall at `compile-program` - ❌ jcode sources themselves — identical sources build in ~2 min on macOS - ❌ `JERBOA_BINARY_DETERMINISTIC_LINK` — set (0) in all runs including the working macOS one ## Hypotheses for the fixer (ranked) 1. **`compile-program` (R6RS top-level program mode) on the full jcode import closure blows up on the FreeBSD/ta6fb bundled Chez 10.4.0** — an optimization/expansion pass allocates a combinatorially growing structure (GC-churn signature ~ `sweep_generation` continuance is the giveaway: endless mark/sweep, RSS plateaus as the collector reclaims, output never advances). Likely a path/bug that is x86_64-freebsd-specific (macOS arm64 `tarm64osx` build unaffected). Prime suspects: a specific large module in the closure (see last-writes above), a huge literal/`case`/macro expansion, or `letrec*` closure-conversion on a large `let`-chain. 2. **Bisect the closure** to isolate the triggering module: run the generated `cross-wpo-compile.ss` helper standalone against progressively larger subsets of the libdirs; once a minimal repro is found, reduce to bare Chez (`chezscheme`) `compile-program` for an upstream report. 3. **Verify against a locally built Chez on FreeBSD** (`JERBOA_WPO_SCHEME` pointing at a source-built `ta6fb` `scheme`, see the `jerboa-freebsd-build-from-source` build recipe) to distinguish bundled-runtime build flags (dtrace probes, machine type, `--enable-harden`) from a genuine Chez 10.4.0 FreeBSD codegen bug. The bundled runtime is the current suspect; a plain `gmake`-built `ta6fb` `scheme` may dodge it. 4. Instrument the stuck subprocess (add `(collect-trip-bytes)`/GC notify around `compile-program`, interrupt with gdb and unwind the Scheme continuation) to identify the exact pass/continuation. ## Success criteria for a fix - `gmake binary` completes to `=== Build complete: .../jcode ===` on FreeBSD 15.1 amd64 with these exact sources and 0.12.6 **and** 0.12.9, OR - a faithful self-contained reproducer (repo → minimal `.ss` → `chezscheme` `compile-program`) that documents the FreeBSD-specific Chez bug, with the upstream workaround/flag documented. ## Files/dirs referenced on the reproducer host ``` ~/mine/jerboa-code # ober/jerboa-code @ 7cb9f19 ~/.cache/jerbuild/<sha>/{bundle,lib} # toolchain bundle (complete) /tmp/jerbuild-binary-*/cross-wpo-compile.ss # generated helper (survives, can be replayed) /var/folders/.../jerbuild-binary-*/ # macOS equivalent (macOS run completed) ```
Author
Owner

Verified resolved on FreeBSD 15.1-RELEASE-p3 amd64 (biggus) using fresh Jerboa master 2b00a04277 (VERSION 0.13.0) and the pinned consumer ober/jerboa-code 7cb9f19bb27a13213222d5391a409a469043f928.

Two clean default-WPO gmake binary runs completed with JERBOA_BINARY_FORCE_INPROCESS unset and the isolated [1/5] Program compile in isolated subprocess (multicall self) path. Both produced nonempty program.so, program.wpo, program.wp.so, and executable jcode. The pinned builds passed clean-environment --version and --help both in place and after copying only jcode to an otherwise empty staging directory (all exit 0, no stderr). A separate pinned JERBOA_BINARY_WPO=0 build also completed successfully.

For confirmation against the requested latest consumer, current ober/jerboa-code master 990f54917685810737da670b1f2c4da95f33eadb completed the same isolated default-WPO build and clean/staged startup checks (all exit 0, no stderr).

Evidence was retained on biggus:

  • /home/user/work/jerboa-code-verify-78/dist/issue78-pinned-run1/
  • /home/user/work/jerboa-code-verify-78-run2/dist/issue78-pinned-run2/
  • /home/user/work/jerboa-code-verify-78-run2/dist/issue78-pinned-wpo-off/
  • /home/user/work/jerboa-code-verify-78-current/dist/issue78-current-run1/

This verifies that the original FreeBSD compile hang no longer reproduces. Closing issue 78.

Verified resolved on FreeBSD 15.1-RELEASE-p3 amd64 (biggus) using fresh Jerboa master 2b00a042773d73c848f0bf1520366930a07c41d6 (VERSION 0.13.0) and the pinned consumer ober/jerboa-code 7cb9f19bb27a13213222d5391a409a469043f928. Two clean default-WPO gmake binary runs completed with JERBOA_BINARY_FORCE_INPROCESS unset and the isolated [1/5] Program compile in isolated subprocess (multicall self) path. Both produced nonempty program.so, program.wpo, program.wp.so, and executable jcode. The pinned builds passed clean-environment --version and --help both in place and after copying only jcode to an otherwise empty staging directory (all exit 0, no stderr). A separate pinned JERBOA_BINARY_WPO=0 build also completed successfully. For confirmation against the requested latest consumer, current ober/jerboa-code master 990f54917685810737da670b1f2c4da95f33eadb completed the same isolated default-WPO build and clean/staged startup checks (all exit 0, no stderr). Evidence was retained on biggus: - /home/user/work/jerboa-code-verify-78/dist/issue78-pinned-run1/ - /home/user/work/jerboa-code-verify-78-run2/dist/issue78-pinned-run2/ - /home/user/work/jerboa-code-verify-78-run2/dist/issue78-pinned-wpo-off/ - /home/user/work/jerboa-code-verify-78-current/dist/issue78-current-run1/ This verifies that the original FreeBSD compile hang no longer reproduces. Closing issue 78.
ober closed this issue 2026-09-22 01:42:43 -04:00
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ober/jerboa#78
No description provided.