Fresh compiler processes produce byte-different FASLs and secmon binaries; investigate deterministic gensym identity #97

New issue

Open

opened 2026-09-25 12:02:12 -04:00 by ober · 0 comments

ober commented

2026-09-25 12:02:12 -04:00

Owner

Evidence and scope

Secmon qualification checkout at 050752d plus uncommitted build changes, using locked Jerboa 020d49ffcc and jsqlite 73dfdf0703e41ef7bf1b5ab89839975b9d9b97b8 plus its recorded dependency patch.
Saved unpatched reproducibility report: daemon_binaries_status=mismatch, keygen_binary_status=mismatch; typed lock, typed static library, release inputs, and source manifest all match. Both builds complete all five binaries.
Earlier isolated investigation recorded byte differences in fresh jsqlite/cache.ss FASLs before boot construction, despite resetting random-seed, gensym-prefix, and gensym-count. Symbol encounter order differed even when the recorded symbol-key multiset matched.
A diagnostic modification to vendored Chez s/5_7.ss allowed make-session-key to read JERBOA_DETERMINISTIC_GENSYM_SESSION, retaining the ordinary path when unset. A distinct key per binary output, forwarded through secmon's restricted compiler-child environment, yielded matching stage-2 artifact manifests and a saved overall status=match report. The handoff also records passing binary smoke/security for that isolated candidate after a separate macOS linker correction.
A single global fixed key failed with: Exception in intern-gensym: unique name "jerboa-release-gensym-session-1255" already interned. Do not ship a global fixed key.

These are diagnostic results, not an independently repeated minimal upstream regression or proof that the proposed environment hook is production-safe. The downstream checkout has other changes, and the hook has not been validated in a newly packaged self-contained Jerboa binary.

Source evidence / hypothesis

In Jerboa 2604e79232, vendor/ChezScheme/s/5_7.ss constructs the private session key using (cs)unique_id. In s/newhash.ss, symbol-hash can materialize a gensym unique name. s/fasl.ss traverses hashtable entries during graph discovery and emission. The experiments support session-dependent identity/hash ordering as a cause. Existing prefix rewriting after serialization is insufficient in the observed downstream builds; this does not prove that every possible canonical serializer is impossible.

current-generate-id is already configured by secmon's compiler helper but does not cover all private session-generated identities. Maintainer investigation should decide whether to add a supported build-scoped identity facility or fix deterministic serialization at another layer.

Correction to prior investigation: JERBOA_BINARY_STRIP_FASL and JERBOA_BINARY_CANONICALIZE_WPO were set in some secmon runs, but the inspected secmon builder/scripts do not consume them. Those runs do not establish that enabling those features caused any change. Explicit secmon WPO selection did execute and still produced mismatching reports.

Reproduction assets for the local maintainer

~/conduit-blockers.md contains chronological experiments; its latest correction supersedes the overstated policy/normalization conclusions.
~/work/jerboa-secmon-detector: unpatched qualification candidate, scripts/reproducibility-report.sh, dist/reproducibility/{report.txt,first-build.log,second-build.log}, and both sets of saved binaries.
~/work/jerboa-gensym-trace-diagnostic: patched diagnostic candidate and saved matching report, with first-stage2/second-stage2 manifests. Diagnostic wrapper/traces are not release changes.
~/work/jerboa-gensym-session-inspection: branch conduit-deterministic-gensym, uncommitted make-session-key hook only; make chez completed, packaged multicall binary verification remains undone.

First preserve the existing reports: the report script deletes its output directory on rerun. Reduce to a fresh-process jsqlite/cache compilation through a project make target or supported Jerboa compiler entry point, then compare both FASLs before boot assembly. Compare unpatched vs candidate using identical source/configuration. Include the parent/child environment boundary in the reproduction.

Acceptance criteria

Regression reproduces the variance and proves the chosen fix in independent compiler processes with identical inputs.
Independent modules/artifacts retain distinct identities; dependent modules preserve shared identity. Load both independently built and mutually dependent FASLs together without collisions.
Ordinary runtime gensym behavior remains unchanged unless a documented build mode is explicitly selected; validate any new inputs and namespace policy.
Fresh packaged self-contained Jerboa supports the solution without external build-tree prerequisites.
Downstream two-build hashes, runtime smoke, and security checks pass with diagnostics removed. Keep raw-byte equality distinct from semantic FASL equivalence.

Please investigate/implement in Jerboa and its bundled Chez as appropriate. This issue requests a supported capability, not adoption of the experimental environment hook verbatim. The user will handle this upstream work. Related but distinct fixed runtime-discovery issue: #93.

Observed on macOS arm64, 2026-09-25. This blocks the strict byte-for-byte secmon release reproducibility gate, not execution of Jerboa or Conduit. Users should not need a separate Chez installation, object-code tree, or repaired cache: the eventual supported solution must work through the self-contained Jerboa distribution. ## Evidence and scope - Secmon qualification checkout at 050752d plus uncommitted build changes, using locked Jerboa 020d49ffcc121bfe7a47424dbd23ae467969d805 and jsqlite 73dfdf0703e41ef7bf1b5ab89839975b9d9b97b8 plus its recorded dependency patch. - Saved unpatched reproducibility report: daemon_binaries_status=mismatch, keygen_binary_status=mismatch; typed lock, typed static library, release inputs, and source manifest all match. Both builds complete all five binaries. - Earlier isolated investigation recorded byte differences in fresh jsqlite/cache.ss FASLs before boot construction, despite resetting random-seed, gensym-prefix, and gensym-count. Symbol encounter order differed even when the recorded symbol-key multiset matched. - A diagnostic modification to vendored Chez s/5_7.ss allowed make-session-key to read JERBOA_DETERMINISTIC_GENSYM_SESSION, retaining the ordinary path when unset. A distinct key per binary output, forwarded through secmon's restricted compiler-child environment, yielded matching stage-2 artifact manifests and a saved overall status=match report. The handoff also records passing binary smoke/security for that isolated candidate after a separate macOS linker correction. - A single global fixed key failed with: Exception in intern-gensym: unique name "jerboa-release-gensym-session-1255" already interned. Do not ship a global fixed key. These are diagnostic results, not an independently repeated minimal upstream regression or proof that the proposed environment hook is production-safe. The downstream checkout has other changes, and the hook has not been validated in a newly packaged self-contained Jerboa binary. ## Source evidence / hypothesis In Jerboa 2604e79232f31f5efd721b73b669fce10ebf0c66, vendor/ChezScheme/s/5_7.ss constructs the private session key using (cs)unique_id. In s/newhash.ss, symbol-hash can materialize a gensym unique name. s/fasl.ss traverses hashtable entries during graph discovery and emission. The experiments support session-dependent identity/hash ordering as a cause. Existing prefix rewriting after serialization is insufficient in the observed downstream builds; this does not prove that every possible canonical serializer is impossible. current-generate-id is already configured by secmon's compiler helper but does not cover all private session-generated identities. Maintainer investigation should decide whether to add a supported build-scoped identity facility or fix deterministic serialization at another layer. Correction to prior investigation: JERBOA_BINARY_STRIP_FASL and JERBOA_BINARY_CANONICALIZE_WPO were set in some secmon runs, but the inspected secmon builder/scripts do not consume them. Those runs do not establish that enabling those features caused any change. Explicit secmon WPO selection did execute and still produced mismatching reports. ## Reproduction assets for the local maintainer - ~/conduit-blockers.md contains chronological experiments; its latest correction supersedes the overstated policy/normalization conclusions. - ~/work/jerboa-secmon-detector: unpatched qualification candidate, scripts/reproducibility-report.sh, dist/reproducibility/{report.txt,first-build.log,second-build.log}, and both sets of saved binaries. - ~/work/jerboa-gensym-trace-diagnostic: patched diagnostic candidate and saved matching report, with first-stage2/second-stage2 manifests. Diagnostic wrapper/traces are not release changes. - ~/work/jerboa-gensym-session-inspection: branch conduit-deterministic-gensym, uncommitted make-session-key hook only; make chez completed, packaged multicall binary verification remains undone. First preserve the existing reports: the report script deletes its output directory on rerun. Reduce to a fresh-process jsqlite/cache compilation through a project make target or supported Jerboa compiler entry point, then compare both FASLs before boot assembly. Compare unpatched vs candidate using identical source/configuration. Include the parent/child environment boundary in the reproduction. ## Acceptance criteria 1. Regression reproduces the variance and proves the chosen fix in independent compiler processes with identical inputs. 2. Independent modules/artifacts retain distinct identities; dependent modules preserve shared identity. Load both independently built and mutually dependent FASLs together without collisions. 3. Ordinary runtime gensym behavior remains unchanged unless a documented build mode is explicitly selected; validate any new inputs and namespace policy. 4. Fresh packaged self-contained Jerboa supports the solution without external build-tree prerequisites. 5. Downstream two-build hashes, runtime smoke, and security checks pass with diagnostics removed. Keep raw-byte equality distinct from semantic FASL equivalence. Please investigate/implement in Jerboa and its bundled Chez as appropriate. This issue requests a supported capability, not adoption of the experimental environment hook verbatim. The user will handle this upstream work. Related but distinct fixed runtime-discovery issue: https://git.jerboa.sh/ober/jerboa/issues/93.

ober referenced this issue

2026-09-25 13:29:16 -04:00

Fix deterministic compilation with scoped gensym identities #98

ober referenced this issue from ober/jerboa-secmon

2026-09-25 15:50:16 -04:00

Adopt Jerboa 0.13.3 and qualify reproducible Conduit telemetry #8