Issue #75: attributable, actionable DTrace profiles #90

Merged

ober merged 39 commits from fix/issue-75-attributable-traces into master 2026-09-22 00:52:31 -04:00

Conversation 0 Commits 39 Files changed 70 +6191 -816

ober commented 2026-09-21 18:46:58 -04:00

Owner

Copy link

Issue #75: attributable, actionable DTrace profiles

Closes #75.

What

End-to-end DTrace measurement pipeline: capture -> validated bundle -> actionable reports, plus the overhead bench harness and attribution workload.

Capture and bundle (steps 1-2)

• tools/dtrace/capture.sh: live FreeBSD capture producing a self-contained bundle (retained binary, argv, cohort identity, six raw families, checksums, atomic publish with INCOMPLETE fail-closed staging).
• support/dtrace/process-identity.c: native process incarnation identity (boot_id/pid/start_ns) for attributable cohorts; FreeBSD fix (sys/user.h include).
• Cohort model (tools/dtrace/cohort.ss): helper-backed identity, sealed cohorts, readiness gates; fixed a Gerbil-only untaint in the identity parse found by live capture.
• Validators and collectors hardened against real FreeBSD dtrace semantics found during live verification:
  • probe specs must use hyphen display names (allocation-sample, not allocation__sample)
  • USDT family gating and readiness must query exact probes: FreeBSD's pid provider lists every process function under any invented provider<pid> name, and dtrace -l exits 0 with zero matches
  • wait for USDT DOF registration before SIGSTOP, serialize collector grabs, escalate INT->TERM->KILL for backgrounded dtrace
  • D language fixes: no top-level assignments, no void* arithmetic, copyin takes uintptr_t, cpu collector records frame unknown (profile probes forbid func as a scalar; usym/ustack cannot be stringified)
  • 16m gc buffers (1M+ pause floods dropped records at 4m)

Reports (step 3)

• tools/dtrace/report.ss: report.json/report.md/cpu.folded/syscalls.csv/io.csv/allocations.csv/gc.csv/spans.csv from any bundle; complete = verdict pass && malformed=0; fail-closed integration into capture finalize (stub -> validate run1 -> real reports -> regen sums -> final validate -> publish).

Benchmarks and attribution (2g)

• benchmarks/dtrace/bench-overhead.ss with control lines, attach barrier, startup/elapsed split; bench-settings.env single source of truth; run-overhead.sh A/B driver (4 modes, alternating order, absolute binary paths); compare-overhead.sh gated comparison engine; deterministic attribution.ss workload (index-build/validate/io-overlap phases with user-event markers and io probes).
• syscalls-freebsd.d: flock/ioctl/dup2 probes.

Validation fixes found by live verification

• empty bundle members hashed as missing (get-bytevector-all returns eof for empty files) -> universal checksum mismatches; syscall fd/request/ret now accept signed integers (openat AT_FDCWD dirfd).

Verification

• make test-dtrace-unit green on macOS (78 PASS) and FreeBSD biggus (gmake DTRACE=on DTRACE_REQUIRED=1 test-dtrace-unit, 196 PASS lines).
• Live end-to-end on FreeBSD 15.1 (biggus, dtraceall loaded): six collectors attached serially, capture.sh --preset full published a validated bundle (rc=0), validate.ss rc=0, report.ss rc=0 report: complete (pass); gc.csv pause_count=60046, aggregated syscalls, cpu samples, control events present.
• make binary clean; jerboa-bin --version 0.13.0; VERSION bumped per policy.
• Cookbook entries for the FreeBSD dtrace lessons added to data/howtos.sexp.

# Issue #75: attributable, actionable DTrace profiles Closes #75. ## What End-to-end DTrace measurement pipeline: capture -> validated bundle -> actionable reports, plus the overhead bench harness and attribution workload. ### Capture and bundle (steps 1-2) - `tools/dtrace/capture.sh`: live FreeBSD capture producing a self-contained bundle (retained binary, argv, cohort identity, six raw families, checksums, atomic publish with INCOMPLETE fail-closed staging). - `support/dtrace/process-identity.c`: native process incarnation identity (boot_id/pid/start_ns) for attributable cohorts; FreeBSD fix (`sys/user.h` include). - Cohort model (`tools/dtrace/cohort.ss`): helper-backed identity, sealed cohorts, readiness gates; fixed a Gerbil-only `untaint` in the identity parse found by live capture. - Validators and collectors hardened against real FreeBSD dtrace semantics found during live verification: - probe specs must use hyphen display names (`allocation-sample`, not `allocation__sample`) - USDT family gating and readiness must query exact probes: FreeBSD's pid provider lists every process function under any invented `provider<pid>` name, and `dtrace -l` exits 0 with zero matches - wait for USDT DOF registration before SIGSTOP, serialize collector grabs, escalate INT->TERM->KILL for backgrounded dtrace - D language fixes: no top-level assignments, no `void*` arithmetic, `copyin` takes `uintptr_t`, cpu collector records frame `unknown` (profile probes forbid `func` as a scalar; `usym`/`ustack` cannot be stringified) - 16m gc buffers (1M+ pause floods dropped records at 4m) ### Reports (step 3) - `tools/dtrace/report.ss`: report.json/report.md/cpu.folded/syscalls.csv/io.csv/allocations.csv/gc.csv/spans.csv from any bundle; complete = verdict pass && malformed=0; fail-closed integration into capture finalize (stub -> validate run1 -> real reports -> regen sums -> final validate -> publish). ### Benchmarks and attribution (2g) - `benchmarks/dtrace/bench-overhead.ss` with control lines, attach barrier, startup/elapsed split; `bench-settings.env` single source of truth; `run-overhead.sh` A/B driver (4 modes, alternating order, absolute binary paths); `compare-overhead.sh` gated comparison engine; deterministic `attribution.ss` workload (index-build/validate/io-overlap phases with user-event markers and io probes). - `syscalls-freebsd.d`: flock/ioctl/dup2 probes. ### Validation fixes found by live verification - empty bundle members hashed as missing (`get-bytevector-all` returns eof for empty files) -> universal checksum mismatches; syscall fd/request/ret now accept signed integers (openat AT_FDCWD dirfd). ## Verification - `make test-dtrace-unit` green on macOS (78 PASS) and FreeBSD biggus (`gmake DTRACE=on DTRACE_REQUIRED=1 test-dtrace-unit`, 196 PASS lines). - Live end-to-end on FreeBSD 15.1 (biggus, dtraceall loaded): six collectors attached serially, `capture.sh --preset full` published a validated bundle (rc=0), `validate.ss` rc=0, `report.ss` rc=0 `report: complete (pass)`; gc.csv pause_count=60046, aggregated syscalls, cpu samples, control events present. - `make binary` clean; `jerboa-bin --version` 0.13.0; VERSION bumped per policy. - Cookbook entries for the FreeBSD dtrace lessons added to `data/howtos.sexp`.

ober added 29 commits 2026-09-21 18:46:58 -04:00

issue #75 step 1: bundle contract, strict validator, cohort registry ... 6606dcbf9a

- tools/dtrace/raw-schema.json: measurement-bundle contract v1 (layout,
  manifest/cohort/settings/capabilities/collectors JSON, raw record
  format with common ns-timestamped prefix, event pairing rules, exit
  contract)
- tools/dtrace/validate.ss: fail-closed bundle validator (SHA256SUMS
  coverage+integrity, manifest boundaries/clock units, cohort
  incarnation binding, collector lifecycle incl attach ack/exit/signal/
  drops/errors, per-producer clock monotonicity, ovf loss markers,
  span/io/gc pairing + parent graph, control ordering, syscall-open
  boundary tails, events_observed cross-check, artifact hashes, family
  state mapping); exit 0 only for a valid complete bundle; uses core
  sha256-bytevector (no crypto library imports) so it runs under
  bin/jerboa and jerboa-bin alike
- tools/dtrace/cohort.ss: cohort registry keyed by boot id + pid +
  native start_ns + runtime nonce (helper: support/dtrace/
  process-identity.c), synthetic mode for fixtures, USDT predicate and
  selector generation
- tests/fixtures/dtrace: make-good.sh (checked-in good bundle +
  drift detection), gen-variants.sh (15 bad variants, each isolating
  one failure property)
- tests/dtrace/test-bundle-validation.sh: good/copied/variant matrix;
  wired into make test-dtrace-unit; verified green on the repo wrapper
  and jerboa-bin bridge vehicle

issue #75 step 2a+2b: attributable spans and cross-worker context propagation ... 94cde83e03

dtrace.ss:
- %call-with-span rewritten over dynamic-wind: every entry into the
  dynamic extent opens a fresh segment span (same parent, tree-shaped);
  every exit closes it exactly once with outcome ok/error/cancel;
  continuation escape closes with cancel and reentry opens a new
  segment, so no span id is ever double-ended or left unmatched
- current-dtrace-context is now a thread parameter; spans opened by a
  worker cannot leak another thread's context
- new public capture/install API: dtrace-context-capture,
  dtrace-call-with-context; thread-context-wrap-hook installed at the
  end of the library (a mid-body top-level expression splits the Chez
  library body and unbinds forward references)

propagation sites:
- (std misc thread) thread-start! applies the hook in the creator and
  reinstalls the context in the child
- (std actor scheduler) scheduler-submit! wraps submitted tasks with
  the submitter's context
- (std async) spawn-async-task! propagates into task threads
- (std fiber) fiber-spawn wraps fiber bodies; the parameterize extent
  survives engine preemption/resumption

tests:
- test-spans: escape/reentry segment assertions (cancel outcome, new
  segment on reentry, ordering)
- new test-span-propagation (both vehicles: tier-2 exact-context via
  public API, tier-1 bridge-gated recorded parent linkage)
- new test-span-context-workers (repo vehicle: actor, fiber, nested
  actor->fiber exact-context checks)
- build.ss compiles (std actor scheduler) and (std fiber) to .so for
  the new tests; Makefile wires both test lists into test-dtrace-unit

dtrace: raw-schema v1 collectors and capture.sh bundle producer (issue #75 step 2c/2d) ... 5252799551

Collectors rewritten as independent pid-scoped consumers emitting
raw-schema v1 TSV records (jerboa$1/chez$1 pid-suffixed providers plus
explicit pid == $1 predicates for kernel providers; no dtrace -p grab):

- gc-pause.d: gc-start/gc-done/gcpause ns records, pause > 0 guard
- allocation.d: alloc-sample 12-field records with sampling weight
  (est = raw*w >> 16) and runtime/span attribution from the trace
  context
- cpu.d: cpu-sample 97 Hz on-CPU frame sampling with whitespace-frame
  drop counter reported at END
- io.d: io-start/io-done with duration, bytes, errno from io_v1
- user-spans.d: span tree records plus the capture lifecycle control
  records (ready/attached/warmup-done/workload-start at BEGIN,
  workload-done from proc:::exit)
- syscalls-freebsd.d / -macos.d / -linux.d: new per-syscall
  latency/fd/path collectors (openat/open/read/write/close/fsync);
  errno recorded as 0 where the carry flag is not a stable arg

capture.sh rewritten as the bundle producer: SIGSTOP target for the
attach window, data-driven attach acknowledgement per collector,
resume and wait for workload exit (fail-closed), drain, SIGINT
collectors, then finalize per-collector raw files (syscalls path_label
sanitization anchored on the fixed record tail, stable (pid, ts)
sort), emit collectors/settings/capabilities/manifest/cohort JSON,
stub report and validation verdict, checksums, and a two-pass
validate.ss embed ending in a read-only published bundle.  Non-FreeBSD
hosts exit 77 SKIP.

Tests: new test-capture-finalize.sh covers the sanitization and sort
transforms; test-dtrace-unit gains a shell syntax gate for the dtrace
tooling.  README updated for the new capture model.

Verification: make test-dtrace-unit green (19 bundle-validation
checks, 10 finalize checks, span/spans/propagation/workers suites,
syntax gate); capture.sh --help and the macOS SKIP-77 path exercised.
FreeBSD live capture verification lands with the issue #75 step-2e/2f
biggus run.

dtrace: honest capability reporting and incarnation-bound runtime ids (issue #75 step 2e) ... bbf742b8f3

Capability manifest (issue #75 capability 1):
- configure.sh: enabled verdict now requires the working C compiler
  (CC_OK) that was previously only diagnosed on --mode=on; the dtrace -h
  probe's exit status and stderr are preserved; manifest gains
  DTRACE_BIN, DTRACE_TOOL_VERSION, DTRACE_UID, DTRACE_SIP, DTRACE_CC_OK,
  DTRACE_PROBE_RC, DTRACE_PROBE_ERR, and DTRACE_ATTACH_RESULT
  (not_attempted: only a collector run may assert attached)
- doctor.sh: --json emits exactly one JSON object (DTRACE_MODE summary
  line moved to text mode); provider listing preserves per-command
  status and first stderr line instead of swallowing them; adds sip and
  attach_result facts
- jerboa_trace_capabilities: "registered" is now always "unknown"
  (compile-time linkage is reported separately as usdt_compiled); adds
  clock state (monotonic|degraded|unavailable) and clock_degraded
  counter; the monotonic clock fallback no longer substitutes wall-clock
  time - unavailability is counted and reported

Runtime identity (capability 3):
- jerboa_trace_runtime_id rekeys on process-incarnation change: a fork
  child with inherited memory gets a fresh id (forced store is safe in
  the single-threaded child) instead of aliasing the parent
- both runtimes mix a harness per-exec nonce from
  JERBOA_TRACE_RUNTIME_NONCE into the runtime id (unset changes
  nothing); documented in tools/dtrace/README.md
- dtrace.ss capabilities struct gains usdt-compiled, clock-name,
  clock-degraded accessors; test-api asserts the new fields

Tests: new tests/dtrace/test-identity.c fixture (stable in-process id,
fork rekey, parent unchanged, nonce accepted, honest capabilities JSON)
run by test-identity.sh in test-dtrace-unit; make binary clean; full
test-dtrace-unit green (42 PASS lines).

Verification evidence: standalone cc compile of the bridge exercised
capabilities JSON and fork rekey directly (child ids diverge across
repeated runs and under the nonce env); doctor.sh --json parsed as a
single JSON object by python3 on macOS; configure.sh auto/off and
--mode=on diagnostics exercised on Darwin.

dtrace: ignore generated test skip markers ... f79e517a3f

The .skip-test-tools marker is created by test-tools.sh on non-FreeBSD
hosts; it was accidentally committed with the step 2e change.  Drop it
from the index and ignore tests/dtrace/.skip-* so runs never stage it
again.

issue #75 step 3: actionable measurement reports + hardened publish ... b450570f21

report.ss: rewritten as the bundle summarizer. Reads manifest/settings/
validation plus the raw families, emits report.json, report.md, and
per-family extracts (cpu.folded flamegraph input, syscalls.csv, io.csv,
allocations.csv, gc.csv, spans.csv). complete = validation verdict pass
AND zero malformed records; prints a one-line verdict and exits 0 only
for a complete report. Short non-comment lines now count as malformed
(matching the validator), and header/comment lines never do.

capture.sh: --max-bytes bound (default 512 MiB, min 1 MiB, enforced
during collection, fail closed), private $output.staging.$$ assembly dir
under umask 077, atomic single-rename read-only publish with PUBLISHED
flag, INCOMPLETE marker in logs/capture.log on any abort, richer
artifacts manifest (binary captured + sha, debug_info/symbol_maps
not_captured with reasons, source_refs git revision + jerboa version).
Finalize now runs report.ss into the staging bundle after the first
validate embeds the real verdict, regenerates SHA256SUMS, and re-runs
validate over the published content: a bundle that cannot be summarized
must not publish.

tests: test-report.sh rewritten as a subprocess contract (good fixture
exits 0 with complete:true and all artifacts, fail verdict exits 1,
malformed record forces INCOMPLETE despite pass verdict, missing bundle
fails cleanly). New test-bundle-copy.sh: copied and tar-extracted
bundles revalidate standalone; one-byte corruption and missing manifest
are rejected. Both wired into test-dtrace-unit; suite green (165 PASS).

issue #75 step 2g: comparable A/B overhead modes + attribution example ... a37d509109

benchmarks: one settings file (bench-settings.env) now feeds every
overhead mode -- the historical untraced-gc-stress-1000 vs
traced-10000 mismatch is gone. run-overhead.sh runs the standard
modes (mode-off baseline, no-consumer, light-sampling, full-tracing)
on the instrumented binary, alternates normal/reversed mode order
across rounds (order recorded in environment.env), hashes the
settings file per participant, and delegates comparison to the new
compare-overhead.sh: per-pair (traced/no_consumer - 1)*100 with
median/min/max, negatives preserved, baseline-noise marking, exit-3
refusal on settings-hash mismatch, and AB-INVALID (never a
percentage) when runs or attach failed. The workload prints control
ready/attached/workload-start/workload-done boundaries, holds at a
BENCH_ATTACH_FILE readiness barrier (bounded) so consumers attach to
a quiescent target, and reports startup-ms separately from the
steady-state elapsed-ms used for comparisons.

New benchmarks/dtrace/attribution.ss: bounded workload with known
structure (dominant index-build phase, minor validate phase,
overlapping two-reader I/O phase), user-event phase markers, spans,
io probes, explicitly installed thread contexts (raw thread-start!
is not auto-propagated), and deterministic independent counters so
traced/untraced runs reconcile mechanically; temp file cleaned up
via unwind-protect.

syscalls-freebsd.d: flock/ioctl/dup2 entry/return records (fd=arg0,
request=arg1), completing the handoff's syscall set.

tests/dtrace/test-overhead.sh (wired into test-dtrace-unit): settings
parity + no hardcoded iteration counts, compare contract (refusal,
negatives, AB-INVALID), driver smoke on tiny settings (alternating
order + startup capture), attribution determinism + cleanup. Suite
green.

issue #75: bump VERSION to 0.13.0 ... b9710dfe2c

Public capabilities (span attribution, bundle contract + validator,
capture/report tooling, capability reporting) warrant a minor bump
per the repository version policy.

dtrace: include stdlib.h in trace.c for getenv/strtoull ... 570bf2546d

FreeBSD clang makes the implicit getenv declaration from the 2e nonce
change a hard int-to-pointer conversion error; macOS only warned.

dtrace: commit the good fixture's artifacts/libs member ... d4c61eb205

The fake libfixture.so was swallowed by the global *.so ignore, so
fresh clones (FreeBSD biggus) saw a drifted fixture; macOS only
passed on an untracked leftover. Negate the ignore for fixture
bundles and commit the member.

dtrace: retain target binary via procstat when procfs is absent ... b2d71837aa

FreeBSD hosts without a usable /proc/PID/file (biggus) fall back to
procstat -b to name the executable for artifacts retention.

dtrace: capture accepts wrapper or multicall jerboa binaries ... 45f6a18083

Detect once whether $JERBOA takes a run subcommand (repo wrapper) or
the script directly (jerboa-bin) and route cohort/validate/report
through a jrun helper.

dtrace: probe jerboa run-style with a real empty .ss file ... e10229d694

The repo wrapper rejects /dev/null as a script path, which made the
style probe misclassify it as the direct multicall binary under
capture.sh.

dtrace: include sys/user.h for struct kinfo_proc on FreeBSD ... 1b75fe87db

FreeBSD 15 clang rejects the incomplete kinfo_proc type from kvm.h
alone; the struct lives in sys/user.h.

dtrace: drop Gerbil-only untaint from the cohort identity parse ... f0b19c37d1

The live FreeBSD capture exposed it: the ready step never ran under
synthetic fixtures and untaint is not bound in Jerboa.

dtrace: remove illegal top-level assignment from the cpu collector ... f5ca4b1c6c

D scalars are implicitly zero-initialized; a bare top-level
assignment is a compile error under the real FreeBSD dtrace.

dtrace: make cpu collector compile under real FreeBSD dtrace ... bd0c011af1

FreeBSD rejects func as a scalar in profile probes, usym cannot be
stringified, and ustack keys render empty for static Chez binaries,
so v1 records frame unknown and keeps counts/attribution exact.

dtrace: wait for chez USDT registration before stopping target ... 58bb2e9b90

SIGSTOPing before the DOF registration lands leaves chez\$pid
invisible, so every grabbing collector fails to compile; poll the
provider read-only (bounded 10s) and fail with a precise message.

dtrace: fix void* arithmetic in collector copyin sites ... 91411e7e1c

The FreeBSD D compiler rejects void* + int; cast to char* at every
offset into a copyin record. Also settle 0.3s after SIGSTOP so the
stop fully lands before collectors grab the target.

dtrace: copyin takes uintptr_t, not char* 49c284037c

dtrace: serialize collector attaches, ack each before the next ... fb46f641a5

FreeBSD's pid provider rejects concurrent grabs of one process;
launch collectors one at a time behind the existing data-driven
attach ack (target is stopped, so nothing is lost).

dtrace: gate USDT families on exact probe names, not provider listings ... 7e3204c542

FreeBSD's pid provider lists every process function under any
invented provider<pid> name, so a bare chez\$pid:::/jerboa\$pid:::
count always exceeds zero and unlinked providers (jerboa spans/io)
slip through as observed. Query one concrete probe per family.

dtrace: exact-probe readiness gate and collector stop escalation ... 8da394909c

FreeBSD pid-provider listings spoof bare provider queries, so wait
for the real allocation__sample probe before stopping the target;
backgrounded dtrace inherits ignored SIGINT, so escalate INT to TERM
to KILL when stopping collectors (normal path and cleanup).

dtrace: readiness gate must count probe listings, not exit status ... 06f428c678

dtrace -l exits 0 with zero matches, so the previous gate always
passed instantly and froze the target before its USDT DOF landed.

dtrace: probe-name specs need the hyphen display form ... 5acdf9f5de

Exact-name listing and matching only work as allocation-sample /
user-span-start; the underscore form matches nothing on FreeBSD.

dtrace: user-spans collector must use hyphen probe names bcb0bc4dcf

dtrace: hash empty bundle members and accept signed syscall integers ... d1f640b0ce

get-bytevector-all yields eof for empty files, so empty collector
logs hashed as missing and failed every checksum; syscall fd/
request/ret can be negative (openat AT_FDCWD dirfd, failed-call
-1 returns) and need a signed integer token check.

dtrace: raise gc collector buffer to 16m ... c76211dfe0

gc-stress at full tilt emits over a million pauses in ~14s and
4m principal buffers drop tens of thousands of records.

dtrace: cookbook entries for FreeBSD live-capture lessons f13cfcd08d

ober added 1 commit 2026-09-21 19:02:44 -04:00

dtrace: update target-isolation test to the bundle capture contract ... c92a499431

The multi-collector rewrite scopes probes via pid-suffixed
providers and dollar-one predicates instead of dtrace -p, and
incarnation binding moved from the old sidecar note into cohort
members plus per-raw inc header lines.

ober added 1 commit 2026-09-21 19:06:06 -04:00

dtrace: port operator tools test to the bundle capture contract c26df2d8b3

ober added 1 commit 2026-09-21 19:09:16 -04:00

dtrace: fall back to the repo wrapper when no jerboa is on PATH bc06e44597

ober added 1 commit 2026-09-21 19:13:16 -04:00

dtrace: repo wrapper fallback resolves from the repo root 847269a210

ober added 1 commit 2026-09-21 19:22:13 -04:00

dtrace: stat the bundle raw directory itself in the tools test 02562860a6

ober added 1 commit 2026-09-21 19:25:04 -04:00

dtrace: clean expected bundle dirs before the tools capture test 694383d498

ober added 1 commit 2026-09-21 19:27:29 -04:00

dtrace: clean root-owned bundle dirs with matching privilege 51a993efe5

ober added 1 commit 2026-09-21 19:36:23 -04:00

dtrace: published bundles freeze raw dirs to 0555 1f29a05dfe

ober added 1 commit 2026-09-21 19:45:50 -04:00

dtrace: declare the identity helper and its test as native shims 2e1d9991a4

ober added 1 commit 2026-09-21 20:14:58 -04:00

dtrace: build the process-identity helper on demand in capture 83d9837d4d

ober force-pushed fix/issue-75-attributable-traces from 83d9837d4d to b7f95c20f6 2026-09-22 00:44:39 -04:00 Compare

ober scheduled this pull request to auto merge when all checks succeed 2026-09-22 00:52:21 -04:00

ober merged commit 2b00a04277 into master 2026-09-22 00:52:31 -04:00

ober referenced this pull request from a commit 2026-09-22 00:52:32 -04:00

Merge pull request #90

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ober/jerboa!90

No description provided.