Prune persisted row groups with zone maps #43

Merged

ober merged 6 commits from feat/lazy-zone-map-scan into main 2026-09-19 17:58:32 -04:00

Conversation 0 Commits 6 Files changed 10 +309 -17

ober commented 2026-09-15 00:40:42 -04:00

Owner

Copy link

Summary

• preserve persisted 2,048-row group boundaries when reopening columnar tables
• cache per-column zone maps for table chunks and prune impossible groups for simple direct-scan comparison and NULL predicates
• keep full vector predicate evaluation for every candidate group, reverse literal-left comparisons safely, and bypass pruning for semantic casts
• expose synchronized examined/pruned/candidate counters and add persistent reopen regressions
• advance the package and CLI version to 0.20.0

This is the first production SCAN/zone-map slice. Metadata-first file opening, projected-column block reads, bloom filters, and memory-accounted lazy decompression remain explicitly out of scope and documented as outstanding.

The 0.20.0 version is ordered after the currently open transaction PR #40 (#40), CSV PR #41 (#41), and WAL/checkpoint PR #42 (#42).

Verification

• all changed .ss files pass delimiter balance checks
• changed modules compile under jerboa_verify; only established lexical pre-scan false positives remain
• focused zone-map scan, zone-map, row-group, storage, catalog, and engine tests pass
• make build
• make unit
• make verify (653 local SQLLogic checks and 62 upstream subset checks passed; concurrency stress, native analysis, and documentation hygiene passed)
• make binary
• make binary-smoke
• changed-line jerboa_security_scan: no findings
• git diff --check

Review notes

• Production chunk bounds use tagged values so BOOLEAN #f cannot collide with the zone-map format's absent-bound sentinel.
• No-op casts may be unwrapped for pruning; type-changing casts deliberately bypass pruning.
• The source storage file is still fully decoded on open in this slice; query execution avoids materializing/filtering impossible restored chunks but does not yet avoid their initial I/O.

## Summary - preserve persisted 2,048-row group boundaries when reopening columnar tables - cache per-column zone maps for table chunks and prune impossible groups for simple direct-scan comparison and NULL predicates - keep full vector predicate evaluation for every candidate group, reverse literal-left comparisons safely, and bypass pruning for semantic casts - expose synchronized examined/pruned/candidate counters and add persistent reopen regressions - advance the package and CLI version to 0.20.0 This is the first production SCAN/zone-map slice. Metadata-first file opening, projected-column block reads, bloom filters, and memory-accounted lazy decompression remain explicitly out of scope and documented as outstanding. The 0.20.0 version is ordered after the currently open transaction PR #40 (https://git.jerboa.sh/ober/jerboa-duckdb/pulls/40), CSV PR #41 (https://git.jerboa.sh/ober/jerboa-duckdb/pulls/41), and WAL/checkpoint PR #42 (https://git.jerboa.sh/ober/jerboa-duckdb/pulls/42). ## Verification - all changed `.ss` files pass delimiter balance checks - changed modules compile under `jerboa_verify`; only established lexical pre-scan false positives remain - focused zone-map scan, zone-map, row-group, storage, catalog, and engine tests pass - `make build` - `make unit` - `make verify` (653 local SQLLogic checks and 62 upstream subset checks passed; concurrency stress, native analysis, and documentation hygiene passed) - `make binary` - `make binary-smoke` - changed-line `jerboa_security_scan`: no findings - `git diff --check` ## Review notes - Production chunk bounds use tagged values so BOOLEAN `#f` cannot collide with the zone-map format's absent-bound sentinel. - No-op casts may be unwrapped for pruning; type-changing casts deliberately bypass pruning. - The source storage file is still fully decoded on open in this slice; query execution avoids materializing/filtering impossible restored chunks but does not yet avoid their initial I/O.

ober added 1 commit 2026-09-15 00:40:42 -04:00

feat: prune persisted row groups with zone maps b4b1defc21

ober referenced this pull request 2026-09-15 01:01:36 -04:00

Add exact typed decimal aggregate states #44

ober added 7 commits 2026-09-19 16:33:21 -04:00

feat: stream CSV copy with reader options 4a1d1f8de1

test: update CLI smoke version 91287762b3

test: cover WAL and checkpoint crash boundaries a0d0fb996d

Merge main and resolve CSV feature conflicts 21b9c959e8

Retrigger CI after runner-side failure c1d1808916

Stack WAL recovery feature after CSV changes 8a3cf26794

Stack zone-map pruning after WAL recovery changes b1c353b4a1

ober added 3 commits 2026-09-19 17:09:55 -04:00

Bootstrap pinned Jerboa toolchain in CI 6eb1db2e51

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 7feef13435

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 19b4a5e54b

ober added 3 commits 2026-09-19 17:14:29 -04:00

Make close-race assertion platform-neutral 03b9b03fe1

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 3d654e0b2f

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 4438b69640

ober added 3 commits 2026-09-19 17:32:08 -04:00

Make native race smoke self-contained 363975ce4e

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 5f3a053eaa

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 44944ceb7f

ober added 3 commits 2026-09-19 17:46:36 -04:00

Build complete FreeBSD standalone binaries 37bdd45520

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 6e752fa236

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 447da45cad

ober merged commit c7843b8469 into main 2026-09-19 17:58:32 -04:00

ober referenced this pull request from a commit 2026-09-19 17:58:33 -04:00

Merge pull request #43

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ober/jerboa-duckdb!43

No description provided.