Stream CSV COPY and expand reader options #41

Merged

ober merged 8 commits from feat/csv-incremental-options into main 2026-09-19 17:55:17 -04:00

Conversation 0 Commits 8 Files changed 16 +1136 -236

ober commented 2026-09-14 23:57:19 -04:00

Owner

Copy link

Summary

• replace whole-file CSV reads with a cancellation-aware incremental scanner and callback API
• stream COPY FROM conversion/insertion in bounded 1,024-row batches while preserving statement atomicity and constraints
• add CSV reader options, aliases, comments, UTF-8 BOM handling, permissive/error-skipping modes, padding, quoted-NULL control, and bounded delimiter autodetection
• make CSV export cancellation-safe through atomic same-directory replacement and return canonical COPY row counts
• advance the project version to 0.18.0 and document the supported CSV/COPY subset

Verification

• make verify
• make binary
• ./dist/jduckdb --version → v1.5.1 (Variegata) jduckdb 0.18.0
• focused test/unit/csv.ss
• changed-line security scan (the sole finding is the module's pre-existing direct Chez import)
• git diff --check

## Summary - replace whole-file CSV reads with a cancellation-aware incremental scanner and callback API - stream COPY FROM conversion/insertion in bounded 1,024-row batches while preserving statement atomicity and constraints - add CSV reader options, aliases, comments, UTF-8 BOM handling, permissive/error-skipping modes, padding, quoted-NULL control, and bounded delimiter autodetection - make CSV export cancellation-safe through atomic same-directory replacement and return canonical COPY row counts - advance the project version to 0.18.0 and document the supported CSV/COPY subset ## Verification - `make verify` - `make binary` - `./dist/jduckdb --version` → `v1.5.1 (Variegata) jduckdb 0.18.0` - focused `test/unit/csv.ss` - changed-line security scan (the sole finding is the module's pre-existing direct Chez import) - `git diff --check`

ober added 1 commit 2026-09-14 23:57:20 -04:00

feat: stream CSV copy with reader options 4a1d1f8de1

ober added 1 commit 2026-09-15 00:07:05 -04:00

test: update CLI smoke version 91287762b3

ober referenced this pull request 2026-09-15 00:20:18 -04:00

Cover WAL and checkpoint crash boundaries #42

ober referenced this pull request 2026-09-15 00:40:42 -04:00

Prune persisted row groups with zone maps #43

ober referenced this pull request 2026-09-15 01:01:36 -04:00

Add exact typed decimal aggregate states #44

ober added 1 commit 2026-09-19 16:26:35 -04:00

Merge main and resolve CSV feature conflicts 21b9c959e8

ober added 1 commit 2026-09-19 16:29:26 -04:00

Retrigger CI after runner-side failure c1d1808916

ober added 1 commit 2026-09-19 17:09:26 -04:00

Bootstrap pinned Jerboa toolchain in CI 6eb1db2e51

ober added 1 commit 2026-09-19 17:13:36 -04:00

Make close-race assertion platform-neutral 03b9b03fe1

ober added 1 commit 2026-09-19 17:31:37 -04:00

Make native race smoke self-contained 363975ce4e

ober added 1 commit 2026-09-19 17:46:18 -04:00

Build complete FreeBSD standalone binaries 37bdd45520

ober merged commit 621b849d9f into main 2026-09-19 17:55:17 -04:00

ober referenced this pull request from a commit 2026-09-19 17:55:18 -04:00

Merge pull request #41

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ober/jerboa-duckdb!41

No description provided.