CSV reader/writer edge-case fixes and COPY option expansion #54

Merged

ober merged 11 commits from feat/csv-scanner-opts into main 2026-09-19 18:43:46 -04:00

Conversation 1 Commits 11 Files changed 8 +79 -42

ober commented 2026-09-18 12:56:40 -04:00

Owner

Copy link

CSV reader/writer edge cases + COPY option expansion

Implements the audit-fix phase of the CSV/COPY packet (astra-complete.md 7.2) against vendored upstream semantics (pinned duckdb @117e1a46: csv_writer.cpp, copy_csv.cpp, test/sql/copy/csv/).

Reader (lib/jerboa/duckdb/csv.ss)

• NULL marker in a final field without trailing newline now converts exactly like a newline-terminated field (was skipped).
• A quote only opens quoting at field start; quotes inside unquoted fields are literal data (relaxed-quote behavior).
• Completely empty lines are skipped (upstream issue #4933 semantics); a lone quoted-empty field at EOF is a record.
• Fields track quotedness: unquoted null-marker => SQL NULL always; quoted null-marker => SQL NULL only with allow_quoted_nulls (default true, upstream parity).
• New params: allow-quoted-nulls?, skip (drops leading records before header consumption).

Writer

• Values equal to the null string are force-quoted (upstream CSVWriter::RequiresQuotes), so empty string vs SQL NULL stays distinguishable: NULL writes the bare marker, "" writes quoted.

COPY (parser + api)

• COPY ... TO returns a Count result row (was empty result).
• COPY ... TO writes the header by default unless HEADER FALSE is explicit (upstream csv_writer.cpp:89); COPY ... FROM unchanged (header off by default) via a header-mode tri-state on the AST.
• New options: SKIP n, ALLOW_QUOTED_NULLS true|false, SEP/SEPARATOR aliases for DELIMITER.

Tests

• test/unit/csv.ss: EOF null, quoted-empty-at-EOF, empty-line skip, relaxed quotes, allow_quoted_nulls matrix, skip, null-vs-empty write bytes, COPY TO Count + header default + round-trip.
• test/unit/engine.ss: updated for Count result and header-on default.

Verification

make build; make unit x3; make verify (contracts-check, diff, bench, sqllogic 664, sqllogic-upstream 62, sqllogic-events(+failure-check), concurrency-stress, native-race-check, check-docs, binary); make security; compat smoke; git diff --check — all green.

Version: 0.30.1

# CSV reader/writer edge cases + COPY option expansion Implements the audit-fix phase of the CSV/COPY packet (`astra-complete.md` 7.2) against vendored upstream semantics (pinned duckdb @117e1a46: `csv_writer.cpp`, `copy_csv.cpp`, `test/sql/copy/csv/`). ## Reader (`lib/jerboa/duckdb/csv.ss`) - NULL marker in a final field without trailing newline now converts exactly like a newline-terminated field (was skipped). - A quote only opens quoting at field start; quotes inside unquoted fields are literal data (relaxed-quote behavior). - Completely empty lines are skipped (upstream issue #4933 semantics); a lone quoted-empty field at EOF is a record. - Fields track quotedness: unquoted null-marker => SQL NULL always; quoted null-marker => SQL NULL only with allow_quoted_nulls (default true, upstream parity). - New params: `allow-quoted-nulls?`, `skip` (drops leading records before header consumption). ## Writer - Values equal to the null string are force-quoted (upstream `CSVWriter::RequiresQuotes`), so empty string vs SQL NULL stays distinguishable: `NULL` writes the bare marker, `""` writes quoted. ## COPY (parser + api) - `COPY ... TO` returns a Count result row (was empty result). - `COPY ... TO` writes the header by default unless `HEADER FALSE` is explicit (upstream csv_writer.cpp:89); `COPY ... FROM` unchanged (header off by default) via a header-mode tri-state on the AST. - New options: `SKIP n`, `ALLOW_QUOTED_NULLS true|false`, `SEP`/`SEPARATOR` aliases for `DELIMITER`. ## Tests - `test/unit/csv.ss`: EOF null, quoted-empty-at-EOF, empty-line skip, relaxed quotes, allow_quoted_nulls matrix, skip, null-vs-empty write bytes, COPY TO Count + header default + round-trip. - `test/unit/engine.ss`: updated for Count result and header-on default. ## Verification `make build`; `make unit` x3; `make verify` (contracts-check, diff, bench, sqllogic 664, sqllogic-upstream 62, sqllogic-events(+failure-check), concurrency-stress, native-race-check, check-docs, binary); `make security`; compat smoke; `git diff --check` — all green. Version: 0.30.1

ober added 1 commit 2026-09-18 12:56:40 -04:00

Fix CSV reader/writer edge cases and expand COPY options ... cb3a6de060

Reader: null-marker conversion now applies to a final field without a
trailing newline exactly as with one; a quote only opens quoting at
field start so quotes inside unquoted data are literal characters;
completely empty lines are skipped instead of becoming bogus one-field
records; a lone quoted empty field at end of file is a record. Fields
track quotedness so allow_quoted_nulls can keep quoted null markers
as strings (upstream allow_quoted_nulls default true), and skip drops
leading records before header consumption.

Writer: values equal to the null string are force-quoted so SQL NULL
and an equal string value stay distinguishable (upstream csv_writer
RequiresQuotes contract); with the default empty null string that
means empty strings write as quoted and NULL as bare.

COPY: TO returns a Count result row instead of an empty result and
writes the header by default unless HEADER FALSE is explicit (upstream
csv_writer.cpp); FROM accepts SKIP n, ALLOW_QUOTED_NULLS, and the
SEP/SEPARATOR aliases of DELIMITER. Copy statements carry header-mode
tri-state so the writer default does not change FROM behavior.

Verified against vendored upstream csv tests semantics (relaxed
quotes, allow_quoted_nulls, empty-line skip, null round-trip).

Version 0.30.1.

ober commented 2026-09-18 12:57:52 -04:00

Author

Owner

Copy link

CI note: required-ci failures are the instance-wide freebsd-amd64 runner outage also affecting PR #40 and main dispatches (every ci.yaml run since ~12:10 dies in 2-4s; healthy runs take 3-5 min; see runs 208-227). This tree is fully green locally: build, unit x3, verify (incl. sqllogic 664 + upstream 62), security, compat smoke. Will merge once the runner recovers.

**CI note:** required-ci failures are the instance-wide freebsd-amd64 runner outage also affecting PR #40 and main dispatches (every ci.yaml run since ~12:10 dies in 2-4s; healthy runs take 3-5 min; see runs 208-227). This tree is fully green locally: build, unit x3, verify (incl. sqllogic 664 + upstream 62), security, compat smoke. Will merge once the runner recovers.

ober added 44 commits 2026-09-19 17:01:49 -04:00

feat: stream CSV copy with reader options 4a1d1f8de1

test: update CLI smoke version 91287762b3

test: cover WAL and checkpoint crash boundaries a0d0fb996d

feat: prune persisted row groups with zone maps b4b1defc21

feat: add exact typed decimal aggregates 96ef8d24f1

docs: record typed aggregate review state f9c63b9140

feat: add byte-costed selected integer decoding 9d1b32af54

docs: record CODEC-INTEGER review state fbe49117ed

docs: mark CODEC-INTEGER CI green 4720210bbd

feat: add typed hash join build and probe 74feaa45cb

docs: record JOIN-HASH review state de25a5870e

docs: mark JOIN-HASH CI green 6eeb64f317

Implement lexical DECIMAL literals and arithmetic af5fe91242

Record TYPE-NUMERIC parity packet 41d12e425f

Mark TYPE-NUMERIC CI green 234c5a5b32

Run set operation sources through parallel pipelines c5286ef9ea

Record PIPELINE-SETOP parity packet 43c68dbc74

Mark PIPELINE-SETOP CI green 21cb363593

feat: add core regex scalar functions cdd6462199

docs: link regex parity packet to PR 49 4ad13ab086

docs: mark FUNC-TEXT-REGEX CI green 50a1fd8877

Add temporal formatting and parsing functions 031081b1df

Record temporal function pull request 02de9c49b1

Mark temporal function packet CI green c0e29c0e14

Add pinned MAP scalar functions f0e2b47726

Record MAP scalar pull request 4f44d1bfad

Mark MAP scalar packet CI green 7bf42e1b8b

Add univariate statistical aggregates c5b0c43407

Link statistical aggregate parity packet fc2b875a93

Mark statistical aggregate packet CI green 701dafebed

Merge main and resolve CSV feature conflicts 21b9c959e8

Retrigger CI after runner-side failure c1d1808916

Stack WAL recovery feature after CSV changes 8a3cf26794

Stack zone-map pruning after WAL recovery changes b1c353b4a1

Stack decimal aggregates after zone-map changes 0282a75bec

Stack integer codec changes after decimal aggregates 783e862eb6

Stack typed hash join after codec changes 81b3b88bde

Stack decimal arithmetic after typed hash join e6878e2dc0

Stack parallel set operations after decimal arithmetic e19ccc723f

Stack regex functions after parallel pipelines c1353b6206

Stack temporal functions after regex support a13c86adfa

Stack MAP functions after temporal support 5c261e63cc

Stack statistical aggregates after MAP support 4a95079685

Stack CSV scanner fixes after statistical aggregates 57f36e4d7f

ober added 13 commits 2026-09-19 17:10:43 -04:00

Bootstrap pinned Jerboa toolchain in CI 6eb1db2e51

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 7feef13435

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 19b4a5e54b

Merge remote-tracking branch 'origin/feat/lazy-zone-map-scan' into feat/typed-decimal-aggregates f4081a5ea6

Merge remote-tracking branch 'origin/feat/typed-decimal-aggregates' into feat/integer-codec-byte-cost f2de3c3e2c

Merge remote-tracking branch 'origin/feat/integer-codec-byte-cost' into feat/hash-join-typed bd8eda7039

Merge remote-tracking branch 'origin/feat/hash-join-typed' into feat/numeric-width-bucket 8784372ac2

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources cf62784178

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions aea0305b22

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 5b3a65b61b

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest f713d96bfa

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 67b03d7e10

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 6a4cbc5292

ober added 14 commits 2026-09-19 17:22:09 -04:00

Make close-race assertion platform-neutral 03b9b03fe1

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 3d654e0b2f

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 4438b69640

Merge remote-tracking branch 'origin/feat/lazy-zone-map-scan' into feat/typed-decimal-aggregates 7a7f03e1ec

Merge remote-tracking branch 'origin/feat/typed-decimal-aggregates' into feat/integer-codec-byte-cost 41c0ba2a79

Merge remote-tracking branch 'origin/feat/integer-codec-byte-cost' into feat/hash-join-typed db3af1229d

Preserve zone maps and decimal aggregates with hash joins 584ca19d0c

Merge remote-tracking branch 'origin/feat/hash-join-typed' into feat/numeric-width-bucket 95c62a798c

Preserve earlier operators in pipeline integration 6e0323d3f6

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions c26826b10c

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 14410f6090

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 80af6a5b46

Preserve cumulative operators with statistical aggregates 1aa3ebce1b

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 7aa628c791

ober added 1 commit 2026-09-19 17:23:37 -04:00

Correct COPY query count after CSV fixture expansion ddb51ef2db

ober added 7 commits 2026-09-19 17:25:48 -04:00

Import operator instrumentation in engine tests ac947f1cfb

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources 089c004e8b

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions eddde34854

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 945f565c7f

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 2831218666

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 41849b8549

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 903e4f5799

ober added 13 commits 2026-09-19 17:32:55 -04:00

Make native race smoke self-contained 363975ce4e

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 5f3a053eaa

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 44944ceb7f

Merge remote-tracking branch 'origin/feat/lazy-zone-map-scan' into feat/typed-decimal-aggregates 4e37d52e29

Merge remote-tracking branch 'origin/feat/typed-decimal-aggregates' into feat/integer-codec-byte-cost cc00ed7109

Merge remote-tracking branch 'origin/feat/integer-codec-byte-cost' into feat/hash-join-typed 7bce87f884

Merge remote-tracking branch 'origin/feat/hash-join-typed' into feat/numeric-width-bucket d7a5dca543

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources 1593e432bc

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions 9a94301753

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions f688cb0036

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 804e52c366

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates db69437b7d

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 913c013f0f

ober added 13 commits 2026-09-19 17:47:24 -04:00

Build complete FreeBSD standalone binaries 37bdd45520

Merge remote-tracking branch 'origin/feat/csv-incremental-options' into feat/wal-checkpoint-faults 6e752fa236

Merge remote-tracking branch 'origin/feat/wal-checkpoint-faults' into feat/lazy-zone-map-scan 447da45cad

Merge remote-tracking branch 'origin/feat/lazy-zone-map-scan' into feat/typed-decimal-aggregates 00eaae6fc7

Merge remote-tracking branch 'origin/feat/typed-decimal-aggregates' into feat/integer-codec-byte-cost c061ad4387

Merge remote-tracking branch 'origin/feat/integer-codec-byte-cost' into feat/hash-join-typed 21fd064bca

Merge remote-tracking branch 'origin/feat/hash-join-typed' into feat/numeric-width-bucket 75067441f6

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources 92a44ad453

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions 748af906c5

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 1dbe79622e

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 79d0dcba2e

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 020d1d41af

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts bdebb75363

ober added 8 commits 2026-09-19 18:05:50 -04:00

Expect COPY count result in engine test 297b0e53be

Merge remote-tracking branch 'origin/feat/hash-join-typed' into feat/numeric-width-bucket 9391989736

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources 2f7b0ecb60

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions e691d5e913

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions d860cfba2f

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest dfc23f6a3e

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 8403335649

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts ... 2fa5390a39

# Conflicts:
#	test/unit/engine.ss

ober added 7 commits 2026-09-19 18:19:43 -04:00

Cast quantified subquery decimals before comparison 6abc410ce5

Merge remote-tracking branch 'origin/feat/numeric-width-bucket' into feat/pipeline-parallel-sources 27248eec56

Merge remote-tracking branch 'origin/feat/pipeline-parallel-sources' into feat/text-regex-functions 378b747159

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 80e0349e8a

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 3f9dd1b9cb

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 8084eb1f92

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 4b0cced23a

ober added 5 commits 2026-09-19 18:33:34 -04:00

Accept closed outcome in query close race d0e1ae1206

Merge remote-tracking branch 'origin/feat/text-regex-functions' into feat/temporal-timezone-functions 88438d147a

Merge remote-tracking branch 'origin/feat/temporal-timezone-functions' into feat/nested-map-unnest 62ee42fbca

Merge remote-tracking branch 'origin/feat/nested-map-unnest' into feat/statistical-aggregates 2fc1d76052

Merge remote-tracking branch 'origin/feat/statistical-aggregates' into feat/csv-scanner-opts 2e7008e214

ober merged commit f9e705f7d0 into main 2026-09-19 18:43:46 -04:00

ober referenced this pull request from a commit 2026-09-19 18:43:48 -04:00

Merge pull request #54

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ober/jerboa-duckdb!54

No description provided.