ober/jerboa-duckdb
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Per-object transaction commit validation and publication (MVCC write sets) #40

Merged
ober merged 1 commit from feat/tx-write-sets into main 2026-09-19 16:23:45 -04:00
Conversation 1 Commits 1 Files changed 7 +389 -6
ober commented 2026-09-14 23:03:01 -04:00
Owner
Copy link

Per-object transaction commit (MVCC write sets)

What

Replaces whole-database snapshot commit (database-commit-snapshot!, kept as the conservative fallback) with database-commit-objects!: a transaction's commit validates only the catalog objects it wrote against committed object revisions, then publishes exactly those objects from its working snapshot into the shared base.

  • Concurrent transactions writing disjoint objects both commit (previously any concurrent commit conflicted).
  • Same-object write-write still rejects with the canonical TransactionContext Error: Conflict on commit.
  • Object kinds: tables (table:), views (view:), indexes (index:), sequences (sequence:), schemas (schema: create/drop). Unknown keys (e.g. database:*) fall back to the conservative whole-database check + swap.
  • Table drop/rename merges drop dependent indexes (remove-indexes-for-table!), mirroring the catalog paths.
  • Sequence values merge as non-transactional gaps (furthest-advanced value wins, never regresses).

Known limitation

Each transaction fork consumes nextval from its own snapshot of the sequence state, so concurrent transactions may draw the same value; commit merging guarantees only that the furthest-advanced value survives (test documents this: two concurrent first draws → next value is 2, not 3).

Tests

test/unit/concurrency.ss additions: disjoint-table commits, same-table conflict preserved, disjoint DDL (table vs view), DML+DDL commute, sequence gap merge, drop-vs-insert commute. Existing interrupt/snapshot/GC/history tests unchanged and green.

Verification

  • make build
  • make unit x3 (concurrency suite green every run)
  • make verify (contracts-check, diff, bench, sqllogic, sqllogic-upstream, sqllogic-events, sqllogic-events-failure-check, concurrency-stress, native-race-check, check-docs, binary build) — all green
  • make security, sh test/cli/compat.sh dist/jduckdb, git diff --check — clean

Version: 0.17.15

# Per-object transaction commit (MVCC write sets) ## What Replaces whole-database snapshot commit (`database-commit-snapshot!`, kept as the conservative fallback) with `database-commit-objects!`: a transaction's commit validates **only the catalog objects it wrote** against committed object revisions, then publishes exactly those objects from its working snapshot into the shared base. - Concurrent transactions writing **disjoint objects both commit** (previously any concurrent commit conflicted). - Same-object write-write still rejects with the canonical `TransactionContext Error: Conflict on commit`. - Object kinds: tables (`table:`), views (`view:`), indexes (`index:`), sequences (`sequence:`), schemas (`schema:` create/drop). Unknown keys (e.g. `database:*`) fall back to the conservative whole-database check + swap. - Table drop/rename merges drop dependent indexes (`remove-indexes-for-table!`), mirroring the catalog paths. - Sequence values merge as non-transactional gaps (furthest-advanced value wins, never regresses). ## Known limitation Each transaction fork consumes `nextval` from its own snapshot of the sequence state, so concurrent transactions may draw the same value; commit merging guarantees only that the furthest-advanced value survives (test documents this: two concurrent first draws → next value is 2, not 3). ## Tests `test/unit/concurrency.ss` additions: disjoint-table commits, same-table conflict preserved, disjoint DDL (table vs view), DML+DDL commute, sequence gap merge, drop-vs-insert commute. Existing interrupt/snapshot/GC/history tests unchanged and green. ## Verification - `make build` - `make unit` x3 (concurrency suite green every run) - `make verify` (contracts-check, diff, bench, sqllogic, sqllogic-upstream, sqllogic-events, sqllogic-events-failure-check, concurrency-stress, native-race-check, check-docs, binary build) — all green - `make security`, `sh test/cli/compat.sh dist/jduckdb`, `git diff --check` — clean Version: 0.17.15
Implement per-object transaction commit validation and publication
All checks were successful
version-policy / required (pull_request) Successful in 4m46s
Details
required-ci / required (pull_request) Successful in 14m31s
Details
4cc04a575a 
Transactions now track which catalog objects they wrote (tables,
views, indexes, sequences, schemas) and commit validates only those
object revisions against committed state, publishing exactly the
written objects from the working snapshot into the shared base.
Concurrent transactions writing disjoint objects both commit;
same-object write-write conflicts still reject with the canonical
Conflict on commit error. Sequence values merge as non-transactional
gaps taking the furthest-advanced value. Unknown object universes
fall back to the conservative whole-database conflict check.

Version 0.17.15.
ober force-pushed feat/tx-write-sets from 4cc04a575a
All checks were successful
version-policy / required (pull_request) Successful in 4m46s
Details
required-ci / required (pull_request) Successful in 14m31s
Details
to f48bfff08a
Some checks failed
required-ci / required (pull_request) Failing after 3m43s
Details
version-policy / required (pull_request) Successful in 3m43s
Details
2026-09-18 12:10:07 -04:00 Compare
ober commented 2026-09-18 12:35:47 -04:00
Author
Owner
Copy link

CI status note (2026-09-18 ~12:35 UTC-4): required-ci failures on this PR (runs 208/210/212) are infrastructure, not the change:

  • All three die in 2-4 seconds (healthy runs of make verify on freebsd-amd64 take 3-5 minutes).
  • workflow_dispatch of the same ci.yaml on main (run 214, commit b392ffe) fails identically in 3s — that exact commit passed at 10:42 (run 207, 3.5 min).
  • The last healthy ci.yaml run anywhere on the instance was 10:42:58; every ci.yaml run since 12:10 (mine + main dispatches 214-224) has failed instantly. version-policy runs succeed on the same events.
  • The original pre-rebase commit 4cc04a5 passed required-ci in full (run 141).

The rebased f48bfff is fully green locally: make build, make unit x4, make verify, make security, compat smoke, git diff --check. Will retrigger and merge once the runner recovers.

**CI status note (2026-09-18 ~12:35 UTC-4):** `required-ci` failures on this PR (runs 208/210/212) are infrastructure, not the change: - All three die in 2-4 seconds (healthy runs of `make verify` on freebsd-amd64 take 3-5 minutes). - `workflow_dispatch` of the **same ci.yaml on main** (run 214, commit b392ffe) fails identically in 3s — that exact commit passed at 10:42 (run 207, 3.5 min). - The last healthy ci.yaml run anywhere on the instance was 10:42:58; every ci.yaml run since 12:10 (mine + main dispatches 214-224) has failed instantly. version-policy runs succeed on the same events. - The original pre-rebase commit 4cc04a5 passed required-ci in full (run 141). The rebased f48bfff is fully green locally: `make build`, `make unit` x4, `make verify`, `make security`, compat smoke, `git diff --check`. Will retrigger and merge once the runner recovers.
ober merged commit 12eb0ae079 into main 2026-09-19 16:23:45 -04:00
ober referenced this pull request from a commit 2026-09-19 16:23:46 -04:00
Merge pull request #40
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ober/jerboa-duckdb!40
No description provided.