Make sync recover multipart uploads reliably #63

Merged

ober merged 14 commits from fix/astra-jobs16-part-writes into main 2026-09-21 15:23:09 -04:00

Conversation 0 Commits 14 Files changed 9 +638 -117

ober commented 2026-09-21 03:04:39 -04:00

Owner

Copy link

Fix sync reliability for high-job-count uploads by capping simultaneous file uploads at four, splitting the unchanged JDv2 encrypted frame stream into S3-compliant 8 MiB multipart parts, and using smaller TLS request writes. Refresh upload deadlines on retry and drain nested part workers safely after cancellation or failure.

Validation:

• make test passes, including multipart frame-preservation and nested cancellation/failure regressions.
• macOS make clean, make binary, and make binary-smoke pass.
• make linux-amd64, make linux-arm64, and make freebsd-amd64 pass.
• git diff --check passes.

The requested full-home jd sync -v -d . jd:mac2/ was exercised but did not complete: an active QEMU process was writing ~/.android/avd/jmail_export_api26.avd/userdata-qemu.img.qcow2, and sync correctly rejected the changing source file. The emulator was left running.

Fix sync reliability for high-job-count uploads by capping simultaneous file uploads at four, splitting the unchanged JDv2 encrypted frame stream into S3-compliant 8 MiB multipart parts, and using smaller TLS request writes. Refresh upload deadlines on retry and drain nested part workers safely after cancellation or failure. Validation: - `make test` passes, including multipart frame-preservation and nested cancellation/failure regressions. - macOS `make clean`, `make binary`, and `make binary-smoke` pass. - `make linux-amd64`, `make linux-arm64`, and `make freebsd-amd64` pass. - `git diff --check` passes. The requested full-home `jd sync -v -d . jd:mac2/` was exercised but did not complete: an active QEMU process was writing `~/.android/avd/jmail_export_api26.avd/userdata-qemu.img.qcow2`, and sync correctly rejected the changing source file. The emulator was left running.

ober added 1 commit 2026-09-21 03:04:39 -04:00

Make sync recover multipart uploads reliably 656571c022

ober added 1 commit 2026-09-21 03:09:42 -04:00

ci: retry required checks after local verification 58fa103c91

ober added 1 commit 2026-09-21 03:48:00 -04:00

Continue sync when a source file changes c380322913

ober added 1 commit 2026-09-21 03:51:03 -04:00

ci: report required check stages separately ffd832111c

ober added 1 commit 2026-09-21 03:51:48 -04:00

ci: use GNU make on FreeBSD runner 623ca49249

ober added 1 commit 2026-09-21 03:56:09 -04:00

ci: isolate required test stages 0506ff9082

ober added 1 commit 2026-09-21 04:02:55 -04:00

ci: isolate sync regression checks 766239a2ca

ober added 1 commit 2026-09-21 04:07:38 -04:00

ci: group unit assertions for diagnosis fc86f17206

ober added 1 commit 2026-09-21 04:12:03 -04:00

ci: pinpoint FreeBSD unit test failures bfc4c68175

ober added 1 commit 2026-09-21 04:14:28 -04:00

ci: isolate FreeBSD unit assertion failures b3b90dfc57

ober added 1 commit 2026-09-21 04:51:10 -04:00

fix: skip unrepresentable S3 object keys 1be0b35d8d

ober added 1 commit 2026-09-21 08:44:07 -04:00

fix: allow long multipart operations to recover ae64572a16

ober added 1 commit 2026-09-21 12:09:33 -04:00

fix: extend timeout for multi-hour uploads 3a6f585715

ober added 1 commit 2026-09-21 14:49:30 -04:00

fix: skip S3 keys rejected as too long 58e884f3ac

ober scheduled this pull request to auto merge when all checks succeed 2026-09-21 15:22:56 -04:00

ober merged commit befc285829 into main 2026-09-21 15:23:09 -04:00

ober referenced this pull request from a commit 2026-09-21 15:23:10 -04:00

Merge pull request #63

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ober/jerboa-drive!63

No description provided.