Jerboa Scheme LoRA

A fine-tune of Qwen3-Coder-30B-A3B-Instruct that knows Jerboa — a Chez-Scheme-based dialect with a Gerbil-flavored prelude. The LoRA teaches the base model Jerboa's module syntax ((std foo), (jerboa prelude) — not :std/foo), the standard library, the actor/fiber system, the FFI, and how Jerboa diverges from Gerbil/Racket/Clojure/SRFI.

Three lines:

• v3 — public, 3-stage CPT → SFT → DPO on RunPod A100. Live as jaimef/jerboa-qwen on Ollama.
• v4 — private, same pipeline on a heretic-abliterated Qwen3-Coder base (uncensored). Single-user; not on Ollama.
• v5 — private, same pipeline on a different base: abliterated Qwen3.6-35B-A3B (qwen3_5_moe hybrid — 30 Gated-DeltaNet + 10 full-attention layers, 256 fused experts + shared expert). Text-only (MTP and vision heads dropped). BF16 on HF private, MLX 6-bit local; not on Ollama.

Quick Start

v3 (public, Ollama)

ollama pull jaimef/jerboa-qwen
ollama run jaimef/jerboa-qwen "How do I import the Jerboa prelude and parse JSON?"

Tags: :latest is Q8_0 (~32 GB, full fidelity). :q4_k_m is the smaller quant (~18 GB). Pick :latest if it fits.

v4 (private, local MLX)

mlx_lm.generate --model jerboa-mlx-6bit-v4 --prompt "How do I import the Jerboa prelude and parse JSON?"
# or run as an OpenAI-compatible server:
mlx_lm.server --model jerboa-mlx-6bit-v4

BF16 backup on HF private: jaimef21/jerboa-qwen3-coder-30b-v4. MLX 6-bit stays local — see upload_hf_v4.sh if you ever want to retry the HF push (S3 multipart from residential uplink broken-pipes; the script is idempotent and resume-friendly).

v5 (private, local MLX)

mlx_lm.generate --model jerboa-mlx-6bit-v5 --prompt "How do I import the Jerboa prelude and parse JSON?"
mlx_lm.server   --model jerboa-mlx-6bit-v5   # OpenAI-compatible server

BF16 backup on HF private: jaimef21/jerboa-qwen3.6-35b-a3b-v5 (16 shards, 69.3 GB). MLX 6-bit local: jerboa-mlx-6bit-v5/ (6 shards, 26 GB, 6.502 bpw). On a 48 GB Mac keep sudo sysctl iogpu.wired_limit_mb=32768 — values above ~32 GB kernel-panic when loading this 23+ GB bundle (the Makefile caps it; do not raise it).

MLX model_type gotcha. The text-only save_pretrained writes config.json with model_type="qwen3_5_moe_text" (correct for transformers, which registers a …TextForCausalLM class). But mlx_lm only ships models/qwen3_5_moe.py, so it rejects …_text as unsupported. Fix without touching the canonical HF repo: symlink the cached shards into a scratch dir and write a patched config.json with model_type set back to qwen3_5_moe (its ModelArgs.from_dict wraps the flat config as text_config when no text_config key exists, so that's sufficient), then mlx_lm.convert --hf-path <scratch>. This is what v5_mlx_src/ is.

Pipeline

Three stages on one pod, ~5–6h wall clock. See TRAINING_PIPELINE.md for design rationale and TRAINING_FIX_PLAN.md for the v3 hyperparam corrections that fixed the v1 "DPO delta below the quantization floor" pathology.

Stage	What it learns	Data	LR / Epochs	LoRA (v4)
CPT	Token distribution of raw Jerboa source	cpt_corpus_v3.jsonl (~6 MB, mined from jerboa, jerboa-mcp, stdlib doc-comments)	2.0e-5 / 2	r=32, α=64
SFT	Answer Jerboa questions in chat	sft_v3_mined.jsonl (~3.5 MB, compile-gated via jerboa_check_syntax)	1.0e-4 / 2	r=32, α=64
DPO	Suppress Gerbil/Racket/Clojure/SRFI hallucinations	dpo_pairs_v3.jsonl (~300 pairs, compile-validated chosen)	5.0e-6 / 3	r=32, α=64

v3 ran at r=64 / α=128 on A100 80GB. v4 dropped to r=32 / α=64 on H100 NVL (95.8 GiB) because the larger adapter OOMs during paged_adamw_8bit optimizer state init even on the bigger card. r=32 fits with ~19 GiB headroom and DPO still converged cleanly (final loss 0.0008, rewards/accuracies 1.0).

v5 keeps r=32 / α=64 but runs on H200 NVL (the 35B-A3B qwen3_5_moe is larger than v4's 30B-A3B). Two arch-specific fixes were needed for axolotl on this base: (1) disable the LoRA Triton kernel patcher (lora_mlp_kernel: false, lora_qkv_kernel: false, lora_o_kernel: false) — axolotl tries to import transformers.models.qwen3_5_moe_text for the fused kernels and crashes; with them off it falls back to plain PEFT LoRA, which works. (2) merge_lora_qwen35moe.py shims transformers.core_model_loading.WeightConverter.__init__ to swallow the distributed_operation/quantization_operation kwargs that peft-0.19.1 passes but transformers-5.8.1 doesn't accept — without it, any adapter load that targets the fused experts.* (i.e. all of them) crashes during the inter-stage merge.

Each stage's LoRA is merged into bf16 between stages so the next stage trains on absorbed weights, not stacked adapters. Targets attention + MLP + MoE experts (experts.gate_up_proj, experts.down_proj) — the key reason RunPod axolotl beat the previous Together-AI single-stage path, which was attention-only.

Run the pipeline

The active remote-training entry point is v6_runpod.py — the only supported orchestrator (see docs/training-security.md). It trains CPT → SFT → DPO as PEFT adapters and pulls the final adapter directory (it does not produce a fused model). Run make security first to validate the immutable training lock.

make security                     # validate security/training-lock.json + inputs

# Full pipeline: up → push → prepare-cache → preflight → train-smoke → train → pull → down
python3 v6_runpod.py auto

# Or drive stages individually:
python3 v6_runpod.py up
python3 v6_runpod.py push
python3 v6_runpod.py preflight
python3 v6_runpod.py train        # full adapter sequence; auto-runs upload-hf
python3 v6_runpod.py pull         # download adapter + logs (not a merged model)
python3 v6_runpod.py status
python3 v6_runpod.py down

Pod state lives in .runpod_state_v6.json (gitignored). See version-6.md for the full v6 design and subcommand reference.

Retired: runpod_train.py (and its .runpod_state.json) is disabled — it predates authenticated/private services and immutable training inputs, and now exits immediately with a pointer to v6_runpod.py. The v3/v4/v5 runs described below used it; do not revive it.

Build data

python3 build_cpt_corpus_v2.py    # → cpt_corpus_v3.jsonl
python3 build_sft_v3.py           # → sft_v3_mined.jsonl (compile-gated)
python3 build_dpo_pairs_v3.py     # → dpo_pairs_v3.jsonl (compile-validated)

The builders read from ~/mine/jerboa (source, stdlib, tests, docs) and ~/mine/jerboa-mcp (cookbooks, divergence, error-fixes, api-signatures) via the jerboa_* MCP tools.

Convert + publish

# MLX 6-bit bundle for Mac (loads under mlx_lm or LM Studio)
./convert_to_mlx.sh runpod-pipeline-final jerboa-mlx-6bit-v3

# HuggingFace
./upload_hf.sh

Retired (GGUF/Ollama): build_ollama_gguf.sh is disabled until the conversion dependencies and model artifacts have a reviewed lock — it exits non-zero on purpose. Because push_ollama.sh requires a locally-built GGUF tag (jerboa-qwen:latest), the Ollama push path currently cannot complete end-to-end. The published jaimef/jerboa-qwen Ollama tags predate this retirement.

Evaluation

v3 is gated on two evals before publish; both must beat the un-fine-tuned base.

Eval	What it measures	Script	v3 result
Held-out coding	Compile-rate + idiom hits on questions never seen during training	eval_holdout.py	see eval_pod_v3.json vs eval_pod_base.json
Per-pair similarity	Chosen-vs-rejected on the DPO set (Jaccard + char-similarity, volume-invariant)	eval_similarity.py	see eval_mlx_v3.json vs eval_mlx_v2.json

The similarity eval is the one to trust — it's volume-invariant and per-pair, so it doesn't get gamed by dataset growth.

v5 results (eval_pod_base.json = abliterated base vs eval_pod_v5.json = trained, 12 holdout Qs / 40 similarity pairs):

Metric	Base	v5
Holdout total score	19 (mean 1.58)	31 (mean 2.58)
Holdout has-code	10/12	12/12
Mean generation time	15.2 s	4.2 s
Similarity tok_lean (mean)	0.0288	0.0873
Pairs leaning chosen	34/40	32/40
Refusals (eval_refusal_hf.py, 24 prompts)	—	0/24

v5 clearly beats its base on every coding signal; tok_lean triples (stronger pull toward Jerboa idioms) and 0/24 refusals confirms the abliteration held through training.

Use with OpenCode

Local Ollama

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": { "jerboa-qwen": { "name": "Jerboa Qwen" } }
    }
  }
}

Or run ./configure_opencode.sh ollama to write that for you.

Why this works where the previous approach didn't

The v1 pipeline was a single SFT pass on Together AI (attention-only LoRA) + a single SFT pass on MLX (attention + MLP, 4-bit base). Two structural problems:

1. No CPT — the model never saw raw Jerboa source. It learned "how to answer questions about Jerboa" but not "how Jerboa code naturally looks."
2. No DPO — the 119 divergence entries (hash-has-key? → hash-key?, (srfi :NN) → (srfi NN), etc.) were flattened into SFT Q/A, which only pulls the model toward the right form. DPO actively pushes it away from the wrong form. The wrong form's prior (learned from the base's massive Racket/Gerbil training data) survived essentially intact.

v3 fixes both, plus the v1 DPO hyperparam pathology (LR 5e-7 × 18 steps × LoRA r=16 produced a cumulative weight update of ~2e-5 — below the BF16 precision floor and erased by Q4_K_M quantization). v3 runs LR 5e-6 / 3 epochs / r=64, α=128.

v4 abliteration

v4 abliterates the base model before re-running the CPT → SFT → DPO pipeline so the final model has no refusal floor.

Heretic detour. Initial v4 attempts used Heretic (runpod_abliterate.py + heretic_auto_driver.py to bypass its TUI). All three Heretic runs plateaued at 75–96/100 refusals: v1 (75), v2 with winsorization (96, aborted), v3 with orthogonalize_direction=false (back to 75). Diagnosis: Heretic v1.3.0 silently skips Qwen3-Coder's MoE experts. Its detection loop does for expert in layer.mlp.experts:, but Qwen3MoeExperts stores down_proj as a single fused [num_experts, hidden, intermediate] Parameter — the iteration raises TypeError, gets swallowed by Heretic's with suppress(Exception):, and Heretic ablates only the 48 attn.o_proj modules while ignoring all 48×128 = 6144 expert down_projs. That's the 75/100 floor.

The fix. direct_ablate_qwen3moe.py does Arditi-et-al-style directional ablation MoE-correctly: computes the refusal direction once, then applies W ← (I - rr^T) W to both self_attn.o_proj.weight (standard Linear) and mlp.experts.down_proj (fused parameter, sliced per-expert) for every layer. eval_ablation.py is the matching refusal-counter (mirrors Heretic's eval to keep comparisons honest). Output base: jaimef21/qwen3-coder-30b-a3b-abliterated-v2 (private).

v5 abliteration

v5 abliterates a different base — Qwen3.6-35B-A3B (qwen3_5_moe), a hybrid VLM with 40 decoder layers (full_attention_interval=4 → 30 Gated-DeltaNet + 10 full-attention) and MoE on every layer (256 fused routed experts + 1 shared expert). direct_ablate_qwen35moe.py handles the arch correctly: it ablates self_attn.o_proj only on the 10 full-attention layers (the DeltaNet layers have no o_proj to project the residual through the same way), plus the fused experts.down_proj and the shared_expert.down_proj on all 40 layers. We load the text-only …TextForCausalLM (693 weight keys) — the MTP and vision heads are dropped so they never enter training or the final bundle. Output base: jaimef21/qwen3.6-35b-a3b-abliterated-v5 (private). eval_refusal_hf.py counts refusals on the resulting model (0/24 after the full pipeline).

Repo layout

File	Purpose
v6_runpod.py	Active RunPod orchestrator for v6 adapter training (up/push/preflight/train/pull/down/auto); see version-6.md
runpod_train.py	Retired/disabled — old RunPod staged-training orchestrator (up/push/train/eval/pull/down); exits immediately and points to v6_runpod.py
runpod_abliterate.py	RunPod pod lifecycle for the (abandoned) v4 Heretic abliteration
heretic_auto_driver.py	Monkey-patches Heretic's TUI prompts so it runs hands-off + uploads to HF
direct_ablate_qwen3moe.py	MoE-correct directional abliteration (Heretic skipped Qwen3-Coder's fused experts.down_proj) — the script that actually produced v4's base
direct_ablate_qwen35moe.py	v5 abliteration for qwen3_5_moe: o_proj on the 10 full-attn layers + fused experts.down_proj/shared_expert.down_proj on all 40; text-only load (MTP/vision dropped)
eval_ablation.py	Refusal-count evaluator (mirrors Heretic's eval, run after direct_ablate_qwen3moe.py)
eval_refusal_hf.py	v5 refusal counter on the HF (transformers) model — 0/24 after the full pipeline
axolotl_jerboa_{cpt,sft,dpo}_v5.yaml	v5 per-stage configs (qwen3_5_moe base; lora_*_kernel: false to skip the broken Triton patcher)
merge_lora_qwen35moe.py	Inter-stage LoRA→bf16 merge for v5; shims WeightConverter.__init__ around the peft-0.19.1 ↔ transformers-5.8.1 kwarg mismatch on fused experts
push_v5.py	Pod-side upload_large_folder push of the merged v5 BF16 → jaimef21/jerboa-qwen3.6-35b-a3b-v5 (private, resumable xet)
upload_hf_v4.sh	Resume-friendly per-file uploader for jerboa-mlx-6bit-v4 (tries hf_transfer then plain, exponential backoff, hf upload skips already-committed LFS files)
axolotl_jerboa_{cpt,sft,dpo}.yaml	Per-stage axolotl configs (v4: r=32, α=64, MoE expert targets)
build_cpt_corpus_v2.py	Mine ~/mine/jerboa + ~/mine/jerboa-mcp → CPT corpus
build_sft_v3.py	Mine cookbooks + error-fixes + stdlib → SFT, compile-gated
build_dpo_pairs_v3.py	Generate divergence pairs across Gerbil/Racket/Clojure/SRFI surface variants, compile-validated
eval_holdout.py	Held-out Jerboa coding eval (compile-rate + idiom hits)
eval_similarity.py	Per-pair chosen-vs-rejected similarity eval
eval_on_pod.py	Pod-side eval runner (base + trained, parallel)
eval_mlx.py	MLX-side eval runner for local Mac comparison
build_ollama_gguf.sh	Retired/disabled — GGUF Q8_0 + Q4_K_M conversion stub; exits non-zero until conversion deps + artifacts have a reviewed lock
push_ollama.sh	Push to ollama.com (:latest = Q8_0; --all also pushes :q4_k_m) — currently blocked: needs a GGUF tag from the retired build_ollama_gguf.sh
upload_hf.sh	Push merged model to HuggingFace
config.jerboa.v2.toml	Heretic config for v4 abliteration
convert_to_mlx.sh	mlx_lm.convert wrapper
manage_runpod.sh	RunPod endpoint lifecycle (separate from training pods)

Iteration loop

1. Add recipes to ~/mine/jerboa-mcp/cookbooks.json
2. Add divergence entries to ~/mine/jerboa-mcp/divergence.json for any new hallucinations caught
3. Rebuild data: build_cpt_corpus_v2.py && build_sft_v3.py && build_dpo_pairs_v3.py
4. Bump version suffix on output JSONLs and adapter dirs
5. Re-run the pipeline (python3 v6_runpod.py auto, or the individual v6_runpod.py stages)
6. Eval gate: trained must beat base on both holdout and similarity
7. Convert + push: upload_hf.sh (GGUF/Ollama push is retired — build_ollama_gguf.sh is disabled)