|
41417
|
Renma by @Kaorii | Stardance
|
|
0
|
12.0
|
0
|
[]
|
[{"post_id": 38098, "ts": "202 [{"post_id": 38098, "ts": "2026-07-31T09:49:12Z", "hours": 4.0, "text": "Devlog 2026-07-31 — Ported training to actual hardware, kicked off full run The training pipeline (README, configs/qlora.yaml ) was written for a 16 GB AMD RX 6800 XT (ROCm, gfx1030). This machine has no such card — its only GPUs are an NVIDIA RTX 5060 Laptop (8 GB, CUDA) and a weak AMD iGPU. The config’s RDNA2-only choices were actively hurting: fp16-only, 4-bit disabled, and flash-linear-attention never installed. Changes: Installed CUDA torch ( 2.13.0+cu130 ) instead of the ROCm wheel. Installed flash-linear-attention — enables the Gated-DeltaNet fast path transformers otherwise silently falls back from (up to ~18x slower per the README’s own measurement). Added to pyproject.toml . configs/qlora.yaml : re-enabled quantization.load_in_4bit and switched compute_dtype to bfloat16 . Both were disabled specifically for RDNA2’s missing bf16-dot instruction, which doesn’t apply on NVIDIA. Drops base weights ~8.1 GB → ~2.6 GB. 00_check_env.py : was ROCm-only and hard-failed a working CUDA setup ( /dev/kfd checks, “torch is a ROCm build”). Branched it so AMD-only gates no longer block a valid NVIDIA run; VRAM threshold now reads the actual load_in_4bit setting instead of a hardcoded 15 GB. max_seq_length : 2048 → 1024. Smoke run measured 6.64/7.5 GB peak VRAM at 1024 — 2048 would very likely OOM mid-run on this 8 GB card. Smoke run (20 steps): eval_loss 0.966 → 0.741, eval accuracy 0.756 → 0.793. Real learning signal, despite the script’s own crude first/last train-loss check flagging “NOT DECREASING” (too few steps, too noisy to read that way). Full run started ~16:00, effective batch 16, ~1288 steps, ~27s/step → ~9.5h ETA. Checkpoints every 50 steps to training/out/renma-qlora . Known non-issue: TRL logs “Mismatch between tokenized prompt and the start of tokenized prompt+completion” on essentially every example. Expected — the dataset intentionally ends prompts mid-template (dangling <think> , see commit e04b564 ), so the prompt/completion split doesn’t land on a token boundary. Cosmetic, not a masking bug (eval loss/accuracy both improve cleanly through it)."}, {"post_id": 37026, "ts": "2026-07-29T14:32:58Z", "hours": 8.0, "text": "Renma Devlog #1 Dataset built, smoke test passed, full QLoRA run not started. Dataset: 12,000 examples (11,760 train / 240 val) from MathNet (18,823 kept of 27,817) + Hendrycks MATH (2,354 kept of 7,500, tiers 1–2 only), 149 dupes removed. Tier mix: 1—2.5%, 2—8.7%, 3—43.9%, 4—36.1%, 5—8.8%. Open question: tier 1 is thin (2.5%) — decide before full run whether to upsample or accept the skew. LoRA targeting: Qwen3.5-4B is hybrid — 24/32 layers are Gated-DeltaNet ( in_proj_qkv / in_proj_z / out_proj ), only 8 are standard attention ( q/k/v/o_proj ). Targeting just the usual four names would silently train ¼ of the network with no error. qlora.yaml now targets both. Still need to verify empirically via trainable param count after get_peft_model() . ROCm/bitsandbytes: gfx1030 + rocm70 build segfaults with no traceback (no gfx1030 code objects despite support matrix). Bisected to rocm714 build working — now forced via BNB_ROCM_VERSION=714 in rocm.py . Also: flash-linear-attention / causal_conv1d are CUDA-only, so Gated-DeltaNet falls back to torch_chunk_gated_delta_rule (correct, 2–3x slower). Next: Run full QLoRA training."}]...
|
Title: Renma by @Kaorii | Stardance
Devlogs:
Renm Title: Renma by @Kaorii | Stardance
Devlogs:
Renma Devlog #1 Dataset built, smoke test passed, full QLoRA run not started. Dataset: 12,000 examples (11,760 train / 240 val) from MathNet (18,823 kept of 27,817) + Hendrycks MATH (2,354 kept of 7,500, tiers 1–2 only), 149 dupes removed. Tier mix: 1—2.5%, 2—8.7%, 3—43.9%, 4—36.1%, 5—8.8%. Open question: tier 1 is thin (2.5%) — decide before full run whether to upsample or accept the skew. LoRA targeting: Qwen3.5-4B is hybrid — 24/32 layers are Gated-DeltaNet ( in_proj_qkv / in_proj_z / out_proj ), only 8 are standard attention ( q/k/v/o_proj ). Targeting just the usual four names would silently train ¼ of the network with no error. qlora.yaml now targets both. Still need to verify empirically via trainable param count after get_peft_model() . ROCm/bitsandbytes: gfx1030 + rocm70 build segfaults with no traceback (no gfx1030 code objects despite support matrix). Bisected to rocm714 build working — now forced via BNB_ROCM_VERSION=714 in rocm.py . Also: flash-linear-attention / causal_conv1d are CUDA-only, so Gated-DeltaNet falls back to torch_chunk_gated_delta_rule (correct, 2–3x slower). Next: Run full QLoRA training.
Devlog 2026-07-31 — Ported training to actual hardware, kicked off full run The training pipeline (README, configs/qlora.yaml ) was written for a 16 GB AMD RX 6800 XT (ROCm, gfx1030). This machine has no such card — its only GPUs are an NVIDIA RTX 5060 Laptop (8 GB, CUDA) and a weak AMD iGPU. The config’s RDNA2-only choices were actively hurting: fp16-only, 4-bit disabled, and flash-linear-attention never installed. Changes: Installed CUDA torch ( 2.13.0+cu130 ) instead of the ROCm wheel. Installed flash-linear-attention — enables the Gated-DeltaNet fast path transformers otherwise silently falls back from (up to ~18x slower per the README’s own measurement). Added to pyproject.toml . configs/qlora.yaml : re-enabled quantization.load_in_4bit and switched compute_dtype to bfloat16 . Both were disabled specifically for RDNA2’s missing bf16-dot instruction, which doesn’t apply on NVIDIA. Drops base weights ~8.1 GB → ~2.6 GB. 00_check_env.py : was ROCm-only and hard-failed a working CUDA setup ( /dev/kfd checks, “torch is a ROCm build”). Branched it so AMD-only gates no longer block a valid NVIDIA run; VRAM threshold now reads the actual load_in_4bit setting instead of a hardcoded 15 GB. max_seq_length : 2048 → 1024. Smoke run measured 6.64/7.5 GB peak VRAM at 1024 — 2048 would very likely OOM mid-run on this 8 GB card. Smoke run (20 steps): eval_loss 0.966 → 0.741, eval accuracy 0.756 → 0.793. Real learning signal, despite the script’s own crude first/last train-loss check flagging “NOT DECREASING” (too few steps, too noisy to read that way). Full run started ~16:00, effective batch 16, ~1288 steps, ~27s/step → ~9.5h ETA. Checkpoints every 50 steps to training/out/renma-qlora . Known non-issue: TRL logs “Mismatch between tokenized prompt and the start of tokenized prompt+completion” on essentially every example. Expected — the dataset intentionally ends prompts mid-template (dangling <think> , see commit e04b564 ), so the prompt/completion split doesn’t land on a token boundary. Cosmetic, not a masking bug (eval loss/accuracy both improve cleanly through it)....
|
1792907317.8155088
|
1793771317.8155088
|