Align a model with DPO preference data
Make your assistant prefer your good answers over your bad ones with Direct Preference Optimization on Llama-3.1-8B.
Dataset format: JSONL (preference) — {prompt, chosen, rejected}
The flow
An interactive miniature of the actual training screen — step through it.
New training job
Fine-tune an open model on your data. GPU auto-selected by model size.
Detected: JSONL (instruct) · 52,002 rows · validated
Summary
Llama, Qwen3, Gemma, Mistral… VRAM estimated per method, GPU auto-selected.
Step by step
- 1
Prepare your dataset
Format ~200–2,000 examples as (prompt, chosen, rejected) preference triples — start from a SFT adapter. Quality over quantity — a few hundred clean rows beats tens of thousands of noisy ones. Upload it, or point at a HuggingFace dataset.
- 2
Pick the base model
Choose the recommended base below (or any HF repo id). The dashboard auto-selects a right-sized GPU; you can override it.
- 3
Configure the fine-tune
DPO needs a reference; train on top of your SFT/LoRA adapter. Beta 0.1, low LR (5e-6), 1–2 epochs to avoid over-optimizing.
- 4
Train
Fast fine-tuning finishes in minutes. You get a standard safetensors adapter and a fit-quality receipt.
- 5
Deploy & serve
One click deploys your adapter onto a shared base-model GPU and gives you an OpenAI-compatible endpoint. You pay per token, not per idle GPU-hour. Swap one base_url and you're live.
Build the model alignment model
Standard safetensors out, OpenAI-compatible endpoint in. Export anytime — your weights are yours.