DPO vs RLHF: which should you use to align a model?
Start with DPO: same preference labels, no reward model or sampling loop. Switch to RLHF with PPO when you need online exploration. Full cost math.
reinforcement-learning
4 posts with the tag “reinforcement-learning”
What compute do you need for RL training of LLM agents?
Agent RL needs GPUs for generation and training plus one isolated sandbox per live rollout. Cost math showing GPU wait time, not sandboxes, drives the bill.
PPO vs DPO vs GRPO: which RL method should you use?
Pick by the signal you have: preference pairs mean DPO, a checker you can run in code means GRPO, a learned reward model means PPO. Memory and cost math inside.
What is GRPO, and how do you run GRPO training on GPUs?
GRPO trains an LLM by scoring groups of sampled answers against each other, with no critic model. How it works, memory math for 8B, and how to run it.