Run RL with verifiable rewards using Training Hub
Reinforcement learning with verifiable rewards (RLVR) trains a language model with a
reward that can be checked by a program. For example, a math verifier can return 1 when
the final answer is equivalent to the expected answer and 0 otherwise. Because the
reward does not require a human label or a learned reward model, it is reproducible and
easy to validate.
This guide runs Group Relative Policy Optimization (GRPO) through Training Hub's
lora_grpo API. The verl backend uses vLLM for rollouts and FSDP for distributed
training. Set lora_r=0 when you want full-parameter GRPO; the example uses LoRA so it
can run with a smaller GPU footprint.
TOC
RequirementsPrepare verifiable-reward dataStart a GRPO runMonitor and validate the resultUse a custom verifierTroubleshootingRequirements
- An Alauda AI Workbench with an NVIDIA GPU. Use multiple GPUs for the
verlbackend. - Internet access, or an internal mirror for PyPI and the model repository.
- A base causal language model that supports the Transformers chat template.
- Training Hub 0.11 or later. The
verlbackend requires thegrpo-verlextra.
Install the dependencies in the Workbench terminal:
The extra installs Training Hub's GRPO dependencies and verl. The backend also needs a
working CUDA, PyTorch, and vLLM environment; use a Workbench image that matches the GPU
driver and CUDA version.
Prepare verifiable-reward data
Create a JSONL file with one math problem per line. The problem is sent to the model,
and answer is passed to the verifier as ground truth:
Keep a separate validation file when you evaluate a run, and make sure answers are
unambiguous. The built-in math reward extracts the final answer (prefer \\boxed{...})
and uses math_verify for semantic comparison, with a normalized-string fallback.
Start a GRPO run
Save the following as run_rlvr.py in the Workbench. Use a small model and a small
n_train value for a smoke test before increasing the dataset and iteration count.
Run it after activating the environment:
prompt_batch_size * group_size must be divisible by n_gpus * 4 with the default
verl agent-worker configuration. If the run reports a divisibility error, increase
prompt_batch_size to the value suggested in the error. The backend writes prepared
Parquet data under ckpt_output_dir/verl_data and checkpoints under
ckpt_output_dir/checkpoints.
Monitor and validate the result
The training logs include the mean reward and the fraction of rollouts with a full match.
Check the generated training_metrics.jsonl and the checkpoint directory before using a
checkpoint for inference:
Evaluate on problems that were not used for training. A high training reward can mean that the policy memorized the examples or learned to exploit the verifier. Include format checks and adversarial cases in the evaluation set, and inspect responses for the required final-answer format.
Use a custom verifier
For domains such as code execution or structured tool calls, provide a Python reward
function. The function must be importable and use the verl reward signature:
Put the function in a .py file and pass it to lora_grpo:
The verifier should be deterministic, bounded, and independent of the model's training process. Return partial rewards only when they represent a meaningful, testable property; otherwise a binary pass/fail reward is easier to interpret.
Troubleshooting
- CUDA or vLLM import errors: use a Workbench image with a CUDA and PyTorch version
supported by the installed
verland vLLM packages, then reinstall the extra in the activated environment. - Out-of-memory during rollout: lower
gpu_memory_utilization,max_tokens, orgroup_size; useparam_offload=Truefor the actor when necessary. - No learning signal: inspect sample responses and the verifier directly. A verifier
that always returns
0or always returns1cannot train a useful policy. - Checkpoint resume: rerun with the same
ckpt_output_dir; theverlbackend uses its saved checkpoints to resume automatically.
See Fine-tuning LLMs with Training Hub for SFT, OSFT, QLoRA, and continued pre-training workflows.