Toward DeepSeek-R1: Building Thinking Models Using GRPO

---

---

Toward DeepSeek-R1: Building Thinking Models Using GRPO

The journey from DeepSeek‑V3 to DeepSeek‑R1 marks a pivotal shift in how we think about language models: not just as pattern recognizers, but as reasoning agents. While V3 focused on architectural innovations (e.g., multi-head latent attention (MLA), mixture of experts (MoE), and multi‑token prediction (MTP)), R1 introduced a new dimension: the ability to think through problems step by step. At the heart of this breakthrough lies Group Relative Policy Optimization (GRPO), a reinforcement learning technique designed to refine models beyond next‑token prediction, enabling them to tackle complex reasoning tasks with verifiable outcomes.

Unlike traditional fine‑tuning methods, GRPO leverages the idea of relative advantage within groups of solutions. Instead of rewarding a model for producing a single correct answer, GRPO evaluates multiple attempts, compares their relative quality, and uses this structured feedback to guide learning. This makes it particularly powerful for domains (e.g., mathematical reasoning), where correctness can be objectively verified and where diverse solution paths provide rich signals for improvement. In practice, GRPO can further train small language models (e.g., Qwen3) using verifiable rewards for answer correctness and output format.

In this bonus lesson, we will explore how GRPO works, why it excels in reasoning tasks, and how to apply it to post‑train smaller models. We will walk through the core GRPO objective, its clipped surrogate function, and Kullback-Leibler (KL) divergence regularization, before diving into a hands-on demonstration of training Qwen3 with verifiable reward functions. By the end, you will see how GRPO bridges the gap between raw generative ability and structured reasoning, laying the foundation for building true thinking models.

To learn how to post-train Qwen3 for mathematical reasoning using GRPO, just keep reading.

---

How Group Relative Policy Optimization (GRPO) Works

Group Relative Policy Optimization (GRPO) represents a fundamental departure from traditional policy optimization methods by eliminating the need for a separate value function while maintaining stable and efficient training dynamics (Figure 1). The mathematical foundation of GRPO is built on the principle of group-relative advantage estimation, which provides both computational efficiency and variance reduction.

---

Core GRPO Objective

The GRPO objective function can be formally expressed as (Figure 2):

where:

• : represents the input prompt or question
• : represents a group of outputs/completions generated for the same prompt
• : is the clipped advantage function
• : is the KL divergence penalty term
• : is the regularization coefficient

---

Group Relative Advantage Estimation

The key innovation of GRPO lies in its advantage estimation mechanism. Instead of relying on a learned value function, as Proximal Policy Optimization (PPO) typically does, GRPO computes advantages relative to the group of sampled outputs (Figure 3):

where:

• : is the reward for output
• : is the group mean reward
• : is the group standard deviation

This normalization serves multiple purposes. It provides a natural baseline by comparing each output against its peers, reduces variance by localizing comparisons within each group, and eliminates the need for a separate value function approximator.

---

Clipped Surrogate Objective

Following PPO’s approach, GRPO employs a clipped surrogate objective to maintain training stability:

where constrains to the range , preventing excessively large policy updates that could destabilize training.

---

KL Divergence Regularization

The KL (Kullback-Leibler) divergence term in the GRPO objective serves to keep the updated policy close to a reference policy , typically the initial supervised fine-tuned model:

This regularization term is particularly important in language model training to prevent catastrophic forgetting of the model’s pre-trained capabilities.

---

Why GRPO Excels in Mathematical Reasoning

Group Relative Policy Optimization (GRPO) has rapidly established itself as a leading fine-tuning approach for mathematical reasoning in large language models (LLMs), owing to several structural features that make it especially well-suited to this domain.

Recent large-scale studies (e.g., those involving the DeepSeekMath and DeepSeek-R1 models) demonstrate that swapping PPO for GRPO in reinforcement learning from human feedback (RLHF) pipelines yields significant improvements on mathematical reasoning benchmarks (e.g., GSM8K, MATH datasets).

For example, DeepSeekMath training with GRPO outperformed PPO-based methods on math reasoning accuracy while reducing compute costs (Figure 4).

---

Intrinsic Support for Verifiable Rewards

Mathematics problem-solving offers an unusually clear and verifiable reward mechanism: answers are either correct or incorrect, and partial credit can be precisely defined. GRPO capitalizes on this by scoring multiple solutions generated for each problem using a well-defined reward model, often simply marking correct steps or final answers. This direct and reliable reward assignment enhances the signal-to-noise ratio of model updates, as each policy improvement step is based on unambiguous performance feedback.

In contrast, open-ended tasks (e.g., creative writing or dialogue) often rely on subjective or noisy reward models, making stable policy optimization far more difficult. GRPO’s per-group advantage estimation is highly compatible with settings where reward computation is transparent and objective, as in math (Figure 5).

---

Efficient Learning from Diverse Solution Attempts

Mathematical reasoning often requires the model to try multiple solution paths before arriving at a correct answer or elegant derivation. GRPO’s core mechanism (i.e., group sampling) directly supports this, as multiple outputs (“grouped rollouts”) are generated for each problem instance.

By comparing each solution’s reward to the group mean, GRPO enables the model to reinforce approaches that outperform common errors or naive guesses in the group. This fosters systematic exploration of problem-solving strategies and mitigates overfitting to spurious patterns that might arise from individual samples.

In practical terms, this approach enables rapid discrimination between valuable reasoning sequences and ineffective or “shortcut” answers commonly found in early-stage LLM outputs.

---

Post-Training Qwen3 for Math Reasoning with GRPO

In this section, we will see how to post-train small language models (e.g., Qwen3) using GRPO to improve their mathematical reasoning capabilities.

For this implementation, we will use the AI-MO/NuminaMath-TIR dataset, which contains approximately 70,000 problems from the NuminaMath-CoT dataset, focusing on those with numerical outputs, most of which are integers.

The original NuminaMath-CoT dataset (containing 860,000 pairs of competition math problems and solutions) is designed to enhance the mathematical reasoning capabilities of large language models (LLMs). The NuminaMath dataset includes problems ranging from high school level to advanced competition level, all meticulously annotated with accompanying chain-of-thought traces (Table 1).

We will start by installing the necessary libraries.

pip install trl==0.14.0 peft math_verify
pip install -U datasets
pip install "transformers==4.51.3"
pip install -U "torchao>=0.16.0"

---

Loading and Exploring the NuminaMath Dataset

After setting up the environment, we will begin by loading our dataset and visualizing a few examples.

from datasets import load_dataset

dataset_id = "AI-MO/NuminaMath-TIR"
train_set, test_set = load_dataset(dataset_id, split=['train[:1%]', 'test[:10%]'])

print("Train set: ", len(train_set))
print("Test set: ", len(test_set))
from IPython.display import Markdown, display

display(Markdown(r"**Problem Statement**: " + train_set[0]["problem"]))
display(Markdown(train_set[1]["solution"]))

Output:

On Line 1, we import the load_dataset function from the datasets library, which enables access to a specific dataset by its ID. On Line 3, we define the dataset identifier. Then on Line 4, we use load_dataset to fetch small subsets of the train (1%) and test (10%) splits from the NuminaMath-TIR collection.

On Lines 6 and 7, we print the sizes of each subset for confirmation. Finally, on Lines 8-11, we use IPython’s Markdown display to render the problem statement from the 1st training sample and the solution from the 2nd training sample directly in the notebook (Figure 6).

Next, we convert our train and test data problem statements into conversational-style prompts.

SYSTEM_PROMPT = """
## Instruction:
Given a Problem statement.
1. Provide reasoning and final answer in following JSON format:
```json
{
  "THINKING": "This is my thinking",
  "ANSWER": "\( \\boxed{312} \)"
}
```
"""

def make_conversation(example):
    problem = "## Problem:\n" + example["problem"] + "\n## Solution:\n"
   
    return {
        "prompt": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": problem},
        ],
    }

train_dataset = train_set.map(make_conversation)
test_dataset = test_set.map(make_conversation)

On Lines 24-32, we define a helper function make_conversation that transforms each dataset example into a structured dictionary with a single "prompt" entry. This includes the math problem as user input, formatted for chat-style training.

On Lines 34 and 35, we apply this mapping function to the train and test datasets, thereby converting the raw problem data into conversational form.

---

Loading Qwen3 and Configuring LoRA

Now, we start loading our model and defining our low-rank adaptation (LoRA) config for parameter-efficient fine-tuning (PEFT) (rather than fully finetuning all model parameters, which is costly) of Qwen3.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3-0.6B"

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

processor = AutoTokenizer.from_pretrained(model_id, padding_side="left")

# Qwen3 is a hybrid thinking model. Its chat template only injects the empty
# <think></think> block when enable_thinking=False is passed explicitly, and
# GRPOTrainer calls apply_chat_template() internally without that argument,
# so we make non-thinking the default in the template itself.
if "enable_thinking is defined and enable_thinking is false" in (processor.chat_template or ""):
    processor.chat_template = processor.chat_template.replace(
        "{%- if enable_thinking is defined and enable_thinking is false %}",
        "{%- if enable_thinking is not defined or enable_thinking is false %}",
    )

from peft import LoraConfig, get_peft_model

# Configure LoRA
peft_config = LoraConfig(
    r=8,
    lora_alpha=8,
    lora_dropout=0.1,
    target_modules=["q_proj", "v_proj"],
)

# Apply PEFT model adaptation
peft_model = get_peft_model(model, peft_config)

# Print trainable parameters
peft_model.print_trainable_parameters()

Output:

trainable params: 1,146,880 || all params: 597,196,800 || trainable%: 0.1920

On Lines 36 and 37, we import the necessary PyTorch and Hugging Face classes to load a causal language model and its associated tokenizer. On Line 39, we specify the model ID for Qwen3-0.6B. Then, on Lines 42-46, we load the model using AutoModelForCausalLM.from_pretrained() with auto device placement and bfloat16 precision for memory efficiency. On Line 48, we load the corresponding tokenizer, setting padding_side="left" because GRPO generates completions in batches. Immediately afterwards, on Lines 54-58, we patch the chat template so that non-thinking mode is the default. Qwen3 is a hybrid thinking model: its template only injects the empty <think></think> block when enable_thinking=False is passed explicitly, and GRPOTrainer calls apply_chat_template() internally without that argument. Without this patch every rollout would emit a long free-form reasoning trace, be truncated at max_completion_length, and score zero on both reward functions.

On Line 60, we bring in the PEFT utilities for LoRA-based fine-tuning. On Lines 63-68, we set up LoraConfig, tweaking parameters like rank r, scaling factor lora_alpha, dropout rate, and targeting attention projection layers. On Line 71, we adapt the original model using this LoRA configuration to inject low-rank updates into the model weights. Finally, on Line 74, we print the trainable parameter count, giving us insight into how many parameters will be updated during training.

Below, we implement an inference function to generate the solution given a prompt using the above model.

import time

def generate_solution(model, prompt):
  text = processor.apply_chat_template(
    prompt,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
  )
  inputs = processor([text], return_tensors="pt").to(model.device)

  # Generate text without gradients
  start_time = time.time()
  with torch.no_grad():
      output_ids = model.generate(**inputs, max_new_tokens=512, temperature=0.01, top_p=1.0)
  end_time = time.time()

  # Decode and extract model response
  generated_text = processor.decode(output_ids[0], skip_special_tokens=True)

  # Get inference time
  inference_duration = end_time - start_time

  # Get number of generated tokens
  num_input_tokens = inputs['input_ids'].shape[1]
  num_generated_tokens = output_ids.shape[1] - num_input_tokens

  return generated_text, inference_duration, num_generated_tokens

On Line 75, we import the time module to help track inference duration. Then on Line 77, we define generate_solution, a function that takes in a model and a structured prompt, joining the chat entries into a single text string. On Line 84, we tokenize this prompt using processor and move the inputs to the same device as the model for seamless execution.

On Lines 87-90, we measure the time taken to generate a response. We use torch.no_grad() to avoid computing gradients during inference. On Line 93, we decode the generated tokens into readable text, skipping any special tokens.

Finally, on Line 102, we return the generated output, its inference time, and token count for easy evaluation of model performance.

---

Implementing Verifiable Reward Functions

Since we are training our model for mathematical reasoning, we can use verifiable rewards for GRPO. To verify whether or not a generated solution is correct with respect to the original solution, we will use the Hugging Face Math-Verify library.

Math-Verify is a robust mathematical expression evaluation system designed for assessing Large Language Model outputs in mathematical tasks. This evaluator achieves the highest accuracy and the most correct scores compared to existing evaluators on the MATH dataset.

For training, the DeepSeek-R1 authors used an accuracy-based reward model evaluates whether the response is correct, alongside a format-based reward that ensures the model places its reasoning process between <think></think> tags. We can simply define and implement these reward functions as generic Python functions.

First, we implement a format enforcement reward function that ensures that the generation follows a specific JavaScript Object Notation (JSON) format with "THINKING" and "ANSWER" keys for reasoning and answer.

import json
def format_reward(completions, **kwargs):
    """Reward function that checks if the completion has a specific format."""

    completion_contents = [completion[0]["content"] for completion in completions]
    rewards_list = []
    for completion in completion_contents:
      try:
        completion_str = completion.split("```json")[1].split('```')[0]
        completion_json = json.loads(completion_str)
        if "THINKING" in completion_json and "ANSWER" in completion_json:
          rewards_list.append(1.0)
        else:
          rewards_list.append(0.0)
      except:
        rewards_list.append(0.0)
    return rewards_list

On Line 104, we define theformat_reward function, which takes in a list of completions and optional keyword arguments. On Line 107, we extract the "content" field from each completion and store them in completion_contents. We then initialize an empty list rewards_list to hold the reward scores. Moving to Lines 109-112, we loop through each completion string. Inside the try block, we attempt to split the text to isolate the JSON portion between ```json and ```. If successful, we parse it with json.loads. The check on Line 113 ensures that both "THINKING" and "ANSWER" keys exist in the parsed JSON. If they do, we append a reward of 1.0; otherwise, we append 0.0.

In essence, we are rewarding completions that follow the expected structured format with both "THINKING" and "ANSWER" fields, while penalizing those that fail to meet this requirement. This way, we enforce structured reasoning outputs in our model training.

Next, the code below implements the accuracy reward function, which assigns a higher reward if the generated answer is correct.

from math_verify import LatexExtractionConfig, parse, verify
def accuracy_reward(completions, **kwargs):
    """Reward function that checks if the completion is the same as the ground truth."""
    solutions = kwargs['solution']
    rewards = []
    for completion, solution in zip(completions, solutions):
        try:
          content = json.loads(completion[0]["content"].split("```json")[1].split('```')[0])
          gold_parsed = parse(solution, extraction_mode="first_match", extraction_config=[LatexExtractionConfig()])
          answer_parsed = parse(content["ANSWER"], extraction_mode="first_match")
 
          if len(gold_parsed) != 0:
            rewards.append(float(verify(answer_parsed, gold_parsed)))
          else:
              rewards.append(1.0)
        except:
          rewards.append(0.0)
    return rewards

On Line 120, we import 3 components from the math_verify module: LatexExtractionConfig, parse, and verify. These are used to extract and compare mathematical expressions from model completions and ground truth solutions. On Lines 121 and 122, we define the accuracy_reward function, which evaluates how closely each model-generated completion matches the correct answer.

On Lines 123 and 124, we retrieve the list of ground truth solutions from the keyword arguments and initialize an empty rewards list. Then, for each pair of completion and solution, we enter a try block (Line 126) to safely parse and compare the outputs. On Line 127, we extract the JSON portion of the completion and load it. On Lines 128 and 129, we parse both the ground truth solution and the model’s predicted "ANSWER" using the parse function, configured to extract the first matching LaTeX expression.

On Lines 131 and 132, we check if the parsed ground truth (gold_parsed) is non-empty. If it is, we use the verify function to compare it with the parsed answer and append the resulting float score to rewards. If the ground truth is empty, we assign a default reward of 1.0. On Lines 135 and 136, any parsing or verification errors result in a fallback reward of 0.0. Finally, on Line 137, we return the list of reward scores. This function helps us quantify how accurate each model completion is in solving math problems, using structured verification rather than simple string matching.

The above code provides a basic implementation of a verifiable reward function that can be used in GRPO. Now we will test our model (without any GRPO training yet) using the accuracy reward function on a test data example to see how it performs.

example = test_dataset[0]

## Testing Generation
generated_text, inference_duration, num_generated_tokens = generate_solution(model, example['prompt'])
print(generated_text)
print(f"Inference Duration: {inference_duration} seconds")
print(f"Number of Generated Tokens: {num_generated_tokens}")

## Testing Reward Function
solutions = [example['solution']]
completions = [[{"content": generated_text.split('assistant')[1]}]]
print(f"Accuracy Reward: {accuracy_reward(completions, solution=solutions)}")
print(f"Format Reward: {format_reward(completions, solution=solutions)}")

Output:

As can be seen, the Qwen3 model, without any GRPO training, yet fails to answer the questions correctly and rather produces a coherent derivation that still reaches the wrong answer and ignores the required JSON output format (Figure 7). In the next section, we will see if GRPO training can improve the model’s reasoning capabilities on the same example.

---

Training Qwen3 with GRPOTrainer

We have everything in place for our GRPO training: the model, dataset, and reward function. It is time to configure and start our GRPO training. We will utilize the Hugging Face Transformer Reinforcement Learning (TRL) GRPOTrainer and GRPOConfig for quick and clean implementation.

from trl import GRPOConfig, GRPOTrainer

# Configure training arguments using GRPOConfig
training_args = GRPOConfig(
    output_dir="Qwen3-0.6B-Thinking",
    learning_rate=1e-5,
    remove_unused_columns=False, # to access the solution column in accuracy_reward
    gradient_accumulation_steps=16,
    num_train_epochs=1,
    bf16=True,

    # Parameters that control the data preprocessing
    max_completion_length = 128,
    num_generations=4,
    max_prompt_length=512,

    # Parameters related to reporting and saving
    logging_steps=10,
    report_to="none",  # Colab ships wandb; the default prompts for an API key
    save_strategy="steps",
    save_steps=10,
)

trainer = GRPOTrainer(
    model=model,
    reward_funcs=[format_reward, accuracy_reward],
    args=training_args,
    processing_class=processor,  # reuse the non-thinking chat template
    train_dataset=train_dataset.remove_columns(['messages', 'problem'])
)

trainer.train()
trainer.save_model(training_args.output_dir)

On Line 151, we import GRPOConfig and GRPOTrainer from the trl library to set up and run GRPO-style reinforcement learning. Then, on Lines 154-172, we configure the training parameters via GRPOConfig, setting the output directory, learning rate, LoRA compatibility (remove_unused_columns=False), batch accumulation, epochs, and precision type (bf16).

We also define generation controls (e.g., maximum prompt/completion lengths, and the number of completions per sample). Logging and checkpointing settings ensure progress is tracked and saved every few steps.

On Lines 174-180, we instantiate GRPOTrainer using the LoRA-adapted model, our accuracy_reward function, the training arguments, and a filtered dataset (excluding non-essential columns). Finally, on Lines 182 and 183, we launch the training and save the fine-tuned model to the configured directory.

Once training is complete, let us try out our trained model on the same test data example and see how it performs.

example = test_dataset[5]

## Testing Generation
generated_text, inference_duration, num_generated_tokens = generate_solution(model, example['prompt'])
display(Markdown(generated_text))
print(f"Inference Duration: {inference_duration} seconds")
print(f"Number of Generated Tokens: {num_generated_tokens}")

## Testing Reward Function
solutions = [example['solution']]
completions = [[{"content": generated_text}]]
print(f"Accuracy Reward: {accuracy_reward(completions, solution=solutions)}")

Output:

As we see, after GRPO training and with <1000 examples, the model already demonstrates the ability to generate the correct "THINKING" and "ANSWER" tags (Figure 8), even though the solution itself is incorrect. Given the inference time and the number of generated tokens, this approach shows potential benefits.

---

---

Summary

In this lesson, we explored how Group Relative Policy Optimization (GRPO) serves as the foundation for building thinking models in DeepSeek‑R1. We began by breaking down the core GRPO objective, examining how group relative advantage estimation, the clipped surrogate objective, and KL divergence regularization come together to create a stable yet flexible training framework. By understanding these components, we set the stage for appreciating why GRPO is uniquely suited to reasoning tasks compared to traditional reinforcement learning approaches.

We then highlighted why GRPO excels in mathematical reasoning. Unlike conventional methods that reward single correct answers, GRPO thrives on diverse solution attempts and leverages verifiable rewards to guide learning. This intrinsic support for correctness and efficiency allows models to learn from multiple reasoning paths, reinforcing both accuracy and adaptability. Through this lens, we saw how GRPO transforms small language models into structured problem solvers capable of step‑by‑step reasoning.

Finally, we moved into practice by applying GRPO to post‑train Qwen3. We walked through loading and visualizing the dataset, preparing the Qwen3 model, and implementing verifiable reward functions tailored to mathematical tasks. With these foundations, we demonstrated how GRPO training can elevate a generative model into a true thinking model. By the end, we established a clear path from DeepSeek‑V3’s architectural innovations to DeepSeek‑R1’s reasoning breakthroughs, showing how GRPO bridges the gap between generation and structured thought.

---

Citation Information

Mangla, P. “Toward DeepSeek-R1: Building Thinking Models Using GRPO,” PyImageSearch, S. Huot, A. Sharma, and P. Thakur, eds., 2026, https://pyimg.co/70rm1

@incollection{Mangla_2026_deepseek-r1-building-thinking-models-using-grpo,
  author = {Puneet Mangla},
  title = {{Toward DeepSeek-R1: Building Thinking Models Using GRPO}},
  booktitle = {PyImageSearch},
  editor = {Susan Huot and Aditya Sharma and Piyush Thakur},
  year = {2026},
  url = {https://pyimg.co/70rm1},
}

---

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!