Decoding DeepSeek R1's Advanced Reasoning Capabilities

DeepSeek-R1’s advanced reasoning capabilities have made it the new leader in the generative LLM field. It has caused a stir in the AI industry, with reports of Nvidia’s $600 billion loss post-launch. But what makes DeepSeek-R1 so famous overnight? In this article, we’ll explore why DeepSeek-R1 is gaining so much attention, delve into its groundbreaking capabilities, and analyze how its reasoning powers are reshaping real-world applications. Stay tuned as we break down the model’s performance through a detailed, structured analysis.

Learning Objectives

• Understand DeepSeek-R1’s advanced reasoning capabilities and its impact on the LLM landscape.
• Learn how Group Relative Policy Optimization (GRPO) enhances reinforcement learning without a Critic model.
• Explore the differences between DeepSeek-R1-Zero and DeepSeek-R1 in terms of training and performance.
• Analyze the evaluation metrics and benchmarks that showcase DeepSeek-R1’s superiority in reasoning tasks.
• Discover how DeepSeek-R1 optimizes STEM and coding tasks with scalable, high-throughput AI models.

This article was published as a part of theData Science Blogathon.

Table of contents

• What is Deepseek-R1?
• What is Group Relative Policy Optimization (GRPO)?
• Training Process and Optimization in DeepSeek-R1-Zero
• How does the GRPO Work?
• Enhancing Reasoning and General Capabilities in DeepSeek-R1
• Evaluation of DeepSeek-R1
• Evaluating Reasoning Capabilities of DeepSeek-R1-7B
• Advanced Reasoning and Problem-Solving Scenario
• Conclusion
• Frequently Asked Questions

What is Deepseek-R1?

In simple words, DeepSeek-R1 is a cutting-edge language model series developed by DeepSeek, established in 2023 by Liang Wenfeng. It achieved advanced reasoning capabilities in LLMs through reinforcement learning(RL). There are two variants:

DeepSeek-R1-Zero

It is trained purely via RL on the base model without supervised fine-tuned (SFT), and it autonomously develops advanced reasoning behavior like self-verification and multi-step reflection, achieving 71% accuracy on the AIME 2024 benchmark

DeepSeek-R1

It was enhanced with cold-start data and multi-stage training (RL SFT), it addresses readability issues and outperforms OpenAI’s o1 on tasks like MATH-500 (97.3% accuracy) and coding challenges (Codeforces rating 2029)

DeepSeek uses Group Relative Policy Optimization(GRPO), an RL technique that does not use the Critic model and saves RL’s training costs. GRPO optimizes policies by grouping outputs and normalizing rewards, eliminating the need for the Critic models.

The project also distills its reasoning patterns into smaller models (1.5B-70B), enabling efficient deployment. According to the benchmark It’s 7B model surpasses GPT-4o.

DeepSeek-R1 Paper here.

Comparison Chart

Model	GPQA	LiveCode	Diamond Bench	CodeForces pass@1 cons@64	CodeForces pass@1	Rating
OpenAI-01-mini	63.6	80.0	90.0	60.0	53.8	1820
OpenAI-01-0912	74.4	83.3	94.8	77.3	63.4	1843
DeepSeek-R1-Zero	71.0	86.7	95.9	73.3	50.0	1444

Accuracy Plot of Deepseek-R1-Zero on AIME Dataset

DeepSeek open-sourced the models, training pipelines, and benchmarks aim to democratize RL-driven reasoning research, offering scalable solutions for STEM, coding, and knowledge-intensive tasks. DeepSeek-R1 directs a path to the new era of low-cost, high-throughput SLMs and LLMs.

What is Group Relative Policy Optimization (GRPO)?

Before going into the cutting-edge GRPO, let’s surf on some basics of Reinforcement Learning(RL).

Reinforcement Learning is the interaction between the Agent and Environment. During training, the agent takes actions so that it maximizes the cumulative rewards. Think about a bot playing Chess or a Robot on a factory floor trying to do tasks with actual items.

The agent is learning by doing. It gets a reward when it does things right; otherwise, it gets negative. By doing these repetitive trials, it will be on a journey to find the optimal strategy to adapt to the unknown environment.

Here is the simple diagram of Reinforcement Learning, It has 3 components:

Core RL Loop

• Agent which takes actions based on the learned policy.
• Action is the decision made by the agent at a given state.
• The environment is the external system (game, workshop floor, flying drone, etc) where the agent operates and learns by interacting.
• The environment provides feedback to the agent in the form of new state and rewards.

Agent Components

• Value function estimates how good a particular state or action is in terms of long-term rewards
• Policy is a strategy that defines the agent’s action selection.
• The value function informs the policy by helping it improve decision-making
• The policy guides (Guides Relationship) the agent in choosing actions in the RL Loops

Learning Elements

• Experience, here the agent collects transactions while interacting with the environment.
• Optimization or Policy updates use the experience to refine the policy and important decision-making.

Training Process and Optimization in DeepSeek-R1-Zero

The experience gathered is used to update the policy through optimization. The value function provides insights to refine the policy. The policy guides the agent, which interacts with the environment to collect new experiences and the cycle goes on until the agent learns the optimum strategy or improves to adapt to the environment.

In the training of DeepSeek-R1-Zero, they use Group Relative Policy optimization or GRPO, it eliminate the Critic Model and lowers the training cost.

As for my understanding of the DeepSeek-R1 Research Paper, here is the schematic training process of the DeepSeek-R1-Zero and DeepSeek-R1 models.

Tentative DeepSeek-R1-Zero and R1 Training Diagram

How does the GRPO Work?

For each question q, GRPO samples a group of output {o1, o2, o2..} from the old policy and optimizes the policy model by maximizing the below objective:

Here epsilon and beta are hyper-parameters, and A_i is the advantage computed using a group of rewards {r1, r2, r3…rG} corresponding to the output within each group.

Advantage Calculation

In the Advantage calculation, Normalize rewards within group outputs, r_i is the reward for output I and r_group is the rewards of all output in the group.

To maximize the clipped policy updates with KL penalty,

Kullback-Leibler Divergence

The KL Divergence also known as Relative Entropy is a statistical distance function, that measures the difference between the models’s probability distribution (Q) and true probability distribution (P).

For more KL-Divergence

The below equation is the mathematical form of KL-Divergence:

Relative entropy or KL distance is always a non-negative real number. It has the lowest value of 0 if and only if the Q and P are identical. That means both the Model Probability distribution(Q) and True Probability distribution (P) overlap or a perfect system.

Example of KL Divergence

Here are simple examples to showcase KL divergence,

We will use the entropy function from the Scipy Statistical package, It will calculate the relative entropy between two distributions.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import entropy

# Define two probability distributions P and Q
x = np.linspace(-3, 3, 100)
P = np.exp(-(x**2))  # Gaussian-like distribution
Q = np.exp(-((x - 1) ** 2))  # Shifted Gaussian

# Normalize to ensure they sum to 1
P /= P.sum()
Q /= Q.sum()

# Compute KL divergence
kl_div = entropy(P, Q)

Our P and Q as Gaussian-like and shifted Gaussian distribution respectively.

plt.style.use("ggplot")
plt.figure(figsize=(12, 8))
plt.plot(x, P, label="P (Original)", line, color="blue")
plt.plot(x, Q, label="Q (Shifted)", line, color="red")
plt.fill_between(x, P, Q, color="yellow", alpha=0.3, label="Difference")
plt.title(f"KL Divergence: {kl_div:.4f}")
plt.xlabel("x")
plt.ylabel("Probability Density")
plt.legend()
plt.show()

The yellow portion is the KL difference between P and Q.

In the GRPO equation, GRPO samples a group of outputs for each query and computes advantages relative to the group’s mean and standard deviation. This avoids training a separate critic model. The objective includes a clipped ratio and KL penalty to stay close to the reference policy.

The ratio part is the probability ratio of the new and old policy.Clip(ratio) is bound between 1-epsilon and 1 epsilon.

The conversation process between User and Assistant

The user asks a question, and the model or assistant solves it by first thinking about the reasoning process and then responding to the user.

The reasoning and answer are enclosed in the below diagram.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import entropy

The Self-Evolution Process of DeepSeek-R1-Zero demonstrates how Reinforcement Learning can improve the model’s reasoning capabilities autonomously. The chart shows how the model’s reasoning capabilities for handling complex reasoning tasks evolve.

Enhancing Reasoning and General Capabilities in DeepSeek-R1

DeepSeek-R1, answers two significant questions that arise after promising results of the Zero model.

• Can reasoning performance be further improved?
• How can we train a user-friendly model that not only produces a clear and coherent Chain Of Thought (CoT) but also demonstrates strong general capabilities?

The DeepSeek-R1 uses Cold-Start Data in a format where the developer collects thousands of cold-start data to fine-tune the DeepSeek-V3-Base as a starting point of RL.

These data have two important advantages compared to DeepSeek-R1-zero.

• Readability: A key limitation of the Zero model is that its content is not suitable for reading. The responses are mixed with many languages, and not well formatted to highlight answers for users.
• Potential: Expert lead designing the pattern for cold-start data to help better performance against DeepSeek-R1-Zero.

Evaluation of DeepSeek-R1

According to the DeepSeek-R1 paper, They (the developer)set the maximum generation length to 32768 tokens for the models. They found long output reasoning model result in higher repetition rates with greedy decoding and significant variability. Therefore, they use pass@k evaluation, It use a sampling temperature of 0.6 and a top-p value of 0.95 to generate k numbers response for each question.

Pass@1 is then calculated as:

Here, P_i denotes the correctness of the i-th response, according to the research paper this method ensures more reliable performance estimates.

We can see that the education-oriented knowledge benchmarks such as MMLU, MMLU-Pro, GPQA Diamond, and DeepSeek-R1 perform better compared to DeepSeek-V3. It has primarily enhanced accuracy in STEM-related questions. DeepSeek-R1 also delivers great results on IF-Eval, a benchmark data designed to assess the model’s ability to follow format instructions.

Enough maths and theoretical understanding has been done, which I wish significantly boost your overall knowledge of Reinforcement Learning and its cutting-edge application on DeepSeek-R1 model development. Now we will get our hands on DeepSeek-R1 using Ollama and taste the newly minted LLM.

Evaluating Reasoning Capabilities of DeepSeek-R1-7B

The evaluation of DeepSeek-R1-7B focuses on its enhanced reasoning capabilities, particularly its performance in complex problem-solving scenarios. By analyzing key benchmarks, this assessment provides insights into how effectively the model handles intricate reasoning tasks compared to its predecessors.

What We Want to Achieve

• Evaluate DeepSeek-R1’s reasoning capabilities across different cognitive domains
• Identify strengths and limitations in specific reasoning tasks
• Understand the model’s potential real-world applications

Setup the Environment

• Install Ollama fromhere
• After installing it to your system open your terminal and type the below command, it will download and start the DeepSeek-R1 7B model.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import entropy

Now I put a Linear inequality question from NCERT

Q.Solve 4x 3 < 6x 7

and the response is:

Which is accurate according to the book.

Amazing!!

Now will set up a testing environment using Llamaindex which will be a more prominent way to do this.

Setup Testing Environment

# Define two probability distributions P and Q
x = np.linspace(-3, 3, 100)
P = np.exp(-(x**2))  # Gaussian-like distribution
Q = np.exp(-((x - 1) ** 2))  # Shifted Gaussian

# Normalize to ensure they sum to 1
P /= P.sum()
Q /= Q.sum()

# Compute KL divergence
kl_div = entropy(P, Q)

Now we install the necessary packages

Install Packages

plt.style.use("ggplot")
plt.figure(figsize=(12, 8))
plt.plot(x, P, label="P (Original)", line, color="blue")
plt.plot(x, Q, label="Q (Shifted)", line, color="red")
plt.fill_between(x, P, Q, color="yellow", alpha=0.3, label="Difference")
plt.title(f"KL Divergence: {kl_div:.4f}")
plt.xlabel("x")
plt.ylabel("Probability Density")
plt.legend()
plt.show()

Now Open VScode and create a Jupyter Notebook name prompt_analysis.ipynb root of the project folder.

Import Libraries

<think> reasoning process</think>
<answer> answer here </answer>

USER: Prompt
Assistant: Answer



</h4>
<p>You must stay running ollama deepseek-r1:7b on your terminal.</p>



<p>Now, start with the mathematical problem</p>



<p><b>Imporant:</b> OUTPUT will be very long so the output in this blog will be abridged, For full output you must see the blog’s code repository here.</p>
<h2 id="Advanced-Reasoning-and-Problem-Solving-Scenario">Advanced Reasoning and Problem-Solving Scenario</h2>



<p>This section explores complex problem-solving tasks that require a deep understanding of various reasoning techniques, from mathematical calculations to ethical dilemmas. By engaging with these scenarios, you will enhance your ability to think critically, analyze data, and draw logical conclusions across diverse contexts.</p>



<h3 id="Mathematical-Problem-Discount-and-Loyalty-Card-Calculation">Mathematical Problem: Discount and Loyalty Card Calculation</h3>



<p>A store offers a 20% discount on all items. After applying the discount, there’s an additional 10% off for loyalty card members. If an item originally costs $150, what is the final price for a loyalty card member? Show your step-by-step calculation and explain your reasoning.</p>



<div class="code" style="position:relative; padding:0px; margin:0px;"><div class="code" style="position:relative; padding:0px; margin:0px;"><div class="code" style="position:relative; padding:0px; margin:0px;"><div class="code" style="position:relative; padding:0px; margin:0px;"><div class="code" style="position:relative; padding:0px; margin:0px;"><pre class="brush:php;toolbar:false">import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import entropy

Copy after login

Copy after login