Which is the Higher SLM?

The evolution of AI fashions has reached new heights, significantly in small language fashions (SLMs), the place effectivity and efficiency are key. Among the many newest contenders, Phi-4-mini and o1-mini stand out as superior and environment friendly fashions. On this article, we’ll be doing a Phi-4 mini vs o1-mini comparability to take a look at their person expertise, pace, and efficiency on STEM functions and coding duties. We shall be assessing their strengths in programming, debugging, and general effectivity to see which mannequin performs higher. By the top, you’ll have a transparent perspective on which mannequin aligns together with your wants.

What’s Phi-4-mini?

Phi-4-mini is a state-of-the-art SLM designed for high-performance reasoning and coding duties. It strikes a steadiness between effectivity and accuracy, making it a powerful contender in AI-driven functions. The mannequin is designed for high-accuracy textual content technology and complicated reasoning duties whereas being computationally environment friendly, making it well-suited for edge computing environments

Structure Overview

Phi-4-mini is a dense, decoder-only transformer mannequin with 3.8 billion parameters and a 128K token context window. It helps a vocabulary measurement of 200,064 tokens and incorporates Grouped Question Consideration (GQA) to optimize useful resource effectivity whereas sustaining excessive efficiency.

Grouped Question Consideration (GQA) is an environment friendly consideration mechanism that balances the pace of multi-query consideration (MQA) with the standard of multi-head consideration (MHA) by grouping question heads and sharing key/worth heads, enhancing inference pace for language fashions

Key Options

• Shared Enter-Output Embeddings: Reduces reminiscence overhead by reusing embeddings for each enter and output.
• Coaching Information: Skilled on 5 trillion tokens, together with high-quality academic materials, coding examples, and artificial information tailor-made for reasoning.
• Efficiency: Excels in reasoning, arithmetic, coding, and instruction-following, with the flexibility to combine exterior APIs by means of perform calling.

Additionally Learn: Phi-4 vs GPT-4o-mini Face-Off

What’s o1-mini?

o1-mini is a light-weight and cost-efficient SLM geared toward balancing affordability and efficiency. It prioritizes environment friendly processing whereas sustaining an inexpensive degree of accuracy for basic AI functions.

Structure Overview

o1-mini follows an ordinary transformer structure, with fewer parameters than Phi-4-mini (precise measurement undisclosed). It additionally helps a 128K token context window however focuses on cost-effective processing quite than architectural optimizations like GQA.

Additionally Learn: OpenAI’s o1-preview vs o1-mini: A Step Ahead to AGI

Mannequin Comparability: Phi-4-mini vs o1-mini

Phi-4-mini is a robust mannequin designed for duties like reasoning, math, and coding, whereas o1-mini follows an easier design centered on cost-effective coding. The desk beneath highlights their key variations:

Phi-4-mini stands out with superior options like GQA and shared embeddings, making it superior in reasoning, coding, and API integration. In distinction, o1-mini is a lighter, cost-effective various optimized for coding, although it lacks the architectural refinements seen in Phi-4-mini. Selecting between the 2 will depend on whether or not the precedence is excessive accuracy and reasoning energy or affordability and effectivity in particular duties.

Analysis of Reasoning Efficiency

This part appears at how the Phi-4-mini and o3-mini fashions carry out in reasoning in comparison with their bigger fashions. It focuses on how properly they clear up complicated issues and make logical conclusions, highlighting the variations in accuracy, effectivity, and readability between the smaller and bigger fashions.

Phi-4-mini and o1-mini vs Bigger Fashions

The reasoning capabilities of the reasoning-enhanced Phi-4-mini and o1-mini had been evaluated throughout a number of benchmarks, together with AIME 2024, MATH-500, and GPQA Diamond. These benchmarks assess superior mathematical reasoning and basic problem-solving expertise, offering a foundation for comparability in opposition to a number of bigger fashions from DeepSeek, Bespoke, and OpenThinker.

Regardless of having solely 3.8 billion parameters, the reasoning-trained Phi-4-mini demonstrates robust efficiency, surpassing bigger fashions comparable to:

• DeepSeek-R1-Distill-Llama-8B (8B parameters)
• Bespoke-Stratos-7B (7B parameters)
• OpenThinker-7B (7B parameters)

Moreover, it achieves efficiency akin to DeepSeek-R1-Distill-Qwen-7B, a considerably bigger 7B mannequin, additional highlighting its effectivity. Nonetheless, o1-mini, regardless of its undisclosed parameter measurement, leads throughout a number of benchmarks, making it a powerful contender in AI reasoning duties.

Benchmark Comparisons

The efficiency of each fashions, as proven within the offered picture, highlights their competitiveness in opposition to bigger fashions:

• AIME Benchmark:
  • o1-mini scores 63.6, the very best amongst all fashions.
  • Phi-4-mini (reasoning skilled) scores 50.0, a fivefold enchancment over its base model (10.0).
• MATH-500 Benchmark:
  • Phi-4-mini (90.4) barely outperforms o1-mini (90.0), making it extremely efficient for complicated mathematical reasoning duties.
• GPQA Diamond:
  • o1-mini leads with 60.0, showcasing superior basic problem-solving capabilities.
  • Phi-4-mini (49.0) outperforms a number of 7B and 8B fashions, proving its effectivity in reasoning duties.

These outcomes point out that o1-mini dominates usually problem-solving and reasoning, whereas Phi-4-mini (reasoning-trained) excels in mathematical benchmarks regardless of its smaller measurement (3.8B parameters). Each fashions reveal distinctive effectivity, difficult and even outperforming considerably bigger fashions throughout key AI benchmarks.

Phi-4-mini vs o1-mini: Reasoning and Coding Capabilities

Now we are going to evaluate the reasoning and programming capabilities of Phi-4-mini and o1-mini. For that, we’re going to give the identical immediate to each fashions and consider their responses and we shall be utilizing API to load the mannequin. Listed below are the duties we’ll be attempting out on this comparability:

1. Analyzing constructing order relationships
2. Logical reasoning in Math
3. Discovering the Longest Substring

Job 1: Analyzing Constructing Order Relationships

This process requires the mannequin to infer the relative positions of buildings primarily based on the given constraints and establish the center constructing.

Immediate: “There are 5 buildings referred to as V, W, X, Y and Z in a row (not essentially in that order). V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y. Which is the constructing within the center?
Choices:
A) V
B) W
C) X
D) Y”

Enter to o1-mini

from openai import OpenAI
import time
import tiktoken
from IPython.show import show, Markdown
with open("path_to_api_key") as file:
   api_key = file.learn().strip()


task1_start_time = time.time()
shopper = OpenAI(api_key=api_key)
messages = [
   {
       "role": "user",
       "content": """
   There are five buildings called V, W, X, Y and Z in a row (not necessarily in that order).
   V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y.
   Which is the building in the middle?
   Options:
   A) V
   B) W
   C) X
   D) Y
"""
   }
]
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
task1_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for process 1:----------------- ", task1_end_time - task1_start_time)
# Show consequence
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

Enter to Phi 4-mini

from transformers import pipeline
import time
from IPython.show import show, Markdown


from transformers import AutoModelForCausalLM, BitsAndBytesConfig
quantization_config = BitsAndBytesConfig(load_in_8bit=True)


# Load mannequin instantly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct", trust_remote_code=True)
mannequin = AutoModelForCausalLM.from_pretrained("microsoft/Phi-4-mini-instruct", trust_remote_code=True, quantization_config=quantization_config) 


task1_start_time = time.time()
messages = [
   {"role": "system", "content": "You are an expert in solving numerical and general reasoning questions."},
   {"role": "user", "content": """There are five buildings called V, W, X, Y and Z in a row (not necessarily in that order).
   V is to the West of W. Z is to the East of X and the West of V, W is to the West of Y.Which is the building in the middle? Options:
A) V
B) W
C) X
D) Y"""},
]


pipe = pipeline(
   "text-generation",
   mannequin=mannequin,
   tokenizer=tokenizer,
)
generation_args = {
   "max_new_tokens": 1024,
   "return_full_text": False,
   "temperature": 0.0,
   "do_sample": False,
}
output = pipe(messages, **generation_args) #,
task1_end_time = time.time()
print("----------------=Whole Time Taken for process 1:----------------- ", task1_end_time - task1_start_time)
show(Markdown((output[0]['generated_text'])))
pipe = pipeline("text-generation", mannequin="microsoft/Phi-4-mini-instruct", trust_remote_code=True)
pipe(messages)

Response by Phi 4-mini

Comparative Evaluation

o1-mini is best than Phi-4-mini in each pace and accuracy for this process. o1-mini rapidly figures out the proper reply (“V”) with only a few steps, whereas Phi-4-mini takes for much longer as a result of it goes by means of every element step-by-step. Even with all that effort, Phi-4-mini nonetheless will get the flawed reply (“Z”), which isn’t even one of many selections. This reveals that Phi-4-mini struggles with easy logic issues, whereas o1-mini handles them rapidly and appropriately. Phi-4-mini’s detailed pondering is perhaps helpful for more durable issues, however on this case, it solely precipitated delays and errors.

Job 2:  Logical Reasoning in Math

This process requires the mannequin to acknowledge the sample in a given quantity sequence and establish the lacking quantity.

Immediate: “Choose the quantity from among the many given choices that may change the query mark (?) within the following collection:16, 33, 100, 401, ?
Choices:A) 1235
B) 804
C) 1588
D) 2006″

Enter to o1-mini

task2_start_time = time.time()

shopper = OpenAI(api_key=api_key)

messages = [
   {
       "role": "user",
       "content": """Select the number from among the given options that can replace the question mark (?) in the following series.16, 33, 100, 401, ?
       A) 1235
       B) 804
       C) 1588
       D) 2006"""
   }
]


# Use a appropriate encoding (cl100k_base is the most suitable choice for brand spanking new OpenAI fashions)
encoding = tiktoken.get_encoding("cl100k_base")


# Calculate token counts
input_tokens = sum(len(encoding.encode(msg["content"])) for msg in messages)
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
output_tokens = len(encoding.encode(completion.selections[0].message.content material))


task2_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for process 2:----------------- ", task2_end_time - task2_start_time)
# Show consequence
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

Enter to Phi 4-mini

task2_start_time = time.time()
messages = [
   {"role": "system", "content": "You are an expert in solving numerical and general reasoning questions."},
   {"role": "user", "content": """Select the number from among the given options
   that can replace the question mark (?) in the following series.16, 33, 100, 401, ?
A) 1235
B) 804
C) 1588
D) 2006"""},
]
pipe = pipeline(
   "text-generation",
   mannequin=mannequin,
   tokenizer=tokenizer,
)
generation_args = {
   "max_new_tokens": 1024,
   "return_full_text": False,
   "temperature": 0.0,
   "do_sample": False,
}
output = pipe(messages, **generation_args) #,
task2_end_time = time.time()
print("----------------=Whole Time Taken for process 2:----------------- ", task2_end_time - task2_start_time)
show(Markdown((output[0]['generated_text'])))

Response by Phi 4-mini

Comparative Evaluation

o1-mini carried out higher than Phi-4-mini in each pace and accuracy for this quantity sample process. o1-mini rapidly acknowledged the sample and appropriately selected 2006 in simply 10.77 seconds. However, Phi-4-mini took for much longer (50.25 seconds) and nonetheless acquired the flawed reply (120). In the meantime, o1-mini adopted a transparent and direct strategy, fixing the issue appropriately and effectively. This reveals that o1-mini is best at recognizing quantity patterns rapidly, whereas Phi-4-mini tends to overcomplicate easy issues, resulting in errors and delays.

Job 3: Discovering the Longest Substring

This drawback asks you to seek out the size of the longest substring inside a given string that doesn’t include any repeating characters. For instance, within the string “abcabcbb”, the longest substring with out repeating characters can be “abc”, and its size is 3.

Immediate: “Given a string s, discover the size of the longest substring with out repeating characters.
Write a perform lengthOfLongestSubstring(s: str) -> int that returns the size of the longest substring with out repeating characters.”

Enter to o1-mini

task3_start_time = time.time()
shopper = OpenAI(api_key=api_key)
messages = [
   {
       "role": "user",
       "content": """
Given a string s, find the length of the longest substring without repeating characters.


Write a function lengthOfLongestSubstring(s: str) -> int that returns the length of the longest substring without repeating characters.
"""
   }
]
# Use a appropriate encoding (cl100k_base is the most suitable choice for brand spanking new OpenAI fashions)
encoding = tiktoken.get_encoding("cl100k_base")
# Calculate token counts
input_tokens = sum(len(encoding.encode(msg["content"])) for msg in messages)
completion = shopper.chat.completions.create(
   mannequin="o1-mini-2024-09-12",
   messages=messages
)
output_tokens = len(encoding.encode(completion.selections[0].message.content material))
task3_end_time = time.time()
# Print outcomes
print(completion.selections[0].message)
print("----------------=Whole Time Taken for process 3:----------------- ", task3_end_time - task3_start_time)
# Show consequence
from IPython.show import Markdown
show(Markdown(completion.selections[0].message.content material))

Response by o1-mini

def lengthOfLongestSubstring(s: str) -> int:
   """
   Returns the size of the longest substring with out repeating characters within the string s.


   Parameters:
   s (str): The enter string.


   Returns:
   int: The size of the longest substring with out repeating characters.
   """
   # Dictionary to retailer the final positions of every character.
   char_map = {}
   # Initialize the left pointer of the window.
   left = 0
   # Initialize the utmost size.
   max_length = 0


   # Iterate over the string with the correct pointer.
   for proper, char in enumerate(s):
       # If the character is present in char_map and its final place is >= left pointer,
       # transfer the left pointer to at least one place proper of its final prevalence to keep away from duplicates.
       if char in char_map and char_map[char] >= left:
           left = char_map[char] + 1


       # Replace/insert the character's newest index.
       char_map[char] = proper


       # Calculate the size of the present window and replace max_length if crucial.
       current_window_length = proper - left + 1
       if current_window_length > max_length:
           max_length = current_window_length


       # Debug statements to hint the inner state (could be eliminated in manufacturing)
       # print(f"Present character: {char}, Left: {left}, Proper: {proper}, Present Window: '{s[left:right+1]}', Max Size: {max_length}")


   return max_length


# Instance utilization:
if __name__ == "__main__":
   test_strings = [
       "abcabcbb",
       "bbbbb",
       "pwwkew",
       "",
       "a",
       "dvdf"
   ]
   for s in test_strings:
       print(f"Longest substring with out repeating characters in '{s}' is of size {lengthOfLongestSubstring(s)}.")

Output

Enter to Phi 4-mini

from transformers import pipeline
messages = [
   {"role": "user",
    "content":"""Given a string s, find the length of the longest substring without repeating characters.
Write a function lengthOfLongestSubstring(s: str) -> int that returns the length of the longest substring without repeating characters."""},
]
pipe = pipeline("text-generation", mannequin="microsoft/Phi-4-mini-instruct", trust_remote_code=True)
pipe(messages)

Response by Phi 4-mini

def lengthOfLongestSubstring(s: str) -> int:
   char_index_map = {}
   left = 0
   max_length = 0
   for proper in vary(len(s)):
       if s[right] in char_index_map and char_index_map[s[right]] >= left:
           left = char_index_map[s[right]] + 1
       char_index_map[s[right]] = proper
       max_length = max(max_length, proper - left + 1)


   return max_length
print("Longest substring with out repeating characters in 'abcabcbb' is of size ",lengthOfLongestSubstring("abcabcbb"))
print("Longest substring with out repeating characters in 'bbbbb' is of size  ",lengthOfLongestSubstring("bbbbb"))

Output

Comparative Evaluation

Each o1-mini and Phi-4-mini used the sliding window methodology appropriately, however o1-mini’s code was extra organized and simpler to know. It included clear explanations, feedback, take a look at instances, and easy-to-read variable names. Phi-4-mini’s resolution was shorter however lacked explanations and construction, making it more durable to observe in larger tasks. o1-mini was additionally sooner and produced a cleaner, extra readable resolution, whereas Phi-4-mini centered extra on retaining the code transient.

Total Comparative Evaluation

Right here’s the general comparative evaluation for all 3 duties:

Conclusion

Total, o1-mini excelled in structured reasoning, accuracy, and coding finest practices, making it extra appropriate for complicated problem-solving and maintainable code. Whereas Phi-4-mini was sooner, its exploratory strategy sometimes led to inefficiencies or incorrect conclusions, particularly in reasoning duties. In coding, o1-mini offered well-documented and readable options, whereas Phi-4-mini prioritized brevity at the price of readability. If pace is the primary concern, Phi-4-mini is a strong selection, however for precision, readability, and structured problem-solving, o1-mini stands out as the higher possibility.

Often Requested Questions

Good day! I am Vipin, a passionate information science and machine studying fanatic with a powerful basis in information evaluation, machine studying algorithms, and programming. I’ve hands-on expertise in constructing fashions, managing messy information, and fixing real-world issues. My aim is to use data-driven insights to create sensible options that drive outcomes. I am desirous to contribute my expertise in a collaborative atmosphere whereas persevering with to be taught and develop within the fields of Information Science, Machine Studying, and NLP.