OpenAI o3 achieves 96.7% accuracy in competition-level math problems and 87.7% accuracy in PhD-level science questions. These numbers exceed the performance of the OpenAI o1 model and set a new industry standard for reasoning. However, high performance on benchmark datasets does not necessarily align with real-world tasks, on-the-ground performance, or business value. While the model achieves high scores on math and science tests, the high cost of high-compute modes and the unexpected reliability issues in enterprise workflows suggest that businesses must proceed with significant caution when they plan their AI budgets.
The reality of benchmark scores
The reported performance for OpenAI o3 shows a massive leap over previous models. In the ARC-AGI datasets, which test spatial reasoning and pattern recognition, o3 scores between 75.7% and 87.5% accuracy. A high-compute configuration using 172x the compute of the low-efficiency mode reaches 87.5% accuracy. This is comparable to human performance at about 85% accuracy. In general intelligence metrics, o3 scores 1380 on Chatbot Arena ELO, which is higher than the 1280 score of GPT-4o.
The model also shows strength in specialized domains. o3 scores 84.0% on MMLU-Pro compared to the 72.6% scored by GPT-4o. In coding, o3 scores 69.1% on SWE-bench Verified, which is a significant increase over the 30.7% score of GPT-4o. It also reaches 70.0% on LiveCodeBench, while GPT-4o reaches 38.0%. In math, o3 scores 98.4% on AIME 2025, which dwarfs the 9.3% score of GPT-4o. It also scores 96.0% on MATH, exceeding the 74.6% of GPT-4o. In reasoning tasks, o3 scores 83.3% on GPQA Diamond, while GPT-4o scores 50.3%. It also scores 95.8% on BBH and 94.0% on Winogrande.
Other evaluations show similar trends. o3 scores 92.0% on IFEval, while GPT-4o scores 85.3%. o3 scores 49.4% on SimpleQA, compared to 40.1% for GPT-4o. o3 scores 86.0% on TruthfulQA, which is higher than the 83.5% for GPT-4o. In coding tasks, o3 reaches 90.0% on HumanEval+, which is higher than the 87.2% of GPT-4o. It also scores 1074 on Design Arena ELO and 1048 on Website Arena ELO, both of which exceed the scores of GPT-4o.
The cost of reasoning
High benchmark performance does not guarantee business value because the cost of using these models remains high. For instance, it costs about $17 to $20 every time o3 completes a visual reasoning task in its low-compute mode. The cost of its high-compute mode is believed to be even higher, and it can run into thousands of dollars per task. Reasoning models like o3 bill internal reasoning tokens at output rates. This means effective costs run 3x to 10x the base rate depending on how complex the task is.
The pricing for the OpenAI API follows a specific structure where output costs are much higher than input costs. For the GPT-5.6 family, output costs are exactly 6x the input costs across every tier. For the o-series, the price depends on the specific model chosen.
| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|---|---|---|
| o3 | $2.00 | $8.00 |
| o3-pro | $20.00 | $80.00 |
| o4-mini | $1.10 | $4.40 |
| Claude 4 Sonnet | $3.00 | $15.00 |
| Claude 4 Opus | $15.00 | $75.00 |
Users must also account for long-context multipliers. Any prompt that exceeds 272K input tokens triggers a separate long-context meter. In these cases, the input price doubles and the output price rises by 50%. You likely understand that a high score on a test does not guarantee a model will perform correctly in your specific business environment.
Infrastructure stability and outages
Reliability remains a significant hurdle for enterprise adoption. On September 3, 2026, OpenAI logged a major disruption that affected both ChatGPT and Codex. Downdetector tracked more than 74,000 reports during the incident. The disruption lasted several hours, with one report placing the window between 10:58 AM ET and 2:56 PM ET. OpenAI confirmed elevated errors across its products, and users reported that login screens stalled or messages failed to send.
Codex users experienced different problems. Coding workflows interrupted mid-task, which stalled engineering pipelines. This incident was larger than an earlier disruption in April 2026. In 2026, OpenAI also saw errors in the Asia-Pacific region on September 19 and 20. These incidents show that a single vendor outage can impact many different industries at once.
The concentration of AI usage on a few cloud providers creates shared risks. Earlier in 2026, an Azure-side failure caused ChatGPT, Claude, and Grok to go offline together for 90 minutes. This shows that even if different companies build the models, they often share the same underlying infrastructure.
Regression and reliability issues
A different set of problems emerged around August 31 and September 1, 2026. During this time, users noticed that GPT-5.6 Sol began to behave like a lower-tier fallback model. Complex requests that once took time to process started returning almost instant responses. These responses often skip deeper reasoning and ignore instructions that previously required analysis.
File handling also became unreliable during this period. ChatGPT sometimes fails to process the actual contents of an attached PDF. The interface shows the file is present, but the model provides answers that suggest it did not read the document. It often infers or guesses instead of grounding the answer in the content. This issue is problematic because no upload error appears to the user.
Python and container-based workflows also failed. Users encountered errors like caas.internal.errors.ClientError and found that /mnt/data files were not available. Deep Research also showed signs of failure. The tool appears to run for several minutes without performing any actual web searches or retrieving sources. This leaves the final report looking as if it was generated without any external research.
Competition from Anthropic
Anthropic provides a strong alternative through its Claude 4 models. Claude 4 Opus and Claude 4 Sonnet are hybrid models that offer near-instant responses or extended thinking for deeper reasoning. Claude 4 Sonnet delivers 72.7% on the SWE-bench coding benchmark, which is higher than the 69.1% scored by o3.
Claude 4 Sonnet also shows better efficiency in certain tasks. It has a latency of 1.9s and provides a balance of performance and cost. Claude 4 Opus is the most powerful model in its lineup and leads on SWE-bench with 72.5%. Claude 4 models also use tools in parallel and feature improved memory capabilities.
The competition in pricing is also intense. Claude 4 Sonnet costs $3 per million input tokens and $15 per million output tokens. This is lower than the $20 per million input and $80 per million output required for o3-pro.
Enterprise deployment strategies
OpenAI is attempting to address deployment challenges through the OpenAI Deployment Company. This business unit helps organizations build and deploy AI systems they can rely on. OpenAI launched this company with more than $4 billion in initial investment. It is a partnership with 19 firms including TPG, Advent, Bain Capital, and Brookfield. It also includes investment from Goldman Sachs, SoftBank Corp., and McKinsey & Company.
The company acquired Tomoro to add 150 Forward Deployed Engineers to its team. These engineers work with business leaders to identify where AI can make an impact. They help redesign organizational infrastructure and critical workflows around AI. This move aims to help companies move from identifying opportunities to building production systems that deliver results.
Deployment is not just about the model. Successful implementation requires addressing data governance, cost structures, and the need for explainability. The OpenAI Deployment Company works to turn AI gains into durable systems by connecting models to customer data, tools, and business processes.
The human element in AI oversight
As AI models improve, the difficulty of human review increases. When an AI performs well most of the time, humans develop a mindset that the tool is reliable. This can lead to overlooked errors. If a model has 98% accuracy, humans often lack the tools or expertise to find the 2% of errors that remain.
In some cases, users discard AI output entirely because they need 100% accuracy. This creates a tension between the desire for automation and the requirement for perfection. Companies must focus on creating user-friendly workflows that allow staff to verify AI work without massive increases in manual labor.
The ability of a model to adapt to new contexts is also a factor in human trust. Most reasoning models struggle when a well-known puzzle is slightly modified. For example, if a user changes the constraints of the Monty Hall problem, the model might still rely on its training data instead of the new context. This tendency to overfit on training data makes it difficult for humans to trust the model in novel, real-world scenarios.
The mechanics of reasoning
The reasoning capabilities of o3 stem from a specific architecture. The model uses natural language program search and execution within token space. At test time, the model searches over a space of possible Chains of Thought (CoTs) that describe the steps required to solve a task. It calculates multiple chains of thought and selects the one with the most likely successful outcome. This process is similar to AlphaZero-style Monte-Carlo tree search.
This search process requires massive amounts of computation. The model generates and executes its own programs where the program itself becomes the artifact of knowledge recombination. This allows the model to handle tasks it has never encountered before.
However, these programs are natural language instructions rather than executable symbolic programs. This means the model must evaluate the fitness of its own plans using another model. This reliance on an evaluator model can lead to errors when the system operates out of distribution. Will the high cost of reasoning tokens eventually drive enterprises toward smaller, more efficient models?