The OpenAI transition
OpenAI removed o3 from the ChatGPT interface on August 26, 2026. This follows the 90-day sunset period that started on May 28, 2026. While the ChatGPT interface now uses GPT-5.4 Thinking as the primary reasoning model, the o3 API remains available for developers without a confirmed retirement date. The removal of o3 from ChatGPT forces enterprises to migrate their existing workflows to the o4 or the GPT-5 family of models if they require the specific reasoning capabilities provided by the o3 architecture. You know that reasoning models consume more compute than standard LLMs. Because o3 uses a reinforcement learning approach to run through intermediate reasoning steps, it produces different results than pattern matching models. This architectural shift allows the model to perform problem decomposition, exploration, evaluation, backtracking, verification, and synthesis.
Reasoning benchmark scores
Reasoning benchmarks demonstrate the divergence between standard models and thinking models. OpenAI o3 scores 96.7% on the 2024 AIME math competition. It reaches 25.2% on FrontierMath, which tests unseen research-level mathematical problems. On GPQA Diamond, o3 achieves 87.7% in PhD-level science questions. However, o3 scores only 2.9% on the newer ARC-AGI-2 benchmark, whereas Gemini 3.1 Pro reaches 77.1% on that same test. These figures show that performance on one metric does not guarantee dominance in another. Gemini 2.5 Pro and o3 compete in several other high-stakes categories. o3 reaches a reasoning score of 73.4, while Gemini 2.5 Pro reaches 69.5. In math, o3 scores 37.7 against Gemini 2.5 Pro’s 35.1. Gemini 2.5 Pro leads in multimodal performance with a score of 70.4, which exceeds o3’s 63.3. Instruction following shows a large gap where o3 scores 86.0 and Gemini 2.5 Pro scores 56.4.
Software engineering and coding analysis
The coding landscape favors Anthropic models. Claude Opus 4.5 reaches 76.8% on the SWE-bench leaderboard. o3 reached 71.7% on SWE-bench Verified. Claude Fable 5 reached 95.0% on SWE-bench before its suspension in June. Most public benchmarks like HumanEval or MBPP measure code generation rather than code analysis. Code review requires a model to read existing code, identify problems, and explain them without writing new code. A benchmark of 112 pull requests from open-source repositories tested models on specific issues. These issues included 18 concurrency bugs, 16 security vulnerabilities, 14 logic errors, 12 resource leaks, 10 API contract violations, 8 performance regressions, and 6 error handling gaps. Reasoning models like o1, o3-mini, and o4-mini use chain-of-thought to trace variable state across function boundaries and evaluate whether a new code path maintains invariants.
| Model | SWE-bench Verified Score |
|---|---|
| Claude Fable 5 | 95.0% |
| Claude Opus 4.5 | 76.8% |
| Gemini 3 Pro Preview | 77.40% |
| o3 | 71.7% |
| GPT-5 | 74.40% |
Enterprise data complexity
DevRev released the Enterprise AI Agent Benchmark to test how models handle messy data organization and permissions. The benchmark uses four levels of autonomy, with L1 and L2 covering retrieval and multi-step thinking in a single domain. L1 tasks involve retrieval and synthesis across a maximum of two or three sources. L2 tasks involve multi-step thinking in a single domain, where planning is evident but the agent does not take action. Most enterprise tasks involve moderate complexity but massive data variety. Data organization involves the source, the shape, and the permissions that control access to the information. DevRev’s Computer agent achieved 22 to 35 points higher accuracy than Claude Code in L1 and L2 tasks. Computer also used 268,000 tokens per correct answer, which is 3.4 times fewer than the 902,000 tokens used by Claude Code. Scaling the dataset from 1x to 64x increased Computer’s token consumption by 11 percent, while Claude Code’s increased by 55 percent.
Context window and large scale workloads
Gemini 2.5 Pro includes a 1 million token context window. This allows users to feed entire datasets, legal contracts, or mid-sized codebases into a single prompt. This architectural difference eliminates the need for complex retrieval-augmented generation (RAG) pipelines because the model handles the entire context directly. In contrast, the GPT-4 generation relied on a 128K window. Users of smaller windows must chunk documents, build RAG pipelines, and manage context carefully. Gemini 2.5 Pro is a better choice for tasks requiring long-context Q&A over codebases, reports, or contracts. It is also a better choice if cost efficiency on long-context workloads is a priority.
| Model Capability | Gemini 2.5 Pro | GPT-4 Class |
|---|---|---|
| Context Window | 1,000,000 tokens | 128,000 tokens |
| Reasoning Type | Multimodal Chain-of-Thought | Standard LLM |
| Best Use Case | Long-context & Multimodal | Real-time & Latency-sensitive |
Economics of inference and thinking tokens
Thinking tokens increase the cost of reasoning models. These tokens are generated during the model’s internal reasoning process and are invisible to the user in the final response. Google bills these tokens at the standard output rate. If a model generates 3,000 reasoning tokens for a 500-token response, the user pays for 3,500 output tokens. For Gemini 2.5 Pro, the price is $1.25 per million input tokens and $10.00 per million output tokens. Gemini 3.1 Pro is more expensive at $2.00 per million input tokens and $12.00 per million output tokens. Gemini 3.5 Flash is $1.50 per million input tokens and $9.00 per million output tokens. Gemini 3.5 Flash-Lite costs $0.10 per million input tokens and $0.25 per million output tokens. Vertex AI pricing runs 10 to 20 percent higher than Google AI Studio for the same models.
| Model Name | Input Price (per 1M) | Output Price (per 1M) |
|---|---|---|
| Gemini 2.5 Pro | $1.25 | $10.00 |
| Gemini 3.1 Pro | $2.00 | $12.00 |
| Gemini 3.5 Flash | $1.50 | $9.00 |
| Gemini 3.5 Flash-Lite | $0.10 | $0.25 |
| o3-mini (API) | $1.10 | $4.40 |
Multimodal capabilities and speed
Gemini 2.5 Pro is a multimodal model that handles images, videos, and text. Google describes the model as being engineered for maximum quality. OpenAI’s o3-Pro positions itself as a high-reliability variant that thinks longer for challenging questions. The key tradeoff between these models is speed versus depth. OpenAI recommends o3-Pro for challenging questions where reliability matters more than speed. Gemini 2.5 Pro is a better choice for tasks involving multimodal reasoning or long-document analysis. GPT-4 models lead in real-time multimodal interaction, specifically regarding live audio and video input. GPT-4 can take live audio and video input natively, which makes it the practical choice for customer-facing voice applications.
The operational verdict
The decision between Gemini 2.5 Pro and o3 depends on the specific engineering requirements of the task. Gemini 2.5 Pro is the superior choice for tasks requiring massive context windows, multimodal analysis, or high-volume tasks via Flash variants. The model handles complex reasoning over 2 million token contexts more effectively through its specialized architecture. o3 is the superior choice for high-accuracy mathematical reasoning and instruction following where latency is not a priority. The model excels at finding correct answers for problems with a verifiable solution. Which model will dominate the enterprise market as context windows continue to expand? Gemini 2.5 Pro is the best model for long-context workflows and multimodal reasoning.
