When you ask a reasoning model a complex question-like diagnosing a rare medical condition or predicting legal outcomes-it doesn’t just spit out an answer. It pauses. It thinks. And during that pause, it generates dozens, sometimes hundreds, of hidden think tokens-intermediate steps that aren’t shown to you but are critical to getting the right result. This isn’t magic. It’s math. And it comes with a price tag.
What Are Think Tokens, Really?
Think tokens are the hidden reasoning steps a model generates before giving you its final answer. They’re not visible in the output, but they’re what separate a reasoning model like OpenAI’s o3 or Anthropic’s Claude 3.7 from a standard LLM. Think of them like a student showing their work on a math test. The teacher doesn’t need to see every scratch-out, but they know the student didn’t just guess. OpenAI’s o1 models, launched in late 2023, were the first to make this approach commercially viable. They use up to 5x more tokens than standard models just to think through a problem. For example, on the GPQA Diamond benchmark-a set of expert-level science questions-Qwen2.5-14B-Instruct generates 1,200 to 1,800 reasoning tokens per query. That’s a lot of processing power just to answer one question. The goal? Higher accuracy. And it works. Qwen2.5-14B-Instruct jumps from 38.2% to 47.3% accuracy on GPQA when using full reasoning. That’s a 9.1-point gain. But here’s the catch: you pay for every token. And not just in compute time-in dollars.The Cost of Thinking
OpenAI charges $0.015 per 1,000 reasoning tokens. For standard GPT-4 output, it’s $0.003. That’s a fivefold increase for the same output quality. If your application runs 50,000 queries a month, switching to a reasoning model can spike your bill from $1,200 to $6,800. That’s not a typo. Refuel.ai’s data shows that fine-tuning models with reasoning traces increases output token counts by 400-600%. And for every 5% accuracy gain, you’re spending 5.3x more tokens on average. The math is brutal: you get better answers, but at a cost that scales faster than the benefit. Enterprise users know this. On G2, 82% of companies using reasoning models say cost is their biggest concern. Some are turning to techniques like Conditional Token Selection (CTS), developed by Zhang et al., which cuts reasoning tokens by 75.8% with only a 5% drop in accuracy. That’s the kind of optimization that turns a luxury feature into a practical tool.When Thinking Helps-and When It Doesn’t
Not all problems need deep thinking. Apple’s research breaks reasoning models into three performance zones:- Low complexity (1-3 steps): Standard models win. They’re faster, cheaper, and more accurate by 4.7-8.2 percentage points. Reasoning adds noise, not clarity.
- Medium complexity (4-7 steps): This is where reasoning models shine. Accuracy jumps 9.1-12.3 points. Think financial risk modeling, drug interaction checks, or multi-step legal analysis.
- High complexity (8+ steps): Both models collapse. Accuracy drops below 5%. The model doesn’t get smarter-it just gives up. Apple’s team calls it “complete accuracy collapse.”
Why Some Reasoning Chains Are Nonsense
Not all thinking is useful. OpenAI’s o3 models generate reasoning chains that are 50% rated as “largely inscrutable” by human evaluators. Compare that to Claude 3.7, where only 15% are unreadable, and Qwen2.5 at 28%. What does this mean? A model can generate a long, detailed explanation that sounds smart but is logically broken. This is the illusion of reasoning. The model isn’t understanding. It’s mimicking. It’s learned that long outputs correlate with high scores on benchmarks, so it generates them-even if the steps don’t make sense. That’s dangerous in high-stakes applications. Imagine a legal AI generating a 20-step analysis that looks thorough but misses a key precedent because it’s following a learned pattern, not real logic.How to Use Reasoning Models Without Going Broke
If you’re serious about using reasoning models, here’s what works:- Match the model to the task. Don’t use reasoning for simple queries. Reserve it for problems with 4-7 clear steps.
- Use dynamic token budgeting. Let the system decide how many think tokens to spend based on question complexity. Tools like CTS can help automate this.
- Test with reference models. Run a subset of queries through a simpler model first. If the answer is confident and accurate, skip the heavy reasoning.
- Avoid fine-tuning standard LLMs. Refuel.ai found applying reasoning traces to models like Llama-3 without prior training drops accuracy by 12%. Use models trained for reasoning from the start.
- Monitor latency spikes. Reasoning tokens cause unpredictable delays. If your SLA is under 2 seconds, test under peak load. 63% of users report violations.
The Market Is Split
The reasoning model market hit $2.8 billion in Q4 2024. But adoption isn’t even. OpenAI leads with 47% of commercial API usage. Anthropic has 28%. Open-source models like Qwen hold 15%. The rest? Niche players like Refuel.ai, focused on data extraction. Fortune 500 companies are all in: 78% of financial firms use them for risk modeling, 63% of pharma companies for drug discovery, and 41% of legal tech platforms for case analysis. But only 22% of small businesses use them-even though 68% see the potential. The barrier? Cost and complexity.What’s Next?
OpenAI’s next model, o4, will feature adaptive reasoning depth-automatically adjusting token use based on problem difficulty. That’s a big step. Gartner predicts that by 2026, 80% of enterprise reasoning systems will use token compression techniques like CTS. The race isn’t about bigger models anymore. It’s about smarter thinking. But there’s a warning from the Santa Fe Institute. Dr. Melanie Mitchell says next-token prediction architectures may never achieve human-like efficiency. We’re hitting a wall. More tokens don’t always mean more understanding. At some point, the model just runs out of tricks.Final Thought: Think Smarter, Not Harder
Reasoning models are powerful-but they’re not a cure-all. They’re tools for specific problems. Use them when the complexity justifies the cost. Skip them when they add noise. And always, always test the output-not just the accuracy, but the logic. The future of AI isn’t just about bigger models. It’s about knowing when to make them think-and when to let them stay quiet.Do reasoning models always give better answers than standard LLMs?
No. For simple tasks-like answering basic facts or summarizing short texts-standard LLMs often perform better. Reasoning models add noise and latency without improving accuracy. They only outperform standard models on medium-complexity problems with 4-7 logical steps. Beyond that, both types collapse.
How much more do reasoning models cost to run?
OpenAI charges 5x more for reasoning tokens than standard output. For example, $0.015 per 1,000 reasoning tokens vs. $0.003 for standard GPT-4. If your app runs 50,000 queries monthly, switching to a reasoning model can raise costs from $1,200 to $6,800. Fine-tuning with reasoning traces can increase token use by 400-600%, making cost the top concern for 82% of enterprises.
Can I improve reasoning efficiency without losing accuracy?
Yes. Conditional Token Selection (CTS), developed by Zhang et al., reduces reasoning tokens by 75.8% with only a 5% accuracy drop on GPQA. Other methods include dynamic budgeting-allocating more tokens for complex questions and fewer for simple ones-and using reference models to skip reasoning when unnecessary. Organizations using these techniques save 30-45% on costs while keeping 95% of accuracy gains.
Why do some reasoning models produce nonsense explanations?
Because they’re not actually reasoning-they’re mimicking. These models learn that long, detailed outputs correlate with higher benchmark scores, so they generate them even if the logic is flawed. OpenAI’s o3 models have 50% of their reasoning chains rated as “largely inscrutable” by humans. Claude 3.7 and Qwen are better, but the issue persists. Always validate the output, not just the chain.
Are reasoning models worth it for small businesses?
Usually not. Only 22% of small-to-medium businesses use them, even though 68% see their potential. The high cost and technical complexity make them impractical unless you’re solving high-value, multi-step problems like financial risk modeling or rare disease diagnosis. For most SMEs, standard models are faster, cheaper, and good enough.
What’s the biggest risk of using reasoning models?
The biggest risk is over-trusting the output. Just because a model gives a long, detailed reasoning chain doesn’t mean it’s correct. In legal, medical, or financial applications, this can lead to serious errors. Always pair reasoning models with human review, especially for critical decisions. Also, watch for latency spikes-63% of users report SLA violations during peak usage.
