What Are LLM Tokens and Why Do They Matter?
Every large language model — whether it's Claude, GPT-4o, or Gemini — doesn't read text the way humans do. Instead, it breaks text into tokens: subword units that sit between individual characters and whole words. The tokenizer (a separate algorithm trained alongside the model) converts your text into sequences of integer IDs, each corresponding to a token in the model's vocabulary. These IDs are what actually flows through the neural network.
Understanding tokens is foundational to working with LLM APIs because every API provider bills by the token. You pay separately for input tokens (everything you send: system prompt, history, user message) and output tokens (everything the model generates). Since input tokens are processed in parallel and output tokens are generated sequentially, providers typically charge 3–5× more for output than input.
How Token Counting Works
Different models use different tokenization algorithms. OpenAI's GPT family uses tiktoken (a BPE-based tokenizer). Anthropic's Claude uses a custom tokenizer trained with Claude. Google's Gemini uses SentencePiece. Because each tokenizer has its own vocabulary and merging rules, the same text can produce different token counts across models — though the differences are usually small (within 5–10%) for English text.
A useful rule of thumb: for standard English prose, 1 token ≈ 4 characters or ¾ of a word. So a 1,000-word article is roughly 1,333 tokens. For dense code, JSON, or XML, the ratio is closer to 3 characters per token because identifiers and symbols are common in the tokenizer's vocabulary. For non-English languages (Chinese, Arabic, Korean), the ratio worsens significantly — often 2–3× more tokens per word than English — because these languages are underrepresented in most tokenizer training data.
Context Windows: The Token Budget
Every LLM has a context window — the maximum number of tokens it can process in a single API request. This window encompasses both input and output. If your input alone exceeds the context window, the API will return an error (or, in some implementations, silently truncate). The context window also determines how much conversation history, document context, and system instructions you can include.
Context windows have grown dramatically over the last two years. Early GPT-4 had a 8,192-token window. Today, Claude claude-sonnet-5 supports 200,000 input tokens — enough to process a full novel. Gemini 1.5 Pro extends this to 1,000,000 tokens, enabling analysis of an entire codebase or hours of transcribed audio in a single call. GPT-4o's 128,000-token window covers most enterprise document processing needs.
2026 LLM Pricing: A Full Comparison
API pricing has fallen substantially year over year as model efficiency improves and competition intensifies. Here's the current landscape for the most widely-used models:
- Claude claude-sonnet-5 ($3/$15 per MTok): Anthropic's flagship model for complex reasoning, coding, and analysis. Best-in-class for instruction following and multi-step tasks.
- Claude claude-haiku-4-5 ($0.80/$4 per MTok): Fast and affordable for high-volume production workloads. Excellent for extraction, classification, and structured output.
- GPT-4o ($2.50/$10 per MTok): OpenAI's omnimodel with strong multimodal capabilities and broad tool-use support. 128K context window.
- GPT-4o-mini ($0.15/$0.60 per MTok): The most cost-effective OpenAI model for straightforward tasks. Dramatically cheaper than GPT-4o for simple use cases.
- Gemini 1.5 Pro ($1.25/$5 per MTok): Google's long-context specialist. 1M token window makes it unique for document-heavy workflows.
- Gemini 1.5 Flash ($0.075/$0.30 per MTok): The cheapest model in this comparison. Ideal for high-volume, latency-sensitive applications where cost is the primary constraint.
API Cost Optimization Strategies for 2026
Prompt Caching is the single most impactful optimization for production LLM applications. Both Anthropic and OpenAI allow you to mark portions of your prompt as cacheable. When the same cached prefix is reused across requests, you pay the cache read rate (typically 80–90% cheaper) instead of the full input rate. For apps with a static system prompt or shared document context, caching alone can reduce costs by 60–80%.
Model Routing is the architectural pattern of dispatching different tasks to different models based on complexity. A routing layer (often a lightweight classifier or rule-based system) sends simple tasks (keyword extraction, sentiment classification, short Q&A) to cheap models like Haiku or GPT-4o-mini, reserving expensive models for complex reasoning. Teams implementing routing typically see 40–60% cost reductions without noticeable quality degradation.
Streaming vs. Batch: For non-interactive use cases (document processing, data extraction pipelines), batch APIs (where available) can offer significant discounts. Anthropic's Message Batches API provides 50% off standard pricing for workloads that don't require real-time responses. OpenAI's Batch API similarly offers 50% discounts with 24-hour completion SLAs.
Output Length Control: Setting a tight max_tokens limit prevents unexpectedly long outputs that inflate costs. For structured data extraction, instruct the model to output only JSON with no explanation. For summarization, specify a word limit. Controlling output length is especially important when using high-cost models like Claude claude-sonnet-5 or GPT-4o, where output tokens are billed at premium rates.
Tokens in Multi-Turn Conversations
In chat applications, each API call typically includes the full conversation history to give the model context. A 10-turn conversation where each turn is 200 tokens means the 10th call includes ~2,000 tokens of history alone — before your latest message or system prompt. As conversations grow, costs compound rapidly. Production chat applications must implement history truncation (keeping only recent turns) or history summarization (compressing older context) to control per-conversation costs.
A well-designed conversation management system limits effective context to the last 4–6 turns plus a rolling summary of earlier conversation. This keeps input tokens predictable and costs bounded, while preserving enough context for coherent multi-turn dialogue. Tools like LangChain's ConversationSummaryBufferMemory or custom summary chains automate this pattern.