LLM Tokenization and Token Costs: How to Use Tokens Wisely When Testing
Every request you send to a large language model is measured, billed, and limited in tokens. LLM tokenization turns your text into those tokens, so it sits behind every cost line in an AI testing budget.
This guide explains how tokenization works, where token costs build up when you test LLM applications or use AI agents for test automation. It also offers tips for achieving lower spend without weakening coverage.
The piece will help QA leads and engineers who are deciding how to run LLM-based testing.
- What is LLM tokenization?
- How does tokenization work?
- Types of LLM tokenization
- Tokenization examples you’ll see in testing
- Token limit: what it means for a test run
- Where token costs come from when you test LLM applications
- Best strategy to reduce LLM token usage and cost in testing
- Claude Code token usage: what to watch in agent-driven testing
- AI test automation cost: how to judge the full picture
- What to ask before you commit to an approach
- Diagnose: Token spend in testing comes from long inputs such as logs and page content, output tokens that cost more than input tokens, and reruns of tests that nothing changed.
- The Fix: Count tokens first. Then cap judge outputs. Cache the repeated prompt prefix, batch overnight runs, and give simple judgments to a smaller model.
- Proof: Re-measure after every change. A newer tokenizer can raise token counts by roughly 30% for the same text, so a model switch needs a fresh baseline.
Quick Answer
LLM tokenization is the process of splitting text into small units called tokens and mapping each one to a number the model can process. A token can be a whole word, part of a word, a single character, or a punctuation mark. Providers count, bill, and limit usage in tokens. The same text can produce different token counts on different models.
What is LLM Tokenization?
A token is the smallest unit the LLM model reads and writes.
OpenAI’s article gives a rough rule for common English text: about four characters per token, or about 100 tokens for 75 words. Treat that as a starting estimate.
Anthropic notes that Claude 4.7 and later models use a newer tokenizer that produces approximately 30% more tokens for the same text than earlier models; the exact increase depends on content. Two teams sending identical test prompts to different models can see different bills.
Testers should care about this because:
- Cost. Providers price usage per million tokens, with different rates for input and output.
- Limits. Every model has a maximum number of tokens it can handle in one request.
- Behavior. The way text is split affects how a model reads IDs, numbers, and non-English text; all common in test data.
How Does Tokenization Work?
Tokenization works in four steps: the tokenizer cleans the text, splits it into pieces using a fixed vocabulary, maps each piece to an integer ID, and passes those IDs to the model. The model returns IDs as output, and the tokenizer converts them back into readable text.
Most modern LLM tokenizers use subword methods. The best-known is byte pair encoding (BPE). It handles rare and unknown words by encoding them as sequences of subword units.
Imagine a training text that contains the words “test”, “tests” and “testing”.
- BPE starts with single characters, then repeatedly merges the pair of neighbors that appears most often.
- First, “t” and “e” join into “te”, then “te” and “s” into “tes”, then “tes” and “t” into “test”.
- Common words end up as one token. Rare words stay split into smaller pieces.
- Frequent words are kept whole and rare words are broken into meaningful subwords.
The vocabulary is learned once from a large body of text and then fixed. A model is trained with a specific tokenizer. This is why counts from one provider do not carry over to another.
Types of LLM Tokenization
The main types of LLM tokenization are word-based, character-based, and subword tokenization, with byte-level variants that can represent any input. Most current LLMs use subword methods such as byte pair encoding, WordPiece, or Unigram.
Word-based tokenization
The text is split on spaces or rules. Each word becomes a token. It is easy to read, but the vocabulary grows very large. Any word outside the vocabulary becomes an unknown token.
Character-based tokenization
This treats every character as a token. The vocabulary is tiny, and unknown words disappear, because any word can be spelled out.
The cost is length. Each token carries little meaning on its own, and the same sentence becomes a much longer sequence. Longer sequences mean more tokens. On a paid API, that’s more tokens to pay for.
Subword tokenization
Subword methods are between the previous two. They keep frequent words whole and split rare ones. Common variants include byte pair encoding, WordPiece, and Unigram, with SentencePiece as a widely used toolkit.
Byte-level tokenization
Byte-level variants start from raw bytes instead of characters. So any text, including emoji and unusual symbols, can be represented without an unknown token. It’s a way to keep the base vocabulary small while covering all input.
Tokenization Examples You’ll See in Testing
Tokenization examples in testing fall into these patterns: ordinary sentences, identifiers and structured data, text in other languages, and page or screenshot content. Each pattern costs a different number of tokens per character.
- Ordinary sentences: A common English word is usually one token. A long or unusual word is split into pieces.
- Identifiers and payloads: Test IDs, UUIDs, hashes, long numbers, and encoded payloads are rare strings with few repeating patterns. Tokenizers tend to break them into many small pieces.
- Other languages: A NeurIPS 2023 study found that the same text can need up to 15 times more tokens in some languages than in others. A localization test suite can therefore cost more to run in some languages than in English.
- Web pages: Anthropic estimates that an average web page of about 10 kB is roughly 2,500 tokens, and a 100 kB documentation page is roughly 25,000 tokens.
- Screenshots and agent tools: Screenshots returned by browser or computer tools are billed as image input. Adding Anthropic’s browser tool to a request also adds about 6,600 input tokens.
Check in your pipeline: Pick one sample of each input type your tests send, count the tokens, and write the numbers next to the test. That table becomes your cost model.
Token Limit: What It Means for a Test Run
A token limit is the maximum number of tokens a model can handle in one request, counting what you send and what it returns. Providers call this the context window. The size varies by model.
Three factors affect test design:
- Thinking tokens count toward the limit and are billed as output.
- Compaction summarizes earlier turns so a session can continue past the window.
- If a response reaches the limit, the API returns the stop reason model_context_window_exceeded. A test can check for it and fail instead of passing a half-finished answer.
Long conversations are where limits kick in. ACCELQ’s LLM verification commands include a conversational metric for knowledge retention, which checks whether a chatbot keeps information across a conversation. Those tests grow in token count with every turn, so it’s best to budget them separately.
Where Token Costs Come From When You Test LLM Applications
Token costs of testing LLM applications come from the calls your application makes while under test, the calls a judge model makes to score answers, the calls an AI agent makes to write or run tests, and the reruns that repeat all of them. Output usually costs more than input.
Several smaller items add to the total cost:
- Tool definitions count as input. Anthropic lists a 286-token system prompt for tool use on Sonnet 5.5, plus the tokens in your own tool schemas.
- Web search results retrieved during a conversation. They are counted as input tokens in every later turn.
- Reasoning costs. Thinking tokens are billed as output.
A worked example
This is illustrative arithmetic using Anthropic’s list prices as of October 2026.
Take a nightly regression of 500 test cases, each sent to a Claude Sonnet 5.5 judge with 1,500 input tokens and 300 output tokens. Of the input, 1,000 tokens are the same scoring rubric every time.
-
Baseline: 750,000 input tokens cost $1.50 and 150,000 output tokens cost $1.50. One run costs $3.00.
-
Cap the output: Ask for a score and a short reason, about 60 tokens. Output cost falls to $0.30. One run costs $1.80.
-
Cache the rubric: Cached tokens cost $0.10 per million instead of $2. One run costs about $0.85.
-
Batch it: The Batch API takes 50% off, and the discount stacks with caching. One run costs about $0.43.
Over 30 nights, that is $90 against about $12.75. Treat the final figure as a best case, since cache hits depend on request timing. You also have to check that shorter outputs still judge accurately.
Also Read: LLM's in Software Testing
Best strategy to reduce LLM token usage and cost in testing
The best strategy to reduce LLM token usage in testing is to: measure first, use LLM checks only where they add value, shrink inputs, cap outputs, cache repeated prompt prefixes, batch what can wait, and move simple judgments to a smaller model.
Note: Together, these fixes would take the worked example above from $3.00 to about $0.43 per run.
1. Measure before you change anything
Count tokens per test with the token counting endpoint. Read the usage fields returned with each response, which the pricing documentation shows. Track cost per test.
2. Use LLM checks only where you judge meaning
Keep LLM calls for questions of meaning, such as relevance, faithfulness, and tone. In ACCELQ, LLM verification commands are added only where you want them. They run on an API key you supply, so usage appears with your own provider.
3. Shrink the input
Send the judge only the answer under test, the question, and the retrieved context. Skip full logs and whole pages. Anthropic’s Claude Code guide shows that a hook that filters test output can cut context from tens of thousands of tokens to hundreds.
4. Cap the output
Output tokens cost five times more than input tokens on Sonnet 5.5. Thinking tokens are billed as output. Ask a judge for a score and a one-sentence reason, and lower effort for simple tasks. Compare short and long outputs on a sample first; a very short reason can make a failed test harder to debug.
5. Cache the repeated prefix
Rubrics, system prompts, tool definitions, and example answers repeat on every call. Cache prefixes are built in the order tools, system, then messages. Put the cache marker on the last block that is identical across requests, and keep volatile content, such as test case data, after it. A five-minute cache write costs 1.25 times the base input price, and a hit on Sonnet 5.5 costs 5% of it.
6. Batch what can wait
Nightly regressions, evaluation sets, and bulk test data generation rarely need an instant answer. The Batch API takes 50% off input and output tokens. Keep interactive debugging on standard calls.
7. Right-size the model
Anthropic recommends Claude Haiku 5.5 for classification, extraction, and routing. For prompts up to 100,000 tokens, its list price is $0.10 input and $0.50 output per million tokens, one twentieth of Sonnet 5.5. Above that, it costs more.
Before you move a judge to a smaller model, run human evaluations on a subset of cases and switch only when the automated scores are close to the human ones.
8. Run less, re-baseline when the model changes
Run only the LLM-judged tests that a change can affect. When you switch models, count again: the tokenizer on Claude 4.7 and later can produce roughly 30% more tokens for the same suite. Anthropic advises counting with the specific model ID.
Claude Code Token Usage: What to watch in agent-driven testing
Claude Code is billed by API token consumption. Anthropic reports an average of about $13 per developer per active day, or $150 to $250 per month. The same guide suggests piloting with a small group to set a baseline before wider rollout. Testing teams should do the same.
Usage climbs in long sessions because Claude Code sends the full conversation with every request. Caching lowers the cost of that history. But the first message after a break longer than the cache lifetime reprocesses everything on an API key that lifetime is five minutes.
Habits that help lower costs in test work:
- Clear between tasks. Use /clear, and add compaction instructions such as focusing on test output and code changes.
- Filter test output. Anthropic’s example hook rewrites test commands so only failures come back.
- Delegate test runs to subagents. Verbose output stays in the subagent’s context. A smaller model keeps their cost down.
- Pick the right model and effort. Sonnet handles most coding tasks for less than Opus. Thinking cannot be turned off on Opus 5.5, Sonnet 5.5, Haiku 5.5 or the Fable models, so lower the effort for simple work.
- Keep teams small. Teams use about seven times more tokens than standard sessions when teammates run in plan mode.
- Track spend. Run /usage, and set workspace spend limits in the Console for API organizations.
If your team tests from Claude Code, it will fit into ACCELQ’s MCP server. Claude Code is a supported assistant, MCP is part of ACCELQ Autopilot, and the assistant works as you, with your own permissions. Claude Code also defers MCP tool definitions by default, so adding a server does not load every tool definition up front.
The ACCELQ MCP tools follow a review-first pattern. The assistant proposes a test plan, creates tests after you approve it, and shows a run plan before anything starts. Results come back as passed, failed, and not-run counts. Trend analysis returns a verdict that groups failures as persistent, flaky, regression or recovered. That gives the assistant short, structured answers to reason over. Compare /usage before and after on a pilot to see what it saves.
AI Test Automation Cost: How to Judge the Full Picture
AI test automation cost is token spend plus the platform, the compute that runs the tests, and the people time spent creating and maintaining them. Token spend is the easiest to measure, because providers report usage with each response and publish per-million-token rates.
Track three numbers: cost per test run, cost per suite run, and monthly spend per developer using an AI agent. Estimate people time separately, such as maintenance hours before and after AI assistance. Test any vendor’s savings claims against your own baseline.
What to ask before you commit to an approach
Before you commit to an approach for LLM-based testing, check these: who pays for tokens, whether you can see and cap spend, how results return to the model, whether you can cache and batch, and how thresholds are validated.
-
Who pays for tokens. ACCELQ’s LLM verification commands use your own API key, so your provider bills the usage. You choose the model.
-
Spend visibility. Anthropic offers workspace spend limits and a usage page in the Console. Ask any vendor for the equivalent.
-
Result format. Counts and verdicts are shorter than raw logs. ACCELQ’s MCP tools return job counts and trend verdicts.
-
Caching and batching. Confirm your setup can use the provider’s caching and batch features.
-
Thresholds. Ask how pass scores are set. Baseline your current production model. Calibrate the scores against human evaluation.
FAQs
What is a token in an LLM?
How does tokenization work in large language models?
What is character-based tokenization?
How do I reduce LLM token usage and cost when testing?
How do I count tokens before a test run?
Why did my token count rise after I switched models?
What happens when a test exceeds the token limit?
How much does Claude Code cost per developer?
Whose API key do ACCELQ's LLM verification commands use?
You Might Also Like:
Manual Testers Matter: How They Drive Modern Automation
Manual Testers Matter: How They Drive Modern Automation
The Top 7 UI Test Automation Best Practices
The Top 7 UI Test Automation Best Practices
Flaky Tests in 2026 – How to Identify, Fix, and Prevent Them