What is Token?
Tokenisation is the process of splitting text into tokens using a vocabulary of subword units. Modern LLMs use Byte Pair Encoding (BPE) or SentencePiece tokenisation, which splits common words into single tokens and rare words into multiple subword tokens. For example, 'running' might be a single token, while 'tokenisation' might be split into ['token', 'isation']. This means token counts vary by language and content type: English prose uses approximately 1.3 tokens per word; code uses more tokens per character due to special syntax; languages with non-Latin scripts (Chinese, Arabic, Hindi) use significantly more tokens per character than English.
Token economics matter enormously for enterprise AI cost management. An enterprise RAG system that prepends 8,000 tokens of document context to every query is burning 8,000 input tokens per request. At scale — millions of queries per month — this becomes a significant cost. Enterprise AI architects use techniques like selective context inclusion (only adding the most relevant retrieved chunks, not all retrieved chunks), prompt compression (using a smaller model to compress verbose context before sending to the main model), and KV caching (reusing computed representations for repeated prompt sections) to manage token costs without sacrificing response quality.
Also known as: Subword Unit, BPE Token
Key Points
Core idea
One English token is approximately 0.75 words or 4 characters. A 10-page document (~5,000 words) is approximately 6,500 tokens — useful for estimating context window usage and API costs.
Why it matters
Non-Latin scripts and low-resource languages often use more tokens per word than English, increasing costs and reducing effective context window capacity for multilingual enterprise applications.
Enterprise use
Cloud LLM APIs charge per input and output token. A cost-per-query analysis requires accurate token count estimation for the system prompt, context, and typical response length.
How Token works
Define the purpose, inputs, and success criteria that Token must support.
Apply Token in the relevant workflow while recording its inputs, configuration, and outputs.
Evaluate the result against representative data, operational constraints, and human review before expanding production use.
Token-efficient enterprise AI workflows.
Fluid AI's orchestration layer manages token budgets across long enterprise workflows, chunking documents and caching context to maximise quality while minimising compute cost.
Explore ArchitectureTopics Covered
- LLM token enterprise cost
- token count context window
- tokenization LLM enterprise
- API token cost enterprise AI
- token budget RAG system
- LLM token economics enterprise
- context window token management
- multilingual token count enterprise