Tokens and Context Windows: A Versioned Engineering Guide

The Engineering Model

A Token is an identifier produced by a particular tokenizer. A context window is a provider/model-specific limit over a request and its generated output. Neither is a universal unit like a character or a word.

For a request, the usable input budget is approximately:

window_limit
- system/instruction tokens
- conversation and evidence tokens
- reserved output tokens
- protocol/tool overhead

The exact accounting depends on the provider, message format, tool schema, images, cached prefixes, and safety wrappers. Treat the provider’s usage fields and limits as authoritative.

Tokenization Is Model-Specific

BPE, WordPiece, Unigram, and byte-level variants segment text differently. The same string can have different token IDs and counts across models, revisions, or encodings. A rough character ratio is useful only for an early estimate; it is not suitable for a budget or invoice.

Tokenization can be affected by:

• language and script;
• whitespace and punctuation;
• Unicode normalization;
• code, URLs, JSON, and identifiers;
• special tokens and chat templates;
• tokenizer revision.

Do not say that one language always uses a fixed number of characters per token. Measure representative data with the tokenizer used in production.

Count With the Actual Tokenizer

Use the library and encoding documented for the target model:

from dataclasses import dataclass
from typing import Protocol

class Tokenizer(Protocol):
    def encode(self, text: str) -> list[int]: ...
    def decode(self, tokens: list[int]) -> str: ...

@dataclass(frozen=True)
class Count:
    characters: int
    tokens: int

def count_text(text: str, tokenizer: Tokenizer) -> Count:
    tokens = tokenizer.encode(text)
    return Count(characters=len(text), tokens=len(tokens))

This fragment deliberately does not import a provider SDK. A real counter must use the exact model encoding and account for message, tool, image, and template overhead. Record tokenizer/model revision with each measurement.

Round-Trip Tests

Some tokenizers do not preserve arbitrary text byte-for-byte through a naïve decode. Test representative Unicode, code, JSON, and delimiter cases before using decode for truncation. A token boundary is not necessarily a safe semantic boundary.

Context-Window Metadata

Do not embed a permanent comparison table in application logic or documentation. Store a versioned capability record:

{
  "provider": "recorded-provider",
  "model": "recorded-model@revision",
  "input_limit_tokens": "verified-from-current-docs",
  "output_limit_tokens": "verified-from-current-docs",
  "tokenizer": "recorded-encoding@revision",
  "tool_overhead": "measured-by-harness",
  "price_table": "provider-price-2026-07-01",
  "checked_at": "recorded-time"
}

A marketed maximum may differ from an API route, deployment region, subscription tier, or effective output limit. Verify the endpoint you actually call.

Budget the Request

Reserve output before selecting input:

def available_input(
    window_limit: int,
    system_tokens: int,
    history_tokens: int,
    tool_tokens: int,
    reserved_output: int,
) -> int:
    used = system_tokens + history_tokens + tool_tokens + reserved_output
    return max(0, window_limit - used)

Fail clearly when the budget is insufficient. Do not silently drop the system policy, authorization context, or the middle of a structured record. Prefer selection, pagination, retrieval, or a reviewed summary over arbitrary truncation.

Safe Truncation and Chunking

Truncating by token count can split a code block, JSON value, URL, Unicode sequence, or permission condition. Safer options are:

• split at a parser or document boundary;
• keep headers and source identifiers with each chunk;
• preserve overlap only when its cost and duplication are measured;
• include a chunk revision and source span;
• validate that required claims remain present;
• reject an incomplete structured object instead of guessing.

For retrieval, chunk size and overlap are workload parameters. Evaluate evidence recall, answer support, latency, storage, and duplicate retrieval rather than using a universal number.

Long-Context Behavior

A larger limit does not guarantee better use of information. Quality can degrade through:

• irrelevant or contradictory evidence;
• stale state;
• attention dilution or positional sensitivity;
• lost delimiters and source boundaries;
• increased latency and cost;
• prompt injection surface.

“Lost in the middle” describes observed positional sensitivity in some settings, not a universal law that placing a task at the end fixes. Test ordering with the model, task, and evidence distribution you use.

RoPE, ALiBi, sliding-window attention, recurrence, retrieval, and compression are implementation choices with different tradeoffs. They are not interchangeable promises of long-context comprehension. This article focuses on the application budget; consult the model architecture paper and current documentation for implementation details.

Conversation History

Keep history as structured state:

{
  "message_id": "m-17",
  "role": "user",
  "created_at": "recorded-time",
  "content_ref": "approved-store://message/m-17",
  "tenant_id": "tenant-a",
  "visibility": "conversation",
  "superseded_by": null
}

Summaries should preserve source references, unresolved questions, decisions, and uncertainty. They should have an owner, purpose, retention, and deletion path. A summary cannot override current policy or identity.

Cost Accounting

Token estimates are not bills. Reconcile:

• input, output, cached, and reasoning usage when exposed;
• tool and downstream charges;
• retries, cancellations, and failed requests;
• currency, discounts, minimums, and price-table revision;
• storage, retrieval, and observability costs.

def estimated_charge(
    input_tokens: int,
    output_tokens: int,
    input_rate: float,
    output_rate: float,
) -> float:
    if min(input_tokens, output_tokens) < 0:
        raise ValueError("token_counts_must_be_non_negative")
    return (
        input_tokens * input_rate
        + output_tokens * output_rate
    )

Keep the currency, rate unit, source, and estimated flag. Compare cost per successful task, not only cost per request or per million tokens.

Caching

Caching may reduce repeated work, but its behavior is provider- and workload-specific. Verify:

• cache key and prefix boundaries;
• TTL and invalidation after policy/source changes;
• tenant isolation and sensitive-data handling;
• billing and hit/miss semantics;
• stale-result behavior and fallback.

Measure hit rate, p50/p95 latency, cost, and incorrect reuse. Do not promise a fixed saving.

Multilingual and Structured Data Testing

Build a corpus that represents production:

• languages and scripts;
• punctuation, emoji, and normalization;
• source code and stack traces;
• JSON, CSV, Markdown, URLs, and long identifiers;
• adversarial delimiters and injection text.

For each tokenizer revision, record counts and budget failures. A language “efficiency” claim without corpus, tokenizer, normalization, and confidence information is not portable.

Context Security

Token budgeting does not provide security. Enforce separately:

• identity, tenant, and object authorization;
• source access before retrieval;
• secret and personal-data minimization;
• bounded tool results and memory writes;
• command and network policy;
• provenance and taint for untrusted content;
• deletion propagation to indexes, caches, backups, and exports.

Do not put bearer tokens, private keys, or unrestricted production records in a prompt merely because they fit the window.

Evaluation Harness

Test a context strategy with:

Keep raw samples and model/provider metadata. Use deterministic checks for structure and permissions; use calibrated human or model review for open-ended quality.

Common Failure Modes

• using a character-per-token ratio as a production budget;
• copying an old model window or price table;
• counting text but ignoring chat/tool/image overhead;
• reserving no output budget;
• truncating in the middle of a permission or structured record;
• assuming a large window understands an entire codebase;
• treating a summary, cache, citation, or tokenizer as authorization;
• caching across tenants or after policy changes;
• optimizing token count while hiding failed tasks and rework;
• recommending a formatting or encoding tool unrelated to the current decision.

Checklist

• [ ] Record model, endpoint, tokenizer, limit, template, and price revisions.
• [ ] Count representative production-shaped fixtures with the actual tokenizer.
• [ ] Reserve output and protocol/tool overhead before selecting input.
• [ ] Select or chunk at semantic and permission boundaries.
• [ ] Preserve source IDs, revisions, uncertainty, and deletion metadata.
• [ ] Treat summaries, caches, and retrieved text as non-authoritative input.
• [ ] Test multilingual, code, JSON, Unicode, and injection cases.
• [ ] Reconcile provider usage with a versioned price table.
• [ ] Measure successful-task cost, quality, latency, and cache behavior.
• [ ] Enforce identity, access, secrets, tools, and side effects outside the model.

Conclusion

Tokens and context windows are implementation constraints, not quality guarantees. Use the actual tokenizer and endpoint metadata, reserve output and protocol overhead, preserve semantic and permission boundaries, and reconcile usage with provider records. A large window can support a good system, but only workload evidence can show whether more context, chunking, compression, or caching improves the result.

Primary Sources

• SentencePiece: A simple and language independent subword tokenizer
• BERT: Pre-training of Deep Bidirectional Transformers
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• ALiBi: Train Short, Test Long
• NIST AI Risk Management Framework