AIForum.

How to Handle Token Limits in Multi-Turn Chatbots

On this page

THE SHORT ANSWER

Multi-turn chatbots hit token limits because every turn re-sends the whole conversation inside a single shared context budget. The fix is a budgeting discipline: measure prompt tokens per turn, reserve room for the reply, and choose deliberately between sliding windows, rolling summaries, and retrieval over replay.
Quick facts

Multi-turn chatbots run into token limits because every turn re-sends the whole conversation, and the model’s context window is a single shared budget covering the system prompt, tool definitions, the full message history, and the reply. The practical answer is a budgeting discipline rather than one setting: measure usage on every request, reserve room for the response, and decide deliberately what leaves the context — the oldest turns, a compressed summary of them, or retrieved fragments pulled back in on demand.

This guide is written for developers and technically curious builders running chat assistants through a hosted API or a local runtime. It covers how the limits are defined, how to measure them, which history strategies hold up in real conversations, and where each one fails.

What a token limit actually constrains

A model’s context window is the maximum number of tokens it can process in a single request, counting input and output together. That ceiling is per model, and the maximum output is a separate, usually smaller, number. ILMU’s API documentation lists, for example, ilmu-glm-5.3 with a 1,000,000-token context window and a 128,000-token maximum output, while ilmu-vision-v1.3 is listed at 128,000 tokens of context with a 32,000-token maximum output (ILMU, Context Windows).

Two details matter more than the headline numbers. First, the window is shared: the same documentation states that the context window is shared between your input (system prompt, messages, tool definitions) and the model’s output, so a 150,000-token input on a 200,000-token model leaves at most 50,000 tokens for the response even though the model’s maximum output is 128,000. Second, not every model counts the same way — ILMU notes that its two text-to-speech models cap at 10,000 characters per request rather than tokens, and that embedding and reranking models consume a context window but return vectors and scores instead of generated tokens.

The practical consequence: a chatbot does not have “a 200K memory.” It has a 200K per-request budget that must simultaneously hold everything the model needs to see and everything it is allowed to say.

Why multi-turn conversations grow faster than expected

Stateless chat APIs require the client to resend the conversation each time. Turn one might cost 800 tokens; turn twenty costs the system prompt plus nineteen prior exchanges plus the new message. Growth is roughly linear in turn count and in the length of each turn, which is why a chat that feels fine in testing can fail after a long support session or a pasted document.

For illustration, consider a chatbot with a 1,200-token system prompt, 900 tokens of tool definitions, and turns averaging 350 tokens. By turn ten the prompt is roughly 1,200 + 900 + (10 × 350) ≈ 5,600 tokens. By turn one hundred it is roughly 37,100 tokens, before any retrieved documents. These numbers are arithmetic, not measurements, but they show the shape of the problem: history, not the system prompt, dominates over long sessions.

Three components are easy to underestimate:

  • System prompts and instructions. ILMU’s guidance is explicit that system prompts are included in every request and that a lengthy system prompt eats into the space available for conversation.
  • Tool and function definitions. These are part of the input on every call, and verbose schemas can consume thousands of tokens before the user says anything.
  • Tool results and retrieved documents. A single search result or file excerpt can be larger than the entire human conversation around it.

Step 1: Measure before you trim

Every response from a metered API includes a usage object. ILMU documents the shape as:

{
  "usage": {
    "prompt_tokens": 1250,
    "completion_tokens": 483,
    "total_tokens": 1733
  }
}

Log prompt_tokens for every turn, not just the total. The prompt figure is what tells you how close the next request is to the ceiling, and it is the number your trimming policy has to control. A useful habit is to record prompt tokens alongside turn index and the strategy in use, so a regression in a summarizer or a retrieval step shows up as a curve rather than a surprise error.

Local runtimes usually expose equivalent counters through their server logs or API responses. If your runtime does not, estimate with the model’s tokenizer rather than dividing characters by four — the ratio varies by language and content type, and code and JSON are especially token-dense.

Step 2: Budget the window explicitly

Write the budget down before writing the trimming code. The following is an illustrative allocation for a 200,000-token model, not a recommendation from any vendor:

ComponentIllustrative budget (tokens)
System prompt and tool definitions4,000
Retrieved context20,000
Conversation history96,000
Reserved for the response32,000
Headroom for estimation error48,000

Two rules keep this honest. Reserve the response budget first, because a request that fills the window with input leaves the model no room to answer. Then keep real headroom: token counters differ slightly between tokenizers and providers, and a budget with no slack turns a rounding difference into a failed request.

Step 3: Choose a history strategy

Sliding window

Keep the system prompt, the most recent N turns, and drop everything older. It is cheap, predictable, and easy to reason about. Its weakness is that anything the user established early — a name, a constraint, a decision — disappears without warning.

Rolling summarization

When history exceeds a threshold, ask the model to compress the oldest turns into a summary and replace them with it. This preserves continuity far longer than a sliding window. The costs are an extra model call, added latency, and drift: summaries lose specifics, and errors compound each time a summary is summarized.

Retrieval over replay

Instead of carrying the whole conversation, store turns externally and retrieve only the passages relevant to the current message. This scales to long-lived assistants and keeps the prompt small. It depends on the retrieval step actually finding the right material, and it adds an embedding or search component to maintain.

Hybrid policies

Most production chatbots combine these: a sliding window for recent turns, a pinned block for durable facts, a summary for the middle, and retrieval for anything that must be recovered on demand. Pin facts explicitly rather than hoping a summarizer preserves them.

Step 4: Cap both sides of the exchange

Trimming input alone is not enough. Set a maximum output length so a single verbose reply cannot consume the reserve, and cap the size of tool results and retrieved chunks before they enter the prompt. A retrieval step that returns ten documents when three would do is a token leak that no history policy will fix.

Step 5: Test the policy reproducibly

Trimming rules break silently, so test them against a fixed transcript. OpenAI’s advanced usage documentation explains that Chat Completions are non-deterministic by default and that you can move toward mostly deterministic outputs by setting the seed parameter to a fixed integer, keeping all other parameters identical, and using the system_fingerprint response field to detect when the backend configuration changes (OpenAI, Advanced usage).

That combination is useful for comparing two trimming strategies on the same input, but it is not a guarantee. Treat any single comparison run as weak evidence and repeat it, and expect determinism to be affected by factors outside your control.

A minimal implementation pattern

The following is illustrative pseudocode, not a tested implementation:

def build_prompt(system, tools, history, new_message, budget):
    fixed = count(system) + count(tools)
    reserve = budget.response
    available = budget.window - fixed - reserve - budget.headroom

    # Pinned facts always survive
    pinned = [m for m in history if m.pinned]
    recent = [m for m in history if not m.pinned]

    # Drop oldest turns until the prompt fits
    while count(pinned) + count(recent) + count(new_message) > available:
        if len(recent) > 1:
            recent.pop(0)
        else:
            recent[0] = summarize(recent[0])   # last resort
            break

    return [system] + pinned + recent + [new_message]

Note the order of last resort: drop first, summarize second. Summarizing on every overflow hides the fact that your budget is too tight.

Failure modes and tradeoffs

  • Silent quality loss. The most common failure is not an error but a chatbot that forgets a constraint. Log which turns were dropped so you can correlate complaints with trimming events.
  • Summarization drift. Each compression is lossy. Keep the original transcript in storage so a summary can be regenerated from the source rather than from another summary.
  • Retrieval misses. A retrieved-context design fails when the relevant turn is never surfaced. Measure retrieval recall on real conversations, not on synthetic queries.
  • Cost and latency of memory maintenance. Summarization and embedding calls are extra requests. Budget for them in both money and response time.
  • Provider differences. Exact error behavior when a request exceeds the window varies by provider and should be checked in that provider’s documentation rather than assumed.

What to do next

Start by instrumenting prompt_tokens per turn on your current chatbot and plotting it against turn index. That single measurement usually reveals whether you need a simple cap, a summarizer, or a retrieval layer. Then set an explicit budget, implement the cheapest strategy that fits, and replay a fixed transcript to confirm the behaviour before and after the change.

If you are working through a specific limit — a local model with a small window, a runaway tool output, or a summarizer that keeps losing detail — bring your model, runtime, and observed token counts to the community and compare notes with others running similar setups.

FAQ

Does the context window include the assistant's reply?

Yes. The window is shared between input and output, so a long prompt directly reduces the space available for the response.

What happens when a request exceeds the context window?

The request cannot be served as-is. The exact behavior and error text vary by provider and runtime, so check your provider's documentation rather than assuming silent truncation.

Can I make trimming tests fully reproducible?

Not fully. OpenAI documents that outputs are non-deterministic by default and that the seed parameter plus a fixed system_fingerprint can move you toward mostly deterministic results when all other parameters match.

Is summarization always better than dropping old turns?

No. Summarization preserves continuity but adds latency, cost, and drift. Dropping turns is cheaper and more predictable but loses early constraints entirely.

References

  1. https://docs.ilmu.ai/docs/capabilities/context-windows
  2. https://platform.openai.com/docs/advanced-usage

Related terms

Context window
The maximum number of tokens a model can process in one request, counting input and output together.
Prompt tokens
The tokens counted for everything sent to the model: system prompt, messages, tool definitions, and retrieved context.
Completion tokens
The tokens the model generates in its reply, counted separately in the usage object.
Sliding window
A history strategy that keeps only the most recent turns and discards older ones.
Rolling summarization
Replacing older conversation turns with a model-generated summary once history exceeds a threshold.
KV cache
The attention key and value state a runtime keeps for tokens already processed; it grows with context length and consumes memory beyond the model weights.

Leave a Reply

Your email address will not be published. Required fields are marked *