How Context Length Affects Local AI Memory Usage
Raising a local model's context window from 2K to 32K changed generation speed by essentially nothing in a published llama.cpp sweep, but added about 1.7 GB of…
TAGGED GUIDES
#context window
Explore the guides, ideas, and practical explanations in this topic.
4 published articlesWhat Are Tokens in AI? A Practical Explanation
A token is the unit of text an AI model actually reads and writes — it can be a single character or a whole word, depending on…
What Is an AI Context Window? Tokens, Limits, and Practical Workarounds
A context window is the amount of input an AI system can process at one time, measured in tokens and including the response it is generating. This…
How Large Language Models Work: Tokens, Context, Training, and Inference
Large language models are next-token predictors, and almost every practical constraint they have follows from four ideas: how text becomes tokens, how the context window limits what…