Local AI: A Complete Guide to Running Language Models on Your Computer
Local AI means running a language model on hardware you control instead of sending prompts to a hosted service. This guide explains how local inference works, how…
TAGGED GUIDES
#quantization
Explore the guides, ideas, and practical explanations in this topic.
10 published articlesHow Context Length Affects Local AI Memory Usage
Raising a local model's context window from 2K to 32K changed generation speed by essentially nothing in a published llama.cpp sweep, but added about 1.7 GB of…
What Quantization Changes in a Language Model (and What It Leaves Alone)
Quantization rewrites a model's weights in lower precision while leaving its architecture, tokenizer, and parameter count intact. The memory savings are predictable; the quality cost is not,…
How to Choose a Local AI Model for Your Computer
Choosing a local AI model is a hardware-fit problem before it is a model-quality problem. Define the job, find your real memory budget, estimate weights plus KV…
Why a Local AI Model Runs Slowly: A Troubleshooting Checklist
A slow local AI model is usually not one problem. Separate model loading, prompt processing, and token generation first, then check GPU offload, context length, quantization, and…
RAM vs VRAM for Local AI: What Each One Does
When you run a local LLM, VRAM is where inference happens on the GPU, while system RAM provides capacity for loading models, CPU inference, and offloading. Both…
How to Compare Local Language Models with Reproducible Tests
A reproducible local model comparison pins the model, runtime, hardware, prompt template, context settings, and sampling parameters before measuring quality, latency, memory, and failure rates. This guide…
AI Hardware Explained: RAM, VRAM, Quantization, and Model Size
Running a local language model is mostly a memory-planning problem. This guide explains how parameter count, quantization, VRAM, RAM, context length, and runtime overhead determine whether a…
How Large Language Models Work: Tokens, Context, Training, and Inference
Large language models are next-token predictors, and almost every practical constraint they have follows from four ideas: how text becomes tokens, how the context window limits what…
How to Record a Reproducible Local Model Benchmark
A reproducible local model benchmark is a record, not a number. It needs the exact model bytes and hashes, the runtime version, the hardware and power state,…