How to Handle Timeouts and Retries for Local LLM APIs (Ollama, LM Studio)

Local LLM APIs fail differently from cloud APIs. When a hosted endpoint hangs, the cause is usually the network or the provider; when Ollama or LM Studio hangs, the machine is often still working — loading weights into VRAM, allocating a KV cache, or evaluating a long prompt before the first token appears. The practical fix is to treat those as separate failure classes: give the read timeout enough room to cover model load plus generation, and reserve retries for errors that are genuinely transient.

This guide is aimed at developers and self-hosters calling a local server over HTTP — Ollama’s native /api/chat and /api/generate endpoints, or the OpenAI-compatible /v1/chat/completions surface that both Ollama and LM Studio expose. It covers how to classify errors, how to size timeouts, how to add backoff, and when retrying makes things worse.

Why a local timeout is not automatically an error

With a hosted API, latency is mostly network transit plus queueing on someone else’s hardware. With a local runtime, several extra stages sit between the HTTP request and the first token: the server may have to load model weights into memory, allocate KV cache for the requested context length, and run prompt evaluation before generating anything. Requests can also queue behind other in-flight requests when the server has a limited number of parallel slots.

That means a timeout on a local endpoint is frequently a budget problem rather than a failure. If the same request succeeds when you wait longer, the correct fix is a larger read timeout, not a retry loop. If the request fails again with the model already resident, you are probably looking at a deterministic error — out of memory, a context length the loaded model cannot serve, or a malformed payload — and retrying will simply repeat it.

Classify the failure before you retry

The ollama-proxy retry and fallback documentation draws a useful line between transient and non-transient conditions, and it is a reasonable default taxonomy for any local setup:

• Transient: HTTP 429 and 5xx responses, connection errors, timeouts, and provider-specific errors flagged as transient.
• Non-transient: 4xx responses other than 429, validation errors, and authentication failures. In ollama-proxy these fail immediately — no retry and no fallback.

For a local server you should extend that list with conditions the HTTP status code does not capture:

Configure two timeouts, not one

Most HTTP clients separate a connect timeout from a read timeout, and that distinction matters far more for local endpoints than for remote ones. Because the server is on localhost or your LAN, connect should be short — a few seconds is generous. The read timeout is the one that needs care, because it has to cover everything from request acceptance to the last token.

A workable rule is: read timeout ≥ model load time + prompt evaluation time + expected generation time, plus margin. Measure those three numbers for your typical prompt instead of guessing; the diagnostic steps later in this article show how.

Equally important is the total budget across attempts. The Instructor integration guide for Ollama documents its timeout parameter as a total across all retry attempts rather than a per-attempt limit, so a 3-second timeout stays 3 seconds in total instead of multiplying to 9 seconds or more. Apply the same reasoning to any wrapper you write: three attempts with a 300-second read timeout each means a worst case of roughly 15 minutes plus backoff, which is rarely what a caller expects.

Backoff, jitter, and retry budgets

ollama-proxy uses exponential backoff capped at 10 seconds, producing a sequence of 1s, 2s, 4s, 8s, 10s, 10s, and so on. Capping the delay matters more than the exact base, because an uncapped exponential can leave a client sleeping for hours after a handful of failures.

For local servers, aggressive backoff is usually not the bottleneck — the model load is. A short delay of one or two seconds between attempts, combined with a generous read timeout, generally behaves better than many fast retries against a server that is still paging weights into memory. Adding a small random jitter to each delay prevents several clients from retrying in lockstep after a shared outage.

An illustrative retry wrapper for Ollama

The following example is illustrative rather than a tested artifact: it shows how the pieces — status classification, a connect/read timeout pair, capped backoff, and a final re-raise — fit together. Adapt the endpoint and model name to your setup.

import random
import time

import requests

RETRYABLE_STATUS = {429, 500, 502, 503, 504}

def chat_with_retry(prompt, model='llama3.2', attempts=3, read_timeout=300):
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': prompt}],
        'stream': False,
    }
    for attempt in range(attempts):
        try:
            response = requests.post(
                'http://localhost:11434/api/chat',
                json=payload,
                timeout=(5, read_timeout),  # (connect, read)
            )
            if response.status_code in RETRYABLE_STATUS:
                raise requests.HTTPError(f'retryable status {response.status_code}')
            response.raise_for_status()
            return response.json()
        except (requests.ConnectionError, requests.Timeout, requests.HTTPError):
            if attempt == attempts - 1:
                raise
            delay = min(10, 2 ** attempt) + random.uniform(0, 0.5)
            time.sleep(delay)

Expected behaviour: a cold-start timeout on the first attempt is absorbed by the second attempt once the model is resident, while a 404 for an unknown model propagates immediately because raise_for_status() raises and the status is not in the retryable set.

Structured output retries are a different mechanism

If you use Instructor with Ollama for JSON-schema-constrained responses, its max_retries parameter retries when the model returns output that fails validation — a semantic failure, not a transport one. The same call also accepts a total timeout, which the documentation notes is especially important for Ollama because it prevents retries from multiplying the total wait time.

from pydantic import BaseModel
import instructor

class Character(BaseModel):
    name: str
    age: int

client = instructor.from_provider('ollama/llama2', mode=instructor.Mode.JSON)

resp = client.create(
    messages=[{'role': 'user', 'content': 'Tell me about Harry Potter'}],
    response_model=Character,
    max_retries=2,
    timeout=10.0,  # total timeout across all retry attempts
)

Keep the two retry types separate in your design. A validation retry re-sends the prompt and costs another full generation; a transport retry may cost nothing if the first request never reached the model. Mixing them into one counter makes it hard to reason about worst-case latency.

Proxy-level retries and fallback

If you would rather not implement retry logic in every client, a proxy in front of the local server can own it. The ollama-proxy configuration is per provider and applies to both Ollama-style and OpenAI-style endpoints:

providers:
  openai:
    retries: 2
    fallback: groq/llama-3.3-70b-versatile
  myvllm:
    api_key: EMPTY
    base_url: http://localhost:8000/v1
    retries: 3   # retries only, no fallback

Two documented behaviours are worth internalising before you enable a fallback. First, the fallback gets exactly one attempt — there is no recursive retry on the fallback itself. Second, the client sees a successful response from the fallback with no special indicator that the primary failed, so your own logging is the only place the substitution will be visible. The retries field defaults to 0, meaning no retry unless you ask for one.

Fallback also has a privacy dimension that the configuration file does not express. A local-first setup exists partly because prompts stay on your machine; a fallback to a hosted provider moves that data off the device. If you enable fallback for a local backend, decide deliberately which request types may leave the machine — and consider whether the fallback belongs behind an explicit opt-in per workload rather than as a global default.

Diagnosing a timeout step by step

Work from the cheapest, least invasive check to the most involved:

1. Time the raw request. Send the same payload with curl and a timing template. If it completes in 90 seconds when your client allows 30, the timeout is simply too small. Example:

   curl -s -o /dev/null -w 'connect=%{time_connect}s total=%{time_total}sn' 
     http://localhost:11434/api/chat 
     -H 'Content-Type: application/json' 
     -d '{"model":"llama3.2","messages":[{"role":"user","content":"hi"}],"stream":false}'

2. Check whether the model is loaded. Ollama lists currently resident models via /api/ps. If the model is absent, you are paying a cold-start cost on the first request after idle.
3. Read the server log. Look for model load failures, memory allocation errors, or repeated load/unload cycles. These are deterministic and retrying will not clear them.
4. Shrink the prompt. A tiny prompt that succeeds where a large one times out points at prompt evaluation time or context length rather than the server being down.
5. Test one request at a time. If sequential requests are fast but concurrent ones time out, you are queueing behind limited parallel slots. Ollama exposes queue and parallelism controls such as OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE; check your installed version’s documentation before changing them.
6. Check hardware pressure. If the model does not fit in VRAM and spills to system memory, generation slows dramatically and timeouts become routine. This is a capacity problem, not a retry problem.

For LM Studio, the same sequence applies; its local server also loads models on demand and queues requests, so the first request after a model has been unloaded is the slowest. Exact queue and timeout controls live in the server settings of your installed version, so verify them there rather than assuming defaults.

Streaming, duplicate work, and idempotency

Streaming changes the failure mode. With stream: true, the read timeout applies to the gap between chunks rather than to the whole response, so a stalled stream can be detected quickly — but a retry after partial output has already been delivered will produce duplicate tokens unless the client discards what it received. For interactive chat that is usually acceptable; for anything that writes to a database or triggers a downstream action, it is not.

Retries also cost real compute. Every retried generation re-runs prompt evaluation and sampling, which on a single-GPU machine competes with whatever else is running. Keep retry counts low for local endpoints — two or three attempts is usually enough to absorb a cold start — and prefer fixing the timeout over adding attempts.

What retries cannot fix

Retrying is a remedy for transient availability and transport problems. It does not fix a model that is too large for your GPU, a context window that the loaded model cannot serve, an invalid API payload, or output that is subtly wrong. In ollama-proxy’s taxonomy, validation errors and non-429 4xx responses fail immediately for exactly this reason. If your logs show the same error text across every attempt, stop retrying and change the configuration instead.

A practical starting point

Measure how long your typical prompt takes end to end with the model already loaded, then add the cold-start time you observe after an idle period. Set the read timeout above that sum, keep the connect timeout short, and enable at most two or three retries with capped backoff. If cold starts are the dominant cost, keep the model resident — Ollama accepts a keep_alive parameter on requests and a corresponding server-level setting — and the timeouts will stop firing in the first place.

If you have a timeout pattern that does not match any of the categories above, share your runtime, model, context length, hardware, and the exact error text in the community so others can compare it against their own setups.