Retries and cut-off answers
· 2 min read
Two kinds of waste hide in almost every LLM application: requests that fail and are sent again, and answers that stop at the output limit and are asked again. Each looks small. Together they are often a steady few percent of the bill, and they slow your product down.
Retries
A retry is the same request sent again after the first attempt failed: a timeout, a rate limit (HTTP 429), a server error (5xx), or a dropped connection.
What it costs. Providers do not bill most error responses. But a client timeout is different: your code gave up, while the provider may have finished the work and billed it. Long answers with short timeouts are the usual cause. Retries also cost time: a user waits for two attempts.
How to spot it. Count calls whose request is identical to a failed call made in the previous minute. Report it as a retry rate per feature ("18% of checkout-bot calls are retries"), not only in dollars, because the dollars understate the problem.
How to fix it.
- Set timeouts from real response times. A long answer can take tens of seconds.
- On 429, back off with growing waits, and lower your concurrency. Retrying at once makes the rate limit worse.
- Do not retry errors that will fail again: a request that is too long, invalid parameters, a refused request.
- Mind double retries. The OpenAI and Anthropic Python clients already retry some errors themselves (
max_retries). A retry loop of your own on top multiplies them.
Cut-off answers
An answer is cut off when it reaches the output limit you set (max_tokens). It stops mid-sentence or mid-JSON. The signal is in the response:
| Provider | Field | Value when cut off |
|---|---|---|
| OpenAI Chat Completions | finish_reason | length |
| OpenAI Responses | status, incomplete_details.reason | incomplete, max_output_tokens |
| Anthropic | stop_reason | max_tokens |
What it costs. If your code notices and asks again, often with a higher limit, the first answer was paid for and thrown away. If it does not notice, you ship a broken answer, which is worse.
How to fix it.
response = client.chat.completions.create(model=model, messages=messages, max_tokens=800)
if response.choices[0].finish_reason == "length":
log.warning("answer cut off", extra={"feature": "summary", "max_tokens": 800})- Log every cut-off with the feature name, and look at how often it happens.
- Size
max_tokensfrom real answer lengths for each feature, with room to spare. - If answers are often too long, ask for a shorter answer in the prompt rather than raising the limit forever.
- For structured output, keep the schema small; long JSON is the most common thing to get cut.
How Zexl finds both
Zexl reads the finish reason, status code and error class of every call (never the text). A failed call followed by the same request within 60 seconds is a retry; Zexl shows both the retry rate and the dollars. An answer that hit the output limit and was requested again within 10 minutes is cut off and redone, and the cut-off attempt is counted as waste. See waste patterns.
Create a free account, or see How to reduce your LLM API costs.
See where your own AI spend goes
One import in a Python app, or five environment variables for Claude Code. Prompts never leave your servers.