Retries and cut-off answers

· 2 min read

Two kinds of waste hide in almost every LLM application: requests that fail and are sent again, and answers that stop at the output limit and are asked again. Each looks small. Together they are often a steady few percent of the bill, and they slow your product down.

Retries

A retry is the same request sent again after the first attempt failed: a timeout, a rate limit (HTTP 429), a server error (5xx), or a dropped connection.

What it costs. Providers do not bill most error responses. But a client timeout is different: your code gave up, while the provider may have finished the work and billed it. Long answers with short timeouts are the usual cause. Retries also cost time: a user waits for two attempts.

How to spot it. Count calls whose request is identical to a failed call made in the previous minute. Report it as a retry rate per feature ("18% of checkout-bot calls are retries"), not only in dollars, because the dollars understate the problem.

How to fix it.

  • Set timeouts from real response times. A long answer can take tens of seconds.
  • On 429, back off with growing waits, and lower your concurrency. Retrying at once makes the rate limit worse.
  • Do not retry errors that will fail again: a request that is too long, invalid parameters, a refused request.
  • Mind double retries. The OpenAI and Anthropic Python clients already retry some errors themselves (max_retries). A retry loop of your own on top multiplies them.

Cut-off answers

An answer is cut off when it reaches the output limit you set (max_tokens). It stops mid-sentence or mid-JSON. The signal is in the response:

ProviderFieldValue when cut off
OpenAI Chat Completionsfinish_reasonlength
OpenAI Responsesstatus, incomplete_details.reasonincomplete, max_output_tokens
Anthropicstop_reasonmax_tokens

What it costs. If your code notices and asks again, often with a higher limit, the first answer was paid for and thrown away. If it does not notice, you ship a broken answer, which is worse.

How to fix it.

python
response = client.chat.completions.create(model=model, messages=messages, max_tokens=800)
if response.choices[0].finish_reason == "length":
    log.warning("answer cut off", extra={"feature": "summary", "max_tokens": 800})
  • Log every cut-off with the feature name, and look at how often it happens.
  • Size max_tokens from real answer lengths for each feature, with room to spare.
  • If answers are often too long, ask for a shorter answer in the prompt rather than raising the limit forever.
  • For structured output, keep the schema small; long JSON is the most common thing to get cut.

How Zexl finds both

Zexl reads the finish reason, status code and error class of every call (never the text). A failed call followed by the same request within 60 seconds is a retry; Zexl shows both the retry rate and the dollars. An answer that hit the output limit and was requested again within 10 minutes is cut off and redone, and the cut-off attempt is counted as waste. See waste patterns.

Create a free account, or see How to reduce your LLM API costs.

See where your own AI spend goes

One import in a Python app, or five environment variables for Claude Code. Prompts never leave your servers.