# What Zexl is Zexl shows how much of your AI model spend did useful work, and how much was wasted. It names the waste in plain words, puts a dollar figure on each kind, and shows you the runs behind it. Zexl only records **metadata** about each model call: the model, token counts, timings, the outcome, and salted hashes. **Prompt and response text never leave your process.** See [What leaves your process](https://tryzexl.xyz/docs/privacy.md). ## What you get - **Useful spend share.** Of every dollar, how much produced something you used. - **Waste by pattern.** Retries, loops, abandoned responses, answers cut off and redone, and prompt caching not used. See [Waste patterns](https://tryzexl.xyz/docs/patterns.md). - **Spend by feature, model, customer and person.** - **Run detail.** One agent run, step by step, with the repeated or failed steps marked. - **Budget and forecast.** Where the month is likely to end, and when it passes your budget. See [Forecast and budget](https://tryzexl.xyz/docs/forecast.md). - **Ask your AI assistant.** The [MCP server](https://tryzexl.xyz/docs/mcp.md) answers "what did we waste last week?" inside Claude Code, Claude Desktop or Cursor. ## How it works 1. Your app sends call metadata to Zexl, from the [Python SDK](https://tryzexl.xyz/docs/quickstart.md) or from [Claude Code's telemetry](https://tryzexl.xyz/docs/claude-code.md). 2. Zexl prices each call with the providers' published prices (or your own). 3. Every 2 minutes, Zexl sorts each call into one category: useful, or one of the waste patterns. 4. The dashboard and the MCP server show the result. ## Ways to connect | Way | Best for | Effort | |---|---|---| | [Python SDK](https://tryzexl.xyz/docs/quickstart.md) | Apps that use the OpenAI or Anthropic Python clients | One import and one environment variable | | [With your AI agent](https://tryzexl.xyz/docs/connect-with-ai.md) | Letting Claude Code or Cursor set it all up | Approve one link | | [Claude Code](https://tryzexl.xyz/docs/claude-code.md) | Seeing what your own coding sessions cost | Five environment variables | | [Gateway](https://tryzexl.xyz/docs/gateway.md) | Any language, no code change, and guardrails that can stop calls | Run one container; apps change the base URL and key only | Supported today: OpenAI and Anthropic. The SDK is for Python; the gateway works with any language. ## Guides Practical guides on cutting AI spend, whether or not you use Zexl: [How to reduce your LLM API costs](https://tryzexl.xyz/guides/reduce-llm-api-costs.md), [prompt caching](https://tryzexl.xyz/guides/prompt-caching.md), [agent loops](https://tryzexl.xyz/guides/ai-agent-loops.md), and [all guides](https://tryzexl.xyz/guides). --- # Quickstart: Python SDK About 5 minutes. You need a Python app that calls OpenAI or Anthropic with their official Python packages. ## 1. Create a workspace and a key 1. [Create an account](https://tryzexl.xyz/sign-up) with email or Google. 2. Name your workspace. One workspace is one team or product. 3. Choose **Python SDK**, choose the environment (usually **Production**), and press **Create key**. 4. Copy the key. It looks like `zx__.` and is shown once. The part after the dot is the **salt**. It is made in your browser and Zexl never receives it. The SDK uses it to hash prompts locally, so nobody, including Zexl, can match a hash to text. ## 2. Install and turn it on ```bash pip install zexl ``` ```bash export ZEXL_API_KEY="zx_..." ``` Add one import at your app's entry point, before the OpenAI or Anthropic client is created: ```python import zexl.auto # records OpenAI and Anthropic calls; prompts never leave the process ``` That is the whole setup. Typical entry points: `main.py` or `app.py` (FastAPI, Flask), `wsgi.py` or `asgi.py` (Django), the worker start module (Celery), the handler module (AWS Lambda). If you load `.env` in code (for example with `python-dotenv`), put the import after `load_dotenv()`. ## 3. Make a call and watch it arrive Run your app so it makes one model call. The setup page shows the call within a few seconds. Waste analysis follows within about 3 minutes. ## 4. Label your spend (recommended) ```python import zexl with zexl.feature("support-reply"): # which product feature made the calls ... with zexl.run("research", feature="report-builder"): # one unit of agent work ... zexl.set_customer(account.id) # your customer's id, never an email ``` Without labels, the feature is the `module.function` that made the call, and agent loops are grouped into runs automatically. See [Features, runs and customers](https://tryzexl.xyz/docs/labels.md). ## Short-lived processes Serverless functions, scripts and CLI tools should call `zexl.flush()` before they exit, so the last records are sent. ## Settings | Variable | Meaning | Default | |---|---|---| | `ZEXL_API_KEY` | Your key. Required | none | | `ZEXL_ENV` | Environment label. Ignored when the key has its own environment | `production` | | `ZEXL_DISABLED` | `1` turns the SDK off completely | off | | `ZEXL_OPENAI_STREAM_USAGE` | `1` asks OpenAI for token usage on streamed chat calls | off | ## What is covered | Provider | Calls | |---|---| | OpenAI | `chat.completions.create`, `.parse`, `.stream`; `responses.create`, `.parse`; streaming; sync and async | | Anthropic | `messages.create`, `.parse`, `.stream`; `beta.messages`; streaming; sync and async | Your calls are never blocked or changed. If Zexl is down, records are dropped and your app carries on. `zexl.status()` shows whether capture is on and how many records were sent. --- # Connect with your AI agent Let Claude Code, Cursor or another coding agent add Zexl to your project. You approve one link; the agent does the rest. The key never passes through the chat. ## 1. Add the Zexl MCP server to your agent Claude Code: ```bash claude mcp add zexl -- uvx zexl-mcp ``` Claude Desktop, Cursor and other tools with a JSON config: ```json { "mcpServers": { "zexl": { "command": "uvx", "args": ["zexl-mcp"] } } } ``` `uvx` comes with [uv](https://docs.astral.sh/uv/). ## 2. Ask > Add Zexl to this project. ## 3. What happens 1. The agent calls `zexl_connect_start`. It shows you a link and a code, like `KX4P-9QRM`. 2. You open the link. Sign up or sign in (email or Google). 3. The page shows the same code. **Check that it matches.** If it does not, or you did not ask for this, press Deny. 4. Choose a workspace (or make a new one) and an environment. Development is the usual choice for a laptop. Press **Approve**. 5. The agent calls `zexl_connect_finish`. On your machine, it: - writes `ZEXL_API_KEY` into the project's `.env` file. The agent only sees a masked form like `zx_ab12…`; - makes the salt locally. If the project already had a Zexl key, it keeps that salt so earlier calls still match; - adds `.env` to `.gitignore` when git would otherwise commit it; - saves a read key for the MCP tools in `~/.config/zexl/credentials.json`, readable only by you. 6. The agent follows `zexl_setup_guide`: it adds `zexl` to your dependencies, adds `import zexl.auto` at the entry point, and suggests feature labels. Review the diff. 7. You run the app once. The agent calls `zexl_check_connection` and tells you what arrived. ## What the agent gets Two keys, both visible and revocable in **Settings > API keys**: | Key | Can | Cannot | |---|---|---| | Send key (in `.env`) | Send call metadata for this project, filed under the environment you chose | Read anything | | Read key (for the MCP tools) | Read this workspace's figures | Send data, change budgets, keys, prices or members | A request lasts 15 minutes and gives keys once. ## After that Ask the agent questions such as: - "What did we waste last week, and how do I fix the biggest one?" - "Why was Tuesday so expensive?" - "Will we stay within budget this month?" See [MCP server](https://tryzexl.xyz/docs/mcp.md) for every tool. --- # Track Claude Code Claude Code can export its usage with OpenTelemetry. Point it at Zexl to see what your coding sessions cost, by model and by repository. No SDK and no code change. ## Set up 1. In the dashboard, open **Setup** (or onboarding), choose **Claude Code** and an environment (usually **Development**), and create the key. 2. Put these in your shell profile (`~/.zshrc` or `~/.bashrc`), with your key: ```bash export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_ENDPOINT=https://zexl-intake.fly.dev export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer zx__" ``` 3. Open a new terminal and start `claude`. Send a prompt. The call appears on the setup page within seconds. The header takes the key **without** the part after the dot. Claude Code does not hash prompts, so it does not need the salt. You can also put the same variables under `"env"` in Claude Code's `settings.json`, or use your organisation's managed settings to turn it on for a whole team. ## What is sent Per API request: model, input and output tokens, cache read and cache write tokens, cost estimate, duration, and errors with their status code. Prompts and code are not sent unless you turn that on yourself, and Zexl drops them if they arrive. ## What you see | Field | From | |---|---| | Run | One Claude Code session | | Feature | Repository name when Claude Code reports it, else the kind of request | | Actor | Claude Code's user id | | Cost | Claude Code's own estimate | Patterns that apply: prompt caching not used, growing context, and retries from API errors. Loops and cut-off answers need hashes that Claude Code does not send. ## Pro and Max plans On a Pro or Max plan you are not billed per token. The cost shown is what the same usage would cost on the API, which is useful for comparing sessions, models and repositories. ## Further reading - [Claude Code cost tracking](https://tryzexl.xyz/guides/claude-code-cost-tracking.md): what to look at once the data arrives. --- # Environments and keys A workspace is one team or product. Inside it, every call carries an **environment** label: `production`, `staging`, `development`, or a short label of your own. ## Why it matters - The dashboard shows **Production** by default, so test traffic does not inflate your real figures. - The switch under the workspace picker changes it: Production, Staging or All. - Prompt caching findings are counted on production only. - The monthly budget counts every environment, because every environment is on your provider bill. ## The key decides When you create a key you choose its environment. Every call sent with that key is filed under it, **whatever the app says**. A staging server cannot write into production by mistake. Make one key per environment: | Where it runs | Key environment | |---|---| | Production servers | Production | | Staging or preview deploys | Staging | | Your laptop, CI, coding agents | Development | Keys made before environments existed show **any**. For those, the app's own label is used: `ZEXL_ENV` for the SDK (default `production`), and `development` for Claude Code. ## Key kinds | Kind | Used by | Can | |---|---|---| | Send | The SDK, Claude Code | Send call metadata. Cannot read anything | | Read | The MCP server, AI assistants | Read one workspace's figures. Cannot send or change anything | Create either kind in **Settings > API keys**. A read key can be limited to one environment or read all of them. ## The salt and new keys A send key ends with `.`. The salt is made in your browser (or by the MCP server on your machine) and never sent to Zexl. Calls are only compared with calls hashed with the same salt. When you create a second send key, Settings asks for an existing key so it can copy the salt **in your browser**. Keep the same salt across all keys of a workspace, so a loop that spans two services is still recognised. ## Revoking Revoke a key in **Settings > API keys**. Zexl stops accepting it within a minute. --- # Features, runs and customers Labels turn "we spent $4,000" into "the report builder spent $1,900, and $600 of it was loops". There are three labels. Each answers one question: | Label | Question it answers | Typical value | |---|---|---| | Feature | Which part of the product spent it? | `support-reply`, `report-builder` | | Run | Which single task spent it? | One agent job, one chat turn, one document | | Customer | Who was it spent for? | Your account or tenant id | All three are optional. Start with none: Zexl fills in a feature and groups most agent runs by itself. Add labels where the automatic ones are too coarse. ## Feature Which part of your product made the call. ```python import zexl with zexl.feature("support-reply"): reply = client.chat.completions.create(...) ``` Without a label, Zexl uses the HTTP route of the current request when there is one, else the `module.function` that made the call. If your function names are sensitive, set the feature yourself. ## Run One unit of agent work: all the calls an agent made to finish one task. Waste patterns like loops are found within a run, and **Run detail** shows a run step by step. Zexl groups runs **automatically** when each request continues the previous answer (the usual agent loop), or when you use the OpenAI Responses API's `previous_response_id`. If your app already has an OpenTelemetry trace, the trace is the run. Group a run yourself when the agent trims or summarises its history, or uses sub-agents: ```python with zexl.run("research", feature="report-builder"): agent.solve(task) @zexl.run() # or as a decorator def summarise(doc): ... ``` ## Customer Which of your customers the spend was for. ```python zexl.set_customer(account.id) # an id, never an email or a name ``` ## Labels through the gateway With the [gateway](https://tryzexl.xyz/docs/gateway.md), set the same labels as HTTP headers on the request. Any language works, because the headers are added by your HTTP client, not by an SDK. | Header | Label | Used when the header is missing | |---|---|---| | `x-zexl-feature` | Feature | The LiteLLM virtual key's alias, then its team's alias | | `x-zexl-run` | Run | Calls are grouped when each request continues the previous answer | | `x-zexl-customer` | Customer (an id, never an email) | The request's `user` field (OpenAI) or `metadata.user_id` (Anthropic) | ```python from openai import OpenAI client = OpenAI(base_url="https://your-gateway/v1", default_headers={"x-zexl-feature": "support-reply"}) client.chat.completions.create(..., extra_headers={"x-zexl-run": task.id}) ``` The gateway removes the `x-zexl-*` headers before the request goes to the provider. ## Which labels to add first - **Several products or teams share one API key:** add features. Without them every call lands in one bucket. - **You run agents:** check **Run detail** first. If one task shows up as many short runs, or many tasks as one long run, add a run label. - **You charge customers or have usage-based plans:** add customers to see which accounts cost more than they pay. - **Function names or routes are sensitive:** set the feature yourself so the automatic name is never used. Labels are short text values. Use stable names (not ids that change on every deploy) for features, and ids for runs and customers. ## Environment See [Environments and keys](https://tryzexl.xyz/docs/environments.md). --- # Waste patterns explained Every call gets exactly one category: **useful**, or one of five waste patterns. Waste is charged to the call that produced nothing, not to the call that later worked. A figure marked **≈** includes an estimate. ## Retries **What it means.** A call failed, timed out or was cut off, and the same request was sent again within 60 seconds. **What it costs.** The failed call's cost. Providers do not bill most error responses, so the dollar figure is often small; the **retry rate** ("18% of calls are retries") is shown as well, because retries also cost time. **How to fix.** Find why calls fail: timeouts on long answers, rate limits, or requests that will always fail. Raise the client timeout, back off on 429 responses, and stop retrying errors that cannot succeed. ## Loops **What it means.** Inside one run, the agent repeated itself: the same full request 4 or more times, the same tool result 4 or more times, or the same tool calls 4 or more times. The first time is useful; the repeats are waste. **What it costs.** The full cost of each repeat. Measured, not estimated. **How to fix.** Give the agent a step limit, and stop or change course when a tool returns the same result twice. Open a run in **Run detail** to see which step repeats. ## Abandoned responses **What it means.** A streamed answer was started and never finished, or the client gave up (timeout or cancel), and it was not retried. **What it costs.** The input plus the output generated before it stopped (estimated from characters). **How to fix.** Cancel requests you no longer need as early as possible, or do not start them. Check client timeouts and that your code reads streams to the end. ## Cut off and redone **What it means.** An answer hit the output token limit, and the same thing was asked again soon after. **What it costs.** The full cost of the cut-off answer. Measured. **How to fix.** Raise `max_tokens` for this feature, or ask for a shorter answer, so the first attempt is enough. ## Prompt caching not used **What it means.** The same long prompt start (system prompt and tools) was sent 20 or more times in a day, close together, with under 10% of input read from the provider's cache. Production only. **What it costs.** The difference between the normal input price and the cached price on the repeated part. Estimated. **How to fix.** Keep the fixed part of the prompt first and byte-for-byte the same. Anthropic needs `cache_control` on it. OpenAI caches prompts over 1,024 tokens automatically when the start is identical. ## Growing context (not waste) **What it means.** In a run of 5 or more steps, the uncached input grew at nearly every step, to at least 3 times its starting size. **What it costs.** Shown separately as extra input cost. **Not counted as waste**, because the extra context may be needed. **How to fix.** Summarise or trim the history, cache the stable part, or pass large tool results by reference. ## Useful Everything else. A call is useful unless one of the rules above applies. ## Further reading - [Prompt caching with OpenAI and Anthropic](https://tryzexl.xyz/guides/prompt-caching.md) - [AI agent loops: how to find and stop them](https://tryzexl.xyz/guides/ai-agent-loops.md) - [Retries and cut-off answers](https://tryzexl.xyz/guides/retries-and-cut-off-answers.md) --- # Forecast and budget ## Monthly budget Set one monthly budget for the workspace on the **Budget** page. It counts **every environment**, because every environment is on your provider bill. The overview shows what is left. Zexl never blocks a call. ## Forecast The **Forecast** page, the overview, the budget page and the MCP tool `zexl_forecast` all show where this month's spend is likely to end. How it is worked out, in plain statistics: 1. Zexl takes the daily spend of the last 28 full days. 2. It fits a straight line through them. Recent days count more: a day's weight halves every 7 days back. 3. The trend is **damped**: each future day continues it at 90% of the day before, so one sharp rise does not run away. 4. With 3 weeks of data, busy and quiet weekdays are taken into account. 5. Forecast = spent so far + the rest of today + each remaining day. 6. The **likely range** comes from how far real days strayed from the line. The month should end inside it about 4 times out of 5. With under 3 full days of data, the forecast uses the **pace so far** instead, with a wide range. The forecast page also shows the day the running total is expected to pass the budget, if it does. A launch, a new feature or a fixed loop changes the line. The forecast follows within a few days. ## Stop at the budget With the [gateway](https://tryzexl.xyz/docs/gateway.md), the Guardrails page can stop calls once the month budget is used up. Start in Alert mode to see what it would stop. ## Coming soon - Alerts at 50%, 80% and 100% of the budget, and when the forecast passes it. - Budgets per feature and per environment. --- # Using the dashboard The left sidebar has every page. Under the workspace picker is the **environment switch** (Production, Staging, All). Most pages also have a **period** (7, 30 or 90 days). Both are kept in the URL, so you can share a view. ## Spend | Page | Shows | |---|---| | **Overview** | Useful-spend share, tracked spend, waste, budget left, the month-end forecast, where the money went, and spend per day | | **Spend analytics** | This period against the previous one, by day, feature and model | | **Forecast** | Where the month will end, the likely range, and when it passes the budget. See [Forecast and budget](https://tryzexl.xyz/docs/forecast.md) | | **Monthly summary** | This month so far against the same days last month, and the top findings | | **Feature portfolio** | Each feature's spend against its waste, to see where to look first | | **Breakdown** | Spend, waste and useful share by feature, model, customer or actor | ## Waste | Page | Shows | |---|---| | **Insights** | One card per pattern and feature, with its dollar size and a sentence. Open one to see the runs behind it, then a run to see it step by step | | **Fix tracker** | Open findings and a setup checklist | **Run detail** lists every step of one run: model, tokens and cost, and which earlier step it repeated, which failed, and which was cut off. Hashes themselves are never shown. ## Data | Page | Shows | |---|---| | **Calls explorer** | The newest calls as they arrive | | **Pipeline health** | Whether data is arriving, and totals | ## Control | Page | Shows | |---|---| | **Budget** | The monthly budget, spend so far and the forecast. Set or remove the budget here | | **Guardrails** | Rules the gateway applies to every call (allowed models, steps and cost per run, the month budget), and the calls they stopped or would have stopped | | **Gateway** | Whether gateway calls are arriving, how to run the gateway, and the label headers | ## Workspace | Page | Shows | |---|---| | **Setup** | Connect an app: SDK or Claude Code, environment, key, and a live check for the first call | | **Settings** | API keys (send and read), members and invites, budget, provider invoices, and your own model prices | ## Invoices and coverage Enter your provider invoice total per month in **Settings**. Zexl compares it with what it tracked and shows the **coverage**: how much of your bill Zexl sees. Low coverage means some calls are not instrumented yet. ## Your own prices If a model has no known price, its calls count as $0 and the overview says so. Set a price per million tokens in **Settings > Model prices**; earlier calls are priced again. ## More workspaces and members Use **+ New workspace…** in the workspace picker. Invite people in **Settings > Members**: Zexl makes an invite link for one email address, which you send yourself. --- # MCP server `zexl-mcp` lets AI assistants read your Zexl figures and set up new projects. It runs on your machine and talks to the Zexl API with a **read key**: it cannot send data or change anything. ## Install Claude Code: ```bash claude mcp add zexl -- uvx zexl-mcp ``` Claude Desktop, Cursor and others: ```json { "mcpServers": { "zexl": { "command": "uvx", "args": ["zexl-mcp"] } } } ``` ## Log in Either let the agent connect ([Connect with your AI agent](https://tryzexl.xyz/docs/connect-with-ai.md)), or create a **Read** key in **Settings > API keys** and pass it: ```bash claude mcp add zexl -e ZEXL_READ_KEY=zx_... -- uvx zexl-mcp ``` ## Tools | Tool | Answers | |---|---| | `zexl_status` | Is it connected, to which workspace, and when did the last call arrive? | | `zexl_connect_start` | Starts connecting a project; returns a link and a code for you | | `zexl_connect_finish` | Waits for your approval, writes `ZEXL_API_KEY` to `.env`, saves the read key | | `zexl_setup_guide` | Exact steps to add the SDK to a Python app, or to track Claude Code | | `zexl_check_connection` | Did the first call arrive, and what was recorded? | | `zexl_spend_summary` | Spend, waste by pattern, useful share, budget left, invoice coverage | | `zexl_forecast` | Where the month will end, the likely range, and when it passes the budget | | `zexl_top_waste` | The biggest findings, each with a fix | | `zexl_breakdown` | Spend and waste by feature, model, customer or actor | | `zexl_explain_spike` | Why a day cost more: its waste, and the features and models that changed | | `zexl_pattern_runs` | The runs behind one finding, worst first | | `zexl_run_detail` | One run step by step | Reading tools take `days` (default 30) and `environment` (default `production`; also `staging`, `development`, `all`). The prompt `integrate_zexl` walks an agent through setup. ## Things to ask - "What did we waste in production last week? Fix the biggest one." - "Which feature has the worst useful-spend share?" - "Explain the spike on the 10th." - "Show me one of the looping runs in the researcher feature." - "Will we stay within budget this month?" ## Settings | Variable | Meaning | Default | |---|---|---| | `ZEXL_READ_KEY` | A read key | the saved credentials | | `ZEXL_API_URL` | The Zexl API | `https://zexl-api.fly.dev` | | `ZEXL_CONFIG_DIR` | Where credentials are saved | `~/.config/zexl` | ## Hosted server A hosted version for claude.ai connectors and other remote clients, with sign-in instead of a key, is planned. --- # What leaves your process The SDK builds every record from a fixed allowlist. This is the complete list of what is sent: | Sent | Examples | |---|---| | Provider, API and model names | `openai`, `chat`, `gpt-4o` | | Token counts reported by the provider | input, output, cache read, cache write, reasoning | | Character counts | system prompt, tool definitions, input, output | | Timings | start, end, time to first token | | Outcome | finish reason, HTTP status, exception class name, whether a stream completed | | Tool names | the name of each tool the model called, and how many tools were defined | | Salted hashes | of the request, system prompt, tool definitions, newest input, output, and each tool call | | Run id and step number | made locally | | Labels you set | feature, customer, environment | | SDK details | SDK version, and an 8-character check value derived from your salt | **Never sent:** message text, system prompts, tool descriptions and schemas, tool arguments, tool results, model output, images, exception messages, your provider API keys, and the salt. ## Hashes The part of your key after the dot is a salt. It stays in your process. Every hash is an HMAC-SHA256 keyed with it. Without the salt, a hash cannot be matched to text, including by Zexl, and hashes from different workspaces cannot be compared. If you lose the key, the salt cannot be recovered; a new salt starts a new hash history. ## Things to know - Tool names and model names are sent as they are. Do not put secrets in tool names. - The feature label defaults to `module.function`. Set it yourself if your function names are sensitive. - Lengths are sent. They show how long a prompt was, not what it said. ## On the server - The intake checks every record against its own allowlist and drops anything else, even from senders that are not the SDK. - Send keys can only send. Read keys can only read one workspace. Revoked keys stop working within a minute. - Every query is limited to the workspace you are a member of. ## How this is tested A leak test plants marker strings in every place content can appear (system prompt, messages, images, tool definitions and results, responses, streamed chunks, error bodies) and fails if a marker or the salt appears in any byte the SDK sends. It gates every release. --- # Gateway and guardrails The Zexl gateway is for teams that cannot add the SDK, use a language other than Python, call providers other than OpenAI and Anthropic, or want rules that **stop** a call: a step limit for agents, a cost limit per run, a list of allowed models, or a hard stop at the month budget. It is the open-source [LiteLLM](https://github.com/BerriAI/litellm) proxy with the Zexl plug-in, used as a **pure proxy**. You run it in your own cloud, with your provider keys. Prompts never leave your network: Zexl receives the same call metadata the SDK sends. **What changes for your apps: the base URL and the key. Nothing else.** Apps keep their OpenAI or Anthropic SDK, their model names and their request format. Requests, responses, streams and errors pass through exactly as they are, and the gateway adds no retries. The one exception is a guardrail in Block mode, which you turn on yourself. ``` Your app ──▶ Zexl gateway (in your cloud) ──▶ OpenAI, Anthropic, Bedrock, Gemini, … └──▶ call metadata only ──▶ Zexl ``` ## 1. Create a key In the dashboard, open **Setup**, choose **Gateway**, pick the environment the gateway serves, and create the key. Copy the whole key, including the part after the dot: that part is the salt, and it stays in the gateway. ## 2. Run the gateway Build the image from the Zexl repository: ```bash docker build -f gateway/Dockerfile -t zexl-gateway . ``` The image ships with the right `config.yaml`: no model list (every model your apps ask for is passed through), no retries or fallbacks, and the Zexl plug-in. Run it with your provider keys: ```bash docker run -d -p 4000:4000 \ -e ZEXL_API_KEY="zx_..." \ -e LITELLM_MASTER_KEY="sk-..." \ -e OPENAI_API_KEY -e ANTHROPIC_API_KEY \ zexl-gateway ``` LiteLLM refuses to start without a master key. Make one long and random (`sk-` followed by `openssl rand -hex 24`). ## 3. Point your apps at it Change the base URL, and use the gateway key (the LiteLLM master key, or a LiteLLM virtual key per app) instead of the provider key: | SDK | Base URL | |---|---| | OpenAI (any language) | `http://zexl-gateway:4000/openai/v1` | | Anthropic (any language) | `http://zexl-gateway:4000/anthropic` | ```python from openai import OpenAI client = OpenAI(base_url="http://zexl-gateway:4000/openai/v1", api_key="sk-...") from anthropic import Anthropic client = Anthropic(base_url="http://zexl-gateway:4000/anthropic", api_key="sk-...") ``` Everything else in your code stays as it is. ## Labels | Header | Sets | Without it | |---|---|---| | `x-zexl-feature` | Feature | The LiteLLM virtual key's alias, then its team's alias | | `x-zexl-run` | Run label: every call with the same value is shown as one run | Agent loops are grouped automatically when each request continues the previous answer | | `x-zexl-customer` | Customer (use an id, never an email) | The request's `user` field (OpenAI) or `metadata.user_id` (Anthropic) | Headers are optional, and the provider never receives them. Give each app its own LiteLLM virtual key with a clear alias, and features are labelled with no header at all. ## Guardrails Set in the dashboard under **Guardrails**. Each gateway fetches the rules every minute. | Rule | A call breaks it when | |---|---| | Allowed models | The model it asks for is not on the list. `claude-*` matches every name that starts with `claude-` | | Most steps in one run | Its run already has that many steps. Runs are grouped by conversation (an agent resending its history), which is what a loop looks like. Stops agent loops | | Most a run may cost | Its run has already cost that much, by LiteLLM's prices | | Stop at the month budget | This month's spend, in every environment, has reached the budget on the Budget page | Three modes: - **Off.** Nothing is checked. - **Alert.** Every call goes through. Calls that break a rule are listed on the Guardrails page as "would stop". Start here, and watch for a week. - **Block.** Calls that break a rule are stopped with HTTP 403 and a message naming the rule, before they reach the provider and cost anything. 403 is used because OpenAI and Anthropic clients do not retry it. This is the only way the gateway ever changes a call. If Zexl cannot be reached, the gateway keeps the last rules it had and calls go through. To turn rules off on one gateway whatever the dashboard says, set `ZEXL_GUARDRAILS=off`. The Python SDK records calls but never stops them; guardrails need the gateway. ## Settings | Variable | Meaning | Default | |---|---|---| | `ZEXL_API_KEY` | The full send key, with the part after the dot. Without it the plug-in does nothing and LiteLLM works as usual | none | | `ZEXL_ENV` | Environment, for keys made before environments | `production` | | `ZEXL_GUARDRAILS` | `off` turns rules off on this gateway | `on` | | `ZEXL_ENDPOINT`, `ZEXL_API_URL` | Where to send calls and fetch rules | Zexl's hosted services | ## What is sent The same allowlist as the SDK: provider, model, token counts (converted to each provider's own convention), timings, outcome, tool names, salted hashes, run, feature and customer. Never prompt or response text, and never the salt. The plug-in is tested against a real LiteLLM proxy with marker text in every request, and the test fails if any of it reaches Zexl. See [What leaves your process](https://tryzexl.xyz/docs/privacy.md). ## Providers OpenAI (chat completions and the Responses API) and Anthropic (Messages), plain and streamed. Other providers follow. ## What stays the same | | Through the gateway | |---|---| | Your SDK, model names, request format | Unchanged | | Responses, streams, errors and status codes | Exactly as the provider sent them | | Retries | Only your client's own; the gateway adds none | | Headers sent to the provider | Yours, minus the `x-zexl-*` labels; the provider key replaces the gateway key | | Extra time per call | One network hop inside your cloud | | If Zexl is down | Calls go through; rules stay as they were | | If the gateway is down | Calls fail, as with any proxy: keep a fallback to the provider URL if that matters | --- # Troubleshooting ## No calls arrive 1. Make sure the app made a model call through the `openai` or `anthropic` Python package. 2. Check `import zexl.auto` runs **before** the client is created. 3. Check `ZEXL_API_KEY` is set in **that** process, with the part after the dot. 4. Print `zexl.status()` in the app. It shows whether capture is on, which hooks are active, and how many records were sent or dropped. 5. Look for a warning on the `zexl` logger. 6. Check the dashboard's environment switch: a development key's calls are not in Production. Try **All**. ## A model shows $0 Zexl does not know its price. Set one in **Settings > Model prices**. ## Calls are in the wrong environment The key decides. Create a key for the right environment, or set `ZEXL_ENV` if the key shows **any**. ## "ZEXL_API_KEY has no salt part" Use the full key shown at creation, including the part after the dot. If it is lost, create a new key; it starts a new hash history. ## Claude Code calls do not show Open a new terminal after editing your shell profile, and check `OTEL_LOGS_EXPORTER=otlp` is set. The header takes the key without the part after the dot. ## The MCP server says it is not connected Run "connect Zexl" again with your agent, or set `ZEXL_READ_KEY` to a **read** key. A send key does not work for reading. ## Does the SDK slow my app? About 0.05 ms per call, and sending happens on a background thread. If Zexl is down, your app carries on and records are dropped. ## How do I turn it off? Set `ZEXL_DISABLED=1`, or remove the import. ## The gateway does not record calls Check that `ZEXL_API_KEY` is set in the gateway's container (the full key, with the part after the dot) and that `config.yaml` lists `zexl_gateway.litellm_callback.proxy_handler_instance` under `litellm_settings.callbacks`. The gateway logs a line starting with `zexl-gateway` when something is wrong. ## A guardrail stopped a call it should not have Set the mode to Alert on the Guardrails page; gateways apply it within a minute. On one gateway, `ZEXL_GUARDRAILS=off` turns rules off at once. --- # Best way to track wasted spend on OpenAI API calls To track wasted spend on OpenAI API calls, use two layers. Start with the OpenAI Usage Dashboard to see what you spent by project, key and model. Then record each call in your own app, so you can see which calls were repeats, loops or cache misses that bought nothing. ## What the OpenAI dashboard tells you The Usage Dashboard is the source of truth for the bill. Viewing it needs an [organization owner or the Usage Dashboard permission](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). - You can [filter usage by API source, key, model, user or tier](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs), and view tokens [per day, hour or minute](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). - It shows [input, uncached, cached and cache write tokens, plus output tokens](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). A low cached share on a prompt you send often is the first sign of caching not doing its job. - You can [export monthly usage and cost as CSV](https://help.openai.com/en/articles/20001072), as [Activity data or Cost data](https://help.openai.com/en/articles/20001072). Activity data can be [grouped by project, user, API key, model, batch or service tier](https://help.openai.com/en/articles/20001072). - The recommended export interval is [1 day](https://help.openai.com/en/articles/20001072). To match the invoice, [group by Line item](https://help.openai.com/en/articles/20001072). Two things catch teams out. Data is [shown in UTC](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs), and recent usage [may take time to appear](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). Playground tokens also [count toward API usage](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). For Enterprise customers, invoices from [April 1, 2026 no longer include detailed API costs](https://help.openai.com/en/articles/20001072), so the dashboard and exports matter more. ## Where the dashboard stops The dashboard counts tokens. It does not know why a call was made. If several apps share a project or key, [the total alone may not show which one sent a request](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). It also cannot tell a useful call from a wasted one, such as: - a request your code re-sent after a timeout - an agent repeating the same tool call in a loop - a streamed answer the user closed before it finished - an answer cut off at the token limit and asked for again To see those, record a small entry for each call in your app: model, tokens, timing, the run it belonged to, and a hash of the request so repeats can be matched without storing prompt text. Group the entries by run and look for the same request sent twice or more. When you compare these logs with the dashboard, [use UTC](https://help.openai.com/en/articles/10478918-reviewing-api-usage-and-costs). [Zexl](https://tryzexl.xyz/what-is-zexl) does this for Python apps with one import: it records tokens, timing and hashed fingerprints, never prompt text, and prices each wasted pattern. It is [free during early access](https://tryzexl.xyz/pricing). ## FAQ ### Can the OpenAI dashboard show duplicate calls? No. It shows totals by project, key, model and time. Spotting a duplicate needs a record of each call with a fingerprint of the request, kept on your side. ### Does OpenAI send spend alerts? Check your organization's billing settings for limits and notices. For alerts on waste, such as a loop in one agent, you need call level data from your own app. ### Why do my logs not match the dashboard? The dashboard is in UTC and recent usage can lag. Playground use also counts. Compare whole UTC days, and give each app its own project or key. ### How do I see if prompt caching works on OpenAI? Compare cached input tokens with total input tokens for the model in the Usage Dashboard. If a prompt you send many times a day shows almost no cached tokens, the start of the prompt is changing between calls. --- # Token cost calculation Token cost calculation is the way you work out what a call to an LLM API costs: you count the input tokens and output tokens, multiply each by the model's price for that kind of token, and add the two. Prices are quoted per million tokens, and output tokens usually cost more than input. ## The formula The basic formula is [Total = input tokens × input price + output tokens × output price](https://www.mindstudio.ai/blog/token-based-pricing). Divide each token count by one million first when the price is per million tokens. Two adjustments matter in practice: - **Cached input.** Many providers charge a lower rate for input they served from a prompt cache. Split input into cached and uncached tokens and price each at its own rate. - **Peak or promotional rates.** Some prices change by time or date. For example, [DeepSeek peak rates are double](https://tokencalculator.ai/), and Gemini promotional pricing [runs through Dec 31, 2026, then doubles to $1.50 input and $7.50 output](https://tokencalculator.ai/) per 1M tokens. ## How many tokens is my text? A token is a piece of a word. As a rough guide, [1 token is about 4 characters](https://tokencalculator.ai/) of English, and [1,000 tokens is about 750 words](https://www.mindstudio.ai/blog/token-based-pricing). Short common words are often one token: ["hello" is 1 token](https://www.mindstudio.ai/blog/token-based-pricing), while ["ChatGPT" is about 2 tokens](https://tokencalculator.ai/). Language changes the count. [Non-English text needs 20-30% more tokens](https://www.mindstudio.ai/blog/token-based-pricing) for the same meaning, so the same feature can cost more for users who write in other languages. For an exact count, use the tokenizer for your model or read the token counts the API returns with each response. ## Example prices per million tokens These are list prices from one price tracker. Check the provider's own pricing page before you budget, because prices change. | Model | Input | Cached input | Output | |---|---|---|---| | GPT-6 Astra | [$10.00](https://tokencalculator.ai/) | [$1.00](https://tokencalculator.ai/) | [$50.00](https://tokencalculator.ai/) | | Claude Opus 5.5 | [$4.00](https://tokencalculator.ai/) | [$0.20](https://tokencalculator.ai/) | [$20.00](https://tokencalculator.ai/) | | Grok 4.7 | [$2.00](https://tokencalculator.ai/) | Not stated | [$6.00](https://tokencalculator.ai/) | | Gemini 3.8 Flash | [$0.75](https://tokencalculator.ai/) | [$0.075](https://tokencalculator.ai/) | [$3.75](https://tokencalculator.ai/) | | DeepSeek Flash V4.1 | [$0.15](https://tokencalculator.ai/) (cache miss) | [$0.003](https://tokencalculator.ai/) (cache hit) | [$0.60](https://tokencalculator.ai/) | Across the market, input prices ran from [$0.15 to $5.00 per million](https://www.mindstudio.ai/blog/token-based-pricing) and output from [$0.60 to $25.00 per million](https://www.mindstudio.ai/blog/token-based-pricing) as of January 2026. Output typically costs [3-5x input](https://www.mindstudio.ai/blog/token-based-pricing). ## Why the calculated cost and the bill differ The formula gives the cost of one call. The bill is the sum of every call, including the ones that did not help anyone: - a request re-sent after a timeout, paid twice - an agent that repeats the same step, resending its whole history each time - a prompt sent many times a day that never hits the cache - a streamed answer the user closed early, still billed for what was generated - an answer cut off at the token limit and asked for again Each of these is priced with the same formula, so per call math looks fine while the monthly total grows. To find them you need a record of each call, grouped by the task or agent run it belonged to. [Zexl](https://tryzexl.xyz/what-is-zexl) keeps that record from token counts and timing only and prices these patterns, without storing prompt text. ## FAQ ### How is token cost calculated? Multiply input tokens by the input price and output tokens by the output price, then add them. Price cached input at its own lower rate if the provider has one. ### How much do 500,000 tokens cost? It depends on the model and on how the tokens split between input and output. Use the formula above with your model's prices: divide each token count by one million and multiply by the price per million. ### How many words are 10,000 tokens? At [about 750 words per 1,000 tokens](https://www.mindstudio.ai/blog/token-based-pricing), scale that rate up for English text. Other languages give fewer words for the same number of tokens. ### What does 1,000 tokens cost? Take the price per million tokens and divide it by one thousand. Do it separately for input and output, since output usually costs more. --- # Langfuse alternatives and when to switch The main Langfuse alternatives are LangSmith for teams already deep in LangChain, Braintrust for teams that want evals and tracing in one hosted product, and a cost-focused tool when the question is spend rather than traces. Langfuse itself is open source and self-hostable, so check whether you need to switch at all. ## What Langfuse does well Langfuse is a strong default. The points in its favor: - **License.** Langfuse is [MIT-licensed](https://langfuse.com/compare/langsmith). Tracing, evals, prompt management, experiments, annotation and the playground are [in the MIT distribution with no usage limits](https://langfuse.com/compare/langsmith). - **Self-hosting.** Self-hosting is [available on every tier including free](https://langfuse.com/compare/langsmith). On self-hosted Langfuse OSS, [SSO and org-level RBAC are free](https://langfuse.com/compare/langsmith). - **Integrations.** It [ingests OpenTelemetry and 100+ frameworks](https://langfuse.com/compare/langsmith), so it is not tied to one agent library. - **Scale and history.** It [runs on ClickHouse](https://langfuse.com/compare/langsmith), Langfuse Pro [keeps 3 years of history](https://langfuse.com/compare/langsmith), and it is [used at 21 of the Fortune 50](https://langfuse.com/compare/langsmith). On Langfuse Cloud, SSO and project RBAC [need the Teams add-on or Enterprise](https://langfuse.com/compare/langsmith), and Cloud Enterprise is [$2,499/month](https://langfuse.com/compare/langsmith). Teams that want those controls without that plan can self-host instead. ## Why teams look elsewhere People search for alternatives for a few different reasons, and each one points to a different tool: - They build on LangChain or LangGraph and want the vendor's own tracing. - They want evals, CI checks and alerts in one hosted product. - They do not need traces at all. They need to know which AI calls are wasting money. ## LangSmith LangSmith is the like for like alternative. It is [proprietary](https://langfuse.com/compare/langsmith), and LangSmith Cloud [moved traces to SmithDB in May 2026](https://langfuse.com/compare/langsmith). Things to weigh before choosing it: - Pricing is per seat: LangSmith charges [$39 per invited seat](https://langfuse.com/compare/langsmith). In Langfuse's own comparison, 500k traces a month with 5 users comes to [$621 on Langfuse Pro](https://langfuse.com/compare/langsmith) and [$3,895 on LangSmith Plus](https://langfuse.com/compare/langsmith). That comparison is published by Langfuse, so run your own numbers. - SaaS retention is [capped at 180 days](https://langfuse.com/compare/langsmith). - Self-hosting is an [Enterprise-only add-on](https://langfuse.com/compare/langsmith), as are [SSO and RBAC](https://langfuse.com/compare/langsmith), and [bulk export](https://langfuse.com/compare/langsmith). Pick it if your team lives in the LangChain stack and wants one vendor for framework and tracing. ## Braintrust Braintrust puts evals, tracing and cost views in one hosted product. Its own figures: its Brainstore database runs trace queries [about 80x faster than traditional databases](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026), and Notion went from [triaging 3 issues a day to 30](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - The [Starter plan is free with 1 GB, 10K scores and unlimited users](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). Paid plans [start at $249/month](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - It integrates with [OTel, Vercel AI SDK, OpenAI Agents SDK, LangChain, LangGraph and Google ADK](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - A [GitHub Action runs evals on every PR](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026), and [webhook alerts go to Slack and PagerDuty](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). Pick it if evals in CI matter as much as traces. ## Other open source options If the open source license is the reason you chose Langfuse, other open source tools exist: [Opik is Apache 2.0](https://www.firecrawl.dev/blog/best-llm-observability-tools), [MLflow is Apache 2.0](https://www.firecrawl.dev/blog/best-llm-observability-tools), [W&B Weave is open source](https://www.firecrawl.dev/blog/best-llm-observability-tools), and so are [Helicone, Portkey, OpenLLMetry, TruLens, Phoenix and Evidently](https://www.firecrawl.dev/blog/best-llm-observability-tools). Each has a different focus, from gateways to evaluation, so compare them on the job you need done. ## A cost-only tool Some teams reach for Langfuse only to answer one question: where is the AI money going? A tracing tool can show that, alongside full traces. [Zexl](https://tryzexl.xyz/what-is-zexl) is narrower. Its Python SDK records only tokens, timing and hashed fingerprints, never prompt text, and prices retries, agent loops, unused caching, abandoned streams and cut off answers that were asked again. It works with OpenAI and Anthropic. It is not a tracing tool, so it is the wrong pick if you need full prompt content or a gateway for fallback. It is [free during early access](https://tryzexl.xyz/pricing). ## How to choose Start from the question you need answered, not the feature list. - **"Why did this answer go wrong?"** You need traces with prompt content. Stay on Langfuse, or pick LangSmith if you want the LangChain vendor. - **"Did this change make answers worse?"** You need evals tied to your pull requests. Look at Braintrust, or the eval features already in Langfuse. - **"Which calls are burning money?"** You need call level cost data grouped by run. A cost-only tool can answer that without storing prompts. - **"Must the data stay on our servers?"** Shortlist the tools you can self-host on the plan you can afford. Some teams run two tools: one for traces and evals, one for spend. ## Side by side | | Langfuse | LangSmith | Braintrust | Cost-only (Zexl) | |---|---|---|---|---| | License | [MIT](https://langfuse.com/compare/langsmith) | [Proprietary](https://langfuse.com/compare/langsmith) | Not stated | Not stated | | Self-hosting | [Every tier](https://langfuse.com/compare/langsmith) | [Enterprise add-on](https://langfuse.com/compare/langsmith) | Not stated | Not stated | | SSO and RBAC | [Free on self-hosted OSS](https://langfuse.com/compare/langsmith) | [Enterprise only](https://langfuse.com/compare/langsmith) | Not stated | Not stated | | Retention | [3 years on Pro](https://langfuse.com/compare/langsmith) | [180 days on SaaS](https://langfuse.com/compare/langsmith) | Not stated | Not stated | | Entry price | Not stated | [$39 per seat](https://langfuse.com/compare/langsmith) | [Free Starter](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026) | [Free in early access](https://tryzexl.xyz/pricing) | Prices change often. Check current prices on each vendor's pricing page before you decide. ## FAQ ### What is the difference between Langfuse and LangSmith? Langfuse is MIT-licensed and can be self-hosted on any tier. LangSmith is proprietary, with self-hosting, SSO, RBAC and bulk export on Enterprise only, and per seat pricing. ### What are LangSmith's competitors? Langfuse is a direct one. Braintrust covers tracing plus evals, and open source tools such as Opik, MLflow and Phoenix cover parts of the same ground. ### How does LangWatch compare with Langfuse? We have no verified figures on LangWatch, so we do not compare it here. Check its license, self-hosting terms and retention against the table above. ### Is Langfuse part of LangChain? No. Langfuse is its own open source project under the MIT license. LangSmith is the tracing product from the LangChain team. --- # Best LLM cost monitoring tools, grouped by the job they do The right LLM cost monitoring tool depends on the question you need answered. Tracing platforms show cost per trace next to prompts and evals. Gateways meter every request at one point. Cloud cost tools put AI spend next to the rest of your cloud bill. Waste finders show which calls bought nothing. You may need only one tool from one of these groups, not one of each. This guide sorts the main options by the job they do, so you can pick a group first and a product second. ## Why AI spend needs its own monitoring AI spend is growing fast. Zylo reports that AI-native software spend [rose 334% year over year](https://zylo.com/blog/best-ai-cost-management-tools) as of September 23, 2026, and [rose 507% in organizations with more than 10,000 employees](https://zylo.com/blog/best-ai-cost-management-tools). In organizations with 2,501 to 10,000 employees it [rose 447%](https://zylo.com/blog/best-ai-cost-management-tools). Spend on SaaS with AI features [rose 143%](https://zylo.com/blog/best-ai-cost-management-tools). Surprises happen: [78% of IT leaders saw unexpected charges](https://zylo.com/blog/best-ai-cost-management-tools). Part of the reason is that the cost of one request is not fixed. Long contexts, retries, tool loops and reasoning models can [multiply per-request cost by 10x before the invoice shows it](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). A monthly invoice tells you the total. It does not tell you which feature, agent or retry produced it. That is the gap these tools fill. ## What to look for in a cost monitoring tool Before comparing products, decide which of these you need: - **Granularity.** Per model and per day is enough for budgeting. Finding waste needs per call data grouped by the run or task it belonged to. - **Attribution.** Can you split cost by feature, team, customer or environment? Without labels, a shared API key hides who spent what. - **Prompt handling.** Some tools store full prompts and responses. That helps debugging, and it raises privacy and retention questions. Others record only token counts and timing. - **Where it sits.** An SDK or exporter in your code, a proxy or gateway in the request path, or a billing feed from your cloud provider. Each has different setup and failure modes. - **Alerts.** A dashboard you check weekly will not catch a loop that runs overnight. Look for alerts to the channels your team already watches. - **License and hosting.** If data must stay on your servers, shortlist tools you can self-host. ## Tracing and evaluation platforms These tools record each request as a trace, with prompts, responses, tool calls and cost. They are built for debugging and quality work, and cost views come along with the traces. ### Braintrust Braintrust combines tracing, evals and cost views in one hosted product. Its own figures: the Brainstore database runs trace queries [about 80x faster than traditional databases](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026), and Notion went from [triaging 3 issues a day to 30](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - Integrations: [OTel, Vercel AI SDK, OpenAI Agents SDK, LangChain, LangGraph and Google ADK](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - Logging is [async](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026), so it stays out of the request path. - [Webhook alerts to Slack and PagerDuty](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026), and a [GitHub Action that runs evals on every PR](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - An [AI Gateway that works with a base URL change](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). - Pricing: the [Starter plan is free with 1 GB, 10K scores and unlimited users](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026); paid plans [start at $249/month](https://www.braintrust.dev/articles/best-tools-tracking-llm-costs-2026). Good fit for teams that want cost, quality and CI checks in one place. ### Open source tracing tools If you want to run the tool yourself, several open source options exist. [Langfuse is MIT](https://www.firecrawl.dev/blog/best-llm-observability-tools). [Opik](https://www.firecrawl.dev/blog/best-llm-observability-tools) and [MLflow](https://www.firecrawl.dev/blog/best-llm-observability-tools) are Apache 2.0. [W&B Weave is open source](https://www.firecrawl.dev/blog/best-llm-observability-tools), as are [OpenLLMetry, TruLens, Phoenix and Evidently](https://www.firecrawl.dev/blog/best-llm-observability-tools). They differ in focus. Some lead with tracing and prompt management, some with evaluation, some with OpenTelemetry instrumentation that sends data to a backend you already run. All of them can show token counts per trace; check how each one prices models and whether it can group cost by your own labels. Good fit for teams that need traces and want the data on their own servers. ## Gateways and proxies A gateway sits between your app and the model providers. Every request passes through it, so it can count tokens, apply limits and route to other models. - [Portkey](https://www.firecrawl.dev/blog/best-llm-observability-tools) supports [250+ models with 20-40ms overhead](https://www.firecrawl.dev/blog/best-llm-observability-tools) and is open source. - [Helicone](https://www.firecrawl.dev/blog/best-llm-observability-tools) is open source. The strength of a gateway is coverage: nothing reaches a provider without being counted. The cost is a new hop in the request path, which adds latency and becomes something you have to keep running. Good fit for teams that also want fallback, routing or rate limits. ## Cloud and finance cost tools These tools read billing data rather than requests. They are built for finance and FinOps teams who look at AI spend next to compute, storage and SaaS. - [CloudZero reads cloud and billing feeds plus Snowflake and Databricks](https://zylo.com/blog/best-ai-cost-management-tools), and has [per-token and per-model views for Anthropic and OpenAI usage billed through the cloud](https://zylo.com/blog/best-ai-cost-management-tools). - [IBM Apptio Cloudability covers multi-cloud and GenAI spend](https://zylo.com/blog/best-ai-cost-management-tools). It is [not built for token-level API tracking](https://zylo.com/blog/best-ai-cost-management-tools), so pair it with an engineering tool if you need that. Good fit for answering "what did AI cost the company this quarter" across teams and vendors. They do not see inside your app, so they cannot tell which feature or agent drove the spend. ## Waste finders The groups above tell you what you spent. A waste finder tells you which part of it bought nothing: requests re-sent after a timeout, agents that repeat the same step, prompts that never hit the cache, streamed answers closed early, and answers cut off and asked for again. [Zexl](https://tryzexl.xyz/what-is-zexl) is in this group. Its Python SDK wraps the OpenAI and Anthropic SDKs, records only tokens, timing and hashed fingerprints, never prompt text, groups calls into agent runs and prices each wasted pattern by named rules. It also works with Claude Code. It is [free during early access](https://tryzexl.xyz/pricing). It is not a tracing tool or a gateway, so it is the wrong pick if you need full prompt content or fallback routing. Good fit for engineering leads who need to know where to cut first. ## How the groups compare | Group | Sees prompts | Where it sits | Question it answers | |---|---|---|---| | Tracing and evals | Yes, usually | SDK or exporter | Why did this request cost what it did? | | Gateways | Can, by design | In the request path | What did every request cost? | | Cloud cost tools | No | Billing feeds | What did AI cost across the company? | | Waste finders | Not necessarily | SDK | Which spend bought nothing? | Prices and plans change. Check current prices on each vendor's pricing page before you decide. ## How to choose 1. **Write down the question.** "What did we spend" is a billing question. "Why did it go up" is a tracing question. "What can we cut" is a waste question. 2. **Check what data you are allowed to send.** If prompts cannot leave your servers, rule out hosted tools that store them, or self-host. 3. **Start with your provider's own usage dashboard.** It is free and it is the source of truth for the bill. Add a tool when you need a split it cannot give you. 4. **Label calls from day one.** Whatever you pick, tag calls by feature and environment. Cost without attribution is hard to act on. 5. **Set one alert.** A daily spend alert per project can catch a runaway loop before it becomes an invoice. ## What to measure once a tool is in place A tool only helps if you look at the right numbers. Start with these, in this order: - **Spend per feature per day.** This shows which part of the product drives the bill and when it changed. - **Retries and repeated calls.** A request sent twice is paid twice. See [how retries and cut off answers add cost](https://tryzexl.xyz/guides/retries-and-cut-off-answers). - **Agent run length.** Count the calls in each agent run and watch for runs that repeat a step. See [how agent loops waste tokens](https://tryzexl.xyz/guides/ai-agent-loops). - **Cache hit rate.** If you use prompt caching, check that cached tokens show up on the bill. See [the prompt caching guide](https://tryzexl.xyz/guides/prompt-caching). For the arithmetic behind each call, see [how token cost is calculated](https://tryzexl.xyz/guides/token-cost-calculation). ## FAQ ### How do I pick a tool to track LLM costs? No single tool fits every team. Pick by job: a tracing platform for debugging, a gateway for coverage and routing, a cloud cost tool for finance, or a waste finder to see what to cut. ### Can I monitor LLM costs without storing prompts? Yes. Cost needs token counts, the model and timing, not prompt text. Cloud billing tools and some SDK based tools work from those alone. ### Do I need a gateway to track costs? No. A gateway gives full coverage, but an SDK or your provider's usage export also works. Choose a gateway if you want routing or fallback too. ### Are there free LLM cost monitoring tools? Yes. Several tracing tools are open source and can be self-hosted, and some hosted tools have free plans. Your provider's usage dashboard is also free. --- # LLM cost monitoring ROI: metrics that show value To measure the ROI of LLM spend, divide the value a feature or agent produces by its full cost, and track that per task, not per month. The cost side comes from token usage, retries and the people who check the output. The value side comes from time saved, revenue or margin. Cost monitoring is what makes the first half of that division accurate. This guide covers the gap between expected and realized returns, the metrics that close it, and how long each kind of project takes to pay back. ## Why AI ROI is hard to prove Expected returns and realized returns do not match. BCG found companies expected [a 14.1% return on AI but realized 11.2%](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi). McKinsey reports that [88% of organizations use AI, but only 39% report any EBIT impact](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi). Pilots struggle too. Dynatrace cites that [about 5% of pilots extract value, despite $30-40B spent](https://www.dynatrace.com/news/blog/how-to-measure-ai-roi-with-business-observability/). Agents carry extra risk: Gartner expects [more than 40% of agentic AI initiatives to be canceled by the end of 2027](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi), and Elastic reports that [over 70% of agentic use cases fail to deliver the expected value](https://www.elastic.co/blog/how-it-leaders-measure-roi-of-agentic-ai-initiatives). Data problems slow things down before ROI can even be measured: [86.9% of organizations delayed deployments over data issues](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi). Meanwhile, budgets are under watch. [52% of CIOs say cost reduction becomes more important](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi). Adoption is still early: [17% of CIOs have adopted agents and 42% plan to](https://www.elastic.co/blog/how-it-leaders-measure-roi-of-agentic-ai-initiatives). That leaves a short window to set up measurement before spend grows. ## The cost side: what to track ROI is only as good as its cost number. For LLM features, the full cost has more parts than the invoice line. - **Token cost per task.** Input and output tokens at the model's price, summed across every call the task made. See [how token cost is calculated](https://tryzexl.xyz/guides/token-cost-calculation). - **Wasted calls.** Retries, repeated agent steps, cache misses and answers that were cut off and asked for again. They add cost and no value. See [how retries and cut off answers add cost](https://tryzexl.xyz/guides/retries-and-cut-off-answers). - **Human review time.** If a person checks or fixes every output, their time is part of the cost. - **Platform and tooling.** Hosting, vector stores, observability and gateway fees. Ownership matters as much as the data. AvePoint notes that agent cost tracking [needs a named owner and a cost trail](https://www.avepoint.com/blog/strategy-blog/how-to-measure-ai-roi). Without a name next to each agent, nobody is asked why its spend went up. ### Tools that report cost Several observability products now break AI cost down. Datadog Cloud Cost Management [breaks spend down by token, model or project](https://www.datadoghq.com/blog/manage-ai-cost-and-performance-with-datadog/), and its Agent Observability [shows token usage and cost](https://www.datadoghq.com/blog/manage-ai-cost-and-performance-with-datadog/). For a wider list, see [LLM cost monitoring tools grouped by job](https://tryzexl.xyz/guides/best-llm-cost-monitoring-tools). ## The value side: what to pair it with Cost alone says nothing about return. Pair it with one value metric per use case: - **Time saved**, measured against the old way of doing the task. - **Revenue** the feature brings in or protects. - **Margin** gained by doing the same work at lower cost. Count time saved honestly. Naitive gives a clear example: if a person writes a draft in 60 minutes and the AI writes one in 20, but fixing it takes 25 minutes, the saving is [15 minutes, not 40](https://blog.naitive.cloud/metrics-ai-roi-evaluation/). Measure the full loop, including review. How much review a system needs changes the math. In Naitive's figures, [25% of AI systems run autonomously, 40% need active human involvement and 35% need passive monitoring](https://blog.naitive.cloud/metrics-ai-roi-evaluation/). A system that needs active review has a higher real cost than its token bill shows. ## Three metrics for agents Elastic describes three metrics built for agentic work: - **AVM (agentic value multiple):** [savings plus revenue plus margin, divided by total cost](https://www.elastic.co/blog/how-it-leaders-measure-roi-of-agentic-ai-initiatives). This is the headline ROI number. - **ACCT:** [the cost per successfully completed task](https://www.elastic.co/blog/how-it-leaders-measure-roi-of-agentic-ai-initiatives). Failed and abandoned runs still cost money, so they raise this number. - **CMOS:** [the input tokens needed per task](https://www.elastic.co/blog/how-it-leaders-measure-roi-of-agentic-ai-initiatives). It shows context bloat and long prompts before they show on the bill. ACCT is the one to watch first. It moves when an agent loops, retries or fails, which are the patterns that raise cost without raising value. See [how agent loops waste tokens](https://tryzexl.xyz/guides/ai-agent-loops). ## Tie the metrics to business goals Technical metrics need a link to the numbers leadership reads. Dynatrace found that only [28% of organizations align observability with business KPIs](https://www.dynatrace.com/news/blog/how-to-measure-ai-roi-with-business-observability/), and that high performers are [3x more likely to transform](https://www.dynatrace.com/news/blog/how-to-measure-ai-roi-with-business-observability/). In practice that means one dashboard per use case with three rows: cost per completed task, the value metric, and the trend of both. If cost per task rises while value stays flat, something in the system changed. Pick value metrics the business already tracks. A support team already counts tickets closed and handle time. A sales team already counts meetings booked. Reusing those numbers means nobody has to trust a new metric, and the AI feature is judged on the same scale as everything else. Report the cost side in the same units. Cost per ticket closed or cost per meeting booked is easier to discuss than tokens per day, and it makes the trade off between model choice and value visible to people outside engineering. ## How long ROI takes Set expectations by project type. Naitive gives these ranges: | Project type | Time to results | |---|---| | Simple tasks | [3-6 months to results](https://blog.naitive.cloud/metrics-ai-roi-evaluation/) | | Complex projects | [6-9 months to break even](https://blog.naitive.cloud/metrics-ai-roi-evaluation/) | | Predictive maintenance | [12+ months](https://blog.naitive.cloud/metrics-ai-roi-evaluation/) | Measure cost from day one even if value takes months. Waste found in the first weeks shortens the time to break even. ## A simple measurement plan You do not need a finance team to start. Five steps cover a single use case: 1. **Pick one use case.** A support reply drafter, a code review agent, a document summarizer. Measuring the whole AI program at once hides which part pays off. 2. **Write down the baseline.** How long does the task take today without AI, and who does it? This is the number time saved is measured against. 3. **Tag every call.** Label each LLM call with the feature, the environment and the agent run it belongs to. Without tags, a shared API key mixes every use case into one bill. 4. **Track cost per completed task weekly.** Count only tasks that finished and were accepted. Divide the week's full cost by that count. 5. **Review the trend with the owner.** The named owner explains changes. A rising cost per task with flat value is the signal to look for waste before adding more budget. Keep the first version small. A spreadsheet with weekly rows is enough until the numbers prove the use case is worth more tooling. ## Common mistakes when measuring AI ROI - **Using the invoice as the cost.** The invoice misses review time and tooling, and it does not show which feature caused the spend. - **Counting time saved before review.** As the drafting example shows, the gross saving can be larger than the real one. - **Dividing by all tasks, not completed ones.** Failed runs still cost money. Dividing by attempts makes cost per task look lower than it is. - **Ignoring waste.** Retries, loops and cache misses raise cost with no value. Cutting them improves ROI without touching the model or the product. - **Measuring once.** Prompts, models and traffic change. A number from launch week says little about this month. - **No owner.** Spend without a named person behind it grows quietly. ## Where cost monitoring fits Cost is the side of ROI you can change quickly. Value takes months to show; wasted calls can be cut in a sprint. A tool that separates useful spend from waste, such as [Zexl](https://tryzexl.xyz/what-is-zexl), feeds the cost side of AVM and ACCT with numbers you can act on. It is [free during early access](https://tryzexl.xyz/pricing). ## FAQ ### How do you measure AI ROI? Divide the value a use case produces (time saved, revenue or margin) by its full cost: tokens, wasted calls, human review and tooling. Track it per completed task. ### What are ROI indicators for AI? Cost per completed task, input tokens per task, time saved after review, and the value multiple of savings plus revenue plus margin over total cost. ### What two things should you consider when measuring AI ROI? The full cost, including review time and waste, and a value metric tied to a business goal. You need both to get an ROI number you can act on. ### How do you measure AI performance? Track task success rate, cost per successful task and how much human review each output needs. Pair those with the business metric the use case is meant to move. --- # How to reduce your LLM API costs The fastest way to cut an OpenAI or Anthropic bill is not a cheaper model. It is to stop paying for calls that produced nothing: requests sent twice, agents repeating a step, answers cut off and asked again, and long prompts sent without caching. In most applications these are a large share of the bill, and fixing them changes nothing your users see. This guide is a checklist, in the order that usually saves the most for the least work. ## 1. Find out where the money goes Your provider invoice gives one number per model per day. It does not say which feature spent it, which customer it was for, or whether the call did anything useful. Before changing code, you need three answers: - **Which features cost the most?** A support bot and a nightly report job can use the same model and have very different bills. - **Which calls were wasted?** A call is wasted when nothing used its result: it failed and was retried, it repeated an earlier step, it was abandoned, or it was cut off and redone. - **What is the trend?** A bill that grows with usage is fine. A bill that grows faster than usage means waste is growing. Record the model, token counts, timing and outcome of every call, with a label for the feature that made it. You do not need the prompt text for this, and keeping prompts out of your monitoring is safer. See [LLM cost monitoring: what to measure](https://tryzexl.xyz/guides/llm-cost-monitoring.md). ## 2. Turn on prompt caching If your requests start with the same long system prompt, tool definitions or reference document, both providers can serve that part from a cache at a fraction of the normal input price. - **OpenAI** caches automatically when the start of the prompt is identical and long enough (about 1,024 tokens on current models). Newer models also let you mark cache breakpoints and choose how long the cache lives. - **Anthropic** caches when you add `cache_control`, either once at the top of the request (automatic caching) or on specific content blocks. The default cache lives 5 minutes; a 1-hour option exists. The most common reason caching does not work is a prompt that changes at the start: a timestamp, a user name or a shuffled tool list placed before the stable part. Put stable content first, changing content last. Full details: [Prompt caching with OpenAI and Anthropic](https://tryzexl.xyz/guides/prompt-caching.md). ## 3. Stop agent loops An agent loop is an agent calling the model again and again without making progress: the same tool call with the same arguments, the same tool result coming back, the same request resent. Every repeat is billed in full, and because agents resend their whole history each step, later repeats cost more than earlier ones. Give every agent a step limit, stop when a tool returns the same result twice, and log the run id with every call so you can see a whole run at once. See [AI agent loops: how to find and stop them](https://tryzexl.xyz/guides/ai-agent-loops.md). ## 4. Fix retries and cut-off answers Two smaller leaks add up: - **Retries.** A request times out or hits a rate limit and your code sends it again. Most provider errors are not billed, but timeouts on long answers often are, because the model did the work. Raise timeouts for long answers and back off on rate limits instead of hammering. - **Cut-off answers.** When an answer hits `max_tokens`, it stops mid-sentence (`finish_reason` is `length` on OpenAI, `stop_reason` is `max_tokens` on Anthropic). If your code then asks again, you paid for the first attempt for nothing. Set the limit to fit the answers you actually need. See [Retries and cut-off answers](https://tryzexl.xyz/guides/retries-and-cut-off-answers.md). ## 5. Keep context from growing without limit Agents and chat features usually resend the whole conversation on every call. A 20-step run whose input grows each step can cost many times more than the same work with a trimmed history. Summarise old turns, drop tool results once they have been used, and cache the stable start of the prompt. ## 6. Then choose models per feature Once the waste is gone, look at model choice feature by feature. A classification step or a short extraction rarely needs the largest model. Compare cost per feature, not per model: moving one high-volume feature to a smaller model often saves more than any other change. Check the providers' current prices on their pricing pages, because they change often. ## 7. Use batch processing for work that can wait Both OpenAI and Anthropic offer a batch API for requests that do not need an answer straight away, at a lower price. Nightly reports, evaluations and bulk classification are good candidates. ## 8. Watch the month as it happens Set a monthly budget and look at the forecast during the month, not at the invoice after it. A loop introduced on the 3rd costs a month of spend if nobody sees it until the bill arrives. ## How Zexl helps Zexl does steps 1 to 5 for you. One import in a Python app (or a few environment variables for Claude Code) records every OpenAI and Anthropic call, without sending prompt text. Every call is sorted into useful work or one of five kinds of waste, with a dollar figure, the feature it came from, and the runs behind it. The forecast shows where the month will end. [Create a free account](https://tryzexl.xyz/sign-up) or read the [quickstart](https://tryzexl.xyz/docs/quickstart.md). --- # Prompt caching with OpenAI and Anthropic Prompt caching lets the provider reuse the start of a prompt it has already processed. When it works, the repeated part is billed at a fraction of the normal input price and the answer starts sooner. When it does not, you pay full price for the same system prompt on every call, and nothing tells you. This guide explains how each provider caches, the mistakes that stop it from working, and how to check. ## How caching works Both providers cache a **prefix**: the beginning of the request, up to some point. The next request reuses the cache only if its beginning is **identical, byte for byte**, up to that point. Anything that changes early in the prompt (a date, a user name, a request id, tools in a different order) breaks the match for everything after it. So the rule is the same everywhere: 1. Stable content first: system prompt, tool definitions, reference documents, few-shot examples. 2. Changing content last: the user's message, the conversation so far, today's date. ## OpenAI - Caching is **automatic** for prompts whose stable start is long enough. On current models the minimum is 1,024 tokens; for some earlier models it depends on request settings. - Newer models also support **explicit breakpoints** and a choice of **cache lifetime**, so you can say exactly where the stable part ends and how long to keep it. - `prompt_cache_key` groups related requests (for example, one per customer) so they reuse the same cache and keep separate cache accounting. - The response's usage shows how many input tokens came from the cache (`cached_tokens`). ## Anthropic - Caching needs `cache_control`. Add it once at the top level of the request for **automatic caching** (the API places the breakpoints as the conversation grows), or on individual content blocks for exact control. - The default cache lives **5 minutes**, renewed each time it is used. A **1-hour** lifetime is available. - Writing to the cache costs a little more than normal input; reading from it costs much less. With the 5-minute cache, it pays for itself after one cache read. Check Anthropic's pricing page for the current multipliers. - The response's usage reports `cache_creation_input_tokens` (written) and `cache_read_input_tokens` (read). ```python import anthropic client = anthropic.Anthropic() response = client.messages.create( model="claude-sonnet-5-5", max_tokens=1024, cache_control={"type": "ephemeral"}, # automatic caching system=LONG_STABLE_SYSTEM_PROMPT, messages=[{"role": "user", "content": question}], ) print(response.usage.cache_read_input_tokens) ``` ## Why the cache misses - **A timestamp or id at the top of the system prompt.** Move it to the end of the user message. - **Tools built in a different order each time.** Sort them, or build the list once. - **Personalised system prompts.** "You are helping Priya" changes the prefix for every user. Put the name in the user message. - **Calls too far apart.** A cache that lives 5 minutes does not help a feature called once an hour. Use a longer lifetime where the provider offers one, or accept the miss. - **Prefix too short.** Below the provider's minimum nothing is cached. - **Different models.** Caches are per model. ## Check that it works Look at the cached-token counts in the usage of each response, per feature. A feature that sends the same long prompt many times a day with a cached share near zero is paying full price for nothing. That is the signal to fix, and it can be found from token counts alone, without reading any prompt. ## How Zexl finds missed caching Zexl groups calls that share the same system prompt, tools and model (using salted hashes, never the text). When a group has 20 or more calls in a day, close enough together for the cache to help, a long enough shared start, and under 10% of input read from cache, the calls are marked **prompt caching not used**, with the money caching would have saved. See [waste patterns](https://tryzexl.xyz/docs/patterns.md). [Create a free account](https://tryzexl.xyz/sign-up) to see it on your own calls, or read [How to reduce your LLM API costs](https://tryzexl.xyz/guides/reduce-llm-api-costs.md) for the other big savings. --- # AI agent loops: how to find and stop them An agent loop is an agent that keeps calling the model without making progress: it repeats the same tool call, gets the same result, and tries again. It does not crash and it does not raise an error. It just spends money until it hits a step limit, a timeout, or your budget. Loops are one of the most expensive kinds of waste in agent applications, because each repeat resends the whole conversation so far. The tenth repeat costs more than the first. ## Why agents loop - **A tool returns an error the model cannot fix.** The model retries the same call with the same arguments, hoping for a different answer. - **A tool returns nothing useful.** An empty search result, a "not found", a truncated page. The model searches again. - **The goal is unclear.** The model cannot tell that it is done, so it keeps checking. - **History was trimmed.** The agent drops old steps to save tokens, forgets it already tried something, and tries it again. - **Two tools bounce off each other.** Tool A suggests calling tool B, which suggests calling tool A. ## The three signs of a loop You can detect a loop from call data alone, without reading prompts: 1. **The same request is sent again** within one run. 2. **The same tool result comes back again**: the newest input to the model is identical to an earlier one. 3. **The same tool calls are made again**: identical tool names and arguments. One repeat can be fine; a model sometimes needs to check twice. Four or more of the same thing in one run is a loop. ## Guards that stop loops Put these in every agent, whatever framework you use: ```python MAX_STEPS = 15 seen_calls: set[str] = set() for step in range(MAX_STEPS): reply = call_model(history) if not reply.tool_calls: break key = repr([(c.name, c.arguments) for c in reply.tool_calls]) if key in seen_calls: history.append(note("You already made this exact call. Try something else or answer.")) seen_calls.add(key) history += run_tools(reply.tool_calls) else: log.warning("agent hit the step limit", extra={"run_id": run_id}) ``` - **A step limit.** Every agent needs one. Pick it from real runs: the longest normal run plus a margin. - **Duplicate-call detection.** When the same tool call repeats, tell the model, or stop. - **Useful tool errors.** "Not found: no order with id 123" lets the model change course; an empty string does not. - **A cost limit per run.** Stop a run that has spent more than a set amount. - **One run id per task.** Log it with every model call, so a whole run can be seen at once. ## Measure before and after A step limit stops the worst runs, but it does not tell you how much you were losing or whether the fix worked. Measure the share of spend in repeated steps per feature, fix the biggest one, and check that the share drops. ## How Zexl finds loops Zexl groups calls into runs automatically when each request continues the previous answer, or from `zexl.run()` when you mark them yourself. Within each run it compares salted hashes of the request, the newest input and the tool calls. Four or more identical ones make a loop, and every repeat after the first is counted as waste, measured, not estimated. The run detail page shows the run step by step with the repeats marked. See [Features, runs and customers](https://tryzexl.xyz/docs/labels.md). [Create a free account](https://tryzexl.xyz/sign-up), or read [How to reduce your LLM API costs](https://tryzexl.xyz/guides/reduce-llm-api-costs.md). --- # Retries and cut-off answers Two kinds of waste hide in almost every LLM application: requests that fail and are sent again, and answers that stop at the output limit and are asked again. Each looks small. Together they are often a steady few percent of the bill, and they slow your product down. ## Retries A retry is the same request sent again after the first attempt failed: a timeout, a rate limit (HTTP 429), a server error (5xx), or a dropped connection. **What it costs.** Providers do not bill most error responses. But a **client timeout** is different: your code gave up, while the provider may have finished the work and billed it. Long answers with short timeouts are the usual cause. Retries also cost time: a user waits for two attempts. **How to spot it.** Count calls whose request is identical to a failed call made in the previous minute. Report it as a **retry rate** per feature ("18% of checkout-bot calls are retries"), not only in dollars, because the dollars understate the problem. **How to fix it.** - Set timeouts from real response times. A long answer can take tens of seconds. - On 429, back off with growing waits, and lower your concurrency. Retrying at once makes the rate limit worse. - Do not retry errors that will fail again: a request that is too long, invalid parameters, a refused request. - Mind double retries. The OpenAI and Anthropic Python clients already retry some errors themselves (`max_retries`). A retry loop of your own on top multiplies them. ## Cut-off answers An answer is cut off when it reaches the output limit you set (`max_tokens`). It stops mid-sentence or mid-JSON. The signal is in the response: | Provider | Field | Value when cut off | |---|---|---| | OpenAI Chat Completions | `finish_reason` | `length` | | OpenAI Responses | `status`, `incomplete_details.reason` | `incomplete`, `max_output_tokens` | | Anthropic | `stop_reason` | `max_tokens` | **What it costs.** If your code notices and asks again, often with a higher limit, the first answer was paid for and thrown away. If it does not notice, you ship a broken answer, which is worse. **How to fix it.** ```python response = client.chat.completions.create(model=model, messages=messages, max_tokens=800) if response.choices[0].finish_reason == "length": log.warning("answer cut off", extra={"feature": "summary", "max_tokens": 800}) ``` - Log every cut-off with the feature name, and look at how often it happens. - Size `max_tokens` from real answer lengths for each feature, with room to spare. - If answers are often too long, ask for a shorter answer in the prompt rather than raising the limit forever. - For structured output, keep the schema small; long JSON is the most common thing to get cut. ## How Zexl finds both Zexl reads the finish reason, status code and error class of every call (never the text). A failed call followed by the same request within 60 seconds is a **retry**; Zexl shows both the retry rate and the dollars. An answer that hit the output limit and was requested again within 10 minutes is **cut off and redone**, and the cut-off attempt is counted as waste. See [waste patterns](https://tryzexl.xyz/docs/patterns.md). [Create a free account](https://tryzexl.xyz/sign-up), or see [How to reduce your LLM API costs](https://tryzexl.xyz/guides/reduce-llm-api-costs.md). --- # Claude Code cost tracking Claude Code can report its own usage: every request it makes to the model, with tokens, cost estimate, timing and errors. Turned on for a team, this answers questions the plan page cannot: which repositories use the most, which models, which sessions were unusually expensive, and whether prompt caching is working. ## What Claude Code reports Claude Code exports telemetry with OpenTelemetry when you turn it on. For each API request it sends an event with: - the model; - input, output, cache-read and cache-write tokens; - its own cost estimate (`cost_usd`); - the duration; - errors, with the status code and attempt number; - the session id, user id, and (when available) the repository. Prompts and code are **not** exported unless the user explicitly turns that on. ## Turn it on Five environment variables, in the shell profile or in Claude Code's `settings.json` under `env`: ```bash export CLAUDE_CODE_ENABLE_TELEMETRY=1 export OTEL_LOGS_EXPORTER=otlp export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector.example.com export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " ``` `OTEL_LOGS_EXPORTER=otlp` is the one people miss: the per-request events are log events, so without it nothing about individual requests is sent. For a team, set the same values in your organisation's managed settings so every developer reports without doing anything. ## Pro and Max plans On a Pro or Max subscription you are not billed per token, but Claude Code still reports `cost_usd`: what the same usage would cost on the API. That is the right number for comparing sessions, models and repositories, and for deciding whether a team is better off on subscriptions or on the API. ## What to look at - **Cost per repository.** Some codebases cost far more per session, usually because of how much context Claude Code reads. - **Cache-read share.** Claude Code relies on prompt caching. A low share of input read from cache means you pay more for the same work. - **Model mix.** How much runs on the largest model. - **Long sessions.** Context grows with every turn; a very long session costs more per step at the end than at the start. - **API errors.** Rate limits and overloaded errors slow developers down and show up as retries. ## How Zexl tracks Claude Code Point the endpoint at Zexl's intake (`https://zexl-intake.fly.dev`) with a Zexl key in the header. Each Claude Code session becomes a run, the repository becomes the feature, and the person becomes the actor. Zexl shows spend per repository, model and developer, flags missed caching and growing context, and drops any prompt text that arrives. Setup takes two minutes: [Track Claude Code](https://tryzexl.xyz/docs/claude-code.md). [Create a free account](https://tryzexl.xyz/sign-up). --- # LLM cost monitoring: what to measure LLM cost monitoring means knowing, at any time, what your AI features cost, why, and whether the money did useful work. Provider dashboards answer the first part. The other two need data from your own application. ## What to measure **Per call:** - provider and model; - input, output and cached tokens (as reported by the provider); - cost, from the provider's current price for that model; - latency; - outcome: finish reason, status code, error class, whether a stream finished. **Labels on each call:** - **feature**: which part of your product made it; - **run**: which unit of agent work it belonged to; - **customer**: who it was for, as an id; - **environment**: production, staging or development. **Over time:** - spend per feature, model and customer; - **useful spend share**: the part of spend that produced something used, after retries, loops, abandoned and redone calls; - cache-read share per feature; - a month-end forecast against your budget. The useful spend share is the number most dashboards miss. Total spend tells you the bill grew; the useful share tells you whether to worry. ## Three ways to collect it | | Provider dashboard | Proxy or gateway | SDK in your app | |---|---|---|---| | Setup | None | Change the base URL | One import | | Spend per model and day | Yes | Yes | Yes | | Spend per feature and customer | No | Through headers you add | Yes, from labels and code location | | Agent runs and loops | No | Partly | Yes | | Extra network hop | No | Yes | No | | Prompts pass through a third party | No | Yes, unless you host it | No | Many teams start with the provider dashboard, add an SDK or a proxy when the bill becomes a line item someone asks about, and add budgets and alerts last. ## Do you need the prompts? For cost, no. Everything above comes from metadata: token counts, model names, timings and outcomes. Even repeated requests can be found by comparing **salted hashes** of the text instead of the text itself: two identical prompts give the same hash, and nobody can turn the hash back into the prompt without the salt. Keeping prompts out of your cost tool means one less copy of customer data, one less vendor to review, and no question about what happens if that tool is breached. ## Check your numbers Compare tracked spend with your provider invoice each month. If the tool tracks 95% or more of the invoice, you can trust its breakdowns. A big gap means some calls are not instrumented. ## How Zexl does it Zexl is an SDK for Python apps (OpenAI and Anthropic clients) and a receiver for Claude Code's telemetry. It records the fields above, labels calls by feature, run, customer and environment, and sorts every call into useful work or one of five kinds of waste. Prompt text never leaves your process: [what is sent](https://tryzexl.xyz/docs/privacy.md). Enter your invoices and Zexl shows how much of the bill it covers. [Create a free account](https://tryzexl.xyz/sign-up) or read the [quickstart](https://tryzexl.xyz/docs/quickstart.md).