Best LLM cost monitoring tools, grouped by the job they do

· 7 min read

The right LLM cost monitoring tool depends on the question you need answered. Tracing platforms show cost per trace next to prompts and evals. Gateways meter every request at one point. Cloud cost tools put AI spend next to the rest of your cloud bill. Waste finders show which calls bought nothing.

You may need only one tool from one of these groups, not one of each. This guide sorts the main options by the job they do, so you can pick a group first and a product second.

Why AI spend needs its own monitoring

AI spend is growing fast. Zylo reports that AI-native software spend rose 334% year over year as of September 23, 2026, and rose 507% in organizations with more than 10,000 employees. In organizations with 2,501 to 10,000 employees it rose 447%. Spend on SaaS with AI features rose 143%.

Surprises happen: 78% of IT leaders saw unexpected charges. Part of the reason is that the cost of one request is not fixed. Long contexts, retries, tool loops and reasoning models can multiply per-request cost by 10x before the invoice shows it.

A monthly invoice tells you the total. It does not tell you which feature, agent or retry produced it. That is the gap these tools fill.

What to look for in a cost monitoring tool

Before comparing products, decide which of these you need:

  • Granularity. Per model and per day is enough for budgeting. Finding waste needs per call data grouped by the run or task it belonged to.
  • Attribution. Can you split cost by feature, team, customer or environment? Without labels, a shared API key hides who spent what.
  • Prompt handling. Some tools store full prompts and responses. That helps debugging, and it raises privacy and retention questions. Others record only token counts and timing.
  • Where it sits. An SDK or exporter in your code, a proxy or gateway in the request path, or a billing feed from your cloud provider. Each has different setup and failure modes.
  • Alerts. A dashboard you check weekly will not catch a loop that runs overnight. Look for alerts to the channels your team already watches.
  • License and hosting. If data must stay on your servers, shortlist tools you can self-host.

Tracing and evaluation platforms

These tools record each request as a trace, with prompts, responses, tool calls and cost. They are built for debugging and quality work, and cost views come along with the traces.

Braintrust

Braintrust combines tracing, evals and cost views in one hosted product. Its own figures: the Brainstore database runs trace queries about 80x faster than traditional databases, and Notion went from triaging 3 issues a day to 30.

Good fit for teams that want cost, quality and CI checks in one place.

Open source tracing tools

If you want to run the tool yourself, several open source options exist. Langfuse is MIT. Opik and MLflow are Apache 2.0. W&B Weave is open source, as are OpenLLMetry, TruLens, Phoenix and Evidently.

They differ in focus. Some lead with tracing and prompt management, some with evaluation, some with OpenTelemetry instrumentation that sends data to a backend you already run. All of them can show token counts per trace; check how each one prices models and whether it can group cost by your own labels.

Good fit for teams that need traces and want the data on their own servers.

Gateways and proxies

A gateway sits between your app and the model providers. Every request passes through it, so it can count tokens, apply limits and route to other models.

The strength of a gateway is coverage: nothing reaches a provider without being counted. The cost is a new hop in the request path, which adds latency and becomes something you have to keep running. Good fit for teams that also want fallback, routing or rate limits.

Cloud and finance cost tools

These tools read billing data rather than requests. They are built for finance and FinOps teams who look at AI spend next to compute, storage and SaaS.

Good fit for answering "what did AI cost the company this quarter" across teams and vendors. They do not see inside your app, so they cannot tell which feature or agent drove the spend.

Waste finders

The groups above tell you what you spent. A waste finder tells you which part of it bought nothing: requests re-sent after a timeout, agents that repeat the same step, prompts that never hit the cache, streamed answers closed early, and answers cut off and asked for again.

Zexl is in this group. Its Python SDK wraps the OpenAI and Anthropic SDKs, records only tokens, timing and hashed fingerprints, never prompt text, groups calls into agent runs and prices each wasted pattern by named rules. It also works with Claude Code. It is free during early access. It is not a tracing tool or a gateway, so it is the wrong pick if you need full prompt content or fallback routing.

Good fit for engineering leads who need to know where to cut first.

How the groups compare

GroupSees promptsWhere it sitsQuestion it answers
Tracing and evalsYes, usuallySDK or exporterWhy did this request cost what it did?
GatewaysCan, by designIn the request pathWhat did every request cost?
Cloud cost toolsNoBilling feedsWhat did AI cost across the company?
Waste findersNot necessarilySDKWhich spend bought nothing?

Prices and plans change. Check current prices on each vendor's pricing page before you decide.

How to choose

  1. Write down the question. "What did we spend" is a billing question. "Why did it go up" is a tracing question. "What can we cut" is a waste question.
  2. Check what data you are allowed to send. If prompts cannot leave your servers, rule out hosted tools that store them, or self-host.
  3. Start with your provider's own usage dashboard. It is free and it is the source of truth for the bill. Add a tool when you need a split it cannot give you.
  4. Label calls from day one. Whatever you pick, tag calls by feature and environment. Cost without attribution is hard to act on.
  5. Set one alert. A daily spend alert per project can catch a runaway loop before it becomes an invoice.

What to measure once a tool is in place

A tool only helps if you look at the right numbers. Start with these, in this order:

For the arithmetic behind each call, see how token cost is calculated.

FAQ

How do I pick a tool to track LLM costs?

No single tool fits every team. Pick by job: a tracing platform for debugging, a gateway for coverage and routing, a cloud cost tool for finance, or a waste finder to see what to cut.

Can I monitor LLM costs without storing prompts?

Yes. Cost needs token counts, the model and timing, not prompt text. Cloud billing tools and some SDK based tools work from those alone.

Do I need a gateway to track costs?

No. A gateway gives full coverage, but an SDK or your provider's usage export also works. Choose a gateway if you want routing or fallback too.

Are there free LLM cost monitoring tools?

Yes. Several tracing tools are open source and can be self-hosted, and some hosted tools have free plans. Your provider's usage dashboard is also free.

See where your own AI spend goes

One import in a Python app, or five environment variables for Claude Code. Prompts never leave your servers.