Choose by how you want to start. If you need traces of every model call first, an open tracing platform is the quickest route. If prompt versions and evaluations matter more, a prompt management tool fits. If you want one gateway in front of every model provider, start there and add deeper tracing later.
If your app already makes model calls in production, you've probably got logs. What you might not have is a clear view of which prompt, which model and which input produced a bad answer, and whether last week's change made things better or worse. That's the gap Lunary set out to fill as an observability and prompt management platform, and it's the gap any replacement has to fill too.
The seven tools below are listed in no particular order. Each description reflects what the vendor says about its own product, and the groups underneath show where each one expects you to begin.
Seven LLM tools at a glance
| Tool | Category | Good for |
|---|---|---|
| Arize | Agent observability and evaluation | Improving agents through tracing, evals and experiments |
| Braintrust | AI observability and evals | Catching regressions before they reach users |
| Galileo | AI observability and evaluation | Stopping failures, not only logging them |
| Langfuse | Open source tracing and evals | Tracing and evaluating agents on an open platform |
| Portkey | AI gateway | One gateway in front of many model providers |
| PromptLayer | Prompt management | Versioning prompts and running evals as a team |
| Traceloop | Evals and monitoring loop | Checking every release against continuous evals |
Where each one starts
Tracing and evaluation platforms
Langfuse, Arize, Braintrust and Galileo all centre on tracing what your app does and scoring the results. One is open source, which matters if you want to self-host or keep trace data in your own cloud. The others are hosted platforms that pitch evaluation, experiments and regression checks alongside tracing.
Prompt management first
PromptLayer starts from the prompt. Versions, evals and logs sit around the prompt itself, which suits teams where product managers or writers edit prompts alongside engineers.
Release checks as a loop
Traceloop frames its product as a continuous loop between evaluations and monitors, so each release gets checked against what came before. That shape fits teams shipping prompt or model changes often.
A gateway in front of the models
Portkey sits between your app and the model providers, routing calls through one gateway. Its homepage now says the product runs under the Prisma AIRS AI Gateway name, so check the current setup and terms before you build on it.
The seven, alphabetically
In no particular order, sorted A to Z. Each line covers what the vendor says the tool is for.
Arize
Best for: AI teams improving agents over time through observability, evaluation, tracing and experiments in one platform, as the vendor describes its focus.
Braintrust
Best for: Engineering teams tracing production traffic and running evals to catch regressions before users see them, which is the vendor's core pitch.
Galileo
Best for: Teams that want observability paired with evaluation, where the vendor's stated aim is stopping AI failures, not only reporting them.
Langfuse
Best for: Developers who prefer an open source platform for tracing and evaluating agents, using production data to find and fix weak behaviour, per the vendor.
Portkey
Best for: Organisations putting generative AI into production across many teams; the vendor's homepage says the product now runs as Prisma AIRS AI Gateway.
PromptLayer
Best for: Teams treating prompts like code, with versioning, evals, tracing, logs and regression sets for agents in production, according to the vendor.
Traceloop
Best for: Product engineers wanting evals and monitors joined in one feedback loop, so each release is checked as it ships, per the vendor.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
How much code changes on day one
This is the question that decides most switches, and it rarely appears on a homepage. Some tools wrap your model client, so you change an import and a few lines. Some ask you to add tracing decorators around functions. Gateways ask you to change the base address your app calls, which is quick but puts a new service in the path of every request.
Before you choose, list every place your app calls a model: the main chat flow, background jobs, agents and any retrieval steps. Then check how each tool traces those, and whether it follows open standards for tracing so you could move again later without rewriting everything.
Evaluations need the same care. Ask how test sets are stored, whether you can score with your own rules as well as model-graded checks, and whether results can block a release or only report on it. A tool that traces beautifully but can't run your evals in your release process will leave half the job undone.
Advice for AI teams before you switch tracing tools
Start with one flow, not the whole app. Pick the feature users complain about most, instrument it in two shortlisted tools, and run the same test set through both. You'll learn more from comparing how each one shows the same bad answer than from any demo.
Keep your prompts and test cases in your own repository as well as in whichever tool you pick, so moving later is a copy job rather than a rebuild. And don't forget the people writing prompts: if they're not engineers, the tool needs a workspace they can use without touching code. If your team is still building its prompt skills, my guide to prompt engineering basics and this ChatGPT prompts guide are good places to start.
Do you make LLM observability or evaluation software? Tool makers can ask to be considered for a featured place on this list.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What is LLM observability?
It's tracing and logging every call your app makes to a language model, with the prompt, response, timing and context, so you can see why an answer went wrong and fix it.
What's the difference between tracing and evaluation?
Tracing records what happened on each call. Evaluation scores the results, either against your own rules or with a model acting as a grader, so you can tell whether a change helped.
Do I need an AI gateway as well as tracing?
Not always. A gateway helps when you call several model providers and want routing, fallbacks and one place for keys. If you use a single provider, tracing alone is often enough.
Can I self-host an LLM observability tool?
Some tools offer open source or self-hosted editions, which helps if trace data has to stay in your own cloud. Data rules vary by country and sector, so check what your regulator expects.
How hard is it to move traces between tools?
Old traces rarely move cleanly. Most teams export what they need for audits, then start fresh in the new tool and keep test sets in their own repository.
