Monitoring an LLM app means recording every model call as a trace, so you can see the prompt, the answer, the latency and the tokens used, then reviewing and scoring a sample. Decide what to log, add an SDK or gateway, group calls into sessions, score live traffic, set how long you keep data and review regularly.
An LLM app fails differently from ordinary software. It rarely crashes. It answers, confidently, and the answer is slow, wrong or strange, and you only hear about it when a user complains. Monitoring is how you find those answers before the complaint arrives. It works in layers: what you record, how you capture it, how long you keep it, and how you review it.
Layer one: decide what you'll need to answer later
You can only debug what you recorded. Before you add any tooling, list the questions you'll want to answer when something goes wrong:
- Which prompt was sent, and which model answered it?
- What did the model return?
- How long did the call take?
- How many tokens did it use?
- Which user session or conversation was it part of?
That list is your logging spec. A single failed answer usually involves several steps, retrieval, a tool call and a model call, so the session identifier matters as much as the prompt.
Layer two: three ways to capture every call
There are three common routes for getting calls into a monitoring tool, and they differ mostly in how much code you change.
A vendor SDK
You add the tool's library to your app and wrap your model calls. This gives the richest detail, because the SDK knows about your prompts, sessions and custom steps.
An OpenTelemetry-based library
You instrument the app with an open standard and send traces wherever you like, including an observability stack you already run. This keeps you from being tied to one vendor's format.
A gateway
You point your existing client at a gateway instead of the model provider, and requests show up in a dashboard. It's the smallest code change, though you see what passes through the gateway rather than every internal step.
Layer three: retention that changes without you touching it
This is the layer people set last, and it's the one to set first. How long traces are kept affects your privacy obligations and how far back you can look when debugging.
LangSmith's docs, at the time of writing (as of Nov 2026), describe two trace tiers: base retention of 14 days and extended retention of 180 days. They also say that actions such as online evaluators and automation rules can upgrade a trace to the longer tier, which keeps the data longer. Matching one run in a trace upgrades the whole trace, and a rule set on threads upgrades every trace in the thread.
So switching on an evaluator to score quality can also change how long customer data sits in your traces, without you ever opening a retention setting. LangSmith's docs add that changes to retention apply to new traces only, which is why the decision belongs before you start sending traces, not after. Whichever tool you use, check what each feature does to retention before customer data flows through it, and keep traces only as long as your debugging and privacy duties need.
Six tools that cover the layers
Langfuse
An open-source AI engineering platform for tracing, prompt management and evaluation. It's self-hostable and extensible, with data exports to blob storage.
Helicone
LLM observability with an AI gateway. You change the base URL and API key in your OpenAI-style client, and requests show up in its dashboard.
LangSmith
A tracing and evaluation platform that traces runs of LLM apps and can score them with online evaluators and automation rules on live traces, with retention organised in tiers.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
Arize Phoenix
Open-source tracing, evaluation, prompt engineering and datasets, built on OpenTelemetry and OpenInference instrumentation. The company also offers a managed enterprise platform.
OpenLLMetry
An open-source project from Traceloop for monitoring and debugging LLM app execution, built on OpenTelemetry, so you can export traces to Traceloop or to the observability stack you already have.
Braintrust
Observability and evals: you send traces from your app, run evals on prompts and models, and it surfaces recurring problems across agent traces.
There's a wider comparison of LLM observability tools if you want more options for the capture layer.
Six steps from first trace to a review routine
- Write your logging spec. Prompt, model, latency, tokens and session, from layer one. This step is a document, not a tool.
- Add tracing. Choose a route from layer two: Helicone for a gateway, OpenLLMetry for an OpenTelemetry-based library, or your chosen tool's SDK. One small code change starts recording every call.
- Group calls into traces and sessions. A whole user conversation or agent run should read as one story. Langfuse groups calls into traces and sessions for this.
- Score a sample of live traces. Run automatic scoring and add human review for unclear cases. Braintrust and Arize Phoenix both support evaluation, and LangSmith runs online evaluators. Dashboards show volume; scores show quality.
- Set retention before customer data arrives. Check what each feature does to that setting, using the LangSmith example above as your warning.
- Review on a schedule. Look at slow, failing and low-scoring traces regularly and turn the worst into test cases. Braintrust's pattern discovery helps spot recurring problems. Monitoring pays back when it feeds your fixes and your evaluation set.
Once you're scoring traces, a dedicated look at LLM evaluation tools helps you build the test set, and guardrails that check outputs before users see them cover the problems you'd rather prevent than catch.
My take on where to begin
Start with the boring layer: write down what you'll need to answer later, add tracing with the smallest change that captures it, and set your retention before a single customer conversation passes through. Scoring and dashboards are worth adding, but only once you know how long the data they touch will stay around.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What should I log from an LLM app?
At minimum the prompt, the response, the model, latency, token use and a session or user identifier, so one conversation can be followed end to end.
What is tracing for an LLM app?
A trace is a step-by-step record of one run, including model calls, retrieval, tool use and custom logic, so you can see where time is spent.
Can I monitor an LLM app without a commercial tool?
Yes. Several tools here are open source, including Arize Phoenix and OpenLLMetry. Check each project's current terms before relying on a hosted version.
Do I need to change my code to monitor calls?
Often a little. Helicone's quickstart changes the base URL and API key of an OpenAI-style client, while SDK-based tools are initialised inside your app.
How long should I keep traces?
Keep them as briefly as your debugging and privacy duties allow. LangSmith's docs link retention to GDPR and CCPA obligations to delete personal data once it's no longer needed.
