Asset 20 8 2
Does AI recommend your business? Run the free check →

Join 15,000 business owners, marketers and entrepreneurs. The Sunday newsletter you'll be annoyed only arrives once a week.

Article

Best LLM Caching Tools in 2026: 10 Picks for Engineers

LLM caching tools fall into four types: AI gateways that cache responses in front of your model calls, open source libraries you wire into your own code, managed semantic caches, and prompt caching built into the model provider's API. A gateway with exact-match caching is the quickest win; add semantic caching only when you can measure that similar prompts deserve the same answer.

A cache that only matches identical requests will do nothing for a chatbot where every user words the same question differently. That's why the first thing to settle isn't the vendor but the kind of match your traffic needs. Run through the list below before you shortlist anything; each item is something you can confirm in the docs in a few minutes.

Nine things to confirm before you choose an LLM cache

  • Does it match only identical requests, or similar ones by meaning, and does the semantic option need a separate vector database you'll have to run?
  • Where does the cache live: inside each worker process, in a shared store such as Redis, or on an edge network in front of your app?
  • How do you switch it on and off per request, for example with a header or a config flag, so time-sensitive prompts always skip it?
  • Which response types will it store: text only, or images and other outputs your app returns as well?
  • How do you set an expiry time and clear stale entries the day you change a system prompt or swap the model?
  • Does the cache key include the model name, parameters and system prompt, so two different requests can never share one answer?
  • Which plan carries the feature you need, since semantic matching can sit on a higher tier than simple caching?
  • Is it generally available or still in preview, and what support comes with that status?
  • Will hits and misses show up in your logs, so you can prove the saving rather than assume it?

Two of those items, the type of match and where the cache lives, account for most of the difference between the tools below. The rest are what turn a cache that works in a demo into one you can trust in production.

Exact-match or semantic: the decision that shapes the shortlist

Exact-match caching stores a response against the full request and returns it only when the next request is the same, character for character. It's cheap and predictable, because a hit is always an answer to the identical question, though that answer goes stale if you change the system prompt and forget to clear the cache. It shines on batch jobs, evaluation runs, retries, agents that call the same tool prompt over and over, and any feature where the prompt is built from a template with a small set of inputs.

Semantic caching embeds the prompt, looks for a stored prompt that's close enough in meaning, and returns that answer. It catches the support question asked twenty different ways, which is where the bigger savings live. The price is risk: set the similarity threshold too loose and someone asking about cancelling an order gets the answer meant for changing one. It also needs somewhere to keep the vectors, so you're running or paying for a vector store as well as the cache. I'd only switch it on for a narrow set of routes where the answers are stable and a near miss is harmless.

Provider prompt caching is a third thing again. It doesn't return a stored answer at all; it lets the model reuse a long, repeated prefix such as a system prompt or a big document, so the call is faster and cheaper but the model still writes a fresh reply. On the Claude API, for instance, you mark the reusable part with cache_control. It sits happily alongside a response cache rather than replacing one.

Gateway, library or datastore: where the cache sits

A gateway or proxy sits between your app and the model providers, so caching is one switch alongside routing, retries and logging. It's the least code, and if you already run one for observability, caching may be a header away. If you're still choosing that layer, the wider field of LLM gateways is worth a look first, because caching is rarely the only reason to adopt one.

A library lives inside your application. You get fine control over keys, thresholds and storage, and nothing leaves your infrastructure, but you own the plumbing. A managed semantic cache or datastore takes the vector side off your hands and charges for it. One detail catches self-hosters out: an in-memory cache lives inside one worker process, so a proxy running four workers keeps four separate caches, and your hit rate looks far worse than it should. Point it at a shared store from day one.

LLM caching tools worth shortlisting

GPTCache

An open source semantic cache library that slots into LangChain and llama_index projects, for teams happy to run their own storage.

Helicone

Gateway and observability in one hosted service: it keeps complete responses on Cloudflare's edge, and you switch caching on per request with a header.

Portkey

A hosted AI gateway offering simple caching on every plan, with semantic caching reserved for select enterprise plans that include a vector database.

LiteLLM

Engineers who'd rather host their own proxy pick this open source option; pair it with a shared store, because in-memory caches stay inside one worker.

Redis LangCache

Redis's managed semantic cache, still in preview, returns a stored answer when a new prompt is close enough in meaning to an earlier one.

Bifrost

Open source gateway bundling failover, load balancing and semantic caching behind a single OpenAI-compatible API that you run on your own infrastructure.

Work with me

Want AI doing the heavy lifting in your marketing?

I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.

Upstash Semantic Cache

A small library that treats the cache as a fuzzy key-value store, matching on semantic similarity rather than exact text, on top of hosted Redis.

Vercel AI Gateway

For apps already on Vercel, this model gateway adds a prompt-caching setting across providers, while caching full responses stays in your own code.

ModelCache

Codefuse's open source semantic cache stores query and result pairs, so similar requests come back faster without another trip to the model.

Anthropic prompt caching

Not a separate product but a Claude API feature: mark reusable prompt sections with cache_control, either automatically or with explicit breakpoints.

How the bill adds up

Open source libraries and proxies cost nothing to license, but they're not free to run: you pay for the hosting, the shared store and, for semantic matching, the embedding calls and vector database behind it. Some hosted gateways list a free plan with paid tiers above it, and semantic caching can be the feature that moves you up a tier. Managed semantic caches charge for the store and the matching. Provider prompt caching is billed inside your normal token usage rather than as a separate product.

None of that is worth guessing. Each vendor's pricing page has the current terms, and the provider's pricing page tells you how cached tokens are billed. The sum that matters is the one you'll run in the test below: model spend saved by hits, minus what the cache costs to host and query.

Do you make LLM caching tools or AI gateways? Tool makers can ask to be considered for a featured place on this list.

Replay a day of prompts before you commit

Export a day of real prompts from your logs, strip anything personal, and replay them through the cache twice. The first pass fills it; the second shows your exact-match hit rate. That number is the honest ceiling for simple caching on today's traffic, and on a chatbot it's often humbling.

Then switch on semantic matching at a strict threshold and replay again. For every new hit, read the stored answer next to the prompt that triggered it. Count the ones you'd be embarrassed to send. If you find even a handful at a strict setting, keep semantic caching off that route and leave exact match in place. Your tracing tool should show the latency difference per route; if you don't have one yet, the LLM observability tools that log cache hits make this test far easier to repeat each month.

Picking by the app you're running

For a prototype or a small internal tool, I'd use the caching built into whichever gateway you already have, exact-match only, and stop there. For a high-volume product with long, repeated system prompts, turn on provider prompt caching first; it's the cheapest change with no risk to answer quality. Support bots and search-style assistants are where semantic caching earns its place, ideally through a managed cache or a gateway plan that includes it, so you're not maintaining a vector store just for this. And if data can't leave your network, a self-run proxy or library on a shared store is the only route that keeps everything in-house.

Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.

Published and maintained by the Lilach Bullock team, covering marketing, AI and business growth.
Your buyers are asking AI who to use. Does it say you?

See for free whether ChatGPT, Claude, Perplexity, Gemini and Google name you, and get the plan to become the answer.

Check my AI visibility →
Sundays only

Get the Sunday newsletter.

One email a week. AI experiments, marketing tactics, and the workflows Lilach is building right now in her own business.

Subscribe free

Let’s get your marketing running on AI.

Book a free 30-minute call

We figure out what you need, where AI fits in, and what working together would look like.

Book the call →

Or take the 30-second calculator

You’ll see the hours and the money quietly leaking out of your week, and the three workflows worth building first.

Take the calculator →

Or grab the free AI resource library

Prompt packs, templates, checklists, and swipe files. The exact tools I build for paying clients. Yours, free.

Get the library →
Keep reading

More from the blog.