- LLM gateways side by side
- Eleven LLM gateways to compare
- Where to start for your team
- Why OWASP wants limits on every AI app
- Routing, fallbacks and observability explained
- Questions to ask before you choose an AI gateway
- Signs your AI features need a gateway
- My plain advice on choosing an LLM gateway
- Frequently asked questions
The best LLM gateways sit between your app and the AI models it calls, so you can switch models, set limits, log every request and keep things running when one provider has a bad day. Some are open source proxies you host yourself, others are managed services. Eleven options follow in no particular order.
The first AI feature in a product usually calls one model directly. By the third or fourth feature, most teams are juggling several providers, several keys and no clear view of who's using what.
The eleven gateways below are listed alphabetically and in no particular order, and every description comes from the tool's own website, read this week. If you're new to model APIs, my plain guide to what the OpenAI API is explains the basics first.
LLM gateways side by side
I checked the free plan column on each tool's own website, as of Oct 2026. Yes means the site names a free plan, a free account, a free tier or an open source version. Trial only means a time-limited trial. Not stated means the pages I read named neither, even where there's a button inviting you to start for free, so check the vendor's page before you sign up.
| Tool | Type | Good for | Free plan |
|---|---|---|---|
| Bifrost | Open source enterprise AI gateway | A fast open source gateway | Yes |
| Cloudflare AI Gateway | AI gateway on Cloudflare's network | Teams already on Cloudflare | Not stated |
| Eden AI | European AI gateway and LLM router | A European, GDPR-minded router | Not stated |
| Helicone | AI gateway with LLM observability | Seeing every request and its logs | Trial only |
| Kong AI Gateway | AI gateway from the Kong API platform | Companies already running Kong | Yes |
| LiteLLM | Open source AI gateway and LLM proxy | Self-hosting a model proxy | Yes |
| OpenRouter | Unified interface for many AI models | One API key for many models | Not stated |
| Portkey | Production stack and AI gateway | Governing AI use across a company | Not stated |
| Requesty | AI gateway and LLM router | Routing across many models | Not stated |
| TrueFoundry | Enterprise AI gateway and MCP gateway | Agents and MCP servers as well | Not stated |
| Vercel AI Gateway | AI gateway from Vercel | Apps already hosted on Vercel | Not stated |
Eleven LLM gateways to compare
Bifrost
Best for: Engineering teams that want an open source AI gateway they can run themselves, with an enterprise trial if they need more.
Cloudflare AI Gateway
Best for: Developers already using Cloudflare who want AI requests routed, logged and controlled through the same network.
Eden AI
Best for: European companies that want one gateway and router for many AI models, with data protection in mind.
Helicone
Best for: Product teams that want an AI gateway with observability, so they can see requests, errors and usage in a dashboard.
Kong AI Gateway
Best for: Platform teams that already manage APIs with Kong and want AI traffic secured and governed the same way.
LiteLLM
Best for: Developers who want an open source LLM proxy that puts many model providers behind one familiar interface.
OpenRouter
Best for: Builders who want one API key and one interface for hundreds of models, with no subscription to manage.
Portkey
Best for: Enterprises that want to observe, govern and secure AI requests, now offered as the PRISMA AIRS AI Gateway.
Requesty
Best for: Startups that want an AI gateway and router giving them access to a very wide range of models through one API.
TrueFoundry
Best for: Larger teams that want an AI gateway plus gateways for MCP servers and agents under one roof.
Vercel AI Gateway
Best for: Web developers building on Vercel who want automatic fallbacks and routing between models without extra setup.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
Where to start for your team
- Self-hosting an open source proxy: LiteLLM or Bifrost.
- One key for hundreds of models: OpenRouter.
- Seeing every request in a dashboard: Helicone.
- Apps already hosted on Vercel: Vercel AI Gateway.
- European data protection in mind: Eden AI.
Why OWASP wants limits on every AI app
OWASP lists unbounded consumption in its Top 10 for LLM applications. Its advice includes rate limiting and user quotas that restrict how many requests a single source can make in a given time, plus timeouts for heavy operations.
Those controls are exactly what a gateway gives you in one place. Without them, one runaway script or one abusive user can drain your usage allowance and slow the product down for everyone else.
Routing, fallbacks and observability explained
Routing decides which model handles each request, so you can send simple jobs to a smaller model and harder ones to a stronger one. Fallbacks switch to a backup provider automatically when your main provider fails.
Observability means logs and dashboards showing every request, response, error and token count. It's how you find out which feature is busiest and which prompts keep failing.
Questions to ask before you choose an AI gateway
- Can I host it myself, or is it managed only?
- Which model providers does it support?
- Does it offer automatic fallbacks between providers?
- Can I set limits per user, team or key?
- Does it log prompts, and can I switch that off?
- How much delay does it add to each request?
Signs your AI features need a gateway
- API keys live in several places. Centralise them.
- A provider outage took you offline. Add fallbacks.
- You can't see who uses what. Turn on request logs.
- Usage spikes surprise you. Set quotas per user.
Do you make an LLM or AI gateway? Tool makers can ask to be considered for a featured place on this list.
My plain advice on choosing an LLM gateway
Add a gateway before you need one. It's far easier to route every call through one place from the start than to untangle a dozen direct connections after a provider outage.
Then set sensible limits on day one and review the logs every week. They'll tell you which features people use and where the models struggle. If you're comparing model access options, my walkthrough on using one interface for many models shows how that works in practice.
Frequently asked questions
What is an LLM gateway?
An LLM gateway is a layer between your app and AI model providers that routes requests, applies limits, logs usage and can switch providers automatically.
Do I need an AI gateway for one model?
Not always, but it still helps with logging, limits and key management, and it makes adding a second model much easier later.
Are there open source LLM gateways?
Yes. Several gateways are open source and can be hosted on your own servers. The free plan column above shows what each tool's own site says.
What's the difference between a gateway and a router?
A router mainly decides which model handles each request. A gateway usually adds limits, logging, security and key management too.
Does a gateway slow down AI responses?
It adds a small step to each request. Ask each vendor how much delay to expect and test it with your own traffic.
Can a gateway help control AI usage?
Yes. Rate limits, quotas and timeouts let you cap how much any user, team or feature can use.
