- Two products hide behind one name
- All eight in one table
- Self-hosted open-source gateways
- Managed gateways with no servers to run
- Inside an API-management stack
- Open source, or open core?
- Questions to settle before you shortlist
- When staying put makes sense
- A two-week trial that won't break production
- Frequently asked questions
Swap only once you know which half you're replacing. Teams that want traffic kept in-house usually pick a self-hosted open-source gateway, teams that would rather run no servers lean towards a managed gateway, and companies already standardised on an API-management stack often add an AI gateway to it for tighter governance. A few options also bundle logging and observability.
LiteLLM is an open-source library that gives developers one interface for calling lots of large language models in the OpenAI format. If you're a developer or platform engineer routing several model providers through it, something has probably shifted. Maybe a security review wants the gateway run by someone else, maybe your platform team wants AI traffic inside the tooling it already looks after, or maybe you'd simply like better visibility into what every request is doing.
Whatever the trigger, this page sorts the options by how they run rather than by fame, because a hosted service and a self-hosted binary aren't competing for the same team.
Two products hide behind one name
Start with the project's own docs, which list a Python SDK and a separate AI Gateway as distinct products. One is a library you import into your code. The other is a service that sits between your apps and the model providers. Which half you rely on decides which alternatives are even relevant.
If you only use the SDK inside a single app, most of the names below are a bigger change than you need, because nearly all of them are gateways: something your requests travel through, not something you call from inside your code. That's fine if you want a central point for keys, logs and fallbacks, but it's a new moving part to deploy and watch.
If you run the gateway half for a whole team, you're in the right place. Your shortlist is about where that gateway lives, who patches it, and what it records about each call.
All eight in one table
| Name | Group | Offered as | Worth a look when |
|---|---|---|---|
| Bifrost | Self-hosted | Open source | Speed is the headline worry |
| Cloudflare AI Gateway | Managed | Service | You want caching and retries handled |
| Helicone AI Gateway | Self-hosted | Open source | You like the OpenAI SDK shape |
| Kong AI Gateway | API platform | Commercial | Governance sits with one platform team |
| OpenRouter | Managed | Service | One endpoint for many models appeals |
| Portkey | Self-hosted | Open source with a managed tier | You might move to hosting later |
| TensorZero | Self-hosted | Open source | You need more than routing |
| Vercel AI Gateway | Managed | Service | Keys and failover belong in one spot |
If you've been told to keep every prompt inside your own network, the managed rows drop out straight away. A two-person startup with no one on call can cross off most of the self-hosted ones just as quickly, which usually leaves a shortlist of two or three.
Self-hosted open-source gateways
These keep the shape you're used to: a gateway you run on your own machines, with the code open for your team to read. You still own upgrades and uptime, so only choose this group if that's a feature for you rather than a chore.
Bifrost
Its README pitches a high-performance, OpenAI-compatible gateway.
Helicone AI Gateway
One API for many providers, via the OpenAI SDK format.
Portkey
Its FAQ says the Gateway is open source, with hosting on top.
TensorZero
An LLMOps platform whose README says it includes a gateway.
Managed gateways with no servers to run
If the main reason you're looking is that nobody wants to babysit another service, a managed gateway hands that job to the vendor. The trade is that your requests pass through infrastructure you don't control, which some security teams won't sign off. We've a wider round-up of gateways for routing model traffic if you'd like more names in this group.
Cloudflare AI Gateway
Adds analytics, caching, rate limits, retries and model fallback.
OpenRouter
Calls itself a unified API for AI models.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
Vercel AI Gateway
Centralises credentials, logs requests and fails over between providers.
Inside an API-management stack
Larger companies often already run an API gateway for everything else. Adding AI traffic to that same layer means one set of policies, one place for audits and one team that knows the tooling.
Kong AI Gateway
Kong's docs call it a connectivity and governance layer.
Open source, or open core?
Several names here describe themselves as open source, yet the details differ. Portkey's own FAQ says its Gateway is open source and free to use, with a managed version on top. Helicone's comparison table marks Helicone as open source and OpenRouter as not. TensorZero is open too, but its README presents the gateway as one piece of a wider LLMOps platform.
For your shortlist, that means reading the licence and the repository, not the headline. Check which features sit in the open code and which arrive only with a hosted plan. If keeping everything in-house is the reason you're moving, make sure the parts you need are in the version you can run yourself.
Questions to settle before you shortlist
- Does every prompt have to stay inside your own network, or is a third party allowed to see traffic?
- Who holds the provider keys today, and who should hold them after the move?
- Which logs do you need per request, and who's allowed to read them?
- What should happen when one provider fails: retry, fall back to another model, or stop?
- Is anyone on your team happy to own upgrades for a self-hosted service?
Write the answers down before you open a single docs page, and share them with whoever signs off on security. They'll knock half the list out on their own, and the rest becomes a fair fight between a handful of names.
When staying put makes sense
If your current setup routes requests reliably, your keys are in one place and the team understands it, moving for the sake of a newer name is a sideways step. The good reasons to switch are concrete: a policy that bans self-hosting, a platform team that insists on its own API layer, or a need for request logs you can't get today. If you're unsure, our OpenRouter vs LiteLLM comparison walks through one common pairing in more depth.
A two-week trial that won't break production
- Pick one low-risk internal app and point only its traffic at the candidate gateway.
- Keep the old route live so you can switch back with a config change.
- Send the same set of prompts through both routes and compare responses and errors.
- Block one provider on purpose and watch how the fallback behaves.
- Open the request history and try tracing one call end to end.
- Ask whoever will own it day to day whether they'd be happy doing so for a year.
- Only then move a second app across.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
Is there a hosted version of the open-source gateways?
Portkey offers one. For the rest, read each project's docs.
Do I need a gateway at all if I only call two models?
Not always. A library inside one app can be enough.
Will my existing OpenAI-style code keep working?
Helicone and Bifrost describe OpenAI-compatible APIs; test your own calls before you move.
Which option suits a company that already manages its APIs centrally?
Kong's AI Gateway, which sits inside that layer.
Why do Portkey's docs mention Prisma AIRS?
Its docs site is titled 'Portkey (Prisma AIRS AI Gateway) Docs'. Check the vendor page for what that naming means for the plan you'd be on.
