The right replacement for an AI gateway depends on what you used it for and where you want it to run. Routers send each call to the right model, full gateways add governance and tracing, and open source data planes handle agent traffic. Decide first whether you'll self-host or use a hosted service, then compare by job.
Portkey's homepage describes an AI gateway with observability, guardrails, governance and prompt management in one platform for AI teams. Teams looking for a replacement usually relied on one or two of those jobs more than the rest, and that's the best place to start.
There's one more thing to know before the list, and it changes what you're comparing against. Nine tools follow in no particular order, unranked, with no links, and Portkey isn't one of the rows.
What Portkey's homepage now says
Its homepage banner now says Portkey is PRISMA AIRS AI Gateway, generally available for enterprises, and its docs title carries the same name. So the product you're weighing alternatives against is presented today as an enterprise security gateway, not the independent startup many older comparison pages still describe.
That doesn't make it better or worse for you, but it does change the question. If you chose it as a lightweight developer tool, check whether its direction still matches yours. And when you look at alternatives, check who's behind each one and what it's built for, because the tools here range from community open source projects to products from established infrastructure companies.
They also do different jobs. Some are routers that pick or switch models. Some are full gateways with caching, failover and tracing. Some are data planes for agent traffic, and one only chooses which model to call. Lining them up as if they were the same product is how teams end up with the wrong one.
The nine options, briefly
Agent Router
Best for: Platform teams wanting an open source control plane for agentic traffic powered by Envoy; its homepage says it was formerly Envoy AI Gateway.
agentgateway (Solo.io)
Best for: Infrastructure teams wanting a Rust-based gateway for LLM, MCP and inference traffic with OpenTelemetry support.
LangDB
Best for: Agent builders on LangChain, Google ADK, OpenAI and other frameworks who want routing with tracing, monitoring and optimisation built in.
LiteLLM
Best for: Engineering teams that want an open source gateway they can self-host, with one OpenAI-compatible key, spend tracking and routing.
Not Diamond
Best for: Developers of coding agents who only need something to choose the model for each request, not a full gateway with governance.
OpenRouter
Best for: Builders who'd rather not run anything themselves and want one hosted interface to many models, with model comparison for their prompts.
Plano
Best for: Teams building agentic apps who want an open source proxy and data plane on GitHub for LLM routing, observability and agent orchestration.
Requesty
Best for: Companies wanting smart routing, caching, failover and observability behind one OpenAI-compatible API, with EU data residency support (vendor claim).
TensorZero
Best for: Production LLM teams that want gateway, observability, optimisation, evaluations and experimentation from one open source project.
Self-hosted or hosted, router or full gateway
Open source you run yourself. LiteLLM is an open source gateway you can self-host, putting your models behind one OpenAI-compatible key with spend tracking and routing. TensorZero bundles a gateway with observability, evals and experimentation in one open source stack. Plano is an open source proxy and data plane for agentic apps, on GitHub. Agent Router is an open source control plane for agent traffic built on Envoy, and its homepage says it was formerly Envoy AI Gateway.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
Gateways from infrastructure vendors. Solo.io's agentgateway handles LLM, MCP and inference traffic, is written in Rust and supports OpenTelemetry, which suits teams already running service meshes or API gateways.
Hosted gateways and routers. Requesty routes every call through one OpenAI-compatible API with routing, caching, failover and observability, and says it supports EU data residency. LangDB leads with agent debugging and tracing as well as routing, and works with several agent frameworks. OpenRouter gives you one hosted interface to many models, so you don't run anything at all.
Model choice only. Not Diamond picks the model for each request, aimed at coding agents. It's a router, not a gateway with governance, so pair it with something else if you need logging or policies.
Portkey alternatives side by side
| Tool | Type | Good for |
|---|---|---|
| Agent Router | Open source control plane | Envoy-based control of agent traffic |
| agentgateway (Solo.io) | LLM and MCP gateway | One gateway for LLM, MCP and inference traffic |
| LangDB | Gateway with tracing | Debugging agents as well as routing them |
| LiteLLM | Open source gateway | Self-hosting one OpenAI-compatible key for every model |
| Not Diamond | Model router | Picking the right model for coding agents |
| OpenRouter | Hosted model router | Trying many models through one hosted interface |
| Plano | Open source agent proxy | Routing and orchestrating agent traffic you host |
| Requesty | Gateway and router | Failover and caching with EU data residency |
| TensorZero | Open source gateway and optimisation | Gateway, evals and experiments in one open stack |
My advice for teams leaving Portkey
If governance and guardrails were the reason you chose it, look hard at the full gateways and the infrastructure options, and ask each vendor how policies, keys and audit logs work. If you mainly used it to route calls across providers, a router or a hosted gateway is a lighter replacement. If you used its tracing, check whether your new gateway does enough or whether you'll add a separate observability tool.
Self-hosting gives you control over where prompts and keys live, but you're the one keeping it up. My guide to the best LLM gateways covers the wider field, my list of API gateways helps if you'd rather extend the gateway you already run, and my roundup of LLM observability tools covers tracing you can add on top.
A short checklist for moving gateway traffic
- List every app, key and provider that currently routes through the gateway.
- Note which features each app depends on: fallbacks, caching, guardrails, spend limits per key or prompt templates.
- Point one low-risk service at the new gateway and compare latency and error rates.
- Recreate policies and rate limits before moving anything customer-facing.
- Move the remaining services one at a time, keeping the old route available for rollback.
Because most of these tools speak an OpenAI-compatible API, the code change is often small. The settings around it are where switches go wrong, so give them the most attention.
Do you make AI gateways or LLM routers? Tool makers can ask to be considered for a featured place on this list.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What's the difference between an LLM router and an AI gateway?
A router decides which model or provider handles each request. A gateway sits in front of your model calls and usually adds caching, failover, keys, policies and logging as well as routing.
Is there an open source alternative to Portkey?
Yes. Several gateways and agent proxies are open source and can be self-hosted, which keeps prompts and keys on your own infrastructure.
Do I need to change my code to switch AI gateways?
Often only slightly, if both gateways use an OpenAI-compatible API. Model names, headers and gateway-specific features are where changes usually show up.
Should I self-host an AI gateway or use a hosted one?
Self-hosting gives you control over data and keys, but you run upgrades and scaling yourself. A hosted gateway is quicker to start, so weigh control against upkeep.
What is an MCP gateway?
It's a gateway that handles Model Context Protocol traffic between agents and the tools they call, so you can apply the same routing, security and logging to tool calls as to model calls.
