A Comparative Survey of LLM Gateways: LiteLLM, Portkey, Kong AI, and Bifrost

As production systems began routing requests across multiple model providers, an intermediary layer emerged to handle provider failover, response caching, rate limiting, and cost accounting. These LLM gateways now constitute a distinct infrastructure category. This article compares four widely used gateways on their architecture, feature scope, and reported performance, and notes the provenance of the performance figures.

The role of a gateway

Without a gateway, an application couples directly to each provider and must implement failover and rate-limit handling itself:

graph LR
    APP[Application] --> OAI[OpenAI]
    APP --> ANT[Anthropic]
    OAI --> |rate limited| F1[failure]

A gateway centralises this concern behind a single interface:

graph LR
    APP[Application] --> GW[Gateway]
    GW --> |primary| OAI[OpenAI]
    GW --> |fallback| ANT[Anthropic]
    OAI --> |rate limited| GW
    GW --> |failover| ANT

Surveyed gateways

LiteLLM

LiteLLM [1] is a Python-native gateway exposing a unified API across a large number of providers, with configurable fallbacks.

from litellm import completion

response = completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    fallbacks=["claude-3-sonnet", "gemini-pro"],
)

Its principal strengths are breadth of provider support and an active open-source community; as a Python service, its throughput ceiling is lower than that of compiled alternatives [4].

Portkey

Portkey [2] is oriented toward production use, emphasising guardrails (for example, PII and content filtering), semantic caching, and observability dashboards.

import Portkey from 'portkey-ai';

const portkey = new Portkey({
  apiKey: 'PORTKEY_API_KEY',
  config: { retry: { attempts: 3, onStatusCodes: [429, 503] } },
});

Kong AI Gateway

Kong AI Gateway [3] extends the established Kong API gateway with AI-specific routing, and targets enterprise deployments already using the Kong ecosystem.

services:
  - name: ai-service
    plugins:
      - name: ai-proxy
        config:
          route_type: llm/v1/chat
          model: { provider: openai, name: gpt-4o }

Bifrost

Bifrost [4] is a gateway implemented in Go, prioritising low per-request overhead and small resource footprint at the cost of a smaller feature set and ecosystem.

Reported performance

Independent, standardised benchmarks across gateways are scarce; most published figures originate from vendors and use differing methodologies, so they should be read as indicative rather than definitive. Kong published a benchmark comparing Kong AI Gateway, Portkey, and LiteLLM, conducted on AWS EKS with a mocked (WireMock) OpenAI-compatible backend to isolate gateway overhead from provider variability [5]. As with any vendor-run comparison, methodology and incentives should be considered when interpreting the results; readers are encouraged to consult the published methodology [5] and, where performance is material, to benchmark against their own workload.

Qualitatively, the architectural expectation is consistent across sources: compiled proxies (Go-based, such as Bifrost, and the Kong data plane) impose lower per-request overhead than a Python service such as LiteLLM, while hosted gateways add network round-trip latency in exchange for managed features [1–5].

Feature scope

(Feature availability changes over time; verify against each project's current documentation [1–4].)

Selection criteria

The gateways occupy different points in a design space rather than a single quality ordering:

• Provider breadth and rapid prototyping favour LiteLLM [1].
• Guardrails and managed dashboards favour Portkey [2].
• Enterprise integration and an existing Kong deployment favour Kong AI [3].
• Minimal overhead and footprint favour Bifrost [4].

Discussion

A gateway addresses provider-level reliability—failover, retries, caching—but does not by itself provide workflow-level reliability such as state persistence or execution replay. The two concerns are complementary, and a production architecture typically requires both. Gateway selection should therefore be made jointly with, not in isolation from, the execution layer.

References

1. LiteLLM documentation. https://docs.litellm.ai/
2. Portkey AI Gateway (repository). https://github.com/Portkey-AI/gateway
3. Kong AI Gateway. https://konghq.com/products/kong-ai-gateway
4. Bifrost (Maxim). https://github.com/maximhq/bifrost
5. Kong, "AI Gateway Benchmark: Kong AI Gateway, Portkey, and LiteLLM" (vendor benchmark). https://konghq.com/blog/engineering/ai-gateway-benchmark-kong-ai-gateway-portkey-litellm