Match each AI gateway to its distinctive 2026 differentiator

Match pairs·Medium·4.0 · 0·~2 min·Asked atEy Turbopuffer Wandb

Attempt it

Drag each answer to line up with its matching prompt

LiteLLM

Free tier edge gateway with caching, analytics, rate limit at the CDN edge

Portkey

Open-source OpenAI compatible interface in front of 100+ providers, self-hostable proxy

Vercel AI Gateway

Hosted gateway tightly tied to Vercel and Next.js apps with edge routing

Cloudflare AI Gateway

Drop in HTTP proxy that logs every LLM call with one base URL change

Helicone

Hosted gateway plus observability and guardrails, strong on caching and reliability config

An **AI gateway** sits between your app and the LLM providers. It gives you one interface in front of many providers and handles routing, retries, caching, and logging. By 2026 the landscape settled into five names. **LiteLLM** is the open-source one. It exposes an OpenAI-compatible API in front of 100+ providers (Anthropic, Google, Bedrock, Vertex, hosted open weights) and you run the proxy yourself. The de facto open standard. **Portkey** is a hosted gateway focused on reliability config: fallback chains, caching, retries, guardrails, observability all in one product. **Vercel AI Gateway** is hosted by Vercel and integrates with Next.js apps; it does routing at Vercel's edge so latency stays low for web-app workloads. **Cloudflare AI Gateway** sits at the CDN edge with a generous free tier; caching, analytics, and rate-limits run on the edge network itself. **Helicone** is a drop-in HTTP proxy whose pitch is one-line setup: change the base URL and every call is logged. Observability-first rather than routing-first.

**The five gateways split along two axes.** Axis one: **self-hosted versus hosted**. LiteLLM is the open-source self-hostable proxy (you can use their hosted version too, but the OSS is the differentiator). The other four are hosted-first. Axis two: **routing-first versus observability-first**. LiteLLM, Portkey, Vercel, and Cloudflare lead with routing and reliability; Helicone leads with logging and observability. **LiteLLM** is the open-source de facto standard. Single OpenAI-compatible interface in front of 100+ providers, runnable as a Python library or a standalone proxy. Teams that want to avoid hosted-gateway lock-in start here. The catch is that the operational surface (deployment, scaling, alerting) is yours. **Portkey** packages the routing and reliability story as a hosted product: fallback chains, smart retries, semantic caching, guardrails, observability dashboards. Strong fit for teams that want one vendor for everything from routing to logging without standing up the proxy themselves. **Vercel AI Gateway** is the Vercel-flavored hosted gateway. Edge routing, tight integration with Next.js apps, and a billing relationship most Vercel customers already have. Picks up the workload of Vercel-hosted web apps almost by default. **Cloudflare AI Gateway** sits at the CDN edge with a generous free tier. Caching, analytics, rate-limit, and provider failover all run on the same edge network Cloudflare already operates. Strong fit for high-volume read traffic and global workloads. **Helicone** is observability-first: one base-URL change and every call is logged with traces, costs, and user attribution. Routing is supported but the pitch is 'see your traffic in five minutes,' not 'replace your routing layer.'

TL;DR

The 2026 AI-gateway landscape splits along self-host vs hosted and routing-first vs observability-first; each major gateway anchors a distinct quadrant.

Memory aid

Easy to grasp

Picture five different toll booths on the highway to the LLM providers. One is open-source and you build your own booth (LiteLLM). One is a fancy hosted booth that also takes pictures of every car (Portkey). One sits on the same on-ramp as a popular city highway (Vercel). One is built into the road itself so it is free for small traffic (Cloudflare). And one is just a clear glass booth that logs everyone who passes (Helicone). They all let your traffic through, but each one is designed for a different driver.

Key concepts

LLMOps

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

By 2026 the AI-gateway landscape converged on five names that teams routinely have to choose between: LiteLLM, Portkey, Vercel AI Gateway, Cloudflare AI Gateway, and Helicone. They all do the headline thing (one interface in front of many providers, with routing, retries, and logging), but each one anchors a different posture and serves a different team shape. Picking one is a platform decision more than a feature decision, and the wrong choice locks you into a runtime, a billing model, or an observability story that fights the rest of your stack.

This walkthrough explains the differentiator of each gateway, the two axes that organize them, and the decision frame that maps team context to a clean answer.

Mental model: the five gateways live in different quadrants of (self-hosted vs hosted) and (routing-first vs observability-first). Match the quadrant to the team's context and the answer is usually obvious.

LiteLLM: the OSS de facto standard

What it is

LiteLLM is an open-source library and proxy that exposes an OpenAI Chat Completions interface in front of 100+ providers. Anthropic Messages, Google Generative AI, Bedrock, Vertex, hosted open weights (Together, Replicate, Anyscale), and self-hosted vLLM all land behind one schema. It has two deployment modes: as a Python library (import and call from your app) and as a standalone proxy server (deploy and point apps at it).

The differentiator

It is the dominant open-source option, by a wide margin in GitHub-star and adoption metrics. Teams choose it when they want to avoid hosted-gateway lock-in or when compliance requires the gateway run inside their own VPC. The codebase is permissively licensed and the OpenAI-compatible interface is the closest thing to a standard in this space.

What it does not give you

Deep observability. Routing, retries, and per-key budgets are all there, but the dashboard and tracing story is intentionally minimal; the expectation is that you pair LiteLLM with Langfuse, Phoenix, or another observability backend. Operations are yours too: scaling, alerting, and high-availability for the proxy.

Hosted LiteLLM

LiteLLM also offers a hosted version. That collapses some of the operational story but the OSS posture is the differentiator most teams choose it for.

Portkey, Vercel, Cloudflare: three hosted shapes

Helicone: observability-first proxy

Decision frame and common pitfalls

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

LiteLLM is the most-starred OSS LLM gateway and the standard 'thin proxy' choice in teams that want to swap providers without rewriting call sites.
Portkey is widely adopted at mid-stage startups that need hosted reliability config (fallback chains, semantic caching) without standing up their own proxy.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhen would you self-host LiteLLM as a proxy versus import it as a Python library?

Library mode is fine for single-app contexts where the call site can absorb the dependency. Proxy mode wins when multiple apps and languages need the same routing layer, when you need a single audit point, or when you want to enforce per-tenant rate limits at the gateway.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Treating gateways as interchangeable. The differentiators are real and the wrong choice locks you into the wrong platform shape (CDN, Vercel, self-host).

60 second bullets to scan on the way to the call.

The five major 2026 AI gateways and their distinctive postures
Why LiteLLM is the OSS de facto standard and what it does not give you

Primary sources. Browse if you want the original framing.