AI Gateway Benchmark: Highflame vs Bifrost, Portkey, LiteLLM

Highflame Identity is now open source: agent identity on open standards. Read the launch →

▸ BENCHMARKS

Your gateway is fine, until everyone hits it at once

A hundred developers on Claude Code average about one tool call every twenty seconds. Every gateway can serve that. Then a CI run, a multi-agent workflow, and the morning rush land together, and the gateways stop looking alike. We put four of them through the same crush on a two-host AWS rig. A call that comes back in under a second through Highflame drags out to seventeen seconds through LiteLLM, and most of LiteLLM's never come back at all.

See what a rush does → Talk to engineering

MEASURED AWS t2.xlarge, us-east-1, five-minute sustained runs, with the load generator on a separate host so every request crosses a real network. Full method below.

▸ WHAT A RUSH DOES

Same rush, same hardware, four gateways

Response time when 5,000 connections land at once, and the throughput each gateway holds for five minutes. Switch between them.

Under the rush Sustained rate

Highflame held 14,331 requests a second, its p99 drifting under half a percent across the five minutes and memory flat at 47 MB, a little over twice Bifrost's throughput on the same box. Portkey and LiteLLM never cleared a few hundred a second because each does its real work inside one interpreted process. Nobody dropped a request at this load; the gap here is raw headroom.

So we pushed past the ceiling

We ramped each gateway to its limit, then deliberately offered about 50% more, to see whether it queues or falls over.

Every gateway kept returning 200s; none errored. Pushed half again past its ceiling, Highflame queued the excess and still delivered 13,595 a second at 100% success, its p99 pinned near 12 milliseconds. Bifrost sat at its own ceiling and held. Highflame's ceiling is a little over twice Bifrost's on identical hardware, and neither compiled gateway collapses at the edge.

▸ WHY THE TAILS DIVERGE

It comes down to the runtime

The two natively concurrent compiled gateways stay flat under load. The interpreted, largely single-threaded runtimes fall into whole seconds when a rush lands because one slow moment stalls everything queued behind it. We have a theory about the exact mechanism in Portkey's case and have not confirmed it with a profiler, so the table shows the measured number, not the diagnosis.

▸ METHODOLOGY

How we ran this

Honesty about test conditions is a feature, not a footnote.

Real hardware, load on its own host The gateway and its backend sit on a t2.xlarge (4 vCPU, 16 GiB, us-east-1). The load generator runs on a separate 8-vCPU EC2 box, so every request crosses a real network hop rather than loopback.

Five-minute sustained runs Every headline figure is a continuous five-minute run held at load, not a sixty-second burst. A one-second sampler records CPU, memory, and TCP state on both hosts throughout.

An instant backend, on purpose The backend is an nginx mock that answers in roughly zero milliseconds, so each number is the gateway's own cost, with the backend-direct figure recorded alongside as the floor.

Four gateways, one harness Highflame, Bifrost, Portkey, and LiteLLM, each through the same load generator and mock, one at a time with no contention between them.

Rush response is not overhead The p99 while holding 5,000 connections is what a user feels under a brief overload, not a clean gateway-overhead number. The clean comparison is the sustained table.

The caveat we are still closing Responses are small fixed-size bodies here, so work a gateway spends parsing and re-serializing large responses is undercounted. The next run standardizes larger responses to close that gap.

▸ FAQ

Reading the numbers

Are these numbers real or a laptop?

Real AWS hardware. The gateway and its backend run on a t2.xlarge in us-east-1, with the load generator on a separate EC2 box so every request crosses a real network. Each figure is a five-minute sustained run.

Why did you measure at a rush instead of average load?

Because average load never stresses a gateway. A hundred developers doing organic tool calls average about one request every twenty seconds. The pressure comes from bursts, CI fan-out, multi-agent workflows, everyone starting the day at once, and that is where the gateways stop behaving alike.

Why is LiteLLM so far behind?

Under thousands of concurrent connections a single Python worker process cannot keep up, so requests queue for seconds and some time out. It is a runtime limit, not a configuration mistake.

Is Highflame just a faster proxy, then?

No. Highflame inspects every LLM and MCP call inline against your policy before forwarding it. Speed is what makes running that inspection on the hot path viable instead of a tax teams switch off.

Can I reproduce this?

Yes. The harness, the configurable mock, and the gateway adapters are in the firehog performance suite. Get in touch and we will point you at it.

SPEED IS THE EASY PART

The interesting question is what runs on the hot path

Highflame stays at the front of that tail-latency chart so it can do the thing a plain proxy does not: inspect every LLM and MCP call inline, against your policy, before it forwards. See what that looks like.