AI Gateway Load Testing: What Breaks at Concurrency (and How to Fix It)

The morning your whole team opens Claude Code at once, a tool call that should come back in under a second can take seventeen. The model is fine; the wait lives in the gateway in front of it.

Load testing an AI gateway means driving concurrent traffic through it and measuring what happens at the peak, not the average. A gateway that is invisible at one request every twenty seconds can add seventeen seconds of latency the morning a hundred developers land at once.

Most of the day that gateway is invisible. A hundred developers doing normal work average about one request every twenty seconds, and any gateway ever written can serve that. The one you choose only starts to matter in the few seconds when everyone lands together: the morning rush, a CI run fanning out, three agent workflows firing at once.

So we measured exactly those seconds. Four gateways, the same AWS hardware, the same rush.

ONE TOOL CALL · 5,000 CONNECTIONS AT ONCE · SAME AWS BOX

Forget the average

Real developer traffic is spiky. Everyone starts their morning inside the same half hour. A CI run fans out a dozen tool calls at once. An agentic workflow chains six calls back to back inside a single session, and three of those workflows fire together. For a few seconds at a time, "one request every twenty seconds" turns into hundreds of requests in flight, and then it falls back to nothing.

A day of one team's tool calls requests in flight · illustrative shape, org telemetry-informed

average ≈ 1 request / 20s morning rush + CI fan-out: hundreds at once

How we ran it

We rebuilt the benchmark Maxim publishes for their Bifrost gateway, then made it harder to argue with. The load generator runs on its own EC2 box, separate from the gateway, so every request crosses a real network hop instead of a loopback shortcut. And every number here comes from a five-minute run held at a constant load, not a sixty-second burst.

The gateway under test and its backend sit on a t2.xlarge in us-east-1. The backend is an nginx mock that answers in about zero milliseconds, so whatever latency or throughput you read is the gateway’s own cost, not the model’s. Four gateways went through it: Highflame (Rust), Bifrost (Go), Portkey (Node), and LiteLLM (Python).

Under steady load, Highflame barely moves

Before the rush, the steady state. Hold a heavy, constant load on each gateway for five minutes and watch for drift.

Highflame held 14,331 requests a second for the full five minutes: its p99 drifted four tenths of a percent, memory stayed flat at 47 MB, and not one request failed across roughly 4.3 million. Bifrost held 6,799, a little under half that. Portkey and LiteLLM never cleared a few hundred a second, each capped by its single interpreted process.

Headroom for when you scale

A rate you can sustain is only half the answer. The other half is what happens when a spike asks for more than the gateway can serve.

Highflame~14,700 rps

pushed 50% over: still 13,595 at 100%, p99 ~12 ms

Bifrost~6,900 rps

held flat at its own ceiling

Highflame's ceiling is a little over twice Bifrost's. Offered half again more than either can serve, both queue the overflow and hold 100% success; neither collapses.

Now everyone hits at once

Here is the version that matches the morning-standup moment: thousands of requests in flight at once, each holding a connection open while it waits. We held 5,000 open at once and kept them there for five minutes. This is the response time a developer actually feels, at p99.

Under the rush Sustained rate

With 5,000 connections held open at once, the compiled gateways answer in under a second while Portkey and LiteLLM stall into seconds and drop between a third and two-thirds of their requests. This is the response time a developer feels, not a clean overhead number.

The pattern comes down to what actually runs the request.

The two natively concurrent compiled gateways, Highflame on Rust and Bifrost on Go, serve thousands of connections at once and stay under a second even under the crush. The two built on an interpreted, largely single-threaded runtime fall off a cliff, for one slow moment stalls every request queued behind it.

What this buys your team

Put Highflame on the hot path and the rush your team makes every morning never turns into a support ticket. Tool calls come back in about the time it takes to notice they left, at the rush and at the ceiling both, so nobody sits on a spinner or re-runs a command to check whether the agent hung.

Caveats

The load generator ran on a separate box from each gateway, the same way for all four, so the comparison between them is fair; whether that matches every detail of Maxim’s own rig, we cannot say, so treat our numbers against theirs with that in mind.

Frequently asked questions

What is AI gateway load testing?

Driving many concurrent requests through the gateway and measuring latency and error rate at the peak, not the average. It tells you what users feel during the morning rush, not during a quiet afternoon.

Why does an AI gateway only slow down under concurrency?

Most of the day the traffic is light enough that any gateway keeps up. The differences show only in the seconds when everyone lands together, when a gateway that queues or blocks starts adding seconds of latency.

How many concurrent connections should an AI gateway handle?

Enough to cover your worst simultaneous burst, not your average load. In these tests we held 5,000 connections open at once; Highflame answered them in under a second while a Python-based gateway took 17 seconds and dropped a third of the calls.

What makes a gateway drop requests under load?

Usually a runtime that can’t keep enough work in flight: a global lock, a thread-per-request model, or unbounded queuing that eventually times out.