API Gateway vs Load Balancer in SaaS Backend Architectures

An API gateway and a load balancer both sit in the traffic path between a client and a backend. That's the whole reason people mix them up. Both can end TLS. Both look at an HTTP request and decide where it goes next. Squint at an architecture diagram and they look like the same box, drawn twice for no reason.

They're not the same box, and mixing them up costs real money later. A load balancer decides which server picks up a connection. An API gateway decides whether the request should happen at all, and what happens to it on the way through. Most teams get this backwards: they treat the gateway as a fancier load balancer, instead of a policy checkpoint that happens to sit near one.

For a SaaS company, that split stops being academic fast.

Take multi-tenancy. Rate limiting one tenant without throttling everyone else needs per-request, per-identity logic, and a load balancer has no concept of "tenant." It sees a connection, not a customer. Auth works the same way: every API call needs to prove who's making it, and that check belongs at the gateway, not buried inside a load-balancing rule where nobody will find it during an incident. Usage-based billing needs call-level counting per customer, which only a gateway gives you. And routing across microservices, sending /auth one place, /billing another, /inference somewhere else based on path and method, is gateway work, full stop.

Most companies now run more than one API gateway at once, often without anyone drawing a clean line between "gateway problem" and "load balancer problem." The failure shows up the same way almost every time: teams end SSL twice, add a latency hop nobody asked for, or leave a security gap because each box assumed the other one was already handling it.

The canonical layered architecture: how load balancer and API gateway fit together

Picture three layers, stacked in order.

An external load balancer sits at the edge. It spreads incoming connections across several API gateway instances, so no single gateway becomes a single point of failure. SSL ends here too, before traffic gets anywhere near application code.

The API gateway sits in the middle. It takes what the edge balancer hands it, checks who's calling, decides if they're allowed to call, applies rate limits, changes the request if it needs changing, and sends it to the right downstream service based on path or method.

A service-level load balancer sits per microservice. The gateway already picked a service. Something still has to pick which instance of that service handles the request, and that's a separate, smaller job, done again, closer to the metal.

None of this is overkill, whatever your gut says when you see three layers doing what feels like one job. Pull out any layer and something specific breaks. Remove the edge balancer and your gateway becomes the single point of failure it was supposed to prevent. Remove the gateway and nothing enforces auth or per-tenant limits, so every service has to rebuild that logic badly, on its own, differently each time. Remove the service-level balancer and one hot instance falls over while its neighbors sit there idle.

Running several gateway instances behind a balancer spreads CPU load across them, and one gateway crashing doesn't take the whole API down with it. "API gateway vs. load balancer" is a false question in any SaaS backend that's actually shipped to production. The real question is where each piece sits in the stack, and how cleanly traffic hands off between them.

A concrete SaaS architecture: the AWS API Gateway and ALB pattern for multi-tenant products

Here's how this plays out in a documented production setup for a multi-tenant SaaS MVP built on AWS.

AWS API Gateway sits as the single public entry point. Every external request hits it first, no exceptions. Behind it, an internal Application Load Balancer (ALB) routes traffic to a monolith running on EC2 instances inside an Auto Scaling Group. The frontend, a React app, lives on AWS Amplify. File uploads take a shortcut: the client asks API Gateway for a pre-signed S3 URL, then uploads straight to S3, skipping both the gateway and the monolith for the heavy payload. That makes sense, since nobody wants a 200MB video file eating through gateway request limits meant for small JSON payloads.

Security gets layered instead of bolted on in one spot. WAF protections sit at the API Gateway, right at public ingress. A custom Lambda authorizer checks JWTs for tenant-specific access, chosen over a managed auth provider specifically so the team isn't locked into one vendor's identity system. The /upload endpoint gets a usage plan capping requests at 10 per second, per tenant, so one noisy customer can't starve the rest.

Both pieces earn their keep because they absorb different kinds of load, and this is where people get it backwards: they assume one of the two is redundant. It isn't. API Gateway auto-scales for bursty API traffic without anyone pre-provisioning capacity for it. The ALB, paired with its Auto Scaling Group, handles the monolith's steady grind, scaling on CPU thresholds instead. Neither one does the other's job well, because the traffic shapes don't match. Asking the ALB to do gateway-style per-tenant throttling is like asking a bouncer to also do the bar's books.

Cost at low volume is almost comically cheap. The ALB runs about $18.36 a month, fixed, and the whole setup lands somewhere around $23 to $28 a month at low traffic. API Gateway charges roughly $3.50 per million requests, and that's the number that starts to matter once request volume climbs past MVP scale.

The pattern has real ceilings, worth knowing before anyone copies it wholesale. API Gateway tops out around 600 requests per second before a limit increase is needed, which takes planning, not a support ticket filed at 2am in a panic. Past a certain volume, Gateway's per-request fees can pass the ALB's flat cost, and that crossover is a real trigger to rethink the setup, not a hypothetical someone brings up in a meeting. This build also runs in a single region with no Route 53 failover, a deliberate MVP trade-off. It's not a template for production at real scale, and treating it like one is how a Tuesday afternoon outage happens.

One small detail shows how independently the ALB manages itself: it runs HTTP health checks against a /health endpoint, 15 seconds apart, with a 5-second timeout. The gateway never touches this. The ALB just quietly checks its own pulse, on its own schedule, answering to nobody.

When to use only a load balancer, only an API gateway, or both

A load balancer alone fits when traffic is internal, service-to-service, and already passed through gateway checks upstream. It fits a simple web service with identical servers and no need to know who's calling. It also fits the job of spreading load across gateway instances themselves, which is still a load balancer doing load balancer things, not some secret gateway replacement. If the only real requirement is uptime and raw throughput, with no API lifecycle to manage, skip the gateway.

An API gateway alone fits when traffic is low-volume, internal, or developer-facing, and one gateway instance carries no real availability risk. It fits a serverless backend, where the cloud provider's own gateway already scales sideways, folding the load-balancing job into the platform itself. It's also the right call early, in an MVP, where simplicity beats redundancy: the AWS setup above runs $23 to $28 a month at low traffic.

Both together is the right default for any SaaS product actually charging money, and treating this as a toss-up is the mistake worth naming directly. Multi-tenant products that need per-tenant rate limits, authentication, and usage metering, alongside real uptime and horizontal scale, need the full stack, no shortcuts. So do microservices setups where the gateway fans out to several backend pools, each with its own balancer. AI-native products add another twist: the gateway applies token budgets and model routing on top of ordinary API policy, a genuinely different concern from deciding which server picks up a connection.

The test is simple. Ask "which server should handle this?" and that's a load balancer problem. Ask "who is this, are they allowed to do this, how much have they already used, and which version of the service should answer them?" and that's a gateway problem. In SaaS, both questions get asked on nearly every request, which is exactly why both pieces show up together so often.

One more distinction worth making: a standard API gateway handles transport, routing, and auth. An LLM gateway adds model-aware routing, tracks token spend and cost per call, and can switch between model providers based on meaning, not just availability. That's a specialization stacked on top of the standard gateway and balancer setup. It doesn't replace either one.

What this architecture decision implies for the rest of your backend

Put JWT checks, OAuth 2.0, and API key validation at the gateway, and every downstream service gets requests that are already authenticated. Nobody writes their own auth logic five times over in five different services, which is exactly the kind of redundancy that quietly rots into five different bugs, each one slightly different from the others.

Rate limits enforced at the gateway aren't just a security control. They're the same data a usage-based billing system needs. Per-tenant quotas at the gateway layer double as the metering layer that tells finance what to invoice, without anyone building a second tracking system just for money.

Watching the system needs both layers, and neither substitutes for the other. Gateway logs show per-customer, per-endpoint detail: who called what, how often, and when it failed. Load balancer metrics show total throughput and which servers are healthy. One tells you about a customer. The other tells you about a machine, and mixing up the two means debugging the wrong problem at 11pm.

The failure worth watching for is fragmentation: auth in one tool, rate limiting in another, usage metering bolted onto a third. Every extra system is another place those numbers drift apart, and reconciling three sources of truth in the middle of an incident is nobody's idea of a good time. The gateway should be the one place policy gets enforced, not one of three tools all half-doing the same job while pointing fingers at each other during a postmortem.

Most IT teams say flat out they'd rather run one unified system than juggle a pile of point tools for automation, discovery, management, security, and spend. That's the same instinct showing up at the business layer that the layered gateway-and-balancer setup expresses at the infrastructure layer: keep policy in one place, spread load in another, and don't let the two jobs blur together. A gateway enforcing clean, well-defined per-tenant rules is what lets a backend tell a founder, in real time, who just signed up, how much they're using, and what that usage should actually cost them.