API Gateway in System Design with Real Examples

An API gateway in system  design is the single front door that every client request walks through before it reaches your backend services. Netflix is a good example of how much rides on that door. When Netflix open-sourced its gateway, Zuul 2, in 2018, its engineering team wrote that they ran more than 80 clusters of Zuul 2, sending traffic to about 100 backend service clusters, at more than 1 million requests per second. Phones, TVs, laptops and game consoles didn't need to know where any of those backend services lived. They talked to the gateway and the gateway handled the rest.

That's the core idea and it's simple. The details are where interviews and production systems get interesting: what the gateway should do, what it should never do, how it limits traffic, how it checks identity, how it avoids becoming a bottleneck or a single point of failure and how it differs from a load balancer, a reverse proxy or a service mesh.

This guide explains all of it in plain English. Each section is written so you can understand it on its own and explain it in an interview without needing another source. The code examples were run and the outputs shown are real.

API gateway in 60 seconds

• What: a server that sits between clients and your backend services. Every request goes through it.
• Why: clients get one address instead of dozens and common work (login checks, rate limits, logging, TLS) happens once instead of in every service.
• Core jobs: routing, authentication, rate limiting, request aggregation, protocol translation, caching, timeouts and retries, monitoring.
• Main risks: it can become a single point of failure, a performance bottleneck or a dumping ground for business logic.
• Not the same as: a load balancer (spreads traffic across copies of one service) or a service mesh (handles traffic between services inside the system).
• Popular options: Amazon API Gateway, Kong, Envoy-based gateways, NGINX, Spring Cloud Gateway, Azure API Management, Google Apigee.

What Is an API Gateway?

An API gateway is a server that acts as the single entry point for client requests to a group of backend services. It receives every request, applies shared rules such as authentication and rate limiting, sends the request to the right service and returns the response to the client.

First, two quick definitions:

• An API (Application  Programming Interface) is a defined way for one program to ask another program for data or actions. When a food delivery app asks "show me restaurants near me," it's calling an API.
• A microservice is a small, independent service that owns one part of a system, such as users, orders, payments or search. Large apps are often split into dozens or hundreds of them.

A simple analogy: the reception desk

Think of a large office building with many departments. Visitors don't wander the corridors looking for the right room. They go to the reception desk. The receptionist:

• checks their ID (authentication),
• checks whether they're allowed to visit that department (authorization),
• tells them which floor to go to or calls the department for them (routing),
• stops a crowd from rushing in all at once (rate limiting),
• writes their name in the visitor log (logging).

The departments don't each need their own security guard and visitor log. They trust the reception desk for the common work and focus on their real job. An API gateway is that reception desk for your backend.

Key terms in plain English

Interview answer: "An API gateway is the single entry point for clients into a microservices system. It routes requests to the right service and handles shared concerns like authentication, rate limiting, TLS and logging in one place, so clients stay simple and services can focus on business logic."

Why Do We Need an API Gateway?

The easiest way to understand the value of an API Gateway is to examine what happens when a system operates without one.

The problem: clients talking to every service directly

Imagine an e-commerce app with separate services for users, products, carts, orders, payments and reviews. Without a gateway, the mobile app talks to all of them directly:

WITHOUT A GATEWAY                          WITH A GATEWAY

 Mobile app ──► users.internal:8081         Mobile app ─┐
            ──► products.internal:8082      Web app ────┼──► API Gateway ──► users
            ──► carts.internal:8083         Partner ────┘    (one address)  ──► products
            ──► orders.internal:8084                                         ──► carts
            ──► payments.internal:8085                                       ──► orders
            ──► reviews.internal:8086                                        ──► payments
                                                                             ──► reviews
 Every client knows every address.
 Every service re-implements auth,
 rate limits, logging and CORS.

That creates six real problems:

1. Clients know too much. The app must know the address of every service. If you split the orders service into two or move it, every app version in users' pockets has to change. Old app versions that people never update will break.
2. Shared work is repeated everywhere. Every service has to check tokens, enforce rate limits, handle CORS (the browser rule about which websites may call your API), log requests and terminate TLS. Some teams will do it well, some badly and some will forget. Inconsistent security is a security hole.
3. Mobile apps become "chatty." Showing one home screen might need data from five services. That's five round trips over a slow mobile network. Each one adds latency and drains battery.
4. Internal details leak out. Exposing every service to the internet reveals how your system is built and gives attackers more doors to try.
5. Protocols don't match. Your services might talk gRPC internally, but browsers and partners expect REST and JSON.
6. Change is hard. Renaming an endpoint, versioning an API or migrating a service becomes a coordinated change across every client.

What the gateway fixes

If the interviewer pushes back: "Isn't a gateway just one more hop and one more thing to fail?" Yes and that's a fair trade-off to acknowledge. The answer is that the hop is usually small compared with service work, it can even reduce total latency by aggregating calls and the failure risk is handled by running it as redundant, stateless instances. Without it, every service carries its own copy of auth and rate limiting, which is a bigger and less visible risk.

How Does an API Gateway Work? The Request Lifecycle

When a request arrives, a gateway runs it through a series of steps, often called a filter chain or plugin pipeline. The exact order varies by product and configuration, but a typical flow looks like this:

 Client request: GET https://api.shop.com/v1/orders/7   (with token)
        │
        ▼
 ① TLS termination ........ decrypt HTTPS
        │
        ▼
 ② Basic checks ........... request size, allowed methods, CORS, IP allow/deny, WAF rules
        │
        ▼
 ③ Authentication ......... who is this? (verify JWT, API key or mTLS certificate)
        │                    fail → 401 Unauthorized
        ▼
 ④ Coarse authorization ... may this client call this route at all? (scopes, roles)
        │                    fail → 403 Forbidden
        ▼
 ⑤ Rate limit / quota ..... has this client used up its allowance?
        │                    fail → 429 Too Many Requests (+ Retry-After)
        ▼
 ⑥ Routing ................ match path, host, method, headers → pick the target service
        │                    no match → 404 Not Found
        ▼
 ⑦ Request transformation . add X-Request-ID and user info headers, rewrite path, convert protocol
        │
        ▼
 ⑧ Forward to service ..... with load balancing, a timeout and maybe a retry or circuit breaker
        │                    timeout → 504 Gateway Timeout; service down → 502/503
        ▼
 ⑨ Response transformation  remove internal headers, reshape or filter fields, cache if allowed
        │
        ▼
 ⑩ Logging, metrics, traces, then reply to the client

A few things about this flow are worth understanding properly, because interviewers often ask about them:

• Cheap checks come first. Rejecting a bad request at step ② or ③ costs almost nothing. Forwarding it to a service and then rejecting it wastes resources. Good gateways fail fast.
• Authentication happens before rate limiting in most setups, so limits can be applied per user or per API key rather than per IP address. Some systems also apply a rough per-IP limit even before authentication, to protect the authentication step itself from floods.
• The gateway passes identity forward. After verifying a token, the gateway usually adds trusted headers, such as the user ID, so services don't need to verify the token again. Services must only trust these headers when the request genuinely came from the gateway (for example, over mTLS or a private network) or anyone could fake them.
• Status codes tell you who failed. A 401, 403, 404 or 429 usually comes from the gateway itself. A 502 or 504 means the gateway was fine but the service behind it wasn't. A 500 usually comes from the service. Knowing this makes debugging much faster.

Interview answer: "A request goes through a filter chain: TLS termination, validation, authentication, authorization, rate limiting, routing, transformation, then it's forwarded with a timeout and load balancing. The response comes back through transformation, logging and metrics. Cheap checks run first so bad requests are rejected before they cost anything."

Core Features of an API Gateway

1. Request routing

Routing is the gateway's most basic job: deciding which backend service should handle each request.

Gateways can route on almost any part of a request:

Weighted routing deserves special mention. It lets you release a new version of a service to a small share of traffic, watch error rates and increase the share step by step. If something goes wrong, you set the weight back to 0% without redeploying anything. This makes the gateway an important deployment tool, not just a traffic cop.

Routing also enables path rewriting. The public URL /api/v1/orders/7 can become /orders/7 on the service. Clients see a clean, versioned public API and services keep their own internal paths.

2. Authentication and authorization

Authentication answers "who are you?" Authorization answers "what are you allowed to do?" Gateways are very good at the first and only partly suitable for the second.

Common ways a gateway authenticates clients:

How JWT verification works at the gateway. A JWT has three parts separated by dots: a header (which algorithm signed it), a payload (claims such as user ID and expiry time, exp) and a signature. The gateway recomputes the signature with the key it trusts and compares. If the signature matches and the token hasn't expired, the claims can be trusted.

The program below checks a token the way a gateway would. It uses the official HS256 example token and key from RFC 7515, the JSON Web Signature standard:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.Base64;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

// What a gateway checks on an HS256 JWT: the algorithm, the signature and the expiry.
// Test token and key are the HMAC example from RFC 7515 (JSON Web Signature), Appendix A.1.
public class JwtCheck {
    static final String TOKEN =
        "eyJ0eXAiOiJKV1QiLA0KICJhbGciOiJIUzI1NiJ9" +
        ".eyJpc3MiOiJqb2UiLA0KICJleHAiOjEzMDA4MTkzODAsDQogImh0dHA6Ly9leGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ" +
        ".dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk";
    static final byte[] KEY = Base64.getUrlDecoder().decode(
        "AyM1SysPpbyDfgZld3umj1qzKObwVMkoqQ-EstJQLr_T-1qS0gZH75aKtMN3Yj0iPS4hcgUuTwjAzZr1Z9CAow");

    static String check(String token, long nowEpochSec) throws Exception {
        String[] parts = token.split("\\.");
        if (parts.length != 3) return "REJECT: not a JWT";
        String header = new String(Base64.getUrlDecoder().decode(parts[0]), StandardCharsets.UTF_8);
        if (!header.replaceAll("\\s", "").contains("\"alg\":\"HS256\"")) return "REJECT: unexpected algorithm";

        Mac mac = Mac.getInstance("HmacSHA256");
        mac.init(new SecretKeySpec(KEY, "HmacSHA256"));
        byte[] expected = mac.doFinal((parts[0] + "." + parts[1]).getBytes(StandardCharsets.US_ASCII));
        byte[] actual = Base64.getUrlDecoder().decode(parts[2]);
        if (!MessageDigest.isEqual(expected, actual)) return "REJECT: bad signature";

        String payload = new String(Base64.getUrlDecoder().decode(parts[1]), StandardCharsets.UTF_8);
        Matcher m = Pattern.compile("\"exp\"\\s*:\\s*(\\d+)").matcher(payload);
        if (!m.find()) return "REJECT: no expiry";
        long exp = Long.parseLong(m.group(1));
        if (nowEpochSec >= exp) return "REJECT: expired at " + exp;
        return "ACCEPT: " + payload.replaceAll("\\s+", " ");
    }

    public static void main(String[] args) throws Exception {
        long beforeExpiry = 1_300_000_000L;   // a moment in 2011, before the token's exp
        long today = System.currentTimeMillis() / 1000;
        String tampered = TOKEN.replace("eyJpc3MiOiJqb2Ui", "eyJpc3MiOiJldmUi");   // "iss":"joe" -> "iss":"eve"

        System.out.println("valid token, checked in 2011:  " + check(TOKEN, beforeExpiry));
        System.out.println("same token, checked today:     " + check(TOKEN, today));
        System.out.println("payload changed by attacker:   " + check(tampered, beforeExpiry));
    }
}

Output:

valid token, checked in 2011:  ACCEPT: {"iss":"joe", "exp":1300819380, "http://example.com/is_root":true}
same token, checked today:     REJECT: expired at 1300819380
payload changed by attacker:   REJECT: bad signature

Three lessons from those three lines:

1. A correct signature is not enough. The RFC's token is genuinely signed, but it expired in 2011. A gateway must check claims like exp too (and usually issuer iss and audience aud).
2. Any change to the payload breaks the signature. Changing joe to eve was detected immediately.
3. Only accept the algorithm you expect. The code rejects anything that isn't HS256. Trusting whatever algorithm the token claims has caused real vulnerabilities, including tokens that claim "alg": "none". In production, use a well-tested JWT library rather than hand-written parsing like this demo and prefer asymmetric keys (RS256 or ES256) so the gateway only needs a public key.

JWT details that matter in production:

• Where the keys come from. With asymmetric signing, the gateway fetches the identity provider's public keys from its JWKS endpoint (JSON Web Key Set, defined in RFC 7517; OpenID Connect providers publish its address as jwks_uri). The gateway caches these keys and refreshes them periodically.
• Key rotation. Each token's header carries a kid (key ID) that says which key signed it. When the provider rotates keys, it publishes the new key alongside the old one for a while. If a token arrives with an unknown kid, the gateway refreshes its key cache (with a rate limit on refreshes, so attackers can't force constant fetches).
• Check all the important claims, not only exp: iss (the issuer is your identity provider), aud (the token was meant for this API, not some other service) and nbf (not valid before this time), allowing a small clock skew of a few seconds.
• The revocation trade-off. Checking JWTs locally is fast because the gateway never calls anyone. The downside is that a stolen or revoked token stays valid until it expires. That's why access tokens are kept short-lived (often minutes) and paired with refresh tokens and why OAuth token introspection (RFC 7662) exists for cases where instant revocation matters more than speed.

Where authorization belongs. The gateway can do coarse-grained authorization: "this token has the orders:read scope, so it may call GET /orders/*." But it usually can't do fine-grained authorization: "may user 42 see order 7?" That requires knowing who owns order 7, which only the order service knows. Leaving that check out of the service is the number one risk on the OWASP API Security Top 10 (2023): Broken Object Level Authorization. The gateway confirms identity; the service still has to check ownership.

3. Rate limiting and throttling

Rate limiting caps how many requests a client can make in a given time. It protects your services from overload, stops one heavy user from hurting everyone else, slows down scraping and brute-force attacks and enforces paid plan limits.

When a client goes over the limit, the gateway returns 429 Too Many Requests (defined in RFC 6585), ideally with a Retry-After header telling the client when to try again.

What to limit by: API key, user ID, tenant (customer company), IP address or route. Most production systems combine several, such as 100 requests per minute per user and 10,000 per minute per tenant, plus a stricter limit on expensive routes like search or login.

The main rate limiting algorithms:

Token bucket is the one most worth understanding, because it's so common. Amazon API Gateway, for example, documents its throttling in terms of a steady rate and a burst, using the token bucket idea. The account-level default is 10,000 requests per second per Region, with additional burst capacity (AWS quotas page).

Picture a bucket that holds 3 tokens and gets 1 new token every second. A client that's been quiet has a full bucket, so it can send 3 requests instantly (a burst). After that, it can only send about 1 per second, as tokens drip back in. The mini gateway later in this article implements exactly this.

Rate limiting across many gateway servers. If you run 10 gateway instances and each keeps its own counters in memory, a client can get up to 10 times its limit by spreading requests across them. Fixes:

• Shared store: keep counters in Redis and update them atomically (for example, with a Lua script). Accurate, but adds a network call to every request.
• Local limits with a share of the budget: each of 10 instances allows a tenth of the limit. Fast, but uneven traffic makes it less accurate.
• Hybrid: local counting with periodic syncing to a shared store.

Stripe's engineering blog post "Scaling your API with rate limiters" describes running several kinds of limiters in production, including a token-bucket request rate limiter backed by Redis, a limiter on concurrent requests and load shedders that drop less important traffic when the system is under strain. That last idea, load shedding, is worth mentioning in interviews: under extreme load, it's better to reject low-priority requests deliberately than to let everything slow down.

Rate limits vs quotas. A rate limit controls speed (requests per second). A quota controls volume over a long period (say, 1 million calls per month on a paid plan). API products usually need both.

4. Request aggregation (API composition)

Aggregation means the gateway calls several services for one client request and combines their responses into a single reply. It cuts round trips, which matters most on mobile networks.

A home screen that needs a user's profile, recent orders and recommendations would otherwise need three calls from the phone. With aggregation, the phone makes one call and the gateway (sitting in the same data center as the services) makes three fast internal calls in parallel.

Two rules make aggregation safe:

1. Call services in parallel, not one after another. Total time becomes the slowest call, not the sum.
2. Give optional data a timeout and a fallback. If recommendations are slow, return the page without them rather than making everyone wait.

import java.util.concurrent.*;

// Gateway-side aggregation: one client request fans out to three services in parallel.
// A slow, non-essential service gets a timeout and a fallback instead of delaying the whole page.
public class Aggregator {
    static final ExecutorService pool = Executors.newFixedThreadPool(8);

    static CompletableFuture<String> callService(String name, long latencyMs, String result) {
        return CompletableFuture.supplyAsync(() -> {
            try { Thread.sleep(latencyMs); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            return result;
        }, pool);
    }

    static String homeScreen(long recommendationsLatencyMs) {
        long start = System.nanoTime();
        CompletableFuture<String> profile = callService("profile", 120, "\"name\":\"Asha\"");
        CompletableFuture<String> orders  = callService("orders", 180, "\"activeOrders\":2");
        CompletableFuture<String> recs    = callService("recommendations", recommendationsLatencyMs, "\"recs\":[\"shoes\",\"socks\"]")
                .completeOnTimeout("\"recs\":[]", 250, TimeUnit.MILLISECONDS);   // optional data: fall back to empty

        String body = "{" + profile.join() + "," + orders.join() + "," + recs.join() + "}";
        long ms = (System.nanoTime() - start) / 1_000_000;
        return String.format("%4d ms  %s", ms, body);
    }

    public static void main(String[] args) {
        homeScreen(150);   // warm-up run, not printed
        System.out.println("sequential calls would take about 120 + 180 + 150 = 450 ms");
        System.out.println("parallel, all healthy:        " + homeScreen(150));
        System.out.println("parallel, recs takes 3 s:     " + homeScreen(3_000));
        pool.shutdownNow();
    }
}

Output (timings from one run; they vary slightly by machine):

sequential calls would take about 120 + 180 + 150 = 450 ms
parallel, all healthy:         181 ms  {"name":"Asha","activeOrders":2,"recs":["shoes","socks"]}
parallel, recs takes 3 s:      251 ms  {"name":"Asha","activeOrders":2,"recs":[]}

Parallel calls took about 181 ms, close to the slowest service (180 ms), instead of roughly 450 ms in sequence. When recommendations took 3 seconds, the page still came back in about 251 ms with an empty recommendations list.

A warning: aggregation logic can grow until the gateway contains real business logic ("if the user is premium, also fetch…"). When that starts happening, move the aggregation into a dedicated Backend for Frontend service (covered in the patterns section) or a GraphQL layer and keep the shared gateway thin.

5. Protocol translation

Clients and services don't always speak the same language. The gateway can translate:

• REST/JSON → gRPC: browsers and partners call a normal JSON API; the gateway converts to gRPC for internal services. Envoy, for example, has a gRPC-JSON transcoder for this.
• gRPC-Web → gRPC: browsers can't make native gRPC calls, so a gateway or proxy bridges gRPC-Web to gRPC.
• HTTP → message queue: a request like "generate report" is accepted by the gateway and placed on a queue and the gateway returns 202 Accepted with a link to check status later.
• REST → SOAP: wrapping an old SOAP system behind a modern JSON API, so new clients never see the XML.
• WebSocket handling: many gateways can accept and route long-lived WebSocket connections, which need different timeouts and load balancing than normal requests.

6. Request and response transformation

Gateways can change requests and responses on the way through:

• Add headers: a request ID, the authenticated user ID, the client's IP as seen by your edge (X-Forwarded-For; see the security section for how to read it safely).
• Remove headers: strip internal headers like X-Powered-By or debug headers before responses leave your network.
• Rewrite paths and versions: /v1/orders and /v2/orders can map to different services or versions (see API versioning below).
• Reshape bodies: rename fields or remove sensitive ones.

API versioning at the gateway. When you need to make a breaking change (removing a field, changing a type, changing behavior), you publish a new version and keep the old one running until clients move. The gateway makes this manageable because it can route each version to a different backend. The common styles:

URL path versioning is the most common and the easiest to explain in an interview. Whatever style you pick, version only for breaking changes (adding optional fields shouldn't need a new version), announce deprecation dates and use gateway metrics to see which clients still call old versions before you turn them off.

Header and path changes are cheap and common. Body transformations are more expensive (the gateway must read and parse every body) and easy to overuse. Heavy reshaping usually belongs in a BFF or in the service itself.

7. Caching

A gateway can cache responses for repeated, non-personal requests, such as a product catalog or exchange rates. A cache hit skips the backend entirely. The same rules as CDN caching apply: never cache personalized responses in a shared cache, include everything that changes the response in the cache key and keep TTLs short for data that changes. Amazon API Gateway's REST APIs, for example, offer an optional cache per stage.

For public, cacheable content, a CDN in front of the gateway usually does this job better and closer to users. Gateway caching is most useful for API responses that a CDN can't easily cache.

8. Load balancing and service discovery

Behind each route there are usually several copies (instances) of a service. The gateway or a load balancer it sends traffic to, spreads requests across them using strategies like round robin, least connections or latency-aware routing.

In dynamic environments (Kubernetes, auto-scaling groups), instances come and go constantly. Service discovery keeps the gateway's list current, using the platform's DNS, a registry like Consul or Eureka or Kubernetes Services. A gateway with a stale instance list sends traffic to servers that no longer exist, so health checks matter: the gateway should stop sending traffic to instances that fail them.

9. Resilience: timeouts, retries, circuit breakers

The gateway sits on every request path, so how it handles slow or failing services decides whether a small problem stays small.

• Timeouts on every upstream call. Without them, one hung service ties up gateway connections and threads until the gateway itself stops responding to everyone. Return 504 Gateway Timeout instead. Managed gateways enforce limits too: Amazon API Gateway REST APIs, for example, document an integration timeout of 50 ms to 29 seconds by default and since June 2024, that ceiling can be raised for Regional and private REST APIs.
• Retries only for safe requests. Retrying a GET is usually fine. Retrying a POST /payments can charge a customer twice, unless the API uses idempotency keys. Retry only on errors that might succeed next time, add jitter and cap attempts.
• Circuit breakers. If a service keeps failing, stop sending it traffic for a short time and fail fast (or return a fallback), then test with a few requests before fully reopening. This gives the struggling service room to recover.
• Load shedding. Under extreme load, deliberately reject low-priority traffic (analytics, background syncs) to protect critical paths (login, checkout).

10. Observability

Because every request passes through it, the gateway is the best place to see your whole API at once:

• Request IDs: generate a unique ID for each request (or accept one from the client), pass it to every service in a header like X-Request-ID and return it in the response. When a user reports a problem, that ID finds every log line across every service.
• Access logs: route, client, status code, latency and bytes for every request.
• Metrics: request rate, error rate and latency per route and per client.
• Distributed tracing: start or continue a trace (using the W3C traceparent header) so tools like OpenTelemetry can show the full path of a request.

11. Edge security basics

Gateways also commonly handle:

• TLS termination and certificate management.
• CORS rules for browser clients.
• Request validation: reject bodies that don't match the API schema, oversized payloads or unexpected content types before they reach services. (Amazon API Gateway, for example, documents a hard 10 MB payload limit for REST APIs.)
• IP allow and deny lists and integration with a WAF (Web Application Firewall) to block common attacks like SQL injection.

Interview answer: "The core gateway features are routing, authentication, coarse authorization, rate limiting, aggregation, protocol translation, transformation, caching, load balancing, resilience with timeouts and circuit breakers and observability. I'd keep fine-grained authorization and business logic in the services."

Building a Mini API Gateway in Java

Reading about features is one thing; seeing them work together is better. The program below is a deliberately small API gateway built only with the JDK. It starts three pretend services, then a gateway that:

• authenticates clients with API keys,
• rate-limits each key with a token bucket (burst of 3, then 1 per second),
• routes by path prefix,
• adds an X-Request-ID header that services can see,
• enforces a 500 ms timeout on the backend,
• returns the right status code for each failure.

import com.sun.net.httpserver.*;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.net.http.*;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.*;

// A teaching-sized API gateway: one entry point that authenticates, rate-limits,
// tags requests with an ID, routes by path prefix and enforces a timeout on backends.
public class MiniGateway {

    // ---------- Token bucket: refills `ratePerSec` tokens per second, holds at most `capacity` ----------
    static final class TokenBucket {
        private final double capacity, ratePerSec;
        private double tokens;
        private long lastNanos = System.nanoTime();
        TokenBucket(double capacity, double ratePerSec) { this.capacity = capacity; this.ratePerSec = ratePerSec; this.tokens = capacity; }
        synchronized boolean tryTake() {
            long now = System.nanoTime();
            tokens = Math.min(capacity, tokens + (now - lastNanos) / 1e9 * ratePerSec);
            lastNanos = now;
            if (tokens >= 1) { tokens -= 1; return true; }
            return false;
        }
    }

    static final Map<String, String> ROUTES = new LinkedHashMap<>();       // path prefix -> backend base URL
    static final Set<String> API_KEYS = Set.of("key-mobile-app", "key-partner-acme");
    static final Map<String, TokenBucket> BUCKETS = new ConcurrentHashMap<>();
    static final HttpClient CLIENT = HttpClient.newBuilder().connectTimeout(Duration.ofMillis(200)).build();

    public static void main(String[] args) throws Exception {
        // Two pretend microservices
        startBackend(9001, "/users/42", 0, "{\"id\":42,\"name\":\"Asha\"}");
        startBackend(9002, "/orders/7", 0, "{\"id\":7,\"status\":\"PACKED\"}");
        startBackend(9003, "/reports/daily", 2_000, "{\"rows\":120}");          // a slow service
        ROUTES.put("/api/users/", "http://localhost:9001/users/");
        ROUTES.put("/api/orders/", "http://localhost:9002/orders/");
        ROUTES.put("/api/reports/", "http://localhost:9003/reports/");

        HttpServer gateway = HttpServer.create(new InetSocketAddress(9000), 0);
        gateway.createContext("/", MiniGateway::handle);
        gateway.setExecutor(Executors.newFixedThreadPool(16));
        gateway.start();

        // ---------- Try it ----------
        call("GET /api/users/42 (valid key)", "/api/users/42", "key-mobile-app");
        call("GET /api/orders/7 (valid key)", "/api/orders/7", "key-mobile-app");
        call("GET /api/users/42 (no key)", "/api/users/42", null);
        call("GET /api/payments/1 (no route)", "/api/payments/1", "key-mobile-app");
        call("GET /api/reports/daily (slow backend)", "/api/reports/daily", "key-partner-acme");
        for (int i = 1; i <= 4; i++) call("burst #" + i + " from partner", "/api/orders/7", "key-partner-acme");
        System.exit(0);
    }

    static void handle(HttpExchange ex) throws IOException {
        String requestId = UUID.randomUUID().toString().substring(0, 8);
        String path = ex.getRequestURI().getPath();
        String apiKey = ex.getRequestHeaders().getFirst("X-API-Key");

        // 1. Authentication
        if (apiKey == null || !API_KEYS.contains(apiKey)) { reply(ex, 401, "{\"error\":\"missing or invalid API key\"}", requestId); return; }

        // 2. Rate limiting per client: burst of 3, then 1 request per second
        if (!BUCKETS.computeIfAbsent(apiKey, k -> new TokenBucket(3, 1)).tryTake()) {
            ex.getResponseHeaders().set("Retry-After", "1");
            reply(ex, 429, "{\"error\":\"rate limit exceeded\"}", requestId); return;
        }

        // 3. Routing: first matching path prefix wins (real gateways also match host, method, headers)
        String target = null;
        for (Map.Entry<String, String> r : ROUTES.entrySet()) {
            if (path.startsWith(r.getKey())) { target = r.getValue() + path.substring(r.getKey().length()); break; }
        }
        if (target == null) { reply(ex, 404, "{\"error\":\"no route\"}", requestId); return; }

        // 4. Forward with a request ID and a hard timeout
        HttpRequest forward = HttpRequest.newBuilder(URI.create(target))
                .header("X-Request-ID", requestId)
                .timeout(Duration.ofMillis(500))
                .GET().build();
        try {
            HttpResponse<String> resp = CLIENT.send(forward, HttpResponse.BodyHandlers.ofString());
            reply(ex, resp.statusCode(), resp.body(), requestId);
        } catch (HttpTimeoutException e) {
            reply(ex, 504, "{\"error\":\"upstream timed out\"}", requestId);
        } catch (Exception e) {
            reply(ex, 502, "{\"error\":\"upstream unavailable\"}", requestId);
        }
    }

    static void reply(HttpExchange ex, int status, String body, String requestId) throws IOException {
        byte[] b = body.getBytes(StandardCharsets.UTF_8);
        ex.getResponseHeaders().set("Content-Type", "application/json");
        ex.getResponseHeaders().set("X-Request-ID", requestId);
        ex.sendResponseHeaders(status, b.length);
        ex.getResponseBody().write(b);
        ex.close();
    }

    static void startBackend(int port, String path, long delayMs, String body) throws IOException {
        HttpServer s = HttpServer.create(new InetSocketAddress(port), 0);
        s.createContext(path, ex -> {
            try { Thread.sleep(delayMs); } catch (InterruptedException ignored) { }
            String rid = ex.getRequestHeaders().getFirst("X-Request-ID");
            byte[] b = body.replace("}", ",\"seenRequestId\":" + (rid != null) + "}").getBytes(StandardCharsets.UTF_8);
            ex.sendResponseHeaders(200, b.length);
            ex.getResponseBody().write(b);
            ex.close();
        });
        s.setExecutor(Executors.newCachedThreadPool());
        s.start();
    }

    static void call(String label, String path, String apiKey) throws Exception {
        HttpRequest.Builder b = HttpRequest.newBuilder(URI.create("http://localhost:9000" + path));
        if (apiKey != null) b.header("X-API-Key", apiKey);
        HttpResponse<String> r = HttpClient.newHttpClient().send(b.build(), HttpResponse.BodyHandlers.ofString());
        System.out.printf("%-38s -> %d %s%n", label, r.statusCode(), r.body());
    }
}

Run it with java MiniGateway.java. Output (the same on repeated runs):

GET /api/users/42 (valid key)          -> 200 {"id":42,"name":"Asha","seenRequestId":true}
GET /api/orders/7 (valid key)          -> 200 {"id":7,"status":"PACKED","seenRequestId":true}
GET /api/users/42 (no key)             -> 401 {"error":"missing or invalid API key"}
GET /api/payments/1 (no route)         -> 404 {"error":"no route"}
GET /api/reports/daily (slow backend)  -> 504 {"error":"upstream timed out"}
burst #1 from partner                  -> 200 {"id":7,"status":"PACKED","seenRequestId":true}
burst #2 from partner                  -> 200 {"id":7,"status":"PACKED","seenRequestId":true}
burst #3 from partner                  -> 429 {"error":"rate limit exceeded"}
burst #4 from partner                  -> 429 {"error":"rate limit exceeded"}

Reading the output line by line:

• Lines 1 and 2: valid requests were routed to two different services through one address and each service saw the request ID the gateway added.
• Line 3: no API key, so the gateway answered 401 itself. The user service never saw the request.
• Line 4: /api/payments has no route, so 404 from the gateway.
• Line 5: the reports service takes 2 seconds, but the gateway gave up after 500 ms and returned 504, protecting its own threads.
• Bursts: the partner key's bucket started with 3 tokens. The slow reports call used one token and the half token that refilled during its 500 ms timeout wasn't enough for a third request. So two burst requests succeeded (about 2.5 tokens, minus 2) and the rest got 429 with a Retry-After header. Rate limits count every request, including ones that later fail upstream.

A timeout protects the gateway, not the service. When the gateway returned 504 after 500 ms, the reports service didn't know and kept working for the full 2 seconds. If the client now retries, the service gets a second copy of the same expensive job while still doing the first one. Repeat that across many clients and you have a retry storm against a service that was only slow. The fix is deadline propagation: the gateway tells the service how much time is left (gRPC does this automatically with deadlines; for HTTP, a header carrying the remaining time budget works) and the service stops work or skips it entirely, once the deadline has passed. Pair it with retry limits and jitter on the client side.

This is a teaching example, not production code. A real gateway would also handle TLS, use non-blocking I/O to hold many connections cheaply, share rate-limit state across instances, load-balance across service instances, stream large bodies instead of buffering them and load routes from configuration instead of code.

API Gateway with a Monolith (and the Strangler Fig Pattern)

API gateways are usually discussed with microservices, but they're also useful in front of a monolith, a single application that contains all the features in one codebase.

What the gateway does for a monolith. It can still terminate TLS, authenticate requests, apply rate limits, standardize error responses, log every request and spread traffic across several copies of the monolith. Routing is simpler, because every route goes to the same application.

The biggest reason to put a gateway in front of a monolith: migration. Martin Fowler named this approach the Strangler Fig pattern, after a vine that slowly grows around a tree until it replaces it. The gateway makes it practical:

STEP 1: put the gateway in front
  Client ─► Gateway ─► Monolith (everything)

STEP 2: move one feature out
  Client ─► Gateway ─┬─► New Orders service   (/api/orders/**)
                     └─► Monolith             (everything else)

STEP 3: keep going, one route at a time
  Client ─► Gateway ─┬─► Orders service
                     ├─► Users service
                     ├─► Payments service
                     └─► Monolith (small leftovers or gone)

1. Put the gateway in front of the monolith. Clients now only know the gateway's address.
2. Build one new service (say, orders) and change one route: /api/orders/** goes to the new service. Everything else still goes to the monolith. Clients notice nothing.
3. Repeat, one feature at a time, using weighted routing to send a small share of traffic to each new service first.

This turns a risky "big rewrite" into many small, reversible steps, which is why the gateway is often the first thing teams add when they start breaking up a monolith.

API Gateway Patterns: Single Gateway, BFF and More

Single gateway (one front door for everything)

One gateway serves all clients: web, mobile, partners.

• Good: simple to run, one place for policies, easy to understand.
• Bad: different clients need different things (a TV app wants big, image-heavy responses; a smartwatch wants tiny ones). One gateway serving everyone either gets bloated with client-specific logic or serves nobody perfectly. It can also become a team bottleneck when every change needs the gateway team.

Backend for Frontend (BFF)

A Backend for Frontend is a separate backend layer built for one specific type of client, such as the iOS app, the web app or a partner API. Sam Newman popularized the pattern in a 2015 article. Each BFF calls the internal services, combines and shapes the data exactly the way its client needs and is usually owned by the same team that builds that client.

  iOS app ─────► iOS BFF ──────┐
  Web app ─────► Web BFF ──────┼──► users, orders, catalog, payments ...
  Partners ────► Partner API ──┘
        (often behind one thin shared gateway for auth, TLS and rate limits)

• Good: each client gets exactly the data it needs; client teams move independently; client-specific logic stays out of the shared gateway.
• Bad: more services to run; logic can get duplicated across BFFs; needs discipline about what belongs in a BFF vs a shared service.

A common real-world shape is a thin shared edge gateway (TLS, authentication, rate limits, WAF) in front of several BFFs.

Gateways per domain or team

In large organizations, separate gateways (or separate gateway configurations) are owned by different business domains, for example one for payments and one for content. This avoids one central team becoming a bottleneck, but needs shared standards so security and logging stay consistent.

Edge gateway vs internal gateway

• An edge gateway faces the internet: strict authentication, rate limits, WAF, TLS with public certificates.
• An internal gateway sits between internal domains or teams, with lighter rules. As internal traffic grows, many companies move this job to a service mesh instead.

Named patterns you might hear

Microsoft's Azure Architecture Center describes three related patterns that map neatly onto gateway jobs:

• Gateway Routing: route requests to multiple services through one endpoint.
• Gateway Aggregation: combine multiple backend calls into one.
• Gateway Offloading: move shared work like TLS, authentication and rate limiting from services into the gateway.

Using these names in an interview shows you know the vocabulary.

Checklist for choosing a pattern in an interview:

• One client type or a small system? One gateway is enough.
• Clients need very different data (mobile, web, TV, partners)? Thin shared gateway + one BFF per client type, owned by that client's team.
• Many teams or business domains? Gateways or gateway configurations per domain, with shared security and logging standards.

API Gateway vs Load Balancer vs Reverse Proxy vs Service Mesh

These four get confused constantly, partly because modern products overlap. The clearest way to separate them is by what problem each one is mainly built to solve.

Reverse proxy

A reverse proxy is a server that receives requests on behalf of backend servers and forwards them. It can terminate TLS, cache, compress and hide backend addresses. NGINX and HAProxy are classic examples. An API gateway is a specialized reverse proxy with API-focused features added on top: authentication, per-client rate limits, API keys, quotas, transformations and developer-facing management.

Load balancer

A load balancer spreads traffic across multiple copies of the same service so no single instance is overloaded and it removes unhealthy instances. A Layer 4 load balancer works with TCP/UDP connections and doesn't look inside requests. A Layer 7 load balancer understands HTTP and can route by path or host, which overlaps with basic gateway routing. A load balancer doesn't usually know about API keys, users or quotas.

Service mesh

A service mesh (Istio, Linkerd, Consul) manages traffic between services inside your system. It usually runs a small proxy next to every service (a sidecar, often Envoy) or at the node level and provides mTLS between services, retries, timeouts, traffic splitting and detailed metrics for east-west traffic. A gateway handles north-south traffic: clients coming in from outside.

Kubernetes Ingress and the Gateway API

In Kubernetes, an Ingress resource defines basic HTTP routing from outside the cluster to services inside, implemented by an ingress controller (often NGINX-based). Kubernetes' newer Gateway API reached general availability (v1.0) in October 2023 and offers a richer, more standard way to describe gateways, routes and traffic splitting across many implementations. Many API gateway products can act as the implementation behind these resources.

Comparison table

Real architectures often use several at once:

Internet ─► CDN/WAF ─► Load balancer ─► API gateway (several instances) ─► services
                                                                            │
                                                          service mesh handles traffic
                                                          between the services

Interview answer: "A load balancer spreads traffic across copies of a service. A reverse proxy forwards requests for hidden servers. An API gateway is a reverse proxy specialized for APIs, adding auth, per-client rate limits, aggregation and API management, for north-south traffic. A service mesh handles east-west traffic between services with mTLS, retries and observability."

Popular API Gateway Tools Compared

Choosing a gateway mostly comes down to two questions: managed or self-hosted and how much API management do you need (developer portals, API keys, usage plans, monetization) versus pure traffic handling.

Managed vs self-hosted in one sentence each: managed gateways remove operational work and scale automatically, but you accept their limits, pricing model and lock-in. Self-hosted gateways give full control and can be cheaper at very high volume, but you run, patch, scale and secure them yourself.

Know your managed gateway's limits. For example, Amazon API Gateway documents (at the time of writing) a 10 MB payload limit, a default REST integration timeout ceiling of 29 seconds (raisable for Regional and private REST APIs) and an HTTP API timeout of 30 seconds. Limits like these shape your design: large file uploads usually go directly to storage with pre-signed URLs instead of through the gateway and long-running work uses the 202 Accepted pattern instead of holding a request open.

Real-World Examples

Netflix Zuul: a gateway at very large scale

Netflix announced Zuul as its edge service in 2013. In 2016, its engineering team described rebuilding it as Zuul 2 on an asynchronous, non-blocking architecture (using Netty), so a single server could hold far more connections at once. The rewrite took a long time and was hard to get right: asynchronous code is more difficult to write, test and debug than blocking code. In 2018, Netflix open-sourced Zuul 2 and reported running more than 80 clusters of it, routing to about 100 backend service clusters, at more than 1 million requests per second.

Two takeaways for system design:

1. At scale, the gateway's connection model matters. A blocking, thread-per-request gateway ties up a thread for every slow backend call. Non-blocking gateways (Netty, Envoy, NGINX) handle many more concurrent connections with fewer resources.
2. The gateway is a platform. Netflix used it for routing, monitoring, resilience and security, which is why it ran so many clusters. A gateway at this scale is a product with its own team, not a configuration file.

Stripe: layered rate limiting in production

Stripe's "Scaling your API with rate limiters" describes using more than one kind of limit at once: a per-user request rate limiter (token bucket in Redis), a limiter on concurrent requests for expensive endpoints and load shedders that reserve capacity for critical traffic when the whole system is under stress. It's a good model for interviews: one rate limit is rarely enough. You want fairness between users, protection for expensive endpoints and a way to protect the most important traffic during an emergency.

Amazon API Gateway: limits that shape your design

Amazon API Gateway's documented quotas, such as the 10,000 requests per second account-level default per Region, the 10 MB payload limit and the integration timeouts above, are good reminders that a gateway isn't a pass-through pipe. It's a system with its own capacity, limits and failure modes. You design around it: request quota increases ahead of big launches, send large uploads straight to storage, move slow work to asynchronous processing and watch for 429 responses caused by the gateway's own throttling, not your services.

T-Mobile (2023): one unprotected API is enough

In a January 2023 filing with the US Securities and Exchange Commission (Form 8-K), T-Mobile reported that on January 5, 2023, it identified a bad actor obtaining data through a single API without authorization. The company said the actor first retrieved data around November 25, 2022 and obtained data for approximately 37 million customer accounts, including names, billing addresses, emails, phone numbers and dates of birth (though many accounts didn't include the full data set). T-Mobile said it traced and stopped the activity within a day of detecting it.

The filing doesn't say how the API was deployed or whether a gateway was involved, so it would be wrong to blame any specific product. The general lessons are still clear and widely applicable:

• Every API needs authentication and authorization, including internal-looking or "legacy" ones. A gateway with a default-deny policy (no route is public unless explicitly configured) makes forgotten endpoints much less likely.
• Keep an inventory of your APIs. "Improper Inventory Management" is its own item on the OWASP API Security Top 10. You can't protect endpoints you don't know exist.
• Watch for unusual volume per client. Bulk data extraction looks very different from normal usage. Gateway metrics per consumer and alerts on anomalies shorten the time between "started" and "detected."

API Gateway Security: What It Can and Can't Protect

A gateway is a strong security layer, but only for the problems it can actually see. The OWASP API Security Top 10 (2023 edition) is a useful checklist for being precise about this:

The pattern is clear: the gateway handles "who are you, how much can you do and which doors exist." Services must handle "is this specific thing yours."

Other security practices for the gateway itself:

• Default deny: nothing is reachable unless a route explicitly allows it.
• Lock down the services: services should only accept traffic from the gateway (private network, security groups or mTLS). Otherwise attackers can skip the gateway entirely.
• Never trust identity headers from outside. If the gateway adds X-User-ID, it must strip any X-User-ID sent by the client first.
• Don't trust X-Forwarded-For blindly. This header lists the IP addresses a request passed through and each proxy appends the address it received the request from. But a client can send its own fake X-Forwarded-For: 1.2.3.4, which ends up at the left of the list. If you rate-limit, allow-list or log by IP, take the client address from the right side of the list: skip the entries added by your own trusted proxies (load balancer, CDN) and use the first address that isn't one of them. Using the left-most value lets anyone dodge IP rate limits or pretend to come from an allowed address. The standardized Forwarded header (RFC 7239) has the same rule.
• Don't leak internals in errors. Return clean error messages; log details internally.
• Protect the gateway's own admin API and configuration. Whoever controls routing controls everything.
• Patch it promptly. The gateway parses untrusted traffic from the whole internet. Protocol-level attacks (like 2023's HTTP/2 "Rapid Reset") hit proxies and gateways first.

Scaling an API Gateway and Avoiding a Single Point of Failure

Every request goes through the gateway, so if it's slow, everything is slow and if it's down, everything is down. These practices keep that from happening.

Keep gateway instances stateless. Don't store sessions or important state in gateway memory. Then you can run many identical instances behind a load balancer, add more under load and replace any of them at any time. State that must be shared, like rate-limit counters, goes in a fast shared store such as Redis and the gateway should keep working (perhaps with local limits) if that store is briefly unavailable.

Run across multiple availability zones, and for critical systems, multiple regions. An availability zone is a separate data center within a cloud region; losing one shouldn't take your gateway down.

Keep the gateway thin. No business logic, no database calls for normal requests, no heavy body transformations unless necessary. The more the gateway does, the slower and riskier every request becomes. The microservices article by James Lewis and Martin Fowler (2014) summed up the preferred style as "smart endpoints and dumb pipes": intelligence lives in the services and the communication layer stays simple.

Use non-blocking I/O. Gateways spend most of their time waiting on networks. Event-driven gateways (Envoy, NGINX, Netty-based ones) hold many thousands of connections with few threads.

Budget the latency. A gateway adds an extra network hop plus its own processing. For a well-configured gateway in the same data center, that overhead is usually small compared with the service's own work, but measure it: track gateway processing time separately from upstream time.

Reuse connections to services. Opening a new TCP and TLS connection for every forwarded request wastes time. Gateways keep pools of open connections to each service.

Treat configuration like code. Many large outages start with a bad configuration change, not a software bug. Keep routes and policies in version control, review changes, test them and roll them out gradually with an easy rollback.

Plan for failure of the gateway itself. Health checks so the load balancer removes broken instances, autoscaling with headroom for spikes and rehearsed procedures for rolling back a bad configuration.

Interview answer: "I'd run the gateway as stateless instances behind a load balancer across multiple availability zones, autoscale them, keep rate-limit state in Redis, use non-blocking I/O, keep business logic out and manage configuration through version control with gradual rollouts. That stops the gateway from being a bottleneck or single point of failure."

Monitoring an API Gateway

Because the gateway sees every request, it's the best single place to understand the health of your whole API.

The key metrics, per route and per client:

A common framework for these is RED: Rate, Errors and Duration for every route.

Log every request with a request ID and propagate it (and the W3C traceparent header for tracing) to all services. When a user says "my order failed," the request ID shown in the response lets support find every log line and trace span involved.

Alert on symptoms users feel (error rate and latency on important routes) rather than on every internal signal. And be careful what you log: never log full tokens, passwords or sensitive personal data.

Disadvantages: When Not to Use an API Gateway

A gateway adds real value, but also real costs.

Disadvantages:

• Another moving part: something to deploy, configure, scale, secure, monitor and pay for.
• Extra latency: one more network hop and processing step on every request.
• Single point of failure risk: if it isn't run redundantly, its outage is your outage.
• Bottleneck for teams: if one central team must approve every route change, delivery slows down.
• Business logic creep: convenient "just this once" logic accumulates until the gateway is a fragile monolith that every team depends on.
• Vendor lock-in with managed gateways, through custom configuration formats and features.
• Configuration complexity: with hundreds of routes and policies, it gets hard to know what's exposed and why. A wrong rule can expose an API or weaken access controls for many services at once.

Benefits and challenges at a glance:

When you might not need one (yet):

• A single monolithic application with one backend. A reverse proxy or load balancer is enough.
• A small system with one client and a few services, where a simple reverse proxy handles TLS and routing.
• Purely internal service-to-service traffic, which is usually better handled by a service mesh or direct calls with good client libraries.

When you almost certainly want one:

• Many microservices behind a public API.
• Multiple client types (web, mobile, partners) with different needs.
• Public or partner APIs that need keys, quotas and usage tracking.
• A need for consistent security and monitoring across many teams' services.

API Gateway Best Practices Checklist

A quick checklist that pulls together the advice from this article:

• Encrypt everything: TLS from clients to the gateway and TLS or mTLS from the gateway to services.
• Default deny: nothing is public unless a route explicitly exposes it; services accept traffic only from the gateway.
• Authenticate at the gateway, authorize objects in services.
• Rate-limit per client and per expensive route, with shared state across instances and return 429 with Retry-After.
• Set timeouts on every upstream call, propagate deadlines, retry only safe or idempotent requests and use circuit breakers.
• Keep business logic out. Move client-specific shaping into BFFs.
• Aggregate carefully: parallel calls, timeouts and fallbacks for optional data and no deep chains of calls.
• Use consistent error responses. Every error should have the same shape, the right status code and the request ID. The IETF standard Problem Details for HTTP APIs (RFC 9457, which replaced RFC 7807) defines a common JSON format for this, served as application/problem+json.
• Version APIs for breaking changes and track usage of old versions before removing them.
• Run redundantly: stateless instances across availability zones, autoscaled, with health checks.
• Treat configuration as code: version control, review, tests, gradual rollout, fast rollback.
• Monitor per route and per client: rate, errors (split by gateway vs upstream), latency percentiles and unusual volume.
• Keep an inventory of every exposed route and remove what's no longer used.

Common Mistakes

How to Explain an API Gateway in a System Design Interview

In most system design interviews, the gateway appears in your first diagram. Interviewers want to see that you know what it does, what it doesn't do and how it stays reliable. A good structure:

1. Place it: "Clients connect through a CDN and load balancer to a fleet of API gateway instances, which route to our services."
2. List its jobs for this system: "It terminates TLS, validates JWTs, applies per-user rate limits, routes by path and adds request IDs."
3. Say what stays in services: "Object-level authorization and business logic stay in the services."
4. Cover reliability: "Gateways are stateless and multi-AZ, autoscaled, with rate-limit state in Redis, timeouts and circuit breakers on every upstream."
5. Mention a pattern if clients differ: "The mobile app gets a BFF that aggregates the home screen into one call."
6. Add one number if it helps: "With a token bucket of 100 requests per minute and a burst of 20 per user, a single abusive client can't overwhelm search."

Worked example: API layer for a food delivery app

(This is an illustrative design, not a description of any specific company.)

 Customer app ─┐         │
 Restaurant app├──► Load balancer ──► API gateway fleet (multi-AZ, stateless)
 Delivery app ─┘                       │  TLS, JWT check, per-user rate limits,
 Partner APIs ───► (separate keys,     │  request IDs, routing, timeouts
                    quotas, mTLS)      │
                                       ├──► Customer BFF ──► catalog, cart, orders, payments
                                       ├──► Restaurant BFF ─► menu, orders
                                       ├──► Delivery BFF ───► dispatch, location
                                       └──► Partner API ────► orders (limited scope)
                       Redis: shared rate-limit counters
                       Logs, metrics, traces: every request carries X-Request-ID + traceparent
                       Images and static files: served by a CDN, not the gateway

Decisions to explain:

• Separate partner access with API keys, quotas and narrower scopes, so a partner bug can't hurt customer traffic.
• BFFs per app because a customer app, a restaurant tablet and a delivery partner's phone need very different data.
• Order placement uses idempotency keys, so the gateway or app can retry safely after a timeout.
• Live order tracking uses Server-Sent Events or WebSockets routed through the gateway with longer idle timeouts.
• Large photo uploads (restaurant menus) go straight to object storage with pre-signed URLs, not through the gateway.

API Gateway Interview Questions (with Answers)

FAQ

Key Takeaways