How to Build a Scalable API Gateway with Node.js for Enterprise Applications

The fastest reliable way to build a scalable node.js api gateway is to put a lean HTTP framework such as Fastify in front of your services. Use a pooled HTTP client (undici) for upstream calls. Handle authentication, rate limiting, timeouts, and observability at the gateway layer. Keep the gateway stateless and push shared state into Redis so you can scale it horizontally behind a load balancer. The rest of this post walks through each of those decisions with working code, the trade-offs we weigh on real projects, and the mistakes that tend to surface only under production load.

The Problem a Gateway Actually Solves

If you run more than a handful of services, you have probably seen the symptoms. Every service implements its own JWT validation, slightly differently. Rate limits exist in some places and not others. Clients hard-code internal hostnames. When an incident happens, nobody can trace a request across service boundaries.

A gateway fixes this by giving you a single, policy-enforcing entry point. Microsoft's Azure Architecture Center describes this as two related patterns: Gateway Routing and Gateway Offloading. Routing hides your topology. Offloading centralizes cross-cutting concerns like TLS, auth, and throttling.

Build vs. buy

Before writing code, be honest about whether you should. Managed API management products and mature open-source gateways cover many needs well. We recommend building a custom Node.js gateway when:

• You need custom request composition (aggregating several backend calls into one response for a mobile client).
• Your auth or tenancy logic is domain-specific and awkward to express in a declarative config.
• Your team already operates Node.js in production and can own the gateway as a product, with on-call responsibilities.

If none of those apply, a configured off-the-shelf gateway is usually the lower-risk choice.

Reference Architecture for a Node.js API Gateway

Here is the shape we aim for:

1. Load balancer terminates external traffic and distributes it across gateway instances.
2. Stateless gateway instances (Node.js) handle auth, routing, rate limiting, and resilience.
3. Redis holds shared state: rate-limit counters, optionally cached JWKS or session data.
4. Upstream services are reached by internal DNS or a service registry.
5. Telemetry pipeline (OpenTelemetry collector) receives traces, metrics, and logs.

The key property is statelessness. Any gateway instance must be able to serve any request. That is what lets you scale by adding replicas instead of rewriting.

Step 1: Choose a Lean HTTP Foundation

We default to Fastify for gateways. It has low per-request overhead, a well-defined plugin encapsulation model, and first-party proxy support through @fastify/http-proxy, which is built on undici, the HTTP client maintained under the Node.js project.

npm install fastify @fastify/http-proxy @fastify/rate-limit \
  @fastify/under-pressure ioredis jose

A minimal routing skeleton:

// gateway.js
import Fastify from 'fastify';
import proxy from '@fastify/http-proxy';

const app = Fastify({
  logger: { level: process.env.LOG_LEVEL ?? 'info' },
  requestIdHeader: 'x-request-id',
  trustProxy: true, // we sit behind a load balancer
});

const routes = [
  { prefix: '/orders',   upstream: process.env.ORDERS_URL },
  { prefix: '/catalog',  upstream: process.env.CATALOG_URL },
  { prefix: '/accounts', upstream: process.env.ACCOUNTS_URL },
];

for (const r of routes) {
  app.register(proxy, {
    upstream: r.upstream,
    prefix: r.prefix,
    rewritePrefix: r.prefix,
    undici: {
      connections: 128,        // pooled keep-alive connections per upstream
      headersTimeout: 5_000,
      bodyTimeout: 15_000,
    },
  });
}

await app.listen({ port: 8080, host: '0.0.0.0' });

Why connection pooling matters: without keep-alive, each upstream call pays a TCP (and possibly TLS) handshake. Pooling is one of the simplest ways to keep gateway latency predictable as traffic grows.

Step 2: Authenticate at the Edge

Validate tokens once, at the gateway, and pass verified identity downstream. We use the jose library because it supports remote JWKS with built-in caching and key rotation.

// auth.js
import fp from 'fastify-plugin';
import { createRemoteJWKSet, jwtVerify } from 'jose';

const JWKS = createRemoteJWKSet(new URL(process.env.JWKS_URL));

export default fp(async (app) => {
  app.decorateRequest('user', null);

  app.addHook('onRequest', async (req, reply) => {
    if (req.url.startsWith('/health')) return;

    const header = req.headers.authorization ?? '';
    const [scheme, token] = header.split(' ');
    if (scheme !== 'Bearer' || !token) {
      return reply.code(401).send({ error: 'missing_token' });
    }

    try {
      const { payload } = await jwtVerify(token, JWKS, {
        issuer: process.env.JWT_ISSUER,
        audience: process.env.JWT_AUDIENCE,
      });
      req.user = payload;
      // Forward verified identity; strip anything the client tried to set.
      req.headers['x-user-id'] = String(payload.sub);
      req.headers['x-tenant-id'] = String(payload.tenant ?? '');
    } catch {
      return reply.code(401).send({ error: 'invalid_token' });
    }
  });
});

Two rules we hold to:

• Never trust identity headers from the client. Overwrite them after verification, and make sure upstream services only accept traffic from the gateway (network policy or mTLS).
• Always check iss and aud. Signature validation alone does not prove the token was issued for your API.

Step 3: Distributed Rate Limiting

In-memory rate limiting breaks the moment you run two replicas, because each instance keeps its own counters. Back the limiter with Redis so limits hold across the fleet.

// rate-limit.js
import Redis from 'ioredis';
import rateLimit from '@fastify/rate-limit';

export async function registerRateLimit(app) {
  const redis = new Redis(process.env.REDIS_URL, {
    enableOfflineQueue: false, // fail fast rather than queue unbounded
    maxRetriesPerRequest: 1,
  });

  await app.register(rateLimit, {
    redis,
    max: (req) => (req.user?.plan === 'enterprise' ? 1000 : 100),
    timeWindow: '1 minute',
    keyGenerator: (req) => req.user?.sub ?? req.ip,
    skipOnError: true, // if Redis is down, prefer availability
  });
}

skipOnError: true is a deliberate policy choice, not a default to copy blindly. For a public API vulnerable to abuse, you may prefer to fail closed. Decide explicitly and document it. Return 429 Too Many Requests (defined in RFC 6585) with a Retry-After header so well-behaved clients can back off.

Step 4: Resilience: Timeouts, Breakers, and Load Shedding

A gateway that waits forever on a slow upstream will exhaust its own resources and take down healthy routes with it. We layer three defenses.

Timeouts on every upstream call

The headersTimeout and bodyTimeout settings in Step 1 cover this. Set them per route based on that service's real latency profile, not a single global guess.

Circuit breakers for fragile dependencies

For routes where the gateway composes responses itself (rather than pure proxying), wrap upstream calls in a breaker such as opossum:

import CircuitBreaker from 'opossum';
import { request } from 'undici';

const fetchInventory = async (sku) => {
  const res = await request(`${process.env.INVENTORY_URL}/sku/${sku}`);
  if (res.statusCode >= 500) throw new Error(`upstream ${res.statusCode}`);
  return res.body.json();
};

const breaker = new CircuitBreaker(fetchInventory, {
  timeout: 2000,
  errorThresholdPercentage: 50,
  resetTimeout: 10_000,
});
breaker.fallback(() => ({ available: null, degraded: true }));

Load shedding when the process itself is struggling

Node.js runs your JavaScript on a single event loop. When that loop is saturated, every request slows down. @fastify/under-pressure rejects new work with a 503 before the process tips over:

import underPressure from '@fastify/under-pressure';

app.register(underPressure, {
  maxEventLoopDelay: 1000,
  maxHeapUsedBytes: 0.9 * 1024 ** 3,
  retryAfter: 5,
  exposeStatusRoute: '/health/live',
});

The Google SRE book's chapter on Handling Overload makes the case well: shedding some load cleanly beats degrading all of it.

Step 5: Observability You Can Use During an Incident

At minimum, every request through the gateway should produce:

• A structured log line with request ID, route, status, latency, and tenant (Fastify's built-in Pino logger handles this).
• A trace span propagated to upstreams via W3C Trace Context headers. The OpenTelemetry Node.js SDK auto-instruments Fastify and undici.
• RED metrics (rate, errors, duration) per route and per upstream.

Load the OpenTelemetry SDK before anything else, typically with node --import ./otel.js gateway.js, so instrumentation patches modules before they are required.

Step 6: Scaling Horizontally

Because the gateway is stateless, scaling is mostly operational:

1. Run one process per container and let the orchestrator (for example Kubernetes) manage replicas. This is generally simpler to reason about than the Node.js cluster module inside a container.
2. Autoscale on meaningful signals. CPU is a reasonable start; event loop delay and p99 latency are better indicators of user impact.
3. Shut down gracefully. On SIGTERM, stop accepting connections and drain in-flight requests:

const shutdown = async () => {
  app.log.info('draining');
  await app.close(); // closes server, runs onClose hooks
  process.exit(0);
};
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

1. Separate liveness from readiness. Liveness says "restart me." Readiness says "stop sending me traffic." Conflating them causes restart storms.

Common Pitfalls We See

• Putting business logic in the gateway. It becomes a distributed monolith. Keep it to routing, policy, and light composition.
• CPU-heavy work on the event loop. Large payload transformations or synchronous crypto block every other request. Move them to worker threads or downstream services.
• No per-route configuration. One timeout and one rate limit for every route rarely fits real traffic.
• Unversioned route config. Treat routing tables as code, reviewed and deployed through CI.

Where to Go From Here

A gateway is a small amount of code with a large blast radius, so it deserves the same rigor as any tier-one service: load testing, chaos testing of upstream failures, and clear ownership. If you want a second set of eyes on your design, see how we approach platform and API engineering across our capabilities, or explore how gateway requirements differ across regulated industries such as finance and healthcare.

FAQ

Is Node.js fast enough for an enterprise API gateway?

For I/O-bound work like proxying, auth checks, and rate limiting, Node.js is well suited because its non-blocking model handles many concurrent connections efficiently. It becomes a poor fit when the gateway does heavy CPU work per request. Keep that work off the event loop and benchmark with your own traffic shape.

Should I use Express or Fastify for a Node.js API gateway?

Both work. We lean toward Fastify for gateways because of its lower overhead, plugin encapsulation, and first-party undici-based proxy. If your team has deep Express expertise and modest throughput needs, Express with a proxy middleware is a valid choice.

How do I handle rate limiting across multiple gateway instances?

Store counters in a shared backend such as Redis rather than in process memory. Key limits on verified identity (user or tenant ID) where possible, fall back to IP for anonymous traffic, and decide explicitly whether to fail open or closed if Redis is unavailable.

Do I still need a service mesh if I have an API gateway?

They solve different problems. A gateway manages north-south traffic entering your system. A service mesh manages east-west traffic between services, often adding mTLS and retries. Many enterprise platforms run both, but a mesh is not a prerequisite for a well-built gateway.

How should I version APIs through the gateway?

Route on a path prefix (such as /v1/orders) or a header, and map each version to the appropriate upstream. Keep version routing in configuration, not code branches, so you can retire old versions without redeploying gateway logic.