Building a Circuit Breaker Pattern in Node.js

When your Redis connection fails, what happens to your application? If you're not careful, every single request starts waiting for a connection timeout. Your response times jump from 200ms to 30 seconds. Users see loading spinners. Your monitoring lights up. And you're scrambling at 2 AM wondering why the whole system is down when it's just one dependency.

This is the exact problem the circuit breaker pattern solves. And it's one of the first reliability patterns I implemented in NeedThisDone.com.

The Problem: Cascading Failures

Here's what happens without a circuit breaker:

1. Redis goes down (network blip, memory pressure, whatever)
2. Every request tries to connect to Redis
3. Every connection attempt waits for the timeout (usually 5-30 seconds)
4. Your request queue backs up
5. Your entire application becomes unresponsive
6. Users leave. Revenue drops. You get paged.

The failure of one dependency takes down everything.

The Solution: Three States

A circuit breaker has three states, just like an electrical circuit breaker:

• Closed (normal): Requests flow through normally. The breaker monitors for failures.
• Open (tripped): After too many failures, the breaker "trips." All requests fail immediately — no waiting for timeouts.
• Half-Open (testing): After a cooldown period, the breaker lets one request through to test if the dependency recovered.

The Implementation

Here's the actual pattern from my production code. The state machine tracks connection failures:

// State tracking
let connectionAttempts = 0;
let lastConnectionError: Date | null = null;

const MAX_CONNECTION_FAILURES = 3;
const FAILURE_WINDOW_MS = 60_000; // 1 minute

The key function checks whether we should even try connecting:

function isCircuitBreakerOpen(): boolean {
  // No recent errors — circuit is closed, proceed normally
  if (!lastConnectionError) return false;

  const timeSinceLastError =
    Date.now() - lastConnectionError.getTime();

  // 3+ failures within the last 60 seconds — circuit is OPEN
  if (
    connectionAttempts >= MAX_CONNECTION_FAILURES &&
    timeSinceLastError < FAILURE_WINDOW_MS
  ) {
    return true;
  }

  // Outside the failure window — auto-reset
  return false;
}

Before every Redis operation, we check the breaker:

async function ensureConnected(): Promise<void> {
  if (isCircuitBreakerOpen()) {
    throw new Error(
      `Redis circuit breaker open: ${connectionAttempts} failures in last 60s`
    );
  }

  // Reset counters if we're outside the failure window
  if (
    lastConnectionError &&
    Date.now() - lastConnectionError.getTime() >= FAILURE_WINDOW_MS
  ) {
    connectionAttempts = 0;
    lastConnectionError = null;
  }

  if (!redis.isOpen) {
    try {
      await redis.connect();
      // Success — reset everything
      connectionAttempts = 0;
      lastConnectionError = null;
    } catch (error) {
      connectionAttempts++;
      lastConnectionError = new Date();
      throw error;
    }
  }
}

Why This Works

The magic is in the timing:

• First failure: We retry immediately. Network blips happen.
• Second failure: We retry again. Maybe the server is restarting.
• Third failure: The circuit opens. Something is actually wrong.
• For the next 60 seconds: Every request fails instantly instead of waiting for a timeout.
• After 60 seconds: The window resets. We try one more connection.

This means your application goes from "30-second hangs on every request" to "instant error responses while Redis recovers." Users see an error message instead of an infinite loading spinner. Your other features (the ones that don't need Redis) keep working.

Graceful Degradation

The circuit breaker is only half the story. The other half is what your application does when the breaker is open.

For my caching layer, the answer is simple: skip the cache and hit the database directly.

async function getCachedData<T>(key: string, fetcher: () => Promise<T>): Promise<T> {
  try {
    const cached = await redis.get(key);
    if (cached) return JSON.parse(cached);
  } catch {
    // Circuit breaker tripped or Redis unavailable
    // Fall through to the database
  }

  // Always have a fallback
  return fetcher();
}

The application is slower without the cache, but it's still working. That's the difference between a 5-minute incident and a 5-hour outage.

Lessons Learned

After running this in production:

1. 60 seconds is the sweet spot for the failure window. Short enough to recover quickly, long enough to avoid hammering a struggling server.
2. 3 failures before tripping catches real outages while ignoring single-request blips.
3. Log when the breaker trips. That log line is your early warning system.
4. Always have a fallback. A circuit breaker without graceful degradation just fails faster — which is better than hanging, but not as good as actually working.

The circuit breaker pattern isn't glamorous. It doesn't make your app faster or add features. But when Redis goes down at 2 AM, it's the difference between sleeping through the recovery and getting paged.

Build your reliability patterns before you need them. By the time you're in an incident, it's too late to add them.

If you're building something that needs this level of reliability, let's connect.