Skip to content
Sand Software Solutions

Retrying fetch without making an outage worse

Published · 2 min read

"Just retry it" is how a 30-second upstream blip becomes a 10-minute outage: every client retries at the same moment, the recovering service gets flattened again, and everyone retries again. Good retrying has four parts: waits that grow, random spread (jitter), a way to give up, and respect for what the server tells you.

npm install @sandsoftwaresolutions/retry

A complete example

import { createServer } from "node:http";
import { retry, StopRetrying, retryAfterMs } from "@sandsoftwaresolutions/retry";

// A flaky upstream: 503, then 429 with Retry-After: 1, then OK. /gone is a hard 404.
let hits = 0;
const upstream = createServer((req, res) => {
  if (req.url === "/gone") { res.statusCode = 404; return res.end(); }
  hits++;
  if (hits === 1) { res.statusCode = 503; return res.end(); }
  if (hits === 2) { res.statusCode = 429; res.setHeader("Retry-After", "1"); return res.end(); }
  res.setHeader("content-type", "application/json");
  res.end(JSON.stringify({ ok: true, hits }));
}).listen(0);
await new Promise((r) => upstream.once("listening", r));
const base = `http://127.0.0.1:${upstream.address().port}`;

async function getJson(path) {
  return retry(async (attempt, signal) => {
    const res = await fetch(base + path, { signal });
    if (res.status === 404) throw new StopRetrying(new Error("not found"));
    if (!res.ok) {
      const err = new Error(`HTTP ${res.status}`);
      err.status = res.status;
      err.retryAfterMs = retryAfterMs(res.headers.get("retry-after"));
      throw err;
    }
    return res.json();
  }, {
    retries: 4, minMs: 100, maxMs: 2000, timeoutPerAttemptMs: 3000,
    onRetry: ({ attempt, delayMs, error }) => console.log(`attempt ${attempt} failed (${error.message}), waiting ${Math.round(delayMs)} ms`),
  });
}

const t = Date.now();
console.log(await getJson("/data"), `after ${Date.now() - t} ms`);
try { await getJson("/gone"); } catch (e) { console.log("gone ->", e.message); }
upstream.close();

Real output (the upstream answers 503, then 429 with Retry-After: 1, then 200):

attempt 1 failed (HTTP 503), waiting 81 ms
attempt 2 failed (HTTP 429), waiting 1000 ms
{ ok: true, hits: 3 } after 1098 ms
gone -> not found

What happened:

  • Growing, jittered waits. The first wait was 81 ms, somewhere between 0 and the 100 ms base. The default is "full jitter": a random value between 0 and the current ceiling, which spreads a crowd of clients out better than adding a little noise.
  • The server was obeyed. The 429 carried Retry-After: 1. We copied it onto the error as retryAfterMs, and the library waited at least that long (1000 ms) even though its own schedule said less. retryAfterMs(header) understands both Retry-After: 120 and an HTTP date, and returns undefined (not 0) for anything else.
  • A 404 is not retried. Throwing StopRetrying stops at once and rethrows your wrapped error. Retrying a "not found" or a validation error only wastes time.
  • Every attempt has a deadline and a cancel signal. timeoutPerAttemptMs aborts a hung attempt, and the signal you get is passed to fetch, so the socket is actually closed.

Decide what is safe to retry

The library can't know this; you have to. A rule that works for most HTTP APIs:

Situation Retry?
Network error, timeout Yes, for idempotent requests (GET, PUT, DELETE)
429, 503 Yes, honouring Retry-After
500, 502, 504 Yes, a few times
Other 4xx No: the request is wrong, it will be wrong again
POST that might have succeeded before the connection dropped Only with an idempotency key

That last row matters most. If a payment POST times out, you don't know whether it was processed. Send an Idempotency-Key header (Stripe and Resend, among others, support one) and generate it once, outside the retry function so every attempt reuses it:

const key = crypto.randomUUID();
await retry(() => fetch(url, { method: "POST", headers: { "Idempotency-Key": key }, body }), { retries: 3 });

Failure is data

After the last attempt, retry throws a RetryError with attempts, all errors, and cause (the last one), so your log shows the whole story rather than just the final error. With retries: 0 the original error is thrown unchanged. Obviously-wrong options (NaN, negative waits, a timeoutPerAttemptMs of 0) throw a RangeError immediately instead of silently becoming a 1 ms wait.

Cap the total, too

Backoff makes each wait longer, but it doesn't cap the total time. If a user is waiting on this call, pass an AbortSignal.timeout(10_000) as signal: the wait ends immediately and the promise rejects with the signal's reason. A request that retries for three minutes in the background of a page load is a bug, however polite its backoff.

← All posts

A question or an idea?

Send us an email. You talk directly to the people who build the software and get an answer within 2 working days.