Retrying fetch without making an outage worse
Published · 2 min read
"Just retry it" is how a 30-second upstream blip becomes a 10-minute outage: every client retries at the same moment, the recovering service gets flattened again, and everyone retries again. Good retrying has four parts: waits that grow, random spread (jitter), a way to give up, and respect for what the server tells you.
npm install @sandsoftwaresolutions/retry
A complete example
import { createServer } from "node:http";
import { retry, StopRetrying, retryAfterMs } from "@sandsoftwaresolutions/retry";
// A flaky upstream: 503, then 429 with Retry-After: 1, then OK. /gone is a hard 404.
let hits = 0;
const upstream = createServer((req, res) => {
if (req.url === "/gone") { res.statusCode = 404; return res.end(); }
hits++;
if (hits === 1) { res.statusCode = 503; return res.end(); }
if (hits === 2) { res.statusCode = 429; res.setHeader("Retry-After", "1"); return res.end(); }
res.setHeader("content-type", "application/json");
res.end(JSON.stringify({ ok: true, hits }));
}).listen(0);
await new Promise((r) => upstream.once("listening", r));
const base = `http://127.0.0.1:${upstream.address().port}`;
async function getJson(path) {
return retry(async (attempt, signal) => {
const res = await fetch(base + path, { signal });
if (res.status === 404) throw new StopRetrying(new Error("not found"));
if (!res.ok) {
const err = new Error(`HTTP ${res.status}`);
err.status = res.status;
err.retryAfterMs = retryAfterMs(res.headers.get("retry-after"));
throw err;
}
return res.json();
}, {
retries: 4, minMs: 100, maxMs: 2000, timeoutPerAttemptMs: 3000,
onRetry: ({ attempt, delayMs, error }) => console.log(`attempt ${attempt} failed (${error.message}), waiting ${Math.round(delayMs)} ms`),
});
}
const t = Date.now();
console.log(await getJson("/data"), `after ${Date.now() - t} ms`);
try { await getJson("/gone"); } catch (e) { console.log("gone ->", e.message); }
upstream.close();
Real output (the upstream answers 503, then 429 with Retry-After: 1, then 200):
attempt 1 failed (HTTP 503), waiting 81 ms
attempt 2 failed (HTTP 429), waiting 1000 ms
{ ok: true, hits: 3 } after 1098 ms
gone -> not found
What happened:
- Growing, jittered waits. The first wait was 81 ms, somewhere between 0 and the 100 ms base. The default is "full jitter": a random value between 0 and the current ceiling, which spreads a crowd of clients out better than adding a little noise.
- The server was obeyed. The 429 carried
Retry-After: 1. We copied it onto the error asretryAfterMs, and the library waited at least that long (1000 ms) even though its own schedule said less.retryAfterMs(header)understands bothRetry-After: 120and an HTTP date, and returnsundefined(not 0) for anything else. - A 404 is not retried. Throwing
StopRetryingstops at once and rethrows your wrapped error. Retrying a "not found" or a validation error only wastes time. - Every attempt has a deadline and a cancel signal.
timeoutPerAttemptMsaborts a hung attempt, and thesignalyou get is passed tofetch, so the socket is actually closed.
Decide what is safe to retry
The library can't know this; you have to. A rule that works for most HTTP APIs:
| Situation | Retry? |
|---|---|
| Network error, timeout | Yes, for idempotent requests (GET, PUT, DELETE) |
| 429, 503 | Yes, honouring Retry-After |
| 500, 502, 504 | Yes, a few times |
| Other 4xx | No: the request is wrong, it will be wrong again |
| POST that might have succeeded before the connection dropped | Only with an idempotency key |
That last row matters most. If a payment POST times out, you don't know whether it was processed. Send an Idempotency-Key header (Stripe and Resend, among others, support one) and generate it once, outside the retry function so every attempt reuses it:
const key = crypto.randomUUID();
await retry(() => fetch(url, { method: "POST", headers: { "Idempotency-Key": key }, body }), { retries: 3 });
Failure is data
After the last attempt, retry throws a RetryError with attempts, all errors, and cause (the last one), so your log shows the whole story rather than just the final error. With retries: 0 the original error is thrown unchanged. Obviously-wrong options (NaN, negative waits, a timeoutPerAttemptMs of 0) throw a RangeError immediately instead of silently becoming a 1 ms wait.
Cap the total, too
Backoff makes each wait longer, but it doesn't cap the total time. If a user is waiting on this call, pass an AbortSignal.timeout(10_000) as signal: the wait ends immediately and the promise rejects with the signal's reason. A request that retries for three minutes in the background of a page load is a bug, however polite its backoff.