A Self-Hosted Webhook Gateway With Retries And A Dead-Letter Queue

The Problem

Stripe, your CRM, and three SaaS tools all post webhooks directly into automation scenarios. When one of those scenarios is down, the event is gone - the sender retried twice into a 500 and gave up. Nobody finds out until the numbers disagree at month end.

Prerequisites & Tool Stack

Everything this blueprint calls. Check you have them before you start - the usual reason a build stalls halfway is a key that takes a day to get approved.

A VPS with a public IP

Webhook senders need to reach it. The smallest shared-vCPU instance handles a few hundred events a second.

Free tier: From about $4.50 a month

Coolify

Gives the box a deploy pipeline, automatic TLS, and log access without building a platform first. Free and self-hosted.

Free tier: Free, self-hosted

Redis

Backs the queue. The compose file runs it locally; Upstash is the managed option if you would rather not.

Free tier: 10,000 commands a day

A domain and DNS access

One A record. Coolify issues the certificate once it resolves.

Some links above are partner links and we earn a commission if you sign up through them. It costs you nothing extra, and it does not decide what goes in a blueprint - the self-hosted option is recommended wherever it is genuinely the better call.

What It Costs To Run

Monthly, at the volumes this blueprint was tested against. Worth comparing against what the manual version of this work costs you in hours.

Component	Cost	At what volume
Hetzner CX22	$4.59/mo	Gateway, worker, and Redis together
Coolify	$0	Self-hosted, no licence
Managed Redis	$0 - $10/mo	Only if you skip the local container

The files

Get the deploy bundle

The repository runs locally in a minute. Getting it onto a box with TLS, log retention, and a replay path is the part worth having written down.

.env template with every secret the gateway reads, annotated

Coolify deploy notes, including the health check that avoids a restart loop

The replay CLI for draining the dead-letter queue after an outage

Signature verification for Stripe, GitHub, and HubSpot as working code

Technical Breakdown

The decisions that matter, in the order you will meet them. Everything here is a thing that broke in testing before it was a rule.

01.Verify the signature before you read the body

Compute the HMAC over the raw bytes, compare in constant time, and reject anything that fails. Parsing JSON first means you are running your parser on unauthenticated input, which is exactly the wrong order.

javascript

import { createHmac, timingSafeEqual } from 'node:crypto';

export function verify(rawBody, header, secret) {
  const expected = createHmac('sha256', secret).update(rawBody).digest();
  const received = Buffer.from(header ?? '', 'hex');

  // Length check first: timingSafeEqual throws on a mismatch rather than
  // returning false, and an exception here is a 500 instead of a 401.
  return (
    received.length === expected.length && timingSafeEqual(received, expected)
  );
}

02.Acknowledge in milliseconds, work later

The handler verifies, enqueues, and returns 200. Nothing else. Every sender has a timeout measured in seconds and treats a slow response as a failure, which turns your slow subscriber into duplicate deliveries.

03.One job per subscriber

Fanning out inside a single job means one failing destination retries all of them. Separate jobs give each subscriber its own retry state, so a broken Slack webhook cannot cause duplicate CRM writes.

04.Make delivery idempotent

Send an idempotency key derived from the event id with every downstream call and have subscribers honour it. At-least-once delivery is the only guarantee a queue can offer, so the receiver has to be safe to repeat.

javascript

await queue.add(
  'deliver',
  { subscriber, event },
  {
    jobId: `${subscriber.id}:${event.id}`,   // dedupes a replayed event
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: { age: 86_400 },
  }
);

05.Park failures where someone will look

After five attempts the job moves to the dead-letter queue with its full payload and error history, and an alert fires. A dead-letter queue nobody is told about is just a slower way of losing the event.

When It Goes Wrong

The difference between a demo and something you can leave running is entirely in this table. Every row is a failure the blueprint handles explicitly rather than hoping about.

Failure	What happens
Invalid or missing signature	Rejected 401 before parsing. Counted per sender, so a rotated secret is visible within minutes.
Redis unavailable at receive time	Gateway returns 503 so the sender retries. Accepting an event you cannot persist is worse than refusing it.
Subscriber returns 5xx	Five attempts with jittered exponential backoff, then dead-lettered with the response body attached.
Duplicate event id	BullMQ jobId collision drops the second copy silently, which is the intended behaviour.

Payload Example

A dead-lettered job, as stored.

json

{
  "job_id": "crm-sync:evt_1QxR2mK8",
  "subscriber": "crm-sync",
  "event": { "id": "evt_1QxR2mK8", "type": "invoice.paid" },
  "attempts": 5,
  "first_failed_at": "2026-09-11T02:14:55Z",
  "last_error": "POST https://crm.internal/hooks -> 502 Bad Gateway",
  "replayable": true
}

Not sure this is the right platform?

Long-running services that receive, verify, and fan out events to everything downstream.

Browse All Blueprints

Related blueprints

AI Support Triage That Routes Mail Before A Human Reads ItMake

Self-Hosted Lead Enrichment That Scores A Signup In Four Secondsn8n

A Price Scraper That Survives The Site Changing Its MarkupPython