Webhooks Explained: Signed Payloads and Reliable Retries

Backend2026-11-11TryQuickToolBox

You ship a payment integration and everything works in staging. Then in production a customer pays, your endpoint times out for two seconds, and the order never activates. Meanwhile the payment provider retries the same event three times, and your app creates three duplicate orders for the one that did succeed. Webhooks look simple — just an HTTP POST — but the hard parts are trust and delivery guarantees.

This article covers the two things that break webhook integrations most often: verifying that a request really came from the sender, and handling retries without duplicating side effects.

What a webhook actually is

A webhook is a reverse API call. Instead of your server polling the provider every minute asking "anything new?", the provider pushes an HTTP request to a URL you registered when an event happens. The payload is usually JSON, and the event type is either in the body or in a header.

POST /webhooks/payments HTTP/1.1
Host: api.example.com
Content-Type: application/json
X-Event-Id: evt_8f2c1a
X-Timestamp: 1735689600
X-Signature: t=1735689600,v1=5d41402abc4b2a76b9719d911017c592

{
  "id": "evt_8f2c1a",
  "type": "payment.succeeded",
  "created": 1735689600,
  "data": { "order_id": "ord_123", "amount": 4999, "currency": "USD" }
}

Two design decisions matter on the sender side and you should check both when evaluating any provider: does it sign the payload, and does it retry on failure?

Why unsigned webhooks are dangerous

Your webhook URL is public. Anyone who learns it can POST a fake payment.succeeded event and mark an unpaid order as paid. An IP allowlist helps a little, but providers change IPs, run behind CDNs, and you may not get a stable list.

The robust answer is an HMAC signature over the raw request body using a shared secret that only you and the provider know. The provider computes the signature and sends it in a header; you recompute it and compare.

Verifying a signature correctly

Three details are easy to get wrong:

// Node.js / Express example
const crypto = require('crypto');

function verifySignature(rawBody, header, secret, toleranceSeconds = 300) {
  const parts = Object.fromEntries(
    header.split(',').map(p => p.split('='))
  );
  const timestamp = Number(parts.t);
  if (!timestamp) return false;

  // Reject stale requests (replay protection)
  const now = Math.floor(Date.now() / 1000);
  if (Math.abs(now - timestamp) > toleranceSeconds) return false;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(`${parts.t}.${rawBody}`)
    .digest('hex');

  const a = Buffer.from(expected);
  const b = Buffer.from(parts.v1 || '');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

app.post('/webhooks/payments', express.raw({ type: '*/*' }), (req, res) => {
  if (!verifySignature(req.body.toString(), req.get('X-Signature'), SECRET)) {
    return res.status(401).send('invalid signature');
  }
  const event = JSON.parse(req.body.toString());
  // ... handle event
  res.status(200).send('ok');
});

Note the express.raw() middleware. If you mount express.json() before this route, the body is already parsed and the signature check silently fails — a very common bug.

Designing retries that don't lose events

Webhook delivery is at-least-once, not exactly-once. Networks fail, your deploy restarts a container, and the provider has no idea whether your 200 response was real. So the provider retries with exponential backoff, often for hours or days.

Respond fast, process later

Your endpoint should do the minimum: verify, persist the raw event, return 2xx. Heavy work — sending emails, calling other APIs, generating invoices — should happen in a background job. If processing takes longer than the provider's timeout, the request is treated as failed and retried even though you did the work.

  1. Verify the signature and timestamp.
  2. Insert the event into a table with a unique constraint on the event ID.
  3. Return 200 immediately.
  4. Process the stored event asynchronously, marking it done.

Idempotency: the real fix for duplicates

Because retries happen, your handler must be safe to run twice. The event ID is the natural idempotency key.

INSERT INTO webhook_events (event_id, type, payload, received_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (event_id) DO NOTHING;

If the insert affects zero rows, you have seen this event before — acknowledge it and move on. For business logic, apply the same idea: use the order ID as a unique key when activating a payment, so a second event cannot create a second activation.

Choosing status codes intentionally

StatusProvider behaviorWhen to use
200 / 204Mark deliveredEvent accepted (even if already processed)
400Usually no retryMalformed payload you can never handle
401Usually no retrySignature mismatch — check your secret
429Retry with backoffYou are rate limiting
500 / 503Retry with backoffTemporary failure, DB down, deploy in progress

Returning 200 for an event you could not store is the worst option: the provider stops retrying and the event is gone forever. Return a 5xx and let the retry do its job.

Practical checklist for production

Testing webhooks locally

You cannot easily receive a public POST on localhost. Options: a tunneling tool that exposes your dev port, the provider's CLI if it has one, or a small script that sends a signed payload to your handler so you can exercise the verification path.

The most valuable test is the duplicate: send the same event twice and confirm your system ends up in the same state. Then send an event with an old timestamp and confirm it is rejected. Then tamper with one byte of the body and confirm the signature check fails.

Debugging payloads

Webhook bodies are JSON, and malformed or deeply nested payloads are painful to read in logs. A formatter helps you inspect the structure quickly before writing parsing code. The JSON formatter runs in the browser, so you can paste a captured payload without sending customer data to a server.

FAQ

What is the difference between a webhook and an API?

An API is pull-based: your code calls the provider when it needs data. A webhook is push-based: the provider calls your endpoint when an event happens. Most integrations use both — webhooks for event notifications and the API to fetch full details or perform actions.

Why does my HMAC signature never match?

The usual cause is signing a re-serialized body instead of the raw bytes. Whitespace, key order, and unicode escaping all change the hash. Read the raw request body before any JSON middleware, and confirm the secret and hash algorithm match the provider's documentation.

Are webhook retries guaranteed?

No. Providers typically retry for a limited window with exponential backoff, then give up. Treat webhooks as a fast notification channel rather than a source of truth: reconcile periodically against the provider's API so missed events are eventually corrected.

If you are building or debugging a webhook receiver today, start by locking down signature verification and idempotent storage — those two changes eliminate the majority of real-world webhook incidents.