Webhooks Explained: Signed Payloads and Reliable Retries

You ship a payment integration, the provider fires a webhook, and your endpoint returns 200 — but three hours later support reports missing orders. Or worse: the same event arrives four times and the customer is charged twice. Webhooks look simple (an HTTP POST to your URL) but the hard parts are trust and delivery guarantees.

This guide covers both: how to verify that a payload actually came from the sender using signed payloads, and how to build a retry model that survives downtime, timeouts, and duplicate deliveries.

What a webhook actually is

A webhook is a reverse API call. Instead of your server polling GET /events every minute, the provider pushes an HTTP request to a URL you registered when something happens. The payload is usually JSON, the method is almost always POST, and the response you return is a signal: 2xx means "received", anything else means "try again later".

That last sentence is the whole contract. Everything else — signatures, retries, ordering — exists to make that contract safe over an unreliable network.

Why unsigned webhooks are dangerous

Your webhook endpoint is a public URL. Anyone who discovers it can POST a fake payload claiming an order was paid. If you trust the body without verification, you have built an open door into your billing logic.

IP allowlists help a little but break behind CDNs, serverless egress, and provider IP changes. The robust answer is a shared secret plus an HMAC signature over the raw request body.

How signed payloads work

The provider and you share a secret key. On each delivery the provider computes an HMAC (usually SHA-256) over the raw body — sometimes over a timestamp plus the body — and sends the hex digest in a header such as X-Signature. You recompute the same digest with your copy of the secret and compare.

Two details matter more than people expect:

• Sign the raw bytes. If your framework parses JSON and re-serializes it, key order and whitespace change and the digest will not match. Read the raw body first.
• Compare in constant time. A naive == leaks timing information. Use a constant-time comparison function from your language's crypto library.

A minimal verification example (Node.js)

const crypto = require('crypto');

function verifySignature(rawBody, headerSig, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)          // raw Buffer or string, not parsed JSON
    .digest('hex');

  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(headerSig || '', 'utf8');
  if (a.length !== b.length) return false;
  return crypto.timingSafeEqual(a, b);
}

The same pattern works in Python with hmac.compare_digest, in Go with hmac.Equal, and in Java with MessageDigest.isEqual. The principle is identical everywhere: HMAC over raw bytes, constant-time compare.

Replay protection with timestamps

A valid signature proves the payload came from someone holding the secret — but a captured request can be replayed forever. Providers that care about this include a timestamp in the signed string, for example t=1712345678,v1=<hex>. On your side:

1. Parse the timestamp from the header.
2. Reject the request if it is older than a tolerance window (often 5 minutes).
3. Recompute the HMAC over timestamp + "." + rawBody.

This bounds the damage of a leaked request to a few minutes instead of forever.

Designing retries that actually deliver

Networks fail, your service restarts, deploys roll pods. A serious provider retries failed deliveries on a schedule. A serious receiver must assume every event can arrive more than once.

Sender-side retry rules

• Treat any non-2xx response, connection error, or timeout as a failure.
• Retry with exponential backoff and jitter — for example 1s, 5s, 30s, 5m, 30m, 2h — so a provider outage does not produce a thundering herd when you recover.
• Cap the number of attempts (commonly 5–10) and record the final failure so the user or operator can see it.
• Set a short client timeout (5–10 seconds). A webhook handler should acknowledge fast and do the work asynchronously.

Receiver-side rules

1. Verify first, parse second. Reject bad signatures with 401 before touching your database.
2. Persist the event, then return 200. Write the raw payload and event ID to a queue or table, respond immediately, and process in a worker. If processing fails, retry internally.
3. Make handlers idempotent. Use the provider's event ID as a unique key. If it already exists, return 200 without re-running side effects.
4. Return errors honestly. If your database is down, return 500 so the provider retries — do not return 200 and silently drop the event.

Idempotency in practice

Idempotency is not a buzzword here — it is the only thing standing between a retry and a double charge. The pattern:

INSERT INTO webhook_events (event_id, payload, received_at)
VALUES ($1, $2, now())
ON CONFLICT (event_id) DO NOTHING;

If the insert affects zero rows, you have already seen this event; acknowledge and move on. This works even under concurrent deliveries because the unique constraint serializes them at the database.

For side effects that must happen exactly once (sending an email, charging a card), pass your own idempotency key downstream — most payment and email APIs accept one.

Ordering, debugging, and observability

Do not assume events arrive in order. Retries can reorder deliveries, and parallel workers will too. If order matters, include a sequence number or timestamp in the payload and reconcile state rather than applying deltas blindly.

Log every delivery attempt with the event ID, signature result, response code, and processing time. When a customer says "my order never arrived", you want to answer in seconds: was the webhook received, was the signature valid, did processing succeed? A structured log line per event is enough.

For endpoints that receive a lot of traffic, watch for retry storms after a deploy. A quick way to spot them is to look at the request patterns in your access logs — a spike of identical paths from one IP with 5xx responses is a classic retry loop. Tools like the Nginx log analyzer make that pattern obvious without writing a query by hand.

Common mistakes to avoid

• Parsing JSON before verifying the signature.
• Using the same secret across environments, or never rotating it.
• Doing heavy work (PDF generation, third-party calls) inside the request handler.
• Returning 200 on processing errors, so the provider never retries.
• Trusting the payload's amount or status without re-fetching the resource from the provider's API when it matters.

FAQ

How do I verify a webhook signature if the body is JSON?

Read the raw request body as bytes before any JSON parsing, compute the HMAC over those bytes, and compare with the header using a constant-time function. Re-serialized JSON will produce a different digest.

How many times should a webhook be retried?

There is no universal number, but 5–10 attempts over several hours with exponential backoff and jitter is common. The important part is that receivers are idempotent, so retries are safe even when they are frequent.

What status code should my webhook endpoint return?

Return 2xx only after you have durably stored the event. Return 401 for invalid signatures, 400 for malformed payloads, and 5xx or 429 when you want the sender to retry.

If you are testing webhook payloads or logging the JSON you receive, the JSON formatter is a quick way to pretty-print and validate a captured event before you build parsing logic around it.