Webhooks Explained: Signed Payloads and Reliable Retries

Your integration works in staging: the payment provider fires an event, your endpoint receives it, the order flips to paid. Then in production, a customer pays twice, a delivery silently disappears during a deploy, and someone replays a captured request to mark an order as refunded. All three failures share the same root cause: the webhook was treated as a simple HTTP POST instead of a security-critical, at-least-once delivery channel.

This guide covers the two halves of a correct webhook implementation — verifying that a payload is authentic, and making delivery survive retries — with code you can adapt to any stack.

What a webhook really is

A webhook is a server-to-server HTTP request that one system sends to another when an event happens. It inverts the usual polling model: instead of your app asking "anything new?" every minute, the producer pushes the event to a URL you registered.

Two properties follow from that design, and both matter:

• Your endpoint is public. Anyone on the internet can POST to it, so every request must be authenticated.
• Delivery is at-least-once. Networks fail, servers restart, and timeouts are ambiguous — the sender cannot know whether your handler ran. So it retries, and you may see the same event more than once.

Everything below is a consequence of those two facts.

Signing payloads with HMAC

The standard approach is a shared secret plus an HMAC over the raw request body. The sender computes a signature and puts it in a header; you recompute it and compare. If they match, the payload came from someone who knows the secret and was not modified in transit.

Three details separate a working implementation from a broken one:

1. Sign the raw bytes. Parse JSON only after verification. Re-serializing the body changes whitespace and key order, and the signature will never match.
2. Use a constant-time comparison. A naive == leaks timing information. Use the language's built-in safe compare.
3. Include a timestamp in the signed string. Signing timestamp + "." + body lets you reject old requests, which blocks replay attacks.

Verifying a signature in Node.js

const crypto = require('crypto');

function verifyWebhook(rawBody, headers, secret, toleranceSeconds = 300) {
  const signature = headers['x-signature'];   // e.g. "t=1712345678,v1=abc123..."
  const timestamp = headers['x-timestamp'];
  if (!signature || !timestamp) return false;

  // 1. Reject stale requests (replay protection)
  const age = Math.floor(Date.now() / 1000) - Number(timestamp);
  if (!Number.isFinite(age) || age > toleranceSeconds) return false;

  // 2. Recompute over the exact bytes received
  const signedPayload = `${timestamp}.${rawBody}`;
  const expected = crypto
    .createHmac('sha256', secret)
    .update(signedPayload, 'utf8')
    .digest('hex');

  // 3. Constant-time compare
  const a = Buffer.from(expected, 'hex');
  const b = Buffer.from(signature, 'hex');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

In Express, this requires express.raw({ type: 'application/json' }) on the webhook route so the body arrives as a Buffer. If a global express.json() middleware runs first, the raw body is gone and verification will fail — a very common bug.

Verifying in Python

import hmac, hashlib, time

def verify(raw_body: bytes, timestamp: str, signature: str, secret: bytes, tolerance: int = 300) -> bool:
    if abs(time.time() - int(timestamp)) > tolerance:
        return False
    signed = timestamp.encode() + b"." + raw_body
    expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

Always return 401 on a failed check and log the request ID — but never log the secret or the full signature.

Designing retries that actually work

Retries are only useful if your endpoint behaves predictably. The contract is simple: return 2xx quickly once you have durably accepted the event, and let everything else be retried.

Two rules keep this healthy:

• Acknowledge fast. Verify, persist the event to a queue or table, respond 200. Do the real work asynchronously. A handler that calls three APIs before responding will time out and trigger duplicate deliveries.
• Use exponential backoff with jitter. If a sender retries immediately, a brief outage becomes a thundering herd. A schedule like 10s, 30s, 2m, 10m, 1h with random jitter spreads the load.

Make handlers idempotent

Because delivery is at-least-once, the same event ID can arrive twice — once after a timeout where your handler actually succeeded. Store processed event IDs and skip duplicates:

INSERT INTO processed_events (event_id, received_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- If 0 rows inserted, this is a duplicate: return 200 and stop.

Where possible, make the business operation itself idempotent — set order status to paid rather than incrementing a counter, and use the event ID as an idempotency key when calling downstream services.

A checklist before going live

1. Verify the HMAC over raw bytes with a constant-time compare.
2. Reject timestamps older than a few minutes to stop replays.
3. Persist the event, then return 2xx within a second or two.
4. Deduplicate on the event ID before doing any work.
5. Log the event ID, type, and result — never the secret.
6. Expose a way to replay events manually from your dashboard or admin tool.
7. Test with deliberately broken signatures, duplicate deliveries, and slow handlers.

When you are debugging a failing integration, the raw request body is the thing you will stare at most. Pasting it into a JSON formatter makes nested event objects readable without changing the bytes you already captured.

FAQ

Why does my signature check fail even though the payload looks identical?

Almost always because the body was parsed and re-serialized before hashing. JSON key order, whitespace, and unicode escaping change during parse/stringify, so the bytes no longer match what the sender signed. Capture the raw body and hash that.

Should I return 200 before or after processing the event?

After durably storing it, before doing the heavy work. If you process first, slow downstream calls cause timeouts, the sender retries, and you get duplicates. Queue the event, respond immediately, and process asynchronously.

Is HTTPS enough to secure a webhook endpoint?

No. TLS protects data in transit but does not prove who sent the request — anyone can POST to a public URL. HMAC signing with a shared secret authenticates the sender, and a signed timestamp prevents replay of captured requests.

Bottom line

Treat webhooks like any untrusted input: verify the signature over raw bytes, check the timestamp, deduplicate by event ID, and acknowledge only after the event is safely stored. Do that, and retries stop being a source of duplicate charges and become what they were meant to be — a safety net.