Webhooks Explained: Signed Payloads and Retries

You've built an API and now need to notify other services when events happen. You set up a webhook endpoint, but soon discover that anyone can send fake payloads, and when your server is down, events are lost forever. This is a common pitfall: webhooks are simple in concept but require careful design for security and reliability.

In this guide, we'll walk through the essentials of webhooks: how to sign payloads to prevent tampering, how to verify signatures on the receiving end, and how to implement retry logic with exponential backoff to handle failures. We'll also cover idempotency to avoid duplicate processing. By the end, you'll have a robust webhook system that you can trust.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks. When an event occurs in a source system (e.g., a payment is processed), it sends an HTTP POST request to a URL you specify, with a payload containing event details. This allows real-time communication between services without polling.

However, webhooks introduce two challenges:

• Security: How does the receiver know the request genuinely came from the sender?
• Reliability: What if the receiver is temporarily unavailable? How do we ensure the event is eventually delivered?

Securing Webhooks with Signed Payloads

To prevent forged requests, the sender should sign the payload using a shared secret. The receiver can then verify the signature to authenticate the sender. The most common method is HMAC (Hash-based Message Authentication Code) with SHA-256.

How HMAC Signing Works

1. The sender and receiver share a secret key (ideally a long, random string).
2. For each webhook, the sender computes an HMAC of the request body using the secret and includes it in a header (e.g., X-Signature).
3. The receiver computes the same HMAC on the received body and compares it with the header value. If they match, the request is authentic.

It's crucial to sign the raw request body, not a parsed version, to avoid discrepancies due to JSON serialization differences.

Example: Signing a Payload in Node.js

const crypto = require('crypto');

function signPayload(payload, secret) {
  return crypto.createHmac('sha256', secret).update(payload).digest('hex');
}

// On the sender side:
const payload = JSON.stringify({ event: 'payment.success', amount: 100 });
const signature = signPayload(payload, 'your-secret-key');
// Include signature in header: X-Signature: signature

Verifying the Signature on the Receiver

On the receiving end, you must verify the signature before processing the payload. Use a constant-time comparison to prevent timing attacks.

const crypto = require('crypto');

function verifySignature(payload, signature, secret) {
  const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

// In your webhook endpoint:
app.post('/webhook', (req, res) => {
  const signature = req.headers['x-signature'];
  const payload = req.rawBody; // raw body as string
  if (!verifySignature(payload, signature, 'your-secret-key')) {
    return res.status(401).send('Invalid signature');
  }
  // Process the event
  res.status(200).send('OK');
});

Note: Always use HTTPS to prevent man-in-the-middle attacks. Also, consider including a timestamp in the signed payload to prevent replay attacks.

Implementing Retries with Exponential Backoff

Even with a reliable receiver, network issues or temporary outages can cause webhook delivery to fail. A robust sender should retry failed deliveries. However, naive retries can overwhelm the receiver or cause duplicate processing. The solution: exponential backoff with jitter and a maximum retry limit.

Retry Strategy

• Retry on failure: Retry when the receiver returns a non-2xx status code or times out.
• Exponential backoff: Increase the delay between retries exponentially (e.g., 1s, 2s, 4s, 8s...).
• Jitter: Add random variation to the delay to avoid thundering herd problems.
• Maximum attempts: Set a limit (e.g., 5 retries) and then mark the webhook as failed (perhaps alerting an admin).

Example: Retry Logic in Node.js

async function deliverWebhook(url, payload, secret, maxRetries = 5) {
  const signature = signPayload(payload, secret);
  let attempt = 0;
  while (attempt < maxRetries) {
    try {
      const response = await fetch(url, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json', 'X-Signature': signature },
        body: payload,
      });
      if (response.ok) return;
      throw new Error(`HTTP ${response.status}`);
    } catch (error) {
      attempt++;
      if (attempt >= maxRetries) throw error;
      const delay = Math.min(1000 * 2 ** attempt + Math.random() * 1000, 30000);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

This function retries up to 5 times with exponential backoff and jitter, capping the delay at 30 seconds. Adjust parameters based on your needs.

Ensuring Idempotency

Retries can cause duplicate deliveries, so your webhook handler must be idempotent. Include a unique event ID in the payload and have the receiver track processed IDs to avoid double processing.

// On the receiver:
const processedEvents = new Set();

app.post('/webhook', (req, res) => {
  const { id } = req.body;
  if (processedEvents.has(id)) {
    return res.status(200).send('Already processed');
  }
  // Process event
  processedEvents.add(id);
  res.status(200).send('OK');
});

In production, use a persistent store (e.g., Redis or a database) to track processed IDs across restarts.

Comparison: Signed vs. Unsigned Webhooks

FAQ

What is a webhook signature?

A webhook signature is a cryptographic hash (usually HMAC) of the payload, created with a shared secret. It allows the receiver to verify that the request came from the expected sender and that the payload wasn't altered.

How many times should I retry a failed webhook?

There's no fixed number, but common practice is 3 to 5 retries with exponential backoff. After that, mark the delivery as failed and alert your team. Consider the criticality of the event: payment notifications might warrant more retries than non-critical updates.

Can I use webhooks without HTTPS?

You should never use webhooks over plain HTTP in production. HTTPS encrypts the payload in transit, preventing eavesdropping and man-in-the-middle attacks. Even with signatures, HTTP exposes sensitive data.

Ready to test your webhook payloads? Use our JSON Formatter to pretty-print and validate JSON payloads before sending or debugging them.