Webhooks Explained: Signed Payloads and Reliable Retries

You've just integrated a third-party payment gateway or a CI/CD service, and now you need real-time notifications. Webhooks are the answer—but without proper security and reliability, they can become a source of frustration. Attackers can spoof requests, and network glitches can cause missed events. In this guide, we'll walk through securing webhooks with signed payloads and implementing robust retry mechanisms.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks triggered by specific events. When an event occurs (e.g., a payment succeeds), the source service sends an HTTP POST request to a URL you provide, containing event data. This push model is more efficient than polling and enables real-time reactions.

However, webhooks introduce two challenges: authenticity (ensuring the request is from the expected sender) and reliability (ensuring you don't miss events).

Securing Webhooks with Signed Payloads

A signed payload uses a secret shared between the sender and receiver to create a cryptographic signature. The sender includes this signature in a header, and the receiver verifies it to confirm the payload's integrity and origin.

How HMAC Signing Works

HMAC (Hash-based Message Authentication Code) is the standard approach. The sender computes an HMAC over the request body using the shared secret and includes it in a header like X-Signature. The receiver recomputes the HMAC and compares it.

// Example: Generating a signature in Node.js
const crypto = require('crypto');

function generateSignature(payload, secret) {
  return crypto
    .createHmac('sha256', secret)
    .update(JSON.stringify(payload))
    .digest('hex');
}

// On the receiving end, verify:
function verifySignature(payload, secret, receivedSignature) {
  const expected = generateSignature(payload, secret);
  // Use timing-safe comparison to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(expected, 'hex'),
    Buffer.from(receivedSignature, 'hex')
  );
}

Key points:

• Always use a strong, unique secret per webhook endpoint.
• Compare signatures using a timing-safe function to avoid leaking information.
• Include a timestamp in the signature to prevent replay attacks (e.g., sign timestamp + payload).

Additional Security Measures

• HTTPS: Always use HTTPS to encrypt data in transit.
• IP allowlisting: If the provider publishes IP ranges, restrict incoming requests to those IPs.
• Payload size limits: Reject overly large payloads to mitigate DoS.

Implementing Reliable Retries

Networks are unreliable. A webhook might fail due to timeouts, server errors, or temporary outages. A robust retry strategy ensures eventual delivery without overwhelming your system or the sender.

Retry Best Practices

1. Exponential backoff: Increase the delay between retries (e.g., 1s, 2s, 4s, 8s...).
2. Maximum retry limit: Cap the number of attempts (e.g., 5) to avoid infinite loops.
3. Dead-letter queue: After exhausting retries, store failed events for manual inspection.
4. Idempotency: Ensure that processing the same event multiple times doesn't cause duplicate side effects.

Idempotency in Practice

Include a unique event ID in the payload. On the receiver side, track processed IDs (in a database or cache) and ignore duplicates. For example:

// Pseudocode for idempotent processing
if (eventStore.exists(event.id)) {
  return 200; // Already processed
}
// Process event...
eventStore.save(event.id);

Handling Failures Gracefully

Respond quickly with a 2xx status to acknowledge receipt. If processing takes time, enqueue the event and process asynchronously. Return 5xx only for transient errors that warrant a retry; for permanent errors (e.g., invalid payload), return 4xx to prevent futile retries.

Comparison: Push vs. Poll

Testing Your Webhook Implementation

Use tools like ngrok to expose your local server and simulate webhook deliveries. Verify signature validation and retry behavior by intentionally failing requests.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (often HMAC) that allows the receiver to verify the request's authenticity and integrity using a shared secret.

How many times should I retry a failed webhook?

There's no universal number, but 3–5 retries with exponential backoff are common. After that, log the failure for manual review.

Can I use webhooks without HTTPS?

Technically yes, but it's highly insecure. Always use HTTPS to prevent eavesdropping and man-in-the-middle attacks.

Ready to inspect webhook payloads? Use our JSON formatter to pretty-print and validate JSON payloads quickly.