Webhooks Explained: Signed Payloads and Retries

Backend2026-11-16TryQuickToolBox

You've just integrated a payment provider or a CI/CD service, and now you need to receive real-time events. Webhooks are the standard way to push data between systems, but many implementations fail in production due to security flaws or unreliable delivery. This guide explains how to secure webhooks with signed payloads and implement robust retry mechanisms.

What Are Webhooks and Why They Matter

A webhook is an HTTP callback: when an event occurs in a source system (e.g., a new order), it sends an HTTP POST request to a URL you provide. Unlike polling, webhooks are efficient and near real-time. However, because they are HTTP-based, you must handle security and reliability explicitly.

Common use cases include payment notifications, CI/CD pipeline triggers, chat notifications, and syncing data between SaaS applications.

Securing Webhooks with Signed Payloads

Webhooks are publicly accessible endpoints, so attackers can send fake requests. To prevent this, providers sign each payload with a secret key. The receiver verifies the signature to ensure authenticity and integrity.

How HMAC Signatures Work

The most common method is HMAC-SHA256. The provider computes a hash of the request body using a shared secret and includes it in a header (e.g., X-Signature). The receiver recomputes the hash and compares it.

// Node.js example: generating a signature
const crypto = require('crypto');
const secret = 'your_webhook_secret';
const payload = JSON.stringify({ event: 'order.created', id: 123 });
const signature = crypto.createHmac('sha256', secret).update(payload).digest('hex');
// Send signature in header: X-Signature: sha256=...

Verifying Signatures on the Receiver

Always use a constant-time comparison to prevent timing attacks. In Node.js, use crypto.timingSafeEqual.

// Express.js middleware example
const crypto = require('crypto');

function verifySignature(req, res, next) {
  const signature = req.headers['x-signature'];
  const payload = JSON.stringify(req.body); // raw body needed
  const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');
  const trusted = Buffer.from(`sha256=${expected}`, 'utf8');
  const received = Buffer.from(signature, 'utf8');
  if (trusted.length !== received.length || !crypto.timingSafeEqual(trusted, received)) {
    return res.status(401).send('Invalid signature');
  }
  next();
}

Important: Use the raw request body for verification, not the parsed JSON, to avoid discrepancies.

Additional Security Measures

Implementing Reliable Retries

Webhooks can fail due to network issues, server downtime, or bugs. Providers typically retry failed deliveries with exponential backoff. As a receiver, you must be idempotent and handle duplicate events.

Designing for Idempotency

Each webhook event should have a unique ID. Store processed IDs to ignore duplicates. Use a database with a unique constraint or a cache like Redis.

// Pseudocode for idempotent processing
if (await redis.exists(`webhook:${event.id}`)) {
  return res.status(200).send('Already processed');
}
await processEvent(event);
await redis.set(`webhook:${event.id}`, '1', 'EX', 86400); // 24h TTL

Retry Strategies

Providers usually retry with increasing delays (e.g., 1s, 5s, 30s, 5m, 1h). Ensure your endpoint responds quickly (within 5 seconds) to avoid timeouts. If processing takes longer, acknowledge receipt and process asynchronously.

Here's a comparison of common retry patterns:

PatternProsCons
Fixed intervalSimpleCan hammer a recovering server
Exponential backoffReduces load during outagesDelayed delivery for transient errors
Exponential backoff with jitterPrevents thundering herdSlightly more complex

Monitoring and Dead Letter Queues

Log all webhook attempts and failures. After a maximum number of retries, move events to a dead letter queue (DLQ) for manual inspection. Alert on high failure rates.

Putting It All Together: A Step-by-Step Implementation

  1. Set up an HTTPS endpoint that accepts POST requests and reads the raw body.
  2. Verify the signature using the shared secret and constant-time comparison.
  3. Check the timestamp to prevent replay attacks (if provided).
  4. Ensure idempotency by checking the event ID against a cache or database.
  5. Process the event asynchronously if it might take time; respond with 200 OK immediately.
  6. Implement retry logic on the sender side with exponential backoff and jitter.
  7. Monitor and alert on failures, and use a DLQ for persistent errors.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request came from a trusted source and wasn't tampered with.

How do I handle duplicate webhook events?

Use the unique event ID provided in the payload to track processed events. Store IDs in a database or cache with a TTL to ignore duplicates.

What should my webhook endpoint return?

Return a 2xx status code (e.g., 200 OK) quickly to acknowledge receipt. If processing fails, return a 5xx to trigger a retry, but only for transient errors.

When testing webhook payloads, you might need to inspect JSON structures. Use the JSON Formatter to validate and pretty-print webhook payloads for easier debugging.