Webhooks Explained: Signed Payloads and Retry Logic

You've built an API and now need to notify other services when events happen. You decide to implement webhooks: HTTP callbacks that push data to subscriber URLs. But soon you face two critical questions: How do you ensure the payload hasn't been tampered with? And what happens when the receiving server is down or returns an error? Without proper safeguards, webhooks can become a security hole and a reliability nightmare.

In this guide, we'll walk through the essentials of securing webhooks with signed payloads and implementing robust retry logic. You'll learn how to generate and verify HMAC signatures, design idempotent handlers, and build a retry mechanism that handles failures gracefully. By the end, you'll have a production-ready approach to webhooks.

Why Webhooks Need Security and Reliability

Webhooks are essentially server-to-server HTTP requests triggered by events. Unlike API calls initiated by a client, webhooks are pushed from your server to a subscriber's endpoint. This push model introduces two main challenges:

• Security: The payload travels over the internet and could be intercepted or spoofed. Subscribers must verify that the request genuinely came from your service.
• Reliability: The subscriber's server might be temporarily unavailable, slow, or return an error. You need a strategy to retry failed deliveries without causing duplicate processing.

Ignoring these challenges can lead to data breaches, lost events, or inconsistent state across systems. Let's tackle them one by one.

Securing Webhooks with Signed Payloads

A signed payload uses a cryptographic hash to prove the integrity and authenticity of the request. The most common approach is HMAC (Hash-based Message Authentication Code) with a shared secret. Here's how it works:

1. Shared secret: Both the sender (your service) and the receiver (subscriber) share a secret key. This key is generated when the subscriber registers their webhook endpoint.
2. Signature generation: For each webhook request, the sender computes an HMAC of the request body (and sometimes headers like timestamp) using the secret. The signature is sent in a header, e.g., X-Signature.
3. Signature verification: The receiver recomputes the HMAC using the same secret and compares it to the received signature. If they match, the payload is authentic and unmodified.

Let's see a code example in Node.js for generating and verifying signatures.

const crypto = require('crypto');

// Sender: generate signature
function generateSignature(payload, secret) {
  return crypto.createHmac('sha256', secret)
    .update(JSON.stringify(payload))
    .digest('hex');
}

// Receiver: verify signature
function verifySignature(payload, secret, receivedSignature) {
  const expectedSignature = generateSignature(payload, secret);
  // Use timingSafeEqual to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(expectedSignature, 'hex'),
    Buffer.from(receivedSignature, 'hex')
  );
}

Important: Always use a constant-time comparison function like timingSafeEqual to avoid timing attacks. Also, include a timestamp in the signed data to prevent replay attacks. For example, sign timestamp + '.' + payload and reject requests older than a few minutes.

Best Practices for Signed Payloads

• Use HMAC-SHA256 or stronger.
• Include a timestamp in the signature and enforce a maximum age (e.g., 5 minutes).
• Never transmit the secret in the request; it's only used for signing.
• Rotate secrets periodically and support multiple active secrets during rotation.

Implementing Retry Logic for Reliable Delivery

Even with security in place, webhooks can fail due to network issues, server errors, or timeouts. A robust retry mechanism ensures events are eventually delivered. Here's a step-by-step approach:

1. Set a timeout: Define a reasonable timeout for webhook requests (e.g., 10 seconds). If the subscriber doesn't respond in time, consider it a failure.
2. Classify responses: Treat 2xx status codes as success. For 4xx errors (except 429), retrying is usually pointless because the request is invalid. For 5xx errors and timeouts, retry with exponential backoff.
3. Exponential backoff with jitter: Schedule retries at increasing intervals (e.g., 1s, 5s, 30s, 5m, 30m) and add random jitter to avoid thundering herd problems.
4. Limit retries: Set a maximum number of attempts (e.g., 5) and then mark the delivery as failed. Optionally, notify the subscriber via email or a dashboard.
5. Idempotency: Since retries can cause duplicate deliveries, include a unique event ID in the payload and header. Subscribers should use this ID to deduplicate processing.

Here's a simplified retry queue implementation in Python using requests and time:

import requests
import time
import random

def deliver_webhook(url, payload, secret, max_retries=5):
    signature = generate_signature(payload, secret)
    headers = {'X-Signature': signature, 'Content-Type': 'application/json'}
    for attempt in range(max_retries):
        try:
            response = requests.post(url, json=payload, headers=headers, timeout=10)
            if 200 <= response.status_code < 300:
                return True
            elif response.status_code == 429 or 500 <= response.status_code < 600:
                # Retryable error
                pass
            else:
                # Non-retryable error
                return False
        except requests.exceptions.RequestException:
            # Network error, retry
            pass
        # Exponential backoff with jitter
        delay = min(30, 2 ** attempt) + random.uniform(0, 1)
        time.sleep(delay)
    return False

Idempotency: Handling Duplicate Deliveries

Because retries can cause the same event to be delivered multiple times, subscribers must implement idempotent processing. The standard approach is to include a unique event ID (e.g., a UUID) in the payload and have the receiver store processed IDs for a certain period. If a duplicate ID is received, the receiver acknowledges it without reprocessing.

Here's a simple idempotency check in a webhook handler:

processed_events = set()

def handle_webhook(event):
    event_id = event['id']
    if event_id in processed_events:
        return 'already processed'
    # Process event
    processed_events.add(event_id)
    return 'success'

In production, use a persistent store like Redis or a database with TTL to track processed IDs.

Comparison: Signed Payloads vs. Other Security Measures

HMAC signed payloads strike a good balance between security and simplicity for most webhook use cases.

Putting It All Together

To build a reliable and secure webhook system:

• Generate a unique secret per subscriber and sign each payload with HMAC.
• Include a timestamp and event ID in the signed data.
• Implement retries with exponential backoff and jitter for transient failures.
• Ensure idempotency on the receiver side using event IDs.
• Monitor delivery success rates and alert on failures.

When debugging webhook deliveries, inspecting the raw HTTP requests and responses is invaluable. If you're working with Nginx logs to trace incoming webhook requests, the Nginx Log Analyzer can help you parse and filter log entries to identify failed deliveries or suspicious patterns.

FAQ

What is a signed webhook payload?

A signed webhook payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request's authenticity and integrity. The sender computes the signature using a shared secret and includes it in a header; the receiver recomputes and compares it.

How many times should I retry a failed webhook?

A common practice is 3 to 5 retries with exponential backoff. After that, mark the delivery as failed and notify the subscriber. The exact number depends on your SLA and the criticality of the event.

Why is idempotency important for webhooks?

Retries can cause duplicate deliveries. Idempotency ensures that processing the same event multiple times has the same effect as processing it once, preventing data corruption or duplicate actions.