Webhooks Explained: Signed Payloads and Retries

You've built an API and now need to notify other services when events happen. You set up a webhook endpoint, but you quickly realize that delivering HTTP requests reliably is harder than it looks. Network failures, malicious actors, and duplicate events can cause chaos. This guide explains how to implement webhooks with signed payloads and retries so you can build a robust integration.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks. When an event occurs in your application, you send an HTTP POST request to a URL provided by the subscriber. This allows real-time communication between services without polling.

For example, a payment processor might send a webhook to your app when a charge succeeds. Your app can then update the order status immediately.

Why Sign Webhook Payloads?

Webhook endpoints are publicly accessible, so anyone can send fake requests. Without verification, your app might process fraudulent events. Signing payloads ensures that the request originated from your service and hasn't been tampered with.

Signing typically uses a secret shared between the sender and receiver. The sender computes a hash-based message authentication code (HMAC) over the payload and includes it in a header. The receiver recomputes the HMAC and compares it.

How to Generate and Verify Signatures

Most implementations use HMAC-SHA256. Here's a simple example in Python:

import hmac
import hashlib
import json

def generate_signature(payload, secret):
    body = json.dumps(payload, separators=(',', ':')).encode('utf-8')
    signature = hmac.new(secret.encode('utf-8'), body, hashlib.sha256).hexdigest()
    return signature

def verify_signature(payload, signature, secret):
    expected = generate_signature(payload, secret)
    return hmac.compare_digest(expected, signature)

When sending the webhook, include the signature in a header like X-Signature. The receiver extracts the raw body, recomputes the HMAC, and compares. Always use a constant-time comparison function to prevent timing attacks.

Implementing Retries for Reliable Delivery

Webhooks are sent over the internet, so failures are inevitable. Your server might be down, the network might be slow, or the subscriber might return an error. Retries with exponential backoff are essential.

Retry Strategy

A common approach is to retry with increasing delays: 1 minute, 5 minutes, 30 minutes, 2 hours, etc. Stop after a maximum number of attempts (e.g., 10) or a total time window (e.g., 24 hours).

Only retry on specific conditions: network errors, 5xx responses, or 429 (Too Many Requests). Do not retry on 4xx errors (except 429) because they indicate a client mistake that won't be fixed by retrying.

Example Retry Logic

Here's a simplified retry mechanism in Python using the requests library:

import requests
import time
from requests.exceptions import RequestException

def send_webhook(url, payload, secret, max_retries=5):
    headers = {'Content-Type': 'application/json'}
    signature = generate_signature(payload, secret)
    headers['X-Signature'] = signature
    
    for attempt in range(max_retries):
        try:
            response = requests.post(url, json=payload, headers=headers, timeout=10)
            if response.status_code < 500 and response.status_code != 429:
                return response.status_code
        except RequestException:
            pass
        
        # Exponential backoff with jitter
        delay = (2 ** attempt) + (random.random() * 0.1)
        time.sleep(delay)
    
    return None  # Failed after retries

In production, you'd store failed attempts in a queue and process them asynchronously. Tools like Celery or cloud queues (AWS SQS, Google Cloud Tasks) can help.

Best Practices for Webhook Security

• Use HTTPS: Always send webhooks over TLS to prevent eavesdropping.
• Include a timestamp: Add a timestamp to the payload and reject requests older than a few minutes to prevent replay attacks.
• Rotate secrets: Periodically change the signing secret and support multiple active secrets during rotation.
• Validate the URL: When registering a webhook, ensure the URL is reachable and returns a 2xx to a test request.
• Rate limiting: Protect your webhook sender from being abused by limiting the number of requests per subscriber.

Handling Duplicates and Idempotency

Retries can cause duplicate deliveries. Include a unique event ID in the payload and have the receiver track processed IDs. If an event ID has been seen before, ignore it.

Alternatively, design your event handlers to be idempotent. For example, updating a record with the same data multiple times should have the same effect as doing it once.

Testing Webhooks

Use tools like ngrok to expose your local server and test webhook delivery. Simulate failures by returning error codes and verify that retries occur as expected.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request came from a trusted source and wasn't altered.

How many times should I retry a failed webhook?

It depends on your reliability needs. A common practice is 5-10 retries with exponential backoff over 24 hours. Always allow subscribers to manually trigger a retry.

Can I use webhooks without signing?

You can, but it's risky. Without signing, anyone can send fake events. Always sign webhooks in production to prevent fraud.

Ready to test your webhook payloads? Use our JSON Formatter to validate and pretty-print JSON payloads before sending.