Webhooks Explained: Signed Payloads and Retries

You've just integrated a third-party payment provider. The user completes a purchase, but your database never gets updated. Why? Because the webhook that was supposed to notify your app failed silently. Without proper signature verification and retry logic, webhooks can become a source of mysterious bugs and security holes.

In this article, we'll explain how webhooks work, why you must verify signed payloads, and how to design a robust retry mechanism. We'll also show you how to test and debug webhooks effectively.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks. They allow one system to send real-time data to another when an event occurs. For example, when a payment is processed, the payment provider sends an HTTP POST request to a URL you specify, containing details about the event.

Unlike polling, where your app repeatedly asks for updates, webhooks push data to you as soon as it's available. This makes them efficient and near real-time.

The Security Problem: Why Verify Signatures?

Webhook endpoints are publicly accessible URLs. Anyone can send a POST request to them. Without verification, an attacker could forge a request to, say, mark an order as paid without actual payment. To prevent this, providers sign each payload with a secret key.

How HMAC Signatures Work

Most providers use HMAC (Hash-based Message Authentication Code) with SHA-256. They compute a hash of the request body using a shared secret and include it in a header, like X-Signature. Your server recomputes the hash and compares it to the header. If they match, the request is authentic.

Here's a simple example in Python:

import hmac
import hashlib

def verify_signature(payload, secret, signature):
    expected = hmac.new(secret.encode(), payload, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

Always use a constant-time comparison function like hmac.compare_digest to avoid timing attacks.

Common Mistakes to Avoid

• Using the wrong secret: Each environment (test, live) should have its own secret.
• Not verifying the timestamp: Some providers include a timestamp to prevent replay attacks. Reject requests older than a few minutes.
• Logging secrets: Never log the secret or the full signature in production.

Designing Reliable Retries

Webhooks are sent over the internet, so failures are inevitable. Networks drop, servers restart, and endpoints time out. A good webhook system retries failed deliveries with exponential backoff.

How Retries Typically Work

Providers usually retry with increasing delays: after 1 minute, 5 minutes, 30 minutes, etc., up to a maximum number of attempts (e.g., 5-10). If all retries fail, the event is marked as failed and may be available for manual replay.

Your endpoint should respond quickly with a 2xx status code to acknowledge receipt. If you return a non-2xx code or time out, the provider will retry.

Implementing Idempotency

Because retries can cause duplicate deliveries, your endpoint must be idempotent. Use the event ID (included in the payload or headers) to track processed events. Before processing, check if you've already handled that ID. Store processed IDs in a database with a TTL.

Example idempotency check:

def process_event(event_id, data):
    if db.exists(f"processed:{event_id}"):
        return  # already processed
    # process data
    db.setex(f"processed:{event_id}", 86400, "1")  # expire after 24h

Building a Webhook Receiver: Step-by-Step

1. Expose an HTTPS endpoint: Use TLS to protect data in transit.
2. Verify the signature: Reject requests with invalid signatures with a 401.
3. Check the timestamp: If provided, ensure it's within an acceptable window (e.g., 5 minutes).
4. Parse the payload: Validate the JSON structure and required fields.
5. Check idempotency: Use the event ID to avoid duplicates.
6. Process asynchronously: Respond with 200 immediately, then process in a background job to avoid timeouts.
7. Log and monitor: Record failures and set up alerts.

Testing Webhooks Locally

During development, you need a way to receive webhooks on your local machine. Tools like ngrok or localtunnel create a public URL that forwards to your localhost. Alternatively, many providers offer a CLI to trigger test events.

Always test signature verification with both valid and invalid signatures. Simulate retries by returning error codes and observing the provider's retry behavior.

Comparison: Webhooks vs. Polling

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request came from the expected sender and wasn't tampered with.

How many times should a webhook be retried?

Most providers retry 5-10 times with exponential backoff. The exact number depends on the provider's policy. Your endpoint should be idempotent to handle duplicates.

What HTTP status code should my webhook return?

Return a 2xx status code (e.g., 200 OK) to acknowledge successful receipt. Any other code (4xx or 5xx) signals failure and triggers retries.

Ready to test your webhook payloads? Use our JSON Formatter to validate and pretty-print JSON payloads quickly.