Webhooks Explained: Signed Payloads and Retries

You've just integrated a third-party payment provider. The docs say: "We'll send a webhook when a payment succeeds." Simple, right? But then you realize: how do you know the request really came from them? What if your server is down when they send it? What if they send the same event twice?

Webhooks are powerful but require careful handling. This guide covers the essentials: signing payloads to verify authenticity, implementing retries to handle failures, and ensuring idempotency to avoid duplicate processing.

What Are Webhooks?

A webhook is an HTTP callback: instead of your app polling an API for changes, the provider sends an HTTP POST to a URL you specify when an event occurs. This makes integrations efficient and near real-time.

Common examples include payment notifications (Stripe, PayPal), CI/CD triggers (GitHub, GitLab), and messaging platforms (Slack, Discord).

Why Secure Webhooks with Signatures?

Your webhook endpoint is publicly accessible. Anyone can send a POST request to it. Without verification, an attacker could forge events—for example, marking an unpaid order as paid.

Signing payloads with a shared secret allows you to verify that the request originated from the expected sender and that the payload hasn't been tampered with.

How HMAC Signatures Work

The provider and you share a secret key. For each webhook, the provider computes an HMAC (Hash-based Message Authentication Code) of the payload using that secret and includes the signature in a header (e.g., X-Signature). You recompute the HMAC on your side and compare.

HMAC ensures integrity and authenticity because only someone with the secret can produce a valid signature.

Implementing Signed Payloads

Let's walk through a typical implementation in Node.js (Express) and Python (Flask). The principles apply to any language.

Step 1: Obtain and Store the Secret

When you register the webhook, the provider gives you a secret. Store it securely—environment variables or a secrets manager—never in code.

Step 2: Capture the Raw Body

Signature verification requires the exact raw body bytes. If your framework parses JSON before you verify, re-serializing may change the bytes and break verification.

In Express, use express.raw({ type: 'application/json' }) for the webhook route. In Flask, access request.get_data() before parsing.

Step 3: Compute and Compare Signatures

Use a constant-time comparison function to prevent timing attacks.

// Node.js (Express)
const crypto = require('crypto');

app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const expected = crypto.createHmac('sha256', secret).update(req.body).digest('hex');
  
  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }
  
  // Process event
  const event = JSON.parse(req.body);
  // ... handle event
  res.status(200).send('OK');
});

In Python, use hmac.compare_digest:

# Python (Flask)
import hmac, hashlib, os
from flask import request, abort

@app.route('/webhook', methods=['POST'])
def webhook():
    signature = request.headers.get('X-Signature')
    secret = os.environ['WEBHOOK_SECRET'].encode()
    expected = hmac.new(secret, request.get_data(), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(signature, expected):
        abort(401)
    event = request.get_json()
    # ... handle event
    return 'OK', 200

Step 4: Handle Timestamps (Optional but Recommended)

Some providers include a timestamp in the signature to prevent replay attacks. For example, Stripe's scheme signs timestamp.payload. You should reject requests with timestamps too far in the past (e.g., older than 5 minutes).

Implementing Retries

Networks fail. Your server might be down, or the request might time out. Reliable webhook systems retry failed deliveries.

On the Provider Side

If you're building the webhook sender, implement retries with exponential backoff. A common pattern: retry after 1 minute, then 5 minutes, then 30 minutes, etc., up to a maximum number of attempts (e.g., 5). Include a unique event ID in the payload so the receiver can deduplicate.

On the Receiver Side

As a receiver, you should:

• Respond quickly (within a few seconds) with a 2xx status to acknowledge receipt. Do heavy processing asynchronously.
• Return non-2xx for errors that warrant a retry (e.g., 500). But be careful: if you return 4xx, the provider may not retry.
• Make your endpoint idempotent so duplicate deliveries don't cause issues.

Idempotency: Handling Duplicate Events

Retries can lead to duplicate events. Use the event ID (included by most providers) to track processed events. Before processing, check if the ID has been seen; if so, skip.

Store processed IDs in a database with a TTL, or use an idempotency key in your business logic.

Testing Webhooks Locally

During development, your local server isn't publicly accessible. Tools like ngrok or localtunnel create a public URL that tunnels to your machine. Alternatively, use a webhook testing service that captures requests and lets you replay them.

Common Pitfalls and Best Practices

• Don't trust the payload before verification. Always verify the signature first.
• Use HTTPS. Webhooks carry sensitive data; encrypt in transit.
• Log everything. Log the raw request (headers and body) for debugging, but redact sensitive info.
• Monitor failures. Set up alerts for repeated webhook failures.
• Version your events. Include a version in the payload so you can evolve without breaking receivers.

Comparison: Signed vs. Unsigned Webhooks

FAQ

How do I verify a webhook signature?

Compute an HMAC of the raw request body using the shared secret, then compare it to the signature header using a constant-time function. Always use the raw body, not parsed JSON.

What should my webhook endpoint return?

Return a 2xx status code quickly to acknowledge receipt. If processing fails, return a 5xx to trigger a retry, but ensure your endpoint is idempotent to handle duplicates.

How many times will a webhook be retried?

It depends on the provider. Most retry several times with exponential backoff over a period of hours or days. Check the provider's documentation for specifics.

When debugging webhook deliveries, inspecting server logs is often necessary. If you're using Nginx, the Nginx Log Analyzer can help you parse and analyze access logs to spot failed requests or unusual patterns.