Webhooks Explained: Signed Payloads and Retries

You've just integrated a payment provider's webhook. The docs say "we'll POST to your endpoint when a payment succeeds." Simple, right? Then you notice duplicate events, missed notifications, and a security alert about forged requests. Webhooks are simple in concept but tricky in production. This article explains how to implement webhooks correctly using signed payloads and retries, so you can build reliable and secure integrations.

What Are Webhooks?

A webhook is a reverse API: instead of your application polling a service for updates, the service pushes events to your endpoint via HTTP POST. It's efficient and near real-time. Common uses include payment notifications, CI/CD triggers, and chat integrations.

But webhooks introduce challenges: how do you know the request really came from the provider? What if your server is down? What if the same event arrives twice? Let's tackle these one by one.

Securing Webhooks with Signed Payloads

Anyone can send an HTTP request to your endpoint. Without verification, attackers could forge events, leading to fraud or data corruption. The standard solution is to sign the payload with a shared secret.

How HMAC Signatures Work

The provider and you share a secret key. For each event, the provider computes an HMAC (Hash-based Message Authentication Code) of the request body using that secret and sends the signature in a header (e.g., X-Signature). You recompute the HMAC on your side and compare. If they match, the request is authentic and untampered.

Here's a minimal example in Python using Flask:

import hmac
import hashlib
from flask import Flask, request, abort

app = Flask(__name__)
SECRET = b'your_shared_secret'

@app.route('/webhook', methods=['POST'])
def webhook():
    signature = request.headers.get('X-Signature')
    if not signature:
        abort(400, 'Missing signature')
    body = request.get_data()
    expected = hmac.new(SECRET, body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, signature):
        abort(401, 'Invalid signature')
    # Process the event
    return '', 200

Always use hmac.compare_digest for constant-time comparison to prevent timing attacks. Also, verify the signature before parsing the JSON body to avoid unnecessary processing.

Best Practices for Signatures

• Use a strong secret: generate a random key of at least 32 bytes and store it securely (e.g., environment variable).
• Include a timestamp: to prevent replay attacks, providers often include a timestamp in the signed payload. Reject requests older than a few minutes.
• Rotate secrets periodically: support multiple active secrets during rotation to avoid downtime.
• Document the signature algorithm: specify the hash function (SHA-256 is standard) and header name.

Handling Failures with Retries

Webhooks are fire-and-forget from the provider's perspective, but your endpoint might be temporarily unavailable. To ensure delivery, providers implement retry policies. As a receiver, you must be prepared for retries and duplicates.

Retry Strategies Used by Providers

Most providers retry with exponential backoff: e.g., immediately, then after 1 minute, 5 minutes, 30 minutes, etc., up to a maximum number of attempts (often 5–10). Some also allow you to configure a retry schedule. The exact behavior varies, so read the provider's docs.

Designing Your Endpoint for Retries

Your endpoint should:

• Respond quickly: acknowledge receipt with a 2xx status within a few seconds. Do heavy processing asynchronously (e.g., queue the event).
• Be idempotent: the same event may be delivered multiple times. Use an idempotency key (often provided in the event payload or header) to deduplicate. Store processed event IDs for a reasonable period.
• Return appropriate status codes: 2xx for success, 4xx for client errors (provider may not retry), 5xx for server errors (provider will retry).

Example of idempotency check in Python:

processed_events = set()  # In production, use a database or cache

@app.route('/webhook', methods=['POST'])
def webhook():
    # ... signature verification ...
    event = request.json
    event_id = event['id']
    if event_id in processed_events:
        return '', 200  # Already processed
    # Process event
    processed_events.add(event_id)
    return '', 200

Testing Your Webhook Implementation

Before going live, test your endpoint thoroughly:

1. Simulate signed requests: write unit tests that generate valid and invalid signatures.
2. Test retry scenarios: simulate failures and ensure your endpoint handles duplicates.
3. Use provider tools: many providers offer a CLI or dashboard to send test events.
4. Monitor and log: log incoming events (without sensitive data) and set up alerts for signature failures or high error rates.

Comparison: Polling vs Webhooks

FAQ

How do I verify a webhook signature?

Compute the HMAC of the raw request body using the shared secret and compare it to the signature header using a constant-time comparison function. Always verify before processing the payload.

What should my endpoint return to acknowledge a webhook?

Return a 2xx status code (e.g., 200 OK) as soon as you've received and queued the event for processing. Avoid long processing in the request handler to prevent timeouts.

How can I handle duplicate webhook deliveries?

Use an idempotency key from the event payload (or a unique event ID) to track processed events. Store these IDs in a database or cache and skip processing if the ID has been seen before.

Ready to test your webhook payloads? Use our JSON Formatter to pretty-print and validate the JSON body of incoming webhooks, making debugging easier.