Webhooks Explained: Signed Payloads and Retries

You've built a webhook endpoint to receive events from a third-party service. Everything works in testing, but in production you notice duplicate events, missed notifications, and occasional security concerns about forged requests. These are common pitfalls when implementing webhooks without proper safeguards.

Webhooks are HTTP callbacks that let services notify your application when something happens. Unlike polling, they provide real-time updates. But to use them reliably, you need to address two critical aspects: authenticity (ensuring the request came from the expected sender) and delivery (handling failures and retries). This article explains both with practical code examples.

How Webhooks Work

A webhook is a user-defined HTTP POST endpoint. When an event occurs, the source service sends a request to that URL with a payload describing the event. For example, a payment processor might send a webhook when a charge succeeds.

The flow is simple:

1. You register a webhook URL with the service.
2. The service sends an HTTP POST to that URL when an event fires.
3. Your endpoint processes the payload and returns a 2xx status to acknowledge receipt.

Without proper verification, anyone can send requests to your endpoint. Without retry handling, transient failures can cause lost events. Let's tackle each.

Securing Webhooks with Signed Payloads

To ensure a webhook originated from the expected sender, services sign the payload using a shared secret. The signature is typically an HMAC (Hash-based Message Authentication Code) of the request body, sent in a header like X-Signature.

How HMAC Signing Works

The sender computes an HMAC using a secret key and the raw request body, then includes the hex-encoded signature in a header. Your endpoint recomputes the HMAC with the same secret and compares it to the received signature. If they match, the request is authentic.

Here's a minimal example in Node.js using Express:

const crypto = require('crypto');
const express = require('express');
const app = express();

// Use raw body for signature verification
app.use(express.raw({ type: 'application/json' }));

app.post('/webhook', (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const payload = req.body;

  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }

  // Process the event
  const event = JSON.parse(payload);
  // ... handle event
  res.status(200).send('OK');
});

Key points:

• Always use the raw request body for verification, not a parsed object. Parsing can alter the payload (e.g., whitespace) and break the signature.
• Use a timing-safe comparison to prevent timing attacks.
• Store secrets securely (environment variables, secret managers) and rotate them periodically.

Additional Security Measures

• Timestamp validation: Include a timestamp in the signature (e.g., t=timestamp,v1=signature) and reject requests older than a few minutes to prevent replay attacks.
• IP allowlisting: If the provider publishes IP ranges, restrict your endpoint to those IPs.
• HTTPS: Always use HTTPS to protect the payload in transit.

Implementing Reliable Retries

Webhook delivery is not guaranteed. Networks fail, your server might be down, or you might return a non-2xx status. To handle this, senders typically retry with exponential backoff. As a receiver, you must design your endpoint to be idempotent and to respond quickly.

On the Sender Side (If You Send Webhooks)

If you're building a service that sends webhooks, implement retries with backoff. A common strategy:

• Retry on network errors or 5xx responses.
• Do not retry on 4xx responses (except 429 Too Many Requests).
• Use exponential backoff with jitter: wait 1s, 2s, 4s, 8s, etc., up to a maximum.
• Set a maximum number of retries (e.g., 5) and consider a dead-letter queue for failures.

Example retry logic in Python:

import time
import requests
from requests.exceptions import RequestException

def send_webhook(url, payload, max_retries=5):
    for attempt in range(max_retries):
        try:
            response = requests.post(url, json=payload, timeout=5)
            if 200 <= response.status_code < 300:
                return True
            elif 400 <= response.status_code < 500 and response.status_code != 429:
                # Client error, don't retry
                return False
        except RequestException:
            pass
        # Exponential backoff with jitter
        delay = (2 ** attempt) + random.uniform(0, 1)
        time.sleep(delay)
    return False

On the Receiver Side

Your endpoint should:

• Acknowledge quickly: Return a 2xx within a few seconds. If processing takes longer, enqueue the event and process asynchronously.
• Be idempotent: Use an event ID from the payload to deduplicate. Store processed IDs and ignore repeats.
• Handle concurrency: Use database transactions or locks to avoid race conditions when processing the same event concurrently.

Idempotency example using a database unique constraint:

CREATE TABLE processed_events (
  event_id VARCHAR(255) PRIMARY KEY,
  processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Before processing, attempt to insert the event ID. If it violates the primary key, skip processing.

Testing Webhooks Locally

During development, you need a public URL for the service to reach your local server. Tools like ngrok or localtunnel expose your localhost. Always use test secrets and verify signatures in development to catch issues early.

Comparison: Polling vs Webhooks

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) generated with a shared secret. It allows the receiver to verify that the request came from the expected sender and wasn't tampered with.

How do I handle duplicate webhook deliveries?

Make your endpoint idempotent by using a unique event ID from the payload. Store processed IDs and ignore duplicates. This ensures that retries or network issues don't cause double processing.

Should I retry failed webhook deliveries?

Yes, but only for transient errors (network issues, 5xx). Use exponential backoff with jitter and set a maximum retry limit. Avoid retrying on 4xx errors (except 429) as they indicate a problem with the request itself.

When debugging webhook payloads, you often need to inspect JSON data. The JSON Formatter can help you pretty-print and validate JSON payloads quickly.