Webhooks Explained: Signed Payloads and Retries

You've integrated a payment gateway or a CI/CD service, and now you need to receive real-time events. Webhooks are the standard solution, but they come with two big challenges: verifying that the request actually came from the expected sender, and handling failures when your endpoint is down or slow. This article explains how to implement signed webhook payloads and robust retry logic, with practical code examples.

What Are Webhooks?

A webhook is a user-defined HTTP callback. Instead of your application polling an API for updates, the provider sends an HTTP POST request to a URL you specify whenever an event occurs. This makes integrations efficient and near real-time.

However, webhooks are push-based, which means your endpoint must be publicly accessible, handle unexpected load, and deal with duplicate or malicious requests. That's where signing and retries come in.

Why Sign Webhook Payloads?

Without verification, anyone who knows your webhook URL could send fake events. Signing payloads with a shared secret allows you to authenticate the sender and ensure the payload hasn't been tampered with.

The most common approach is HMAC (Hash-based Message Authentication Code) using SHA-256. The provider computes a signature over the raw request body using a secret key, and includes it in a header (e.g., X-Signature). Your server recomputes the signature and compares them.

Implementing HMAC Verification

Here's a simple example in Node.js using Express:

const crypto = require('crypto');
const express = require('express');
const app = express();

// Use raw body for signature verification
app.use(express.raw({ type: 'application/json' }));

app.post('/webhook', (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const payload = req.body; // Buffer

  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }

  // Process the event
  const event = JSON.parse(payload);
  console.log('Received event:', event.type);
  res.status(200).send('OK');
});

app.listen(3000, () => console.log('Webhook server running'));

Key points:

• Use raw body: Parse the body as raw bytes, not JSON, because any transformation changes the signature.
• Timing-safe comparison: Use crypto.timingSafeEqual to prevent timing attacks.
• Secret management: Store secrets securely (environment variables, secret managers).

Handling Retries: Making Webhooks Reliable

Networks fail, servers crash, and deployments cause downtime. A reliable webhook system must retry failed deliveries. But naive retries can cause duplicate processing and overload your system.

Retry Strategies

Common retry patterns include:

• Exponential backoff: Wait longer between each retry (e.g., 1s, 2s, 4s, 8s...).
• Maximum retry limit: Stop after N attempts (e.g., 5) and mark the delivery as failed.
• Jitter: Add randomness to avoid thundering herd problems.

Most providers implement something like: retry up to 5 times with exponential backoff over 24 hours.

Idempotency: The Key to Safe Retries

Because retries can deliver the same event multiple times, your endpoint must be idempotent. Include a unique event ID in the payload and store processed IDs to skip duplicates.

Example using a simple in-memory set (use a database in production):

const processedEvents = new Set();

app.post('/webhook', (req, res) => {
  // ... signature verification ...
  const event = JSON.parse(payload);
  if (processedEvents.has(event.id)) {
    return res.status(200).send('Already processed');
  }
  // Process event
  processedEvents.add(event.id);
  res.status(200).send('OK');
});

For production, use a database with a unique constraint on the event ID.

Best Practices for Webhook Consumers

• Respond quickly: Acknowledge receipt with a 2xx status before doing heavy processing. Queue the event for async handling.
• Validate timestamp: If the provider includes a timestamp, reject requests older than a few minutes to prevent replay attacks.
• Use HTTPS: Always expose your webhook endpoint over HTTPS to protect payloads in transit.
• Log everything: Log incoming requests, signatures, and processing results for debugging.
• Monitor failures: Set up alerts for repeated delivery failures.

Comparison of Retry Policies

FAQ

How do I verify a webhook signature?

Compute the HMAC of the raw request body using the shared secret and compare it to the signature header using a timing-safe function. Never compare with simple equality.

What should I do if my webhook endpoint is down?

The provider should retry with exponential backoff. Ensure your endpoint returns a 2xx quickly and processes events asynchronously to minimize downtime impact.

Can I use webhooks without signing?

It's not recommended. Without signatures, you cannot verify the sender, making your endpoint vulnerable to fake events. Always use signatures when available.

When testing webhook payloads, you might need to inspect JSON data. Use the JSON Formatter to pretty-print and validate payloads quickly.