Webhooks Explained: Signed Payloads and Retries

You've just integrated a payment gateway, and now you need to update your database when a payment succeeds. Polling the API every few seconds is wasteful and slow. Webhooks solve this by pushing events to your server as they happen. But without proper security and reliability, webhooks can become a source of bugs and vulnerabilities. This article explains how to implement webhooks correctly, focusing on signed payloads and retries.

What Are Webhooks?

A webhook is an HTTP callback: an event occurs in a service, and that service sends an HTTP POST request to a URL you provide. The payload typically contains JSON data about the event. For example, when a customer completes a payment, Stripe sends a charge.succeeded event to your endpoint.

Webhooks are push-based, so they reduce latency and server load compared to polling. However, they introduce challenges: how do you know the request is genuine? What if your server is down when the event fires?

Why Signed Payloads Matter

Anyone can send an HTTP POST to your webhook endpoint. Without verification, an attacker could forge events, leading to unauthorized actions like marking an order as paid. Signed payloads solve this by including a cryptographic signature that only the sender and you can generate.

The most common method is HMAC (Hash-based Message Authentication Code) with a shared secret. The sender computes a hash of the payload using the secret and includes it in a header. You recompute the hash and compare. If they match, the request is authentic.

How to Verify a Signed Payload

Here's a step-by-step approach using Node.js and Express:

1. Retrieve the raw request body as a string. Do not parse it as JSON before verification, as parsing can alter the byte sequence.
2. Extract the signature from the header (e.g., X-Signature).
3. Compute the HMAC using your secret and the raw body.
4. Compare the computed signature with the received one using a constant-time comparison to prevent timing attacks.

const crypto = require('crypto');

function verifySignature(rawBody, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signature)
  );
}

app.post('/webhook', express.raw({type: 'application/json'}), (req, res) => {
  const signature = req.headers['x-signature'];
  if (!verifySignature(req.body, signature, process.env.WEBHOOK_SECRET)) {
    return res.status(401).send('Invalid signature');
  }
  const event = JSON.parse(req.body);
  // Process event
  res.status(200).send('OK');
});

Always use a strong secret and rotate it periodically. Never expose it in client-side code.

Implementing Retries with Backoff

Webhooks are delivered over the internet, so failures happen. Your server might be down, or the network might hiccup. A robust webhook system retries failed deliveries with exponential backoff.

Typically, the sender retries if it doesn't receive a 2xx response within a timeout (e.g., 5 seconds). The retry schedule might be: immediate, then after 1 minute, 5 minutes, 30 minutes, 2 hours, etc., up to a maximum number of attempts (e.g., 5).

As the receiver, you must respond quickly to avoid timeouts. Acknowledge receipt with a 200 OK and process the event asynchronously.

Handling Idempotency

Because retries can cause duplicate deliveries, your event processing must be idempotent. Include a unique event ID in the payload and store processed IDs. Before processing, check if the ID has already been handled.

async function processEvent(event) {
  const { id, type, data } = event;
  if (await db.processedEvents.findOne({ id })) {
    return; // Already processed
  }
  await db.processedEvents.insertOne({ id });
  // Handle event based on type
}

Use a database transaction to ensure the event is marked as processed only after successful handling.

Best Practices for Webhook Security

• Always verify signatures before processing.
• Use HTTPS to prevent eavesdropping.
• Validate the timestamp if included, to prevent replay attacks.
• Limit payload size to avoid denial-of-service.
• Log all webhook attempts for debugging and auditing.

Comparison: Polling vs Webhooks

FAQ

How do I test webhooks locally?

Use a tool like ngrok to expose your local server to the internet. Configure the webhook provider to send events to your ngrok URL.

What if my server is down when a webhook is sent?

The sender should retry with backoff. Ensure your server is highly available and responds quickly to avoid timeouts.

Can I use asymmetric signatures instead of HMAC?

Yes, some providers use RSA or ECDSA signatures. You verify with their public key. HMAC is simpler but requires a shared secret.

When debugging webhook payloads, it's helpful to format the JSON for readability. Try our JSON Formatter to pretty-print and validate webhook payloads instantly.