Webhooks Explained: Signed Payloads and Retries

You've just integrated a payment gateway. The docs say: "We'll send a webhook to your endpoint when a payment succeeds." You set up a simple route, test it locally, and deploy. Then, in production, you notice duplicate events, missed notifications, and mysterious 500 errors. Webhooks seem simple—until they aren't.

This guide explains how webhooks really work, why signed payloads matter, and how to build retry logic that keeps your system in sync even when things go wrong.

What Is a Webhook?

A webhook is an HTTP callback: one system sends an HTTP POST request to a URL you provide when a specific event occurs. Unlike polling, where you repeatedly ask "Did anything happen?", webhooks push events to you in near real-time.

Common uses include:

• Payment processors notifying you of successful charges.
• Git hosting services triggering CI/CD pipelines on push.
• Communication platforms alerting you to new messages.

The sender (e.g., Stripe, GitHub) is the webhook provider; your server is the webhook consumer.

Why Signed Payloads Are Essential

Your webhook endpoint is publicly accessible. Anyone who knows the URL can send fake events. Without verification, an attacker could spoof a "payment succeeded" event and you might ship goods for free.

Signed payloads solve this by including a cryptographic signature in the request headers, generated with a shared secret known only to you and the provider. When you receive a webhook, you recompute the signature and compare. If they match, the payload is authentic.

How HMAC Signatures Work

Most providers use HMAC-SHA256. The provider computes:

signature = HMAC-SHA256(secret, payload)

and sends it in a header like X-Signature. Your server does the same calculation and checks equality. Because the secret is secret, only the provider can generate a valid signature.

Verifying a Signature in Practice

Here's a minimal Node.js example using Express:

const crypto = require('crypto');
const express = require('express');
const app = express();

app.post('/webhook', express.raw({type: 'application/json'}), (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const payload = req.body; // raw Buffer

  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (signature !== expected) {
    return res.status(401).send('Invalid signature');
  }

  // Process the event
  const event = JSON.parse(payload);
  console.log('Received event:', event.type);
  res.status(200).send('OK');
});

app.listen(3000);

Important: Always use the raw request body for signature verification. If you parse JSON first, the serialized string may differ (whitespace, key order) and the signature check will fail.

Common Pitfalls

• Using the wrong secret: Some providers have separate secrets for test and live modes.
• Timing attacks: Use a constant-time comparison function (e.g., crypto.timingSafeEqual) to avoid leaking information.
• Ignoring timestamp headers: Many providers include a timestamp to prevent replay attacks. Reject requests older than a few minutes.

Retries: Handling Failures Gracefully

Networks fail, servers restart, and deployments happen. Webhook providers expect that your endpoint might be temporarily unavailable, so they retry failed deliveries. But retries introduce new challenges: duplicate events, out-of-order delivery, and infinite loops if you're not careful.

How Providers Retry

Most providers use exponential backoff: they wait a short time after the first failure, then progressively longer intervals. For example, Stripe retries up to 3 days with increasing delays. GitHub retries up to 8 times. The exact schedule varies, but the pattern is consistent.

Your endpoint should return a 2xx status code quickly to acknowledge receipt. Any other status (4xx, 5xx) signals failure and triggers a retry.

Building Idempotent Handlers

Because retries can deliver the same event multiple times, your handler must be idempotent: processing the same event twice should have the same effect as processing it once.

Common strategies:

• Event ID tracking: Store the unique event ID (provided in the payload or headers) in a database. Before processing, check if it's already been handled.
• Upserts: Use database operations that are naturally idempotent (e.g., INSERT ... ON CONFLICT DO NOTHING).
• State machines: Design your business logic to tolerate repeated transitions (e.g., "mark order as paid" only if not already paid).

Example using a simple in-memory set (for illustration; use a persistent store in production):

const processedEvents = new Set();

app.post('/webhook', (req, res) => {
  const event = req.body;
  if (processedEvents.has(event.id)) {
    return res.status(200).send('Already processed');
  }
  // Process event...
  processedEvents.add(event.id);
  res.status(200).send('OK');
});

Responding Quickly

Providers often have a timeout (e.g., 5 seconds). If your handler does heavy processing, acknowledge immediately and process asynchronously. For example, push the event to a queue and return 200.

Best Practices for Reliable Webhooks

1. Always verify signatures using the raw body and a constant-time comparison.
2. Use HTTPS for your endpoint to prevent eavesdropping and tampering.
3. Implement idempotency to handle duplicate deliveries.
4. Log everything — incoming payloads, signatures, and your responses — for debugging.
5. Monitor failures and set up alerts for repeated errors.
6. Test with real events using provider dashboards or CLI tools.
7. Handle out-of-order events by including timestamps or sequence numbers in your logic.

Comparison: Polling vs. Webhooks

FAQ

What is a webhook signature?

A webhook signature is a cryptographic hash (usually HMAC-SHA256) of the payload, generated with a shared secret. It allows the receiver to verify that the request came from the expected sender and wasn't tampered with.

How do I handle duplicate webhook events?

Make your handler idempotent by tracking event IDs or using database operations that are naturally idempotent. Check if an event has already been processed before applying changes.

What should my webhook endpoint return?

Return a 2xx status code (e.g., 200 OK) as soon as you've received and validated the event. If you need to do heavy processing, acknowledge immediately and handle the work asynchronously.

Ready to test your webhook payloads? Use our JSON Formatter to inspect and validate the JSON structure of incoming events.