Webhooks Explained: Signed Payloads and Retry Logic

You've just integrated a payment gateway. The docs say: "We'll send a webhook when a payment succeeds." Great. But then you wonder: how do I know the request really came from them? What if my server is down when they send it? What if they send the same event twice?

Webhooks are powerful but come with pitfalls. This guide walks through the mechanics, security, and reliability patterns you need to handle them correctly.

What Is a Webhook?

A webhook is an HTTP POST request that one system sends to another when a specific event occurs. Instead of your app polling an API every few seconds ("Any new orders? Any new orders?"), the provider pushes data to you as soon as something happens.

Typical flow:

1. You register a URL (e.g., https://yourapp.com/webhooks/payments) with the provider.
2. An event occurs (payment succeeded, user signed up, file uploaded).
3. The provider sends a POST request to your URL with a JSON body describing the event.
4. Your server processes the event and responds with 2xx to acknowledge receipt.

If you don't respond quickly (usually within 5–30 seconds), the provider considers the delivery failed and may retry.

Why You Must Verify Signed Payloads

Your webhook endpoint is publicly accessible. Anyone who discovers the URL can send fake events. Without verification, an attacker could forge a "payment succeeded" event and get free goods.

The solution: signed payloads. The provider computes a cryptographic signature over the request body using a shared secret (known only to you and them) and includes it in a header. You recompute the signature on your end and compare.

How HMAC Signing Works

Most providers use HMAC-SHA256. The steps:

1. Provider concatenates the raw request body with a timestamp (to prevent replay attacks).
2. Provider computes HMAC-SHA256(secret, timestamp + '.' + body).
3. Provider sends the signature in a header, e.g., X-Signature: t=1234567890,v1=abc123....
4. You extract the timestamp and signature, recompute the HMAC using your copy of the secret, and compare using a constant-time comparison function.

Here's a minimal Node.js example:

const crypto = require('crypto');

function verifySignature(req, secret) {
  const signatureHeader = req.headers['x-signature'];
  const [tPart, vPart] = signatureHeader.split(',');
  const timestamp = tPart.split('=')[1];
  const receivedSig = vPart.split('=')[1];

  const payload = timestamp + '.' + JSON.stringify(req.body);
  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  // Constant-time comparison to prevent timing attacks
  return crypto.timingSafeEqual(
    Buffer.from(receivedSig, 'hex'),
    Buffer.from(expectedSig, 'hex')
  );
}

Important: Always use the raw request body for signature verification. If your framework parses JSON and re-serializes it, the byte sequence may differ, causing verification to fail. Capture the raw body before parsing.

Timestamp Validation

Check that the timestamp is recent (e.g., within 5 minutes). This prevents replay attacks where an attacker captures a valid request and resends it later.

Retry Logic: Handling Failures Gracefully

Your server might be down, overloaded, or experiencing a network blip. Providers handle this by retrying failed deliveries. But you must design your endpoint to handle retries safely.

Exponential Backoff with Jitter

Providers typically retry with increasing delays: 1 minute, 5 minutes, 30 minutes, 2 hours, etc. This is called exponential backoff. Some add random jitter to avoid thundering herd problems.

From your side, you don't control the provider's retry schedule, but you should:

• Respond with 2xx as soon as you've safely stored the event (even if processing is async).
• Return 4xx for permanent failures (e.g., malformed payload) so the provider stops retrying.
• Return 5xx for transient failures (e.g., database down) to trigger retries.

Idempotency: The Key to Safe Retries

Retries mean you might receive the same event multiple times. Your processing must be idempotent—processing the same event twice should have the same effect as processing it once.

Common approach: include a unique event ID in the payload. Before processing, check if you've already handled that ID. If yes, return 200 and skip.

async function handleWebhook(event) {
  const existing = await db.events.findOne({ id: event.id });
  if (existing) {
    return { status: 200, body: 'Already processed' };
  }

  await db.events.insert({ id: event.id, processedAt: new Date() });
  await processEvent(event);
  return { status: 200, body: 'OK' };
}

This pattern ensures that even if the provider sends the event three times, you only act once.

Comparison: Polling vs Webhooks

Webhooks win for real-time updates, but polling can be a fallback for critical events if you're worried about missed deliveries.

Best Practices Checklist

• Always verify signatures. Never trust an unverified webhook.
• Use HTTPS. Webhook payloads often contain sensitive data.
• Respond fast. Acknowledge receipt quickly, then process asynchronously if needed.
• Implement idempotency. Store event IDs to avoid duplicate processing.
• Log everything. Keep a record of received events for debugging and auditing.
• Monitor failures. Set up alerts for repeated 5xx responses or signature verification failures.
• Test with real payloads. Use the provider's test mode or send sample events to your endpoint.

Debugging Webhook Issues

When webhooks fail, the first step is to inspect the raw request. Common problems:

• Signature mismatch: You're using the parsed body instead of raw. Fix by capturing raw body.
• Timeout: Your endpoint takes too long to respond. Move heavy processing to a background job.
• Missing events: Check the provider's dashboard for delivery logs and retry history.

If you're debugging HTTP traffic, tools like Nginx Log Analyzer can help you spot patterns in request failures and response codes.

FAQ

What is a webhook secret and how do I store it?

The secret is a shared key used to sign payloads. Store it in environment variables or a secrets manager—never in code or version control. Rotate it periodically if the provider supports it.

How many times will a provider retry a failed webhook?

It varies by provider. Some retry for 24 hours with exponential backoff, others for 3 days. Check your provider's documentation for the exact schedule and maximum attempts.

Can I use webhooks with serverless functions?

Yes, but be mindful of execution time limits. Acknowledge the webhook quickly (return 200) and offload heavy processing to a queue or background function to avoid timeouts.

Ready to build robust webhook integrations? Start by verifying signatures and implementing idempotency—these two steps will save you hours of debugging and prevent costly security incidents.