Webhooks Explained: Signed Payloads and Retries

You've just integrated a payment gateway. The docs say: "We'll send a POST request to your endpoint when a payment succeeds." Easy, right? You build a simple route that accepts JSON, updates your database, and returns 200. Then it happens: a customer's payment succeeds, but your server is briefly down. The webhook never arrives. Or worse, it arrives twice, and you double-credit the account. Webhooks seem simple until they fail.

This article explains how webhooks really work, why signed payloads are essential for security, and how to build retry logic that keeps your integration reliable—even when things go wrong.

What Is a Webhook?

A webhook is an HTTP callback: instead of your application polling an API for changes, the provider sends an HTTP POST to a URL you specify when an event occurs. It's push instead of pull.

Common examples include payment confirmations (Stripe, PayPal), CI/CD build notifications (GitHub, GitLab), and messaging platforms (Slack, Discord). The pattern is simple: you register a URL, the provider sends JSON payloads to that URL, and your server processes them.

But this simplicity hides two critical challenges: authenticity (how do you know the request really came from the provider?) and reliability (what happens if your endpoint is down or slow?).

Why Signed Payloads Matter

Your webhook endpoint is a public URL. Anyone can send a POST request to it. Without verification, an attacker could forge a payment success event, trigger a shipment, or corrupt your data.

Signed payloads solve this by including a cryptographic signature in the request headers. The provider computes an HMAC (hash-based message authentication code) using a shared secret and the request body, then sends it alongside the payload. Your server recomputes the HMAC and compares it. If they match, the request is authentic.

How HMAC Signing Works

1. The provider and you share a secret key (provided during setup).
2. For each webhook, the provider concatenates the timestamp and the raw request body, then computes an HMAC-SHA256 hash using the secret.
3. The signature is sent in a header, often something like X-Signature or Stripe-Signature.
4. Your server recomputes the HMAC using the same secret and compares it to the received signature.
5. If they match, you process the event; otherwise, you reject it with a 401.

Always use the raw request body for HMAC computation. If your framework parses JSON before you verify, the body may be altered (whitespace, key order), causing verification to fail.

Example: Verifying a Webhook Signature in Node.js

const crypto = require('crypto');
const express = require('express');
const app = express();

// Use raw body for signature verification
app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const payload = req.body; // Buffer

  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }

  // Process the event
  const event = JSON.parse(payload.toString());
  // ... handle event

  res.status(200).send('OK');
});

Note the use of timingSafeEqual to prevent timing attacks. Never use a simple === comparison for signatures.

Designing Retry Logic

Even with signatures, webhooks can fail. Your server might be down, the network might drop the request, or your processing might time out. Providers typically retry failed deliveries, but you should also design your system to handle retries gracefully.

What Counts as a Failure?

Most providers consider any response other than 2xx as a failure. This includes 3xx redirects, 4xx client errors, and 5xx server errors. Some also enforce a timeout (e.g., 5 seconds). If you don't respond quickly, the delivery is marked as failed and retried.

Exponential Backoff with Jitter

Providers usually retry with exponential backoff: first retry after 1 minute, then 5 minutes, 30 minutes, etc. Adding jitter (random variation) prevents thundering herds if many webhooks fail simultaneously.

If you're building a webhook sender, implement backoff like this:

• Retry 1: after 10 seconds
• Retry 2: after 1 minute
• Retry 3: after 5 minutes
• Retry 4: after 30 minutes
• Retry 5: after 2 hours

After a maximum number of attempts, move the event to a dead-letter queue for manual inspection.

Idempotency: Handling Duplicate Deliveries

Retries mean your endpoint may receive the same event multiple times. Your processing must be idempotent: processing the same event twice should have the same effect as processing it once.

Common strategies:

• Event ID deduplication: Store processed event IDs in a database or cache. Before processing, check if the ID has been seen.
• Upsert instead of insert: Use database operations that naturally handle duplicates (e.g., INSERT ... ON CONFLICT DO NOTHING).
• Idempotency keys: For operations like charging a card, pass a unique key so the payment provider ignores duplicate requests.

Best Practices for Webhook Consumers

Also consider validating the timestamp in the signature to prevent replay attacks. Many providers include a timestamp in the signed payload; reject events older than a few minutes.

Best Practices for Webhook Producers

If you're building a service that sends webhooks, follow these guidelines:

• Sign every payload with HMAC-SHA256.
• Include a unique event ID and timestamp.
• Retry with exponential backoff and jitter.
• Provide a way for consumers to view delivery logs and manually replay events.
• Document your signature verification process clearly.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) computed from the request body and a shared secret. The receiver verifies the signature to ensure the request is authentic and unmodified.

How many times should a webhook be retried?

There's no universal number, but most providers retry 3–5 times over several hours. After that, the event is typically marked as failed and may be available for manual replay. Your system should handle duplicates idempotently regardless of retry count.

Can I use webhooks without HTTPS?

Technically yes, but you should always use HTTPS. Webhooks often contain sensitive data, and without TLS, an attacker could intercept or tamper with the payload. HTTPS also ensures you're sending to the intended server.

Debugging Webhooks with Nginx Logs

When webhooks fail, the first place to look is your web server logs. If you're using Nginx, the access and error logs record every request, including status codes and response times. You can quickly spot patterns: repeated 401s (signature verification failing), 500s (application errors), or timeouts.

For a faster analysis, try the Nginx Log Analyzer to parse and summarize webhook traffic, filter by status code, and identify problematic endpoints.