Webhooks Explained: Signed Payloads and Retry Strategies

You've built an API and now need to notify other services when events happen. Polling is inefficient, so you implement webhooks. But soon you face two common problems: how to ensure the payload hasn't been tampered with, and what to do when the receiving endpoint is down. This article explains how to use signed payloads and retry strategies to make your webhooks secure and reliable.

Why Webhooks Need Signatures and Retries

Webhooks are HTTP callbacks triggered by events. They are simple to implement but introduce security and reliability challenges. Without signatures, an attacker could forge requests to your endpoint, leading to data corruption or unauthorized actions. Without retries, transient failures (like network blips or temporary server errors) cause missed events.

Signed payloads solve authenticity: the sender signs the payload with a secret, and the receiver verifies it. Retries solve reliability: the sender retries failed deliveries with backoff.

How Signed Payloads Work

A signed payload includes a cryptographic signature generated from the request body and a shared secret. The most common method is HMAC (Hash-based Message Authentication Code) with SHA-256.

Generating the Signature

The sender computes the HMAC of the raw request body using the secret key. The signature is sent in a header, often along with a timestamp to prevent replay attacks.

// Node.js example: generating HMAC signature
const crypto = require('crypto');

function signPayload(payload, secret) {
  const timestamp = Math.floor(Date.now() / 1000);
  const signedPayload = `${timestamp}.${JSON.stringify(payload)}`;
  const signature = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');
  return { timestamp, signature };
}

Include the timestamp and signature in headers, e.g., X-Webhook-Timestamp and X-Webhook-Signature.

Verifying the Signature

The receiver must reconstruct the signed payload using the received timestamp and raw body, then compute the HMAC and compare it securely.

// Node.js example: verifying signature
function verifySignature(rawBody, timestamp, receivedSignature, secret) {
  const signedPayload = `${timestamp}.${rawBody}`;
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');
  // Use timing-safe comparison
  return crypto.timingSafeEqual(
    Buffer.from(expectedSignature, 'hex'),
    Buffer.from(receivedSignature, 'hex')
  );
}

Always use a timing-safe comparison to prevent timing attacks. Also, check that the timestamp is recent (e.g., within 5 minutes) to mitigate replay attacks.

Designing a Retry Strategy

Retries are essential for reliable webhook delivery. But naive retries can overwhelm the receiver or cause duplicate processing. A good retry strategy includes:

• Exponential backoff: Wait longer between each retry (e.g., 1s, 2s, 4s, 8s...).
• Maximum retries: Limit the number of attempts (e.g., 5) to avoid infinite loops.
• Dead-letter queue: After max retries, store the event for manual inspection.
• Idempotency: Include a unique event ID so the receiver can deduplicate.

Here's a simple retry loop in pseudocode:

function deliverWithRetry(url, payload, maxRetries = 5) {
  let attempt = 0;
  while (attempt < maxRetries) {
    try {
      const response = await fetch(url, { method: 'POST', body: payload });
      if (response.ok) return;
    } catch (error) {
      // Log error
    }
    attempt++;
    const delay = Math.pow(2, attempt) * 1000; // exponential backoff
    await new Promise(resolve => setTimeout(resolve, delay));
  }
  // Move to dead-letter queue
}

Consider adding jitter to the backoff to avoid thundering herd problems.

Comparison: Polling vs Webhooks with Retries

Step-by-Step: Implementing Webhooks with Signatures and Retries

1. Generate a secret key for each webhook endpoint and share it securely.
2. On event occurrence, construct the payload (JSON) and compute the signature using HMAC-SHA256 with the secret and a timestamp.
3. Send the POST request to the endpoint with headers: Content-Type: application/json, X-Webhook-Timestamp, and X-Webhook-Signature.
4. On the receiver side, verify the signature using the raw body and timestamp. Reject if invalid or timestamp is too old.
5. If delivery fails (non-2xx response or network error), retry with exponential backoff up to a maximum number of attempts.
6. After max retries, log the failure and optionally alert; store the event in a dead-letter queue for manual replay.
7. Ensure idempotency on the receiver by using an event ID to ignore duplicates.

Common Pitfalls and Best Practices

• Not using raw body: Many frameworks parse JSON and alter the body. Use the raw body for signature verification.
• Ignoring timestamps: Without timestamp validation, replay attacks are possible.
• No idempotency: Retries can cause duplicate processing. Use idempotency keys.
• Infinite retries: Always set a max retry limit and a dead-letter queue.
• Not logging: Log delivery attempts and failures for debugging.

FAQ

What is a signed webhook payload?

A signed webhook payload includes a cryptographic signature (e.g., HMAC) that allows the receiver to verify the request originated from the expected sender and wasn't tampered with.

How many times should I retry a failed webhook?

A common practice is 3–5 retries with exponential backoff. The exact number depends on your SLA and the criticality of the event. Always use a dead-letter queue for events that exceed the retry limit.

Can I use webhooks without signatures?

It's not recommended. Without signatures, your endpoint is vulnerable to forged requests. Always use signatures (like HMAC) and validate timestamps to prevent replay attacks.

When debugging webhook deliveries, inspecting server logs is crucial. If you're using Nginx, the Nginx Log Analyzer can help you parse and analyze access logs to trace webhook requests and identify failures.