Webhooks Explained: Signed Payloads and Retry Strategies

You've just built a webhook endpoint to receive events from a third-party service. Everything works in testing, but in production you notice some events never arrive, or worse, you receive duplicate events causing data inconsistencies. These are classic webhook pitfalls that stem from two critical aspects: payload authenticity and delivery reliability.

Webhooks are HTTP callbacks triggered by events, pushing data to your application in real-time. Unlike polling, they're efficient and near-instant. But they introduce challenges: how do you know the request truly came from the expected sender, and how do you handle failures when the receiver is temporarily down? This article dives into signed payloads and retry mechanisms, giving you practical patterns to build resilient webhook integrations.

Why Webhooks Need Signatures

Webhook endpoints are publicly accessible URLs. Without verification, anyone can send forged requests to your endpoint, potentially triggering unauthorized actions or data corruption. Signed payloads solve this by adding a cryptographic signature to each request, allowing you to verify the sender's identity and ensure the payload hasn't been tampered with.

Most providers use HMAC (Hash-based Message Authentication Code) with a shared secret. The sender computes a hash of the payload using the secret and includes it in a header. You recompute the hash on your side and compare. If they match, the request is legitimate.

Implementing HMAC Verification

Let's walk through a typical implementation in Node.js. Suppose the provider sends a signature in the X-Signature header, computed as HMAC-SHA256 of the raw request body using your secret.

const crypto = require('crypto');
const express = require('express');
const app = express();

// Use raw body parser to preserve exact payload
app.use(express.raw({ type: 'application/json' }));

app.post('/webhook', (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  
  const expected = crypto
    .createHmac('sha256', secret)
    .update(req.body)
    .digest('hex');
  
  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }
  
  // Process the event
  const event = JSON.parse(req.body);
  // ... handle event
  
  res.status(200).send('OK');
});

Key points:

• Always use the raw request body for verification, not the parsed JSON. Parsing can alter whitespace or key order, breaking the hash.
• Use a timing-safe comparison to prevent timing attacks.
• Keep your secret secure; rotate it periodically and support multiple active secrets during rotation.

Designing a Retry Strategy

Even with signatures, deliveries can fail due to network issues, server downtime, or timeouts. A robust retry strategy ensures eventual delivery without overwhelming your system or the sender.

1. Acknowledge Quickly, Process Asynchronously

Your webhook endpoint should respond with a 2xx status as soon as possible, ideally within a few seconds. If processing takes longer, queue the event for background processing. This prevents timeouts and allows the sender to consider the delivery successful.

2. Implement Exponential Backoff with Jitter

When a delivery fails (non-2xx response or timeout), the sender should retry with increasing delays. Exponential backoff (e.g., 1s, 2s, 4s, 8s...) prevents hammering your endpoint. Adding random jitter avoids thundering herd problems when many webhooks fail simultaneously.

Example retry schedule:

• Attempt 1: immediate
• Attempt 2: after 30 seconds
• Attempt 3: after 2 minutes
• Attempt 4: after 10 minutes
• Attempt 5: after 1 hour

After a maximum number of retries (e.g., 5), the sender should stop and notify you (e.g., via email or dashboard).

3. Handle Duplicates with Idempotency

Retries can cause duplicate deliveries. Your endpoint must be idempotent: processing the same event multiple times should have the same effect as processing it once. Use a unique event ID from the payload and store it. Before processing, check if the ID has been seen; if so, return 200 without reprocessing.

Common approaches:

• Database unique constraint on event ID.
• Redis set with TTL for recently processed IDs.
• Idempotency keys in your business logic.

Comparison: Signed Payloads vs. Other Security Measures

HMAC signatures are the de facto standard because they're stateless, efficient, and provide strong security when implemented correctly.

Best Practices for Webhook Consumers

• Validate signatures on every request, even in development.
• Respond fast; offload heavy processing to a queue.
• Log everything—headers, payload, and processing outcome—for debugging.
• Monitor failures and set up alerts for repeated errors.
• Use HTTPS to protect payloads in transit.
• Rotate secrets regularly and support multiple active secrets during rotation.

Best Practices for Webhook Producers

• Sign payloads with a secret unique to each consumer.
• Include a unique event ID to aid idempotency.
• Retry with backoff and cap attempts.
• Provide a dashboard for consumers to view delivery logs and replay events.
• Document your signature scheme clearly.

Testing Your Webhook Implementation

Use tools like ngrok to expose your local server and simulate webhook deliveries. Write unit tests for signature verification and idempotency logic. For load testing, consider tools that can send signed requests at scale.

When debugging, inspect the raw payload and headers. A JSON formatter can help visualize the payload structure. Try our JSON Formatter to pretty-print and validate webhook payloads quickly.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the sender's identity and ensure the payload hasn't been altered. It's computed using a shared secret and the request body.

How many times should a webhook be retried?

There's no universal number, but common practice is 3–5 retries with exponential backoff. After that, the event is considered failed and the consumer should be notified to investigate or replay manually.

Can I use JWT instead of HMAC for webhook signatures?

Yes, JWT can work, but it adds complexity (token issuance, expiry). HMAC is simpler and sufficient for most use cases. Choose based on your existing infrastructure and requirements.

Ready to inspect and debug webhook payloads? Use our free JSON Formatter to format and validate JSON instantly.