Webhooks Explained: Signed Payloads and Retry Strategies

You've just integrated a payment gateway or a CI/CD service, and now you need to receive real-time events. You set up an endpoint, but soon you face two critical questions: How do you ensure the request actually came from the service? And what happens if your server is down when an event fires? This article answers both by explaining signed payloads and retry strategies—the backbone of reliable webhooks.

What Are Webhooks?

Webhooks are user-defined HTTP callbacks. When an event occurs in a source system (e.g., a new commit, a payment success), it sends an HTTP POST request to a URL you provide. Unlike polling, webhooks push data in near real-time, reducing latency and server load. However, this push model introduces challenges: verifying authenticity and handling delivery failures.

Why Signed Payloads Matter

Anyone can send an HTTP request to your endpoint. Without verification, attackers could forge events, leading to unauthorized actions. Signed payloads solve this by including a cryptographic signature computed over the request body using a shared secret. The receiver recomputes the signature and compares it, ensuring the payload is untampered and originates from the expected sender.

How HMAC Signatures Work

Most services use HMAC-SHA256. The sender creates a signature by hashing the payload with a secret key and includes it in a header (e.g., X-Signature). The receiver does the same and compares using a constant-time function to prevent timing attacks.

Step-by-Step: Verifying a Signed Payload

1. Extract the signature from the header (e.g., X-Signature).
2. Get the raw request body as a string or buffer—do not parse it first, as formatting changes break the signature.
3. Compute the expected signature using HMAC-SHA256 with your secret key.
4. Compare signatures using a constant-time comparison function.
5. Reject the request with HTTP 401 if they don't match.

Here's a Node.js example using Express:

const crypto = require('crypto');
const express = require('express');
const app = express();

app.post('/webhook', express.raw({type: 'application/json'}), (req, res) => {
  const signature = req.headers['x-signature'];
  const secret = process.env.WEBHOOK_SECRET;
  const payload = req.body; // raw buffer

  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');

  if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
    return res.status(401).send('Invalid signature');
  }

  // Process the event
  const event = JSON.parse(payload);
  // ... handle event
  res.status(200).send('OK');
});

Note: Always use the raw body for verification. Middleware that parses JSON will alter the payload, causing signature mismatches.

Designing a Retry Strategy

Even with signed payloads, network issues or server downtime can cause delivery failures. A robust retry strategy ensures events aren't lost. Key considerations:

1. Acknowledge Quickly

Respond with a 2xx status as soon as you've validated the signature and queued the event for processing. This prevents the sender from timing out and retrying unnecessarily.

2. Idempotency

Retries may deliver the same event multiple times. Use an idempotency key (often included in the payload or headers) to deduplicate. Store processed event IDs and ignore duplicates.

3. Exponential Backoff with Jitter

If your endpoint returns an error (non-2xx), the sender should retry with increasing delays. Exponential backoff (e.g., 1s, 2s, 4s, 8s) reduces load during outages. Adding random jitter prevents thundering herd problems.

Typical retry schedule:

4. Dead Letter Queue

After maximum retries, move the event to a dead letter queue for manual inspection. This prevents endless retries and data loss.

5. Monitoring and Alerts

Track delivery success rates, latency, and failure reasons. Set up alerts for spikes in failures to detect issues early.

Common Pitfalls and Best Practices

• Always verify signatures—never trust incoming data blindly.
• Use HTTPS to protect payloads in transit.
• Rotate secrets periodically and support multiple active secrets during rotation.
• Log webhook events (with sensitive data redacted) for debugging.
• Test with sample payloads using tools that simulate webhook deliveries.

FAQ

What is a signed payload in webhooks?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request's authenticity and integrity.

How many times should a webhook be retried?

There's no fixed number; common practices range from 3 to 10 retries with exponential backoff. After that, events are often sent to a dead letter queue.

Can I use webhooks without signatures?

It's possible but highly discouraged. Without signatures, your endpoint is vulnerable to forged requests and data breaches.

Ready to test your webhook payloads? Use our JSON Formatter to inspect and validate JSON payloads quickly.