Webhooks Explained: Signed Payloads and Retry Strategies

You've just integrated a third-party service that sends webhooks, and everything works fine in testing. But in production, you notice occasional missed events, duplicate deliveries, or worse—a malicious actor sending fake payloads to your endpoint. Webhooks are simple in concept but tricky in practice. This article explains how to secure webhooks with signed payloads and implement robust retry strategies so you can trust your event-driven integrations.

What Are Webhooks and Why They Matter

Webhooks are user-defined HTTP callbacks triggered by specific events. When an event occurs—like a payment success or a new user signup—the source service sends an HTTP POST request to a URL you provide. This push-based model is more efficient than polling, reducing latency and server load. However, it introduces challenges: verifying authenticity and handling delivery failures.

Securing Webhooks with Signed Payloads

Without verification, anyone can send requests to your webhook endpoint. Signed payloads solve this by including a cryptographic signature that proves the request originated from the trusted sender.

How HMAC Signatures Work

Most providers use HMAC (Hash-based Message Authentication Code) with a shared secret. The sender computes a hash of the payload using the secret and includes it in a header (e.g., X-Signature). Your server recomputes the hash and compares it. If they match, the request is authentic.

// Example: Verifying a webhook signature in Node.js
const crypto = require('crypto');

function verifySignature(payload, signature, secret) {
  const hmac = crypto.createHmac('sha256', secret);
  hmac.update(payload);
  const digest = hmac.digest('hex');
  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(digest));
}

Important: Always use a timing-safe comparison to prevent timing attacks. Also, verify the signature against the raw request body, not a parsed JSON object, because parsing can alter the payload.

Best Practices for Signature Verification

• Use HTTPS: Always serve webhooks over TLS to prevent eavesdropping.
• Rotate secrets: Periodically rotate your signing secret and support multiple active secrets during rotation.
• Validate timestamp: Include a timestamp in the signature to prevent replay attacks. Reject requests older than a few minutes.
• Log failures: Monitor signature verification failures to detect attacks or misconfigurations.

Implementing Reliable Retries

Even with signed payloads, network issues or temporary outages can cause webhook delivery to fail. A robust retry strategy ensures events are eventually delivered without overwhelming your system.

Exponential Backoff with Jitter

Retrying immediately can worsen congestion. Instead, use exponential backoff: wait longer between each retry. Add jitter (random variation) to avoid thundering herd problems when many webhooks fail simultaneously.

// Example: Retry schedule with exponential backoff and jitter
function getRetryDelay(attempt) {
  const base = 1000; // 1 second
  const max = 30000; // 30 seconds
  const delay = Math.min(base * Math.pow(2, attempt), max);
  return delay + Math.random() * 1000; // add up to 1s jitter
}

Idempotency: Handling Duplicates

Retries can cause duplicate deliveries. Your webhook handler must be idempotent—processing the same event multiple times should have the same effect as processing it once. Use the event ID from the payload to track processed events.

• Store processed event IDs in a database with a TTL.
• Before processing, check if the event ID already exists; if so, return a success response immediately.
• Design operations to be idempotent (e.g., use upserts instead of inserts).

Respond Quickly and Acknowledge

Your webhook endpoint should respond with a 2xx status code as quickly as possible—ideally within a few seconds. If processing takes longer, queue the event for asynchronous handling and return 202 Accepted. This prevents timeouts and unnecessary retries.

Comparison: Webhook vs. Polling

Step-by-Step: Building a Secure Webhook Receiver

1. Set up an HTTPS endpoint with a valid certificate.
2. Extract the signature header (e.g., X-Signature) and timestamp.
3. Verify the signature using the shared secret and raw body.
4. Check the timestamp to reject old requests (e.g., older than 5 minutes).
5. Parse the payload and check for duplicate event IDs.
6. Process the event asynchronously if needed, and respond with 200 OK.
7. Log the event for auditing and debugging.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request came from a trusted source and wasn't tampered with.

How many times should I retry a failed webhook?

Most systems retry 3–5 times with exponential backoff. After that, the event is typically marked as failed and may require manual intervention or dead-letter queue handling.

Can I use webhooks without HTTPS?

It's strongly discouraged. Without HTTPS, payloads and signatures can be intercepted or modified. Always use TLS for webhook endpoints.

When debugging webhook payloads or logs, formatting JSON correctly is essential. Use our JSON Formatter to pretty-print and validate JSON payloads quickly.