Webhooks Explained: Signed Payloads and Retries

Why Webhooks Are Tricky

Webhooks are a simple idea: instead of polling an API for changes, the provider pushes events to your endpoint via HTTP POST. But in practice, webhooks introduce a set of challenges that trip up many developers: verifying that the request actually came from the provider, handling duplicate deliveries, and dealing with network failures. If you ignore these, you risk security breaches, data corruption, and angry users.

This article walks through the two most critical aspects of production-grade webhooks: signed payloads and retries. We’ll cover how to verify signatures, how to implement retries with exponential backoff, and how to make your processing idempotent.

Signed Payloads: Proving Authenticity

Anyone can send an HTTP POST to your endpoint. To ensure the request came from your provider, the provider signs the payload with a secret key and sends the signature in a header. Your job is to recompute the signature and compare it.

Most providers use HMAC-SHA256. The signature is typically computed over the raw request body (sometimes with a timestamp) and hex-encoded. For example, Stripe uses a header like Stripe-Signature containing a timestamp and signature.

How HMAC Works

HMAC (Hash-based Message Authentication Code) combines a secret key with a hash function. Given the same key and message, the output is deterministic. Without the key, an attacker cannot forge a valid signature.

Here’s a minimal example in Node.js:

const crypto = require('crypto');

function verifySignature(payload, signature, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('hex');
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}

In Python, it’s equally straightforward:

import hmac
import hashlib

def verify_signature(payload: bytes, signature: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)

Best Practices for Signature Verification

• Use constant-time comparison to prevent timing attacks. Both crypto.timingSafeEqual and hmac.compare_digest are safe.
• Verify the raw body, not a parsed JSON object. Parsing and re-serializing can change whitespace and break the signature.
• Include a timestamp in the signed data to prevent replay attacks. Reject requests older than a few minutes.
• Store secrets securely—use environment variables or a secrets manager, never hardcode them.

Handling Timestamps and Replay Attacks

Many providers include a timestamp in the signature header. For example, the header might be t=1614556800,v1=abc123.... You should:

1. Extract the timestamp and signature.
2. Check that the timestamp is within an acceptable window (e.g., 5 minutes).
3. Recompute the signature using the timestamp and payload.
4. Compare signatures securely.

This prevents an attacker from replaying an old valid request.

Retries: Handling Failures Gracefully

Webhooks are delivered over the public internet, so failures are inevitable. Your endpoint might be down, slow, or return a 500 error. A robust provider will retry failed deliveries, but you also need to handle retries on your side if you are the provider.

Exponential Backoff with Jitter

When retrying, avoid hammering the endpoint with immediate retries. Use exponential backoff: wait longer after each failure. Add jitter (randomness) to avoid thundering herd problems when many webhooks fail simultaneously.

A typical retry schedule might look like:

After a maximum number of attempts, mark the delivery as failed and alert.

Idempotency: The Key to Safe Retries

Because retries can cause duplicate deliveries, your webhook handler must be idempotent. That means processing the same event multiple times should have the same effect as processing it once.

How to achieve idempotency:

• Use a unique event ID provided in the payload (e.g., id field). Store processed IDs in a database with a unique constraint.
• Design operations to be idempotent—for example, use upserts instead of inserts, or check if a resource already exists before creating it.
• Return 2xx quickly and process asynchronously if the work is heavy. This reduces the chance of timeouts that trigger retries.

Implementing Retries on the Provider Side

If you are building a webhook system, you need a reliable delivery mechanism. Here’s a simplified approach:

1. When an event occurs, enqueue a delivery job with the payload and target URL.
2. A worker picks up the job and sends the HTTP POST with a timeout (e.g., 10 seconds).
3. If the response status is not 2xx, schedule a retry with exponential backoff.
4. Track attempts and give up after a maximum (e.g., 5 attempts).
5. Log failures and provide a way for users to manually replay.

Use a persistent queue (e.g., Redis, RabbitMQ, or a database table) so that retries survive restarts.

Putting It All Together: A Webhook Receiver Checklist

When building a webhook receiver, follow this checklist:

• Verify the signature using the provider’s secret and constant-time comparison.
• Check the timestamp to prevent replay attacks.
• Respond with 2xx quickly—acknowledge receipt before doing heavy processing.
• Process asynchronously if needed, using a queue.
• Ensure idempotency by tracking event IDs.
• Log everything for debugging and auditing.
• Monitor failures and set up alerts for repeated retries.

FAQ

What is a signed webhook payload?

A signed payload includes a cryptographic signature (usually HMAC-SHA256) generated with a shared secret. The receiver recomputes the signature to verify the request came from the expected sender and wasn’t tampered with.

Why do webhooks need retries?

Network issues, server downtime, or temporary errors can cause a webhook delivery to fail. Retries with exponential backoff ensure the event is eventually delivered without overwhelming the receiver.

How do I prevent duplicate webhook processing?

Use idempotency: include a unique event ID in the payload, store processed IDs, and design your operations to be idempotent (e.g., upserts). This way, even if a webhook is delivered multiple times, the effect is the same.

If you need to inspect or debug webhook payloads, the JSON Formatter can help you pretty-print and validate JSON data quickly.