Webhooks Explained: Signed Payloads and Retries for Reliable Integrations
You've just built an endpoint to receive webhooks from a payment provider. The first test works fine, but in production you notice missing events, duplicate charges, and no way to tell if a request is genuine. Webhooks are simple in concept but tricky to get right. This guide explains how to secure webhooks with signed payloads and implement robust retry logic so your integrations don't break silently.
What Are Webhooks and Why They Matter
Webhooks are user-defined HTTP callbacks triggered by events. Instead of your application polling an API for changes, the service pushes data to your endpoint when something happens. This makes integrations efficient and near real-time.
Common use cases include payment notifications (e.g., Stripe), CI/CD triggers (e.g., GitHub), and messaging platforms (e.g., Slack). But with this push model, you lose control over when requests arrive. That's why you need to handle security and reliability explicitly.
The Security Problem: Unsigned Webhooks Are Dangerous
An unsigned webhook is just an HTTP POST to a public URL. Anyone can send a request that looks like it came from your payment provider. Without verification, attackers could forge events, leading to fraudulent orders, data leaks, or unauthorized actions.
The solution is to sign the payload. The sender computes a cryptographic signature over the request body using a shared secret, and includes it in a header. Your endpoint recomputes the signature and compares it. If they match, you know the request is authentic and hasn't been tampered with.
How to Sign Webhook Payloads with HMAC
HMAC (Hash-based Message Authentication Code) is the standard for webhook signing. It uses a secret key and a hash function (like SHA-256) to produce a signature that is unique to the payload and secret.
Here's how a sender might sign a payload in Python:
import hmac
import hashlib
import json
secret = b'your_shared_secret'
payload = json.dumps({'event': 'payment.success', 'amount': 1000}).encode('utf-8')
signature = hmac.new(secret, payload, hashlib.sha256).hexdigest()
# Include signature in header: X-Signature:
On the receiving side, you must verify the signature before processing the payload:
import hmac
import hashlib
def verify_signature(payload, received_signature, secret):
expected = hmac.new(secret, payload, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, received_signature)
Important: Always use a constant-time comparison function like hmac.compare_digest to prevent timing attacks. Also, verify the signature against the raw request body, not a parsed version, because JSON serialization can differ.
Best Practices for Signed Payloads
- Use a strong secret: Generate a random secret of at least 32 bytes and store it securely (e.g., environment variable or secrets manager).
- Include a timestamp: Add a timestamp to the signed payload and reject requests older than a few minutes to prevent replay attacks.
- Use HTTPS: Always transmit webhooks over TLS to protect the payload and signature in transit.
- Rotate secrets periodically: Support multiple active secrets during rotation to avoid downtime.
Handling Failures: Retries and Idempotency
Webhooks can fail: your server might be down, the network might drop, or your endpoint might return an error. A reliable sender will retry delivery, but you must design your receiver to handle duplicates and out-of-order events.
Retry Strategies
Most webhook providers use exponential backoff with jitter. For example, retry after 1 minute, then 5 minutes, then 30 minutes, up to a maximum number of attempts. This gives your system time to recover without overwhelming it.
If you're building the sender side, implement a queue with scheduled retries. Store the payload and attempt count, and use a worker to process retries.
Idempotency: The Key to Safe Retries
Because retries can cause duplicate deliveries, your endpoint must be idempotent. That means processing the same event multiple times should have the same effect as processing it once.
Common approaches:
- Use a unique event ID: Include an ID in the payload and store processed IDs in a database. Ignore events with IDs you've already seen.
- Design operations to be idempotent: For example, use upserts instead of inserts, or check if a resource already exists before creating it.
Here's a simple idempotency check in Python:
def process_event(event_id, payload):
if db.exists('processed_events', event_id):
return # already processed
# process payload
db.insert('processed_events', event_id)
Ensure the check and the processing happen in a single transaction to avoid race conditions.
Comparison: Polling vs. Webhooks
| Aspect | Polling | Webhooks |
|---|---|---|
| Latency | Depends on interval (e.g., 5 min) | Near real-time |
| Server load | Constant requests even when no changes | Requests only on events |
| Complexity | Simple to implement | Requires signing, retries, idempotency |
| Reliability | You control retries | Depends on sender's retry policy |
Step-by-Step: Implementing a Secure Webhook Receiver
- Set up an HTTPS endpoint that accepts POST requests and reads the raw body.
- Extract the signature header (e.g.,
X-Signature) and the timestamp header if present. - Verify the signature using HMAC and your secret. Reject if invalid.
- Check the timestamp to prevent replay attacks (e.g., reject if older than 5 minutes).
- Parse the payload and extract the event ID.
- Check idempotency by looking up the event ID in your database.
- Process the event within a transaction that also records the event ID.
- Return a 2xx status quickly to acknowledge receipt. If processing takes time, enqueue the job and return 202 Accepted.
Testing Your Webhook Implementation
Use tools like ngrok to expose your local server and simulate webhooks. Write unit tests for signature verification, idempotency, and retry logic. For example, test that a duplicate event ID is ignored and that an invalid signature returns 401.
When debugging, log the received payload and signature (but never log secrets). If you're working with JSON payloads, a JSON formatter can help you inspect and validate the structure quickly.
FAQ
What is a signed webhook payload?
A signed payload includes a cryptographic signature (usually HMAC) computed over the request body with a shared secret. The receiver verifies the signature to ensure the request is authentic and unaltered.
How many times should a webhook be retried?
There's no universal number, but common practice is 3–5 retries with exponential backoff. After that, the event is often marked as failed and may require manual intervention or a dead-letter queue.
Can I use webhooks without HTTPS?
You should never use webhooks over plain HTTP. Without TLS, attackers can intercept the payload and signature, and replay or modify requests. Always use HTTPS.
Ready to inspect and format webhook payloads? Try our free JSON formatter to validate and beautify JSON data instantly.