Webhooks Explained: Signed Payloads and Retries
Why Webhooks Break in Production
You've built a webhook endpoint, tested it locally, and everything works. Then production hits: a provider sends a duplicate event, your endpoint times out, or an attacker spoofs a request. Suddenly, your integration is unreliable and insecure.
Webhooks are simple in concept—HTTP callbacks triggered by events—but production-grade implementations require careful handling of security and reliability. This guide covers two critical aspects: signed payloads to verify authenticity and retry strategies to handle failures.
What Are Webhooks?
A webhook is a user-defined HTTP callback. When an event occurs in a source system (e.g., a payment succeeds), it sends an HTTP POST request to a URL you provide, containing data about the event. This push model is more efficient than polling.
But webhooks introduce challenges:
- Security: How do you know the request came from the legitimate sender?
- Reliability: What if your endpoint is down or slow?
- Duplicates: Networks can cause retries; you might receive the same event multiple times.
Let's tackle security first.
Securing Webhooks with Signed Payloads
Signed payloads use a shared secret to create a cryptographic signature of the request body. The sender includes this signature in a header (e.g., X-Signature). Your endpoint recomputes the signature and compares it. If they match, the request is authentic.
How HMAC Signing Works
Most providers use HMAC-SHA256. The signature is computed over the raw request body using the shared secret. Here's the flow:
- Provider and consumer share a secret key.
- Provider creates the payload, computes HMAC-SHA256(secret, payload), and sends it in a header.
- Consumer receives the request, reads the raw body, computes the same HMAC, and compares.
Important: Use the raw body exactly as received—parsing and re-serializing JSON can change the bytes and break the signature.
Implementing Signature Verification
Here's a Node.js example using Express:
const crypto = require('crypto');
const express = require('express');
const app = express();
// Use raw body parser for webhook routes
app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
const signature = req.headers['x-signature'];
const secret = process.env.WEBHOOK_SECRET;
const expected = crypto
.createHmac('sha256', secret)
.update(req.body)
.digest('hex');
if (!crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))) {
return res.status(401).send('Invalid signature');
}
// Process the event
const event = JSON.parse(req.body);
// ... handle event
res.status(200).send('OK');
});
Always use a constant-time comparison like crypto.timingSafeEqual to prevent timing attacks.
Other Security Considerations
- Timestamp validation: Include a timestamp in the signed payload and reject old requests to prevent replay attacks.
- HTTPS: Always use HTTPS for webhook endpoints.
- Secret rotation: Support multiple active secrets to allow rotation without downtime.
Implementing Reliable Retries
Even with a secure endpoint, network issues or temporary failures can cause webhook delivery to fail. A robust retry mechanism ensures events aren't lost.
Sender-Side Retry Strategy
If you're building a webhook sender, implement exponential backoff with jitter. For example:
- Retry 1: after 1 minute
- Retry 2: after 5 minutes
- Retry 3: after 30 minutes
- Retry 4: after 2 hours
- Retry 5: after 24 hours
Add random jitter to avoid thundering herd problems. Stop after a maximum number of attempts and alert.
Receiver-Side Best Practices
As a webhook consumer, you can't control the sender's retry policy, but you can make your endpoint resilient:
- Respond quickly: Acknowledge receipt with a 2xx status before processing heavy work. Use a queue.
- Idempotency: Handle duplicate events gracefully. Use the event ID to deduplicate.
- Monitor failures: Log failures and set up alerts.
Handling Duplicates with Idempotency
Webhook providers often retry on timeouts, so you may receive the same event multiple times. Make your processing idempotent:
- Store processed event IDs in a database with a unique constraint.
- Before processing, check if the ID already exists; if so, return 200 OK without reprocessing.
- For state-changing operations, use idempotency keys.
Example using a database unique index:
async function handleEvent(event) {
try {
await db.query('INSERT INTO processed_events (id) VALUES ($1)', [event.id]);
} catch (err) {
if (err.code === '23505') { // unique violation
return; // already processed
}
throw err;
}
// process event
}
Comparison: Webhook Security Approaches
| Approach | Pros | Cons |
|---|---|---|
| HMAC signature | Strong integrity, widely supported | Requires secret management |
| Basic Auth | Simple | Less secure, credentials in header |
| IP allowlist | Easy to implement | IPs can change, not robust |
HMAC signatures are the de facto standard for webhook security.
Testing Webhooks Locally
Tools like ngrok or localtunnel expose your local server to the internet for testing. Always test signature verification and retry logic with simulated failures.
When inspecting JSON payloads from webhooks, formatting them can help debug. Try our JSON Formatter to pretty-print and validate webhook payloads.
FAQ
How do I verify a webhook signature?
Compute an HMAC of the raw request body using the shared secret, then compare it to the signature header using a constant-time comparison function.
What should I do if my webhook endpoint is down?
Most providers retry with exponential backoff. Ensure your endpoint returns a 2xx quickly and process asynchronously to minimize downtime impact.
Can I use webhooks without signatures?
It's risky. Without signatures, attackers can spoof requests. Always use signed payloads or at least basic authentication over HTTPS.
Conclusion
Webhooks are powerful but require attention to security and reliability. Implement HMAC signature verification to ensure authenticity, and design your system to handle retries and duplicates gracefully. With these practices, your webhook integrations will be robust and production-ready.