Webhooks Explained: Signed Payloads and Reliable Retries
You've just integrated a third-party service that sends webhooks to your endpoint. Everything works in testing, but in production, you notice missed events, duplicate deliveries, and even forged requests. Webhooks are simple in concept but tricky to get right. This article explains how to secure webhooks with signed payloads and implement reliable retry logic so your integrations are robust and trustworthy.
What Are Webhooks and Why They Matter
Webhooks are user-defined HTTP callbacks triggered by events. Instead of polling an API for changes, you register a URL, and the service sends a POST request with event data whenever something happens. This push model reduces latency and server load, making it ideal for real-time integrations like payment notifications, CI/CD triggers, and chat messages.
However, webhooks introduce challenges: verifying authenticity, handling failures, and ensuring exactly-once processing. Let's tackle these one by one.
Securing Webhooks with Signed Payloads
Without verification, anyone can send fake requests to your webhook endpoint. Signed payloads solve this by including a cryptographic signature in the request headers, generated using a shared secret. You verify the signature to confirm the request came from the expected sender and wasn't tampered with.
How HMAC Signatures Work
Most services use HMAC (Hash-based Message Authentication Code) with SHA-256. The sender computes a hash of the request body (and sometimes a timestamp) using the secret, and includes it in a header like X-Signature. You recompute the hash on your side and compare.
Here's a typical verification flow in Node.js:
const crypto = require('crypto');
function verifySignature(payload, signature, secret) {
const expected = crypto
.createHmac('sha256', secret)
.update(payload)
.digest('hex');
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(expected)
);
}
Always use a timing-safe comparison to prevent timing attacks. Also, include a timestamp in the signed data and reject requests older than a few minutes to prevent replay attacks.
Best Practices for Signature Verification
- Use HTTPS to protect the payload in transit.
- Store secrets securely (e.g., environment variables, secret managers).
- Validate the timestamp and reject stale requests.
- Log verification failures for monitoring.
Implementing Reliable Retries
Networks fail, servers restart, and endpoints time out. A robust webhook system must retry failed deliveries. But retries can cause duplicates, so you need idempotency.
Retry Strategies
Most providers use exponential backoff with jitter. For example, retry after 1 minute, then 5 minutes, then 30 minutes, up to a maximum number of attempts (e.g., 5). Include a unique event ID in the payload so the receiver can deduplicate.
On the receiving side, respond quickly with a 2xx status to acknowledge receipt, then process asynchronously. If processing fails, you can't rely on the sender to retry indefinitely—implement your own dead-letter queue or alerting.
Idempotency Keys
To handle duplicate deliveries, use an idempotency key (often the event ID) and store processed IDs for a reasonable period. Before processing, check if the ID has been seen; if so, return success without re-processing.
// Example: idempotency check in Express
app.post('/webhook', async (req, res) => {
const eventId = req.body.id;
if (await db.seenEvents.has(eventId)) {
return res.status(200).send('Already processed');
}
await db.seenEvents.add(eventId);
// process event asynchronously
res.status(200).send('OK');
});
Comparison: Polling vs Webhooks
| Aspect | Polling | Webhooks |
|---|---|---|
| Latency | High (interval-based) | Low (near real-time) |
| Server Load | High (frequent requests) | Low (event-driven) |
| Complexity | Simple to implement | Requires signature verification and retries |
| Reliability | Depends on polling frequency | Depends on retry mechanism |
Step-by-Step: Building a Secure Webhook Receiver
- Set up an endpoint that accepts POST requests over HTTPS.
- Verify the signature using the shared secret and reject invalid requests with 401.
- Check the timestamp to prevent replay attacks.
- Respond immediately with 200 OK after basic validation, then process asynchronously.
- Implement idempotency using event IDs to avoid duplicate processing.
- Log and monitor failures and retries.
FAQ
What is a signed webhook payload?
A signed payload includes a cryptographic signature (usually HMAC) that allows the receiver to verify the request's authenticity and integrity using a shared secret.
How many times should I retry a failed webhook?
Common practice is 3–5 retries with exponential backoff. After that, move the event to a dead-letter queue for manual inspection.
Can I use webhooks without HTTPS?
No, HTTPS is essential to protect the payload and signature from eavesdropping and tampering. Always use TLS.
Ready to test your webhook payloads? Use our JSON formatter to pretty-print and validate JSON payloads quickly.