Webhooks Explained: Signed Payloads and Retry Strategies
You've built an API that sends webhooks to notify external services about events. But what happens when the receiving endpoint is down? Or when a malicious actor tries to forge a webhook request? Without proper safeguards, your webhooks can become a source of data inconsistency and security vulnerabilities. This article explains how to implement signed payloads and retry mechanisms to make your webhooks reliable and secure.
What Are Webhooks and Why Do They Need Security?
Webhooks are user-defined HTTP callbacks that are triggered by specific events. They enable real-time communication between systems, such as notifying a payment service when an order is placed. However, because webhooks are publicly accessible endpoints, they are susceptible to spoofing and replay attacks. Additionally, network issues or receiver downtime can cause missed events. Therefore, two critical aspects must be addressed: authenticity (via signed payloads) and reliability (via retries).
Signed Payloads: Ensuring Authenticity and Integrity
A signed payload allows the receiver to verify that the webhook originated from a trusted source and that the data hasn't been tampered with. This is typically achieved using HMAC (Hash-based Message Authentication Code) with a shared secret.
How HMAC Signing Works
The sender and receiver share a secret key. For each webhook request, the sender computes an HMAC signature using the secret and the request payload (often including a timestamp to prevent replay attacks). The signature is sent in a header, such as X-Signature. The receiver recomputes the signature and compares it with the received one. If they match, the request is authentic.
Here's an example in Python using HMAC-SHA256:
import hmac
import hashlib
import time
secret = b'your_shared_secret'
payload = b'{"event":"order.created","id":123}'
timestamp = str(int(time.time())).encode()
message = timestamp + b'.' + payload
signature = hmac.new(secret, message, hashlib.sha256).hexdigest()
# Send headers: X-Signature: signature, X-Timestamp: timestamp
On the receiver side, verify the signature and check the timestamp to ensure the request is recent (e.g., within 5 minutes). This prevents replay attacks.
Best Practices for Signed Payloads
- Use a strong secret: Generate a random, high-entropy secret and store it securely.
- Include a timestamp: Reject requests with timestamps outside an acceptable window.
- Use constant-time comparison: Prevent timing attacks by using functions like
hmac.compare_digest. - Rotate secrets periodically: Allow multiple active secrets during rotation.
Retry Mechanisms: Ensuring Reliable Delivery
Even with signed payloads, network failures or receiver errors can cause webhook delivery to fail. A robust retry strategy ensures that events are eventually delivered without overwhelming the receiver.
Designing a Retry Policy
A common approach is exponential backoff with jitter. For example, retry after 1 minute, then 5 minutes, then 30 minutes, up to a maximum number of attempts (e.g., 5). Include a unique event ID in the payload so the receiver can deduplicate.
Example retry schedule:
| Attempt | Delay |
|---|---|
| 1 | Immediate |
| 2 | 1 minute |
| 3 | 5 minutes |
| 4 | 30 minutes |
| 5 | 2 hours |
After exhausting retries, move the event to a dead-letter queue for manual inspection.
Idempotency: Handling Duplicate Deliveries
Because retries can cause duplicate deliveries, receivers must be idempotent. Include an event ID in the payload and have the receiver store processed IDs to ignore duplicates. Alternatively, design operations to be naturally idempotent (e.g., using PUT instead of POST).
Implementing Webhooks: A Step-by-Step Guide
- Define events and payload structure: Document the events your system emits and the JSON schema for each.
- Generate a secret: Create a unique secret per webhook endpoint and share it securely.
- Sign each request: Compute HMAC over the payload and timestamp, include in headers.
- Implement retries with backoff: Use a queue system to schedule retries.
- Provide a way to verify signatures: Document the verification process for receivers.
- Monitor and log: Track delivery attempts and failures for debugging.
Common Pitfalls and How to Avoid Them
- Ignoring TLS: Always use HTTPS to protect payloads in transit.
- Not validating the timestamp: Allows replay attacks.
- Retrying indefinitely: Can cause resource exhaustion; set a maximum.
- Assuming immediate delivery: Design for asynchronous processing.
FAQ
What is a signed webhook payload?
A signed webhook payload includes a cryptographic signature (e.g., HMAC) that allows the receiver to verify the sender's authenticity and data integrity.
How many times should I retry a failed webhook?
Typically 3 to 5 retries with exponential backoff. The exact number depends on your SLA and the criticality of the event.
Can I use JWT instead of HMAC for webhook signatures?
Yes, JWT can be used, but HMAC is simpler and more common for webhook signing. JWT adds overhead and is better suited for authentication tokens.
Ready to test your webhook payloads? Use our JSON formatter to validate and beautify your JSON payloads before sending.