Webhooks Explained: Signed Payloads and Retries
You've just integrated a payment gateway using webhooks. Everything works in testing, but in production, events sometimes go missing, or you receive duplicate notifications that cause double charges. The root cause often lies in how you handle signed payloads and retries. This article explains the mechanics of webhooks and provides practical steps to build a robust, secure integration.
What Are Webhooks?
Webhooks are user-defined HTTP callbacks. When an event occurs in a source system (e.g., a payment is processed), it sends an HTTP POST request to a URL you specify, containing event data. Unlike polling, webhooks push data in near real-time, reducing latency and server load.
However, webhooks introduce challenges: verifying authenticity, handling failures, and ensuring exactly-once processing. Let's tackle each.
Signed Payloads: Verifying Authenticity
Since webhook endpoints are publicly accessible, anyone could send fake requests. To prevent this, providers sign the payload with a secret key. You verify the signature to ensure the request is genuine.
How HMAC Signatures Work
Most providers use HMAC (Hash-based Message Authentication Code) with SHA-256. The provider computes a signature using the request body and a shared secret, then includes it in a header (e.g., X-Signature). You recompute the signature on your side and compare.
Example in Python:
import hmac
import hashlib
def verify_signature(payload, secret, received_signature):
expected = hmac.new(secret.encode(), payload, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, received_signature)
Important: Use the raw request body, not parsed JSON, as parsing can alter whitespace and break the signature.
Common Pitfalls
- Using parsed data: Always verify against the raw bytes.
- Timing attacks: Use constant-time comparison (e.g.,
hmac.compare_digest). - Ignoring timestamps: Some providers include a timestamp in the signature to prevent replay attacks. Validate it's within a tolerance window (e.g., 5 minutes).
Retries: Handling Failures Gracefully
Your endpoint might be temporarily down, or network issues could cause failures. Reliable webhook systems retry failed deliveries with exponential backoff.
Retry Strategies
Providers typically retry on non-2xx responses or timeouts. Common patterns:
| Strategy | Description | Example |
|---|---|---|
| Fixed interval | Retry every N seconds | Every 30s, up to 5 times |
| Exponential backoff | Double the delay each attempt | 1s, 2s, 4s, 8s... |
| Exponential with jitter | Add randomness to avoid thundering herd | 1s ± 0.5s, 2s ± 1s... |
As a receiver, you can't control the sender's retry policy, but you can design your endpoint to be resilient.
Best Practices for Receivers
- Respond quickly: Return 2xx within a few seconds. Offload processing to a background job.
- Be idempotent: Use an idempotency key from the payload to avoid duplicate processing.
- Log everything: Store incoming webhooks for debugging and replay.
- Monitor failures: Set up alerts for repeated failures.
Implementing Idempotency
Duplicates happen: the sender might retry because your response was slow, or you might accidentally process the same event twice. Idempotency ensures that processing an event multiple times has the same effect as once.
Use a unique event ID (often provided in the payload or headers) and store it in a database with a unique constraint. Before processing, check if the ID exists; if so, skip.
def process_event(event_id, data):
if EventLog.exists(event_id):
return # already processed
EventLog.create(event_id)
# process data...
For high-throughput systems, use a distributed lock or a database transaction to avoid race conditions.
Security Considerations
Beyond signature verification, consider:
- HTTPS: Always use TLS to prevent eavesdropping.
- IP allowlisting: If the provider publishes IP ranges, restrict incoming requests.
- Rate limiting: Protect your endpoint from abuse.
- Payload validation: Even after signature verification, validate the data schema to avoid injection attacks.
Testing Webhooks Locally
During development, you need a public URL to receive webhooks. Tools like ngrok or localtunnel expose your local server. Alternatively, many providers offer a CLI to forward events to your localhost.
Simulate failures to test retry handling: return 500 errors intentionally and observe the provider's retry pattern.
FAQ
How do I verify a webhook signature?
Use HMAC with the shared secret. Compute the hash of the raw request body and compare it to the signature header using a constant-time comparison function.
What should I do if I receive duplicate webhooks?
Implement idempotency by storing event IDs. Check if the event has already been processed before executing business logic.
Can I rely on webhook retries alone for reliability?
No. Retries help but can't guarantee delivery. For critical events, combine webhooks with a polling fallback or a message queue that persists events.
When debugging webhook payloads, you might need to inspect JSON data. Use our JSON Formatter to pretty-print and validate payloads quickly.