Webhooks

Polling has a built-in delay: on average, you discover an event half your polling interval after it happened. Poll every 5 seconds? Average 2.5-second delay before you know. Poll every 1 second to reduce delay? Now you've 5x'd your request volume.

Webhooks break this tradeoff entirely. Delivery latency is typically under 1 second (GitHub delivers within 10 seconds; Stripe is usually under 5), and the request count is proportional to actual events — not clock ticks.

With polling, your server load is proportional to time × users, regardless of whether anything is happening. At 3 AM when nobody's shopping, your polling loops are still firing 2,000 requests/second — paying for compute and bandwidth to hear "nothing changed" 172 million times a day.

With webhooks, load is proportional to actual events. Zero events at 3 AM? Zero requests. Black Friday surge with 50,000 orders per hour? 50,000 webhook deliveries — exactly the traffic you'd expect, and exactly what you need to handle. The infrastructure costs track the business, not the clock.

Well-designed webhook payloads follow a consistent pattern. Every payload should tell you three things: what happened (event type), when it happened (timestamp), and the details (the changed resource). Some providers send the full resource in the payload; others send only an ID and expect you to fetch the full details — this is called a "thin" vs "fat" payloadA 'fat' payload includes all the data you need right in the webhook body. A 'thin' payload only sends an event type and resource ID — you then call the API to get the details. Fat is faster to process; thin is more secure (less data in transit)..

Here's a scary thought: your webhook URL is just a public endpoint. Anyone who knows it could send a fake POST pretending to be Stripe, telling your app "this payment succeeded" when it didn't. You'd ship products to someone who never paid. This is why signature verificationA way to prove a message really came from who it claims to be from. The sender signs the message with a secret key, and you verify it using the same key. If the signature matches, the message is authentic. is absolutely critical.

The solution is called HMACHash-based Message Authentication Code. A cryptographic method where the sender hashes the message body together with a shared secret key. The receiver does the same hash — if they match, the message is authentic and hasn't been tampered with. — Hash-based Message Authentication Code. Here's how it works in plain English:

import hmac
import hashlib

def verify_webhook(payload_body, signature_header, secret):
    """Verify the webhook really came from the provider."""
    # Compute what the signature SHOULD be
    expected = hmac.new(
        key=secret.encode('utf-8'),
        msg=payload_body,
        digestmod=hashlib.sha256
    ).hexdigest()

    # Compare securely (prevents timing attacks)
    if not hmac.compare_digest(expected, signature_header):
        raise ValueError("Invalid signature — possible forgery!")

    return True  # Signature matches, safe to process

const crypto = require('crypto');

function verifyWebhook(payloadBody, signatureHeader, secret) {
  // Compute what the signature SHOULD be
  const expected = crypto
    .createHmac('sha256', secret)
    .update(payloadBody)
    .digest('hex');

  // Compare securely (constant-time comparison)
  const isValid = crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signatureHeader)
  );

  if (!isValid) throw new Error('Invalid signature!');
  return true; // Safe to process
}

Networks are unreliable. Your server might be down for maintenance, or it might return a 500 error because of a temporary database outage. If the provider sends a webhook and doesn't get a 200 OK back, it needs to try again. But it can't just retry immediately a thousand times — that would overwhelm your server the moment it comes back up.

The solution is exponential backoffA retry strategy where the wait time doubles after each failure: first retry after 1 minute, then 2 minutes, then 4, then 8, etc. This prevents hammering a server that's already struggling.: wait a little, then wait longer, then even longer. Typically: 1 minute → 5 minutes → 30 minutes → 2 hours → 12 hours. After a certain number of attempts (usually 5-10), the provider gives up and marks the delivery as failed.

What happens when all retry attempts are exhausted and the webhook still can't be delivered? The event goes to a dead letter queueA holding area for messages that couldn't be delivered after all retry attempts. Named after the 'dead letter office' in postal services — where undeliverable mail ends up. Engineers review these to fix the root cause and replay failed events. (DLQ). Think of it like the "return to sender" pile at the post office — letters that couldn't be delivered after multiple attempts.

A dead letter queue is critical because it means no events are silently lost. You can inspect the queue, fix whatever was wrong (maybe your server was misconfigured, or the endpoint URL changed), and replay the failed events. Most providers like Stripe give you a dashboard where you can see failed webhook deliveries and manually retry them.

Imagine a customer creates an order, then updates the shipping address, then cancels it. Three events, in that order. But webhooks might arrive as: cancel → create → update. Why? Because each webhook is an independent HTTP request. Network conditions, retry timing, and provider-side queuing can all scramble the order.

Your handler needs to handle out-of-order delivery gracefully. Common strategies:

• Timestamps: Each event has a timestamp. Before processing, check if you already have a newer version of this resource. If so, ignore the older event.
• Version numbers: Some providers include a sequence number. Only process events with a higher sequence than what you've stored.
• Fetch latest state: Instead of trusting the payload, use it as a signal and always fetch the current state from the provider's API. This guarantees you have the freshest data regardless of delivery order.

HMAC signatures verify the sender, but there are more security layers to add:

• HTTPS only: Never expose a webhook endpoint over plain HTTP. The payload and signature would be visible to anyone on the network.
• IP allowlisting: Some providers publish the IP addresses they send webhooks from. You can configure your firewall to reject requests from any other IP. Stripe, for example, publishes their webhook IPs.
• Timestamp validation: Check that the event timestamp is recent (within the last 5 minutes). This prevents replay attacksAn attacker captures a legitimate webhook and re-sends it hours or days later. If your handler just checks the signature (which is still valid), it would process the stale event. Checking the timestamp prevents this. — where an attacker captures a valid webhook and re-sends it days later.
• Secret rotation: Periodically generate new signing secrets. Good providers let you have two active secrets during rotation so you don't miss events during the switch.

Most providers give your endpoint 5-30 seconds to respond with a 200. If your handler takes longer (maybe it's writing to a slow database or calling another API), the provider considers it a failure and retries. This creates a nasty loop: slow processing → timeout → retry → more slow processing → more timeouts.

The golden rule: accept fast, process later. Your webhook endpoint should do exactly three things: (1) verify the signature, (2) drop the event onto an internal queue, (3) return 200 OK. A background worker then picks up the event from the queue and does the real processing — updating databases, sending emails, calling other APIs. This decoupling means your endpoint always responds in milliseconds, regardless of how complex the processing is.

GitHub webhooks trigger on repository activity: pushes, pull requests, issues, releases, workflow runs. This is what powers CI/CD pipelines — when you push code, GitHub sends a webhook to Jenkins/CircleCI/GitHub Actions telling it to run your build.

• Signature: HMAC-SHA256 in the X-Hub-Signature-256 header.
• Retries: Failed deliveries are retried for up to 3 days.
• Events: You choose which events to subscribe to (push, pull_request, issues, etc.) — you don't get everything.
• Ping event: When you first set up a webhook, GitHub sends a ping event to verify your endpoint is reachable.

Twilio uses webhooks to notify your app about incoming SMS messages, call status changes, and delivery receipts. When someone texts your Twilio number, Twilio sends a webhook to your URL with the message body, sender, and phone number.

• Signature: Uses an authentication token and request URL to generate a signature in the X-Twilio-Signature header.
• Unique twist: Twilio webhooks can return TwiMLTwilio Markup Language — an XML format that tells Twilio what to do in response to a webhook. Your endpoint can return TwiML saying 'play this audio' or 'forward this call to another number.' in the response body to control the call/message flow. Your webhook isn't just receiving data — it's sending instructions back.
• Fallback URL: You can configure a backup URL. If your primary webhook fails, Twilio automatically tries the fallback before giving up.

Strong answer: "Webhooks are inherently a push model — if the provider can't push, you won't receive events during the outage. That's why production systems should never rely 100% on webhooks alone. I'd combine webhooks with a periodic reconciliation jobA scheduled process (e.g., running every hour or daily) that pulls records from the provider's API and compares them with your database to catch any discrepancies or missed events. — say, every hour, my system fetches recent transactions from the provider's API and compares them with what I've received via webhooks. Any missing events get processed. This 'belt and suspenders' approach means neither a webhook failure nor a provider outage causes permanent data loss."

Strong answer: "Three strategies: (1) Use the provider's CLI tool (like stripe listen --forward-to localhost:4242/webhook) which tunnels real events to my local machine. (2) Use ngrok to create a temporary public URL that points to my local server. (3) For automated tests, I mock the webhook request by constructing the payload and computing the HMAC signature with my test secret. I always test the signature verification path — both valid and invalid signatures — to make sure my security logic is correct."

Strong answer: "Webhooks are one implementation of event-driven architecture that works across organizational boundaries — company A notifies company B via HTTP. Event-driven architecture is the broader pattern: when something happens, publish an event, and interested parties react. Inside a single system, you'd typically use a message broker like Kafka or RabbitMQ rather than HTTP webhooks. Webhooks are the HTTP-based, cross-boundary version of event-driven communication — they're simpler to set up (just a URL) but less reliable and performant than internal message queues."