The event fired but the customer never got it: designing webhooks other teams can rely on

When an integrator asks you to "notify us the moment an order is created," the work does not end with a POST to their URL. A webhook system that holds up has four parts: producing the event reliably inside your application (in the same transaction as the data change, written to a queue), a delivery layer (timeouts, exponential backoff, a dead letter path), verifiability (a signature over the body, a timestamp, replay protection) and observability (a delivery record for every attempt, plus manual resend). Skip any of the four and the conversation always ends in the same place. The customer says the notification never arrived, and you have no screen that shows whether it did.

First decision: webhooks, polling, or both

Webhooks lower latency. Polling guarantees completeness. They are not alternatives, they are two halves of the same integration. Shopify says this plainly in its own developer docs: webhook delivery is not always guaranteed, so apps should run reconciliation jobs that pull data with updated_at filters. Stripe closes the same gap with an events list endpoint.

The practical rule: give customers a webhook and an endpoint that answers "what changed since this timestamp." When a delivery is missed, they should be able to heal themselves without opening a ticket. Updating a screen that is open in a browser right now is a different problem, and that is where WebSocket and SSE come in (real-time update methods).

Where is the event produced?

The most common mistake is sending the webhook from inside the HTTP request handler. It breaks in two ways. If the transaction rolls back, the event has already left and the customer processes an order that does not exist on your side. And if the receiver is slow, your user's request is slow; if the receiver is down, your user's request fails.

The fix is the outbox pattern: write an event row in the same database transaction as the state change, and have a separate worker read those rows and deliver them. The event only exists if the data actually changed, nothing is lost when the queue is briefly unavailable, and production order is preserved in the database. Queue layout, worker pools and retry settings are covered in the background jobs post.

Thin payload or full snapshot?

Two options. A thin payload carries the event id, the type, the time and the id of the affected record, and the receiver fetches current data from your API. A snapshot puts every field of the record, as it was at that moment, into the body.

Stripe now recommends thin events for new integrations and expects the receiver to fetch the related object. The Standard Webhooks specification says to keep payloads under roughly 20 KB and to send a link rather than embedding large data. Thin payloads buy you three things: the receiver always sees current data, sensitive fields do not travel unnecessarily, and out-of-order delivery hurts less. The cost is an extra API call per event, which means the traffic your webhooks generate has to be planned on the API side too (rate limiting and quotas).

Whichever you pick, four fields belong in every body: a unique event id, the event type, when the event happened, and a schema version marker. Without that marker, the first field rename breaks every integrator you have (API versioning and backward compatibility).

Signing: in a header, over the raw body

A webhook endpoint is a publicly reachable door with no login behind it. The receiver needs a way to prove the body came from you, and that way is an HMAC signature generated with a shared secret. Three widely used designs, all variations on one idea:

• Standard Webhooks: the webhook-id, webhook-timestamp and webhook-signature headers. The signed content is id.timestamp.body joined with full stops, signed with HMAC-SHA256, and the secret is stored with a whsec_ prefix. The spec also defines asymmetric signing with ed25519, where the receiver holds only a public key instead of your shared secret.
• Stripe: a Stripe-Signature header carrying t= for the timestamp and v1= for the signature. The signed content is the timestamp, a full stop and the raw body.
• GitHub: the X-Hub-Signature-256 header, a hex HMAC-SHA256 with a sha256= prefix. The docs explicitly say never to use a plain == for the comparison and to call a constant-time compare function instead.

Three rules worth writing down. One: the signature covers the raw body, so any middleware that parses the JSON and re-serialises it will break verification. That is the single most common verification failure in Stripe's own troubleshooting guide. Two: put the timestamp inside the signed content so the receiver can detect a replayed request. Stripe's libraries default to a five minute tolerance, and Slack uses the same window, rejecting timestamps older than five minutes. Three: plan secret rotation from day one. Stripe keeps the old secret alive for up to 24 hours and sends one signature per active secret, so the receiver can update code without downtime.

Never put the secret in the URL. GitHub calls this out separately, and for good reason: URLs end up in access logs, error traces and screenshots (secret management).

At-least-once delivery, no ordering guarantee

Write those two facts in your integrator docs, or the other side will discover them in production. Stripe does not guarantee that events arrive in the order they were generated, and tells you not to use the created field for ordering or for duplicate detection, but to track event ids instead. Shopify is blunter still: there is no ordering guarantee within a topic or across topics for the same resource, and a products/update can land before its products/create.

So give the receiver three things. A unique event id that stays the same across retries (Stripe's event id, GitHub's X-GitHub-Delivery, Shopify's X-Shopify-Webhook-Id all do this), a field carrying the record's version or last-updated time, and one line of instruction in the docs: if you see the same id twice, skip it; if a version arrives that is older than what you hold, drop it. What duplicate processing costs when money is involved is covered in the payment integration post.

Retry schedule and status code policy

Retries use exponential backoff with jitter. Without jitter, every pending event comes back in the same second after an outage and you are the one who knocks the receiver over (surviving third-party outages).

Three reference points for the window: Stripe retries delivery for up to three days in live mode, the example schedule in the Standard Webhooks spec ends about 75.5 hours after the first attempt, and Slack makes three attempts on event subscriptions (immediately, after about a minute, after about five minutes). Which shape you pick depends on your product, but the schedule has to be written down so integrators know how long they are covered.

How you react to status codes is part of the design too. The policy Standard Webhooks recommends is compact: 2xx is success, 3xx counts as failure and redirects are not followed (Stripe also treats redirects as failures), 410 is a signal to disable the endpoint, and 429, 502 and 504 mean throttle. Honour a Retry-After header when you get one.

Have a rule for disabling endpoints that fail for a long stretch, and warn the customer before you pull the switch. Slack is a measured example here: if more than 95% of attempts fail within 60 minutes, event subscriptions are temporarily disabled and the app owner gets an email, while apps receiving fewer than 1,000 events an hour are exempt from the automatic shutoff.

Time budgets and the queue one slow receiver blocks

Publish how long you wait for a response, because that number shapes the integrator's architecture. GitHub expects a 2xx within 10 seconds, Slack within three. Stripe puts the point in one sentence: return a 2xx before any complex logic, and return 200 before you mark the invoice paid in your accounting system.

The same discipline applies on your side as the sender. Keep your timeout to a few seconds, and give every destination its own queue or at least its own concurrency budget. In a single-queue design, one customer's dead endpoint blocks the head of the queue and every other customer's events wait behind it. Tripping a circuit breaker on a destination that keeps failing beats retrying at full speed, for both sides.

A customer-supplied URL is an attack surface

If the customer types in the webhook target, they can make your server send a request to any address they choose. That is the textbook shape of server-side request forgery: the target can be an internal admin panel, a database console, or the cloud provider's instance metadata at 169.254.169.254. The Standard Webhooks spec puts this first in its security section and recommends two concrete controls: route outbound webhook traffic through a filtering proxy that blocks internal IP ranges (it names Smokescreen as an example), and run webhook workers in a private subnet with no access to internal services.

Four rules to add on top: accept HTTPS only, do not follow redirects, validate the resolved IP at connection time rather than at configuration time (DNS rebinding lives in exactly that gap), and cap how much of the response body you read. For the neighbouring doors at the application layer, see our API security post.

The screen you open when someone says "we never got it"

What makes a webhook system debuggable is the delivery record. For each attempt, store the event id, the destination, the attempt number, the HTTP status code, a truncated response body, the duration and the time of the next attempt. Stripe's dashboard shows each event as Delivered, Pending or Failed, and names connection errors, TLS errors and timeouts separately. That distinction matters, because a 404 and a TLS error are two different problems for two different teams.

Plan manual resend from the start and state its window. Stripe allows resending from the dashboard for 15 days after the event was created and from the CLI for 30 days, and warns that a manual resend does not cancel the automatic retries. Open that screen to customers and most of the "did it arrive or not" traffic disappears from your support queue.

Finally, alert per destination. Your overall success rate can sit at 99% while one customer's endpoint has been completely down for two days, and nothing in the aggregate chart will show it. Thresholds and alert design are covered in the SLO and error budget post.

Three steps that fit in this week

One: trace a single event type end to end. Where is it produced (inside the transaction or after it), does the body carry an event id and a version marker, does the signature cover the raw body.

Two: if you have no delivery record, build that first. Without it, every discussion is stuck between "it shows as sent on our side" and "it never reached us," and nobody can move.

Three: add four sentences to your integrator docs. Delivery is at least once, order is not guaranteed, return 2xx before doing the work, verify the signature over the raw body. Next to them, publish your timeout and your retry schedule.

At Wedevit we build the webhook layer on the outbox pattern, design the signing and secret rotation scheme to match the standards, isolate queues and circuit breakers per destination, and turn the delivery log and resend screen into something integrators can use without calling you. A concrete first step: if a customer said today that last night's order notification never arrived, could you show them which screen holds the attempts and what each one returned?