Webhooks 101 for Engineering Workflows

TL;DR — A webhook is a callback URL the sender POSTs to when an event happens. Always verify signatures. Respond fast (under 5 seconds). Make handlers idempotent. Plan for retries. Treat the URL as a credential and rotate when leaked.

After several webhook-driven workflows in May (Jira auto-assign, Slack slash commands, Notion sync), time for the protocol primer. Most teams use webhooks for years without understanding the failure modes. This post is the things-that-bite-you list.

What a webhook is

A POST request from one service to a URL the receiver controls, fired when an event happens. Inverse of polling: instead of you asking “anything new?”, the sender tells you.

GitHub PR opened
    → POST https://your-app.com/webhooks/github
       Body: { action: "opened", pull_request: {...} }

Your endpoint receives, processes, responds 2xx. Sender forgets about you (or retries on non-2xx).

That’s it conceptually. The details kill you.

What can go wrong

The mistakes I keep seeing:

1. No signature verification. Anyone who knows the URL can POST anything to it. Easy to fix.
2. Slow processing. Your handler takes 8 seconds; sender retries because it thinks you failed. You process twice.
3. Not idempotent. Same event processed twice creates duplicate tickets/messages/data.
4. No replay protection. Captured webhook from yesterday replayed today still works.
5. Trusting payload data without verification. Sender claims to be GitHub but isn’t.
6. Logging the body in plain text. Often contains secrets (OAuth tokens, JWTs).
7. Returning 200 immediately, then crashing. Sender thinks you succeeded; you didn’t process.

Each has a standard solution. Most aren’t implemented in 80% of webhook handlers I’ve audited.

Signature verification

Every reputable webhook sender supports it. Pattern: sender includes an HMAC of the body in a header; receiver recomputes with a shared secret; rejects if mismatch.

GitHub:

Header: X-Hub-Signature-256: sha256=<hex>
Algorithm: HMAC-SHA256
Secret: configured when you create the webhook

Stripe:

Header: Stripe-Signature: t=<ts>,v1=<hex>
Algorithm: HMAC-SHA256 of ts.body
Secret: webhook signing secret from Stripe dashboard

Slack:

Header: X-Slack-Signature: v0=<hex>
Algorithm: HMAC-SHA256 of "v0:" + ts + ":" + body
Secret: app signing secret

In n8n, do this in a Code node right after the Webhook trigger:

const crypto = require('crypto');
const item = $input.first();
const body = item.json.bodyRaw || '';
const sig = item.json.headers['x-hub-signature-256'] || '';

const expected = 'sha256=' + crypto
  .createHmac('sha256', $env.GITHUB_WEBHOOK_SECRET)
  .update(body)
  .digest('hex');

if (!crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected))) {
  throw new Error('invalid signature');
}

return [{ json: JSON.parse(body) }];

Use timingSafeEqual — comparing strings with === is vulnerable to timing attacks. Tiny attack surface, but it’s free to defend.

Webhook trigger needs “Raw Body” enabled so you get the unparsed body for HMAC.

Respond fast

Most senders give you 5-10 seconds. Some less. If your processing takes longer:

Pattern A — respond immediately, queue for processing.

[Webhook]
    → respond 200 in ~50ms
    → push event to internal queue (DB / Redis / RabbitMQ)

[Worker process]
    → pull from queue
    → process at leisure

n8n doesn’t have a great built-in queue, but you can write the event to a Postgres table and have a second workflow consume it on a schedule.

Pattern B — handle synchronously when fast enough.

For workflows that finish in under a second, just process synchronously. Most n8n workflows fit here.

The pivot point: if your processing involves a slow external API, queue. If it’s all fast nodes, sync.

Idempotency

Senders retry on failure. The same event can arrive twice, ten times, or once a day later.

Your handler MUST be safe to run with the same input multiple times.

Two patterns:

Pattern A — natural idempotency. Use the source’s unique ID to upsert into your DB:

INSERT INTO jira_tickets (id, status, updated_at) VALUES (?, ?, ?)
ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, updated_at = EXCLUDED.updated_at;

Re-running with the same event produces the same DB state.

Pattern B — dedupe at receipt. Store every event ID you’ve processed (e.g., GitHub’s X-GitHub-Delivery header is a UUID per delivery). If already seen, return 200 and skip.

const deliveryId = $input.first().json.headers['x-github-delivery'];

const seen = await db.query('SELECT 1 FROM processed_deliveries WHERE id = $1', [deliveryId]);
if (seen.rows.length > 0) return [{ json: { duplicate: true } }];

await db.query('INSERT INTO processed_deliveries (id, received_at) VALUES ($1, NOW())', [deliveryId]);
// ... process

Pattern B is more work; sometimes necessary when the action isn’t naturally idempotent (e.g., “send a notification”).

Replay protection

Even with signatures verified, an attacker who captured a webhook can resend it. The defense: check the timestamp.

const ts = parseInt(headers['x-slack-request-timestamp'], 10);
const now = Math.floor(Date.now() / 1000);
if (Math.abs(now - ts) > 300) {   // > 5 minutes
  throw new Error('event too old');
}

Most signing protocols include a timestamp in the signed string (Slack, Stripe). Verify both the signature AND the timestamp freshness. A signature alone proves authenticity; the timestamp proves recency.

Retries — sender side

Most senders retry failed deliveries automatically. GitHub: a few times with backoff. Stripe: up to 3 days with exponential backoff. Slack: a few times within an hour.

Implications:

• Don’t return 5xx on permanent failures (e.g., “invalid signature”) — return 4xx so the sender stops retrying. 5xx says “retry, this is transient.”
• A successful workflow run that has a 5xx response confuses the sender. Return 2xx if you successfully received and queued, even if downstream processing later fails.

Treat the URL as a credential

Webhook URLs are often the only thing protecting your endpoint from arbitrary input. The URL itself is the secret.

• Don’t log webhook URLs in app logs
• Don’t paste them into Slack
• Rotate them if you suspect a leak
• Prefer signature verification so the URL alone isn’t enough

In-development webhook receiving

For developing webhook workflows on localhost, you need a public URL. Two tools:

• ngrok — ngrok http 5678 gives you a public URL forwarding to your local n8n
• cloudflared tunnel — Cloudflare’s alternative, free

Configure the sender to point at the public URL during dev. Re-point to your production n8n URL after.

Some senders (GitHub, Stripe) have a “redeliver” button to replay a captured webhook from their UI. Use it for testing instead of asking your colleague to click “merge” 14 times.

Common Pitfalls

Trusting User-Agent: GitHub-Hookshot/. Trivial to forge. Always verify signatures, never trust headers.

Logging the raw body to stdout. Some bodies contain OAuth tokens, JWTs, secrets. Redact known sensitive fields or log structured subsets.

Returning 200 OK before processing. If the next step crashes, the sender thinks you got it. Either process synchronously, or have a robust queue with its own retries.

Reusing the same URL for prod + staging. Staging tests fire at prod handler. Dedicated URLs per environment.

Hardcoded webhook URLs in version control. Even for read-only repos, public webhook URLs are credentials. Use env vars or n8n’s webhook auth.

Ignoring rate-limit headers from senders. Some senders include X-RateLimit-* headers. If you’re firing back at their API, respect them.

Single-instance n8n. If n8n goes down, you miss webhooks. Sender may retry; may not. Run two n8n instances behind a load balancer for anything critical.

Wrapping Up

Webhooks are the protocol that ties every event-driven workflow together. Verify signatures, respond fast, be idempotent, protect against replay, treat URLs as credentials. Friday: error handling and retries in n8n — what happens when the workflow itself fails.