Webhook Troubleshooting

When webhook deliveries break — Helius → dashboard ingest, dashboard → merchant fan-out, retries, dedup, ordering — this is the runbook.

The webhook stack has two halves:

• Inbound (ingest): Helius Enhanced webhooks → dashboard Worker → D1 + Queues.
• Outbound (fan-out): D1 events → dashboard dispatch worker → merchant endpoints (with retries / DLQ).

Most failures show up as either "events I expected didn't reach D1" (ingest broken) or "events reached D1 but the merchant didn't receive a delivery" (fan-out broken). Always determine which half is failing first.

Commerce events differ from protocol ingest: payment and refund state writes a UUID event into commerce_event_outbox in the same D1 batch as the business transition. The outbox routes directly by merchant_id. Protocol events continue to originate from Helius and use mandate-based routing.

Common Failure Signatures

Signature	Likely cause	Section
Synthetic worker passes pull but the dashboard UI doesn't update	Ingest pipeline stalled	Ingest Stalled
On-chain event exists; D1 row missing	Helius didn't deliver, or signature failed	Missing D1 Row
D1 row exists; merchant didn't receive	Fan-out failed	Fan-out Failed
Payment/refund changed but no commerce delivery exists	Outbox drain or merchant routing failed	Inspect commerce_event_outbox and dispatch rows by event ID
Merchant received the same delivery twice	Dedup key collision or replay during rotation	Dedup Issues
Merchant received events out of order	Queue ordering or retry interleave	Ordering Issues
Queue depth growing	Consumer keeping up but slow, or consumer stuck	Queue Backlog

Ingest Stalled

Detect:

• Synthetic worker passes the on-chain pull but reports the D1 backfill check failed.
• D1 billing_events ingestion lag > 30s for sustained period.

Triage:

1. Helius dashboard → webhook delivery log. Are deliveries succeeding (200 OK from our worker)?
2. If Helius shows non-200: check the dashboard worker logs (Cloudflare → Workers → vela-dashboard-worker → Live Logs). Likely an HMAC mismatch or a parse error.
3. If Helius shows 200 but D1 is empty: the worker is succeeding superficially but failing to write. Look for D1 batch failed in logs.

Fix:

• HMAC mismatch: confirm the Helius webhook secret matches what the worker expects (env var). Rotate if there's reason to suspect.
• Parse error: check whether @velapay/webhook schema-diff CI was bypassed in the last release. If schemas drift on-chain ahead of vela-webhook, ingest will fail. Bump and redeploy.
• D1 batch failed: read the typed error. Usually a constraint violation from a bad migration; see Database Migrations →.

Missing D1 Row

Detect: Operator queries D1 for an event known to exist on-chain (SELECT * FROM billing_events WHERE signature = '<sig>') and gets nothing.

Triage:

1. Check Helius delivery log for that transaction signature. Did Helius deliver?
2. If Helius didn't deliver: Helius may have missed the slot. Trigger backfill (see below).
3. If Helius delivered but worker rejected: check worker logs for that delivery's response code and error.
4. If worker accepted but didn't write: D1 transaction failure — check D1 audit logs.

Fix — backfill from on-chain:

cd vela-dashboard
bun run scripts/backfill-events.ts \
    --from-signature <SIG> \
    --to-signature <SIG>      # or a slot range

The backfill script reads the protocol accounts directly via vela-sdk, reconstructs the missed events, and writes them to D1 with a synthetic helius_id so they don't collide with real Helius deliveries. After backfill, fan-out will trigger automatically for any merchant subscribed to those events.

Fan-out Failed

Detect: D1 row exists; webhook_dispatch.status = 'failed' for the relevant delivery.

For Commerce OS, search with the UUID event id. Never deduplicate or diagnose solely by transaction signature: multiple business events can refer to on-chain transactions independently. Replay must preserve the original event ID and remain safe if the merchant already processed it.

Triage:

1. Look at webhook_dispatch.last_error for the typed error.
2. Common errors:
   • 403 / 401 — merchant rotated their HMAC secret; their server now rejects.
   • 5xx from merchant — their server is down.
   • timeout — their server is slow or unreachable.

Fix:

• Merchant secret rotation: trigger dual-secret rotation (admin console → endpoint → Rotate). The dispatcher will use the new secret on next retry.
• Merchant server down: nothing for VelaPay to do beyond exhausting retries → DLQ. Operator may notify the merchant proactively for high-volume endpoints.
• Manual replay: bunx vela admin webhook replay --dispatch-id <UUID>.

Dedup Issues

The dashboard's idempotency key derives from the on-chain transaction signature + event type. Collisions are nearly impossible unless:

• A backfill writes the same event a second time after Helius delivers it.
• A retry storm causes the dispatcher to re-emit during the retry window.

Triage:

1. SELECT count(*) FROM webhook_dispatch WHERE idempotency_key = '<KEY>' GROUP BY idempotency_key HAVING count(*) > 1.
2. If the duplicates have different helius_id: backfill collision — safe, the merchant should see the same delivery twice but with the same idempotency key (their verifier should dedup).
3. If the duplicates have the same helius_id: dispatcher bug. Page protocol team.

Fix:

• For backfill collisions: educate merchants to dedup on the idempotency key (this is documented in vela-docs).
• For dispatcher bugs: hotfix and replay affected dispatches.

Ordering Issues

Vela does not guarantee strict ordering of webhook deliveries. Merchants must reconcile via the on-chain timestamp in each event.

If a merchant complains about ordering:

1. Confirm the events were emitted in the expected order on-chain (event.timestamp field).
2. Confirm both deliveries have correct payloads.
3. Educate the merchant: deliveries are at-least-once, unordered. This is documented in vela-docs.

If ordering correctness becomes a frequent merchant ask, escalate as a product decision (the protocol could grow a strict-ordering mode).

Queue Backlog

Detect: webhook_dispatch queue depth growing in admin console; consumer is processing but not catching up.

Triage:

1. Check Cloudflare Queues dashboard for the queue (vela-webhook-events-dev, vela-webhook-dispatch-dev).
2. Worker CPU pressure? Check Cloudflare Workers analytics.
3. Stuck on a single failing endpoint? Sometimes one slow merchant blocks the consumer.

Fix:

• Increase consumer concurrency in wrangler.toml (max_batch_size, max_concurrency).
• Quarantine the problematic endpoint to a separate queue lane (admin console → endpoint → Quarantine).
• If the queue is at the platform limit, shard by hash of merchant ID across multiple consumer Workers.

Manual Replay Procedure

# Single dispatch
bunx vela admin webhook replay --dispatch-id <UUID>

# All failed deliveries for a merchant in a window
bunx vela admin webhook replay-failed --merchant <ID> --since <ISO> --until <ISO>

# Backfill from on-chain (when Helius missed the event)
cd vela-dashboard
bun run scripts/backfill-events.ts --from-signature <SIG>

Reconciliation Cron (background)

A scheduled reconciliation job runs every hour in the dashboard worker:

1. For each active mandate in D1, read the on-chain Mandate.pull_count.
2. Compare with the count of billing_events of type pull.executed in D1.
3. On divergence > 0: emit an alert + auto-trigger backfill from the last known good signature.

This is the safety net for missed Helius deliveries. It runs hourly, not real-time, so don't rely on it for near-real-time correctness — but it catches drift overnight.

Related

• Helius webhook configuration: Helius Infrastructure →
• Webhook verifier package: vela-webhook →
• Keeper-side failures: Keeper Operations →
• Database constraint failures: Database Migrations →