Incident Response

This page defines how Vela Labs responds when something breaks.

The priority order is:

1. protect subscriber funds
2. preserve protocol integrity
3. stop bad writes or bad pulls
4. restore customer-facing availability
5. preserve evidence
6. communicate clearly

Severity Levels

Severity	Definition	Examples	Response
SEV-0	Funds at risk or unauthorized pulls possible	Transfer hook bypass, wrong mint enforcement, compromised upgrade authority	Stop releases, pause if available, assemble incident channel immediately
SEV-1	Billing correctness or protocol availability broken	Pulls fail globally, webhook fanout corrupts settlement data, SDK builds wrong instructions	Freeze deploys, mitigate, targeted hotfix
SEV-2	Major product flow unavailable	Checkout cannot subscribe, dashboard cannot load plans, portal cannot cancel	Fix in owning app, verify adjacent flows
SEV-3	Degraded non-critical function	Docs broken, analytics delayed, UI display issue	Normal priority with tracked follow-up

First Five Minutes

1. Name the incident and severity.
2. Stop related deploys.
3. Identify affected surface: protocol, SDK, dashboard, webhook, checkout, widget, portal, admin, docs.
4. Decide whether protocol pause/emergency controls are needed.
5. Preserve logs, transaction signatures, payloads, and commit SHAs.

Do not start broad refactors during incident response.

Protocol Incident Checklist

Use for SEV-0 or SEV-1 protocol concerns.

• [ ] Confirm cluster and program IDs.
• [ ] Confirm whether issue is deployed program, IDL, SDK, or consumer misuse.
• [ ] Fetch relevant transactions and logs.
• [ ] Check upgrade authority status.
• [ ] Check whether emergency pause is available and appropriate.
• [ ] Run bun run verify:deployed.
• [ ] Run targeted failing test or reproduction.
• [ ] Prepare hotfix artifact from controlled commit.
• [ ] Run bun run verify:rollout.
• [ ] Upgrade only after verification passes or emergency authority approves.

Webhook/Event Incident Checklist

Use when downstream systems reject, rewrite, or lose events.

• [ ] Capture raw event payload.
• [ ] Validate with @velapay/sdk/events.
• [ ] Validate with @velapay/webhook.
• [ ] Check dashboard queue event processor.
• [ ] Check DLQ and retry schedule.
• [ ] Confirm HMAC secret rotation state.
• [ ] Confirm mint and token_symbol handling.
• [ ] Replay from raw payload if safe.

Checkout/Widget/Portal Incident Checklist

Use when users cannot create or manage mandates.

• [ ] Identify whether failing path is wrapped-USDC or non-USDC Token-2022.
• [ ] Confirm the plan has the expected billingMint.
• [ ] Confirm TokenConfig is enabled.
• [ ] Confirm wallet signs the expected transaction.
• [ ] Confirm browser-safe SDK exports are current.
• [ ] Confirm D1 session state is not stale.
• [ ] Reproduce with a local build if possible.

Admin Incident Checklist

Use when emergency controls, config, or monitoring break.

• [ ] Confirm admin auth wallet matches configured authority.
• [ ] Confirm service binding to dashboard Worker.
• [ ] Confirm protocol config deserialization.
• [ ] Confirm audit log write path.
• [ ] Do not bypass audit logging unless funds are actively at risk.

Communication Template

# Incident <ID>: <short title>

Severity:
Started:
Detected by:
Owner:

## Impact

## Current Status

## What We Know

## What We Are Doing

## Customer/User Communication

## Follow-Up

Post-Incident Review

Within 24 hours for SEV-0/SEV-1, write:

• timeline
• root cause
• blast radius
• what detected it
• what mitigated it
• what failed in process
• tests or monitors added
• docs/runbooks updated

Emergency Controls

Emergency controls exist to protect users, not to make debugging convenient.

Use emergency controls when:

• unauthorized pulls are plausible
• wrong token mint can be pulled
• transfer hook behavior is suspect
• keeper behavior can cause repeated bad writes
• webhook replay can duplicate external side effects

Do not use emergency controls when:

• issue is a cosmetic UI bug
• docs are stale
• one isolated user has a recoverable checkout failure
• root cause is not understood and emergency action would increase ambiguity

Evidence to Preserve

• transaction signatures
• slot numbers
• raw webhook payloads
• queue message IDs
• D1 rows related to the incident
• program byte hashes
• IDL hash/commit SHA
• SDK package version or local commit
• dashboard/worker deployment identifier
• screenshots only when they clarify user impact