Protocol Deployment Runbook

Comprehensive runbook for deploying and upgrading Vela protocol programs on Solana without losing program identity, drifting config, or breaking downstream consumers.

This page exists because the biggest operational risk for vela-protocol is not ordinary code shipping. It is program identity drift:

• losing the program keypair and silently creating a new program ID
• publishing SDK/docs/apps against the wrong mainnet address
• rotating one program (vela-transfer-hook) without updating the other (vela-protocol)
• confusing the program keypair with the upgrade authority wallet

For Vela, protocol deployment is a ceremony, not a casual CLI step.

---

Scope

This checklist applies to:

• vela_protocol main program
• vela_transfer_hook transfer hook program
• any future Vela program deployed in the same operational model

This page covers:

• key custody
• devnet upgrade rehearsal
• build and deploy preparation
• first mainnet deployment
• later upgrades
• downstream propagation into SDK/docs/apps
• recovery expectations

---

Non-Negotiable Rules

1. Mainnet program keypairs are generated once and treated as permanent identities.
2. Program keypairs are never stored only in target/deploy/.
3. Mainnet deploys are never performed from an unprepared laptop session with ad hoc keys.
4. declare_id!(), Anchor.toml, and checked-in program-ID manifests must match the intended mainnet public keys before deploy.
5. vela_protocol and vela_transfer_hook are separate programs with separate identities and must be tracked independently.
6. Upgrade authority custody must be explicitly planned and documented before launch.
7. No mainnet deploy is complete until downstream repos are updated and verified.

---

Current Devnet Deployment

The current devnet program identities are:

Program	Devnet program ID
vela_protocol	CVM6UqbwKgHckZzm8R2qbN3BWhCTdk1PsSeEQLchkwKT
vela_transfer_hook	3agVoFp4NZFuKbVqCV8HbjSZn1xW4Utk4U1Wir3TKjZ9

The current devnet upgrade authority wallet is:

EkyMDqChwLDyx8XUxEFB7Ngvk9Yfp2DGEJ6VzqqbgErD

The expected devnet program keypair location is:

~/.config/velapay/keys/devnet/vela_protocol-keypair.json
~/.config/velapay/keys/devnet/vela_transfer_hook-keypair.json

Those keypair files define the program addresses. The deployer wallet does not.

WARNING

Never regenerate these files to "fix" a deployment. If a keypair is missing, stop and restore from backup. A newly generated program keypair means a new program address.

---

Deployment Types

There are three different operations. Do not blur them.

Operation	Address changes?	When to use
Program upgrade	No	Normal protocol changes using the same program keypair and upgrade authority
Fresh deployment	Yes	New environment, lost pre-launch identity, or intentional new program ID
IDL-only update	No	Only when on-chain program bytes already match the intended code and the IDL needs publishing

Most Vela protocol releases should be program upgrades, not fresh deployments.

DANGER

Do not use a fresh deployment to "upgrade" production unless the intent is to change every downstream program ID reference. Fresh deployment is a breaking ecosystem event.

---

Devnet Safe Upgrade Path

The devnet upgrade path lives in vela-protocol.

Use:

cd /Users/laitsky/Developments/vela-labs/vela-protocol

bun run verify:rollout
bun run upgrade:devnet
bun run verify:deployed

Equivalent alias:

bun run deploy:devnet

The safe upgrade script performs these checks before and after deployment:

Gate	Purpose
Program ID manifest check	Confirms source IDs and config/program-ids.json agree
Program keypair restore	Copies saved keypairs into target/deploy
IDL sync check	Ensures SDK IDLs match protocol IDLs
Upgrade authority check	Confirms the current wallet can upgrade the deployed programs
SDK checks	Type-checks SDK and verifies IDL/event parser compatibility
Webhook checks	Verifies event schemas, fixtures, and empty token_symbol handling
Dashboard worker checks	Verifies webhook ingestion, fanout, and stream event processing
Checkout/widget checks	Verifies transaction-building consumers compile
Byte hash verification	Dumps deployed program bytes and compares them with local .so files
On-chain IDL verification	Fetches deployed IDLs and compares them with local IDL artifacts

WARNING

SKIP_CONSUMER_CHECKS=1 bun run verify:rollout is for diagnostics only. Never use it as a release gate.

---

Devnet Upgrade Checklist

Use this every time a protocol change touches instruction accounts, account layout, event shape, token semantics, or transfer-hook behavior.

1. Freeze the protocol interface

Before deployment, confirm:

• final Rust code is built from the intended commit
• declare_id!() values are unchanged unless intentionally rotating IDs
• config/program-ids.json contains the expected devnet IDs
• generated IDLs in target/idl are final
• copied SDK IDLs match target/idl

2. Confirm key and wallet state

Run:

solana config get
solana-keygen pubkey
solana address -k ~/.config/velapay/keys/devnet/vela_protocol-keypair.json
solana address -k ~/.config/velapay/keys/devnet/vela_transfer_hook-keypair.json

Expected:

• RPC URL is devnet
• wallet pubkey is the current upgrade authority
• program keypair addresses match the current devnet program IDs

Abort if any of those are wrong.

3. Run compatibility gate

bun run verify:rollout

This must pass before deploying.

If it fails:

• do not deploy
• identify the repo that failed
• update that repo or the SDK/IDL
• rerun the gate from the beginning

4. Upgrade devnet

bun run upgrade:devnet

The script upgrades both deployable units and publishes/verifies IDLs.

5. Prove deployed bytes and IDLs

bun run verify:deployed

This must report byte-hash matches for both programs and IDL matches for both IDL accounts.

6. Runtime smoke check

At minimum, run a production-like devnet smoke flow:

1. create a wrapped-USDC periodic plan
2. create a non-USDC periodic plan, such as PYUSD or EURC
3. subscribe to each plan
4. execute one pull for each path
5. create and settle a non-USDC stream mandate
6. inspect dashboard/webhook events

Expected:

• selected billing mint is enforced
• mint is correct in emitted events
• token_symbol is preserved as raw display metadata
• consumers accept empty token_symbol for stream events
• no consumer rewrites non-USDC events to sUSDC

---

Multi-Repo Rollout Order

For protocol changes that touch public interfaces, use this order.

Blocking release pair

1. vela-protocol
2. vela-sdk

The SDK is the compatibility adapter for every app and worker. Do not upgrade runtime consumers against a protocol IDL that the SDK has not absorbed.

Same rollout window

3. vela-dashboard
4. vela-webhook

These consume events and/or build runtime instructions.

Immediately after runtime rollout

5. vela-docs
6. vela-demo

These keep external examples and developer education aligned.

Audit and patch before broad release

7. vela-admin
8. vela-checkout
9. vela-widget
10. vela-portal

These can contain direct transaction-building flows, generated IDs, or UI token label assumptions.

---

Change Impact Matrix

Use this to know which repos probably need changes.

Protocol change	Repos to review
Instruction args/accounts	vela-sdk, vela-dashboard, vela-checkout, vela-widget, vela-demo
Account layout	vela-sdk, vela-dashboard, vela-admin, indexers, docs
Event fields or encoding	vela-sdk/events, vela-webhook, vela-dashboard, vela-docs
Program IDs	every repo with generated IDs, IDLs, explorer links, or env config
Token semantics	vela-sdk, vela-dashboard, vela-checkout, vela-widget, vela-portal, docs
Stream behavior	vela-sdk, vela-dashboard, vela-webhook, docs
Pull/settlement behavior	SDK, keepers, dashboard worker, webhook consumers
CLI surface	vela-sdk, vela-docs, vela-demo

When unsure, search all repos:

cd /Users/laitsky/Developments/vela-labs

rg -n "CVM6UqbwKgHckZzm8R2qbN3BWhCTdk1PsSeEQLchkwKT|3agVoFp4NZFuKbVqCV8HbjSZn1xW4Utk4U1Wir3TKjZ9|billingMint|token_symbol|wrappedUsdcMint|sUSDC|UNKNOWN" \
  vela-{protocol,sdk,dashboard,webhook,checkout,widget,portal,demo,docs,admin}

Then classify hits:

• legitimate compatibility path
• generated ID or IDL reference
• stale assumption that needs patching
• test fixture that should be updated

---

Multi-Token Rollout Rules

The current multi-token protocol behavior has two canonical rules:

1. billingMint / TokenConfig decides settlement token.
2. Event mint is authoritative; token_symbol is display metadata only.

Consumer requirements:

• plan creation must accept optional billingMint
• omitted billingMint means wrapped-USDC compatibility default
• subscription must resolve the plan billing mint
• non-USDC periodic pulls must pass the selected billing mint
• stream events must not hardcode sUSDC
• empty token_symbol is valid for non-USDC stream events
• downstream systems may enrich display labels from mint maps, but must not overwrite protocol event truth

DANGER

Never key accounting, settlement, reconciliation, or authorization off token_symbol. Use mint.

---

Critical Concepts

Program Keypair vs Upgrade Authority

These are not the same thing.

Program keypair

• Defines the program's public key, which becomes the program ID.
• If this keypair changes, the program ID changes.
• If this keypair is lost before a deploy flow that relies on it, you cannot recreate the same program ID.

Upgrade authority

• Controls whether the deployed upgradeable program can be upgraded later.
• Usually a wallet, multisig, or governance-controlled authority.
• Can remain intact even if you no longer have a convenient local copy of the deploy keypair.

Operational takeaway

You must protect both:

• the program keypair, because it anchors identity
• the upgrade authority, because it anchors control

---

EVM Mental Model

This section exists for engineers coming from EVM, because Solana program identity behaves differently from Ethereum contract deployment.

What feels familiar

• there is still a deployer/operator wallet
• there is still an admin-style authority for later upgrades
• there is still a deployment ceremony where code is built, published, and then referenced by SDKs and apps

What is different

In EVM, contract addresses are typically derived from:

• deployer address + nonce, or
• CREATE2 inputs

In Solana upgradeable programs, the program's identity is tied to a program keypair.

That means:

• your normal wallet from solana-keygen pubkey is not the program ID
• the program ID comes from the program keypair JSON
• if a different program keypair is used, a different program ID is produced

The two identities to keep separate

1. Program keypair

• file like vela_protocol-keypair.json
• defines the program address
• should be thought of as the identity anchor for the deployed program

2. Wallet / deployer / upgrade authority

• your normal Solana wallet
• pays for deployment transactions
• may also hold upgrade authority
• is not the same thing as the program keypair

Practical mapping for EVM developers

EVM concept	Solana concept	Notes
Deployer EOA	Operator wallet / upgrade authority wallet	Pays for deploys, may control upgrades
Contract address	Program ID	Public address other systems integrate with
Address derivation from deployer + nonce	Program ID from program keypair	Different keypair means different program ID
Proxy admin / owner	Upgrade authority	Controls upgrades for upgradeable programs

Important operational consequence

If you come from EVM, the easiest mistake is assuming:

"my wallet determines the deployed program address."

For Vela on Solana, that is not true.

The safer mental model is:

• wallet key = who is allowed to deploy or upgrade
• program keypair = what address the program lives at

Why builds can surprise people

Some Solana/Anchor workflows will create deploy keypairs automatically if they do not already exist in the expected location.

That means a build or deploy flow can accidentally do this:

1. fail to find the original program keypair
2. generate a fresh one
3. build or deploy against a new program identity

This is why Vela treats program keypairs as permanent, backed-up artifacts instead of disposable build output.

---

Program Inventory

Maintain a tracked table like this before mainnet launch:

Program	Purpose	Mainnet Program ID	Program Keypair Location	Upgrade Authority	Status
vela_protocol	Billing primitive	TBD	Secure storage	TBD	Pre-launch
vela_transfer_hook	Token-2022 enforcement hook	TBD	Secure storage	TBD	Pre-launch

This table should live in private ops docs and be updated after every mainnet ceremony.

---

Key Custody Practice

Required storage layout

Keep three copies of each mainnet program keypair:

1. Working copy

   • Secure operator machine or controlled deployment workstation
   • Used only for build/deploy preparation

2. Encrypted cloud backup

   • Example: 1Password Secure Document or encrypted archive in private cloud storage
   • Must be encrypted before upload if stored outside a secret manager

3. Offline backup

   • Encrypted USB or hardware-backed offline archive
   • Stored separately from the working machine

Required files

At minimum:

• vela_protocol-keypair.json
• vela_transfer_hook-keypair.json

Do not do this

• do not commit keypairs to git
• do not leave them only in target/deploy/
• do not keep only one copy
• do not store raw, unencrypted keypair JSON in a random cloud folder
• do not assume a past deploy machine will always remain available

---

Suggested Storage Convention

Local operator path:

~/.config/velapay/keys/mainnet/vela_protocol-keypair.json
~/.config/velapay/keys/mainnet/vela_transfer_hook-keypair.json

Optional environment override for scripts:

VELA_PROGRAM_KEY_DIR=/secure/path/to/mainnet-keys

The exact path is less important than the discipline:

• stable
• documented
• backed up
• not disposable

---

Pre-Mainnet Checklist

Complete all of these before generating final mainnet keys or deploying anything.

Protocol readiness

• all protocol features intended for initial launch are frozen
• account layouts are reviewed for upgrade compatibility
• transfer-hook account dependencies are reviewed
• ExtraAccountMetaList capacity is reviewed for future needs
• failure modes are documented and fail-closed behavior is confirmed

Testing readiness

• Rust tests pass
• TypeScript integration tests pass
• devnet deploy path has been exercised end to end
• upgrade rehearsal on devnet has been performed at least once
• SDK and docs are synced to the latest devnet/protocol shape

Operational readiness

• designated deploy operator is known
• upgrade authority holder is known
• recovery path is documented
• downstream repo owners know the post-deploy propagation steps
• rollback expectations are documented

Key readiness

• final mainnet program keypairs are generated intentionally
• public keys are recorded in private ops notes
• all three backups exist
• restore drill has been performed at least once

---

Mainnet ID Freeze Process

Before first deploy, freeze the intended mainnet IDs.

1. Generate final mainnet program keypairs once.
2. Record their public keys.
3. Update:
   • declare_id!()
   • Anchor.toml
   • protocol-owned program ID manifest
4. Run repo-local checks to ensure code and config match.
5. Back up the keypairs before proceeding.

Do not begin deploy prep if any of those are still ambiguous.

---

First Mainnet Deployment Ceremony

Treat first deployment as a checklist-driven ceremony.

1. Prepare the environment

• use a clean, trusted machine session
• confirm Solana CLI, Anchor, Rust, Bun, and Arcium versions
• confirm target cluster is mainnet, not devnet
• confirm wallet being used for upgrade authority steps is the intended wallet

2. Restore the correct mainnet program keypairs

• copy the saved keypairs into the workspace deploy location
• verify derived addresses match the intended mainnet program IDs

3. Verify checked-in source matches the intended IDs

Confirm:

• programs/vela-protocol/src/lib.rs
• programs/vela-transfer-hook/src/lib.rs
• Anchor.toml
• config/program-ids.json or equivalent manifest

4. Build artifacts

• run Arcium build if required
• run Anchor build
• verify output bytecode corresponds to the intended program IDs

5. Deploy in the intended order

Recommended order:

1. vela_transfer_hook
2. vela_protocol

Reason:

• the main program may reference the hook ID or depend on hook behavior
• deploying the hook first reduces identity mismatch risk during first launch

6. Verify on-chain state immediately

Check:

• deployed program IDs
• upgrade authority
• executable state
• IDL publication/upgrade if used
• hook-related account expectations

7. Record ceremony output

Write down:

• exact deploy date/time
• operator
• machine used
• git commit SHA
• deployed program IDs
• upgrade authority
• verification commands/results

No mainnet deploy should rely on memory.

---

Upgrade Ceremony (Post-Launch)

Mainnet upgrades should follow the same discipline, but they are not identical to first deploys.

Before upgrade

• confirm the existing deployed program ID
• confirm the current upgrade authority
• confirm the new build is based on the same intended program identity
• verify no unintended declare_id!() drift occurred
• confirm downstream repos that will need updates if IDLs or behavior changed

During upgrade

• restore the same saved program keypairs used for the canonical identity
• build from a clean, tagged, or otherwise controlled commit
• use the safe upgrade flow, not an ad hoc anchor deploy
• if using buffer-based upgrade flow, record buffer address and final upgrade transaction

After upgrade

• verify on-chain program still has the expected program ID
• verify upgrade authority was not changed unintentionally
• refresh IDL if required
• run downstream smoke checks

---

Downstream Propagation Checklist

Mainnet deployment is not done when the on-chain upgrade succeeds. It is done when all downstream consumers are aligned.

Update and verify:

• vela-sdk
  • checked-in IDLs
  • program ID constants/manifests
  • tests referencing concrete program IDs
• vela-docs
  • mainnet reference pages
  • quickstart examples
  • explorer links
• vela-admin
  • protocol constants/config
  • environment/config variables
• vela-dashboard
  • generated constants or runtime config
• any service or worker that builds transactions, validates accounts, or links to explorer pages

For every downstream repo, answer:

• does it reference the protocol program ID?
• does it reference the transfer hook program ID?
• does it embed an IDL that changed?
• does it link to explorer pages or operational docs using the old value?

---

Post-Deploy Verification Checklist

Run all of these after mainnet deploy/upgrade:

• verify vela_protocol program ID on chain
• verify vela_transfer_hook program ID on chain
• verify upgrade authority on both
• verify any hook-specific config or extra account meta setup
• verify SDK still constructs transactions with the intended IDs
• verify docs build with updated references
• verify admin/dashboard builds still pass
• verify at least one dry-run or smoke flow on mainnet infrastructure assumptions

Suggested smoke surface:

• protocol account fetch
• plan read path
• mandate address derivation
• transfer-hook-aware instruction construction

---

Incident Scenarios

Scenario: program keypair lost, no deploy yet

Impact:

• must generate a new keypair
• mainnet program ID changes before launch

Action:

• update manifests, source IDs, SDK/docs/constants
• restart the checklist with the new canonical IDs

Scenario: program keypair lost after deploy, upgrade authority still intact

Impact:

• existing deployed program may still be upgradeable
• identity recreation from scratch becomes riskier

Action:

• do not improvise
• document current on-chain authority state
• confirm whether the planned upgrade flow truly requires the saved program keypair
• restore from backup immediately

Scenario: wrong program ID published downstream

Impact:

• clients and operators may point at the wrong program

Action:

• stop publish/release propagation
• correct SDK/docs/app constants
• rebuild and verify all consumer repos
• publish a clear ops note internally

Scenario: transfer hook updated, protocol references stale hook ID

Impact:

• pull execution can fail or behave inconsistently

Action:

• update protocol-owned constants and manifests
• rebuild protocol
• refresh SDK IDLs/constants
• verify docs/admin/dashboard alignment

---

Things We Must Never Do on Mainnet

• never treat target/ as the source of truth for identity
• never generate “temporary” mainnet keypairs
• never run raw deploy commands without checking active cluster and intended wallet
• never upgrade one of the two programs and assume the other repo surfaces do not need review
• never ship SDK/docs/apps before program IDs and IDLs are reconciled
• never leave upgrade authority custody undocumented

---

Minimal Mainnet Runbook

This is the condensed version operators should memorize:

1. Restore the correct saved mainnet program keypairs.
2. Verify program IDs from those keypairs.
3. Verify declare_id!(), Anchor.toml, and manifest match.
4. Build from a controlled commit.
5. Deploy/upgrade in the intended order.
6. Verify on-chain IDs and upgrade authority.
7. Update SDK/docs/apps.
8. Record the ceremony in ops notes.

---

Vela-Specific Recommendations

For Vela, the most important practical controls are:

• keep vela_protocol and vela_transfer_hook keypairs backed up separately and clearly labeled
• maintain a protocol-owned canonical manifest for mainnet IDs
• use guarded build/deploy scripts, not one-off commands
• make downstream repo propagation part of the deploy checklist, not an afterthought
• rehearse the full ceremony on devnet first

If the team does those five things consistently, mainnet deployment risk drops substantially.