Arcium MXE Operations

The operational guide for VelaPay's Arcium integration. Arcium runs the encrypted-compute layer that validates billing on ciphertext — it is on the critical path for every keeper-initiated pull since v1.1. When Arcium is unhappy, billing is unhappy.

This page is the operational companion to Decisions / Why Arcium. The decision page explains why we chose Arcium; this page explains how to keep it running.

What Arcium Does for VelaPay

Every keeper-initiated pull (default production path) follows this round-trip:

Keeper                          Arcium MXE                   Vela on-chain
   │                                                              │
   │   1. request_validation                                      │
   │   (encrypts mandate inputs)                                  │
   │   ─────────────────────────►                                 │
   │                                                              │
   │                       2. circuit evaluates                   │
   │                       on ciphertext                          │
   │                                                              │
   │   3. encrypted callback                                      │
   │   (PullApproval params)                                      │
   │   ◄─────────────────────────                                 │
   │                                                              │
   │   4. arcium_callback writes PullApproval ──────────────────► │
   │                                                              │
   │   5. execute_pull → transfer_checked → hook validates ─────► │

If Arcium is unavailable at step 2 or 3, the round-trip fails closed: no PullApproval is written, no transfer happens, an ArciumUnavailableEvent is emitted on-chain.

Components

Component	Where	Owner
Arcium cluster	Arcium-managed (cloud)	Arcium team
Vela MXE circuits	vela-protocol/arcium/ (TBD path — confirm in repo)	Protocol team
Arcium client	@arcium-hq/client 0.9.3 (in vela-sdk and consumer apps)	VelaPay
Arcium.toml	vela-protocol/Arcium.toml	Protocol team
Cluster offset	Worker env (ARCIUM_CLUSTER_OFFSET=456 for devnet)	VelaPay

Health Signals

Signal	Healthy value	Unhealthy threshold
Round-trip latency p50	< 2s	> 5s
Round-trip latency p95	< 5s	> 15s
ArciumUnavailableEvent rate	~0/hr	> 5/hr
Circuit version mismatch errors	0	any
Encrypted callback verification failures	0	any

These signals surface in the admin console under Protocol → Arcium Health.

Common Failure Modes

Cluster outage

Symptom: All pulls failing with ArciumUnavailable. request_validation returns network errors.

Action:

1. Confirm with Arcium status dashboard / status channel.
2. If transient (< 30 min): keepers retry on next cycle. No operator action.
3. If sustained (> 30 min): page the protocol team. Coordinate with Arcium engineering.
4. Communicate to merchants if billing is materially delayed.

Circuit version mismatch

Symptom: Round-trip succeeds but the program rejects the callback with a typed circuit-version error.

Cause: A circuit was deployed without bumping the program-side expected-version constant, or vice versa. A coordination bug.

Action:

1. Identify which side is stale (read program-side expected_circuit_version, read deployed circuit version from Arcium).
2. Bring them into alignment via the side that needs to update (deploy new circuit, or upgrade program).
3. Pause the protocol during the rollout if the mismatch causes merchant-visible failures.

Callback verification failure

Symptom: Encrypted callback received but the cryptographic verification fails on-chain.

Cause: This should not happen with correctly-deployed circuits. If it does, treat as a security incident — the callback authenticity contract is the foundation of fail-closed.

Action:

1. Pause the protocol immediately.
2. Page the protocol team and Arcium.
3. Investigate root cause with full logs preserved.
4. Do not unpause until the verification path is restored and an audit confirms no spurious approvals were accepted.

Decryption failure (merchant-side billing record)

Symptom: Merchant cannot decrypt their billing records via the dashboard.

Cause: The merchant's decryption key was not properly captured at subscribe time, or they have rotated keys.

Action:

1. Confirm whether the merchant has a current valid key.
2. If lost: billing records cannot be retroactively decrypted (this is by design — Arcium never has access to decrypted state). Only the merchant can read them.
3. Document the lost-key recovery limitation in merchant onboarding so this surprise doesn't recur.

Cluster offset misconfigured

Symptom: All requests fail with cluster-not-found errors.

Cause: Wrong ARCIUM_CLUSTER_OFFSET in Worker env (devnet uses 456; mainnet uses a different value).

Action: Update env, redeploy worker.

Circuit Deployment

Circuits are versioned alongside the protocol program. Coordinated rollout:

1. Author the circuit in vela-protocol/arcium/circuits/.
2. Test locally with the Arcium localnet (Cerberus backend), specified in Arcium.toml.
3. Bump expected_circuit_version constant in vela-protocol.
4. Deploy circuit via Arcium CLI.
5. Upgrade program with new expected_circuit_version.
6. Verify via the synthetic worker that the new round-trip succeeds.
7. Update Environment Registry with the new circuit version.

Either side deployed without the other = mismatch error. Always coordinate.

Local Development

# Start Arcium localnet (in vela-protocol)
arcium localnet start

# Deploy circuits to localnet
arcium localnet deploy

# Run program tests against localnet
anchor test --skip-local-validator

Local tests use the localnet cluster offset (also documented in Arcium.toml). The Arcium localnet is dockerized — first startup takes a few minutes.

Pre-Release Checklist (Arcium-touching releases)

• [ ] Synthetic worker green against the new circuit version on staging for ≥ 1 hour.
• [ ] Round-trip latency p95 < 5s on staging.
• [ ] Zero callback verification failures on staging.
• [ ] Circuit version coordination doc updated.
• [ ] Rollback plan documented (previous circuit version still deployable).

Privacy Posture (operational view)

Data	Visible to operator	Visible to Arcium nodes	Visible to validators
Mandate plaintext inputs (amount, period)	Pre-encryption only (operator-side memory before send)	No (only ciphertext is shipped to MXE)	No
MXE intermediate state	No (computed on shares across nodes)	No (split across nodes; no single node sees plaintext)	No
PullApproval parameters	After callback decryption	Encrypted form only	Encrypted form only
Merchant billing records	Only the merchant (who holds the decryption key)	No	No
ArciumUnavailableEvent (the metadata that an outage occurred)	Yes (intentional)	Yes	Yes

Related

• Decision: Why Arcium →
• Architecture: Architecture / Billing Execution Path →
• Keeper integration: Keeper Operations →
• Incident response: Incident Response →