Keeper Operations

How VelaPay's keepers work, how to monitor them, how to recover when they fail, and how to rotate the keeper authority. Keepers are the cron-driven Workers that execute scheduled pulls and stream settlements on behalf of merchants — when keepers stop, billing stops.

Keeper Inventory

Keeper	Cadence	What it does	Worker config
Periodic keeper	Per-period	Executes execute_pull for active periodic mandates whose period is due	vela-dashboard/worker/wrangler.toml (main worker)
Stream keeper	Higher frequency	Settles StreamMandate accrued debt via execute_stream	vela-dashboard/worker/wrangler.stream-keeper.toml (separate worker, separate cadence)
Synthetic worker	Every 15 min	Drives a full subscribe → pull → cancel funnel against staging — not a real keeper but the same operational tier	vela-synthetic/wrangler.toml

All keepers are authorized via ProtocolConfig.keeper_authority. The transfer hook fails closed for any non-authorized signer.

How Keepers Decide What to Pull

Each cron firing reads from D1 (the dashboard's authoritative billing-event store), filtered to mandates whose next_pull_at ≤ now. For every candidate:

1. Pre-flight check — is the mandate still active on-chain? Has it been cancelled since D1 was last updated?
2. Arcium MXE round-trip (default path since v1.1) — request validation; receive encrypted PullApproval parameters.
3. execute_pull instruction — submit the transaction.
4. Result handling — on success, advance next_pull_at and write a billing event. On typed error, log and surface in the admin console.

The stream keeper differs only in step 3: it submits execute_stream and runs the settle-then-mutate invariant.

Monitoring

Keepers are healthy when:

Signal	Healthy value
Cron last-run timestamp	Within the last cadence window
Pull success rate (rolling 24h)	> 99%
Arcium round-trip latency p95	< 5s
ArciumUnavailableEvent rate	~0
D1 billing_events ingestion lag	< 30s after on-chain confirmation

These signals are visible in the admin console under Protocol → Keeper Health. Page on:

• Cron worker silent for > 2 cadence windows (no last_run_at update).
• Pull success rate < 95% over a rolling hour.
• ArciumUnavailableEvent count > 5 per hour.

Common Failure Modes

Arcium unavailable

Symptom: rising ArciumUnavailableEvent rate; pulls failing with ArciumUnavailable typed error.

Action:

1. Check Arcium status (their dashboard / status page).
2. If transient: keeper retries on next cycle — no operator action needed.
3. If sustained > 30 minutes: page the protocol team. Coordinate with Arcium.
4. Optional: enable the Phase 0 fallback path for affected mandates if the outage is expected to last > 1 hour and merchants opt in.

Subscriber insufficient balance

Symptom: InsufficientFundsError typed errors. Often part of dunning.

Action: Already routed by the dashboard's dunning pipeline — sends dunning template via vela-mail, schedules retry on the merchant's configured cadence. No keeper-level fix needed.

Pull executed too early

Symptom: PullTooEarlyError. Usually a clock-skew bug between D1's next_pull_at and the on-chain Mandate.period_end.

Action:

1. Open admin console → mandate detail.
2. Compare D1 next_pull_at against on-chain Mandate.period_start + plan.period.
3. If D1 is wrong: trigger the reconciliation flow (admin console → Reconcile mandate).
4. If on-chain is wrong: investigate — this is a protocol-side bug.

Keeper authority compromised

Symptom: unauthorized pulls observed on-chain, or the keeper keypair is suspected leaked.

Action: see Keeper Authority Rotation below. Pause the protocol first.

Keeper Worker quota exceeded

Symptom: Cloudflare Worker logs show CPU-time or daily-request quota errors.

Action:

1. Inspect the dashboard worker's CPU profile in Cloudflare.
2. The most common cause is an Arcium round-trip pile-up — adjust concurrency limits in the worker config.
3. If genuinely at quota, upgrade the Cloudflare plan or shard the keeper across multiple workers.

Manual Pull Execution

When a keeper is down and a pull must happen now:

cd vela-sdk
bunx vela pull --mandate <PUBKEY> --as-keeper

The CLI does the same thing the keeper Worker does, signed by the local keeper keypair (must be the keeper authority). Use only when the Worker is incapacitated.

Recovery From Missed Periods

If the keeper was down for several pull windows:

1. Read the affected mandates from D1: SELECT * FROM mandates WHERE next_pull_at < strftime('%s','now') - 3600.
2. For each, decide whether to:
   • Pull now (charge the subscriber for the missed period — usually the merchant's preference).
   • Skip (forgive the missed period).
   • Pull and refund (charge then immediately refund — useful for accounting clarity).
3. Use the bulk-replay tool in the admin console (or bunx vela pull --batch <file>).
4. Confirm each replay results in a webhook event in D1 — see Webhook Troubleshooting → if not.

Keeper Authority Rotation

Required on operator turnover or compromise. Coordinate with the protocol team — this changes a global protocol parameter.

1. Generate a new keeper keypair on the new operator's machine.
2. Pause the protocol (optional but recommended for compromise — it stops all pulls during the rotation):

   bunx vela admin pause

3. Update ProtocolConfig.keeper_authority via the admin instruction:

   bunx vela admin update-keeper-config --new-authority <PUBKEY>

4. Deploy the keeper Worker with the new keypair as a Worker secret:

   bunx wrangler secret put KEEPER_KEYPAIR
   bunx wrangler deploy

5. Resume the protocol:

   bunx vela admin unpause

6. Run vela-synthetic once to confirm end-to-end the new authority works.
7. Update Environment Registry with the new keeper authority pubkey.
8. Securely retire the old keypair.

Pre-flight Before Each Release

Before deploying any release that touches the dashboard worker, the SDK, or the protocol:

• [ ] Synthetic worker green (15-min cron passing).
• [ ] No ArciumUnavailableEvent in the last hour.
• [ ] Pull success rate ≥ 99% in last 24h.
• [ ] No D1 ingestion lag > 30s in last hour.
• [ ] Manual pull via CLI succeeds against staging.

Related

• Verification Matrix →
• Webhook Troubleshooting →
• Incident Response →
• Architecture / Billing Flows →