Monitoring and runbooks

Endpoint
GET /health	Per-chain request counts by status
GET /metrics	Prometheus metrics

Key metrics:

Metric	Meaning
verity_fulfilled_total, verity_fulfilments_submitted_total	Throughput
verity_requests{status=…}	Local database by status (pending, submitted, fulfilled, closed, expired, failed)
verity_fulfil_latency_blocks (summary)	Request → our fulfilment, in blocks
verity_tx_account_balance_wei, verity_low_balance	Gas money
verity_last_tick_timestamp_seconds, verity_tick_errors_total	Liveness and RPC health
verity_blockhashes_stored_total, verity_expired_total	Old-request handling

The Docker stack ships a Grafana dashboard and five Prometheus alert rules: low balance, node stalled, requests stuck, expired requests, and failures. Independently of Prometheus, the node posts a low-balance alert to alert_webhook_url (Slack and Discord compatible).

Runbooks

In verity-node/docs/runbooks/:

Situation	Runbook
Low balance	low-balance.md
Requests pending but nothing fulfilled	stuck-nonce.md
Node stalled or RPC errors	rpc-outage.md
Requests older than 256 blocks	expired-requests.md (verity bhs backfill)
Replacing a VRF key	key-rotation.md
Backup nodes and failover	failover.md