Runbook

The on-call reference for BabyFX Pay. If something is on fire, start here. Every section follows the same shape: symptom ? diagnosis ? fix ? prevention.

How to read this

Each section is a self-contained recipe. Skip to the one matching your alert or log line. The order of sections is roughly "most likely" to "least likely".

A. Sweep failures

Symptom: "sweep_status=FAILED" on a confirmed invoice

Diagnosis:

SELECT id, sweep_tx_hash, sweep_error, updated_at
FROM invoices
WHERE sweep_status = 'FAILED'
ORDER BY updated_at DESC
LIMIT 10;

The sweep_error column tells you the reason. The top three are:

Prevention: alert when sweep wallet balance < 0.5× the 24h gas spend.

Symptom: invoices stuck in CONFIRMED, no sweep tx

Diagnosis: pay-sweep is down or its DB connection is broken.

docker ps --filter "name=pay-sweep"
docker logs pay-sweep --tail 50

Fix: restart the container. If the DB connection is broken, restart Postgres first, then pay-sweep.

B. Settlement lag

Symptom: invoices in PENDING for more than 5 minutes

Diagnosis: pay-settlement is behind on block polling, or the chain is down.

docker logs pay-settlement --tail 50 | Select-String "block"

Look for "skipping block" or "RPC error" lines. A healthy settlement logs block N height M every 1.5s.

Fix: if the chain is down, see C. Chain issues. Otherwise, restart pay-settlement.

C. Chain issues

Symptom: chain container restarting

Diagnosis: the most common cause is corrupted WAL. Check docker logs babychain-node --tail 100.

Fix:

docker compose -f chain\docker-compose.yml down
# remove the data volume ONLY if you are OK losing local state
docker volume rm babychain_data
docker compose -f chain\docker-compose.yml up -d

Symptom: chain stuck at a specific height

Diagnosis: CometBFT is wedged. The evmd process is up but not progressing.

Fix: graceful restart — docker restart babychain-node is safe; the node will replay the WAL and resume.

D. Gateway issues

Symptom: read API returns 500 for every call

Diagnosis: Postgres is unreachable. Check docker logs babychain-api --tail 50.

Fix: check babychain-postgres is up. If it is, check disk space — the read API will 500 if Postgres cannot extend a relation.

Symptom: proxy returns 502 to every JSON-RPC call

Diagnosis: the proxy cannot reach the chain. Check:

docker exec babychain-rpc-proxy curl -s http://host.docker.internal:8545 -X POST -H "Content-Type: application/json" --data ''{"jsonrpc":"2.0","method":"eth_chainId","params":[],"id":1}''

Fix: if that fails inside the container, the issue is network. If it works, the issue is that the chain is up but not bound to 0.0.0.0:8545. Check the chain''s --json-rpc.address flag.

E. Webhook failures

Symptom: merchant reports missed webhooks

Diagnosis:

SELECT id, invoice_id, response_status, attempt, last_attempt_at
FROM webhook_deliveries
WHERE delivered = FALSE
ORDER BY last_attempt_at DESC
LIMIT 20;

Fix:

• If response_status = 4xx, the merchant''s endpoint is broken. Notify them.
• If response_status = 5xx or NULL (timeout), the retry ladder will catch up. Wait.
• If attempt = 8 and still failing, the delivery is permanently failed. Notify the merchant and offer manual replay.

F. Database issues

Symptom: Postgres running out of disk

Fix: archive old invoice rows. The retention policy is configurable; the default is 365 days. Run pay/scripts/archive-invoices.ts to move old rows to invoices_archive.

Symptom: indexer cannot catch up

Fix: the indexer has a backfill mode. Stop the indexer, set INDEXER_BACKFILL=true, restart. It will walk from genesis to tip before tailing.

When all else fails

1. Snapshot the logs of every container: docker compose logs > bundle-$(date +%s).log.
2. Snapshot the relevant DB tables.
3. Page the second on-call.
4. Open a postmortem doc — the template is in runbook/postmortem-template.md in the monorepo.