Ctrl K
GitHubDocs
Security

Secret handling

Last updated: 2026-07-23

This page enumerates every secret in the system, where it lives at rest, who can read it, and how it is rotated. If a secret is not on this page, it should not exist.

The local dev stack uses environment variables for every secret for convenience. None of this is safe for testnet, let alone mainnet. The "Production" column tells you where it has to move before any non-dev deployment.

Secret inventory

Secret Local dev Production Rotation
PAY_MNEMONIC env var HSM / KMS Once on compromise; never on schedule
PAY_SECRET_KEY (KEK for webhook secrets) env var KMS, separate from PAY_MNEMONIC Annual
PAY_INTERNAL_JWT_SECRET env var KMS Quarterly
GAS_FUNDER_PRIVATE_KEY env var HSM (separate from PAY_MNEMONIC) On compromise
POSTGRES_PASSWORD env var Managed Postgres (IAM / password rotation) Per managed-PG policy
CHAIN_RPC_URL (for pay/settlement ? chain) env var Secrets manager N/A — URL only, not a secret
CHAIN_RPC_AUTH (if RPC is auth-gated) env var Secrets manager Per RPC provider policy
WEBHOOK_HMAC_KEY (legacy, unused in v1) n/a n/a n/a

Storage rules

  1. At rest, all secrets are encrypted with envelope encryption (PAY_SECRET_KEY as the KEK).
  2. In transit, secrets are never logged. The application uses a redacting logger that replaces any value matching a known secret prefix (bfx_, whsec_, hex strings of length 64, etc.) with ***.
  3. In memory, secrets are read once at startup into a sealed struct that does not implement Stringer or fmt.Stringer (Go) / toString (JS). Stack traces will show the address of the struct, not its contents.
  4. In backups, the database dump is encrypted at rest by the managed Postgres provider. The webhook_secret_ciphertext rows are useless without PAY_SECRET_KEY.

Local dev guidance

# .env (NEVER commit)
PAY_MNEMONIC="word1 word2 ... word12"
PAY_SECRET_KEY="$(openssl rand -hex 32)"
PAY_INTERNAL_JWT_SECRET="$(openssl rand -hex 32)"
GAS_FUNDER_PRIVATE_KEY="0x..."

The pay/ services load this file at startup. In local dev, that is fine. In any other environment, it is not.

.env is in .gitignore. Do not commit it. Do not paste it into chat. Do not put it in a screenshot.

Rotation playbook

Rotate PAY_SECRET_KEY (the KEK)

  1. Generate the new key: openssl rand -hex 32.
  2. Store it in KMS under a new alias (pay-secret-key-v2).
  3. Update the application to read the new alias.
  4. Re-encrypt every row in merchants.webhook_secret_ciphertext:
    • For each row: read ciphertext, decrypt with old key, encrypt with new key, write back.
    • Do this in a single transaction per merchant to avoid partial state.
  5. Delete the old alias from KMS.

There is a one-time CLI in pay/scripts/rekey.ts that does this. It is idempotent and can be re-run safely.

Rotate GAS_FUNDER_PRIVATE_KEY

  1. Generate a new EOA.
  2. Fund it with native 2BABY for gas.
  3. Set GAS_FUNDER_PRIVATE_KEY to the new key.
  4. Restart pay/sweep and pay/settlement.
  5. Drain the old gas-funder wallet to the new one (operator action).

Rotate a merchant''s api_key

  1. Soft-delete the merchant (DELETE /v1/merchant).
  2. Recreate with the same name and webhook_url.
  3. The new api_key and webhook_secret are returned exactly once.
There is no self-serve api_key rotation endpoint in v1. This is a deliberate scope cut. If you need it, raise an issue and we will add POST /v1/merchant/rotate-keys for the next release.

On-call

If you suspect any secret is compromised:

  1. Page the security on-call.
  2. Stop writing to the chain by setting the pay/sweep replica count to 0.
  3. Rotate PAY_MNEMONIC (this re-derives every future deposit address, but past addresses still need to be swept — coordinate with the on-call).
  4. Force-rotate every api_key by re-creating all merchants and notifying each one out-of-band.
  5. Force-rotate every webhook_secret by re-encrypting the webhook_secret_ciphertext column with a new PAY_SECRET_KEY.