trade-docs/doc/sol-cutover-checklist.md

sol cutover checklist

Date: 2026-04-12

Goal

Finalize public cutover of the reconstructed R001 trade surface from mevnode_bot to sol.

Already running on sol

• agave-validator
• host Postgres
• host Redis
• Hasura
• trade-api
• trade-frontend
• trade-ingestor
• dlob-publisher-hot
• dlob-publisher-all
• dlob-hot-redis-to-postgres-raw-writer
• dlob-hot-postgres-to-postgres-derived-writer
• dlob-all-redis-to-postgres-derived-writer

Public cutover steps

• Sync org secret SOL_SSH_PRIVATE_KEY_B64 for trade-next.
• Open 80/tcp and 443/tcp on sol from the trade-next public-edge workflow.
• Apply trade-next/trade-gitops/environments/sol/public-edge.
• Verify Traefik is listening on 80 and 443.
• Verify cert-manager and ClusterIssuer/letsencrypt-prod are Ready.
• Change authoritative DNS record trade.mpabi.pl A 149.50.96.162.
• Wait for Certificate/trade-mpabi-pl-tls to become Ready=True.
• Verify https://trade.mpabi.pl/healthz.
• Verify https://trade.mpabi.pl/ login and UI load.
• Re-run canary smoke check after DNS/TLS cutover.

Current status

• Done on 2026-04-12 20:16 CEST: public HTTP on 149.50.96.162:80 returns 301 to https://trade.mpabi.pl/....
• Done on 2026-04-12 20:16 CEST: https://trade.mpabi.pl/healthz answers 200 when forced to 149.50.96.162 with curl --resolve, currently using the default Traefik certificate before Let's Encrypt issuance completes.
• Remaining blocker for production TLS: the authoritative trade.mpabi.pl A record still points away from 149.50.96.162, so the ACME HTTP-01 challenge remains pending.

What still lived only on mevnode_bot

• old trade-staging namespace and its historical runtime state
• old DNS destination behind trade.mpabi.pl before the record is repointed
• old rollback source for secrets and manifests before the sol snapshot/offline bootstrap path was created

What is no longer dependent on mevnode_bot

• trade-r001-canary runtime on sol
• canary secret resync, because sol now has a local secret snapshot at $HOME/.local/share/trade-bootstrap/sol/trade-r001-canary-secrets
• host Postgres bootstrap on sol, because it can now read trade-postgres.json from the local snapshot

Open risk

• I could not confirm the final live-vs-live diff on mevnode_bot immediately before shutdown because the host stopped answering over SSH during the last checks.
• Because of that, the checklist above is authoritative for the reconstructed R001 on sol, but not a forensic inventory of every leftover object on the old host.
• The trade-next workflow definitions are committed and current, but the latest deploy-sol-public-edge trigger did not materialize a fresh Gitea Actions run after the workflow update, so the live cluster state above was applied from the same committed manifests outside the runner path.