diff --git a/doc/migration-todo.md b/doc/migration-todo.md index 2633a7c..721bb05 100644 --- a/doc/migration-todo.md +++ b/doc/migration-todo.md @@ -19,6 +19,7 @@ Date: `2026-04-12` ## Current Findings - `mevnode_bot` is reachable again after restart. +- Current `R001` mapping still shows no separate live `trade-bot` deployment in `trade-staging`; the live runtime observed for this migration wave is `hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, and the DLOB stack. - Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`. - I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset. - `k3s` on `mevnode_bot` is up and the node is `Ready`. @@ -369,10 +370,14 @@ Date: `2026-04-12` ### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout -- In progress. +- Done on `2026-04-12`. - Initial validation on `2026-04-12`: after the DLOB hot/all-path rollout, sampled Agave slot lag on `sol` at `0` versus `https://api.mainnet-beta.solana.com`, with local health still reporting `ok`. - Done on `2026-04-12`: adjusted the canary rollout model to match the namespace guardrails by switching deployments to `RollingUpdate { maxSurge: 0, maxUnavailable: 1 }`, adding explicit low CPU limits to `api-token-seed`, and making the workflow restart the surface sequentially instead of all at once. - Done on `2026-04-12`: after the sequential smoke rollout on `sol`, all canary deployments returned to `Available=1/1`, `trade-api /v1/ticks` and `/v1/chart` returned data, and the validator still reported `lag=0`, `health=ok`. +- Done on `2026-04-12`: added operator script `environments/sol/trade-r001-canary/scripts/check-sol-canary-smoke.sh`, which waits for deployment availability, checks `agave-validator` and `k3s`, samples Agave lag, validates derived-table plus `drift_ticks` freshness, and exercises `trade-api` and `trade-frontend` end to end. +- Done on `2026-04-12`: switched canary publisher liveness probes on `sol` from `/health` to `/startup` after confirming that `/health` returned false negatives during steady-state while `drift_ticks` stayed fresh; the smoke-check is now the primary end-to-end validation gate for canary rollouts on `sol`. +- Done on `2026-04-12`: completed a full smoke-check run at `2026-04-12T17:11:33+00:00`; Agave reported `lag=0`, `health=ok`, all canary deployments were `Available=1/1`, `drift_ticks` remained fresh, `trade-api` returned live `SOL-PERP` ticks and candles, and `trade-frontend` returned `200`. +- Done on `2026-04-12`: snapshotted the `trade-r001-canary` secrets from `sol` to `$HOME/.local/share/trade-bootstrap/sol/trade-r001-canary-secrets` with directory mode `700` and file mode `600`, and verified that `sync-live-secrets.sh` can now reapply secrets to `sol` directly from the local snapshot without contacting `mevnode_bot`. ### 24. Keep `mevnode_bot` as a rollback target until the new stack is stable @@ -401,6 +406,6 @@ Date: `2026-04-12` - [x] 19. Bootstrap an organization-scoped Gitea Actions runner on `mevnode`. - [x] 20. Prepare a minimal canary namespace for trade on `mevnode`. - [x] 21. Bring up `Hasura`, API, and frontend on `mevnode`. -- [ ] 22. Migrate publishers and ingest only after storage and Redis validation pass. -- [ ] 23. Validate host health, agave health, trade health, and end-to-end flow after rollout. +- [x] 22. Migrate publishers and ingest only after storage and Redis validation pass. +- [x] 23. Validate host health, agave health, trade health, and end-to-end flow after rollout. - [ ] 24. Keep `mevnode_bot` as a rollback target until the new stack is stable.