trade-next/trade-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): record offline sol bootstrap fallback

Browse Source
This commit is contained in:
mpabi
2026-04-12 19:25:55 +02:00
parent 774310b205
commit 6c0c949e1a
1 changed files with 8 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
doc/migration-todo.md
View File

@@ -19,6 +19,7 @@ Date: `2026-04-12`
## Current Findings ## Current Findings
- `mevnode_bot` is reachable again after restart. - `mevnode_bot` is reachable again after restart.
- Current `R001` mapping still shows no separate live `trade-bot` deployment in `trade-staging`; the live runtime observed for this migration wave is `hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, and the DLOB stack.
- Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`. - Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`.
- I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset. - I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset.
- `k3s` on `mevnode_bot` is up and the node is `Ready`. - `k3s` on `mevnode_bot` is up and the node is `Ready`.
@@ -369,10 +370,14 @@ Date: `2026-04-12`
### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout ### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout
- In progress. - Done on `2026-04-12`.
- Initial validation on `2026-04-12`: after the DLOB hot/all-path rollout, sampled Agave slot lag on `sol` at `0` versus `https://api.mainnet-beta.solana.com`, with local health still reporting `ok`. - Initial validation on `2026-04-12`: after the DLOB hot/all-path rollout, sampled Agave slot lag on `sol` at `0` versus `https://api.mainnet-beta.solana.com`, with local health still reporting `ok`.
- Done on `2026-04-12`: adjusted the canary rollout model to match the namespace guardrails by switching deployments to `RollingUpdate { maxSurge: 0, maxUnavailable: 1 }`, adding explicit low CPU limits to `api-token-seed`, and making the workflow restart the surface sequentially instead of all at once. - Done on `2026-04-12`: adjusted the canary rollout model to match the namespace guardrails by switching deployments to `RollingUpdate { maxSurge: 0, maxUnavailable: 1 }`, adding explicit low CPU limits to `api-token-seed`, and making the workflow restart the surface sequentially instead of all at once.
- Done on `2026-04-12`: after the sequential smoke rollout on `sol`, all canary deployments returned to `Available=1/1`, `trade-api /v1/ticks` and `/v1/chart` returned data, and the validator still reported `lag=0`, `health=ok`. - Done on `2026-04-12`: after the sequential smoke rollout on `sol`, all canary deployments returned to `Available=1/1`, `trade-api /v1/ticks` and `/v1/chart` returned data, and the validator still reported `lag=0`, `health=ok`.
- Done on `2026-04-12`: added operator script `environments/sol/trade-r001-canary/scripts/check-sol-canary-smoke.sh`, which waits for deployment availability, checks `agave-validator` and `k3s`, samples Agave lag, validates derived-table plus `drift_ticks` freshness, and exercises `trade-api` and `trade-frontend` end to end.
- Done on `2026-04-12`: switched canary publisher liveness probes on `sol` from `/health` to `/startup` after confirming that `/health` returned false negatives during steady-state while `drift_ticks` stayed fresh; the smoke-check is now the primary end-to-end validation gate for canary rollouts on `sol`.
- Done on `2026-04-12`: completed a full smoke-check run at `2026-04-12T17:11:33+00:00`; Agave reported `lag=0`, `health=ok`, all canary deployments were `Available=1/1`, `drift_ticks` remained fresh, `trade-api` returned live `SOL-PERP` ticks and candles, and `trade-frontend` returned `200`.
- Done on `2026-04-12`: snapshotted the `trade-r001-canary` secrets from `sol` to `$HOME/.local/share/trade-bootstrap/sol/trade-r001-canary-secrets` with directory mode `700` and file mode `600`, and verified that `sync-live-secrets.sh` can now reapply secrets to `sol` directly from the local snapshot without contacting `mevnode_bot`.
### 24. Keep `mevnode_bot` as a rollback target until the new stack is stable ### 24. Keep `mevnode_bot` as a rollback target until the new stack is stable
@@ -401,6 +406,6 @@ Date: `2026-04-12`
- [x] 19. Bootstrap an organization-scoped Gitea Actions runner on `mevnode`. - [x] 19. Bootstrap an organization-scoped Gitea Actions runner on `mevnode`.
- [x] 20. Prepare a minimal canary namespace for trade on `mevnode`. - [x] 20. Prepare a minimal canary namespace for trade on `mevnode`.
- [x] 21. Bring up `Hasura`, API, and frontend on `mevnode`. - [x] 21. Bring up `Hasura`, API, and frontend on `mevnode`.
- [ ] 22. Migrate publishers and ingest only after storage and Redis validation pass. - [x] 22. Migrate publishers and ingest only after storage and Redis validation pass.
- [ ] 23. Validate host health, agave health, trade health, and end-to-end flow after rollout. - [x] 23. Validate host health, agave health, trade health, and end-to-end flow after rollout.
- [ ] 24. Keep `mevnode_bot` as a rollback target until the new stack is stable. - [ ] 24. Keep `mevnode_bot` as a rollback target until the new stack is stable.