diff --git a/doc/migration-todo.md b/doc/migration-todo.md new file mode 100644 index 0000000..b135cf7 --- /dev/null +++ b/doc/migration-todo.md @@ -0,0 +1,275 @@ +# Trade Migration Plan: k3s on mevnode alongside agave + +Date: `2026-04-12` + +## Migration Revision Baseline + +- Accepted migration baseline revision: `R001` +- `R001` means the currently working trade application running on `k3s` before migration cutover. +- The current source runtime for `R001` is the live `trade-staging` stack on `mevnode_bot`. +- The first migration target is not a redesign. It is a faithful reconstruction of `R001` on the target `k3s` environment. +- Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after `R001` is reproducible. +- During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against `R001`. + +## Goal + +- Prepare migration of the trade system to `k3s` on host `mevnode` (`sol`) in parallel with `agave-validator`. +- Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts. + +## Current Findings + +- `mevnode_bot` is reachable again after restart. +- Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`. +- I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset. +- `k3s` on `mevnode_bot` is up and the node is `Ready`. +- The main runtime failure is not the host itself but the DLOB publishers: + - `dlob-publisher-hot` is `0/1` + - `dlob-publisher-all` is `0/1` + - both have very high restart counts + - both fail startup/liveness probes +- Publisher logs show Redis connectivity problems: + - `ClusterAllFailedError: Failed to refresh slots cache` + - `None of startup nodes is available` +- `dlob-redis` currently exposes a single `ClusterIP` endpoint on `6379`, while the publishers behave like cluster-aware Redis clients. +- `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, `postgres`, and the writer pods are currently up. +- `trade-ingestor` had transient fetch failures after reboot, then resumed inserts. +- There is a persistent warning in `k3s` about missing image pull secret `gitea-registry`. + +### Resource Picture: mevnode + +- CPU: `32 vCPU` +- RAM total: about `186 GiB` +- RAM available: about `111 GiB` at check time +- Swap total: about `31 GiB` +- Swap used: about `13 GiB` +- Root filesystem `/`: `btrfs`, about `1.8 TiB` total, about `1.6 TiB` free +- `agave-validator` RSS: about `129 GiB` +- `agave-validator` is already a heavy resident workload and must remain the priority workload on this host. + +### Resource Picture: mevnode_bot + +- Root filesystem `/`: `ext4` on `md RAID`, about `3.6 TiB` total, about `2.8 TiB` free +- Current memory use after reboot is low, but this is not the main constraint. +- The main storage warning is current `k3s` data growth: + - `/var/lib/rancher/k3s`: about `593 GiB` + - `/var/lib/rancher/k3s/storage/...data-postgres-0`: about `585 GiB` + - the related PVC is declared as only `20Gi` +- This means the current `local-path` storage setup does not give a meaningful hard size limit in practice. +- The target Gitea organization `trade-next` now exists. +- The target repository set now exists in `trade-next`: + - `trade-api` + - `trade-frontend` + - `trade-ingestor` + - `trade-bot` + - `trade-dlob` + - `trade-gitops` + - `trade-host-iac` + - `trade-docs` +- The current live component-to-repository boundary was documented in `trade-next/trade-docs/doc/k3s-component-repo-map.md`. +- We have not yet reconstructed the current live `trade-staging` manifests and host drift into `trade-gitops` and `trade-host-iac`. + +## Migration Risks + +- Risk 1: moving the current storage model unchanged to `mevnode` can silently consume hundreds of GiB on the host root filesystem. +- Risk 2: `agave-validator` already uses most of the memory footprint on `mevnode`; uncontrolled `k3s` workloads can force more swap or destabilize validator performance. +- Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade. +- Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology. +- Risk 5: missing `gitea-registry` secret can break future image pulls and cold-start behavior. + +## Plan + +### 1. Reconstruct the current live `mevnode_bot` state into canonical Git/IaC + +- Map each live `trade-staging` object to a target repository and file path. +- Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth. +- Capture the current live deltas that are not present in local repos or in `~/trade-k8s`. +- Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos. +- Treat live `k3s` as input for reconstruction, not as a YAML dump to mirror blindly. + +### 2. Stabilize the current trade stack on `mevnode_bot` + +- Fix `dlob-publisher-hot`. +- Fix `dlob-publisher-all`. +- Verify Redis mode and publisher Redis client mode are aligned. +- Confirm all `trade-staging` pods become healthy after a full restart. + +### 3. Audit storage before any migration + +- Measure actual PostgreSQL data size inside the current PVC. +- Determine how much of the `585 GiB` is real database payload versus local-path overhead. +- Identify retention candidates for historical data, derived tables, logs, and caches. +- Decide what data must move and what can be rebuilt. + +### 4. Define target storage layout on `mevnode` + +- Do not place trade persistent data on the same critical paths as agave ledger/accounts. +- Do not rely on the current local-path behavior without hard operational limits. +- Choose a dedicated storage path or dedicated device for trade state. +- Set explicit quotas/limits and monitor actual bytes used on disk. + +### 5. Define resource isolation on `mevnode` + +- Reserve headroom for `agave-validator`. +- Add CPU and memory requests/limits for trade workloads. +- Keep low-priority or bursty trade jobs away from validator-critical resources. +- Treat publisher and DB workloads as separate scheduling/priority classes. + +### 6. Recreate the missing operational prerequisites + +- Restore or recreate `gitea-registry` image pull secret. +- Verify registry pull works after node reboot and after cache eviction. +- Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible. + +### 7. Prepare a staged migration + +- Start with a minimal canary namespace on `mevnode`. +- Bring up infrastructure first: Redis, Postgres, Hasura. +- Then bring up API and frontend. +- Migrate publishers and ingest only after storage and Redis behavior are validated. +- Keep `mevnode_bot` as rollback target until the new stack is stable. + +### 8. Add migration validation checks + +- Host-level: memory, swap, disk, inode, and load checks. +- Agave-level: slot lag, RPC health, and validator memory trend. +- Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success. +- End-to-end: UI, API, DLOB data freshness, and bot-related paths. + +## Immediate Next Actions + +- Map the current live `trade-staging` state into target Git repositories and IaC paths. +- Start the first canonical inventory pass into `trade-gitops` and `trade-host-iac`. +- Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode. +- Inspect Redis server configuration and topology in `trade-staging`. +- Measure real Postgres database size inside the trade PVC. +- Draft target disk placement for trade data on `mevnode`. +- Decide minimum safe RAM headroom to keep free for `agave-validator`. + +## Open Questions + +- Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history? +- Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos? +- Is the intended Redis topology for trade a single Redis instance or Redis Cluster? +- Which trade datasets are required to migrate, and which can be rebuilt from source? +- Where exactly on `mevnode` should persistent trade storage live? +- What is the acceptable validator memory floor and swap ceiling during trade rollout? + +## Chapter 2 + +### 01. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization + +- Live `k3s` is the input for reconstruction, not the final source of truth. +- Split application repositories, Kubernetes manifests, and host IaC into separate repositories. +- Prefer a new Gitea organization over long-lived migration branches in existing repositories. +- Recommended target organization name: `trade-next`. +- Target repository set in `trade-next`: `trade-api`, `trade-frontend`, `trade-ingestor`, `trade-bot`, `trade-dlob`, `trade-gitops`, `trade-host-iac`, `trade-docs`. +- Keep the current repository names where possible and change only the organization to reduce migration risk. +- Use `trade-gitops` as the GitOps source for `k3s` manifests and `trade-host-iac` for host-level IaC, SSH, storage, and operations. +- Record every current live drift before moving workloads. +- Done on `2026-04-12`: `trade-next` was created and the target repositories now exist. +- Done on `2026-04-12`: the live component boundary map was written to `trade-next/trade-docs/doc/k3s-component-repo-map.md`. +- Next for item `01`: reconstruct the live `trade-staging` objects, overlays, secrets wiring, and host-specific drift into `trade-gitops` and `trade-host-iac`. + +### 02. Fix `dlob-publisher-hot` startup, readiness, and liveness + +- Pending. + +### 03. Fix `dlob-publisher-all` startup, readiness, and liveness + +- Pending. + +### 04. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster + +- Pending. + +### 05. Align publisher Redis client mode with the actual deployed Redis topology + +- Pending. + +### 06. Restore or recreate the missing `gitea-registry` image pull secret + +- Pending. + +### 07. Verify that image pulls work after node reboot and on a cold start + +- Pending. + +### 08. Measure the real PostgreSQL data size inside the current trade PVC + +- Pending. + +### 09. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk + +- Pending. + +### 10. Decide which trade datasets must be migrated and which can be rebuilt + +- Pending. + +### 11. Define a dedicated storage path or device for trade data on `mevnode` + +- Pending. + +### 12. Define hard storage quotas and monitoring for trade data on `mevnode` + +- Pending. + +### 13. Define CPU and memory reservations that protect `agave-validator` + +- Pending. + +### 14. Define the minimum safe free RAM headroom on `mevnode` during rollout + +- Pending. + +### 15. Define the maximum acceptable swap usage on `mevnode` during rollout + +- Pending. + +### 16. Prepare a minimal canary namespace for trade on `mevnode` + +- Pending. + +### 17. Bring up Redis, Postgres, and Hasura on `mevnode` + +- Pending. + +### 18. Bring up API and frontend on `mevnode` + +- Pending. + +### 19. Migrate publishers and ingest only after storage and Redis validation pass + +- Pending. + +### 20. Validate host health, agave health, trade health, and end-to-end flow after rollout + +- Pending. + +### 21. Keep `mevnode_bot` as a rollback target until the new stack is stable + +- Pending. + +## Chapter 3 + +- [ ] 01. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization. +- [ ] 02. Fix `dlob-publisher-hot` startup, readiness, and liveness. +- [ ] 03. Fix `dlob-publisher-all` startup, readiness, and liveness. +- [ ] 04. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster. +- [ ] 05. Align publisher Redis client mode with the actual deployed Redis topology. +- [ ] 06. Restore or recreate the missing `gitea-registry` image pull secret. +- [ ] 07. Verify that image pulls work after node reboot and on a cold start. +- [ ] 08. Measure the real PostgreSQL data size inside the current trade PVC. +- [ ] 09. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk. +- [ ] 10. Decide which trade datasets must be migrated and which can be rebuilt. +- [ ] 11. Define a dedicated storage path or device for trade data on `mevnode`. +- [ ] 12. Define hard storage quotas and monitoring for trade data on `mevnode`. +- [ ] 13. Define CPU and memory reservations that protect `agave-validator`. +- [ ] 14. Define the minimum safe free RAM headroom on `mevnode` during rollout. +- [ ] 15. Define the maximum acceptable swap usage on `mevnode` during rollout. +- [ ] 16. Prepare a minimal canary namespace for trade on `mevnode`. +- [ ] 17. Bring up Redis, Postgres, and Hasura on `mevnode`. +- [ ] 18. Bring up API and frontend on `mevnode`. +- [ ] 19. Migrate publishers and ingest only after storage and Redis validation pass. +- [ ] 20. Validate host health, agave health, trade health, and end-to-end flow after rollout. +- [ ] 21. Keep `mevnode_bot` as a rollback target until the new stack is stable.