trade-next/trade-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(migration): add R001 migration plan in markdown

Browse Source
This commit is contained in:
u1
2026-04-12 12:57:27 +00:00
parent 108c315361
commit a127cfbde3
1 changed files with 275 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

275
doc/migration-todo.md Normal file
View File

@@ -0,0 +1,275 @@
# Trade Migration Plan: k3s on mevnode alongside agave
Date: `2026-04-12`
## Migration Revision Baseline
- Accepted migration baseline revision: `R001`
- `R001` means the currently working trade application running on `k3s` before migration cutover.
- The current source runtime for `R001` is the live `trade-staging` stack on `mevnode_bot`.
- The first migration target is not a redesign. It is a faithful reconstruction of `R001` on the target `k3s` environment.
- Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after `R001` is reproducible.
- During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against `R001`.
## Goal
- Prepare migration of the trade system to `k3s` on host `mevnode` (`sol`) in parallel with `agave-validator`.
- Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts.
## Current Findings
- `mevnode_bot` is reachable again after restart.
- Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`.
- I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset.
- `k3s` on `mevnode_bot` is up and the node is `Ready`.
- The main runtime failure is not the host itself but the DLOB publishers:
- `dlob-publisher-hot` is `0/1`
- `dlob-publisher-all` is `0/1`
- both have very high restart counts
- both fail startup/liveness probes
- Publisher logs show Redis connectivity problems:
- `ClusterAllFailedError: Failed to refresh slots cache`
- `None of startup nodes is available`
- `dlob-redis` currently exposes a single `ClusterIP` endpoint on `6379`, while the publishers behave like cluster-aware Redis clients.
- `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, `postgres`, and the writer pods are currently up.
- `trade-ingestor` had transient fetch failures after reboot, then resumed inserts.
- There is a persistent warning in `k3s` about missing image pull secret `gitea-registry`.
### Resource Picture: mevnode
- CPU: `32 vCPU`
- RAM total: about `186 GiB`
- RAM available: about `111 GiB` at check time
- Swap total: about `31 GiB`
- Swap used: about `13 GiB`
- Root filesystem `/`: `btrfs`, about `1.8 TiB` total, about `1.6 TiB` free
- `agave-validator` RSS: about `129 GiB`
- `agave-validator` is already a heavy resident workload and must remain the priority workload on this host.
### Resource Picture: mevnode_bot
- Root filesystem `/`: `ext4` on `md RAID`, about `3.6 TiB` total, about `2.8 TiB` free
- Current memory use after reboot is low, but this is not the main constraint.
- The main storage warning is current `k3s` data growth:
- `/var/lib/rancher/k3s`: about `593 GiB`
- `/var/lib/rancher/k3s/storage/...data-postgres-0`: about `585 GiB`
- the related PVC is declared as only `20Gi`
- This means the current `local-path` storage setup does not give a meaningful hard size limit in practice.
- The target Gitea organization `trade-next` now exists.
- The target repository set now exists in `trade-next`:
- `trade-api`
- `trade-frontend`
- `trade-ingestor`
- `trade-bot`
- `trade-dlob`
- `trade-gitops`
- `trade-host-iac`
- `trade-docs`
- The current live component-to-repository boundary was documented in `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- We have not yet reconstructed the current live `trade-staging` manifests and host drift into `trade-gitops` and `trade-host-iac`.
## Migration Risks
- Risk 1: moving the current storage model unchanged to `mevnode` can silently consume hundreds of GiB on the host root filesystem.
- Risk 2: `agave-validator` already uses most of the memory footprint on `mevnode`; uncontrolled `k3s` workloads can force more swap or destabilize validator performance.
- Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade.
- Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology.
- Risk 5: missing `gitea-registry` secret can break future image pulls and cold-start behavior.
## Plan
### 1. Reconstruct the current live `mevnode_bot` state into canonical Git/IaC
- Map each live `trade-staging` object to a target repository and file path.
- Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth.
- Capture the current live deltas that are not present in local repos or in `~/trade-k8s`.
- Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos.
- Treat live `k3s` as input for reconstruction, not as a YAML dump to mirror blindly.
### 2. Stabilize the current trade stack on `mevnode_bot`
- Fix `dlob-publisher-hot`.
- Fix `dlob-publisher-all`.
- Verify Redis mode and publisher Redis client mode are aligned.
- Confirm all `trade-staging` pods become healthy after a full restart.
### 3. Audit storage before any migration
- Measure actual PostgreSQL data size inside the current PVC.
- Determine how much of the `585 GiB` is real database payload versus local-path overhead.
- Identify retention candidates for historical data, derived tables, logs, and caches.
- Decide what data must move and what can be rebuilt.
### 4. Define target storage layout on `mevnode`
- Do not place trade persistent data on the same critical paths as agave ledger/accounts.
- Do not rely on the current local-path behavior without hard operational limits.
- Choose a dedicated storage path or dedicated device for trade state.
- Set explicit quotas/limits and monitor actual bytes used on disk.
### 5. Define resource isolation on `mevnode`
- Reserve headroom for `agave-validator`.
- Add CPU and memory requests/limits for trade workloads.
- Keep low-priority or bursty trade jobs away from validator-critical resources.
- Treat publisher and DB workloads as separate scheduling/priority classes.
### 6. Recreate the missing operational prerequisites
- Restore or recreate `gitea-registry` image pull secret.
- Verify registry pull works after node reboot and after cache eviction.
- Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible.
### 7. Prepare a staged migration
- Start with a minimal canary namespace on `mevnode`.
- Bring up infrastructure first: Redis, Postgres, Hasura.
- Then bring up API and frontend.
- Migrate publishers and ingest only after storage and Redis behavior are validated.
- Keep `mevnode_bot` as rollback target until the new stack is stable.
### 8. Add migration validation checks
- Host-level: memory, swap, disk, inode, and load checks.
- Agave-level: slot lag, RPC health, and validator memory trend.
- Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success.
- End-to-end: UI, API, DLOB data freshness, and bot-related paths.
## Immediate Next Actions
- Map the current live `trade-staging` state into target Git repositories and IaC paths.
- Start the first canonical inventory pass into `trade-gitops` and `trade-host-iac`.
- Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode.
- Inspect Redis server configuration and topology in `trade-staging`.
- Measure real Postgres database size inside the trade PVC.
- Draft target disk placement for trade data on `mevnode`.
- Decide minimum safe RAM headroom to keep free for `agave-validator`.
## Open Questions
- Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history?
- Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos?
- Is the intended Redis topology for trade a single Redis instance or Redis Cluster?
- Which trade datasets are required to migrate, and which can be rebuilt from source?
- Where exactly on `mevnode` should persistent trade storage live?
- What is the acceptable validator memory floor and swap ceiling during trade rollout?
## Chapter 2
### 01. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization
- Live `k3s` is the input for reconstruction, not the final source of truth.
- Split application repositories, Kubernetes manifests, and host IaC into separate repositories.
- Prefer a new Gitea organization over long-lived migration branches in existing repositories.
- Recommended target organization name: `trade-next`.
- Target repository set in `trade-next`: `trade-api`, `trade-frontend`, `trade-ingestor`, `trade-bot`, `trade-dlob`, `trade-gitops`, `trade-host-iac`, `trade-docs`.
- Keep the current repository names where possible and change only the organization to reduce migration risk.
- Use `trade-gitops` as the GitOps source for `k3s` manifests and `trade-host-iac` for host-level IaC, SSH, storage, and operations.
- Record every current live drift before moving workloads.
- Done on `2026-04-12`: `trade-next` was created and the target repositories now exist.
- Done on `2026-04-12`: the live component boundary map was written to `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- Next for item `01`: reconstruct the live `trade-staging` objects, overlays, secrets wiring, and host-specific drift into `trade-gitops` and `trade-host-iac`.
### 02. Fix `dlob-publisher-hot` startup, readiness, and liveness
- Pending.
### 03. Fix `dlob-publisher-all` startup, readiness, and liveness
- Pending.
### 04. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster
- Pending.
### 05. Align publisher Redis client mode with the actual deployed Redis topology
- Pending.
### 06. Restore or recreate the missing `gitea-registry` image pull secret
- Pending.
### 07. Verify that image pulls work after node reboot and on a cold start
- Pending.
### 08. Measure the real PostgreSQL data size inside the current trade PVC
- Pending.
### 09. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk
- Pending.
### 10. Decide which trade datasets must be migrated and which can be rebuilt
- Pending.
### 11. Define a dedicated storage path or device for trade data on `mevnode`
- Pending.
### 12. Define hard storage quotas and monitoring for trade data on `mevnode`
- Pending.
### 13. Define CPU and memory reservations that protect `agave-validator`
- Pending.
### 14. Define the minimum safe free RAM headroom on `mevnode` during rollout
- Pending.
### 15. Define the maximum acceptable swap usage on `mevnode` during rollout
- Pending.
### 16. Prepare a minimal canary namespace for trade on `mevnode`
- Pending.
### 17. Bring up Redis, Postgres, and Hasura on `mevnode`
- Pending.
### 18. Bring up API and frontend on `mevnode`
- Pending.
### 19. Migrate publishers and ingest only after storage and Redis validation pass
- Pending.
### 20. Validate host health, agave health, trade health, and end-to-end flow after rollout
- Pending.
### 21. Keep `mevnode_bot` as a rollback target until the new stack is stable
- Pending.
## Chapter 3
- [ ] 01. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization.
- [ ] 02. Fix `dlob-publisher-hot` startup, readiness, and liveness.
- [ ] 03. Fix `dlob-publisher-all` startup, readiness, and liveness.
- [ ] 04. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster.
- [ ] 05. Align publisher Redis client mode with the actual deployed Redis topology.
- [ ] 06. Restore or recreate the missing `gitea-registry` image pull secret.
- [ ] 07. Verify that image pulls work after node reboot and on a cold start.
- [ ] 08. Measure the real PostgreSQL data size inside the current trade PVC.
- [ ] 09. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk.
- [ ] 10. Decide which trade datasets must be migrated and which can be rebuilt.
- [ ] 11. Define a dedicated storage path or device for trade data on `mevnode`.
- [ ] 12. Define hard storage quotas and monitoring for trade data on `mevnode`.
- [ ] 13. Define CPU and memory reservations that protect `agave-validator`.
- [ ] 14. Define the minimum safe free RAM headroom on `mevnode` during rollout.
- [ ] 15. Define the maximum acceptable swap usage on `mevnode` during rollout.
- [ ] 16. Prepare a minimal canary namespace for trade on `mevnode`.
- [ ] 17. Bring up Redis, Postgres, and Hasura on `mevnode`.
- [ ] 18. Bring up API and frontend on `mevnode`.
- [ ] 19. Migrate publishers and ingest only after storage and Redis validation pass.
- [ ] 20. Validate host health, agave health, trade health, and end-to-end flow after rollout.
- [ ] 21. Keep `mevnode_bot` as a rollback target until the new stack is stable.