# Trade Migration Plan: k3s on mevnode alongside agave Date: `2026-04-12` ## Migration Revision Baseline - Accepted migration baseline revision: `R001` - `R001` means the currently working trade application running on `k3s` before migration cutover. - The current source runtime for `R001` is the live `trade-staging` stack on `mevnode_bot`. - The first migration target is not a redesign. It is a faithful reconstruction of `R001` on the target `k3s` environment. - Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after `R001` is reproducible. - During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against `R001`. ## Goal - Prepare migration of the trade system to `k3s` on host `mevnode` (`sol`) in parallel with `agave-validator`. - Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts. ## Current Findings - `mevnode_bot` is reachable again after restart. - Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`. - I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset. - `k3s` on `mevnode_bot` is up and the node is `Ready`. - The main runtime failure is not the host itself but the DLOB publishers: - `dlob-publisher-hot` is `0/1` - `dlob-publisher-all` is `0/1` - both have very high restart counts - both fail startup/liveness probes - Publisher logs show Redis connectivity problems: - `ClusterAllFailedError: Failed to refresh slots cache` - `None of startup nodes is available` - `dlob-redis` currently exposes a single `ClusterIP` endpoint on `6379`, while the publishers behave like cluster-aware Redis clients. - `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, `postgres`, and the writer pods are currently up. - `trade-ingestor` had transient fetch failures after reboot, then resumed inserts. - There is a persistent warning in `k3s` about missing image pull secret `gitea-registry`. ### Resource Picture: mevnode - CPU: `32 vCPU` - RAM total: about `186 GiB` - RAM available: about `111 GiB` at check time - Swap total: about `31 GiB` - Swap used: about `13 GiB` - Root filesystem `/`: `btrfs`, about `1.8 TiB` total, about `1.6 TiB` free - `/data/ledger` lives on a separate `1.8 TiB` NVMe device. - `/data/state` lives on another separate `1.8 TiB` NVMe device. - The current candidate place for trade persistent storage on `sol` is the root NVMe filesystem `/`, not the validator ledger or accounts devices. - `agave-validator` RSS: about `129 GiB` - `agave-validator` is already a heavy resident workload and must remain the priority workload on this host. ### Resource Picture: mevnode_bot - Root filesystem `/`: `ext4` on `md RAID`, about `3.6 TiB` total, about `2.8 TiB` free - Current memory use after reboot is low, but this is not the main constraint. - The main storage warning is current `k3s` data growth: - `/var/lib/rancher/k3s`: about `593 GiB` - `/var/lib/rancher/k3s/storage/...data-postgres-0`: about `585 GiB` - the related PVC is declared as only `20Gi` - This means the current `local-path` storage setup does not give a meaningful hard size limit in practice. - The target Gitea organization `trade-next` now exists. - The target repository set now exists in `trade-next`: - `trade-api` - `trade-frontend` - `trade-ingestor` - `trade-bot` - `trade-dlob` - `trade-gitops` - `trade-host-iac` - `trade-docs` - The current live component-to-repository boundary was documented in `trade-next/trade-docs/doc/k3s-component-repo-map.md`. - We have not yet reconstructed the current live `trade-staging` manifests and host drift into `trade-gitops` and `trade-host-iac`. ## Migration Risks - Risk 1: moving the current storage model unchanged to `mevnode` can silently consume hundreds of GiB on the host root filesystem. - Risk 2: `agave-validator` already uses most of the memory footprint on `mevnode`; uncontrolled `k3s` workloads can force more swap or destabilize validator performance. - Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade. - Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology. - Risk 5: missing `gitea-registry` secret can break future image pulls and cold-start behavior. ## Plan ### 1. Reconstruct the current live `mevnode_bot` state into canonical Git/IaC - Map each live `trade-staging` object to a target repository and file path. - Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth. - Capture the current live deltas that are not present in local repos or in `~/trade-k8s`. - Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos. - Treat live `k3s` as input for reconstruction, not as a YAML dump to mirror blindly. ### 2. Stabilize the current trade stack on `mevnode_bot` - Fix `dlob-publisher-hot`. - Fix `dlob-publisher-all`. - Verify Redis mode and publisher Redis client mode are aligned. - Confirm all `trade-staging` pods become healthy after a full restart. ### 3. Audit storage before any migration - Measure actual PostgreSQL data size inside the current PVC. - Determine how much of the `585 GiB` is real database payload versus local-path overhead. - Identify retention candidates for historical data, derived tables, logs, and caches. - Decide what data must move and what can be rebuilt. ### 4. Define target storage layout on `mevnode` - Do not place trade persistent data on the same critical paths as agave ledger/accounts. - Do not rely on the current local-path behavior without hard operational limits. - Choose a dedicated storage path or dedicated device for trade state. - Set explicit quotas/limits and monitor actual bytes used on disk. ### 5. Define resource isolation on `mevnode` - Reserve headroom for `agave-validator`. - Add CPU and memory requests/limits for trade workloads. - Keep low-priority or bursty trade jobs away from validator-critical resources. - Treat publisher and DB workloads as separate scheduling/priority classes. ### 6. Recreate the missing operational prerequisites - Restore or recreate `gitea-registry` image pull secret. - Verify registry pull works after node reboot and after cache eviction. - Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible. ### 7. Prepare a staged migration - Start with a minimal canary namespace on `mevnode`. - Bring up infrastructure first: Redis, Postgres, Hasura. - Then bring up API and frontend. - Migrate publishers and ingest only after storage and Redis behavior are validated. - Keep `mevnode_bot` as rollback target until the new stack is stable. ### 8. Add migration validation checks - Host-level: memory, swap, disk, inode, and load checks. - Agave-level: slot lag, RPC health, and validator memory trend. - Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success. - End-to-end: UI, API, DLOB data freshness, and bot-related paths. ## Immediate Next Actions - Map the current live `trade-staging` state into target Git repositories and IaC paths. - Start the first canonical inventory pass into `trade-gitops` and `trade-host-iac`. - Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode. - Inspect Redis server configuration and topology in `trade-staging`. - Measure real Postgres database size inside the trade PVC. - Draft target disk placement for trade data on `mevnode`. - Decide minimum safe RAM headroom to keep free for `agave-validator`. ## Open Questions - Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history? - Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos? - Is the intended Redis topology for trade a single Redis instance or Redis Cluster? - Which trade datasets are required to migrate, and which can be rebuilt from source? - Where exactly on `mevnode` should persistent trade storage live? - What is the acceptable validator memory floor and swap ceiling during trade rollout? ## Chapter 2 ### 01. Define hard database and cache constraints - Pending. - Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests. - `Postgres` must not be able to grow beyond the actually allocated disk budget. - `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom. - `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat. - `Redis` must have explicit memory limits and an intentional eviction/persistence policy. - `Redis` must not be allowed to consume unbounded RAM or disk on the target host. - Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full. - Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover. ### 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization - Live `k3s` is the input for reconstruction, not the final source of truth. - Split application repositories, Kubernetes manifests, and host IaC into separate repositories. - Prefer a new Gitea organization over long-lived migration branches in existing repositories. - Recommended target organization name: `trade-next`. - Target repository set in `trade-next`: `trade-api`, `trade-frontend`, `trade-ingestor`, `trade-bot`, `trade-dlob`, `trade-gitops`, `trade-host-iac`, `trade-docs`. - Keep the current repository names where possible and change only the organization to reduce migration risk. - Use `trade-gitops` as the GitOps source for `k3s` manifests and `trade-host-iac` for host-level IaC, SSH, storage, and operations. - Record every current live drift before moving workloads. - Done on `2026-04-12`: `trade-next` was created and the target repositories now exist. - Done on `2026-04-12`: the live component boundary map was written to `trade-next/trade-docs/doc/k3s-component-repo-map.md`. - Next for item `02`: reconstruct the live `trade-staging` objects, overlays, secrets wiring, and host-specific drift into `trade-gitops` and `trade-host-iac`. ### 03. Fix `dlob-publisher-hot` startup, readiness, and liveness - Pending. ### 04. Fix `dlob-publisher-all` startup, readiness, and liveness - Pending. ### 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster - Pending. ### 06. Align publisher Redis client mode with the actual deployed Redis topology - Pending. ### 07. Restore or recreate the missing `gitea-registry` image pull secret - Pending. ### 08. Verify that image pulls work after node reboot and on a cold start - Pending. ### 09. Measure the real PostgreSQL data size inside the current trade PVC - Pending. ### 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk - Pending. ### 11. Decide which trade datasets must be migrated and which can be rebuilt - Pending. ### 12. Define a dedicated storage path or device for trade data on `mevnode` - Pending. - Current preferred target layout on `sol` is to place trade persistent storage on the root NVMe filesystem `/`, because validator ledger and validator state already have their own dedicated devices. - Do not place `Postgres` or `Redis` on `/data/ledger` or `/data/state`. - Reserve a dedicated `1 TiB` storage budget for `Postgres` on `sol` as the first target sizing for `R001`. - Reserve a separate small storage budget for `Redis` on `sol`, initially `2 GiB` unless persistence requirements change. - Prefer a dedicated path such as `/srv/trade/postgres` and `/srv/trade/redis` over implicit storage under `/var/lib/rancher/k3s/storage`. - On the current `btrfs` root filesystem, the preferred implementation path is a dedicated subvolume plus enforced `qgroup` quota for each persistent component. - Done on `2026-04-12`: created `btrfs` subvolumes on `sol` at `/srv/trade`, `/srv/trade/postgres`, and `/srv/trade/redis`. ### 13. Define hard storage quotas and monitoring for trade data on `mevnode` - Pending. - Use enforceable limits, not only declarative PVC sizes without backend enforcement. - Add warning and critical thresholds for disk usage, inode usage, and PVC growth. - Target hard quota for `Postgres` on `sol`: `1 TiB`. - Target hard quota for `Redis` on `sol`: `2 GiB` unless later requirements justify persistence growth. - Keep enough unallocated free space on the root NVMe for the OS, `k3s`, container images, logs, and rollback operations after trade quotas are reserved. - Done on `2026-04-12`: enabled `btrfs` qgroups on `/` at `sol` and applied hard quotas of `1 TiB` to `/srv/trade/postgres` and `2 GiB` to `/srv/trade/redis`. ### 14. Define CPU and memory reservations that protect `agave-validator` - Pending. ### 15. Define the minimum safe free RAM headroom on `mevnode` during rollout - Pending. ### 16. Define the maximum acceptable swap usage on `mevnode` during rollout - Pending. ### 17. Install and bootstrap `k3s` on `mevnode` - Pending. - Define the target `k3s` installation model on `mevnode`. - Define the target storage class, host paths, and bootstrap dependencies needed before application rollout. - Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed. ### 18. Install `Postgres` and `Redis` on `mevnode` - Pending. - Bring up the data plane first: `Postgres` and `Redis`. - Validate persistence, storage placement, service discovery, restart behavior, and cold start. - Validate that `Postgres` lands on the dedicated `1 TiB` trade storage area and that `Redis` lands on its separate small quota before application rollout. - Do not install application components until `Postgres` and `Redis` are healthy. ### 19. Prepare a minimal canary namespace for trade on `mevnode` - Pending. ### 20. Bring up `Hasura`, API, and frontend on `mevnode` - Pending. - Bring up `Hasura` only after `Postgres` is healthy. - Bring up API and frontend only after `Hasura`, `Postgres`, and `Redis` are healthy. ### 21. Migrate publishers and ingest only after storage and Redis validation pass - Pending. ### 22. Validate host health, agave health, trade health, and end-to-end flow after rollout - Pending. ### 23. Keep `mevnode_bot` as a rollback target until the new stack is stable - Pending. ## Chapter 3 - [ ] 01. Define hard database and cache constraints. - [ ] 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization. - [ ] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness. - [ ] 04. Fix `dlob-publisher-all` startup, readiness, and liveness. - [ ] 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster. - [ ] 06. Align publisher Redis client mode with the actual deployed Redis topology. - [ ] 07. Restore or recreate the missing `gitea-registry` image pull secret. - [ ] 08. Verify that image pulls work after node reboot and on a cold start. - [ ] 09. Measure the real PostgreSQL data size inside the current trade PVC. - [ ] 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk. - [ ] 11. Decide which trade datasets must be migrated and which can be rebuilt. - [ ] 12. Define a dedicated storage path or device for trade data on `mevnode`. - [ ] 13. Define hard storage quotas and monitoring for trade data on `mevnode`. - [ ] 14. Define CPU and memory reservations that protect `agave-validator`. - [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout. - [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout. - [ ] 17. Install and bootstrap `k3s` on `mevnode`. - [ ] 18. Install `Postgres` and `Redis` on `mevnode`. - [ ] 19. Prepare a minimal canary namespace for trade on `mevnode`. - [ ] 20. Bring up `Hasura`, API, and frontend on `mevnode`. - [ ] 21. Migrate publishers and ingest only after storage and Redis validation pass. - [ ] 22. Validate host health, agave health, trade health, and end-to-end flow after rollout. - [ ] 23. Keep `mevnode_bot` as a rollback target until the new stack is stable.