trade-docs/doc/migration-todo.md

# Trade Migration Plan: k3s on mevnode alongside agave

Date: `2026-04-12`

## Migration Revision Baseline

- Accepted migration baseline revision: `R001`
- `R001` means the currently working trade application running on `k3s` before migration cutover.
- The current source runtime for `R001` is the live `trade-staging` stack on `mevnode_bot`.
- The first migration target is not a redesign. It is a faithful reconstruction of `R001` on the target `k3s` environment.
- Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after `R001` is reproducible.
- During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against `R001`.

## Goal

- Prepare migration of the trade system to `k3s` on host `mevnode` (`sol`) in parallel with `agave-validator`.
- Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts.

## Current Findings

- `mevnode_bot` is reachable again after restart.
- Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`.
- I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset.
- `k3s` on `mevnode_bot` is up and the node is `Ready`.
- The main runtime failure is not the host itself but the DLOB publishers:
  - `dlob-publisher-hot` is `0/1`
  - `dlob-publisher-all` is `0/1`
  - both have very high restart counts
  - both fail startup/liveness probes
- Publisher logs show Redis connectivity problems:
  - `ClusterAllFailedError: Failed to refresh slots cache`
  - `None of startup nodes is available`
- `dlob-redis` currently exposes a single `ClusterIP` endpoint on `6379`, while the publishers behave like cluster-aware Redis clients.
- `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, `postgres`, and the writer pods are currently up.
- `trade-ingestor` had transient fetch failures after reboot, then resumed inserts.
- There is a persistent warning in `k3s` about missing image pull secret `gitea-registry`.

### Resource Picture: mevnode

- CPU: `32 vCPU`
- RAM total: about `186 GiB`
- RAM available: about `111 GiB` at check time
- Swap total: about `31 GiB`
- Swap used: about `13 GiB`
- Root filesystem `/`: `btrfs`, about `1.8 TiB` total, about `1.6 TiB` free
- `/data/ledger` lives on a separate `1.8 TiB` NVMe device.
- `/data/state` lives on another separate `1.8 TiB` NVMe device.
- The current candidate place for trade persistent storage on `sol` is the root NVMe filesystem `/`, not the validator ledger or accounts devices.
- `agave-validator` RSS: about `129 GiB`
- `agave-validator` is already a heavy resident workload and must remain the priority workload on this host.

### Resource Picture: mevnode_bot

- Root filesystem `/`: `ext4` on `md RAID`, about `3.6 TiB` total, about `2.8 TiB` free
- Current memory use after reboot is low, but this is not the main constraint.
- The main storage warning is current `k3s` data growth:
  - `/var/lib/rancher/k3s`: about `593 GiB`
  - `/var/lib/rancher/k3s/storage/...data-postgres-0`: about `585 GiB`
  - the related PVC is declared as only `20Gi`
- This means the current `local-path` storage setup does not give a meaningful hard size limit in practice.
- The target Gitea organization `trade-next` now exists.
- The target repository set now exists in `trade-next`:
  - `trade-api`
  - `trade-frontend`
  - `trade-ingestor`
  - `trade-bot`
  - `trade-dlob`
  - `trade-gitops`
  - `trade-host-iac`
  - `trade-docs`
- The current live component-to-repository boundary was documented in `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- We have not yet reconstructed the current live `trade-staging` manifests and host drift into `trade-gitops` and `trade-host-iac`.

## Migration Risks

- Risk 1: moving the current storage model unchanged to `mevnode` can silently consume hundreds of GiB on the host root filesystem.
- Risk 2: `agave-validator` already uses most of the memory footprint on `mevnode`; uncontrolled `k3s` workloads can force more swap or destabilize validator performance.
- Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade.
- Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology.
- Risk 5: missing `gitea-registry` secret can break future image pulls and cold-start behavior.

## Plan

### 1. Reconstruct the current live `mevnode_bot` state into canonical Git/IaC

- Map each live `trade-staging` object to a target repository and file path.
- Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth.
- Capture the current live deltas that are not present in local repos or in `~/trade-k8s`.
- Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos.
- Treat live `k3s` as input for reconstruction, not as a YAML dump to mirror blindly.

### 2. Stabilize the current trade stack on `mevnode_bot`

- Fix `dlob-publisher-hot`.
- Fix `dlob-publisher-all`.
- Verify Redis mode and publisher Redis client mode are aligned.
- Confirm all `trade-staging` pods become healthy after a full restart.

### 3. Audit storage before any migration

- Measure actual PostgreSQL data size inside the current PVC.
- Determine how much of the `585 GiB` is real database payload versus local-path overhead.
- Identify retention candidates for historical data, derived tables, logs, and caches.
- Decide what data must move and what can be rebuilt.

### 4. Define target storage layout on `mevnode`

- Do not place trade persistent data on the same critical paths as agave ledger/accounts.
- Do not rely on the current local-path behavior without hard operational limits.
- Choose a dedicated storage path or dedicated device for trade state.
- Set explicit quotas/limits and monitor actual bytes used on disk.

### 5. Define resource isolation on `mevnode`

- Reserve headroom for `agave-validator`.
- Add CPU and memory requests/limits for trade workloads.
- Keep low-priority or bursty trade jobs away from validator-critical resources.
- Treat publisher and DB workloads as separate scheduling/priority classes.

### 6. Recreate the missing operational prerequisites

- Restore or recreate `gitea-registry` image pull secret.
- Verify registry pull works after node reboot and after cache eviction.
- Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible.

### 7. Prepare a staged migration

- Start with a minimal canary namespace on `mevnode`.
- Bring up infrastructure first: Redis, Postgres, Hasura.
- Then bring up API and frontend.
- Migrate publishers and ingest only after storage and Redis behavior are validated.
- Keep `mevnode_bot` as rollback target until the new stack is stable.

### 8. Add migration validation checks

- Host-level: memory, swap, disk, inode, and load checks.
- Agave-level: slot lag, RPC health, and validator memory trend.
- Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success.
- End-to-end: UI, API, DLOB data freshness, and bot-related paths.

## Immediate Next Actions

- Map the current live `trade-staging` state into target Git repositories and IaC paths.
- Start the first canonical inventory pass into `trade-gitops` and `trade-host-iac`.
- Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode.
- Inspect Redis server configuration and topology in `trade-staging`.
- Measure real Postgres database size inside the trade PVC.
- Draft target disk placement for trade data on `mevnode`.
- Decide minimum safe RAM headroom to keep free for `agave-validator`.

## Open Questions

- Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history?
- Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos?
- Is the intended Redis topology for trade a single Redis instance or Redis Cluster?
- Which trade datasets are required to migrate, and which can be rebuilt from source?
- Where exactly on `mevnode` should persistent trade storage live?
- What is the acceptable validator memory floor and swap ceiling during trade rollout?

## Chapter 2

### 01. Define hard database and cache constraints

- In progress.
- Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests.
- In core `Postgres`, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached.
- `Postgres` must not be able to grow beyond the actually allocated disk budget.
- `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom.
- `Postgres` should use `temp_file_limit` so large sorts and hashes fail before they can consume uncontrolled temporary disk space.
- `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat.
- `Postgres` should use bounded `max_wal_size` and, if replication slots are present, bounded `max_slot_wal_keep_size`, while treating WAL limits as early control signals rather than the final hard stop.
- `Postgres` should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota.
- `Postgres` should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit.
- `Redis` must have explicit memory limits and an intentional eviction/persistence policy.
- `Redis` must use explicit `maxmemory` below the container memory limit and must set an intentional `maxmemory-policy` instead of relying on the default unlimited dataset behavior.
- `Redis` should set `maxmemory-clients` so client buffers cannot grow without bound under connection pressure.
- `Redis` must not be allowed to consume unbounded RAM or disk on the target host.
- Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full.
- Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover.
- Done on `2026-04-12`: installed `PostgreSQL 16` and `Redis 7` on `sol` and started both services.
- Done on `2026-04-12`: moved `Postgres` data directory to `/srv/trade/postgres/16/main`.
- Done on `2026-04-12`: applied `Postgres` guardrails on `sol`: `shared_buffers=2GB`, `effective_cache_size=8GB`, `work_mem=16MB`, `maintenance_work_mem=512MB`, `temp_file_limit=8GB`, `max_wal_size=4GB`, `min_wal_size=512MB`, `max_slot_wal_keep_size=2GB`, `idle_in_transaction_session_timeout=10min`.
- Done on `2026-04-12`: prepared and tested operational `Postgres` read-only switch at `/usr/local/bin/trade-postgres-readonly`.
- Done on `2026-04-12`: applied `Redis` guardrails on `sol`: `maxmemory=384MB`, `maxmemory-clients=64MB`, `maxmemory-policy=noeviction`, `save \"\"`, `appendonly no`.
- Done on `2026-04-12`: initialized `Redis` as a single-node cluster with all `16384` slots assigned and `cluster_state: ok`.

### 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization

- Live `k3s` is the input for reconstruction, not the final source of truth.
- Split application repositories, Kubernetes manifests, and host IaC into separate repositories.
- Prefer a new Gitea organization over long-lived migration branches in existing repositories.
- Recommended target organization name: `trade-next`.
- Target repository set in `trade-next`: `trade-api`, `trade-frontend`, `trade-ingestor`, `trade-bot`, `trade-dlob`, `trade-gitops`, `trade-host-iac`, `trade-docs`.
- Keep the current repository names where possible and change only the organization to reduce migration risk.
- Use `trade-gitops` as the GitOps source for `k3s` manifests and `trade-host-iac` for host-level IaC, SSH, storage, and operations.
- Record every current live drift before moving workloads.
- Done on `2026-04-12`: `trade-next` was created and the target repositories now exist.
- Done on `2026-04-12`: the live component boundary map was written to `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- Next for item `02`: reconstruct the live `trade-staging` objects, overlays, secrets wiring, and host-specific drift into `trade-gitops` and `trade-host-iac`.

### 03. Fix `dlob-publisher-hot` startup, readiness, and liveness

- Pending.

### 04. Fix `dlob-publisher-all` startup, readiness, and liveness

- Pending.

### 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster

- Pending.

### 06. Align publisher Redis client mode with the actual deployed Redis topology

- Pending.

### 07. Restore or recreate the missing `gitea-registry` image pull secret

- Pending.

### 08. Verify that image pulls work after node reboot and on a cold start

- Pending.

### 09. Measure the real PostgreSQL data size inside the current trade PVC

- Pending.

### 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk

- Pending.

### 11. Decide which trade datasets must be migrated and which can be rebuilt

- Pending.

### 12. Define a dedicated storage path or device for trade data on `mevnode`

- Done on `2026-04-12`.
- Current preferred target layout on `sol` is to place trade persistent storage on the root NVMe filesystem `/`, because validator ledger and validator state already have their own dedicated devices.
- Do not place `Postgres` or `Redis` on `/data/ledger` or `/data/state`.
- Reserve a dedicated `1 TiB` storage budget for `Postgres` on `sol` as the first target sizing for `R001`.
- Reserve a separate small storage budget for `Redis` on `sol`, initially `2 GiB` unless persistence requirements change.
- Prefer a dedicated path such as `/srv/trade/postgres` and `/srv/trade/redis` over implicit storage under `/var/lib/rancher/k3s/storage`.
- On the current `btrfs` root filesystem, the preferred implementation path is a dedicated subvolume plus enforced `qgroup` quota for each persistent component.
- Done on `2026-04-12`: created `btrfs` subvolumes on `sol` at `/srv/trade`, `/srv/trade/postgres`, and `/srv/trade/redis`.

### 13. Define hard storage quotas and monitoring for trade data on `mevnode`

- In progress.
- Use enforceable limits, not only declarative PVC sizes without backend enforcement.
- Add warning and critical thresholds for disk usage, inode usage, and PVC growth.
- Target hard quota for `Postgres` on `sol`: `1 TiB`.
- Target hard quota for `Redis` on `sol`: `2 GiB` unless later requirements justify persistence growth.
- Keep enough unallocated free space on the root NVMe for the OS, `k3s`, container images, logs, and rollback operations after trade quotas are reserved.
- Done on `2026-04-12`: enabled `btrfs` qgroups on `/` at `sol` and applied hard quotas of `1 TiB` to `/srv/trade/postgres` and `2 GiB` to `/srv/trade/redis`.

### 14. Define CPU and memory reservations that protect `agave-validator`

- Pending.

### 15. Define the minimum safe free RAM headroom on `mevnode` during rollout

- Pending.

### 16. Define the maximum acceptable swap usage on `mevnode` during rollout

- Pending.

### 17. Install and bootstrap `k3s` on `mevnode`

- Pending.
- Define the target `k3s` installation model on `mevnode`.
- Define the target storage class, host paths, and bootstrap dependencies needed before application rollout.
- Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed.

### 18. Install `Postgres` and `Redis` on `mevnode`

- Done on `2026-04-12` for host-level bootstrap.
- Bring up the data plane first: `Postgres` and `Redis`.
- Validate persistence, storage placement, service discovery, restart behavior, and cold start.
- Validate that `Postgres` lands on the dedicated `1 TiB` trade storage area and that `Redis` lands on its separate small quota before application rollout.
- Do not install application components until `Postgres` and `Redis` are healthy.
- Done on `2026-04-12`: `Postgres` and `Redis` are running on `sol` as host services, both bound to `127.0.0.1` at this stage.
- Done on `2026-04-12`: validated `Postgres` on `/srv/trade/postgres/16/main`, validated `Redis` on `/srv/trade/redis`, and confirmed `btrfs` quotas are active for both paths.

### 19. Prepare a minimal canary namespace for trade on `mevnode`

- Pending.

### 20. Bring up `Hasura`, API, and frontend on `mevnode`

- Pending.
- Bring up `Hasura` only after `Postgres` is healthy.
- Bring up API and frontend only after `Hasura`, `Postgres`, and `Redis` are healthy.

### 21. Migrate publishers and ingest only after storage and Redis validation pass

- Pending.

### 22. Validate host health, agave health, trade health, and end-to-end flow after rollout

- Pending.

### 23. Keep `mevnode_bot` as a rollback target until the new stack is stable

- Pending.

## Chapter 3

- [ ] 01. Define hard database and cache constraints.
- [ ] 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization.
- [ ] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness.
- [ ] 04. Fix `dlob-publisher-all` startup, readiness, and liveness.
- [ ] 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster.
- [ ] 06. Align publisher Redis client mode with the actual deployed Redis topology.
- [ ] 07. Restore or recreate the missing `gitea-registry` image pull secret.
- [ ] 08. Verify that image pulls work after node reboot and on a cold start.
- [ ] 09. Measure the real PostgreSQL data size inside the current trade PVC.
- [ ] 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk.
- [ ] 11. Decide which trade datasets must be migrated and which can be rebuilt.
- [x] 12. Define a dedicated storage path or device for trade data on `mevnode`.
- [ ] 13. Define hard storage quotas and monitoring for trade data on `mevnode`.
- [ ] 14. Define CPU and memory reservations that protect `agave-validator`.
- [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout.
- [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout.
- [ ] 17. Install and bootstrap `k3s` on `mevnode`.
- [x] 18. Install `Postgres` and `Redis` on `mevnode`.
- [ ] 19. Prepare a minimal canary namespace for trade on `mevnode`.
- [ ] 20. Bring up `Hasura`, API, and frontend on `mevnode`.
- [ ] 21. Migrate publishers and ingest only after storage and Redis validation pass.
- [ ] 22. Validate host health, agave health, trade health, and end-to-end flow after rollout.
- [ ] 23. Keep `mevnode_bot` as a rollback target until the new stack is stable.