trade-docs/doc/migration-todo.md

Trade Migration Plan: k3s on mevnode alongside agave

Date: 2026-04-12

Migration Revision Baseline

• Accepted migration baseline revision: R001
• R001 means the currently working trade application running on k3s before migration cutover.
• The current source runtime for R001 is the live trade-staging stack on mevnode_bot.
• The first migration target is not a redesign. It is a faithful reconstruction of R001 on the target k3s environment.
• Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after R001 is reproducible.
• During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against R001.

Goal

• Prepare migration of the trade system to k3s on host mevnode (sol) in parallel with agave-validator.
• Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts.

Current Findings

• mevnode_bot is reachable again after restart.
• Previous reset on mevnode_bot looks like ACPI power state transition occurred.
• I did not find a clear OOM, kernel panic, watchdog, or I/O error trail explaining the previous host reset.
• k3s on mevnode_bot is up and the node is Ready.
• The main runtime failure is not the host itself but the DLOB publishers:
  • dlob-publisher-hot is 0/1
  • dlob-publisher-all is 0/1
  • both have very high restart counts
  • both fail startup/liveness probes
• Publisher logs show Redis connectivity problems:
  • ClusterAllFailedError: Failed to refresh slots cache
  • None of startup nodes is available
• dlob-redis currently exposes a single ClusterIP endpoint on 6379, while the publishers behave like cluster-aware Redis clients.
• Hasura, trade-api, trade-frontend, trade-ingestor, postgres, and the writer pods are currently up.
• trade-ingestor had transient fetch failures after reboot, then resumed inserts.
• There is a persistent warning in k3s about missing image pull secret gitea-registry.

Resource Picture: mevnode

• CPU: 32 vCPU
• RAM total: about 186 GiB
• RAM available: about 111 GiB at check time
• Swap total: about 31 GiB
• Swap used: about 13 GiB
• Root filesystem /: btrfs, about 1.8 TiB total, about 1.6 TiB free
• /data/ledger lives on a separate 1.8 TiB NVMe device.
• /data/state lives on another separate 1.8 TiB NVMe device.
• The current candidate place for trade persistent storage on sol is the root NVMe filesystem /, not the validator ledger or accounts devices.
• agave-validator RSS: about 129 GiB
• agave-validator is already a heavy resident workload and must remain the priority workload on this host.

Resource Picture: mevnode_bot

• Root filesystem /: ext4 on md RAID, about 3.6 TiB total, about 2.8 TiB free
• Current memory use after reboot is low, but this is not the main constraint.
• The main storage warning is current k3s data growth:
  • /var/lib/rancher/k3s: about 593 GiB
  • /var/lib/rancher/k3s/storage/...data-postgres-0: about 585 GiB
  • the related PVC is declared as only 20Gi
• This means the current local-path storage setup does not give a meaningful hard size limit in practice.
• The target Gitea organization trade-next now exists.
• The target repository set now exists in trade-next:
  • trade-api
  • trade-frontend
  • trade-ingestor
  • trade-bot
  • trade-dlob
  • trade-gitops
  • trade-host-iac
  • trade-docs
• The current live component-to-repository boundary was documented in trade-next/trade-docs/doc/k3s-component-repo-map.md.
• We have not yet reconstructed the current live trade-staging manifests and host drift into trade-gitops and trade-host-iac.

Migration Risks

• Risk 1: moving the current storage model unchanged to mevnode can silently consume hundreds of GiB on the host root filesystem.
• Risk 2: agave-validator already uses most of the memory footprint on mevnode; uncontrolled k3s workloads can force more swap or destabilize validator performance.
• Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade.
• Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology.
• Risk 5: missing gitea-registry secret can break future image pulls and cold-start behavior.

Plan

1. Reconstruct the current live mevnode_bot state into canonical Git/IaC

• Map each live trade-staging object to a target repository and file path.
• Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth.
• Capture the current live deltas that are not present in local repos or in ~/trade-k8s.
• Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos.
• Treat live k3s as input for reconstruction, not as a YAML dump to mirror blindly.

2. Stabilize the current trade stack on mevnode_bot

• Fix dlob-publisher-hot.
• Fix dlob-publisher-all.
• Verify Redis mode and publisher Redis client mode are aligned.
• Confirm all trade-staging pods become healthy after a full restart.

3. Audit storage before any migration

• Measure actual PostgreSQL data size inside the current PVC.
• Determine how much of the 585 GiB is real database payload versus local-path overhead.
• Identify retention candidates for historical data, derived tables, logs, and caches.
• Decide what data must move and what can be rebuilt.

4. Define target storage layout on mevnode

• Do not place trade persistent data on the same critical paths as agave ledger/accounts.
• Do not rely on the current local-path behavior without hard operational limits.
• Choose a dedicated storage path or dedicated device for trade state.
• Set explicit quotas/limits and monitor actual bytes used on disk.

5. Define resource isolation on mevnode

• Reserve headroom for agave-validator.
• Add CPU and memory requests/limits for trade workloads.
• Keep low-priority or bursty trade jobs away from validator-critical resources.
• Treat publisher and DB workloads as separate scheduling/priority classes.

6. Recreate the missing operational prerequisites

• Restore or recreate gitea-registry image pull secret.
• Verify registry pull works after node reboot and after cache eviction.
• Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible.

7. Prepare a staged migration

• Start with a minimal canary namespace on mevnode.
• Bring up infrastructure first: Redis, Postgres, Hasura.
• Then bring up API and frontend.
• Migrate publishers and ingest only after storage and Redis behavior are validated.
• Keep mevnode_bot as rollback target until the new stack is stable.

8. Add migration validation checks

• Host-level: memory, swap, disk, inode, and load checks.
• Agave-level: slot lag, RPC health, and validator memory trend.
• Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success.
• End-to-end: UI, API, DLOB data freshness, and bot-related paths.

Immediate Next Actions

• Map the current live trade-staging state into target Git repositories and IaC paths.
• Start the first canonical inventory pass into trade-gitops and trade-host-iac.
• Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode.
• Inspect Redis server configuration and topology in trade-staging.
• Measure real Postgres database size inside the trade PVC.
• Draft target disk placement for trade data on mevnode.
• Decide minimum safe RAM headroom to keep free for agave-validator.

Open Questions

• Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history?
• Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos?
• Is the intended Redis topology for trade a single Redis instance or Redis Cluster?
• Which trade datasets are required to migrate, and which can be rebuilt from source?
• Where exactly on mevnode should persistent trade storage live?
• What is the acceptable validator memory floor and swap ceiling during trade rollout?

Chapter 2

01. Define hard database and cache constraints

• In progress.
• Storage limits for Postgres and Redis must be enforced by the real storage backend, not only declared in Kubernetes manifests.
• In core Postgres, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached.
• Postgres must not be able to grow beyond the actually allocated disk budget.
• Postgres must have explicit CPU and memory requests/limits consistent with host headroom.
• Postgres should use temp_file_limit so large sorts and hashes fail before they can consume uncontrolled temporary disk space.
• Postgres must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat.
• Postgres should use bounded max_wal_size and, if replication slots are present, bounded max_slot_wal_keep_size, while treating WAL limits as early control signals rather than the final hard stop.
• Postgres should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota.
• Postgres should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit.
• Redis must have explicit memory limits and an intentional eviction/persistence policy.
• Redis must use explicit maxmemory below the container memory limit and must set an intentional maxmemory-policy instead of relying on the default unlimited dataset behavior.
• Redis should set maxmemory-clients so client buffers cannot grow without bound under connection pressure.
• Redis must not be allowed to consume unbounded RAM or disk on the target host.
• Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full.
• Full-disk and out-of-memory behavior for Postgres and Redis must be tested before application cutover.
• Done on 2026-04-12: installed PostgreSQL 16 and Redis 7 on sol and started both services.
• Done on 2026-04-12: moved Postgres data directory to /srv/trade/postgres/16/main.
• Done on 2026-04-12: applied Postgres guardrails on sol: shared_buffers=2GB, effective_cache_size=8GB, work_mem=16MB, maintenance_work_mem=512MB, temp_file_limit=8GB, max_wal_size=4GB, min_wal_size=512MB, max_slot_wal_keep_size=2GB, idle_in_transaction_session_timeout=10min.
• Done on 2026-04-12: prepared and tested operational Postgres read-only switch at /usr/local/bin/trade-postgres-readonly.
• Done on 2026-04-12: applied Redis guardrails on sol: maxmemory=384MB, maxmemory-clients=64MB, maxmemory-policy=noeviction, save \"\", appendonly no.
• Done on 2026-04-12: initialized Redis as a single-node cluster with all 16384 slots assigned and cluster_state: ok.

02. Reconstruct the current live mevnode_bot and trade-staging state into canonical Git/IaC in a new Gitea organization

• Live k3s is the input for reconstruction, not the final source of truth.
• Split application repositories, Kubernetes manifests, and host IaC into separate repositories.
• Prefer a new Gitea organization over long-lived migration branches in existing repositories.
• Recommended target organization name: trade-next.
• Target repository set in trade-next: trade-api, trade-frontend, trade-ingestor, trade-bot, trade-dlob, trade-gitops, trade-host-iac, trade-docs.
• Keep the current repository names where possible and change only the organization to reduce migration risk.
• Use trade-gitops as the GitOps source for k3s manifests and trade-host-iac for host-level IaC, SSH, storage, and operations.
• Record every current live drift before moving workloads.
• Done on 2026-04-12: trade-next was created and the target repositories now exist.
• Done on 2026-04-12: the live component boundary map was written to trade-next/trade-docs/doc/k3s-component-repo-map.md.
• Next for item 02: reconstruct the live trade-staging objects, overlays, secrets wiring, and host-specific drift into trade-gitops and trade-host-iac.

03. Fix dlob-publisher-hot startup, readiness, and liveness

• Pending.

04. Fix dlob-publisher-all startup, readiness, and liveness

• Pending.

05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster

• Pending.

06. Align publisher Redis client mode with the actual deployed Redis topology

• Pending.

07. Restore or recreate the missing gitea-registry image pull secret

• Pending.

08. Verify that image pulls work after node reboot and on a cold start

• Pending.

09. Measure the real PostgreSQL data size inside the current trade PVC

• Pending.

10. Explain why the declared 20Gi PVC currently consumes about 585 GiB on disk

• Pending.

11. Decide which trade datasets must be migrated and which can be rebuilt

• Pending.

12. Define a dedicated storage path or device for trade data on mevnode

• Done on 2026-04-12.
• Current preferred target layout on sol is to place trade persistent storage on the root NVMe filesystem /, because validator ledger and validator state already have their own dedicated devices.
• Do not place Postgres or Redis on /data/ledger or /data/state.
• Reserve a dedicated 1 TiB storage budget for Postgres on sol as the first target sizing for R001.
• Reserve a separate small storage budget for Redis on sol, initially 2 GiB unless persistence requirements change.
• Prefer a dedicated path such as /srv/trade/postgres and /srv/trade/redis over implicit storage under /var/lib/rancher/k3s/storage.
• On the current btrfs root filesystem, the preferred implementation path is a dedicated subvolume plus enforced qgroup quota for each persistent component.
• Done on 2026-04-12: created btrfs subvolumes on sol at /srv/trade, /srv/trade/postgres, and /srv/trade/redis.

13. Define hard storage quotas and monitoring for trade data on mevnode

• In progress.
• Use enforceable limits, not only declarative PVC sizes without backend enforcement.
• Add warning and critical thresholds for disk usage, inode usage, and PVC growth.
• Target hard quota for Postgres on sol: 1 TiB.
• Target hard quota for Redis on sol: 2 GiB unless later requirements justify persistence growth.
• Keep enough unallocated free space on the root NVMe for the OS, k3s, container images, logs, and rollback operations after trade quotas are reserved.
• Done on 2026-04-12: enabled btrfs qgroups on / at sol and applied hard quotas of 1 TiB to /srv/trade/postgres and 2 GiB to /srv/trade/redis.

14. Define CPU and memory reservations that protect agave-validator

• Pending.

15. Define the minimum safe free RAM headroom on mevnode during rollout

• Pending.

16. Define the maximum acceptable swap usage on mevnode during rollout

• Pending.

17. Install and bootstrap k3s on mevnode

• Pending.
• Define the target k3s installation model on mevnode.
• Define the target storage class, host paths, and bootstrap dependencies needed before application rollout.
• Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed.

18. Install Postgres and Redis on mevnode

• Done on 2026-04-12 for host-level bootstrap.
• Bring up the data plane first: Postgres and Redis.
• Validate persistence, storage placement, service discovery, restart behavior, and cold start.
• Validate that Postgres lands on the dedicated 1 TiB trade storage area and that Redis lands on its separate small quota before application rollout.
• Do not install application components until Postgres and Redis are healthy.
• Done on 2026-04-12: Postgres and Redis are running on sol as host services, both bound to 127.0.0.1 at this stage.
• Done on 2026-04-12: validated Postgres on /srv/trade/postgres/16/main, validated Redis on /srv/trade/redis, and confirmed btrfs quotas are active for both paths.

19. Prepare a minimal canary namespace for trade on mevnode

• Pending.

20. Bring up Hasura, API, and frontend on mevnode

• Pending.
• Bring up Hasura only after Postgres is healthy.
• Bring up API and frontend only after Hasura, Postgres, and Redis are healthy.

21. Migrate publishers and ingest only after storage and Redis validation pass

• Pending.

22. Validate host health, agave health, trade health, and end-to-end flow after rollout

• Pending.

23. Keep mevnode_bot as a rollback target until the new stack is stable

• Pending.

Chapter 3

• ☐
  1. Define hard database and cache constraints.
• ☐
  2. Reconstruct the current live mevnode_bot and trade-staging state into canonical Git/IaC in a new Gitea organization.
• ☐
  3. Fix dlob-publisher-hot startup, readiness, and liveness.
• ☐
  4. Fix dlob-publisher-all startup, readiness, and liveness.
• ☐
  5. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster.
• ☐
  6. Align publisher Redis client mode with the actual deployed Redis topology.
• ☐
  7. Restore or recreate the missing gitea-registry image pull secret.
• ☐
  8. Verify that image pulls work after node reboot and on a cold start.
• ☐
  9. Measure the real PostgreSQL data size inside the current trade PVC.
• ☐
  10. Explain why the declared 20Gi PVC currently consumes about 585 GiB on disk.
• ☐
  11. Decide which trade datasets must be migrated and which can be rebuilt.
• ☒
  12. Define a dedicated storage path or device for trade data on mevnode.
• ☐
  13. Define hard storage quotas and monitoring for trade data on mevnode.
• ☐
  14. Define CPU and memory reservations that protect agave-validator.
• ☐
  15. Define the minimum safe free RAM headroom on mevnode during rollout.
• ☐
  16. Define the maximum acceptable swap usage on mevnode during rollout.
• ☐
  17. Install and bootstrap k3s on mevnode.
• ☒
  18. Install Postgres and Redis on mevnode.
• ☐
  19. Prepare a minimal canary namespace for trade on mevnode.
• ☐
  20. Bring up Hasura, API, and frontend on mevnode.
• ☐
  21. Migrate publishers and ingest only after storage and Redis validation pass.
• ☐
  22. Validate host health, agave health, trade health, and end-to-end flow after rollout.
• ☐
  23. Keep mevnode_bot as a rollback target until the new stack is stable.