trade-docs/doc/migration-todo.md

Trade Migration Plan: k3s on mevnode alongside agave

Date: 2026-04-12

Migration Revision Baseline

• Accepted migration baseline revision: R001
• R001 means the currently working trade application running on k3s before migration cutover.
• The current source runtime for R001 is the live trade-staging stack on mevnode_bot.
• The first migration target is not a redesign. It is a faithful reconstruction of R001 on the target k3s environment.
• Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after R001 is reproducible.
• During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against R001.

Goal

• Prepare migration of the trade system to k3s on host mevnode (sol) in parallel with agave-validator.
• Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts.

Current Findings

• mevnode_bot is reachable again after restart.
• Previous reset on mevnode_bot looks like ACPI power state transition occurred.
• I did not find a clear OOM, kernel panic, watchdog, or I/O error trail explaining the previous host reset.
• k3s on mevnode_bot is up and the node is Ready.
• The main runtime failure is not the host itself but the DLOB publishers:
  • dlob-publisher-hot is 0/1
  • dlob-publisher-all is 0/1
  • both have very high restart counts
  • both fail startup/liveness probes
• Publisher logs show Redis connectivity problems:
  • ClusterAllFailedError: Failed to refresh slots cache
  • None of startup nodes is available
• dlob-redis currently exposes a single ClusterIP endpoint on 6379, while the publishers behave like cluster-aware Redis clients.
• Hasura, trade-api, trade-frontend, trade-ingestor, postgres, and the writer pods are currently up.
• trade-ingestor had transient fetch failures after reboot, then resumed inserts.
• There is a persistent warning in k3s about missing image pull secret gitea-registry.

Resource Picture: mevnode

• CPU: 32 vCPU
• RAM total: about 186 GiB
• RAM available: about 111 GiB at check time
• Swap total: about 31 GiB
• Swap used: about 13 GiB
• Root filesystem /: btrfs, about 1.8 TiB total, about 1.6 TiB free
• /data/ledger lives on a separate 1.8 TiB NVMe device.
• /data/state lives on another separate 1.8 TiB NVMe device.
• The current candidate place for trade persistent storage on sol is the root NVMe filesystem /, not the validator ledger or accounts devices.
• agave-validator RSS: about 129 GiB
• agave-validator is already a heavy resident workload and must remain the priority workload on this host.

Resource Picture: mevnode_bot

• Root filesystem /: ext4 on md RAID, about 3.6 TiB total, about 2.8 TiB free
• Current memory use after reboot is low, but this is not the main constraint.
• The main storage warning is current k3s data growth:
  • /var/lib/rancher/k3s: about 593 GiB
  • /var/lib/rancher/k3s/storage/...data-postgres-0: about 585 GiB
  • the related PVC is declared as only 20Gi
• This means the current local-path storage setup does not give a meaningful hard size limit in practice.
• The target Gitea organization trade-next now exists.
• The target repository set now exists in trade-next:
  • trade-api
  • trade-frontend
  • trade-ingestor
  • trade-bot
  • trade-dlob
  • trade-gitops
  • trade-host-iac
  • trade-docs
• The current live component-to-repository boundary was documented in trade-next/trade-docs/doc/k3s-component-repo-map.md.
• We have not yet reconstructed the current live trade-staging manifests and host drift into trade-gitops and trade-host-iac.
• The first canonical bootstrap module now exists in trade-next/trade-gitops: bootstrap/gitea-actions.
• An organization-scoped Gitea Actions runner for trade-next is now online on sol and can reach the target k3s cluster using the K3S_KUBECONFIG_B64 org secret.
• The first canary environment now exists in trade-next/trade-gitops under environments/sol/trade-r001-canary.
• Namespace trade-r001-canary is live on sol with initial quota and limit guardrails, and can reach the shared Postgres and Redis services in trade-infra.
• The first application surface for R001 is now live in trade-r001-canary: Hasura, trade-api, and trade-frontend.
• The target host Postgres on sol now has TimescaleDB installed and a matching application database/user provisioned for R001.
• trade-ingestor is now live in trade-r001-canary on sol in a schema-compatible canary mode that reads from dlob_stats_latest.
• Current live trade-staging drift was confirmed for trade-ingestor: the live script now reads dlob_hot_derived_latest and dlob_all_derived_latest, but that derived writer chain has not yet been reconstructed in the sol canary schema.
• agave RPC on sol is currently healthy; repeated samples after the canary rollout show slot lag of about 2 versus api.mainnet-beta.solana.com.

Migration Risks

• Risk 1: moving the current storage model unchanged to mevnode can silently consume hundreds of GiB on the host root filesystem.
• Risk 2: agave-validator already uses most of the memory footprint on mevnode; uncontrolled k3s workloads can force more swap or destabilize validator performance.
• Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade.
• Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology.
• Risk 5: missing gitea-registry secret can break future image pulls and cold-start behavior.

Plan

1. Reconstruct the current live mevnode_bot state into canonical Git/IaC

• Map each live trade-staging object to a target repository and file path.
• Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth.
• Capture the current live deltas that are not present in local repos or in ~/trade-k8s.
• Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos.
• Treat live k3s as input for reconstruction, not as a YAML dump to mirror blindly.

2. Stabilize the current trade stack on mevnode_bot

• Fix dlob-publisher-hot.
• Fix dlob-publisher-all.
• Verify Redis mode and publisher Redis client mode are aligned.
• Confirm all trade-staging pods become healthy after a full restart.

3. Audit storage before any migration

• Measure actual PostgreSQL data size inside the current PVC.
• Determine how much of the 585 GiB is real database payload versus local-path overhead.
• Identify retention candidates for historical data, derived tables, logs, and caches.
• Decide what data must move and what can be rebuilt.

4. Define target storage layout on mevnode

• Do not place trade persistent data on the same critical paths as agave ledger/accounts.
• Do not rely on the current local-path behavior without hard operational limits.
• Choose a dedicated storage path or dedicated device for trade state.
• Set explicit quotas/limits and monitor actual bytes used on disk.

5. Define resource isolation on mevnode

• Reserve headroom for agave-validator.
• Add CPU and memory requests/limits for trade workloads.
• Keep low-priority or bursty trade jobs away from validator-critical resources.
• Treat publisher and DB workloads as separate scheduling/priority classes.

6. Recreate the missing operational prerequisites

• Restore or recreate gitea-registry image pull secret.
• Verify registry pull works after node reboot and after cache eviction.
• Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible.

7. Prepare a staged migration

• Start with a minimal canary namespace on mevnode.
• Bring up infrastructure first: Redis, Postgres, Hasura.
• Then bring up API and frontend.
• Migrate publishers and ingest only after storage and Redis behavior are validated.
• Keep mevnode_bot as rollback target until the new stack is stable.

8. Add migration validation checks

• Host-level: memory, swap, disk, inode, and load checks.
• Agave-level: slot lag, RPC health, and validator memory trend.
• Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success.
• End-to-end: UI, API, DLOB data freshness, and bot-related paths.

Immediate Next Actions

• Map the current live trade-staging state into target Git repositories and IaC paths.
• Start the first canonical inventory pass into trade-gitops and trade-host-iac.
• Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode.
• Inspect Redis server configuration and topology in trade-staging.
• Measure real Postgres database size inside the trade PVC.
• Draft target disk placement for trade data on mevnode.
• Decide minimum safe RAM headroom to keep free for agave-validator.

Open Questions

• Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history?
• Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos?
• Is the intended Redis topology for trade a single Redis instance or Redis Cluster?
• Which trade datasets are required to migrate, and which can be rebuilt from source?
• Where exactly on mevnode should persistent trade storage live?
• What is the acceptable validator memory floor and swap ceiling during trade rollout?

Chapter 2

01. Define hard database and cache constraints

• In progress.
• Storage limits for Postgres and Redis must be enforced by the real storage backend, not only declared in Kubernetes manifests.
• In core Postgres, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached.
• Postgres must not be able to grow beyond the actually allocated disk budget.
• Postgres must have explicit CPU and memory requests/limits consistent with host headroom.
• Postgres should use temp_file_limit so large sorts and hashes fail before they can consume uncontrolled temporary disk space.
• Postgres must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat.
• Postgres should use bounded max_wal_size and, if replication slots are present, bounded max_slot_wal_keep_size, while treating WAL limits as early control signals rather than the final hard stop.
• Postgres should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota.
• Postgres should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit.
• Redis must have explicit memory limits and an intentional eviction/persistence policy.
• Redis must use explicit maxmemory below the container memory limit and must set an intentional maxmemory-policy instead of relying on the default unlimited dataset behavior.
• Redis should set maxmemory-clients so client buffers cannot grow without bound under connection pressure.
• Redis must not be allowed to consume unbounded RAM or disk on the target host.
• Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full.
• Full-disk and out-of-memory behavior for Postgres and Redis must be tested before application cutover.
• Done on 2026-04-12: installed PostgreSQL 16 and Redis 7 on sol and started both services.
• Done on 2026-04-12: moved Postgres data directory to /srv/trade/postgres/16/main.
• Done on 2026-04-12: applied Postgres guardrails on sol: shared_buffers=2GB, effective_cache_size=8GB, work_mem=16MB, maintenance_work_mem=512MB, temp_file_limit=8GB, max_wal_size=4GB, min_wal_size=512MB, max_slot_wal_keep_size=2GB, idle_in_transaction_session_timeout=10min.
• Done on 2026-04-12: prepared and tested operational Postgres read-only switch at /usr/local/bin/trade-postgres-readonly.
• Done on 2026-04-12: applied Redis guardrails on sol: maxmemory=384MB, maxmemory-clients=64MB, maxmemory-policy=noeviction, save \"\", appendonly no.
• Done on 2026-04-12: initialized Redis as a single-node cluster with all 16384 slots assigned and cluster_state: ok.

02. Reconstruct the current live mevnode_bot and trade-staging state into canonical Git/IaC in a new Gitea organization

• Live k3s is the input for reconstruction, not the final source of truth.
• Split application repositories, Kubernetes manifests, and host IaC into separate repositories.
• Prefer a new Gitea organization over long-lived migration branches in existing repositories.
• Recommended target organization name: trade-next.
• Target repository set in trade-next: trade-api, trade-frontend, trade-ingestor, trade-bot, trade-dlob, trade-gitops, trade-host-iac, trade-docs.
• Keep the current repository names where possible and change only the organization to reduce migration risk.
• Use trade-gitops as the GitOps source for k3s manifests and trade-host-iac for host-level IaC, SSH, storage, and operations.
• Record every current live drift before moving workloads.
• Done on 2026-04-12: trade-next was created and the target repositories now exist.
• Done on 2026-04-12: the live component boundary map was written to trade-next/trade-docs/doc/k3s-component-repo-map.md.
• Done on 2026-04-12: bootstrapped trade-next/trade-gitops with the first runnable IaC module for sol under bootstrap/gitea-actions.
• Next for item 02: reconstruct the live trade-staging objects, overlays, secrets wiring, and host-specific drift into trade-gitops and trade-host-iac.

03. Fix dlob-publisher-hot startup, readiness, and liveness

• Pending.

04. Fix dlob-publisher-all startup, readiness, and liveness

• Pending.

05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster

• Pending.

06. Align publisher Redis client mode with the actual deployed Redis topology

• Pending.

07. Restore or recreate the missing gitea-registry image pull secret

• Pending.

08. Verify that image pulls work after node reboot and on a cold start

• Pending.

09. Measure the real PostgreSQL data size inside the current trade PVC

• Pending.

10. Explain why the declared 20Gi PVC currently consumes about 585 GiB on disk

• Pending.

11. Decide which trade datasets must be migrated and which can be rebuilt

• Pending.

12. Define a dedicated storage path or device for trade data on mevnode

• Done on 2026-04-12.
• Current preferred target layout on sol is to place trade persistent storage on the root NVMe filesystem /, because validator ledger and validator state already have their own dedicated devices.
• Do not place Postgres or Redis on /data/ledger or /data/state.
• Reserve a dedicated 1 TiB storage budget for Postgres on sol as the first target sizing for R001.
• Reserve a separate small storage budget for Redis on sol, initially 2 GiB unless persistence requirements change.
• Prefer a dedicated path such as /srv/trade/postgres and /srv/trade/redis over implicit storage under /var/lib/rancher/k3s/storage.
• On the current btrfs root filesystem, the preferred implementation path is a dedicated subvolume plus enforced qgroup quota for each persistent component.
• Done on 2026-04-12: created btrfs subvolumes on sol at /srv/trade, /srv/trade/postgres, and /srv/trade/redis.

13. Define hard storage quotas and monitoring for trade data on mevnode

• In progress.
• Use enforceable limits, not only declarative PVC sizes without backend enforcement.
• Add warning and critical thresholds for disk usage, inode usage, and PVC growth.
• Target hard quota for Postgres on sol: 1 TiB.
• Target hard quota for Redis on sol: 2 GiB unless later requirements justify persistence growth.
• Keep enough unallocated free space on the root NVMe for the OS, k3s, container images, logs, and rollback operations after trade quotas are reserved.
• Done on 2026-04-12: enabled btrfs qgroups on / at sol and applied hard quotas of 1 TiB to /srv/trade/postgres and 2 GiB to /srv/trade/redis.

14. Define CPU and memory reservations that protect agave-validator

• Pending.

15. Define the minimum safe free RAM headroom on mevnode during rollout

• Pending.

16. Define the maximum acceptable swap usage on mevnode during rollout

• Pending.

17. Install and bootstrap k3s on mevnode

• Done on 2026-04-12.
• Define the target k3s installation model on mevnode.
• Define the target storage class, host paths, and bootstrap dependencies needed before application rollout.
• Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed.
• Done on 2026-04-12: installed single-node k3s on sol as v1.34.6+k3s1.
• Done on 2026-04-12: installed k3s in a minimal profile with traefik, servicelb, local-storage, and network-policy disabled.
• Done on 2026-04-12: validated core addons on sol; both coredns and metrics-server are healthy.
• Done on 2026-04-12: prepared shared namespace trade-infra with ClusterIP services postgres-host and redis-host backed by host EndpointSlice targets on 149.50.96.162.
• Done on 2026-04-12: opened ufw access on cni0 only for pod CIDR 10.42.0.0/24 to 6443, 10250, 5432, and 6379.
• Done on 2026-04-12: validated pod -> service -> host connectivity from a test pod to both Postgres and Redis.

18. Install Postgres and Redis on mevnode

• Done on 2026-04-12 for host-level bootstrap.
• Bring up the data plane first: Postgres and Redis.
• Validate persistence, storage placement, service discovery, restart behavior, and cold start.
• Validate that Postgres lands on the dedicated 1 TiB trade storage area and that Redis lands on its separate small quota before application rollout.
• Do not install application components until Postgres and Redis are healthy.
• Done on 2026-04-12: Postgres and Redis are running on sol as host services, bound to 127.0.0.1 and 149.50.96.162 for k3s pod access.
• Done on 2026-04-12: validated Postgres on /srv/trade/postgres/16/main, validated Redis on /srv/trade/redis, and confirmed btrfs quotas are active for both paths.
• Done on 2026-04-12: Postgres listens with pod CIDR access in pg_hba.conf, and Redis announces 149.50.96.162 for cluster-aware clients inside k3s.
• Done on 2026-04-12: installed TimescaleDB 2.26.2 packages on host Postgres for sol, enabled shared_preload_libraries=timescaledb, and provisioned application role/database admin/crypto matching the live trade-postgres secret from trade-staging.

19. Bootstrap an organization-scoped Gitea Actions runner on mevnode

• Done on 2026-04-12.
• Use trade-next/trade-gitops as the canonical bootstrap source, not an ad-hoc shell history on the host.
• Scope the runner at the trade-next organization level so every migration repository can reuse the same execution plane.
• Keep the first bootstrap simple: one runner replica on sol, one deployer service account, and one org secret carrying a kubeconfig for cluster access.
• Keep the runner state persistent only where necessary; avoid long-lived Docker layer storage until image caching strategy is explicitly planned.
• Done on 2026-04-12: created trade-next/trade-gitops initial main branch with bootstrap/gitea-actions manifests, scripts, and a smoke workflow.
• Done on 2026-04-12: deployed trade-next-act-runner in namespace gitea-actions on sol using docker.io/gitea/act_runner:latest with a docker:27-dind sidecar.
• Done on 2026-04-12: created trade-gitops-deployer service account and initial cluster-admin binding for first-pass GitOps bootstrap.
• Done on 2026-04-12: synced K3S_KUBECONFIG_B64 as an organization secret in trade-next.
• Done on 2026-04-12: verified the runner is online in Gitea with labels ubuntu-latest and k3s-deploy.
• Done on 2026-04-12: verified workflow execution from trade-next/trade-gitops; commit c566978 completed with status success for runner-smoke / smoke (push).

20. Prepare a minimal canary namespace for trade on mevnode

• Done on 2026-04-12.
• Use a dedicated canary namespace for R001, not the default namespace and not the infra namespace.
• Put namespace-level CPU, memory, object-count, and storage caps in place before the first application manifests are applied.
• Keep the first canary envelope conservative until the validator protection budget is finalized.
• Validate that the canary namespace can resolve and reach postgres-host.trade-infra.svc.cluster.local:5432 and redis-host.trade-infra.svc.cluster.local:6379.
• Done on 2026-04-12: created trade-next/trade-gitops environment module at environments/sol/trade-r001-canary.
• Done on 2026-04-12: applied namespace trade-r001-canary on sol with revision label R001 and canary environment labels.
• Done on 2026-04-12: applied ResourceQuota in trade-r001-canary with caps requests.cpu=2, limits.cpu=6, requests.memory=4Gi, limits.memory=12Gi, pods=20, services=10, configmaps=20, secrets=30, persistentvolumeclaims=4, requests.storage=100Gi.
• Done on 2026-04-12: applied LimitRange in trade-r001-canary with default request 100m/128Mi, default limit 1 CPU/1Gi, and per-container max 2 CPU/4Gi.
• Done on 2026-04-12: verified deployment through Gitea Actions; commit 060dcc3 completed with status success for deploy-trade-r001-canary / apply (push).
• Done on 2026-04-12: verified an in-cluster connectivity smoke test from trade-r001-canary to host-backed Postgres and Redis services.

21. Bring up Hasura, API, and frontend on mevnode

• Done on 2026-04-12.
• Bring up Hasura only after Postgres is healthy.
• Bring up API and frontend only after Hasura, Postgres, and Redis are healthy.
• Use trade-next/trade-gitops as the source of truth for the canary application surface, not ad-hoc manifests on the host.
• Keep the first application rollout internal to the cluster: ClusterIP services and health checks first, public ingress later.
• Reuse the live R001 application images and live application secrets, but point them at the target host Postgres on sol.
• Done on 2026-04-12: added Hasura, trade-api, and trade-frontend manifests to trade-next/trade-gitops under environments/sol/trade-r001-canary.
• Done on 2026-04-12: added operator scripts to trade-next/trade-gitops for sol host preparation, gitea-registry secret creation, and live secret sync from mevnode_bot/trade-staging into trade-r001-canary.
• Done on 2026-04-12: prepared trade-r001-canary secrets on sol for trade-postgres, trade-hasura, trade-api, trade-frontend-tokens, trade-basic-auth, and gitea-registry.
• Done on 2026-04-12: ran postgres-migrate and hasura-bootstrap jobs successfully on sol.
• Done on 2026-04-12: deployed Hasura, trade-api, and trade-frontend to trade-r001-canary; all three deployments reached Available.
• Done on 2026-04-12: verified internal HTTP health checks 200 for http://hasura:8080/healthz, http://trade-api:8787/healthz, and http://trade-frontend:8081/healthz.
• Done on 2026-04-12: verified GitOps execution through Gitea Actions; commit 6672e10 completed with status success for deploy-trade-r001-canary / apply (push).

22. Migrate publishers and ingest only after storage and Redis validation pass

• In progress.
• Substep A is now in place on sol: deploy trade-ingestor first in a schema-compatible canary mode and verify its secret wiring, rollout behavior, and Hasura connectivity.
• Substep B remains pending: reconstruct the live R001 DLOB derived writer chain and the dlob_*_derived_latest tables so the canary ingestor can switch from dlob_stats_latest to the exact current live query path.
• Done on 2026-04-12: extended trade-next/trade-gitops canary environment with trade-ingestor, synced trade-ingestor-tokens, and added trade-ingestor rollout checks to the deploy-trade-r001-canary workflow.
• Done on 2026-04-12: manually validated trade-ingestor on sol; deployment reached Available, restart count stayed at 0, and startup logs confirmed HASURA_GRAPHQL_URL=http://hasura:8080/v1/graphql, markets SOL-PERP,PUMP-PERP, and mode dlob_stats_ticks.
• Done on 2026-04-12: after push of commit b8f1303, the GitOps path on sol rolled a fresh trade-ingestor pod and it remained healthy with restart count 0.

23. Validate host health, agave health, trade health, and end-to-end flow after rollout

• Pending.

24. Keep mevnode_bot as a rollback target until the new stack is stable

• Pending.

Chapter 3

• ☐
  1. Define hard database and cache constraints.
• ☐
  2. Reconstruct the current live mevnode_bot and trade-staging state into canonical Git/IaC in a new Gitea organization.
• ☐
  3. Fix dlob-publisher-hot startup, readiness, and liveness.
• ☐
  4. Fix dlob-publisher-all startup, readiness, and liveness.
• ☐
  5. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster.
• ☐
  6. Align publisher Redis client mode with the actual deployed Redis topology.
• ☐
  7. Restore or recreate the missing gitea-registry image pull secret.
• ☐
  8. Verify that image pulls work after node reboot and on a cold start.
• ☐
  9. Measure the real PostgreSQL data size inside the current trade PVC.
• ☐
  10. Explain why the declared 20Gi PVC currently consumes about 585 GiB on disk.
• ☐
  11. Decide which trade datasets must be migrated and which can be rebuilt.
• ☒
  12. Define a dedicated storage path or device for trade data on mevnode.
• ☐
  13. Define hard storage quotas and monitoring for trade data on mevnode.
• ☐
  14. Define CPU and memory reservations that protect agave-validator.
• ☐
  15. Define the minimum safe free RAM headroom on mevnode during rollout.
• ☐
  16. Define the maximum acceptable swap usage on mevnode during rollout.
• ☒
  17. Install and bootstrap k3s on mevnode.
• ☒
  18. Install Postgres and Redis on mevnode.
• ☒
  19. Bootstrap an organization-scoped Gitea Actions runner on mevnode.
• ☒
  20. Prepare a minimal canary namespace for trade on mevnode.
• ☒
  21. Bring up Hasura, API, and frontend on mevnode.
• ☐
  22. Migrate publishers and ingest only after storage and Redis validation pass.
• ☐
  23. Validate host health, agave health, trade health, and end-to-end flow after rollout.
• ☐
  24. Keep mevnode_bot as a rollback target until the new stack is stable.