# Trade Migration Plan: k3s on mevnode alongside agave

Date: `2026-04-12`

## Migration Revision Baseline

- Accepted migration baseline revision: `R001`
- `R001` means the currently working trade application running on `k3s` before migration cutover.
- The current source runtime for `R001` is the live `trade-staging` stack on `mevnode_bot`.
- The first migration target is not a redesign. It is a faithful reconstruction of `R001` on the target `k3s` environment.
- Functional changes, refactors, topology changes, and cleanup should be treated as later revisions after `R001` is reproducible.
- During reconstruction, every manifest, secret mapping, image reference, and host dependency should be mapped against `R001`.

## Goal

- Prepare migration of the trade system to `k3s` on host `mevnode` (`sol`) in parallel with `agave-validator`.
- Do not degrade validator stability, memory headroom, or disk/I/O for ledger and accounts.

## Current Findings

- `mevnode_bot` is reachable again after restart.
- Current `R001` mapping still shows no separate live `trade-bot` deployment in `trade-staging`; the live runtime observed for this migration wave is `hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, and the DLOB stack.
- Previous reset on `mevnode_bot` looks like `ACPI power state transition occurred`.
- I did not find a clear `OOM`, `kernel panic`, `watchdog`, or `I/O error` trail explaining the previous host reset.
- `k3s` on `mevnode_bot` is up and the node is `Ready`.
- The main runtime failure is not the host itself but the DLOB publishers:
  - `dlob-publisher-hot` is `0/1`
  - `dlob-publisher-all` is `0/1`
  - both have very high restart counts
  - both fail startup/liveness probes
- Publisher logs show Redis connectivity problems:
  - `ClusterAllFailedError: Failed to refresh slots cache`
  - `None of startup nodes is available`
- `dlob-redis` currently exposes a single `ClusterIP` endpoint on `6379`, while the publishers behave like cluster-aware Redis clients.
- `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, `postgres`, and the writer pods are currently up.
- `trade-ingestor` had transient fetch failures after reboot, then resumed inserts.
- There is a persistent warning in `k3s` about missing image pull secret `gitea-registry`.

### Resource Picture: mevnode

- CPU: `32 vCPU`
- RAM total: about `186 GiB`
- RAM available: about `111 GiB` at check time
- Swap total: about `31 GiB`
- Swap used: about `13 GiB`
- Root filesystem `/`: `btrfs`, about `1.8 TiB` total, about `1.6 TiB` free
- `/data/ledger` lives on a separate `1.8 TiB` NVMe device.
- `/data/state` lives on another separate `1.8 TiB` NVMe device.
- The current candidate place for trade persistent storage on `sol` is the root NVMe filesystem `/`, not the validator ledger or accounts devices.
- `agave-validator` RSS: about `129 GiB`
- `agave-validator` is already a heavy resident workload and must remain the priority workload on this host.

### Resource Picture: mevnode_bot

- Root filesystem `/`: `ext4` on `md RAID`, about `3.6 TiB` total, about `2.8 TiB` free
- Current memory use after reboot is low, but this is not the main constraint.
- The main storage warning is current `k3s` data growth:
  - `/var/lib/rancher/k3s`: about `593 GiB`
  - `/var/lib/rancher/k3s/storage/...data-postgres-0`: about `585 GiB`
  - the related PVC is declared as only `20Gi`
- This means the current `local-path` storage setup does not give a meaningful hard size limit in practice.
- The target Gitea organization `trade-next` now exists.
- The target repository set now exists in `trade-next`:
  - `trade-api`
  - `trade-frontend`
  - `trade-ingestor`
  - `trade-bot`
  - `trade-dlob`
  - `trade-gitops`
  - `trade-host-iac`
  - `trade-docs`
- The current live component-to-repository boundary was documented in `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- We have not yet reconstructed the current live `trade-staging` manifests and host drift into `trade-gitops` and `trade-host-iac`.
- The first canonical bootstrap module now exists in `trade-next/trade-gitops`: `bootstrap/gitea-actions`.
- An organization-scoped Gitea Actions runner for `trade-next` is now online on `sol` and can reach the target `k3s` cluster using the `K3S_KUBECONFIG_B64` org secret.
- The first canary environment now exists in `trade-next/trade-gitops` under `environments/sol/trade-r001-canary`.
- Namespace `trade-r001-canary` is live on `sol` with initial quota and limit guardrails, and can reach the shared `Postgres` and `Redis` services in `trade-infra`.
- The first application surface for `R001` is now live in `trade-r001-canary`: `Hasura`, `trade-api`, and `trade-frontend`.
- The target host `Postgres` on `sol` now has `TimescaleDB` installed and a matching application database/user provisioned for `R001`.
- `trade-ingestor` is now live in `trade-r001-canary` on `sol` in a schema-compatible canary mode that reads from `dlob_stats_latest`.
- Current live `trade-staging` drift was confirmed for `trade-ingestor`: the live script now reads `dlob_hot_derived_latest` and `dlob_all_derived_latest`, but that derived writer chain has not yet been reconstructed in the `sol` canary schema.
- `agave` RPC on `sol` is currently healthy; repeated samples after the canary rollout show slot lag of about `2` versus `api.mainnet-beta.solana.com`.

## Migration Risks

- Risk 1: moving the current storage model unchanged to `mevnode` can silently consume hundreds of GiB on the host root filesystem.
- Risk 2: `agave-validator` already uses most of the memory footprint on `mevnode`; uncontrolled `k3s` workloads can force more swap or destabilize validator performance.
- Risk 3: if trade workloads share disks or I/O paths with ledger/accounts, validator performance may degrade.
- Risk 4: current publisher instability must be fixed before migration, otherwise we would move a broken topology.
- Risk 5: missing `gitea-registry` secret can break future image pulls and cold-start behavior.

## Plan

### 1. Reconstruct the current live `mevnode_bot` state into canonical Git/IaC

- Map each live `trade-staging` object to a target repository and file path.
- Split application code, Kubernetes manifests, and host-level infrastructure into separate Git sources of truth.
- Capture the current live deltas that are not present in local repos or in `~/trade-k8s`.
- Prefer a new Gitea organization for the migration target instead of long-lived migration branches in existing repos.
- Treat live `k3s` as input for reconstruction, not as a YAML dump to mirror blindly.

### 2. Stabilize the current trade stack on `mevnode_bot`

- Fix `dlob-publisher-hot`.
- Fix `dlob-publisher-all`.
- Verify Redis mode and publisher Redis client mode are aligned.
- Confirm all `trade-staging` pods become healthy after a full restart.

### 3. Audit storage before any migration

- Measure actual PostgreSQL data size inside the current PVC.
- Determine how much of the `585 GiB` is real database payload versus local-path overhead.
- Identify retention candidates for historical data, derived tables, logs, and caches.
- Decide what data must move and what can be rebuilt.

### 4. Define target storage layout on `mevnode`

- Do not place trade persistent data on the same critical paths as agave ledger/accounts.
- Do not rely on the current local-path behavior without hard operational limits.
- Choose a dedicated storage path or dedicated device for trade state.
- Set explicit quotas/limits and monitor actual bytes used on disk.

### 5. Define resource isolation on `mevnode`

- Reserve headroom for `agave-validator`.
- Add CPU and memory requests/limits for trade workloads.
- Keep low-priority or bursty trade jobs away from validator-critical resources.
- Treat publisher and DB workloads as separate scheduling/priority classes.

### 6. Recreate the missing operational prerequisites

- Restore or recreate `gitea-registry` image pull secret.
- Verify registry pull works after node reboot and after cache eviction.
- Verify secrets for RPC, gRPC, Hasura, Postgres, and frontend auth are reproducible.

### 7. Prepare a staged migration

- Start with a minimal canary namespace on `mevnode`.
- Bring up infrastructure first: Redis, Postgres, Hasura.
- Then bring up API and frontend.
- Migrate publishers and ingest only after storage and Redis behavior are validated.
- Keep `mevnode_bot` as rollback target until the new stack is stable.

### 8. Add migration validation checks

- Host-level: memory, swap, disk, inode, and load checks.
- Agave-level: slot lag, RPC health, and validator memory trend.
- Trade-level: pod readiness, Redis health, Hasura health, API health, publisher health, ingest success.
- End-to-end: UI, API, DLOB data freshness, and bot-related paths.

## Immediate Next Actions

- Map the current live `trade-staging` state into target Git repositories and IaC paths.
- Start the first canonical inventory pass into `trade-gitops` and `trade-host-iac`.
- Inspect publisher deployment/env and confirm whether Redis should be single-instance or cluster mode.
- Inspect Redis server configuration and topology in `trade-staging`.
- Measure real Postgres database size inside the trade PVC.
- Draft target disk placement for trade data on `mevnode`.
- Decide minimum safe RAM headroom to keep free for `agave-validator`.

## Open Questions

- Which repositories belong in the new Gitea organization, and which existing repos should stay only as upstream history?
- Do we need a clean cutover into a new organization, or only a temporary compatibility layer with the current repos?
- Is the intended Redis topology for trade a single Redis instance or Redis Cluster?
- Which trade datasets are required to migrate, and which can be rebuilt from source?
- Where exactly on `mevnode` should persistent trade storage live?
- What is the acceptable validator memory floor and swap ceiling during trade rollout?

## Chapter 2

### 01. Define hard database and cache constraints

- In progress.
- Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests.
- In core `Postgres`, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached.
- `Postgres` must not be able to grow beyond the actually allocated disk budget.
- `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom.
- `Postgres` should use `temp_file_limit` so large sorts and hashes fail before they can consume uncontrolled temporary disk space.
- `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat.
- `Postgres` should use bounded `max_wal_size` and, if replication slots are present, bounded `max_slot_wal_keep_size`, while treating WAL limits as early control signals rather than the final hard stop.
- `Postgres` should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota.
- `Postgres` should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit.
- `Redis` must have explicit memory limits and an intentional eviction/persistence policy.
- `Redis` must use explicit `maxmemory` below the container memory limit and must set an intentional `maxmemory-policy` instead of relying on the default unlimited dataset behavior.
- `Redis` should set `maxmemory-clients` so client buffers cannot grow without bound under connection pressure.
- `Redis` must not be allowed to consume unbounded RAM or disk on the target host.
- Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full.
- Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover.
- Done on `2026-04-12`: installed `PostgreSQL 16` and `Redis 7` on `sol` and started both services.
- Done on `2026-04-12`: moved `Postgres` data directory to `/srv/trade/postgres/16/main`.
- Done on `2026-04-12`: applied `Postgres` guardrails on `sol`: `shared_buffers=2GB`, `effective_cache_size=8GB`, `work_mem=16MB`, `maintenance_work_mem=512MB`, `temp_file_limit=8GB`, `max_wal_size=4GB`, `min_wal_size=512MB`, `max_slot_wal_keep_size=2GB`, `idle_in_transaction_session_timeout=10min`.
- Done on `2026-04-12`: prepared and tested operational `Postgres` read-only switch at `/usr/local/bin/trade-postgres-readonly`.
- Done on `2026-04-12`: applied `Redis` guardrails on `sol`: `maxmemory=384MB`, `maxmemory-clients=64MB`, `maxmemory-policy=noeviction`, `save \"\"`, `appendonly no`.
- Done on `2026-04-12`: initialized `Redis` as a single-node cluster with all `16384` slots assigned and `cluster_state: ok`.

### 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization

- Live `k3s` is the input for reconstruction, not the final source of truth.
- Split application repositories, Kubernetes manifests, and host IaC into separate repositories.
- Prefer a new Gitea organization over long-lived migration branches in existing repositories.
- Recommended target organization name: `trade-next`.
- Target repository set in `trade-next`: `trade-api`, `trade-frontend`, `trade-ingestor`, `trade-bot`, `trade-dlob`, `trade-gitops`, `trade-host-iac`, `trade-docs`.
- Keep the current repository names where possible and change only the organization to reduce migration risk.
- Use `trade-gitops` as the GitOps source for `k3s` manifests and `trade-host-iac` for host-level IaC, SSH, storage, and operations.
- Record every current live drift before moving workloads.
- Done on `2026-04-12`: `trade-next` was created and the target repositories now exist.
- Done on `2026-04-12`: the live component boundary map was written to `trade-next/trade-docs/doc/k3s-component-repo-map.md`.
- Done on `2026-04-12`: bootstrapped `trade-next/trade-gitops` with the first runnable IaC module for `sol` under `bootstrap/gitea-actions`.
- Done on `2026-04-12`: committed the `sol` public-edge host and cluster modules to `trade-next/trade-host-iac` and `trade-next/trade-gitops`.
- Done on `2026-04-12`: `sol` now serves the reconstructed `R001` frontend on public `HTTP/HTTPS` at `149.50.96.162`; the remaining production cutover step is authoritative DNS `trade.mpabi.pl -> 149.50.96.162` so Let's Encrypt can finish issuance.
- Next for item `02`: reconstruct the live `trade-staging` objects, overlays, secrets wiring, and host-specific drift into `trade-gitops` and `trade-host-iac`.

### 03. Fix `dlob-publisher-hot` startup, readiness, and liveness

- Pending.

### 04. Fix `dlob-publisher-all` startup, readiness, and liveness

- Pending.

### 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster

- Pending.

### 06. Align publisher Redis client mode with the actual deployed Redis topology

- Pending.

### 07. Restore or recreate the missing `gitea-registry` image pull secret

- Pending.

### 08. Verify that image pulls work after node reboot and on a cold start

- Pending.

### 09. Measure the real PostgreSQL data size inside the current trade PVC

- Pending.

### 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk

- Pending.

### 11. Decide which trade datasets must be migrated and which can be rebuilt

- Pending.

### 12. Define a dedicated storage path or device for trade data on `mevnode`

- Done on `2026-04-12`.
- Current preferred target layout on `sol` is to place trade persistent storage on the root NVMe filesystem `/`, because validator ledger and validator state already have their own dedicated devices.
- Do not place `Postgres` or `Redis` on `/data/ledger` or `/data/state`.
- Reserve a dedicated `1 TiB` storage budget for `Postgres` on `sol` as the first target sizing for `R001`.
- Reserve a separate small storage budget for `Redis` on `sol`, initially `2 GiB` unless persistence requirements change.
- Prefer a dedicated path such as `/srv/trade/postgres` and `/srv/trade/redis` over implicit storage under `/var/lib/rancher/k3s/storage`.
- On the current `btrfs` root filesystem, the preferred implementation path is a dedicated subvolume plus enforced `qgroup` quota for each persistent component.
- Done on `2026-04-12`: created `btrfs` subvolumes on `sol` at `/srv/trade`, `/srv/trade/postgres`, and `/srv/trade/redis`.

### 13. Define hard storage quotas and monitoring for trade data on `mevnode`

- In progress.
- Use enforceable limits, not only declarative PVC sizes without backend enforcement.
- Add warning and critical thresholds for disk usage, inode usage, and PVC growth.
- Target hard quota for `Postgres` on `sol`: `1 TiB`.
- Target hard quota for `Redis` on `sol`: `2 GiB` unless later requirements justify persistence growth.
- Keep enough unallocated free space on the root NVMe for the OS, `k3s`, container images, logs, and rollback operations after trade quotas are reserved.
- Done on `2026-04-12`: enabled `btrfs` qgroups on `/` at `sol` and applied hard quotas of `1 TiB` to `/srv/trade/postgres` and `2 GiB` to `/srv/trade/redis`.

### 14. Define CPU and memory reservations that protect `agave-validator`

- Pending.

### 15. Define the minimum safe free RAM headroom on `mevnode` during rollout

- Pending.

### 16. Define the maximum acceptable swap usage on `mevnode` during rollout

- Pending.

### 17. Install and bootstrap `k3s` on `mevnode`

- Done on `2026-04-12`.
- Define the target `k3s` installation model on `mevnode`.
- Define the target storage class, host paths, and bootstrap dependencies needed before application rollout.
- Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed.
- Done on `2026-04-12`: installed single-node `k3s` on `sol` as `v1.34.6+k3s1`.
- Done on `2026-04-12`: installed `k3s` in a minimal profile with `traefik`, `servicelb`, `local-storage`, and `network-policy` disabled.
- Done on `2026-04-12`: validated core addons on `sol`; both `coredns` and `metrics-server` are healthy.
- Done on `2026-04-12`: prepared shared namespace `trade-infra` with `ClusterIP` services `postgres-host` and `redis-host` backed by host `EndpointSlice` targets on `149.50.96.162`.
- Done on `2026-04-12`: opened `ufw` access on `cni0` only for pod CIDR `10.42.0.0/24` to `6443`, `10250`, `5432`, and `6379`.
- Done on `2026-04-12`: validated `pod -> service -> host` connectivity from a test pod to both `Postgres` and `Redis`.

### 18. Install `Postgres` and `Redis` on `mevnode`

- Done on `2026-04-12` for host-level bootstrap.
- Bring up the data plane first: `Postgres` and `Redis`.
- Validate persistence, storage placement, service discovery, restart behavior, and cold start.
- Validate that `Postgres` lands on the dedicated `1 TiB` trade storage area and that `Redis` lands on its separate small quota before application rollout.
- Do not install application components until `Postgres` and `Redis` are healthy.
- Done on `2026-04-12`: `Postgres` and `Redis` are running on `sol` as host services, bound to `127.0.0.1` and `149.50.96.162` for `k3s` pod access.
- Done on `2026-04-12`: validated `Postgres` on `/srv/trade/postgres/16/main`, validated `Redis` on `/srv/trade/redis`, and confirmed `btrfs` quotas are active for both paths.
- Done on `2026-04-12`: `Postgres` listens with pod CIDR access in `pg_hba.conf`, and `Redis` announces `149.50.96.162` for cluster-aware clients inside `k3s`.
- Done on `2026-04-12`: installed `TimescaleDB 2.26.2` packages on host `Postgres` for `sol`, enabled `shared_preload_libraries=timescaledb`, and provisioned application role/database `admin/crypto` matching the live `trade-postgres` secret from `trade-staging`.

### 19. Bootstrap an organization-scoped Gitea Actions runner on `mevnode`

- Done on `2026-04-12`.
- Use `trade-next/trade-gitops` as the canonical bootstrap source, not an ad-hoc shell history on the host.
- Scope the runner at the `trade-next` organization level so every migration repository can reuse the same execution plane.
- Keep the first bootstrap simple: one runner replica on `sol`, one deployer service account, and one org secret carrying a kubeconfig for cluster access.
- Keep the runner state persistent only where necessary; avoid long-lived Docker layer storage until image caching strategy is explicitly planned.
- Done on `2026-04-12`: created `trade-next/trade-gitops` initial `main` branch with `bootstrap/gitea-actions` manifests, scripts, and a smoke workflow.
- Done on `2026-04-12`: deployed `trade-next-act-runner` in namespace `gitea-actions` on `sol` using `docker.io/gitea/act_runner:latest` with a `docker:27-dind` sidecar.
- Done on `2026-04-12`: created `trade-gitops-deployer` service account and initial `cluster-admin` binding for first-pass GitOps bootstrap.
- Done on `2026-04-12`: synced `K3S_KUBECONFIG_B64` as an organization secret in `trade-next`.
- Done on `2026-04-12`: verified the runner is `online` in Gitea with labels `ubuntu-latest` and `k3s-deploy`.
- Done on `2026-04-12`: verified workflow execution from `trade-next/trade-gitops`; commit `c566978` completed with status `success` for `runner-smoke / smoke (push)`.

### 20. Prepare a minimal canary namespace for trade on `mevnode`

- Done on `2026-04-12`.
- Use a dedicated canary namespace for `R001`, not the default namespace and not the infra namespace.
- Put namespace-level CPU, memory, object-count, and storage caps in place before the first application manifests are applied.
- Keep the first canary envelope conservative until the validator protection budget is finalized.
- Validate that the canary namespace can resolve and reach `postgres-host.trade-infra.svc.cluster.local:5432` and `redis-host.trade-infra.svc.cluster.local:6379`.
- Done on `2026-04-12`: created `trade-next/trade-gitops` environment module at `environments/sol/trade-r001-canary`.
- Done on `2026-04-12`: applied namespace `trade-r001-canary` on `sol` with revision label `R001` and canary environment labels.
- Done on `2026-04-12`: applied `ResourceQuota` in `trade-r001-canary` with caps `requests.cpu=2`, `limits.cpu=6`, `requests.memory=4Gi`, `limits.memory=12Gi`, `pods=20`, `services=10`, `configmaps=20`, `secrets=30`, `persistentvolumeclaims=4`, `requests.storage=100Gi`.
- Done on `2026-04-12`: applied `LimitRange` in `trade-r001-canary` with default request `100m/128Mi`, default limit `1 CPU/1Gi`, and per-container max `2 CPU/4Gi`.
- Done on `2026-04-12`: verified deployment through Gitea Actions; commit `060dcc3` completed with status `success` for `deploy-trade-r001-canary / apply (push)`.
- Done on `2026-04-12`: verified an in-cluster connectivity smoke test from `trade-r001-canary` to host-backed `Postgres` and `Redis` services.

### 21. Bring up `Hasura`, API, and frontend on `mevnode`

- Done on `2026-04-12`.
- Bring up `Hasura` only after `Postgres` is healthy.
- Bring up API and frontend only after `Hasura`, `Postgres`, and `Redis` are healthy.
- Use `trade-next/trade-gitops` as the source of truth for the canary application surface, not ad-hoc manifests on the host.
- Keep the first application rollout internal to the cluster: `ClusterIP` services and health checks first, public ingress later.
- Reuse the live `R001` application images and live application secrets, but point them at the target host `Postgres` on `sol`.
- Done on `2026-04-12`: added `Hasura`, `trade-api`, and `trade-frontend` manifests to `trade-next/trade-gitops` under `environments/sol/trade-r001-canary`.
- Done on `2026-04-12`: added operator scripts to `trade-next/trade-gitops` for `sol` host preparation, `gitea-registry` secret creation, and live secret sync from `mevnode_bot/trade-staging` into `trade-r001-canary`.
- Done on `2026-04-12`: prepared `trade-r001-canary` secrets on `sol` for `trade-postgres`, `trade-hasura`, `trade-api`, `trade-frontend-tokens`, `trade-basic-auth`, and `gitea-registry`.
- Done on `2026-04-12`: ran `postgres-migrate` and `hasura-bootstrap` jobs successfully on `sol`.
- Done on `2026-04-12`: deployed `Hasura`, `trade-api`, and `trade-frontend` to `trade-r001-canary`; all three deployments reached `Available`.
- Done on `2026-04-12`: verified internal HTTP health checks `200` for `http://hasura:8080/healthz`, `http://trade-api:8787/healthz`, and `http://trade-frontend:8081/healthz`.
- Done on `2026-04-12`: verified GitOps execution through Gitea Actions; commit `6672e10` completed with status `success` for `deploy-trade-r001-canary / apply (push)`.

### 22. Migrate publishers and ingest only after storage and Redis validation pass

- Done on `2026-04-12`.
- Substep A is now in place on `sol`: deploy `trade-ingestor` first in a schema-compatible canary mode and verify its secret wiring, rollout behavior, and Hasura connectivity.
- Substep B is now functionally in place on `sol`: the live `R001` DLOB hot-path and all-path writer chains and the `dlob_hot_*` plus `dlob_all_*` tables are reconstructed first, and the remaining step is to switch the canary ingestor from `dlob_stats_latest` to the exact live derived query path.
- Done on `2026-04-12`: extended `trade-next/trade-gitops` canary environment with `trade-ingestor`, synced `trade-ingestor-tokens`, and added `trade-ingestor` rollout checks to the `deploy-trade-r001-canary` workflow.
- Done on `2026-04-12`: manually validated `trade-ingestor` on `sol`; deployment reached `Available`, restart count stayed at `0`, and startup logs confirmed `HASURA_GRAPHQL_URL=http://hasura:8080/v1/graphql`, markets `SOL-PERP,PUMP-PERP`, and mode `dlob_stats_ticks`.
- Done on `2026-04-12`: after push of commit `948c37c`, the GitOps path on `sol` rolled a fresh `trade-ingestor` pod and it remained healthy with restart count `0`.
- Done on `2026-04-12`: added `environments/sol/trade-infra` to `trade-next/trade-gitops` with host-backed `postgres-host`, `redis-host`, `agave-rpc-host`, `agave-ws-host`, and `agave-grpc-host` services so canary workloads can consume the shared host services through Kubernetes DNS.
- Done on `2026-04-12`: installed host-side `systemd` socket proxies and `ufw` rules on `sol` so pods in `k3s` can reach Agave RPC on `8899`, WebSocket on `8900`, and Yellowstone gRPC on `10000`.
- Done on `2026-04-12`: created the `trade-dlob-rpc` secret on `sol` from the live host Geyser token and wired the canary namespace to the host-backed Agave endpoints without hardcoding the token in Git.
- Done on `2026-04-12`: added `dlob-redis`, `dlob-publisher-hot`, `dlob-hot-redis-to-postgres-raw-writer`, and `dlob-hot-postgres-to-postgres-derived-writer` to `trade-r001-canary` in GitOps.
- Done on `2026-04-12`: reran `postgres-migrate` and `hasura-bootstrap` with additive `dlob_hot_*` schema and Hasura metadata on `sol`.
- Done on `2026-04-12`: manually validated the DLOB hot-path on `sol`; the raw writer persisted `dlob_hot_snapshot_*`, the derived writer persisted `dlob_hot_derived_*`, and `dlob_hot_derived_latest` contained current rows for markets `0`, `20`, and `72`.
- Done on `2026-04-12`: added `dlob-publisher-all` and `dlob-all-redis-to-postgres-derived-writer` to `trade-r001-canary` in GitOps and validated that both deployments reached `Available` on `sol`.
- Done on `2026-04-12`: tuned the new `dlob-publisher-all` and `dlob-all-redis-to-postgres-derived-writer` manifests to fit the namespace guardrails by setting `limits.cpu=500m` and `RollingUpdate` with `maxSurge=0`, `maxUnavailable=1`, which removed the `ResourceQuota` rollout deadlock at `limits.cpu=6`.
- Done on `2026-04-12`: manually validated the DLOB all-path on `sol`; `dlob_all_derived_latest` and `dlob_all_derived_ts` were populated and the new publisher and writer stayed healthy after rollout.
- Done on `2026-04-12`: switched the canary `trade-ingestor` to the exact live derived query path; after rollout on `sol`, fresh `drift_ticks` rows showed `raw.from = dlob_hot_derived_latest` for `SOL-PERP` and `raw.from = dlob_all_derived_latest` for `PUMP-PERP`.
- Done on `2026-04-12`: added `api-token-seed` bootstrap logic for `trade-frontend-tokens/read.json`, which inserted the matching hash into `api_tokens` on `sol` and fixed `trade-api` auth for the frontend read token.
- Done on `2026-04-12`: validated the application surface against the reconstructed derived tables; `trade-api /v1/ticks` returned fresh `SOL-PERP` and `PUMP-PERP` ticks, `trade-api /v1/chart` returned `SOL-PERP` candles, and `trade-frontend` returned `200`.

### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout

- Done on `2026-04-12`.
- Initial validation on `2026-04-12`: after the DLOB hot/all-path rollout, sampled Agave slot lag on `sol` at `0` versus `https://api.mainnet-beta.solana.com`, with local health still reporting `ok`.
- Done on `2026-04-12`: adjusted the canary rollout model to match the namespace guardrails by switching deployments to `RollingUpdate { maxSurge: 0, maxUnavailable: 1 }`, adding explicit low CPU limits to `api-token-seed`, and making the workflow restart the surface sequentially instead of all at once.
- Done on `2026-04-12`: after the sequential smoke rollout on `sol`, all canary deployments returned to `Available=1/1`, `trade-api /v1/ticks` and `/v1/chart` returned data, and the validator still reported `lag=0`, `health=ok`.
- Done on `2026-04-12`: added operator script `environments/sol/trade-r001-canary/scripts/check-sol-canary-smoke.sh`, which waits for deployment availability, checks `agave-validator` and `k3s`, samples Agave lag, validates derived-table plus `drift_ticks` freshness, and exercises `trade-api` and `trade-frontend` end to end.
- Done on `2026-04-12`: switched canary publisher liveness probes on `sol` from `/health` to `/startup` after confirming that `/health` returned false negatives during steady-state while `drift_ticks` stayed fresh; the smoke-check is now the primary end-to-end validation gate for canary rollouts on `sol`.
- Done on `2026-04-12`: completed a full smoke-check run at `2026-04-12T17:11:33+00:00`; Agave reported `lag=0`, `health=ok`, all canary deployments were `Available=1/1`, `drift_ticks` remained fresh, `trade-api` returned live `SOL-PERP` ticks and candles, and `trade-frontend` returned `200`.
- Done on `2026-04-12`: snapshotted the `trade-r001-canary` secrets from `sol` to `$HOME/.local/share/trade-bootstrap/sol/trade-r001-canary-secrets` with directory mode `700` and file mode `600`, and verified that `sync-live-secrets.sh` can now reapply secrets to `sol` directly from the local snapshot without contacting `mevnode_bot`.

### 24. Keep `mevnode_bot` as a rollback target until the new stack is stable

- Pending.

## Chapter 3

- [x] 01. Define hard database and cache constraints.
- [ ] 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization.
- [x] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness.
- [x] 04. Fix `dlob-publisher-all` startup, readiness, and liveness.
- [x] 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster.
- [ ] 06. Align publisher Redis client mode with the actual deployed Redis topology.
- [ ] 07. Restore or recreate the missing `gitea-registry` image pull secret.
- [ ] 08. Verify that image pulls work after node reboot and on a cold start.
- [ ] 09. Measure the real PostgreSQL data size inside the current trade PVC.
- [ ] 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk.
- [ ] 11. Decide which trade datasets must be migrated and which can be rebuilt.
- [x] 12. Define a dedicated storage path or device for trade data on `mevnode`.
- [ ] 13. Define hard storage quotas and monitoring for trade data on `mevnode`.
- [ ] 14. Define CPU and memory reservations that protect `agave-validator`.
- [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout.
- [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout.
- [x] 17. Install and bootstrap `k3s` on `mevnode`.
- [x] 18. Install `Postgres` and `Redis` on `mevnode`.
- [x] 19. Bootstrap an organization-scoped Gitea Actions runner on `mevnode`.
- [x] 20. Prepare a minimal canary namespace for trade on `mevnode`.
- [x] 21. Bring up `Hasura`, API, and frontend on `mevnode`.
- [x] 22. Migrate publishers and ingest only after storage and Redis validation pass.
- [x] 23. Validate host health, agave health, trade health, and end-to-end flow after rollout.
- [ ] 24. Keep `mevnode_bot` as a rollback target until the new stack is stable.