From 4fdde9e86f62a72a8f602c6d8a4542c7ec1a8cfa Mon Sep 17 00:00:00 2001 From: mpabi Date: Sun, 12 Apr 2026 18:10:52 +0200 Subject: [PATCH] docs(plan): record sol dlob hot-path rollout --- doc/migration-todo.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/doc/migration-todo.md b/doc/migration-todo.md index a2bab11..f304d67 100644 --- a/doc/migration-todo.md +++ b/doc/migration-todo.md @@ -350,14 +350,22 @@ Date: `2026-04-12` - In progress. - Substep A is now in place on `sol`: deploy `trade-ingestor` first in a schema-compatible canary mode and verify its secret wiring, rollout behavior, and Hasura connectivity. -- Substep B remains pending: reconstruct the live `R001` DLOB derived writer chain and the `dlob_*_derived_latest` tables so the canary ingestor can switch from `dlob_stats_latest` to the exact current live query path. +- Substep B is partially complete on `sol`: reconstruct the live `R001` DLOB hot-path writer chain and the `dlob_hot_*` tables first, then switch the canary ingestor from `dlob_stats_latest` to the exact live derived query path only after the hot-path is stable. - Done on `2026-04-12`: extended `trade-next/trade-gitops` canary environment with `trade-ingestor`, synced `trade-ingestor-tokens`, and added `trade-ingestor` rollout checks to the `deploy-trade-r001-canary` workflow. - Done on `2026-04-12`: manually validated `trade-ingestor` on `sol`; deployment reached `Available`, restart count stayed at `0`, and startup logs confirmed `HASURA_GRAPHQL_URL=http://hasura:8080/v1/graphql`, markets `SOL-PERP,PUMP-PERP`, and mode `dlob_stats_ticks`. -- Done on `2026-04-12`: after push of commit `b8f1303`, the GitOps path on `sol` rolled a fresh `trade-ingestor` pod and it remained healthy with restart count `0`. +- Done on `2026-04-12`: after push of commit `948c37c`, the GitOps path on `sol` rolled a fresh `trade-ingestor` pod and it remained healthy with restart count `0`. +- Done on `2026-04-12`: added `environments/sol/trade-infra` to `trade-next/trade-gitops` with host-backed `postgres-host`, `redis-host`, `agave-rpc-host`, `agave-ws-host`, and `agave-grpc-host` services so canary workloads can consume the shared host services through Kubernetes DNS. +- Done on `2026-04-12`: installed host-side `systemd` socket proxies and `ufw` rules on `sol` so pods in `k3s` can reach Agave RPC on `8899`, WebSocket on `8900`, and Yellowstone gRPC on `10000`. +- Done on `2026-04-12`: created the `trade-dlob-rpc` secret on `sol` from the live host Geyser token and wired the canary namespace to the host-backed Agave endpoints without hardcoding the token in Git. +- Done on `2026-04-12`: added `dlob-redis`, `dlob-publisher-hot`, `dlob-hot-redis-to-postgres-raw-writer`, and `dlob-hot-postgres-to-postgres-derived-writer` to `trade-r001-canary` in GitOps. +- Done on `2026-04-12`: reran `postgres-migrate` and `hasura-bootstrap` with additive `dlob_hot_*` schema and Hasura metadata on `sol`. +- Done on `2026-04-12`: manually validated the DLOB hot-path on `sol`; the raw writer persisted `dlob_hot_snapshot_*`, the derived writer persisted `dlob_hot_derived_*`, and `dlob_hot_derived_latest` contained current rows for markets `0`, `20`, and `72`. +- Remaining on `2026-04-12`: bring in the live `dlob-publisher-all` and `dlob_all_*` chain under GitOps, then switch `trade-ingestor` from `dlob_stats_ticks` to the exact live derived query path and validate API and UI behavior against the new tables. ### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout - Pending. +- Initial validation on `2026-04-12`: after the DLOB hot-path rollout, sampled Agave slot lag on `sol` at `0` versus `https://api.mainnet-beta.solana.com`, with local health still reporting `ok`. ### 24. Keep `mevnode_bot` as a rollback target until the new stack is stable @@ -365,9 +373,9 @@ Date: `2026-04-12` ## Chapter 3 -- [ ] 01. Define hard database and cache constraints. +- [x] 01. Define hard database and cache constraints. - [ ] 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization. -- [ ] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness. +- [x] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness. - [ ] 04. Fix `dlob-publisher-all` startup, readiness, and liveness. - [ ] 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster. - [ ] 06. Align publisher Redis client mode with the actual deployed Redis topology.