From 138a4bd95394b0dd23a682e2f508b177a42348e5 Mon Sep 17 00:00:00 2001 From: mpabi Date: Sun, 12 Apr 2026 18:19:59 +0200 Subject: [PATCH] docs(plan): record canary dlob all-path rollout --- doc/migration-todo.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/doc/migration-todo.md b/doc/migration-todo.md index f304d67..26af980 100644 --- a/doc/migration-todo.md +++ b/doc/migration-todo.md @@ -350,7 +350,7 @@ Date: `2026-04-12` - In progress. - Substep A is now in place on `sol`: deploy `trade-ingestor` first in a schema-compatible canary mode and verify its secret wiring, rollout behavior, and Hasura connectivity. -- Substep B is partially complete on `sol`: reconstruct the live `R001` DLOB hot-path writer chain and the `dlob_hot_*` tables first, then switch the canary ingestor from `dlob_stats_latest` to the exact live derived query path only after the hot-path is stable. +- Substep B is now functionally in place on `sol`: the live `R001` DLOB hot-path and all-path writer chains and the `dlob_hot_*` plus `dlob_all_*` tables are reconstructed first, and the remaining step is to switch the canary ingestor from `dlob_stats_latest` to the exact live derived query path. - Done on `2026-04-12`: extended `trade-next/trade-gitops` canary environment with `trade-ingestor`, synced `trade-ingestor-tokens`, and added `trade-ingestor` rollout checks to the `deploy-trade-r001-canary` workflow. - Done on `2026-04-12`: manually validated `trade-ingestor` on `sol`; deployment reached `Available`, restart count stayed at `0`, and startup logs confirmed `HASURA_GRAPHQL_URL=http://hasura:8080/v1/graphql`, markets `SOL-PERP,PUMP-PERP`, and mode `dlob_stats_ticks`. - Done on `2026-04-12`: after push of commit `948c37c`, the GitOps path on `sol` rolled a fresh `trade-ingestor` pod and it remained healthy with restart count `0`. @@ -360,7 +360,10 @@ Date: `2026-04-12` - Done on `2026-04-12`: added `dlob-redis`, `dlob-publisher-hot`, `dlob-hot-redis-to-postgres-raw-writer`, and `dlob-hot-postgres-to-postgres-derived-writer` to `trade-r001-canary` in GitOps. - Done on `2026-04-12`: reran `postgres-migrate` and `hasura-bootstrap` with additive `dlob_hot_*` schema and Hasura metadata on `sol`. - Done on `2026-04-12`: manually validated the DLOB hot-path on `sol`; the raw writer persisted `dlob_hot_snapshot_*`, the derived writer persisted `dlob_hot_derived_*`, and `dlob_hot_derived_latest` contained current rows for markets `0`, `20`, and `72`. -- Remaining on `2026-04-12`: bring in the live `dlob-publisher-all` and `dlob_all_*` chain under GitOps, then switch `trade-ingestor` from `dlob_stats_ticks` to the exact live derived query path and validate API and UI behavior against the new tables. +- Done on `2026-04-12`: added `dlob-publisher-all` and `dlob-all-redis-to-postgres-derived-writer` to `trade-r001-canary` in GitOps and validated that both deployments reached `Available` on `sol`. +- Done on `2026-04-12`: tuned the new `dlob-publisher-all` and `dlob-all-redis-to-postgres-derived-writer` manifests to fit the namespace guardrails by setting `limits.cpu=500m` and `RollingUpdate` with `maxSurge=0`, `maxUnavailable=1`, which removed the `ResourceQuota` rollout deadlock at `limits.cpu=6`. +- Done on `2026-04-12`: manually validated the DLOB all-path on `sol`; `dlob_all_derived_latest` and `dlob_all_derived_ts` were populated and the new publisher and writer stayed healthy after rollout. +- Remaining on `2026-04-12`: switch `trade-ingestor` from `dlob_stats_ticks` to the exact live derived query path and validate API and UI behavior against `dlob_hot_derived_latest` plus `dlob_all_derived_latest`. ### 23. Validate host health, agave health, trade health, and end-to-end flow after rollout @@ -376,7 +379,7 @@ Date: `2026-04-12` - [x] 01. Define hard database and cache constraints. - [ ] 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization. - [x] 03. Fix `dlob-publisher-hot` startup, readiness, and liveness. -- [ ] 04. Fix `dlob-publisher-all` startup, readiness, and liveness. +- [x] 04. Fix `dlob-publisher-all` startup, readiness, and liveness. - [ ] 05. Confirm whether trade Redis is meant to run as a single instance or as Redis Cluster. - [ ] 06. Align publisher Redis client mode with the actual deployed Redis topology. - [ ] 07. Restore or recreate the missing `gitea-registry` image pull secret.