From 2f95a70359fe08d22344283ff77f3fff5b2ab97b Mon Sep 17 00:00:00 2001 From: u1 Date: Sun, 12 Apr 2026 13:16:58 +0000 Subject: [PATCH] docs(migration): add enforced database and cache constraints --- doc/migration-todo.md | 76 +++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 24 deletions(-) diff --git a/doc/migration-todo.md b/doc/migration-todo.md index b135cf7..18148e4 100644 --- a/doc/migration-todo.md +++ b/doc/migration-todo.md @@ -213,40 +213,66 @@ Date: `2026-04-12` ### 12. Define hard storage quotas and monitoring for trade data on `mevnode` - Pending. +- Use enforceable limits, not only declarative PVC sizes without backend enforcement. +- Add warning and critical thresholds for disk usage, inode usage, and PVC growth. -### 13. Define CPU and memory reservations that protect `agave-validator` +### 13. Define hard database and cache constraints + +- Pending. +- Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests. +- `Postgres` must not be able to grow beyond the actually allocated disk budget. +- `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom. +- `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat. +- `Redis` must have explicit memory limits and an intentional eviction/persistence policy. +- `Redis` must not be allowed to consume unbounded RAM or disk on the target host. +- Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full. +- Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover. + +### 14. Define CPU and memory reservations that protect `agave-validator` - Pending. -### 14. Define the minimum safe free RAM headroom on `mevnode` during rollout +### 15. Define the minimum safe free RAM headroom on `mevnode` during rollout - Pending. -### 15. Define the maximum acceptable swap usage on `mevnode` during rollout +### 16. Define the maximum acceptable swap usage on `mevnode` during rollout - Pending. -### 16. Prepare a minimal canary namespace for trade on `mevnode` +### 17. Install and bootstrap `k3s` on `mevnode` + +- Pending. +- Define the target `k3s` installation model on `mevnode`. +- Define the target storage class, host paths, and bootstrap dependencies needed before application rollout. +- Verify that the base cluster, registry access, namespaces, and secret distribution work before any application components are installed. + +### 18. Install `Postgres` and `Redis` on `mevnode` + +- Pending. +- Bring up the data plane first: `Postgres` and `Redis`. +- Validate persistence, storage placement, service discovery, restart behavior, and cold start. +- Do not install application components until `Postgres` and `Redis` are healthy. + +### 19. Prepare a minimal canary namespace for trade on `mevnode` - Pending. -### 17. Bring up Redis, Postgres, and Hasura on `mevnode` +### 20. Bring up `Hasura`, API, and frontend on `mevnode` + +- Pending. +- Bring up `Hasura` only after `Postgres` is healthy. +- Bring up API and frontend only after `Hasura`, `Postgres`, and `Redis` are healthy. + +### 21. Migrate publishers and ingest only after storage and Redis validation pass - Pending. -### 18. Bring up API and frontend on `mevnode` +### 22. Validate host health, agave health, trade health, and end-to-end flow after rollout - Pending. -### 19. Migrate publishers and ingest only after storage and Redis validation pass - -- Pending. - -### 20. Validate host health, agave health, trade health, and end-to-end flow after rollout - -- Pending. - -### 21. Keep `mevnode_bot` as a rollback target until the new stack is stable +### 23. Keep `mevnode_bot` as a rollback target until the new stack is stable - Pending. @@ -264,12 +290,14 @@ Date: `2026-04-12` - [ ] 10. Decide which trade datasets must be migrated and which can be rebuilt. - [ ] 11. Define a dedicated storage path or device for trade data on `mevnode`. - [ ] 12. Define hard storage quotas and monitoring for trade data on `mevnode`. -- [ ] 13. Define CPU and memory reservations that protect `agave-validator`. -- [ ] 14. Define the minimum safe free RAM headroom on `mevnode` during rollout. -- [ ] 15. Define the maximum acceptable swap usage on `mevnode` during rollout. -- [ ] 16. Prepare a minimal canary namespace for trade on `mevnode`. -- [ ] 17. Bring up Redis, Postgres, and Hasura on `mevnode`. -- [ ] 18. Bring up API and frontend on `mevnode`. -- [ ] 19. Migrate publishers and ingest only after storage and Redis validation pass. -- [ ] 20. Validate host health, agave health, trade health, and end-to-end flow after rollout. -- [ ] 21. Keep `mevnode_bot` as a rollback target until the new stack is stable. +- [ ] 13. Define hard database and cache constraints. +- [ ] 14. Define CPU and memory reservations that protect `agave-validator`. +- [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout. +- [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout. +- [ ] 17. Install and bootstrap `k3s` on `mevnode`. +- [ ] 18. Install `Postgres` and `Redis` on `mevnode`. +- [ ] 19. Prepare a minimal canary namespace for trade on `mevnode`. +- [ ] 20. Bring up `Hasura`, API, and frontend on `mevnode`. +- [ ] 21. Migrate publishers and ingest only after storage and Redis validation pass. +- [ ] 22. Validate host health, agave health, trade health, and end-to-end flow after rollout. +- [ ] 23. Keep `mevnode_bot` as a rollback target until the new stack is stable.