From bf1a1e2e2d0b39b6cfb4e38781e2f9eba454bd2b Mon Sep 17 00:00:00 2001 From: u1 Date: Sun, 12 Apr 2026 14:03:17 +0000 Subject: [PATCH] docs(migration): record sol database bootstrap status --- doc/migration-todo.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/doc/migration-todo.md b/doc/migration-todo.md index 8043101..21b79bd 100644 --- a/doc/migration-todo.md +++ b/doc/migration-todo.md @@ -161,15 +161,28 @@ Date: `2026-04-12` ### 01. Define hard database and cache constraints -- Pending. +- In progress. - Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests. +- In core `Postgres`, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached. - `Postgres` must not be able to grow beyond the actually allocated disk budget. - `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom. +- `Postgres` should use `temp_file_limit` so large sorts and hashes fail before they can consume uncontrolled temporary disk space. - `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat. +- `Postgres` should use bounded `max_wal_size` and, if replication slots are present, bounded `max_slot_wal_keep_size`, while treating WAL limits as early control signals rather than the final hard stop. +- `Postgres` should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota. +- `Postgres` should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit. - `Redis` must have explicit memory limits and an intentional eviction/persistence policy. +- `Redis` must use explicit `maxmemory` below the container memory limit and must set an intentional `maxmemory-policy` instead of relying on the default unlimited dataset behavior. +- `Redis` should set `maxmemory-clients` so client buffers cannot grow without bound under connection pressure. - `Redis` must not be allowed to consume unbounded RAM or disk on the target host. - Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full. - Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover. +- Done on `2026-04-12`: installed `PostgreSQL 16` and `Redis 7` on `sol` and started both services. +- Done on `2026-04-12`: moved `Postgres` data directory to `/srv/trade/postgres/16/main`. +- Done on `2026-04-12`: applied `Postgres` guardrails on `sol`: `shared_buffers=2GB`, `effective_cache_size=8GB`, `work_mem=16MB`, `maintenance_work_mem=512MB`, `temp_file_limit=8GB`, `max_wal_size=4GB`, `min_wal_size=512MB`, `max_slot_wal_keep_size=2GB`, `idle_in_transaction_session_timeout=10min`. +- Done on `2026-04-12`: prepared and tested operational `Postgres` read-only switch at `/usr/local/bin/trade-postgres-readonly`. +- Done on `2026-04-12`: applied `Redis` guardrails on `sol`: `maxmemory=384MB`, `maxmemory-clients=64MB`, `maxmemory-policy=noeviction`, `save \"\"`, `appendonly no`. +- Done on `2026-04-12`: initialized `Redis` as a single-node cluster with all `16384` slots assigned and `cluster_state: ok`. ### 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization @@ -223,7 +236,7 @@ Date: `2026-04-12` ### 12. Define a dedicated storage path or device for trade data on `mevnode` -- Pending. +- Done on `2026-04-12`. - Current preferred target layout on `sol` is to place trade persistent storage on the root NVMe filesystem `/`, because validator ledger and validator state already have their own dedicated devices. - Do not place `Postgres` or `Redis` on `/data/ledger` or `/data/state`. - Reserve a dedicated `1 TiB` storage budget for `Postgres` on `sol` as the first target sizing for `R001`. @@ -234,7 +247,7 @@ Date: `2026-04-12` ### 13. Define hard storage quotas and monitoring for trade data on `mevnode` -- Pending. +- In progress. - Use enforceable limits, not only declarative PVC sizes without backend enforcement. - Add warning and critical thresholds for disk usage, inode usage, and PVC growth. - Target hard quota for `Postgres` on `sol`: `1 TiB`. @@ -263,11 +276,13 @@ Date: `2026-04-12` ### 18. Install `Postgres` and `Redis` on `mevnode` -- Pending. +- Done on `2026-04-12` for host-level bootstrap. - Bring up the data plane first: `Postgres` and `Redis`. - Validate persistence, storage placement, service discovery, restart behavior, and cold start. - Validate that `Postgres` lands on the dedicated `1 TiB` trade storage area and that `Redis` lands on its separate small quota before application rollout. - Do not install application components until `Postgres` and `Redis` are healthy. +- Done on `2026-04-12`: `Postgres` and `Redis` are running on `sol` as host services, both bound to `127.0.0.1` at this stage. +- Done on `2026-04-12`: validated `Postgres` on `/srv/trade/postgres/16/main`, validated `Redis` on `/srv/trade/redis`, and confirmed `btrfs` quotas are active for both paths. ### 19. Prepare a minimal canary namespace for trade on `mevnode` @@ -304,13 +319,13 @@ Date: `2026-04-12` - [ ] 09. Measure the real PostgreSQL data size inside the current trade PVC. - [ ] 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk. - [ ] 11. Decide which trade datasets must be migrated and which can be rebuilt. -- [ ] 12. Define a dedicated storage path or device for trade data on `mevnode`. +- [x] 12. Define a dedicated storage path or device for trade data on `mevnode`. - [ ] 13. Define hard storage quotas and monitoring for trade data on `mevnode`. - [ ] 14. Define CPU and memory reservations that protect `agave-validator`. - [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout. - [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout. - [ ] 17. Install and bootstrap `k3s` on `mevnode`. -- [ ] 18. Install `Postgres` and `Redis` on `mevnode`. +- [x] 18. Install `Postgres` and `Redis` on `mevnode`. - [ ] 19. Prepare a minimal canary namespace for trade on `mevnode`. - [ ] 20. Bring up `Hasura`, API, and frontend on `mevnode`. - [ ] 21. Migrate publishers and ingest only after storage and Redis validation pass.