trade-next/trade-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(migration): record sol database bootstrap status

Browse Source
This commit is contained in:
u1
2026-04-12 14:03:17 +00:00
parent 6913224ead
commit bf1a1e2e2d
1 changed files with 21 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
doc/migration-todo.md
View File

@@ -161,15 +161,28 @@ Date: `2026-04-12`
### 01. Define hard database and cache constraints
- Pending.
- In progress.
- Storage limits for `Postgres` and `Redis` must be enforced by the real storage backend, not only declared in Kubernetes manifests.
- In core `Postgres`, the hard stop for total on-disk growth should still be treated as the filesystem quota; PostgreSQL itself should add earlier guardrails before that limit is reached.
- `Postgres` must not be able to grow beyond the actually allocated disk budget.
- `Postgres` must have explicit CPU and memory requests/limits consistent with host headroom.
- `Postgres` should use `temp_file_limit` so large sorts and hashes fail before they can consume uncontrolled temporary disk space.
- `Postgres` must have operating thresholds for free disk, WAL growth, autovacuum, and table/index bloat.
- `Postgres` should use bounded `max_wal_size` and, if replication slots are present, bounded `max_slot_wal_keep_size`, while treating WAL limits as early control signals rather than the final hard stop.
- `Postgres` should have retention and pruning policies for time-series data so historical chunks are dropped before storage pressure reaches the OS quota.
- `Postgres` should have an operational read-only switch prepared, so new write traffic can be stopped before the filesystem quota is actually hit.
- `Redis` must have explicit memory limits and an intentional eviction/persistence policy.
- `Redis` must use explicit `maxmemory` below the container memory limit and must set an intentional `maxmemory-policy` instead of relying on the default unlimited dataset behavior.
- `Redis` should set `maxmemory-clients` so client buffers cannot grow without bound under connection pressure.
- `Redis` must not be allowed to consume unbounded RAM or disk on the target host.
- Database and cache capacity alerts must fire before any hard limit is reached, not after the filesystem is already effectively full.
- Full-disk and out-of-memory behavior for `Postgres` and `Redis` must be tested before application cutover.
- Done on `2026-04-12`: installed `PostgreSQL 16` and `Redis 7` on `sol` and started both services.
- Done on `2026-04-12`: moved `Postgres` data directory to `/srv/trade/postgres/16/main`.
- Done on `2026-04-12`: applied `Postgres` guardrails on `sol`: `shared_buffers=2GB`, `effective_cache_size=8GB`, `work_mem=16MB`, `maintenance_work_mem=512MB`, `temp_file_limit=8GB`, `max_wal_size=4GB`, `min_wal_size=512MB`, `max_slot_wal_keep_size=2GB`, `idle_in_transaction_session_timeout=10min`.
- Done on `2026-04-12`: prepared and tested operational `Postgres` read-only switch at `/usr/local/bin/trade-postgres-readonly`.
- Done on `2026-04-12`: applied `Redis` guardrails on `sol`: `maxmemory=384MB`, `maxmemory-clients=64MB`, `maxmemory-policy=noeviction`, `save \"\"`, `appendonly no`.
- Done on `2026-04-12`: initialized `Redis` as a single-node cluster with all `16384` slots assigned and `cluster_state: ok`.
### 02. Reconstruct the current live `mevnode_bot` and `trade-staging` state into canonical Git/IaC in a new Gitea organization
@@ -223,7 +236,7 @@ Date: `2026-04-12`
### 12. Define a dedicated storage path or device for trade data on `mevnode`
- Pending.
- Done on `2026-04-12`.
- Current preferred target layout on `sol` is to place trade persistent storage on the root NVMe filesystem `/`, because validator ledger and validator state already have their own dedicated devices.
- Do not place `Postgres` or `Redis` on `/data/ledger` or `/data/state`.
- Reserve a dedicated `1 TiB` storage budget for `Postgres` on `sol` as the first target sizing for `R001`.
@@ -234,7 +247,7 @@ Date: `2026-04-12`
### 13. Define hard storage quotas and monitoring for trade data on `mevnode`
- Pending.
- In progress.
- Use enforceable limits, not only declarative PVC sizes without backend enforcement.
- Add warning and critical thresholds for disk usage, inode usage, and PVC growth.
- Target hard quota for `Postgres` on `sol`: `1 TiB`.
@@ -263,11 +276,13 @@ Date: `2026-04-12`
### 18. Install `Postgres` and `Redis` on `mevnode`
- Pending.
- Done on `2026-04-12` for host-level bootstrap.
- Bring up the data plane first: `Postgres` and `Redis`.
- Validate persistence, storage placement, service discovery, restart behavior, and cold start.
- Validate that `Postgres` lands on the dedicated `1 TiB` trade storage area and that `Redis` lands on its separate small quota before application rollout.
- Do not install application components until `Postgres` and `Redis` are healthy.
- Done on `2026-04-12`: `Postgres` and `Redis` are running on `sol` as host services, both bound to `127.0.0.1` at this stage.
- Done on `2026-04-12`: validated `Postgres` on `/srv/trade/postgres/16/main`, validated `Redis` on `/srv/trade/redis`, and confirmed `btrfs` quotas are active for both paths.
### 19. Prepare a minimal canary namespace for trade on `mevnode`
@@ -304,13 +319,13 @@ Date: `2026-04-12`
- [ ] 09. Measure the real PostgreSQL data size inside the current trade PVC.
- [ ] 10. Explain why the declared `20Gi` PVC currently consumes about `585 GiB` on disk.
- [ ] 11. Decide which trade datasets must be migrated and which can be rebuilt.
- [ ] 12. Define a dedicated storage path or device for trade data on `mevnode`.
- [x] 12. Define a dedicated storage path or device for trade data on `mevnode`.
- [ ] 13. Define hard storage quotas and monitoring for trade data on `mevnode`.
- [ ] 14. Define CPU and memory reservations that protect `agave-validator`.
- [ ] 15. Define the minimum safe free RAM headroom on `mevnode` during rollout.
- [ ] 16. Define the maximum acceptable swap usage on `mevnode` during rollout.
- [ ] 17. Install and bootstrap `k3s` on `mevnode`.
- [ ] 18. Install `Postgres` and `Redis` on `mevnode`.
- [x] 18. Install `Postgres` and `Redis` on `mevnode`.
- [ ] 19. Prepare a minimal canary namespace for trade on `mevnode`.
- [ ] 20. Bring up `Hasura`, API, and frontend on `mevnode`.
- [ ] 21. Migrate publishers and ingest only after storage and Redis validation pass.