fix(sol): serialize canary rollouts under quota
All checks were successful
deploy-trade-r001-canary / apply (push) Successful in 6m46s
All checks were successful
deploy-trade-r001-canary / apply (push) Successful in 6m46s
This commit is contained in:
@@ -58,20 +58,26 @@ jobs:
|
|||||||
kubectl get ns trade-r001-canary --show-labels
|
kubectl get ns trade-r001-canary --show-labels
|
||||||
kubectl -n trade-r001-canary get svc,resourcequota,limitrange
|
kubectl -n trade-r001-canary get svc,resourcequota,limitrange
|
||||||
|
|
||||||
- name: Restart application surface
|
- name: Restart application surface sequentially
|
||||||
env:
|
env:
|
||||||
KUBECONFIG: /tmp/kubeconfig
|
KUBECONFIG: /tmp/kubeconfig
|
||||||
run: |
|
run: |
|
||||||
kubectl -n trade-r001-canary rollout restart \
|
restart_and_wait() {
|
||||||
deploy/hasura \
|
name="$1"
|
||||||
deploy/trade-api \
|
timeout="$2"
|
||||||
deploy/trade-frontend \
|
kubectl -n trade-r001-canary rollout restart "deploy/${name}"
|
||||||
deploy/trade-ingestor \
|
kubectl -n trade-r001-canary rollout status "deploy/${name}" --timeout="${timeout}"
|
||||||
deploy/dlob-publisher-hot \
|
}
|
||||||
deploy/dlob-publisher-all \
|
|
||||||
deploy/dlob-hot-redis-to-postgres-raw-writer \
|
restart_and_wait hasura 300s
|
||||||
deploy/dlob-hot-postgres-to-postgres-derived-writer \
|
restart_and_wait trade-api 300s
|
||||||
deploy/dlob-all-redis-to-postgres-derived-writer
|
restart_and_wait trade-frontend 300s
|
||||||
|
restart_and_wait trade-ingestor 300s
|
||||||
|
restart_and_wait dlob-hot-redis-to-postgres-raw-writer 300s
|
||||||
|
restart_and_wait dlob-hot-postgres-to-postgres-derived-writer 300s
|
||||||
|
restart_and_wait dlob-all-redis-to-postgres-derived-writer 300s
|
||||||
|
restart_and_wait dlob-publisher-hot 420s
|
||||||
|
restart_and_wait dlob-publisher-all 420s
|
||||||
|
|
||||||
- name: Wait for database and metadata bootstrap
|
- name: Wait for database and metadata bootstrap
|
||||||
env:
|
env:
|
||||||
@@ -85,15 +91,6 @@ jobs:
|
|||||||
env:
|
env:
|
||||||
KUBECONFIG: /tmp/kubeconfig
|
KUBECONFIG: /tmp/kubeconfig
|
||||||
run: |
|
run: |
|
||||||
kubectl -n trade-r001-canary rollout status deploy/hasura --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/trade-api --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/trade-frontend --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/trade-ingestor --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/dlob-publisher-hot --timeout=420s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/dlob-publisher-all --timeout=420s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/dlob-hot-redis-to-postgres-raw-writer --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/dlob-hot-postgres-to-postgres-derived-writer --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary rollout status deploy/dlob-all-redis-to-postgres-derived-writer --timeout=300s
|
|
||||||
kubectl -n trade-r001-canary get deploy,pods -o wide
|
kubectl -n trade-r001-canary get deploy,pods -o wide
|
||||||
|
|
||||||
- name: Verify trade-ingestor runtime
|
- name: Verify trade-ingestor runtime
|
||||||
|
|||||||
@@ -54,6 +54,7 @@ Minimal canary namespace for migration baseline `R001` on `sol`.
|
|||||||
before it waits for `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, and the DLOB hot/all-path deployments to become healthy.
|
before it waits for `Hasura`, `trade-api`, `trade-frontend`, `trade-ingestor`, and the DLOB hot/all-path deployments to become healthy.
|
||||||
- The canary `trade-ingestor` now follows the live `R001` path on `sol`: it reads `dlob_hot_derived_latest` first for hot markets and falls back to `dlob_all_derived_latest`.
|
- The canary `trade-ingestor` now follows the live `R001` path on `sol`: it reads `dlob_hot_derived_latest` first for hot markets and falls back to `dlob_all_derived_latest`.
|
||||||
- `api-token-seed` restores the frontend read token in `api_tokens`, so `trade-api` and `trade-frontend` can be validated against the reconstructed derived tables after each deploy.
|
- `api-token-seed` restores the frontend read token in `api_tokens`, so `trade-api` and `trade-frontend` can be validated against the reconstructed derived tables after each deploy.
|
||||||
|
- Canary deployments use `RollingUpdate` with `maxSurge=0` and `maxUnavailable=1`, and the workflow restarts them sequentially, because the namespace `limits.cpu=6` guardrail does not tolerate surge-based restarts of the full surface at once.
|
||||||
|
|
||||||
## Operator Flow
|
## Operator Flow
|
||||||
|
|
||||||
|
|||||||
@@ -12,6 +12,13 @@ spec:
|
|||||||
- name: seed
|
- name: seed
|
||||||
image: postgres:16-alpine
|
image: postgres:16-alpine
|
||||||
imagePullPolicy: IfNotPresent
|
imagePullPolicy: IfNotPresent
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 64Mi
|
||||||
|
limits:
|
||||||
|
cpu: 250m
|
||||||
|
memory: 128Mi
|
||||||
command:
|
command:
|
||||||
- sh
|
- sh
|
||||||
- -lc
|
- -lc
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
|
app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
|
app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: dlob-publisher-hot
|
app.kubernetes.io/name: dlob-publisher-hot
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app: hasura
|
app: hasura
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: trade-api
|
app.kubernetes.io/name: trade-api
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: trade-frontend
|
app.kubernetes.io/name: trade-frontend
|
||||||
|
|||||||
@@ -5,6 +5,11 @@ metadata:
|
|||||||
namespace: trade-r001-canary
|
namespace: trade-r001-canary
|
||||||
spec:
|
spec:
|
||||||
replicas: 1
|
replicas: 1
|
||||||
|
strategy:
|
||||||
|
type: RollingUpdate
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 0
|
||||||
|
maxUnavailable: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: trade-ingestor
|
app.kubernetes.io/name: trade-ingestor
|
||||||
|
|||||||
Reference in New Issue
Block a user