chore(snapshot): sync workspace state

kustomize/overlays/mevnode-bot/README.md

@@ -6,15 +6,13 @@ Zakres:
 
 - `postgres`
 - `hasura`
-- `trade-api`
 - `trade-frontend` (`NodePort 30081`)
-- `trade-ingestor`
 - `dlob-redis`
-- `dlob-publisher`
-- `dlob-server`
-- `dlob-worker`
-- `dlob-depth-worker`
-- `dlob-slippage-worker`
+- `dlob-publisher-hot`
+- `dlob-publisher-all`
+- `dlob-hot-redis-to-postgres-raw-writer`
+- `dlob-hot-postgres-to-postgres-derived-writer`
+- `dlob-all-redis-to-postgres-derived-writer`
 
 Założenia runtime:
 
@@ -26,11 +24,47 @@ Założenia runtime:
 - `trade-frontend` jest wystawiany przez `NodePort 30081`.
 - Namespace pozostaje `trade-staging`.
 
+Topologia DLOB w tym overlayu:
+
+- `dlob-publisher-hot` to hot-path na wybrane markety (`PERP_MARKETS_TO_LOAD=0,20,72`, czyli `SOL-PERP`, `JTO-PERP`, `ADA-PERP`) i prefix Redis `DLOB_HOT` (`dlob-hot:`).
+- `dlob-publisher-all` to szeroki feed do wszystkich marketów na prefix Redis `DLOB_ALL` (`dlob-all:`).
+- `dlob-hot-redis-to-postgres-raw-writer` to schema-first writer: czyta `dlob-hot:` z Redis i zapisuje kanoniczny raw DLOB do PostgreSQL (`dlob_hot_snapshot_latest`, `dlob_hot_snapshot_ts`).
+- `dlob-hot-postgres-to-postgres-derived-writer` czyta raw hot z PostgreSQL i buduje normalized layer (`dlob_hot_derived_latest`, `dlob_hot_derived_ts`).
+- `LIVE` na `k3s`: sciezka `hot` jest juz aktywna jako `Redis -> raw PG -> derived PG`.
+- `dlob-publisher-all` celowo nie ma patcha persistent store, żeby nie dublowac zapisow do Postgresa.
+- `dlob-all-redis-to-postgres-derived-writer` zapisuje dla `all` tylko warstwe pochodna do PostgreSQL; bez `raw_ts`, zeby nie zjadac dysku w kilka dni.
+- `LIVE` na `k3s`: sciezka `all` jest aktywna jako `Redis -> derived PG`.
+- `all` zapisuje pelne `derived_ts` dla wszystkich marketow z `dlob-all:*`, bez samplingu i bez shortlisty marketow.
+
+Planowany zakres zapisu dla `all`:
+
+- tozsamosc i czas: `source`, `market_type`, `market_index`, `market_name`, `is_indicative`, `event_ts/ts_ms`, `slot`, `market_slot`
+- cena i mikrostruktura: `mark_price`, `oracle_price`, `best_bid_price`, `best_ask_price`, `mid_price`, `spread_quote`, `spread_bps`
+- plynnosc: `depth_levels`, `bid_levels`, `ask_levels`, `top_bid_size`, `top_ask_size`, `top_bid_notional`, `top_ask_notional`, `depth_bid_base`, `depth_ask_base`, `depth_bid_quote`, `depth_ask_quote`, `imbalance`
+- drabinka do modelu/UI: `bids_norm`, `asks_norm` jako top `10` leveli z polami `price`, `sizeBase`, `notional`, `sources`
+- kontrola jakosci i dedupe: `raw_payload_hash`, `updated_at` / `inserted_at`
+- celowo pomijamy: pelny `raw payload`, `raw latest`, `raw ts`, `L3`, `best_makers`
+
+Polityka `hot` vs `all`:
+
+- `hot` i `all` rozniamy po roli operacyjnej, nie po samym mechanizmie subskrypcji.
+- `hot` = `dlob-publisher-hot`: subset marketow, najnizsza latencja, najwyzszy priorytet operacyjny.
+- `all` = `dlob-publisher-all`: pelne pokrycie marketow, feed do coverage/read-side/analityki, nizszy priorytet operacyjny niz `hot`.
+- `USE_ORDER_SUBSCRIBER=true` moze byc wlaczone na obu publisherach; to nie definiuje roli `hot` vs `all`.
+- `hot` powinien byc broniony jako pierwszy przy problemach z CPU/RAM/RPC i nie powinien byc degradowany przed `all`.
+- Jesli trzeba ograniczac load albo robic fallback, najpierw degradujemy `all`, dopiero potem `hot`.
+
+Zalecana polityka runtime:
+
+- `hot`: utrzymywac `OrderSubscriber + gRPC` jako sciezke docelowa.
+- `all`: domyslnie tez `OrderSubscriber + gRPC`, ale to pierwszy kandydat do przejscia na slabszy tryb przy presji zasobow.
+- Kolejnosc degradacji dla `all`: `gRPC -> websocket -> polling`.
+- Wylaczenie `USE_ORDER_SUBSCRIBER` traktowac jako tryb awaryjny/testowy, nie jako stan docelowy.
+
 Wymagane sekrety:
 
 - `trade-postgres`
 - `trade-hasura`
-- `trade-api`
 - `trade-frontend-tokens`
 - `trade-dlob-rpc`
 

kustomize/overlays/mevnode-bot/dlob-all-redis-to-postgres-derived-writer.yaml

@@ -0,0 +1,66 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dlob-all-redis-to-postgres-derived-writer
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: dlob-all-redis-to-postgres-derived-writer
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: dlob-all-redis-to-postgres-derived-writer
+    spec:
+      priorityClassName: dlob-all
+      imagePullSecrets: []
+      containers:
+        - name: writer
+          image: gitea.mpabi.pl/trade/trade-dlob-server:all-derived-20260317-014016
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: DLOB_SOURCE
+              value: mevnode_bot_all_derived
+            - name: REDIS_HOST
+              value: dlob-redis
+            - name: REDIS_PORT
+              value: "6379"
+            - name: REDIS_KEY_PREFIX
+              value: "dlob-all:"
+            - name: DLOB_POLL_MS
+              value: "1000"
+            - name: NORMALIZED_DEPTH
+              value: "10"
+            - name: PRICE_PRECISION
+              value: "1000000"
+            - name: BASE_PRECISION
+              value: "1000000000"
+            - name: PGHOST
+              value: postgres
+            - name: PGPORT
+              value: "5432"
+            - name: PGUSER
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_USER
+            - name: PGPASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_PASSWORD
+            - name: PGDATABASE
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_DB
+          command: ["node", "/lib/scripts/dlobAllRedisToPostgresDerivedWriter.js"]
+          resources:
+            requests:
+              cpu: 250m
+              memory: 512Mi
+            limits:
+              cpu: 1000m
+              memory: 2Gi

kustomize/overlays/mevnode-bot/dlob-hot-postgres-to-postgres-derived-writer.yaml

@@ -0,0 +1,62 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dlob-hot-postgres-to-postgres-derived-writer
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
+    spec:
+      priorityClassName: dlob-hot
+      imagePullSecrets: []
+      containers:
+        - name: writer
+          image: gitea.mpabi.pl/trade/trade-dlob-server:hot-pg-events-20260320-205517
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: DLOB_SOURCE
+              value: mevnode_bot_hot_derived
+            - name: RAW_SOURCE
+              value: mevnode_bot_hot_raw
+            - name: DLOB_POLL_MS
+              value: "5000"
+            - name: NORMALIZED_DEPTH
+              value: "10"
+            - name: PRICE_PRECISION
+              value: "1000000"
+            - name: BASE_PRECISION
+              value: "1000000000"
+            - name: PGHOST
+              value: postgres
+            - name: PGPORT
+              value: "5432"
+            - name: PGUSER
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_USER
+            - name: PGPASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_PASSWORD
+            - name: PGDATABASE
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_DB
+          command: ["node", "/lib/scripts/dlobHotPostgresToPostgresDerivedWriter.js"]
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi

kustomize/overlays/mevnode-bot/dlob-hot-redis-to-postgres-raw-writer.yaml

@@ -0,0 +1,60 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dlob-hot-redis-to-postgres-raw-writer
+  annotations:
+    argocd.argoproj.io/sync-wave: "5"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
+    spec:
+      priorityClassName: dlob-hot
+      imagePullSecrets: []
+      containers:
+        - name: writer
+          image: gitea.mpabi.pl/trade/trade-dlob-server:hot-pg-events-20260320-205517
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: DLOB_SOURCE
+              value: mevnode_bot_hot_raw
+            - name: REDIS_HOST
+              value: dlob-redis
+            - name: REDIS_PORT
+              value: "6379"
+            - name: REDIS_KEY_PREFIX
+              value: "dlob-hot:"
+            - name: DLOB_POLL_MS
+              value: "5000"
+            - name: PGHOST
+              value: postgres
+            - name: PGPORT
+              value: "5432"
+            - name: PGUSER
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_USER
+            - name: PGPASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_PASSWORD
+            - name: PGDATABASE
+              valueFrom:
+                secretKeyRef:
+                  name: trade-postgres
+                  key: POSTGRES_DB
+          command: ["node", "/lib/scripts/dlobHotRedisToPostgresRawWriter.js"]
+          resources:
+            requests:
+              cpu: 100m
+              memory: 256Mi
+            limits:
+              cpu: 500m
+              memory: 1Gi

kustomize/overlays/mevnode-bot/dlob-ingestor.mjs

@@ -1,219 +0,0 @@
-import process from 'node:process';
-import { setTimeout as sleep } from 'node:timers/promises';
-
-function getIsoNow() {
-  return new Date().toISOString();
-}
-
-function envString(name, fallback) {
-  const v = process.env[name];
-  if (v == null) return fallback;
-  const s = String(v).trim();
-  return s ? s : fallback;
-}
-
-function envInt(name, fallback, { min, max } = {}) {
-  const v = process.env[name];
-  if (v == null) return fallback;
-  const n = Number.parseInt(String(v), 10);
-  if (!Number.isFinite(n)) return fallback;
-  const low = typeof min === 'number' ? min : n;
-  const high = typeof max === 'number' ? max : n;
-  return Math.max(low, Math.min(high, n));
-}
-
-function envList(name, fallbackCsv) {
-  const raw = process.env[name] ?? fallbackCsv;
-  return String(raw)
-    .split(',')
-    .map((s) => s.trim())
-    .filter(Boolean);
-}
-
-function toIntOrNull(v) {
-  if (v == null) return null;
-  if (typeof v === 'number') return Number.isFinite(v) ? Math.trunc(v) : null;
-  if (typeof v === 'string') {
-    const s = v.trim();
-    if (!s) return null;
-    const n = Number.parseInt(s, 10);
-    return Number.isFinite(n) ? n : null;
-  }
-  return null;
-}
-
-function numStr(v) {
-  if (v == null) return null;
-  if (typeof v === 'number') return Number.isFinite(v) ? String(v) : null;
-  if (typeof v === 'string') {
-    const s = v.trim();
-    return s ? s : null;
-  }
-  return null;
-}
-
-function isoFromEpochMs(v) {
-  const n = typeof v === 'number' ? v : typeof v === 'string' ? Number(v.trim()) : NaN;
-  if (!Number.isFinite(n) || n <= 0) return null;
-  const d = new Date(n);
-  const ms = d.getTime();
-  if (!Number.isFinite(ms)) return null;
-  return d.toISOString();
-}
-
-function resolveConfig() {
-  const hasuraUrl = envString('HASURA_GRAPHQL_URL', 'http://hasura:8080/v1/graphql');
-  const hasuraAdminSecret = envString('HASURA_ADMIN_SECRET', '');
-  if (!hasuraAdminSecret) throw new Error('Missing HASURA_ADMIN_SECRET');
-
-  const markets = envList('DLOB_MARKETS', 'SOL-PERP,PUMP-PERP');
-  const pollMs = envInt('TICKS_POLL_MS', 1000, { min: 250, max: 60000 });
-  const source = envString('TICKS_SOURCE', 'dlob_stats');
-
-  return { hasuraUrl, hasuraAdminSecret, markets, pollMs, source };
-}
-
-async function graphqlRequest(cfg, query, variables) {
-  const res = await fetch(cfg.hasuraUrl, {
-    method: 'POST',
-    headers: {
-      'content-type': 'application/json',
-      'x-hasura-admin-secret': cfg.hasuraAdminSecret,
-    },
-    body: JSON.stringify({ query, variables }),
-    signal: AbortSignal.timeout(10000),
-  });
-
-  const text = await res.text();
-  if (!res.ok) throw new Error(`Hasura HTTP ${res.status}: ${text}`);
-
-  let json;
-  try {
-    json = JSON.parse(text);
-  } catch {
-    throw new Error(`Hasura: invalid json: ${text}`);
-  }
-
-  if (json.errors?.length) throw new Error(json.errors.map((e) => e.message).join(' | '));
-  return json.data;
-}
-
-async function fetchStats(cfg) {
-  const query = `
-    query DlobStatsLatest($markets: [String!]!) {
-      dlob_stats_latest(where: { market_name: { _in: $markets } }) {
-        market_name
-        market_index
-        ts
-        slot
-        oracle_price
-        mark_price
-        mid_price
-        best_bid_price
-        best_ask_price
-        updated_at
-      }
-    }
-  `;
-
-  const data = await graphqlRequest(cfg, query, { markets: cfg.markets });
-  const rows = Array.isArray(data?.dlob_stats_latest) ? data.dlob_stats_latest : [];
-  return rows;
-}
-
-async function insertTicks(cfg, objects) {
-  if (!objects.length) return 0;
-
-  const mutation = `
-    mutation InsertTicks($objects: [drift_ticks_insert_input!]!) {
-      insert_drift_ticks(objects: $objects) { affected_rows }
-    }
-  `;
-
-  const data = await graphqlRequest(cfg, mutation, { objects });
-  return Number(data?.insert_drift_ticks?.affected_rows || 0);
-}
-
-async function main() {
-  const cfg = resolveConfig();
-  const lastUpdatedAtByMarket = new Map();
-
-  console.log(
-    JSON.stringify(
-      {
-        service: 'trade-ingestor',
-        mode: 'dlob_stats_ticks',
-        startedAt: getIsoNow(),
-        hasuraUrl: cfg.hasuraUrl,
-        markets: cfg.markets,
-        pollMs: cfg.pollMs,
-        source: cfg.source,
-      },
-      null,
-      2
-    )
-  );
-
-  while (true) {
-    try {
-      const rows = await fetchStats(cfg);
-      const nowIso = getIsoNow();
-
-      const objects = [];
-      for (const r of rows) {
-        const marketName = String(r?.market_name || '').trim();
-        if (!marketName) continue;
-
-        const updatedAt = r?.updated_at ? String(r.updated_at) : '';
-        if (updatedAt && lastUpdatedAtByMarket.get(marketName) === updatedAt) continue;
-        if (updatedAt) lastUpdatedAtByMarket.set(marketName, updatedAt);
-
-        const marketIndex = toIntOrNull(r?.market_index) ?? 0;
-        const dlobIso = isoFromEpochMs(r?.ts);
-        const tsIso = dlobIso || nowIso;
-
-        const oraclePrice = numStr(r?.oracle_price) || numStr(r?.mark_price) || numStr(r?.mid_price);
-        const markPrice = numStr(r?.mark_price) || numStr(r?.mid_price) || oraclePrice;
-        if (!oraclePrice) continue;
-
-        objects.push({
-          ts: tsIso,
-          market_index: marketIndex,
-          symbol: marketName,
-          oracle_price: oraclePrice,
-          mark_price: markPrice,
-          oracle_slot: r?.slot == null ? null : String(r.slot),
-          source: cfg.source,
-          raw: {
-            from: 'dlob_stats_latest',
-            market_name: marketName,
-            market_index: marketIndex,
-            dlob: {
-              ts: r?.ts ?? null,
-              slot: r?.slot ?? null,
-              best_bid_price: r?.best_bid_price ?? null,
-              best_ask_price: r?.best_ask_price ?? null,
-              mid_price: r?.mid_price ?? null,
-              updated_at: updatedAt || null,
-            },
-          },
-        });
-      }
-
-      const inserted = await insertTicks(cfg, objects);
-      if (inserted) {
-        console.log(`[dlob-ticks] inserted=${inserted} ts=${nowIso}`);
-      }
-    } catch (err) {
-      console.error(`[dlob-ticks] error: ${String(err?.message || err)}`);
-      await sleep(2000);
-    }
-
-    await sleep(cfg.pollMs);
-  }
-}
-
-main().catch((err) => {
-  console.error(String(err?.stack || err));
-  process.exitCode = 1;
-});

kustomize/overlays/mevnode-bot/dlob-publisher-all-service.yaml

@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: dlob-publisher-all
+  annotations:
+    argocd.argoproj.io/sync-wave: "4"
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "9464"
+    prometheus.io/path: /metrics
+spec:
+  selector:
+    app.kubernetes.io/name: dlob-publisher-all
+  ports:
+    - name: http
+      port: 8080
+      targetPort: 8080
+    - name: metrics
+      port: 9464
+      targetPort: 9464

kustomize/overlays/mevnode-bot/dlob-publisher-all.yaml

@@ -0,0 +1,102 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: dlob-publisher-all
+  annotations:
+    argocd.argoproj.io/sync-wave: "4"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: dlob-publisher-all
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: dlob-publisher-all
+    spec:
+      imagePullSecrets: []
+      containers:
+        - name: publisher
+          image: gitea.mpabi.pl/trade/trade-dlob-server:grpc-reconnect-20260319-023351
+          imagePullPolicy: IfNotPresent
+          ports:
+            - name: http
+              containerPort: 8080
+          env:
+            - name: RUNNING_LOCAL
+              value: "true"
+            - name: LOCAL_CACHE
+              value: "true"
+            - name: ENV
+              value: mainnet-beta
+            - name: USE_WEBSOCKET
+              value: "true"
+            - name: USE_ORDER_SUBSCRIBER
+              value: "true"
+            - name: USE_GRPC
+              value: "true"
+            - name: DISABLE_GPA_REFRESH
+              value: "true"
+            - name: KILLSWITCH_SLOT_DIFF_THRESHOLD
+              value: "1000000000"
+            - name: ENABLE_TOB_MONITORING
+              value: "false"
+            - name: ELASTICACHE_HOST
+              value: dlob-redis
+            - name: ELASTICACHE_PORT
+              value: "6379"
+            - name: REDIS_CLIENT
+              value: DLOB_ALL
+            - name: FETCH_CONNECT_TIMEOUT_MS
+              value: "15000"
+            - name: FETCH_HEADERS_TIMEOUT_MS
+              value: "300000"
+            - name: FETCH_BODY_TIMEOUT_MS
+              value: "300000"
+            - name: GRPC_CLIENT
+              value: "yellowstone"
+            - name: ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: trade-dlob-rpc
+                  key: ENDPOINT
+            - name: WS_ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: trade-dlob-rpc
+                  key: WS_ENDPOINT
+            - name: GRPC_ENDPOINT
+              valueFrom:
+                secretKeyRef:
+                  name: trade-dlob-rpc
+                  key: GRPC_ENDPOINT
+            - name: TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: trade-dlob-rpc
+                  key: TOKEN
+          command: ["node", "/lib/publishers/dlobPublisher.js"]
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 180
+          readinessProbe:
+            httpGet:
+              path: /startup
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 24
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 20
+            timeoutSeconds: 3
+            failureThreshold: 10

kustomize/overlays/mevnode-bot/dlob-redis-retention.yaml

@@ -0,0 +1,63 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: dlob-redis-retention
+data:
+  clean.sh: |
+    #!/bin/sh
+    set -eu
+    now_s="$(date +%s)"
+
+    for pattern in 'dlob-hot:*' 'dlob-all:*'; do
+      max_age_s=900
+      if [ "$pattern" = 'dlob-all:*' ]; then
+        max_age_s=3600
+      fi
+
+      for key in $(redis-cli -h dlob-redis --scan --pattern "$pattern"); do
+        value="$(redis-cli -h dlob-redis --raw get "$key" || true)"
+        ts="$(printf '%s' "$value" | sed -n 's/.*"ts":\([0-9][0-9]*\).*/\1/p')"
+        if [ -z "$ts" ]; then
+          ts="$(printf '%s' "$value" | sed -n 's/.*"updatedAtTs":\([0-9][0-9]*\).*/\1/p')"
+        fi
+        [ -n "$ts" ] || continue
+        if [ $((now_s - ts / 1000)) -gt "$max_age_s" ]; then
+          redis-cli -h dlob-redis del "$key" >/dev/null
+        fi
+      done
+    done
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: dlob-redis-retention
+spec:
+  schedule: "*/5 * * * *"
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 1
+  jobTemplate:
+    spec:
+      template:
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: cleaner
+              image: redis:7-alpine
+              command:
+                - /bin/sh
+                - /scripts/clean.sh
+              resources:
+                requests:
+                  cpu: 10m
+                  memory: 32Mi
+                limits:
+                  cpu: 100m
+                  memory: 128Mi
+              volumeMounts:
+                - name: script
+                  mountPath: /scripts
+          volumes:
+            - name: script
+              configMap:
+                name: dlob-redis-retention
+                defaultMode: 365

kustomize/overlays/mevnode-bot/kustomization.yaml

@@ -12,30 +12,25 @@ resources:
   - ../../base/hasura/service.yaml
   - ../../base/hasura/deployment.yaml
   - ../../base/hasura/job-bootstrap.yaml
-  - ../../base/api/service.yaml
-  - ../../base/api/deployment.yaml
-  - ../../base/ingestor/deployment.yaml
   - ../../base/frontend/service.yaml
   - ../../base/frontend/deployment.yaml
   - ../../base/dlob/redis.yaml
+  - priority-classes.yaml
   - ../../base/dlob/publisher-deployment.yaml
-  - ../../base/dlob/server-service.yaml
-  - ../../base/dlob/server-deployment.yaml
-  - ../../base/dlob-worker/deployment.yaml
-  - ../../base/dlob-depth-worker/deployment.yaml
-  - ../../base/dlob-slippage-worker/deployment.yaml
+  - dlob-publisher-all.yaml
+  - dlob-publisher-all-service.yaml
+  - dlob-hot-redis-to-postgres-raw-writer.yaml
+  - dlob-hot-postgres-to-postgres-derived-writer.yaml
+  - dlob-all-redis-to-postgres-derived-writer.yaml
+  - dlob-redis-retention.yaml
 
 patchesStrategicMerge:
-  - patch-api.yaml
   - patch-frontend.yaml
   - patch-frontend-service.yaml
-  - patch-ingestor.yaml
   - patch-dlob-publisher.yaml
   - patch-dlob-publisher-storage.yaml
-  - patch-dlob-server.yaml
-  - patch-dlob-worker.yaml
-  - patch-dlob-depth-worker.yaml
-  - patch-dlob-slippage-worker.yaml
+  - patch-dlob-publisher-hot-priority.yaml
+  - patch-dlob-publisher-all-priority.yaml
 
 configMapGenerator:
   - name: postgres-initdb
@@ -44,24 +39,6 @@ configMapGenerator:
   - name: hasura-bootstrap-script
     files:
       - ../../base/hasura/hasura-bootstrap.mjs
-  - name: trade-api-wrapper
-    files:
-      - ../../base/api/wrapper.mjs
-  - name: trade-api-upstream
-    files:
-      - ../../base/api/server.mjs
-  - name: dlob-worker-script
-    files:
-      - ../../base/dlob-worker/worker.mjs
-  - name: dlob-depth-worker-script
-    files:
-      - ../../base/dlob-depth-worker/worker.mjs
-  - name: dlob-slippage-worker-script
-    files:
-      - ../../base/dlob-slippage-worker/worker.mjs
-  - name: trade-dlob-ingestor-script
-    files:
-      - dlob-ingestor.mjs
 
 generatorOptions:
   disableNameSuffixHash: true

kustomize/overlays/mevnode-bot/patch-dlob-depth-worker.yaml

@@ -1,12 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: dlob-depth-worker
-spec:
-  template:
-    spec:
-      containers:
-        - name: worker
-          env:
-            - name: DLOB_MARKETS
-              value: SOL-PERP,PUMP-PERP

kustomize/overlays/mevnode-bot/patch-dlob-publisher-all-priority.yaml

@@ -1,8 +1,8 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: trade-api
+  name: dlob-publisher-all
 spec:
   template:
     spec:
-      imagePullSecrets: []
+      priorityClassName: dlob-all

kustomize/overlays/mevnode-bot/patch-dlob-publisher-hot-priority.yaml

@@ -1,8 +1,8 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: dlob-server
+  name: dlob-publisher-hot
 spec:
   template:
     spec:
-      imagePullSecrets: []
+      priorityClassName: dlob-hot

kustomize/overlays/mevnode-bot/patch-dlob-publisher-storage.yaml

@@ -1,7 +1,7 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: dlob-publisher
+  name: dlob-publisher-hot
 spec:
   replicas: 1
   template:
@@ -12,7 +12,7 @@ spec:
             - name: ENABLE_PERSISTENT_STORE
               value: "true"
             - name: DLOB_SOURCE
-              value: "mevnode_bot"
+              value: "mevnode_bot_hot"
             - name: PRICE_PRECISION
               value: "1000000"
             - name: BASE_PRECISION

kustomize/overlays/mevnode-bot/patch-dlob-publisher.yaml

@@ -1,7 +1,7 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: dlob-publisher
+  name: dlob-publisher-hot
 spec:
   template:
     spec:
@@ -10,13 +10,19 @@ spec:
         - name: publisher
           env:
             - name: PERP_MARKETS_TO_LOAD
-              value: "0,75"
+              value: "0,20,72"
             - name: USE_GRPC
               value: "true"
             - name: USE_WEBSOCKET
               value: "true"
             - name: DISABLE_GPA_REFRESH
               value: "true"
+            - name: FETCH_CONNECT_TIMEOUT_MS
+              value: "15000"
+            - name: FETCH_HEADERS_TIMEOUT_MS
+              value: "300000"
+            - name: FETCH_BODY_TIMEOUT_MS
+              value: "300000"
             - name: GRPC_CLIENT
               value: "yellowstone"
             - name: GRPC_ENDPOINT
@@ -29,3 +35,27 @@ spec:
                 secretKeyRef:
                   name: trade-dlob-rpc
                   key: TOKEN
+          startupProbe:
+            httpGet:
+              path: /startup
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 90
+          readinessProbe:
+            httpGet:
+              path: /startup
+              port: http
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 3
+            failureThreshold: 12
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: http
+            initialDelaySeconds: 30
+            periodSeconds: 20
+            timeoutSeconds: 3
+            failureThreshold: 10

kustomize/overlays/mevnode-bot/patch-dlob-slippage-worker.yaml

@@ -1,12 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: dlob-slippage-worker
-spec:
-  template:
-    spec:
-      containers:
-        - name: worker
-          env:
-            - name: DLOB_MARKETS
-              value: SOL-PERP,PUMP-PERP

kustomize/overlays/mevnode-bot/patch-dlob-worker.yaml

@@ -1,12 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: dlob-worker
-spec:
-  template:
-    spec:
-      containers:
-        - name: worker
-          env:
-            - name: DLOB_MARKETS
-              value: SOL-PERP,PUMP-PERP

kustomize/overlays/mevnode-bot/patch-ingestor.yaml

@@ -1,39 +0,0 @@
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: trade-ingestor
-spec:
-  template:
-    spec:
-      imagePullSecrets: []
-      containers:
-        - name: ingestor
-          image: node:20-slim
-          imagePullPolicy: IfNotPresent
-          env:
-            - name: HASURA_GRAPHQL_URL
-              value: http://hasura:8080/v1/graphql
-            - name: HASURA_ADMIN_SECRET
-              valueFrom:
-                secretKeyRef:
-                  name: trade-hasura
-                  key: HASURA_GRAPHQL_ADMIN_SECRET
-            - name: DLOB_MARKETS
-              value: SOL-PERP,PUMP-PERP
-            - name: TICKS_POLL_MS
-              value: "1000"
-            - name: TICKS_SOURCE
-              value: "dlob_stats"
-          command:
-            - node
-          args:
-            - /opt/dlob/dlob-ingestor.mjs
-          volumeMounts:
-            - name: dlob-script
-              mountPath: /opt/dlob/dlob-ingestor.mjs
-              subPath: dlob-ingestor.mjs
-              readOnly: true
-      volumes:
-        - name: dlob-script
-          configMap:
-            name: trade-dlob-ingestor-script

kustomize/overlays/mevnode-bot/priority-classes.yaml

@@ -0,0 +1,15 @@
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata:
+  name: dlob-hot
+value: 100000
+globalDefault: false
+description: Higher scheduling priority for hot DLOB publisher.
+---
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata:
+  name: dlob-all
+value: 50000
+globalDefault: false
+description: Lower scheduling priority for all-market DLOB publisher.