trade/trade-deploy
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(snapshot): sync workspace state

Browse Source
This commit is contained in:
mpabi
2026-03-29 13:14:30 +02:00
parent 71b00f80bb
commit 4354de5c83
46 changed files with 959 additions and 5718 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
kustomize/overlays/mevnode-bot/README.md
View File

@@ -6,15 +6,13 @@ Zakres:
- `postgres`
- `hasura`
- `trade-api`
- `trade-frontend` (`NodePort 30081`)
- `trade-ingestor`
- `dlob-redis`
- `dlob-publisher`
- `dlob-server`
- `dlob-worker`
- `dlob-depth-worker`
- `dlob-slippage-worker`
- `dlob-publisher-hot`
- `dlob-publisher-all`
- `dlob-hot-redis-to-postgres-raw-writer`
- `dlob-hot-postgres-to-postgres-derived-writer`
- `dlob-all-redis-to-postgres-derived-writer`
Założenia runtime:
@@ -26,11 +24,47 @@ Założenia runtime:
- `trade-frontend` jest wystawiany przez `NodePort 30081`.
- Namespace pozostaje `trade-staging`.
Topologia DLOB w tym overlayu:
- `dlob-publisher-hot` to hot-path na wybrane markety (`PERP_MARKETS_TO_LOAD=0,20,72`, czyli `SOL-PERP`, `JTO-PERP`, `ADA-PERP`) i prefix Redis `DLOB_HOT` (`dlob-hot:`).
- `dlob-publisher-all` to szeroki feed do wszystkich marketów na prefix Redis `DLOB_ALL` (`dlob-all:`).
- `dlob-hot-redis-to-postgres-raw-writer` to schema-first writer: czyta `dlob-hot:` z Redis i zapisuje kanoniczny raw DLOB do PostgreSQL (`dlob_hot_snapshot_latest`, `dlob_hot_snapshot_ts`).
- `dlob-hot-postgres-to-postgres-derived-writer` czyta raw hot z PostgreSQL i buduje normalized layer (`dlob_hot_derived_latest`, `dlob_hot_derived_ts`).
- `LIVE` na `k3s`: sciezka `hot` jest juz aktywna jako `Redis -> raw PG -> derived PG`.
- `dlob-publisher-all` celowo nie ma patcha persistent store, żeby nie dublowac zapisow do Postgresa.
- `dlob-all-redis-to-postgres-derived-writer` zapisuje dla `all` tylko warstwe pochodna do PostgreSQL; bez `raw_ts`, zeby nie zjadac dysku w kilka dni.
- `LIVE` na `k3s`: sciezka `all` jest aktywna jako `Redis -> derived PG`.
- `all` zapisuje pelne `derived_ts` dla wszystkich marketow z `dlob-all:*`, bez samplingu i bez shortlisty marketow.
Planowany zakres zapisu dla `all`:
- tozsamosc i czas: `source`, `market_type`, `market_index`, `market_name`, `is_indicative`, `event_ts/ts_ms`, `slot`, `market_slot`
- cena i mikrostruktura: `mark_price`, `oracle_price`, `best_bid_price`, `best_ask_price`, `mid_price`, `spread_quote`, `spread_bps`
- plynnosc: `depth_levels`, `bid_levels`, `ask_levels`, `top_bid_size`, `top_ask_size`, `top_bid_notional`, `top_ask_notional`, `depth_bid_base`, `depth_ask_base`, `depth_bid_quote`, `depth_ask_quote`, `imbalance`
- drabinka do modelu/UI: `bids_norm`, `asks_norm` jako top `10` leveli z polami `price`, `sizeBase`, `notional`, `sources`
- kontrola jakosci i dedupe: `raw_payload_hash`, `updated_at` / `inserted_at`
- celowo pomijamy: pelny `raw payload`, `raw latest`, `raw ts`, `L3`, `best_makers`
Polityka `hot` vs `all`:
- `hot` i `all` rozniamy po roli operacyjnej, nie po samym mechanizmie subskrypcji.
- `hot` = `dlob-publisher-hot`: subset marketow, najnizsza latencja, najwyzszy priorytet operacyjny.
- `all` = `dlob-publisher-all`: pelne pokrycie marketow, feed do coverage/read-side/analityki, nizszy priorytet operacyjny niz `hot`.
- `USE_ORDER_SUBSCRIBER=true` moze byc wlaczone na obu publisherach; to nie definiuje roli `hot` vs `all`.
- `hot` powinien byc broniony jako pierwszy przy problemach z CPU/RAM/RPC i nie powinien byc degradowany przed `all`.
- Jesli trzeba ograniczac load albo robic fallback, najpierw degradujemy `all`, dopiero potem `hot`.
Zalecana polityka runtime:
- `hot`: utrzymywac `OrderSubscriber + gRPC` jako sciezke docelowa.
- `all`: domyslnie tez `OrderSubscriber + gRPC`, ale to pierwszy kandydat do przejscia na slabszy tryb przy presji zasobow.
- Kolejnosc degradacji dla `all`: `gRPC -> websocket -> polling`.
- Wylaczenie `USE_ORDER_SUBSCRIBER` traktowac jako tryb awaryjny/testowy, nie jako stan docelowy.
Wymagane sekrety:
- `trade-postgres`
- `trade-hasura`
- `trade-api`
- `trade-frontend-tokens`
- `trade-dlob-rpc`

66
kustomize/overlays/mevnode-bot/dlob-all-redis-to-postgres-derived-writer.yaml Normal file
View File

@@ -0,0 +1,66 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-all-redis-to-postgres-derived-writer
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dlob-all-redis-to-postgres-derived-writer
template:
metadata:
labels:
app.kubernetes.io/name: dlob-all-redis-to-postgres-derived-writer
spec:
priorityClassName: dlob-all
imagePullSecrets: []
containers:
- name: writer
image: gitea.mpabi.pl/trade/trade-dlob-server:all-derived-20260317-014016
imagePullPolicy: IfNotPresent
env:
- name: DLOB_SOURCE
value: mevnode_bot_all_derived
- name: REDIS_HOST
value: dlob-redis
- name: REDIS_PORT
value: "6379"
- name: REDIS_KEY_PREFIX
value: "dlob-all:"
- name: DLOB_POLL_MS
value: "1000"
- name: NORMALIZED_DEPTH
value: "10"
- name: PRICE_PRECISION
value: "1000000"
- name: BASE_PRECISION
value: "1000000000"
- name: PGHOST
value: postgres
- name: PGPORT
value: "5432"
- name: PGUSER
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_USER
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_PASSWORD
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_DB
command: ["node", "/lib/scripts/dlobAllRedisToPostgresDerivedWriter.js"]
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 1000m
memory: 2Gi

62
kustomize/overlays/mevnode-bot/dlob-hot-postgres-to-postgres-derived-writer.yaml Normal file
View File

@@ -0,0 +1,62 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-hot-postgres-to-postgres-derived-writer
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
template:
metadata:
labels:
app.kubernetes.io/name: dlob-hot-postgres-to-postgres-derived-writer
spec:
priorityClassName: dlob-hot
imagePullSecrets: []
containers:
- name: writer
image: gitea.mpabi.pl/trade/trade-dlob-server:hot-pg-events-20260320-205517
imagePullPolicy: IfNotPresent
env:
- name: DLOB_SOURCE
value: mevnode_bot_hot_derived
- name: RAW_SOURCE
value: mevnode_bot_hot_raw
- name: DLOB_POLL_MS
value: "5000"
- name: NORMALIZED_DEPTH
value: "10"
- name: PRICE_PRECISION
value: "1000000"
- name: BASE_PRECISION
value: "1000000000"
- name: PGHOST
value: postgres
- name: PGPORT
value: "5432"
- name: PGUSER
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_USER
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_PASSWORD
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_DB
command: ["node", "/lib/scripts/dlobHotPostgresToPostgresDerivedWriter.js"]
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi

60
kustomize/overlays/mevnode-bot/dlob-hot-redis-to-postgres-raw-writer.yaml Normal file
View File

@@ -0,0 +1,60 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-hot-redis-to-postgres-raw-writer
annotations:
argocd.argoproj.io/sync-wave: "5"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
template:
metadata:
labels:
app.kubernetes.io/name: dlob-hot-redis-to-postgres-raw-writer
spec:
priorityClassName: dlob-hot
imagePullSecrets: []
containers:
- name: writer
image: gitea.mpabi.pl/trade/trade-dlob-server:hot-pg-events-20260320-205517
imagePullPolicy: IfNotPresent
env:
- name: DLOB_SOURCE
value: mevnode_bot_hot_raw
- name: REDIS_HOST
value: dlob-redis
- name: REDIS_PORT
value: "6379"
- name: REDIS_KEY_PREFIX
value: "dlob-hot:"
- name: DLOB_POLL_MS
value: "5000"
- name: PGHOST
value: postgres
- name: PGPORT
value: "5432"
- name: PGUSER
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_USER
- name: PGPASSWORD
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_PASSWORD
- name: PGDATABASE
valueFrom:
secretKeyRef:
name: trade-postgres
key: POSTGRES_DB
command: ["node", "/lib/scripts/dlobHotRedisToPostgresRawWriter.js"]
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 1Gi

219
kustomize/overlays/mevnode-bot/dlob-ingestor.mjs
View File

@@ -1,219 +0,0 @@
import process from 'node:process';
import { setTimeout as sleep } from 'node:timers/promises';
function getIsoNow() {
return new Date().toISOString();
}
function envString(name, fallback) {
const v = process.env[name];
if (v == null) return fallback;
const s = String(v).trim();
return s ? s : fallback;
}
function envInt(name, fallback, { min, max } = {}) {
const v = process.env[name];
if (v == null) return fallback;
const n = Number.parseInt(String(v), 10);
if (!Number.isFinite(n)) return fallback;
const low = typeof min === 'number' ? min : n;
const high = typeof max === 'number' ? max : n;
return Math.max(low, Math.min(high, n));
}
function envList(name, fallbackCsv) {
const raw = process.env[name] ?? fallbackCsv;
return String(raw)
.split(',')
.map((s) => s.trim())
.filter(Boolean);
}
function toIntOrNull(v) {
if (v == null) return null;
if (typeof v === 'number') return Number.isFinite(v) ? Math.trunc(v) : null;
if (typeof v === 'string') {
const s = v.trim();
if (!s) return null;
const n = Number.parseInt(s, 10);
return Number.isFinite(n) ? n : null;
}
return null;
}
function numStr(v) {
if (v == null) return null;
if (typeof v === 'number') return Number.isFinite(v) ? String(v) : null;
if (typeof v === 'string') {
const s = v.trim();
return s ? s : null;
}
return null;
}
function isoFromEpochMs(v) {
const n = typeof v === 'number' ? v : typeof v === 'string' ? Number(v.trim()) : NaN;
if (!Number.isFinite(n) || n <= 0) return null;
const d = new Date(n);
const ms = d.getTime();
if (!Number.isFinite(ms)) return null;
return d.toISOString();
}
function resolveConfig() {
const hasuraUrl = envString('HASURA_GRAPHQL_URL', 'http://hasura:8080/v1/graphql');
const hasuraAdminSecret = envString('HASURA_ADMIN_SECRET', '');
if (!hasuraAdminSecret) throw new Error('Missing HASURA_ADMIN_SECRET');
const markets = envList('DLOB_MARKETS', 'SOL-PERP,PUMP-PERP');
const pollMs = envInt('TICKS_POLL_MS', 1000, { min: 250, max: 60000 });
const source = envString('TICKS_SOURCE', 'dlob_stats');
return { hasuraUrl, hasuraAdminSecret, markets, pollMs, source };
}
async function graphqlRequest(cfg, query, variables) {
const res = await fetch(cfg.hasuraUrl, {
method: 'POST',
headers: {
'content-type': 'application/json',
'x-hasura-admin-secret': cfg.hasuraAdminSecret,
},
body: JSON.stringify({ query, variables }),
signal: AbortSignal.timeout(10000),
});
const text = await res.text();
if (!res.ok) throw new Error(`Hasura HTTP ${res.status}: ${text}`);
let json;
try {
json = JSON.parse(text);
} catch {
throw new Error(`Hasura: invalid json: ${text}`);
}
if (json.errors?.length) throw new Error(json.errors.map((e) => e.message).join(' | '));
return json.data;
}
async function fetchStats(cfg) {
const query = `
query DlobStatsLatest($markets: [String!]!) {
dlob_stats_latest(where: { market_name: { _in: $markets } }) {
market_name
market_index
ts
slot
oracle_price
mark_price
mid_price
best_bid_price
best_ask_price
updated_at
}
}
`;
const data = await graphqlRequest(cfg, query, { markets: cfg.markets });
const rows = Array.isArray(data?.dlob_stats_latest) ? data.dlob_stats_latest : [];
return rows;
}
async function insertTicks(cfg, objects) {
if (!objects.length) return 0;
const mutation = `
mutation InsertTicks($objects: [drift_ticks_insert_input!]!) {
insert_drift_ticks(objects: $objects) { affected_rows }
}
`;
const data = await graphqlRequest(cfg, mutation, { objects });
return Number(data?.insert_drift_ticks?.affected_rows || 0);
}
async function main() {
const cfg = resolveConfig();
const lastUpdatedAtByMarket = new Map();
console.log(
JSON.stringify(
{
service: 'trade-ingestor',
mode: 'dlob_stats_ticks',
startedAt: getIsoNow(),
hasuraUrl: cfg.hasuraUrl,
markets: cfg.markets,
pollMs: cfg.pollMs,
source: cfg.source,
},
null,
2
)
);
while (true) {
try {
const rows = await fetchStats(cfg);
const nowIso = getIsoNow();
const objects = [];
for (const r of rows) {
const marketName = String(r?.market_name || '').trim();
if (!marketName) continue;
const updatedAt = r?.updated_at ? String(r.updated_at) : '';
if (updatedAt && lastUpdatedAtByMarket.get(marketName) === updatedAt) continue;
if (updatedAt) lastUpdatedAtByMarket.set(marketName, updatedAt);
const marketIndex = toIntOrNull(r?.market_index) ?? 0;
const dlobIso = isoFromEpochMs(r?.ts);
const tsIso = dlobIso || nowIso;
const oraclePrice = numStr(r?.oracle_price) || numStr(r?.mark_price) || numStr(r?.mid_price);
const markPrice = numStr(r?.mark_price) || numStr(r?.mid_price) || oraclePrice;
if (!oraclePrice) continue;
objects.push({
ts: tsIso,
market_index: marketIndex,
symbol: marketName,
oracle_price: oraclePrice,
mark_price: markPrice,
oracle_slot: r?.slot == null ? null : String(r.slot),
source: cfg.source,
raw: {
from: 'dlob_stats_latest',
market_name: marketName,
market_index: marketIndex,
dlob: {
ts: r?.ts ?? null,
slot: r?.slot ?? null,
best_bid_price: r?.best_bid_price ?? null,
best_ask_price: r?.best_ask_price ?? null,
mid_price: r?.mid_price ?? null,
updated_at: updatedAt || null,
},
},
});
}
const inserted = await insertTicks(cfg, objects);
if (inserted) {
console.log(`[dlob-ticks] inserted=${inserted} ts=${nowIso}`);
}
} catch (err) {
console.error(`[dlob-ticks] error: ${String(err?.message || err)}`);
await sleep(2000);
}
await sleep(cfg.pollMs);
}
}
main().catch((err) => {
console.error(String(err?.stack || err));
process.exitCode = 1;
});

19
kustomize/overlays/mevnode-bot/dlob-publisher-all-service.yaml Normal file
View File

@@ -0,0 +1,19 @@
apiVersion: v1
kind: Service
metadata:
name: dlob-publisher-all
annotations:
argocd.argoproj.io/sync-wave: "4"
prometheus.io/scrape: "true"
prometheus.io/port: "9464"
prometheus.io/path: /metrics
spec:
selector:
app.kubernetes.io/name: dlob-publisher-all
ports:
- name: http
port: 8080
targetPort: 8080
- name: metrics
port: 9464
targetPort: 9464

102
kustomize/overlays/mevnode-bot/dlob-publisher-all.yaml Normal file
View File

@@ -0,0 +1,102 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-publisher-all
annotations:
argocd.argoproj.io/sync-wave: "4"
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: dlob-publisher-all
template:
metadata:
labels:
app.kubernetes.io/name: dlob-publisher-all
spec:
imagePullSecrets: []
containers:
- name: publisher
image: gitea.mpabi.pl/trade/trade-dlob-server:grpc-reconnect-20260319-023351
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
- name: RUNNING_LOCAL
value: "true"
- name: LOCAL_CACHE
value: "true"
- name: ENV
value: mainnet-beta
- name: USE_WEBSOCKET
value: "true"
- name: USE_ORDER_SUBSCRIBER
value: "true"
- name: USE_GRPC
value: "true"
- name: DISABLE_GPA_REFRESH
value: "true"
- name: KILLSWITCH_SLOT_DIFF_THRESHOLD
value: "1000000000"
- name: ENABLE_TOB_MONITORING
value: "false"
- name: ELASTICACHE_HOST
value: dlob-redis
- name: ELASTICACHE_PORT
value: "6379"
- name: REDIS_CLIENT
value: DLOB_ALL
- name: FETCH_CONNECT_TIMEOUT_MS
value: "15000"
- name: FETCH_HEADERS_TIMEOUT_MS
value: "300000"
- name: FETCH_BODY_TIMEOUT_MS
value: "300000"
- name: GRPC_CLIENT
value: "yellowstone"
- name: ENDPOINT
valueFrom:
secretKeyRef:
name: trade-dlob-rpc
key: ENDPOINT
- name: WS_ENDPOINT
valueFrom:
secretKeyRef:
name: trade-dlob-rpc
key: WS_ENDPOINT
- name: GRPC_ENDPOINT
valueFrom:
secretKeyRef:
name: trade-dlob-rpc
key: GRPC_ENDPOINT
- name: TOKEN
valueFrom:
secretKeyRef:
name: trade-dlob-rpc
key: TOKEN
command: ["node", "/lib/publishers/dlobPublisher.js"]
startupProbe:
httpGet:
path: /startup
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 180
readinessProbe:
httpGet:
path: /startup
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 24
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 10

63
kustomize/overlays/mevnode-bot/dlob-redis-retention.yaml Normal file
View File

@@ -0,0 +1,63 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: dlob-redis-retention
data:
clean.sh: |
#!/bin/sh
set -eu
now_s="$(date +%s)"
for pattern in 'dlob-hot:*' 'dlob-all:*'; do
max_age_s=900
if [ "$pattern" = 'dlob-all:*' ]; then
max_age_s=3600
fi
for key in $(redis-cli -h dlob-redis --scan --pattern "$pattern"); do
value="$(redis-cli -h dlob-redis --raw get "$key" || true)"
ts="$(printf '%s' "$value" | sed -n 's/.*"ts":\([0-9][0-9]*\).*/\1/p')"
if [ -z "$ts" ]; then
ts="$(printf '%s' "$value" | sed -n 's/.*"updatedAtTs":\([0-9][0-9]*\).*/\1/p')"
fi
[ -n "$ts" ] || continue
if [ $((now_s - ts / 1000)) -gt "$max_age_s" ]; then
redis-cli -h dlob-redis del "$key" >/dev/null
fi
done
done
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: dlob-redis-retention
spec:
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: cleaner
image: redis:7-alpine
command:
- /bin/sh
- /scripts/clean.sh
resources:
requests:
cpu: 10m
memory: 32Mi
limits:
cpu: 100m
memory: 128Mi
volumeMounts:
- name: script
mountPath: /scripts
volumes:
- name: script
configMap:
name: dlob-redis-retention
defaultMode: 365

41
kustomize/overlays/mevnode-bot/kustomization.yaml
View File

@@ -12,30 +12,25 @@ resources:
- ../../base/hasura/service.yaml
- ../../base/hasura/deployment.yaml
- ../../base/hasura/job-bootstrap.yaml
- ../../base/api/service.yaml
- ../../base/api/deployment.yaml
- ../../base/ingestor/deployment.yaml
- ../../base/frontend/service.yaml
- ../../base/frontend/deployment.yaml
- ../../base/dlob/redis.yaml
- priority-classes.yaml
- ../../base/dlob/publisher-deployment.yaml
- ../../base/dlob/server-service.yaml
- ../../base/dlob/server-deployment.yaml
- ../../base/dlob-worker/deployment.yaml
- ../../base/dlob-depth-worker/deployment.yaml
- ../../base/dlob-slippage-worker/deployment.yaml
- dlob-publisher-all.yaml
- dlob-publisher-all-service.yaml
- dlob-hot-redis-to-postgres-raw-writer.yaml
- dlob-hot-postgres-to-postgres-derived-writer.yaml
- dlob-all-redis-to-postgres-derived-writer.yaml
- dlob-redis-retention.yaml
patchesStrategicMerge:
- patch-api.yaml
- patch-frontend.yaml
- patch-frontend-service.yaml
- patch-ingestor.yaml
- patch-dlob-publisher.yaml
- patch-dlob-publisher-storage.yaml
- patch-dlob-server.yaml
- patch-dlob-worker.yaml
- patch-dlob-depth-worker.yaml
- patch-dlob-slippage-worker.yaml
- patch-dlob-publisher-hot-priority.yaml
- patch-dlob-publisher-all-priority.yaml
configMapGenerator:
- name: postgres-initdb
@@ -44,24 +39,6 @@ configMapGenerator:
- name: hasura-bootstrap-script
files:
- ../../base/hasura/hasura-bootstrap.mjs
- name: trade-api-wrapper
files:
- ../../base/api/wrapper.mjs
- name: trade-api-upstream
files:
- ../../base/api/server.mjs
- name: dlob-worker-script
files:
- ../../base/dlob-worker/worker.mjs
- name: dlob-depth-worker-script
files:
- ../../base/dlob-depth-worker/worker.mjs
- name: dlob-slippage-worker-script
files:
- ../../base/dlob-slippage-worker/worker.mjs
- name: trade-dlob-ingestor-script
files:
- dlob-ingestor.mjs
generatorOptions:
disableNameSuffixHash: true

12
kustomize/overlays/mevnode-bot/patch-dlob-depth-worker.yaml
View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-depth-worker
spec:
template:
spec:
containers:
- name: worker
env:
- name: DLOB_MARKETS
value: SOL-PERP,PUMP-PERP

4
kustomize/overlays/mevnode-bot/patch-api.yaml → kustomize/overlays/mevnode-bot/patch-dlob-publisher-all-priority.yaml
View File

@@ -1,8 +1,8 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: trade-api
name: dlob-publisher-all
spec:
template:
spec:
imagePullSecrets: []
priorityClassName: dlob-all

4
kustomize/overlays/mevnode-bot/patch-dlob-server.yaml → kustomize/overlays/mevnode-bot/patch-dlob-publisher-hot-priority.yaml
View File

@@ -1,8 +1,8 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-server
name: dlob-publisher-hot
spec:
template:
spec:
imagePullSecrets: []
priorityClassName: dlob-hot

4
kustomize/overlays/mevnode-bot/patch-dlob-publisher-storage.yaml
View File

@@ -1,7 +1,7 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-publisher
name: dlob-publisher-hot
spec:
replicas: 1
template:
@@ -12,7 +12,7 @@ spec:
- name: ENABLE_PERSISTENT_STORE
value: "true"
- name: DLOB_SOURCE
value: "mevnode_bot"
value: "mevnode_bot_hot"
- name: PRICE_PRECISION
value: "1000000"
- name: BASE_PRECISION

34
kustomize/overlays/mevnode-bot/patch-dlob-publisher.yaml
View File

@@ -1,7 +1,7 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-publisher
name: dlob-publisher-hot
spec:
template:
spec:
@@ -10,13 +10,19 @@ spec:
- name: publisher
env:
- name: PERP_MARKETS_TO_LOAD
value: "0,75"
value: "0,20,72"
- name: USE_GRPC
value: "true"
- name: USE_WEBSOCKET
value: "true"
- name: DISABLE_GPA_REFRESH
value: "true"
- name: FETCH_CONNECT_TIMEOUT_MS
value: "15000"
- name: FETCH_HEADERS_TIMEOUT_MS
value: "300000"
- name: FETCH_BODY_TIMEOUT_MS
value: "300000"
- name: GRPC_CLIENT
value: "yellowstone"
- name: GRPC_ENDPOINT
@@ -29,3 +35,27 @@ spec:
secretKeyRef:
name: trade-dlob-rpc
key: TOKEN
startupProbe:
httpGet:
path: /startup
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 90
readinessProbe:
httpGet:
path: /startup
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 12
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 30
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 10

12
kustomize/overlays/mevnode-bot/patch-dlob-slippage-worker.yaml
View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-slippage-worker
spec:
template:
spec:
containers:
- name: worker
env:
- name: DLOB_MARKETS
value: SOL-PERP,PUMP-PERP

12
kustomize/overlays/mevnode-bot/patch-dlob-worker.yaml
View File

@@ -1,12 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: dlob-worker
spec:
template:
spec:
containers:
- name: worker
env:
- name: DLOB_MARKETS
value: SOL-PERP,PUMP-PERP

39
kustomize/overlays/mevnode-bot/patch-ingestor.yaml
View File

@@ -1,39 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: trade-ingestor
spec:
template:
spec:
imagePullSecrets: []
containers:
- name: ingestor
image: node:20-slim
imagePullPolicy: IfNotPresent
env:
- name: HASURA_GRAPHQL_URL
value: http://hasura:8080/v1/graphql
- name: HASURA_ADMIN_SECRET
valueFrom:
secretKeyRef:
name: trade-hasura
key: HASURA_GRAPHQL_ADMIN_SECRET
- name: DLOB_MARKETS
value: SOL-PERP,PUMP-PERP
- name: TICKS_POLL_MS
value: "1000"
- name: TICKS_SOURCE
value: "dlob_stats"
command:
- node
args:
- /opt/dlob/dlob-ingestor.mjs
volumeMounts:
- name: dlob-script
mountPath: /opt/dlob/dlob-ingestor.mjs
subPath: dlob-ingestor.mjs
readOnly: true
volumes:
- name: dlob-script
configMap:
name: trade-dlob-ingestor-script

15
kustomize/overlays/mevnode-bot/priority-classes.yaml Normal file
View File

@@ -0,0 +1,15 @@
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: dlob-hot
value: 100000
globalDefault: false
description: Higher scheduling priority for hot DLOB publisher.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: dlob-all
value: 50000
globalDefault: false
description: Lower scheduling priority for all-market DLOB publisher.

81
kustomize/overlays/prod/frontend-server.mjs
View File

@@ -16,8 +16,6 @@ const STARTED_AT = new Date().toISOString();
const STATIC_DIR = process.env.STATIC_DIR || '/srv';
const BASIC_AUTH_FILE = process.env.BASIC_AUTH_FILE || '/tokens/frontend.json';
const API_READ_TOKEN_FILE = process.env.API_READ_TOKEN_FILE || '/tokens/read.json';
const API_UPSTREAM = process.env.API_UPSTREAM || process.env.API_URL || 'http://api:8787';
const HASURA_UPSTREAM = process.env.HASURA_UPSTREAM || 'http://hasura:8080';
const HASURA_GRAPHQL_PATH = process.env.HASURA_GRAPHQL_PATH || '/v1/graphql';
const GRAPHQL_CORS_ORIGIN = process.env.GRAPHQL_CORS_ORIGIN || process.env.CORS_ORIGIN || '*';
@@ -70,13 +68,6 @@ function loadBasicAuth() {
return { username, password };
}
function loadApiReadToken() {
const j = readJson(API_READ_TOKEN_FILE);
const token = (j?.token || '').toString();
if (!token) throw new Error(`Invalid API_READ_TOKEN_FILE: ${API_READ_TOKEN_FILE}`);
return token;
}
function send(res, status, headers, body) {
res.statusCode = status;
for (const [k, v] of Object.entries(headers || {})) res.setHeader(k, v);
@@ -439,56 +430,6 @@ function readBody(req, limitBytes = 1024 * 16) {
});
}
function proxyApi(req, res, apiReadToken) {
const upstreamBase = new URL(API_UPSTREAM);
const inUrl = new URL(req.url || '/', `http://${req.headers.host || 'localhost'}`);
const prefix = '/api';
const strippedPath = inUrl.pathname === prefix ? '/' : inUrl.pathname.startsWith(prefix + '/') ? inUrl.pathname.slice(prefix.length) : null;
if (strippedPath == null) {
send(res, 404, { 'content-type': 'text/plain; charset=utf-8' }, 'not_found');
return;
}
const target = new URL(upstreamBase.toString());
target.pathname = strippedPath || '/';
target.search = inUrl.search;
const isHttps = target.protocol === 'https:';
const lib = isHttps ? https : http;
const headers = stripHopByHopHeaders(req.headers);
delete headers.authorization; // basic auth from client must not leak upstream
headers.host = target.host;
headers.authorization = `Bearer ${apiReadToken}`;
const upstreamReq = lib.request(
{
protocol: target.protocol,
hostname: target.hostname,
port: target.port || (isHttps ? 443 : 80),
method: req.method,
path: target.pathname + target.search,
headers,
},
(upstreamRes) => {
const outHeaders = stripHopByHopHeaders(upstreamRes.headers);
res.writeHead(upstreamRes.statusCode || 502, outHeaders);
upstreamRes.pipe(res);
}
);
upstreamReq.on('error', (err) => {
if (!res.headersSent) {
send(res, 502, { 'content-type': 'text/plain; charset=utf-8' }, `bad_gateway: ${err?.message || err}`);
} else {
res.destroy();
}
});
req.pipe(upstreamReq);
}
function withCors(res) {
res.setHeader('access-control-allow-origin', GRAPHQL_CORS_ORIGIN);
res.setHeader('access-control-allow-methods', 'GET,POST,OPTIONS');
@@ -734,26 +675,6 @@ async function handler(req, res) {
}
}
if (req.url?.startsWith('/api') && (req.url === '/api' || req.url.startsWith('/api/'))) {
if (AUTH_MODE !== 'off' && AUTH_MODE !== 'none' && AUTH_MODE !== 'disabled') {
const user = resolveAuthenticatedUser(req);
if (!user) {
unauthorized(res);
return;
}
}
let token;
try {
token = loadApiReadToken();
} catch (e) {
send(res, 500, { 'content-type': 'text/plain; charset=utf-8' }, String(e?.message || e));
return;
}
proxyApi(req, res, token);
return;
}
serveStatic(req, res);
}
@@ -797,11 +718,9 @@ server.listen(PORT, () => {
service: 'trade-frontend',
port: PORT,
staticDir: STATIC_DIR,
apiUpstream: API_UPSTREAM,
hasuraUpstream: HASURA_UPSTREAM,
basicAuthFile: BASIC_AUTH_FILE,
basicAuthMode: BASIC_AUTH_MODE,
apiReadTokenFile: API_READ_TOKEN_FILE,
authUserHeader: AUTH_USER_HEADER,
authMode: AUTH_MODE,
htpasswdFile: HTPASSWD_FILE,

219
kustomize/overlays/staging/dlob-ingestor.mjs
View File

@@ -1,219 +0,0 @@
import process from 'node:process';
import { setTimeout as sleep } from 'node:timers/promises';
function getIsoNow() {
return new Date().toISOString();
}
function envString(name, fallback) {
const v = process.env[name];
if (v == null) return fallback;
const s = String(v).trim();
return s ? s : fallback;
}
function envInt(name, fallback, { min, max } = {}) {
const v = process.env[name];
if (v == null) return fallback;
const n = Number.parseInt(String(v), 10);
if (!Number.isFinite(n)) return fallback;
const low = typeof min === 'number' ? min : n;
const high = typeof max === 'number' ? max : n;
return Math.max(low, Math.min(high, n));
}
function envList(name, fallbackCsv) {
const raw = process.env[name] ?? fallbackCsv;
return String(raw)
.split(',')
.map((s) => s.trim())
.filter(Boolean);
}
function toIntOrNull(v) {
if (v == null) return null;
if (typeof v === 'number') return Number.isFinite(v) ? Math.trunc(v) : null;
if (typeof v === 'string') {
const s = v.trim();
if (!s) return null;
const n = Number.parseInt(s, 10);
return Number.isFinite(n) ? n : null;
}
return null;
}
function numStr(v) {
if (v == null) return null;
if (typeof v === 'number') return Number.isFinite(v) ? String(v) : null;
if (typeof v === 'string') {
const s = v.trim();
return s ? s : null;
}
return null;
}
function isoFromEpochMs(v) {
const n = typeof v === 'number' ? v : typeof v === 'string' ? Number(v.trim()) : NaN;
if (!Number.isFinite(n) || n <= 0) return null;
const d = new Date(n);
const ms = d.getTime();
if (!Number.isFinite(ms)) return null;
return d.toISOString();
}
function resolveConfig() {
const hasuraUrl = envString('HASURA_GRAPHQL_URL', 'http://hasura:8080/v1/graphql');
const hasuraAdminSecret = envString('HASURA_ADMIN_SECRET', '');
if (!hasuraAdminSecret) throw new Error('Missing HASURA_ADMIN_SECRET');
const markets = envList('DLOB_MARKETS', 'SOL-PERP,DOGE-PERP,JUP-PERP');
const pollMs = envInt('TICKS_POLL_MS', 1000, { min: 250, max: 60_000 });
const source = envString('TICKS_SOURCE', 'dlob_stats');
return { hasuraUrl, hasuraAdminSecret, markets, pollMs, source };
}
async function graphqlRequest(cfg, query, variables) {
const res = await fetch(cfg.hasuraUrl, {
method: 'POST',
headers: {
'content-type': 'application/json',
'x-hasura-admin-secret': cfg.hasuraAdminSecret,
},
body: JSON.stringify({ query, variables }),
signal: AbortSignal.timeout(10_000),
});
const text = await res.text();
if (!res.ok) throw new Error(`Hasura HTTP ${res.status}: ${text}`);
let json;
try {
json = JSON.parse(text);
} catch {
throw new Error(`Hasura: invalid json: ${text}`);
}
if (json.errors?.length) throw new Error(json.errors.map((e) => e.message).join(' | '));
return json.data;
}
async function fetchStats(cfg) {
const query = `
query DlobStatsLatest($markets: [String!]!) {
dlob_stats_latest(where: { market_name: { _in: $markets } }) {
market_name
market_index
ts
slot
oracle_price
mark_price
mid_price
best_bid_price
best_ask_price
updated_at
}
}
`;
const data = await graphqlRequest(cfg, query, { markets: cfg.markets });
const rows = Array.isArray(data?.dlob_stats_latest) ? data.dlob_stats_latest : [];
return rows;
}
async function insertTicks(cfg, objects) {
if (!objects.length) return 0;
const mutation = `
mutation InsertTicks($objects: [drift_ticks_insert_input!]!) {
insert_drift_ticks(objects: $objects) { affected_rows }
}
`;
const data = await graphqlRequest(cfg, mutation, { objects });
return Number(data?.insert_drift_ticks?.affected_rows || 0);
}
async function main() {
const cfg = resolveConfig();
const lastUpdatedAtByMarket = new Map();
console.log(
JSON.stringify(
{
service: 'trade-ingestor',
mode: 'dlob_stats_ticks',
startedAt: getIsoNow(),
hasuraUrl: cfg.hasuraUrl,
markets: cfg.markets,
pollMs: cfg.pollMs,
source: cfg.source,
},
null,
2
)
);
while (true) {
try {
const rows = await fetchStats(cfg);
const nowIso = getIsoNow();
const objects = [];
for (const r of rows) {
const marketName = String(r?.market_name || '').trim();
if (!marketName) continue;
const updatedAt = r?.updated_at ? String(r.updated_at) : '';
if (updatedAt && lastUpdatedAtByMarket.get(marketName) === updatedAt) continue;
if (updatedAt) lastUpdatedAtByMarket.set(marketName, updatedAt);
const marketIndex = toIntOrNull(r?.market_index) ?? 0;
const dlobIso = isoFromEpochMs(r?.ts);
const tsIso = dlobIso || nowIso;
const oraclePrice = numStr(r?.oracle_price) || numStr(r?.mark_price) || numStr(r?.mid_price);
const markPrice = numStr(r?.mark_price) || numStr(r?.mid_price) || oraclePrice;
if (!oraclePrice) continue;
objects.push({
ts: tsIso,
market_index: marketIndex,
symbol: marketName,
oracle_price: oraclePrice,
mark_price: markPrice,
oracle_slot: r?.slot == null ? null : String(r.slot),
source: cfg.source,
raw: {
from: 'dlob_stats_latest',
market_name: marketName,
market_index: marketIndex,
dlob: {
ts: r?.ts ?? null,
slot: r?.slot ?? null,
best_bid_price: r?.best_bid_price ?? null,
best_ask_price: r?.best_ask_price ?? null,
mid_price: r?.mid_price ?? null,
updated_at: updatedAt || null,
},
},
});
}
const inserted = await insertTicks(cfg, objects);
if (inserted) {
console.log(`[dlob-ticks] inserted=${inserted} ts=${nowIso}`);
}
} catch (err) {
console.error(`[dlob-ticks] error: ${String(err?.message || err)}`);
await sleep(2_000);
}
await sleep(cfg.pollMs);
}
}
main().catch((err) => {
console.error(String(err?.stack || err));
process.exitCode = 1;
});

487
kustomize/overlays/staging/fake-ingestor.mjs
View File

@@ -1,487 +0,0 @@
import fs from 'node:fs';
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
function envString(name, fallback) {
const v = process.env[name];
if (v == null) return fallback;
const s = String(v).trim();
return s ? s : fallback;
}
function envNumber(name, fallback) {
const v = process.env[name];
if (v == null) return fallback;
const n = Number(v);
return Number.isFinite(n) ? n : fallback;
}
function envInt(name, fallback, { min, max } = {}) {
const v = process.env[name];
if (v == null) return fallback;
const n = Number.parseInt(String(v), 10);
if (!Number.isInteger(n)) return fallback;
const nn = Math.max(min ?? n, Math.min(max ?? n, n));
return nn;
}
function readJson(filePath) {
try {
const raw = fs.readFileSync(filePath, 'utf8');
return JSON.parse(raw);
} catch {
return undefined;
}
}
function readTokenFromFile(filePath) {
const json = readJson(filePath);
const raw = json?.token || json?.jwt || json?.authToken;
const tok = typeof raw === 'string' ? raw.trim() : '';
return tok ? tok : undefined;
}
function urlWithPath(baseUrl, path) {
const u = new URL(baseUrl);
const basePath = u.pathname && u.pathname !== '/' ? u.pathname.replace(/\/$/, '') : '';
u.pathname = `${basePath}${path}`;
return u;
}
async function httpJson(url, options) {
const res = await fetch(url, options);
const text = await res.text();
let json;
try {
json = JSON.parse(text);
} catch {
json = undefined;
}
return { ok: res.ok, status: res.status, json, text };
}
function mean(values) {
if (!values.length) return 0;
return values.reduce((a, b) => a + b, 0) / values.length;
}
function stddev(values) {
if (!values.length) return 0;
const m = mean(values);
const v = values.reduce((acc, x) => acc + (x - m) * (x - m), 0) / values.length;
return Math.sqrt(v);
}
function quantile(values, q) {
if (!values.length) return 0;
const sorted = values.slice().sort((a, b) => a - b);
const pos = (sorted.length - 1) * q;
const base = Math.floor(pos);
const rest = pos - base;
if (sorted[base + 1] == null) return sorted[base];
return sorted[base] + rest * (sorted[base + 1] - sorted[base]);
}
function randn() {
// Box–Muller
let u = 0;
let v = 0;
while (u === 0) u = Math.random();
while (v === 0) v = Math.random();
return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}
function clamp(v, min, max) {
if (v < min) return min;
if (v > max) return max;
return v;
}
function isTruthy(value) {
const v = String(value ?? '')
.trim()
.toLowerCase();
if (!v) return false;
return !['0', 'false', 'off', 'no', 'none', 'disabled'].includes(v);
}
function parsePriceFromTick(t) {
const mp = t?.mark_price ?? t?.markPrice;
const op = t?.oracle_price ?? t?.oraclePrice ?? t?.price;
const v = mp != null ? Number(mp) : Number(op);
return Number.isFinite(v) && v > 0 ? v : null;
}
function computeLogReturns(prices) {
const out = [];
for (let i = 1; i < prices.length; i++) {
const a = prices[i - 1];
const b = prices[i];
if (!(a > 0) || !(b > 0)) continue;
const r = Math.log(b / a);
if (Number.isFinite(r)) out.push(r);
}
return out;
}
class BlockSampler {
#values;
#minBlock;
#maxBlock;
#idx = 0;
#end = 0;
#remaining = 0;
constructor(values, { minBlock, maxBlock }) {
this.#values = values.slice();
this.#minBlock = Math.max(1, minBlock);
this.#maxBlock = Math.max(this.#minBlock, maxBlock);
this.#startNewBlock();
}
#startNewBlock() {
const n = this.#values.length;
if (n <= 1) {
this.#idx = 0;
this.#end = n;
this.#remaining = n;
return;
}
const start = Math.floor(Math.random() * n);
const span = this.#maxBlock - this.#minBlock + 1;
const len = this.#minBlock + Math.floor(Math.random() * span);
this.#idx = start;
this.#end = Math.min(n, start + len);
this.#remaining = this.#end - this.#idx;
}
next() {
if (this.#values.length === 0) return 0;
if (this.#remaining <= 0 || this.#idx >= this.#end) this.#startNewBlock();
const v = this.#values[this.#idx];
this.#idx += 1;
this.#remaining -= 1;
return Number.isFinite(v) ? v : 0;
}
}
async function fetchSeedTicksPage({ apiBase, readToken, symbol, source, limit, to }) {
const u = urlWithPath(apiBase, '/v1/ticks');
u.searchParams.set('symbol', symbol);
u.searchParams.set('limit', String(limit));
if (source) u.searchParams.set('source', source);
if (to) u.searchParams.set('to', to);
const res = await httpJson(u.toString(), {
method: 'GET',
headers: {
authorization: `Bearer ${readToken}`,
},
});
if (!res.ok) throw new Error(`seed_ticks_http_${res.status}: ${res.text}`);
if (!res.json?.ok) throw new Error(`seed_ticks_error: ${res.json?.error || res.text}`);
const ticks = Array.isArray(res.json?.ticks) ? res.json.ticks : [];
return ticks;
}
async function fetchSeedTicksPaged({ apiBase, readToken, symbol, source, desiredLimit, pageLimit, maxPages }) {
const pages = [];
let cursorTo = null;
for (let page = 0; page < maxPages; page++) {
const remaining = desiredLimit - pages.reduce((acc, p) => acc + p.length, 0);
if (remaining <= 0) break;
const limit = Math.min(pageLimit, remaining);
const ticks = await fetchSeedTicksPage({ apiBase, readToken, symbol, source, limit, to: cursorTo });
if (!ticks.length) break;
// Server returns ascending ticks; we unshift to keep overall chronological order.
pages.unshift(ticks);
const oldestTs = String(ticks[0]?.ts || '').trim();
if (!oldestTs) break;
const oldestMs = Date.parse(oldestTs);
if (!Number.isFinite(oldestMs)) break;
cursorTo = new Date(oldestMs - 1).toISOString();
}
const flat = pages.flat();
if (!flat.length) return flat;
// Best-effort de-duplication (in case of overlapping `to` bounds).
const seen = new Set();
const out = [];
for (const t of flat) {
const ts = String(t?.ts || '');
const key = ts ? ts : JSON.stringify(t);
if (seen.has(key)) continue;
seen.add(key);
out.push(t);
}
return out;
}
async function ingestTick({ apiBase, writeToken, tick }) {
const u = urlWithPath(apiBase, '/v1/ingest/tick');
const res = await httpJson(u.toString(), {
method: 'POST',
headers: {
'content-type': 'application/json',
authorization: `Bearer ${writeToken}`,
},
body: JSON.stringify(tick),
});
if (!res.ok) throw new Error(`ingest_http_${res.status}: ${res.text}`);
if (!res.json?.ok) throw new Error(`ingest_error: ${res.json?.error || res.text}`);
return res.json?.id || null;
}
function formatNumeric(value) {
if (!Number.isFinite(value)) return '0';
// Avoid scientific notation for small values while keeping reasonable precision.
const abs = Math.abs(value);
if (abs === 0) return '0';
if (abs >= 1) return value.toFixed(8).replace(/\.?0+$/, '');
if (abs >= 0.01) return value.toFixed(10).replace(/\.?0+$/, '');
if (abs >= 0.0001) return value.toFixed(12).replace(/\.?0+$/, '');
return value.toFixed(16).replace(/\.?0+$/, '');
}
async function main() {
const apiBase = envString('INGEST_API_URL', 'http://trade-api:8787');
const symbol = envString('MARKET_NAME', envString('SYMBOL', 'PUMP-PERP'));
const source = envString('SOURCE', 'drift_oracle');
const intervalMs = envInt('INTERVAL_MS', 1000, { min: 50, max: 60_000 });
const readTokenFile = envString('FAKE_READ_TOKEN_FILE', envString('READ_TOKEN_FILE', '/tokens/read.json'));
const writeTokenFile = envString('FAKE_WRITE_TOKEN_FILE', envString('WRITE_TOKEN_FILE', '/app/tokens/alg.json'));
const seedLimitDesired = envInt('FAKE_SEED_LIMIT', 50_000, { min: 50, max: 200_000 });
const seedPageLimit = envInt('FAKE_SEED_PAGE_LIMIT', 5000, { min: 50, max: 5000 });
const seedMaxPages = envInt(
'FAKE_SEED_MAX_PAGES',
Math.ceil(seedLimitDesired / seedPageLimit) + 2,
{ min: 1, max: 200 }
);
const seedSourceRaw = process.env.FAKE_SEED_SOURCE;
const seedSource = seedSourceRaw == null ? source : String(seedSourceRaw).trim();
const minBlock = envInt('FAKE_BLOCK_MIN', 120, { min: 1, max: 50_000 });
const maxBlock = envInt('FAKE_BLOCK_MAX', 1200, { min: 1, max: 50_000 });
const volScale = envNumber('FAKE_VOL_SCALE', 1);
const noiseScale = envNumber('FAKE_NOISE_SCALE', 0.05);
const meanReversion = envNumber('FAKE_MEAN_REVERSION', 0.0002);
const markNoiseBps = envNumber('FAKE_MARK_NOISE_BPS', 5);
const logEvery = envInt('FAKE_LOG_EVERY', 30, { min: 1, max: 10_000 });
const marketIndexEnv = process.env.MARKET_INDEX;
const marketIndexFallback = Number.isInteger(Number(marketIndexEnv)) ? Number(marketIndexEnv) : 0;
const startPriceEnv = envNumber('FAKE_START_PRICE', 0);
const clampEnabled = isTruthy(process.env.FAKE_CLAMP ?? '1');
const clampQLow = clamp(envNumber('FAKE_CLAMP_Q_LOW', 0.05), 0.0, 0.49);
const clampQHigh = clamp(envNumber('FAKE_CLAMP_Q_HIGH', 0.95), 0.51, 1.0);
const clampLowMult = envNumber('FAKE_CLAMP_LOW_MULT', 0.8);
const clampHighMult = envNumber('FAKE_CLAMP_HIGH_MULT', 1.2);
const readToken = readTokenFromFile(readTokenFile);
const writeToken = readTokenFromFile(writeTokenFile);
if (!writeToken) throw new Error(`Missing write token (expected JSON token at ${writeTokenFile})`);
let seedTicks = [];
let seedPrices = [];
let marketIndex = marketIndexFallback;
let seedFromTs = null;
let seedToTs = null;
if (!readToken) {
console.warn(`[fake-ingestor] No read token at ${readTokenFile}; running without seed data.`);
} else {
seedTicks = await fetchSeedTicksPaged({
apiBase,
readToken,
symbol,
source: seedSource ? seedSource : undefined,
desiredLimit: seedLimitDesired,
pageLimit: seedPageLimit,
maxPages: seedMaxPages,
});
seedFromTs = seedTicks.length ? String(seedTicks[0]?.ts || '') : null;
seedToTs = seedTicks.length ? String(seedTicks[seedTicks.length - 1]?.ts || '') : null;
for (const t of seedTicks) {
const p = parsePriceFromTick(t);
if (p != null) seedPrices.push(p);
}
const lastTick = seedTicks.length ? seedTicks[seedTicks.length - 1] : null;
const mi = lastTick?.market_index ?? lastTick?.marketIndex;
if (Number.isInteger(Number(mi))) marketIndex = Number(mi);
}
if (!seedPrices.length && startPriceEnv > 0) {
seedPrices = [startPriceEnv];
}
if (!seedPrices.length) {
throw new Error(
'No seed prices available. Provide FAKE_START_PRICE (and optionally MARKET_INDEX) or mount a read token to seed from /v1/ticks.'
);
}
const returnsRaw = computeLogReturns(seedPrices);
const mu = mean(returnsRaw);
const sigma = stddev(returnsRaw);
const center = envString('FAKE_CENTER_RETURNS', '1') !== '0';
const returns = returnsRaw.length
? center
? returnsRaw.map((r) => r - mu)
: returnsRaw
: [];
const sampler = new BlockSampler(returns.length ? returns : [0], { minBlock, maxBlock });
const pLow = quantile(seedPrices, clampQLow);
const p50 = quantile(seedPrices, 0.5);
const pHigh = quantile(seedPrices, clampQHigh);
const clampMin = pLow > 0 ? pLow * clampLowMult : Math.min(...seedPrices) * clampLowMult;
const clampMax = pHigh > 0 ? pHigh * clampHighMult : Math.max(...seedPrices) * clampHighMult;
let logPrice = Math.log(seedPrices[seedPrices.length - 1]);
const targetLog = Math.log(p50 > 0 ? p50 : seedPrices[seedPrices.length - 1]);
console.log(
JSON.stringify(
{
service: 'trade-fake-ingestor',
apiBase,
symbol,
source,
intervalMs,
marketIndex,
seed: {
ok: Boolean(seedTicks.length),
desiredLimit: seedLimitDesired,
pageLimit: seedPageLimit,
maxPages: seedMaxPages,
source: seedSource || null,
ticks: seedTicks.length,
prices: seedPrices.length,
fromTs: seedFromTs,
toTs: seedToTs,
},
model: {
type: 'block-bootstrap-returns',
centerReturns: center,
mu,
sigma,
volScale,
noiseScale,
meanReversion,
block: { minBlock, maxBlock },
clamp: clampEnabled ? { min: clampMin, max: clampMax } : { disabled: true },
},
},
null,
2
)
);
let stopping = false;
process.on('SIGINT', () => {
stopping = true;
});
process.on('SIGTERM', () => {
stopping = true;
});
let tickCount = 0;
let nextAt = Date.now();
while (!stopping) {
const now = Date.now();
if (now < nextAt) await sleep(nextAt - now);
nextAt += intervalMs;
const r = sampler.next();
const noise = (Number.isFinite(sigma) ? sigma : 0) * noiseScale * randn();
const revert = Number.isFinite(meanReversion) ? meanReversion * (targetLog - logPrice) : 0;
const step = (Number.isFinite(r) ? r : 0) * volScale + noise + revert;
logPrice += step;
let oraclePrice = Math.exp(logPrice);
if (clampEnabled && Number.isFinite(clampMin) && Number.isFinite(clampMax) && clampMax > clampMin) {
oraclePrice = clamp(oraclePrice, clampMin, clampMax);
logPrice = Math.log(oraclePrice);
}
const markNoise = (markNoiseBps / 10_000) * randn();
const markPrice = oraclePrice * (1 + markNoise);
const ts = new Date().toISOString();
const tick = {
ts,
market_index: marketIndex,
symbol,
oracle_price: formatNumeric(oraclePrice),
mark_price: formatNumeric(markPrice),
source,
raw: {
fake: true,
model: 'block-bootstrap-returns',
seeded: seedTicks.length
? {
symbol,
source: seedSource || null,
desiredLimit: seedLimitDesired,
pageLimit: seedPageLimit,
maxPages: seedMaxPages,
fromTs: seedFromTs,
toTs: seedToTs,
}
: {
symbol,
source: null,
desiredLimit: 0,
pageLimit: seedPageLimit,
maxPages: seedMaxPages,
fromTs: null,
toTs: null,
},
},
};
try {
await ingestTick({ apiBase, writeToken, tick });
tickCount += 1;
if (tickCount % logEvery === 0) {
console.log(
`[fake-ingestor] ok ticks=${tickCount} ts=${ts} px=${tick.oracle_price} mark=${tick.mark_price} clamp=[${formatNumeric(
clampMin
)},${formatNumeric(clampMax)}]`
);
}
} catch (err) {
console.warn(`[fake-ingestor] ingest failed: ${String(err?.message || err)}`);
await sleep(Math.min(5000, Math.max(250, intervalMs)));
}
}
console.log('[fake-ingestor] stopped');
}
main().catch((err) => {
console.error(String(err?.stack || err));
process.exitCode = 1;
});

81
kustomize/overlays/staging/frontend-server.mjs
View File

@@ -16,8 +16,6 @@ const STARTED_AT = new Date().toISOString();
const STATIC_DIR = process.env.STATIC_DIR || '/srv';
const BASIC_AUTH_FILE = process.env.BASIC_AUTH_FILE || '/tokens/frontend.json';
const API_READ_TOKEN_FILE = process.env.API_READ_TOKEN_FILE || '/tokens/read.json';
const API_UPSTREAM = process.env.API_UPSTREAM || process.env.API_URL || 'http://api:8787';
const HASURA_UPSTREAM = process.env.HASURA_UPSTREAM || 'http://hasura:8080';
const HASURA_GRAPHQL_PATH = process.env.HASURA_GRAPHQL_PATH || '/v1/graphql';
const GRAPHQL_CORS_ORIGIN = process.env.GRAPHQL_CORS_ORIGIN || process.env.CORS_ORIGIN || '*';
@@ -70,13 +68,6 @@ function loadBasicAuth() {
return { username, password };
}
function loadApiReadToken() {
const j = readJson(API_READ_TOKEN_FILE);
const token = (j?.token || '').toString();
if (!token) throw new Error(`Invalid API_READ_TOKEN_FILE: ${API_READ_TOKEN_FILE}`);
return token;
}
function send(res, status, headers, body) {
res.statusCode = status;
for (const [k, v] of Object.entries(headers || {})) res.setHeader(k, v);
@@ -439,56 +430,6 @@ function readBody(req, limitBytes = 1024 * 16) {
});
}
function proxyApi(req, res, apiReadToken) {
const upstreamBase = new URL(API_UPSTREAM);
const inUrl = new URL(req.url || '/', `http://${req.headers.host || 'localhost'}`);
const prefix = '/api';
const strippedPath = inUrl.pathname === prefix ? '/' : inUrl.pathname.startsWith(prefix + '/') ? inUrl.pathname.slice(prefix.length) : null;
if (strippedPath == null) {
send(res, 404, { 'content-type': 'text/plain; charset=utf-8' }, 'not_found');
return;
}
const target = new URL(upstreamBase.toString());
target.pathname = strippedPath || '/';
target.search = inUrl.search;
const isHttps = target.protocol === 'https:';
const lib = isHttps ? https : http;
const headers = stripHopByHopHeaders(req.headers);
delete headers.authorization; // basic auth from client must not leak upstream
headers.host = target.host;
headers.authorization = `Bearer ${apiReadToken}`;
const upstreamReq = lib.request(
{
protocol: target.protocol,
hostname: target.hostname,
port: target.port || (isHttps ? 443 : 80),
method: req.method,
path: target.pathname + target.search,
headers,
},
(upstreamRes) => {
const outHeaders = stripHopByHopHeaders(upstreamRes.headers);
res.writeHead(upstreamRes.statusCode || 502, outHeaders);
upstreamRes.pipe(res);
}
);
upstreamReq.on('error', (err) => {
if (!res.headersSent) {
send(res, 502, { 'content-type': 'text/plain; charset=utf-8' }, `bad_gateway: ${err?.message || err}`);
} else {
res.destroy();
}
});
req.pipe(upstreamReq);
}
function withCors(res) {
res.setHeader('access-control-allow-origin', GRAPHQL_CORS_ORIGIN);
res.setHeader('access-control-allow-methods', 'GET,POST,OPTIONS');
@@ -734,26 +675,6 @@ async function handler(req, res) {
}
}
if (req.url?.startsWith('/api') && (req.url === '/api' || req.url.startsWith('/api/'))) {
if (AUTH_MODE !== 'off' && AUTH_MODE !== 'none' && AUTH_MODE !== 'disabled') {
const user = resolveAuthenticatedUser(req);
if (!user) {
unauthorized(res);
return;
}
}
let token;
try {
token = loadApiReadToken();
} catch (e) {
send(res, 500, { 'content-type': 'text/plain; charset=utf-8' }, String(e?.message || e));
return;
}
proxyApi(req, res, token);
return;
}
serveStatic(req, res);
}
@@ -797,11 +718,9 @@ server.listen(PORT, () => {
service: 'trade-frontend',
port: PORT,
staticDir: STATIC_DIR,
apiUpstream: API_UPSTREAM,
hasuraUpstream: HASURA_UPSTREAM,
basicAuthFile: BASIC_AUTH_FILE,
basicAuthMode: BASIC_AUTH_MODE,
apiReadTokenFile: API_READ_TOKEN_FILE,
authUserHeader: AUTH_USER_HEADER,
authMode: AUTH_MODE,
htpasswdFile: HTPASSWD_FILE,

36
kustomize/overlays/staging/ingestor-dlob-patch.yaml
View File

@@ -1,36 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: trade-ingestor
spec:
template:
spec:
containers:
- name: ingestor
image: node:20-slim
imagePullPolicy: IfNotPresent
env:
- name: HASURA_GRAPHQL_URL
value: http://hasura:8080/v1/graphql
- name: HASURA_ADMIN_SECRET
valueFrom:
secretKeyRef:
name: trade-hasura
key: HASURA_GRAPHQL_ADMIN_SECRET
- name: DLOB_MARKETS
value: SOL-PERP,DOGE-PERP,JUP-PERP
- name: TICKS_POLL_MS
value: "1000"
- name: TICKS_SOURCE
value: "dlob_stats"
command: ["node"]
args: ["/opt/dlob/dlob-ingestor.mjs"]
volumeMounts:
- name: dlob-script
mountPath: /opt/dlob/dlob-ingestor.mjs
subPath: dlob-ingestor.mjs
readOnly: true
volumes:
- name: dlob-script
configMap:
name: trade-dlob-ingestor-script

54
kustomize/overlays/staging/ingestor-fake-patch.yaml
View File

@@ -1,54 +0,0 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: trade-ingestor
spec:
template:
spec:
containers:
- name: ingestor
env:
- name: FAKE_READ_TOKEN_FILE
value: "/tokens/read.json"
- name: FAKE_WRITE_TOKEN_FILE
value: "/app/tokens/alg.json"
- name: FAKE_SEED_LIMIT
value: "50000"
- name: FAKE_SEED_PAGE_LIMIT
value: "5000"
- name: FAKE_SEED_MAX_PAGES
value: "20"
- name: FAKE_BLOCK_MIN
value: "300"
- name: FAKE_BLOCK_MAX
value: "1800"
- name: FAKE_VOL_SCALE
value: "1.8"
- name: FAKE_NOISE_SCALE
value: "0.03"
- name: FAKE_MEAN_REVERSION
value: "0.0001"
- name: FAKE_CLAMP_Q_LOW
value: "0.02"
- name: FAKE_CLAMP_Q_HIGH
value: "0.98"
- name: FAKE_CLAMP_LOW_MULT
value: "0.7"
- name: FAKE_CLAMP_HIGH_MULT
value: "1.3"
command: ["node"]
args: ["/opt/fake/fake-ingestor.mjs"]
volumeMounts:
- name: fake-script
mountPath: /opt/fake
readOnly: true
- name: read-tokens
mountPath: /tokens
readOnly: true
volumes:
- name: fake-script
configMap:
name: trade-fake-ingestor-script
- name: read-tokens
secret:
secretName: trade-frontend-tokens

4
kustomize/overlays/staging/kustomization.yaml
View File

@@ -13,12 +13,8 @@ patchesStrategicMerge:
- hasura-patch.yaml
- frontend-auth-patch.yaml
- frontend-graphql-proxy-patch.yaml
- ingestor-dlob-patch.yaml
configMapGenerator:
- name: trade-dlob-ingestor-script
files:
- dlob-ingestor.mjs
- name: trade-frontend-server-script
files:
- frontend-server.mjs