Runbook (mpabi): Agave RPC (mainnet, private) + Geyser plugin -> Postgres

Cel:
- Na mpabi odpalic prywatny Agave RPC (non-voting) po WireGuard (wg0).
- Zbudowac Agave ze zrodel (pinned release).
- (Po starcie) dodac Geyser plugin zapisujacy do PostgreSQL.

Skrot (kolejnosc):
1) Preconditions: WG + UFW (SSH tylko po wg0).
2) Storage: SOLACCT->/var/lib/solana/accounts, SOLLEDG->/var/lib/solana/ledger (+ decyzja: wipe czy kontynuacja).
3) Pakiety + sysctl/limits.
4) /opt/solana/bin: Anza tools (solana-keygen itd.).
5) Build Agave ze zrodel (pin tag) + install do /opt/solana/bin/agave-validator.
6) Identity (0600).
7) Genesis + snapshot (mainnet).
8) Systemd: agave-validator (User=solana) + RPC smoke test po WG.
9) Geyser plugin -> Postgres + dopiecie do systemd + weryfikacja.

Konwencja:
- Komendy uruchamiasz jako user: `user` (z sudo).
- Validator dziala jako uzytkownik systemowy: `solana`.
- Biny instalujemy do /opt/solana/bin (root-owned, stabilne sciezki).
- Dane runtime:
  - ledger:   /var/lib/solana/ledger   (SOLLEDG = /dev/nvme1n1p1)
  - accounts: /var/lib/solana/accounts (SOLACCT = /dev/nvme0n1p1)

Parametry hosta (mpabi):
- Public iface: enp6s0, public IP: 149.50.116.219
- WireGuard iface: wg0, WG IP: 10.66.66.1
- RPC: 8899, WS: 8900
- Cluster inbound UDP: 8000-8020 (na public)

=====================
0) Preconditions
=====================
1) WireGuard up:
   ip -4 a show wg0
   wg show

2) SSH powinno byc tylko po wg0 (defense-in-depth):
   sudo ufw status verbose

3) Upewnij sie, ze masz sudo do operacji operatorskich (systemctl/journalctl/ufw).

=====================
1) Storage: mount SOLACCT/SOLLEDG
=====================
UWAGA: ten wariant NIE formatuje dyskow (bez kasowania), tylko montuje istniejace XFS.

UUIDs (wg lsblk -f):
- SOLACCT: 055bf4e7-00f0-45a0-a995-2413b14428c4  (/dev/nvme0n1p1)
- SOLLEDG: 6ac7a8c6-efc7-4803-a3dc-a66ea2ef0125  (/dev/nvme1n1p1)

1) (Opcjonalnie) obejrzyj co zajmuje SOLLEDG (zanim podepniesz pod /var/lib/solana/ledger):
   sudo mkdir -p /mnt/solledg
   sudo mount /dev/nvme1n1p1 /mnt/solledg
   sudo du -sh /mnt/solledg/* 2>/dev/null | sort -h | tail
   sudo umount /mnt/solledg

2) Utworz usera solana + mountpointy:
   sudo getent group solana >/dev/null || sudo groupadd --system solana
   sudo id -u solana >/dev/null 2>&1 || sudo useradd --system --gid solana \
     --home-dir /var/lib/solana --create-home --shell /bin/bash solana

   sudo install -d -o solana -g solana -m 0750 /var/lib/solana/accounts /var/lib/solana/ledger

3) Dodaj wpisy do /etc/fstab (sprawdz czy nie ma juz duplikatow):
   grep -nE '(/var/lib/solana/(accounts|ledger))' /etc/fstab || true

   UUID_ACCTS="055bf4e7-00f0-45a0-a995-2413b14428c4"
   UUID_LEDGER="6ac7a8c6-efc7-4803-a3dc-a66ea2ef0125"

   echo "UUID=$UUID_ACCTS /var/lib/solana/accounts xfs noatime,nodiratime 0 2" | sudo tee -a /etc/fstab >/dev/null
   echo "UUID=$UUID_LEDGER /var/lib/solana/ledger   xfs noatime,nodiratime 0 2" | sudo tee -a /etc/fstab >/dev/null

   sudo mount -a
   sudo systemctl daemon-reload

4) Permissions (krytyczne):
   sudo chown solana:solana /var/lib/solana/accounts /var/lib/solana/ledger
   sudo chmod 750 /var/lib/solana/accounts /var/lib/solana/ledger

5) Verify:
   df -hT /var/lib/solana/accounts /var/lib/solana/ledger
   findmnt /var/lib/solana/accounts
   findmnt /var/lib/solana/ledger
   stat -c '%U:%G %a %n' /var/lib/solana/accounts /var/lib/solana/ledger

DECYZJA: co robimy z SOLLEDG (63% used)?
- Opcja A: zostawiamy (jesli to stary ledger Solany i chcesz kontynuowac).
- Opcja B: czyscimy i robimy sync od nowa (destrukcyjne, rekomendowane dla powtarzalnosci):
  sudo systemctl stop agave-validator 2>/dev/null || true
  sudo -u solana -H bash -lc '
    cd /var/lib/solana/ledger
    rm -rf rocksdb snapshots accounts_index accounts_hash_cache admin.rpc ledger.lock
  '

=====================
2) Pakiety + sysctl/limits
=====================
1) Pakiety:
   sudo apt-get update
   sudo apt-get install -y \
     chrony curl jq git ca-certificates ripgrep \
     smartmontools nvme-cli \
     build-essential pkg-config libssl-dev \
     clang llvm-dev libclang-dev \
     cmake protobuf-compiler libudev-dev zlib1g-dev

2) Sysctl:
   sudo tee /etc/sysctl.d/90-agave.conf >/dev/null <<'EOF'
fs.file-max = 2000000
vm.max_map_count = 1000000
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 250000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
EOF
   sudo sysctl --system

3) ulimit:
   sudo tee /etc/security/limits.d/90-solana.conf >/dev/null <<'EOF'
* soft nofile 1000000
* hard nofile 1000000
EOF

=====================
3) Firewall (UFW) dla prywatnego RPC
=====================
Docelowo:
- SSH 22/tcp: tylko wg0
- RPC 8899/tcp, WS 8900/tcp: tylko wg0 (dodatkowo bind na 10.66.66.1)
- WireGuard: 51820/udp (public)
- Cluster sync: 8000-8020/udp (public)

Komendy (jesli jeszcze nie ustawione):
  sudo ufw default deny incoming
  sudo ufw default allow outgoing
  sudo ufw allow 51820/udp

  sudo ufw allow in on wg0 to any port 22 proto tcp
  sudo ufw deny  in on enp6s0 to any port 22 proto tcp

  sudo ufw allow in on wg0 to any port 8899 proto tcp
  sudo ufw allow in on wg0 to any port 8900 proto tcp
  sudo ufw deny  in on enp6s0 to any port 8899 proto tcp
  sudo ufw deny  in on enp6s0 to any port 8900 proto tcp

  sudo ufw allow in on enp6s0 to any port 8000:8020 proto udp

  sudo ufw enable
  sudo ufw status verbose

=====================
4) Instalacja narzedzi Solana/Anza do /opt/solana/bin
=====================
1) Installer (jako solana):
   sudo install -d -o root -g root -m 0755 /opt/solana/bin
   sudo -u solana -H bash -lc 'sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)"'

2) Skopiuj biny do /opt/solana/bin:
   sudo cp -a /var/lib/solana/.local/share/solana/install/active_release/bin/. /opt/solana/bin/
   sudo find /opt/solana/bin -maxdepth 1 -type f -exec chmod 0755 {} \;

3) Verify:
   /opt/solana/bin/solana --version || true
   /opt/solana/bin/solana-keygen --version

=====================
5) Build + install agave-validator (pinned latest stable)
=====================
Patrz:
- trade-iac/doc/step-002f-agave-build-from-source-latest.txt

Skrót:
1) Wybierz TAG (np. v3.0.x) i zbuduj jako solana do /var/lib/solana/src/
2) Zainstaluj binarke do /opt/solana/bin/agave-validator
3) Verify:
   /opt/solana/bin/agave-validator --version

=====================
6) Identity
=====================
1) Utworz raz:
   if [ ! -f /var/lib/solana/identity.json ]; then
     sudo -u solana -H /opt/solana/bin/solana-keygen new --no-passphrase -o /var/lib/solana/identity.json
     sudo chown solana:solana /var/lib/solana/identity.json
     sudo chmod 0600 /var/lib/solana/identity.json
   fi

2) Zapisz pubkey do notatek:
   sudo -u solana -H /opt/solana/bin/solana-keygen pubkey /var/lib/solana/identity.json

=====================
7) Genesis + snapshot (mainnet)
=====================
1) Genesis:
   patrz: trade-iac/doc/step-002c-genesis-download-mainnet.txt

2) Snapshot (wymagany na mainnet):
   patrz: trade-iac/doc/step-002d-snapshot-download-mainnet.txt

=====================
8) Systemd: agave-validator (RPC-only)
=====================
Unit template:
- trade-iac/doc/step-002b-agave-validator-systemd.txt

Wazne:
- `User=solana`
- `--rpc-bind-address 10.66.66.1`
- `--private-rpc`
- nie uzywaj `\\` w ExecStart (tylko pojedynczy `\` na koncu linii)

Start/verify:
  sudo systemctl daemon-reload
  sudo systemctl enable --now agave-validator
  systemctl is-active agave-validator
  sudo journalctl -u agave-validator -n 120 --no-pager

RPC smoke test (po WG):
  curl -sS -H 'content-type: application/json' \
    --data '{"jsonrpc":"2.0","id":1,"method":"getVersion"}' \
    http://10.66.66.1:8899 | jq .

Manual run (exports + komenda w plikach na mpabi):
  # Zawiera "solana export" (PATH + SOLANA_RPC_URL) i helper `agave_run()`:
  # - /etc/solana/agave-env.sh
  # - /etc/solana/agave-validator.args
  sudo -u solana -H bash -lc 'source /etc/solana/agave-env.sh; agave_run'

=====================
9) Geyser plugin -> PostgreSQL
=====================
Krok po kroku:
- trade-iac/doc/step-003-geyser-plugin-postgres.txt

Skrót:
1) Postgres:
   sudo apt-get install -y postgresql
   sudo systemctl enable --now postgresql

2) DB/user:
   sudo -u postgres psql -v ON_ERROR_STOP=1 <<'SQL'
   DO $$
   BEGIN
     IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'geyser') THEN
       CREATE ROLE geyser LOGIN PASSWORD 'CHANGE_ME';
     END IF;
   END $$;
   CREATE DATABASE geyser OWNER geyser;
   SQL

3) Build plugin (.so) i instaluj do /opt/solana/plugins/
4) Skopiuj example JSON config do /etc/solana/geyser-postgres.json, ustaw DSN, selektory.
5) Dopisz do unit ExecStart:
     --geyser-plugin-config /etc/solana/geyser-postgres.json
   i zrestartuj:
     sudo systemctl daemon-reload
     sudo systemctl restart agave-validator

Verify:
  sudo journalctl -u agave-validator -n 200 --no-pager | rg -i 'geyser|plugin|postgres' || true
  sudo -u postgres psql -d geyser -c '\\dt' | head