trade-iac/doc/step-002-plan-solana-rpc-mainnet-private.txt

Step 002: Plan wdrozenia Solana RPC (Agave mainnet, non-voting, private RPC + Geyser->Postgres + monitoring + NOC tmux)

Zalozenia (default):
- WG iface: wg0, WG subnet: 10.66.66.0/24, WG IP baremetala: 10.66.66.1
- Public iface: eth0, public IP: 149.50.116.219
- RPC: 8899 (HTTP), WS: 8900 (WS = RPC_PORT+1)
- Postgres (lokalnie): 5432 (DB: geyser, user: geyser)
- (Optional) Yellowstone gRPC: 10000 (tylko po WG)
- Prometheus: 9090, Grafana: 3000, node_exporter: 9100 (tylko po WG)
- Dynamic port range (cluster): 8000-8020/udp (public eth0)
- User: solana
- Dirs:
  - /var/lib/solana (ledger/accounts)
  - /etc/solana (configs)
  - /var/log/solana (logs)

0) Preconditions (check)
  # WG dziala
  ip -4 a show wg0
  wg show

  # SSH po WG dziala (z laptop/VPS):
  #   ssh user@10.66.66.1

1) Packages
  sudo apt-get update
  sudo apt-get install -y \
    tmux btop \
    sysstat iotop \
    ethtool nstat \
    chrony curl jq git ca-certificates \
    ripgrep \
    smartmontools nvme-cli \
    build-essential pkg-config libssl-dev \
    clang llvm-dev libclang-dev \
    cmake protobuf-compiler libudev-dev zlib1g-dev

2) Users/Dirs
  sudo getent group solana >/dev/null || sudo groupadd --system solana
  sudo id -u solana >/dev/null 2>&1 || sudo useradd --system --gid solana \
    --home-dir /var/lib/solana --create-home --shell /bin/bash solana

  sudo install -d -o root -g root -m 0755 /etc/solana /opt/solana/bin
  sudo install -d -o solana -g solana -m 0750 \
    /var/lib/solana /var/lib/solana/ledger /var/lib/solana/accounts /var/log/solana

  # identity permissions (po utworzeniu pliku):
  #   sudo chown solana:solana /var/lib/solana/identity.json
  #   sudo chmod 600 /var/lib/solana/identity.json

2A) Storage (rekomendowane)
  # Variant A: jesli /dev/nvme0n1 jest wolny i chcesz go podzielic 50/50:
  #   doc/step-002a-storage-nvme0n1-xfs.txt
  #
  # Variant B (mpabi): masz juz 2x NVMe z XFS:
  #   - SOLACCT na accounts (nvme0n1p1)
  #   - SOLLEDG na ledger (nvme1n1p1)
  #   doc/step-002a-storage-existing-solacct-solledg-xfs.txt

3) Firewall/WG binding (private-only)
  # UFW: deny everything inbound by default
  sudo ufw default deny incoming
  sudo ufw default allow outgoing

  # WireGuard (public)
  sudo ufw allow 51820/udp

  # SSH only via WG
  sudo ufw allow in on wg0 to any port 22 proto tcp
  sudo ufw deny  in on eth0 to any port 22 proto tcp

  # RPC/WS only via WG
  sudo ufw allow in on wg0 to any port 8899 proto tcp
  sudo ufw allow in on wg0 to any port 8900 proto tcp
  sudo ufw deny  in on eth0 to any port 8899 proto tcp
  sudo ufw deny  in on eth0 to any port 8900 proto tcp

  # Yellowstone gRPC only via WG
  sudo ufw allow in on wg0 to any port 10000 proto tcp
  sudo ufw deny  in on eth0 to any port 10000 proto tcp

  # Monitoring only via WG
  sudo ufw allow in on wg0 to any port 9100 proto tcp
  sudo ufw allow in on wg0 to any port 9090 proto tcp
  sudo ufw allow in on wg0 to any port 3000 proto tcp
  sudo ufw deny  in on eth0 to any port 9100 proto tcp
  sudo ufw deny  in on eth0 to any port 9090 proto tcp
  sudo ufw deny  in on eth0 to any port 3000 proto tcp

  # Cluster ports (public, inbound) for sync (UDP)
  sudo ufw allow in on eth0 to any port 8000:8020 proto udp
  # (opcjonalnie) jesli bedziesz mial problemy z reachability:
  # sudo ufw allow in on eth0 to any port 8000:8020 proto tcp

  sudo ufw enable
  sudo ufw status verbose

4) Validator (Agave) config + systemd (non-voting RPC-only)
  # sysctl baseline (Agave startup checks)
  sudo tee /etc/sysctl.d/90-agave.conf >/dev/null <<'EOF'
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
EOF
  sudo sysctl --system

  # ulimit baseline
  sudo tee /etc/security/limits.d/90-solana.conf >/dev/null <<'EOF'
* soft nofile 1000000
* hard nofile 1000000
EOF

  # Agave tools (solana-keygen etc.) for solana user
  sudo -u solana -H bash -lc 'sh -c "$(curl -sSfL https://release.anza.xyz/stable/install)"'

  # Copy binaries to /opt/solana/bin (avoid symlink permission gotchas)
  sudo cp -a /var/lib/solana/.local/share/solana/install/active_release/bin/. /opt/solana/bin/
  sudo find /opt/solana/bin -maxdepth 1 -type f -exec chmod 0755 {} \;

  /opt/solana/bin/solana --version || true
  /opt/solana/bin/solana-keygen --version

  # Build agave-validator (pin tag + toolchain)
  # Uwaga: dla nowszych stable release'ow binarka moze nie byc publikowana,
  # wiec build ze zrodel jest wymagany.
  #   doc/step-002f-agave-build-from-source-latest.txt

  # Identity (create once)
  if [ ! -f /var/lib/solana/identity.json ]; then
    sudo -u solana -H /opt/solana/bin/solana-keygen new --no-passphrase -o /var/lib/solana/identity.json
    sudo chown solana:solana /var/lib/solana/identity.json
    sudo chmod 600 /var/lib/solana/identity.json
  fi

  # Genesis hash (mainnet)
  GENESIS_HASH="$(curl -sS -H 'content-type: application/json' \
    --data '{"jsonrpc":"2.0","id":1,"method":"getGenesisHash"}' \
    https://api.mainnet-beta.solana.com | jq -r .result)"
  echo "$GENESIS_HASH"
  # Oczekiwane (mainnet): 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d

  # systemd unit (bind RPC to WG/localhost only)
  # UWAGA: w unit file NIE uzywaj `\\` na koncu linii w ExecStart.
  # Jesli wpiszesz `\\`, agave-validator dostanie literalny argument `\` i padnie z:
  #   "error: The subcommand '\\' wasn't recognized"
  #
  # Najprosciej: ExecStart w 1 linii, bez "kontynuacji" backslashami.
  # Ale: zeby uniknac problemow z wklejaniem/lamaniem linii, ponizej jest bezpieczna wersja wielolinijkowa.
  sudo tee /etc/systemd/system/agave-validator.service >/dev/null <<'EOF'
[Unit]
Description=Agave validator (mainnet, RPC-only, non-voting)
After=network-online.target wg-quick@wg0.service
Wants=network-online.target wg-quick@wg0.service

[Service]
Type=simple
User=solana
Group=solana
WorkingDirectory=/var/lib/solana
Environment=RUST_LOG=info
LimitNOFILE=1048576
Restart=always
RestartSec=5
TimeoutStopSec=120
ExecStart=/opt/solana/bin/agave-validator \
  --identity /var/lib/solana/identity.json \
  --no-voting \
  --private-rpc \
  --ledger /var/lib/solana/ledger \
  --accounts /var/lib/solana/accounts \
  --log /var/log/solana/validator.log \
  --bind-address 0.0.0.0 \
  --rpc-bind-address 10.66.66.1 \
  --rpc-port 8899 \
  --dynamic-port-range 8000-8020 \
  --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
  --expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
  --full-rpc-api \
  --limit-ledger-size 50000000
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ReadWritePaths=/var/lib/solana/ledger /var/lib/solana/accounts /var/log/solana /var/lib/solana

[Install]
WantedBy=multi-user.target
EOF

  sudo systemctl daemon-reload
  sudo systemctl enable --now agave-validator

  systemctl is-active agave-validator
  journalctl -u agave-validator -n 80 --no-pager

  # Jesli padnie na genesis (brak genesis.tar.bz2):
  #   doc/step-002c-genesis-download-mainnet.txt
  #
  # Jesli padnie na snapshot (mainnet wymaga snapshot):
  #   doc/step-002d-snapshot-download-mainnet.txt
  #
  # Jesli chcesz najpierw odpalic validator z CLI (test), zanim przepniesz na systemd:
  #   doc/step-002e-agave-validator-run-cli.txt

5) Geyser plugin -> Postgres
  # Krok po kroku:
  #   doc/step-003-geyser-plugin-postgres.txt
  #
  # W skrocie:
  # - budujesz plugin .so i wrzucasz do /opt/solana/plugins/
  # - tworzysz /etc/solana/geyser-postgres.json
  # - dopisujesz do ExecStart:
  #     --geyser-plugin-config /etc/solana/geyser-postgres.json

6) Drift DLOB (2 markets) (on VPS, not baremetal) (placeholder)
  # VPS env:
  #   PERP_MARKETS_TO_LOAD=0,75
  #   USE_GRPC=true
  #   GRPC_ENDPOINT=http://10.66.66.1:10000

7) Monitoring (bind to WG only) (placeholder)
  # node_exporter (prefer bind to WG):
  #   sudo apt-get install -y prometheus-node-exporter
  #   sudo systemctl enable --now prometheus-node-exporter
  #
  # Prometheus/Grafana (bind to 10.66.66.1):
  #   sudo apt-get install -y prometheus grafana
  #   sudo systemctl enable --now prometheus grafana-server

8) NOC tmux (btop) (placeholder)
  # Create tmux session:
  #   btop
  #   iostat -x 1
  #   vmstat 1
  #   nstat -az
  #   ethtool -S eth0 | head -n 40 (wrap in watch)
  #   journalctl -fu agave-validator
  #   while true; do curl getHealth/getSlot; sleep 1; done

9) Troubleshooting checklist
  # Services:
  systemctl status agave-validator --no-pager
  journalctl -u agave-validator -n 200 --no-pager

  # Sockets and binds:
  ss -lntp | egrep ':22\\b|:8899\\b|:8900\\b|:10000\\b|:9100\\b|:9090\\b|:3000\\b' || true

  # Firewall:
  sudo ufw status verbose

  # RPC local:
  curl -sS -H 'content-type: application/json' \
    --data '{"jsonrpc":"2.0","id":1,"method":"getHealth"}' \
    http://127.0.0.1:8899 | jq .

5-min quick verification checklist
  wg show
  sudo ufw status verbose
  systemctl is-active agave-validator
  ss -lntp | egrep ':8899\\b|:8900\\b|:10000\\b|:9090\\b|:3000\\b|:9100\\b' || true
  curl -sS -H 'content-type: application/json' \
    --data '{"jsonrpc":"2.0","id":1,"method":"getSlot"}' \
    http://10.66.66.1:8899 | jq .