substreams-sink

Expert assistant for consuming Solana/SVM Substreams data — building production-grade sinks and data pipelines.

Core Concepts

What is a Substreams Sink?

A Substreams sink is an application that:

• Connects to a Substreams endpoint via gRPC
• Streams processed blockchain data from a Substreams package (.spkg)
• Handles cursor persistence for resumability
• Manages chain reorganizations gracefully
• Processes the data into your destination (database, queue, etc.)

Note: Before building a custom sink, consider using existing solutions:

• substreams-sink-sql — For PostgreSQL and ClickHouse. Handles cursor management, reorgs, batching, and schema management out of the box.
• substreams-sink-kv — For key-value stores.
• substreams-sink-files — For file-based outputs (JSON, CSV, Parquet).

SVM-Specific Considerations

• Solana endpoint: solana.substreams.pinax.network:443
• Output types: Solana modules output protobuf types like Events (per-DEX) or DatabaseChanges (aggregate)
• Address encoding: Solana addresses are base58-encoded (not hex)
• High throughput: Solana produces blocks every ~400ms with many transactions; sinks must handle high data volume

Authentication

export SUBSTREAMS_API_KEY="your-api-key"
# Or
export SUBSTREAMS_API_TOKEN="your-jwt-token"

Get your API key from thegraph.market or pinax.network.

References

See the references/ directory for detailed guides:

• rust-sink.md — Rust sink implementation
• cursor-reorg.md — Cursor management & reorg handling

Quick Start: SQL Sink (Recommended)

For most use cases, use substreams-sink-sql with the pre-built SVM SPKGs:

ClickHouse

substreams-sink-sql run \
  "clickhouse://default:@localhost:9000/solana" \
  "https://github.com/pinax-network/substreams-svm/releases/download/svm-dex-v0.3.1/clickhouse-svm-dex-v0.3.1.spkg" \
  db_out

PostgreSQL

substreams-sink-sql run \
  "postgres://user:pass@localhost:5432/solana?sslmode=disable" \
  "https://github.com/pinax-network/substreams-svm/releases/download/svm-dex-v0.3.1/postgres-svm-dex-v0.3.1.spkg" \
  db_out

Available SVM SPKGs

| Module | Description | Output | |--------|-------------|--------| | svm-dex | All DEX swaps (Raydium, Jupiter, Orca, PumpFun, Meteora, etc.) | DatabaseChanges | | svm-transfers | SPL token + native SOL transfers | DatabaseChanges | | svm-balances | Token balance changes | DatabaseChanges | | svm-accounts | Account creation/updates | DatabaseChanges | | svm-metadata | Metaplex token metadata | DatabaseChanges |

Each is available in ClickHouse and PostgreSQL variants.