sails-indexer

What this skill does

# Sails Indexer

## Role

Design and implement the read-side backend for a standard Gear/Vara Sails app.

Use this skill when the frontend or integration layer needs history, pagination, filtering, search, aggregates, charts, timelines, or a fast read API that should not be rebuilt from direct chain queries on every screen load.

Treat the indexer as a projection pipeline:

`chain/archive -> Sails event decode -> optional on-chain enrichment -> Postgres read model -> thin GraphQL API`

## Local Handbook

- `../../references/gear-execution-model.md`
- `../../references/gear-messaging-and-replies.md`
- `../../references/sails-program-and-service-architecture.md`
- `../../references/sails-idl-client-pipeline.md`
- `../../references/sails-cheatsheet.md`
- `../../references/sails-gtest-and-local-validation.md`
- `../../references/scale-binary-decoding-guide.md`
- `../../references/contract-interface-evolution.md`
- `../../references/sails-indexer-patterns.md`
- `../../references/sails-idl-v2-syntax.md` — IDL v2 event syntax (`throws`, `@query`, service-scoped types)

## Standard Defaults

- Default to an IDL-driven read-side indexer, not a generic command-side backend.
- Treat the program `.idl` as the event and route contract for normal Sails decoding work.
- Default stack: Subsquid-style archive ingestion, PostgreSQL read model, ORM-backed entities, and a thin GraphQL API.
- Always expose GraphQL as the default read surface for frontend and integration consumers. Do not downgrade the default to REST.
- Mount a real GraphQL endpoint such as `/graphql` and make it reachable from local frontend development without browser CORS failures.
- Prefer one of two frontend-safe API exposure strategies and name the choice explicitly: enable CORS on the indexer API, or serve GraphQL through the frontend dev proxy or same-origin gateway.
- Use a real ingestion adapter wired to chain or archive sources. Do not leave placeholder, null, or demo adapters in the runtime.
- Prefer event-driven projections first. Use direct on-chain queries only for initial entity bootstrap or explicit source-of-truth confirmation, and keep them out of hot event-processing paths unless the design justifies the cost.
- Keep projection logic in processor or handler services. Keep the API layer thin.
- Model restart, replay, backfill, and duplicate safety as first-class requirements, not as later hardening.
- Prefer deterministic entity IDs derived from chain facts such as program ID, message ID, block number, extrinsic position, or explicit domain keys.
- If the repo already has an indexer stack, extend it instead of replacing it with a new framework casually.
- Treat PostGraphile as a v4-compatible template constraint, not as a product requirement for every indexer. The bundled API template uses the v4 bootstrap/config API (`postgraphile(...)`, `PostGraphileOptions`, `appendPlugins`), so keep `postgraphile` on the 4.x line and `postgraphile-plugin-connection-filter` on the 2.x line when copying these assets. Do not switch to v5 by changing package versions alone; update the API bootstrap and template config together.
- Follow the canonical runtime order from `../../references/sails-indexer-patterns.md`, especially `Start with configuration, not hardcoded endpoints`, `Build one shared batch processor and export its derived types`, `Centralize all IDL decoding behind one decoder`, `Wire the runtime in main.ts, but do not place projection logic there`, `Give every handler the same lifecycle contract`, `Process a batch in a stable order`, `Save in groups and only write what changed`, and `Expose a thin API layer on top of PostgreSQL`.

## Route Here When

- the builder needs historical views, activity feeds, portfolio pages, or timelines
- the frontend needs pagination, filtering, sorting, or joins that are awkward or expensive through direct chain queries
- the project needs aggregated metrics, snapshots, counters, rankings, charts, or search
- the app has many dynamic child programs that cannot be enumerated upfront and whose events must be indexed
- the builder needs a query API backed by projected data rather than live state reads only
- the work requires event decoding from `.idl` and mapping those events into off-chain entities

## Do Not Route Here When

- the task is really a command-side backend, auth service, session service, cron worker, or general Node API
- the task only needs a few direct Sails queries from frontend or scripts and no persistent read model
- the project is Vara.eth or ethexe-first rather than standard Gear/Vara Sails
- the main problem is contract architecture, not read-side projection design
- the request is purely about frontend wiring without a new read model

## Required Input Contract

Before implementation, make these inputs explicit:

- target programs to index: fixed IDs, discovery source, or hybrid
- `.idl` files or generated decode artifacts for each indexed program
- `fromBlock` or backfill start rule
- list of events, services, and payloads that matter
- required read entities and the UI or integration surfaces that consume them
- enrichment needs: event-only, event-plus-query, or periodic derived updates
- GraphQL endpoint contract: route, local development origin policy, and whether frontend reaches it through direct CORS or dev proxy
- restart, replay, and cutover constraints for existing deployments

If any of these are unknown, stop and write the missing assumptions explicitly before coding.

Cross-check against `../../references/sails-indexer-patterns.md`:

- `Start with configuration, not hardcoded endpoints`, `Build one shared batch processor and export its derived types`, and `Create a typed boundary for Gear events before touching business payloads` for config, processor boundary, and typed Gear event boundary
- `Centralize all IDL decoding behind one decoder` for decoder setup and query encoding or decoding
- `Keep the SQL schema read-model oriented` before freezing the SQL or ORM schema

## Quick Start

Use this cold-start path when the repo does not already contain a working indexer scaffold.

Templates live in `skills/sails-indexer/assets/`:

- `package.json`
- `tsconfig.json`
- `docker-compose.yml`
- `.env.example`

Cold-start defaults:

- Node.js 20+
- PostgreSQL available locally (prefer a local service install; Docker is an alternative if a local service is not available)
- `reflect-metadata` imported once at the top of each runtime entrypoint before TypeORM entities or data source code is loaded
- `experimentalDecorators: true` and `emitDecoratorMetadata: true` enabled in `tsconfig.json`
- separate Subsquid gateway URL (`VARA_ARCHIVE_URL`) and Vara archive RPC URL (`VARA_RPC_URL`); Subsquid performs historical queries that require an archive node on the RPC side, so do not point `VARA_RPC_URL` at a live non-archive endpoint

Suggested first-run order:

1. Copy the files from `assets/` into the new indexer repo.
2. Fill `.env` from `.env.example`.
3. Install dependencies with `npm install`.
4. Start PostgreSQL (local service, or `docker compose up -d` if using Docker).
5. Generate TypeORM entities from `schema.graphql` with `npm run codegen`.
6. Build once with `npm run build`.
7. Generate a migration with `npm run db:generate`.
8. Apply the migration with `npm run db:apply`.
9. Start the processor with `npm run dev:processor`.
10. Start the API with `npm run dev:api`.
11. Verify that `/graphql` is reachable through the chosen local access path.

For schema updates, use this order:

1. update `schema.graphql`
2. `npm run codegen`
3. `npm run build`
4. `npm run db:generate`
5. `npm run db:apply`

For fixed-program projects, `.env` can stay minimal with one `VARA_PROGRAM_ID` and one `VARA_IDL_PATH`.
For discovery-driven projects, replace that pair with the explicit root IDs that serve as discovery sources — `REGISTRY_PROGRAM_ID`, `FACTORY_PROGRAM_ID`, or other domain-specific anchors — and keep discovery logic separate from projection