🎛️Components

Description about components that combine to make up Datafast-Runtime

Generally, the runtime is divided into 6 main logical components plus 4 globally-available components, including

Main logical components:

Manifest

The Manifest component is responsible for loading subgraph output manifest - which can be generated by @graph-protocol/cli graph build command.

The manifest bundle of subgraph handled by Manifest includes subgraph's version, datastore schema, wasm modules etc

BlockSource

BlockSource is the client connecting to the external blockchain data store (block-store)

BlockSource polls the block-store for archived blockchain data & also new data. When using with Delta-Ingestor, BlockSource uses datafusion SQL API to download block data from DeltaLake for processing.

BlockSource also deals with data serialization - serializing the block-store format to compatible chain-based data format used in runtime's processing work.

Valve

Valve is used to regulate BlockSouce flow: when a batch of data is downloaded, the runtime needs some time to process it thus Valve will temporarily stop the BlockSource's download job until the runtime is ready to process the next batch of data.

Inspector

Inspector (or BlockInspector) 's job is to ensure the block data received by runtime is valid to process in term of continuity.

Inspector can silently reject duplicated blocks, and wait until a block with valid block-number is received. It can also panic if the data received from block-store is not in correct order or conflict with the output data found in Database

DataFilter

DataFilter component is used to filter relevant data. It pulls the indexing requirements from Manifest and determine which part of data in data batch received by BlockSource is needed for indexing.

In case of Ethereum, DataFilter filters and selects only ethereum logs/events related to the contracts defined in subgraph's manifest bundle.

Database

Being self-explanatory, Database commits result of indexing work to external database & reads relevant data from database during processing.

Database is made up of 2 parts: an in-memory caching component + an external datastore. The caching component helps reducing read/write to actual database and its data is committed to external database once the processing of a single data batch is finished.

Subgraph

Subgraph component puts pieces together: it creates & re-creates wasm-hosts using wasm modules from Manifest, processes data validated by DataFilter, sends data store operational request to Database & receives result during the work.

Global components

These global components are supporting / special components that does not directly involes in the main logic but needed in different parts of the work.

Schema

Schema is a component created by parsing the subgraph's manifest bundle. It helps dealing with data validation & with each external-database's implementation

Wasm-Runtime

Wasm-runtime is built using Wasmer. It loads an external wasm-modules from subgraph's manifest bundle and includes built-in functions for subgraph's author to interact with using AssemblyScript API

JSON-RPC Client

Used to interact with blockchain's JSON-RPC. It requires specific implementations for each type of Blockchain JSON-RPC. Currently only Ethereum is supported, but can be extended to work with different chain.