Features

Works with any web3-compatible node

As long as you have a web3-compatible node with an accessible JSON-RPC endpoint, you're good to go. It even works with Infura's API.

Automatic backfilling

No matter if the tool has been offline for a period of time or it is the first time it starts, if the backfilling feature is enabled, it will automatically take care of queueing the necessary blocks for scraping.

• if the feature is enabled, when the process first starts, it will add all the blocks in the [0, best block] interval into the todo list

• if the process is stopped for a while, it will automatically add all the blocks in the [highest DB block, best block] interval into the todo list

• since the redis list used for task management is a sorted set, the highest blocks always have priority - this means new blocks will be processed as soon as they appear while the backfilling will be happening in the otherwise-idle time

Chain reorganisations (reorg) handling

Whenever a reorg is detected we check the db for a block with the same number but different hash and replace it with the newest version. Note: this depends on the behavior of the node to which Memento is connected and it doesn't guarantee the data is 100% reorg-proof.

Lag function

If you don't want to deal with reorgs, enable the lag function, set a value (default 10 blocks) and enjoy your reorg-free data.

• when this function is enabled, Memento will wait a number of blocks (or confirmations) before scraping and indexing a specific block

• for example, if the lag feature is enabled and the value is set to value: 10, Memento will only scrape the block 100 when the best block in the network is higher than 110

Data validator

The tool includes a validator that checks if the data makes sense from a logical integrity standpoint for every block scraped. This avoids any inconsistencies that may occur throughout the scraping process.

Automatic migrations

If the feature is enabled, whenever the run function is called, it will automatically try to execute the database migrations. No need to worry when doing an upgrade.