4.1.0

Migration from 4.0 (optional)

Run dipdup schema init on the existing database to enable dipdup_head_status view and REST endpoint.

big_map indexes allow achieving faster processing times than operation ones when storage updates are the only on-chain data your dapp needs to function. With this DipDup release, you can go even further and index only the current storage state, ignoring historical changes.

indexes:
  foo:
    kind: big_map
    ...
    skip_history: never|once|always

When this option is set to once, DipDup will skip historical changes only on initial sync and switch to regular indexing afterward. When the value is always, DipDup will fetch all big map keys on every restart. Preferrable mode depends on your workload.

All big map diffs DipDup pass to handlers during fast sync have action field set to BigMapAction.ADD_KEY. Keep in mind that DipDup fetches all keys in this mode, including ones removed from the big map. If needed, you can filter out the latter by BigMapDiff.data.active field.

New datasource for contract and token metadata

Since the first version DipDup allows to fetch token metadata from Better Call Dev API with bcd datasource. Now it's time for a better solution. Firstly, BCD is far from being reliable in terms of metadata indexing. Secondly, spinning up your own instance of BCD requires significant effort and computing power. Lastly, we plan to deprecate Better Call Dev API soon (but do not worry - it won't affect the explorer frontend).

Luckily, we have dipdup-metadata, a standalone companion indexer for DipDup written in Go. Configure a new datasource in the following way:

datasources:
  metadata:
    kind: metadata
    url: https://metadata.dipdup.net
    network: mainnet|ghostnet|limanet

Now you can use it anywhere in your callbacks:

datasource = ctx.datasources['metadata']
token_metadata = await datasource.get_token_metadata(address, token_id)

bcd datasource will remain available for a while, but we discourage using it for metadata processing.

Nested packages for hooks and handlers

Callback modules are no longer have to be in top-level hooks/handlers directories. Add one or multiple dots to the callback name to define nested packages:

package: indexer
hooks:
  foo.bar:
    callback: foo.bar

After running init command, you'll get the following directory tree (shortened for readability):

indexer
├── hooks
│   ├── foo
│   │   ├── bar.py
│   │   └── __init__.py
│   └── __init__.py
└── sql
    └── foo
        └── bar
            └── .keep

The same rules apply to handler callbacks. Note that callback field must be a valid Python package name - lowercase letters, underscores, and dots.

New CLI commands and flags

schema init is a new command to prepare a database for running DipDip. It will create tables based on your models, then call on_reindex SQL hook to finish preparation - the same things DipDup does when run on a clean database.
hasura configure --force flag allows to configure Hasura even if metadata hash matches one saved in database. It may come in handy during development.
init --keep-schemas flag makes DipDup preserve contract JSONSchemas. Usually, they are removed after generating typeclasses with datamodel-codegen, but you can keep them to convert to other formats or troubleshoot codegen issues.

Built-in `dipdup_head_status` view and REST endpoint

DipDup maintains several internal models to keep its state. As Hasura generates GraphQL queries and REST endpoints for those models, you can use them for monitoring. However, some SaaS monitoring solutions can only check whether an HTTP response contains a specific word or not. For such cases dipdup_head_status view was added - a simplified representation of dipdup_head table. It returns OK when datasource received head less than two minutes ago and OUTDATED otherwise. Latter means that something's stuck, either DipDup (e.g., because of database deadlock) or TzKT instance. Or maybe the whole Tezos blockchain, but in that case, you have problems bigger than indexing.

$ curl "http://127.0.0.1:41000/api/rest/dipdupHeadStatus?name=https%3A%2F%2Fapi.tzkt.io" 
{"dipdupHeadStatus":[{"status":"OUTDATED"}]}%

Note that dipdup_head update may be delayed during sync even if the --early-realtime flag is enabled, so don't rely exclusively on this endpoint.

Changelog

Added

cli: Added schema init command to initialize database schema.
cli: Added --force flag to hasura configure command.
codegen: Added support for subpackages inside callback directories.
hasura: Added dipdup_head_status view and REST endpoint.
index: Added an ability to skip historical data while synchronizing big_map indexes.
metadata: Added metadata datasource.
tzkt: Added get_big_map and get_contract_big_maps datasource methods.