5.1.0

Migration from 5.0 (optional)

• Run init command. Now you have two conflicting hooks: on_rollback and on_index_rollback. Follow the guide below to perform the migration. ConflictingHooksError exception will be raised until then.

What's New

Per-index rollback hook

In this release, we continue to improve the rollback-handling experience, which became much more important since the Ithaca protocol reached mainnet. Let's briefly recap how DipDup currently processes chain reorgs before calling a rollback hook:

1. If the buffer_size option of a TzKT datasource is set to a non-zero value, and there are enough data messages buffered when a rollback occurs, data is just dropped from the buffer, and indexing continues.
2. If all indexes in the config are operation ones, we can attempt to process a single-level rollback. All operations from rolled back block must be presented in the next one for rollback to succeed. If some operations are missing, the on_rollback hook will be called as usual.
3. Finally, we can safely ignore indexes with a level lower than the rollback target. The index level is updated either on synchronization or when at least one related operation or bigmap diff has been extracted from a realtime message.

If none of these tricks have worked, we can't process a rollback without custom logic. Here's where changes begin. Before this release, every project contained the on_rollback hook, which receives datasource: IndexDatasource argument and from/to levels. Even if your deployment has thousands of indexes and only a couple of them are affected by rollback, you weren't able to easily find out which ones.

Now on_rollback hook is deprecated and superseded by the on_index_rollback one. Choose one of the following options:

• You haven't touched the on_rollback hook since project creation. Run init command and remove hooks/on_rollback and sql/on_rollback directories in project root. Default action (reindexing) has not changed.
• You have some custom logic in on_rollback hook and want to leave it as-is for now. You can ignore introduced changes at least till the next major release.
• You have implemented per-datasource rollback logic and are ready to switch to the per-index one. Run init, move your code to the on_index_rollback hook and delete on_rollback one. Note, you can access rolled back datasource via index.datasource.

Token transfer index

Sometimes implementing an operation index is overkill for a specific task. An existing alternative is to use a big_map index to process only the diffs of selected big map paths. However, you still need to have a separate index for each contract of interest, which is very resource-consuming. A widespread case is indexing FA1.2/FA2 token contracts. So, this release introduces a new token_transfer index:

indexes:
  transfers:
    kind: token_transfer
    datasource: tzkt
    handlers:
      - callback: transfers

The TokenTransferData object is passed to the handler on each operation, containing only information enough to process a token transfer.

config env command to generate env-files

Generally, It's good to separate a project config from deployment parameters, and DipDup has multiple options to achieve this. First of all, multiple configs can be chained successively, overriding top-level sections. Second, the DipDup config can contain docker-compose-style environment variable declarations. Let's say your config contains the following content:

database:
  kind: postgres
  host: db
  port: 5432
  user: ${POSTGRES_USER:-dipdup}
  password: ${POSTGRES_PASSWORD:-changeme}
  database: ${POSTGRES_DB:-dipdup}

You can generate an env-file to use with this exact config:

$ dipdup -c dipdup.yml -c dipdup.docker.yml config env
POSTGRES_USER=dipdup
POSTGRES_PASSWORD=changeme
POSTGRES_DB=dipdup

The environment of your current shell is also taken into account:

$ POSTGRES_DB=foobar dipdup -c dipdup.yml -c dipdup.docker.yml config env
POSTGRES_USER=dipdup
POSTGRES_PASSWORD=changeme
POSTGRES_DB=foobar  # <- set from current env

Use -f <filename> option to save output on disk instead of printing to stdout. After you have modified the env-file according to your needs, you can apply it the way which is more convenient to you:

With dipdup --env-file / -e option:

dipdup -e prod.env <...> run

When using docker-compose:

services:
  indexer:
    ...
    env_file: prod.env

Keeping framework up-to-date

A bunch of new tags is now pushed to the Docker Hub on each release in addition to the X.Y.Z one: X.Y and X. That way, you can stick to a specific release without the risk of leaving a minor/major update unattended (friends don't let friends use latest 😉). The -pytezos flavor is also available for each tag.

FROM dipdup/dipdup:5.1
...

In addition, DipDup will poll GitHub for new releases on each command which executes reasonably long and print a warning when running an outdated version. You can disable these checks with advanced.skip_version_check flag.

Pro tip: you can also enable notifications on the GitHub repo page with 👁 Watch -> Custom -> tick Releases -> Apply to never miss a fresh DipDup release.

Changelog

See full 5.1.0 changelog here.