4.0.0

⚠ Breaking Changes

• run --oneshot option is removed. The oneshot mode (DipDup stops after the sync is finished) applies automatically when last_level field is set in the index config.
• clear-cache command is removed. Use cache clear instead.

Migration from 3.x

• Run dipdup init command to generate on_synchronized hook stubs.
• Run dipdup schema approve command on every database you want to use with 4.0.0. Running dipdup migrate is not necessary since spec_version hasn't changed in this release.

What's New

Performance optimizations

Overall indexing performance has been significantly improved. Key highlights:

• Configuration files are loaded 10x times faster. The more indexes in the project, the more noticeable difference is.
• Significantly reduced CPU usage in realtime mode.
• Datasource default HTTP connection options optimized for a reasonable balance between resource consumption and indexing speed.

Also, two new flags were added to improve DipDup performance in several scenarios: merge_subscriptions and early_relatime. See this paragraph for details.

Configurable action on reindex

There are several reasons that trigger reindexing:

Now it is possible to configure desirable action on reindexing triggered by the specific reason.

To configure actions for each reason, add the following section to DipDup config:

advanced:
  ...
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception

New CLI commands and flags

advanced top-level config section

This config section allows users to tune system-wide options, either experimental or unsuitable for generic configurations.

CLI flags have priority over self-titled AdvancedConfig fields.

aiosignalrcore replaced with pysignalr

It may not be the most noticeable improvement for end-user, but it still deserves a separate paragraph in this article.

Historically, DipDup used our own fork of signalrcore library named aiosignalrcore. This project aimed to replace the synchronous websocket-client library with asyncio-ready websockets. Later we discovered that required changes make it hard to maintain backward compatibility, so we have decided to rewrite this library from scratch. So now you have both a modern and reliable library for SignalR protocol and a much more stable DipDup. Ain't it nice?

Changes since 3.1.3

This is a combined changelog of -rc versions released since the last stable release until this one.

Added

• cli: Added run --early-realtime flag to establish a realtime connection before all indexes are synchronized.
• cli: Added'run --merge-subscriptions` flag to subscribe to all operations/big map diffs during realtime indexing.
• cli: Added status command to print the current status of indexes from the database.
• cli: Added config export [--unsafe] command to print config after resolving all links and variables.
• cli: Added cache show command to get information about file caches used by DipDup.
• config: Added first_level and last_level optional fields to TemplateIndexConfig. These limits are applied after ones from the template itself.
• config: Added daemon boolean field to JobConfig to run a single callback indefinitely. Conflicts with crontab and interval fields.
• config: Added advanced top-level section.
• hooks: Added on_synchronized hook, which fires each time all indexes reach realtime state.

Fixed

• cli: Fixed config not being verified when invoking some commands.
• cli: Fixed crashes and output inconsistency when piping DipDup commands.
• cli: Fixed missing schema approve --hashes argument.
• cli: Fixed schema wipe --immune flag being ignored.
• codegen: Fixed contract address used instead of an alias when typename is not set.
• codegen: Fixed generating callback arguments for untyped operations.
• codegen: Fixed missing imports in handlers generated during init.
• coinbase: Fixed possible data inconsistency caused by caching enabled for method get_candles.
• hasura: Fixed unnecessary reconfiguration in restart.
• http: Fixed increasing sleep time between failed request attempts.
• index: Fixed CallbackError raised instead of ReindexingRequiredError in some cases.
• index: Fixed crash while processing storage of some contracts.
• index: Fixed incorrect log messages, remove duplicate ones.
• index: Fixed invocation of head index callback.
• index: Fixed matching of untyped operations filtered by source field (@pravin-d).
• tzkt: Fixed filtering of big map diffs by the path.
• tzkt: Fixed get_originated_contracts and get_similar_contracts methods whose output was limited to HTTPConfig.batch_size field.
• tzkt: Fixed lots of SignalR bugs by replacing aiosignalrcore library with pysignalr.
• tzkt: Fixed processing operations with entrypoint default.
• tzkt: Fixed regression in processing migration originations.
• tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
• tzkt: Fixed sending useless subscription requests when adding indexes in runtime.

Changed

• cli: schema wipe command now requires confirmation when invoked in the interactive shell.
• cli: schema approve command now also causes a recalculation of schema and index config hashes.
• index: DipDup will recalculate respective hashes if reindexing is triggered with config_modified: ignore or schema_modified: ignore in advanced config.

Removed

• cli: Removed deprecated run --oneshot argument and clear-cache command.

Performance

• config: Configuration files are loaded 10x times faster.
• index: Checks performed on each iteration of the main DipDup loop are slightly faster now.
• index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
• tzkt: Improved performance of response deserialization.
• tzkt: Rate limit was increased. Try to set connection_timeout to a higher value if requests fail with ConnectionTimeout exception.