5.0.0

⚠ Breaking Changes

Python versions 3.8 and 3.9 are no longer supported.
bcd datasource has been removed.
Two internal tables were added, dipdup_contract_metadata and dipdup_token_metadata.
Some methods of tzkt datasource have changed their signatures and behavior.
Dummy advanced.oneshot config flag has been removed.
Dummy schema approve --hashes command flag has been removed.
docker init command has been removed.
ReindexingReason enumeration items have been changed.

Migration from 4.x

Ensure that you have a python = "^3.10" dependency in pyproject.toml.
Remove bcd datasources from config. Use metadata datasource instead to fetch contract and token metadata.
Update tzkt datasource method calls as described below.
Run the dipdup schema approve command on every database you use with 5.0.0.
Update usage of ReindexingReason enumeration if needed.

Chain reorgs have occurred much recently since the Ithaca protocol reached mainnet. The preferable way to deal with rollbacks is the on_rollback hook. But if the logic of your indexer is too complex, you can buffer an arbitrary number of levels before processing to avoid reindexing.

datasources:
  tzkt_mainnet:
    kind: tzkt
    url: https://api.tzkt.io
    buffer_size: 2

DipDup tries to remove backtracked operations from the buffer instead emitting rollback. Ithaca guarantees operations finality after one block and blocks finality after two blocks, so to completely avoid reorgs, buffer_size should be 2.

BCD API takedown

Better Call Dev API was officially deprecated in February. Thus, it's time to go for bcd datasource. In DipDup, it served the only purpose of fetching contract and token metadata. Now there's a separate metadata datasource which do the same thing but better. If you have used bcd datasource for custom requests, see How to migrate from BCD to TzKT API article.

TzKT batch request pagination

Historically, most TzktDatasource methods had a page iteration logic hidden inside. The quantity of items returned by TzKT in a single request is configured in HTTPConfig.batch_size and defaulted to 10.000. Before this release, three requests would be performed by the get_big_map method to fetch 25.000 big map keys, leading to performance degradation and extensive memory usage.

affected method	response size in 4.x	response size in 5.x
`get_similar_contracts`	unlimited	max. `datasource.request_limit`
`get_originated_contracts`	unlimited	max. `datasource.request_limit`
`get_big_map`	unlimited	max. `datasource.request_limit`
`get_contract_big_maps`	unlimited	max. `datasource.request_limit`
`get_quotes`	first `datasource.request_limit`	max. `datasource.request_limit`

All paginated methods now behave the same way. You can either iterate over pages manually or use iter_... helpers.

datasource = ctx.get_tzkt_datasource('tzkt_mainnet')
batch_iter = datasource.iter_big_map(
    big_map_id=big_map_id,
    level=last_level,
)
async for key_batch in batch_iter:
    for key in key_batch:
        ...

Metadata interface for TzKT integration

Starting with 5.0 you can store and expose custom contract and token metadata in the same format DipDup Metadata service does for TZIP-compatible metadata.

Enable this feature with advanced.metadata_interface flag, then update metadata in any callback:

await ctx.update_contract_metadata(
    network='mainnet',
    address='KT1...',
    metadata={'foo': 'bar'},
)

Metadata stored in dipdup_contract_metadata and dipdup_token_metadata tables and available via GraphQL and REST APIs.

Prometheus integration

This version introduces initial Prometheus integration. It could help you set up monitoring, find performance issues in your code, and so on. To enable this integration, add the following lines to the config:

prometheus:
  host: 0.0.0.0

Changes since 4.2.7

Added

config: Added custom section to store arbitrary user data.
metadata: Added metadata_interface feature flag to expose metadata in TzKT format.
prometheus: Added ability to expose Prometheus metrics.
tzkt: Added ability to process realtime messages with lag.
tzkt: Added missing fields to the HeadBlockData model.
tzkt: Added iter_... methods to iterate over item batches.

Fixed

config: Fixed default SQLite path (:memory:).
prometheus: Fixed invalid metric labels.
tzkt: Fixed pagination in several getter methods.
tzkt: Fixed data loss when skip_history option is enabled.
tzkt: Fixed crash in methods that do not support cursor pagination.
tzkt: Fixed possible OOM while calling methods that support pagination.
tzkt: Fixed possible data loss in get_originations and get_quotes methods.

Changed

tzkt: Added offset and limit arguments to all methods that support pagination.

Removed

bcd: Removed bcd datasource and config section.
cli: Removed docker init command.
cli: Removed dummy schema approve --hashes flag.
config: Removed dummy advanced.oneshot flag.

Performance

dipdup: Use fast orjson library instead of built-in json where possible.