6.0.0

⚠ Breaking Changes

Project models in models.py must be subclassed from dipdup.models.Model instead of tortoise.Model.
The deprecated on_rollback event hook has been removed in favor of on_index_rollback.
HTTP datasources no longer use caching. cache show and cache clear commands have been removed. http.cache config flag has been removed.
--logging-config option has been removed. Use the logging config section or set up logging manually.
Feature flag options of run command have been removed. Use the advanced config section instead.

Migration from 5.x

Replace tortoise.Model import with dipdup.models.Model in your models.py module.
Remove the on_rollback event hook if it still persists in your project. Ensure that on_index_rollback.py contains ctx.rollback call, or remove it and call dipdup init.
If you have used buffer_size config option, remove it to use database-level rollbacks.
Run schema approve command with every schema you want to use with DipDup 6.0.

The era of handling chain reorgs manually is finally over! Now when DipDup receives a reorg message from TzKT it just rewinds a database to the previous state reverting changes in backtracked blocks level by level. To make this possible, DipDup catches all database modifications and saves diffs in a separate table, dipdup_model_update (you don't need to access it directly).

# INSERT saved with no data, just drop this row on reorg.
trader = Trader(name='Alice', balance=100, active=True)
await trader.save()

# UPDATE saved with data old values to set them on reorg.
# Diff: {'balance': 100}
trader.balance = 200
await trader.save() 

# DELETE saved with full copy of data. On reorg this row will be recreated with the same PK.
# Diff: {'name': 'Alice', 'balance': 200, 'active': True}
await trader.delete()

Bulk class methods like bulk_insert and bulk_update are supported too. However, for raw queries, DipDup uses prefetching (additional SELECT) to save original values. So, ReallyHugeTable.filter().delete() will create efficiently a full copy of the table in dipdup_model_update. Most likely you will never need to perform such queries in handlers, but keep that detail in mind.

Since the Ithacanet protocol, only two last blocks may be backtracked. We do not need to store older diffs, they are removed automatically. If you need to keep more levels or disable this feature, adjust rollback_depth config option.

advanced:
  rollback_depth: 2  # 0 to disable

on_index_rollback event hook now looks like this:

from dipdup.context import HookContext
from dipdup.index import Index


async def on_index_rollback(
    ctx: HookContext,
    index: Index,  # type: ignore[type-arg]
    from_level: int,
    to_level: int,
) -> None:
    await ctx.execute_sql('on_index_rollback')
    await ctx.rollback(
        index=index.name,
        from_level=from_level,
        to_level=to_level,
    )

TzKT buffer_size option remains available, but it's not required to handle chain reorgs anymore.

Crash dumps and automatic reporting

Now when DipDup catches unhandled exceptions, a crash dump will be saved to the temporary directory.

dipdup.exceptions.CallbackError: An error occured during callback execution
________________________________________________________________________________

`demo_token.hooks.on_restart` callback execution failed:

  Exception: 

Eliminate the reason of failure and restart DipDup.
________________________________________________________________________________

Crashdump saved to `/tmp/dipdup/crashdumps/veb7kz07.json`

This JSON file is the same data Sentry collects on crashes if integration is enabled. It includes a stack trace, local variables of each frame, and other information useful when investigating a crash. Attach this file when sending bug reports to GitHub Issues.

When preparing a crash dump Sentry can detect sensitive information like database passwords in the crash dump and remove it from the report. So it's generally safe to share the crash dump with the developers. Now you can also send these crash reports automatically to the Baking Bad team.

Your privacy matters; crash reporting is disabled by default. Simulate a crash with a random exception and inspect a crash dump before enabling this option to ensure that report doesn't contain secrets. Then add the following lines to your config:

advanced:
  crash_reporting: True

Changes since 5.2.5

Added

cli: Added config export --full flag to resolve templates before printing config.
config: Added advanced.crash_reporting flag to enable reporting crashes to Baking Bad.
config: Added advanced.rollback_depth field, a number of levels to keep in a database for rollback.
context: Added rollback method to perform database rollback.
database: Added an internal ModelUpdate model to store the latest database changes.
dipdup: Save Sentry crashdump in /tmp/dipdup/crashdumps/XXXXXXX.json on a crash.

Fixed

config: Do not perform env variable substitution in commented-out lines.
prometheus: Fixed updating dipdup_index_handlers_matched_total metric.

Changed

codegen: on_index_rollback hook calls ctx.rollback by default.
database: Project models must be subclassed from dipdup.models.Model

Removed

cli: --logging-config option is removed.
cli: All run command flags are removed. Use the advanced section of the config.
cli: cache show and cache clear commands are removed.
config: http.cache flag is removed.
hooks: Removed deprecated on_rollback hook.
index: Do not try to avoid single-level rollbacks by comparing operation hashes.