Upcoming Database Changes in Protocol 21
Author
Garand Tyson
Publishing date
In Protocol 21, Soroban is getting its first big upgrade with the implementation of five new CAPs that introduce some exciting features, such as passkey signing support, an improvement to state archival, and cost improvements for smart contract transactions. In addition to these new features, Stellar validators are also getting one of the largest performance optimizations to date: a brand new database backend called BucketListDB.
For those who haven’t heard, BucketListDB is a database built from the ground up for the Stellar blockchain. BucketListDB provides much faster reads and writes than Stellar’s current SQL database backends, while also requiring less memory and disk space. Most excitingly, BucketListDB also supports conflict-free parallel IO which will be pivotal to future Protocol upgrades that introduce parallel Soroban execution and multithreaded Overlay networking. If you’re curious how everything works under the hood, check out this tech talk.
BucketListDB has been an experimental feature in stellar-core and after a year of rigorous testing, it’s time to switch it on by default. This means that starting with stellar-core release 21.0, SQL and in-memory mode are officially being deprecated. During this transition period, SQL and in-memory mode will still be supported, but may see significant performance degradation compared to BucketListDB and fall behind the rest of the network.
Note that SQL isn’t fully going away. SQL will continue to be used for storing small amounts of non-ledger related metainformation and for some DEX queries. However, the majority of queries and most ledger state will only be served by BucketListDB.
What Validator Operators Need to Do
Stellar-core version 21.0 will introduce a new config flag called DEPRECATED_SQL_LEDGER_STATE. This flag replaces EXPERIMENTAL_BUCKETLIST_DB. To support migration, both flags will be supported in the stellar-core 21.0 release. However EXPERIMENTAL_BUCKETLIST_DB will be removed in stellar-core 21.1. EXPERIMENTAL_BUCKETLIST_DB=true is identical to DEPRECATED_SQL_LEDGER_STATE=false. While both flags are supported, only one can be used in the configuration file.
Either DEPRECATED_SQL_LEDGER_STATE or EXPERIMENTAL_BUCKETLIST_DB must be defined or stellar-core 21.0 will not be able to start. One of these flags must be set when a node upgrades to the stellar-core 21.0 package. One of these flags must be set when the package is deployed, not when the network actually upgrades to Protocol 21.
This flag’s default setting, and the setting that most validator operators should use, is DEPRECATED_SQL_LEDGER_STATE=false (or equivalently EXPERIMENTAL_BUCKETLIST_DB=true). If DEPRECATED_SQL_LEDGER_STATE=true (or equivalently EXPERIMENTAL_BUCKETLIST_DB=false), the node may experience performance degradation and fall behind the rest of the network.
DEPRECATED_SQL_LEDGER_STATE=true (or equivalently EXPERIMENTAL_BUCKETLIST_DB=false) should only be set to true if you either:
- Must run stellar-core with the “in-memory” mode flag.
- Directly query data from stellar-core’s SQL backend, and the data being queried is not supported by BucketListDB (see “Deprecated Features” below).
Only operators running a standalone validator or watcher node will need to set this flag. If captive-core is running as part of Horizon or RPC, the flag will automatically be set and no action is required.
DEPRECATED_SQL_LEDGER_STATE is a new flag only supported in stellar-core 21.0+. This means this flag cannot be set on any stellar-core 20.x packages. In order to prevent downtime when picking up the new package, it is recommended to upgrade stellar-core with the following steps:
- In the current stellar-core 20.x config, set EXPERIMENTAL_BUCKETLIST_DB=true. This will trigger a one-time migration to the new BucketListDB backend (see Migration Guide below). This migration takes some time, so it is recommended that only one node’s config is updated at a time.
- Upgrade stellar-core to 21.0. Since the node has already migrated and EXPERIMENTAL_BUCKETLIST_DB is defined, stellar-core should restart quickly without significant downtime.
- After upgrading, change EXPERIMENTAL_BUCKETLIST_DB=true to DEPRECATED_SQL_LEDGER_STATE=false. These flags are equivalent, but the EXPERIMENTAL_BUCKETLIST_DB will be removed in a future release.
Migration Guide
When DEPRECATED_SQL_LEDGER_STATE=false (or equivalently EXPERIMENTAL_BUCKETLIST_DB=true) is set for the first time, there is a one-time migration performed from SQL to BucketListDB. This migration occurs automatically the first time either `stellar-core run` or `stellar-core upgrade-db` is executed and requires permission to perform DB schema changes. This migration involves dropping SQL tables and setting up indexes required for BucketListDB and is disk bound. On NVME SSD drives, this migration takes around 3 minutes. On slower EBS backed drives, migration may take up to 45 minutes. It is recommended that operators upgrade and migrate each node individually.
Following the one time migration, validator startup time and catchup time are significantly reduced compared to SQL.Following the migration, stellar-core disk usage is expected to drop approximately 65%. While memory usage of the stellar-core process is expected to increase by around 1 GB, there is a significant reduction in memory usage by the SQL backend (note that specific SQL backend memory usage is heavily dependent on operator configuration). It is expected that BucketListDB is strictly more efficient than the preexisting SQL backends, so validator requirements and resource utilization overall should not increase.
Deprecated Features
When DEPRECATED_SQL_LEDGER_STATE=false (or equivalently EXPERIMENTAL_BUCKETLIST_DB=true) the following tables in stellar-core’s SQL database are dropped:
- ACCOUNT
- TRUSTLINE
- DATA
- CLAIMABLE_BALANCE
- LIQUIDITY_POOL
- CONTRACT_DATA
- CONTRACT_CODE
- CONFIG_SETTING
- TTL
This includes all ledger state tables (with the exception of OFFERS).Additionally, the TXMETA column in the TXHISTORY table will be dropped.
Note that when DEPRECATED_SQL_LEDGER_STATE=false (or equivalently EXPERIMENTAL_BUCKETLIST_DB=true), this data is deleted on the first execution of `stellar-core run` or `stellar-core upgrade-db`. To recover this information, it will be necessary to set DEPRECATED_SQL_LEDGER_STATE=true, run `stellar-core new-db`, then replay transaction data.In general, we advise against using stellar-core to provide this information because querying core databases directly may cause the node to lose sync with the network, and the best practice is to run a Horizon or RPC instance. However, if querying this information is required by your application, it is recommended to set DEPRECATED_SQL_LEDGER_STATE=true in the short term. This flag will be removed in future releases and these tables will no longer be supported. As a long term solution, consider running a Horizon or RPC instance.
Additionally, in-memory mode is being deprecated and will be removed in a future update. BucketListDB provides significantly faster startup time and can catch up to the network much faster than SQL, removing the need for in-memory mode.
Questions? Comments? Please raise them on the Stellar Dev Discord #validators channel.