Blog Article
Not All Data is Equal: How Soroban is Solving State Bloat
Author
Tomer Weller
Publishing date
Soroban
State expiration
State bloat
Soroban designers have been discussing how to address one of blockchain’s elephants in the room for several months — that elephant being state bloat. State in blockchain refers to the current snapshot of all the data recorded on the ledger at a particular point in time. And state bloat refers to the fact that the blockchain’s state is always getting bigger as more account, asset, and smart contract information and associated data are stored, ultimately causing massive scalability and sustainability issues.
State bloat has been a known issue in blockchain development for several years and many networks are trying to address it, including Ethereum’s proposals to use danksharding and zero-knowledge rollups (ZK-rollups). However, although these methods help to slow state growth, they don’t limit it, and act more like a band-aid than a fix. It’s hard to retroactively tell contracts that their state expires, so the fact that Ethereum and other chains have been deploying smart contracts and building a DeFi ecosystem for years makes implementing a full state expiration plan much more difficult.
Soroban, on the other hand, benefits from the second mover advantage, meaning its design is optimized based on other networks’ experiences. This is true for many previously discussed design decisions and today we’ll focus on one more: how Soroban’s design is tackling state bloat by implementing a state expiration solution from the outset.
Blockchain’s state size problem
As mentioned earlier, a blockchain’s state refers to the current information recorded on the network at a given point in time. The specific information included in the state can vary depending on the chain, but with Stellar and Soroban, state includes all accounts, smart contracts, assets, and their respective balances or data.
Copies of a blockchain’s state are stored by each validator node and grow with every new ledger entry. For example, if Suzie airdrops 100 SPINACH tokens to Stephan, the state size would increase. In the case of Stellar, this would add one claimable balance entry to the ledger. Since blockchains can process thousands of (much more complex) transactions per second, state can change and grow exponentially.
As a blockchain’s state increases over time, validators’ hardware requirements also increase and it takes much longer to sync a new validator to the network. Those running a validator node must be able to afford the storage and computational resources necessary to store the blockchain’s entire state and keep up with processing new blocks and transactions. These ever-increasing technical requirements make maintaining a validator more demanding and raise the barrier to entry for new users, which undermines one of the main purposes of blockchain: decentralization.
In addition, allowing state size to grow unbounded has the potential to negatively affect a network’s transactions per second (TPS). As the ledger state grows, it’s harder for validators to find the information they’re looking for — like looking for a needle in a haystack with the haystack constantly getting exponentially bigger. This causes transactions to slow and lowers TPS since it’s more difficult for validators to find the information they need to validate transactions.
So, allowing blockchain state to grow uninhibited 1) undermines decentralization by raising the barrier to entry for validators, and 2) decreases network performance and TPS in the long run. These are two foundational problems that need to be solved if blockchain is ever going to scale and be successful as a real-world system.
Soroban’s state bloat solution
Come Mainnet launch, Stellar will be the first blockchain to employ state expiration of this design to solve for state bloat (keep in mind this solution only applies to smart contract entries, `ContractData` and `ContractCode`), proving that it’s being built for longevity and real-world usability.
Initially taking inspiration from Ethereum’s state expiry proposals from 2021, Soroban designers spent several months researching, collaborating (internally and with the community), and iterating to come up with Soroban’s current state expiration model. Although there are still some technical decisions to be made, the overall design is solid and includes: a database to store expired data entries (Expired State Store), two distinct types of data storage (Temporary and Persistent), and a rent/fee model for entries.
Expired State Store (ESS)
The ESS is a downstream system similar to Horizon that stores the persistent data entries that have been expired and deleted from Stellar’s current database, BucketListDB.
When a user needs to restore an expired (Persistent) entry, they execute a `Restore` operation on the Stellar network. ESS nodes see this operation and provide the necessary cryptographic proofs to restore the entry back to the BucketList. When an entry is restored back to the BucketList, it can be used as if it was a normal LedgerEntry that never expired.
Temporary and Persistent data storage types
Currently, the Stellar network (and every other blockchain) treats every data entry equally. 1,000 unaccepted SPINACH token claimable balance entries take up as much storage on the network as more important and useful entries, such as 1,000 USDC balance entries. With two distinct storage types, we can remove redundant or out of date entries that are clogging up the network to make room for more useful entries.
When a developer is writing a smart contract on Soroban, they must define in the contract whether the data entries are either 1) Temporary, or 2) Persistent. Both storage types function the same with respect to their interface but differ in their fees and expiration behavior.
Temporary Storage
The Temporary storage entry type is the cheapest storage option. When it expires from the BucketList, it doesn’t move to the ESS, meaning it is permanently deleted and can never be restored.
Temporary entries are healthy for the network and great for information that is only relevant for short periods of time or that can be arbitrarily recreated.
Persistent Storage
The persistent storage entry type is more expensive to maintain on the BucketList but is sent to the ESS upon expiration and can be restored with a RestoreFootprintOp. This makes it suitable for information you can’t afford to lose, like a user’s token balance.
Rent
Currently, whether you’re storing an account balance or issuing a SPINACH token, every data entry requires just a one-time fee to burden validator nodes forever as part of the state. Implementing a pricing model for expiration allows for important data to be prioritized and less important data to expire and be deprioritized.
In Soroban’s state expiration design, every smart contract entry must pay rent to exist on the ledger before it expires. This is like paying “rent” for the space each entry takes up on the network. The rate is based on how long the entry should live on the ledger and how big the entry is. And like transactions fees, rent will go into the fee pool – which is currently locked, can only be unlocked by a validator-approved change to the protocol, and isn’t collected by SDF or any other individual or organization.
Of course, there’s a lot more to Soroban’s state expiration model than the high-level overview described above. If you’re interested in better understanding or contributing to the structural and technical design decisions, searching the soroban-dev channel in the Stellar Developer Discord or listening to the design discussions available on Soroban’s YouTube channel are great places to start.
Designing state expiration
Although Soroban’s state expiration proposal was initially created by SDF engineers, community engagement has had a tremendous impact on its design thereafter. With help from ecosystem feedback, the proposal has gone through at least ten iterations, and smaller technical questions are discussed in Discord constantly. Designing something as pioneering as state expiration cannot be done in a vacuum, and listening to feedback from the developer community is vital to ensure we are delivering something the ecosystem supports.
There have been numerous discussions around various aspects of the state expiration configuration in Discord, where many design proposals are public-facing documents.
Some deeply debated subjects include:
- Open questions around TemporaryEntry’s time-to-live (TTL) and expiration were discussed in Discord and during April 20th’s design discussion;
- How to treat keys for various storage types was heavily discussed in Discord;
- A rousing conversation regarding the various storage entry lifetimes was had during April 24th’s design discussion;
- An extremely controversial topic, state expiration semantics, was also a popular issue in Discord.
Keep in mind that Soroban’s development is happening extremely quickly. Some of the items discussed in these threads and proposals are now obsolete (I’m looking at you, mergeable entries).
Soroban state expiration, moving forward
Introducing state expiration presents a paradigm shift for blockchain. An aspect of the technology that has been frequently talked about throughout the years is that all data on a blockchain will be accessible all the time, forever. While this is a nice sentiment, it is not sustainable or realistic if blockchain is ever going to gain widespread use in the real world.
State expiration does require more work from the smart contract developer at the outset — they have to consider the different data storage types and ensure they are paying rent. However, this slightly more complex storage interface is a small tradeoff for the improved performance and decentralization the network will experience in the future. Implementing state expiration further shows how Stellar and Soroban are being built for scalability and real-world usability.
The first phase of Soroban’s state expiration implementation is live as part of the tenth preview release on Futurenet. This first phase will provide the user interface rent model and data expiration. However, since validators have a built-in expired entry cache, we won’t need to send entries to the ESS at first. Further phases of state expiration will be rolled out over the next year. Be sure to keep up to date on all Soroban-related news in the Stellar Developer Discord or by following the Soroban Twitter account!