SDF’s Horizon: Limiting Data to 1 year

Let us explain. Initially, the SDF public Horizon service was meant to provide an easy building block for ecosystem developers, encourage builders to experiment with the network and provide some visibility into on-chain data. However, even though the docs have advised against it, many projects used it to power the backends of production applications.

This reliance on SDF's Horizon has, in turn, created a single point of failure in the network. Network activity drops almost 90% when our instances are experiencing degraded performance or downtime. This centralized point of failure is dangerous for the health of a decentralized network, and it has become increasingly tenuous as the network has scaled along with Horizon.

Horizon’s complete historical database is a staggering 40TB and spans almost a decade, yet our analyses show that 89% of requests only care about the last month. These requests are serving the intended use case of SDF’s Horizon: a building block to support developers and their users to use the network for payments or other applications. These users need a fast, reliable, and responsive service to iterate quickly.

The other 10%, though, are often large, complex queries of historical data. Providing the same availability and latency for this data causes unsustainable operational overhead and complexity, yet our intention has never been to serve these use cases (let alone at this service level). So, we are making that intention explicit by continuing to support the large user and developer base with a highly available free service while encouraging users with different needs to use one of the many other options available in our ecosystem.

This change will likely not impact most of you, so you won't need to do anything.

This change will only impact you if you rely on horizon.stellar.org to retrieve network history >1 year old.

If this change does impact your project, first assess your needs. What's your goal in accessing historical data? Is it an audit? Is it limited to a handful of assets? Do you want to stream all network changes? Each of these needs has a different, better way to continue to view historical data:

• Find an infrastructure provider: The ecosystem offers several options for accessing a Horizon instance managed by a third party.
• Use our data warehouse: We already have a better way to query historical data: Hubble, our public BigQuery dataset. In Hubble, filters or pagination limitations won't restrict you, and you can make queries directly against the entire network's history. Check out the blog post or the documentation to get started!
• Store the data yourself: If the data you need is well-bounded, you can use this transition period (or Hubble) to set up your own data storage system with the full flexibility you desire.
• Run Horizon yourself: Before diving into this, remember: assess your needs! Maybe you only care about history related to Circle’s USDC. Maybe you just need two years of history and can limit the rest. Think carefully, then check out the Admin Guide to get started.
• Try out Soroban RPC: This simpler, lightweight service will soon provide a rolling week of basic transaction history and is much easier to run yourself!

While this change may pose challenges, it's a step in the right direction toward a more decentralized, self-sustaining ecosystem. And we are committed to making this transition as smooth as possible with alternative solutions to help meet your needs and consistent communication about upcoming milestones.

Stay tuned for a series of upcoming posts where we’ll explore the truncation itself, and share more on new initiatives and products that facilitate access to the network. Notable amongst those will be SDF’s composable data platform, a modular architecture we're developing to provide flexible and efficient insights into arbitrary data spans.
Until then, we would love to hear your feedback on this change, so please get in touch with us in the #horizon channel on our Developer Discord.