Blog Article
Horizon just got a lot better: our new ingestion engine and other great stuff
Autor
Stellar Development Foundation
Fecha de publicación
Horizon
Ingestion
API
Over a year ago, the StellarX team added a GitHub issue about a strange 500 Internal Server Error returned by Horizon from time to time. After checking it out, we realized the cause: Horizon was getting offers from the Stellar-Core database, and some of those offers were coming in from ledgers not yet ingested by Horizon. To get a creation time for offers, Horizon checks the ledger close time, and if a ledger has yet to be ingested, that check comes up empty. Hence the server error.
Meanwhile, we were getting reports about very long response times for the /paths endpoint. Stellar has a built-in decentralized order book that allows users to exchange one asset for another, and the /paths endpoint helps you find the best set of intermediate trades so that you can exchange the minimum possible amount of the source asset for the maximum possible amount of the destination asset. Horizon relied on the Stellar-Core database, which just isn’t optimized for path-finding queries.
Those (mostly minor) issues weren’t exactly new — we’d been aware of similar problems for years — but they were becoming increasingly common. To address them, we had to make a fundamental change to Horizon’s architecture, and to build a completely new system for ingesting data. Over the past few months, that’s exactly what we’ve done.
The new ingestion system, which is included in the Horizon v0.20.1 release, is ready for testing, and in this short blog post I’ll explain the reasoning behind it and outline two experimental features in Horizon that wouldn’t have been possible in the old system. It has some big advantages over the old ingestion system — it’s more consistent and developer-friendly, allows user configuration, and doesn’t overtax Stellar-Core — and eventually we plan to move over to it completely. At the moment, both systems are working concurrently, and if you enable a special feature flag in Horizon you can test the new system and its new features right now!
The new ingestion Golang package can also be used outside Horizon to build custom apps and services. In the coming months, we will publish documentation, examples, and a new blog post explaining how to use it.
Accounts for Signer
“Accounts for Signers” was a highly popular feature request in our GitHub. You can use it to look up all the accounts that match a given signer, which makes it easier for Stellar clients to implement multi-sig in a user-friendly way. If, for instance, you have a primary account that also co-signs several multi-sig accounts — a business account, an account you share with your partner, a family trust account — your Stellar wallet can find them all using your primary account, no extra effort required on your part. With a single key, you can easily manage multiple multi-sig accounts.
Before the new ingestion system, we couldn’t build the “Accounts for Signers” view because Stellar-Core has no index on the signer field in the signer table, and the query was not fast enough for real-time, production use. To make matters more complicated, a later version of Stellar-Core added a performance update that removed the signers table completely and moved the XDR encoded array of signers to the accounts table.
Because the new system can recreate the state, we were able to create a table in Horizon’s database with all signers and proper indexes. This feature has been available since 0.19.0, but is hidden behind a feature flag. You can also try it out on our public Horizon clusters.
What’s next?
In Q3 2019 we will be working on moving all the Horizon features to the new system. We will also publish documentation for the new ingestion package that can be used to build custom, advanced apps and services without Horizon.