Transaction Submission, Timeouts, and Dynamic Fees FAQ

No. The network is humming along nicely. It has processed almost 2 billion operations since its inception, and about 500 million in the past 6 months alone. Ledgers continue to close in about 5 seconds, and throughput remains high. For a quick visual, check out the network dashboard.

What is happening, however, is that there is more network activity, which is increasing competition for ledger space. When the number of operations submitted to the network exceeds the ledger limit, the network enters surge pricing mode, at which point, fees serve as bids, and transactions offering higher fees per operation are prioritized for inclusion in the transaction set that's applied to the ledger. If your fee bid is too low, you may get priced out.

Currently, the public network is configured to allow 1,000 operations per ledger.

Quick reminder: in the Stellar vernacular, an operation is an individual command that modifies the ledger. At the moment, there are 18 possible operations, including the payment operation, path payment operations, and operations to place buy and sell orders on the DEX.

A transaction is a bundle of 1-100 operations. To modify the ledger — in other words, to do something on Stellar — you take operations, bundle them into a transaction, and wrap that in a transaction envelope containing necessary signatures, which you then submit to a Stellar Core node. Generally, you create and sign the transaction using a Stellar SDK, and generally, that Stellar SDK uses Horizon, the Stellar API, to submit it.

Assuming a transaction is valid, the Stellar Core node that receives it shares it with other Stellar Core nodes, who work together to create a combined set of valid transactions. Nodes that are armed to validate then ratify that transaction set via a multi-stage voting process, and apply it to change the state of the ledger. For more on that process, check out the Transaction Lifecycle doc.

Since the Protocol 11 upgrade in 2019, the maximum size of a transaction set — aka the ledger limit — has been measured in operations/ledger rather than transactions/ledger. The rationale: operations are what get applied to alter accounts, balances, and orders, and they're not variable in size like transactions, so using them to set the ledger limit allows for more consistent throughput. The protocol essentially peers into transactions, counts the number of operations inside, and tries to accommodate as many transactions as possible given the ledger limit.

The ledger limit is a network configuration, and like all network configurations, it's determined by validators, who vote on it just like they vote to apply a transaction set to the ledger. Shortly after the Protocol 11 upgrade, validators set the ledger limit to 1,000 ops/ledger, and it's remained there ever since.

When setting the ledger limit, validators attempt to strike a balance: they want to allow enough throughput to support network usage, but they also want to make sure that nodes across the world with access to lower end hardware and slower connectivity can still keep up and participate in consensus.

At any time, Stellar-network validators can vote to increase the ledger limit. However, given that a large portion of current network traffic consists of bot-submitted arbitrage transactions destined to fail (more on that below), it is unlikely that increasing the ledger limit right now would do much to reduce surge pricing. It would just give the bots more room to operate.

If you are reading this, and have an opinion about ledger limits, keep in mind that you can run your own validator to participate in the network and vote on where to set them. Who can run a Stellar validator? Anyone can. You can!

Fees on Stellar are dynamic. There is a minimum fee required for all transactions, which, like the ledger limit, is a network configuration determined by validators. Currently, it is 100 stroops per operation. That's 0.00001 XLM.

When network activity is below the ledger limit, all valid transactions make the ledger and incur the minimum fee. When the number of operations submitted exceeds the ledger limit, the network enters surge pricing mode. Since not every transaction can make the ledger, transactions are prioritized based on the fee they specify. The transactions offering the highest fee per operation make the ledger first. For more information on how that works, see the Fees doc.

Think of fees as bids. How much are you willing to pay to get your transaction processed? That's the fee you should bid. The maximum amount you are willing to pay. You will actually be charged the minimum amount necessary to make the ledger.

How much are people actually getting charged? To get a sense, I used the public BigQuery Stellar dataset to pull fee data for the month of January, 2021. Here's what I discovered:

• The average fee/op charged in January 2021 was 1,221 stroops. That's 0.0001221 XLM (about $0.00004762 at the time of writing).
• 52% of transactions incurred the minimum fee of 100 stroops/op.
  

In other words, even though surge pricing was frequently in effect in January, fees were still a fraction of a fraction of a cent.

Generally, consumer-facing apps and other services that want to ensure their transactions make the ledger in a timely fashion specify a fee of ~100,000 stroops. That's 0.01 XLM, which most users still find incredibly cheap to move money around the world. As network activity continues to increase, you may find you need to adjust your fee bid to remain competitive, but for now, you should be able to set it around 100,000 stroops and forget it.

Most people (including you, dear reader, if you're getting this error) use Horizon, the Stellar API, to submit transactions to the network. Transaction submission — and the subsequent steps required to ratify a transaction and apply it to the ledger — is a complicated, asynchronous process. Horizon is middleware designed to make that process simpler for the user. It posts transactions to a Stellar Core node, and waits to hear results before returning an HTTP response.

When you submit a transaction to Horizon, Horizon first attempts to decode it. If it can't — usually because there's some structural error with the XDR — Horizon returns a 400 error, and lets you know that the transaction is malformed. If the transaction is well formed, Horizon posts it to a Stellar Core node, and that node performs a validity check. If the node discovers the transaction is invalid — say it doesn't have the right sequence number, or the fee it offers is below the 100-stroop network minimum — Horizon returns a 400 error, and lets you know what went wrong.

If the transaction is well formed and passes the validity check, the Stellar Core node propagates it to the rest of the network for possible inclusion in the ledger. When network activity is below the ledger limit, the transaction goes through and incurs the minimum fee, Stellar Core notifies Horizon, and Horizon returns a 200 Success response. The happy path!

However, if the network is in surge pricing mode and the transaction's fee bid is too low to make the ledger, the Stellar Core node hangs onto it, and tries to resubmit it for the next 3 ledgers. Horizon doesn't have direct insight into that process, so it waits 30 seconds, and if it doesn't hear back from Stellar Core, it concludes that there is no success response coming, and lets the user know that the request has timed out. That's the best it can do.

That means that when you get a 504, there's still a chance that your transaction will make the ledger — Horizon doesn't know for sure whether or not Stellar Core has discarded the transaction — which is why we strongly suggest using timebounds to limit the submission window. However, if you are getting a lot of 504s, it is likely that your fee bids are generally too low. Submit higher fees.

As mentioned at the top of the FAQ, we will publish a technical guide to handling errors gracefully. For details and code examples, you should check that out as soon as it's available.

Generally, though, you should do three things when submitting a transaction:

1. Set a timebound. That way, if the transaction doesn't succeed within the time specified, you know it's not hanging around waiting to be processed. You can try again knowing you won't accidentally, say, duplicate a payment.
2. Set the highest maximum fee you are willing to pay. You will actually pay the minimum amount necessary to make the ledger. Under normal circumstances, even with a higher max fee set, you will pay the network minimum — currently 100 stroops.
3. Implement a retry loop with increasing delay (e.g. 30s, 60s, 90s). It should only execute once you've exceeded the timebound you set.
   

If you do those three things, you should be able to sidestep or work around timeouts, and to consistently get your transactions onto the ledger.

No. If you are using the public, SDF-maintained Horizon instance, you will get a 429 error if you try to submit more than 3,600 requests per hour. It's a free, public API, and other people need to use it, too. Don't be a resource hog! If you're getting rate limited, it's time to set up your own Horizon instance.

If you are trying to submit a high volume of transactions, you may be experiencing issues related to sequence number management. The best practice is to use a channel account which you can read all about in the Channels doc.

If you look at the network dashboard, you may notice a high volume of failed transactions included in the ledger. Those are caused by a slew of trading bots attempting to take advantage of a small number of arbitrage opportunities.

Stellar has a unique set of operations called path payments, which allow the simultaneous sending and conversion of currency — I send USD; you receive NGN — and they make it incredibly easy to use the network for cross-border and cross-currency transactions.

Path payments convert currency by consuming orders in Stellar’s built-in order books, and sometimes an inefficiency in the order books gives rise to a pricing mismatch. We won’t get into the details here, but savvy developers realized that — every once in a while — you can submit a path payment and end up with a tiny bit more money than you started with, and they built bots to look for those opportunities, and to try to capitalize on them.

A lot of people built arbitrage bots, and they all look for the same opportunities. When one comes up, it’s a race: the winning bot submits a transaction that claims the opportunity and succeeds; the remaining bots submit transactions conditioned on the existence of that opportunity, and since it’s no longer available, those transactions fail.

Because those transactions met the minimum fee requirement, they fail after they’re included in the ledger rather than before, so they end up in everyone else’s way. It’s like a passel of pigeons hovering around a park bench: you drop a crumb on the sidewalk, they all dive after it. One pigeon gets the crumb, the rest stay hungry, and while the losers sit there cooing and strutting and wishing for what might have been, they block the sidewalk so pedestrians can’t use it.

Not really. Stellar is open participation: anyone can submit transactions to the network, including people who build bots that try to take advantage of arbitrage opportunities.

Back in July, there was a discussion about raising the minimum fee to try to price out the arbitrage bots, but doing so would have had an immediate impact on everyone in the ecosystem — especially market makers, who submit a high volume of DEX orders to provide liquidity — and there wasn't consistent support for the idea.

Instead, you, dear reader, should plan to outbid the bots. Remember: they get no return from failed transactions, and as fees increase, it becomes more expensive for them to continue diving after bread crumbs.

Fees do not go to the Stellar Development Foundation or to network validators: they go into the fee pool, which is tracked in the ledger header. At the moment, that's where they remain.

In the days of yore, Stellar had an inflation operation, and the fee pool was distributed every time it ran. However, after a ton of feedback from the ecosystem, a proposal to disable inflation was implemented in Protocol 12, and when validators voted to upgrade the network to that version of the protocol on 10/28/2019, the inflation operation was officially deprecated. There is no inflation on Stellar. Also, there is no staking on Stellar.

At some point, someone could come up with a new proposal to distribute the fee pool, that proposal could be implemented in a major protocol release, and the network validators could vote to upgrade the network to that version of the protocol. That someone could be you! Remember: Stellar is open source, and anyone can contribute to the codebase.

At the moment, however, fees are so low on Stellar — even with surge pricing — that the fee pool isn't that big, and it doesn't grow that quickly. It would take a lot of time and engineering effort to redistribute it, and it's probably not worth worrying about in the short term.

While fees don't enrich anyone, they do serve a purpose: they discourage large-scale bad behavior, and they give everyone a fair chance to bid for limited ledger space.