Growing Anguishes – Sia Blog
For the past month we’ve been working closely with Poloniex and Bittrex to fix wallet issues that were preventing deposits and withdrawals. It shows up things are now embarking to function as expected. Bittrex withdrawals have been working for several days, and Poloniex is clearing out its backlog of failed withdrawals. Most fresh Poloniex withdrawals are successful, and many users are receiving withdrawals from weeks ago.
Thank you all so much for your patience and support as we worked with the exchanges to fix the issues. In this post, we will explain what went wrong, how we responded, and what we are doing to make sure these issues don’t reoccur anytime soon.
What Went Wrong
Overall, the wallet issues were caused by a major spike in transaction volume on the Sia blockchain.
We plan for growth, but we were not expecting so much growth, so prompt, without warning. Siacoin entered May with a market cap of
$30M, and quickly enlargened to over $400M by early June. As users transferred their coins out of exchanges, and became hosts and renters on the Sia network, the number of transactions significantly grew.
This yam-sized spike in transaction volume meant that our transaction pool (aka mempool), at Two MB in size, periodically packed up at peak times. If the pool is total, fresh transactions get thrown away. Since miners pick transactions from the pool to form blocks, any transaction that is not included in the pool will not be added to the Sia blockchain.
This is our fault. We made the decision two years ago to cap the transaction pool at Two MB, due to significant spectacle issues above that size, and postponed the engineering work that would permit for a larger pool.
So, as users requested withdrawals from Poloniex and Bittrex, the exchanges broadcasted the transactions to the network — but didn’t check to make sure they actually were added to the blockchain. The transactions were marked as accomplish as soon as they were broadcasted. This caused a lot of confusion, since users with ended transactions were not eyeing their coins appearing in their wallets.
How We Responded
After receiving user reports of Poloniex wallet issues, we emailed them on May 22 to ask if we could help. We provided a quick fix on May 27 that would permit exchanges to re-broadcast transactions that were not added to the transaction pool. Poloniex re-enabled the wallet on May 30, but had to disable it overnight due to similar problems occurring (the transaction pool quickly packed up again, and they were not rebroadcasting transactions automatically).
We continued to communicate with Poloniex, and provided them with the IP addresses of a major mining pool on May 31 (so that they could broadcast transactions directly to the pool), and better logging on June 1. We then implemented a send-to-many endpoint on June Five, so that exchanges could batch up to 250 withdrawals in a single transaction.
Poloniex implemented these fixes and went back online on June 8. They are working through the backlog, and we expect them to finish that today. This means things should be pretty much back to normal.
We worked with Bittrex at the same time to implement the fixes and they responded indeed quick — getting back up and running in just a few days.
In total, we merged 14 fixes into the Sia code in a span of two weeks. These include both the short-term fixes mentioned above and long-term upgrades to permit the Sia network to better scale as transaction volume proceeds to increase.
Fixes and Upgrades
Permits exchanges to grab a raw transaction associated with an ID and re-broadcast it to the network.
Improve logging so that exchanges can more lightly know if a transaction confirmed.
Every reverted transaction will now be retried to be added in the pool. Long-term upgrade for transaction pool to make reorgs less disruptive.
Permits transaction pool to naturally clear of glut, preventing obtuse network fragmentation.
Permits exchanges to process 25x the withdrawals per hour, using 1/Trio the blockchain space per withdrawal.
Add a fee market to permit high priority transactions to go through.
Spectacle optimization to securely permit for a thicker transaction pool.
Optimization to permit safe larger transaction pool.
Scalability upgrade with stress tests to permit safe larger transaction pool.
Most of the above pull requests concentrate on long-term upgrades to permit the Sia network to better scale.
We’ll be enhancing the size of the transaction pool from 2mb to 5mb, with the capability to automatically grow further if there are large quantities of high-fee transactions. We will also be creating a fee market, similar to Bitcoin, so that miners can prioritize transactions based on fees. With the upgrades implemented, the Sia network should be able to securely scale by an order of magnitude.
These scalability upgrades are a main priority, and will be included in the v1.Trio.0 Capricorn release later this month, along with other significant features like remote file repair — which will permit Sia serve as a viable cloud backup solution. In order to practice these scalability upgrades, a large portion of Sia users will need to upgrade to v1.Three.0 upon its release.
Once again, thank you all so much for your patience and support in this time of extreme growth!
By clapping more or less, you can signal to us which stories truly stand out.
Blog posts related to Sia and cryptocurrency.