This blog post captures the highlights from Patrick McCorry's talk at EthDenver 2021. Patrick is Master of Relays at Infura and is passionately working to solve transaction management friction for the industry. We outline three critical problems that currently prevent developers and DeFi users from getting more efficient transactions on Ethereum, and how Infura Transactions seeks to eliminate this friction to create a better end user experience.
What is the problem behind stuck transactions?
The Ethereum network is like a funnel. Transactions waiting inside the funnel can be considered pending, and they are only mined after they have passed through the funnel. A block is generated every 14 seconds, and every block contains around 12MM gas. When there are too many pending transactions in the funnel, and it exceeds the block capacity, it’s physically impossible for every pending transaction to get into the next block. Generally, this is the source of stuck transactions.
Ethereum as a global permissionless fee market. It is up to YOU to entice the miner.
Given this list of pending transactions, miners get to decide which transactions to include in the next block and which transactions to ignore. This is why we have a fee market. It is up to you to set a high fee that can outcompete all other pending transactions and thus entice miners to include your transaction over others.
Problem #1: Transactions get dropped by the network
Let's consider the first source of stuck transactions. You pick your transaction, set a fee you are happy with, sign it, and broadcast it to the network. Then, for some reason, the transaction gets dropped and it won't be picked up by miners because they are simply unaware of it.
Dropped transactions are typical when the network is under high congestion. Your transaction must be propagated over the devp2p network, and any node between you and the miners can decide to drop your transaction for any reason - especially when the mempool is congested. So, let’s assume an Ethereum node can only hold 4,096 transactions (default node configuration setting) and there are 4,097 transactions in the pending pool, then one transaction has to get dropped. If your transaction is the one with the lowest fee transaction in the pending pool, it will be the one that is most likely to get dropped.
There are also various different (and often awkward) network rules that have an impact on why transactions get stuck. For example: you might have 17 transactions in the pending pool, but it only guarantees to keep around 16, so the 17th gets dropped as seen by the go-ethereum code and config.
A wider problem with dropped transactions is called the nonce gap issue. Let’s say you send 6 transactions in a row. For some reason, transaction 3 gets dropped from the network. As a result, its child transactions (in this case transaction 4, 5, and 6) are going to get stuck, and will never be confirmed by a miner. That’s because there’s a special network rule: for every Ethereum transaction, the nonce must increment strictly by 1. In this case however, transaction 3 got dropped and makes it impossible for any proceeding transaction to get mined. That’s an extremely frustrating situation to be in.
How theoretical is the nonce gap problem? Do we really incur dropped transactions? Let’s look at this research from Blocknative. Their forensic mempool investigation surfaced the events on March 12 and 13 2020, when over 50,000 transactions an hour were being dropped from the pending pool as you can see here:
One company that suffered during Black Thursday (and highlighted in the Blocknative blog post) was MakerDAO.
When the price of ETH collapsed on March 12, 2020, a large number of CDPs on MakerDAO immediately became undercollateralized and eligible for forcible liquidation. When a CDP is forcefully liquidated, it triggers an auction process that lets anyone (keeper bots) place a bid to purchase the remaining collateral. However, on Black Thursday, many Keeper transactions were adversely affected as the network congestion led to miner nodes dropping transactions at an unusually high rate.
This led to the keeper's bot non-dropped transactions becoming stuck. They tried to bump the gas price of the stuck transactions including setting an outrageous fee of 4,500 gwei (10x the normal gas fee) - and even that transaction got stuck! This is because the dropped transactions resulted in a nonce gap that prohibited miners from including any of the child transactions. The solution was not to bump the gas price, but to re-publish the dropped transactions and fix the nonce gap issue.
Of course, some users noticed that keeper bots were struggling to submit bids and they subsequently won the the auctions by submitting 0 DAI bids. Overall, the keeper bots just had a bug in their relay infrastructure.
Problem #2: Transaction queues get stuck
The second problem is how a bot manages the transaction fees for each transaction in their queue of transactions.
It can become problematic if the same fee schedule is not applied across the entire transaction queue. For example, you can see in the above that if there is a low fee transaction in the middle of the queue, then it can be nearly as bad as having a dropped transaction. This is because Ethereum miners don't always look at your dependent transactions, and as a result the queue can get stuck because of that one low fee. (This is slightly different to Bitcoin, as miners do consider dependent transactions. This is called "child pays for parent" and it helps bump up the transaction fee.)
Problem #3: Paying too much… bad fee management
Finally, the third problem: setting the gas price too high. Commonly, people visit Eth Gas Station, get the trader rate, and then send that to the network. For the most part, that works. You pay a premium fee, and your transaction will get mined in the next few blocks. Of course, the prices are so volatile that even that doesn’t work, because the fees skyrocket by the time you send your transaction. Consequently, everyone overpays the fee and you still don't have a guarantee that the fee will be sufficient for the transaction to get mined.
Solving these problems is not trivial.
Most teams that build their own in-house relayers would agree that transaction relayer software is really fiddly. In fact, one team we spoke with recently, described their in-house relayer as a complex beast.
Writing robust fee-replacement and transaction monitoring software is complex and time consuming. Transaction monitoring also requires dedicated infrastructure, resources, and effort running 24/7 to work reliably. And even if you build your own relayer, it’s really difficult to get satisfactory guarantees. Unless done properly transactions won’t be mined at a reliable rate, resulting in bad UX. If you get the relayer software wrong, you can end up with a potential loss of funds. Some transactions are time-critical (e.g. a CDP) and fragile transaction infrastructure could lose you money.
Just like the issues faced by the keeper bots competing in the liquidation auction on Black Thursday...most of the problems could have been avoided with better relayer software.
...Enter Infura Transactions
So really, the whole point of Infura Transactions (ITX) is to help you to no longer deal with stuck transactions and to ensure you do not need to re-invent the wheel by building your own in-house relayer.
How does it work? You can simply hook into the Infura Transactions API and send your relay transaction to Infura. We'll guarantee that your transaction gets delivered to Ethereum and that you will no longer deal with stuck transactions.
How does ITX solve the problem?
The ITX relayer is built with the following features:
- Republish periodically. ITX re-publishes the transaction every few minutes, so even if your transaction gets dropped, we don't need to check if it was dropped or not, we will always just republish it.
- Highest fee first. Our transaction queue is organized such that the highest fee paying transaction is at the start of the queue and the lowest fee paying transaction is at the end of the queue.
- Real-time fee adjustments. Our initial gas price estimate for sending the transaction aims to pay the minimum price while targeting a quick confirmation.
- Escalator algorithm. We have an increasingly sophisticated bidding strategy such that we pay the best price at any time.
Our list of features and ITX is built with reliability in mind and to help avoid users paying unnecessarily excess fees. There are plenty of gotchas at the network level that we have discovered while building ITX and we hope no one else will need to re-implement the wheel to overcome the same issues.
Who may find ITX useful?
You may find ITX useful if you are working on the following products:
- Crypto wallet providers. There are several wallet providers who send transactions on behalf of their users including Gnosis, Argent, Dharma and Loopring. Our infrastructure is close to plug-and-play for most smart contract wallet providers.
- Sequencers and rollup providers. A sequencer needs to send very large bocks (~12 million gas for an optimism block) and thus they need to effectively outcompete the rest of the network to get their block confirmed. Our fee bumping strategy ensures that we'll eventually get the rollup block mined at the best price.
- Server-side scripts & DeFi bots. An example includes the keeper bot's for the liquidation auctions and ensuring the auction bids are reliably processed in time.
- Dapps. An example is the Ethereum Name service as it requires ~3-4 transactions before the domain name is set up. Ideally, the user can simply send ETH to the ENS service, and then the server-side script finishes the process on behalf of the user.
How do we get started?
We hope that you are now hyped up about using the ITX service for your project.
To get started:
- ETH is money. You can deposit directly into our service as a pre-payment to cover your gas fees. This will act as your virtual gas tank and we will use that ETH to pay for gas on the network. ETH is the payment mechanism, ETH is money, use it as money, and deposit into the ITX service.
- Ethereum account. We use Ethereum signing keys to handle authentication for new relay requests to the ITX service. You sign it and then simply give us the job.
- Coding project. You need a script or a project that will send a transaction to the network.
How to set up your gas tank:
All deposits are handled by an on-chain smart contract and it has the same address across all networks:
- Send deposit. You can send the funds using the
- Wait. Your balance will be updated on the ITX service after 10 blocks.
- You are ready. Just send us a transaction and watch it get mined.
Just to remind everyone. ETH is money. ETH is how we take the payment. And ETH is used in your gas tank.
Of course, since the deposit is handled by a smart contract. You can simply automate the top-up process in your application and you may even use ITX to perform the continuous top-ups.
How have we implemented ITX:
We have three new JSON RPC commands. If you are using ethers.js, you can simply use
provider.send(["jsonRpcName","parameters"]) to send us the command.
Let's briefly cover each command:
relay_getBalance: Given an Ethereum account, it will return the balance of the user's gas tank in WEI.
relay_sendTransaction: Given a pre-signed relay transaction, it will return a relay transaction has to let you track it's progress.
relay_getTransactionStatus: Given a relay transaction hash, it will return a list of broadcast Ethereum Transactions (and then you can check if one is mined).
Our Status API returns a list of transaction hashes due to how fee bumping works in Ethereum. Every time we bump the network fee on your behalf it will produce a new Ethereum Transaction (and a new hash). We decided to provide all hashes to let you decide how your users will track the progress of their transaction.
Find out more:
We hope you have enjoyed this article and hopefully you will decide to give ITX instead of trying to build an in-house relayer (i.e., a complex beast).
If you would like to find out more information about ITX and the API, then please our documentation and submit a request for alpha access. If you want something slightly more hands-on, then our ITX demo project is available for you to try.