I was wondering if there is any explanation available to some unexpected behavior when trying to make multiple transactions to the same smart contract.
Transaction
We have an FA1.2 token whereby we are calling the transfer
entrypoint (gas cost: 0.083230ꜩ) through another smart contract's default
entrypoint (gas cost: 0.076003ꜩ) which in total has a gas cost of 0.159233ꜩ.
For the context of this question: I will call this the "transaction".
Throughput
We are currently testing throughput on edo2net
whereby we want to maximize the total number of transactions we can include into a block. Since we are testing in hundreds (and eventually, thousands), we also want to sustain that maximum throughput over a period of time and record the overall performance.
Findings
A few interesting points that collectively bring together the question. So, here is where we are at now:
- Maximum "transactions" per operation. We have not been able to get beyond 12. To be clear, this is a single inject call with 12 transactions and resulting in the node returning a single operation hash. This calculates to 0.159233ꜩ x 12 = 1.910796ꜩ gas costs. (This isn't an issue by the way, as throughput does not have to hinge on cramming everything into a single operation as there are limits)
- Maximum "transactions" per block. It is the same as above. Which is the first part of the question... Why is the node refusing to allow us to exceed 12 per block?
Stability
The node has some strange behavior where a large "queue" of items will quickly lead to the node responding with counter issues due to the mempool getting congested with operations pending and more interestingly the node will park the pending operations for several blocks before including it. I'll call this "locked out", since we can no longer sustain the inject requests due to, what seems to be, an invalid mempool state. Here is what is interesting:
- Queuing 12 transactions in a single operation within a single block time is OK.
- Queuing 12 transactions in two operations (i.e. 6 transactions x 2 operations) is OK.
- Queuing 12 transactions in three operations (i.e. 4 transactions x 3 operations) will cause the node to "lock out"
- Queuing 10 transactions split into single operations (i.e. 1 transaction x 10 operations) will cause the node to "lock out".
- Queuing 24 transactions in two separate operations (i.e. 12 transaction x 2 operations) will cause the node to "lock out".
I don't know why this is happening either, and the stability impacts our throughput results, as we cannot sustain repeated injects / operations over time and results in a lot of "missed" blocks. Why is this happening?
Notes: All the test above were tested in various setups [a] single node / single key (1:1), [b] single node / multiple keys (1:N), [c] multiple node / multiple keys (1:1 x N)