Operation injected to node, but not added to blockchain | Wallet counter error

Question

I use my own babbylon node and previously i have added some wallet from tezos faucet and send some coins to my generated wallet with tezos-client:

./tezos-client transfer 8500 from test_w1 to tz1b3RxJP2TDQZEvmdBtobCZ6QpcBHSYs6xA --burn-cap 258

Send via eztz.js

Now i'm trying to send tezos from one generated wallet to another with eztz.js lib. Use this code:

        async function send(toAddress, keys, amount, fee) {
            //function (contract, from, keys, amount, parameter, fee, gasLimit, storageLimit) 
            await eztz.contract.send(toAddress, keys.pkh, keys, amount, "", fee, 20000, 20000);
        }

        await send(
            "tz1QHRFZJcPJoNiqWQnuUuHHAmKLN7AdUmSN",
            {
                pk: "edpkvVbT6jFiT2MHKEp9BeVubxmEnK7GUYo3iRwxWWY3Y4Rk8VrRcA",
                pkh: "tz1b3RxJP2TDQZEvmdBtobCZ6QpcBHSYs6xA",
                sk: "edskRny9Tbjrjfcks1TWEQfM789RHbhdscHRuSm4huvQoYqNQsPeupLqahC8q2Xmy7vmov5gfTTT1PxBVmu8EigRuocE3hB8tQ"
            },
            300000000,
            10000000
        );

When i execute it first time - node inject my operation and even return address of transaction, but this transfer wansn't added to blockchain, my balance wasn't changed. First time operation logs:

When tried to send second time - get counter error:

Error while applying operation onnqHbU5bigtJbCRZD8HYPtVZKsyioNSM2tbvGMS9unMVsfhnBG:\nbranch refused (Error:\n                  Counter 128324 already used for contract tz1b3RxJP2TDQZEvmdBtobCZ6QpcBHSYs6xA (expected 128326)\n)"}]

Second time logs:

If manually set counter to 128326 - will receive error counter_in_the_future.

Counter error gone after half an hour, but still if i try to send coins - operation will be not included to blockchain and i will receive counter error again.

Send via tezos-client

I imported my generated private key as test_w2 and try to send coins with tezos-client:

./tezos-client transfer 8500 from test_w2 to tz1UqBubMBKtEVpN6jkNQwc3TAoBNdtmtiyM --burn-cap 258

First time i've get same counter error:

But after half an hour it worked:

So it means, error not in wallets generating or tezos node, error can be only in eztz.js client or passed values. Maybe i've passed wrong fee / gas_limit / amount?

How can i fix this?

Note. Actually, i use forked Tz.Net library (c#), but it has same workflow as eztz.js and i receive same errors.

Answer 1

TLDR: the first operation you injected in your node is valid but will be never be injected by other nodes/bakers because it has too low fees. It is then stuck in the mempool and make the the next operations you're trying to forge invalid because the next valid counter you want to use is already being taken by the too low fee operation. You can restart your node or wait for 60 blocks to pass to unlock the situation.

---

What I believe happens here is that the first operation you injected is considered invalid by your other peers. As you said, it can be rejected for multiple reasons (not enough fees being the most common case).

So, you forged an operation which is valid regarding the protocol and therefore valid in the mempool. Your node accepted injecting the operation because some bakers might be accepting and including 0 fees operations. However, by default, this is not the case. The operation you injected in the node is thus living in the mempool waiting for a baker to include it in a block.

So now comes the technical part. When you request the node to forge an operation for you, the client (or the API you're using) will request from the node the counter associated to your account (side note: the counter is here to prevent replay attacks). To grab this information, the node will query the data using the current state of the chain (i.e. the state that results from the application of the last block received) and will return, for example, 1000. For the operation you're forging to be valid, its counter must then be 1001. If this is not the case, it will give you a counter in the past or counter in the future error when the node is trying to validate it.

In your case, you injected a too low fee operation with a counter 128324. Your node considers that you know what you're doing (e.g. trying to get a low-fee baker to inject it) and thus does not check for the fees. If there are no low-fee baker running in the network, your operation will never be included and stuck in the mempool for 60 blocks (20min minimum in babylonnet). After those 60 blocks, the operation will considered too old and purged from the mempool. This is why, after some times, the situation was unlocked.

Now, while this invalid operation is stuck in the mempool, the next operation you forge using the client will also request the counter from the node and as your last operation was not included, the previous counter 128324 will be used again which collides with the previous transaction. When this happens, the mempool is not happy because it will try to validate the operations one after the other. As your first operation is valid (even if it will never be included), it will be applied on its internal state which is an intermediate state where your first operation was applied thus making the next expected counter increased by one. The next time an operation is received (for the same account) the mempool will expect the counter to be increased to 128325. If that's not the case, then it will reject it with a counter in the past error. If you try to manually increase the counter, you might get counter in the future.

How to get out of this situation:

• you may wait for 60 blocks to pass by and then your transaction will be discarded for being too old;
• you can restart your node if you have control over it. The mempool is erased when the node stops;
• you can forge and inject your transaction using another node. One, which will not have the invalid operation stuck in its mempool.

In the next shell release, we aim to include an admin RPC that will be used to remove a specific operation from the mempool.

P.S. You may check which operations live in the mempool using the RPC /chains/main/mempool/pending_operations