Downsides to running multiple endorsers?

Question

We've been having lots of missed endorsements lately for unknown reasons. The logs for the endorser show a properly injected operation, and shows the hash. But searching for the hash reveals 404 Not found (on tzscan) and looking at the block we see it not there. Sometimes we are the only endorsement missing; sometimes we are 1 of 2 or 3 others. We consistently have over 300 p2p connections and run on SSDs.

We now know we are the victims of this issue, https://gitlab.com/tezos/tezos/issues/559 in some situations, as we are experiencing the same behavior with our node RPC timing out.

Until the devs fix the issue, to combat this, we want to run multiple endorsers on different nodes to achieve a type of "high availability". If our main endorser can't contact our local node, due to the issue above, our second endorser will attempt the same endorsement on it's local node.

Is this double endorsing? It's the same block, and the same endorsement slots. In theory, both servers should produce the same operation hash, which is just a duplicate; not a double. Right?

Answer 1

This is correct but extremely and dangerously (for yourself) optimistic.

Yes, in theory, as long are the 2 nodes you attach an endorser to are perfectly aligned "It's the same block, and the same endorsement slots. [so] the same operation hash, which is just a duplicate; not a double."

But, necessarily, in practice, there will be a time where the 2 nodes won't be perfectly in the exact same state. At that point, the 2 endorsers won't endorse the same block and THAT will be double endorsing which will make you loose all the benefit of your high availability and much much more.

For a realistic dark scenario takes:

There is a block baked by baker priority 0 at level 41 containing 31 endorsements and the late 32th (late) endorsement of block level 40 that are concurrently propagating themselves on the network. - Your node A receives the 32th endorsements of 40 first and then the block level 41. It "refuses" it because it lacks an endorsement and wait for baker priority 1 at level 41 to bake a block with the whole 32 endorsements. Endorser A endorses this block of baker of priority 1 - Your node B receives the block level 41 first (and therefore accepts it) and then the 32th endorsements of block 40 (and discard it, too late now). Endorser B endorses the block at level 41 of baker priority 0.

You've just double endorsed, Goodbye security deposit and rewards!!!