We've been having lots of missed endorsements lately for unknown reasons. The logs for the endorser show a properly injected operation, and shows the hash. But searching for the hash reveals 404 Not found (on tzscan) and looking at the block we see it not there. Sometimes we are the only endorsement missing; sometimes we are 1 of 2 or 3 others. We consistently have over 300 p2p connections and run on SSDs.
We now know we are the victims of this issue, https://gitlab.com/tezos/tezos/issues/559 in some situations, as we are experiencing the same behavior with our node RPC timing out.
Until the devs fix the issue, to combat this, we want to run multiple endorsers on different nodes to achieve a type of "high availability". If our main endorser can't contact our local node, due to the issue above, our second endorser will attempt the same endorsement on it's local node.
Is this double endorsing? It's the same block, and the same endorsement slots. In theory, both servers should produce the same operation hash, which is just a duplicate; not a double. Right?