Why did the network ignore my baked block?

Question

I had baking priority slot 0. I successfully baked the block. I successfully injected my block to the network (as shown in bakers log below). Yet, for some reason, the baker at priority 1 actually baked the block and received rewards. Why?

Feb  6 12:31:00 - client.baking: Injected block BLr8PWEwAZfk for usbaker after BLc2zLq3aB9E (level 301867, priority 0, fitness 00::00000000008d6a29, operations 21+0+0+1).

I understand that my block has lower fitness than the priority 1 block, but still want to know how that occurred. It was my understanding that priority 1 cannot even “attempt” to inject his block until AFTER my 60s has expired. Is this not the way the protocol works? Do priority 0/1/2/3/N all inject at the same time and it's up to the endorsers to make sure they endorse priority 0 block first?

It seems that is what happened in my case. The logs clearly show that I had priority 0, but for whatever reason, the endorsers decided to endorse priority 1's block, not mine. The question I’d like a protocol dev to answer is “why”? If I did in fact inject my priority 0 block within the specified time, why did endorsers pick priority 1?

Answer 1

Here is how it works: Just to keep things simple, let's say that the chain (all the blocks included up to now) has a fitness of 100. In other words, 100 total endorsements were included on all the blocks in the chain. Your baker now has priority 0 rights to bake the next block (block #5), and you have 30 endorsements in your mempool for the previous block (block #4).

You go ahead and bake a block at height 5 which has a total fitness of 130 (the total of previous fitness + all new endorsements you were able to include).

The priority 1 baker has 32 endorsements for the previous block in their mempool, maybe because they have better peer connectivity, or perhaps learned of 2 more endorsements than you did.

The priority 1 baker sees your block arrive and notices it has 130 fitness. At T+15 seconds (the time you baked your priority 0 block + 15 seconds), the priority 1 baker realizes that it can construct a block that has 132 fitness by including the 2 endorsements that you didn't, so they go ahead and inject the block.

The rest of the network sees that the priority 1 block has fitness of 132, and your priority 0 block only has fitness 130, so the network changes branch to make the priority 1 block the new head of the chain.

So, to summarize, there are a couple common misconceptions here:

• Whether bakers endorse your block determines whether it gets accepted or not. Endorsements never arrive until after your block is baked, and only increase the fitness of the chain you added your block to, not the block itself.
• A priority 1 baker has to wait an additional 30 seconds before they can bake a block that becomes the new head of the chain. They really only need to wait 15 seconds before they can inject a block that has a higher fitness than yours.

Answer 2

The shell currently ignores a new head if it is less fit than the hypothetical next head -- baked with all the endorsements in the mempool. This hypothetical next fitness is called context_fitness in the code there.

This means that if your baked block includes fewer endorsements than endorsers see in their mempool, they will ignore your block and wait for the priority 1 block.

Answer 3

Most likely poor connection to peers, or at least the ones that mattered at the point in time (the endorsers for your block, and baker of the next).

Endorsers will endorse the block with the higher fitness, but they can't double endorse. That means if they saw your block and endorsed it, they can't change their mind when the p1 block came along (even if it had a higher fitness level).

In your case, the endorsers never received your block fast enough allowing the potential of endorsing the p1 block. That p1 block had a higher fitness, so the endorsers endorsed that block.

Why would endorsers go with a higher priority block? Because it has a higher chance of being the canonical block, and endorsers only get paid if the block the endorse succeeds.