Not able to run Baker and Endorser using docker image after docker image upgrade

Question

I ran out of disk space on my server, so i stop the server and increase the disk space and start the mainnet.sh script again. So it updated the script and start the node but it could not start the baker and endorser. I even run the command ./mainnet.sh baker start and it gave the output that baker is now running but when i check the status of baker using ./mainnet baker status, it shows baker is not running. I tried restarting the script but baker and endorser are not able to run.

sachint@ubuntu-s-3vcpu-1gb-blr1-01:~/tezos$ ./mainnet.sh start
mainnet: Pulling from tezos/tezos
Digest: sha256:5a97a580673976cac52ce2bb02a056042aa300928a7d5faa5c4cfa5d6548f9b3
Status: Image is up to date for tezos/tezos:mainnet
Creating network "mainnet_default" with the default driver
Creating mainnet_node_1                  ... 
Creating mainnet_endorser-003-PsddFKi3_1 ... done
Creating mainnet_endorser-004-Pt24m4xi_1 ... done
Creating mainnet_accuser-003-PsddFKi3_1  ... done
Creating mainnet_accuser-004-Pt24m4xi_1  ... done
Creating mainnet_baker-004-Pt24m4xi_1    ... done
Creating mainnet_baker-003-PsddFKi3_1    ... done

sachint@ubuntu-s-3vcpu-1gb-blr1-01:~/tezos$ ./mainnet.sh status
Node is running
Baker is not running
Endorser is not running
The script is up to date.

Answer 1

With your node running, you need to run tezos-admin-client unmark all invalid blocks. If you hit a space issue once, it becomes a “permanent” error until you clear the knowledge. This affected me once a long time ago. Supposedly there was a code-fix put in many months ago (but clearly not working). Someone in Slack had this same issue yesterday and this fix is still the workaround solution.

Answer 2

So right now i just downloaded a new mainnet.sh file using command

wget -O mainnet.sh https://gitlab.com/tezos/tezos/raw/master/scripts/alphanet.sh

now my baker and endorser are running properly.

As soon as i do ./mainnet.sh update_script, again the endorser and baker stops and won't just start using start command.

So for now i am using the mainnet.sh without updating it until i find a better solution.

And yes the command

tezos-admin-client unmark all invalid blocks

do help in case you get any IO error in your nodes log.