How to recover from "No space left on device" error?

Question

The drive the node uses ran out of disk space. This resulted in a series of errors in a format similar to:

Apr 16 01:58:36 - validator.block: Validation of block BKyJr1hPHwi5eXd71vSmVy6pFiXrFJBMBiByyEtqExBSbS3XYcx failed
Apr 16 01:58:36 - validator.block: Pushed: 2019-04-16T01:58:36Z, Treated: 2019-04-16T01:58:36Z, Failed: 2019-04-16T01:58:36Z
Apr 16 01:58:36 - validator.block: No space left on device                       
Apr 16 01:58:36 - node.validator.bootstrap_pipeline: Unexpected error (validator): Error:
Apr 16 01:58:36 - node.validator.bootstrap_pipeline:                                 No space left on device
Apr 16 01:58:36 - node.validator.bootstrap_pipeline:        
Apr 16 01:58:37 - validator.peer(13): Worker crashed [NetXXXXXXXXXX:idXXXXXXXXXX]:                          
Apr 16 01:58:37 - validator.peer(13): No space left on device

The node stopped syncing and all peers became disconnected.

I freed up adequate disk space and restarted the node, but it continued to repeat the above errors about being out of disk space and refused to sync, disconnecting from all peers.

How can I get my node to realize there is free space on the drive and resume syncing?

Answer 1

The clue to the solution is in the part of the error saying that Validation of block BK... failed.

It seems that it wasn't checking to see if there was any new free disk space available, it was just reading back the error of why it failed in the first place. And when it failed to validate the block, the block was marked as invalid.

After running ./tezos-admin-client unmark all invalid blocks and restarting the tezos-node, the node is properly connecting to peers and syncing again.