6

The drive the node uses ran out of disk space. This resulted in a series of errors in a format similar to:

Apr 16 01:58:36 - validator.block: Validation of block BKyJr1hPHwi5eXd71vSmVy6pFiXrFJBMBiByyEtqExBSbS3XYcx failed
Apr 16 01:58:36 - validator.block: Pushed: 2019-04-16T01:58:36Z, Treated: 2019-04-16T01:58:36Z, Failed: 2019-04-16T01:58:36Z
Apr 16 01:58:36 - validator.block: No space left on device                       
Apr 16 01:58:36 - node.validator.bootstrap_pipeline: Unexpected error (validator): Error:
Apr 16 01:58:36 - node.validator.bootstrap_pipeline:                                 No space left on device
Apr 16 01:58:36 - node.validator.bootstrap_pipeline:        
Apr 16 01:58:37 - validator.peer(13): Worker crashed [NetXXXXXXXXXX:idXXXXXXXXXX]:                          
Apr 16 01:58:37 - validator.peer(13): No space left on device

The node stopped syncing and all peers became disconnected.

I freed up adequate disk space and restarted the node, but it continued to repeat the above errors about being out of disk space and refused to sync, disconnecting from all peers.

How can I get my node to realize there is free space on the drive and resume syncing?

11

The clue to the solution is in the part of the error saying that Validation of block BK... failed.

It seems that it wasn't checking to see if there was any new free disk space available, it was just reading back the error of why it failed in the first place. And when it failed to validate the block, the block was marked as invalid.

After running ./tezos-admin-client unmark all invalid blocks and restarting the tezos-node, the node is properly connecting to peers and syncing again.

  • 1
    Just want to add that this may fix related errors. I was getting validator.peer: Worker crashed followed by a generic IO error. The underlying issue was the same -- storage was maxed out and a block was marked invalid. – latte_jed May 1 at 5:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.