I/O error while sync

Question

I have following error

validator.block: Validation of block BLr9zngSpapX4V6Bd4vkdy3LGLmkXD6hQLR96QzRA3924SfuPZV failed
8/19/2019 5:05:43 PMAug 19 14:05:43 - validator.block: Pushed: 2019-08-19T14:05:43-00:00, Treated: 2019-08-19T14:05:43-00:00, Failed: 2019-08-19T14:05:43-00:00
8/19/2019 5:05:43 PMAug 19 14:05:43 - validator.block: I/O error

8/19/2019 6:30:53 PMAug 19 15:30:53 - node.validator.bootstrap_pipeline:                                 I/O error
8/19/2019 6:30:53 PMAug 19 15:30:53 - node.validator.bootstrap_pipeline:
8/19/2019 6:30:53 PMAug 19 15:30:53 - validator.peer_245: Worker crashed [NetXdQprcVkpa:idrMAtAJZMXA]:
8/19/2019 6:30:53 PMAug 19 15:30:53 - validator.peer_245: I/O error
8/19/2019 6:30:58 PMAug 19 15:30:58 - validator.block: Validation of block BLr9zngSpapX4V6Bd4vkdy3LGLmkXD6hQLR96QzRA3924SfuPZV failed
8/19/2019 6:30:58 PMAug 19 15:30:58 - validator.block: Pushed: 2019-08-19T15:30:58-00:00, Treated: 2019-08-19T15:30:58-00:00, Failed: 2019-08-19T15:30:58-00:00
8/19/2019 6:30:58 PMAug 19 15:30:58 - validator.block: I/O error
8/19/2019 6:30:58 PMAug 19 15:30:58 - node.validator.bootstrap_pipeline: Unexpected error (validator): Error:
8/19/2019 6:30:58 PMAug 19 15:30:58 - node.validator.bootstrap_pipeline:                                 I/O error
8/19/2019 6:30:58 PMAug 19 15:30:58 - node.validator.bootstrap_pipeline:
8/19/2019 6:30:58 PMAug 19 15:30:58 - validator.peer_210: Worker crashed [NetXdQprcVkpa:idsFj3Xkb3kV]:
8/19/2019 6:30:58 PMAug 19 15:30:58 - validator.peer_210: I/O error
8/19/2019 6:31:03 PMAug 19 15:31:03 - validator.block: Validation of block BLr9zngSpapX4V6Bd4vkdy3LGLmkXD6hQLR96QzRA3924SfuPZV failed
8/19/2019 6:31:03 PMAug 19 15:31:03 - validator.block: Pushed: 2019-08-19T15:31:03-00:00, Treated: 2019-08-19T15:31:03-00:00, Failed: 2019-08-19T15:31:03-00:00
8/19/2019 6:31:03 PMAug 19 15:31:03 - validator.block: I/O error
8/19/2019 6:31:03 PMAug 19 15:31:03 - node.validator.bootstrap_pipeline: Unexpected error (validator): Error:
8/19/2019 6:31:03 PMAug 19 15:31:03 - node.validator.bootstrap_pipeline:                                 I/O error
8/19/2019 6:31:03 PMAug 19 15:31:03 - node.validator.bootstrap_pipeline:
8/19/2019 6:31:03 PMAug 19 15:31:03 - validator.peer_233: Worker crashed [NetXdQprcVkpa:idqWQvWjXsHA]:
8/19/2019 6:31:03 PMAug 19 15:31:03 - validator.peer_233: I/O error
8/19/2019 6:31:07 PMAug 19 15:31:07 - validator.block: Validation of block BLr9zngSpapX4V6Bd4vkdy3LGLmkXD6hQLR96QzRA3924SfuPZV failed
8/19/2019 6:31:07 PMAug 19 15:31:07 - validator.block: Pushed: 2019-08-19T15:31:07-00:00, Treated: 2019-08-19T15:31:07-00:00, Failed: 2019-08-19T15:31:07-00:00
8/19/2019 6:31:07 PMAug 19 15:31:07 - validator.block: I/O error
8/19/2019 6:31:07 PMAug 19 15:31:07 - node.validator.bootstrap_pipeline: Unexpected error (validator): Error:
8/19/2019 6:31:07 PMAug 19 15:31:07 - node.validator.bootstrap_pipeline:                                 I/O error
8/19/2019 6:31:07 PMAug 19 15:31:07 - node.validator.bootstrap_pipeline:
8/19/2019 6:31:07 PMAug 19 15:31:07 - validator.peer_232: Worker crashed [NetXdQprcVkpa:idtSKCZfLiqc]:
8/19/2019 6:31:07 PMAug 19 15:31:07 - validator.peer_232: I/O error
8/19/2019 6:31:09 PMAug 19 15:31:09 - validator.peer_269: Worker started for NetXdQprcVkpa:idsJFAtMfLo2
8/19/2019 6:31:09 PMAug 19 15:31:09 - validator.peer_269: Worker terminated [NetXdQprcVkpa:idsJFAtMfLo2]

??

Answer 1

I/O errors can be triggered by many events. Most of the time, it is raised when the disk storage is full and the last validated block cannot be stored.

When an error occurs while validating a block, it is marked as rejected in the storage. The error will be raised each time the validator tries to validate that block.

To solve this problem you should first list the rejected blocks using the following command:

./tezos-admin-client list rejected blocks (In your case, it should give you the BLr9zngSpapX4V6Bd4vkdy3LGLmkXD6hQLR96QzRA3924SfuPZV block)

Then, you should call:

tezos-admin-client unmark invalid <the_block_hash>. The validation process should restart normally.