1

I ran of space with my node, i reset corrected the problem and restarted the node, now i have a weird problem when i run my code, it no longer works. The following query does not work: http://localhost:8732/chains/main/blocks/head~670000/header it returns empty info, if i run it with tezos-client rpc get /chains/main/blocks/head~670000/header it returns did not find service, but doing http://localhost:8732/chains/main/blocks/head/header returns data. This is huge problem for us, we depend on getting block info. And i am stumped figuring out what the problem is.

Improve this question
2
  • It seems that your storage is in a weird state with a (maybe) corrupted storage. Is the node running well ? Is there only one block not responding well to rpcs ?
    – vect0r
    Nov 3, 2019 at 18:25
  • Almost the entire blocks cannot be queried, 6500 down are there, if i stop start the node this number increases. I need to start the node and force to fix before connecting.
    – user277
    Nov 3, 2019 at 21:38

1 Answer 1

Reset to default
2

In my experience when the tezos-node process fails hard its store and context data (in ~/.tezos-node, typically) can be left in an inconsistent state such that that chain data has to be thrown away and recreated.

That chain data can be recreated by syncing from scratch or by importing a snapshot and then syncing from that snapshot point. https://tezosshots.com/ is one source for such snapshots.

The kind of hard failures that can interrupt tezos-node and leave its data in bad shape include: running out of free disk space (as you experienced), getting killed by the OOM-killer when RAM is exhausted, getting a SIGKILL interrupt such as from kill -9, and by sudden loss of power to the server.

(There is reportedly work underway to modify tezos-node to be less susceptible to this kind of error, perhaps by enforcing more ACID guarantees in the storage layer.)

Improve this answer
2
  • Yeah restarting from scratch implies resynching an archive node over a week minimum. It would be good to have som db tools to try to repair part of it. The funny thing is, there us a process that seem to run at the start that repair the srart, but seems to stop. So a stop start of the node fixes a bit more each time. Do clearly there is a process that checks the db. I need to keep running and finish.
    – user277
    Nov 3, 2019 at 21:33
  • And using a snapshot is nit an option, since i run an archive node. No ine has such snapshot, it’s too big.
    – user277
    Nov 3, 2019 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.