I ran of space with my node, i reset corrected the problem and restarted the node, now i have a weird problem when i run my code, it no longer works. The following query does not work: http://localhost:8732/chains/main/blocks/head~670000/header it returns empty info, if i run it with tezos-client rpc get /chains/main/blocks/head~670000/header it returns did not find service, but doing http://localhost:8732/chains/main/blocks/head/header returns data. This is huge problem for us, we depend on getting block info. And i am stumped figuring out what the problem is.
2
-
It seems that your storage is in a weird state with a (maybe) corrupted storage. Is the node running well ? Is there only one block not responding well to rpcs ?– vect0rNov 3, 2019 at 18:25
-
Almost the entire blocks cannot be queried, 6500 down are there, if i stop start the node this number increases. I need to start the node and force to fix before connecting.– Zx1984Nov 3, 2019 at 21:38
Add a comment
|
1 Answer
In my experience when the tezos-node process fails hard its store and context data (in ~/.tezos-node, typically) can be left in an inconsistent state such that that chain data has to be thrown away and recreated.
That chain data can be recreated by syncing from scratch or by importing a snapshot and then syncing from that snapshot point. https://tezosshots.com/ is one source for such snapshots.
The kind of hard failures that can interrupt tezos-node and leave its data in bad shape include: running out of free disk space (as you experienced), getting killed by the OOM-killer when RAM is exhausted, getting a SIGKILL interrupt such as from kill -9, and by sudden loss of power to the server.
(There is reportedly work underway to modify tezos-node to be less susceptible to this kind of error, perhaps by enforcing more ACID guarantees in the storage layer.)
-
Yeah restarting from scratch implies resynching an archive node over a week minimum. It would be good to have som db tools to try to repair part of it. The funny thing is, there us a process that seem to run at the start that repair the srart, but seems to stop. So a stop start of the node fixes a bit more each time. Do clearly there is a process that checks the db. I need to keep running and finish.– Zx1984Nov 3, 2019 at 21:33
-
And using a snapshot is nit an option, since i run an archive node. No ine has such snapshot, it’s too big.– Zx1984Nov 3, 2019 at 21:35