Issues with the Tezos Node CPR script to restart the node after failure

Question

I'm facing some issues with the Tezos Node CPR script from Etom Knudsen, which serves to automatically restart the node after failure. The script has a bug in the function to call network stats in line 35, which I solved using the solution suggested by a user on GitHub.

The script calls the network statistics from the node to check if it is connected to other peers:

curl -s 127.0.0.1:8732/network/stat/

If it is not, the call will return something like:

{"total_sent":"8898729","total_recv":"9544788","current_inflow":0,"current_outflow":0}

It also checks when the node received the last block from the network, by calling:

curl -s 127.0.0.1:8732/chains/main/blocks/head/header/

I suspect that the latter has never worked, as during network failures the rpc call seems to return an empty line, preventing the script to determine when the last block arrived. By default the script is configured to restart the node after 180s when no block arrived during this time. However, it fails to do so, as indicated, for instance, by the following line:

2019-03-23T06:55:56Z Waiting. Last block was 515 secs ago

I believe since the most recent node update (commit hash: 366f64f3...) also the network stat function fails, as the node seems to receive and send some data also when it is not connected to any other peer. This is indicated by the following:

2019-03-23T06:55:35Z Waiting. Last block was 494 secs ago)
2019-03-23T06:55:40Z Found p2p activity
2019-03-23T06:55:40Z Looking for p2p avtivity - will wait for max 90 secs
2019-03-23T06:55:40Z Waiting. Last block was 499 secs ago)
2019-03-23T06:55:45Z Found p2p activity
2019-03-23T06:55:45Z Looking for p2p avtivity - will wait for max 90 secs
2019-03-23T06:55:45Z Waiting. Last block was 504 secs ago)
2019-03-23T06:55:51Z Found p2p activity

Both issues result in the script not working as intended and the node does not automatically restart any longer.

Has anyone also observed these issues and revised the script accordingly? Or does anyone know another restart script for the Tezos node that works better?

Answer 1

As I also noted on the GitHub page of Etom Knudsen, it could be that there is a bug in the getTimeSinceLastBlock function in line 33.

Strangely, the moment I manually restart the node, the CPR script also fires and its log shows the following error message:

Looking for p2p avtivity - will wait for max 90 secs
Waiting. Last block was 510 secs ago)
Waiting. Last block was 515 secs ago)
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.6/json/__init__.py", line 299, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
date: invalid date '+%s'
/usr/local/bin/tezos-node-cpr.sh: 33: /usr/local/bin/tezos-node-cpr.sh: arithmetic expression: expecting primary: "1553324161-"
Network OK. No p2p activity and/or too long ( secs since last block. Restarting node!`

So it seems that there is a bug in the conversion of seconds to string time to be used in python, preventing the script to determine the time since the last block.