What are some of the methods people are using to make sure a baking node remains up and running 24/7? I see all these baking services with "enterprise server solutions", whatever that means. However, when node falls over, it still needs someone to fix/restart etc. What if the owner of the node is not currently available or perhaps had to travel to a different country. Would SSH remote access be an option? What are some of the ways people reduce risks here? Thanks
We are or hire system architects who design a HA (high availability) infrastructure for our bakers 😉 You can see some discussions here. SSH is definately one tool in the toolbelt here, but if you want a serious HA architecture you need to find skilled people to build and operate it 👍
A lot of us bakers old timers depend on the work done by etomknudsen called "Tezos Node CPR".
Main repo has systemd scripts to let systemd manage and log the tezos binaries. Systemd does a great job keeping services up as it's battle tested in enterprise level services, it handles process dependencies and restarting of services on errors.
The Tezos Node CPR is an optional service that monitors and restarts the above Tezos services as it detects problems such as:
- Node hasn't gotten new blocks for N seconds
- Network disconnected
- RPC on the node is down
The common problem it solves is IP changes due to dynamic ip. The node itself doesn't have logic to "reset" itself into a working state once the node becomes an invalid peer. The only way to correct the problem is by restarting the process.
That said, Tezos node CPR doesn't replace an actual administrator as baking may require an update to the binaries at any moment due to hotfixes and features upgrades. Baking is not a complete passive endeavor.