4

I have a private node and a public node. I have added each other address in respective trust address list using below command:

./mainnet.sh admin-client trust address address_to_trust:9732

Initially the private node won't connect to public node so i have to use connect command from public node as mentioned below:

./mainnet.sh admin-client connect address address_to_connect:9732

Now my private node is connected to my public node and start syncing the blockchain. But the connection breaks after sometime automatically and syncing stops. This has happened thrice in 2 days.

I am not confident enough to start baking in private node if the only single connection breaks automatically.

Anything else i should do to ensure 24x7 connection uptime?

4

Update 3/21/19 Core developers had pushed out a fix as of 366f64f3df266cf02a06412d6760f73626d0a2bf commit on the mainnet branch that addresses this issue that I described below.

In the setting where you have a node in --private-mode (to bake) that connects itself to front (public) nodes, you must explicitly set your front nodes to trust your private node (using --peer argument of tezos-node or the config file of tezos-node or RPCs)! Else, there is a chance that at some point your front nodes decide to disconnect themselves from your private node in favour of a fresh new peer. The bug was that there were cases of disconnection of trusted-private-peers (not really visible because nodes used to favour old connections over new ones but still). we've now shielded those connections.

Essentially, there is a new feature that allows you to set your private node as a peer on your public nodes. That will favor your private node over new peers on your public node.


Original Answer

This problem has been around since the beginning. The problem stems from that the private node is treated the same as every other peer in your public node(even setting one as trusted). Basically it's not prioritizing your private node higher and sometimes your public node may decide that your private node is less "useful" than others. I mean if you think about it, private nodes are just leechers, not really contributing to your public node(s) so your public node might disconnect your private node for another node in the network. Also if your private node somehow "misbehaves" such as not functioning correctly or taking up too much resources, your public node might even graylist it and not allow connections from your private node. Though you would expect setting as trusted should alleviate that!

Most of us early bakers solve this problem by increasing the number of redundancies(public nodes) so when one disconnects it's not a big deal and it should reconnect sooner or later. I think in general, this is a good idea as you'll likely need to upgrade your nodes and having no redundancy will make the process much trickier. These days, I actually find that my private node almost never disconnects unlike the early days. My suspicion is that the private node might have gain "credibility", kind of opposite of being graylisted.

Until a new feature is added that can ensure an undisrupted prioritized connection, your best options are really limited to bringing up more public nodes or adding the foundation nodes(likely trustworthy) as your trusted nodes. One thing you could try is quicksync or copy the chain data to your private node so you don't have to bootstrap your private node. It's possible that during the syncing process, it's getting disconnected due to the resources it's requiring. The problem could be your private node's workers, not your public node disconnecting it.

| improve this answer | |
  • Can you specify list of trusted foundation nodes? – Sachin Tomar Mar 8 '19 at 5:47
  • how about running a script on our public node that keep checking the connection status with private node and if it finds no connection it runs a command to connect to private node automatically. Does this seems to be the solution apart from having multiple public nodes? – Sachin Tomar Mar 14 '19 at 6:59
  • @SachinTomar that won't work because private nodes will reject external connections. Maybe it can work if you detect it on the private node side? – Frank Mar 14 '19 at 23:01
  • Actually i am facing opposite of what you said. When i run ./mainnet.sh admin-client connect address private-node-ip:9732 from public node, it creates a connection to private node. But same command on private node with public-node ip in command fails with message "Cannot connect. Node is running in private node" NOTE: i have added the public-node ip in peer list in private node. – Sachin Tomar Mar 15 '19 at 4:27
4

While this does not directly answer your question, Kiln has a feature that can notify you when your Private Node loses connections.

When you configure Kiln to monitor the Private Node, include a 'minimum number of peers' in the UI. If your Private Node's peers drop below that number, you'll receive a notification.

https://gitlab.com/obsidian.systems/tezos-bake-monitor

| improve this answer | |
  • Thanks. Setting monitoring and alerting really helps to ensure our node is running in healthy state. I don't need to check my node again and again now. – Sachin Tomar Mar 14 '19 at 6:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.