Extremely slow node sync on alphanet

Question

So, I've set up a Tezos node on an AWS EC2 t2.medium instance. I followed the instructions here, but for alphanet instead of mainnet.

I got as far as: ./tezos-node run --rpc-addr :8732

It's running, and syncing, but VERY slowly. In a couple of hours of running, the data returned from 'client get timestamp' has moved forward only a day or two. I'm also frequently getting messages like p2p.maintenance: Too few connections (5)

I have tried opening ALL ports into and out of the instance just to make sure no traffic was being blocked. It makes no difference. What's wrong? What do I need to change?

Answer 1

You could try adding more peers if you are seeing the too few peers message. This script should work for the alphanet (you need jq installed)

for j in 0 1; do
  for i in `curl -s "http://api.alphanet.tzscan.io/v3/network?state=running&p=$j&number=50" \
    | jq -r '.[] | .point_id' | xargs`; do

    # handle ipv4 or ipv6
    numparts=$(echo $i | awk -F: '{print NF}')
    basenum=$((numparts-1))
    port=$(echo $i | cut -d: -f$numparts)
    base=$(echo $i | cut -d: -f1-$basenum)
    formatted="[$base]:$port"

    echo "Connecting $formatted..."
    ~/tezos/tezos-admin-client connect address $formatted
  done
done

Credit goes to creator of this script.

Answer 2

T2 instances are general purpose compute instances which have low to moderate network performance and are not optimised for Input/Output operations (IOPS). Tezos is very IO intensive by all accounts.

Try spinning up a Storage Optimised instance (H1/I3/D2) and I suspect it will provide better performance.

See - https://aws.amazon.com/ec2/instance-types/

Answer 3

solution : instance type changing

while syncing the node for Alphanet I have tried multiple instances : from T2.micro to T3.xlarge.

At some point I thought that RAM size or Network performance may play a role here. But Even T3.xlarge instance did not bring the whole node synced fast.

What really helped is changing the instance types periodically. You may have noticed that node is syncing much faster in very beginning, when it just started. Then, after some time, it became very slow again.

I've made an observation that even bigger AWS instance type won't allow you to finish this operation fast in one take .

The plan may be:

1. Stop the baker, endorser, accuser and then stop the node itself
2. Stop the t2.medium instance
3. Change the type of your instance to t2.small
4. Start the t2.small instance
5. Start the node and then baker, endorser, accuser. While starting these processes do not forget to redirect the output to log files respectively: tezos.log, baker.log, endorser.log and accuser.log
6. Start watching how fast new blocks are getting synced on a new instance. Use tail -f tezos.log . You have to remember the block that new instance has started from
7. Leave it alone for sometime. You may want to come later and see how many blocks have been synced since you started the node. If I recall correctly, it may sync very fast up to 10 000 blocks or so, though it depends on the instance you have chosen. Instead of t2.small, you may have selected t2.large for instance.
8. When sync process will eventually slow down, repeat operation again. This time migrate from t2.small to t2.medium. It will give you another 10k blocks synced fast.

This approach worked, though it required some manual interventions.

PS: for better results you may use t2.large + t2.medium as a changing pair, not t2.small + t2.medium as described above. But the difference won't be significant.