4

When trying to bootstrap a node in a VM on Google Cloud Platform I see the performance running fine for approximately 30 minutes before it completely tails off.

You can see this from the graphs below which show the CPU + IO for the past day on the instance. You can clearly see the point at which I start the tezos-node as everything spikes up. CPU goes up to 20% - about 80% of a single CPU (I guess Tezos doesn't make use of multiple CPUs) and the IOPS steadliy climbs up, peaking at about 3,500 IOPS or 16MB/S.

However after 30 minutes everything just tanks to the point where it takes several seconds for each block to validate. Not sure what's going on here.

This is a n1-highmem-4 instance (4 vCPUs, 26 GB memory), although I have seen the same behaviour on various other instance type in GCP. And can be reproduced very easily on any instance type I have tried.

This one has been running for about 17 hours now and only got to block BL6SwDDDhqHkQEpfGWVBFBZ6LX1Byg4eibHSEvQbmS3bL7Z8b89 (3-Aug)

CPU + Network IO

Disk I/O

Update

After the comment about disk size I provisioned a bigger SSD (200GB) and this appears to resolve the problem. Will do some research to see what the minimum disk size is to achieve the consistent 3,500 IOPS that Tezos seems to need.

Improve this question
4
  • Describe the disk? Did you try increasing the disk size (and throughput) to see if the disk is the bottleneck?
    – Tom
    Apr 8, 2019 at 14:14
  • Make sure you are not running out of inodes. df -i
    – asbjornenge
    Apr 9, 2019 at 7:25
  • @Tom seems the disk is the issue alright, reading up a bit on GCP the disk performance increases linearly with size. The node in the question was the default one (I had provisioned a local SSD but forgot to mount it!). I've provisioned a 200GB SSD and the node is humming along nicely at approx 3,500 IOPS for the last 12 hours
    – xtzbaker
    Apr 9, 2019 at 8:24
  • @Tom nice! Feel free to hit "answer your own question" and write up the correct answer here - so your question get's answered and other people can find the solution more easily 👍
    – asbjornenge
    Apr 11, 2019 at 12:32

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.