3

I have a local version of the TzKT indexer and API (v1.11.1) running for mainnet on a server without any apparent issues. I am running a program that retrieves all transactions in the history of the blockchain in batches of 10K by querying the indexer with the following URL:

http://[indexer]:5000/v1/operations/transactions?limit=10000&offset=[X]

Where [indexer] is just a placeholder for the real domain (or localhost if I use it from the server itself) and [X] increases by 10K for each consecutive query.

For initial queries where offset is still relatively small, the execution time is below 8 seconds, which I find acceptable. However, when coming into higher values like offset=10100000 the execution times go up to around 3 minutes, while the same query on https://api.tzkt.io/ takes about 30 seconds. Is there a way to prevent the query times from increasing so much?

1 Answer 1

3

I'd recommend you to paginate over the data with a cursor, instead of the offset, when it's possible. It will be much faster, because it will enable DB to do index scan, instead of sequential scan over the whole table.

In your case you can use ?id.gt={lastReceivedId} as a cursor. For example:

As you can see, there can be multiple ways to get the same data from the API, so if some query seems slow, just try a different approach. A general advice - avoid sequential scan in case of huge tables, and always try to make the DB do index scan.

As for the performance difference between your instance and api.tzkt.io, it's just a matter of 1) hardware setup and 2) DB settings optimization. So, try to improve your hardware setup and try to optimize your DB, according to your hardware and use case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.