Performance Optimization for eth_getLogs/eth_getFilterLogs with Large-scale Query Parameters

[0xbigapple]

Background

eth_getLogs and eth_getFilterLogs are interfaces for querying smart contract event logs. To quickly find blocks containing target events, these interfaces use Bloom filters to index data. The partialMatch method searches the Section-Bloom database using these filters. However, when query conditions contain a large number of topics or addresses (e.g. querying TRC20 transfers for multiple addresses or querying multiple contracts for a single address), it causes many duplicate database queries and slows down the response. While version 4.8.0 added limits on topic counts and block ranges to prevent long request times, these restrictions force users to split their queries into multiple smaller requests.

We have optimized the partialMatch method to avoid duplicate database queries and improve interface performance. This optimization allows nodes to support larger queries, reducing the number of requests users need to make.

Rationale

Why should this feature exist?

• Large queries are slow due to duplicate database operations. Users have to split their queries into smaller ones to work around current limits, which is not user-friendly.

What are the use-cases?

• NFT exchanges & DEX: Monitor Transfer events for multiple tokens
• Data analysis platforms: Batch query historical events for analysis and visualization

Specification

Let's take eth_getLogs as an example to see how event log queries locate target blocks:

1. Converts query conditions (address, topics) into Bloom filter positions
2. For each section (2048 blocks), call partialMatch to find possible blocks
3. The partialMatch method queries Section-Bloom database for matching blocks

Due to Bloom filter properties, duplicate queries occur: with 2048 total bits and 3 bits per condition, when condition count reaches 683 (683 * 3 > 2048), bit collisions are inevitable, leading to duplicate database queries.

Test results using random from-address from USDT transfer records:

Test Set	BitIndex Duplication Rate
10 topics	0%
100 topics	7%
500 topics	28.8%
1000 topics	48.4%

Optimization of partialMatch method:

1. Preprocess bitIndexes[][] to extract unique bitIndexes
2. Query database only once for each unique bitIndex
3. Reassemble final BitSet using query results

This reduces IO operations and improves query speed.

Test Specification

• Verifies result consistency before and after optimization
• Compares performance with same inputs

Scope Of Impact

Affected Modules:

• JSON-RPC interface layer (EthApi)

Affected interfaces:

• eth_getLogs
• eth_getFilterLogs

Implementation

Do you have ideas regarding the implementation of this feature?
Y

Are you willing to implement this feature?
Y

[jwrct]

@0xbigapple After this optimization, what is the max query that the node can support? And how is its performance?

[0xbigapple]

@0xbigapple After this optimization, what is the max query that the node can support? And how is its performance?

Hi, @jwrct The max query depends on the node machine's configuration, and I haven't specifically tested this yet. Based on my current data, comparing the execution times before and after optimization for the partialMatch method, the response time shows an almost direct positive correlation with the bit index repetition rate. I will update here when I have more detailed test results.

[waynercheung]

@0xbigapple It's a good idea to extract unique bitIndexes before fetching it from DB, but I also have questions as belows:

• Will it increase memory occupation?
• In partialMatch, it uses Future to fetch data from DB in parallel, in your optimization, is there any change about it?
• Your solution maybe work for request with large topics which will increase the BitIndex Duplication Rate, but for normal request, does it has some negative impact? After all, the count of subtopics for most requests are less than 20, so it is also important to ensure the performance of normal requests.
• The parameter also has addresses except the topics, do you have also considered it?

comparing the execution times before and after optimization for the partialMatch method

Besides, looking forward to your test results about the execution times you mentioned above.

[waynercheung]

Test results using random from-address from USDT transfer records:

Test Set BitIndex Duplication Rate
10 topics 0%
100 topics 7%
500 topics 28.8%
1000 topics 48.4%

@0xbigapple Hi, about this Duplication Rate, can you explain more detail about how do you calculating the rate? or what does the rate mean?

[0xbigapple]

@0xbigapple It's a good idea to extract unique bitIndexes before fetching it from DB, but I also have questions as belows:

• Will it increase memory occupation?
• In partialMatch, it uses Future to fetch data from DB in parallel, in your optimization, is there any change about it?
• Your solution maybe work for request with large topics which will increase the BitIndex Duplication Rate, but for normal request, does it has some negative impact? After all, the count of subtopics for most requests are less than 20, so it is also important to ensure the performance of normal requests.
• The parameter also has addresses except the topics, do you have also considered it?

comparing the execution times before and after optimization for the partialMatch method

Besides, looking forward to your test results about the execution times you mentioned above.

Hi, @waynercheung Thanks for your discussion

Let me address your concerns:

• About memory usage: we use HashSet to store unique bit indexes with a maximum size of 2048, which means minimal memory overhead that is released immediately after query completion.
• About parallel processing: the partialMatch method still maintains its original parallel execution using Futures with a default thread pool size of 5, which remains unchanged in this optimization.
• About performance impact: this optimization is particularly effective for queries with large numbers of topics or addresses (especially since there's no limit on address list size). For typical requests with less than 20, the extra cost is tiny since it only loops less than 60 times. My tests running 100 times for 5 rounds showed no noticeable slowdown for these small queries.

partialMatch execution times: 10 topics test was performed locally and the results might be slightly affected by other running programs on the machine.

Round	Original(ms)	Optimized(ms)
1	1.79	1.69
2	2.61	2.32
3	2.29	2.06
4	2.61	2.18
5	2.10	2.38
Avg	2.28	2.13

[0xbigapple]

Test results using random from-address from USDT transfer records:
Test Set BitIndex Duplication Rate
10 topics 0%
100 topics 7%
500 topics 28.8%
1000 topics 48.4%

@0xbigapple Hi, about this Duplication Rate, can you explain more detail about how do you calculating the rate? or what does the rate mean?

The Duplication rate shows the percentage of repeated bit indexes. For example: if 1000 topics generate 3000 bitIndexes in total but only 1000 are unique, the duplication rate is 1 -1000/3000 = 66.67%.

[waynercheung]

The Duplication rate shows the percentage of repeated bit indexes. For example: if 1000 topics generate 3000 bitIndexes in total but only 1000 are unique, the duplication rate is 1 -1000/3000 = 66.67%.

Got it, thanks.

[0xbigapple]

@waynercheung @jwrct The following results(partialMatch execution times) were obtained from local testing on my Mac. The average value with 5 rounds(100 times per round).

Metric	10 Topics	100 Topics	500 Topics	1000 Topics	2000 Addresses
Original (ms)	2.28	2.51	9.35	21.41	40.15
Optimized (ms)	2.13	2.13	6.95	12.80	15.24
Improvement Rate	6.58%	15.14%	25.67%	40.21%	62.04%
Duplication Rate	0%	7.60%	29.41%	47.31%	67.62%

[jwrct]

@0xbigapple Great!

While version 4.8.0 added limits on topic counts and block ranges to prevent long request times, these restrictions force users to split their queries into multiple smaller requests.

What are the limits on the number of topics and block ranges in version 4.8.0? Are there plans to relax these limits with this optimization?

[0xbigapple]

@0xbigapple Great!

While version 4.8.0 added limits on topic counts and block ranges to prevent long request times, these restrictions force users to split their queries into multiple smaller requests.

What are the limits on the number of topics and block ranges in version 4.8.0? Are there plans to relax these limits with this optimization?

@jwrct The default value is followed, but node can customize it in the configuration file, we have no plans to modify the default value at this time.

    PARAMETER.jsonRpcMaxBlockRange = 5000;
    PARAMETER.jsonRpcMaxSubTopics = 1000;