Import all torrents statistics from the tracker (even with high load and many torrents)

[josecelano]

Discussed in #428

^{Originally posted by josecelano January 4, 2024}
The Index has a cronjob called "statistics importer". It imports torrent statistics from the associated tracker. The default configuration executed the process every hour:

[tracker_statistics_importer]
torrent_info_update_interval = 3600
port = 3002

This tasks:

1. Gets all torrents from the database. A compact version with only the internal ID and the infohash (SELECT torrent_id, info_hash FROM torrust_torrents).
2. It iterates over all torrents requesting the statistics from the Tracker REST API.

That solution has some problems:

• We do not have the same torrents in the Index and the Tracker. The tracker could have 1 torrent and the Index 1000 or the opposite. If the tracker only has one we are making 999 extra requests. The opposite case is not a problem because we are only making one request.
• If we have too many torrents in the Index making all the requests could take more the the interval you set in the configuration (default to 1h). That means it could take more than 1 hour to import all the statistics.

It seems that the process does scale well and it's not even efficient. There are different solutions depending on which aspect you want to improve. For example, if you want to scale it you can add an importation date and just fetch statistics continuously only importing the torrents that have not been imported in the last hour. This solution does not solve the problem of making request to the tracker for torrents it does not have.

Proposed solution 1

A naive implementation could be:

• Ask the tracker for the torrent statistics that have changed in the last hour.
• Use pagination for that response. The tracker could have millions of torrents.
• The index only has to process the tracker torrents and update the statistics.

Proposed solution 2

We could also not import statistics at all for all torrents and only import statistics for a single torrent when the users load the torrent details page. That would require an extra request to the tracker every time you load the page. It could work if we do not need statistics for other cases.

Other solutions

• Use other types of APIs where you can subscribe to changes in statistics.
• Use message queues to subscribe to statistics changes events.
• Etcetera

I will go with solution 2 and that would make the tracker API even simpler. This is related to using a DashMap for the tracker data. We would not need to use a BTreeMap to order torrents, because right now we only need it for the API. In addition, background tasks are hard to maintain:

• It's hard to detect problems.
• There can be deadlock, overlapping, ...
• Performance can be a problem as I described.
• ...

cc @da2ce7 @WarmBeer @mario-nt @grmbyrn

[josecelano]

We have now 9945 torrents on the live demo and we are having this nice pattern:

I suppose it's the importer running every hour.

[josecelano]

Hi, today @da2ce7 and I were discussing a solution. @da2ce7 proposed:

• Make sure we don't overlap executions of the importer. I think the current solution does not overlap executions. It simply waits one hour after finishing the importation. If the process takes less than 1 hour statistics will be imported every hour. If the process takes longer, for example, 3 hours, then statistics will be imported every 4 hours (3 hours for the process + 1 hour waiting for the next tick). I guess the intention was to update statistics at least once in an hour, right @WarmBeer? In that case, maybe we could just wait the remaining time between 1 hour and the duration of the importation process. @WarmBeer @da2ce7 ?
• We can also improve the logs. See Overhaul logging #468 (comment)
• We should also take advantage of threads. @da2ce7 proposed to import a batch of torrents at the same time using tokio-spawned tasks. We are making a single request per torrent. In the future, we could add a new endpoint to the tracker to get the statistics for more than one torrent at the same time.
• We should use pagination for the query to get all torrents from the Index. We can take for example 50 and import them in parallel. And then continue with the next page (order by the implicit DB table order).
• If there is a problem with the tracker connection while importing a torrent, the current behavior is just resetting statistics for the torrent in the Index and trying again in the next importation.

        let interval = std::time::Duration::from_secs(torrent_info_update_interval);
        let mut interval = tokio::time::interval(interval);

        interval.tick().await; // first tick is immediate...

        loop {
            interval.tick().await;

            info!("Running tracker statistics importer ...");

            if let Err(e) = send_heartbeat(importer_port).await {
                error!("Failed to send heartbeat from importer cronjob: {}", e);
            }

            if let Some(tracker) = weak_tracker_statistics_importer.upgrade() {
                drop(tracker.import_all_torrents_statistics().await);
            } else {
                break;
            }
        }

Relates to: #468 (comment)

[josecelano]

It seems that as the number of torrents increases the server is having more problems importing all the statistics in 1 hour.

We have now 16823 torrents.

[josecelano]

We have 27570 torrents in the live demo.

It seems the server is always busy finally.

[josecelano]

By the way @WarmBeer @da2ce7 @mario-nt what do you think about my Proposed solution 2: import statistics on the fly.

We are importing statistics for all torrents every hour even if maybe we do not need them because we do not have users in the Index or the users are only interested in 20% of the torrents. As far as I know we only show the statistics on the list and details page.

We could:

• Remove statistics from the Index database.
• Import statistics only when we need them (list and detail pages).
• We can use a in-memory cache valid for one hour. Before fetching them from the tracker we check if we have fresh data in the cache (less than 1 hour).

Pros:

• If the Index does not have users we don't overload the Tracker.
• We only get the data we use. If only 20% of the torrents are listed or viewed in detail we only import statistics for those torrents.

Cons:

• Assuming a high load in the Index and users interested in 100% of the torrents equally distributed, it takes longer to build the response because it's not a direct SQL query. But this should not be too slow.

[josecelano]

Other potencial solutions that can be combined from ChatGPT:

To improve the efficiency and speed of importing statistics from your BitTorrent Tracker to your BitTorrent Index, especially given the large scale of torrents and the potential mismatch between the two, you can consider several strategies. Each of these methods leverages different aspects of system design and algorithmic efficiency:

1. Batch Processing with Enhanced API

• Idea: Enhance the Tracker's API to support batch processing. This means enabling the API to accept requests for statistics of multiple torrents at once, rather than one at a time.
• Implementation: Update the Tracker's API to allow for batch requests. You can set a reasonable limit on the number of torrents per request to balance load.
• Advantages: Reduces the number of API calls drastically, decreasing network overhead and API load.

2. Differential Update Strategy

• Idea: Instead of importing all torrent statistics every hour, determine which torrents have likely changed and update only those.
• Implementation: Implement a mechanism (like a timestamp or a change log) on the Tracker to identify torrents that have been updated since the last import. The Index then requests statistics only for these torrents.
• Advantages: Significantly reduces the amount of data transferred and processed.

3. Using Bloom Filters

• Idea: Use Bloom filters to quickly check if a torrent in the Index is also in the Tracker, and update statistics accordingly.
• Implementation:
  • The Tracker maintains a Bloom filter of all its torrents.
  • The Index queries this Bloom filter to check if its torrents are in the Tracker before making an API call.
  • Implement a scheduled task to update the Bloom filter periodically.
• Advantages: Bloom filters are space-efficient and fast for membership checking, which reduces unnecessary API calls for torrents not in the Tracker.

4. Webhooks or Push Mechanism

• Idea: Instead of pulling data from the Tracker, have the Tracker push updates to the Index.
• Implementation: Implement a webhook system in the Tracker that sends updates to the Index whenever torrent statistics change.
• Advantages: Real-time updates and reduced load on both systems, as data is transferred only when there is a change.

5. Database Replication or Shared Access

• Idea: If both services can securely access a shared database, replication or direct querying could be used.
• Implementation: Implement a shared database or a replication mechanism where the Tracker updates a shared dataset that the Index can read directly.
• Advantages: Eliminates the need for API calls and allows the Index to directly query the data it needs.

6. Caching and Incremental Updates

• Idea: Cache statistics in the Index and only request updates for those statistics.
• Implementation: Implement a caching layer in the Index that stores the most recent statistics. Periodically, request updates for these statistics from the Tracker.
• Advantages: Reduces API calls to only those torrents whose cached data is out of date.

Conclusion

Each of these strategies has its advantages and can be combined for optimal performance. Given that both services are implemented in Rust, you can leverage Rust's performance and concurrency features to implement these solutions efficiently. The choice of strategy will depend on your specific requirements, such as the frequency of updates, the volume of data, and the infrastructure setup of your Tracker and Index.

[mario-nt]

I liked the idea of sharing a database, but that option couples the tracker and the index completely.

My favourite one is number 4: Webhooks or Push Mechanism, as that way, we only push the data once it is updated, but it may still be slow if there are a lot of changes.

And bloom filters look good too.

[josecelano]

By the way @WarmBeer @da2ce7 @mario-nt what do you think about my Proposed solution 2: import statistics on the fly.

We are importing statistics for all torrents every hour even if maybe we do not need them because we do not have users in the Index or the users are only interested in 20% of the torrents. As far as I know we only show the statistics on the list and details page.

We could:

• Remove statistics from the Index database.
• Import statistics only when we need them (list and detail pages).
• We can use a in-memory cache valid for one hour. Before fetching them from the tracker we check if we have fresh data in the cache (less than 1 hour).

Pros:

• If the Index does not have users we don't overload the Tracker.
• We only get the data we use. If only 20% of the torrents are listed or viewed in detail we only import statistics for those torrents.

Cons:

• Assuming a high load in the Index and users interested in 100% of the torrents equally distributed, it takes longer to build the response because it's not a direct SQL query. But this should not be too slow.

It's not possible because we allow users to sort torrents by the number of seeders/leechers

So we need that info in the database to run the query.