search_calls_per_second needs to be dialed down

[edsu]

I was running some fairly simple data retrieval in this Notebook(see the Wayback section) and I discovered that I got completely blocked from accessing web.archive.org! Luckily I remembered that there was Internet Archive's #wayback-researchers Slack channel, where I got this reponse.

Hi edsu - I found your /cdx requests from 4:29UTC. Those requests are limited to an average of 60/min. Over that and we start sending 429s. If 429s are ignored for more than a minute we block the IP at the firewall (no connection) for 1 hour, which is what happened to you. Subsequent 429s over a given period will double that time each occurrence. If you can keeping your api request < 60/minute you will prevent this from happening.

I thought that the openwayback module's defaults would have prevented me from going over the 60 requests per minute (one per second) and I thought wayback's support for handling 429 responses would have backed off sufficiently fast. I suspect that the goal posts on the server side have changed recently because I had code that worked a month or so ago, which stopped working (resulting in the block).

I was able to get around this by using a custom WaybackSession where I set the search_calls_per_second to 0.5, but I suspect 1.0 would probably work better. Maybe the default could me moved down from 1.5 to 1.0?

Also, perhaps there needs to be some logic to make sure to wait a minute when encountering a 429 as well?

[Mr0grog]

Keeping this open to track other adjustments mentioned here to handle things better in general here (e.g. when we hit a 429) rather than just adjusting the limits (done in #140).

This is also tied in with a long-running goal I’ve had to revamp rate limiting, since part of the problem here is that we implement the throttle on calls to search() and get_memento() instead of in the actual send() method (or around calls to it), where it should be happening. (Elsewhere in EDGI code, we have a nice RateLimit object that works like a lock that I’d like to use here, so we can pass a rate limit into send() or otherwise share a single lock around the codebase as needed.)

[Mr0grog]

Sketching out some more detailed thoughts here:

• Should bring in the RateLimit class from web-monitoring-processing and have the session hold two instances of that (one for the CDX server and one for the Memento server, since they do, or at least did, have independent rate limits) instead of just integers.

• How do the above actually get applied? Some options:

  1. Used automatically in send() based on the URL requested. Pros: easy to implement (otherwise there is a layer of indirection between the client methods and send() — they call request(), which prepares a request to send() and I have to think about how to get through that); don’t have to worry about calling code making mistakes. Cons: maybe too magic, hard to override. Doesn’t give us a way to burst for requests that go together (e.g. following redirects from a memento request).
  2. Have the client pass the rate limit to use into the session when making a request. This keeps the current model, where client methods are in charge of what types of requests get which limits, and means we don’t have to magically associate limits with URLs, just with use cases. But we need to figure out how to actually pass that info all the way down to send().
  3. Have the client use the limits directly. This is kind of like today, but we just need to move where the limits get applied to be directly around calls to request() or send() rather than at the start of methods like get_memento() or search(). This requires those high-level methods to be more careful (and probably more error prone), but keeps the application of the limits in the place where we have more contextual information to do interesting things with (e.g. if we wanted to add the ability to burst briefly for requests that go together).

  I suspect (2) may present pretty hard-to-address logistical hurdles, so either (1) or (3) are better (at least until we get rid of requests — see Pool connections across threads and/or make WaybackClient thread-safe #58). I like that (1) just works, but (3) gives us more flexibility.

• Consider adding some kind of burst-ability to the RateLimit object (e.g. for CDX, the real budget is 60/min, not 1/s, so you could do a few requests faster than 1/s as long as you don’t blow the 1-minute budget). I’m not sure this is that useful for CDX, but it would be super useful for mementos. We get around this right now by not implementing rate limits correctly (that is, implementing them at the call to get_memento() and not to the actual requests that method makes).

• Add a more restrictive lock to WaybackSession to handle 429s. The idea here is that a 429 response would lock the session for a little while, and maybe subsequent 429s would lock the session for longer if they have a Retry-After header (IIRC, we do not currently obey Retry-After because we discovered they were frequently a lie and excessively long, but it’s possible that’s no longer true.)

  Note some care needs to be taken here to respect timeouts. If the retry is going to wait for too long, we may just want to abandon early. Or make timeouts predictable by waiting until the timeout and then raising. Dunno. (We may also want to add another value to the tuple version of timeouts that controls the overall timeout, rather than just the connect and read timeouts.)

[Mr0grog]

@edsu while doing some final cleanup on the patch release for this (sorry for the delays; I’ve been a bit distracted lately), I was reading the docstring for RateLimitError and it certainly sounds like I had intended to raise on any 429 response instead of just when we’ve exceeded the maximum number of retries like we currently do. Thinking about it now, it feels like that might be more correct, too.

Raising instead of retrying seemed like a big change for v0.4.4, so for that release I just made a longer retry delay (see #142). But I’m thinking about making it always raise instead of ever retrying (responses with status code 429) in v0.5.0.

As a user of this tool, do you have any thoughts on that? Would you prefer the current behavior (retrying but with a very long delay) instead? (Another option might be to make this configurable, but I’d prefer to avoid that complexity and just choose one behavior or the other.)

[edsu]

Yes, I like raising an exception when a 429 is encountered. I think it will help wayback users avoid a situation where they are getting blocked?

[Mr0grog]

Great, that’s about what I was thinking as well. (I think the reason it was being handled here in the first place was that, at EDGI, we used to use this as part of a nightly bulk processing script that took a few hours to run, and having it automatically handle rate limits would have been useful there since you probably don’t want a job that big to stop in the middle.)