AO3 gives 403 error with httpx on Windows, even with spoofed User-Agent

[RodneyWarrick]

I have found that, recently, attempting to get archiveofourown.org (AO3) URLs from httpx on Windows invariably returns a 403 error (with a CloudFlare "prove you're not a bot" page). Requests and curl don't seem to have this issue. I couldn't get this to happen on Linux.

Windows 11
Python 3.13
httpx 0.28.1
requests 2.32.5

Test script:

#!/usr/bin/env python3

import httpx
import requests

TEST_URL = 'https://archiveofourown.org/tags/721553/feed.atom'

HEADERS = {'User-Agent' : 'python-requests/2.32.5'}

print("requests result:", requests.get(TEST_URL).status_code)

print("httpx result:", httpx.get(TEST_URL).status_code)

print("httpx (requests UA) result:", httpx.get(TEST_URL, headers=HEADERS).status_code)

Output for me on Windows:

requests result: 200
httpx result: 403
httpx (requests UA) result: 403

[lovelydinosaur]

That's disappointing.

Wondering if httpx is being misused by over prolific crawlers etc. I doubt we'll be able to get any useful feedback from cloudflare around problematic behaviours they might be seeing.

Might be a good pointer towards having some default rate limiting built-in perhaps?