Allow overriding MAX_URL_LENGTH via environment variable

[ruimartinsptl]

Motivation

Currently, HTTPX enforces a hardcoded MAX_URL_LENGTH (default value: 65536) inside the file httpx/_urlparse.py.

In some exceptional use cases, users may need to send requests with longer URLs.

I have a use case that if uses requests, it works well, but using httpx, I have an error URL too long

Proposal

Allow overriding the MAX_URL_LENGTH constant via an environment variable, e.g., HTTPX_MAX_URL_LENGTH.
(If the environment variable is not set, the default value (65536) would be used)

Example implementation:

Replace this:

MAX_URL_LENGTH = 65536

by this:

import os
MAX_URL_LENGTH = int(os.environ.get("HTTPX_MAX_URL_LENGTH", 65536))

Alternatives considered

• Monkey patching the constant at runtime (not reliable and not recommended)

import httpx
import httpx._urlparse
import sys
httpx._urlparse.MAX_URL_LENGTH = sys.maxsize

• Forking the library and changing the value manually (not maintainable)
• Override the variable via a new argument in the httpx.Client (and httpx.AsyncClient) constructor (several disadvantages)

Benefits

• Makes HTTPX more flexible for advanced users and testing scenarios
• No impact on default behavior for most users

Would this change be welcome? If so, I can open an issue (and then a pull request)

Thanks!

[lovelydinosaur]

You're welcome to expand on your use case?

[ruimartinsptl]

Thank you for your reply!

My use case:

We have a microservices architecture, and due to a limitation in one of our services (which is not easy to fix at the moment), there is a specific HTTP GET endpoint that requires sending a very large list of IDs as query parameters.

For example, with 2 IDs:

GET /users?user_id=8b72185e-a4cf-419b-b9d7-7518e0ebbd89&user_id=04f26298-ce3c-48fe-8ba9-e9537c297fab&...

At this time, splitting this request into multiple smaller requests is not an option, it must be a single, large request (more than 65536).

When I use the requests library, this works perfectly, the request is sent, and the server handles it.

However, with httpx library, I get a client-side "URL too long" error before the request is even sent, due to the hardcoded MAX_URL_LENGTH.
If I increase the value of MAX_URL_LENGTH in the file /Users/rui/miniconda3/lib/python3.12/site-packages/httpx/_urlparse.py it will work well
(I also tested with the "Monkey patching" solution presented above and also worked well, but is easier to configure in our Dockerfile if we can define it directly as an Environment Variable)

Let me know if you need more details.

Thank you

[ruimartinsptl]

I'll add here the scripts that I used for testing, then you can reproduce,

First, create a server inside app.py:

from fastapi import FastAPI, Request
from typing import List, Optional

app = FastAPI()

@app.get("/test")
def test_endpoint(request: Request, id: Optional[List[str]] = None):
    ids = id if id is not None else []
    return {
        "ids_length": len(ids), 
        "url_length": len(str(request.url)),
    }

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000) 

You can start the server:

• with uvicorn (python app.py)
• or with gunicorn (gunicorn -w 4 -k uvicorn.workers.UvicornWorker app:app --bind 0.0.0.0:8000), as you prefer

Now test the httpx limits:

import httpx
BASE_URL = "http://localhost:8000/test"

MAX_IDS = 10_000_000
STEP = 1_000

num_ids = STEP
while num_ids <= MAX_IDS:
    ids = [f"id={i}" for i in range(num_ids)]
    url_params = '&'.join(ids)
    url = f"{BASE_URL}?{url_params}"
    print(f"\nTesting with {num_ids} query parameters (URL length: {len(url)})")
    try:
        response = httpx.get(url)
        if response.status_code == 414:
            print(f"Hit limit on server: {num_ids} parameters (status 414 URI Too Long)")
            break
        elif response.status_code >= 400:
            print(f"HTTP Error on server {response.status_code} with {num_ids} parameters")
            break
        else:
            print("OK")
    except Exception as e:
        print(f"Client Exception: {e}")
        break
    num_ids += STEP 

Results:

$ python test_url_length_httpx_65536.py
Testing with 1000 query parameters (URL length: 6916)
OK

(...)

Testing with 8000 query parameters (URL length: 62916)
OK

Testing with 9000 query parameters (URL length: 70916)
Client Exception: URL too long

---

If we test the same code with requests, we won't have problems:

import requests

BASE_URL = "http://localhost:8000/test"

MAX_IDS = 10_000_000
STEP = 1_000

num_ids = STEP
while num_ids <= MAX_IDS:
    ids = [f"id={i}" for i in range(num_ids)]
    url_params = '&'.join(ids)
    url = f"{BASE_URL}?{url_params}"
    print(f"\nTesting with {num_ids} query parameters (URL length: {len(url)})")
    try:
        response = requests.get(url)
        if response.status_code == 414:
            print(f"Hit limit on server: {num_ids} parameters (status 414 URI Too Long)")
            break
        elif response.status_code >= 400:
            print(f"HTTP Error on server {response.status_code} with {num_ids} parameters")
            break
        else:
            print("OK")
    except Exception as e:
        print(f"Exception: {e}")
        break
    num_ids += STEP 

Results:

$ python test_url_length_requests.py

Testing with 1000 query parameters (URL length: 6916)
OK

(...)

Testing with 31000 query parameters (URL length: 267916)
OK
^C

---

Now, applying the Monkey patch to replace the MAX_URL_LENGTH

import httpx
import httpx._urlparse
import sys
# httpx._urlparse.MAX_URL_LENGTH = 65536
# httpx._urlparse.MAX_URL_LENGTH = 100_000
httpx._urlparse.MAX_URL_LENGTH = sys.maxsize

# print(httpx.__file__)

BASE_URL = "http://localhost:8000/test"

MAX_IDS = 10_000_000
STEP = 1_000

num_ids = STEP
while num_ids <= MAX_IDS:
    ids = [f"id={i}" for i in range(num_ids)]
    url_params = '&'.join(ids)
    url = f"{BASE_URL}?{url_params}"
    print(f"\nTesting with {num_ids} query parameters (URL length: {len(url)})")
    try:
        response = httpx.get(url)
        if response.status_code == 414:
            print(f"Hit limit on server: {num_ids} parameters (status 414 URI Too Long)")
            break
        elif response.status_code >= 400:
            print(f"HTTP Error on server{response.status_code} with {num_ids} parameters")
            break
        else:
            print("OK")
    except Exception as e:
        print(f"Exception: {e}")
        break
    num_ids += STEP 

Result:

$ python test_url_length_httpx_mp.py

Testing with 1000 query parameters (URL length: 6916)
OK

(...)

Testing with 31000 query parameters (URL length: 267916)
OK
^C

---

Conclusion:

• I can override the limit by applying monkey patching, but it is not reliable because future updates of HTTPx can break my code
• Having an official way to override this limit (e.g., via environment variable) would make HTTPX more robust and flexible for advanced/test scenarios, and would avoid relying on fragile monkey patching.

Thank you