Stream error after 5 minutes

[EthanBlackburn]

Describe the bug

Streaming output from gpt4 consistently fails with Connection broken: InvalidChunkLength(got length b'', 0 bytes read) for prompts that trigger long-form responses. I've been able to reproduce this by prompting gpt to rewrite a long document.

A few observations

1. This failure occurs exactly at the 5 minute mark every time, which leads me to believe there's a timeout somewhere.
2. The python implementation of stream reader assumes that an empty string will never be returned by the server.
3. I sent the same request using a non-streaming client with a timeout at 10min. The timeout wasn't triggered until the 10min mark,so this problem seems isolated to streaming

Code snippet from urllib3/response.py:

def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()  # self._fp.fp.readline() returns b''
        line = line.split(b";", 1)[0]
        try:
            self.chunk_left = int(line, 16)  # this raises a value error
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise InvalidChunkLength(self, line)

It seems like a problem on the server side but Im not confident

To Reproduce

1. Create gpt4 streaming client
2. Prompt it to rewrite a very long text. For example, verses 1:1 -> 4:10 in the book of genesis

Code snippets

No response

OS

macOs

Python version

Python 3.10.9

Library version

v.0.26.4

[karim-attia]

I have the exact same issue. I can generate a little over 2'000 tokens via GPT-4 stream and I stop receiving stream updates after exactly 300 seconds.

Running on something similar as this code: https://github.com/Nutlope/twitterbio/blob/e799871764e559cd96608bfefcc3976e7daf346a/utils/OpenAIStream.ts#L26

[oomti]

The default timeout of the connection with aiohttp library(that openai librayr utilizes) is 300 seconds.

Have you changed the timeout period when you send your request?

It's an optional parameter timeout_period and you set it's value in seconds.

[oomti]

@EthanBlackburn if you provide me a generator prompt/model-name for a long text I can test in my custom API client if the connection is closed from server side or it's an issue with the openAPI python library.

[karim-attia]

@oomti This prompt should exceed 5min with GPT-4.

Please write a JSON that counts from 1 to 500 like this:
{
"One": 1,
"Two": 2,
}

[oomti]

Thanks @karim-attia

Tested with a 600seconds timeout on my part.

After 5 minutes the stream has ended with the following error message:

openai-chatgpt\openai\Lib\site-packages\aiohttp\streams.py", line 304, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

This indicates that the socket from the serverside is disconnected after 5 minutes, which is probably a default setting.

[oomti]

Doublechecked and tested with various settings -
timeout=600, stream=true - connection closed in 5 minutes
timeout=600, stream=false - connection closed in 5 minutes(so won't receive a response)

Also tested with the official web UI - completes only 87 lines and closes response after(there might be other limitations there)

I think the large token count was designed with a larger input in mind and a large sized output was not considered.

[yannoth]

Same bug here

[ficapy]

same here

requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

________________________________________________________
Executed in  301.97 secs    fish           external
   usr time    2.23 secs    0.09 millis    2.23 secs
   sys time    1.41 secs    1.76 millis    1.40 secs

[KaedinLian]

I met the same issue too with 4000 toknes prompt and asking near 4000 toknes.
timeout=1000, stream=true - connection closed in 5 minutes (call: invalid chunk Connection broken: InvalidChunkLength(got length b'', 0 bytes read)
timeout=1000, stream=false - connection closed in 5 minutes(so won't receive a response)

[jizs]

We find out this is not a bug of client, but a bug of the openai server

[wdeng]

I have the exact same error, and interact with ChatGPT and documentation, it seems to be a server side error.

"
From the information you've provided, it does indeed seem that there's a timeout issue with the OpenAI GPT-4 streaming API that's closing the connection after exactly 5 minutes (300 seconds). This could be due to a server-side setting.

As for the error you're encountering, InvalidChunkLength(got length b'', 0 bytes read), it's likely that when the server-side socket disconnects after 5 minutes, the client is still trying to read from the connection and encountering an empty string, which leads to this error. This seems to be supported by the fact that the error is thrown when trying to parse an empty string as an integer, which is invalid.

However, without having access to the underlying implementation of the OpenAI API, it's hard to know for sure. It's also worth noting that as an AI developed by OpenAI, I don't have the ability to access or manipulate the OpenAI servers in real-time.

As I recommended earlier, the best course of action would be to reach out to OpenAI's support or raise an issue on the OpenAI GitHub repository. They would be best equipped to address this issue and potentially provide a fix or workaround.

""

[wdeng]

anyone has possible solutions?

[takatost]

I have the same issue when I generate long text using GPT-4.

[jackwh]

Same issue with long GPT-4 outputs, although I'm using the PHP client. After extensive debugging I'm pretty certain the issue is server-side with OpenAI, but besides this thread, this is the only place I've found it being discussed.

The issue seems to affect both standard and streamed responses. With standard responses on nginx, the OpenAI server closes the connection after 5 minutes, and nginx reports this as a Gateway Timeout.

With streamed responses, the final delta doesn't specify a finish_reason, suggesting the model is still running. But the stream is terminated anyway.

I can reproduce this on both macOS and Ubuntu. Changing nginx directives makes no difference, nor does overriding HTTP client timeouts.

This appears to be a recent change, it worked as expected for me up until last week. Hopefully this is fixed soon — I've sent a link to this comment to OpenAI's support, hopefully someone sees it 🤞