Decompression Interceptor

[FelixVaughan]

This relates to...

Address proposal brought up here
and is related to #4316

Rationale

This PR implements a decompression interceptor for Undici, as discussed in the linked proposal and issue. The goal is to provide automatic response decompression for request() (matching the behavior of fetch()), reducing boilerplate and improving developer experience when working with compressed HTTP responses.

Changes

• Adds a new decompression interceptor for Undici clients
• Supports gzip, deflate, and brotli content-encoding
• Skips decompression for 4xx/5xx, 204, and 304 responses (per spec and browser behavior)
• Removes content-encoding and content-length headers when decompressing
• Streams decompressed data for memory efficiency
• Additional tests for supported encodings and edge cases

Features

• Automatic decompression for gzip, deflate, brotli, and zstd responses
• Transparent header cleanup
• Skips decompression for error and no-content responses
• Streaming implementation for large responses
• Case-insensitive encoding support
• Pass-through for unsupported encodings

Status

• I have read and agreed to the Developer's Certificate of Origin
• Tested
• Benchmarked (optional)
• Documented
• Review ready
• In review
• Merge ready

[metcoder95]

Nice job!

I'd just suggest to implement the decompressor in accordance to RFC-9110, in that way decompressor can work with different combination in according to standards.

I believe the only missing part might be the support for multiple encodings on a response

[FelixVaughan]

@metcoder95 Thanks! All the above have been implemented along with documentation. Let me know if anything else is required

[metcoder95]

Nice job! I left few commenets.

Do not forget about documentation and TS types

[metcoder95]

Pipeline returns a stream which you can subscribe for the chunks of data

[metcoder95]

LGTM, just tests seems failing

[FelixVaughan]

Failures were due to lack of createZstdCompress support prior to v22.15.0. Added a skip condition to handle this.

If everything else looks good, I'd like to get out a final commit to clean up some code

[FelixVaughan]

@Uzlopak these optimizations are sound and the docs are a nice touch!

I noticed annotations are missing on onResponseData, and onResponseStart, onResponseEnd and onResponeError dont have return types on their annotations but lgtm otherwise

[Uzlopak]

@FelixVaughan

I mean, can you integrate them into this PR please? I dont know the types for those methods, and did not want to dig deeper. LOL.

[Uzlopak]

Applied my recommended changes.
It adds the said performance optimizations.

Also I added jsdoc which resulted to basically understand the full PR and despite the before existing performance issues, the code is as far as I can see correct.

@metcoder95
can you please review my changes and approve and merge?

[Uzlopak]

After talk with Matteo, need some experimental warning or use different approach, readable stream instead of data-event

[metcoder95]

We can start with an experimental warning and take it from there

[Uzlopak]

Exactly. Can you take care of it?

[FelixVaughan]

@Uzlopak I looked at dispatcher.md and it looks like the handler methods return void in all cases.

Do you think its worth making changes to the jsdocs and code at this stage?

Everything still functions as expected

[Uzlopak]

We did not merge yet, and if you can make the correct jsdoc, then please add them.

Also we need to add an experimentalwarning OR you need to change it to a readable stream.

[FelixVaughan]

Sounds good, I updated the jsdocs (some code as well) and added the experimental flag

[metcoder95]

Small not added for documentation, rest lgtm

[mcollina]

lgtm with a nit.

[mcollina]

It would be better to use a normal object instead, this is just initialization overhead.

[mcollina]

lgtm

[gurgunday]

lgtm as well

[Uzlopak]

Just for documentation purposes:

After a chat with Matteo I understand the issue, he sees with this PR.

While the interceptor is working, the majority of its code is about managing the streams. All that logic on when to pause and when to resume is a potential beesnest for errors. 

Unfortunately there is no good example in the project on how to do it properly. 

Other interceptors have the handler logic extracted in the handlers folder. This one is an amalgam of interceptor and handler. 

I am personally not well versed in these parts of the project. I really wished it was somehow more intuitive or atleast there was a handler and interceptor which was the gold standard for handlers and interceptors. Then we could use that as a blueprint to standardize handlers and/or interceptors.

E.g. i had a look at the dump interceptor and it seemed that it is more like we need it. But it is storing the payload into the memory. Not the logic on how we can stream the payload out.

IIRC The cache interceptor seems to be better, but it is too complicated to grok it in short time.

Anyhow. There was for sure some thought when interceptors and handlers were designed, but it is not easy to grok. 

So we have onResponseStart, onResponseData and onResponseEnd. So they are like events from a classical stream. But there is nothing intuitive in it, that I could just code a simple pipelining of the incoming response?!

I guess for all the other parts than the body of http requests/response the Interceptors are good?! But for handling the body it feels clunky.

Anyhow… i will make this week a deep dive into the interceptor stuff. 

[metcoder95]

So we have onResponseStart, onResponseData and onResponseEnd. So they are like events from a classical stream. But there is nothing intuitive in it, that I could just code a simple pipelining of the incoming response?!

Yeah, but this is aside of the interceptors, is more about the Dispatch handler.

They are intertwined in some way, but the Request events are the ones that manages the data flow, not necessarily the interceptors (the interceptors are just abstractions on top of the Dispatcher handlers).

I can see the potential issue with the body pipelining into the decompressors (same problem I saw when I was playing at the beginning when the issue was made). Everything boils down to the body processing with the decompressor streams, it might be a bit noisy and potentially have some issues, but cannot think on a better way (right now).

I can revisit and adjust later on and based on issues, but not see another way currently.

[FelixVaughan]

Happy to revisit and improve upon this as well. In the meantime, it would be great if we could merge this