Plans for 8da4w quantization

[sanchitintel]

Hi,

From #430, it seems that 8da4w is primarily for Executorch, and is set to be deprecated. Please advise if there are any plans to enable it for CUDA & CPU as well, such that int4 weights could be converted to int8 just before computation?

Thanks!

cc @jerryzh168

[Xia-Weiwen]

Hi @jerryzh168 I saw your pointer here:
https://github.com/pytorch/ao/tree/main/torchao/quantization#to-be-deprecated-a8w8-dynamic-quantization
However, we need 8da4w for CPU and XPU and we don't want it deprecated. May I know any concern from your side? Thanks

CC. @jgong5 @leslie-fang-intel

[jerryzh168]

this is used for executorch before, but it seems that we have people adding kernels here: #880. we are open to adding kernels for this

[Xia-Weiwen]

this is used for executorch before, but it seems that we have people adding kernels here: #880. we are open to adding kernels for this

Got it. Thanks for the clarification!

[jerryzh168]

@Xia-Weiwen that link is talking about the quantizer API since we are updating to the quantize_ API, we'll be using something like https://github.com/pytorch/ao/tree/main/torchao/quantization#a8w8-dynamic-quantization but with

ao/torchao/quantization/quant_api.py

Line 83 in 8236a87

"int8_dynamic_activation_int4_weight",

as the second argument. i.e.:

quantize_(model, int8_dynamic_activation_int4_weight())

[Xia-Weiwen]

Thanks for the pointers!

[sanchitintel]

Thanks for clarifying, @jerryzh168!