Very slow - any way to speed up?

[luke-jr]

Per #10,

Noting that the processing time is considerably shorter than the length of speech,

Yet even using 64 threads, it's taking days to process minutes of audio on my POWER9.

Has something changed since #10, or is there something I am doing wrong?

[RndyP]

What is your platform?

[ggerganov]

@luke-jr
I'm not familiar with POWER9, but from a quick ChatGPT search, it seems this CPU has a RISC architecture:

Currently, whisper.cpp supports only x86 and ARM architectures. By support, it means that it uses the available SIMD instruction set to make the computation efficient. On other architectures, it will fallback to non-SIMD computation which is multiple times slower.

Adding support for Power ISA (or whatever the instruction set is called) should not be very difficult. The matrix multiplication routines in ggml.c need to be extended to support the respective instruction set and the corresponding compile flags added to the Makefile.

I don't have experience with this architecture, so hopefully someone contributes.
It will be very interesting to see what is the performance on these CPUs.

[luke-jr]

Yeah, I'm not surprised it isn't optimised for PPC64, but I wouldn't expect it to be magnitudes slower either. Real-time to days is a huge difference. :/

[ggerganov]

For example on my Ryzen 9 5950X if I remove the -mavx -mavx2 -mfma -mf16c flags I observed about x50 slower computation of the bench tool. Removing those flags is similar to what you have on the PPC64 - no SIMD, no F16C support.

SIMD can make a huge difference

[fitzsim]

ChatGPT is out-of-date regarding the Power ISA being proprietary. It is open source now, just like RISC-V. See https://openpowerfoundation.org/.

[luke-jr]

After #320, ./main -m models/ggml-base.en.bin -f samples/jfk.wav takes 15.7 seconds.

Additional options	Time
-p 64	77s
-t 64	28s
-t 1	59.6s
-t 16	5.8s
-t 32	5.1s
-m models/ggml-large.bin -t 32	111.1s

ChatGPT is out-of-date regarding the Power ISA being proprietary.

In my experience, ChatGPT tends to be wrong quite often.

[ggerganov]

@fitzsim @luke-jr
I am planning to merge a refactored version of the SIMD routines in ggml which I think will make things easier to maintain in the future. The PR is pretty much ready in #324

All instruction sets fit quite nicely in the proposed pattern, but I'm having a little trouble with the ppc64le stuff since I don't have a way to test it. So for the moment, I've special-cased it, which is not great.

If you are interested and have some free time, you can take a look at the implementation and see if you can fit it in the new pattern. Or at the very least - run a test and see that it still works after the changes.

Regarding the new performance: 5s for jfk.wav using base still seems quite a lot. Not sure why the performance is so bad

[fitzsim]

@ggerganov, sure, I'll try to fit the POWER9 optimizations into the main SIMD structure, some time after #324 lands in the master branch.

Agreed regarding 5s likely not being optimal. @luke-jr, can you add the whisper_print_timings lines to your table? They may contain hints about further optimization efforts.

[luke-jr]

$ time ./main -m models/ggml-base.en.bin -f samples/jfk.wav -t 32
whisper_model_load: loading model from 'models/ggml-base.en.bin'
whisper_model_load: n_vocab       = 51864
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head  = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 512
whisper_model_load: n_text_head   = 8
whisper_model_load: n_text_layer  = 6
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 2
whisper_model_load: adding 1607 extra tokens
whisper_model_load: mem_required  =  506.00 MB
whisper_model_load: ggml ctx size =  140.60 MB
whisper_model_load: memory size   =   22.83 MB
whisper_model_load: model size    =  140.54 MB

system_info: n_threads = 32 / 64 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | 

main: processing 'samples/jfk.wav' (176000 samples, 11.0 sec), 32 threads, 1 processors, lang = en, task = transcribe, timestamps = 1 ...


[00:00:00.000 --> 00:00:11.000]   And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.


whisper_print_timings:     load time =   110.77 ms
whisper_print_timings:      mel time =    49.83 ms
whisper_print_timings:   sample time =     8.41 ms
whisper_print_timings:   encode time =  3631.16 ms / 605.19 ms per layer
whisper_print_timings:   decode time =  1374.97 ms / 229.16 ms per layer
whisper_print_timings:    total time =  5175.76 ms

real    0m5.187s
user    2m31.675s
sys     0m1.078s

[fitzsim]

The remaining slowness seems to be in the short-to-fp32 conversion. Would it make sense to try a GGML_TYPE_F32 version of ggml-base.en.bin, to eliminate the conversion steps? Can someone outline steps for trying that?