deepseek-coder-6.7B-instruct-GGUF crashes the app when selected

[PedzacyKapec]

System Info

win10,
ver 2.5.2

Information

• The official example notebooks/scripts
• My own modified scripts

Reproduction

1. select the model

Expected behavior

to work

[cebtenzzre]

There are upstream issues with this model's vocab too. I don't know if these files work anywhere yet. Give it some time.

ggml-org/llama.cpp#3633 (comment)

[TheBloke]

To my knowledge, the GGUFs are now fine with llama.cpp. All issues I've been made aware of are now fixed, and I re-uploaded fixed GGUFs to all repos over the last couple of hours.

Eg 6.7B Q4_K_M in llama.cpp:

llm_load_tensors: offloaded 35/35 layers to GPU
llm_load_tensors: VRAM used: 3821.75 MB
..................................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 100000.0
llama_new_context_with_model: freq_scale = 0.25
llama_kv_cache_init: offloading v cache to GPU
llama_kv_cache_init: offloading k cache to GPU
llama_kv_cache_init: VRAM kv self = 256.00 MB
llama_new_context_with_model: kv self size  =  256.00 MB
llama_build_graph: non-view tensors processed: 740/740
llama_new_context_with_model: compute buffer total size = 77.63 MB
llama_new_context_with_model: VRAM scratch buffer: 71.00 MB
llama_new_context_with_model: total VRAM used: 4148.75 MB (model: 3821.75 MB, context: 327.00 MB)

system_info: n_threads = 56 / 112 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
sampling:
	repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
	top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0


You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.
### Instruction:
write a quick sort algorithm in python.
### Response:
Sure, here is an implementation of the Quick Sort algorithm in Python:

def quicksort(arr):
    if len(arr) <= 1:
        return arr
    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]
    return quicksort(left) + middle + quicksort(right)

This function works by selecting a 'pivot' element from the array and partitioning the other elements into two sub-arrays, according to whether they are less than or greater than the pivot. The sub-arrays are then recursively sorted. This algorithm is efficient for large datasets as it has an average time complexity of O(n log n).
 [end of text]

llama_print_timings:        load time =    2662.64 ms
llama_print_timings:      sample time =      84.93 ms /   201 runs   (    0.42 ms per token,  2366.71 tokens per second)
llama_print_timings: prompt eval time =      66.51 ms /    75 tokens (    0.89 ms per token,  1127.62 tokens per second)
llama_print_timings:        eval time =    2006.03 ms /   200 runs   (   10.03 ms per token,    99.70 tokens per second)
llama_print_timings:       total time =    2207.53 ms
Log end

I am aware of issues with various downstream clients, like GPT4All and llama-cpp-python, which I think might relate to the BPE vocab; but I'm not 100% sure on that.

So I imagine this issue is connected to that, not due to any issue with the GGUFs with regards to llama.cpp.

[cebtenzzre]

Ah, I didn't notice the BPE vocab. We don't support BPE models converted within the last few weeks, since there was a breaking change and we haven't updated yet.

[cebtenzzre]

Correction: This model doesn't use a GPT2-style BPE vocab, it uses sentencepiece (LLamaTokenizer/LLamaTokenizerFast), but it does not have a tokenizer.model and I believe it still depends on ggml-org/llama.cpp#3252 for decoding the contents of tokenizer.json. This will be resolved soon.

[Phil209]

Just letting you know the new deepseek-llm-7B-chat-GGUF from the Bloke crashes GPT4All. I'm assuming it's crashing for the same token reason as this older deepseek coder LLM.

[cebtenzzre]

This was fixed by #1706 which will be included in the next release.

[cebtenzzre]

The v2.6.1 release includes the necessary llama.cpp update.