Add BPE pre-tokenization for Command-R/R+.#7063

ggerganov merged 3 commits intoggml-org:masterggml-org/llama.cpp:masterfrom

dranger003:bpe-pretok-command-r-2dranger003/llama.cpp:bpe-pretok-command-r-2Copy head branch name to clipboard