[BUG] GSM8k Zero Accuracy

[candemircan]

Describe the bug

When I run the GSM8K eval, I get 0 accuracy. If I save details, I see that the predictions look like follows:

['Please reason step by step, and put your final answer within \\boxed{}.ichtig\n']

I'm not sure if this is related to the closed issue #202 , but I've seen others having this issue as well

To Reproduce

export VLLM_WORKER_MULTIPROC_METHOD=spawn
model="Qwen/Qwen2.5-Math-1.5B"
seed=0
MODEL_ARGS="pretrained=$model,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95,seed:$seed}"

lighteval vllm "$MODEL_ARGS" "leaderboard|gsm8k|0|0" --use-chat-template --output-dir "data" --save-details

Expected behavior

I do not have this issue with other evals. e.g. gpqa:diamond runs fine. It generates outputs and nonzero accuracies

Version info

lighteval is version '0.8.1', and vllm is version '0.8.4'

Thank you for your help!

[ScottHoang]

yes. the gsm8k parser doesn't work very well. if you want you can use my custom gsm8k_normalizer

def gsm8k_normalizer(text: str) -> str:
    # step 1: if the model generated too much, get the first part
    text = text.split("\nQ:")[0]

    # step 2: extract \boxed{ }
    extracted_nums = re.findall(r"\\boxed\{(.*?)\}", repr(text))
    if extracted_nums:
        extracted_nums = extracted_nums[-1]
    else:
        extracted_nums = text
    # step 2: pick the last number
    # try:
    extracted_nums = re.sub(r"[^a-zA-Z0-9\s.,-]", " ", extracted_nums)

    return greedy_extract_xml_answer(extracted_nums)

def greedy_extract_xml_answer(text: str) -> str:
    """
    Greedy extract will try to extract valid answer within \\boxed{..}
    If no valid answer is found, it returns the last digit in the text pharse
    """
    pattern = r"(?<!\S)(?P<prefix>(?:-\$|\$-|\$|-)?)(?P<number>(?:(?:\d{1,3}(?:,\d{3})+)|\d+)(?:\.\d*)?)(?:%)?(?!\S)"
    match = re.findall(pattern, text)
    if match:
        prefix = match[-1][0]  # may include '-' if the number is negative
        number = match[-1][1]  # match.group("number")

        # Remove commas from the number (e.g., "1,765" -> "1765")
        number = number.replace(",", "")

        # Remove a trailing ".00" if present.
        if number.endswith(".00"):
            number = number[:-3]
        # If there's a lone trailing dot (e.g., "32."), remove it.
        elif number.endswith("."):
            number = number[:-1]

        # Determine if a minus sign appears anywhere in the prefix.
        sign = "-" if "-" in prefix else ""
        return sign + number
    try:
        text = text.replace(",", "")
        extracted_nums = [s for s in re.findall(r"-?\d+\.?\d*", text)]
        return str(float(extracted_nums[-1]))
    except Exception:
        return "[invalid]"

[zjlariel]

I found it takes longer to use lighteval:gsm8k than to use leaderboard:gsm8k

leaderboard:gsm8k

lighteval:gsm8k

why did lighteval:gsm8k execute twice?

[candemircan]

I found it takes longer to use lighteval:gsm8k than to use leaderboard:gsm8k

leaderboard:gsm8k

lighteval:gsm8k

why did lighteval:gsm8k execute twice?

In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found this paper GSM-Plus, where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs

UPDATE: nvm, see below

[CurryxIaoHu]

I encounter the same bug

[NathanHB]

Hi ! Thanks for the issue

In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found this paper GSM-Plus, where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs

This is because we use two metrics for gs8mk, one of them requiring to run the same prompt 8 times, so 10552 questions total)

@candemircan you are having weird generations because your model is not instruct and you are using chat templates, I would recommend not using chat templates and using 5 shots for base models.

I have oopened a PR (#688) to change the metric funciton we use to a better one.

lighteval vllm "model_name=Qwen/Qwen2.5-Math-1.5B,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95}" "lighteval|gsm8k|5|0"  --output-dir "data" --save-details --max-samples 100

yields

|      Task       |Version|     Metric     |Value|   |Stderr|
|-----------------|------:|----------------|----:|---|-----:|
|all              |       |extractive_match| 0.74|±  |0.0441|
|lighteval:gsm8k:5|      0|extractive_match| 0.74|±  |0.0441|

[candemircan]

hi @NathanHB, thanks for the quick response and for the clarification! this resolves the issue for me

PS: model_name=Qwen/Qwen2.5-Math-1.5B does not work for me with vllm, but pretrained=Qwen/Qwen2.5-Math-1.5B does. Not sure if this is just me, but I saw your README also uses model_name whereas examples on the huggingface page use pretrained. Sharing in case its unintentional

[NathanHB]

the documentation for main uses model_name, we haven't made the release yet !

[Guncuke]

I found it's hard to evaluate thinking models, for example, deepseek qwen 7b will output like this:

"Alright, let's tackle this problem step by step. So, Tim has a box containing 7 blue shoe boxes and 9 red shoe boxes. That means initially, he has a total of 7 + 9 = 16 shoe boxes.\n\nNow, Tim uses 3 blue shoe boxes for going fishing. So, subtracting those from the blue shoe boxes, we have 7 - 3 = 4 blue shoe boxes remaining.\n\nNext, he uses 1/3 of his red shoe boxes for fishing. He originally has 9 red shoe boxes. Calculating 1/3 of 9 gives us 3 red shoe boxes used. Therefore, the remaining red shoe boxes are 9 - 3 = 6.\n\nFinally, to find the total number of shoe boxes left, we add the remaining blue and red shoe boxes: 4 + 6 = 10.\n\nSo, Tim has 10 shoe boxes left in his box after using some for fishing.\n</think>\n\nTim has a box with **7 blue shoe boxes** and **9 red shoe boxes**, totaling **16 shoe boxes**.\n\nHe uses **3 blue shoe boxes** for fishing, leaving him with:\n\\[ 7 \\text{ blue shoe boxes} - 3 = 4 \\text{ blue shoe boxes} \\]\n\nHe also uses **\\(\\frac{1}{3}\\)** of his **9 red shoe boxes** for fishing:\n\\[ \\frac{1}{3} \\times 9 = 3 \\text{ red shoe boxes} \\]\nSo, the remaining red shoe boxes are:\n\\[ 9 \\text{ red shoe boxes} - 3 = 6 \\text{ red shoe boxes} \\]\n\nAdding the remaining blue and red shoe boxes:\n\\[ 4 \\text{ blue shoe boxes} + 6 \\text{ red shoe boxes} = 10 \\text{ shoe boxes} \\]\n\n**Final Answer:**  \nThere are **10 shoe boxes** left in Tim's box."

They don't follow the output format. I can deal with this by remove the --use-chat-template, but if I do that, the model will no thinking. How can I deal with this.

[passing2961]

Hi ! Thanks for the issue

In this case, lighteval creates two splits. The first one has 1319 questions, which is the test set of GSM8k. The second one has 10552 questions. I'm not sure what these are, but I found this paper GSM-Plus, where they have exactly 10552 questions, so that could be. Although I couldn't find any references to it in the lighteval docs

This is because we use two metrics for gs8mk, one of them requiring to run the same prompt 8 times, so 10552 questions total)

@candemircan you are having weird generations because your model is not instruct and you are using chat templates, I would recommend not using chat templates and using 5 shots for base models.

I have oopened a PR (#688) to change the metric funciton we use to a better one.

lighteval vllm "model_name=Qwen/Qwen2.5-Math-1.5B,dtype=bfloat16,max_model_length=4096,gpu_memory_utilization=0.9,generation_parameters={max_new_tokens:4096,temperature:0.6,top_p:0.95}" "lighteval|gsm8k|5|0"  --output-dir "data" --save-details --max-samples 100

yields

|      Task       |Version|     Metric     |Value|   |Stderr|
|-----------------|------:|----------------|----:|---|-----:|
|all              |       |extractive_match| 0.74|±  |0.0441|
|lighteval:gsm8k:5|      0|extractive_match| 0.74|±  |0.0441|

@NathanHB Hi, I ran the same CLI, but I got different results which is shown in the below.

How can I reproduce the performance that you reported.