Evaluation broken in e2b

[YijuGuo]

Model introduction

When running the provided bash script with the specified configurations, successful generation is achieved. However, during the evaluation process, an error occurs as shown below:

File "/usr/local/lib/python3.10/site-packages/bigcodebench/evaluate.py", line 209, in evaluate
    passk = [int(k) for k in pass_k.split(",")]
AttributeError: 'tuple' object has no attribute 'split'

DATASET=bigcodebench
MODEL=baseline_llama_1b
BACKEND=vllm
NUM_GPU=2
SPLIT=complete
SUBSET=full
export E2B_API_KEY=""
bigcodebench.evaluate
--model $MODEL
--split $SPLIT
--execution "e2b"
--subset $SUBSET
--backend $BACKEND
--check_gt_only

Model URL

https://huggingface.co/meta-llama/Llama-3.1-8B

Additional instructions (Optional)

No response

Author

No

Security

• I confirm that the model is safe to run which does not contain any malicious code or content.

Integrity

• I confirm that the model comes from unique and original work and does not contain any plagiarism.

[terryyz]

Thanks for reporting the issue @YijuGuo! I just updated the e2b docker image. Please check if it works now. However, please still consider gradio (better to duplicate your own HF space), which is much faster than e2b.

[MaiziXiao]

@terryyz https://huggingface.co/spaces/bigcode/bigcodebench-evaluator is currently down

[YijuGuo]

Thank you very much for your reply.
But when I used gradio, the eval_results.json is empty. What do I need to do to solve this @problem?

{
  "model": "__dummy__.jsonl",
  "split": "complete",
  "subset": "full",
  "calibrated": true,
  "gt_pass_rate": 0.9982456140350877,
  "failed_tasks": [
    "BigCodeBench/823",
    "BigCodeBench/1005"
  ]
}

{
  "date": "2025-02-13 03:25",
  "eval": {}
}

[terryyz]

@terryyz https://huggingface.co/spaces/bigcode/bigcodebench-evaluator is currently down

thanks! restarted :)

[terryyz]

Thank you very much for your reply. But when I used gradio, the eval_results.json is empty. What do I need to do to solve this @problem?

{
  "model": "__dummy__.jsonl",
  "split": "complete",
  "subset": "full",
  "calibrated": true,
  "gt_pass_rate": 0.9982456140350877,
  "failed_tasks": [
    "BigCodeBench/823",
    "BigCodeBench/1005"
  ]
}

{
  "date": "2025-02-13 03:25",
  "eval": {}
}

It seems you passed --check_gt_only. This is the expected output to check the ground truth passing rate.
Please refer to the ADVANCED USAGE for more details.

[terryyz]

fixed

[YijuGuo]

I'm working with the Llama 1B model and noticed that the gt_pass_rate is showing 99.8%. This seems unusually high.

I'm using this benchmark to test the accuracy of a local model. I was wondering if I need to adjust --check_gt_only parameter to get accurate results for my local model evaluation, or if the default setting is sufficient.

[terryyz]

I'm working with the Llama 1B model and noticed that the gt_pass_rate is showing 99.8%. This seems unusually high.

I'm using this benchmark to test the accuracy of a local model. I was wondering if I need to adjust --check_gt_only parameter to get accurate results for my local model evaluation, or if the default setting is sufficient.

gt_pass_rate is the ground truth pass rate, and it should be close to 100% (may not be 100% due to some different config in the environment). --check_gt_only is used to check the ground truth pass rate, not the model pass rate. If you want to evaluate the model, you should not pass --check_gt_only into the eval.

[YijuGuo]

Thank you very much for your reply. When I submit the jsonl file on this website (https://huggingface.co/spaces/bigcode/bigcodebench-evaluator ) , it shows an error.

[terryyz]

Thank you very much for your reply. When I submit the jsonl file on this website (https://huggingface.co/spaces/bigcode/bigcodebench-evaluator ) , it shows an error.

You don't need to submit the file manually using bigcodebench.evaluate with an arg of --samples xxxx.jsonl will simply work.
Please refer to the ADVANCED USAGE for more details.