ValueError: Cannot find tasks community|alghafa:mcq_exams_test_ar in task list or in custom task registry)

[BobTsang1995]

This FrameWork is a great job！but it seems you guys didn't say how to eval the community benchmark？I am wandering what should I do to eval the list of benchmark in examples/tasks/OALL_v2_tasks.txt

[clefourrier]

Hi @BobTsang1995 !
You'll need to use the --custom-tasks argument and point to the correct file. @alielfilali01 , want to share your usual command line to launch the evals?

[BobTsang1995]

Hi @BobTsang1995 ! You'll need to use the --custom-tasks argument and point to the correct file. @alielfilali01 , want to share your usual command line to launch the evals?
I try to use this command,but failed.
accelerate launch --multi_gpu --num_processes=8 -m lighteval accelerate "pretrained=/mnt/Qwen2-7B-CT-latest/checkpoint-1150,dtype=bfloat16,max_length=16384" "community|alghafa:mcq_exams_test_ar|0|0" --custom-tasks "community_tasks/arabic_evals.py" --output-dir="./evals/"
it reminds me two type error below:
TypeError: LightevalTaskConfig.init() got an unexpected keyword argument 'output_regex'
TypeError: LightevalTaskConfig.init() got an unexpected keyword argument 'frozen'

so i comment out the code "output_regex" and "frozen"
It may be because I commented out the above two parameters, which caused another error to occur at the same time.

/mnt/sg_nas/liheng/lighteval/src/lighteval/models/abstract_model.py:200 in tok_encode_pair │
[rank0]: │ │
[rank0]: │ 197 │ │ │ │
[rank0]: │ 198 │ │ │ return context_enc, continuation_enc │
[rank0]: │ 199 │ │ │
[rank0]: │ ❱ 200 │ │ whole_enc = self.tok_encode(context + continuation) │
[rank0]: │ 201 │ │ context_enc = self.tok_encode(context) │
[rank0]: │ 202 │ │ context_enc_len = len(context_enc) │
[rank0]: │ 203 │ │ # In case continuation tokens merge with context tokens we use the merged token │
[rank0]: │ │
[rank0]: │ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
[rank0]: │ │ context = 'الأسئلة التالية هي أسئلة متعددة الإختيارات مع الجواب الصحيح\n\nالسؤال: صل │ │
[rank0]: │ │ أولى دول'+98 │ │
[rank0]: │ │ continuation = 245 │ │
[rank0]: │ │ n_spaces = 0 │ │
[rank0]: │ │ pairwise = False │ │
[rank0]: │ │ self = <lighteval.models.transformers.base_model.BaseModel object at 0x7fc53c0ea680> │ │
[rank0]: │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
[rank0]: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
[rank0]: TypeError: can only concatenate str (not "int") to str

[clefourrier]

Hi! Are you using the latest version of lighteval on main?

[BobTsang1995]

Hi! Are you using the latest version of lighteval on main?

yeah，I'm pretty sure

[clefourrier]

Hm, I'll take some time to investigate today, thanks a lot for the report then :)

[BobTsang1995]

It seems that the parent class LightevalTaskConfig does not have the parameters output_regex and frozen.

[clefourrier]

Yep, these params have been removed in a PR yesterday, but I'm more concerned about the second error you're getting

[clefourrier]

Hi! Can you test if the code in this PR: #440 fixes your issue?

[BobTsang1995]

Hi! Can you test if the code in this PR: #440 fixes your issue?
Of course I'd love to test this fork, just give me a few minutes

[BobTsang1995]

thanks god it works, Merci, mon ami français

[BobTsang1995]

bro, something went wrong...
I found that the asas-ai/AraTrust-categorized dataset does not exist on hugging face

[clefourrier]

Hi @BobTsang1995 ! Could you open a new issue, since it's unrelated, and tag @alielfilali01, the community contributor who added the arabic evals? And please follow the bug template :)