[BUG] Issue with LightevalTaskConfig.stop_sequence Attribute When Unset

[ryan-minato]

Describe the bug

When running aimo_evals.py from community tasks, an error occurs indicating that the Tokenizer does not accept None as input. This issue stems from how stop_sequence is handled in LightevalTaskConfig. Since most tasks in lighteval explicitly define stop_sequence, this problem may have gone unnoticed.

lighteval/src/lighteval/tasks/lighteval_task.py

Line 133 in 0977fde

self.stop_sequence = tuple(self.stop_sequence) if self.stop_sequence is not None else None

Proposed Solution

Set the default value for stop_sequence to an empty list ([]) instead of None. Testing shows that this change resolves the issue.

self.stop_sequence = tuple(self.stop_sequence) if self.stop_sequence is not None else tuple()

To Reproduce

lighteval accelerate \
    "pretrained=gpt2" \
    "community|aimo_progress_prize_1|0|0" \
    --custom-tasks "./community_tasks/aimo_evals.py"

Expected behavior

The test should run successfully without any errors.

Version info

This issue occurs in a version installed directly from the main branch using the following command:

pip install -e ".[dev]"