[FT] Support local datasets #604
Description
Issue encountered
Unable to load a local dataset when creating a custom task.
Currently, as part of the Create new task workflow, the class LightevalTaskConfig only supports hugging face datasets via the parameter hf_repo. This does not allow us to test local datasets.
Solution/Feature
Under the hood, load_dataset from the datasets library is called that actually supports local datasets via the parameters data_files and data_dir. These parameters are not exposed thereby preventing us to load local datasets.
Please let me know if there is another way to achieve this if I've missed.