Different results for gsm8k via lighteval compared to internal pipeline

[Mugariya]

Hi,

I get a zero score with lighteval for gsm8k for one of the models and for the same model, the results are a bit better, at least in 20s. Any idea why this could be happening? And how is the lm_eval gsm8k different than lighteval gsm8k?

Thanks

[clefourrier]

Hi!
There are many ways to implement GSM8K - the kind of difference you refer to could be a difference in end of sentence token management on in answer extraction. It's also possible that our parsing is stricter than the one from the harness.

I hope this helps!

[jiangzizi]

Same problem here. Also got a zero acc on both leaderboard|gsm8k|0|0 and lighteval|gsm8k|0|0

[zjlariel]

same +1

[codestar12]

Same

[NathanHB]

newest version of gsm8k in lighteval should work better, also if the model does not perform, add some fewshots it really helps for gsm8k