adds wandb loging of metrics #676
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
|
||
| def push_to_wandb(self, results_dict: dict, details_datasets: dict) -> None: | ||
| self.wandb_run.log( | ||
| {**results_dict["results"]}, |
There was a problem hiding this comment.
Can we log some additional data here like checkpoint_num or consumed_tokens? For the custom X axis like so
There was a problem hiding this comment.
We can probably pass it from nanotron_model, since these metrics are implementation-specific
There was a problem hiding this comment.
not sure what you mean by that ? what's checkpoint_num and is consumed_tokens the total amount of tokens used as input ?
There was a problem hiding this comment.
I mean just arbitrary step counters coming from somewhere else, e.g. we can implement consumed_tokens (consumed_train_samples*seq_len) in nanotron_model.py
| def push_to_wandb(self, results_dict: dict, details_datasets: dict) -> None: | ||
| self.wandb_run.log( | ||
| {**results_dict["results"]}, | ||
| ) |
There was a problem hiding this comment.
nit: IIUC and the metrics are logged as custom|mmlu:astronomy|0/acc_norm , we can .replace(':', '/') to get custom|mmlu/astronomy|0/acc_norm.
Otherwise wandb creates tons of collapsible sections for every subset of the benchmark, instead of just one section for custom|mmlu with all the metrics inside.
No description provided.