Keep the training data continuous and the total batch size constant regardless of changes in the replica world size.#292

zhengchenyu wants to merge 2 commits intometa-pytorch:mainmeta-pytorch/torchft:mainfrom

zhengchenyu:fix.totat.batchzhengchenyu/torchft:fix.totat.batchCopy head branch name to clipboard