What's Changed fix(timeout): larger timeout by @JiaoPL in #495 feat(doc): add GPU memory info for 7B & 20B models by @li126com in #507 feat(model): add rope_base interface by @00INDEX in #512 Feat(QA): Check loss when swapping micro_num and micro_bsz && Check grad norm by @li126com in #510 Fix(QA): the py name in main is wrong by @li126com in #514 fix/feat: small fix and enhancement by @SolenoidWG…
Highlights fix the bug that may have grad overflow when total_steps is small fix the rotary_emb.inv_freq KeyError in tool convert2hf.py add unit test for model What's Changed 🚀 Features feat(core/trainer.py): add more tgs metrics by @li126com in #310 🐞 Bug fixes fix(convert2hf.py): fix the rotary_emb.inv_freq KeyError by @jiangtann in #299 fix(configs/7B_sft.py): model dtype float16 to bfloat16 by…
What's Changed fix(ckpt): fix snapshot none load error and remove file lock by @SolenoidWGT in #298 Full Changelog: v0.2.1dev20230908...v0.2.1dev20230909
Highlights fix the bug that may have NaN value when overlap gradients' allreduce with backward support timeout wrapper and runtime diagnosis support readthedocs Chinese version What's Changed 🚀 Features feat(monitor): add light monitor by @JiaoPL in #275 feat(utils): add timeout wrapper by @SolenoidWGT in #286 feat: add runtime diagnosis by @sunpengsdu in #297 💥 Improvements fix(storage): refactor…
Highlights Support centos and ubuntu dockerfile Support runtime gpu flops and nccl allreduce speed test What's Changed 🚀 Features Implement uniform_init for tensor by @Pryest in #252 Support centos and ubuntu dockerfile by @li126com in #220 #243 Support writer add_scalars for writing dict data by @huangting4201 in #257 Support runtime gpu flops and nccl allreduce speed test by @sunpengsdu in #254…
No content.
Features： Support pipeline parallel, including interleaved and non-interleaved pipeline scheduler. Support sequence parallel. Support model evaluation. Support tf32 with flash-attention. Support tensorboard writer for recording training performance metrics. Support customed uniscale logger. Support calculating model's accuracy and perplexity metrics. Support oss storage and checkpoint asynchronous…