Resources

• Towards Fully FP8 GEMM LLM Training at Scale

• Questions: Clarifying the use of FP8 for Training #99

  • https://github.com/Azure/MS-AMP/issues/99

• Question(weight quantization): Why not write out fp8 after performing weight update?

  • https://github.com/NVIDIA/TransformerEngine/issues/424

• AdamW FP8 optimizer CUDA code

  • https://github.com/Azure/MS-AMP/blob/aed29d6533e8ff86686fdb6fafa6d0b720e9e5f6/msamp/optim/adamw.cu