🎉 Announcing LightLLM v1.1.0: More Efficient, More Powerful! We are thrilled to introduce LightLLM v1.1.0, featuring major architectural and optimization upgrades for higher performance and broader applicability. ✨ Key Highlights 🚀 CPU-GPU Unified Folding Architecture Drastically reduces system-level CPU overhead ⚡ Deep Model Optimizations Enhanced support for DeepSeek and Qwen3-MoE Integration of…
Highlights DeepSeek-R1 Multi-Node H100 Deployment Support FlashInfer Integration XGrammer Integration What's Changed Benchclient by @shihaobai in #740 fix pause reqs by @shihaobai in #741 add RETURN_LIST for tgi_api by @shihaobai in #742 fix: fix a precision bug in the context_flashattention by @blueswhen in #743 Improve the accuracy of deepseekv3 by @hiworldwzj in #744 deepseekv3 bmm noquant and…
New Features Cross-Process Request Object: Retained and optimized the previous three-process architecture design. Introduced a request object that can be accessed across processes, significantly reducing inter-process communication overhead. Folding of scheduling and model inference: Implemented the folding of scheduling and model inference, significantly reducing communication overhead between th…