Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet) Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies Fix freq/rep. penalties causing slowdown on long contexts XTC sampler and token bans Mitigate some latent vulnerabilities in Safetensors loader Other bugfixes, optimizations and QoL improvements Ful…
Faster quantization Windows: Fix JIT compilation error and broken CPU offload performance Fix tokenizer issue breaking decoding of tokens that both end and begin a multi-token character (Korean esp.) Full Changelog: v1.2.0...v1.2.1
Support LagunaForCausalLM (and DFlashLagunaForCausalLM) Experimental CPU offloading support for expert layers Experimental dynamic draft window feature Improved safetensors loader (faster) More graph paths Performance tuning Removed compare_q.py, superseded by qbench.py Various bugfixes Full Changelog: v1.1.0...v1.2.0
Support HYV3ForCausalLM Reduced CPU overhead for TP mode and improved prefill performance in native backend Adjust top-P=1 sampling to match implicit truncating behavior of torch.multinomial Banned strings now supported for recurrent models Bugfixes and QoL improvements Full Changelog: v1.0.0...v1.1.0
-> Small writeup with charts. Remove flash-attention-2 and xformers dependencies New attention kernel with online cache quantization, dual input for SWA layers and attention sinks New conv1d kernel (removes support/need for causal_conv1d) Greatly improved GEMM/GEMV performance on Ampere New INT8 GEMV kernel New MoE kernel ticket scheduler Graph path for all attn/GDN modules Fused sampling kernels…
Fix error when MTP drafting in TP mode Faster quanization Full Changelog: v0.0.42...v0.0.43
Fix MTP drafting when MTP model is not on the target model's output device Full Changelog: v0.0.41...v0.0.42
Add MTP support for Qwen3.5/3.6 Full Changelog: v0.0.40...v0.0.41
Support Gemma4UnifiedForConditionalGeneration Full Changelog: v0.0.39...v0.0.40
Add Step3p7ForConditionalGeneration Full Changelog: v0.0.38...v0.0.39
Support Lfm2MoeForCausalLM (LFM 2.5) Fix regression in GDN inference when bsz > 1 Fix issue causing DFlash to break in TP mode when cudaMallocAsync backend was used QoL improvements Full Changelog: v0.0.37...v0.0.38
Another small bugfix Full Changelog: v0.0.36...v0.0.37
Fix small SD regression Full Changelog: v0.0.35...v0.0.36
Tensor parallel mode for Qwen3.5/3.6 New recurrent state manager avoid dynamics allocation of recurrent states (reduces fragmentation and keeps VRAM overhead constant for Qwen3.5 etc.) Improved checkpointing decision to make better use of available cache space Perform reconstruct-GEMM in slices for large layers, greatly reducing VRAM overhead Fix race condition causing streaming token output to la…
Fix regression causing extra VRAM usage during prefill Add CUDA 13.2.0 wheels (built with cu132 against torch==2.11.0+cu130) Full Changelog: v0.0.33...v0.0.34