Fix NemotronHForCausalLM (support Nemotron-3-Super) Faster quantization Experimental zero-copy pinned arena mode for CPU offload (Linux only for now) Faster MoE prefill and decode (see below) Improved speculative decoding for MoE models Faster dense model prefill on some architectures Remove some sources of nondeterminism Bugfixes and cleanup Test case 4k prefill, t/s decode, t/s, S=1 Qwen3.8-Flas…
Add DeepseekV4ForCausalLM vision tower (supports DeepseekV4-Flash-Vision-Exp) Add Lfm2ForCausalLM (LFM2.5 dense variant) Add Spark2_5ForCausalLM (Spark-X2.5) Add Glm4MoeLiteForCausalLM (GLM4.7-Flash) More accurate autosplit accounting Faster CPU offloading, better Intel support (AVX512BW tier) Fix VRAM leak after vision tower offloading Fix potential worker timeout when offloading large models to…
Add pyproject.toml Cache quantization for DSv4-Flash, Qwen3.8-Flash-Next and GLM5.3-Flash Optimize VRAM allocations and switch to expandable segments by default Fully remove all FA2 code paths Other optimizations and bugfixes Full Changelog: v1.4.7...v1.4.8
Improved CPU MoE performance for all tiers Add DRY sampler Fix recurrent slot leaks Faster quantization for 6, 7 and 8 bit trellises Many other bugfixes, optimizations and QoL improvements Full Changelog: v1.4.6...v1.4.7
Fix some regressions from v1.4.6 Add experimental n-gram streaming for Windows Full Changelog: v1.4.5...v1.4.6
Support Glm5NextForConditionalGeneration (GLM5.3-Flash) Support Qwen4ExpForConditionalGeneration (Qwen3.8-Flash-Next) Improved MoE MTP performance Slab allocation for MoE layers to avoid wasted VRAM for badly aligned expert tensor shapes Work around Triton race condition (autotuner: 'NoneType' object is not a mapping error) Fix Unigram tokenizers Other bugfixes and QoL improvements Full Changelog:…
Fix correctness bug in split MoE CPU-offload mode Better MTP support in expert-parallel mode Validate and default to vision tower quantization Support vision model offloading (streams from system memory, performance penalty is small) Full Changelog: v1.4.3...v1.4.4
Support GlmMoeDsaForCauslLM (GLM 5.2, still probably somewhat WIP) Partial CPU layer expert offloading option with dynamic placement (supersedes expert cache) Improved dynamic draft sizing with auto-calibrated confidence thresholds New (experimental) quant-optimizer pipeline Support for mid-stream text injection in generator (enables reasoning token budget) More precise autosplit allocation Reduce…
Support MuseGlimmerForConditionalGeneration Full Changelog: v1.4.1...v1.4.2
Add Mistral-4 (under Mistral3ForConditionalGeneration) Support logit bias Adopt LLGuidance instead of Formatron (still supported as optional dependency) Allow filters/grammar alongside banned_strings Full Changelog: v1.4.0...v1.4.1
Support DeepseekV4ForCausalLM Improved AVX512 CPU offload performance Faster prefill (all models) Bugfixes QoL features Full Changelog: v1.3.0...v1.4.0
Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet) Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies Fix freq/rep. penalties causing slowdown on long contexts XTC sampler and token bans Mitigate some latent vulnerabilities in Safetensors loader Other bugfixes, optimizations and QoL improvements Ful…
Faster quantization Windows: Fix JIT compilation error and broken CPU offload performance Fix tokenizer issue breaking decoding of tokens that both end and begin a multi-token character (Korean esp.) Full Changelog: v1.2.0...v1.2.1
Support LagunaForCausalLM (and DFlashLagunaForCausalLM) Experimental CPU offloading support for expert layers Experimental dynamic draft window feature Improved safetensors loader (faster) More graph paths Performance tuning Removed compare_q.py, superseded by qbench.py Various bugfixes Full Changelog: v1.1.0...v1.2.0
Support HYV3ForCausalLM Reduced CPU overhead for TP mode and improved prefill performance in native backend Adjust top-P=1 sampling to match implicit truncating behavior of torch.multinomial Banned strings now supported for recurrent models Bugfixes and QoL improvements Full Changelog: v1.0.0...v1.1.0
-> Small writeup with charts. Remove flash-attention-2 and xformers dependencies New attention kernel with online cache quantization, dual input for SWA layers and attention sinks New conv1d kernel (removes support/need for causal_conv1d) Greatly improved GEMM/GEMV performance on Ampere New INT8 GEMV kernel New MoE kernel ticket scheduler Graph path for all attn/GDN modules Fused sampling kernels…
Fix error when MTP drafting in TP mode Faster quanization Full Changelog: v0.0.42...v0.0.43
Fix MTP drafting when MTP model is not on the target model's output device Full Changelog: v0.0.41...v0.0.42
Add MTP support for Qwen3.5/3.6 Full Changelog: v0.0.40...v0.0.41
Support Gemma4UnifiedForConditionalGeneration Full Changelog: v0.0.39...v0.0.40
Add Step3p7ForConditionalGeneration Full Changelog: v0.0.38...v0.0.39
Support Lfm2MoeForCausalLM (LFM 2.5) Fix regression in GDN inference when bsz > 1 Fix issue causing DFlash to break in TP mode when cudaMallocAsync backend was used QoL improvements Full Changelog: v0.0.37...v0.0.38
Another small bugfix Full Changelog: v0.0.36...v0.0.37
Fix small SD regression Full Changelog: v0.0.35...v0.0.36
Tensor parallel mode for Qwen3.5/3.6 New recurrent state manager avoid dynamics allocation of recurrent states (reduces fragmentation and keeps VRAM overhead constant for Qwen3.5 etc.) Improved checkpointing decision to make better use of available cache space Perform reconstruct-GEMM in slices for large layers, greatly reducing VRAM overhead Fix race condition causing streaming token output to la…
Fix regression causing extra VRAM usage during prefill Add CUDA 13.2.0 wheels (built with cu132 against torch==2.11.0+cu130) Full Changelog: v0.0.33...v0.0.34