Support DeepseekV41ForCausalLM Support Lfm3ForCausalLM and Lfm3VlForConditionalGeneration Improved loop detection Greatly reduced VRAM usage during quantization Faster quantization capture pass Fixed disk cache eviction policy (prevent possible churn when switching between many models) More optimizations, bugfixes and QoL improments Full Changelog: v1.6.5...v1.6.6
Support Nemotron-3.5-Super-VL (fix NemotronH config parsing) Reduced VRAM overhead for prefill using cache quantization Lower-latency drafting with hot/sparse vocabulary Faster AVX2 and AVX512-BW inference Faster model loading when offloading experts Support Gemma4 MTP model (Gemma4AssistantForCausalLM) Fix tokenizer regression from 1.6.4 causing prompt caching to fail sometimes Many more fixes an…
Support Lfm2VlForConditionalGeneration Support Nemotron 3.5 Bolt (extend NemotronHForCausalLM) Fairer prefill/decode scheduling for concurrent requests Fix some TP/shmem issues Fix context overrun when requeuing and drafting is enabled Full Changelog: v1.6.3...v1.6.4
Support BailingMoeV3ForCausalLM (Ling-3.0, 3.1) Various bugfixes Full Changelog: v1.6.2...v1.6.3
Faster CPU offloading Support TP models with DFlash2 drafters Less VRAM overhead for GDN/KDA/MLA layers ROCm fixes and faster attention on gfx11* Other bugfixes Full Changelog: v1.6.1...v1.6.2
Reduced (flat) VRAM overhead for GDN/KDA/Mamba drafting Support recurrent draft models, switch DFlash to sliding-attention (reduced VRAM overhead) New hybrid drafting mode (draft model + long-ngram) Workaround for Triton issue breaking kernel autotune on mixed-architecture setups Lots of ROCm fixes and optimizations Turing (sm_75) optimizations AVX2 optimizations Many bugfixes Full Changelog: v1.6…
Preliminary ROCm support (ROCm 10 only, gfx1100+) Faster offloading for AVX2 CPUs Smarter CPU core allocation/pinning by default Reduced system memory overhead Fix vision towers for GLM4, Qwen3-VL Other bugfixes and optimizations Full Changelog: v1.5.4...v1.6.0
Options for token-embedding quantization and disk streaming Improved quantization accuracy Update HF Transformers integration module to Transformers 5, add backprop for wrapped EXL3 layers Faster model loading Performance improvements Reduced system memory and VRAM overhead Fixes for MiMo-V2.6 Lots and lots of (latent) bugfixes Full Changelog: v1.5.3...v1.5.4
Smaller wheels Experimental n-gram corpus feature More VRAM optimizations Support dynamic draft sizing with TP model Various other fixes Full Changelog: v1.5.2...v1.5.3
Add MiMoV2ForCausalLM Experimental Turing support Rein in transient VRAM allocations, fix overly conservative warmup and autosplit-measure passes Various bugfixes and optimizations Full Changelog: v1.5.1...v1.5.2
Add KimiLinearForCausalLM Add DFlash2 TP support for Qwen3.8-Flash-Next, GLM5.3, DeepseekV3/V4 MoE optimizations Fractional-trellis quantization mode Self-calibration pipeline improvements (including highly experimental YAQA support) Fix some memory leaks Fix EXL3_MOE_PINNED_ARENA on Windows and Conda Other optimizations and bugfixes Full Changelog: v1.5.0...v1.5.1
Fix NemotronHForCausalLM (support Nemotron-3-Super) Faster quantization Experimental zero-copy pinned arena mode for CPU offload (Linux only for now) Faster MoE prefill and decode (see below) Improved speculative decoding for MoE models Faster dense model prefill on some architectures Remove some sources of nondeterminism Bugfixes and cleanup Test case 4k prefill, t/s decode, t/s, S=1 Qwen3.8-Flas…
Add DeepseekV4ForCausalLM vision tower (supports DeepseekV4-Flash-Vision-Exp) Add Lfm2ForCausalLM (LFM2.5 dense variant) Add Spark2_5ForCausalLM (Spark-X2.5) Add Glm4MoeLiteForCausalLM (GLM4.7-Flash) More accurate autosplit accounting Faster CPU offloading, better Intel support (AVX512BW tier) Fix VRAM leak after vision tower offloading Fix potential worker timeout when offloading large models to…
Add pyproject.toml Cache quantization for DSv4-Flash, Qwen3.8-Flash-Next and GLM5.3-Flash Optimize VRAM allocations and switch to expandable segments by default Fully remove all FA2 code paths Other optimizations and bugfixes Full Changelog: v1.4.7...v1.4.8
Improved CPU MoE performance for all tiers Add DRY sampler Fix recurrent slot leaks Faster quantization for 6, 7 and 8 bit trellises Many other bugfixes, optimizations and QoL improvements Full Changelog: v1.4.6...v1.4.7
Fix some regressions from v1.4.6 Add experimental n-gram streaming for Windows Full Changelog: v1.4.5...v1.4.6
Support Glm5NextForConditionalGeneration (GLM5.3-Flash) Support Qwen4ExpForConditionalGeneration (Qwen3.8-Flash-Next) Improved MoE MTP performance Slab allocation for MoE layers to avoid wasted VRAM for badly aligned expert tensor shapes Work around Triton race condition (autotuner: 'NoneType' object is not a mapping error) Fix Unigram tokenizers Other bugfixes and QoL improvements Full Changelog:…
Fix correctness bug in split MoE CPU-offload mode Better MTP support in expert-parallel mode Validate and default to vision tower quantization Support vision model offloading (streams from system memory, performance penalty is small) Full Changelog: v1.4.3...v1.4.4
Support GlmMoeDsaForCauslLM (GLM 5.2, still probably somewhat WIP) Partial CPU layer expert offloading option with dynamic placement (supersedes expert cache) Improved dynamic draft sizing with auto-calibrated confidence thresholds New (experimental) quant-optimizer pipeline Support for mid-stream text injection in generator (enables reasoning token budget) More precise autosplit allocation Reduce…
Support MuseGlimmerForConditionalGeneration Full Changelog: v1.4.1...v1.4.2
Add Mistral-4 (under Mistral3ForConditionalGeneration) Support logit bias Adopt LLGuidance instead of Formatron (still supported as optional dependency) Allow filters/grammar alongside banned_strings Full Changelog: v1.4.0...v1.4.1
Support DeepseekV4ForCausalLM Improved AVX512 CPU offload performance Faster prefill (all models) Bugfixes QoL features Full Changelog: v1.3.0...v1.4.0
Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet) Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies Fix freq/rep. penalties causing slowdown on long contexts XTC sampler and token bans Mitigate some latent vulnerabilities in Safetensors loader Other bugfixes, optimizations and QoL improvements Ful…
Faster quantization Windows: Fix JIT compilation error and broken CPU offload performance Fix tokenizer issue breaking decoding of tokens that both end and begin a multi-token character (Korean esp.) Full Changelog: v1.2.0...v1.2.1
Support LagunaForCausalLM (and DFlashLagunaForCausalLM) Experimental CPU offloading support for expert layers Experimental dynamic draft window feature Improved safetensors loader (faster) More graph paths Performance tuning Removed compare_q.py, superseded by qbench.py Various bugfixes Full Changelog: v1.1.0...v1.2.0
Support HYV3ForCausalLM Reduced CPU overhead for TP mode and improved prefill performance in native backend Adjust top-P=1 sampling to match implicit truncating behavior of torch.multinomial Banned strings now supported for recurrent models Bugfixes and QoL improvements Full Changelog: v1.0.0...v1.1.0
-> Small writeup with charts. Remove flash-attention-2 and xformers dependencies New attention kernel with online cache quantization, dual input for SWA layers and attention sinks New conv1d kernel (removes support/need for causal_conv1d) Greatly improved GEMM/GEMV performance on Ampere New INT8 GEMV kernel New MoE kernel ticket scheduler Graph path for all attn/GDN modules Fused sampling kernels…
Fix error when MTP drafting in TP mode Faster quanization Full Changelog: v0.0.42...v0.0.43
Fix MTP drafting when MTP model is not on the target model's output device Full Changelog: v0.0.41...v0.0.42
Add MTP support for Qwen3.5/3.6 Full Changelog: v0.0.40...v0.0.41